BrandonFors/pandas-github-issues huggingface dataset:
[BUG] handle } in line delimited json
State: closed

- [x] closes #14390
- [x] tests added / passed
- [x] passes `git diff upstream/master | flake8 --diff`
- [x] whatsnew entry


Labels:
name: Bug
name: IO JSON
description: read_json, to_json, json_normalize

Comments:
## [Current coverage](https://codecov.io/gh/pydata/pandas/pull/14391?src=pr) is 85.26% (diff: 100%)

> Merging [#14391](https://codecov.io/gh/pydata/pandas/pull/14391?src=pr) into [master](https://codecov.io/gh/pydata/pandas/branch/master?src=pr) will increase coverage by **<.01%**

``` diff
@@             master     #14391   diff @@
==========================================
  Files           140        140          
  Lines         50634      50639     +5   
  Methods           0          0          
  Messages          0          0          
  Branches          0          0          
==========================================
+ Hits          43173      43178     +5   
  Misses         7461       7461          
  Partials          0          0          
```

> Powered by [Codecov](https://codecov.io?src=pr). Last update [d98e982...edb1488](https://codecov.io/gh/pydata/pandas/compare/d98e982ecb0f6b7f3478a63caac34e0be238895c...edb148824da7a321c6e76abf464a6993fe7be714?src=pr)

can you add a benchmark for `lines=True` in `asv_bench/benchmarks/packers.py` (alongside the other json ones)

Not sure why that test failed,, the only change was a line of whitespace

@joshowen can you also post a run for this benchmark (versus previous); can also do it in a %timeit as well. Just checking if any perf issues.

@jreback is there an easy way to do that?  Or should I port the asv test to master and run/compare?

you _can_ run asv if you want, otherwise just do it in ipython (before and after), e.g. something like

```
In [6]: df = DataFrame(dict([('float{0}'.format(i), np.random.randn(N)) for i in range(C)]), index=index).reset_index(drop=True)

In [7]: df.to_json('foo.json',orient='records',lines=True)

In [8]: %timeit df.to_json('foo.json',orient='records',lines=True)
1 loop, best of 3: 3.66 s per loop

In [9]: %timeit df.to_json('foo.json',orient='records')
10 loops, best of 3: 98.8 ms per loop

```

and looking at this, we have a BIG perf hit when `lines=True` (this is a separate issue).

cc @aterrel 

@joshowen can you open another issue about the perf 

```
In [15]: %prun df.to_json('foo.json',orient='records',lines=True)
         100058 function calls (100056 primitive calls) in 3.759 seconds

   Ordered by: internal time

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
        1    2.017    2.017    3.656    3.656 json.py:600(_convert_to_line_delimits)
        1    1.002    1.002    1.002    1.002 {method 'join' of 'str' objects}
        1    0.630    0.630    0.630    0.630 {numpy.core.multiarray.array}
        1    0.078    0.078    0.078    0.078 {pandas.json.dumps}
        1    0.012    0.012    0.012    0.012 {method 'write' of 'file' objects}
```

something odd going on here

lgtm. @jorisvandenbossche 

Thanks @joshowen !
BUG: Allow concat to take string axis names
State: closed

Continued in #14416 

---
- [x] closes #14369 
- [x] tests added / passed
- [x] passes `git diff upstream/master | flake8 --diff`
- [x] whatsnew entry

This uses [`_get_axis_number`](https://github.com/pydata/pandas/blob/master/pandas/core/generic.py#L327-L338) to convert a string `axis` parameter to an integer.  This will allow the `concat` method to use any other aliases given to dataframes axes in the future while not disrupting other uses of `axis` in the `concat` method.  If this pattern works well, it could be used for the other [merge functions](https://github.com/pydata/pandas/blob/master/pandas/tools/merge.py) also.


Labels:
name: Bug
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode

Comments:
## [Current coverage](https://codecov.io/gh/pydata/pandas/pull/14389?src=pr) is 85.26% (diff: 100%)

> Merging [#14389](https://codecov.io/gh/pydata/pandas/pull/14389?src=pr) into [master](https://codecov.io/gh/pydata/pandas/branch/master?src=pr) will decrease coverage by **<.01%**

``` diff
@@             master     #14389   diff @@
==========================================
  Files           140        140          
  Lines         50634      50636     +2   
  Methods           0          0          
  Messages          0          0          
  Branches          0          0          
==========================================
+ Hits          43173      43174     +1   
- Misses         7461       7462     +1   
  Partials          0          0          
```

> Powered by [Codecov](https://codecov.io?src=pr). Last update [d98e982...3f08b07](https://codecov.io/gh/pydata/pandas/compare/d98e982ecb0f6b7f3478a63caac34e0be238895c...3f08b074172760b0f51d0273662621ce451c0f11?src=pr)

Thanks for the feedback!

By the way, I can squash all of these commits into one once we get everything finalized.

> By the way, I can squash all of these commits into one once we get everything finalized.

Squashing is not really needed anymore, we squash when merging anyway.

My most recent commit ( brandonmburroughs/pandas@24eb09d ) hasn't shown up. It seems like the appveyor tests never responded back to Github that it finished. Maybe that has something to do with it?  Any ideas on how to fix this and get the latest commit to show up?

Hmm, not sure why. We moved from pydata org to pandas-dev org, but I would not expect that to cause this issue. But it's maybe a reason the appveyor status is not correct (as it in fact already completed the build)
DOC: Add details to DataFrame groupby transform
State: closed

- [X] closes #13543
- [X] tests added / passed
- [X] passes `git diff upstream/master | flake8 --diff`
- [ ] whatsnew entry

Add requirements for user function in groupby transform

closes #13543
[skip ci]


Labels:
name: Docs

Comments:
@jorisvandenbossche I've taken another stab at explaining the requirements.  Also updated the requirements in the online docs.

also would like to update the doc-string with the same
@bashtage can you rebase / update
I've tried again to explain.  Maybe it would be safer to just say "Don't use f that mutate the data or make use of data in multiple columns to compute the transformation"?
@bashtage can you update
thanks @bashtage I added a templated version to handle SeriesGroupBy as well
BUG: Fix issue with inserting duplicate columns in a dataframe (GH14291)
State: closed

- [x] closes #14291
- [x] tests added / passed
- [x] passes `git diff upstream/master | flake8 --diff`
- [x] whatsnew entry

I've been sitting on this simple fix because it seems a little kludgy. SparseDataFrame doesn't do anything parallel to broadcasting in its _sanitize_column method... should it?


Labels:
name: Bug
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode

Comments:
@paul-mannino Already looking good, apart from the other comments of @bkandel :
- Travis is failing due to some lint error: `pandas/tests/frame/test_mutate_columns.py:171:80: E501 line too long (101 > 79 characters)`
- Can you add a docstring to `_sanitize_columns` where you document the new broadcast parameter? 

## [Current coverage](https://codecov.io/gh/pydata/pandas/pull/14384?src=pr) is 85.26% (diff: 100%)

> Merging [#14384](https://codecov.io/gh/pydata/pandas/pull/14384?src=pr) into [master](https://codecov.io/gh/pydata/pandas/branch/master?src=pr) will not change coverage

``` diff
@@             master     #14384   diff @@
==========================================
  Files           140        140          
  Lines         50634      50634          
  Methods           0          0          
  Messages          0          0          
  Branches          0          0          
==========================================
  Hits          43173      43173          
  Misses         7461       7461          
  Partials          0          0          
```

> Powered by [Codecov](https://codecov.io?src=pr). Last update [d98e982...a00f0fe](https://codecov.io/gh/pydata/pandas/compare/d98e982ecb0f6b7f3478a63caac34e0be238895c...a00f0fe772d29874d0d77a0bd4599e530fb251ef?src=pr)

New changes addressed all of my comments. Thanks!

@paul-mannino There is going something wrong with the PRs that were open before we transferred to another repo. Can you therefore close this one and open a new PR? (just from the same branch)
ENH: feather support in the pandas IO api
State: closed

closes #13092


Labels:
name: Enhancement
name: IO Data
description: IO issues that don't fit into a more specific label

Comments:
cc @wesm 

## [Current coverage](https://codecov.io/gh/pandas-dev/pandas/pull/14383?src=pr) is 84.78% (diff: 82.35%)
> Merging [#14383](https://codecov.io/gh/pandas-dev/pandas/pull/14383?src=pr) into [master](https://codecov.io/gh/pandas-dev/pandas/branch/master?src=pr) will increase coverage by **0.12%**

```diff
@@             master     #14383   diff @@
==========================================
  Files           144        145     +1   
  Lines         51056      51090    +34   
  Methods           0          0          
  Messages          0          0          
  Branches          0          0          
==========================================
+ Hits          43225      43316    +91   
+ Misses         7831       7774    -57   
  Partials          0          0          
```

> Powered by [Codecov](https://codecov.io?src=pr). Last update [72786cc...aa3c4b4](https://codecov.io/gh/pandas-dev/pandas/compare/72786cc8038943cc7ed11affa35fc98734c5a86c...aa3c4b45e507f38c6b402bb221fd63eeb0561bb7?src=pr)
Shall we target this for 0.20?

some of these checks can be obviated by: https://github.com/wesm/feather/pull/244 & https://github.com/wesm/feather/pull/243

this is ready to go.

@jorisvandenbossche 
@wesm 
all comments addressed (except if we want to recommend pip installs)
any final comments. @wesm ?

going to merge in the next day or 2.
ok added pip install instructions as well.
@jreback we should also add feather-format to the list of the doc building requirements (ci/requirements-2.7_DOC_BUILD.run)
yep will do
@jorisvandenbossche https://github.com/pandas-dev/pandas/commit/3e3434b21e3c135c3bd7649689d488380ea1ebc1 

FYI, has to be in the ``.sh`` script because we don't have ``conda-forge`` in the default channel list on travis; we get some weird override behavior with *some* packages, I think because of some incorrect meta-data, but these are really old and not likely to be changed. The .sh solves this problem by simply allowing arbitrary stuff for any build which needs it.

@jreback Seems to be working now! There was only still a small typo: https://github.com/pandas-dev/pandas/commit/975a440b08cce771e286de86706fa213d6479d45
@jreback @wesm A more stylistic question: Jeff now used `.fth` as the file extension, but I think I have mainly seen `.feather` (in blogposts about it). Best to be consistent here. @wesm do you prefer one of both?
We've been using `.feather`, maybe stick with that. 
OK, done: https://github.com/pandas-dev/pandas/commit/e27b29697f0dcf9359f01a19edb2f20c6d728b6c
BUG: pivot_table may raise TypeError without values
State: closed

- [x] tests added / passed
- [x] passes `git diff upstream/master | flake8 --diff`
- [x] whatsnew entry

`pivot_table` raises `TypeError` when `index` or `columns` is array-like and `values` is not specified.

```
df = pd.DataFrame({'A': [1, 2, 3, 4, 5]},
                  index=pd.DatetimeIndex(['2011-01-01', '2011-02-01', '2011-01-02', '2011-01-01', '2011-01-02']))
df.pivot_table(index=df.index.month, columns=df.index.day)
# TypeError: unhashable type: 'numpy.ndarray'
```


Labels:
name: Bug
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode

Comments:
## [Current coverage](https://codecov.io/gh/pandas-dev/pandas/pull/14380?src=pr) is 85.25% (diff: 100%)

> Merging [#14380](https://codecov.io/gh/pandas-dev/pandas/pull/14380?src=pr) into [master](https://codecov.io/gh/pandas-dev/pandas/branch/master?src=pr) will increase coverage by **<.01%**

``` diff
@@             master     #14380   diff @@
==========================================
  Files           140        140          
  Lines         50631      50636     +5   
  Methods           0          0          
  Messages          0          0          
  Branches          0          0          
==========================================
+ Hits          43166      43172     +6   
+ Misses         7465       7464     -1   
  Partials          0          0          
```

> Powered by [Codecov](https://codecov.io?src=pr). Last update [c31ea34...be426db](https://codecov.io/gh/pandas-dev/pandas/compare/c31ea341747ac40d83e64d00bac179548e4b716e...be426db01281650f24aaf38131ba970f9e084c74?src=pr)

@sinhrks I think close and open a new PR. the pydata -> pandas-dev changed caused which things.

thx for the notice. The branch status has been updated properly without re-issuing the PR. Pls take a look when you have a time.

thanks @sinhrks
BUG: Convert float freqstrs to ints at finer resolution
State: closed

- [x] closes #8419 
- [x] tests added / passed
- [x] passes `git diff upstream/master | flake8 --diff`
- [x] whatsnew entry

Passing `'0.5min'` as a frequency string should generate 30 second
intervals, rather than five minute intervals. By recursively increasing
resolution until one is found for which the frequency is an integer,
this commit ensures that that's the case for resolutions from days to
microsecond


Labels:
name: Bug
name: Frequency
description: DateOffsets

Comments:
## [Current coverage](https://codecov.io/gh/pandas-dev/pandas/pull/14378?src=pr) is 85.31% (diff: 100%)
> Merging [#14378](https://codecov.io/gh/pandas-dev/pandas/pull/14378?src=pr) into [master](https://codecov.io/gh/pandas-dev/pandas/branch/master?src=pr) will increase coverage by **<.01%**

```diff
@@             master     #14378   diff @@
==========================================
  Files           144        144          
  Lines         50957      50972    +15   
  Methods           0          0          
  Messages          0          0          
  Branches          0          0          
==========================================
+ Hits          43469      43485    +16   
+ Misses         7488       7487     -1   
  Partials          0          0          
```

> Powered by [Codecov](https://codecov.io?src=pr). Last update [86233e1...65a5163](https://codecov.io/gh/pandas-dev/pandas/compare/86233e15193c3bcd0f646915891ca6c7892335d9...65a516347afafa0f45cc266761e3ae27a8c0079f?src=pr)
can you rebase
just a little docs, otherwise lgtm.
Made the latest changes quickly myself. @jisantuc Thanks a lot for this nice contribution!
BUG: Fix linux Qt clipboard QApplication() creation
State: closed

Fixes #14372

A Qt application cannot instantiate multiple `QApplication` instances,
so we create a new `QApplication` only when the global
`QApplication.instance()` is None.

Failing sample:

```
from PyQt4.QtGui import QApplication
myapp = QApplication([])
from pandas.util.clipboard import clipboard_get  # <--- ERROR

  File "prefix/lib/python2.7/site-packages/pandas/util/clipboard.py", line 164, in <module>

    app = qt4.QtGui.QApplication([])

RuntimeError: A QApplication instance already exists.

```


Labels:
name: IO Data
description: IO issues that don't fit into a more specific label

Comments:
## [Current coverage](https://codecov.io/gh/pydata/pandas/pull/14371?src=pr) is 85.26% (diff: 100%)

> Merging [#14371](https://codecov.io/gh/pydata/pandas/pull/14371?src=pr) into [master](https://codecov.io/gh/pydata/pandas/branch/master?src=pr) will decrease coverage by **<.01%**

``` diff
@@             master     #14371   diff @@
==========================================
  Files           140        140          
  Lines         50632      50632          
  Methods           0          0          
  Messages          0          0          
  Branches          0          0          
==========================================
- Hits          43171      43170     -1   
- Misses         7461       7462     +1   
  Partials          0          0          
```

> Powered by [Codecov](https://codecov.io?src=pr). Last update [ee374ee...09eb695](https://codecov.io/gh/pydata/pandas/compare/ee374ee3bc49b82754e8269f69cf8f79d488bd5e...09eb6951db92bea2fd73562e619f92dec4bb7d05?src=pr)

I don't think any of our clipboard tests actually work on travis, xref https://github.com/pydata/pandas/pull/9693

ideally we can resolve this first.

our pyperclip has been added quite subtantially in master, so pls re-tests and re-open if needed.
Reopened on changed file as https://github.com/pandas-dev/pandas/pull/14815
DOC: add 0.19.1 whatsnew file
State: closed


Labels:
name: Docs

Comments:
## [Current coverage](https://codecov.io/gh/pydata/pandas/pull/14366?src=pr) is 85.26% (diff: 100%)

> Merging [#14366](https://codecov.io/gh/pydata/pandas/pull/14366?src=pr) into [master](https://codecov.io/gh/pydata/pandas/branch/master?src=pr) will increase coverage by **<.01%**

``` diff
@@             master     #14366   diff @@
==========================================
  Files           140        140          
  Lines         50632      50632          
  Methods           0          0          
  Messages          0          0          
  Branches          0          0          
==========================================
+ Hits          43170      43171     +1   
+ Misses         7462       7461     -1   
  Partials          0          0          
```

> Powered by [Codecov](https://codecov.io?src=pr). Last update [96b364a...cad36be](https://codecov.io/gh/pydata/pandas/compare/96b364a4a337b608c92dd6d5a00ceedd80c29315...cad36beb48c3e328171c48af5b4beee0881da715?src=pr)

Merging as other PRs need this
DOC: Remove old warning from dsintro.rst
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [ ] passes `git diff upstream/master | flake8 --diff`
- [ ] whatsnew entry

The warning is about something that have been fixed for almost 3 years. Every time a new user excited about pandas start reading the docs, they have to waste brain-cycles ignoring that big red warning bubble.


Labels:
name: Docs

Comments:
Yes, good idea! Thanks

actually could prob take out the Index warning as well (indexing.rst). Maybe also look for _older_ things like this elsewhere.

Thanks for the quick merge @jorisvandenbossche . This certainly encourages me to keep sending patches like this. :)

@jreback, absolutely. This is my first touch in the code base, so I won't take on that endeavor. But if this inspires a maintainer to do it. I would be ever so glad. :)

Yes, I think we should do a general search on version notices below 0.15 or so (also for notices that something was added in eg version 0.14 -> although maybe we could leave that for the 1.0 docs to do a big cleanup)

Also removing the "experimental" tags on some sections if they're ready

Regarding experimental, there is an issue for that: https://github.com/pydata/pandas/issues/5049
BLD/CI: cython cache pxd files
State: closed

Currently the cython cache on travis doesn't pick up change in `.pxd` files  Most of this commit history is trial and error - but 479c311 shows this working

https://travis-ci.org/pydata/pandas/jobs/166041112

```
$ ci/prep_cython_cache.sh
cython_files.tar  motd.legal-displayed  pip  pyxfiles
Cache available - checking pyx diff
util.pxd has changed:
--- /home/travis/build/pydata/pandas/pandas/src/util.pxd 2016-10-08 13:01:48.255250369 +0000 +++ /home/travis/.cache/pyxfiles/home/travis/build/pydata/pandas/pandas/src/util.pxd 2016-10-06 11:04:00.000000000 +0000 @@ -97,5 +97,6 @@ cdef inline bint _checknan(object val): return not cnp.PyArray_Check(val) and val != val + cdef inline bint is_period_object(object val): return getattr(val, '_typ', '_typ') == 'period'
In a PR
Rebuilding cythonized files
Use cache (Blank if not set) = true
Clear cache (1=YES) = 1
```

xref #14359


Labels:
name: Build
description: Library building on various platforms
name: CI
description: Continuous Integration

Comments:
## [Current coverage](https://codecov.io/gh/pydata/pandas/pull/14363?src=pr) is 85.26% (diff: 100%)

> Merging [#14363](https://codecov.io/gh/pydata/pandas/pull/14363?src=pr) into [master](https://codecov.io/gh/pydata/pandas/branch/master?src=pr) will not change coverage

``` diff
@@             master     #14363   diff @@
==========================================
  Files           140        140          
  Lines         50632      50632          
  Methods           0          0          
  Messages          0          0          
  Branches          0          0          
==========================================
  Hits          43170      43170          
  Misses         7462       7462          
  Partials          0          0          
```

> Powered by [Codecov](https://codecov.io?src=pr). Last update [96b364a...479c311](https://codecov.io/gh/pydata/pandas/compare/96b364a4a337b608c92dd6d5a00ceedd80c29315...479c311ed905b79a1e23ce5fc96f3582b00777ea?src=pr)

@jreback, @jorisvandenbossche - if you wouldn't mind taking a look

@chris-b1 Thanks!
PERF: period factorization
State: closed

Merged in #14419

---
- [x] closes #14338
- [x] tests not needed / passed
- [x] passes `git diff upstream/master | flake8 --diff`
- [x] whatsnew entry

asv 

```
   before     after       ratio
  [c41c6511] [96b364a4]
-     2.44s    46.28ms      0.02  groupby.groupby_period.time_groupby_sum
```

cc @sinhrks , @bmoscon


Labels:
name: Performance
description: Memory or execution speed performance
name: Period
description: Period data type

Comments:
## [Current coverage](https://codecov.io/gh/pydata/pandas/pull/14348?src=pr) is 85.26% (diff: 100%)

> Merging [#14348](https://codecov.io/gh/pydata/pandas/pull/14348?src=pr) into [master](https://codecov.io/gh/pydata/pandas/branch/master?src=pr) will decrease coverage by **<.01%**

``` diff
@@             master     #14348   diff @@
==========================================
  Files           140        140          
  Lines         50634      50634          
  Methods           0          0          
  Messages          0          0          
  Branches          0          0          
==========================================
- Hits          43173      43172     -1   
- Misses         7461       7462     +1   
  Partials          0          0          
```

> Powered by [Codecov](https://codecov.io?src=pr). Last update [d98e982...a77ccc2](https://codecov.io/gh/pydata/pandas/compare/d98e982ecb0f6b7f3478a63caac34e0be238895c...a77ccc21df5316a630d138d4a5afe62e3e1491f6?src=pr)

thanks for taking care of this. It looks like this  is targeted for 0.19.1; any rough idea when that will be released? 

Looks good to me. We only should include a whatsnew notice, depending on the outcome of https://github.com/pydata/pandas/issues/14352 (adding a whatsnew file to put it in in https://github.com/pydata/pandas/pull/14366)

Looks good to me! @jreback 

@chris-b1 whoops, sorry, closed this and I don't see a reopen button ..

Strange, I can't reopen either, will just make a new PR

#14419

Possibly some glitches due to the repo transfer. Older PRs are also stuck on "Checking for ability to merge automatically…" (that was the reason I wanted to close/reopen this one to retrigger it)