BrandonFors/pandas-github-issues huggingface dataset:
BUG: indexing changes to .loc for compat to .ix for several situations
State: closed

handle iterator
handle NamedTuple
.loc retuns scalar selection dtypes correctly, closes #11617

xref #15113


Labels:
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves
name: Dtype Conversions
description: Unexpected or buggy dtype conversions
name: Compat
description: pandas objects compatability with Numpy or Python functions

Comments:
I pulled these out of #15113 to make that a tiny bit cleaner (it will *only* be deprecations, no fixes now). and needed a tiny refactor to handle some slight differences between ``.ix`` and ``.loc``
BUG: Fix to_json lines with escaped characters
State: closed

Updates existing to_json methodology by adding is_escaping variable,
which ensures escaped chars are handled correctly.

Bug description: A simple check of whether the prior char is a backslash
is insufficient because the backslash may itself be escaped.

A test is also included (previously included in #14693).

xref #14693
xref #15096

 - [x] closes #15096 (#14693 was previously closed by author)
 - [x] tests added / passed
 - [x] passes ``git diff upstream/master | flake8 --diff`` (used ``pep8radius master --diff`` instead)
 - [x] whatsnew entry (does not seem necessary, but let me know if I should add)


Labels:
name: Bug
name: IO JSON
description: read_json, to_json, json_normalize

Comments:
## [Current coverage](https://codecov.io/gh/pandas-dev/pandas/pull/15117?src=pr) is 84.75% (diff: 100%)


> No coverage report found for **master** at ab0d236.

> Powered by [Codecov](https://codecov.io?src=pr). Last update [ab0d236...92f4e01](https://codecov.io/gh/pandas-dev/pandas/compare/ab0d23653a6e8264769b50c27b3b8efc95c6ab70...92f4e0122912cb11233a7946caf661a0026f38c1?src=pr)
pls add a whatsnew note (0.20.0 bug fix section)
Thanks, @jreback. I made the changes you requested, rebased to master, and squashed the commits.

Please note: I revised the test to include a test for escaped characters in keys and values (i.e. columns and data of dataframe). Previously I was only testing escaped characters in values/data.
lgtm, minor comments and some linting issues to fix. ping on green.
thanks! nice concise PR! keep em coming
Thanks, @jreback. I appreciate the reviews and feedback.

FYI: There is no change in runtime performance from the bug fix. This is somewhat obvious, but it was a good exercise for me, so I thought I'd share. I reran the following performance tests from #14408:

Master:
```
In [5]: df = DataFrame(dict([('float{0}'.format(i), np.random.randn(N)) for i in range(C)]))
In [6]: %timeit df.to_json('foo.json',orient='records',lines=True)
10 loops, best of 3: 135 ms per loop
```

This branch:
```
In [5]: df = DataFrame(dict([('float{0}'.format(i), np.random.randn(N)) for i in range(C)]))
In [6]: %timeit df.to_json('foo.json',orient='records',lines=True)
10 loops, best of 3: 135 ms per loop
```
@rouzazari great!

note that we already have some asv's for perf testing, I am not sure how much they cover for json though, you can have a look in pandas/asv/benchmarks/packers.py

if you think some cases need more, then please PR!
Got it. I'm not too familiar with this area, but I'll take a look.
BUG: Comparisons result in different dtypes for empty DataFrames #15077
State: closed

 - [x] closes #15077 
 - [x] tests added / passed
 - [x] passes ``git diff upstream/master | flake8 --diff``


Labels:
name: Bug
name: Dtype Conversions
description: Unexpected or buggy dtype conversions

Comments:
## [Current coverage](https://codecov.io/gh/pandas-dev/pandas/pull/15115?src=pr) is 85.67% (diff: 100%)
> Merging [#15115](https://codecov.io/gh/pandas-dev/pandas/pull/15115?src=pr) into [master](https://codecov.io/gh/pandas-dev/pandas/branch/master?src=pr) will decrease coverage by **<.01%**

```diff
@@             master     #15115   diff @@
==========================================
  Files           145        145          
  Lines         51419      51417     -2   
  Methods           0          0          
  Messages          0          0          
  Branches          0          0          
==========================================
- Hits          44057      44054     -3   
- Misses         7362       7363     +1   
  Partials          0          0          
```

> Powered by [Codecov](https://codecov.io?src=pr). Last update [97fd744...a5ca359](https://codecov.io/gh/pandas-dev/pandas/compare/97fd7447ff54d8f964d057dbaad2b5caa362e24d...a5ca35941df8576d9e8d99c95b75761c3853998b?src=pr)
Could you add the examples from #15077 as a test cases? Also a release not in whatsnew/0.20.txt. Thanks.
Yep. Is tests/frame/test_operators.py the right place for the test? 
I've added the test and a line in whatsnew/0.20.0.txt. Build test in progress.
thanks!
TST: Add another test for uneven CSV rows
State: closed

Title is self-explanatory.

xref #15066.
Closes #9549.


Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: IO CSV
description: read_csv, to_csv

Comments:
## [Current coverage](https://codecov.io/gh/pandas-dev/pandas/pull/15114?src=pr) is 84.75% (diff: 100%)
> Merging [#15114](https://codecov.io/gh/pandas-dev/pandas/pull/15114?src=pr) into [master](https://codecov.io/gh/pandas-dev/pandas/branch/master?src=pr) will not change coverage

```diff
@@             master     #15114   diff @@
==========================================
  Files           145        145          
  Lines         51232      51232          
  Methods           0          0          
  Messages          0          0          
  Branches          0          0          
==========================================
  Hits          43420      43420          
  Misses         7812       7812          
  Partials          0          0          
```

> Powered by [Codecov](https://codecov.io?src=pr). Last update [0fe491d...303c340](https://codecov.io/gh/pandas-dev/pandas/compare/0fe491db358204544814aa2371fed4ea947532ab...303c340da125b4ac690b687baf5a3071217bdaef?src=pr)
@gfyoung Is this kind of CSV currently untested?
It is, but to be safe we always include test cases from issues we close.
> It is, but to be safe we always include test cases from issues we close.

Indeed, we do, but in this case it does seem like almost exactly the same test as the one above but with different numbers? (only usage of header or not, but I don't think this should matter?)
@jorisvandenbossche : It shouldn't, but more tests couldn't hurt :smile:
@gfyoung I also changed the 'closes ..', as I would like to keep the issue open for the discussion about the `bad_lines` keyword in there. Will update the title over there
@jorisvandenbossche : I'm changing it back because #5986 and #9729 are better places to have this IMO (we should de-duplicate discussions if possible).
> more tests couldn't hurt

more tests start to hurt when we don't see the wood for the trees anymore
And changed in back again :-) 
#9729 is about too few fields, not about processing rows with too many. 
#5986 is probably a wrong link?

But maybe its better to just open a new issue about it and refer to the other ones.
@jorisvandenbossche : Meant #5686 (sorry!) 

I don't fully understand what you said there (might be the forest-trees idiom)
> I don't fully understand what you said there (might be the forest-trees idiom)

yet I looked it up before posting to be sure it was correct English :-) http://dictionary.cambridge.org/dictionary/english/not-see-the-wood-for-the-trees. With too many tests, it is more difficult to keep the overview
BUG: Fix for GH #14848 for groupby().describe() with tuples as the Index
State: closed

 - [x] closes #14848 
 - [x ] tests added / passed
 - [x ] passes ``git diff upstream/master | flake8 --diff``
 - [x ] whatsnew entry

This isn't the most elegant fix for the bug, but all the tests pass.  There is probably a bigger issue to deal with here, which is to support tuples as an `Index` (as opposed to a `MultiIndex`) throughout, but at least this fix clears the bug.

Labels:
name: Bug
name: Groupby
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves

Comments:
this is not going to be acceptable as a fix. way too hacky and special cased. I don't know where the actual issue is, but this needs more investigation.
@jreback I had a feeling you'd say that.  I will see if there is a better fix.
@Dr-Irv yeah, sorry about that....just want to find the root cause here. thanks!
## [Current coverage](https://codecov.io/gh/pandas-dev/pandas/pull/15110?src=pr) is 84.76% (diff: 100%)
> Merging [#15110](https://codecov.io/gh/pandas-dev/pandas/pull/15110?src=pr) into [master](https://codecov.io/gh/pandas-dev/pandas/branch/master?src=pr) will increase coverage by **<.01%**

```diff
@@             master     #15110   diff @@
==========================================
  Files           145        145          
  Lines         51232      51274    +42   
  Methods           0          0          
  Messages          0          0          
  Branches          0          0          
==========================================
+ Hits          43420      43460    +40   
- Misses         7812       7814     +2   
  Partials          0          0          
```

> Powered by [Codecov](https://codecov.io?src=pr). Last update [0fe491d...c18c6cb](https://codecov.io/gh/pandas-dev/pandas/compare/0fe491db358204544814aa2371fed4ea947532ab...c18c6cbc01e8b2a0fc564efdebc2a203e2201f32?src=pr)
@jreback Found the root cause, so this should be good to go.
you can prob fix it for this issue is fine 
ok, lgtm. ping on green.
thanks @Dr-Irv !
DEPR: deprecate sortlevel in favor of sort_index
State: closed

closes #14279


Labels:
name: Deprecate
description: Functionality to remove in pandas

Comments:
## [Current coverage](https://codecov.io/gh/pandas-dev/pandas/pull/15099?src=pr) is 84.75% (diff: 77.77%)
> Merging [#15099](https://codecov.io/gh/pandas-dev/pandas/pull/15099?src=pr) into [master](https://codecov.io/gh/pandas-dev/pandas/branch/master?src=pr) will decrease coverage by **<.01%**

```diff
@@             master     #15099   diff @@
==========================================
  Files           145        145          
  Lines         51220      51220          
  Methods           0          0          
  Messages          0          0          
  Branches          0          0          
==========================================
- Hits          43415      43414     -1   
- Misses         7805       7806     +1   
  Partials          0          0          
```

> Powered by [Codecov](https://codecov.io?src=pr). Last update [c71f214...c070ba8](https://codecov.io/gh/pandas-dev/pandas/compare/c71f214a7024a3a8d679f8c2eafdd4e870d7c216...c070ba83c9e2bfa847f19072af13c5414301c866?src=pr)
updated
BUG: use more generic type inference for fast plotting
State: closed

xref #15073

https://travis-ci.org/pandas-dev/pandas/jobs/190356982

was failing on tz-aware dtypes. these routines are the same in that they require datetime64ns (but any *also* accepts tz-aware).

Labels:
name: Visualization
description: plotting
name: Dtype Conversions
description: Unexpected or buggy dtype conversions

Comments:
@jorisvandenbossche 
## [Current coverage](https://codecov.io/gh/pandas-dev/pandas/pull/15094?src=pr) is 84.75% (diff: 90.90%)
> Merging [#15094](https://codecov.io/gh/pandas-dev/pandas/pull/15094?src=pr) into [master](https://codecov.io/gh/pandas-dev/pandas/branch/master?src=pr) will increase coverage by **<.01%**

```diff
@@             master     #15094   diff @@
==========================================
  Files           145        145          
  Lines         51232      51234     +2   
  Methods           0          0          
  Messages          0          0          
  Branches          0          0          
==========================================
+ Hits          43420      43423     +3   
+ Misses         7812       7811     -1   
  Partials          0          0          
```

> Powered by [Codecov](https://codecov.io?src=pr). Last update [0fe491d...c64b56b](https://codecov.io/gh/pandas-dev/pandas/compare/0fe491db358204544814aa2371fed4ea947532ab...c64b56bc1357d7036b689c5787311d0386d0a53b?src=pr)
ok, I fixed ``is_datetime64_ns_dtype`` to work with tz-aware (as _any_ does as well, this makes sense). let's see if anything else breaks. added a test as well.
ok, this is actually a tiny public API change, but it was inconsistent / wrong before.
@jorisvandenbossche 
How safe is it to change this? Couldn't there be places in our code where `is_datetime64_ns_dtype` is that assumes it has no timezone?

Just as overview, is this correct?
- `is_datetime64_dtype` -> any resolution, no tz (pure numpy)
- `is_datetime64_any_dtype` -> any resolution, with or without tz
- `is_datetime64tz_dtype` -> ns? resolution, with tz (as other resolution with tz is not possible I think?)
- `is_datetime64_ns_dtype` -> ns resolution, with or without tz

The combination missing is "ns resolution, no tz (pure numpy)", which is actually what `is_datetime64_ns_dtype` was before. But maybe we don't need this? (as you can use the tz one to check it has *no* timezone)
these functions are a bit redundant; the ``_ns_`` and ``_any_`` are barely used at all. so I don't think this is a big deal to change.
Indeed, my search says that `_any_` is exactly used in one spot, and `_ns_` in three :-). So maybe rather something to clean-up? (but not for this PR)
@jorisvandenbossche yeah I'd leave them for now, they *are* useful, even if not used :>

pushing in a sec
@jorisvandenbossche 
yes!
Pypy fixes
State: closed

benign changes visa-vis CPython needed to build pandas for PyPy 2.7

Labels:
name: Compat
description: pandas objects compatability with Numpy or Python functions

Comments:
## [Current coverage](https://codecov.io/gh/pandas-dev/pandas/pull/15080?src=pr) is 84.77% (diff: 100%)
> Merging [#15080](https://codecov.io/gh/pandas-dev/pandas/pull/15080?src=pr) into [master](https://codecov.io/gh/pandas-dev/pandas/branch/master?src=pr) will increase coverage by **<.01%**

```diff
@@             master     #15080   diff @@
==========================================
  Files           145        145          
  Lines         51212      51220     +8   
  Methods           0          0          
  Messages          0          0          
  Branches          0          0          
==========================================
+ Hits          43413      43421     +8   
  Misses         7799       7799          
  Partials          0          0          
```

> Powered by [Codecov](https://codecov.io?src=pr). Last update [42cdb34...b38d7be](https://codecov.io/gh/pandas-dev/pandas/compare/42cdb348cd3800157e04d27b13a7dd0bf52a0d2b...b38d7bee3c7fe25972dc9d3e9620fc95ac0e615b?src=pr)
commit 2e3c256 fixes a bug where substitution of ``PyString_GET_SIZE`` with ``PyUnicode_GET_SIZE`` always happened, on both python 2 and 3. It just turns out that the c-macros ``PyUnicode_GET_SIZE`` and ``PyString_GET_SIZE`` are equivalent on CPython, so the error compiled and ran as advertised. The issue manifested itself on PyPy which is more particular. [EDIT upon review and updated pull request this became changeset 91522c6, thanks for the corrections] [EDIT the suggested fix didn't work, revert to working version}
updated pull request as per reviews and suggestions
An alternative to the ``PyString_GET_SIZE`` vs. ``PyUnicode_GET_SIZE`` would be to just use ``Py_SIZE`` instead on obth python2 and python3, which is less clear but works. Any opinions?
@mattip FYI I have https://github.com/pandas-dev/pandas/pull/15101 to more generally fix the dealloc issue.
@jreback cool, #15101 makes my changes redundant. So in light of the unsatisfactory solution to the ``PyString_GET_SIZE`` vs ``PyUInicode_GET_SIZE`` problem, and #15101, I will close this pull request, and resubmit only the change to pandas/src/ujson/python/JSONtoObj.c 
Sorry for the noise
PERF: improve perf of float-based timeseries plotting
State: closed

xref #15071

`_dt_to_float_ordinal` was already vectorized, so no need to map it on the index.

So now is 'irregular' plotting even faster as period-based plotting:

```
$ asv continuous upstream/master HEAD -b TimeseriesPlotting
...
· Running 4 total benchmarks (2 commits * 1 environments * 2 benchmarks)
[  0.00%] · For pandas commit hash 4fc0dca6:
[  0.00%] ·· Benchmarking conda-py2.7-Cython-matplotlib-numexpr-numpy-openpyxl-pytables-scipy-sqlalchemy-xlrd-xlsxwriter-xlwt
[ 25.00%] ··· Running plotting.TimeseriesPlotting.time_plot_regular             126.57ms
[ 50.00%] ··· Running plotting.TimeseriesPlotting.time_plot_regular_compat      87.57ms
[ 50.00%] · For pandas commit hash de09c988:
[ 50.00%] ·· Benchmarking conda-py2.7-Cython-matplotlib-numexpr-numpy-openpyxl-pytables-scipy-sqlalchemy-xlrd-xlsxwriter-xlwt
[ 75.00%] ··· Running plotting.TimeseriesPlotting.time_plot_regular             126.01ms
[100.00%] ··· Running plotting.TimeseriesPlotting.time_plot_regular_compat      305.58ms    

      before     after       ratio
  [de09c988] [4fc0dca6]
-  305.58ms    87.57ms      0.29  plotting.TimeseriesPlotting.time_plot_regular_compat

```

Labels:
name: Visualization
description: plotting
name: Performance
description: Memory or execution speed performance

Comments:
## [Current coverage](https://codecov.io/gh/pandas-dev/pandas/pull/15073?src=pr) is 84.75% (diff: 100%)
> Merging [#15073](https://codecov.io/gh/pandas-dev/pandas/pull/15073?src=pr) into [master](https://codecov.io/gh/pandas-dev/pandas/branch/master?src=pr) will not change coverage

```diff
@@             master     #15073   diff @@
==========================================
  Files           145        145          
  Lines         51151      51151          
  Methods           0          0          
  Messages          0          0          
  Branches          0          0          
==========================================
  Hits          43355      43355          
  Misses         7796       7796          
  Partials          0          0          
```

> Powered by [Codecov](https://codecov.io?src=pr). Last update [db08da2...3971d56](https://codecov.io/gh/pandas-dev/pandas/compare/db08da2dfdb1740fdb67af3c29ab5867cd2602f8...3971d56ad0cc782a14d6d1d55d95f79e10469fb5?src=pr)
lgtm.
https://travis-ci.org/pandas-dev/pandas/jobs/190356982

I think this broke something on the slow 3.4 build ?
pathlib support in pytables
State: closed

 - [ ] closes #14705 
 - [ ] tests added / passed
 - [ ] passes ``git diff upstream/master | flake8 --diff``
 - [ ] whatsnew entry

As we talked about on gitter, I added the line (https://github.com/pandas-dev/pandas/blob/master/pandas/io/pytables.py#L266) to the HDFStore.open function. Let me know if I misunderstood or should make further edits, thank you!

Labels:
name: Bug
name: IO HDF5
description: read_hdf, HDFStore
name: Compat
description: pandas objects compatability with Numpy or Python functions

Comments:
pls add testing (look how the test r done elsewhere for pathlib)

and a whatsnew note
can you rebase / update
closing as stale, but if you'd like to add tests and update, pls ping.
add the 'name' attribute to dataframes that go through apply_frame_axis0
State: closed


 - [ ] closes #xxxx
 - [x] tests added / passed
 - [x] passes ``git diff upstream/master | flake8 --diff``
 - [ ] whatsnew entry

Previously, if you did `group.name` in the applied function, it would fail and fall back to the slower path because the attribute did not exist.

`shape_before` was unused.

Labels:
name: Groupby
name: Performance
description: Memory or execution speed performance

Comments:
## [Current coverage](https://codecov.io/gh/pandas-dev/pandas/pull/15062?src=pr) is 85.97% (diff: 100%)
> Merging [#15062](https://codecov.io/gh/pandas-dev/pandas/pull/15062?src=pr) into [master](https://codecov.io/gh/pandas-dev/pandas/branch/master?src=pr) will decrease coverage by **<.01%**

```diff
@@             master     #15062   diff @@
==========================================
  Files           140        140          
  Lines         51263      51263          
  Methods           0          0          
  Messages          0          0          
  Branches          0          0          
==========================================
- Hits          44076      44075     -1   
- Misses         7187       7188     +1   
  Partials          0          0          
```

> Powered by [Codecov](https://codecov.io?src=pr). Last update [be3f2ae...722a945](https://codecov.io/gh/pandas-dev/pandas/compare/be3f2aea4a8ad3f6bf4eb84e6ca3269abff5fcfb...722a9451af07f5da5661d97d878654285e170aef?src=pr)
looks innocuous enough (famous last words).

did you have a higher level usage that was triggering this? 
can you add an asv for this?

and add a whatsnew note (perf improvements).
ping @llllllllll 
Sorry about letting this sit idle. I will try to get the asv report and what's new on Monday.
np, ping when ready.
ping: I ran the suite locally but it was pretty noisy. I won't be run this without having other stuff running in the background until sometime tonight.
@llllllllll lgtm.

just report on perf when you can.

I also think that the ``.name`` arg is somewhat undocumented when passed to ``.apply``, if you want to add a note there (or small example), might be useful.
@llllllllll can you rebase; ping on green. i think this was ok.
from what I remember I was going to add a performance case for the asv suite. I had one locally but I actually saw a performance regression and didn't have time to dig deeper. I can rebase but I am not sure if we want to merge with the open question.
@llllllllll looks like you add an example in the asv. if you can rebase and just show/before after on that example would be fine.
thanks @llllllllll rebased and merged.