BrandonFors/pandas-github-issues huggingface dataset:
BUG: Check for NaN after data conversion to numeric
State: closed

In an attempt to squash a Python parser bug in which weirdly-formed floats weren't being checked for `nan`, the bug was traced back to a bug in the `maybe_convert_numeric` function of `pandas/src/inference.pyx`.  Added tests for the bug in `test_lib.py` and adjusted the original `nan` tests in `na_values.py` to test all of the engines.


Labels:
name: Missing-data
description: np.nan, pd.NaT, pd.NA, dropna, isnull, interpolate
name: Dtype Conversions
description: Unexpected or buggy dtype conversions
name: IO CSV
description: read_csv, to_csv

Comments:
## [Current coverage](https://codecov.io/gh/pydata/pandas/pull/13314?src=pr) is **84.22%**

> Merging [#13314](https://codecov.io/gh/pydata/pandas/pull/13314?src=pr) into [master](https://codecov.io/gh/pydata/pandas/branch/master?src=pr) will not change coverage

``` diff
@@             master     #13314   diff @@
==========================================
  Files           138        138          
  Lines         50667      50667          
  Methods           0          0          
  Messages          0          0          
  Branches          0          0          
==========================================
  Hits          42672      42672          
  Misses         7995       7995          
  Partials          0          0          
```

> Powered by [Codecov](https://codecov.io?src=pr). Last updated by [70be8a9...9e17995](https://codecov.io/gh/pydata/pandas/compare/70be8a95ebeb638aa53f465ba885a7b33b6c35d9...9e17995b67c9c1aa86a31803ad799154e78f2977)

I suspect `tools/tests/test_util.py` for `to_numeric` might also be affected, can you add a test there as well?

Can you run related perf tests just to see if any effect.

No need I think: `to_numeric` doesn't pass in any `NaN` values to the funxtion so this branch  has no effect.

Ran perf tests on  windows and didn't see any noticeable changes

@jreback : Made the requested changes, and Travis is giving the green light.   Ready to merge if there are no other concerns.
Fix #13306: Hour overflow in tz-aware datetime conversions.
State: closed

- [x] closes #13306
- [x] tests passed
- [x] passes `git diff upstream/master | flake8 --diff`
- [x] whatsnew entry
      Bug: tz-converting tz-aware DateTimeIndex relied on index being sorted for correct results.


Labels:
name: Bug
name: Timezones
description: Timezone data dtype

Comments:
## [Current coverage](https://codecov.io/gh/pydata/pandas/pull/13313?src=pr) is **84.22%**

> Merging [#13313](https://codecov.io/gh/pydata/pandas/pull/13313?src=pr) into [master](https://codecov.io/gh/pydata/pandas/branch/master?src=pr) will decrease coverage by **<.01%**

``` diff
@@             master     #13313   diff @@
==========================================
  Files           138        138          
  Lines         50713      50666    -47   
  Methods           0          0          
  Messages          0          0          
  Branches          0          0          
==========================================
- Hits          42715      42672    -43   
+ Misses         7998       7994     -4   
  Partials          0          0          
```

> Powered by [Codecov](https://codecov.io?src=pr). Last updated by [fcd73ad...4ca6f0c](https://codecov.io/gh/pydata/pandas/compare/fcd73ad2e7482414b61d47056c6c9c220b11702c...4ca6f0cdff56f9ba0a6ec222a93825ad0d2e572c)

how would you know this worked without tests?

this needs 2 paths 1 if is_montonic (iow is sorted); 1 if not

let me know if you need anything else from me

pls add a whatsnew note as well (Bug fix section)

ok, thanks. i'll do that stuff tonight

sure, ping when ready

can you add a whatsnew entry, otherwise lgtm.

Figured out how to add whatsnew entry :)

sigh, will have to deal with conflict tonight. cannot do it at work. it's in the v0.18.2.txt file. feel free to resolve it yourself if you don't want to wait until tonight

i'm done. take over if you're ok with it

:+1:

thanks @uwedeportivo 

yeah ideally we would check sortedness in the cython code (e.g. `is_monotonic` works for this and is fast. Though I can't produce a counterexample.
DOC: Fix read_stata docstring
State: closed

- [x] passes `git diff upstream/master | flake8 --diff`

Found docstring example is broken:
- http://pandas.pydata.org/pandas-docs/stable/generated/pandas.read_stata.html


Labels:
name: Docs
name: IO Stata
description: read_stata, to_stata

Comments:
## [Current coverage](https://codecov.io/gh/pydata/pandas/pull/13312?src=pr) is **84.22%**

> Merging [#13312](https://codecov.io/gh/pydata/pandas/pull/13312?src=pr) into [master](https://codecov.io/gh/pydata/pandas/branch/master?src=pr) will not change coverage

``` diff
@@             master     #13312   diff @@
==========================================
  Files           138        138          
  Lines         50666      50666          
  Methods           0          0          
  Messages          0          0          
  Branches          0          0          
==========================================
  Hits          42672      42672          
  Misses         7994       7994          
  Partials          0          0          
```

> Powered by [Codecov](https://codecov.io?src=pr). Last updated by [ae2ca83...a9a7357](https://codecov.io/gh/pydata/pandas/compare/ae2ca83d31ebc2f95bb4f71b504f454ec399ee41...a9a7357bf3bc9c4df9e5965324f46678a16a1e3b)
TST: Make numpy_array test strict
State: closed

- [x] tests added / passed
- [x] passes `git diff upstream/master | flake8 --diff`

2 changes to make tests more strict:
- `assert_numpy_array_equal` now checks input is `np.ndarray`
- `assert_almost_equal` now checks inputs are the same class.


Labels:
name: Testing
description: pandas testing functions or related to the test suite

Comments:
## [Current coverage](https://codecov.io/gh/pydata/pandas/pull/13311?src=pr) is **84.22%**

> Merging [#13311](https://codecov.io/gh/pydata/pandas/pull/13311?src=pr) into [master](https://codecov.io/gh/pydata/pandas/branch/master?src=pr) will decrease coverage by **<.01%**

``` diff
@@             master     #13311   diff @@
==========================================
  Files           138        138          
  Lines         50666      50667     +1   
  Methods           0          0          
  Messages          0          0          
  Branches          0          0          
==========================================
  Hits          42672      42672          
- Misses         7994       7995     +1   
  Partials          0          0          
```

> Powered by [Codecov](https://codecov.io?src=pr). Last updated by [ae2ca83...7ae4e93](https://codecov.io/gh/pydata/pandas/compare/ae2ca83d31ebc2f95bb4f71b504f454ec399ee41...7ae4e93f98a436473db50018822e43eaafe0b61c)

thanks!
DOC: remove references to deprecated numpy negation method
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [x] passes `git diff upstream/master | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: Docs

Comments:
## [Current coverage](https://codecov.io/gh/pydata/pandas/pull/13310?src=pr) is **84.22%**

> Merging [#13310](https://codecov.io/gh/pydata/pandas/pull/13310?src=pr) into [master](https://codecov.io/gh/pydata/pandas/branch/master?src=pr) will not change coverage

``` diff
@@             master     #13310   diff @@
==========================================
  Files           138        138          
  Lines         50666      50666          
  Methods           0          0          
  Messages          0          0          
  Branches          0          0          
==========================================
  Hits          42672      42672          
  Misses         7994       7994          
  Partials          0          0          
```

> Powered by [Codecov](https://codecov.io?src=pr). Last updated by [ae2ca83...3b46a1b](https://codecov.io/gh/pydata/pandas/compare/ae2ca83d31ebc2f95bb4f71b504f454ec399ee41...3b46a1b847013b0429bc6dcd2c58ea175c23b5cf)

thanks!
ENH: Respect Key Ordering for OrderedDict List in DataFrame Init
State: closed

Title is self-explanatory.  Closes #13304.


Labels:
name: Enhancement
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode

Comments:
## [Current coverage](https://codecov.io/gh/pydata/pandas/pull/13309?src=pr) is **84.22%**

> Merging [#13309](https://codecov.io/gh/pydata/pandas/pull/13309?src=pr) into [master](https://codecov.io/gh/pydata/pandas/branch/master?src=pr) will increase coverage by **<.01%**

``` diff
@@             master     #13309   diff @@
==========================================
  Files           138        138          
  Lines         50681      50666    -15   
  Methods           0          0          
  Messages          0          0          
  Branches          0          0          
==========================================
- Hits          42682      42672    -10   
+ Misses         7999       7994     -5   
  Partials          0          0          
```

> Powered by [Codecov](https://codecov.io?src=pr). Last updated by [9e7bfdd...87eba3d](https://codecov.io/gh/pydata/pandas/compare/9e7bfdd5996d90c774cf6ebd39c6c0779469e545...87eba3df464effdf3f6afaa16201bc208c535823)

@jreback : Travis is giving the green light.  Ready to merge if there are no other concerns.

ty sir!
Fix series comparison operators when dealing with zero rank numpy arrays
State: closed

- [x] closes #13006
- [x] tests added / passed
- [x] passes `git diff upstream/master | flake8 --diff`
- [x] whatsnew entry


Labels:
name: Bug
name: Dtype Conversions
description: Unexpected or buggy dtype conversions

Comments:
http://stackoverflow.com/questions/4565749/how-detect-length-of-a-numpy-array-with-only-one-element#4565910

Partially based on @sldion 's changes. Thanks

## [Current coverage](https://codecov.io/gh/pydata/pandas/pull/13307?src=pr) is **84.22%**

> Merging [#13307](https://codecov.io/gh/pydata/pandas/pull/13307?src=pr) into [master](https://codecov.io/gh/pydata/pandas/branch/master?src=pr) will decrease coverage by **<.01%**

``` diff
@@             master     #13307   diff @@
==========================================
  Files           138        138          
  Lines         50721      50667    -54   
  Methods           0          0          
  Messages          0          0          
  Branches          0          0          
==========================================
- Hits          42723      42673    -50   
+ Misses         7998       7994     -4   
  Partials          0          0          
```

> Powered by [Codecov](https://codecov.io?src=pr). Last updated by [0c6226c...b49a116](https://codecov.io/gh/pydata/pandas/compare/0c6226cbbc319ec22cf4c957bdcc055eaa7aea99...b49a11685293cd98616c55160675a01be7c2343b)

@jreback Are you good with this PR? Thanks

build is green

thanks @gliptak
BUG: Fix describe(): percentiles (#13104), col index (#13288)
State: closed

- [x] closes #13104, #13288
- [x] tests added / passed
- [x] passes `git diff upstream/master | flake8 --diff`
- [x] whatsnew entry

BUG #13104:
- Percentiles are now rounded to the least precision that keeps
  them unique.
- Supplying duplicates in percentiles will raise ValueError.

BUG #13288
- Fixed a column index of the output data frame.
  Previously, if a data frame had a column index of object type and
  the index contained numeric values, the output column index could
  be corrupt. It led to ValueError if the output was displayed.


Labels:
name: Bug
name: Output-Formatting
description: __repr__ of pandas objects, to_string

Comments:
@pijucha looks really good. just a couple of comments.

## [Current coverage](https://codecov.io/gh/pydata/pandas/pull/13298?src=pr) is **84.23%**

> Merging [#13298](https://codecov.io/gh/pydata/pandas/pull/13298?src=pr) into [master](https://codecov.io/gh/pydata/pandas/branch/master?src=pr) will increase coverage by **<.01%**

``` diff
@@             master     #13298   diff @@
==========================================
  Files           138        138          
  Lines         50681      50681          
  Methods           0          0          
  Messages          0          0          
  Branches          0          0          
==========================================
+ Hits          42682      42687     +5   
+ Misses         7999       7994     -5   
  Partials          0          0          
```

> Powered by [Codecov](https://codecov.io?src=pr). Last updated by [9e7bfdd...39c0676](https://codecov.io/gh/pydata/pandas/compare/9e7bfdd5996d90c774cf6ebd39c6c0779469e545...39c067629998f79184b87330a2ba7ba8ee861782)

@pijucha looks good. just some more validation tests and some doc changes.

@jreback  I've made the changes. (Tests still pending.)

thanks @pijucha nice PR! I slightly edited the docs so take a look when built.

@jreback Thanks
DOC: low_memory in read_csv
State: closed

- [x] closes #5888,  xref #12686
- [x] passes `git diff upstream/master | flake8 --diff`


Labels:
name: Docs
name: IO CSV
description: read_csv, to_csv

Comments:
## [Current coverage](https://codecov.io/gh/pydata/pandas/pull/13293?src=pr) is **84.20%**

> Merging [#13293](https://codecov.io/gh/pydata/pandas/pull/13293?src=pr) into [master](https://codecov.io/gh/pydata/pandas/branch/master?src=pr) will not change coverage

``` diff
@@             master     #13293   diff @@
==========================================
  Files           138        138          
  Lines         50587      50587          
  Methods           0          0          
  Messages          0          0          
  Branches          0          0          
==========================================
  Hits          42593      42593          
  Misses         7994       7994          
  Partials          0          0          
```

> Powered by [Codecov](https://codecov.io?src=pr). Last updated by [b4e2d34...282f1f3](https://codecov.io/gh/pydata/pandas/compare/b4e2d34edcbc404f6c90f76b67bcc5fe26f0945f...282f1f38cdfdcf9fa4d47e46ba2354d6e800b0a4)

@jreback - updated

thanks!
COMPAT: extension dtypes (DatetimeTZ, Categorical) are now Singleton cached objects
State: closed

allows for proper is / == comparisons
Had this odd semantic difference as these were really different objects (though they DID hash the same)
This doesn't actually affect any user code.

```
In [1]: from pandas.core import common as com

In [2]: t1 = com.DatetimeTZDtype('datetime64[ns, US/Eastern]')

In [3]: t2 = com.DatetimeTZDtype('datetime64[ns, US/Eastern]')

In [4]: t1 == t2
Out[4]: True

In [5]: t1 is t2
Out[5]: False

In [6]: hash(t1)
Out[6]: 5756291921003024619

In [7]: hash(t2)
Out[7]: 5756291921003024619
```


Labels:
name: Bug
name: Compat
description: pandas objects compatability with Numpy or Python functions

Comments:
## [Current coverage](https://codecov.io/gh/pydata/pandas/pull/13285?src=pr) is **84.20%**

> Merging [#13285](https://codecov.io/gh/pydata/pandas/pull/13285?src=pr) into [master](https://codecov.io/gh/pydata/pandas/branch/master?src=pr) will increase coverage by **<.01%**

``` diff
@@             master     #13285   diff @@
==========================================
  Files           138        138          
  Lines         50591      50607    +16   
  Methods           0          0          
  Messages          0          0          
  Branches          0          0          
==========================================
+ Hits          42592      42613    +21   
+ Misses         7999       7994     -5   
  Partials          0          0          
```

> Powered by [Codecov](https://codecov.io?src=pr). Last updated by [8749273...fd09c61](https://codecov.io/gh/pydata/pandas/compare/87492737f2f81183700849d453c38d507f128811...fd09c61439ecc8f4e36cfa1c0d7e5f09253f9c2c)
Remove imp and just use importlib to avoid memory error when importin…
State: closed

- [ ] closes #13282

Remove imp from pandas.show_versions() to fix memory problem.


Labels:
name: Build
description: Library building on various platforms

Comments:
hmm, `importlib` is only available in PY3 I think.

can you hack something from this:
https://pypi.python.org/pypi/importlib (don't want to add as a dep)

`importlib` is in 2.7, does pandas still support 2.6?

oh, then that's ok

ok this looks good. ping when green.

thanks @nparley
Change get_dummies() to return columns of dtype=bool instead of np.float64
State: closed

- closes #8725


Labels:
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode
name: Dtype Conversions
description: Unexpected or buggy dtype conversions
name: Categorical
description: Categorical Data Type

Comments:
tests! 

further this needs careful examination to see if you are copying things. 

needs tests
DOC: Added an example of pitfalls when using astype
State: closed

- [x] closes #13260 
- [ ] tests added / passed
- [ ] passes `git diff upstream/master | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: Docs

Comments:
@pfrcks Thanks for the contribution!

Personally, I would reverse the logic in the explanation, by first saying how you should do it, and only then what does not work.

Eg in the sense of "When you want to astype subset of columns -> you have to do it like this: ... -> attention, you cannot use loc in this case like ... as this upcasts"

@jorisvandenbossche I have made the cleanup changes you recommended. Do I need to change the order of the wrong and right way of doing things?
Wrong followed by right way of doing things also sends a clear message I guess. We see how the thing is wrong, then we see how to correct or implement it properly.

Restructured the documentation.

@jorisvandenbossche Made the suggested changes. Hope it is fine now.

Only a small last comment, for the rest looks good!

@jorisvandenbossche I hope that does the trick ;)

@pfrcks Thanks!
DOC: Added additional example for groupby by indexer.
State: closed

- [x] closes #13271 
- [ ] tests added / passed
- [ ] passes `git diff upstream/master | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: Docs

Comments:
Added more documentation and examples to clarify resampling as whole.

@pfrcks Added a few more comments.

@jorisvandenbossche kindly go through the changes and comment.

## [Current coverage](https://codecov.io/gh/pydata/pandas/pull/13276?src=pr) is **84.33%**

> Merging [#13276](https://codecov.io/gh/pydata/pandas/pull/13276?src=pr) into [master](https://codecov.io/gh/pydata/pandas/branch/master?src=pr) will increase coverage by **0.15%**

``` diff
@@             master     #13276   diff @@
==========================================
  Files           138        138          
  Lines         50581      51107   +526   
  Methods           0          0          
  Messages          0          0          
  Branches          0          0          
==========================================
+ Hits          42582      43103   +521   
- Misses         7999       8004     +5   
  Partials          0          0          
```

> Powered by [Codecov](https://codecov.io?src=pr). Last updated by [e0a2e3b...39b7fac](https://codecov.io/gh/pydata/pandas/compare/e0a2e3bc51f9e178f72c44e6de06700ee0bf31c6...39b7fac3cdb5ce6c1f81823012c759ee2acd01b9)

@jorisvandenbossche Sorry, I overlooked that. Have made the necessary changes. Please look and comment.

@pfrcks Thanks!
BUG: Properly validate and parse nrows in read_csv
State: closed

1) Allows `float` values for `nrows` for the Python engine
2) Prevents abuse of the `nrows` argument for the CParser (e.g. you can passing `nrows=1.2`)

Closes #10476.


Labels:
name: Error Reporting
description: Incorrect or improved errors from pandas
name: IO CSV
description: read_csv, to_csv

Comments:
@jreback : Made the requested changes, and Travis is still giving the green light.  Ready to merge if there is nothing else.

short and sweet! nice! thanks @gfyoung
BUG, ENH: Improve infinity parsing for read_csv
State: closed

1) Allow mixed-case infinity strings for the Python engine

Bug was traced back via `lib.maybe_convert_numeric` to the `floatify` function in `pandas/src/parse_helper.h`.  In addition to correcting the bug and adding tests for it, this PR also moves the `test_inf_parsing` test from `c_parser_only.py` to `common.py` in the `pandas/io/tests/parser` dir.

2) Interpret `+inf` as positive infinity for both engines

`float('+inf')` in Python is interpreted as positive infinity, so we should allow it too in parsing.


Labels:
name: Bug
name: IO CSV
description: read_csv, to_csv

Comments:
ok looks reasonable. pls run parser asv's just to be sure (I doubt we have something that specifically has infs in it; you can add if you want).

btw, not averse to signed inf (e.g. `+inf`), if its in the IEEE spec (not sure)

1) Not sure if it's necessary to add a benchmark for this.  A couple of string checks shouldn't have much effect, so unless there is a strong objection, I'll leave it for now.

2) I can't actually run benchmarks at the moment.  There seems to be an issue with linux-64 channels in Anaconda for `tables` and `libpython` that prevents me from running the benchmarks.  Can you confirm on a different architecture?

3) Can't confirm if `+inf` is in IEEE standard (Googled "IEEE infinity," and that didn't turn up anything).  _However_, `float('+inf')` does work in Python, so I guess we should accept it.  However, it is not supported in the CParser either right now, so I'll need to see where infinity parsing is done there.

pls just run the current benchmarks - don't need a new one

you don't need libpython on Linux

I'm just running `asv continuous master floatify-infinity-parsing -b io_bench`, and that's what it's complaining about, nothing special here, right?

@jreback : Added `+inf` parsing to both parsers, and Travis is still giving the green light.  While I cannot run the benchmarks because of that Anaconda issue, I wouldn't be surprised if they remain relatively the same since my changes shouldn't down anything too much.  Ready to merge if there are no other concerns.

lgtm, ping on green. we don't have an associated issue right?

Nope.  Discovered it when I was refactoring `test_parsers`, and someone had inadvertently put `read_csv` instead of `self.read_csv` for that test I moved in this PR.

@jreback : Travis giving the green light .  Ready to merge if there is nothing else.

nice PR @gfyoung 

only comment is that don't put Bug Fixes at the very end, causes conflicts, stick them somewhat randomly in the spaces  within the current list (that's why I leave the space)

@jreback : Ah, fair point.  Bad habit on my part.  Thanks for the reminder!
ENH: support decimal argument in read_html #12907
State: closed

- [x] closes #12907 
- [x] tests added / passed
- [x] passes `git diff upstream/master | flake8 --diff`
- [x] whatsnew entry


Labels:
name: Enhancement
name: IO HTML
description: read_html, to_html, Styler.apply, Styler.applymap

Comments:
@jreback please review latest commit

this looks good.

Do you have a site that this would work on? (I always like to manually test things as well)

@jreback I have generated some data here: http://www.googledrive.com/host/0BzJmIC7_SNINRGxSUlhaT2hBeEk

It should work with:
`df1 = pandas.read_html(url, decimal=',', thousands='.', header=1)`

thanks @camilocot great!
TST: assert_dict_equal to check input type
State: closed

- [x] tests added / passed
- [x] passes `git diff upstream/master | flake8 --diff`

`assert_dict_equal` now checks input is `dict` instance. Fixed some tests which passes `Series` to it.


Labels:
name: Testing
description: pandas testing functions or related to the test suite

Comments:
lgtm, merge on pass.

thanks @sinhrks 

I added a couple of unrelated fixes to correct some numpy warnings.