BrandonFors/pandas-github-issues huggingface dataset:
BUG: to_datetime with unit with Int64
State: closed

- [X] closes #30050
- [X] tests added 1 / passed 1
- [X] passes `black pandas`
- [X] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [X] whatsnew entry

Test output:
```
$ py.test pandas/tests/tools/test_datetime.py
============================= test session starts =============================
platform win32 -- Python 3.7.3, pytest-5.3.1, py-1.8.0, pluggy-0.13.0
rootdir: C:\git_contrib\pandas\pandas, inifile: setup.cfg
plugins: hypothesis-4.50.6, cov-2.8.1, forked-1.1.2, xdist-1.30.0
collected 1 item

pandas\tests\tools\test_datetime.py .                                    [100%]

============================== 1 passed in 0.03s ==============================

```

Notes: Hello. The fix introduces one additional type check if the Series passed to `to_datetime` stores anything except and IntegerArray. It does nothing else in this case. If we try to pass an IntegerArray, it pulls all the values that aren't NaN into an ndarray, converts that into datetime and adds NaT in the places where NaNs were. No precision is lost.

Labels:
name: Datetime
description: Datetime data dtype
name: ExtensionArray
description: Extending pandas with custom dtypes or arrays.

Comments:
@AlexKirko the diff is super confusing because you are pulling on other commits, pls merge in master and repush
> @AlexKirko the diff is super confusing because you are pulling on other commits, pls merge in master and repush

@jreback Apologies for the mess. Still learning how to contribute properly. Merged master in.
@jreback Conflicts in whatsnew removed. Should be ready for another review.
@jreback Merged in master again to remove new conflicts in `datetime.py`
Moved the nan-handling to tslib. Web_and_Docs PR test is broken for now (# 30527), but other than that, things look ready for a review.

@jreback , did you have something like this solution in mind?
@jreback Merged in master to fix Webs_and_Docs PR test bug.
@jreback Addressed all the comments, please take a look.
Hello @AlexKirko! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:





There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2020-01-01 08:03:42 UTC
@jreback @TomAugspurger Added to the test and removed unnecessary `getattr`. Awaiting the decision on where we should implement NaN-handling: in `tslib` or in `datetimes`. I'm for `tslib` (current solution) as we merely expand on the way NaNs are handled and that's, in my opinion, is better than splitting and re-merging arg in `_convert_listlike_datetimes`.
@jreback rebased to updated master, squashed tiny commits, made the needed changes and commented on green
Thanks @AlexKirko!
@jreback @TomAugspurger Thank you! Learned a lot about the process: it should help a lot with the next PR.
API: Return BoolArray for string ops when backed by StringArray
State: closed

ref https://github.com/pandas-dev/pandas/issues/29556

Labels:
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode
name: Dtype Conversions
description: Unexpected or buggy dtype conversions
name: API Design
name: ExtensionArray
description: Extending pandas with custom dtypes or arrays.

Comments:
Do we want to optimize the BooleanArray first before returning it from methods like this? I think could have non-trivial memory impacts doing this right now
Recall this is just for StringDtype, not object-dtype backed Series. So there's no harm to the current users.

And improving performance later is much easier than breaking API.
Planning to merge this in a few hours.
> I'm going to hold off on this. I'm planning to restructure the docs for integer / boolean / NA once all these PRs are in.

What's your current plan? 
In one of the PRs we discussed a little about this (gather them all on a single page vs additional nesting), but don't know where any more.
DOC: Add MultiIndex.get_locs to generated reference
State: closed

This should fix the failing CI, see https://github.com/pandas-dev/pandas/pull/30226

CLoses #30235 

Labels:

Comments:
@jorisvandenbossche I pushed some formatting changes to get this passing.
Just the 32-bit failure being handled elsewhere. Will merge when travis passes.
Things should be good on master once this is merged, thanks to #30237.
Thanks for the fixup!
CI: restore npdev build
State: closed


Labels:
name: CI
description: Continuous Integration

Comments:
So something was reverted on the numpy side for this right? Nothing required yet on our end?
> So something was reverted on the numpy side for this right? Nothing required yet on our end?

I think they reverted the implicit-object warning.

Also I think npdev is now 1.19, so we may need a separate 1.18 build
Thanks!
StringArray comparisions return BooleanArray
State: closed

xref https://github.com/pandas-dev/pandas/issues/29556

cc @jorisvandenbossche 

Labels:
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode
name: API Design
name: ExtensionArray
description: Extending pandas with custom dtypes or arrays.

Comments:
Thanks! Fixed all those I think.

Tangentially related: we'll need a docs section discussing the behavior of `pd.NA` and arrays using `pd.NA` in comparison operations (propagate rather than unequal). I think it makes sense to combine `user_guide/integer_na.rst` and `user_guide/boolean.rst` into a single "nullable types" document which discusses all of this? What do you think?
> I think it makes sense to combine user_guide/integer_na.rst and user_guide/boolean.rst into a single "nullable types" document which discusses all of this? What do you think?

I was floating the idea in my head (but didn't make an issue or so yet) to create a "user_guide/data_types" directory and move the integer and boolean files there (so an extra nesting in the left sidebar). And then the main "data types" page could have a short intro to extension dtypes, pointing to those pages. 
But, if the content is relatively limited, it might also make sense to just combine both on a single page for now.


> I don't think you are really testing this. I think you actually need to take some of our existing indexing tests,

This does not yet add indexing, this only changes the comparison operations. Indexing is one of the next steps on the list in https://github.com/pandas-dev/pandas/issues/29556
> > I don't think you are really testing this. I think you actually need to take some of our existing indexing tests,
> 
> This does not yet add indexing, this only changes the comparison operations. Indexing is one of the next steps on the list in #29556

ok that's fair. 

would at like to at least in this PR put into place an xfail test that assures that if we try to do indexing with a BooleanArray it fails
Indexing seems independent right? BooleanArray comparisons already return BooleanArray, and AFAIK those aren't tested in indexing yet.

I'm happy to write those tests / work on the implementation, but it feels like a separate piece of work.
Yes, let's keep it separate
> Indexing seems independent right? BooleanArray comparisons already return BooleanArray, and AFAIK those aren't tested in indexing yet.
> 
> I'm happy to write those tests / work on the implementation, but it feels like a separate piece of work.

they are separate until sometries to assign a BooleanArray to a column and use it. We have *nothing* for this, certainly do in a separate PR. What I am suggesting is a single test to validate that this raises (for now)
Do we know what the defined behavior is yet? In https://github.com/pandas-dev/pandas/issues/28778 it sounds like we're contemplating either raising or propagating.
This should be good to go.
Thanks!
CLN: replace str.format() in pandas/io/formats
State: closed

- [x] ref #29547 and ref #30020 
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: Code Style
description: Code style, linting, code_checks

Comments:
Merging master should fix the CI issues.
Groupby crash on a single level
State: closed

- [/] closes #30208
- [/] tests added / passed
- [/] passes `black pandas`
- [/] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry


Labels:
name: Bug
name: Groupby

Comments:
Hello @nlepleux! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:





There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2019-12-13 15:23:08 UTC
It is possible to restart the CI process? The git checkout failed on windows
all green
Great thanks @nlepleux
BUG: min/max on empty categorical fails
State: closed

- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry

Currently on master, the following fails:
```python
import pandas as pd

# Empty Dataframe
df = pd.DataFrame({
    "cat": pd.Categorical([], categories=list("ABC"), ordered=True),
    "val": pd.Series([], dtype="int64")
})

df["cat"].max()
```
`ValueError: zero-size array to reduction operation maximum which has no identity`

Other dtypes return `NaN` when calling `min()` or `max()` on an empty Series. This PR changes the behavior of Categoricals to be the same. 

Note that current behavior causes a downstream bug in Dask here: https://github.com/dask/dask/issues/5645, so this PR could fix that bug as well.

Labels:
name: Bug
name: Categorical
description: Categorical Data Type

Comments:
Thanks!
DOC: Documentation for MultiIndex.get_locs() is missing in v0.25 docs
State: closed

Like a comment in #30056. I add "get_locs" to method

- [ ] closes #30056
- [ ] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: Docs

Comments:
Sorry, I totally missed the failing CI here, will fix
BUG: loc-indexing with a slice on a CategoricalIndex
State: closed

- [x] followup to #29922
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry

#29922 didn't handle loc-indexing using a slice on non-string CategoricalIndexes. This fixes that. E.g. we had:

```python
>>> cat_idx = pd.CategoricalIndex([1,2,3], ordered=True)
>>> df = pd.DataFrame({"A": ["foo", "bar", "baz"]}, index=cat_idx)
>>> df.loc[1:2, "A"]
TypeError: cannot do slice indexing on...
```

The above works now.

Labels:
name: Bug
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves
name: Categorical
description: Categorical Data Type

Comments:
thanks @topper-123 very nice
CLN: changed .format to f string in construction.py, dtypes/base.py, and dtypes/cast.py
State: closed

- [X] ref [#29547](https://github.com/pandas-dev/pandas/issues/29547)
- [X] tests added / passed
- [X] passes `black pandas`
- [X] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: Strings
description: String extension data type and string data
name: Code Style
description: Code style, linting, code_checks

Comments:
thanks @katieyounglove
TST: tests for needs-test issues
State: closed

- [x] closes #5798
- [x] closes #6795
- [x] closes #9752
- [x] closes #11220
- [x] closes #13000
- [x] closes #15286
- [x] closes #17198
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: Testing
description: pandas testing functions or related to the test suite

Comments:
ping
Thanks @jbrockmendel great work on this
thanks @jbrockmendel if you can parameterize as indicated would be great
STY: Replace .format() to fstrings and %r to repr()
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry

#29886 

Labels:
name: Code Style
description: Code style, linting, code_checks

Comments:
Hello @EydenVillanueva! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:











There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2019-12-13 16:27:58 UTC
Some test failures, e.g. 

```pytb
self = <pandas.tests.test_nanops.TestEnsureNumeric object at 0x14d026208>

    def test_non_convertable_values(self):
        msg = "Could not convert foo to numeric"
        with pytest.raises(TypeError, match=msg):
>           nanops._ensure_numeric("foo")
E           AssertionError: Pattern 'Could not convert foo to numeric' not found in "Could not convert 'foo' to numeric"

```

Can you check on those?
> Some test failures, e.g.
> 
> ```
> self = <pandas.tests.test_nanops.TestEnsureNumeric object at 0x14d026208>
> 
>     def test_non_convertable_values(self):
>         msg = "Could not convert foo to numeric"
>         with pytest.raises(TypeError, match=msg):
> >           nanops._ensure_numeric("foo")
> E           AssertionError: Pattern 'Could not convert foo to numeric' not found in "Could not convert 'foo' to numeric"
> ```
> 
> Can you check on those?

Sure.
@EydenVillanueva I think a mistake on merging here. Can you rollback to your change, merge master and re-push? Shouldn't need to force push if you go that route
> @EydenVillanueva I think a mistake on merging here. Can you rollback to your change, merge master and re-push? Shouldn't need to force push if you go that route

Hi, something went wrong not sure why, to clean up a little bit I'll close this PR, and make another one without all this mess, haha thank you.
[Doc Fix] Fix typo from Jupyter Notebook to Jupyter Lab in Ecosystem Doc.
State: closed

I saw this typo in the [doc](https://pandas.pydata.org/pandas-docs/stable/ecosystem.html#jupyter-notebook-jupyter-lab).
Decided to create a PR for it. ;)

Labels:
name: Docs

Comments:
@junwoo091400 can you address comment by @gfyoung?  this should be almost ready
Closing as stale, @junwoo091400 if you want to finish this please let us know, and we'll reopen.
Parallelized Build / CI
State: closed

Modeled after https://github.com/cython/cython/pull/3187 which in 0.29.14 added parallel support

A very slight boost to timing from a clean directory (bottleneck is in the Cythonize calls in both cases)

```sh
# Master
$ time python setup.py build_ext --inplace -j 4 : 
62.93s user 3.18s system 97% cpu 1:07.77 total

# this branch
$ time python setup.py build_ext --inplace -j 4:
58.87s user 3.32s system 105% cpu 59.030 total
```



Labels:
name: Build
description: Library building on various platforms

Comments:
LGTM
I think could also reduce the cythonize bottleneck if using the `nthreads` argument:

https://cython.readthedocs.io/en/latest/src/userguide/source_files_and_compilation.html#Cython.Build.cythonize

Locally that yielded this for a clean build:

```sh
69.07s user 3.75s system 334% cpu 21.783 total
```

So 21 seconds from scratch isn't bad; might help a lot to reduce time to run ASVs

that would be a nice speedup.  im not clear on what the difference is between nthreads vs "-j"
ok, it looks like passing `nthreads` to `cythonize` affects the cython->C compilation, and the "-j 4" affects the C->so compilation
Open to suggestions on where to place this, but reusing the argument for parallel compliation in the calls to Cythonize helps a lot. Updated benchmarks:

```sh
$ time python setup.py build_ext --inplace --parallel=4
70.69s user 3.88s system 311% cpu 23.951 total
$ time python setup.py build_ext --inplace -j4
69.88s user 3.80s system 322% cpu 22.817 total
```
I'd roughly say this saves about 4-5 minutes on CI where it works. Have to see what's going on with Linux 36

Also not sure if Windows will really support this, at least not without some upstream changes. Earlier attempts throw a warning about multiprocessing `freeze_support`; put on the back burner for now
the CI errors all look unrelated, just started re-run
lgtm thanks @WillAyd 
I think this might be responsible for some 36 errors on the 32 bit build with environment setup this morning. I'll revert at least the CI piece so we can get that green
ENH: Support string arguments for partition_cols in pandas.to_parquet
State: closed

- [x] closes #27117
- [x] tests added
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry


Labels:
name: IO Parquet
description: parquet, feather

Comments:
lgtm. @jorisvandenbossche or @TomAugspurger if any comments.
@HawkinsBA Thanks!
Fix IntegerArray pow for special cases
State: closed

Split from #29964 

Only https://github.com/pandas-dev/pandas/commit/9e5a69c3a32aad746ef3df29e43df352f4551f4d is relevant. The rest is https://github.com/pandas-dev/pandas/pull/30183, but pushing this up now.

Labels:
name: Bug
name: Dtype Conversions
description: Unexpected or buggy dtype conversions

Comments:
With #30183 merged, I think this needs a rebase to facilitate review and to check if the changes are still good from https://github.com/pandas-dev/pandas/commit/9e5a69c3a32aad746ef3df29e43df352f4551f4d.
My merge in https://github.com/pandas-dev/pandas/pull/30210/commits/b5a4112b16b17d7c89a3f36381bde0c8f240eed0 also includes https://github.com/pandas-dev/pandas/pull/30183
FYI @jorisvandenbossche this is slightly blocking https://github.com/pandas-dev/pandas/pull/29964. It'll make the changes there more focused.
@jorisvandenbossche good to go here?
Cleaned Up setup.py includes
State: closed

Looking to make these more explicit and remove where not required

Labels:
name: Clean

Comments:
Hello @WillAyd! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:



There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2019-12-12 18:28:45 UTC
should rebase this after we merge #30112 
LGTM
needs rebase
DEPR: remove deprecated verbose, private_key kwargs for gbq
State: closed

The deprecation didn't have a whatsnew note or actually use deprecate_kwarg, so im not sure how formal the removal should be.

xref #20564.

Labels:
name: Deprecate
description: Functionality to remove in pandas

Comments:
need to modify these tests too: > test_read_gbq_with_deprecated_kwargs
removed test, added a whatsnew note that didnt make it into #30199
IIRC I remember seeing something about ``private_key`` still in pandas-gbq? @tswast 
That's right. I filed https://github.com/pydata/pandas-gbq/issues/301 to actually remove it. I'll do that and cut a pandas-gbq release today.
so should i revert the private_key change here?  warn?
It's okay to remove. We've been showing a warning since we deprecated it in pandas 0.24.0. Soon it'll hard fail if it's ever supplied.
Thanks @jbrockmendel
STY - fstrings on frame.py & numeric.py
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [x] passes `black pandas`
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry

#https://github.com/pandas-dev/pandas/issues/29886 

Labels:
name: Code Style
description: Code style, linting, code_checks

Comments:
sure
TST: Split test_offsets.py
State: closed

- [x] closes #27085
- [ ] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: Testing
description: pandas testing functions or related to the test suite

Comments:
@Raalsky can you rebase
closing as stale. @Raalsky would love a PR that first moves, then followups to split / parameterize this. make small targeted PRs; we can then merge quickly.
Sorry for being off during the holidays. I've prepared the first step (move) test_offsets to test_date_offsets #31031 Will be following by adding fixtures and then business/date split.
Fixtures added in #31034
REF: dont alter state in pytables read_axes
State: closed


Labels:
name: IO HDF5
description: read_hdf, HDFStore

Comments:
rebased+green
Not sure whats going on here
```
/home/vsts/work/1/s/pandas/core/indexes/multi.py:docstring of pandas.MultiIndex.rst:91: WARNING: autosummary: stub file not found 'pandas.MultiIndex.get_locs'. Check your autosummary_generate setting.
```
thanks
Fixed IntegerArray division by 0
State: closed

When dividing by 0, the result should be `inf`, not `NaN`.

closes https://github.com/pandas-dev/pandas/issues/27398
closes https://github.com/pandas-dev/pandas/issues/27829

Labels:
name: Numeric Operations
description: Arithmetic, Comparison, and Logical operations
name: ExtensionArray
description: Extending pandas with custom dtypes or arrays.

Comments:
As an aside, floor division by zero looks a little suspicious to me.  I'm not 100% sure what the expected behavior for it should be, but off the top of my head I think your test case should return the same values? Certainly doesn't need to be part of this PR though.
```python
In [7]: a // 0
Out[7]: 
<IntegerArray>
[0, 0, 0, NaN]
Length: 4, dtype: Int64

In [8]: a // -0
Out[8]: 
<IntegerArray>
[0, 0, 0, NaN]
Length: 4, dtype: Int64

In [9]: a // 0.0
Out[9]: array([nan, nan, nan, nan])

In [10]: a // -0.0
Out[10]: array([nan, nan, nan, nan])
```
I'd prefer to just merge this as is. I have a followup PR fixing `IntergerArray.__pow__` which depends on this and I can clarify the whatsnew there.
> This can go in first before the pd.NA IntegerArray PR?

Yes, this and my next PR fixing `IntegerArray.__pow__` are splitting off tangentially-related components of that large PR.
> I'd prefer to just merge this as is. I have a followup PR fixing IntergerArray.__pow__ which depends on this and I can clarify the whatsnew there.

Sounds good
BUG: Support categorical targets in IntervalIndex.get_indexer
State: closed

- [X] closes #30063
- [X] tests added / passed
- [X] passes `black pandas`
- [X] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [X] whatsnew entry


Labels:
name: Bug
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves
name: Categorical
description: Categorical Data Type
name: Interval
description: Interval data type

Comments:
thanks @jschendel very nice!