BrandonFors/pandas-github-issues huggingface dataset:
BUG: TDA.__floordiv__ with NaT
State: closed

- [x] closes #35529
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry


Labels:
name: Bug
name: Numeric Operations
description: Arithmetic, Comparison, and Logical operations
name: Timedelta
description: Timedelta data type

Comments:
The return type of a floordiv with a scalar should be an integer. the cast is unavoidable at the moment. Maybe a TODO to change to nullable integer in the future and raising a RuntimeWarning in the meantime for consistency with NumPy.

```
>>> arr = np.array([1, 2, 3])
>>> arr
array([1, 2, 3])
>>>
>>> arr // arr
array([1, 1, 1], dtype=int32)
>>>
>>> arr2 = np.array([1, 2, np.nan])
>>> arr2
array([ 1.,  2., nan])
>>>
>>> arr // arr2
<stdin>:1: RuntimeWarning: invalid value encountered in floor_divide
array([ 1.,  1., nan])
>>>
```
otherwise LGTM
hang on a minute. NumPy now supports floordiv on timedeltas but returns an integer array.

```
>>> np.__version__
'1.19.1'
>>>
>>> left = np.array([1000, 222330, 30], dtype="timedelta64[ns]")
>>> left
array([  1000, 222330,     30], dtype='timedelta64[ns]')
>>>
>>> right = np.array([1000, 222330, None], dtype="timedelta64[ns]")
>>> right
array([  1000, 222330,  'NaT'], dtype='timedelta64[ns]')
>>>
>>> left // right
<stdin>:1: RuntimeWarning: invalid value encountered in floor_divide
array([1, 1, 0], dtype=int64)
>>>
```

UPDATED: this also raises a RuntimeWarning in a clean session.
> The return type of a floordiv with a scalar should be an integer


```
>>> ser = pd.Series([-1, 0, 1])
>>> ser // np.nan
0   NaN
1   NaN
2   NaN
dtype: float64
```

We also don't follow numpy exactly on this (though usually the issue is division by zero, not division by nan).

this looks fine. I think we return float arrays now in other ops (where as numpy does not cast), so think this is ok.
BUG: to_timedelta fails on Int64 Series with null values
State: closed

- [X] closes #35574 
- [X] tests added / passed
- [X] passes `black pandas`
- [X] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry


Labels:
name: Missing-data
description: np.nan, pd.NaT, pd.NA, dropna, isnull, interpolate
name: Timedelta
description: Timedelta data type

Comments:
We'll need a release note. This can go in 1.1.1 I think.
thanks @egoddard 
@Meeseeksdev backport to 1.1.x
Adding function to calculate years since a reference timestamp
State: closed

- [ ] closes #xxxx
- [X] tests added / passed
- [X] passes `black pandas`
- [X] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [X] whatsnew entry


Labels:
name: Needs Discussion
description: Requires discussion from core team before further action

Comments:
Why is this needed?
I'm not sure that "need" is the right standard to use here.

The common use case is calculating a person's age from date of birth. There are numerous questions of this type out on Stack Overflow. Most answers suggest using a time delta and dividing by 365.25, but this is erroneous due to leap years.

A sampling of Stack Overflow questions on the topic:
- https://stackoverflow.com/questions/26788854/pandas-get-the-age-from-a-date-example-date-of-birth
- https://stackoverflow.com/questions/26897098/pandas-calculating-age-from-a-date
- https://stackoverflow.com/questions/51491893/how-do-i-get-an-age-in-years-and-date-on-pandas
- https://stackoverflow.com/questions/46508895/calculating-age-from-date-time-format-in-python-pandas
- https://stackoverflow.com/questions/49726125/how-to-calculate-age-based-on-date-in-python-pandas-data-types-errors
- https://stackoverflow.com/questions/56170259/calculate-age-in-a-pandas-dataframe
- https://stackoverflow.com/questions/49819706/how-to-get-an-age-column-through-date-of-birth-in-pandas
- https://stackoverflow.com/questions/59005559/how-to-calculate-age-from-date-of-birth-in-pandas-pythonjupyter-notebook
Yea I'm also not sure this is useful enough to have a dedicated method for - is there an open issue / discussion for this?
agree here -1 on extending this api 
Thanks @bryan-woods for the PR. Unfortunately I don't think the core team has an appetite for this method, so going to close this for now. 

If it's something you feel strongly about I'd suggest opening an issue for discussion first and can reassess from there
BUG: df.shift(n, axis=1) with multiple blocks
State: closed

- [x] closes #35488
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry


Labels:
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode
name: Regression
description: Functionality that used to work in a prior pandas version
name: Period
description: Period data type

Comments:
thanks
@MeeseeksDev backport to 1.1.x
Ensure _group_selection_context is always reset
State: closed

Context managers will resume with an exception if the with block calling them fails.
This happening is not an excuse to not clean up.

- [ ] closes #xxxx
- [ ] tests added / passed
- [ ] passes `black pandas`
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry

Found by inspecting the code, not by actually finding a failing example.


Labels:
name: Groupby
name: Clean

Comments:
Thanks @eric-wieser for the PR

(just adding a note: #35314 uses _group_selection_context more, so if #35314 gets backported, maybe this should also be backported)
Thanks @eric-wieser
Change check_freq default to False
State: closed

- [x] closes #35570
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Regression
description: Functionality that used to work in a prior pandas version

Comments:
This effectively disables a bunch of existing checks.  passing check_freq=True explicitly in a ton of places is an option i guess.   or could change the default to False and warn that it will become True in a future version
And FYI, this came up on the dev call on Wednesday. The consensus was roughly "we didn't mean to change this for 1.1.0, but now that we have it's best to just leave it."

Apologies for the accidental breaking change @MaxWinterstein, and thanks for working on the PR but we'll pass.

In the future, you might want to subscribe to pandas releases by "watching" the repo for "releases only". We make binaries available for release candidates so it should be easy to test.
> And FYI, this came up on the dev call on Wednesday. The consensus was roughly "we didn't mean to change this for 1.1.0, but now that we have it's best to just leave it."
> 
> Apologies for the accidental breaking change @MaxWinterstein, and thanks for working on the PR but we'll pass.
> 
> In the future, you might want to subscribe to pandas releases by "watching" the repo for "releases only". We make binaries available for release candidates so it should be easy to test.

**First of all, i am totally fine by keeping it that way. No hate.**

I was/am just a little unhappy that this is - in my opinion - a bigger change that should be mentioned in the change logs anywhere. But afaik there is no mention at all. 
As our automated tests on ou rCI failed successfully, there was no hassle or problem on production side, as we keep requirements versioned (as everybody should 😉 ). But when you try to update your requirements and you need to dig inside the projects to find why something is suddenly broken, its a little bit frustrating... 

For sure. This was a process failure where we merged a breaking API change to master. That was noticed during review, but a followup issue was never opened.

As I mentioned in the issue, updating the release notes to explain this API change may still be worthwhile since it might help future upgraders.
DOC: add type BinaryIO to path param #35505
State: closed

- [ ] closes #35505
- [ ] tests added / passed
- [ ] passes `black pandas`
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: Docs
name: IO Excel
description: read_excel, to_excel

Comments:
Seems fine.  Not sure if the extra part of the docstring is necessary.
thanks @alexhtn (comment on a followon PR)
Ensure file is closed promptly in case of error
State: closed

Fixes #35566

Labels:
name: Bug
name: IO SAS
description: SAS: read_sas

Comments:
can you add a test
Very difficult to add a test since the failure is during the constructor meaning that we have no references to the failed state.
I have pulled something together, but it requires a dependency on psutils and assumes that the test file is on a local disk.  Is that acceptable?
A dependency on external libs for this is not great IMO. Are there tests for closed files for other file formats, e.g. excel or csv files? Could you try to emulate them?
OK, let's follow up in the new PR #35587 (sorry, I deleted my repo in the meantime, so need a new PR).
BUG: Ensure rolling groupby doesn't segfault with center=True
State: closed

- [x] closes #35552
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry


Labels:
name: Bug
name: Regression
description: Functionality that used to work in a prior pandas version
name: Window
description: rolling, ewma, expanding

Comments:
can you merge master, looks good
thanks @mroeschke 
@MeeseeksDev backport to 1.1.x
Merge pull request #1 from pandas-dev/master -  #34498 improve the .equals() docstring
State: closed


#34498 improve the .equals() docstring



Labels:
name: Docs

Comments:
Hello @data-RanDan! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:



There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2020-08-04 21:19:37 UTC
@jorisvandenbossche 
Hi
I'm not sure what causes these failures, i only changed the comments..
We actually already have an open PR that is trying to address the same issue: https://github.com/pandas-dev/pandas/pull/34508
well' he seems to be at it for a while..
and i just solved it for you

On Thu, 6 Aug 2020 at 10:31, Joris Van den Bossche <notifications@github.com>
wrote:

> We actually already have an open PR that is trying to address the same
> issue: #34508 <https://github.com/pandas-dev/pandas/pull/34508>
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <https://github.com/pandas-dev/pandas/pull/35555#issuecomment-669759508>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AOJEEFLI5RBWNQOYIUWCO4TR7JL4BANCNFSM4PUZRASA>
> .
>

It's only two weeks, and just a trailing whitespace failure that was remaining in the older PR. So I updated that one.
@data-RanDan I merged the other PR, so going to close this one. But know that your contribution is very much valued nonetheless! On a big project like pandas it is easy to overlook other issues/PRs. It's always a bit annoying if such duplicate work happens, but given that the other PR was older and further in the review process, decided to merge that one.
DOC: Document that read_hdf can use pickle
State: closed

This documents that `read_hdf`, which uses PyTables, might invoke pickle to deserialize arrays that were serialize with pickle.

PyTables clearly documents this at http://www.pytables.org/usersguide/libref/declarative_classes.html#tables.ObjectAtom, but pandas users may not realize that if they aren't aware that pandas uses pytables under the hood.

I've tried to include the warning in all the methods that eventually read data, but it's a bit hard to say if I've gotten them all.

Ideally we would also like to provide an `allow_pickle=False` keyword, but that would be best implemented in PyTables, and used by us. I opened https://github.com/PyTables/PyTables/issues/813 for discussion.

Labels:
name: Docs

Comments:
lgtm ex Jeff's comments
FYI, I tagged this for 1.1.1. Wouldn't normally backport doc-only changes but this is tangentially security related so I think it's better to backport.
thanks @TomAugspurger
Bug fix one element series truncate
State: closed

- [Y] closes #35544 
- [Y] tests added / passed
- [Y] passes `black pandas`
- [Y] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [Y] whatsnew entry

Fix when trying to truncate a one-element series within a correct date range returned an empty series instead of the originally passed-in series. 

Labels:
name: Datetime
description: Datetime data dtype
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves
name: Regression
description: Functionality that used to work in a prior pandas version
name: Series
description: Series data structure

Comments:
ok looks good, can you merge master and ping on green.
> ok looks good, can you merge master and ping on green.

@jreback All green!
thanks @gabicca 
> thanks @gabicca

Awesome, Thanks!
REGR: Fix interpolation on empty dataframe
State: closed

Interpolation on an empty dataframe broke in 1.1 due to a change in how 'all columns are objects' is checked (specifically all(empty set) is True, while before dtype count object = None was checked against size = 0).
This is a complex function and I'm not sure what the proper fix is, suggesting to keep the empty check out of the rest of the logic.

Example code that broke:

```python
import pandas as pd
df = pd.DataFrame([1,2])
df[[]].interpolate(limit_area='inside')
```

- [x] closes #35598
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry


Labels:
name: Missing-data
description: np.nan, pd.NaT, pd.NA, dropna, isnull, interpolate
name: Regression
description: Functionality that used to work in a prior pandas version
name: Algos
description: Non-arithmetic algos: value_counts, factorize, sorting, isin, clip, shift, diff

Comments:
Hello @sanderland! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:





There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2020-08-07 19:37:15 UTC
Can you add a test case for this?
> Can you add a test case for this?

done
@sanderland can you merge master once more to see if that fixes CI?
test failure was

```
=========================== short test summary info ===========================
FAILED pandas/tests/computation/test_eval.py::TestAlignment::test_basic_series_frame_alignment[numexpr-python]
= 1 failed, 72824 passed, 1746 skipped, 1055 xfailed, 58 warnings in 1102.57s (0:18:22) =
```
probably unrelated.

restarted (azure will patch against master)
Thanks @sanderland 
@meeseeksdev backport 1.1.x


I think Simon's call to merge this was appropriate (all the existing review comments were addressed, it was reviewed by several core devs). Yes, you noticed an additional problem (the missing copy) after merge: no problem, that can always happen, and we can do a follow-up to fix this.

> not to mention this is not the correct place for this

See my comment at https://github.com/pandas-dev/pandas/pull/35543#discussion_r469979874 (and @sanderland's own answer above that), your suggestion to move it down into the internals is not possible, so this function is a correct place to put the check.
Backport PR #35440 on branch 1.1.x (BUG: CategoricalIndex.format)
State: closed

Backport PR #35440: BUG: CategoricalIndex.format

Labels:

Comments:
flaky test

```
=========================== short test summary info ===========================
FAILED pandas/tests/io/parser/test_common.py::test_chunks_have_consistent_numerical_type[python]
= 1 failed, 72699 passed, 1743 skipped, 1054 xfailed, 58 warnings in 685.50s (0:11:25) =
```
Check if NPY_NAT is NA for int64 in rank() (#32859)
State: closed

- [x] closes #32859
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry

Used  `missing.isnaobj()` to try to handle the suggestion from @jbrockmendel. Lmk if this might cause other issues.


Labels:
name: NA - MaskedArrays
description: Related to pd.NA and nullable extension arrays

Comments:
Seems reasonable. Can you run the benchmarks for this to make sure it doesn't introduce a regression?
Hmmm, I think I misunderstood what types should return true. I'll revisit this another time if I get the chance.
BUG: handle immutable arrays in tz_convert_from_utc (#35530)
State: closed

- [x] closes #35530 
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry

Labels:
name: Datetime
description: Datetime data dtype

Comments:
Seems reasonable; not sure if CI failure was related so restarted

@jbrockmendel 
small comments, otherwise lgtm
Changed to be a copy (which also lets us remove the const specifiers pointed out), CI green!
LGTM
Thanks @lidavidm 
Thank you for the review!
BUG: Fix assert_equal when check_exact=True for non-numeric dtypes #3…
State: closed

`assert_series_equal(..., check_exact=True)` no longer raises an error when series has a non-numeric dtype. This fixes a bug/ regression introduced in `1.1.0`.

- [x] closes #35446
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry

I've added a very simple test, but could add more. For instance, there could be a check on `assert_frame_equal`, but I'm not sure what the desired strategy is for that.

I've left off the `whatsnew` entry for now since I figure that's the most likely to generate conflicts while this gets reviewed.

Labels:
name: Bug
name: Testing
description: pandas testing functions or related to the test suite

Comments:
@jreback, added the release note, and it's kinda green. It looks like one of the jobs timed out though.
> It looks like one of the jobs timed out though.

restarted
also pls merge master
Made those changes. How is it now?
thanks!
@meeseeksdev backport 1.1.x
BUG: RollingGroupby respects __getitem__
State: closed

- [x] closes #35486
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry


Labels:
name: Bug
name: Groupby
name: Window
description: rolling, ewma, expanding

Comments:
thanks for addressing @mroeschke 
@MeeseeksDev backport to 1.1.x
REGR: Check for float in isnaobj_old
State: closed

- [x] closes #35493
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry

Looks like we lost an isinstance check that caused this regression.

Labels:
name: Regression
description: Functionality that used to work in a prior pandas version
name: IO CSV
description: read_csv, to_csv

Comments:
thanks @dsaxton 
backport to 1.1.x
@simonjayhawkins hmm these don't seem to be triggering a backport action. any idea?
> @simonjayhawkins hmm these don't seem to be triggering a backport action. any idea?

it worked on #35452 but not from #35272
@MeeseeksDev backport to 1.1.x
Added paragraph on creating DataFrame from list of namedtuples
State: closed

- [x] closes #35438 

Is a whatsnew entry necessary for this PR?

Labels:
name: Docs

Comments:
> Is a whatsnew entry necessary for this PR?

no
@arw2019 Thanks for the suggestions, I've updated the text and one of the examples.

I'm confused about the indentation though, I've checked that it consists of 4 spaces, and throughout the file it's a mix of 3- and 4-space indentation actually...
thanks @sjvrijn
Added alignable boolean series and its example to `.loc` docs.
State: closed

- [x] closes #31054
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry

I'm just not sure if I needed to add `IndexingError` to raises section. Added it to be safe, will remove if it's unnecessary.

Also, when someone uses `df.loc[df['col'] > 0]` it really is the same under the hood, `df['col'] > 0` is a boolean series with an alignable index. So there might be some overlap?

Labels:
name: Docs

Comments:
Thanks for the review!
Thanks @MJafarMashhadi
TST: ensure that DataFrameGroupBy.apply does not convert datetime.date to pd.Timestamp
State: closed

- [x] closes #29617 
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`

Previously, there was a bug in `DataFrameGroupBy.apply` where a column of `datetime.date` would raise a `ValueError` if it was included as column in a multi-column grouping. Somewhere in the grouping the values would be converted to `pd.Timestamp` and then when the `pd.Timestamp` couldn't be found in the index of `datetime.date`s it would throw an error. 

This bug persisted until 1.0.5 but was fixed in 1.1.0 (not sure which change fixed it). In 1.1.0 the `datetime.date` are not converted to `pd.Timestamp`, so they are treated like any other dtype=object and left unchanged. 

This PR adds a test to enforce this behaviour to make sure the bug does not arise again. 

Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Groupby
name: Apply
description: Apply, Aggregate, Transform, Map

Comments:
thanks @smithto1
Backport PR #35477 on branch 1.1.x (MAINT: Use float arange when required or intended)
State: closed

Backport PR #35477: MAINT: Use float arange when required or intended

Labels:

Comments:
test failure unrelated

```
=========================== short test summary info ============================
FAILED pandas/tests/io/parser/test_common.py::test_chunks_have_consistent_numerical_type[python]
= 1 failed, 69947 passed, 4254 skipped, 1021 xfailed, 13 warnings in 745.75s (0:12:25) =
```
Fix numpydev CI failure
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [ ] passes `black pandas`
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: CI
description: Continuous Integration
name: Compat
description: pandas objects compatability with Numpy or Python functions

Comments:
LGTM pending green
Adding to 1.1.1 as I think should backport - @TomAugspurger or @simonjayhawkins good to merge?
Yep to backport, whenever you're ready.

On Fri, Jul 31, 2020 at 2:46 PM William Ayd <notifications@github.com>
wrote:

> Adding to 1.1.1 as I think should backport - @TomAugspurger
> <https://github.com/TomAugspurger> or @simonjayhawkins
> <https://github.com/simonjayhawkins> good to merge?
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <https://github.com/pandas-dev/pandas/pull/35496#issuecomment-667320481>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AAKAOIVJQ6OC3IR5J6NEIH3R6MNSVANCNFSM4PQXJ6QQ>
> .
>

> Adding to 1.1.1 as I think should backport - @TomAugspurger or @simonjayhawkins good to merge?

aren't the same tests failing or am I missing something.

we only have Travis running on backport so could wait to backport till after the 2 PRs activating azure and github actions are merged. 
@simonjayhawkins whoops good point. Using the github app misread the checks. Will take a look in a bit