BrandonFors/pandas-github-issues huggingface dataset:
TST/bug: add test for NaT in Categorical.isin with empty string
State: closed

- [x] closes #36550
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


This bug was present in pandas v1.5, but is not present in v2.0.rc1. I added a test for this working as expected now. 

Labels:
name: Testing
description: pandas testing functions or related to the test suite

Comments:
> This bug was present in pandas v1.5, but is not present in v2.0.rc1. I added a test for this working as expected now.

Thanks. Looking at the date of the issue, the bug was present before 1.5.

Things may have changed since I took a break, but we don't normally add a release note when we just add a regression test.

Any idea which PR fixed the issue?
No whatsnew note, this is still correct
> Any idea which PR fixed the issue?

No, I just tried  the code snippet in the issue and saw that the problem is gone. Then I tried it in v1.5, where it was still present :-).

In v1.5 I also got a warning, when I ran the snippet (`FutureWarning: Inferring timedelta64[ns] from data containing strings is deprecated and will be removed in a future version. To retain the old behavior explicitly pass Series(data, dtype=timedelta64[ns])`), so my guess is whatever fixed that warning, also fixed this, but that's just a guess.

I could do a bisect, but IMO that's probably not worth it...

> I could do a bisect, but IMO that's probably not worth it...

no problem. The reason I asked is that the PR that fixed may have had a release note and the issue number could have been appended instead of adding the release note here.
Yeah, I looked through the whatsnew to find canditates and found several candidates and figured it wasn't worth it to figure which one it was...

I've removed the whatsnew entry in the newest commit.
Thanks @topper-123
BUG: not broadcasting when setting empty object
State: closed

- [ ] closes #51450 (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

This is a regression on 1.5.x that we could get rid of because of changes since then. Should go into 2.0 instead of 1.5.x

Labels:
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves
name: Regression
description: Functionality that used to work in a prior pandas version

Comments:
> This is a regression on 1.5.x that we could get rid of because of changes since then. Should go into 2.0 instead of 1.5.x

can you expand a bit on this
I tried removing the condition as I did here on 1.5.x and this does not work. Not overly interested in making an even more complex condition to accommodate all possible special cases on 1.5.x
ok. The issue hasn't been triaged. Maybe worth commenting there.
Thanks @phofl
Add `ScatterPlot._scatter()`, similar to `HistPlot._plot()`
State: closed

- [x] closes #51560 
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Enhancement
name: Visualization
description: plotting
name: Stale

Comments:
I've put the change in docs v2.0.1, under bugs, as I felt this was more appropriate
This pull request is stale because it has been open for thirty days with no activity. Please [update](https://pandas.pydata.org/pandas-docs/stable/development/contributing.html#updating-your-pull-request) and respond to this comment if you're still interested in working on this.
Thanks for the pull request, but I think this feature needs more discussion from the core devs before implementing, especially since this is a private method. Closing for now in favor of more discussion on the issue
DOCS: fix category docs
State: closed

- [x] closes #46770
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

The docs state that 

> all instances of ``CategoricalDtype`` compare equal to a ``CategoricalDtype(None, False)``,

which isn't true, e.g.:

```python
>>> import pandas as pd
>>> c1 = pd.CategoricalDtype(["a", "b"])
>>> c1 == pd.CategoricalDtype(None, False)
False
>>> c1 == pd.CategoricalDtype()
False
```

Users should just compare like they already do, likely `c1 == "category"` or `isinstance(c1, pd.CategoricalDtype)`. 


Labels:
name: Docs
name: Categorical
description: Categorical Data Type

Comments:
Thanks @topper-123
BUG: TDI divison giving bogus .freq
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Bug
name: Timedelta
description: Timedelta data type
name: Frequency
description: DateOffsets
name: Index
description: Related to the Index class or subclasses

Comments:
Thanks @jbrockmendel
PERF: DataFrame.where for EA dtype mask
State: closed

- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/v2.1.0.rst` file if fixing a bug or adding a new feature.

Perf improvement when the mask/cond is "boolean" (EA) dtype. 

Ending up with an EA mask is common when working with EA-backed data:
     
```
import pandas as pd
import numpy as np

df = pd.DataFrame(np.random.randn(10, 2), dtype="Float64")
mask = df > 0

print(mask.dtypes) 

# 0    boolean
# 1    boolean
# dtype: object
```

ASV added:
```
       before           after         ratio
     [ddae6946]       [48684e5c]
     <main>           <where-putmask-ea-boolean>
-      51.0±0.7ms       18.4±0.4ms     0.36  frame_methods.Where.time_where(False, 'Float64')
-      45.3±0.6ms       13.5±0.4ms     0.30  frame_methods.Where.time_where(True, 'Float64')
-      44.4±0.8ms       13.1±0.4ms     0.29  frame_methods.Where.time_where(False, 'float64[pyarrow]')
-      47.2±0.9ms       13.5±0.7ms     0.29  frame_methods.Where.time_where(True, 'float64[pyarrow]')
```

EA-backed frames are not widely covered by ASVs, but there are other methods it may help with, e.g.:
```
       before           after         ratio
     [ddae6946]       [48684e5c]
     <main>           <where-putmask-ea-boolean>
+      11.0±0.7ms       13.6±0.4ms     1.24  frame_methods.Clip.time_clip('float64')
-        95.9±4ms         41.0±2ms     0.43  frame_methods.Clip.time_clip('Float64')
-        82.0±2ms       30.8±0.2ms     0.38  frame_methods.Clip.time_clip('float64[pyarrow]')
```

Note a slight slowdown in the non-EA `clip` ASV. However, I think the simplification is worth it and it puts `clip` back to where it was a few days ago in terms of perf (prior to #51472). In addition, we might be able to improve perf via the discussion in #51547.

Labels:
name: Performance
description: Memory or execution speed performance
name: ExtensionArray
description: Extending pandas with custom dtypes or arrays.

Comments:
I think we can do them separately, but I agree that this should be handled before getting to the manager level
Can you fix conflicts?
thx @lukemanley
CI: Fix failure due to warning
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

cc @mroeschke any idea why this wasn't failing before?

https://app.circleci.com/pipelines/github/pandas-dev/pandas/25348/workflows/8a8ef371-75fe-4c4e-b928-33e74b63f1f9/jobs/59312

Labels:
name: CI
description: Continuous Integration

Comments:
Huh that's odd. Yeah not sure why this wasn't caught.

Should be rectified in https://github.com/pandas-dev/pandas/pull/51545#discussion_r1113736828 I think (there is an unnecessary pyarrow version check here since our min pyarrow version is 7 now)
I'll merge this, checked 5 builds only failures from the other pr. Then we should get a green ci again there
DOC: Clean whatsnew links
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Docs

Comments:
Thanks @phofl
BUG: transpose inferring dtype for dt in object column
State: closed

- [x] closes #51546 (Replace xxxx with the GitHub issue number)
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

cc @jbrockmendel thoughts about doing it for 2.0 or 2.1?

Labels:
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode
name: Dtype Conversions
description: Unexpected or buggy dtype conversions

Comments:
> thoughts about doing it for 2.0 or 2.1?

no strong preference.  i think we're doing more still-merging-to-2.0 in the RC this time around than we usually do, not sure how much is too much
Then lets do for 2.1, no strong preference either on my side
thanks @phofl
DOC Fix EX01 issues in docstrings
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

Towards https://github.com/pandas-dev/pandas/issues/37875

I will be doing a separate PR for the other files I took. I attached the resulting page.
[pandas.DataFrame.kurtosis — documentation.pdf](https://github.com/pandas-dev/pandas/files/10807590/pandas.DataFrame.kurtosis.documentation.pdf)


Labels:
name: Docs

Comments:
Great thanks @DeaMariaLeon 
Thank you
REGR: GH50127 fixes error for converting to / creating Index with dtype `S3`
State: closed

- [x] closes #50127
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions. (Does not apply)
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Regression
description: Functionality that used to work in a prior pandas version
name: Index
description: Related to the Index class or subclasses
name: Astype

Comments:
CI check error seems to be unrelated, since it appears on all latest CIs.

No error other than that, so happy to review it now
~moving off the 2.0 milestone as this is a regression from 1.5~

EDIT: sorry, this one worked on 1.5. Is it a blocker?
Release day, moving this to the 2.1 milestone.
Sorry, my bad, the next release should be 2.0.1, changed the milestone again.
` Ubuntu / Numpy Dev (pull_request)` is failing on all recent PRs, so I guess it is not related to the changes done here.
Thanks for the pull request, but it appears to have gone stale. If interested in continuing, please merge in the main branch, address any review comments and/or failing tests, and we can reopen.
this was close to ready, re-opening.  @Daquisu can you merge main and address any outstanding comments
@Daquisu can you merge main and address any outstanding comments?  ill make reviewing this a priority
Thanks for the pull request, but it appears to have gone stale. If interested in continuing, please merge in the main branch, address any review comments and/or failing tests, and we can reopen.
TST: Add test for groupby rename categories with multiIndexes
State: closed

- [x] closes #51500 (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Testing
description: pandas testing functions or related to the test suite

Comments:
@phofl , I turned it off based on my own judgment, but we can easily turn it back on if necessary.

Edit: I believe that we should also close #51500 since we have already obtained the desired test results previously
DOC: Update doc of parquet compression parameter
State: closed

This commit updates the interface documentation of [`pandas.DataFrame.to_parquet`](https://pandas.pydata.org/docs/dev/reference/api/pandas.DataFrame.to_parquet.html). The documentation of the parameter `compression` is not up-to-date compared to the implementation documentation of [`pandas.io.parquet.to_parquet`](https://github.com/pandas-dev/pandas/blob/fada873232ecb274f86c703770e41866084ae536/pandas/io/parquet.py#L399-L404). This commit copies the documentation of the implementation to the interface.

Labels:
name: Docs
name: IO Parquet
description: parquet, feather

Comments:
Could you merge in the main branch? Appears to be a linting error as well.
Yes, now it has. It says "pandas/core/frame.py:2804:89: E501 Line too long (91 > 88 characters)". It refers to the [line](https://github.com/pandas-dev/pandas/pull/51554/files#diff-421998af5fe0fbc54b35773ce9b6289cae3e8ae607f81783af04ebf1fbcaf077R2804), where you requested to add the default value to get a consistent style. However, this is why the first line of this parameter description exceeds the maximum line width.

- We could move the default value back to the second line.
- We could replace the list of possible values by `str or None, default 'snappy'`. The text still describes possible values.
- We could ignore the failing check or somehow add an expected failure.
- You could extend the allowed line width.

What do you think is the best fix?
> We could replace the list of possible values by str or None, default 'snappy'. The text still describes possible values.

This option sounds best
Should be fine now.
thx @ChristofKaufmann
PERF: DataFrame.first_valid_index/last_valid_index for EA dtypes
State: closed

- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/v2.1.0.rst` file if fixing a bug or adding a new feature.


ASV added:
```
       before                after    ratio
-      21.6±0.1ms          484±9μs     0.02  frame_methods.FindValidIndex.time_last_valid_index('float64[pyarrow]')
-      21.6±0.3ms          427±3μs     0.02  frame_methods.FindValidIndex.time_first_valid_index('float64[pyarrow]')
-      17.0±0.5ms          167±4μs     0.01  frame_methods.FindValidIndex.time_last_valid_index('Float64')
-      16.8±0.2ms          132±2μs     0.01  frame_methods.FindValidIndex.time_first_valid_index('Float64')
```

Labels:
name: Performance
description: Memory or execution speed performance
name: ExtensionArray
description: Extending pandas with custom dtypes or arrays.

Comments:
wow thats a huge difference.  is it all in the len(values) or something?
> wow thats a huge difference. is it all in the len(values) or something?

`self._values` on a multi-column EA-backed frame converts to an object dtype ndarray. This avoids going through the object dtype conversion.
> This is basically the same as with MultiIndex._values?

similar in the sense they're both expensive conversions through object dtype. `MultiIndex._values` has the additional overhead of wrapping in tuples.
Yep, thx, that's what I was thinking
Thx @lukemanley
BUG: parsing pyarrow dtypes corner case
State: closed

- [x] closes #51225 (Replace xxxx with the GitHub issue number)
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

pretty dumb corner case

Labels:
name: Arrow
description: pyarrow functionality

Comments:
release note? I think also should now target 2.1 now that rc is cut as this is not a 2.0.x regression or bugfix for a new feature.
Arrow support is more or less new, I think we can backport relevant fixes there
wasn't pd.ArrowDtype was added in 1.5?
Yes, but support was very limited for arrow stuff in 1.5
sure. it is experimental and this is not a big change so no objections to targeting 2.0
thx @jbrockmendel
TST/CLN: Remove unnecessary pyarrow version checking
State: closed

I think this shouldn't be necessary since 7.0 is the minimum version supported now.


Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Arrow
description: pyarrow functionality

Comments:
FAILED pandas/tests/indexes/test_common.py::test_sort_values_invalid_na_position[string-pyarrow-None] - AssertionError: Did not see expected warning of class 'PerformanceWarning'
FAILED pandas/tests/indexes/test_common.py::test_sort_values_invalid_na_position[string-pyarrow-middle] - AssertionError: Did not see expected warning of class 'PerformanceWarning'

Looks like the (some?) failures are due to https://github.com/pandas-dev/pandas/pull/51483 (should have been more careful with merging after being stricter with the warnings)
Can you remove my tm.assert_produces_warning when you are ready here?
> Can you remove my tm.assert_produces_warning when you are ready here?

Sure, done in the last commit.

Going to merge and backport just in case 2.0 has any PRs that impact `test_arrow.py`
API: ArrowExtensionArray.value_counts returns pyarrow.int64 type
State: closed

- [x] closes #51462 (Replace xxxx with the GitHub issue number)
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Arrow
description: pyarrow functionality

Comments:
Greenish 
thanks @mroeschke