BrandonFors/pandas-github-issues huggingface dataset:
Implement Arrow String Array that is compatible with NumPy semantics
State: closed

cc @jbrockmendel @jorisvandenbossche 

I'll do a couple of pre-cursors to reduce the diff

Context: https://github.com/pandas-dev/pandas/issues/54792

Labels:

Comments:
You will also need to update the na_value use for the dtype:

```diff
--- a/pandas/core/arrays/string_.py
+++ b/pandas/core/arrays/string_.py
@@ -101,7 +101,10 @@ class StringDtype(StorageExtensionDtype):
     #: StringDtype().na_value uses pandas.NA
     @property
     def na_value(self) -> libmissing.NAType:
-        return libmissing.NA
+        if self.storage == "pyarrow_numpy":
+            return np.nan
+        else:
+            return libmissing.NA
```

That eg ensures that getitem returns a NaN instead of pd.NA
Flying today, will look tomorrow/thursday
@jorisvandenbossche would you be ok with doing this as a follow up? I tested this on my train ride but it broke a bunch of tests (obviously), which would make review even more complicated
See #54585 for the na value
Something that also still needs to be done (for this topic, not necessarily in this PR) is to ensure that the constructors and IO methods that follow `pd.options.future.infer_string` now use this pyarrow_numpy dtype instead of the plain pyarrow one?
Yep this will be simple, but needs tear adjustments so easier as a follow up.

the actual logic is in 2 methods, so code adjustments will be minimal 
cc @jbrockmendel this should be ready behaviour wise now
@jbrockmendel @jorisvandenbossche are we good to merge here?
Yes, let's merge this. You have some follow-up PRs in the line anyway, so any further comment can be addressed in those, I think.

(and I'll open some issues for some of the comments I made)
@MeeseeksDev backport to 2.1.x
I'll put up a pr for the infer option and the whatsnew later today
Opened https://github.com/pandas-dev/pandas/issues/54792 as a general follow-up issue tracking the new string dtype topic. And will open some other issues for a few of the remaining discussion items above.
Fixed DatetimeIndex.tz_convert() .freq attribute
State: closed

- [ ] closes #49696
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) 
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).



Labels:

Comments:
Thanks for the pull request, but it appears to have gone stale. If interested in continuing, please merge in the main branch, address any review comments and/or failing tests, and we can reopen.
REF: Factor out string methods from ArrowExtensionArray
State: closed

pre-cursor for a StringArray that follows numpy semantics

cc @jorisvandenbossche maybe? Should be an easy review

Labels:
name: Strings
description: String extension data type and string data
name: Arrow
description: pyarrow functionality

Comments:
Hm I think we didn't think this through today. This probably requires more changes to the "string[pyarrow]" dtype than I am comfortable with between rc and actual release.
BUG: Special-case setting nan into integer series
State: closed

~totally wrong, please ignore, will update~

~have updated, maybe it's not totally wrong anymore, but I'm off to bed now~🛏️ 

have woken up, and I think this is alright, so ready for reviews

Should integer interval dtype also not warn when setting `np.nan`? I've left that one out for now, but interval dtype is far less used than integer so I wouldn't consider that part a blocker to 2.1

Closes #54660

Labels:
name: Bug

Comments:
@MarcoGorelli - looks like you recently updated the title of this PR which seems to be causing other PRs to fail doc builds with:

```
Traceback (most recent call last):
  File "/home/runner/work/pandas/pandas/web/pandas_web.py", line 475, in <module>
    sys.exit(main(args.source_path, args.target_path))
  File "/home/runner/work/pandas/pandas/web/pandas_web.py", line 427, in main
    context = get_context(config_fname, target_path=target_path)
  File "/home/runner/work/pandas/pandas/web/pandas_web.py", line 382, in get_context
    context = preprocessor(context)
  File "/home/runner/work/pandas/pandas/web/pandas_web.py", line 334, in roadmap_pdeps
    for pdep in sorted(pdeps["items"], key=sort_pdep)
  File "/home/runner/work/pandas/pandas/web/pandas_web.py", line 328, in sort_pdep
    raise ValueError(msg)
ValueError: Could not find PDEP number in 'PDEP: Special-case setting nan into integer series (pdep6)'. Please make sure to
                write the title as: 'PDEP-num: PDEP: Special-case setting nan into integer series (pdep6)'.
```
Seems like it doesn’t even allow the “(pdep6)” at the end of the title
I think it's just the label - I've removed that
> Should integer interval dtype also not warn when setting `np.nan`? I've left that one out for now, but interval dtype is far less used than integer so I wouldn't consider that part a blocker to 2.1

Agreed this is not a blocker and can be done later. And I personally also have no strong opinion. I suppose if we follow the same logic, it also shouldn't warn? 
thanks - merging then, have opened https://github.com/pandas-dev/pandas/issues/54790 regarding the interval case
DOC: Series.count remove mention of "level"
State: closed

similar to #54395 (I should have checked `Series.count` in that PR)


Labels:
name: Docs

Comments:
Looks like this is failing due to the following inconsistency in return types for boolean arrays vs other arrays:

```
In [1]: import pandas as pd

In [2]: type(pd.Series([1, 0 , 1]).std())
Out[2]: float

In [3]: type(pd.Series([True, False, True]).std())
Out[3]: numpy.float64
```

This is tested behavior in `test_reductions_return_types`.
Thanks @lukemanley
BLD: Fix version script
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

Had a small issue with the conda packages, since there was a bug in this script where it wasn't picking up the version properly when built from sdist (outside of the package dir).

Labels:
name: Build
description: Library building on various platforms

Comments:
Thanks @lithomas1
Run CI on 2.1.x
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: CI
description: Continuous Integration

Comments:
thanks @phofl
ENH: Allow usage of custom library to serialize with to_json method.
State: closed

- [X] closes #46571
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [X] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [X] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:

Comments:
My intuition is that passing a custom serializer is going to break a ton of stuff.  e.g. would you expect the stdlib json.dumps to work in the existing tests?  im pretty sure it wouldnt
> My intuition is that passing a custom serializer is going to break a ton of stuff. e.g. would you expect the stdlib json.dumps to work in the existing tests? im pretty sure it wouldnt

Yeah adding functionality to add json.dumps did break a lot of tests. That’s why I also added an additional ‘engine_kwargs’ parameter so that I could pass the appropriate parameters for the custom serializer. 

A lot of the I/O methods already support ‘engine_kwargs’ so adding this would follow API standardization. 

@jbrockmendel what do you think? 
What is the actual use case here?  Do you have a serializer that is more performant?  Are there dtypes/scalars that are currently unsupported that you want to be supported?
Yeah I don't think completely sidestepping the existing implementation isn't nicely maintainable here. 
@mroeschke Do we want to close this PR and the corresponding issue then? 

It looks like the `Needs Triage` label was removed for the issue roughly 2 years ago. It sounds like the acceptance criteria for the issue is no longer needed, or needs to be updated?
Yeah let's close this issue pending further discussion. Usually any enhancement request requires more buy in from the core team before a PR
ENH: ArrowExtensionArray(duration) workarounds for pyarrow versions >= 11.0
State: closed

The [issue](https://github.com/apache/arrow/issues/15226) was fixed upstream so the workaround is no longer needed for pyarrow versions >= 11.0


Labels:
name: Upstream issue
description: Issue related to pandas dependency
name: Arrow
description: pyarrow functionality

Comments:
thx @lukemanley
REF: implement Dtype.index_class
State: closed

Allows EAs to specify what Index subclass they want to use.  Avoids special-casing pandas-internal EAs.

Small perf improvement

```
In [2]: dti = pd.date_range("2016-01-01", periods=3, tz="UTC")

In [3]: cat = dti.astype("category")

In [4]: %timeit pd.Index(dti)
6.39 µs ± 80 ns per loop (mean ± std. dev. of 7 runs, 100,000 loops each) # <- main
5.69 µs ± 368 ns per loop (mean ± std. dev. of 7 runs, 100,000 loops each)  # <- PR

In [5]: %timeit pd.Index(cat)
6.37 µs ± 112 ns per loop (mean ± std. dev. of 7 runs, 100,000 loops each)  # <- main
5.8 µs ± 135 ns per loop (mean ± std. dev. of 7 runs, 100,000 loops each)  # <- PR

```

Labels:
name: Refactor
description: Internal refactoring of code
name: ExtensionArray
description: Extending pandas with custom dtypes or arrays.

Comments:
Thanks @jbrockmendel
Speed up StringDtype arrow implementation for merge
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

sorry @jbrockmendel :) I promise I'll work on an EA implementation after that string dtype with NumPy semantics

Labels:
name: Performance
description: Memory or execution speed performance
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode

Comments:
Can you add a whatsnew?

@jbrockmendel @phofl 
How close is to being merged?
Should I move this off the milestone or is this fine to keep?
There already a whatsnew from the previous pr, this should be ready

cc @jbrockmendel
PERF: Dataframe reductions with EA dtypes
State: closed

Avoids re-checking the `_reduce` method signature on every block which can be slow. No whatsnew as I believe the method signature check was only added in the 2.1 cycle.

```
import pandas as pd
import numpy as np

df_wide = pd.DataFrame(np.random.randn(4, 10_000), dtype="float64[pyarrow]")

%timeit df_wide.sum()

# 3.2 s ± 54.5 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)   <- main
# 2.02 s ± 89.9 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)  <- PR    
```


Labels:
name: Performance
description: Memory or execution speed performance
name: ExtensionArray
description: Extending pandas with custom dtypes or arrays.
name: Reduction Operations
description: sum, mean, min, max, etc.

Comments:
thx @lukemanley 
Hello @lukemanley, 

Great work on the performance enhancement! I'm still learning and was curious about how you identified this for improvement. Could you share a bit about your process?

Thank you for your time and contributions!
Enable assert_almost_equal to test numpy-like zero-dimensional arrays
State: closed

- [ ] closes #48919
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).


Labels:
name: Enhancement
name: Testing
description: pandas testing functions or related to the test suite
name: Nested Data
description: Data where the values are collections (lists, sets, dicts, objects, etc.).
name: Stale

Comments:
hi @WillAyd and @jbrockmendel just wanted to bump this in case it got buried. Do you guys want me to make any changes?
This pull request is stale because it has been open for thirty days with no activity. Please [update](https://pandas.pydata.org/pandas-docs/stable/development/contributing.html#updating-your-pull-request) and respond to this comment if you're still interested in working on this.
Thanks for the pull request, but it appears to have gone stale. If interested in continuing, please merge in the main branch, address any review comments and/or failing tests, and we can reopen.
Move Unit Tests for groupby.rolling.corr
State: closed

Moves recently created unit tests for groupby.rolling.corr to pandas/tests/window/test_groupby.py

Resolves comment in #54499

Labels:
name: Testing
description: pandas testing functions or related to the test suite

Comments:
pre-commit.ci autofix
Thanks @omar-elbaz 
@meeseeksdev backport 2.1.x
Fix scatter_matrix lacking ability to use grid parameter
State: closed

- [ ] closes #50818
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Visualization
description: plotting

Comments:
As per #50818 the scatter_matrix function would not produce a scatter matrix with a grid if grid=True was passed. I found this to be a result of the act of setting all non-bottom/left corner subplots of the matrix to have their axes hidden. The matplotlib function 'grid()' would simply not work on subplots that had their axes hidden.

To get around this instead of hiding the x and y axes I found that turning off labels and tick numbers accomplishes the same goal of hiding unnecessary information while allowing grid lines to be shown when grid() is called. Heres what it looks like

<img width="1440" alt="Screenshot 2023-08-11 at 12 14 20 PM" src="https://github.com/pandas-dev/pandas/assets/100482381/b4e36b23-0cf7-4e10-ae1d-dddfae2dc8ae">

If its preferred to give grid lines to diagonals thats a fix I can make as well. I thought turning them off would be better as the tick numbers for the diagonal is incorrect, though thats unrelated to the grid lines.
@mroeschke could you provide any assistance on next steps?
I nor many people have much expertise on this part of the code base so I'm not sure if anyone is available to provide meaningful feedback on the fix
Thanks for the PR here, but as mentioned, I don't think we have anyone on the core team right now to provide meaningful feedback here so closing for now. We can reopen if we have a matplotlib expert to review this PR
#51826_Fixed bug when np.nan is used as index value with .reindex on …
State: closed


- [x] closes #51826 
- [x] [Tests added and passed]
- [x] All [code checks passed]



Labels:
name: Missing-data
description: np.nan, pd.NaT, pd.NA, dropna, isnull, interpolate
name: Interval
description: Interval data type

Comments:
In the query method, when `nan` was passed as index value, it triggered the `else` condition, which extended the result, without properly handling `nan`. This fix ensures proper handling of `nan` value right at the beginning of the query, preventing the unexpected behavior during reindexing.
Unit test for groupby.rolling.corr
State: closed

- [ ] closes #44078  
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) 
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).



Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Groupby
name: Window
description: rolling, ewma, expanding

Comments:
pre-commit.ci autofix
thanks @omar-elbaz
Speed up string inference in maybe_convert_objects
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Strings
description: String extension data type and string data

Comments:
ping on green
Green, merging
@meeseeksdev backport 2.1.x.
Fix inference for fixed with numpy strings with arrow string option
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Strings
description: String extension data type and string data

Comments:
@jbrockmendel gentle ping
Moving off milestone for now.

If Brock reviews in time, can always put it back.
@jbrockmendel ping
DOC: updated required dependencies list
State: closed

- [x] closes #54492 
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).

Labels:
name: Docs

Comments:
@alimcmaster1 tagging as reviewer
@MarcoGorelli could you remind me why we added tzdata as a required dependency?
> @MarcoGorelli could you remind me why we added tzdata as a required dependency?

added here https://github.com/pandas-dev/pandas/pull/51247 
> @MarcoGorelli could you remind me why we added tzdata as a required dependency?

Think its a PEP 615 thing - zoneinfo only works on certain platforms.
https://peps.python.org/pep-0615/#abstract
Right but I'm curious why we didn't make this dependency platform specific
On my phone, but IIRC that was a request from @lithomas1 
I don't think it's necessary on all platforms but doesn't hurt.

We are able to enforce that tzdata exists (the db isn't guaranteed to exist on all platforms) and also enforce a minimum version by putting it as a dep.
Thanks @rsm-23
DEPR: Positional arguments in to_hdf except path_or_buf
State: closed

- [X] xref #54229
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [X] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [X] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: IO HDF5
description: read_hdf, HDFStore

Comments:
@mroeschke Pinging on green
@mroeschke Pinging on green
Thanks just a merge conflict otherwise looks good
@mroeschke Merge conflict should be resolved now
Thanks @rmhowe425
DOC: Prepare release notes for 2.1.0rc0 and remove 2.0.4 release notes
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Docs

Comments:
```
2023-08-10T21:21:21.8021565Z /home/runner/work/pandas/pandas/doc/source/whatsnew/index.rst:38: WARNING: toctree contains reference to nonexisting document 'whatsnew/v2.0.4'
```
CI: Another shot at fixing macOS wheel builds
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

This doesn't repro at all on my fork which is annoying.
This is a last ditch shot at fixing this before the RC

Labels:
name: Build
description: Library building on various platforms

Comments:
pre-commit.ci autofix
Merging since this was green on my branch and I want to get a test in before the RC tomorrow at least.