BrandonFors/pandas-github-issues huggingface dataset:
REF/PERF: MultiIndex.get_indexer with method
State: closed

- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/v2.2.0.rst` file if fixing a bug or adding a new feature.


```
from asv_bench.benchmarks.multiindex_object import Integer

b = Integer()
b.setup()

%timeit b.time_get_indexer_and_backfill()

# 238 ms ± 39.6 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)     -> main
# 50.6 ms ± 2.98 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)  -> PR

%timeit b.time_get_indexer_and_pad()

# 191 ms ± 30.3 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)   -> main
# 50.4 ms ± 1.03 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)  -> PR
```

Labels:
name: Refactor
description: Internal refactoring of code
name: Performance
description: Memory or execution speed performance
name: MultiIndex

Comments:
Thanks @lukemanley (yeah these failures are not related)
REF: make plotting less stateful
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Visualization
description: plotting
name: Clean

Comments:
Thanks @jbrockmendel
BUG: Index.getitem returning wrong result with negative step for arrow
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


```
idx = Index(list("bcdxy"), dtype="string[pyarrow_numpy]")
idx[slice(4, -6, -1)]
```

This omits b for arrow dtypes and looks like a bug on the arrow side, cc @jorisvandenbossche 

Labels:
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves
name: Arrow
description: pyarrow functionality

Comments:
Opened an issue on the Arrow side -> https://github.com/apache/arrow/issues/38768
Thanks @phofl 
Owee, I'm MrMeeseeks, Look at me.

There seem to be a conflict, please backport manually. Here are approximate instructions:

1. Checkout backport branch and update it.

```
git checkout 2.1.x
git pull
```

2. Cherry pick the first parent branch of the this PR on top of the older branch:
```
git cherry-pick -x -m1 196e907a4d88f6cd11062f3fb02b3c76faf51b6b
```

3. You will likely have some merge/cherry-pick conflict here, fix them and commit:

```
git commit -am 'Backport PR #55832: BUG: Index.getitem returning wrong result with negative step for arrow'
```

4. Push to a named branch:

```
git push YOURFORK 2.1.x:auto-backport-of-pr-55832-on-2.1.x
```

5. Create a PR against branch 2.1.x, I would have named this PR:

> "Backport PR #55832 on branch 2.1.x (BUG: Index.getitem returning wrong result with negative step for arrow)"

And apply the correct labels and milestones.

Congratulations — you did some good work! Hopefully your backport PR will be tested by the continuous integration and merged soon!

Remember to remove the `Still Needs Manual Backport` label once the PR gets merged.

If these instructions are inaccurate, feel free to [suggest an improvement](https://github.com/MeeseeksBox/MeeseeksDev).
DOC: replace deprecated freqs in user_guide Period
State: closed

xref #54061

replaced deprecated frequencies in user_guide Period with new frequencies.

Labels:
name: Docs

Comments:
thx @natmokval 
> Can you grep that they aren't anywhere else?

yes, I used grep to check if there are anywhere else frequencies which are deprecated.
I found a couple of places where comments contain deprecated frequencies. For example in `pandas/_libs/tslibs/period.pyx` we have deprecated 'T': `# ('T', 5) but may be passed in as a string like '5T'`

I am not sure, should I change 'T' to "min' in the comment?

Also we have in `asv_bench/benchmarks/` some tests with deprecated frequencies `"S"` and `"H"`. I think it's better to replace them with valid frequencies, are you agree?
Could you please explain why when I run tests in `asv_bench/benchmarks/` with deprecated frequencies they don't fail.
thanks for reviewing this PR @phofl
TYP: plotting
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Visualization
description: plotting
name: Typing
description: type annotations, mypy/pyright type checking

Comments:
Thanks @jbrockmendel
Reorder tests in maybe_downcast_numeric
State: closed

The comment `# if we have any nulls, then we are done` is not consistent with the test `if isna(arr).any()` because `arr` is constructed only from the first element (`r[0]`) not the full ravel'd list of values.  Moreover, calling `np.array()` on some random type can have surprising consequences.

So instead, do the early-out test as intended, just using `r[0]` without going through `np.array()`.  Then test other things about `r[0]`.  Only then should we test all the values (and if we have any nulls, then we are done).

See #55824

- [55824] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Dtype Conversions
description: Unexpected or buggy dtype conversions
name: ExtensionArray
description: Extending pandas with custom dtypes or arrays.

Comments:
While there is a conversation going on in #55824, these changes introduce no new functionality nor user-visible changes (other than tolerating Pint's behavior).  All these changes do is remove a dependency on the behavior of `np.array()` that can be accomplished as well by other means, and to make the implementation match the comments regarding the results of encountering NaN/NA values in the values to be possibly downcasted.
Cool thanks for the PR. Can you add a test and a whatsnew note?
Would be happy to, but...the test case would require loading Pint.  Last time I tried to do some test cases with alien packages I got back a big "No thanks!" from the pandas maintainers.  Also, another PR I submitted (enabling many complex test cases) rejected the WhatsNew changes I wrote because there were no user-visible changes.  Would you like to guide me further?
So to wrap this up...I don't think it's easy to add a test case because that would involve installing Pint, which might not be very CI/CD-friendly.  If there's a good way to do that, please point me to a pattern I can follow.  It's easy for me to describe the change in CHANGES, but its not really user-visible.  But because there is an issue (though no test case), I could write it up anyway.  Thoughts?
@WillAyd 

> Can you add a test and a whatsnew note?

With the current implementation in this PR, I wouldn't expect any behavior changes. You good here?

It does change for Pint because there `np.array([e])` will raise for certain Pint objects `e`, but I don't think we should be testing for this.
Thanks @MichaelTiemannOSC
TST: Fix shares_memory for arrow string dtype
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Strings
description: String extension data type and string data
name: Arrow
description: pyarrow functionality

Comments:
```bash
mypy.....................................................................Failed
- hook id: mypy
- duration: 131.55s
- exit code: 1

pandas/_testing/__init__.py:1051: error: "ExtensionDtype" has no attribute "storage"  [attr-defined]
pandas/_testing/__init__.py:1058: error: "ExtensionDtype" has no attribute "storage"  [attr-defined]
Found 2 errors in 1 file (checked 1445 source files)
```
thx
Thanks @phofl
ENH: infer resolution in array_to_datetime_with_tz
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

towards #55564

Labels:
name: Timezones
description: Timezone data dtype
name: Non-Nano
description: datetime64/timedelta64 with non-nanosecond resolution

Comments:
Thanks @jbrockmendel
BUG: Index.isin raising for arrow strings and null set
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Arrow
description: pyarrow functionality

Comments:
Thanks @phofl 
Owee, I'm MrMeeseeks, Look at me.

There seem to be a conflict, please backport manually. Here are approximate instructions:

1. Checkout backport branch and update it.

```
git checkout 2.1.x
git pull
```

2. Cherry pick the first parent branch of the this PR on top of the older branch:
```
git cherry-pick -x -m1 365448dc665f6d49f3d3071a6c3a04aaccacb3d2
```

3. You will likely have some merge/cherry-pick conflict here, fix them and commit:

```
git commit -am 'Backport PR #55821: BUG: Index.isin raising for arrow strings and null set'
```

4. Push to a named branch:

```
git push YOURFORK 2.1.x:auto-backport-of-pr-55821-on-2.1.x
```

5. Create a PR against branch 2.1.x, I would have named this PR:

> "Backport PR #55821 on branch 2.1.x (BUG: Index.isin raising for arrow strings and null set)"

And apply the correct labels and milestones.

Congratulations — you did some good work! Hopefully your backport PR will be tested by the continuous integration and merged soon!

Remember to remove the `Still Needs Manual Backport` label once the PR gets merged.

If these instructions are inaccurate, feel free to [suggest an improvement](https://github.com/MeeseeksBox/MeeseeksDev).
Backport PR #55761 on branch 2.1.x (BUG: DatetimeIndex.diff raising TypeError)
State: closed


Labels:
name: Bug
name: Index
description: Related to the Index class or subclasses

Comments:
@mroeschke - were you planning to backport anything from #55524? It looks like some of the failures here (in 2.1.x) could use some of the updates you made in that PR. 

e.g `test_parse_tz_aware` is failing due to a units mismatch and it looks like you added an explicit cast to nanoseconds in that PR:

https://github.com/pandas-dev/pandas/blob/32b8cd2c81108dc51d59d2d09287df29d40d255b/pandas/tests/io/parser/test_parse_dates.py#L1005


Ah I see. Definitely should not backport the dependency bumps, but potentially 2.1.x might not have pyarrow pinned like the main branch was and some of these code changes could be ported over there. 

But the failures are not related to this change right?
> But the failures are not related to this change right?

Correct, the failures look unrelated
TST: parametrize over dt64 unit
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Non-Nano
description: datetime64/timedelta64 with non-nanosecond resolution

Comments:
Thanks @jbrockmendel
PERF: IndexEngine.get_indexer_non_unique
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/v2.2.0.rst` file if fixing a bug or adding a new feature.

maybe(?) closes #15364

When resizing result array, grow by factor of 2 rather than fixed amount.

```
import pandas as pd
import numpy as np

idx = pd.Index(np.ones(1_000_000))
target = pd.Index([1])

%timeit idx.get_indexer_for(target)

# 968 ms ± 35.9 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)     -> main
# 73.1 ms ± 2.18 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)  -> PR
```


```
import pandas as pd
import numpy as np

idx = pd.Index(np.arange(1_000_000).tolist() + [0])
target = idx[:-1]

%timeit idx.get_indexer_for(target)

2.17 s ± 22.5 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)  -> main
1.18 s ± 27.1 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)  -> PR
```

Labels:
name: Performance
description: Memory or execution speed performance
name: Index
description: Related to the Index class or subclasses

Comments:
How would performance be if scaling followed 10000 + 2^n where 2^n was about 10000 on the first iteration?
> How would performance be if scaling followed 10000 + 2^n where 2^n was about 10000 on the first iteration?

I tried 10000 + 2^n starting with n=13 (8192) which produced the following timings for the two examples above:

```
79.7 ms ± 3.12 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)
1.34 s ± 71.6 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
```

Pretty close to the timings in this PR. Let me know if you prefer to use this.

If the resizing is the expensive operation here, I think this would be generically more performant for larger cases too so yes
If I understand correctly, I think your suggestion is very close to the current PR as they both end up growing by a factor of 2:

```
In [1]: import pandas as pd

In [2]: pd.DataFrame(
   ...:     {
   ...:         "main": [(n+1) * 10_000 for n in range(10)],
   ...:         "PR": [10_000 * 2**n for n in range(10)],
   ...:         "proposed": [10_000] + [10_000 + 2**(13+n) for n in range(9)]
   ...:     }
   ...: )
Out[2]: 
     main       PR  proposed
0   10000    10000     10000
1   20000    20000     18192
2   30000    40000     26384
3   40000    80000     42768
4   50000   160000     75536
5   60000   320000    141072
6   70000   640000    272144
7   80000  1280000    534288
8   90000  2560000   1058576
9  100000  5120000   2107152
```

Am I understanding your suggestion correctly?

Thanks @lukemanley
DOC: Add "nullable float" link in "dtypes" table of User Guide
State: closed

The formatting of the "nullable float" entry in [this table](https://pandas.pydata.org/docs/user_guide/basics.html#dtypes) was inconsistent with the other entries. As far as I can tell, there is not (currently) a "nullable float" section of the User Guide, so I've added a reference to the API, as is the case for the table's "Boolean (with NA)" entry.

Labels:
name: Docs

Comments:
Thanks @dpoznik
BUG: DatetimeIndex with dayfirst/yearfirst and tz
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

surfaced while implementing #55564

Labels:
name: Bug
name: Timezones
description: Timezone data dtype

Comments:
Thanks @jbrockmendel
BUG: `assert_extension_array_equal` using the wrong dtype
State: closed

- [x] closes #55730 
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added an entry in the latest `doc/source/whatsnew/v2.1.3.rst` file if fixing a bug or adding a new feature.


Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Non-Nano
description: datetime64/timedelta64 with non-nanosecond resolution

Comments:
Thanks @rob-sil
Added code for QuarterBegin class to handle the "_period_dtype_code" attribute
State: closed


Labels:

Comments:
thanks for your PR but I think this should raise, QuarterEnd is already used to create Periods, and Periods don't distinguish between "start" and "end" anyway. It's an implementation detail which freq is used under the hood, the user would just pass `period='Q'`

closing then, but thanks for your PR!
ENH: Allow passing `read_only`, `data_only` and `keep_links` arguments to openpyxl using `engine_kwargs`
State: closed

Previously it was not possible to override the default values for `openpyxl.reader.excel.load_workbook`'s `read_only`, `data_only` and `keep_links` arguments via `pandas.read_excel`'s `engine_kwargs` (see #55027).

Now these options can be changed via `engine_kwargs`.


- [x] closes #55027
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: IO Excel
description: read_excel, to_excel

Comments:
> I think this is a solid improvement over the current implementation, but it is still odd in relation to the other readers. I believe openpyxl is the only reader where we pass non-default values to the load function of the engine. Still, the values we pass seem like they would be the preferred values in most cases to me. Perhaps it's sufficient to document this oddity?

Thanks for your feedback. I don't find it particularly odd that pandas passes non-default arguments to openpyxl. I think these tools just have somewhat different use cases in mind. Openpyxl seems to focus on modifying excel files, so it makes sense that by default they open files with `read_only=False, data_only=False, keep_links=True`, where the latter two are required to not break formulas and links when the modified file is saved. For `pandas.io.excel._openpyxl.OpenpyxlReader.load_workbook` it is only about reading, so choosing the supposedly faster `read_only=True` seems plausible (although `read_only=False` seems slightly more robust, e.g. #39001 but also other cases where the results differ). Also external links are irrelevant, and most users would probably expect to get the results of any formulas (`data_only=True`) rather than the formulas themselves (although with the proposed changes they *can* get them if they wanted to).
Thanks @dweindl
ENH/BUG: infer reso in array_strptime
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Non-Nano
description: datetime64/timedelta64 with non-nanosecond resolution

Comments:
Thanks @jbrockmendel