BrandonFors/pandas-github-issues huggingface dataset:
DEPR: is_period_dtype, is_sparse
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

is_sparse is particularly non-performant bc it uses a runtime import

Labels:
name: Deprecate
description: Functionality to remove in pandas

Comments:
thx @jbrockmendel
CI: Skip testing step in Pypy job
State: closed

@lithomas1 during today's dev call, I asked if we still need the Pypy build since it's actively failing (but passing in the CI) and no one is really ensuring we have compat with Pypy. Thought if we can remove it?

Labels:
name: CI
description: Continuous Integration
name: PyPy

Comments:
Can you just remove the testing phase instead?

We still build PyPy packages on conda (and pandas has failed to compile on PyPy come release time at least once).
The validate min versions script errors on the PyPy build, should investigate if we keep the file around
Sure, let's just skip the testing steps of the Pypy jobs instead

But IIRC it doesn't look like our cibuildwheel is building for pypy which uploads to conda?

https://github.com/pandas-dev/pandas/blob/81b08a9581a72545bee2d3026d3cf5d030ff2223/.github/workflows/wheels.yml#L58
cc @cfbolz we (pandas) don't have anyone championing pypy support.  Can you suggest anyone from the pypy team who might be interested?
> Sure, let's just skip the testing steps of the Pypy jobs instead
> 
> But IIRC it doesn't look like our cibuildwheel is building for pypy which uploads to conda?
> 
> https://github.com/pandas-dev/pandas/blob/81b08a9581a72545bee2d3026d3cf5d030ff2223/.github/workflows/wheels.yml#L58

Although we upload PyPI wheels to Anaconda, we upload them to Anaconda's PyPI repositories, not the conda ones.

The conda packages are built separately by the pandas-feedstock repo, and the packages are built from the Github release tarball.
https://github.com/conda-forge/pandas-feedstock/blob/76269c33f5865e0642232cf9c73fdefb7f9412a7/recipe/meta.yaml#L8
Refactored custom datetime functions
State: closed

Takes some of our custom code out of the vendored numpy module and puts it in a capsule. date_conversions.c is custom and could argue it should entirely be put into the capsule, but didn't want to do that here.

The main motiviation is to make the work @jbrockmendel started in https://github.com/pandas-dev/pandas/pull/51368/ possible


Labels:
name: Refactor
description: Internal refactoring of code
name: Internals
description: Related to non-user accessible pandas implementation

Comments:
Thanks @WillAyd 
@jbrockmendel do you want to try #51368 again? I think the code you had there should translate perfectly into this
My current thought/hope is that numpy will merge https://github.com/numpy/numpy/pull/21199 and we'll get this function from them. If that falls through then I'll revisit #51368.
Sounds good. Just keep in mind with that approach you won't ever be able to do what you were trying to do with using the DateTime API directly, so will just stick with the duck typing
BUG: Logical and comparison ops with ArrowDtype & masked
State: closed

- [x] closes #52625 (Replace xxxx with the GitHub issue number)
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: NA - MaskedArrays
description: Related to pd.NA and nullable extension arrays
name: Arrow
description: pyarrow functionality

Comments:
Owee, I'm MrMeeseeks, Look at me.

There seem to be a conflict, please backport manually. Here are approximate instructions:

1. Checkout backport branch and update it.

```
git checkout 2.0.x
git pull
```

2. Cherry pick the first parent branch of the this PR on top of the older branch:
```
git cherry-pick -x -m1 c85bbc6913b9c4d2acc53c54cc858c84bf0658d1
```

3. You will likely have some merge/cherry-pick conflict here, fix them and commit:

```
git commit -am 'Backport PR #52633: BUG: Logical and comparison ops with ArrowDtype & masked'
```

4. Push to a named branch:

```
git push YOURFORK 2.0.x:auto-backport-of-pr-52633-on-2.0.x
```

5. Create a PR against branch 2.0.x, I would have named this PR:

> "Backport PR #52633 on branch 2.0.x (BUG: Logical and comparison ops with ArrowDtype & masked)"

And apply the correct labels and milestones.

Congratulations — you did some good work! Hopefully your backport PR will be tested by the continuous integration and merged soon!

Remember to remove the `Still Needs Manual Backport` label once the PR gets merged.

If these instructions are inaccurate, feel free to [suggest an improvement](https://github.com/MeeseeksBox/MeeseeksDev).
PERF: read_csv macro updates
State: closed

This gives us a few percentage points back in read_csv performance, especially with larger files. Credit to @Dr-Irv for the idea


Labels:
name: Performance
description: Memory or execution speed performance
name: IO CSV
description: read_csv, to_csv

Comments:
If anyone cared for the details here is the verbose annotated assembly output (from `-fverbose-asm` gcc flag) for before:

```asm
.L246:
# pandas/_libs/src/parser/tokenizer.c:929:                 } else if (IS_DELIMITER(c)) {
	.loc 3 929 28 is_stmt 1
	movq	-136(%rbp), %rax	# self, tmp683
	movl	188(%rax), %eax	# self_452(D)->delim_whitespace, _180
# pandas/_libs/src/parser/tokenizer.c:929:                 } else if (IS_DELIMITER(c)) {
	.loc 3 929 27
	testl	%eax, %eax	# _180
	jne	.L247	#,
# pandas/_libs/src/parser/tokenizer.c:929:                 } else if (IS_DELIMITER(c)) {
	.loc 3 929 28 discriminator 1
	movq	-136(%rbp), %rax	# self, tmp684
	movzbl	184(%rax), %eax	# self_452(D)->delimiter, _181
	cmpb	%al, -49(%rbp)	# _181, c
	je	.L248	#,
.L247:
# pandas/_libs/src/parser/tokenizer.c:929:                 } else if (IS_DELIMITER(c)) {
	.loc 3 929 28 is_stmt 0 discriminator 3
	movq	-136(%rbp), %rax	# self, tmp685
	movl	188(%rax), %eax	# self_452(D)->delim_whitespace, _182
	testl	%eax, %eax	# _182
	je	.L249	#,
# pandas/_libs/src/parser/tokenizer.c:929:                 } else if (IS_DELIMITER(c)) {
	.loc 3 929 28 discriminator 4
	call	__ctype_b_loc@PLT	#
	movq	(%rax), %rdx	# *_183, _184
	movsbq	-49(%rbp), %rax	# c, _185
	addq	%rax, %rax	# _186
	addq	%rdx, %rax	# _184, _187
	movzwl	(%rax), %eax	# *_187, _188
	movzwl	%ax, %eax	# _188, _189
	andl	$1, %eax	#, _190
	testl	%eax, %eax	# _190
	je	.L249	#,
.L248:
# pandas/_libs/src/parser/tokenizer.c:930:                     if (self->delim_whitespace) {
```

and after

```asm
.L243:
# pandas/_libs/src/parser/tokenizer.c:930:                 } else if (IS_DELIMITER(c)) {
	.loc 3 930 27 is_stmt 1
	movzbl	-57(%rbp), %eax	# c, tmp657
	cmpb	-41(%rbp), %al	# delimiter, tmp657
	je	.L244	#,
# pandas/_libs/src/parser/tokenizer.c:930:                 } else if (IS_DELIMITER(c)) {
	.loc 3 930 28 discriminator 1
	cmpl	$0, -40(%rbp)	#, delim_whitespace
	je	.L245	#,
# pandas/_libs/src/parser/tokenizer.c:930:                 } else if (IS_DELIMITER(c)) {
	.loc 3 930 28 is_stmt 0 discriminator 2
	call	__ctype_b_loc@PLT	#
	movq	(%rax), %rdx	# *_171, _172
	movsbq	-57(%rbp), %rax	# c, _173
	addq	%rax, %rax	# _174
	addq	%rdx, %rax	# _172, _175
	movzwl	(%rax), %eax	# *_175, _176
	movzwl	%ax, %eax	# _176, _177
	andl	$1, %eax	#, _178
	testl	%eax, %eax	# _178
	je	.L245	#,
.L244:
# pandas/_libs/src/parser/tokenizer.c:931:                     if (self->delim_whitespace) {
```

Very low level...but this is executed for potentially every character in a file
Can you add a whatsnew?
Thanks @WillAyd
REGR: pivot changing index name of input object
State: closed

- [x] closes #52629 (Replace xxxx with the GitHub issue number)
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode
name: Regression
description: Functionality that used to work in a prior pandas version

Comments:
Thanks @phofl
DEPR: unused keywords in DTI/TDI construtors
State: closed

- [x] closes #49153 (Replace xxxx with the GitHub issue number)
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Deprecate
description: Functionality to remove in pandas

Comments:
Thanks @jbrockmendel
TYP: remove ignore from pandas/tseries/frequencies.py
State: closed

Related to #37715
mypy ignore[assignment] was removed from pandas/tseries/frequencies.py

Labels:
name: Frequency
description: DateOffsets
name: Typing
description: type annotations, mypy/pyright type checking

Comments:
@twoertwein, could you please take a look at my last changes? Looks like there are no failures in ci, but the last check `Doc Build and Upload` is still in progress for about 6 hours.
@natmokval can you merge main
> @natmokval can you merge main

@jbrockmendel, I can merge main but these changes are in main already (see pr #53120). So I think it's better to close this pr. Sorry, I had to do it earlier.
REGR: DataFrame.resample fails on a frame with no columns
State: closed

- [x] closes #52484 (Replace xxxx with the GitHub issue number)
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Regression
description: Functionality that used to work in a prior pandas version
name: Resample
description: resample method

Comments:
Thanks @rhshadrach 
Owee, I'm MrMeeseeks, Look at me.

There seem to be a conflict, please backport manually. Here are approximate instructions:

1. Checkout backport branch and update it.

```
git checkout 2.0.x
git pull
```

2. Cherry pick the first parent branch of the this PR on top of the older branch:
```
git cherry-pick -x -m1 e44daed32234b1afd263e5880a7e743f31218ad1
```

3. You will likely have some merge/cherry-pick conflict here, fix them and commit:

```
git commit -am 'Backport PR #52615: REGR: DataFrame.resample fails on a frame with no columns'
```

4. Push to a named branch:

```
git push YOURFORK 2.0.x:auto-backport-of-pr-52615-on-2.0.x
```

5. Create a PR against branch 2.0.x, I would have named this PR:

> "Backport PR #52615 on branch 2.0.x (REGR: DataFrame.resample fails on a frame with no columns)"

And apply the correct labels and milestones.

Congratulations — you did some good work! Hopefully your backport PR will be tested by the continuous integration and merged soon!

Remember to remove the `Still Needs Manual Backport` label once the PR gets merged.

If these instructions are inaccurate, feel free to [suggest an improvement](https://github.com/MeeseeksBox/MeeseeksDev).
ENH: Implement more str accessor methods for ArrowDtype
State: closed

Most of these don't have corresponding pyarrow compute methods

Labels:
name: Strings
description: String extension data type and string data
name: Arrow
description: pyarrow functionality

Comments:
In general, it's often best to avoid iterating over pyarrow arrays (wrapping each array element in a pyarrow Scalar, and then converting each individually to a python object with `as_py()` has quite some overhead). Typically, first converting the whole array to numpy and iterating over the numpy array can be more efficient:

```
import string, random

In [23]: arr = pa.array(["".join(random.choices(string.ascii_letters, k=5)) for _ in range(1_000_000)])

In [24]: %timeit [val.as_py() for val in arr]
517 ms ± 10.1 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

In [25]: %timeit [val for val in arr.to_numpy(zero_copy_only=False)]
80.7 ms ± 1.17 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)
```

Wrapping in and converting the scalar has quite some overhead, while the conversion to numpy is better optimized and happens in one go at a lower level.
Going to merge this in, I can address any followups if needed.
Owee, I'm MrMeeseeks, Look at me.

There seem to be a conflict, please backport manually. Here are approximate instructions:

1. Checkout backport branch and update it.

```
git checkout 2.0.x
git pull
```

2. Cherry pick the first parent branch of the this PR on top of the older branch:
```
git cherry-pick -x -m1 fbcbdaf70da354801d99a1cebb889153a3eca481
```

3. You will likely have some merge/cherry-pick conflict here, fix them and commit:

```
git commit -am 'Backport PR #52614: ENH: Implement more str accessor methods for ArrowDtype'
```

4. Push to a named branch:

```
git push YOURFORK 2.0.x:auto-backport-of-pr-52614-on-2.0.x
```

5. Create a PR against branch 2.0.x, I would have named this PR:

> "Backport PR #52614 on branch 2.0.x (ENH: Implement more str accessor methods for ArrowDtype)"

And apply the correct labels and milestones.

Congratulations — you did some good work! Hopefully your backport PR will be tested by the continuous integration and merged soon!

Remember to remove the `Still Needs Manual Backport` label once the PR gets merged.

If these instructions are inaccurate, feel free to [suggest an improvement](https://github.com/MeeseeksBox/MeeseeksDev).
DEPR: concat ignoring all-NA columns
State: closed

- [x] closes #40893 (Replace xxxx with the GitHub issue number)
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Missing-data
description: np.nan, pd.NaT, pd.NA, dropna, isnull, interpolate
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode
name: Deprecate
description: Functionality to remove in pandas

Comments:
Thanks @jbrockmendel
DEPR: is_datetime64tz_dtype, is_interval_dtype
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Deprecate
description: Functionality to remove in pandas

Comments:
Thanks @jbrockmendel
BUG: to_numeric(errors='coerce', dtype_backend='pyarrow') with ArrowDtype
State: closed

- [x] closes #52588 (Replace xxxx with the GitHub issue number)
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Arrow
description: pyarrow functionality

Comments:
Owee, I'm MrMeeseeks, Look at me.

There seem to be a conflict, please backport manually. Here are approximate instructions:

1. Checkout backport branch and update it.

```
git checkout 2.0.x
git pull
```

2. Cherry pick the first parent branch of the this PR on top of the older branch:
```
git cherry-pick -x -m1 471c852b3c7c665e4d45b3109cab2b79582a8f6d
```

3. You will likely have some merge/cherry-pick conflict here, fix them and commit:

```
git commit -am "Backport PR #52606: BUG: to_numeric(errors='coerce', dtype_backend='pyarrow') with ArrowDtype"
```

4. Push to a named branch:

```
git push YOURFORK 2.0.x:auto-backport-of-pr-52606-on-2.0.x
```

5. Create a PR against branch 2.0.x, I would have named this PR:

> "Backport PR #52606 on branch 2.0.x (BUG: to_numeric(errors='coerce', dtype_backend='pyarrow') with ArrowDtype)"

And apply the correct labels and milestones.

Congratulations — you did some good work! Hopefully your backport PR will be tested by the continuous integration and merged soon!

Remember to remove the `Still Needs Manual Backport` label once the PR gets merged.

If these instructions are inaccurate, feel free to [suggest an improvement](https://github.com/MeeseeksBox/MeeseeksDev).
                
@meeseeksmachine backport 2.0.x
@meeseeksdev backport 2.0.x
TST: groupby string dtype
State: closed

- [x] closes #40148
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Testing
description: pandas testing functions or related to the test suite

Comments:
Thanks @srkds 
Thanks @mroeschke for the review 🙏
DOC Added that DatetimeIndex needs to be ordered on first()
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [X] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Docs

Comments:
Thank you
BUG: pd.array raising with NumPy array and large dtype
State: closed

- [x] closes #52590 (Replace xxxx with the GitHub issue number)
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

not sure if we want to backport this though...

Labels:
name: Strings
description: String extension data type and string data
name: Arrow
description: pyarrow functionality

Comments:
This should be ready
Thanks @phofl
DEPR: use DeprecationWarning for Categorical fastpath deprecation
State: closed

Follow-up on https://github.com/pandas-dev/pandas/pull/52472, changing the warning type. I expect this fastpath to be used in libraries (at least pyarrow did), and first using DeprecationWarning avoids that users see this warning if it's not coming from their own code (i.e. avoids warnings they can't do anything about).

Labels:
name: Deprecate
description: Functionality to remove in pandas

Comments:
Nice catch, thanks @jorisvandenbossche
CI: Consolidate comment commands GHA
State: closed

Avoids creating a whole other workflow run just to check another type of comment

Labels:
name: CI
description: Continuous Integration

Comments:
Owee, I'm MrMeeseeks, Look at me.

There seem to be a conflict, please backport manually. Here are approximate instructions:

1. Checkout backport branch and update it.

```
git checkout 2.0.x
git pull
```

2. Cherry pick the first parent branch of the this PR on top of the older branch:
```
git cherry-pick -x -m1 5ab10a3af76186f3908dd0816688f33d30cc43fc
```

3. You will likely have some merge/cherry-pick conflict here, fix them and commit:

```
git commit -am 'Backport PR #52587: CI: Consolidate comment commands GHA'
```

4. Push to a named branch:

```
git push YOURFORK 2.0.x:auto-backport-of-pr-52587-on-2.0.x
```

5. Create a PR against branch 2.0.x, I would have named this PR:

> "Backport PR #52587 on branch 2.0.x (CI: Consolidate comment commands GHA)"

And apply the correct labels and milestones.

Congratulations — you did some good work! Hopefully your backport PR will be tested by the continuous integration and merged soon!

Remember to remove the `Still Needs Manual Backport` label once the PR gets merged.

If these instructions are inaccurate, feel free to [suggest an improvement](https://github.com/MeeseeksBox/MeeseeksDev).
                
thx @mroeschke 
Actually I don't think we need to backport this; the `main` config is relevant for the workflows