BrandonFors/pandas-github-issues huggingface dataset:
BUG: low variance arrays' kurtosis wrongfully set to zero
State: closed

- [x] closes #57972 (Replace xxxx with the GitHub issue number)
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

This fixes an issue with low variance arrays' kurtosis. It was previously set to zero due to a cutoff that was chosen too conservatively.

Labels:
name: Stale

Comments:
This pull request is stale because it has been open for thirty days with no activity. Please [update](https://pandas.pydata.org/pandas-docs/stable/development/contributing.html#updating-your-pull-request) and respond to this comment if you're still interested in working on this.
Thanks for the pull request, but it appears to have gone stale. If interested in continuing, please merge in the main branch, address any review comments and/or failing tests, and we can reopen.
TST: Remove deprecation check from test_pyarrow_engine
State: closed

- [x] ref https://github.com/pandas-dev/pandas/issues/57966#issuecomment-2041619807_
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: IO CSV
description: read_csv, to_csv

Comments:
Thanks @Aloqeely
ENH: add numeric_only to Dataframe.cum* methods
State: closed

- [x] closes #53072 
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Enhancement
name: Reduction Operations
description: sum, mean, min, max, etc.

Comments:
Thanks @weikhor
REF: de-duplicate tzinfo-awareness mismatch checks
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Timezones
description: Timezone data dtype
name: Clean

Comments:
Thanks @jbrockmendel
BUG: fix aggregation when using udf with kwarg
State: closed

- [x] closes #58146
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ Not applicable ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ Pending review ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

closes https://github.com/pandas-dev/pandas/issues/39169


Labels:
name: Groupby
name: Apply
description: Apply, Aggregate, Transform, Map

Comments:
https://github.com/pandas-dev/pandas/issues/58146#issuecomment-2039750872

Noted comment that the changes to agg will be breaking changes and I would need a workaround for #58132. This is fairly important as adding a `numeric_only` kwarg will more often push the code down `DataFrameGroupBy._aggregate_frame`. 

I was hoping to get your view on this PR as a temporary fix with minimalist code, it changes the path the code will take to use `GroupByApply.agg` as a list if there is more than one column to operate on - hence the aggregation done as loop of `SeriesGroupBy` if I understood the code correctly.
Could you add a whatsnew note to `v3.0.0.rst` (under `Bug FIx` in the groupby section)?
ENH: New proposed deprecation policy
State: closed

xref #54970

Use `DeprecationWarning` until the last minor version before a major release then we switch to `FutureWarning`

Labels:

Comments:
One thing to note is that when using `pytest.mark.filterwarnings` I have kept the ignore string as `"ignore::DeprecationWarning"` which we will need to change to `FutureWarning` once we change `Pandas40DeprecationWarning`, my reasoning for this is that you have to pass the module path (so `ignore::pandas.util._exceptions.Pandas40DeprecationWarning`) for aliased warnings, which didn't look pretty to me
This pull request is stale because it has been open for thirty days with no activity. Please [update](https://pandas.pydata.org/pandas-docs/stable/development/contributing.html#updating-your-pull-request) and respond to this comment if you're still interested in working on this.
Gentle ping @lithomas1. I would like this to move forward so I don't have to update it every time a new deprecation is added
> Gentle ping @lithomas1. I would like this to move forward so I don't have to update it every time a new deprecation is added

Thanks for the reminder (and sorry that this slipped off my radar).
Will take a look this evening.
One thing we also could consider doing is decoupling the test changes from this PR, and just introducing ``Pandas40DeprecationWarning``, and basic tests that it works as expected.

That way there's less chances of this creating conflicts and getting out of sync.
I think switching to Future warning only one release before the release that will break is too short.  My motion on this is that it would be only 6 months or so, and sometimes less.  Often end users report issues to projects through future warnings. This then requires downstream to make the changes and make a release before the next pandas, or face a break (unless strictly capping, which is not common). 

Producing a FutureWarning 2 releases prior to breaking things seems like a more balanced approach.  
> Producing a FutureWarning 2 releases prior to breaking things seems like a more balanced approach.

That would mean any deprecations released in the last 2 minor releases will immediately be FutureWarnings which means downstream libraries won't have *any* time to handle/suppress these warnings. Had we implemented your idea earlier, v2.1 deprecations would immediately be FutureWarning (there are a lot of them, see #50578), one of them being the cause of #54970 being opened in the first place.

I guess if we think some deprecation needs attention by end users, we can leave it as a FutureWarning and postpone enforcing the deprecation to a later major release.
@Aloqeely we had a discussion about our deprecation policy at the bi-weekly developers meeting, and we think that a PDEP should be created that would formalize the policy and allow for greater discussion.  Are you willing to write up a PDEP with a proposed policy?
Sure, I can give it a go.
Going to close this for now
No longer produce test.tar/test.zip after running test suite
State: closed

- [x] closes #58162 (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Testing
description: pandas testing functions or related to the test suite

Comments:
Thanks @Aloqeely
DEPR: enforce Sparse deprecations
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Clean

Comments:
Thanks @jbrockmendel
PERF: more realistic np datetime c benchmark
State: closed

See my comments in #57951. This change in the benchmark input range has more coverage of the different logic branches in `np_datetime.c`. The benchmark should run about 1.8x as long (most likely a little bit machine dependent) after merging this.

Keeping this separate from my other PR #57988 that improves the `np_datetime.c`performance to distinguish the benchmark impacts.

@WillAyd 


Labels:
name: Benchmark
description: Performance (ASV) benchmarks

Comments:
This pull request is stale because it has been open for thirty days with no activity. Please [update](https://pandas.pydata.org/pandas-docs/stable/development/contributing.html#updating-your-pull-request) and respond to this comment if you're still interested in working on this.
Thanks @dontgoto
DOC: Add notes to nullable types documentation about pd.NA column type
State: closed

- [X] closes #49201 
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

Labels:
name: Docs
name: Missing-data
description: np.nan, pd.NaT, pd.NA, dropna, isnull, interpolate
name: NA - MaskedArrays
description: Related to pd.NA and nullable extension arrays

Comments:
Should I change something else to be merged? Thanks!
Thanks @giormala!
DEPR: PandasArray alias
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Clean

Comments:
Thanks @jbrockmendel
DEPR: Categorical fastpath
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Clean

Comments:
Merge conflict otherwise LGTM
Thanks @jbrockmendel
BUG: Fix read_parquet not working with data type pyarrow list (#57411)
State: closed

- [x] closes #57411
- [x] [Tests added and passed]
- [x] All [code checks passed]
- [ ] Added [type annotations]
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

When writing a DataFrame to a Parquet file, if the data type of the columns of the DataFrame is a PyArrow list, an error is raised when reading it from the Parquet file.

The error occurs because when reading, pandas tries to use NumPy dtypes which do not support PyArrow lists (in the function pandas_dtype in the file ./pandas/core/dtypes/common.py).

To solve this issue, we add an if statement to check if the column's data type is a PyArrow list. If so, we return the data type without converting it into a NumPy dtype (which does not support PyArrow lists).

Labels:
name: Dtype Conversions
description: Unexpected or buggy dtype conversions
name: Arrow
description: pyarrow functionality

Comments:
Thanks for the PR
Thanks for the feedback.
I will try to make some changes and afterward I will update my pull request.
Thanks for the pull request, but it appears to have gone stale. If interested in continuing, please merge in the main branch, address any review comments and/or failing tests, and we can reopen.
ENH: Add support for reading value labels from 108-format and prior Stata dta files
State: closed

- [x] closes #58154
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

This change extends support for reading value labels to Stata 108-format (Stata 6) and earlier dta files.

Labels:
name: IO Stata
description: read_stata, to_stata

Comments:
cc @bashtage
One of the type checks fails because in various places the value table codes are expected to be floats (dict[float, str]) whereas in fact they are stored as ints (dict[int, str]). I think there are two options to make this work:

1. Cast the codes to float when read and update the newly added type hints to match elsewhere, but this would break anything that relies on the existing behaviour
2. Change the existing type hints to expect int keys, but this might not match the intent of the original author
As for typechecking, you can probably need to replace `self._value_label_dict[labname][val[i]]`  with `self._value_label_dict[labname][int(val[i])]` since it will be hard to convince the type checker that `val` is a 1-d array of int which would be needed to let it know that `val[i]` was an int.

> As for typechecking, you can probably need to replace `self._value_label_dict[labname][val[i]]` with `self._value_label_dict[labname][int(val[i])]` since it will be hard to convince the type checker that `val` is a 1-d array of int which would be needed to let it know that `val[i]` was an int.

It seems to be complaining about a mismatch with lines like `def value_labels(self) -> dict[str, dict[float, str]]:` elsewhere in the file. I think I could make it happy by changing the type hints I added to match this, but then this wouldn't match the actual content of the data.
> Some comments on what it here. Overall I think the cyclomatic complexity has gotten too high and I think it would be better to pull these up as two private methods, one for reading 108+ and the other for older formats. Would let you avoid a long `if...else...` clause.

I did wonder about splitting it into functions, but didn't want to change too many lines. I am happy to have a go at implementing this.
Thanks @cmjcharlton
BUG: Fix Index.sort_values with natsort_key raises
State: closed

- [x] closes #56081 (Replace xxxx with the GitHub issue number)
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Index
description: Related to the Index class or subclasses
name: Sorting
description: e.g. sort_index, sort_values

Comments:
An explanation of the fix for this case:
In L570, result is a tuple of tuple. Then it is converted to a MultiIndex in L580, which causes the error.
https://github.com/pandas-dev/pandas/blob/f4232e7cdefcc33afe184891f833aec9c3c9bac2/pandas/core/sorting.py#L570-L580
Thanks @yuanx749
REGR: from_records not initializing subclasses properly
State: closed

- [ ] closes #57008 (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


cc @jbrockmendel would appreciate thoughts, we don't have access to _constructor_from_mgr, so we need a workaround somehow.

Labels:
name: Regression
description: Functionality that used to work in a prior pandas version
name: Stale

Comments:
This pull request is stale because it has been open for thirty days with no activity. Please [update](https://pandas.pydata.org/pandas-docs/stable/development/contributing.html#updating-your-pull-request) and respond to this comment if you're still interested in working on this.
Closing to clear the queue, but feel free to update and reopen anytime
REF: Allow Index._with_infer to also return RangeIndex
State: closed

Would supersede https://github.com/pandas-dev/pandas/pull/58117

I see 8 usages of `Index._with_infer` in the codebase. There are some ops like `groupby` and `CategoricalDtype` that would nicely return a `RangeIndex` result if possible

Labels:
name: Index
description: Related to the Index class or subclasses

Comments:
This may be too aggressive so closing for now
BUG: fixed to_numeric loss in precision when converting decimal type to integer #57213
State: closed

- [X] closes #57213
- [X] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [X] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [X] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions. _(Not applicable)_
- [X] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Dtype Conversions
description: Unexpected or buggy dtype conversions

Comments:
Thanks for the pull request, but it appears to have gone stale. If interested in continuing, please merge in the main branch, address any review comments and/or failing tests, and we can reopen.
BLD: Fix nightlies not building
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Build
description: Library building on various platforms

Comments:
I forgot to delete the other build condition when removing the special build step for numpy 2.0 in my previous PR.
Thanks @lithomas1
ENH: Processing of .mask() for pd.NA #56844
State: closed

- [ ] closes #56844
- [ ] [Tests added and passed]
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

This commit addresses an issue where using NA in conjunction with the mask() method that resulted in unexpected behavior. The problem arose when comparing a Series containing NA as a condition within mask(), causing inconsistencies in the output. This fix ensures that NA behaves consistently as False in logical operations within mask().


Labels:
name: Missing-data
description: np.nan, pd.NaT, pd.NA, dropna, isnull, interpolate

Comments:
Thanks for the pull request, but it appears to have gone stale. If interested in continuing, please merge in the main branch, address any review comments and/or failing tests, and we can reopen.
I am interested in continuing. Should I make another pull request with the merges? @mroeschke 
Yes that would be preferred
DOC: Add return value for Index.slice_locs when label not found #32680
State: closed

- [ ] Add return value for Index.slice_locs when label not found  #32680  
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Docs

Comments:
Thanks @KeiOshima