BrandonFors/pandas-github-issues huggingface dataset:
ENH: Add check for character limmit
State: closed

- [x] closes #56954  (Replace xxxx with the GitHub issue number)
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Enhancement
name: IO Excel
description: read_excel, to_excel
name: Warnings
description: Warnings that appear or should be added to pandas

Comments:
As discussed in the context of https://github.com/pandas-dev/pandas/issues/56954#issuecomment-1911937560, we have incorporated a validation for the maximum row/column number within the code. https://github.com/pandas-dev/pandas/blob/9008ee5810c09bc907b5fdc36fc3c1dff4a50c55/pandas/io/formats/excel.py#L929-L934

Furthermore, I have introduced a character limit check and included corresponding notifications in the documentation.
The CI test failure appears to be linked to the changes made in the pull request, but I'm unable to reproduce it on my local machine (WSL2). Assistance may be needed.
> The CI test failure appears to be linked to the changes made in the pull request

Caused by warnings from other packages, and should be resolved by incorporating  `raise_on_extra_warnings=False`


Thanks @luke396!
ENH: Add skipna to groupby.first and groupby.last
State: closed

- [x] closes #57019 (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

There is a case to put this in 2.2.1 as the ability to agg the first element from every group including NA values was removed by changing the behavior of nth in pandas 2.0.0. But also okay to put this in 3.0.

Labels:
name: Enhancement
name: Groupby
name: Regression
description: Functionality that used to work in a prior pandas version
name: Reduction Operations
description: sum, mean, min, max, etc.

Comments:
Thanks @rhshadrach
REGR: Index.join raising TypeError when joining an empty index to a mixed type index
State: closed

- [x] closes #57048
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/v2.2.1.rst` file if fixing a bug or adding a new feature.


Labels:
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode
name: Regression
description: Functionality that used to work in a prior pandas version

Comments:
Thanks @lukemanley
DOCS: DataFrame.any() and DataFrame.all() documentation specifies the…
State: closed

… wrong return types

- [x] closes #57088  
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Docs
name: Stale
name: Reduction Operations
description: sum, mean, min, max, etc.

Comments:
This pull request is stale because it has been open for thirty days with no activity. Please [update](https://pandas.pydata.org/pandas-docs/stable/development/contributing.html#updating-your-pull-request) and respond to this comment if you're still interested in working on this.
Thanks for the pull request, but it appears to have gone stale. If interested in continuing, please merge in the main branch, address any review comments and/or failing tests, and we can reopen.
Remove parts that reference bfill,ffill,pad,backfill in limit_direction
State: closed

This PR is related to the issue #57092. It removes the parts that reference `bfill`, `ffill`, `pad`, and `backfill` as allowable methods under limit_direction as they are not supported.

- [x] closes #57092 
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Docs
name: Missing-data
description: np.nan, pd.NaT, pd.NA, dropna, isnull, interpolate

Comments:
Thanks for the pull request, but it appears to have gone stale. If interested in continuing, please merge in the main branch, address any review comments and/or failing tests, and we can reopen.
CVE 2020 13091
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:

Comments:
Thanks for the PR, but as mentioned in https://github.com/pandas-dev/pandas/issues/48049 this is a CPython issue and addressing it in pandas is a won't fix so closing.
Sorry, this was supposed to merge 2.2.x into my fork not the other way around. It might be GitHub GUI; I'll try command line.
Remove support for `py.path` objects in IO readers
State: closed

This library is in maintenance mode and recommends `pathlib` objects so removing support for this object in IO methods.

https://py.readthedocs.io/en/latest/path.html#py-path

Labels:
name: IO Data
description: IO issues that don't fit into a more specific label

Comments:
Wow, it's been in maintenance mode since 2017.
BUG: wide_to_long with string columns
State: closed

- [ ] closes #57066 (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

Appears `str.match` never accepted a `re.Pattern` so updated the docs too


Labels:
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode
name: Strings
description: String extension data type and string data

Comments:
thx @mroeschke
Fix mem leak in read_csv
State: closed

Pending validation but I believe this fixes #57039

Initially I misread this function was branching based off a return value that was always 0, but the actual value lies in the side effects

Labels:
name: IO CSV
description: read_csv, to_csv

Comments:
@akrherz do you have a quick way of checking this?
@WillAyd I am sorry, I have never built pandas before, but will try it now. I certainly appreciate the help here.
I was able to build pandas, whew.
main branch memray screenshot (showing RSS) with the reproducing leak script
![Screenshot from 2024-01-26 09-14-05](https://github.com/pandas-dev/pandas/assets/210858/454b9a0b-e676-4081-9f21-6830760b7b40)
running @WillAyd/fix-mem-leak branch
![Screenshot from 2024-01-26 09-19-34](https://github.com/pandas-dev/pandas/assets/210858/4e180ba8-304d-47c2-bae1-b1a2f52ab830)

LGTM!
@WillAyd could you add a `peakmem` ASV benchmark for this case?
I've added, but in this case it unfortunately would not have made a difference because the extension uses the system `free` call instead of `PyMem_Free`, so the Python interpreter does not track this.

Maybe we _should_ use PyMem_Free, although the downside to that would be that it doesn't play nicely with LSAN
I don't wish to waste dev's time here, but I am curious why the introduction of `np.nan` in my code causes this memory leak to trigger?  A terse response of "its complicated" is more than sufficient :) An automated test would not have caught this memory leak, perhaps...
Thanks @WillAyd 
> I don't wish to waste dev's time here, but I am curious why the introduction of `np.nan` in my code causes this memory leak to trigger? A terse response of "its complicated" is more than sufficient :) An automated test would not have caught this memory leak, perhaps...

I think np.nan is a red herring - it is definitely a mistake in the code to get rid of those free blocks and LSAN detects the leak either way

Thanks again for the quick report!
> I think np.nan is a red herring

Thank you, but the memory leak does not happen with that line removed.  Regardless, I am thrilled this is fixed :)
Update install.rst to match the newest code
State: closed

enhancing the Excel files related documents of pandas

Labels:
name: IO Excel
description: read_excel, to_excel

Comments:
Thanks for the PR but I think the order of these dependencies is fine as is so closing
the important changes are completed the usage of notes for each depencies for excel, and added the missing 'odfpy' which exists in pyproject.toml but not in documents.

the order of these dependencies is not important
OK sorry the diff did not make that obvious. Could you remove the re-ordering so the diff is clearer?
ok, should keep the order of dependencies in both the `documents` and `pyproject.toml` be consistently the same? or also doesn't matter?
The ordering in either is not important so I would keep the existing ordering for clarity 
ok the re-ordering is removed
Thanks @NewUserHa
54628 fix find stack level memory leak
State: closed

- [ ] closes #54628 (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Performance
description: Memory or execution speed performance

Comments:
Thanks @lmmx for the initial find
Should we backport this for 2.2.x? (it's a memory leak, although presumably only a small one)
Sure no harm in doing so
@meeseeksdev backport 2.2.x
@meeseeksmachine backport 2.2.x
BUG: PeriodIndex.asfreq silently converts offset to period
State: closed

xref #52064

`PeriodIndex.asfreq` silently converts for offsets such as `offsets.MonthBegin()`, `offsets.BusinessMonthEnd()`, etc. (with no attribute `'_period_dtype_code'`) frequency to period frequency (in this case `'M'`).

Reproducible Example:
```
>>> import pandas as pd
>>> from pandas.tseries import offsets
>>>
>>> index = pd.PeriodIndex(["2020-01-01", "2021-01-01"], freq="Q")
>>> index.asfreq(freq=offsets.MonthBegin(1))
PeriodIndex(['2020-03', '2021-03'], dtype='period[M]')
>>>
>>> index.asfreq(freq=offsets.BusinessMonthEnd(1)) 
PeriodIndex(['2020-03', '2021-03'], dtype='period[M]')
```
the correct behaviour would be raising an Error:
```
>>> index.asfreq(freq=offsets.MonthBegin(1))
ValueError: <MonthBegin> is not supported as period frequency
>>>
>>> index.asfreq(freq=offsets.BusinessMonthEnd(1))
ValueError: <BusinessMonthEnd> is not supported as period frequency
```

corrected the definition of `PeriodIndex.asfreq`and added tests.

Labels:
name: Bug
name: Frequency
description: DateOffsets
name: Deprecate
description: Functionality to remove in pandas

Comments:
@MarcoGorelli, could you please take a look at this PR?
I'd like to think that the suggestion in https://github.com/pandas-dev/pandas/pull/56945#pullrequestreview-1854233737 (i.e. raise if the return value of `to_offset(..., is_period=True)` doesn't have a `_period_dtype_code` attribute, could solve this too?
closed by #56945
BUG: Fix reading of multi_index dataframe with NaN
State: closed

- [x] closes #56929 
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

See #56929 for some of the discussion around this bug. Instead of removing the additional header row as suggested in the issue, I fixed the bug _after_ reading all three rows as a header. The code that errors will have these headers: _header_: ['0', 'Unnamed: 1_level_2'], the code where we explicitly set empty values to NaN before writing has these: _header_: ['0', 'NaN'].  The problem with the empty value stems from this line: https://github.com/pandas-dev/pandas/blob/736868671044605b0f2c2cc23ab5ea09fbc9a2ac/pandas/_libs/parsers.pyx#L694-L698 which checks for an empty string and then assigns Unnamed level to it (this is where the paths for the csv with explicit NaN values diverges as its not assigned the unnamed level)

I think there are several ways of fixing this. In this PR I tried to make sure that the first header is treated like the second by simply changing the comparison in this line: https://github.com/pandas-dev/pandas/blob/736868671044605b0f2c2cc23ab5ea09fbc9a2ac/pandas/_libs/parsers.pyx#L749 
No other tests are failing after recompiling the code, therefore I am fairly confident that this fixes the issue and does not introduce others. However, it would still be good if someone else with more parsing experience looks over it. 

Edit: It seems some tests are breaking due to this, will check it out now. 

Labels:
name: Bug
name: Missing-data
description: np.nan, pd.NaT, pd.NA, dropna, isnull, interpolate
name: IO CSV
description: read_csv, to_csv

Comments:
Thanks for the pull request, but it appears to have gone stale. If interested in continuing, please merge in the main branch, address any review comments and/or failing tests, and we can reopen.
REGR: non-unique, masked dtype index raising IndexError
State: closed

- [x] closes #57027
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/v2.2.1.rst` file if fixing a bug or adding a new feature.


Labels:
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves
name: Regression
description: Functionality that used to work in a prior pandas version

Comments:
Thanks @lukemanley
check ExtensionType in is_datetime64_any_dtype for array-likes
State: closed

- [x] closes #57055
- [x] Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

Labels:
name: ExtensionArray
description: Extending pandas with custom dtypes or arrays.

Comments:
Failures in CI look unrelated to this PR ([logs](https://github.com/pandas-dev/pandas/actions/runs/7647380880/job/20838187996#step:8:730))
> FAILED pandas/tests/arrays/categorical/test_sorting.py::TestCategoricalSort::test_numpy_argsort - Failed: DID NOT RAISE <class 'ValueError'>
FAILED pandas/tests/extension/test_categorical.py::TestCategorical::test_argsort - AssertionError: Series are different
Thanks @mroeschke. The test is failing ([log](https://app.circleci.com/pipelines/github/pandas-dev/pandas/42146/workflows/f2ed35b6-c1e4-44ef-8b7b-4c6a535db568/jobs/142394)) for arrow date types

> FAILED pandas/tests/extension/test_arrow.py::test_is_datetime64_any_dtype[date32[day]] - AssertionError
FAILED pandas/tests/extension/test_arrow.py::test_is_datetime64_any_dtype[date64[ms]] - AssertionError

since for those types `pa.types.is_timestamp` returns `False`. However the current implementation of `is_datetime64_any_dtype` returns `True` when provided as a dtype, e.g. `pd.api.types.is_datetime64_any_dtype(serie_with_pyarrow_date32.dtype)`. What should I do here? Is it expected for that to return `True` or should I adjust the above condition to also check if `pa.types.is_date` returns `False`?

**Edit**
I checked that both of those types translate to datetime64 in numpy:
```python
import pyarrow as pa

type(pa.date32().to_pandas_dtype())
# numpy.dtypes.DateTime64DType 
type(pa.date64().to_pandas_dtype())
# numpy.dtypes.DateTime64DType
```
so I've included those in the test for `True` 
I think the date types should return False. In pandas they don't always behave like date _times_
Agree, for example trying to extract the second raises an error. I'll make the changes.
Thanks @jmoralez
DOC: Add docstrings for fixtures defined from pandas.tests.io.conftest
State: closed

Add docstrings for fixtures defined from pandas.tests.io.conftest

- [x] xref https://github.com/pandas-dev/pandas/issues/19159
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Docs

Comments:
@datapythonista would you please give this PR a review and share any feedback? 
Is there any validation for it we are trying to enforce? I personally don't see much value here otherwise. It'd be nice to have proper documentation for fixtures for sure, but that would take understanding well them and providing useful information on what they are exactly and when they can be used. This PR seems to simply repeat the fixture names in a more verbose way. I'd prefer to not have them rather than this.
Agreed with @datapythonista on the value of this. If you're interested, continuing to enforce numpydoc validations would be worthwhile: https://numpydoc.readthedocs.io/en/latest/validation.html#built-in-validation-checks
@datapythonista @mroeschke 

Thank you gentlemen, I appreciate the feedback and guidance! 

One way I do think this adds value is when running `pytest --fixtures pandas` the output before these changes is: 

<img width="1133" alt="before" src="https://github.com/pandas-dev/pandas/assets/35613487/32d1ab5c-70f2-4493-b9dc-23a5f22285af">

and after adding the docstrings: 

<img width="1320" alt="after" src="https://github.com/pandas-dev/pandas/assets/35613487/01d681ba-f371-4df7-ac36-be0937663bef">


Considering this, is this meaningful to add? If not, I'm happy to revert these changes and move on to working on continuing to enforce numpydoc validations as Matthew suggested. 


> Considering this, is this meaningful to add?

Just to make my last comment clearer - I would still be OK with documenting these pytest fixtures, but I don't think there's a ton of value (impact) in doing so.
@mroeschke I've opened https://github.com/pandas-dev/pandas/issues/57111 to continue work on enforcing numpydoc validations: https://numpydoc.readthedocs.io/en/latest/validation.html#built-in-validation-checks
REGR: groupby.idxmin/idxmax wrong result on extreme values
State: closed

- [x] closes #57040 (Replace xxxx with the GitHub issue number)
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

While working on this, I noticed that the `skipna` argument was also being ignored.

The only way I see solving this is adding a `seen` Boolean array instead of the sentinel values. cc @WillAyd

Labels:
name: Bug
name: Groupby
name: Regression
description: Functionality that used to work in a prior pandas version
name: Reduction Operations
description: sum, mean, min, max, etc.

Comments:
Thanks @rhshadrach