BrandonFors/pandas-github-issues huggingface dataset:
Add Py_mod_gil slot to C extension modules
State: closed

x-ref #59057.

The `Py_mod_gil` slot is needed so that the GIL is not automatically enabled when these C extension modules get imported.

- I checked `pandas_datetime` for thread-safety and it appears to be okay.
- `pandas_parser` is not thread-safe. A discussion is needed on whether we should care about it though, since the `c` engine to `read_csv` and friends is already documented as being thread-unsafe and the thread-safe `pyarrow` alternative is already there.

Labels:

Comments:
Is this slot only available in 3.13?

> * `pandas_parser` is not thread-safe. A discussion is needed on whether we should care about it though,

Do you know what makes it not thread unsafe? I'm surprised given how small of a module it is.

Definitely more of a nice-to-have on the parser though. Understood its probably a very large undertaking to make the CSV reader overall multi-threaded, and most likely not even worth it with pyarrow
> Do you know what makes it not thread unsafe? I'm surprised given how small of a module it is.

I mean that it's thread-unsafe in the sense that there's global state in the `parser_t` struct and there's unsafe reads/writes to it. As far as I understand, `read_csv` only ever creates one instance of it though, which means that running `read_csv` in multiple threads is safe. 
Sorry for the typo - I mean "not thread safe" but I think you got what I was saying. And cool thanks for the response - agree with you on that
Thanks @lysnikolaou
Build 3.13 wheels
State: closed

x-ref #58734 

Labels:
name: Build
description: Library building on various platforms

Comments:
The `Windows` build fails with the following log (as far as I can tell from viewing the raw logs):

```
2024-06-28T17:05:06.0044874Z   ..\..\pandas\meson.build:1:15: ERROR: Command `C:\Users\runneradmin\AppData\Local\Temp\cibw-run-n5pyupdy\cp313-win_amd64\build\venv\Scripts\python.exe -c "
2024-06-28T17:05:06.0046145Z   import os
2024-06-28T17:05:06.0047929Z   import numpy as np
2024-06-28T17:05:06.0049860Z   try:
2024-06-28T17:05:06.0052087Z       # Check if include directory is inside the pandas dir
2024-06-28T17:05:06.0053992Z       # e.g. a venv created inside the pandas dir
2024-06-28T17:05:06.0056390Z       # If so, convert it to a relative path
2024-06-28T17:05:06.0058856Z       incdir = os.path.relpath(np.get_include())
2024-06-28T17:05:06.0061272Z   except Exception:
2024-06-28T17:05:06.0063500Z       incdir = np.get_include()
2024-06-28T17:05:06.0065900Z   print(incdir)
2024-06-28T17:05:06.0068059Z        "` failed with status 3221225477.
2024-06-28T17:05:06.0069766Z 
2024-06-28T17:05:06.0073426Z   A full log can be found at C:\Users\runneradmin\AppData\Local\Temp\cibw-sdist-pm_ss3i2\pandas-3.0.0.dev0+1165.ga926d3a9fc\.mesonpy-xr08_29c\build\meson-logs\meson-log.txt
```
Closing in favor of #59136.
Drop duplicate indices
State: closed

- [x] closes #58648
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

Tests are added but have not been run, as I get this error when I try to run them. I tried all envorment setup options but I end-up having the same error

```shell
root@92c4f0dec9a0:/home/pandas# pytest pandas/tests/frame/methods/test_drop_duplicates.py
ImportError while loading conftest '/home/pandas/pandas/conftest.py'.
pandas/__init__.py:47: in <module>
    from pandas.core.api import (
pandas/core/api.py:46: in <module>
    from pandas.core.groupby import (
pandas/core/groupby/__init__.py:1: in <module>
    from pandas.core.groupby.generic import (
pandas/core/groupby/generic.py:1418: in <module>
    class DataFrameGroupBy(GroupBy[DataFrame]):
pandas/core/groupby/generic.py:2571: in DataFrameGroupBy
    @doc(DataFrame.plot.__doc__)
E   AttributeError: type object 'DataFrame' has no attribute 'plot'
```

Labels:

Comments:
Most of these tests are failing because of the same error mentioned earlier
```bash
AttributeError: type object 'DataFrame' has no attribute 'plot'
```

others like precommit fail from parts that I haven't edited
Thanks for the pull request, but it appears to have gone stale. If interested in continuing, please merge in the main branch, address any review comments and/or failing tests, and we can reopen.
Lodewijk
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:

Comments:
Thanks for the PR, but appears something went wrong in your pull request. I would suggest following our contribution guide https://pandas.pydata.org/docs/development/contributing.html. Closing
DOC: Add DataFrame.median() to the deprecated automatic downcasting list in release notes 2.2.0
State: closed


- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:

Comments:
In version 2.1.4, the median function in pandas automatically performed downcasting from object to float64. However, in version 2.2.0, this automatic downcasting no longer occurs.
Thanks for the pull request, but the project does not unfortunately update old release notes. Sorry for this over site when originally crafting these release notes but closing this PR
DOC: json_normalize breaking changes in pandas 3.0.0
State: closed

For context: #51542 & #57422.

As discussed in the above issues,` json_normalize` has breaking changes in 3.0.0.  This is a quick fix to make that more salient in the update documentation, as I very much expect there to be code in the wild that uses the current json_normalize behaviour.

- [x] Added an entry in the latest `doc/source/whatsnew/v3.0.0.rst` file if fixing a bug or adding a new feature.


Labels:
name: Docs

Comments:
pre-commit.ci autofix
Thanks @JMBurley
BUG:  chokes on pd.DatetimeTZDtype if there are no rows.
State: closed

This is a follow up to #41875

- [X] closes #59126
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode
name: ExtensionArray
description: Extending pandas with custom dtypes or arrays.

Comments:
Is there an issue open for this?
I'd welcome some help to understand where/how to add a proper test. I have added a simple repro (3 lines) in the bug
If you could add a unit test in `pandas/tests/reshape/test_pivot.py` that mimics your reproducer that would be great
Thanks @pcorpet
CLN: unify a ValueError message for removed units T, L, U, N and remove these entries from UnitChoices
State: closed

- unify a ValueError message for removed units T, L, U, N to "Invalid frequency: T"
- removed unused entries T, L, n, u from UnitChoices
- removed a deeprecation  note from the documentation of the class Timedelta 

Labels:
name: Clean

Comments:
Thanks @natmokval
CI: Modify the trailing-whitespace hook to preserve markdown hard linebreaks
State: closed


- [x] closes #59107
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Code Style
description: Code style, linting, code_checks

Comments:
Thanks @Aloqeely
BUG: Allow show_versions to work for any module that raises an exception
State: closed

- [ ] closes #59110 (Replace xxxx with the GitHub issue number)

The use of `import_optional_dependency` only can ignore `ImportError`, but as seen in the issue essentially any exception can be raised with an import.

Also cleans up some redundancy of optional dependencies being listed.


Labels:
name: Bug

Comments:
@mroeschke I think it may be a while until we do a 3.0 release, and with numpy 2.0 now released, we might be seeing more reports of problems with `show_versions()`, so we might need to do a 2.2.3 release with this patch to avoid lots of people reporting issues.
OK we can backport this change just in case
Merging, but happy to follow up if needed
DOC: Update warning message in pandas.eval function
State: closed

### Summary
This pull request updates the pandas.eval documentation to add a security warning about the risks of arbitrary code execution when using the function with untrusted data. This update aims to enhance user awareness and security practices.

### Background
The need for this documentation update was identified by **Duarte Santos** from **Checkmarx's Research Group**. A vulnerability was discovered that allows for arbitrary code execution through the misuse of `pandas.eval` with untrusted inputs.

### Proposed Change
Location: [pandas.eval documentation](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.eval.html)
Update: Insert a warning advising users against the use of `pandas.eval` with untrusted data, highlighting the potential for arbitrary code execution.

### Warning Text:
Warning: Use `pandas.eval` only with trusted data. This function can execute arbitrary code if used with untrusted inputs, similar to the risks associated with [Python's `pickle` module documentation](https://docs.python.org/3/library/pickle.html).


### Rationale
This documentation update is crucial for preventing security issues by making users aware of the risks associated with dynamic expression evaluation in pandas.eval. The update follows a preliminary discussion with the Pandas security team and is now presented for broader community feedback.

Thank you for considering this update to enhance the safety and integrity of code using Pandas.

Regards,
Eilon Cohen
Security Analyst, Checkmarx

Labels:
name: Docs

Comments:
Hi @mroeschke 

Thank you for suggesting the warning change for the `eval` function.
Considering the potential risks with its use, I think a more "aggressive" warning might better communicate the severity to users. What do you think?


> What do you think?

IMO it's preferable to have succinct messaging since this docstring is already long. I suggested to fold in "untrusted data" since that was net new information that could be helpful here.
Thanks @eilonc-cx
TYP: Typing improvements for Index
State: closed

- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
 Add axis to Index.repeat
 Add sort to Index.union
 Add sort to Index.difference
 Add sort and result_name to Index.symmetric_difference
 Add level to Index.isin
 Add start, end, step to Index.slice_locs
 Add loc to Index.delete
 Add axis to Index.argmin
 Add axis to Index.argmax
 Add axis to Index.min
 Add axis to Index.max
 Add dtype to Index.astype

Labels:
name: Typing
description: type annotations, mypy/pyright type checking

Comments:
Note: I've discussed [this error](https://github.com/pandas-dev/pandas/actions/runs/9676397917/job/26695859464?pr=59105#step:12:512) with @phofl at pydata meetup. He told me that this error could likely be ignored, probably some pyright issue
Seems like pyright meant numpy.integer by "integer". Now everything is passed @phofl 
> If you want, you can also add some of these changes to pandas-stubs.
> 
> @Dr-Irv

Yes, a PR in `pandas-stubs` would be welcome.  Some of these methods are already done there, others need updates similar to what is in this PR.

Thank you @AndreyKolomiets !
Added a test for the groupby MultiIndex with codes
State: closed

- [ ] closes #54347
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Groupby
name: MultiIndex

Comments:
@phofl Please review this PR, thanks!
Thanks @MurzNN
TST: .loc dtype change bug
State: closed

- [x] closes #29707 
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

@phofl guided me through this, kudos!


Labels:
name: Testing
description: pandas testing functions or related to the test suite

Comments:
Thanks @MovsisyanM
TYP: Typing improvements for Index
State: closed

Adds type hints to following:

- Add axis to Index.repeat
- Add sort to Index.union
- Add sort to Index.difference
- Add sort and result_name to Index.symmetric_difference
- Add level to Index.isin
- Add start, end, step to Index.slice_locs
- Add loc to Index.delete
- Add axis to Index.argmin
- Add axis to Index.argmax
- Add axis to Index.min
- Add axis to Index.max
- Add dtype to Index.astype


Labels:
name: Typing
description: type annotations, mypy/pyright type checking

Comments:
cc @phofl
Fix AttributeError: 'Series' object has no attribute '_pa_array' in CategoricalDtype
State: closed

- [x] closes #53051
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Stale

Comments:
This pull request is stale because it has been open for thirty days with no activity. Please [update](https://pandas.pydata.org/pandas-docs/stable/development/contributing.html#updating-your-pull-request) and respond to this comment if you're still interested in working on this.
Thanks for the pull request, but it appears to have gone stale. If interested in continuing, please merge in the main branch, address any review comments and/or failing tests, and we can reopen.
BUG: Fix Series.str has no method "isascii()"
State: closed

- [ ] closes #59091
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:

Comments:
can try pc.string_is_ascii() 
import pyarrow as pa
pa.compute.string_is_ascii("hello")
<pyarrow.BooleanScalar: True>

Thanks for the pull request, but it appears to have gone stale. If interested in continuing, please merge in the main branch, address any review comments and/or failing tests, and we can reopen.
DEPS: Make pytz an optional dependency
State: closed

- [ ] closes #34916 (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

This makes pandas return zoneinfo.ZoneInfo objects by default from string inputs

Labels:
name: Timezones
description: Timezone data dtype
name: Dependencies
description: Required and optional dependencies

Comments:
Will merge at the end of the week if there are no further comments
DOC: Clarify interaction of read_csv nrows with other args (#59078)
State: closed

- [x] closes #59078 
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:

Comments:
Thanks for the pull request, but it appears to have gone stale. If interested in continuing, please merge in the main branch, address any review comments and/or failing tests, and we can reopen.
ENH: Add option to only merge column header cells in `ExcelFormatter`.
State: closed

Adds an option to pass `merge_cells="columns"` in `DataFrame.to_excel`. This allows only merging MultiIndex column header cells and prevents index cells to be merged.

- [x] closes #35384
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: IO Excel
description: read_excel, to_excel

Comments:
Thanks for the review @mroeschke !
pre-commit.ci autofix
Thanks @ulido
BUG: Fix read_csv raising TypeError when iterator and nrows are specified without chunksize
State: closed

- [x] closes #59079
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: IO CSV
description: read_csv, to_csv

Comments:
Thanks @Aloqeely
BUG: Add type check for encoding_errors in pd.read_csv
State: closed

- [x] closes #59059 
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Error Reporting
description: Incorrect or improved errors from pandas
name: IO CSV
description: read_csv, to_csv

Comments:
Could you add a unit test to validate the exception message?

> Could you add a unit test to validate the exception message?

Ok, I added a test.
Thanks @Aliebc
DOC: document default behaviour of `sample` methods when `random_state = None`
State: closed

Docs improvement to make it clear what the DataFrame/Series/GroupBy `sample` methods do when no argument is supplied for `random_state`. TLDR it just defers to `np.random` to draw random samples, so the state is whatever the current random state of `np.random` is.

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Docs

Comments:
Thanks @alexmalins