BrandonFors/pandas-github-issues huggingface dataset:
Fix #58421: Index[timestamp[pyarrow]].union with itself return object type
State: closed

- [X] closes #58421
- [X] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [X] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

### Fix Summary:
Previously, the `_make_concat_multiindex` method could silently downgrade extension dtypes (e.g., to object) when creating levels. This PR ensures that the `_concat_indexes` helper uses the correct dtype-aware construction (`array(..., dtype=...)`) to preserve the original dtype of the first index.

### Test added:
Added a test in `pandas/tests/frame/methods/test_concat_arrow_index.py` that covers the preservation of extension dtypes when using `pd.concat` with `keys=` that triggers MultiIndex creation.

The test creates two DataFrames with `timestamp[pyarrow]` indices, then concatenates them with `pd.concat(..., keys=...)` and asserts that:
- The resulting index is a `MultiIndex`
- The second level (`levels[1]`) retains the `ArrowDtype('timestamp[us][pyarrow]')` instead of being downgraded to `object`.

This ensures the dtype preservation fix is validated and regressed against.


Labels:

Comments:
### Note on test failures
Some tests are failing because they expect the old behavior where ``pd.concat(..., keys=...)`` would return an ``Index`` of tuples with ``dtype=object``.

This PR intentionally changes that behavior to **preserve the dtype of the original index** (e.g., ArrowDtype) and produce a proper ``MultiIndex`` with names and levels — which is more consistent and solves the issue.

Errors such as:
- AttributeError: 'Index' object has no attribute 'levels'
- AssertionError due to mismatched Index vs MultiIndex

...are a direct result of this behavior change.
These test failures are expected and reflect outdated assumptions.
If needed, I'm happy to follow up with updates to the relevant tests to align with the new behavior.
Thanks for the pull request, but it appears to have gone stale. If interested in continuing, please merge in the main branch, address any review comments and/or failing tests, and we can reopen.
BUG: OverflowError when fillna on DataFrame with a pd.Timestamp (#61208) 
State: closed

- Now correctly raises OutOfBoundsDatetime
- Added test_fillna_out_of_bounds_datetime()

- [x] closes #61208
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions. (does not apply)
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


### Fix for `fillna` with Out-of-Bounds Datetime Values

**Issue**: Using `fillna` on a `datetime64[ns]` column with an out-of-bounds timestamp (e.g., `'0001-01-01'`) raised an `AssertionError` instead of the expected `OutOfBoundsDatetime`.

**Fix**: Modified the `where` method in `pandas/core/internals/blocks.py` to catch and re-raise `OutOfBoundsDatetime` directly, preventing the `AssertionError`.


**Fix (`inplace=True`)**: Modified the `putmask` method in `pandas/core/internals/blocks.py` to catch and re-raise `OutOfBoundsDatetime` directly, preventing the `AssertionError`.


**Test Added**:
- Created `test_fillna_out_of_bounds_datetime` in `pandas/tests/frame/methods/test_fillna.py`.
- The test:
  - Sets up a DataFrame with a `datetime64[ns]` column containing `NaT`.
  - Attempts to fill `NaT` with `'0001-01-01'`.
  - Expects `OutOfBoundsDatetime`.


Labels:
name: Bug
name: Missing-data
description: np.nan, pd.NaT, pd.NA, dropna, isnull, interpolate
name: Error Reporting
description: Incorrect or improved errors from pandas
name: Timestamp
description: pd.Timestamp and associated methods

Comments:
Thanks @PedroM4rques
Restrict clipping of DataFrame.corr only when cov=False
State: closed

 Closes #61154 `DataFrame.corr` was clipped between `-1` and `1` to handle numerical precision errors. However, this was done regardless of whether `cov` equals `True` or `False`, and should instead only be done when `cov=False`.

- [x] closes #61154 (Replace xxxx with the GitHub issue number)
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: cov/corr

Comments:
@mroeschke here is my pull request for fixing the `dataframe.corr` issue. Thanks!
Thanks! Would you be able to add new unit test that covers this? It seems like we didn't have one that hit this edge case previously.

I think no release note is necessary, since the original one made clear it's only for `corr`.
Thanks @j-hendricks
BUG: OverflowError when fillna on DataFrame with a pd.Timestamp (#61208)
State: closed

- [x] closes #61208
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions. (Does not apply)
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

### Fix for `fillna` with Out-of-Bounds Datetime Values

**Issue**: Using `fillna` on a `datetime64[ns]` column with an out-of-bounds timestamp (e.g., `'0001-01-01'`) raised an `AssertionError` instead of the expected `OutOfBoundsDatetime`.

**Fix**: Modified the `putmask` method in `pandas/core/internals/blocks.py` to catch and re-raise `OutOfBoundsDatetime` directly, preventing the `AssertionError`.

**Test Added**:
- Created `test_fillna_out_of_bounds_datetime` in `pandas/tests/frame/methods/test_fillna.py`.
- The test:
  - Sets up a DataFrame with a `datetime64[ns]` column containing `NaT`.
  - Attempts to fill `NaT` with `'0001-01-01'`.
  - Expects `OutOfBoundsDatetime`.

Labels:

Comments:
I did not open this PR correctly, I'll open another ASAP
Fix #60494: query doesn't work on DataFrame integer column names
State: closed

- [x] closes #60494
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.
Some io code checks failed but they were already failing before the bugfix.
Function _get_cleaned_column_resolvers ignores integer column names, so when it is called on eval function, it returns empty columns which it shouldn't since there is an integer column. Converting the integer columns to strings before calling the _get_cleaned_column_resolvers on eval fucntion fixes this.

Labels:
name: expressions
description: pd.eval, query

Comments:
Thanks @David-msggc
BUG: fix to_json on period
State: closed

- [x] closes #55490 (Replace xxxx with the GitHub issue number)
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Stale

Comments:
Does anyone else want to review it?
This pull request is stale because it has been open for thirty days with no activity. Please [update](https://pandas.pydata.org/pandas-docs/stable/development/contributing.html#updating-your-pull-request) and respond to this comment if you're still interested in working on this.
Can anyone take a look?
Thanks for the pull request, but it appears to have gone stale. If interested in continuing, please merge in the main branch, address any review comments and/or failing tests, and we can reopen.
Fix missing blank line in DataFrame.round docstring (PEP 257 style)
State: closed

This PR fixes a minor style issue in the docstring of `DataFrame.round()`.

### What Changed:
- Adds a missing blank line before the closing triple quotes (`"""`)
- Ensures compliance with PEP 257 and pandas' internal docstring style guidelines
- Helps maintain clean parsing for automated doc tools and improves readability

This is a **non-functional, formatting-only change** intended to improve internal consistency across the API documentation.

---

### ✅ Checklist

- [ ] closes #xxxx (No issue to close — docstring formatting only)
- [ ] Tests added and passed (Not applicable)
- [x] All code checks passed (pre-commit and linting compliant)
- [ ] Added type annotations (Not applicable)
- [ ] Added an entry in `doc/source/whatsnew/` (Not applicable for style-only fix)

Author: Michael Alexander Montoya (@cureprotocols)


Labels:
name: Docs

Comments:
Indeed, pre-commit identifies this as an issue. Closing.
BUG: date comparison fails when series is all pd.NaT values
State: closed

- [x] closes #61188 (Replace xxxx with the GitHub issue number)
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/v3.0.0.rst` file if fixing a bug or adding a new feature.


Labels:
name: Bug
name: Datetime
description: Datetime data dtype
name: Missing-data
description: np.nan, pd.NaT, pd.NA, dropna, isnull, interpolate

Comments:
pre-commit.ci autofix
@rhshadrach I just incorporated the changes! Please let me know if everything looks okay.
> lgtm

Good to hear! Thank you for guidance :)
Looks like this PR re-added the 1.3MB Pandas Cookbook jpeg into main. Looks like we should hard revert this commit?  cc @datapythonista  
I'm travelling and I may not be able to do it for several hours or tomorrow. Do you have permissions to access the project settings and enable the force push option? If you can do that, it's aa easy as clone, git reset --soft head~ and git push -f.

If you don't have access, I can see if I can give you permissons, but I don't think I can. Otherwise let's try npt to merge until then, which will make things significantly more complicated.
@pandas-dev/pandas-core I hard reverted this, as it reintroduced the big image we wanted to avoid having in our history.

@Mohit-Kundu sorry about this. We had to rewrite pandas git history, which likely caused that unrelated image ending up in your PR. Sorry we didn't see you before merging, but now we had to hard undo the resulting commit of this PR to avoid having that image in our history and increasing even more the repository size (it's 433Mb even being extra careful of not adding big files). Given this, can I ask you to please open this PR again with the same changes, but without the image? Thank you!

@mroeschke I see you have the same permissions as me already. I can't add people, but all good for now I think.
Thanks @mroeschke @datapythonista 
@datapythonista I'd be happy to do that! 

However, I'm not exactly sure how to reopen the PR, since on my end it shows that the pull request has been successfully merged and closed. I'd really appreciate your guidance on the best way to proceed; should I open a new PR with the same changes but without the image, or is there another approach you would recommend?

Thanks!


> should I open a new PR with the same changes but without the image

Yes, I think this is the best. Thank you!
BUG: Fix Series comparison fails when index dtypes differ (object vs string) (#61099)
State: closed

- [x] closes #61099
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/v3.0.0.rst` file if fixing a bug or adding a new feature.

Labels:

Comments:
@MayurKishorKumar - looks like your environment got added to your commit.
Oh shoot, my bad! I’ll fix that right away. Thanks for catching it!
Hi @rhshadrach 👋

I’m working on fixing [[#61099](https://github.com/pandas-dev/pandas/issues/61099)] and ran into a failure in `test_mixed_col_index_dtype`.

My fix updates `Index.equals` so that `StringDtype` and `object` dtypes are treated as equivalent when comparing column indexes. As a result, this test now fails because `result.columns.dtype` becomes `"string"` while `expected.columns.dtype` remains `object`.

There are two options I’m considering:

1. **Update the test** to explicitly cast `expected.columns` to `"string"` when `using_infer_string=True`, so it reflects the result.
2. **Adjust internal logic** so the result stays `object`, but that might go against the spirit of treating string/object as equal.

Would updating the test be acceptable in this case?

Thanks!

pre-commit.ci autofix
BUG: Fix AttributeError in pd.eval for method calls on binary operations
State: closed

- [x] closes #61175 
- [x] [Tests added and passed] if fixing a bug or adding a new feature.
- [x] All [code checks passed]
- [x] Added [type annotations] to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

Labels:
name: Bug
name: expressions
description: pd.eval, query

Comments:
Thanks @myenugula
ENH: Add a new parameter to pandas.read_csv #61172
State: closed

- [x] closes #61172
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

### Notes:
I tried to fix this issue by adding a new parameter named `return_empty` for `pandas.read_csv` which is by default _**False**_ and will return an empty DataFrame if set to _**True**_.

I am not familiar with Tests and didn't test it but should not break anythings.

Labels:

Comments:
Thanks for the PR, but as discussed in https://github.com/pandas-dev/pandas/issues/61172#issuecomment-2749363849 there probably isn't much appetite for this feature so closing this PR as won't implement
DOC: User Guide Page on user-defined functions
State: closed

- [x] closes #61126 
- [ ] ~[Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature~
- [ ] ~All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).~
- [ ] ~Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.~
- [ ] ~Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.~


Labels:
name: Docs
name: Apply
description: Apply, Aggregate, Transform, Map

Comments:
Currently writing this, so I would appreciate any feedback on it!
Hi @rhshadrach thanks for the feedback! I agree with you and will push updates soon
/preview
Website preview of this PR available at: https://pandas.pydata.org/preview/pandas-dev/pandas/61195/
@arthurlw I generated a preview of the rendered docs in this PR. If you want to have a look to see if everything looks as expected: https://pandas.pydata.org/preview/pandas-dev/pandas/61195/docs/user_guide/user_defined_functions.html

@rhshadrach can you have a look and see if this can be merged? Even if we want to make some improvements and extend this in the future, I think this PR is already a great first version. So, whenever there is no blocker, probably easier to get this merged and iterate in follow up PRs if needed.
Thanks for the preview! I think adding an example that combines groupby and filter (which takes a UDF) could be beneficial. That said, I do think it might duplicate some existing docs, so not sure if it's worth including here.
Just noting this is still on my radar, should be able to get to it in the next 3 days.
Thanks @arthurlw for all the work here!
Thanks for the guidance on this UDF User Guide @datapythonista @rhshadrach!

I'm interested in diving deeper into this area of pandas. Are there any related issues, features, or improvements you think could be tackled next?

Thanks in advance!
@arthurlw I created #61458 and assigned you to it. I think it's a good one to work in pandas udf. Good to learn more about the status quo, with reasonable complexity, and very useful to the project since it will make the code much clearer for future changes. Also, if you work on this issue, I think you'll realize of inconsistencies, missing documentation, and other related tasks that may be good to work on too. For example, it could be useful to document the executor interface in the documentation about extending pandas. So, third-party library authors can easily learn how to create an execution engine for pandas map/apply.

Please let me know if this is not what you're looking for, I can try to think of something else. Or if you have any question (better to ask them in the other issue).
BUG: Fix pyarrow categoricals not working for pivot and multiindex
State: closed

- [X] closes #53051
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

<details>
<summary>Disclaimer</summary>
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

</details>


Labels:
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode
name: Arrow
description: pyarrow functionality

Comments:
Hey @mroeschke,
Thanks for the Review! All comments should be adressed. Could you have another look if everything looks good?
Hey @mroeschke,
Thanks again for the review. All feedback should be addressed. Can you have another look?
Thanks @robin-mader-bis
DOC: Add details of dropna in DataFrame.pivot_table
State: closed

- [x] closes #61113 
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Docs
name: Missing-data
description: np.nan, pd.NaT, pd.NA, dropna, isnull, interpolate
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode

Comments:
Which `doc/source/whatsnew/vX.X.X.rst` file should I edit? `v2.3.0.rst`?
~@rhshadrach it looks like two warnings are raised because of the bulleted list formatting: ([1](https://github.com/pandas-dev/pandas/actions/runs/14161907429/job/39668656803?pr=61184#step:8:61)), ([2](https://github.com/pandas-dev/pandas/actions/runs/14161907429/job/39668656803?pr=61184#step:8:62)). Is there another way you recommend formatting the docstring? Or do you have an example with a bulleted list that I can work from?~

Think I figured it out in 1a82d2a.
@rhshadrach ready for review
Thanks @it176131 - nice work!
REGR: Interpolate with method=index
State: closed

- [x] closes #61122 (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

Partial revert of #56515. Regression is only on main; hasn't been released yet so no whatsnew.

Labels:
name: Bug
name: Missing-data
description: np.nan, pd.NaT, pd.NA, dropna, isnull, interpolate
name: Regression
description: Functionality that used to work in a prior pandas version

Comments:
Looks like the resample interpolate doctests are failing https://github.com/pandas-dev/pandas/actions/runs/14088933670/job/39460360510?pr=61183
Thanks @rhshadrach