BrandonFors/pandas-github-issues huggingface dataset:
API: Replace na_action parameter in Series/DataFrame/Index.map by the standard skipna
State: open

- [X] xref #61128
- [X] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [X] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [X] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [X] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

We've been consistently using a boolean `skipna` parameter to let users decide whether missing values should be ignored or not in different methods. The `.map` method has an awkward `na_action=None | "igonre"` for the same purpose. I add the standard `skipna` parameter to the methods, and start the deprecation of the `na_action` one.

Labels:
name: API Design
name: Apply
description: Apply, Aggregate, Transform, Map
name: Stale

Comments:
Updated this so the `na_action` parameter is also not removed from the extension arrays, and it warns. The base array since as you said @mroeschke is technically public. `map_array` since it's used by pandas-pint, even if it's not supposed to be public, but better not break their code. And the specific implementations that subclass the base extension array to keep the same signature. mypy complained, and also better to keep the signatures the same even if it's a temporary thing.

I think this should be ready to merge now.
This pull request is stale because it has been open for thirty days with no activity. Please [update](https://pandas.pydata.org/pandas-docs/stable/development/contributing.html#updating-your-pull-request) and respond to this comment if you're still interested in working on this.
DOC: Typo fix for .astype() in cheatsheet
State: closed

- [x] Closes #61523
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Docs

Comments:
@datapythonista You're very welcome! I noticed that too about the PDF size. I'm not too familiar with the MS Office suite and how PowerPoint exports exactly; I just saved it directly from PowerPoint Version 16.97.2 (25052611) for Mac by following what looked like the standard flow through either "Save" or "Export" (they are identical in this case):

<img width="846" alt="Screenshot 2025-06-01 at 8 02 47 AM" src="https://github.com/user-attachments/assets/daa77a7d-1adb-4c3d-a0f8-4d27299fd02b" />

The dialog only provides two options for formatting: "Best for electronic distribution and accessibility" (491kB) and "Best for printing" (475kB).

I judged "electronic and accessibility" to be the best option because to my understanding it embeds accessibility tags and keeps hyperlinks, which bumps the file size slightly. Happy to go with whatever you think is best!
Maybe it's just the version of office. I don't kmow much about office myself, if there isn't much we have control of it's fine. I was just wondering if we knew why it was a bit bigger just by removing a character. No big deal
Thanks @brchristian
DOC: fix ES01 for pandas.plotting.autocorrelation_plot
State: closed

fixes

```
pandas.plotting.autocorrelation_plot ES01
```

This includes a crisp extended summary of the method which talks about what it does, how it does it and whats the value of it. 


Labels:
name: Docs
name: Visualization
description: plotting

Comments:
Thanks @tuhinsharma121 
> lgtm, but I'm not so familiar with autocorrelation plots, I'll let someone else merge. @Dr-Irv maybe you know about this more?

Not my area at all!
BUG: Fix assert_frame_equal dtype handling when check_dtype=False (#61473)
State: open

Fixes a bug in `assert_frame_equal` where `DataFrames` with logically equal values but different dtypes (nullable vs non-nullable) would raise errors even with check_dtype=False. Now, when `check_dtype=False`, dtype mismatches won’t cause failures.

- [x] closes #61473 

 - [x] Tests added and passing

- [x] Pre-commit checks passing

- [x] Added entry to v3.0.0.rst



Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Stale

Comments:
This pull request is stale because it has been open for thirty days with no activity. Please [update](https://pandas.pydata.org/pandas-docs/stable/development/contributing.html#updating-your-pull-request) and respond to this comment if you're still interested in working on this.
hey @mroeschke, addressed the review. mind taking another look when you get a chance?
BUG: Fix pivot_table margins to include NaN groups when dropna=False
State: closed

Fix incorrect margin computation in `pivot_table` when index or columns contain NA values

This PR fixes an issue where the `"All"` row or column (i.e., `margins=True`) in `pd.pivot_table` does not account for rows that contain `NA` values in the index or column dimensions. These rows were incorrectly excluded from the overall aggregation used to compute the margin, leading to incorrect totals.

The fix modifies the margin calculation to ensure that rows with `NA` values are included in the aggregation, consistent with how the data is treated in the main table when `dropna=False`.

- [x] closes #61509 
- [x] Test added and passed
- [x] Ran pre-commit check
- [x] Added entry in `doc/source/whatsnew/v3.0.0.rst` under `Reshaping`

Labels:
name: Bug
name: Missing-data
description: np.nan, pd.NaT, pd.NA, dropna, isnull, interpolate
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode

Comments:
This change fixes the margin behavior in `pivot_table` when `dropna=False`, but as expected, it also affects related logic, a few tests are now failing, especially in `test_crosstab`, where the old margin behavior is still being asserted.

Before I go ahead and update those tests to match the new behavior, I just wanted to double-check if this is the direction we want to take,  treating `dropna=False` more consistently in how margins are computed. If this looks good, I’ll go ahead and fix up the failing tests accordingly
@iabhi4 - thanks for putting this up. The crosstab failures here look like bugs to me as well. E.g. in the penultimate column

```
b    one        two        NaN       All
c   dull shiny dull shiny dull shiny    
a                                       
bar    1     0    1     0    0     0   2
foo    2     0    1     1    0     1   5
All    3     0    2     1    0     0   7
```

`All` should be 1, as this PR is now producing.
@rhshadrach Thanks for confirming! I’ve updated the test cases to reflect the corrected behavior
This pull request is stale because it has been open for thirty days with no activity. Please [update](https://pandas.pydata.org/pandas-docs/stable/development/contributing.html#updating-your-pull-request) and respond to this comment if you're still interested in working on this.
Thanks @iabhi4!
TST: Use external_error_raised for numpy-raised test_error_invalid_values
State: closed

Looks like numpy extended an error message that is failing `test_error_invalid_values` e.g https://github.com/pandas-dev/pandas/actions/runs/15331067593/job/43137707952

Labels:
name: Testing
description: pandas testing functions or related to the test suite

Comments:
Merging to fix the failure on main
BUG: Fix sorting by column named None in DataFrame.sort_values (GH#61512)
State: closed

Fixes a bug in `DataFrame.sort_values` where sorting by a column explicitly named None raised a `KeyError`. Added a conditional check to correctly retrieve and sort the column when `None` is used as a label.

- [x] closes #61512
- [x] Tests added and passed
- [x] All code checks passed via pre-commit
- [x] Added an entry to the latest whatsnew file

Labels:
name: Algos
description: Non-arithmetic algos: value_counts, factorize, sorting, isin, clip, shift, diff

Comments:
Thanks @iabhi4
BUGFIX: escape `quotechar` when `escapechar` is not None (even if quoting=csv.QUOTE_NONE)
State: closed

- [x] closes #61407 (Replace xxxx with the GitHub issue number)
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.
---
Found the issue on `CSVFormatter._initialize_quotechar`, wherein it only returns quotechar when `self.quoting` is not `csvlib.QUOTE_NONE`:
```python
if self.quoting != csvlib.QUOTE_NONE:
    # prevents crash in _csv
    return quotechar
return None
```
to follow the same behavior of `csv.writer` (escape when `escapechar is not None`), I improved the if function to **return quotechar when `self.escapechar is not None`**

in the `CSVFormatter.__init__`, moved the initialization of `self.escapechar` higher than `self.quotechar` since the initial solution was returning errors from calling `self.escapechar` before its initialization

initially marked as draft to let the CIs run & check if there are tests affected by this change

Labels:
name: IO CSV
description: read_csv, to_csv

Comments:
CI errors looks unrelated
Thanks @KevsterAmp
BUG: Fix inconsistent returned objects when applying groupby aggregations
State: closed

- [x] closes #61503 
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] ~Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.~
- [ ] ~Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.~


Labels:
name: Bug
name: Apply
description: Apply, Aggregate, Transform, Map

Comments:
Thanks @arthurlw, can you add a test that checks that the bug is indeed fixed. And a release note as well. Thank you! 
Thanks @arthurlw 
Thanks @arthurlw
@mroeschke : is there any chance this could be added to an earlier release?
I see there is a 2.3.1 milestone for example.
ENH: Implement to_iceberg
State: closed

- [X] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [X] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [X] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [X] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: IO Data
description: IO issues that don't fit into a more specific label

Comments:
Added the `append` parameter. I think it's a great addition, thanks for the feedback @IsaacWarren.

I was thinking that for the parameters that receive PyIceberg objects, one option is to use a generic `**kwargs` like `to_parquet` does, that are sent to the engine (only PyIceberg so far). This wouldn't directly expose PyIceberg details to our API and they could be supported. This would be very simple if only one method from PyIceberg received extra parameters. But there are a couple in `overwrite` where the same could be done, and that makes it a bit trickier. I think it's still best to leave this to a follow up PR, so it can be analyzed and discussed in greater detail. And even if it's done after this is released in 3.0, no big deal since there are no backward compatibility problems.
Use `TypeAlias` in code where types are declared
State: closed

After introducing `TypeAlias` in `_typing.py`, goal of this PR is to use it in any other source files that are creating types in this way.  ~~Also, make all of those types private.~~

I believe that I caught them all via some searching, but may have missed a few.  Couldn't find a rule that enforces use of `TypeAlias`.  Note that in 3.12, the recommendation is to do a `type` declaration, which is probably why there isn't such a rule.


Labels:
name: Clean
name: Typing
description: type annotations, mypy/pyright type checking

Comments:
@rhshadrach are we good to merge this?
Thanks @Dr-Irv
DOC: use Hashable instead of label
State: closed

In https://github.com/pandas-dev/pandas/pull/61455#discussion_r2096069007:
> we should actually be using `Hashable` everywhere as since it's an actual Python type unlike `label`.

- <del>[ ] closes #xxxx (Replace xxxx with the GitHub issue number)</del>
- <del>[ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- <del>[ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.</del>
- <del>[ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.</del>


Labels:
name: Docs

Comments:
Thanks @cmp0xff
DOC: Fix sparse and dense array memory usage comparison.
State: closed

This MR fixes misleading part of sparse user guide, which suggests that dataframes consume far less memory than they really do, due to wrong units calculation.

Also changed usage of `str.format()` to more modern f-string syntax in part of guide which was mentioned above.

Labels:
name: Docs

Comments:
/preview
Website preview of this PR available at: https://pandas.pydata.org/preview/pandas-dev/pandas/61495/
DOC: Fix incorrect reST markups in What's new
State: closed

Fixed some markups so as to render HTML correctly.

Labels:
name: Docs

Comments:
/preview
Website preview of this PR available at: https://pandas.pydata.org/preview/pandas-dev/pandas/61492/
@datapythonista Thanks, I checked the document now renders correctly.
Thanks for the fixes @koyuki7w
Fix GH-61477: Prevent spurious sort warning in concat with unorderable MultiIndex
State: closed

# Fix GH-61477: Stop Spurious Warning When `concat(..., sort=False)` on Mixed-Type `MultiIndex`

## Overview

When you do something like:

```python
pd.concat([df1, df2], axis=1, sort=False)
```
and your two DataFrames have MultiIndex columns that mix tuples and integers, pandas used to try to sort those labels under the hood. Since Python cannot compare tuple < int, you’d see:
```
RuntimeWarning: '<' not supported between instances of 'int' and 'tuple'; sort order is undefined for incomparable objects with multilevel columns
```

This warning is confusing, and worse, you explicitly asked not to sort (sort=False), so pandas should never even try.

# What Changed
1. Short-circuit Index.union when sort=False
Before: Even with sort=False, pandas would call its normal union logic, which might attempt to compare labels.

Now: If you pass sort=False, we simply concatenate the two index arrays with:
``` 
np.concatenate([self._values, other._values])
```
and wrap that in a new Index. No comparisons, no warnings, and your original order is preserved.


2. Guard sorting in MultiIndex._union
Before: pandas would call ```result.sort_values()``` when sort wasn’t False, and if labels were unorderable it would warn you.

Now: We only call ```sort_values()``` when sort is truthy (True), and we wrap it in a ```try/except``` TypeError that silently falls back to the existing order on failure. No warning is emitted.

3. New Regression Test
A pytest test reproduces the original bug scenario, concatenating two small DataFrames with mixed-type MultiIndex columns and ```sort=False.``` The test asserts:

No RuntimeWarning is raised

Column order is exactly “first DataFrame’s columns, then second DataFrame’s columns”

Respects sort=False: If a user explicitly disables sorting, pandas won’t try.

Silences spurious warnings: No more confusing messages about comparing tuples to ints.

Keeps existing behavior for sort=True: You still get a sort or a real error if the labels truly can’t be ordered.

For testing we can try 
```
import numpy as np, pandas as pd

left = pd.DataFrame(
    np.random.rand(5, 2),
    columns=pd.MultiIndex.from_tuples([("A", 1), ("B", (2, 3))])
)
right = pd.DataFrame(
    np.random.rand(5, 1),
    columns=pd.MultiIndex.from_tuples([("C", 4)])
)

# No warning, order preserved:
out = pd.concat([left, right], axis=1, sort=False)
print(out.columns)  # [("A", 1), ("B", (2, 3)), ("C", 4)]

# Sorting still works if requested:
sorted_out = pd.concat([left, right], axis=1, sort=True)
print(sorted_out.columns)  # sorted order or TypeError if impossible
```


Labels:

Comments:
Thanks for the pull request, but it appears to have gone stale. If interested in continuing, please merge in the main branch, address any review comments and/or failing tests, and we can reopen.
BUG: Raise on coercion of ambiguous datetime strings to datetime64
State: closed

This PR addresses a bug where object-dtype arrays containing ambiguous datetime strings (e.g., `"12/01/2020"`, `"13/01/2020"`) were being silently coerced to `datetime64[ns]`, potentially resulting in inconsistent or unintended parsing.

- Introduced stricter input validation during coercion to detect and raise a `ValueError` when an ambiguous format is inferred but cannot be consistently parsed.
- Added tests to cover both direct assignment and constructor-based coercion scenarios.

- [x] closes 61353  
- [x] Tests added and passed
- [ ]  All code checks passed
- [x] Added an entry in the latest `doc/source/whatsnew/v3.0.0.rst` file  

Labels:

Comments:
The `pre-commit` is currently failing due to the `unwanted-patterns-private-import-across-module` hook

This PR imports `_guess_datetime_format_for_array` from `pandas.core.tools.datetimes`, which is a private function.
There’s currently no public alternative that offers this functionality. specifically, inferring a consistent datetime format from an array of strings without coercion.

Happy to refactor this if needed, for example, by moving it to a shared utility module or discussing exposing it publicly if that makes more sense long-term
Closing this PR as the current approach doesn’t feel quite right from a design and maintainability perspective. I believe there might be a cleaner solution that avoids adding complexity across multiple layers
Backport PR #60739 on branch 2.3.x (ENH: pandas.api.interchange.from_dataframe now uses the Arrow PyCapsule Interface if available, only falling back to the Dataframe Interchange Protocol if that fails)
State: closed

OK to backport https://github.com/pandas-dev/pandas/pull/60739/files?

One thing to check is:
- #60739 has the release note in 3.0
- but if it gets backported, then the change will have happened in 2.3

So, not sure how to do this. Is it OK to backport, and then change the whatsnew note on `main`?

Labels:
name: Interchange
description: Dataframe Interchange Protocol

Comments:
Thanks @MarcoGorelli 
thanks! yup, will do soon
BUG: Raise clear error for duplicate id_vars in melt (GH61475)
State: closed

- [x] closes #61475 
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode
name: Error Reporting
description: Incorrect or improved errors from pandas

Comments:
Thanks @ZanirP
BUG: Raise error when filtering HDF5 with tz-aware index (GH#61479)
State: closed

This PR addresses a bug where applying a .select(where=...) query on an HDF5 store with a timezone-aware DatetimeIndex raised a confusing or incorrect error. Since tz-aware filtering isn't currently supported, we now raise a clear ValueError when such filtering is attempted.
What’s included:

-Adds a specific check in select() to detect and error on tz-aware index queries

-Includes a minimal test case reproducing the issue in test_timezone_bug.py

Closes: #61479

Labels:
name: IO HDF5
description: read_hdf, HDFStore
name: Stale

Comments:
@AnushaUKumar - the linked issue is not about timezones, did you perhaps get the number wrong?
Thanks for the pull request, but it appears to have gone stale. If interested in continuing, please merge in the main branch, address any review comments and/or failing tests, and we can reopen.
BUG: Fixed issue where rolling.kurt() calculations would be effected by values outside of scope
State: open

- [x] closes #61416
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

Might have found an unrelated issue when calculating kurtosis for numbers >1e6, but I'll have to look into it more and open an issue if that is the case. 


Labels:
name: Bug
name: Window
description: rolling, ewma, expanding
name: Stale

Comments:
@mroeschke my PR hasn't been reviewed for a while now, just checking if it will be reviewed or if I should just close it.

(sorry if it's a bother, I know you guys probably all have a lot on your plates and I didn't know who to ping)
This pull request is stale because it has been open for thirty days with no activity. Please [update](https://pandas.pydata.org/pandas-docs/stable/development/contributing.html#updating-your-pull-request) and respond to this comment if you're still interested in working on this.