BrandonFors/pandas-github-issues huggingface dataset:
BUG: Fix assert_frame_equal with check_dtype=False for pd.NA dtype differences (GH#61473)
State: open

- [x] closes #61473
- [x] tests added / passed
- [x] Ensure all linting tests pass
- [x] whatsnew entry

## Problem
When comparing two DataFrames containing `pd.NA` values with `check_dtype=False`, `assert_frame_equal` fails when the DataFrames only differ in dtype (object vs Int32). This happens because `pd.NA` and `np.nan` are treated as different values even though they represent the same missing value.

## Solution
Modified `assert_frame_equal` in `pandas/_testing/asserters.py` to normalize `pd.NA` and `np.nan` values when `check_dtype=False` is specified. This ensures that DataFrames with equivalent missing values but different dtypes can be compared successfully.

## Changes Made
- Added normalization logic in `assert_frame_equal` function to handle `pd.NA` and `np.nan` equivalence when `check_dtype=False`
- Added comprehensive unit test in `pandas/tests/util/test_assert_frame_equal.py` to verify the fix

## Testing
- Added unit test `test_assert_frame_equal_pd_na_dtype_difference` that reproduces the original issue and verifies the fix
- Test passes successfully with the implemented solution

Labels:
name: Bug
name: Testing
description: pandas testing functions or related to the test suite

Comments:
Hello,

It seems that the "Downstream Compat" CI job is failing with an `ImportError` related to `statsmodels` and `scipy`. This appears to be an issue with the CI environment's dependencies (e.g., version mismatch between `statsmodels` and `scipy`), and not directly related to the changes introduced in this PR.

The error message is:
```
_______________________________ test_statsmodels _______________________________
E       pytest.PytestDeprecationWarning: 
E       Module 'statsmodels.formula.api' was found, but when imported by pytest it raised:
E           ImportError("cannot import name '_lazywhere' from 'scipy._lib._util' (/home/runner/micromamba/envs/test/lib/python3.11/site-packages/scipy/_lib/_util.py)")
```

I believe this is an infrastructure/dependency issue on the CI side, rather than a bug in my proposed changes for GH#61473.

Could you please confirm this and let me know if there's anything I can do to help resolve this CI issue, or if this is something the core `pandas` team needs to address?
CI: if no docstring, create error GL08 and don't validate - fix for numpydoc 1.9
State: closed

`validate_docstrings` was failing with `numpydoc` 1.9 on cython methods that have no docstrings.  When no docstring, there is nothing to validate.

Partially addresses the CI issue mentioned in #61740 


Labels:
name: CI
description: Continuous Integration
name: Code Style
description: Code style, linting, code_checks

Comments:
is this an alternative to #61725?
> is this an alternative to #61725?

Yes, and it allows us to use `numpydoc` 1.9
Thanks @Dr-Irv 
Owee, I'm MrMeeseeks, Look at me.

There seem to be a conflict, please backport manually. Here are approximate instructions:

1. Checkout backport branch and update it.

```
git checkout 2.3.x
git pull
```

2. Cherry pick the first parent branch of the this PR on top of the older branch:
```
git cherry-pick -x -m1 30a2e7fb239e63beab44c7c459517eb4d2908a0d
```

3. You will likely have some merge/cherry-pick conflict here, fix them and commit:

```
git commit -am "Backport PR #61744: CI: if no docstring, create error GL08 and don't validate - fix for numpydoc 1.9"
```

4. Push to a named branch:

```
git push YOURFORK 2.3.x:auto-backport-of-pr-61744-on-2.3.x
```

5. Create a PR against branch 2.3.x, I would have named this PR:

> "Backport PR #61744 on branch 2.3.x (CI: if no docstring, create error GL08 and don't validate - fix for numpydoc 1.9)"

And apply the correct labels and milestones.

Congratulations — you did some good work! Hopefully your backport PR will be tested by the continuous integration and merged soon!

Remember to remove the `Still Needs Manual Backport` label once the PR gets merged.

If these instructions are inaccurate, feel free to [suggest an improvement](https://github.com/MeeseeksBox/MeeseeksDev).
                
Backport -> https://github.com/pandas-dev/pandas/pull/61745
BUG: Assigning boolean series with boolean indexer
State: closed

Supersedes #60127
- [x] closes #57338 (Replace xxxx with the GitHub issue number)
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Bug
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves
name: PDEP6-related
description: related to PDEP6 (not upcasting during setitem-like Series operations)

Comments:
Thanks @yuanx749
BUG: fillna with DataFrame input should preserve dtype when possible
State: open

When filling a DataFrame with another DataFrame using `fillna`, columns with matching dtypes were being unnecessarily cast to `object` due to use of `np.where`.

This PR updates the logic to use pandas’ `Series.where`, which is dtype-safe and respects extension and datetime types.

- [x] Closes #61568
- [x] Adds a regression test
- [x] Adds a whatsnew entry
- [x] pre-commit checks passed

Labels:
name: Bug
name: Dtype Conversions
description: Unexpected or buggy dtype conversions
name: Stale

Comments:
Since we now operate column-wise and use `Series.where `instead of `np.where`, so it keeps dtype safety as suggested by @jbrockmendel

This also preserves extension dtypes like `string[pyarrow]`, which used to get cast to object. Because of that, `test_fillna_dataframe_preserves_dtypes_mixed_columns` is failing since it expects the downgraded dtype.

Let me know if this behavior change is fine, happy to update the test or tweak the logic based on what’s preferred!
This pull request is stale because it has been open for thirty days with no activity. Please [update](https://pandas.pydata.org/pandas-docs/stable/development/contributing.html#updating-your-pull-request) and respond to this comment if you're still interested in working on this.
DOC: Fix grammar in AUTHORS.md
State: closed

This pull request fixes minor grammatical errors and improves clarity in the AUTHORS.md file to enhance the overall documentation quality. Thank you for considering this contribution!

Labels:
name: Docs

Comments:
Hello, Happy to Contribute first time in this repository ! 
Hi all, just a quick note , the current failing checks are due to unrelated dependency and docstring validation issues.
This PR only contains grammar fixes in author.md and does not affect the code or dependencies.
I kindly suggest these external issues be addressed separately so this documentation improvement can be merged smoothly.
Thank you very much for your time and understanding!


Can you revert the whitespace changes so it is easier to focus on the grammar
Thank you for the feedback!
I’ll revert the whitespace changes right away so the grammar fixes are clearer.
I've reverted the unintended whitespace changes and applied only the requested grammatical fixes to AUTHORS.md. Please let me know if any further changes are needed. Thank you for your guidance!
I'm now seeing nothing _but_ whitespace changes.
Sorry for inconvenience, I think I  have reverted the whitespace and made all the required grammar changes.
Thanks @sangampaudel530
ENH: Added features of issue 61691
State: closed

- [x] closes #61691 
- [ ] [Tests added]
- [ ] All [code checks passed]


Labels:

Comments:
@ishaan1234 generally its best to wait to implement a requested feature until there is a consensus in the issue that it is something we want to do.  Requests for new top-level APIs usually get politely rejected.
Thanks @ishaan1234 but as mentioned, this feature needs discussion before a pull request will be considered so closing this PR for now
> @ishaan1234 generally its best to wait to implement a requested feature until there is a consensus in the issue that it is something we want to do. Requests for new top-level APIs usually get politely rejected.

Alright! Thanks!
> Thanks @ishaan1234 but as mentioned, this feature needs discussion before a pull request will be considered so closing this PR for now

Okay
ENH: Parallelization support for pairwise correlation
State: closed

- [X] closes #40956
- [X] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [X] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [X] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [X] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Used cython.parallel to parallelize the nancorr function. 


Labels:
name: Enhancement
name: Multithreading
description: Parallelism in pandas
name: cov/corr

Comments:
Looks like webAssembly/emscripten environments don't have OpenMP support. To get around this Im thinking of modifying the menson.build and setup.py files to conditionally include OpenMP compiler and linker flags only when not building for WebAssembly/Emscripten. (WIP)
I'm positive on adding parallelization as is being done here, but negative without the proper framework for use across pandas. See https://github.com/pandas-dev/pandas/issues/43313.
Thanks for the PR, but as mentioned the project needs to coordinate & decide a common pattern for parallelism before implementing it in any specific method yet, so closing as that discussion needs resolution first
DOC: Add missing Index.infer_objects link to API reference
State: closed

- [x] closes #61733 
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature - NA
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions - NA.
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Docs

Comments:
pre-commit.ci autofix
Thanks @PreethamYerragudi
BUG: AttributeError in pandas.core.algorithms.diff when passing non-numeric types
State: closed

…g non-numeric types

- [ ] closes #61728
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:

Comments:
Thanks for this PR but closing per https://github.com/pandas-dev/pandas/issues/61728#issuecomment-3020121533
CI: temporarily pin numpydoc<1.7 to unblock docstring validation (GH#61720)
State: closed

Temporarily pin **numpydoc<1.7** to unblock the *docstring-validation* job.

`numpydoc` 1.7.0 raises  
`AttributeError: 'getset_descriptor' object has no attribute '__module__'`  
inside `numpydoc/validate.py`, causing pandas’ *Code Checks / Docstring validation*
step to fail before any pandas code is run (see GH #61720).

This PR

* **pins** `numpydoc<1.7` in `environment.yml`  
  (propagated to `requirements-dev.txt`);
* **fixes** a duplicate **Returns / Yields** section in
  `pandas/_config/config.py::option_context`;
* **marks** an expected warning in `doc/user_guide/timeseries.rst`
  with `:okwarning:` so Sphinx no longer treats it as an error.

Together these changes restore a green CI across all jobs.

### Notes
* All changes are limited to **dev/CI and docs**—no impact on end users.
* Once an upstream fix lands in numpydoc, we’ll remove the version pin.
* No new tests are required; a successful CI run itself demonstrates the fix.

---

- [x] closes #61720  
- [x] CI green locally (`pre-commit run --all-files` & `doc/make.py --warnings-are-errors`)  
- [ ] added to `doc/source/whatsnew/vX.X.X.rst` → **not needed** (CI-only)



Labels:

Comments:
Superseded by #61725
DOC: Pin numpydoc=1.8.0
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Docs
name: Dependencies
description: Required and optional dependencies

Comments:
Is there a way for us to replace our `numpydoc` usage so we could use a `numpydoc>=1.9.0` instead?
> Is there a way for us to replace our `numpydoc` usage so we could use a `numpydoc>=1.9.0` instead?

It probably needs to be fixed from the numpydoc side. We could pin it for now to make the CI green.
Thanks @fangchenli, but I just merged https://github.com/pandas-dev/pandas/pull/61744 which allow us to get around the error (skipping validation for objects that don't have docstrings which is probably the right thing to do).

Going to close since we don't need to pin numpydoc anymore
DOC: add sections about big new features (CoW, string dtype) to 3.0.0 whatsnew notes
State: open

We don't actually yet list the bigger features (string dtype, CoW, no silent downcasting) in the 3.0.0 whatsnew page, so starting to do that here.

Already pushed a section about string dtype, will further add a section about CoW and the downcasting.

Labels:
name: Docs
name: Release
name: Stale

Comments:
This pull request is stale because it has been open for thirty days with no activity. Please [update](https://pandas.pydata.org/pandas-docs/stable/development/contributing.html#updating-your-pull-request) and respond to this comment if you're still interested in working on this.
Going to merge this to have something in the docs, we can always further refine it so more feedback is certainly welcome!
DEPS: bump pyarrow minimum version from 10.0 to 12.0
State: closed

For our support window of 2 years, we can bump the minimum pyarrow version to 12.0.1 (see list of release dates here: https://arrow.apache.org/release/, we could also directly bump to 13 assuming the final 3.0 release will happen in 1-2 months).

Labels:
name: Dependencies
description: Required and optional dependencies
name: Arrow
description: pyarrow functionality

Comments:
@mroeschke this should be ready now (all tests are green)
> Could you also update the table under the `Increased minimum versions for dependencies` section in the `v3.0.0.rst` whatsnew?

Done!
Going to merge this so I can update https://github.com/pandas-dev/pandas/pull/61722
String dtype: turn on by default
State: closed

Now that 2.3.0 is released, time to switch the default on main to prepare for a 3.0 release (and also such that people using nightlies get this)

I assume this might also still uncover some failing tests. And the docstrings will still fail for sure.

Labels:
name: Strings
description: String extension data type and string data

Comments:
cc @rhshadrach I remember you gave the feedback when CoW option was removed (actually removed, without ability to turn CoW off), that it might be good to keep the option working for some time? (so that if you run into issues, you can still turn it off temporarily) 

Shall we do that here?

And if we do that:
- do we want to keep some CI testing that? (for example I could keep the build currently testing turning it on, but then with turning it off)
- add a warning for people setting the option to False that this is only kept working temporarily?
Aside docstrings, the only tests that were failing are the ones with the minimum pyarrow dependency of 10.0.1. For our support window of 2 years, we can bump that to 12.0.1. If that helps, will move it to a separate PR.
> Shall we do that here?

I would like to keep the option for users to switch back to the old string type if needed 
Any idea how many xfailed tests are specific to this? I saw some this morning, will see if I can address those.
looks like 20 tests (76 with parametrization) are xfailed.  At least some of these will be easy-but-tedious of updating `expected`s that I'll get out of the way now.  I don't see any reason why these would be a blocker for this PR.
To be able to move forward with enabling this on main, I added commits to temporarily disable failing CI for doc build or doctest failures. That will require more fixes, but also can only be done when this is enabled, and I would prefer doing that in separate (potentially multiple) PRs instead of doing it all at once. 

Another alternative would be to just keep the option disabled specifically in those builds (by setting `PANDAS_FUTURE_INFER_STRINGS=0` in those builds), so they can keep running and testing for now. But for the docstests, that still gives the same problem that then fixing them needs to be done in one go, so I think for doctests I prefer disabling errors temporarily, and then fixing all failures in separate PRs.
For the main doc build part, I have opened https://github.com/pandas-dev/pandas/pull/61864 to start fixing the build issues with the string dtype enabled
I'm fine with the two-step approach here + #61864.  @rhshadrach @mroeschke happy here?
+1
POC: PDEP16 default to masked nullable dtypes
State: closed

This is the second of several POCs stemming from the discussion in #61618 (see also #61708).  The main goal is to see how invasive it would be.

Specifically, this implements the part of PDEP16 #58988 that changes the default numeric/bool dtypes to use numpy-nullable dtypes.  So `pd.Series(foo)` will behave roughly like `pd.Series(pd.array(foo))` does in main.

Notes:
- For POC purposes this takes the stand that we _never_ give numpy numeric/bool dtypes and always map `dtype=np.int64` to the masked dtype.
- The get_option checks will need to be updated to user a more performant check like for `using_string_dtype`
- The simplification in core.internals.construction will eventually be reverted as the MaskedArrays are updated to support 2D.
- This does *not* incorporate #61708.
- Currently 16773 tests failing locally (with `-m "not slow and not db"`).  705 in window, 2036 in io (almost all of pytables is failing), 1997 (plus a ton more I already xfailed bc they get RecursionError) in computation.  tests.groupby.test_raises has 1110 that look to be mostly about getting the wrong class of exception or exception message.  Many in sparse too, though I don't have a number ATM.  Some of these merit issues:
    - [ ] #61709
    - [ ] #30188
    - [ ] RangeIndex.equals issue, see comments in asserters.py diff.


Labels:
name: PDEP missing values
description: Issues that would be addressed by the Ice Cream Agreement from the Aug 2023 sprint

Comments:
This served its purpose of getting an idea how difficult getting this working would be. Closing.
ENH: Enabled prefix, suffix, and sep to DataFrame.shift 
State: closed

…e periods (#61696)

- [X] closes #61696   
(Replace 61696 with the GitHub issue number)
- [X] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [X] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [X] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [X] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


- Enabled `prefix`, `suffix`, and `sep` arguments to `DataFrame.shift` for iterable periods
- Now it's cleaner and customizable column renaming
- Introduced new test in `test_shift_with_iterable_check_other_arguments`

Labels:

Comments:
Thanks you for the PR, but the original issue needs to be triaged by the team and agreed upon to pursue before opening a PR so going to close until those items are done
POC: NA-only behavior for numpy-nullable dtypes
State: closed

This is the first of several POCs stemming from the discussion in #61618.  The main goal is to see how invasive it would be.

Specifically, this implements the NaN behavior described in PDEP16 #58988.

Functionally this makes it so that:

1) With a Float64Dtype or Float32Dtype, you will *never* get a NaN, only a NA.
2) Users transitioning from numpy dtypes will be maximally-backwards-compatible

As a result, I expect implementing this would solve most issues labeled as "PDEP missing values".  e.g. I just checked and it does address #54876.



Labels:
name: PDEP missing values
description: Issues that would be addressed by the Ice Cream Agreement from the Aug 2023 sprint

Comments:
Im thinking the check should be done once at dtype instantiation
Closing in favor of #62040
DOC: add pandas 3.0 migration guide for the string dtype
State: closed

This PR starts adding a migration guide with some typical issues one might run into regarding the new string dtype when upgrading to pandas 3.0 (or when enabling it in pandas 2.3).

(for now I just added it to the user guide, which is already a long list of pages, so we might need to think about better organizing this or putting it elsewhere)

Closes #59328

Labels:
name: Docs
name: Strings
description: String extension data type and string data

Comments:
@simonjayhawkins thanks a lot for the proofreading!

/preview
Added three more sections based on the items listed in https://github.com/pandas-dev/pandas/issues/59328
cc @rhshadrach since you also already updated some code to work with the future string dtype, feel free to take a look and some feedback is certainly still welcome
TST: Refactor S3 tests
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: IO Network
description: Local or Cloud (AWS, GCS, etc.) IO Issues

Comments:
Thanks @fangchenli