BrandonFors/pandas-github-issues huggingface dataset:
BUG: Add fillna at the beginning of _where not to fill NA. #60729
State: open

- [X] closes #60729  (Replace xxxx with the GitHub issue number)
- [X] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [X] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [X] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

Added fillna at the beginning of _where so that we can fill pd.NA.
Since this is my first PR, please correct me if I'm mistaken. Thanks!

Labels:
name: Bug
name: Missing-data
description: np.nan, pd.NaT, pd.NA, dropna, isnull, interpolate

Comments:
FYI, it seems this has already been discussed at https://github.com/pandas-dev/pandas/pull/53124#issue-1698836456
Many thanks for your feedback, @rhshadrach .  All the changes you requested are now reflected. Could you review the changes?
Hi @mroeschke , Thanks for the suggestion! But it seems some tests fail when we changed the code `np.array()` to `extract_array(cond, extract_numpy=True)`. I'll revert the code for now and see if there is still room for improvement.
Hi @mroeschke ,
I've found that `where()` raises when `cond` is a list which is subset of `{False, True, np.nan}`.
For example:

```python
>>> import pandas as pd
>>> import numpy as np
>>> series = pd.Series([None,1,2,None,3,4,None])
>>> series.where([np.nan, False, False, np.nan, True, True, np.nan], -99)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\Users\sangg\Desktop\pandas-sanggon-from-code\pandas\pandas\core\generic.py", line 10032, in where
    return self._where(cond, other, inplace=inplace, axis=axis, level=level)
  File "C:\Users\sangg\Desktop\pandas-sanggon-from-code\pandas\pandas\core\generic.py", line 9732, in _where
    raise TypeError(msg.format(dtype=cond.dtype))
TypeError: Boolean array expected for the condition, not object
```
I think `where()` raises because `np.asanyarray()` returns a numpy array  with `float64` type, and should work without an error since it does work when `cond` has `pd.NA` in it.
Do I need to open another issue or fix this here?
My suggestion is basically to specify dtype at `L9715` as below.

```python
    def _where(

...

        if isinstance(cond, NDFrame):

...

        else:
            if not hasattr(cond, "shape"):
                dt = None
                if set(cond).issubset({True, False, np.nan}):
                    dt = bool
                cond = np.asanyarray(cond, dtype=dt)
            if cond.shape != self.shape:
                raise ValueError("Array conditional must be same shape as self")
```
[backport 2.3.x] ENH: Enable pytables to round-trip with StringDtype (#60663)
State: closed

(cherry picked from commit 60325b86e28edf40cb02444367efbc8deb2b5231)

Backport of https://github.com/pandas-dev/pandas/pull/60663

Labels:
name: Strings
description: String extension data type and string data

Comments:
Thanks @jorisvandenbossche
TST(string dtype): Fix xfails in test_block_internals.py
State: closed


Labels:
name: Strings
description: String extension data type and string data

Comments:
Owee, I'm MrMeeseeks, Look at me.

There seem to be a conflict, please backport manually. Here are approximate instructions:

1. Checkout backport branch and update it.

```
git checkout 2.3.x
git pull
```

2. Cherry pick the first parent branch of the this PR on top of the older branch:
```
git cherry-pick -x -m1 d38706af66249ef74e42671a480261c68bedfbce
```

3. You will likely have some merge/cherry-pick conflict here, fix them and commit:

```
git commit -am 'Backport PR #60765: TST(string dtype): Fix xfails in test_block_internals.py'
```

4. Push to a named branch:

```
git push YOURFORK 2.3.x:auto-backport-of-pr-60765-on-2.3.x
```

5. Create a PR against branch 2.3.x, I would have named this PR:

> "Backport PR #60765 on branch 2.3.x (TST(string dtype): Fix xfails in test_block_internals.py)"

And apply the correct labels and milestones.

Congratulations — you did some good work! Hopefully your backport PR will be tested by the continuous integration and merged soon!

Remember to remove the `Still Needs Manual Backport` label once the PR gets merged.

If these instructions are inaccurate, feel free to [suggest an improvement](https://github.com/MeeseeksBox/MeeseeksDev).
                
Manual backport -> https://github.com/pandas-dev/pandas/pull/60781
TST (string dtype): follow-up fix for pyarrow 19.0 update
State: closed

Small follow-up on https://github.com/pandas-dev/pandas/pull/60716. This change is already included in the backport PR https://github.com/pandas-dev/pandas/pull/60755, so targeting 3.0 here (on 2.3.x this was needed because there for some reason more jobs were already using pyarrow 19.0)

Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Strings
description: String extension data type and string data
name: IO Parquet
description: parquet, feather

Comments:
```bash
=========================== short test summary info ============================
FAILED pandas/tests/io/test_parquet.py::TestBasic::test_parquet_read_from_url[pyarrow] - AssertionError: DataFrame.columns are different

Attribute "dtype" are different
[left]:  object
[right]: StringDtype(storage=pyarrow, na_value=nan)
```
Thanks @jorisvandenbossche
CI: Update some CI configurations
State: closed

* Bumps mysql/postgres/moto container services
* Makes musl and 32 bit jobs run on Python 3.13
* Bumps musl container image (previous one was EOL last year)
* Consolidates some `pip install`s

Labels:
name: CI
description: Continuous Integration

Comments:
@lithomas1 I reverted to enabling the python-dev build again with 3.13. Not all our optional dependencies have 3.13 wheels yet :(
Going to merge this PR since the test are passing
CI: Remove CircleCI in favor of GHA ARM builds
State: closed

After https://github.com/pandas-dev/pandas/pull/60722, our unit test and wheel builds work successfully on GHA ARM runners. Additionally, now we're successfully uploading nightly ARM wheels to https://anaconda.org/scientific-python-nightly-wheels/pandas/files. Therefore, we can remove CircleCI from our CI

Labels:
name: CI
description: Continuous Integration

Comments:
Looks good - we should target 2.3 right?
We first have to backport https://github.com/pandas-dev/pandas/pull/60722 (or the wheel part of it) then, if we want to backport this as well (and we probably should yes)
Owee, I'm MrMeeseeks, Look at me.

There seem to be a conflict, please backport manually. Here are approximate instructions:

1. Checkout backport branch and update it.

```
git checkout 2.3.x
git pull
```

2. Cherry pick the first parent branch of the this PR on top of the older branch:
```
git cherry-pick -x -m1 f3045db91dbb89306c15b1673987cc70912a76b5
```

3. You will likely have some merge/cherry-pick conflict here, fix them and commit:

```
git commit -am 'Backport PR #60761: CI: Remove CircleCI in favor of GHA ARM builds'
```

4. Push to a named branch:

```
git push YOURFORK 2.3.x:auto-backport-of-pr-60761-on-2.3.x
```

5. Create a PR against branch 2.3.x, I would have named this PR:

> "Backport PR #60761 on branch 2.3.x (CI: Remove CircleCI in favor of GHA ARM builds)"

And apply the correct labels and milestones.

Congratulations — you did some good work! Hopefully your backport PR will be tested by the continuous integration and merged soon!

Remember to remove the `Still Needs Manual Backport` label once the PR gets merged.

If these instructions are inaccurate, feel free to [suggest an improvement](https://github.com/MeeseeksBox/MeeseeksDev).
                
Manual backport -> https://github.com/pandas-dev/pandas/pull/60790
@mroeschke I also removed the circleci webhook in the settings then
Miscellaneous updates for Pyodide 0.27: bump WASM CI and revise Arrow compatibility note
State: closed

- [ ] ~closes #xxxx (Replace xxxx with the GitHub issue number)~
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] ~Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.~
- [ ] ~Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.~

I noticed via #60747 that the Pyodide CI job for `pandas` in `unit-tests.yml` is still using a version of Pyodide[^1] that is quite old by this point; we no longer support it, as we have recently released Pyodide 0.27 this January (https://github.com/pyodide/pyodide/releases/tag/0.27.1). On the other hand, `cibuildwheel` in `wheels.yml` builds against Pyodide 0.26 at the moment.

Other notes around the changes here:
- we unvendored `pyodide-build` from the Pyodide runtime/repository, so its version does not stay in sync with the Pyodide runtime version.
- the Node.js version has been bumped to 20, though 22 should work, too.
- the Pyodide cross-build environment folder has been added to the `.gitignore` file
- PyArrow is now [supported](https://github.com/joemarshall/pyarrow-pyodide/) in Pyodide/WASM ([docs on compilation](https://arrow.apache.org/docs/developers/cpp/emscripten.html)), and I've updated the PDEP-10 document here to remove the point surrounding its previous unavailability in Pyodide/PyScript/other WASM environments.
- I removed an expected failure case in `TestCoercionFloat32.test_setitem` that now xpasses with Emscripten 3.1.58.

Please let me know if you want me to split any of the changes to a different PR. Thank you!

[^1]: I have plans in mind to make these upgrades more seamless via https://github.com/pyodide/pyodide-actions/issues/12 by letting Dependabot do such a job, but I haven't got around to sitting down and spending a bit of time doing so.

Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: CI
description: Continuous Integration

Comments:
> On the other hand, cibuildwheel in wheels.yml builds against Pyodide 0.26 at the moment.

Could you bump pyodide in for the wheel builds then to match the version in our unit tests?
> > On the other hand, cibuildwheel in wheels.yml builds against Pyodide 0.26 at the moment.
> 
> Could you bump pyodide in for the wheel builds then to match the version in our unit tests?

I could,  but it needs this PR to make it to a release: https://github.com/pypa/cibuildwheel/pull/2117
> I could, but it needs this PR to make it to a release: https://github.com/pypa/cibuildwheel/pull/2117

OK no problem. Thanks @agriyakhetarpal
ENH: Support skipna parameter in GroupBy min, max, prod, median, var, std and sem methods
State: closed

- [ ] Closes #15675(Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/v3.0.0.rst` file if fixing a bug or adding a new feature.

Second (and final) batch of GroupBy reductions being enhanced to support the `skipna` parameter.

Labels:
name: Enhancement
name: Groupby
name: Missing-data
description: np.nan, pd.NaT, pd.NA, dropna, isnull, interpolate

Comments:
> Looks great! Can you add a test (adding to your current parametrizations would be fine) where the entire group is NA.

Thanks for the review @rhshadrach. Added the all-NA tests and responded to comments.
Failure on the future infer string is unrelated (and is fixed by #60796). Rerunning Ubuntu 310 just to be sure.
@WillAyd  - you good here?
Thanks @snitish
refactor: deprecate usage of `cursor.execute` statements in favor of the in class implementation of `execute`.
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Refactor
description: Internal refactoring of code
name: IO SQL
description: to_sql, read_sql, read_sql_query

Comments:
Hello @WillAyd.
So this is the follow up of https://github.com/pandas-dev/pandas/pull/60376.
I updated the code base to use panda's `execute` implementation as much as possible. I couldn't replace all places and a simple `git grep '.exec' -- pandas/io/sql.py` will show you that. Anyways what in my opinion is worth mentioning and asking for a review is in the following:
1. [SQLTable._execute_insert](https://github.com/pandas-dev/pandas/blob/ae42f3e1c1b7af55059921b41fd61710fe2dd785/pandas/io/sql.py#L987)
2. [SQLTable._execute_insert_multi](https://github.com/pandas-dev/pandas/blob/ae42f3e1c1b7af55059921b41fd61710fe2dd785/pandas/io/sql.py#L1003)
3. [SQLiteTable._execute_insert](https://github.com/pandas-dev/pandas/blob/ae42f3e1c1b7af55059921b41fd61710fe2dd785/pandas/io/sql.py#L2522)
4. [SQLiteTable._execute_insert_multi](https://github.com/pandas-dev/pandas/blob/ae42f3e1c1b7af55059921b41fd61710fe2dd785/pandas/io/sql.py#L2527)

This should be no problem because we can always wrap that execution around a `try-except` block (as I did in SQLiteTable._execute_insert)

Would you mind taking a look and LMK what you think while in draft ?
Hello @WillAyd ,

All tests passed [here](https://github.com/pandas-dev/pandas/pull/60748/commits/71e0d24548014cd638b435a9494629576ae1a074) and therefore I'm opening this PR to review. I have updated the documentation :).
Looks like this PR needs a rebase with the main branch
Thanks @gmcrocetti
DOC: Clarify deprecation warning for iloc
State: closed

Changed the wording of the deprecation warning to make clearer what exactly is deprecated.

The previous formulation was confusing (see e.g. [here](https://www.reddit.com/r/learnprogramming/comments/1c6a20o/is_pandas_iloc_being_depreciated/), even though the change from the "Deprecated since..." warning in 2.2 to "Changed in version 3.0" makes it slightly better). It is not "returning a tuple" which is deprecated but to use such a callable as an input/index.

Labels:
name: Docs

Comments:
Thanks @grossardt
BUG: Ensure to_datetime raises errors for out-of-bounds scalar inputs
State: closed

- [x] closes #60677 (Replace xxxx with the GitHub issue number)
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.



Labels:
name: Stale

Comments:
This pull request is stale because it has been open for thirty days with no activity. Please [update](https://pandas.pydata.org/pandas-docs/stable/development/contributing.html#updating-your-pull-request) and respond to this comment if you're still interested in working on this.
Thanks for the pull request, but it appears to have gone stale. If interested in continuing, please merge in the main branch, address any review comments and/or failing tests, and we can reopen.
ENH: Support skipna parameter in GroupBy mean and sum
State: closed

- [ ] partially addresses #15675
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/v3.0.0.rst` file if fixing a bug or adding a new feature.

This PR would only partially close #15675. There are other reductions that still need to support `skipna`. Once approved, I plan to address the remaining.

Labels:
name: Groupby

Comments:
Thanks @snitish 
Thanks @mroeschke for approving. Since there are some leftover reduction operations in the original list that still need to support `skipna`, could you please reopen the original issue #15675 ?
Sure thing done.
ENH: `pandas.api.interchange.from_dataframe` now uses the Arrow PyCapsule Interface if available, only falling back to the Dataframe Interchange Protocol if that fails
State: closed

There's some related discussion to the interchange protocol in #56732 

Regardless of whether it gets deprecated, what we can already do is prefer the PyCapsule Interface if it's available. I'd discussed this informally with @WillAyd 

`pandas.api.interchange.from_dataframe` is still used in some places (Seaborn, Plotly<6.0, Altair<5.4) where people may want to upgrade pandas whilst keeping some other deps pinned, so this should make the transition to the PyCapsule Interface seamless for them

Labels:
name: Interchange
description: Dataframe Interchange Protocol

Comments:
Thanks @MarcoGorelli 
@meeseeksdev backport 2.3.x
Awww, sorry MarcoGorelli you do not seem to be allowed to do that, please ask a repository maintainer.
DOC: add SPSS comparison guide structure
State: closed

PR adds a comparison guide for SPSS users transitioning to pandas, addressing #60727. 

## Changes
- Creates new `comparison_with_spss.rst` in getting_started/comparison/
- Documents basic data structure mappings:
  - SPSS data files → pandas DataFrames
  - SPSS variables → pandas Series
  - SPSS case numbers → pandas Index
- Sets up structure for common operations comparison

## Implementation
- Following existing comparison guide formats (R, SQL, etc.)
- Using reStructuredText format
- Including executable code examples
- Cross-referencing pandas documentation

- [ x] closes #60727

Labels:
name: Docs

Comments:
Hi, I am not sure why this check is not passing. Could you please provide some insight?
Thanks @Jacob-Lazar
CLN: Refactor np_can_hold_element to reduce cyclomatic complexity
State: closed

### Overview

This PR refactors the `np_can_hold_element` function to reduce its cyclomatic complexity and improve maintainability. The function was decomposed into smaller helper functions (`_handle_integer_dtype`, `_handle_float_dtype`, etc.), each handling a specific dtype logic. This improves readability and reduces the complexity of the main function.

### Changes

- Refactored `np_can_hold_element` into smaller helper functions:
  - `_handle_integer_dtype`
  - `_handle_float_dtype`
  - `_handle_complex_dtype`
  - `_handle_boolean_dtype`
  - `_handle_string_dtype`
- Added docstrings to the helper functions for better documentation.
- Ensured no changes to functionality or behavior during the refactor.
- Ran `flake8` and fixed style issues to comply with the project's code standards.

### Testing

- Verified that all existing tests in the Pandas test suite pass.
- Confirmed no behavioral changes in `np_can_hold_element`.

### Notes for Reviewers

- The refactor was aimed at maintainability; the logic remains consistent with the original implementation.
- Suggestions for additional tests or improvements are welcome.

Labels:

Comments:
was in pandas/core/dtypes/cast.py:
![image](https://github.com/user-attachments/assets/e7800ea2-7f9e-47c1-857c-1dfc96bfacda)
now in pandas/core/dtypes/cast.py:
![image](https://github.com/user-attachments/assets/2e27030c-490a-4479-8298-3d751a953d04)

Thanks for the effort here but the project doesn't enforce any standards with regards to complexity so I don't think this refactor is worthwhile. Going to close but happy to have contributes toward issues labeled `good first issue`
ENH: Add `Styler.to_typst()`
State: closed

- [x] closes #57617
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Enhancement
name: Styler
description: conditional formatting using DataFrame.style

Comments:
I think this is a nice, minimalist implementation of `Styler.to_typst`, honouring some of the ideas in the original discussion. The coding mirrors the implementation for other formats in terms of signature and template composition.

Provided the tests pass I am supportive of adding this in. Would need a "whatsnew"
Hi @attack68,
I have updated the "What's new". Could you please review it?
@mroeschke want to have a second look?
Thanks @3w36zj6
ENH: Support 'left_anti' and 'right_anti' joins in pd.merge
State: closed

- [ ] closes #42916
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/v3.0.0.rst` file if fixing a bug or adding a new feature.


Labels:
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode

Comments:
Awesome work. Thanks @snitish
DOC: Fix typo "numpy.ndarray.putmask"
State: closed

Should be "numpy.putmask"

- ~~[ ] closes #xxxx (Replace xxxx with the GitHub issue number)~~
- ~~[ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature~~
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- ~~[ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.~~
- ~~[ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.~~

Labels:
name: Docs

Comments:
Thanks @wjandrea