BrandonFors/pandas-github-issues huggingface dataset:
String dtype: allow string dtype for non-raw apply with numba engine
State: closed

https://github.com/pandas-dev/pandas/pull/55104 enabled using numba with `raw=False` in `apply()` (BTW, this seems rather undocumented?), but that PR had some specific hack for object dtype index. Extending that hack to also convert str dtype index to numpy strings under the hood (this might hurt performance, but for now just want to get this working)




xref https://github.com/pandas-dev/pandas/issues/54792

Labels:
name: Strings
description: String extension data type and string data
name: Apply
description: Apply, Aggregate, Transform, Map
name: numba
description: numba-accelerated operations
name: backported

Comments:
Thanks @jorisvandenbossche
String dtype: implement sum reduction
State: closed

Based on the feedback in https://github.com/pandas-dev/pandas/issues/59328, implementing `sum()` for the string dtype

xref https://github.com/pandas-dev/pandas/issues/54792

Labels:
name: Strings
description: String extension data type and string data
name: Reduction Operations
description: sum, mean, min, max, etc.

Comments:
Hmm wouldn't it be better to just make users cast to object if they want the legacy behavior on this? I think it was intentional to not implement sum on the string dtype; while there may be some niche use cases, more often than not I think its a mistake (and huge performance hit) to have string types summed up
> while there may be some niche use cases, more often than not I think its a mistake (and huge performance hit)

This is not my personal experience.


@WillAyd it was indeed intentional in the past to not implement it, but see the thread in https://github.com/pandas-dev/pandas/issues/59328, and if you want to discuss this, let's do that there ;) (I personally don't care too much about this specifically, but can follow some of the arguments given there, and just implementing what has been decided for now)
I am planning to merge this tomorrow (given it is one of the last remaining bigger changes for the string dtype, and including a bunch of test updates / blocking other test updates), but some more review is certainly still welcome
Owee, I'm MrMeeseeks, Look at me.

There seem to be a conflict, please backport manually. Here are approximate instructions:

1. Checkout backport branch and update it.

```
git checkout 2.3.x
git pull
```

2. Cherry pick the first parent branch of the this PR on top of the older branch:
```
git cherry-pick -x -m1 2fdb16b347fc34f78213868a8a973447ac79ab2d
```

3. You will likely have some merge/cherry-pick conflict here, fix them and commit:

```
git commit -am 'Backport PR #59853: String dtype: implement sum reduction'
```

4. Push to a named branch:

```
git push YOURFORK 2.3.x:auto-backport-of-pr-59853-on-2.3.x
```

5. Create a PR against branch 2.3.x, I would have named this PR:

> "Backport PR #59853 on branch 2.3.x (String dtype: implement sum reduction)"

And apply the correct labels and milestones.

Congratulations — you did some good work! Hopefully your backport PR will be tested by the continuous integration and merged soon!

Remember to remove the `Still Needs Manual Backport` label once the PR gets merged.

If these instructions are inaccurate, feel free to [suggest an improvement](https://github.com/MeeseeksBox/MeeseeksDev).
                
Manual backport -> https://github.com/pandas-dev/pandas/pull/60157
typo
State: closed

_cf_: https://github.com/pandas-dev/pandas/pull/59665

@mroeschke, @mattwang44

pandas/io/pytables.py

reverts this change which shouldn't have been implemented

```diff
-         """return a tuple of my permutated axes, non_indexable at the front"""
+         """return a tuple of my permuted axes, non_indexable at the front"""
```

my apologies

@jreback 



`https://github.com/pandas-dev/pandas/pull/2497`

_cf_: ENH: ndim tables in HDFStore

Labels:
name: Docs

Comments:
Thanks @musvaage 
> shouldn't have been implemented

string initially surmised as a typo

just to be clear, my work on prior pulls never assessed which string is contextually correct

so possibly refer to the NumPy docs on `axes`
DOC: Separate out examples - pandas.str.is methods
State: closed

- [x] closes #59806
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).


Labels:
name: Docs
name: Strings
description: String extension data type and string data

Comments:
Hi @rhshadrach, I have refactored the docstrings to separate the `Examples` and `See Also` sections for each `ismethod`, no code changes were made and tested the changes locally. However, I noticed that the unit tests are now failing, which seems to be related to an issue with the CI script. Could you kindly review the changes and provide your feedback when possible? Thank you.


/preview
Website preview of this PR available at: https://pandas.pydata.org/preview/pandas-dev/pandas/59850/
Thanks @saldanhad!
RLS: 2.2.3 testing
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Build
description: Library building on various platforms

Comments:
Looks like this is all green closing
BLD: Build wheels for Python 3.13 on aarch64 as well
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Build
description: Library building on various platforms

Comments:
CircleCI is green, so merging on yellow.
Owee, I'm MrMeeseeks, Look at me.

There seem to be a conflict, please backport manually. Here are approximate instructions:

1. Checkout backport branch and update it.

```
git checkout 2.2.x
git pull
```

2. Cherry pick the first parent branch of the this PR on top of the older branch:
```
git cherry-pick -x -m1 2419343bfea5dba678146139ca9663d831c47b22
```

3. You will likely have some merge/cherry-pick conflict here, fix them and commit:

```
git commit -am 'Backport PR #59847: BLD: Build wheels for Python 3.13 on aarch64 as well'
```

4. Push to a named branch:

```
git push YOURFORK 2.2.x:auto-backport-of-pr-59847-on-2.2.x
```

5. Create a PR against branch 2.2.x, I would have named this PR:

> "Backport PR #59847 on branch 2.2.x (BLD: Build wheels for Python 3.13 on aarch64 as well)"

And apply the correct labels and milestones.

Congratulations — you did some good work! Hopefully your backport PR will be tested by the continuous integration and merged soon!

Remember to remove the `Still Needs Manual Backport` label once the PR gets merged.

If these instructions are inaccurate, feel free to [suggest an improvement](https://github.com/MeeseeksBox/MeeseeksDev).
timedelta on datetime df, series arithmetic
State: closed

- [x] closes #59529 (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:

Comments:
Thanks for the pull request, but it appears to have gone stale. If interested in continuing, please merge in the main branch, address any review comments and/or failing tests, and we can reopen.
ENH: Add kwargs to Series.map
State: closed

- [x] closes #59814
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Enhancement

Comments:
Thanks @nemausus
Backport PR #59840: BLD: Final release prep for 2.2.3
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Build
description: Library building on various platforms

Comments:
/preview
BLD: Final release prep for 2.2.3
State: closed

- [ ] closes #58632  (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Build
description: Library building on various platforms

Comments:
/preview
/preview
Website preview of this PR available at: https://pandas.pydata.org/preview/pandas-dev/pandas/59840/
Owee, I'm MrMeeseeks, Look at me.

There seem to be a conflict, please backport manually. Here are approximate instructions:

1. Checkout backport branch and update it.

```
git checkout 2.2.x
git pull
```

2. Cherry pick the first parent branch of the this PR on top of the older branch:
```
git cherry-pick -x -m1 f1e6cc184ae0534e11c0a2947f4948bc4c5e0a9d
```

3. You will likely have some merge/cherry-pick conflict here, fix them and commit:

```
git commit -am 'Backport PR #59840: BLD: Final release prep for 2.2.3'
```

4. Push to a named branch:

```
git push YOURFORK 2.2.x:auto-backport-of-pr-59840-on-2.2.x
```

5. Create a PR against branch 2.2.x, I would have named this PR:

> "Backport PR #59840 on branch 2.2.x (BLD: Final release prep for 2.2.3)"

And apply the correct labels and milestones.

Congratulations — you did some good work! Hopefully your backport PR will be tested by the continuous integration and merged soon!

Remember to remove the `Still Needs Manual Backport` label once the PR gets merged.

If these instructions are inaccurate, feel free to [suggest an improvement](https://github.com/MeeseeksBox/MeeseeksDev).
BLD: Fix bad Cython annotation
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

Cython changed something, this fixes the 3.13 builds using the latest Cython


Labels:
name: Build
description: Library building on various platforms
name: Compat
description: pandas objects compatability with Numpy or Python functions
name: Python 3.13

Comments:
> thanks @lithomas1 , it was probably my fault for having written it like that

No not at all, it wasn't until very recently that Cython tried to optimize Python annotations.
Backport PR #59136 on branch 2.2.x (Upload 3.13 & free-threaded nightly wheels)
State: closed

Backports of:

- https://github.com/pandas-dev/pandas/pull/59208
- https://github.com/pandas-dev/pandas/pull/59136
- https://github.com/pandas-dev/pandas/pull/59401
- https://github.com/pandas-dev/pandas/pull/59819

which together should enable Python 3.13 wheels

Labels:
name: Build
description: Library building on various platforms

Comments:
Should probably directly include https://github.com/pandas-dev/pandas/pull/59819 as well, once that is merged
wait for #59837
Thanks Joris!
DOC: Adding ArcticDB to the ecosystem.md page
State: closed

This PR adds ArcticDB to the ecosystem.md under the IO section. It includes basic read and write examples to help users get started with saving pandas DataFrames to ArcticDB.

this PR is the fixes from this closed PR https://github.com/pandas-dev/pandas/pull/59820

(I accidently closed the older PR I'm not too sure how I'm not a git expert) 



Labels:
name: Docs
name: Web
description: pandas website

Comments:
Thanks @ms041223
DOC: Emphasize team managed pandas in installation docs
State: closed

- [x] closes #59763 (Replace xxxx with the GitHub issue number)

Additionally:

* Emphasize pandas distributions that the team manages
* Simplify some explanations
* Have the website installation instructions just point to the doc installation instructions to avoid these instructions diverging

Labels:
name: Docs

Comments:
Going to merge, but happy to follow up if needed
Backport PR #59818 on branch 2.2.x (BUG: Remove np._get_promotion_state usage)
State: closed

Backport PR #59818 on branch 2.2.x (BUG: Remove np._get_promotion_state usage)

(cherry picked from commit 081dcdee8d754af90e307cf2311b06b3d02fae2a)


Labels:

Comments:
The CI is not yet expected to pass (see https://github.com/pandas-dev/pandas/pull/59785), but the diff looks good to me, so just going to merge as is.
BLD/RLS: build wheels with released numpy/cython for Python 3.13
State: closed

Now that there is a released numpy that supports Python 3.13, we can clean up parts of our TODOs related to that (from https://github.com/pandas-dev/pandas/pull/59136) in our wheel building workflow.

It was noted in https://github.com/pandas-dev/pandas/issues/59817 that our wheel builds were failing because of testing with numpy dev instead of released numpy, a problem which itself is being fixed by https://github.com/pandas-dev/pandas/pull/59818

Labels:
name: Build
description: Library building on various platforms
name: CI
description: Continuous Integration

Comments:
(note: we should milestone this as 2.2.3, but the previous PRs adding python 3.13 support have not yet been backported)
> (note: we should milestone this as 2.2.3, but the previous PRs adding python 3.13 support have not yet been backported)

Previous PRs are being backported in https://github.com/pandas-dev/pandas/pull/59835
Owee, I'm MrMeeseeks, Look at me.

There seem to be a conflict, please backport manually. Here are approximate instructions:

1. Checkout backport branch and update it.

```
git checkout 2.2.x
git pull
```

2. Cherry pick the first parent branch of the this PR on top of the older branch:
```
git cherry-pick -x -m1 22372175e04f05f73521cab1b26f0818d6766717
```

3. You will likely have some merge/cherry-pick conflict here, fix them and commit:

```
git commit -am 'Backport PR #59819: BLD/RLS: build wheels with released numpy/cython for Python 3.13'
```

4. Push to a named branch:

```
git push YOURFORK 2.2.x:auto-backport-of-pr-59819-on-2.2.x
```

5. Create a PR against branch 2.2.x, I would have named this PR:

> "Backport PR #59819 on branch 2.2.x (BLD/RLS: build wheels with released numpy/cython for Python 3.13)"

And apply the correct labels and milestones.

Congratulations — you did some good work! Hopefully your backport PR will be tested by the continuous integration and merged soon!

Remember to remove the `Still Needs Manual Backport` label once the PR gets merged.

If these instructions are inaccurate, feel free to [suggest an improvement](https://github.com/MeeseeksBox/MeeseeksDev).
                
Backported in https://github.com/pandas-dev/pandas/pull/59835
BUG: Remove np._get_promotion_state usage
State: closed

- [x] closes #59817 
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).


Labels:
name: CI
description: Continuous Integration
name: Compat
description: pandas objects compatability with Numpy or Python functions

Comments:
Owee, I'm MrMeeseeks, Look at me.

There seem to be a conflict, please backport manually. Here are approximate instructions:

1. Checkout backport branch and update it.

```
git checkout 2.2.x
git pull
```

2. Cherry pick the first parent branch of the this PR on top of the older branch:
```
git cherry-pick -x -m1 081dcdee8d754af90e307cf2311b06b3d02fae2a
```

3. You will likely have some merge/cherry-pick conflict here, fix them and commit:

```
git commit -am 'Backport PR #59818: BUG: Remove np._get_promotion_state usage'
```

4. Push to a named branch:

```
git push YOURFORK 2.2.x:auto-backport-of-pr-59818-on-2.2.x
```

5. Create a PR against branch 2.2.x, I would have named this PR:

> "Backport PR #59818 on branch 2.2.x (BUG: Remove np._get_promotion_state usage)"

And apply the correct labels and milestones.

Congratulations — you did some good work! Hopefully your backport PR will be tested by the continuous integration and merged soon!

Remember to remove the `Still Needs Manual Backport` label once the PR gets merged.

If these instructions are inaccurate, feel free to [suggest an improvement](https://github.com/MeeseeksBox/MeeseeksDev).
BUG: read_json silently ignores the dtype when engine=pyarrow
State: closed

- [x] closes #59516  (Replace xxxx with the GitHub issue number)
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:

Comments:
Thanks for the pull request, but it appears to have gone stale. If interested in continuing, please merge in the main branch, address any review comments and/or failing tests, and we can reopen.
CI: Debug failing ARM builds
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


The non musl arm builds have been failing for a while now.

Labels:
name: Build
description: Library building on various platforms

Comments:
I don't have access to an ARM machine, so I'm probably just going to skip the tests for now
(given that the failures are for the sparse extension arrays which aren't really maintained anymore)
pre-commit.ci autofix
DOC: add whatsnew for v2.2.3
State: closed

Add basic whatsnew file skeleton (like https://github.com/pandas-dev/pandas/pull/57759 for 2.2.2), so we can backport this and start add some content.

Labels:
name: Docs

Comments:
/preview
Website preview of this PR available at: https://pandas.pydata.org/preview/pandas-dev/pandas/59811/
thanks @jorisvandenbossche
String dtype: allow string dtype in query/eval with default numexpr engine
State: closed

Currently (on main, with no future options enabled), you can use `query` on an object-dtype column of strings. For exeample:

```python
>>> df = pd.DataFrame({"col" : ['a', 'b', 'c']})
>>> df.query("col < 'b'") 
  col
0   a
```

This works despite that the default engine is "numexpr" (when installed) and that this engine does not actually evaluate the above expression. We do have checks to essentially fall back to evaluate the expression in python if one of the operands is object dtype (and the operator is one of the allowed operators for that). 

But with the future default string dtype, we trigger the warning about explicitly falling back to the python engine:

```python
>>> pd.options.future.infer_string = True
>>> df = pd.DataFrame({"col" : ['a', 'b', 'c']})
>>> df.query("col < 'b'")
RuntimeWarning: Engine has switched to 'python' because numexpr does not support extension array dtypes.
Please set your engine to python manually.
  col
0   a
```

Given that we were already automatically falling back for object dtype strings, I would propose to keep doing that for the future default string dtype as well.  
It is a bit misleading (you might think to get a speedup when using this, which won't be the case), but that is already the case right now anyway, and otherwise a lot of users will start to see that warning (and a lot of people also use `query` for the convenience of writing the expression, I think, and not only for the speed).

xref https://github.com/pandas-dev/pandas/issues/54792

Labels:
name: Strings
description: String extension data type and string data
name: expressions
description: pd.eval, query
name: backported

Comments:
Thanks @jorisvandenbossche
BUG: Fix from_records() column reorder issue, if columns!=None use passed param (#59717)
State: closed

- [x] closes #59717 
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Constructors
description: Series/DataFrame/Index/pd.array Constructors

Comments:
Thanks @VibavariG