BrandonFors/pandas-github-issues huggingface dataset:
PERF: Series.str.get for pyarrow-backed strings
State: closed

- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/v2.1.0.rst` file if fixing a bug or adding a new feature.

```
import pandas as pd
import pandas._testing as tm
import pyarrow as pa

N = 1_000_000

ser = pd.Series(tm.makeStringIndex(N), dtype=pd.ArrowDtype(pa.string()))

%timeit ser.str.get(1)

# 70.1 ms ± 3.13 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)  -> main
# 37.7 ms ± 2.09 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)  -> PR
```

Labels:
name: Performance
description: Memory or execution speed performance
name: Arrow
description: pyarrow functionality

Comments:
```
sort whatsnew entries alphabetically....................................................................Failed
- hook id: sort-whatsnew-items
- duration: 0.04s
- exit code: 1
- files were modified by this hook
pre-commit hook(s) made changes.
If you are seeing this message in CI, reproduce locally with: `pre-commit run --all-files`.
To run `pre-commit` as part of git workflow, use `pre-commit install`.
All changes made by hooks:
diff --git a/doc/source/whatsnew/v2.1.0.rst b/doc/source/whatsnew/v2.1.0.rst
index 44ba2ac..a7ad001 100644
--- a/doc/source/whatsnew/v2.1.0.rst
+++ b/doc/source/whatsnew/v2.1.0.rst
@@ -290,10 +290,10 @@ Performance improvements
 - Performance improvement in :meth:`.DataFrameGroupBy.groups` (:issue:`53088`)
 - Performance improvement in :meth:`DataFrame.loc` when selecting rows and columns (:issue:`53014`)
 - Performance improvement in :meth:`Series.corr` and :meth:`Series.cov` for extension dtypes (:issue:`52502`)
+- Performance improvement in :meth:`Series.str.get` for pyarrow-backed strings (:issue:`53152`)
 - Performance improvement in :meth:`Series.to_numpy` when dtype is a numpy float dtype and ``na_value`` is ``np.nan`` (:issue:`52430`)
 - Performance improvement in :meth:`~arrays.ArrowExtensionArray.to_numpy` (:issue:`52525`)
 - Performance improvement when doing various reshaping operations on :class:`arrays.IntegerArrays` & :class:`arrays.FloatingArray` by avoiding doing unnecessary validation (:issue:`53013`)
-- Performance improvement in :meth:`Series.str.get` for pyarrow-backed strings (:issue:`53152`)
 -
```
Thanks @lukemanley
PERF: Series.add for pyarrow string/binary dtypes
State: closed

- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/v2.1.0.rst` file if fixing a bug or adding a new feature.

```
import pandas as pd
import pandas._testing as tm
import pyarrow as pa

N = 1_000_000
s1 = pd.Series(tm.makeStringIndex(N), dtype=pd.ArrowDtype(pa.string()))
s2 = pd.Series(tm.makeStringIndex(N), dtype=pd.ArrowDtype(pa.string()))

%timeit s1 + s2
# 136 ms ± 3.25 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)   -> main
# 61 ms ± 2.86 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)    -> PR

%timeit s1 + "abc"
# 200 ms ± 12.2 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)   -> main
# 64.5 ms ± 2.95 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)  -> PR
```

Labels:
name: Performance
description: Memory or execution speed performance
name: Arrow
description: pyarrow functionality

Comments:
Nice thanks @lukemanley
MNT: Add support of Cython 3 master
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

This PR adds support for Cython 3 master after PR https://github.com/cython/cython/pull/5386. This PR requires Cython >= 0.29.31 or Cython >= 3.0b3. See https://github.com/pandas-dev/pandas/issues/53125#issuecomment-1539086575 for details.


Labels:
name: Internals
description: Related to non-user accessible pandas implementation

Comments:
Thanks @matusvalo
MNT: Explicitly set cdef functions as noexcept
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

This PR marks functions returning `void` as `noexcept` to avoid checking an exception after every function call. `inline` functions are marked too to avoid unnecessary overhead. The improvement is only in Cython 3. In Cython 0.29.X `noexcept` keyword is ignored. See https://github.com/pandas-dev/pandas/issues/53125#issuecomment-1539086575 for details.

Labels:
name: Internals
description: Related to non-user accessible pandas implementation

Comments:
Thanks @matusvalo
BUG: Setting frame into df with dup cols loses dtypes
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Bug
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves

Comments:
Thanks @phofl
CLN: Cleanup after CoW setitem PRs
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

Follow-up on https://github.com/pandas-dev/pandas/pull/51698 and linked PRs

Labels:
name: Copy / view semantics
name: Stale

Comments:
This pull request is stale because it has been open for thirty days with no activity. Please [update](https://pandas.pydata.org/pandas-docs/stable/development/contributing.html#updating-your-pull-request) and respond to this comment if you're still interested in working on this.
cc @jorisvandenbossche gentle ping
Add test case for applying a function to a groupby object that append…
State: closed

…s each group to a list without using copy() function. The test checks if the groups are correctly appended to the list. This resolves GH issue 17718.

- [ ] closes #17718 (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Testing
description: pandas testing functions or related to the test suite

Comments:
> Hi,
> Some checks are failing because of the formatting of the code.
> here is the [documentation](https://pandas.pydata.org/docs/dev/development/contributing_codebase.html#contributing-pre-commit) for `pre-commit`.
> Could you install it and then commit again?

OK, thank you :)
Thanks @idor980
TYP: remove mypy ignore[assignment] from pandas/core/internals/construction.py
State: closed

Related to #37715
mypy ignore[assignment] was removed from pandas/core/internals/construction.py

Labels:
name: Typing
description: type annotations, mypy/pyright type checking

Comments:
Hi @MarcoGorelli, could you please take a look at this pr?
let's ship it

if you're interested in typing, I'd highly suggest you take a look at https://github.com/pandas-dev/pandas-stubs - that affects the type hints which users get, there's plenty of really useful issues to work on there (and feel free to reach out to Irv if you're not sure where to start)
DOC: Updated applymap docstring
State: closed

- [X] closes #53097 (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

Removed _index part from the docstring for applymap.


Labels:
name: Docs
name: Styler
description: conditional formatting using DataFrame.style

Comments:
Thanks @Rajasvi
TST: Added test for query method with string dtype and null values (#…
State: closed

…31516)

- [x] closes #31516
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Testing
description: pandas testing functions or related to the test suite

Comments:
Thanks for the pull request, but it appears to have gone stale. If interested in continuing, please merge in the main branch, address any review comments and/or failing tests, and we can reopen.
CI: Don't run cron for forks
State: closed

- [n/a] closes #xxxx (Replace xxxx with the GitHub issue number)
- [n/a] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [n/a] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [n/a?] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

The codeql and stale-pr workflows run on a cron:

```yml
on:
  schedule:
    # every day at midnight
    - cron: "0 0 * * *"
```

As does the wheels workflow:

```yml
on:
  schedule:
  #        ┌───────────── minute (0 - 59)
  #        │  ┌───────────── hour (0 - 23)
  #        │  │ ┌───────────── day of the month (1 - 31)
  #        │  │ │ ┌───────────── month (1 - 12 or JAN-DEC)
  #        │  │ │ │ ┌───────────── day of the week (0 - 6 or SUN-SAT)
  #        │  │ │ │ │
  - cron: "27 3 */1 * *"
```

These are useful for the upstream https://github.com/pandas-dev/pandas repo but unnecessary for contributors' forks:

<img width="997" alt="image" src="https://user-images.githubusercontent.com/1324225/236686019-f97c10e6-9fcd-4de4-920e-42a837f6f497.png">

They also trigger emails when they fail on the fork:

<img width="712" alt="image" src="https://user-images.githubusercontent.com/1324225/236686197-c2b4e3cd-eb68-4f02-bb31-8c7b3cb2629f.png">

Similar to autoupdate-pre-commit-config, let's only run the schedule for upstream:

https://github.com/pandas-dev/pandas/blob/956b8008e90845093f57c581c97b15297c72090a/.github/workflows/autoupdate-pre-commit-config.yml#L16


Labels:
name: CI
description: Continuous Integration

Comments:
Makes sense. Thanks @hugovk
ENH: Arithmetic operations on intervals #43629
State: closed

- [ ] closes #43629 (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:

Comments:
Thanks for the pull request, but it appears to have gone stale. If interested in continuing, please merge in the main branch, address any review comments and/or failing tests, and we can reopen.
ENH: add validate parameter to Categorical.from_codes
State: closed

- [x] closes #50975
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Enhancement
name: Performance
description: Memory or execution speed performance
name: Categorical
description: Categorical Data Type

Comments:
Thanks @topper-123
REF: move SparseDtype, ArrowDtype to dtypes.dtypes
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Refactor
description: Internal refactoring of code

Comments:
Yes. Without this we raise at import time bc of circularity

On Tue, May 9, 2023 at 10:45 AM Matthew Roeschke ***@***.***>
wrote:

> ***@***.**** commented on this pull request.
> ------------------------------
>
> In pandas/util/__init__.py
> <https://github.com/pandas-dev/pandas/pull/53116#discussion_r1188939656>:
>
> > @@ -1,11 +1,25 @@
> -# pyright: reportUnusedImport = false
> -from pandas.util._decorators import (  # noqa: F401
> -    Appender,
> -    Substitution,
> -    cache_readonly,
> -)
> -
> -from pandas.core.util.hashing import (  # noqa: F401
> -    hash_array,
> -    hash_pandas_object,
> -)
> +def __getattr__(key: str):
>
> Are these changes related to this PR?
>
> —
> Reply to this email directly, view it on GitHub
> <https://github.com/pandas-dev/pandas/pull/53116#pullrequestreview-1419158994>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AB5UM6B6EB6DE66CNDWQZBDXFJ7CVANCNFSM6AAAAAAXXVJH6M>
> .
> You are receiving this because you authored the thread.Message ID:
> ***@***.***>
>

Thanks @jbrockmendel
TYP: remove mypy ignore from pandas/core/construction.py
State: closed

Related to #37715
mypy ignore[assignment] was removed from pandas/core/construction.py

Labels:
name: Typing
description: type annotations, mypy/pyright type checking

Comments:
Looks like mypy raises an error, I will fix it.
I think the issue might be that `astype` isn't meant to be called on `numpy.ma.MaskedArray` https://numpy.org/doc/stable/reference/routines.ma.html

Calling it, the return type is typed to be `ndarray`, even though if we print it we see it's `numpy.ma.MaskedArray`:
```python
data = np.ma.MaskedArray([1,2,3], mask=[False, False, True])
reveal_type(data)  # Revealed type is "numpy.ma.core.MaskedArray[Any, Any]"
reveal_type(data.astype('int64', copy=True))  # Revealed type is "numpy.ndarray[Any, numpy.dtype[Any]]"
```
Might be worth checking if there's an issue in numpy about it, and if not, then reporting to them? Perhaps this could be made more precise on their end?

The method which says "Convert the input to a masked array of the given data-type" is the one you'd originally used: `numpy.ma.asarray`, so your original solution was probably the correct one, sorry for the noise 🙈 The docs say it that `asarray` doesn't copy if the original data is an ndarray anyway
> I think the issue might be that `astype` isn't meant to be called on `numpy.ma.MaskedArray` https://numpy.org/doc/stable/reference/routines.ma.html
> 
> Calling it, the return type is typed to be `ndarray`, even though if we print it we see it's `numpy.ma.MaskedArray`:
> 
> ```python
> data = np.ma.MaskedArray([1,2,3], mask=[False, False, True])
> reveal_type(data)  # Revealed type is "numpy.ma.core.MaskedArray[Any, Any]"
> reveal_type(data.astype('int64', copy=True))  # Revealed type is "numpy.ndarray[Any, numpy.dtype[Any]]"
> ```
> 
> Might be worth checking if there's an issue in numpy about it, and if not, then reporting to them? Perhaps this could be made more precise on their end?

Thanks, I'll check if there's an issue on GitHub about it.
> 
> The method which says "Convert the input to a masked array of the given data-type" is the one you'd originally used: `numpy.ma.asarray`, so your original solution was probably the correct one, sorry for the noise 🙈 The docs say it that `asarray` doesn't copy if the original data is an ndarray anyway

I rolled back the commit to the first one.
DEPR: downcast inferring round floats to ints
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

xref #40988

Labels:
name: Dtype Conversions
description: Unexpected or buggy dtype conversions
name: Deprecate
description: Functionality to remove in pandas

Comments:
I guess we can do this and then follow up by deprecating the keyword entirely?  its a bit roundabout, but gives anyone who objects lots of chances to do so
@phofl gentle ping
Closing in favor of #53656
ENH: EADtype._find_compatible_dtype
State: closed

- [x] closes #24246
- [x] closes #45349
- [x] closes #32346
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

Tests copied from #52833, so this may close #52235.

Needs a lot of cleanup, in particular docs.  The ArrowDtype._find_compatible_dtype implementation is quite kludgy, would appreciate help from someone more knowledgeable.

I think `_find_compatible_dtype` is the "correct" long-term non-kludge solution for setitem-with-expansion.


Labels:
name: ExtensionArray
description: Extending pandas with custom dtypes or arrays.
name: Stale
name: setitem-with-expansion

Comments:
This pull request is stale because it has been open for thirty days with no activity. Please [update](https://pandas.pydata.org/pandas-docs/stable/development/contributing.html#updating-your-pull-request) and respond to this comment if you're still interested in working on this.
Updated with some added tests, changed EADtype._maybe_promote->_find_compatible_dtype.

There are some TODOs mostly for pyarrow and related to nan-vs-NA, but I don't think any of them are blockers.
_find_compatible_dtype can be used to implement a default implementation of _is_valid_scalar_for, which can be used to implement _from_scalars (both discussed on this week's dev call).