BrandonFors/pandas-github-issues huggingface dataset:
PERF: accessing sliced indexes with populated indexing engines
State: closed

Improves performance of indexes that are sliced from indexes with already-built indexing engines by copying the relevant data from the existing indexing engine, thereby avoiding recomputation.

Performance example:

```python
>>> import pandas as pd
>>>
>>> idx = pd.Index(np.arange(1_000_000))
>>> idx.is_unique, dx.is_monotonic_increasing  # building the engine
(True, True)
>>> %timeit idx[:].is_unique
13.9 ms ± 78.8 µs per loop  # main
2.76 µs ± 9.74 ns per loop  # this PR
>>> %timeit idx[:].is_monotonic_increasing
4.26 ms ± 1.21 µs per loop  # main
2.7 µs ± 3.9 ns per loop  # this PR
>>> %timeit  idx[:].get_loc(999_999)
4.26 ms ± 1.49 µs per loop  # main
3.77 µs ± 41.7 ns per loop  # this PR
```

Not sure how to test this, as the relevant attributes are in cython code, but I don't think we do tests for indexing engines currently?

Labels:
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves
name: Performance
description: Memory or execution speed performance

Comments:
nice! this has been on my todo list for ages but was always intimidating
Thanks @topper-123
Is ExtensionDtype.type a property or a ClassVar?
State: closed

`ExtensionDtype.type` is a `property` but a few (not all) sub-classes define it as a class variable. I assume it should be a `property` but I'm not sure.

There are a few more cases where `type` is used a class variable in pandas/core/arrays/floating.py and pandas/core/arrays/integer.py. I will address them but I first wanted to ask whether `type` is supposed to be a `property` or a class variable. @jbrockmendel 

xref https://github.com/pandas-dev/pandas-stubs/pull/554#discussion_r1118955687

Labels:
name: Typing
description: type annotations, mypy/pyright type checking
name: Stale

Comments:
Either way.  I like the class-level when it is fixed bc it is less verbose.  It's also faster to access (37ns vs 62ns not a huge deal but still)

The only way I can imagine it making a difference at runtime is if a user tries to modify it, which is a corner case im not inclined to defined against.

If you feel strongly about aligning these for type-checking reasons, fine by me.
> Either way. I like the class-level when it is fixed bc it is less verbose. It's also faster to access (37ns vs 62ns not a huge deal but still)

Happy to go with a ClassVar everywhere
This pull request is stale because it has been open for thirty days with no activity. Please [update](https://pandas.pydata.org/pandas-docs/stable/development/contributing.html#updating-your-pull-request) and respond to this comment if you're still interested in working on this.
PERF: ArrowExtensionArray._from_sequence_of_strings(dtype=pa.bool_())
State: closed

- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/v2.1.0.rst` file if fixing a bug or adding a new feature.

`ArrowExtensionArray._from_sequence_of_strings` currently depends on `BooleanArray._from_sequence_of_strings` to parse strings for `boolean[pyarrow]`. This removes the dependency on `BooleanArray` and uses pyarrow casting to accomplish the same thing. I've adding some logic around pyarrow's casting to match existing behavior of `BooleanArray` (assuming we want to maintain existing behavior?)

```
import pyarrow as pa
import numpy as np
from pandas.core.arrays.arrow.array import ArrowExtensionArray

N = 1_000_000
strings_np = np.random.choice(["true", "false", None], N, replace=True)
strings_pa = pa.array(strings_np, type=pa.string())

%timeit ArrowExtensionArray._from_sequence_of_strings(strings_np, dtype=pa.bool_())

# 200 ms ± 4.72 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)    -> main
# 54.5 ms ± 842 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)  -> PR

%timeit ArrowExtensionArray._from_sequence_of_strings(strings_pa, dtype=pa.bool_())

# 298 ms ± 3.45 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)    -> main
# 30.8 ms ± 1.1 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)  -> PR
```

Labels:
name: Performance
description: Memory or execution speed performance
name: Arrow
description: pyarrow functionality

Comments:
Thanks @lukemanley
BUG: read_parquet does not respect index for arrow dtype backend
State: closed

- [x] closes #51717 (Replace xxxx with the GitHub issue number)
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: IO Parquet
description: parquet, feather
name: Arrow
description: pyarrow functionality

Comments:
Hey, thanks for tackling this so quickly, I appreciate it. One additional request is that RangeIndex can have a name, which would be inside `params`.

If it helps,  there's an implementation here already: https://github.com/apache/arrow/blob/45918a90a6ca1cf3fd67c256a7d6a240249e555a/python/pyarrow/pandas_compat.py#L943 and Wes introduced the dict-based index description [here](https://github.com/pandas-dev/pandas/issues/25672). It feels like an under-defined format so could be hard to write tests for this.
thx, adjusted
Was this supposed to be addressed in https://github.com/pandas-dev/pandas/pull/51766?
Not directly, but I added type_mapper support for the index in pyarrow a couple of days ago. We will get this out of the box with pyarrow 12.0, so I wouldn't do anything on our side.
DOC: updated docstrings for EX02 validation and updated code_checks
State: closed

- [x] closes #51236
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

Closes an issue which was worked on by several people.

`./ci/code_checks.sh docstrings` is now passing completely with the `Partially validate docstrings (EX02)` section and its corresponding `--ignore_functions` list deleted and `EX02` moved to the main `Validate docstrings` command.

`pre-commit` passes on the updated files.

Labels:
name: Docs

Comments:
@MarcoGorelli I feel pretty confident about the work here except for the `What's new` description. I put it in the `2.0.0` section, is that correct? I wanted to link to the `main` function in `validate_docstrings.py`, but I couldn't find it in the documentation. I added it to the `Other enhancements` section and wasn't sure if that was correct either. Also, obviously the other contributors and I should be listed in the `What's new` section, but it appears that the `Contributors` section will be filled out later.

Sorry for all of the questions, I'm asking them because the finishing of this issue represents the work of several other people as well and I wanted to make sure that things are correct.
CI: Add bot to track deprecation prs
State: closed

We tend to forget to register deprecation prs in our deprecation log. This bot aims to solve this. Every time a PR with the label Deprecate is merged, it adds a new entry to the deprecation log.

In a future version I'd like to add the option to also check enforced deprecations automatically.

One TODO is how to handle PRs where we enforce deprecations:
- We could create a new label that does not affect the bot
- Add a switch to the bot when we start enforcing deprecations

We could get race conditions in theory, but it is not very likely, so would address if it becomes a problem.

Tried this in my repo and worked reliably

Pinging everyone who is regularly merging prs, hope I did not forget anyone

Labels:
name: CI
description: Continuous Integration

Comments:
i don't know enough about GH workflows to comment on the implementation, but the idea sounds really neat
+1 for trying this out.
Nice, on board with the idea

I'm no security expert, but I guess there's no way for this to be exploited to get the secrets out of it?
I don’t think so, I am basically using the same technique as our assign bot. 

Also, it’s not possible to trigger the bot again after the pr is merged
I'll merge this, we can always revert if we get any problems. I have a deprecation pr ready that can test the bot
DEV: reduce merge conflicts by sorting whatsnew notes by issue number?
State: closed

- [ ] closes #51712 (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


The script which does this is `scripts/sort_whatsnew_note.py`. There's a defensive check inside to validate that it has only reordered the lines of a file, and hasn't changed anything else

---

Rather than tagging all of core/triage, I've just request a review from whoever participated in #50275 - but if this proves controversial we can tag the rest

Labels:

Comments:
Implementation seems fine, im going to post general thoughts in #51712 
Agree implementation looks fine, would be good to see if could be automated as part of the test suite.  Could you add that so one could check the before and after whatsnew file for differences?
yup, done - have run it over all files, you can check what it changes
This feels like a real win to me for reducing whatsnew conflicts.

Thanks!

> Also, going forward, if someone does a PR and updates the whatsnew of the latest version, is there something we can do to automate the sorting as part of either pre-commit or CI?

Yup, it's in pre-commit
One more - does this run in CI and will CI fail if this returns non-zero?  
That's right! You can see it here: https://github.com/pandas-dev/pandas/actions/runs/4312190011/jobs/7522443368

```
 sort whatsnew entries by issue number...................................................................Passed
- hook id: sort-whatsnew-items
- duration: 0.05s
```

And yes, it'll return non-zero exit code if it would've modified any file (like all pre-commit hooks)
I've amended this to exclude 2.0.0, so it'll only be applied going forwards

If in 3.0.0 we want to keep some parts of a section grouped, we can just make subsections, e.g.
```rst
Removal of prior version deprecations/changes
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Non-keyword arguments
^^^^^^^^^^^^^^^^^^^^^
- Disallow passing non-keyword arguments to :func:`read_excel` except ``io`` and ``sheet_name`` (:issue:`34418`)
- Disallow passing non-keyword arguments to :meth:`DataFrame.drop` and :meth:`Series.drop` except ``labels`` (:issue:`41486`)

Behavior changes
^^^^^^^^^^^^^^^^
- Changed behavior of :meth:`Series.quantile` and :meth:`DataFrame.quantile` with :class:`SparseDtype` to retain sparse dtype (:issue:`49583`)
```
Thanks @MarcoGorelli
TST: add test for apply/reconstruct_func
State: closed

- [ ] closes #28472 
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

Add a test for reconstruct_func to ensure this method isn't moved. It is used by other libraries such as dask.

Labels:
name: Groupby
name: Apply
description: Apply, Aggregate, Transform, Map
name: Needs Tests
description: Unit test(s) needed to prevent regressions

Comments:
Thanks @renatocmaciel!
DEPR: pinning name in SeriesGroupBy.agg
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

xref #41090 doesn't fully close it as there are other places that use the deprecated pattern.


Need a good "use X instead" for the warning messsage.


Labels:
name: Groupby
name: Deprecate
description: Functionality to remove in pandas

Comments:
What about:

> If your operation requires utilizing the groupby keys, iterate over the groupby object instead.
Good idea, updated
Thanks @jbrockmendel
REF: Remove sanitize_column from _iset_item
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

this should work ootb. Don't think we need a copy here

Labels:

Comments:
BTW one of your other recent PRs that avoided passing DataFrame to sanitize_column might make it so that the allow_2d=True passed there could be unnecessary?
I think we could still get there with 2D arrays, not sure though. Would have to check
ERR: Add explicit error message for isetitem for DataFrame
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

cc @jbrockmendel 

Labels:
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves
name: Error Reporting
description: Incorrect or improved errors from pandas

Comments:
Doesn't hurt probably, added one
BUG: changing period array formats to specified date formats in `to_csv`
State: closed

- This is intended to fix #51621
- This issue is that PeriodArrays can not change to the specified `date_format` in `to_csv`.
- The cause is that PeriodArray is established as `NDArrayBackedExtensionBlock` instead of `DatetimeLikeBlock` so that `df._mgr.to_native_types` does not change 'PeriodArray' to desired string format. This fix is added in after `df._mgr.to_native_types` to change `Period` to specified `date_format`.
- One unit test is added.


Labels:
name: IO CSV
description: read_csv, to_csv
name: Period
description: Period data type
name: Stale

Comments:
This pull request is stale because it has been open for thirty days with no activity. Please [update](https://pandas.pydata.org/pandas-docs/stable/development/contributing.html#updating-your-pull-request) and respond to this comment if you're still interested in working on this.
Thanks for the pull request, but it appears to have gone stale. If interested in continuing, please merge in the main branch, address any review comments and/or failing tests, and we can reopen.
CoW: Delay copy when setting Series into DataFrame
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Copy / view semantics

Comments:
This tackles one case for now, will add other cases step by step based on the strategy here.

Would not backport this, since it's unlikely that we'll finish everything till 2.0 comes out
This pull request is stale because it has been open for thirty days with no activity. Please [update](https://pandas.pydata.org/pandas-docs/stable/development/contributing.html#updating-your-pull-request) and respond to this comment if you're still interested in working on this.
Best to get this one in first, before looking at https://github.com/pandas-dev/pandas/pull/52062 / https://github.com/pandas-dev/pandas/pull/52051 / https://github.com/pandas-dev/pandas/pull/52061, right?

Yep
Rebased all the other prs
DEP: Deprecate passing a DataFrame to from_records
State: closed

- [x] closes #51353 (Replace xxxx with the GitHub issue number)
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


cc @jbrockmendel  as discussed, restructuring this allows us to simplify to_arrays immediately

Labels:
name: Deprecate
description: Functionality to remove in pandas
name: DataFrame
description: DataFrame data structure

Comments:
small comment, otherwise LGTM
Adds Hamilton to match ecosystem.md
State: closed

Just noticed that ecosystem.rst wasn't in sync with ecosystem.md. So adding part about Hamilton to it.

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Docs

Comments:
Thanks @skrawcz
STYLE Reduce merge conflicts with `force_grid_wrap=1`
State: closed

- [ ] closes #50274 (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

Exact commands I ran:
```
git fetch upstream
git reset --hard upstream/main
sed -i 's/force_grid_wrap = 2/force_grid_wrap = 1/g' pyproject.toml 
pre-commit run isort -a
git commit -a -m 'force_grid_wrap=1' --no-verify
```
I then added an extra commit to update the `future-imports` check to the new style. No other adjustments were necessary

Labels:
name: Code Style
description: Code style, linting, code_checks
name: Stale

Comments:
> Thanks @MarcoGorelli
> 
> This looks like the lesser of the two evils.

I agree.  It's a reasonable compromise between readability and merge-ability.
This pull request is stale because it has been open for thirty days with no activity. Please [update](https://pandas.pydata.org/pandas-docs/stable/development/contributing.html#updating-your-pull-request) and respond to this comment if you're still interested in working on this.
closing for now, doesn't look like there was that much support, and maybe merge conflicts don't annoy others as much as they annoy me 😄
BUG: DataFrame.stack losing EA dtypes with multi-columns and mixed dtypes
State: closed

- [x] closes #45740
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/v2.1.0.rst` file if fixing a bug or adding a new feature.


Labels:
name: Bug
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode
name: ExtensionArray
description: Extending pandas with custom dtypes or arrays.

Comments:
Thanks @lukemanley