BrandonFors/pandas-github-issues huggingface dataset:
BUG: groupby.agg doesn't include grouping columns in result when selected
State: closed

- [x] closes #51186 (Replace xxxx with the GitHub issue number)
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Bug
name: Refactor
description: Internal refactoring of code
name: Groupby
name: Apply
description: Apply, Aggregate, Transform, Map

Comments:
Potential follow-up that this makes possible: _iterate_slices is only used in _python_agg_general, where it iterates over columns calling agg_series.  agg_series iterates over groups.  Instead we could iterate differently over groups, which would look a lot like _aggregate_frame.
Assuming no objection, going to merge when greenish. Currently targeted at 2.1.
CLN: Simplify agg_list_like
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

This logic was added in #41017 to deal with partial failure that can no longer occur.

Labels:
name: Clean
name: Apply
description: Apply, Aggregate, Transform, Map
name: Reduction Operations
description: sum, mean, min, max, etc.

Comments:
Thanks @rhshadrach (failures unrelated)
DEPR: DataFrame.groupby(axis=1)
State: closed

- [x] closes #51203 (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Groupby
name: Deprecate
description: Functionality to remove in pandas

Comments:
Any more thoughts here?
@jbrockmendel - thanks for the ping; just a conflict with the whatsnew.
Thanks @jbrockmendel 
nb  the 2.1.0 docs at https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.groupby.html do not mention this depreciation
Thanks @cfaaron - I've opened #54873.
CI: Pin matplotlib to < 3.7.0
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Visualization
description: plotting
name: CI
description: Continuous Integration

Comments:
whats the condition under which we can revert the pin?
#51394 links a failing build.

Not really familiar with how matplotlib works, but couldn't find a whatsnew that references this
> whats the condition under which we can revert the pin?

[taking plotting out of pandas](https://github.com/pandas-dev/pandas/issues/50059) 😄 
Yep, merging both should get ci green.
Makes reverting easier when doing it separately
Owee, I'm MrMeeseeks, Look at me.

There seem to be a conflict, please backport manually. Here are approximate instructions:

1. Checkout backport branch and update it.

```
git checkout 1.5.x
git pull
```

2. Cherry pick the first parent branch of the this PR on top of the older branch:
```
git cherry-pick -x -m1 dd30415b6e569b66051d41172f86cce79244f519
```

3. You will likely have some merge/cherry-pick conflict here, fix them and commit:

```
git commit -am 'Backport PR #51393: CI: Pin matplotlib to < 3.7.0'
```

4. Push to a named branch:

```
git push YOURFORK 1.5.x:auto-backport-of-pr-51393-on-1.5.x
```

5. Create a PR against branch 1.5.x, I would have named this PR:

> "Backport PR #51393 on branch 1.5.x (CI: Pin matplotlib to < 3.7.0)"

And apply the correct labels and milestones.

Congratulations — you did some good work! Hopefully your backport PR will be tested by the continuous integration and merged soon!

Remember to remove the `Still Needs Manual Backport` label once the PR gets merged.

If these instructions are inaccurate, feel free to [suggest an improvement](https://github.com/MeeseeksBox/MeeseeksDev).
CI: Pin openpyxl to < 3.1.1
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:

Comments:
Owee, I'm MrMeeseeks, Look at me.

There seem to be a conflict, please backport manually. Here are approximate instructions:

1. Checkout backport branch and update it.

```
git checkout 1.5.x
git pull
```

2. Cherry pick the first parent branch of the this PR on top of the older branch:
```
git cherry-pick -x -m1 0e12bfcd4b4d4e6f5ebe6cc6f99c0a5836df4478
```

3. You will likely have some merge/cherry-pick conflict here, fix them and commit:

```
git commit -am 'Backport PR #51391: CI: Pin openpyxl to < 3.1.1'
```

4. Push to a named branch:

```
git push YOURFORK 1.5.x:auto-backport-of-pr-51391-on-1.5.x
```

5. Create a PR against branch 1.5.x, I would have named this PR:

> "Backport PR #51391 on branch 1.5.x (CI: Pin openpyxl to < 3.1.1)"

And apply the correct labels and milestones.

Congratulations — you did some good work! Hopefully your backport PR will be tested by the continuous integration and merged soon!

Remember to remove the `Still Needs Manual Backport` label once the PR gets merged.

If these instructions are inaccurate, feel free to [suggest an improvement](https://github.com/MeeseeksBox/MeeseeksDev).
DOC: Fix EX01 issues in docstrings
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

Towards #37875

Labels:
name: Docs

Comments:
Thanks @DeaMariaLeon (test failures unrelated)
Thank you.
TEST: cut() with nullable Int64 dtype
State: closed

- [x] closes #30787
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: cut
description: cut, qcut
name: NA - MaskedArrays
description: Related to pd.NA and nullable extension arrays

Comments:
thx @kkangs0226
CI fixup pylint
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

I missed this one in https://github.com/pandas-dev/pandas/pull/51367

Labels:
name: CI
description: Continuous Integration
name: Code Style
description: Code style, linting, code_checks

Comments:
merging to get CI green again, this is _super_ minor anyway (just adds a couple of comments)
WARN unnecessary "can't infer datetime format" for out-of-bounds datetimes
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Datetime
description: Datetime data dtype
name: Warnings
description: Warnings that appear or should be added to pandas

Comments:
closing as per https://github.com/pandas-dev/pandas/issues/51379#issuecomment-1431666585, it's not really worth doing this
BUG: GroupBy.quantile with datetimelike and NaT
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Groupby
name: quantile
description: quantile method

Comments:
Thanks @jbrockmendel
ENH: Add Index.filter() method
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
  - New feature - no issue
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
  - `pandas\pandas\tests\indexes\test_base.py:TestIndex.test_filter_string()`
  - `pandas\pandas\tests\indexes\test_base.py:TestIndex.test_filter_int()`
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/v2.0.0.rst` file if fixing a bug or adding a new feature.

This is similar to `DataFrame.filter()`, except it returns an `Index` object, and avoids any under-the-hood things that might be happening with `DataFrame.filter()` in terms of views/copies of the `DataFrame`.   Some examples where this would be useful (and helpful when doing type checking).

- Instead of `df.columns = [x for x in otherdf.columns if "at" in x]`, you can do `df.columns = otherdf.columns.filter(like="at")`
- Instead of `df.drop(columns=[x for x in df.columns if x.startswith("b")]`, you can do `df.drop(columns=df.columns.filter(regex=r"b.*")`
- Instead of `df.set_index([x for x in df.columns if x.endswith("z")])`, you can do `df.set_index(df.columns.filter(r".*z$"))`



Labels:
name: Index
description: Related to the Index class or subclasses

Comments:
-1 here this is a confusing name (well it's an ok name except filter for dataframe does this)
> I might be missing something, but can't you do this mostly with the string accesors as well?

Yes, but it's a bit awkward, because the string accessors return boolean arrays.

Comparison using example `idx = pd.Index(["cat", "dog", "bat", "bird"], dtype=object)`

| Accessors |  With this PR |
| ---- | ---- |
| `idx[idx.isin(["cat", "dog"])]` | `idx.filter(["cat", "dog"])` |
| `idx[idx.str.contains("at")]`  | `idx.filter(like="at")` |
| `idx[idx.str.match(r"b.*")]`  | `idx.filter(match=r"b.*")` |

I also think this PR will be more performant, because you are making just one pass through the index.

> -1 here this is a confusing name (well it's an ok name except filter for dataframe does this)

Yes, but filter for `DataFrame` returns the entire `DataFrame`, and the goal here is to return a filtered `Index` .  Open to using a different name.


Looks like this PR has gone stale and might need some discussion on an issue first before moving forward. Going to close for now, but we can reopen when ready to move forward
REF: use datetime C API instead of getattrs
State: closed

@WillAyd i think there are a couple of paths that must not be reached, since it looks like they pass np.datetime64 objects to these pydatetime-requiring functions.  in python/cython id check that with an `assert False`.  have a favorite way of doing that in c code?

Labels:

Comments:
it works locally, but im seeing build failures from the type checking
Closing as not worth the trouble.
ENH: Improve performance for arrow dtypes in monotonic join
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

```
idx = Index(list(range(1, 1_000_000)), dtype="int64[pyarrow]")
idx2 = Index(list(range(100_000, 1_100_000)), dtype="int64[pyarrow]")
idx.union(idx2)

# main
# %timeit idx.union(idx2)
# 327 ms ± 72.6 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

# pr
# %timeit idx.union(idx2)
# 2.79 ms ± 27.5 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)
```

Labels:
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode
name: Arrow
description: pyarrow functionality

Comments:
no objection here, but eventually we ought to find a way to do this dispatch without special-casing inside the index code (i.e. implement something at the EA level)

hows is perf affected on multi-chunk pyarrow objs?
Arrays have 2 million entries, initial performance 380ms, on this pr

```
%timeit idx.union(idx2)
11.5 ms ± 477 µs per loop (mean ± std. dev. of 7 runs, 1 loop each)
```
2 chunks
I totally agree with your point regarding EA interface, this is a short term solution for 2.0
@jbrockmendel ok to merge?
CLN: idiomatic indexing checks
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves
name: Clean

Comments:
Thanks @jbrockmendel
BLD: Fix uploading of aarch64 wheels
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Build
description: Library building on various platforms

Comments:
Nice, were you able to confirm a similar workflow works locally?
~~It should work(I tested all the way up to the actual uploading). 
Unfortunately, this is the sort of thing where you have to merge and see, so this might take a couple attempts.~~

EDIT: NVM, found a small bug
This is ready now.
Thanks @lithomas1
STYLE: Fix errors in doctests
State: closed

- [x] closes #51341
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Docs
name: Code Style
description: Code style, linting, code_checks

Comments:
@MarcoGorelli I looked through the ```code_checks.sh``` file and it seems like none of the functions edited in my PR are in the script. I ran ```./ci/code_checks.sh docstrings``` and here is the output: 
![image](https://user-images.githubusercontent.com/66008784/218755170-62a646e2-7a84-47e2-bae9-a6e284b36f2b.png)

> @MarcoGorelli I looked through the `code_checks.sh` file and it seems like none of the functions edited in my PR are in the script.

thanks for checking - OK yeah, looks like `EX03` isn't in the list of codes which are currently being checked, so that's why they wouldn't be showing up. That's another one we'll need to gradually enable, but your PR helps reduce the amount of work that'll need doing, thanks 🙌 

for the `{this}` substitution, I'd suggest just restoring it - if `flake8` is erroring, it's because it just checks the code statically rather than trying to run it, so it doesn't know what the symbol means. We can ignore it
@datapythonista @MarcoGorelli Thanks for your reviews, I have created an issue for the template string problem and linked my comment there. https://github.com/pandas-dev/pandas/issues/51377
thanks @kkangs0226
BUG: iterparse on `read_xml` ignores repeated elements
State: closed

- [X] closes #51183
- [X] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [X] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [X] Added an entry in the latest `doc/source/whatsnew/v2.0.0.rst` file if fixing a bug or adding a new feature.


Labels:
name: IO XML
description: read_xml, to_xml

Comments:
Thanks @ParfaitG
TST: xfail flaky test
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Unreliable Test
description: Unit tests that occasionally fail

Comments:
Thanks @jbrockmendel
BUG: Period('') raising
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Period
description: Period data type

Comments:
Thanks @jbrockmendel
STYLE upgrade black formatter
State: closed

gonna have to do this eventually...

I'm trying to tell if it changed anything other than just deleting newlines, any `git` masters know how to do that?

Labels:
name: Code Style
description: Code style, linting, code_checks

Comments:
thanks @rhshadrach for going through the changes - I think they look fine

@lithomas1 is it OK to go ahead with this, or does it need to wait til after the release?
+1 on landing, let's do this before the branch cut.
cool, I've updated - let's merge on green-ish then?
Sorry I just merged a few PRs since this got updated.

Since it's a style change that could impact all PRs, lets rebase once more (I won't merge any other PRs) and get this in
I just tried fetching and merging upstream/main locally, and then did `pre-commit run --all-files`, and everything passed

So I think we should be safe to just merge directly
Ah nice thanks for checking. 
Thanks @MarcoGorelli 
thank you

> So I think we should be safe to just merge directly

If this doesn't work I accept full blame