BrandonFors/pandas-github-issues huggingface dataset:
TST/CLN: Consolidate creation of groupby method args
State: closed

- [ ] closes #xxxx (Replace xxxx with the Github issue number)
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

Wasn't sure if it was better to put the helper in `__init__`  or create `common`.

Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Groupby
name: Clean

Comments:
Thanks @rhshadrach lgtm. and no documentation is too much (imo) so that we are more casual contributor friendly. merge when green.
> As a rule I do not see changing tests as necessarily a benefit. Parameterizing most definitely so that we can test more for consistency.

> no documentation is too much (imo) so that we are more casual contributor friendly

For me, sometimes one of the difficult parts of doing a PR is just figuring out where a test should go. In the same vein as documentation, I think making tests easier to write would be a way to make pandas more contributor friendly. So I do think cleaning up tests where possible would be a benefit.

In fact, I've spent quite some effort trying to figure out ways to better organize and write groupby tests because of this, but rarely do I find a change that's worth doing. The trouble is there are so many ways to parametrize, and how best to do it really depends on the test:

 - the type of object being grouped - i.e. SeriesGroupBy and DataFrameGroupBy
 - the methods being used (e.g. sum, mean, fillna)
 - the arguments (e.g. `.groupby(axis=0)` and `.groupby(axis=1)`), especially `by`
 - the dtypes of the input (including single vs multiindex, and especially categorical)

Various tests parametrize over one or multiple of these, and many more don't parametrize at all but probably should. I've never been able to find a good way to organize tests for parametrizations, but would strongly feel we should refactor if there was one.
Thanks @rhshadrach (failure unrelated).

My 2c: I think test cleanup can be worth it if it's:

1. Test reorganization/renaming such that it's more intuitive to know where to place things
2. Simplifying tests & using more pytest idioms such that debugging a failing test is more direct. 

I think creating a "groupby testing parameterization matrix" of grouping objects x object dtypes x groupby methods x method arguments would be really nice to have running (although maybe redundant with some existing tests), but it would probably be best for it to be a new testing file instead of augmenting what already exists.
Thanks @simonjayhawkins and @mroeschke for sharing your thoughts.
BUG: pivot_table raising TypeError with ea dtype and dropna True
State: closed

- [x] closes #47477 (Replace xxxx with the Github issue number)
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

The aggregation function already provides the correct dtype. In case of numpy dtype we are casting to many cases, see #47971

Labels:
name: Bug
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode
name: NA - MaskedArrays
description: Related to pd.NA and nullable extension arrays

Comments:
Thanks @phofl
TYP: move imports inside TYPE_CHECKING
State: closed

These are found by flake8-type-checking. There are more cases (all of the _typing imports).

If flake8-type-checking should be run on the CI, it would be good to move all the _typing imports inside TYPE_CHECKING (not sure whether we want to do that).

Labels:
name: Typing
description: type annotations, mypy/pyright type checking

Comments:
> There are more cases (all of the _typing imports).

This shouldn't be needed as we guard there so that there are no issues with importing from the top level. (and IIRC I rejected a PR that did that once before)

What problem are we trying to solve?

What you have done here though with the other imports lgtm.
> What problem are we trying to solve?

The goal of flake8-type-checking is to reduce import times and cycling dependencies by moving as much into TYPE_CHECKING. Having it run on the CI would ensure that imports that could be in the TYPE_CHECKING block are there. Probably not worth running it on the CI, would need too many ignores for each line of a _typing import.

I purposefully avoided the _typing imports as I also remember that there were some discussions about not doing that some time ago :)
> The goal of flake8-type-checking is to reduce import times

now that's a fair point that I don't recall discussing before, so potentially open to the idea.
Thanks @twoertwein
CI: fix mypy typing errors (pin numpy) on branch 1.4.x
State: closed

- [ ] closes #47569 (Replace xxxx with the Github issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


cc @phofl @twoertwein

I have checked pyright with `pre-commit run -a --hook-stage manual pyright` whereas on ci we are installing pyright but hopefully the versions are the same.

Labels:
name: CI
description: Continuous Integration
name: Typing
description: type annotations, mypy/pyright type checking

Comments:
Thx @simonjayhawkins
fix : removed extra blank line between sections
State: closed

- [ ] closes #xxxx (Replace xxxx with the Github issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:

Comments:
thanks for your PR, but this is fine as-is - if interested in contributing, please check [good first issue](https://github.com/pandas-dev/pandas/issues?q=is%3Aopen+is%3Aissue+label%3A%22good+first+issue%22)s
ENH: Add `dtype` param for `pandas.Series.str.get_dummies`
State: closed

- [ ] closes #47872 
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Enhancement
name: Strings
description: String extension data type and string data
name: Stale

Comments:
the api should match pd.get_dummies here
This pull request is stale because it has been open for thirty days with no activity. Please [update](https://pandas.pydata.org/pandas-docs/stable/development/contributing.html#updating-your-pull-request) and respond to this comment if you're still interested in working on this.
Thanks for the pull request, but it appears to have gone stale. If interested in continuing, please merge in the main branch, address any review comments and/or failing tests, and we can reopen.
DEBUG: Debug failing macpython wheels
State: closed

might be because of a change in hypothesis


Labels:
name: Testing
description: pandas testing functions or related to the test suite

Comments:
@phofl i've closed the nightly build issue. We may need an issue to unpin hypothesis if this PR gets closed.
Yeah we  should probably open a new issue. Here or in the MacPython repo?
it does need to get unpinned in MacPython eventually, but there is not much visibility there, so an issue in pandas would be better. Do we know why we didn't see the same failures on pandas ci?
If this was an issue in `test_round_sanity` as suggested in https://github.com/pandas-dev/pandas/issues/47933#issuecomment-1205909274, we did xfail(strict=False) this test in https://github.com/pandas-dev/pandas/pull/47803
Still needed?
Opened an issue, can close here
TST: Add test for loc not updating cache correctly
State: closed

- [x] closes #47867 (Replace xxxx with the Github issue number)
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


All the setitem and getitem steps are necessary to produce the bug. Reduced it as far as possible

Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Regression
description: Functionality that used to work in a prior pandas version

Comments:
> All the setitem and getitem steps are necessary to produce the bug. Reduced it as far as possible

you did explicitly check that this exact test fails on 1.4.3?
if so merge when ready. lgtm. Thanks @phofl 
> All the setitem and getitem steps are necessary to produce the bug. Reduced it as far as possible

maybe worth adding a code comment to not alter the test as all steps are necessary to avoid clean-ups or refactors changing it.
Yes, used 1.4.3 to narrow it down to this test
> Yes, used 1.4.3 to narrow it down to this test

sure. the "Reduced it as far as possible" created some doubt. so was just checking.
Good point, added a comment
Owee, I'm MrMeeseeks, Look at me.

There seem to be a conflict, please backport manually. Here are approximate instructions:

1. Checkout backport branch and update it.

```
git checkout 1.4.x
git pull
```

2. Cherry pick the first parent branch of the this PR on top of the older branch:
```
git cherry-pick -x -m1 4485f15597a1be751fc0c4aeccd11012f22782e6
```

3. You will likely have some merge/cherry-pick conflict here, fix them and commit:

```
git commit -am 'Backport PR #47949: TST: Add test for loc not updating cache correctly'
```

4. Push to a named branch:

```
git push YOURFORK 1.4.x:auto-backport-of-pr-47949-on-1.4.x
```

5. Create a PR against branch 1.4.x, I would have named this PR:

> "Backport PR #47949 on branch 1.4.x (TST: Add test for loc not updating cache correctly)"

And apply the correct labels and milestones.

Congratulations — you did some good work! Hopefully your backport PR will be tested by the continuous integration and merged soon!

Remember to remove the `Still Needs Manual Backport` label once the PR gets merged.

If these instructions are inaccurate, feel free to [suggest an improvement](https://github.com/MeeseeksBox/MeeseeksDev).
                
https://github.com/pandas-dev/pandas/pull/47954
DOC: Plotting / Visualization without Jupyter
State: closed

Added a note at the top of the page clarifying that the examples assume you're working jupyter.  @MarcoGorelli asked to be tagged to this PR. Hopefully I did everything correctly as this is my first PR.

- [x] closes #47861 (Replace xxxx with the Github issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Docs

Comments:
Looks like there's an (unrelated) doc build failure, will revisit this later once that's fixed
Hi @MarcoGorelli @tpackard1 ,

should we also mention the `plt.savefig()` function, which allows one to save the pandas object as an image file? This is useful if executed either in a script or in the Python interpreter command line. 

We can probably copy and paste from a push [I made on my fork](https://github.com/nwoodweb/pandas-woodn/commit/98e795469add2868e392220bc057c87152534700).
> Hi @MarcoGorelli @tpackard1 ,
> 
> should we also mention the `plt.savefig()` function, which allows one to save the pandas object as an image file? This is useful if executed either in a script or in the Python interpreter command line.
> 
> We can probably copy and paste from a push [I made on my fork](https://github.com/nwoodweb/pandas-woodn/commit/98e795469add2868e392220bc057c87152534700).

I think that looks good, thanks - at this point I think the simplest thing would be for you to [suggest changes](https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/reviewing-changes-in-pull-requests/commenting-on-a-pull-request#adding-line-comments-to-a-pull-request) on this PR, so then you'll end up as co-committer

In the future, please comment `take` on issues you're working on so the issue is assigned to you - in this case, I'd forgotten that you'd already commented that you were working on this, apologies
BUG: Propagating precision to interval dtype #43913
State: closed

- [x] closes #43913
- [x] [test_frame_equal_tolerance() test added and passed]
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).


Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Interval
description: Interval data type
name: Stale

Comments:
Hey @phofl,
I made changes to the PR. Could you help me review them?
@mroeschke Could you check out the changes I made to the PR?
This pull request is stale because it has been open for thirty days with no activity. Please [update](https://pandas.pydata.org/pandas-docs/stable/development/contributing.html#updating-your-pull-request) and respond to this comment if you're still interested in working on this.
> This pull request is stale because it has been open for thirty days with no activity. Please [update](https://pandas.pydata.org/pandas-docs/stable/development/contributing.html#updating-your-pull-request) and respond to this comment if you're still interested in working on this.

Still working on this 😬
Thanks for the pull request, but it appears to have gone stale. If interested in continuing, please merge in the main branch, address any review comments and/or failing tests, and we can reopen.
BUG: avoid relying on external packaging package in interchange protocol
State: closed

I noticed this by accident, and we have a vendored version of `packaging` that we otherwise use internally. 

(this might be a good case to add a code check for? cc @MarcoGorelli)

Labels:
name: Bug
name: Dependencies
description: Required and optional dependencies

Comments:
Thanks @jorisvandenbossche 

> (this might be a good case to add a code check for? cc @MarcoGorelli)

can always raise an issue for this.

> we have a vendored version of packaging that we otherwise use internally.

xref #41207, do we have an open issue to potentially remove this in the future?
@jorisvandenbossche which are the packages which pandas vendors? Just `version` and `clipboard`?
I am not directly aware of others (except for some C code). And probably `packaging.version` is the main relevant one, as `clipboard` / `pyperclip` might not be that common to use in normal code outside of the clipboard IO code
WEB: Accept PDEP-1
State: closed

I forgot to accept PDEP-1 before merging #47444

Labels:
name: Web
description: pandas website
name: PDEP
description: pandas enhancement proposal

Comments:
Thanks @datapythonista 
@datapythonista on https://pandas.pydata.org/about/roadmap.html there is a link to this PDEP. 

https://pandas.pydata.org/pdeps/accepted/0001-puropose-and-guidelines.html

That gives me a 404.
Ah, I see now. The link was never updated after the change to a flat structure. Will put up a PR to fix.
Fix out-of-bounds heap access in internals.pyx.
State: closed

If blknos is empty, we unconditionally access blknos[start] where start
is 0. This is an out-of-bounds heap access that can be caught by
AddressSanitizer, but it's easy to avoid since
we are going to ignore the result anyway.

No new tests, because this does not change any observable behaviors.

Labels:
name: Internals
description: Related to non-user accessible pandas implementation

Comments:
> If you caught this with ASAN, do you mind explaining how you built + ran with ASAN to find this? (For context, it would be really useful for there to be a snippet in the developer guide about building with ASAN - there some issues especially related to memory leaks where ASAN would be helpful (especially when developing OS's that don't easily support valgrind).

I caught this in a Google-internal build, and I'm actually not sure how to reproduce it outside our environment.

In essence I found this using a statically-linked build where a self-built Python interpreter, NumPy, pandas, etc were all instrumented with ASAN. I don't think you could even find this particular bug without instrumenting *both* NumPy *and* pandas because the dereference here is of a NumPy array. It may or may not be necessary to be using a statically-linked build, which is a nonstandard configuration of Python and its modules we use internally.

While it's probably possible to reproduce the asan report outside our environment with some persistence it doesn't seem easy to do. I'm sorry that's a dissatisfying answer!
> I caught this in a Google-internal build, and I'm actually not sure how to reproduce it outside our environment.
> 
> In essence I found this using a statically-linked build where a self-built Python interpreter, NumPy, pandas, etc were all instrumented with ASAN. I don't think you could even find this particular bug without instrumenting _both_ NumPy _and_ pandas because the dereference here is of a NumPy array. It may or may not be necessary to be using a statically-linked build, which is a nonstandard configuration of Python and its modules we use internally.
> 
> While it's probably possible to reproduce the asan report outside our environment with some persistence it doesn't seem easy to do. I'm sorry that's a dissatisfying answer!

Thanks for explaining - I worried this would be the case. I remember trying to build `pandas` with ASAN on macOS and running into the issue of needing to instrument `numpy` also and running into many build issues and warnings. It got very complicated very quickly :)
Thanks @hawkinsp. Really happy to have more of these types of fixes found by ASAN
ENH: copy kwd for add_suffix/add_prefix
State: closed

- [ ] closes #xxxx (Replace xxxx with the Github issue number)
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Enhancement

Comments:
Thanks @jbrockmendel
ENH: copy keyword to set_axis
State: closed

This also opens the option of deprecating `inplace` here.

- [ ] closes #xxxx (Replace xxxx with the Github issue number)
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Enhancement

Comments:
@Dr-Irv im not clear on what mypy is asking for.  help?
> @Dr-Irv im not clear on what mypy is asking for. help?

I think the issue is as follows.  In `core/generic.py`, you have the `inplace` argument on the 3rd overload typed as:
```python
inplace: bool_t | lib.NoDefault = ...,
```
But in `series.pyi` and `frame.pyi`, the 3rd overload has:
```python
inplace: bool = ...
```

You have to make sure the signatures are the same in the superclass and subclass for each overload, and that the declared method (without `@overload`) has all the possible types of each argument that are specified across all overloads.

green, booyah
Thanks @jbrockmendel
Update config.yml to add Donate page to the nav bar
State: closed

It would also be really great to enable GitHub sponsors specifically for the Pandas Org since people might want to sponsor the org directly on GitHub (dev's / org's want to highlight their sponsorship publicly so GitHub sponsors is a great way to do so).


Labels:
name: Admin
description: Administrative tasks related to the pandas project
name: Web
description: pandas website

Comments:
Yeah that is a fair point! Just wanted to suggest this, let me know if I can be of any help on GitHub sponsors.
I created #47960 to open a discussion with your points, probably better than using this PR. I see that we're using this for GitHub sposnors: https://github.com/pandas-dev/.github/blob/4af72faf8bd1e0f1d3f12401c559a3a4b2a9a68b/FUNDING.yml

If you want to add a quick description on what it'd take what you suggest, that'd be amazing. I don't really know much about it.
Closing this in favor of the discussion that was opened!