BrandonFors/pandas-github-issues huggingface dataset:
BUG: Grouper specified by key not regarded as in-axis
State: closed

- [x] closes #50413 (Replace xxxx with the GitHub issue number)
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

Found this working on #46944. It was a case where we add to exclusions but consider the grouping as not in-axis. This creates a subtle difference in `_obj_with_exclusions` vs `_selected_obj`.

Labels:
name: Bug
name: Groupby

Comments:
Thanks @rhshadrach
STYLE: prohibit parsing exeption messages
State: closed

to fix #50353. 

Welcome any code review and comments.

Labels:
name: Error Reporting
description: Incorrect or improved errors from pandas
name: Stale

Comments:
Except 
```
./pandas/compat/pickle_compat.py:
./pandas/core/internals/blocks.py:
```
has been preliminarily completed, the above two cannot be completed due to my limited ability.
can you merge main?
This pull request is stale because it has been open for thirty days with no activity. Please [update](https://pandas.pydata.org/pandas-docs/stable/development/contributing.html#updating-your-pull-request) and respond to this comment if you're still interested in working on this.
TST: Consolidate tests that raise in groupby
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

Part of #50133

Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Groupby
name: Clean
name: Nuisance Columns
description: Identifying/Dropping nuisance columns in reductions, groupby.add, DataFrame.apply

Comments:
Thanks @rhshadrach
CLN: Remove unused rewrite warning
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

With groupby no longer silently dropping nuisance columns, the warning is no longer raised and so this goes unused.

Labels:
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode
name: Deprecate
description: Functionality to remove in pandas
name: Clean
name: Nuisance Columns
description: Identifying/Dropping nuisance columns in reductions, groupby.add, DataFrame.apply

Comments:
thx @rhshadrach
REF: restore _concat_managers_axis0
State: closed

It was removed in #47372 but removing it was not the point of that PR, just a side-effect.

Labels:
name: Refactor
description: Internal refactoring of code

Comments:
> Motivation for adding this back is that it's more performant?

Mostly simpler.  IIRC it might be a little more performant but very slight. 
Thanks @jbrockmendel
BUG: Index with null value not serialized correctly to json
State: closed

- [ ] closes #31801 (Replace xxxx with the GitHub issue number)
- [ ] closes #28609
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

Restores 0.25.3 behavior, which honestly feels a little weird to me, since it doesn't roundtrip. I'm not too familiar with JSON format, though (I guess the problem is that null can't be a key?).

Also serialize decimal nans correctly.


Labels:
name: Bug
name: IO JSON
description: read_json, to_json, json_normalize
name: Index
description: Related to the Index class or subclasses
name: Stale

Comments:
This pull request is stale because it has been open for thirty days with no activity. Please [update](https://pandas.pydata.org/pandas-docs/stable/development/contributing.html#updating-your-pull-request) and respond to this comment if you're still interested in working on this.
Thanks for the pull request, but it appears to have gone stale. If interested in continuing, please merge in the main branch, address any review comments and/or failing tests, and we can reopen.
STYLE pre-commit check to ensure that test functions name starts with test
State: closed

- [ ] closes #50379 (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Code Style
description: Code style, linting, code_checks

Comments:
Just for the record - my hope is that we can eventually get to 100% coverage, and that then this tool wouldn't be necessary

But in the meantime, it takes ~3 seconds to run on CI, and can help identify issues
BUG/COMPAT: fix assert_* functions for nested arrays with latest numpy
State: closed

This adds tests for `assert_almost_equal` (used by `assert_series_equal` et al in case of object dtype arrays) that all pass on pandas main with released numpy (except for two cases with dict of array), and fixes the implementation of `array_equivalent` (used by `assert_almost_equal`) to return False instead of raising an error if numpy cannot compare the arrays (numpy 1.25.dev starts to raise for this instead of returning False).

I also added a bunch of equivalent tests for `array_equivalent` itself, but those don't all pass (and added xfails). Trying to fix those as well might need more extensive changes, and I would prefer to do that separate from this PR (to keep this one possible to backport).

See https://github.com/pandas-dev/pandas/issues/50360#issuecomment-1362678681 for an illustration of the behaviour that changed in numpy 1.25.dev that causes this.

- [x] closes https://github.com/pandas-dev/pandas/issues/50360
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Compat
description: pandas objects compatability with Numpy or Python functions

Comments:
Any other comments here?
Thanks @jbrockmendel
ENH: Add cumsum to ArrowExtensionArray
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

Don't think a whatsnew is needed since `EA._accumulate` was added in 2.0


Labels:
name: Enhancement
name: Arrow
description: pyarrow functionality

Comments:
thx @mroeschke
CI/WEB: Use Github token to authenticate API calls
State: closed

Hopefully to help the doc and web build sometimes failing because of hitting the rate limit of the GIthub API

```
Traceback (most recent call last):
  File "web/pandas_web.py", line 403, in <module>
    main(args.source_path, args.target_path, args.base_url, args.ignore_io_errors)
  File "web/pandas_web.py", line 349, in main
    context = get_context(config_fname, ignore_io_errors, base_url=base_url)
  File "web/pandas_web.py", line 305, in get_context
    context = preprocessor(context)
  File "/home/runner/work/pandas/pandas/web/pandas_web.py", line 172, in maintainers_add_info
    resp.raise_for_status()
  File "/home/runner/micromamba/envs/test/lib/python3.8/site-packages/requests/models.py", line 1021, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 403 Client Error: rate limit exceeded for url: https://api.github.com/users/shoyer

```

Labels:
name: CI
description: Continuous Integration
name: Web
description: pandas website

Comments:
Haven't looked into this closely, but maybe the problem is that we have too many maintainers 🤣  , causing us to hit the API rate limit(I think it's set really low by default by Github).

We should probably figure out a way to use the GITHUB_TOKEN to call the API on CI, and fall back to unauthenticated API use when its not present(e.g. developing locally).
https://docs.github.com/en/actions/security-guides/automatic-token-authentication#example-2-calling-the-rest-api
Changed to use the `GITHUB_TOKEN` if available as an environment variable. Looks to be building the site correctly in the CI
Thx, lets give it a shot
CI: Test pandas warnings as error on some builds
State: closed

Locally I was validating the recent addition of `-W error:::pandas` and had to change it to `-W error:::$pandas` so validating if the CI will raise as well

Labels:
name: CI
description: Continuous Integration
name: Warnings
description: Warnings that appear or should be added to pandas

Comments:
So it appears the warning filter needs the fully qualified module name therefore we may need a way to generate all possible module paths in the library https://stackoverflow.com/a/73419829
Thx @mroeschke
CI Pin numpy 32 bit
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: CI
description: Continuous Integration

Comments:
yup, here https://github.com/pandas-dev/pandas/issues/50392


from https://github.com/pandas-dev/pandas/pull/49777 it seems there a preference for not pinning numpy unless it's necessary, so I'll just xfail the test
Thanks @MarcoGorelli
ENH: Add std to masked interface
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

The warnings were already shown before I changed the code paths, but not caught


Labels:
name: NA - MaskedArrays
description: Related to pd.NA and nullable extension arrays
name: Reduction Operations
description: sum, mean, min, max, etc.

Comments:
Thanks @phofl
CI try skipping test in 32-bit
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:

Comments:
Closing, the other pr seems to fix the timeout
ENH: Preserve ea dtype in groupby.std
State: closed

- [x] closes #35516 (Replace xxxx with the GitHub issue number)
- [x] closes #43330 (Replace xxxx with the GitHub issue number)
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Groupby
name: NA - MaskedArrays
description: Related to pd.NA and nullable extension arrays

Comments:
Thanks @phofl
Add nanoseconds to timestamps stub file
State: closed

I also found the documentation to be lacking. I had to try it out to determine if `nanoseconds` would be giving me all nanoseconds until the next full second or just until the next full microsecond (the latter is the case).

- [x] no issue exists to track this (probably small enough)
- [x] tests not applicable, I think
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] ~~Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.~~ As suggested by @phofl not needed


Labels:
name: Typing
description: type annotations, mypy/pyright type checking

Comments:
Thanks @robsdedude
Rename duplicate column names in read_json(orient='split')
State: closed

- [X] xref #47718

All our read methods drop duplicated columns (`read_xml` and `read_json(orient='records')`) or rename them with the pattern `x`, `x.1`, `x.2`... (`read_csv`, `read_fwf`, `read_excel`, `read_html`). There is only one exception where the built `DataFrame` contains duplicated columns, which is `read_json(orient='split')`.

Making `read_json(orient='split')` be consistent with the other methods, and rename the duplicate columns here.

Follow ups:

- Use the `dedup_names` function everywhere when columns are renamed #50371
- Add a parameter to the `read_*` functions where the format can have duplicates, to allow better control of the behavior when dealing with duplicate column names #47718

CC: @MarcoGorelli 

Labels:
name: IO JSON
description: read_json, to_json, json_normalize

Comments:
Thanks @datapythonista (failures unrelated). Follow up PRs can be submitted if needed
ENH/API: preserve non-nano in to_datetime
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Non-Nano
description: datetime64/timedelta64 with non-nanosecond resolution

Comments:
Thanks @jbrockmendel
ERR non-ISO formats don't show position of error 
State: closed

- [ ] closes #50361 (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

---



Labels:
name: Datetime
description: Datetime data dtype
name: Error Reporting
description: Incorrect or improved errors from pandas

Comments:
I've aligned them now, thanks - this should  make the diff in #50242  much smaller

----

```python
In [1]: to_datetime(['2020-01-01', '9999-01-01'], format='%Y-%m-%d')
---------------------------------------------------------------------------
OutOfBoundsDatetime: Out of bounds nanosecond timestamp: 9999-01-01 00:00:00 present at position 1

In [2]: to_datetime(['2020-01-01', '9999-01-01'], format='%Y-%d-%m')
---------------------------------------------------------------------------
OutOfBoundsDatetime: Out of bounds nanosecond timestamp: 9999-01-01 00:00:00 present at position 1

In [3]: to_datetime(['2020-01-01', 'foo'], format='%Y-%m-%d')
---------------------------------------------------------------------------
ValueError: time data "foo" at position 1 doesn't match format "%Y-%m-%d" (match)

In [4]: to_datetime(['2020-01-01', 'foo'], format='%Y-%d-%m')
----------------------------------------------------------------------------
ValueError: time data "foo" at position 1 doesn't match format "%Y-%d-%m" (match)
```
Looks like a merge conflict but feel free to merge after that
cool, thanks for reviewing!
Backport PR #50356 on branch 1.5.x (CI temporarily pin numpy)
State: closed

Backport PR #50356: CI temporarily pin numpy

Labels:
name: CI
description: Continuous Integration

Comments:
@MarcoGorelli the 32 bit failure seems legit
yeah - should we backport #49777?
or just pin `numpy` in the `32 Bit Linux` job on 1.5.x...either way, it'd probably be best done in a separate PR, so that this one is just a pure backport

we OK to merge this one for now?
yep ok to merge for now
CI temporarily pin numpy
State: closed

Looks like numba is failing with the latest numpy release https://github.com/numba/numba/issues/8615

Let's temporarily pin numpy to get CI green, and then lift the pin once it's solved in numba?

Labels:
name: CI
description: Continuous Integration

Comments:
I guess you have to pin in the asv config as well?
not strictly necessary for CI, as `--python=same` is used in that job - but yeah, probably better, thanks
thx @MarcoGorelli 
Could we remove numba in some of the envs instead? Because not testing with the latest numpy anywhere doesn't seem like a good situation? 
(I don't know the timeline of a fix fox this in numba, but the issue is already open a while)
the 311-dev job tests with the latest (nightly) numpy, and that one doesn't have numba

perhaps let's wait a few days to see if there's any progress, and if not just remove numba from CI 'til they resolve it?
CI: Enable pytest -W:::pandas on some builds
State: closed

- [ ] xref #48553 (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: CI
description: Continuous Integration

Comments:
thx @mroeschke
DOC: Add drop to MultiIndex docs
State: closed

- [x] closes #50346 (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Docs
name: MultiIndex

Comments:
Thanks @phofl
Update link to `Categorical._codes` removal in 2.0 what's new doc
State: closed

Currently this is pointing to the PR where `Categorical._codes` was deprecated in `pandas=1.3` (https://github.com/pandas-dev/pandas/pull/41429) instead of the PR where it was removed for `pandas=2.0` (https://github.com/pandas-dev/pandas/pull/49258)

Labels:

Comments:
I think we point to the deprecation prs in general.
Ah, happy to close this PR if it's against an agreed upon standard practice. FWIW I will say I found it confusing as the PR wasn't consistent text in the changelog entry. 
Yeah we typically link to the PR that introduces the deprecation (which hopefully links to the PR that enforces the deprecation)