BrandonFors/pandas-github-issues huggingface dataset:
DOC: Add rolling in reference/groupby.rst
State: closed

- [ ] close #51097




Labels:
name: Docs
name: Groupby
name: Window
description: rolling, ewma, expanding

Comments:
Since we have docstring checks, the docstrings of these will need to be improved in this PR

```
Error: /home/runner/work/pandas/pandas/pandas/core/groupby/groupby.py:2787:RT01:pandas.core.groupby.DataFrameGroupBy.rolling:No Returns section found
Error: /home/runner/work/pandas/pandas/pandas/core/groupby/groupby.py:2787:RT01:pandas.core.groupby.SeriesGroupBy.rolling:No Returns section found
Error: /home/runner/work/pandas/pandas/pandas/core/groupby/groupby.py:2787:EX01:pandas.core.groupby.DataFrameGroupBy.rolling:No examples section found
Error: /home/runner/work/pandas/pandas/pandas/core/groupby/groupby.py:2787:EX01:pandas.core.groupby.SeriesGroupBy.rolling:No examples section found
```
> Since we have docstring checks, the docstrings of these will need to be improved in this PR
> 
> ```
> Error: /home/runner/work/pandas/pandas/pandas/core/groupby/groupby.py:2787:RT01:pandas.core.groupby.DataFrameGroupBy.rolling:No Returns section found
> Error: /home/runner/work/pandas/pandas/pandas/core/groupby/groupby.py:2787:RT01:pandas.core.groupby.SeriesGroupBy.rolling:No Returns section found
> Error: /home/runner/work/pandas/pandas/pandas/core/groupby/groupby.py:2787:EX01:pandas.core.groupby.DataFrameGroupBy.rolling:No examples section found
> Error: /home/runner/work/pandas/pandas/pandas/core/groupby/groupby.py:2787:EX01:pandas.core.groupby.SeriesGroupBy.rolling:No examples section found
> ```

I added some docstring, but they definitely need further discussion. 

The parameters section is copied from [pandas.DataFrame.rolling](https://pandas.pydata.org/docs/dev/reference/api/pandas.DataFrame.rolling.html#pandas.DataFrame.rolling), the parameter `step` is not implemented, so I removed it.  Other parameters are implemented, but the exact description I'm not sure if I need to keep changing.
Thanks @luke396
DEPR: remove NumericIndex.__new__ & ._should_fallback_to_positional
State: closed

Remove `NumericIndex.__new__` & `._should_fallback_to_positional` to Index. Next up is to actually remove `NumericIndex` from the code base (there is various scaffolding around, that I need to remove/alter at the same time).

This move `NumericIndex._should_fallback_to_positional` to `Index` as-is. There is an issue #51053 that should be fixed, I'll take that in a follow-up so I can get `NumericIndex` removed here, without worrying about that.

xref #42717.

Labels:
name: Deprecate
description: Functionality to remove in pandas
name: Index
description: Related to the Index class or subclasses

Comments:
I think the ARM failure is unrelated.
Thanks @topper-123
REF: implement groupby std, min, max as EA methods
State: closed

xref #51003 does the same thing for quantile
xref #43682 asking for something similar (but not for these particular reductions)

Allows EA authors to implement performant implementations (including us for e.g. pyarrow dtypes).

Labels:
name: Groupby
name: Reduction Operations
description: sum, mean, min, max, etc.

Comments:
> There any performance impact here?

Not directly, but a big part of the idea is to make it easier to implement performant implementations for our pyarrow arrays (and in my daydreams, for 3rd party authors to do Distributed/GPU EA implementations)
Does any/all/std for sure need a different API then `groupby_op` in https://github.com/pandas-dev/pandas/pull/51166?
> Does any/all/std for sure need a different API then groupby_op in https://github.com/pandas-dev/pandas/pull/51166?

No.  #52053 moves std to use WrappedCythonOp, so it would no longer need a separate method here.  #52043 is a step in the same direction for any/all
Okay so IIUC, `_groupby_any_all` and `_groupby_std` is a bridge to `_groupby_op` in https://github.com/pandas-dev/pandas/pull/51166?
> Okay so IIUC, _groupby_any_all and _groupby_std is a bridge to _groupby_op in https://github.com/pandas-dev/pandas/pull/51166?

Yes.  Though at this point I'm _almost_ ready to close this.
Closing in favor of #52089
ENH: Add use_nullable_dtypes to read_spss
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

Don't want to create another file, so a minimal example to show that the implementation works

Labels:
name: Enhancement
name: NA - MaskedArrays
description: Related to pd.NA and nullable extension arrays
name: Arrow
description: pyarrow functionality

Comments:
Thanks @phofl
DOC: Fix doc example in read_stata
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

This is not rendering properly right now

Labels:
name: Docs

Comments:
Thanks @phofl (the doc and code checks passed)
DOC: Update docs to reflect that Index can hold int64, int32 etc. arrays
State: closed

Add whatsnew and make doc changes in order to document that `Index` can hold int64, int32 etc. arrays. Removes references to `Int64Index` etc. from the docs.

xref #42717.

Labels:
name: Docs

Comments:
I think the ARM failure is unrelated.
Nice whatsnew summary!
DOC Unsetting "copy_on_write" mode to remove EX02 errors
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

Towards #37875 and #27977. Additional info #48998 

@jorisvandenbossche 

Labels:
name: Docs

Comments:
Thanks @DeaMariaLeon 
Thanks to you @mroeschke
BUG: doc string error
State: closed

An error is occurring in `ci/code_checks.sh docstrings` because of a missing rebase:

```
Error: /home/runner/work/pandas/pandas/pandas/core/indexes/base.py:285:EX02:pandas.Index:Examples do not pass tests:
**********************************************************************
Line 49, in pandas.Index
Failed example:
    pd.Index([1, 2, 3], dtype="uint8")
Expected:
    NumericIndex([1, 2, 3], dtype='uint8')
Got:
    Index([1, 2, 3], dtype='uint8')
```

This fixes that.

xref #42717.

Labels:
name: Docs
name: Deprecate
description: Functionality to remove in pandas

Comments:
thx @topper-123
REF: GroupBy.ops don't depend on mutation status
State: closed

`self.mutated` is only used in one place (see one-line edit in first commit) and removing that doesn't break anything.  Conceivably could change some behavior but until we have an example I don't think it is even worth documenting as an API change.

Might be leftover from the old libreduction path?

cc @jreback for the deep cut into the history

Labels:
name: Groupby
name: Clean

Comments:
Thanks @jbrockmendel
DEPR: Remove NumericIndex from tests/arithmetic/
State: closed

Removes `NumericIndex` from `pandas/tests/arithemetic/`.

I've tried quite a lot to generalize the tests here to be for all the numeric numpy dtypes, but it seems almost impossible. The main issue is that numexpr operations behave differently for many dtypes lower than 64-bit than python ones, and its behaviour just seems unpredictably different than the normal python ops.

The tests in this directory concern both indexes and series and non-64-bit dtypes weren't tested before, even though it was possible for Series, and I assume the reason for that is the same difficulty, though I don't know for sure.

xref #42717.

Labels:
name: Deprecate
description: Functionality to remove in pandas

Comments:
thx @topper-123
API: integer values and non-nano dtype
State: closed

I'm open to suggestions for a better wording on the whatsnew.

Note this handles all-int cases but not mixed-int.

timedelta64 analogue coming up.

Labels:
name: Non-Nano
description: datetime64/timedelta64 with non-nanosecond resolution

Comments:
test failures on circleci look related?
> test failures on circleci look related?

yep, and there was even a helpful comment that telling me not to do what i did.  hopefully fixed now
Thanks @jbrockmendel
DOC add examples to Timestamp.unit
State: closed

xref https://github.com/pandas-dev/pandas/issues/37875

Just an example of how to open such a PR

Before submitting, you should run
```
python scripts/validate_docstrings.py pandas.Timestamp.unit
```
and check that `No examples section found` doesn't appear in the "errors found" section (if other errors appear, it's probably fine for now)

You should then also remove the function you've fixed up to the exclusions in `ci/code_checks.sh` under the EX01 section


Labels:
name: Docs
name: Timestamp
description: pd.Timestamp and associated methods

Comments:
Thanks @MarcoGorelli
PERF: avoid copy in _obj_with_exclusions, _selected_obj
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

cc @rhshadrach 

```
import pandas as pd
import numpy as np
import psutil
proc = psutil.Process()

np.random.seed(456123)
df = pd.DataFrame(
    np.random.randn(10_000_000, 5),
    columns=["A", "B", "C", "D", "E"],
)

grps = np.arange(10).repeat(1_000_000)
gb = df.groupby(grps)

gb2 = gb[["C"]]

In [3]: np.shares_memory(gb2._obj_with_exclusions["C"], df["C"])
True   # <- PR
False  # <- main

In [5]: proc.memory_info()
Out[5]: pmem(rss=658894848, vms=36076212224, pfaults=169250, pageins=0)  <- main
Out[5]: pmem(rss=578924544, vms=35993063424, pfaults=149692, pageins=0)  <- PR
```

Labels:
name: Groupby
name: Performance
description: Memory or execution speed performance

Comments:
Do you think the memory perf improvement is worth a note in the whatsnew?
> Do you think the memory perf improvement is worth a note in the whatsnew?

added
Thanks @jbrockmendel 
i think the describe PR will need rebasing (might want to wait until after #51113 is merged) and it should be good to go
DEPR: Move NumericIndex doc string + some minor stuff
State: closed

Moves doc string  of `NumericIndex` to `Index`. Also moves `_can_hold_strings` attribute and removes  `NumericIndex._default_dtype`, & `NumericIndex._typ`.

xref #42717.


Labels:
name: Deprecate
description: Functionality to remove in pandas
name: Index
description: Related to the Index class or subclasses

Comments:
thx @topper-123
TEST: join df with categorical multiIndex
State: closed

- [x] closes #50906
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).

Add test to check index dtype after joining DataFrames with categorical multiIndex.

Labels:
name: Testing
description: pandas testing functions or related to the test suite

Comments:
Thank you, I am looking forward to seeing a cat :)
this is currently marked as draft:

![image](https://user-images.githubusercontent.com/33491632/216652138-8cc3125b-aba2-478d-bab2-e72e70bf678d.png)

it looks ready though, anything else need doing? if you mark as ready for review I think we can ship it
I forgot to change the status of the PR. Now it is "ready for review" :)
BUG: can't round-trip non-nano Timestamp
State: closed

- [ ] xref #51060 (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

This should fix the interger case, but not the float one, so let's still keep the issue open for now

Labels:
name: Non-Nano
description: datetime64/timedelta64 with non-nanosecond resolution

Comments:
Thanks for your review!
API / CoW: return read-only numpy arrays in .values/to_numpy()
State: closed

Context: with the Copy-on-Write implementation (see overview follow up issue https://github.com/pandas-dev/pandas/issues/48998), we can avoid that mutating one pandas object doesn't update another pandas object. But users can still easily get a viewing numpy array, and mutate that one. And at that point, we don't have any control over how this mutation propagates (it might update more objects than just the one from which the user obtained it, for example if other Series/DataFrames were sharing data with this object with CoW). 

This is a draft PR just starting to explore this (see how much fails in our own test suite, to get an idea of the impact)

Labels:
name: Copy / view semantics

Comments:
This needs some more tests, and currently also doesn't yet handle EAs (eg np.asarray(Series[Int64]))
Couple smaller comments. I think this is something we should document explicitly. But can do as follow up I guesss
I added an explicit test for the failing `np.fix` case (currently this was called randomly in an unrelated indexing test). 

The problem with `np.fix` is that this is basically implemented under the hood as:

```
res = np.asanyarray(np.ceil(x))
res = np.floor(x, out=res, where=np.greater_equal(x, 0))
```

And so the `np.asanyarray` result now gives a read-only array, but this is passed in a second step as the `out` parameter, trying to write values into it.

In this case if the `np.asanyarray` and `np.ceil` calls would be inversed, it would work (first converting to read-only numpy array, and then np.ceil operation would effectively return a new writeable array)
thx @jorisvandenbossche 
cc @seberg @phofl my comment above at https://github.com/pandas-dev/pandas/pull/51082#issuecomment-1455882487 is the case that I was remembering where our change to read-only broke a legit case of calling a numpy function on a Series object: normally, the np.ceil call should make it writeable, but because this is wrapped again in a Series (since `np.ceil` is a ufunc), this new writeable array becomes again read-only when converted back to numpy, and then passing this to the `out` param in `np.floor` gives an error. 
This causes an actual regression from the pandas' user point of view, while it could still work. I don't know if for numpy there is a specific reason to do `np.asanyarray(np.ceil(x))` instead of `np.ceil(np.asanyarray(x))`? Although I suppose because you want to allow the object to use its overridden version through `__array_ufunc__`, and converting it to an array first would defeat that.
The `asanyarray` is really there for an obscure purpose probably (numpy converting 0-D arrays to scalars mostly).  But yeah, it is also needed since only arrays are considered valid `out=` arguments (although I have to double check `__array_prepare__` might allow it, but OTOH, I don't think it ever fully worked.

In either case, since we call it along the way, you are right that it would be fine to _add_ an `asanyarray` call at the beginning.
TST: Move xfail to its own test
State: closed

- [x] closes #35443 (Replace xxxx with the GitHub issue number)
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Groupby
name: Clean

Comments:
Thanks @rhshadrach
BUG: astype to string modifying input array inplace
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

inferring is faster than copying in all cases.

Labels:
name: Dtype Conversions
description: Unexpected or buggy dtype conversions
name: Strings
description: String extension data type and string data

Comments:
Thanks!
CI: Address flaky unit tests
State: closed

Noticed `test_basic_series_frame_alignment` and `test_raw_roundtrip` can occasitionally fail in the CI. Marking as xfail strict=False and clearing the clipboard respectively

Labels:
name: CI
description: Continuous Integration
name: Unreliable Test
description: Unit tests that occasionally fail

Comments:
pandas/tests/io/test_clipboard.py:426:12: W1510: Using subprocess.run without explicitly set `check` is not recommended. (subprocess-run-check)
thanks @mroeschke
DEPR: Remove NumericIndex name in repr output
State: closed

This is getting close now.

This PR removes `”NumericIndex”` name from the index repr output and replaces it with `"Index"` in preparation for removal of `NumericIndex`.

Also introduces some scaffolding that will be removed when we remove `NumericIndex`  

xref #42717.

Labels:
name: Deprecate
description: Functionality to remove in pandas
name: Index
description: Related to the Index class or subclasses

Comments:
Thanks @topper-123
REF: use _treat_as_na more
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Groupby
name: Clean

Comments:
Thanks @jbrockmendel
Update copyright year
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Docs

Comments:
Thanks @StepfenShawn
BUG: Series(category).replace to maintain order of categories
State: closed

- [x] closes #51016
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.

No whatsnew as this is a regression on main only.


Labels:
name: Bug
name: Categorical
description: Categorical Data Type

Comments:
Thanks @lukemanley
BUG: Series.tolist to return None in place of pd.NA
State: closed

- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

As discussed here: https://github.com/pandas-dev/pandas/issues/29738#issuecomment-1407693166

Offering this up as a potential solution to make `Series.tolist` return `None` (python native types) in place of `pd.NA`. This would make `Series.tolist` consistent with `Series.to_dict` which at the moment differ:

```
In [1]: import pandas as pd

In [2]: ser = pd.Series([1, pd.NA], dtype="Int64")

In [3]: ser.to_dict()
Out[3]: {0: 1, 1: None}

In [4]: ser.tolist()
Out[4]: [1, <NA>]
```

I don't think this fully closes #29738 as there may still be an open question about `__iter__`. 

Labels:
name: Missing-data
description: np.nan, pd.NaT, pd.NA, dropna, isnull, interpolate
name: Dtype Conversions
description: Unexpected or buggy dtype conversions
name: Stale

Comments:
This pull request is stale because it has been open for thirty days with no activity. Please [update](https://pandas.pydata.org/pandas-docs/stable/development/contributing.html#updating-your-pull-request) and respond to this comment if you're still interested in working on this.