BrandonFors/pandas-github-issues huggingface dataset:
ENH: Do unique check before computing indexer in reindex
State: closed

- [x] closes #51287 (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves
name: Performance
description: Memory or execution speed performance

Comments:
Looks good. Could use a performance whatsnew note 
I'd feel a bit weird advertising that we are raising the error faster now :) But ok to add, if you feel strongly about it
Guess that's fair. Thanks @phofl 
Hello, @phofl @mroeschke  I was wondering if this pull request has been backported? According to the [backporting guidelines](https://pandas.pydata.org/docs/development/maintaining.html#backporting), it seems that PRs with a milestone should be backported. However, I couldn't find a corresponding backport PR linked to this one. Can you please clarify if a backport has been created or if there's a different situation here? 
This pr was released in 2.0, which is what the milestone tells you. 

You’ll have to clarify your question. To which branch should we have backported this pr?
Thank you for your response! 

Sorry for any confusion earlier, I'm not familiar with the open-source management mechanisms on GitHub. I took another look at the documentation and realized indeed when a milestone corresponds to the master version, a backport isn't necessary. 

My question is actually about the process of fixing bugs: when you address a bug, do you first fix it in the master version and then backport that fix to the milestone, which is the target version? Does this mean that other versions won't receive the fix? For example, if the master is at 2.0 and the milestone is set for 1.3.x, does that mean only these two versions will be corrected on GitHub, and other minor versions, such as 1.4.x, 1.5.x, won't be fixed? Additionally, regarding PyPI releases, if a version on GitHub is backported, will the content in the PyPI package be updated with the fix?
pandas only maintains 2 branches during active development: The main branch which reflects the next upcoming minor/major release, e.g. 3.0, and the latest minor release branch, e.g. 2.2.x. A backported PR will apply to main and the latest minor release branch. The latest minor release branch usually only gets 3 or 4 packages on PyPI and conda e.g. 2.2.1, 2.2.2,  2.2.3.

Thank you for your response! Now I get it.
TST: Test mask/where inplace don't modify views with CoW
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Copy / view semantics

Comments:
Failure unrelated
thx @lithomas1
ENH: Enable more Arrow CSV tests/features
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: IO CSV
description: read_csv, to_csv
name: Arrow
description: pyarrow functionality

Comments:
I'll take a look tomorrow
gentle ping @phofl.
thx @lithomas1
TST: avoid mutating DataFrame.values in tests (use iloc instead)
State: closed

This is splitting off some "usefully anyway" test changes from https://github.com/pandas-dev/pandas/pull/51082

We have several places internally where we mutate the `df.values[..] = ..` directly, while this is certainly a discouraged pattern (also only work if you have a single block), and often easily avoided by using `df.iloc[..] = ..` or mutating the array before creating the DataFrame, instead.

Labels:
name: Testing
description: pandas testing functions or related to the test suite

Comments:
thx @jorisvandenbossche
BUG: avoid StringArray.__setitem__ to mutate the value being set as side-effect
State: closed

Encountered this while working on https://github.com/pandas-dev/pandas/pull/51082, and this can be done as independent fix.

```
In [5]: arr = pd.array(["a", "b", "c"], dtype="string")

In [6]: value = np.array(["A", None])

In [7]: arr[[0, 1]] = value

In [8]: value
Out[8]: array(['A', <NA>], dtype=object)  # <-- value got mutated !!
```

I think in general the value that is being set should _never_ be mutated during the setitem operation.

- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Bug
name: Strings
description: String extension data type and string data

Comments:
Thanks @jorisvandenbossche
ENH: enable lazy copy in merge() for CoW
State: closed

xref https://github.com/pandas-dev/pandas/issues/49473

The current `copy=False` only works when merging on the index; when merging on keys, we currently get a [0, 1, 2, ..] join index for both left and right, and so that still results in a copy (but that can be avoided using the `is_range_indexer` utility)

Labels:
name: Copy / view semantics

Comments:
Ah mypy is complaining:

```
pandas/core/reshape/merge.py:155: error: Argument "copy" to "get_result" of "_MergeOperation" has incompatible type "Optional[bool]"; expected "bool"  [arg-type]
``` 
thx @jorisvandenbossche
TST: add CoW test for setitem with Series being set
State: closed

xref https://github.com/pandas-dev/pandas/issues/49473

We already have a test for setitem from https://github.com/pandas-dev/pandas/pull/50692/, but that only handled setting an array (actually, does that copy the array?), and not setting with a pandas object for which we could use lazy copy mechanism. 

This is only adding the test, we seem to actually do an eager copy, so room for optimization.

Labels:
name: Copy / view semantics

Comments:
https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.isetitem.html
The comment in the test was a left-over from copying the previous setitem test, will remove.

But the docstring is actually confusing here, because it _is_ an inplace method in the sense that it is updating the _object_ inplace (we listed this method in the "always inplace" bucket in the draft PDEP). It's only not inplace in the sense that it will not update _values_ inplace.


Yeah good point, a clarification would probably help
Fixed a small typo/copy paste issue. otherwise good to merge on greenish
thx @jorisvandenbossche 

failures were unrelated
TST: add CoW tests for xs() and get()
State: closed

xref https://github.com/pandas-dev/pandas/issues/49473

Adding basic tests for `xs` and `get`. Those rely on indexing methods under the hood, and thus already work correctly for CoW, but adding some basic tests to have independent coverage.

Labels:
name: Copy / view semantics

Comments:
Failures are unrelated, so merging
CI pre-commit autoupdate
State: closed

the automated one didn't run, maybe because the diff is too large?

this touches a lot of files, but it's mostly just `black` removing newlines


Labels:
name: Code Style
description: Code style, linting, code_checks

Comments:
Could also upgrade autotyping to 23.2.0 (pyright as well but that will probably require some code changes).
> but it's mostly just `black` removing newlines

Is there anything else? I cab only see removal of newlines, but there are a lot, so may I'm missing something? 
> Could also upgrade autotyping to 23.2.0 (pyright as well but that will probably require some code changes).

I tried this but got
```
IndexError: string index out of range

Failed to codemod /home/marcogorelli/pandas-dev/pandas/core/arrays/categorical.py
```
I'll see if this needs reporting to them

> Is there anything else? I cab only see removal of newlines, but there are a lot, so may I'm missing something?

I don't think so, but perhaps let's keep that to a separate PR
thanks for reviewing!
CI: Xfail plotting tests
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: CI
description: Continuous Integration

Comments:
relevant jobs passed, merging to get ci green again
thanks!
DOC Fix EX01 errors - added example
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

Towards #37875 

Labels:
name: Docs
name: Code Style
description: Code style, linting, code_checks

Comments:
Thanks again :)
ENH: Add CoW optimization for fillna
State: closed

- [ ] xref #49473 (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Copy / view semantics

Comments:
failure unrelated (flaky test)
DOC: Remove reference to mysql fallback support from user guide.
State: closed

- [x] closes #51263
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Docs
name: IO SQL
description: to_sql, read_sql, read_sql_query

Comments:
Thanks @cdcadman
BUG: can't resample with non-nano dateindex, out-of-nanosecond-bounds
State: closed

- [ ] closes #51024 (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Resample
description: resample method
name: Non-Nano
description: datetime64/timedelta64 with non-nanosecond resolution

Comments:
One comment on the non-test code, otherwise looks good.  i dont know the resample code well enough to know on sight if the test "expected" is correct, but trust you
> One comment on the non-test code, otherwise looks good. i dont know the resample code well enough to know on sight if the test "expected" is correct, but trust you

Thanks for your review @jbrockmendel !

Regarding the expected value: it matches what pandas would currently give for nanosecond resolution:
```python
In [33]:     idx = date_range("1983-01-01", "2000-01-01", freq="Y")
    ...:     ser = Series([1, 4, 2, 8, 5, 7, 1, 4, 2, 8, 5, 7, 1, 4, 2, 8, 5], index=idx)
    ...:     ser.resample("2Y").mean()
Out[33]:
1983-12-31    1.0
1985-12-31    3.0
1987-12-31    6.5
1989-12-31    4.0
1991-12-31    3.0
1993-12-31    6.5
1995-12-31    4.0
1997-12-31    3.0
1999-12-31    6.5
Freq: 2A-DEC, dtype: float64
```

, so if that's right, then this is right.

OK to merge?
Thanks @MarcoGorelli
CI: Don't test min pyarrow version with multiple python versions
State: closed

cc @phofl 

After the minimum pyarrow version was bumped to 7, we shouldn't need to test this over multiple python versions anymore.

Additionally, uses PY3.11 to test multiple pyarrow versions instead of PY3.10

Labels:
name: CI
description: Continuous Integration
name: Arrow
description: pyarrow functionality

Comments:
Looks like the non flaky failures are due to #51006 so merging to reduce the number of builds running
CoW: Ensure that iterrows does not allow mutating parent
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

cc @jorisvandenbossche 

Any idea on how to do this nicer?

Do you think we need a way to hand over a ref object when constructing a Series from an array at some point?

Labels:
name: Copy / view semantics

Comments:
Yeah, if this is a pattern that will be used more, then we should add a helper for it. In general I think we should still try to avoid this kind of pandas object -> values -> pandas object with views _outside of the internals_ to avoid having to think about refs there.

Possible ideas to do this cleaner:

- Rely on indexing (`self.iloc[i]` to iterate over rows), and then refs will already be taken care of. But I suppose that will give a big overhead (because you repeat the dtype calculation for each row). And that's probably the same for `self._mgr.fast_xs(i)` which is special-case method in internals to get a row.
- Add a method to the BlockManager to have the core logic there (like a version of `fast_xs` but then not for a single row but all rows)
Yep I agree with all your points and would also
Like to avoid it as much as possible, but it’s tricky here without hurting Performance I think.

I‘ll explore adding something on the manager level, that sounds like a good compromise
Shall we short-term go for this "ugly" solution? (so at least to have correct behaviour)
Probably a good idea, I think I can get something done before the actual 2.0 is out
ENH: Optimize putmask implementation for CoW
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Copy / view semantics

Comments:
The failing test in the CoW build is the flaky `test_file_descriptor_leak`
CoW failure unrelated (flaky test)
:) funny timing
I was first! ;)
Yep! was a tad to slow :(
DOC Correct EX02 docstring errors
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

Towards #51236 

Labels:
name: Docs
name: Code Style
description: Code style, linting, code_checks

Comments:
Thank you @MarcoGorelli 👍
DOC: Update pct_change documentation
State: closed

Related to SO question: [How to avoid bfill or ffill when calculating pct_change with NaNs](https://stackoverflow.com/questions/75395345/how-to-avoid-bfill-or-ffill-when-calculating-pct-change-with-nans)


Labels:
name: Docs

Comments:
Thanks @corralien 
Thanks @corralien for having fixed this and brought it to our attention!

It actually gets slightly worse: from http://pandas-coverage.herokuapp.com/ it looks like there's no test case with `fill_method` as `None`, and the type hints don't even cover it. I'll open a new issue about
Comments improved.
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:

Comments:
thanks but no thanks - please have a look at https://github.com/pandas-dev/pandas/issues/51236 if you want a good first issue to contribute to
BUG: IntervalArray.shift with invalid NA fill_value
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Missing-data
description: np.nan, pd.NaT, pd.NA, dropna, isnull, interpolate
name: Interval
description: Interval data type

Comments:
Thanks @jbrockmendel
BUG: Groupby.filter with pd.NA
State: closed

@Dr-Irv this is an example of the kind of bug that I have in mind when I said in the Categorical issue that getting pd.NA is a PITA.   We've had 3 years to track them down and we're still finding places where having it unexpectedly causes bugs.  Users will have this long tail until the end of time.

Labels:
name: Groupby
name: Missing-data
description: np.nan, pd.NaT, pd.NA, dropna, isnull, interpolate

Comments:
> @Dr-Irv this is an example of the kind of bug that I have in mind when I said in the Categorical issue that getting pd.NA is a PITA. We've had 3 years to track them down and we're still finding places where having it unexpectedly causes bugs. Users will have this long tail until the end of time.

That's a bit unavoidable at this point, unless we decide to get rid of `pd.NA` , but I don't see that happening

Is there a potential performance impact?
> Is there a potential performance impact?

Hard to see how


My thinking was notna might be more expensive than checking b first
thx @jbrockmendel
BLD: Try stripping wheels
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Build
description: Library building on various platforms

Comments:
This gets Linux wheel sizes down from ~40 MB to ~12 MB to match the wheel sizes built with multibuild.
Sweet thanks @lithomas1