BrandonFors/pandas-github-issues huggingface dataset:
BUG: IntervalIndex.get_indexer raising for read only array
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Interval
description: Interval data type

Comments:
Should we just generally go back to using ndarray instead of memoryviews? If we set that expectation could be a good change for new contributors to pick up
This change is only necessary because cython does not support const with fused types unfortunately
Ah OK. So I guess the pattern is for fused types prefer ndarray (unless that gets fixed/implemented) elsewhere memview?
Yep that makes sense I guess
>  So I guess the pattern is for fused types prefer ndarray (unless that gets fixed/implemented) elsewhere memview?

And that's also what we already do in most places. For example in groupby.pyx, you will see that we typically use ndarray for the input values (which use a fused type and can be const), while mostly use memoryviews for output values (which are writeable) or labels/mask (which don't use fused types). 
(although it's certainly not exact, I see some cases using ndarray that could use memoryview)
merging this, happy to address comments in a follow up.
BUG: concat coercing arrow to object with null type
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode
name: Arrow
description: pyarrow functionality

Comments:
Thanks @phofl
Fixed #52234
State: closed

* Copied fix from https://github.com/ltauro/pandas/tree/nan-not-in-index and applied review comments

- [x] closes #52234
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Missing-data
description: np.nan, pd.NaT, pd.NA, dropna, isnull, interpolate

Comments:
Thanks @ssche
PERF: concatenation of MultiIndexed objects (MultiIndex.append)
State: closed

- [x] closes #53574
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/v2.1.0.rst` file if fixing a bug or adding a new feature.


cc: @jorisvandenbossche - nice find/suggestion.

Perf improvement for `MultiIndex.append` (and `pd.concat` for objects with MultiIndexes):

```
       before           after         ratio
       <main>          <mi-append>
-        67.3±2ms       5.09±0.1ms     0.08  multiindex_object.Append.time_append('datetime64[ns]')
-        68.5±1ms       5.10±0.1ms     0.07  multiindex_object.Append.time_append('int64')
-         147±5ms       5.40±0.3ms     0.04  multiindex_object.Append.time_append('string')
```


Labels:
name: Performance
description: Memory or execution speed performance
name: MultiIndex

Comments:
> Do you think it would be worth testing this with more MIs? (not necessarily in the asvs, can also be just here in the PR) Or are we confident this should be faster (or at least not a big slowdown) for most cases?

Did a quick test with a case of complete unique values in the MI levels (so not even repetition _within_ a level):

```
N = 100_000
idx1 = pd.MultiIndex.from_arrays([range(N), np.random.choice(list(string.ascii_letters), size=(N, 5)).astype(object).sum(axis=1)])
idx2 = pd.MultiIndex.from_arrays([range(N, N*2), np.random.choice(list(string.ascii_letters), size=(N, 5)).astype(object).sum(axis=1)])
%timeit idx1.append(idx2)
```

and even here the new algo is faster (just a smaller difference, "only" around 2x faster). 
My gut feeling says that such a case of all unique level values is the "worse" case for the recoding-based implementation, so that seems to indicate we can assume this to be faster generally (in the end, also with the current implementation of combining the (unique) level values, those have to be encoded afterwards anyway, so recoding shouldn't be worse here).
Very nice thanks @lukemanley 
@lukemanley could you add a test like in the linked dask issue?
Backport PR #53672 on branch 2.0.x (BUG/CoW: is_range_indexer can't handle very large arrays)
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves
name: Copy / view semantics

Comments:
Merging as failures unrelated.
TST: Add test for duplcated columns and usecols GH#11823
State: closed

- [ ] closes #11823 (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Testing
description: pandas testing functions or related to the test suite

Comments:
Thanks @liang3zy22
BUG: series with complex nan
State: closed

- [x] Closes #53627
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests)
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit)
- [x] Added an entry in the latest `doc/source/whatsnew/v2.1.0.rst` file


Labels:
name: Complex
description: Complex Numbers
name: Constructors
description: Series/DataFrame/Index/pd.array Constructors

Comments:
@jbrockmendel suggested changes made, thanks for your review!

One thing to notice is the repr:
```python
>>> import pandas as pd
>>> pd.Series([complex("nan")])
0   NaN
dtype: complex128
>>> pd.Series([1, complex("nan"), 2])
0    1.0+0.0j
1   N000a000N
2    2.0+0.0j
dtype: complex128
```
I don't think this is as expected, but I suppose this should be a separate issue?
Thanks @Charlie-XIAO 

> I don't think this is as expected, but I suppose this should be a separate issue?

Yeah that appears to be a bug. Could you open a separate issue about that?
Sure, will do soon.
BUG: inconsistent behavior of DateOffset #47953
State: closed

- [x] closes #47953
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/v2.0.3.rst` file if fixing a bug or adding a new feature.


Labels:
name: Frequency
description: DateOffsets

Comments:
thanks for your PR - looks like some tests are failing, could you fix them up please?
```
FAILED pandas/tests/tseries/offsets/test_offsets.py::test_dateoffset_add_sub[offset_kwargs0-1970-01-01 00:00:00.000000001] - AssertionError
FAILED pandas/tests/tseries/offsets/test_offsets.py::test_dateoffset_add_sub[offset_kwargs1-1970-01-01 00:00:00.000000005] - AssertionError
FAILED pandas/tests/tseries/offsets/test_offsets.py::test_dateoffset_add_sub[offset_kwargs2-1969-12-31 23:59:59.999999999] - AssertionError
FAILED pandas/tests/tseries/offsets/test_offsets.py::test_dateoffset_add_sub_timestamp_with_nano - AssertionError
```
I am looking into it.
@MarcoGorelli the builds are passing now. 
@MarcoGorelli can you please help me understand the next steps for this PR?
I'll just leave this open a little to see if anyone else has comments/objections, and then will merge
Hi @MarcoGorelli thanks for approving the PR. Any updates on this? TiA!
right, let's ship this 🚀
REF: move values_for_json to EA
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: IO JSON
description: read_json, to_json, json_normalize
name: ExtensionArray
description: Extending pandas with custom dtypes or arrays.

Comments:
Added docstring.  Not confident enough in how this works to add tests yet
@WillAyd OK here?
I don't know that I see the vision for how this will actually work with JSON arguments (i.e. date formatting) but no objection to it either
Unless there are more comments, I'm planning to merge later this week.
Thanks @jbrockmendel
BUG: astype('category') on dataframe backed by non-writeable arrays raises ValueError
State: closed

- [x] closes #53658
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/v2.1.0.rst` file if fixing a bug or adding a new feature.


Labels:
name: Bug
name: Dtype Conversions
description: Unexpected or buggy dtype conversions
name: Categorical
description: Categorical Data Type

Comments:
Thanks @lukemanley
BUG/CoW: is_range_indexer can't handle very large arrays
State: closed

- [ ] closes #53616 (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

I don't have enough RAM to test that this actually works.

(But the test raises ``OverflowError`` before, and doesn't raise after).

Labels:
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves
name: Copy / view semantics

Comments:
thx
Thanks @lithomas1 
@lithomas1 thoughts about back porting this? This is potentially very annoying 
Yeah, for sure. I'll move the whatsnew in a bit then.
@meeseeksdev backport 2.0.x
Owee, I'm MrMeeseeks, Look at me.

There seem to be a conflict, please backport manually. Here are approximate instructions:

1. Checkout backport branch and update it.

```
git checkout 2.0.x
git pull
```

2. Cherry pick the first parent branch of the this PR on top of the older branch:
```
git cherry-pick -x -m1 905fe6b0b90f5de334abb1585e15d987935a592e
```

3. You will likely have some merge/cherry-pick conflict here, fix them and commit:

```
git commit -am "Backport PR #53672: BUG/CoW: is_range_indexer can't handle very large arrays"
```

4. Push to a named branch:

```
git push YOURFORK 2.0.x:auto-backport-of-pr-53672-on-2.0.x
```

5. Create a PR against branch 2.0.x, I would have named this PR:

> "Backport PR #53672 on branch 2.0.x (BUG/CoW: is_range_indexer can't handle very large arrays)"

And apply the correct labels and milestones.

Congratulations — you did some good work! Hopefully your backport PR will be tested by the continuous integration and merged soon!

Remember to remove the `Still Needs Manual Backport` label once the PR gets merged.

If these instructions are inaccurate, feel free to [suggest an improvement](https://github.com/MeeseeksBox/MeeseeksDev).
DEPR: downcast keyword
State: closed

- [x] closes #40988 (Replace xxxx with the GitHub issue number)
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Missing-data
description: np.nan, pd.NaT, pd.NA, dropna, isnull, interpolate
name: Dtype Conversions
description: Unexpected or buggy dtype conversions
name: Deprecate
description: Functionality to remove in pandas

Comments:
Thanks @jbrockmendel
CI: Attempt to fix wheel builds
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Build
description: Library building on various platforms
name: CI
description: Continuous Integration

Comments:
Since the repo is cloned into this job with the sdist, for debugging it may be worth checking that the key files (project.toml, setup.py, meson files) are synced between both
> Since the repo is cloned into this job with the sdist, for debugging it may be worth checking that the key files (project.toml, setup.py, meson files) are synced between both

Can we put this in as a band-aid for now, and see if it fixes the nightlies?

I don't think I have the energy/time to repro/root cause this.
(more details in the comment I pinged you on before.
TLDR: only fails on nightlies. succeeds when you re-run failed builds. has never failed on a manually run workflow or a PR workflow).
> Can we put this in as a band-aid for now, and see if it fixes the nightlies?

Yes most definitely
Thanks @lithomas1
TST: xfail `test_rolling_var_numerical_issues` on Mac
State: closed

- [x] related to #38921. *Does not close the issue*. Test is failing on M2 chip.
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit). *Getting the same warnings on main and this branch on my local machine, so I think it is good.*
- [x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions. *No new functionality.*
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature. *Change to testing only.*

---

Related to discussion starting from [this comment](https://github.com/pandas-dev/pandas/issues/38921#issuecomment-1586318796) on #38921. The test fails on with the Mac M2 chip, so this PR removes the exclusion of Mac from the xfail marker. Mac was previously excluded because the test passes on M1.

Labels:
name: Build
description: Library building on various platforms
name: Window
description: rolling, ewma, expanding
name: ARM
description: aarch64 architecture

Comments:
All checks passed. Now just waiting on review. Please let me know if any changes need to be made.
This test is not passing on my M1 Macbook Pro. I think we should skip it for all arm chip Mac for now. 
> This test is not passing on my M1 Macbook Pro. I think we should skip it for all arm chip Mac for now.

Thank you for checking this!
Thanks @Jython1415
ENH: EA.interpolate
State: closed

- [x] closes #50950 (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

Fixed some incorrect type annotations, de-duplicate some code paths.

Labels:
name: Missing-data
description: np.nan, pd.NaT, pd.NA, dropna, isnull, interpolate
name: ExtensionArray
description: Extending pandas with custom dtypes or arrays.

Comments:
> Looks good. Could use a whatsnew note in 2.1 that EA.interpolate was added

updated+green
Thanks @jbrockmendel
DEPR: downcasting in NDFrame.where, mask, clip
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

Related to #40988.

This doesn't have a good "do X to avoid the warning/get the future behavior".  Could be a use case for the "future" option from #53025

Alternative to #53108

Labels:
name: Deprecate
description: Functionality to remove in pandas

Comments:
@rhshadrach thoughts here?
+1 on the future behavior.

Agree this is a good candidate for a `future.*` option. The other option would be to add an argument to `Series.where` and `DataFrame.where` for the change in behavior. The benefit of the argument would be that it doesn't rely on global state and is more explicit. The downside is that users can't just adopt the future behavior everywhere in a single line of code. No clear winner in my mind - ~I'm good either way.~

Edit: I forgot this was impacting multiple methods. That makes me favor the option rather than argument quite a bit more.
@mroeschke @MarcoGorelli on board for implementing the future option here?
yeah sounds good to me!
> implementing the future option here?

is that something you want to do as part of this PR?
> is that something you want to do as part of this PR?

I was planning to, yes.  But if people were OK on doing this deprecation without a "do X instead" id also be happy do move forward as is
@mroeschke thoughts here?  im making a push to deprecate all our remaining silent downcasting and this is the hard one i think.  im OK either to do this without a "do X instead" or to implement a "future" option.
I think being more conservative with a future option would be better here since there seem to be no current way to work around this warning.
> I think being more conservative with a future option would be better here since there seem to be no current way to work around this warning.

updated with future option.  could plausibly re-use that in #54710, #54261

Thanks @jbrockmendel
PERF: Series.str.get_dummies for ArrowDtype(pa.string())
State: closed

- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/v2.1.0.rst` file if fixing a bug or adding a new feature.

```
import pandas as pd
import pyarrow as pa

data = ["a|b|c", "a|b", "a|c", "b|c", "a", "b", "c"] * 1000
ser = pd.Series(data, dtype=pd.ArrowDtype(pa.string()))

%timeit ser.str.get_dummies() 

# 549 ms ± 43.5 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)     -> main
# 8.67 ms ± 415 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)  -> PR
```


Labels:
name: Performance
description: Memory or execution speed performance
name: Strings
description: String extension data type and string data
name: Arrow
description: pyarrow functionality

Comments:
Nice optimizations as usual. Thanks @lukemanley