BrandonFors/pandas-github-issues huggingface dataset:
PERF: join/merge on subset of MultiIndex
State: closed

- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/v1.6.0.rst` file if fixing a bug or adding a new feature.

Existing code passes a range to algos.take_nd. Passing an ndarray is faster. 

```
       before           after         ratio
     [a712c501]       [00034d83]
     <main>           <multiindex-join-subset>
-    45.8±0.7ms       26.0±0.8ms      0.57  join_merge.JoinMultiindexSubset.time_join_multiindex_subset
```

Labels:
name: Performance
description: Memory or execution speed performance
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode
name: MultiIndex

Comments:
thx @lukemanley
PERF: DatetimeIndex constructor from list
State: closed

- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/v1.6.0.rst` file if fixing a bug or adding a new feature.


Improvement comes from avoiding np.ndim on a large list of objects. 

ASVs added:

```
     [ac648eea]       [5e8af646]
     <main>           <perf-ensure-arraylike-for-datetimelike>
-      41.7±0.6ms         30.1±1ms     0.72  ctors.DatetimeIndexConstructor.time_from_list_of_str
-        239±20ms          110±1ms     0.46  ctors.DatetimeIndexConstructor.time_from_list_of_datetimes
-         243±3ms        111±0.6ms     0.46  ctors.DatetimeIndexConstructor.time_from_list_of_timestamps
-         242±5ms          108±1ms     0.45  ctors.DatetimeIndexConstructor.time_from_list_of_dates
```

Labels:
name: Datetime
description: Datetime data dtype
name: Performance
description: Memory or execution speed performance

Comments:
A general note: I think our rule of thumb regarding benchmarks is: Around 20ms. 
Sorry, I should have remembered. I just updated the asv to reduce the time:

```
       before           after         ratio
     [a712c501]       [a572dc1e]
     <main>           <perf-ensure-arraylike-for-datetimelike>
-     7.47±0.07ms      5.42±0.09ms     0.72  ctors.DatetimeIndexConstructor.time_from_list_of_str
-      44.0±0.4ms       20.5±0.4ms     0.47  ctors.DatetimeIndexConstructor.time_from_list_of_datetimes
-      43.6±0.2ms       19.9±0.2ms     0.46  ctors.DatetimeIndexConstructor.time_from_list_of_timestamps
-      42.9±0.6ms       19.4±0.2ms     0.45  ctors.DatetimeIndexConstructor.time_from_list_of_dates
```
Thx, no worries 
thx @lukemanley
REGR: assert_index_equal raising with non matching pd.NA
State: closed

- [ ] closes #xxxx (Replace xxxx with the Github issue number)
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

I'd like to backport this (not necessarily to 1.5.0, 1.5.1 would be fine too). EAs in Index objects are new in 1.4, so this can be considered a regression.

Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: NA - MaskedArrays
description: Related to pd.NA and nullable extension arrays

Comments:
Thanks @phofl
BUG: MultiIndex.symmetric_difference not keeping ea dtype
State: closed

- [ ] closes #xxxx (Replace xxxx with the Github issue number)
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


```
       before           after         ratio
     [b5632fb3]       [811faf6b]
     <deprecation_notes~1>       <sym_diff>
-     16.0±0.09ms      11.0±0.08ms     0.69  multiindex_object.SetOperations.time_operation('monotonic', 'string', 'symmetric_difference')
-      16.4±0.4ms       10.9±0.1ms     0.66  multiindex_object.SetOperations.time_operation('non_monotonic', 'string', 'symmetric_difference')
-      15.2±0.4ms       9.78±0.2ms     0.64  multiindex_object.SetOperations.time_operation('non_monotonic', 'int', 'symmetric_difference')
-      15.8±0.1ms       10.1±0.2ms     0.64  multiindex_object.SetOperations.time_operation('non_monotonic', 'datetime', 'symmetric_difference')
-      15.1±0.1ms      9.40±0.08ms     0.62  multiindex_object.SetOperations.time_operation('monotonic', 'int', 'symmetric_difference')
-      15.8±0.1ms      9.79±0.04ms     0.62  multiindex_object.SetOperations.time_operation('monotonic', 'datetime', 'symmetric_difference')
```

Labels:
name: MultiIndex
name: NA - MaskedArrays
description: Related to pd.NA and nullable extension arrays

Comments:
Thanks @phofl
BUG: MultiIndex.difference not keeping ea dtype
State: closed

- [ ] closes #xxxx (Replace xxxx with the Github issue number)
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


```
       before           after         ratio
     [b5632fb3]       [c8b6d846]
     <deprecation_notes~1>       <diff>    
-     5.67±0.08ms       4.73±0.1ms     0.83  multiindex_object.Difference.time_difference('string')
-     4.65±0.01ms      3.71±0.03ms     0.80  multiindex_object.Difference.time_difference('int')
-     4.78±0.02ms      3.75±0.01ms     0.78  multiindex_object.Difference.time_difference('ea_int')
-     6.35±0.02ms      4.85±0.02ms     0.76  multiindex_object.Difference.time_difference('datetime')
```

Labels:
name: MultiIndex
name: NA - MaskedArrays
description: Related to pd.NA and nullable extension arrays

Comments:
Thanks @phofl
ENH: MultiIndex.intersection now keeping EA dtypes
State: closed

- [ ] closes #xxxx (Replace xxxx with the Github issue number)
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

Nice performance boost

```
       before           after         ratio
     [b5632fb3]       [ed481f4f]
     <intersection~4>       <intersection>
-      8.81±0.2ms      7.49±0.08ms     0.85  index_object.SetOperations.time_operation('non_monotonic', 'strings', 'symmetric_difference')
-      16.5±0.2ms      12.2±0.05ms     0.74  index_object.SetOperations.time_operation('monotonic', 'date_string', 'intersection')
-      49.7±0.7ms       20.7±0.3ms     0.42  multiindex_object.SetOperations.time_operation('monotonic', 'ea_int', 'intersection')
-      49.9±0.3ms       19.9±0.3ms     0.40  multiindex_object.SetOperations.time_operation('monotonic', 'int', 'intersection')
-         119±1ms       33.2±0.3ms     0.28  multiindex_object.SetOperations.time_operation('monotonic', 'datetime', 'intersection')
-      31.7±0.1ms      4.98±0.05ms     0.16  multiindex_object.SetOperations.time_operation('non_monotonic', 'string', 'intersection')
-      31.8±0.4ms       4.83±0.2ms     0.15  multiindex_object.SetOperations.time_operation('monotonic', 'string', 'intersection')
-      35.5±0.2ms      4.62±0.01ms     0.13  multiindex_object.SetOperations.time_operation('non_monotonic', 'ea_int', 'intersection')
-      34.6±0.4ms       3.35±0.1ms     0.10  multiindex_object.SetOperations.time_operation('non_monotonic', 'int', 'intersection')
-      73.4±0.8ms       3.42±0.1ms     0.05  multiindex_object.SetOperations.time_operation('non_monotonic', 'datetime', 'intersection')
```

Labels:
name: MultiIndex
name: NA - MaskedArrays
description: Related to pd.NA and nullable extension arrays

Comments:
Thanks @phofl
PERF: Avoid fragmentation of DataFrame in read_sas
State: closed

- [x] closes #48595 (Replace xxxx with the Github issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

Not sure how to test this, the op file is 13mb which is way to large for our repository.

Labels:
name: Performance
description: Memory or execution speed performance
name: IO SAS
description: SAS: read_sas

Comments:
I know we want to limit network calls, for testing, how about reading OP's URL that points to a CDC .xpt file. Currently, `read_sas` can parse online files:

```python
pd.read_sas("https://wwwn.cdc.gov/Nchs/Nhanes/2017-2018/DR1TOT_J.XPT")
```

There is also a pytest decorator check in case link changes to use before test. Given the date and government site, link looks reliable over the years:

```python
@pytest.mark.network
@pytest.mark.slow
@tm.network(
    url="https://wwwn.cdc.gov/Nchs/Nhanes/2017-2018/DR1TOT_J.XPT",
    check_before_test=True,
)
```
This runs forever (25secs), since we don't have slow builds anymore, I don't think this is worth it
Thanks @phofl
CI: Fix matplolib release issues
State: closed

- [ ] xref #48602 (Replace xxxx with the Github issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

I am not really familiar with matplotlib, so I've xfailed where the fix was not clear to me

Also gets rid of deprecation warnings


Labels:
name: Visualization
description: plotting
name: CI
description: Continuous Integration

Comments:
cc @datapythonista not sure if this is relevant for the release
Thx, I am merging for now to get ci back to green. Please ping if there is something we could improve here.
DOC: Fix version switcher in old docs
State: closed

- [X] xref #48285

Fixing the version switcher in old versions of the documentation.

I'm also removing here the duplicated 1.4 link in the version switcher.


Labels:
name: Docs

Comments:
I'm adding here `1.5` to the version switcher dropdown. Docs for 1.5 are already published, I just need to change the link to the stable docs once the pypi and conda-forge packages are published. But I guess we can merge this even before, since the dropdown is currently broken, and this should make everything work, except for the 1.5 option that will go to the 1.4 for few more hours.
@phofl do you mind having a quick look at this please?
Thx @datapythonista 

1.6 is fine as milestone?
> 1.6 is fine as milestone?

I think for PRs only affecting the website we don't need to backport anything. Is the milestone useful for anything else other than backporting?

Looks like this PR didn't work as expected. :( Luckily I understand the problem now.

In the new version, the `url` in the json is being used, but in the old versions, it's the `version` field which is relevant. I think we set up things incorrectly at the beginning, that `version` field had to be something like `1.4.4`, and use the `url_template` to provide how to build the whole url with it. But instead, we used `pandas-docs/version/1.4.4` as the `version`: https://github.com/pandas-dev/pandas/pull/45370/files

I can restore the `pandas-docs/version/1.4.4` values in the JSON, and the switcher will work again, except that `version` is used to match to the current version and know in which version of the docs we're in (not quite sure this works). I'll do that for now, and we can discuss later if we want to have the correct `version` values from now on, which can be done if we add a new correct json file for the new versions of the docs, and we keep the old one until the broken versions of the docs are removed from the switcher.

Funny enough, this change in behavior made that even the switcher in the pydata theme page is broken. Try to change the version in their old docs: https://pydata-sphinx-theme.readthedocs.io/en/v0.8.0/user_guide/configuring.html
DOC: Add deprecation infos to deprecated functions
State: closed

- [ ] closes #xxxx (Replace xxxx with the Github issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

These were deprecated in 1.4


Labels:
name: Docs
name: Deprecate
description: Functionality to remove in pandas

Comments:
Thanks @phofl
DOC: include defaults into docs for true_values and false_values
State: closed

- [X] closes #48487 

## Motivation
This PR touches only one docstring, so no other checks are necessary. As outlined in #48487, when we call `pd.read_csv`, we eventually go to `_libs/ops.pyx`, where there are [defaults](https://github.com/pandas-dev/pandas/blob/ca60aab7340d9989d9428e11a51467658190bb6b/pandas/_libs/ops.pyx#L257) for `true_values` and `false_values`:
```python
    # the defaults
    true_vals = {'True', 'TRUE', 'true'}
    false_vals = {'False', 'FALSE', 'false'}

    if true_values is not None:
        true_vals = true_vals | set(true_values)

    if false_values is not None:
        false_vals = false_vals | set(false_values)
```
In addition, there appears to be a `str.lower()` or `str.upper()` call somewhere, although I can't find it, because `trUe` is treated as true-like too. We should outline this in the docstring, so that it's not a surprise to our users.

## Definition of done
- [X] mention the default values for `true_values` and `false_values` in the docstring for `pd.read_csv`

Labels:
name: Docs
name: IO CSV
description: read_csv, to_csv

Comments:
I don't think a whatsnew entry or anything else is needed for a doc-only PR, but please tell me if I've missed anything.
There are a bunch of errors, but it doesn't look like they have anything to do with the docstring change, and they seem to be cropping up over all the current PRs.
Thanks @AlexKirko
WEB: Fix version switcher
State: closed

- [ ] closes #xxxx (Replace xxxx with the Github issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.
cc @datapythonista @mroeschke 

The other prs broke the version switcher. It is looking into our html static path, not the website as root path, hence it can not finde the versions.json. We have to give the full link here. Same issue as the logo


Labels:
name: Docs
name: Web
description: pandas website

Comments:
I see. It seems like an obvious bug in the switcher code, we're providing the right urls, and for whatever reason it's dropping half of it (feels like it's taking only the last component of the url). For what I see, this doesn't happen in the latest version of the docs, where I think everything works as expected. So, I assume there was a bug that has been fixed.

If your changes here fix the problem, happy to get this merged. I can't really tell why the changes here prevent the bug. Seems like the changes here only affect the current docs, and the old docs (the broken ones) will still look for the `json_url` used in them when they were built.

Not sure if you already tried the changes here, but if you didn't, I think the path to `versions.json` can still be relative, but the `url` values in `versions.json` may not to be absolute to work in older versions of pydata-sphinx-theme.
Haven't really got the version switcher to work locally. So if you think this is something else, then we should not merge this.

I read the docs specifically about this and they say that its in the static path 

https://pydata-sphinx-theme.readthedocs.io/en/stable/user_guide/version-dropdown.html#configure-switcher-json-url

The alternative sounded to me as you would have to provide the complete url, but not sure.

True, I didn't realize that it's very difficult to test this before merging, as it only affects old docs.

I don't think this will work. Changes to `conf.py` will only affect the last version of the documentation, which is working fine. I think the change needs to be in the `versions.json`, which is what the old versions of the docs fetch, and the only reasonable thing we can change in the old docs. I opened #48600 which I think should fix the problem.
Ah got it, thx. This makes sense to me.

closing this one
CLN: MultiIndex.drop_duplicates
State: closed

I think there is no reason why `MultiIndex` overwrites `drop_duplicates` from `Index`:

- Both implementations do exactly the same
- Both implementations have exactly the same type annotations
- `MultiIndex.drop_duplicates` does not have a doc-string

There are a few more methods (`set_names`, `rename`) but in those cases, the annotations might actually be different.

Labels:
name: MultiIndex
name: Clean

Comments:
Thanks @twoertwein
BUG: MultiIndex.union losing dtype when right has duplicates
State: closed

- [ ] closes #xxxx (Replace xxxx with the Github issue number)
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: MultiIndex
name: NA - MaskedArrays
description: Related to pd.NA and nullable extension arrays

Comments:
This was fixed by another pr
TYP: reshape.merge
State: closed

- [ ] closes #xxxx (Replace xxxx with the Github issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode
name: Typing
description: type annotations, mypy/pyright type checking

Comments:
Thanks @jbrockmendel
Fix `series.str.startswith(tuple)`
State: closed

- [x] closes #48575 and #38281
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

Should I add a note in `doc/source/whatsnew/v1.5.0.rst`?

Labels:
name: Strings
description: String extension data type and string data

Comments:
> Could you also add a tuple input example in the `Examples` section of the docstrings?

Certainly.
Thanks for the quick fix @janosh! 
@mroeschke You may want to close #38281 manually. I forgot to put a 'closes' in front of its PR number above.
CLN: Simplify factorize
State: closed

- [ ] closes #xxxx (Replace xxxx with the Github issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

#48539 now allows the simplification from #48477 as it will no longer change behavior.

Labels:
name: Algos
description: Non-arithmetic algos: value_counts, factorize, sorting, isin, clip, shift, diff
name: Clean

Comments:
Thanks @rhshadrach