BrandonFors/pandas-github-issues huggingface dataset:
DEPR: allowing unknown array-likes in merge
State: closed

Not sure what the current status is w/r/t new deprecations.

Until we make this change, some of our annotations in this module are lies.

Labels:
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode
name: Deprecate
description: Functionality to remove in pandas

Comments:
Or we could just change it directly for 2.0.  This is a pretty weird corner case
Curious if there was a motivating factor from usage in another library or just self discovered?
Self-discovered

On Fri, Sep 9, 2022 at 10:24 AM Matthew Roeschke ***@***.***>
wrote:

> Curious if there was a motivating factor from usage in another library or
> just self discovered?
>
> —
> Reply to this email directly, view it on GitHub
> <https://github.com/pandas-dev/pandas/pull/48454#issuecomment-1242259472>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AB5UM6B7K6NNVYPHQPLPTF3V5NXFFANCNFSM6AAAAAAQHHFRLA>
> .
> You are receiving this because you authored the thread.Message ID:
> ***@***.***>
>

> Or we could just change it directly for 2.0. This is a pretty weird corner case

Up to you how strongly you feel about avoiding this in 2.0. Personally, I think we should use the 2.0 hard break for functionality that would be a PITA to deprecate normally.
> Up to you how strongly you feel about avoiding this in 2.0.

Not the end of the world to wait until 3.0
Cool, then a whatsnew note in 1.6 would probably be good. But hopefully we can do the 1.x deprecation cleaning spree before adding deprecations in 2.0.
This pull request is stale because it has been open for thirty days with no activity. Please [update](https://pandas.pydata.org/pandas-docs/stable/development/contributing.html#updating-your-pull-request) and respond to this comment if you're still interested in working on this.
This now raises in Int64Factorizer when I pass in a dask.array (in the test this currently adds).  In 1.5.3 passing a dask array worked for `left_on` and `right_on`, but not `on`; I'm guessing @phofl's Factorizer work changed this.

My initial motivation here was to be able to accurately annotate the merge code, which ATM uses is_array_like but doesn't actually require "ndarray | EA | Series | Index".  Seeing as how the API has already been changed, any objection to making that change explicit+documented by checking for the specific types?
I'd be ok with that.

Fixing wouldn't be too hard I guess (we called ensure_...) before, but not sure if it's worth it
Closing, will address in a few months once we confirm no one complains about the changed behavior.
BUG: Index.equals raising with index of tuples that contain NA
State: closed

- [ ] closes #xxxx (Replace xxxx with the Github issue number)
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

This came up when working on MultiIndex.union with ea. Gets converted into a Index of tuples and then raising the error.  The added check is a last resort before raising to avoid impacting performance negatively.

Labels:
name: Missing-data
description: np.nan, pd.NaT, pd.NA, dropna, isnull, interpolate
name: NA - MaskedArrays
description: Related to pd.NA and nullable extension arrays

Comments:
Ok to merge on greenish?
Thanks @phofl
CI: Pin ipython version
State: closed

- [x] closes #48431 (Replace xxxx with the Github issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

Should fix the docbuild till they release the fix


Labels:
name: Docs
name: CI
description: Continuous Integration

Comments:
I checked this in more detail, I don't think we need the okexcept at all, this builds ok locally

<img width="784" alt="image" src="https://user-images.githubusercontent.com/61934744/188961980-3fdc4e74-0a94-4ca9-a0b7-65f64a8766bb.png">

Perfect thanks @phofl 
Ok second thought,
Maybe should backport to 1.4.x too just in case we need another release?
Sure, a manual backport should be fine just in case.
BUG: Fix pyarrow groupby tests
State: closed

orthogonal to #48428

Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Groupby
name: Arrow
description: pyarrow functionality

Comments:
@mroeschke no legitimate reason to rush this, but there are more fixes in test_arrow.py im eager to get to after this and #48428
Owee, I'm MrMeeseeks, Look at me.

There seem to be a conflict, please backport manually. Here are approximate instructions:

1. Checkout backport branch and update it.

```
git checkout 1.5.x
git pull
```

2. Cherry pick the first parent branch of the this PR on top of the older branch:
```
git cherry-pick -x -m1 4e72340d9161c0f89c2e9d96253f75d7b7991cae
```

3. You will likely have some merge/cherry-pick conflict here, fix them and commit:

```
git commit -am 'Backport PR #48443: BUG: Fix pyarrow groupby tests'
```

4. Push to a named branch:

```
git push YOURFORK 1.5.x:auto-backport-of-pr-48443-on-1.5.x
```

5. Create a PR against branch 1.5.x, I would have named this PR:

> "Backport PR #48443 on branch 1.5.x (BUG: Fix pyarrow groupby tests)"

And apply the correct labels and milestones.

Congratulations — you did some good work! Hopefully your backport PR will be tested by the continuous integration and merged soon!

Remember to remove the `Still Needs Manual Backport` label once the PR gets merged.

If these instructions are inaccurate, feel free to [suggest an improvement](https://github.com/MeeseeksBox/MeeseeksDev).
ENH: Add index param to df.to_dict
State: closed

- [x] closes #46398
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: DataFrame
description: DataFrame data structure

Comments:
> Looks pretty good. Could you resolve the merge conflict?

Resolved.
Thanks @yuanx749
BUG/TST: fix a bunch of arraymanager+pyarrow tests
State: closed

- [ ] closes #xxxx (Replace xxxx with the Github issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Arrow
description: pyarrow functionality

Comments:
Owee, I'm MrMeeseeks, Look at me.

There seem to be a conflict, please backport manually. Here are approximate instructions:

1. Checkout backport branch and update it.

```
git checkout 1.5.x
git pull
```

2. Cherry pick the first parent branch of the this PR on top of the older branch:
```
git cherry-pick -x -m1 12dce19a74b7cd5badad0f61ca079b873c1b6089
```

3. You will likely have some merge/cherry-pick conflict here, fix them and commit:

```
git commit -am 'Backport PR #48428: BUG/TST: fix a bunch of arraymanager+pyarrow tests'
```

4. Push to a named branch:

```
git push YOURFORK 1.5.x:auto-backport-of-pr-48428-on-1.5.x
```

5. Create a PR against branch 1.5.x, I would have named this PR:

> "Backport PR #48428 on branch 1.5.x (BUG/TST: fix a bunch of arraymanager+pyarrow tests)"

And apply the correct labels and milestones.

Congratulations — you did some good work! Hopefully your backport PR will be tested by the continuous integration and merged soon!

Remember to remove the `Still Needs Manual Backport` label once the PR gets merged.

If these instructions are inaccurate, feel free to [suggest an improvement](https://github.com/MeeseeksBox/MeeseeksDev).
Backport PR #48264 on branch 1.5.x (BUG: ArrowExtensionArray._from_* accepts pyarrow arrays)
State: closed

Backport PR #48264: BUG: ArrowExtensionArray._from_* accepts pyarrow arrays

Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Constructors
description: Series/DataFrame/Index/pd.array Constructors
name: Arrow
description: pyarrow functionality

Comments:
cc @mroeschke  this is causing failures here
Backport PR #48398 on branch 1.5.x (WARN: Avoid FutureWarnings in tests)
State: closed

Backport PR #48398: WARN: Avoid FutureWarnings in tests

Labels:
name: Warnings
description: Warnings that appear or should be added to pandas

Comments:
I can't reproduce this locally, and the error in the CI doesn't seem to make a lot of sense. @phofl any idea what can be wrong? Doesn't seem related to your changes, but just in case.
This was caused by the ipython upgrade and fixed yesterday (already on 1.5.x)
Great, thanks for the info. I updated this with the latest 1.5.x changes, let's see if this passes the CI now.
Please don't merge yet
We probably don't want to merge the update change, but another change went missing here that we should back in. Will push this after #48468 is merged
cc @mroeschke I'd like to merge this part of the initial pr into 1.5.x, already remove the other part
BUG: ensure to return writable buffer in __dataframe__ interchange for categorical column
State: closed

- [x] closes https://github.com/pandas-dev/pandas/issues/48393
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature


Labels:
name: Interchange
description: Dataframe Interchange Protocol

Comments:
LGTM. Not sure if the stalled CI without any test output is related to this PR or not.
Thanks @jorisvandenbossche (CI stalling unrelated)
Revert set_index inplace and copy keyword changes
State: closed

See the discussion in https://github.com/pandas-dev/pandas/issues/48141

Reverts https://github.com/pandas-dev/pandas/pull/48043 and https://github.com/pandas-dev/pandas/pull/48115 for now.

Labels:
name: Deprecate
description: Functionality to remove in pandas
name: Copy / view semantics

Comments:
Owee, I'm MrMeeseeks, Look at me.

There seem to be a conflict, please backport manually. Here are approximate instructions:

1. Checkout backport branch and update it.

```
git checkout 1.5.x
git pull
```

2. Cherry pick the first parent branch of the this PR on top of the older branch:
```
git cherry-pick -x -m1 a551f1b7b0aeace5d3c6a1c629dce5442fcee85b
```

3. You will likely have some merge/cherry-pick conflict here, fix them and commit:

```
git commit -am 'Backport PR #48417: Revert set_index inplace and copy keyword changes'
```

4. Push to a named branch:

```
git push YOURFORK 1.5.x:auto-backport-of-pr-48417-on-1.5.x
```

5. Create a PR against branch 1.5.x, I would have named this PR:

> "Backport PR #48417 on branch 1.5.x (Revert set_index inplace and copy keyword changes)"

And apply the correct labels and milestones.

Congratulations — you did some good work! Hopefully your backport PR will be tested by the continuous integration and merged soon!

Remember to remove the `Still Needs Manual Backport` label once the PR gets merged.

If these instructions are inaccurate, feel free to [suggest an improvement](https://github.com/MeeseeksBox/MeeseeksDev).
                
Thanks @jorisvandenbossche
BUG: safe_sort losing MultiIndex dtypes
State: closed

- [ ] closes #xxxx (Replace xxxx with the Github issue number)
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

I don't think that this is user visible. This is a precursor for improving ea support in MultiIndex operations

Labels:
name: Dtype Conversions
description: Unexpected or buggy dtype conversions
name: MultiIndex

Comments:
Thanks @phofl 
@meeseeksdev backport 1.5.x
REGR: get_loc for ExtensionEngine not returning bool indexer for na
State: closed

- [ ] closes #xxxx (Replace xxxx with the Github issue number)
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

Stumbled across this when working on something else.

ExtensionEngine was new in 1.5, so no whatsnew. But should backport to 1.5


Labels:
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves

Comments:
One of the failures was related, int32 instead of int64 dtype. Fixed this since it is not really relevant here
DOC: Add missing functions to docs
State: closed

- [X] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [X] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).


Labels:

Comments:
Looks like these were left out since that are in `api`
Sorry, looking in wrong mod
Docs: Added 'cross' option to one part of DataFrame.join() docstring where it was missing
State: closed

- [ ] closes #xxxx (Replace xxxx with the Github issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Docs
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode

Comments:
thx @maharris77
PERF: MultiIndex.argsort / MultiIndex.sort_values
State: closed

- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/v1.6.0.rst` file if fixing a bug or adding a new feature.

Perf improvement in `MultiIndex.argsort` and `MultiIndex.sort_values` for the vast majority of cases (where `MultiIndex.argsort` is called without additional passthrough numpy parameters).

Also, an xfail was removed because np.lexsort does not complain about complex dtype arrays whereas np.argsort will complain about an object array of tuples containing complex and non-complex dtypes. 

ASV added:

```
       before           after         ratio
     [fa211d47]       [b2fea756]
     <main>           <perf-multiindex-argsort>
-      81.0±0.9ms       27.9±0.7ms     0.34  multiindex_object.SortValues.time_sort_values('Int64')
-        96.9±2ms       2.00±0.2ms     0.02  multiindex_object.SortValues.time_sort_values('int64')
```





Labels:
name: Performance
description: Memory or execution speed performance
name: MultiIndex

Comments:
> Could you add a Benchmark for Nullable dtypes too?

Added a benchmark for nullable Int64 and updated the timings in the summary. However, the nullable index used in the asv does not contain any pd.NA values as numpy will raise if it encounters pd.NA. That is true for numpy.argsort (with a list of tuples) and numpy.lexsort so this isn't new behavior. Can handling pd.NA be an enhancement/bug fix for a separate PR?
Thanks @lukemanley
ENH: Add use_nullable_dtypes in csv internals
State: closed

- [x] xref #36712 (Replace xxxx with the Github issue number)
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

This adds the required casting logic for extension array dtypes. Want to get this done as a first step before adding a public option to read_csv. The unit tests can be used as a guideline for the required behaviour.

This always casts to ea dtypes, even when no missing values are present. From a users perspective this makes the most sense imo. You don't want to do a follow up operation and end up with float in a pipeline, when you set nullable dtypes (reindexing, enlargement in indexing ops, ...). That said, if there are concerns about this, we could also do something like

``use_nullable_dtypes=False | always | when necessary``

cc @jorisvandenbossche 


Labels:
name: IO CSV
description: read_csv, to_csv
name: NA - MaskedArrays
description: Related to pd.NA and nullable extension arrays

Comments:
Thanks @phofl
CLN: Refactor groupby._make_wrapper
State: closed

- [x] closes #48028 (Replace xxxx with the Github issue number)
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

Currently keeping the docs the same, plan to make groupby-specific docstrings in the future. Note that most of the SeriesGroupBy docs don't make it into the documentation, I've opened #48399. The only change from a user-perspective here should be that Jupyter / IDEs can now get the correct signature (instead of `*args, **kwargs`).

![image](https://user-images.githubusercontent.com/45562402/188480390-ec9f26c3-ec3d-456d-961b-1843713b8679.png)

![image](https://user-images.githubusercontent.com/45562402/188480951-4e679091-0c7a-402b-8c9e-7e8293f793d1.png)



Labels:
name: Groupby
name: Clean

Comments:
Awesome! A lot more straightforward, thanks @rhshadrach
WARN: Avoid FutureWarnings in tests
State: closed

- [ ] closes #xxxx (Replace xxxx with the Github issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

Would like to backport, otherwise we have to deal with this on the 1.5.x branch

Labels:
name: Warnings
description: Warnings that appear or should be added to pandas

Comments:
Sure fine to backport. Thanks @phofl
WARN: Remove false positive warning for iloc inplaceness
State: closed

- [ ] closes #xxxx (Replace xxxx with the Github issue number)
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

This won't be able to set inplace, so should not warn


Labels:
name: Warnings
description: Warnings that appear or should be added to pandas
name: Copy / view semantics

Comments:
We are trying to cast [nan, 1.0] to integer, which raises the RuntimeWarning, but I don't think that this should leak to our users.
@jbrockmendel ok to merge?
We should try to get this into 1.5 too to avoid confusione
some nitpicks, otherwise LGTM
Thanks @phofl