BrandonFors/pandas-github-issues huggingface dataset:
REGR: MultiIndex.join not resorting levels of new index
State: closed

- [x] closes #53093 (Replace xxxx with the GitHub issue number)
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

Not resorting causes all kinds of problems when indexing

Labels:
name: Regression
description: Functionality that used to work in a prior pandas version
name: MultiIndex

Comments:
Cleaned up, we should aim at fixing this properly for 2.1, because you can still end up like this if you instantiate the MultiIndex directly
Owee, I'm MrMeeseeks, Look at me.

There seem to be a conflict, please backport manually. Here are approximate instructions:

1. Checkout backport branch and update it.

```
git checkout 2.0.x
git pull
```

2. Cherry pick the first parent branch of the this PR on top of the older branch:
```
git cherry-pick -x -m1 ddff84a5ddbf55b5f407976be359720804cd6981
```

3. You will likely have some merge/cherry-pick conflict here, fix them and commit:

```
git commit -am 'Backport PR #53102: REGR: MultiIndex.join not resorting levels of new index'
```

4. Push to a named branch:

```
git push YOURFORK 2.0.x:auto-backport-of-pr-53102-on-2.0.x
```

5. Create a PR against branch 2.0.x, I would have named this PR:

> "Backport PR #53102 on branch 2.0.x (REGR: MultiIndex.join not resorting levels of new index)"

And apply the correct labels and milestones.

Congratulations — you did some good work! Hopefully your backport PR will be tested by the continuous integration and merged soon!

Remember to remove the `Still Needs Manual Backport` label once the PR gets merged.

If these instructions are inaccurate, feel free to [suggest an improvement](https://github.com/MeeseeksBox/MeeseeksDev).
                
@meeseeksdev backport 2.0.x
TYP: added explanation of the join methods within df.align
State: closed

- closes #53092 

Added type annotations of the join methods within df.align - ([here](https://github.com/Antony-evm/pandas-aevmorfop/blob/0.21.x/pandas/core/generic.py))

from  join : {{'outer', 'inner', 'left', 'right'}}, default 'outer'
           
To
 join : {{'outer', 'inner', 'left', 'right'}}, default 'outer'
 Type of alignment to be performed.

            left: use only keys from left frame, preserve key order.
            right: use only keys from right frame, preserve key order.
            outer: use union of keys from both frames, sort keys lexicographically.
            inner: use intersection of keys from both frames, preserve the order of the left keys.




Labels:
name: Docs

Comments:
/preview
Website preview of this PR available at: https://pandas.pydata.org/preview/53101/
Im getting this error with the proposed changes and I fail to see how this could be caused by the documentation:
_error    libmamba Bad conversion of configurable 'remote_connect_timeout_secs' from source '~/.condarc' : yaml-cpp: error at line 14, column 30: bad conversion_

Any ideas? Is it a common issue?

Looks like theres a doc issue: 
```
2023-05-08T16:12:06.8973871Z /home/runner/work/pandas/pandas/pandas/core/generic.py:docstring of pandas.core.generic.NDFrame.align:13: WARNING: Bullet list ends without a blank line; unexpected unindent.
2023-05-08T16:12:06.8974978Z /home/runner/work/pandas/pandas/pandas/core/generic.py:docstring of pandas.core.generic.NDFrame.align:13: WARNING: Bullet list ends without a blank line; unexpected unindent.
```

Also no need to worry about that warning
> Looks like theres a doc issue:
> 
> ```
> 2023-05-08T16:12:06.8973871Z /home/runner/work/pandas/pandas/pandas/core/generic.py:docstring of pandas.core.generic.NDFrame.align:13: WARNING: Bullet list ends without a blank line; unexpected unindent.
> 2023-05-08T16:12:06.8974978Z /home/runner/work/pandas/pandas/pandas/core/generic.py:docstring of pandas.core.generic.NDFrame.align:13: WARNING: Bullet list ends without a blank line; unexpected unindent.
> ```
> 
> Also no need to worry about that warning

Thank you for this - the last line of the list should have been indented apparently. Fixed it and it should be ready to close now. If you could point me to where to look in the logs next time, I'd appreciate it as I spent some time trying to find the error and couldn't find the above for the life of me.

PS: Im quite new to git/github. Apologies for closing/reopening - thought this action would merge the pull request to the main branch. 
Thanks @Antony-evm
Fix bug #37782
State: closed

- [X] closes #37782 
- [X] [Tests added and passed]
- [X] All [code checks passed]
- [X] Added an entry in the latest `doc/source/whatsnew/v2.1.0.rst` file.



Labels:
name: IO JSON
description: read_json, to_json, json_normalize

Comments:
Thanks @felipemaion
Function signature53071
State: closed

- [x] closes #53071  (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

Previously, the skipna and numeric_only parameters could only be passed via the *args or **kwargs parameters, which could make the function signatures less clear and confusing for users. With these changes, the new parameters are included in the function signature and in the calls to the cumprod and cumsum methods inside the functions.

Labels:

Comments:
> You'll have to update the doctoring as well to reflect the new arguments

What about something like this?

```python
@final
@Substitution(name="groupby")
@Appender(_common_see_also)
def cumprod(
    self,
    axis: Axis | lib.NoDefault = lib.no_default,
    skipna: bool = True,
    numeric_only: bool = False,
    *args,
    **kwargs,
) -> NDFrameT:
    """
    Cumulative product for each group.

    Parameters
    ----------
    axis : int, default 0
        The axis to apply the cumulative product along.
    skipna : bool, default True
        Exclude NA/null values when computing the result.
    numeric_only : bool, default False
        Include only float, int, boolean columns. 
    *args, **kwargs
        Additional arguments to be passed to the cumprod method of each group.

    Returns
    -------
    Series or DataFrame
        The cumulative product of each group.
    """
```

Please remove the None part in the numeric_only doctoring
Thanks for the pull request, but it appears to have gone stale. If interested in continuing, please merge in the main branch, address any review comments and/or failing tests, and we can reopen.
ENH: EA._from_scalars
State: closed

- [x] closes #33254 
- [x] closes #31108 (someone double-check me on this)

xref #33254 _from_sequence is generally not very strict.  This implements _from_scalars for a few EAs as a POC to get rid of an ugly kluge in `maybe_cast_pointwise_result`.  The changed behavior in test_resample_categorical_data_with_timedeltaindex looks like a clear improvement to me.

cc @jorisvandenbossche 

Things to work out before this leaves the "WIP" zone:
- [x] implement/document it on the base class, none of this "hasattr" shenanigans
- [x] decide whether dtype should be required.  Not specifying dtype has different implications for different EA subclasses, which is not ideal.  I think part of the motivation is to be able to preserve pyarrow/masked "flavors" when doing pointwise ops.
- [x] see if this fixes #49163, #52411 <b>update</b> Nope!

Labels:
name: Enhancement
name: ExtensionArray
description: Extending pandas with custom dtypes or arrays.

Comments:
This pull request is stale because it has been open for thirty days with no activity. Please [update](https://pandas.pydata.org/pandas-docs/stable/development/contributing.html#updating-your-pull-request) and respond to this comment if you're still interested in working on this.
@jbrockmendel , I have done some investigation based on your PR. I have found one issue:
For pyarrow type, preserve_dtype would be True in agg_series. Then maybe_cast_pointwise_result should be called.
The maybe_cast_pointwise_result function have one default parameter, same_type to be True. So the result would be cast to the original pyarrow types. It caused issue 53030. I have changed the same_type parameter to be False by default. The issue 53030 is fixed, but some pytest test cases failed. 
So in what situation should cast the result to original pyarrow types? And in what situation should cast the result to the corresponding pyarrow types?

@jorisvandenbossche i think the last round of comments and OP issues have been addressed.  any other thoughts?
If there are no further comments, I'm planning to merge this at the end of the week.
PERF: Improve performance when accessing GroupBy.groups
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


```
df = DataFrame(np.random.randint(1, 100, (5_000_000, 5)))
df.groupby([0, 1]).groups

# main
%timeit df.groupby([0, 1]).groups
1.68 s ± 7.52 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
# pr
%timeit df.groupby([0, 1]).groups
504 ms ± 9.24 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
```

Labels:
name: Groupby
name: Performance
description: Memory or execution speed performance

Comments:
Thanks @phofl
CI: Build wheel from sdist
State: closed

Builds the wheels for all platforms/Python versions from the sdist

Other changes:

* Cleans up the `wheel.yml` file
* Uses `delvewheel` to repair the windows wheels instead of manually patching
* Tests windows wheels with `windows-servercore` directly in the `wheels.yml` file
* Runs `wheel unpack` for every wheel for validation, not just windows wheels
* Adds musllinux wheels per https://github.com/MacPython/pandas-wheels/issues/176


Labels:
name: Build
description: Library building on various platforms
name: CI
description: Continuous Integration

Comments:
@mroeschke 

Do you know if this PR is related to the nightly failures?

From a glance, it looks like only macOS is failing sporadically, and with a different error every time. 

Somehow the artifact is getting messed up(missing files I think). 

Downloading the artifact from Github, it looks OK, so I'm thinking there's a race condition or something with uploading/downloading, but not sure where to keep look from here.

https://github.com/pandas-dev/pandas/actions/workflows/wheels.yml?query=event%3Aschedule
CLN: assorted
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Clean
name: Typing
description: type annotations, mypy/pyright type checking

Comments:
Thanks @jbrockmendel
PERF: avoid accessing has_dropped_na
State: closed

When profiling `q1` in the "pandas_queries" from https://github.com/pola-rs/tpch I found the single check to has_dropped_na took 6.5 seconds, 15% of the total runtime.  Turns out we only need has_dropped_na in transform cases, so this PR just avoids accessing it unless it is necessary.

cc @rhshadrach since I think you introduced has_dropped_na.  This seems like something we should be able to improve upon.  Maybe by checking for -1s at the point where we create the codes rather than after we have them?

Labels:

Comments:
> When profiling `q1` in the "pandas_queries" from https://github.com/pola-rs/tpch I found the single check to has_dropped_na took 6.5 seconds, 15% of the total runtime.

Looking at the implementation of has_dropped_na, I'm surprised by this. I'd like to dig into this further - it's a bit hard to discern a reproducible example for this from the link. Do you have anything in this regard? (I can also poke around more).
hmm thats a good point.  in retrospect I bet its the `self.group_info` access that is slow, not the `<0).any()` part.  closing.
CLN: clean Apply._try_aggregate_string_function
State: closed

CC @rhshadrach.

Labels:
name: Clean
name: Apply
description: Apply, Aggregate, Transform, Map

Comments:
@topper-123 - been a while since the tests ran, so just being cautious (paranoid? :laughing:) and wanted to kick them off again. Will merge on green.
Thanks @topper-123
BUG: pd.api.interchange.from_pandas raises with all-null categorical
State: closed

- [ ] closes #53077 (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

pretty simple, hoping this can be bundled together with the other interchange fixes in 2.0.2


Labels:
name: Interchange
description: Dataframe Interchange Protocol

Comments:
Thanks @MarcoGorelli
Added test for sort_index parameter multiindex 'sort_remaining' = False
State: closed

- [x] closes #24247
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests)

Issue fixed on main, added validation tests

Labels:
name: Testing
description: pandas testing functions or related to the test suite

Comments:
Please ensure you run precommit to fix code check issues: https://pandas.pydata.org/docs/development/contributing_codebase.html#contributing-pre-commit
Hi @mroeschke :)
All the checks have passed.
Hi @phofl 
I fixed the code according to your notes, is is ok now?
You didn't address any of my comments
> You didn't address any of my comments

Sorry, I forgot to commit :)
Now you should see it.
thx @NoyHanan
CLN: avoid upcasting in tests where unnecessary (PDEP-6 precursor)
State: closed

another precursor to PDEP6, similar to https://github.com/pandas-dev/pandas/pull/52957

getting there...I think most of the remaining ones are when the test explicitly checks for upcasting behaviour, and so probably can't be changed right now

Labels:
name: Clean

Comments:
Thanks @MarcoGorelli
Fix ns precision isoformat
State: closed

- [x] closes #53020
- [x] [Tests added and passed] Added new tests
- [x] All [code checks passed]
- [x] Added [type annotations]
- [x] Added an entry in the latest `doc/source/whatsnew/v2.1.0.rst` file


Labels:
name: Output-Formatting
description: __repr__ of pandas objects, to_string
name: Timestamp
description: pd.Timestamp and associated methods

Comments:
Please ensure you run precommit to fix code check issues: https://pandas.pydata.org/docs/development/contributing_codebase.html#contributing-pre-commit
Thanks @i1oveMyse1f
REF: implement Manager.concat_vertical, concat_horizontal
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Refactor
description: Internal refactoring of code
name: Internals
description: Related to non-user accessible pandas implementation

Comments:
updated + green
Thanks @jbrockmendel
CLN: avoid infer_dtype
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Clean

Comments:
Thanks @jbrockmendel
REGR: df.loc setitem losing midx names
State: closed

- [x]  closes #53053 (Replace xxxx with the GitHub issue number)
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves
name: Regression
description: Functionality that used to work in a prior pandas version
name: MultiIndex

Comments:
Thanks @phofl
BUG: incorrect is_array_like checks
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Clean

Comments:
Thanks @jbrockmendel
BUG: Fix regression when printing backslash in DataFrame.to_string
State: closed

- [x] closes #53054
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Output-Formatting
description: __repr__ of pandas objects, to_string
name: Regression
description: Functionality that used to work in a prior pandas version

Comments:
Looks okay. Could you address the merge conflict?
> Looks okay. Could you address the merge conflict?

Sure, it's done now
Thanks @GianlucaFicarelli
BUG: `GroupBy.quantile` implicitly sorts `index.levels`
State: closed

- [x] Closes #53009
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests)
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit)
- [x] Added an entry in the latest `doc/source/whatsnew/v2.1.0.rst` file


Labels:
name: Bug
name: Groupby
name: Reduction Operations
description: sum, mean, min, max, etc.

Comments:
Hi @rhshadrach, thanks for your review! I've updated the changelog and the tests as you suggested. However, I'm not sure if `index.levels` is being compared. I tried using
```python
qind = pd.MultiIndex(
    [["A", "B"], [0.2, 0.8]],
    [[1, 1, 0, 0], [0, 1, 0, 1]],
    names=["cat1", None],
)
```
and it still passed the test. Is there any recommended way of fixing this?
Thanks for identifying this @Charlie-XIAO - I opened #53095. For this PR, I think it'd be good to keep the assert_series_equal, and assert the index levels as you had it.
Thanks @rhshadrach! I've retrieved the index levels check.
Thanks for your review @rhshadrach! I've pushed the changes that avoid performance regression when `idx` is not `MultiIndex` as you suggested. Please let me know if similar changes should apply to the case when `idx` is `MultiIndex`.
> Please let me know if similar changes should apply to the case when `idx` is `MultiIndex`.

Yes! It looks to me like these two blocks can now be largely combined.
Thanks @Charlie-XIAO
STY: Bump black to 23.3.0
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Code Style
description: Code style, linting, code_checks

Comments:
technically I think we only need to update `black` once a year, as that's when their changes go out of 'preview' mode - but OK to update now too
Ah cool I was not aware of that. Thanks!