BrandonFors/pandas-github-issues huggingface dataset:
BUG: sort_values raising for dictionary arrow dtype
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Arrow
description: pyarrow functionality

Comments:
Thanks @phofl 
Owee, I'm MrMeeseeks, Look at me.

There seem to be a conflict, please backport manually. Here are approximate instructions:

1. Checkout backport branch and update it.

```
git checkout 2.0.x
git pull
```

2. Cherry pick the first parent branch of the this PR on top of the older branch:
```
git cherry-pick -x -m1 e1b657a7efdc6c8663d6a747d32aa46585ee328b
```

3. You will likely have some merge/cherry-pick conflict here, fix them and commit:

```
git commit -am 'Backport PR #53232: BUG: sort_values raising for dictionary arrow dtype'
```

4. Push to a named branch:

```
git push YOURFORK 2.0.x:auto-backport-of-pr-53232-on-2.0.x
```

5. Create a PR against branch 2.0.x, I would have named this PR:

> "Backport PR #53232 on branch 2.0.x (BUG: sort_values raising for dictionary arrow dtype)"

And apply the correct labels and milestones.

Congratulations — you did some good work! Hopefully your backport PR will be tested by the continuous integration and merged soon!

Remember to remove the `Still Needs Manual Backport` label once the PR gets merged.

If these instructions are inaccurate, feel free to [suggest an improvement](https://github.com/MeeseeksBox/MeeseeksDev).
PERF: fix merging on datetimelike columns to not use object-dtype factorizer
State: closed

See https://github.com/pandas-dev/pandas/pull/53213#issuecomment-1547352853 for context.

Running the added benchmark on current main gives:

```
[100.00%] ··· join_merge.MergeDatetime.time_merge           ok
[100.00%] ··· ============== ============= =================
              --                            tz              
              -------------- -------------------------------
                  units           None      Europe/Brussels 
              ============== ============= =================
               ('ns', 'ns')    16.1±0.3ms      16.1±0.3ms   
               ('ms', 'ms')   20.7±0.07ms      21.0±0.2ms   
               ('ns', 'ms')     167±2ms        16.9±0.4ms   
              ============== ============= =================

```

versus on this branch:

```
[100.00%] ··· ============== ============= =================
              --                            tz              
              -------------- -------------------------------
                  units           None      Europe/Brussels 
              ============== ============= =================
               ('ns', 'ns')   6.33±0.09ms      6.44±0.1ms   
               ('ms', 'ms')    6.95±0.2ms      7.06±0.1ms   
               ('ns', 'ms')   6.90±0.08ms     6.91±0.09ms   
              ============== ============= =================
```

(and running the standard ns/ns example on pandas 1.5, it also gave around 7ms, so this should restore the performance of 1.5)



- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Datetime
description: Datetime data dtype
name: Performance
description: Memory or execution speed performance
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode

Comments:
@jbrockmendel anything else here, or good to go?
Thanks @jorisvandenbossche
DOC: Add note for building with setuptools
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Docs

Comments:
Thanks @phofl
BUG: Correct behavior when reading empty dta files
State: closed

Correct column selection when reading empty dta files Correct omitted dtype information in empty dta files

closes #46240

- [X] closes #46240 (Replace xxxx with the GitHub issue number)
- [X] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [X] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [X] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [X] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Bug
name: IO Stata
description: read_stata, to_stata

Comments:
Thanks @bashtage
BUG: Interpolate not respecting inplace for empty df
State: closed

- [x] closes #53199 (Replace xxxx with the GitHub issue number)
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Missing-data
description: np.nan, pd.NaT, pd.NA, dropna, isnull, interpolate

Comments:
Thanks @phofl
Backport PR #53189 on branch 2.0.x (BUG/CoW: Series.rename not making a lazy copy when passed a scalar)
State: closed

Backport PR #53189: BUG/CoW: Series.rename not making a lazy copy when passed a scalar

Labels:
name: Copy / view semantics

Comments:
cc @lithomas1 looks like this broke something here. Can you take a look?
Hm, I see what's going on. The lazy copy is not taking place since, we're missing #51698 on 2.0.x, that allows df["blah"] = series to do a lazy copy.

Ideally I'd like to continue backporting this, so is it fine if I just revert the test changes for now?
Yep sounds good to me
ENH: Adding engine_kwargs to DataFrame.to_excel
State: closed

- [X] closes #52368
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [X] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [X] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Enhancement
name: IO Excel
description: read_excel, to_excel

Comments:
@rhshadrach Pinging for review since the CI tests aren't all green. Looks like one test failed, and the failure was not related to my enhancement or unit tests. 
@rhshadrach @mroeschke I believe the PR should be good to go at this point. Looks like the test failure for test-arm is unrelated to my push. While writing this comment,  it appears that the test is in a maintenance state. 
Thanks @rmhowe425!
FIX typo in deprecation message of `deprecate_kwarg` decorator
State: closed

- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added an entry in the latest `doc/source/whatsnew/v2.1.0.rst`

Previously, the error message looks has an additional quotation mark, which I assume is a **typo**:
```
FutureWarning: the 'quantile'' keyword is deprecated, use 'q' instead.
```
Now it looks like this:
```
FutureWarning: the 'quantile' keyword is deprecated, use 'q' instead.
```

Btw, does this fix need a changelog?

Labels:
name: Clean

Comments:
thx @Charlie-XIAO, no need for a changelog entry
DEPR Rename keyword "quantile" to "q" in `Rolling.quantile`
State: closed

- [x] Closes #52550
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests)
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit)
- [x] Added an entry in the latest `doc/source/whatsnew/v2.1.0.rst` file


Labels:
name: Deprecate
description: Functionality to remove in pandas
name: Window
description: rolling, ewma, expanding

Comments:
Could you deprecate this on `Expanding` too?
In rolling.py there's also this "private" quantile function

```
    def quantile(
        self,
        quantile: float,
        interpolation: QuantileInterpolation = "linear",
        numeric_only: bool = False,
    ):
```

could you change `quantile` to `q` here too?
> In rolling.py there's also this "private" quantile function, could you change `quantile` to `q` here too?

Sure, and since it is a private function, no deprecation is needed right?
> Sure, and since it is a private function, no deprecation is needed right?

Correct
Thanks @Charlie-XIAO
BUG Merge not behaving correctly when having `MultiIndex` with a single level
State: closed

- [x] Closes #52331
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests)
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit)
- [x] Added an entry in the latest `doc/source/whatsnew/v2.1.0.rst` file

Labels:
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode

Comments:
Could you resolve the merge conflict?
Done. Will fix code style ASAP.
Thanks @Charlie-XIAO
FIX preserve dtype with datetime columns of different resolution when merging
State: closed

- [x] closes #53200
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

Avoid coercing to `object` dtype when merging datetime columns with different resolutions. Instead, use the most fine-grain resolution.

Labels:
name: Bug
name: Non-Nano
description: datetime64/timedelta64 with non-nanosecond resolution

Comments:
thx @glemaitre 
Owee, I'm MrMeeseeks, Look at me.

There seem to be a conflict, please backport manually. Here are approximate instructions:

1. Checkout backport branch and update it.

```
git checkout 2.0.x
git pull
```

2. Cherry pick the first parent branch of the this PR on top of the older branch:
```
git cherry-pick -x -m1 935244a9b9c0fe796de315cd381d6364719bfdc1
```

3. You will likely have some merge/cherry-pick conflict here, fix them and commit:

```
git commit -am 'Backport PR #53213: FIX preserve dtype with datetime columns of different resolution when merging'
```

4. Push to a named branch:

```
git push YOURFORK 2.0.x:auto-backport-of-pr-53213-on-2.0.x
```

5. Create a PR against branch 2.0.x, I would have named this PR:

> "Backport PR #53213 on branch 2.0.x (FIX preserve dtype with datetime columns of different resolution when merging)"

And apply the correct labels and milestones.

Congratulations — you did some good work! Hopefully your backport PR will be tested by the continuous integration and merged soon!

Remember to remove the `Still Needs Manual Backport` label once the PR gets merged.

If these instructions are inaccurate, feel free to [suggest an improvement](https://github.com/MeeseeksBox/MeeseeksDev).
                
One aspect we were still looking into was that the _reason_ this is correctly working for tz-naive datetimes with different resolutions is only because the merging uses the object factorizer (at which point the different resolution doesn't matter). We should also call `_ensure_matching_resos` for tz-naive data somewhere as well.

This also seems to point to a performance regression from 1.5 to 2.0, as looking a bit into it, it seems we are actually also using the object factorizer even for datetimes with the same resolution.

Will do a follow-up PR
Performance fix -> https://github.com/pandas-dev/pandas/pull/53231

There was also one more situation I noticed where we incorrectly cast to object dtype, and that's for an outer/right join if the join key column needs to be combined. Possible short-term workaround -> https://github.com/pandas-dev/pandas/pull/53233
ENH: explicit filters parameter in pd.read_parquet
State: closed

- [x] closes #52238 (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

added filters as a new parameter in pd.read_parquet

Labels:
name: Docs
name: IO Parquet
description: parquet, feather

Comments:
Given that the `filters` keyword is supported by both engines and works the same, I think it would certainly be useful to explicitly document this keyword, and not hide it in the description of the kwargs. We actually already do the same for the `columns` keyword. 
@mroeschke OK with adding this?
Okay yeah that makes sense to add this then
This pull request is stale because it has been open for thirty days with no activity. Please [update](https://pandas.pydata.org/pandas-docs/stable/development/contributing.html#updating-your-pull-request) and respond to this comment if you're still interested in working on this.
@mrastgoo would you have time to add a test?

We have an existing test for the columns keyword (`test_read_columns`), so could add an equivalent `test_read_filters` test case next to it. Something like:

```
    def test_read_filters(self, engine, tmp_path):
        df = pd.DataFrame({"int": list(range(4)), "part": list("aabb"),})

        expected = pd.DataFrame({"int": [0, 1]})
        check_round_trip(
            df,
            engine,
            path=tmp_path,
            expected=expected,
            write_kwargs={"partition_cols": ["part"]},
            read_kwargs={"filters": [("string", "==", "a")], "columns":["int"]},
            repeat=1,
        )
```

I used partition cols to actually see the effect of the filter also for fastparquet (to ensure it is correctly passed through). And with that, the `repeat=1` is needed to not add additional files to the directory when writing a second time (with partition_cols, it does not just overwrite the single file).
@jorisvandenbossche , yes I would do it asap, sorry for the delay in this issue and thanks for the info I will take that into account. 
I just pushed for the test. 
I changed the following line for the test
```
read_kwargs={"filters": [("part", "==", "a")], "columns":["int"]},
```
the error which was raised was not very clear for missing columns, which is not a pandas issue, but thought to mention it. 
```
pyarrow.lib.ArrowInvalid: No match for FieldRef.Name(string) in int: int64
part: dictionary<values=string, indices=int32, ordered=0>
```
Could you merge in main once more?
Thanks @mrastgoo
Use ccache
State: closed

Supersedes #47360

Meson autodetects and autouses ccache so this PR turns out to be very simple.

Shaves off ~ 5 min

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: CI
description: Continuous Integration
name: Stale

Comments:
Is the build failure related to this change?
Seems like the ccache is not being used correctly in one of the code checks?

```
Configure sccache
  /usr/bin/bash -xc env SCCACHE_IDLE_TIMEOUT=0 SCCACHE_DIR='/home/runner/work/pandas/pandas'/.sccache SCCACHE_CACHE_SIZE='500M' sccache --start-server
  + '[' -z '' ']'
  + return
  + case $- in
  + return
  + env SCCACHE_IDLE_TIMEOUT=0 SCCACHE_DIR=/home/runner/work/pandas/pandas/.sccache SCCACHE_CACHE_SIZE=500M sccache --start-server
  sccache: Starting the server...
  sccache: error: Server startup failed: Address in use
```
FYI sccache-action is actually pretty nice because of this: https://github.com/Mozilla-Actions/sccache-action/issues/43#issuecomment-1550356852
Hmm Meson detects sccache but somehow the cache is not filled.
This pull request is stale because it has been open for thirty days with no activity. Please [update](https://pandas.pydata.org/pandas-docs/stable/development/contributing.html#updating-your-pull-request) and respond to this comment if you're still interested in working on this.
Thanks for the PR but it appears to have gone stale. If you want to continue feel free to ping and I can reopen
DOC: add documentation to pandas.core.groupby.SeriesGroupBy.unique
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

towards https://github.com/noatamir/pyladies-berlin-sprints/issues/13
makes the docstring for `pandas.core.groupby.SeriesGroupBy.unique`

cc: @jorisvandenbossche @marenwestermann 

docs used: [here](https://pandas.pydata.org/docs/dev/reference/api/pandas.core.groupby.SeriesGroupBy.unique.html)

Labels:
name: Docs
name: Groupby

Comments:
Thanks @maikia
DOC fix collapsing using new bootstrap v5 API
State: closed

The dev version of the documentation is broken: the collapse buttons of the table in the "Intro to pandas" are not responsive.

This is linked to an update of the `pydata-sphinx-theme` from 0.11 to 0.13 (#53029) that uses Bootstrap v5 that does not rely anymore on jQuery and renamed `data-toogle` and `data-target` to `data-bs-toogle` and `data-bs-target`, respectively.

In addition, `badge-dark` is deprecated in favour of `badge-dark`. However, `bg-dark` does work well with dark theme and `bg-secondary` does a better job.

This is a quick fix knowing that a change to `sphinx-design` would allow to removal the pure HTML file.

Labels:
name: Docs

Comments:
/preview
Website preview of this PR available at: https://pandas.pydata.org/preview/53209/
/preview
Website preview of this PR available at: https://pandas.pydata.org/preview/53209/
/preview
Website preview of this PR available at: https://pandas.pydata.org/preview/53209/
Thanks @glemaitre
BUG: DataFrame.agg not returning a reduced result when providing a lambda
State: closed

- [ ] closes #41768 (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Apply
description: Apply, Aggregate, Transform, Map

Comments:
Actually this solution is somewhat frail, a better fix would probably not to hit apply here so closing
BUG: crosstab(dropna=False) did not keep np.nan in result
State: closed

- [x] closes #10772 (Replace xxxx with the GitHub issue number)
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Missing-data
description: np.nan, pd.NaT, pd.NA, dropna, isnull, interpolate
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode

Comments:
thx @mroeschke 
THANK YOU! @mroeschke
CI: Fix conda_forge_recipe package check
State: closed

Fixing https://github.com/pandas-dev/pandas/actions/runs/4961454335/jobs/8878246555

Labels:
name: Build
description: Library building on various platforms
name: CI
description: Continuous Integration

Comments:
The errors are due to https://github.com/mamba-org/setup-micromamba/issues/68 which will be fixed soon so merging this to fix the package checks