BrandonFors/pandas-github-issues huggingface dataset:
PERF: Groupby.idxmax
State: closed

cc @phofl this is currently failing locally and could use a fresh pair of eyes.

Labels:

Comments:
I'll try to take a look over the weekend
@phofl im mothballing this to clear the queue, just a ping so this doesn't fall off your radar entirely
I think the main problem is that ``_wrap_aggregated_output`` isn't working for unordered categories that are empty. All other reductions that are tested are raising TypeErrors, while idxmax and idxmin return the wrong result. Any ideas what we could do there?
DEPR: concat with mismatched length keys/objs
State: closed

- [x] closes #43485 (Replace xxxx with the GitHub issue number)
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode
name: Deprecate
description: Functionality to remove in pandas

Comments:
Thanks @jbrockmendel
TYP: datetimelike arrays
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Typing
description: type annotations, mypy/pyright type checking

Comments:
Thanks @jbrockmendel
DEPR: groupby with as_index=False not including out-of-axis groupings
State: closed

- [x] closes #49519 (Replace xxxx with the GitHub issue number)
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

cc @jorisvandenbossche 

I only took a cursory look, but I'm hopeful when this deprecation is enforced that we will be able to remove all internal notions of `in_axis` groupings.

Labels:
name: Groupby
name: Deprecate
description: Functionality to remove in pandas

Comments:
Thanks @rhshadrach
Use date32 for pyarrow date
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: datetime.date
description: stdlib datetime.date support
name: Arrow
description: pyarrow functionality

Comments:
@mroeschke the precision is always the same, it's unexpected to store non-fullday value in pa.date64 and date64 is always milliseconds: https://lists.apache.org/thread/q036r1q3cw5ysn3zkpvljx3s9ho18419

The only gain of using that type is the cheaper conversion for frameworks / languages using millisecond based timestamps, but it's slower and uses more memory.
Owee, I'm MrMeeseeks, Look at me.

There seem to be a conflict, please backport manually. Here are approximate instructions:

1. Checkout backport branch and update it.

```
git checkout 2.0.x
git pull
```

2. Cherry pick the first parent branch of the this PR on top of the older branch:
```
git cherry-pick -x -m1 4d1c994da1e9815c87db2cd48a3286b077f2f7fe
```

3. You will likely have some merge/cherry-pick conflict here, fix them and commit:

```
git commit -am 'Backport PR #52332: Use date32 for pyarrow date'
```

4. Push to a named branch:

```
git push YOURFORK 2.0.x:auto-backport-of-pr-52332-on-2.0.x
```

5. Create a PR against branch 2.0.x, I would have named this PR:

> "Backport PR #52332 on branch 2.0.x (Use date32 for pyarrow date)"

And apply the correct labels and milestones.

Congratulations — you did some good work! Hopefully your backport PR will be tested by the continuous integration and merged soon!

Remember to remove the `Still Needs Manual Backport` label once the PR gets merged.

If these instructions are inaccurate, feel free to [suggest an improvement](https://github.com/MeeseeksBox/MeeseeksDev).
                
Thanks for the explanation @alippai 
Thanks for the swift merge and backport @mroeschke
ENH: Implement .dt.year for ArrowExtensionArray
State: closed

- [ ] closes #52321 (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Datetime
description: Datetime data dtype
name: Arrow
description: pyarrow functionality

Comments:
Just noticed:
```
@property
    def _dt_date(self):
        return type(self)(self._pa_array.cast(pa.date64()))
```
Shouldn't this be `date32`? 

EDIT: Created https://github.com/pandas-dev/pandas/pull/52332 instead to test it
Thx @mroeschke
DOC streamline contributing guide a bit
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

Some relatively minor things I noticed

Labels:
name: Docs

Comments:
```
2023-03-31T17:15:59.3514670Z /home/runner/work/pandas/pandas/doc/source/development/contributing_codebase.rst:38: ERROR: Unexpected indentation.
2023-03-31T17:15:59.3517430Z /home/runner/work/pandas/pandas/doc/source/development/contributing_codebase.rst:41: WARNING: Block quote ends without a blank line; unexpected unindent.
```
Thanks @MarcoGorelli
PERF: new_block
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Performance
description: Memory or execution speed performance
name: Internals
description: Related to non-user accessible pandas implementation

Comments:
Any idea on the perf improvement?
> Any idea on the perf improvement?

In the branch in #52291 I'm seeing about 4% of the runtime in check_ndim call that this removes.  #52316 will cause that number to increase.
Thanks  @jbrockmendel
BUG: GH52229 changed code in pandas/pandas/core/reshape/pivot.py _bui…
State: closed

…ld_names_mapper() to match docstring - replaces names of columns/rows

- [x] closes #52229


Labels:

Comments:
Based on the failing test, I think this function just needs to be renamed
Thanks for the pull request, but it appears to have gone stale. If interested in continuing, please merge in the main branch, address any review comments and/or failing tests, and we can reopen.
CLN: Assorted cleanups
State: closed

Ref: 
 - https://github.com/pandas-dev/pandas/commit/871494846489e0964e3b77701c08eb482dd0b10f#r105292840
 - https://github.com/pandas-dev/pandas/pull/52065/files#r1142381009

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


@mroeschke @jbrockmendel 

Labels:
name: Clean

Comments:
thanks @rhshadrach
PERF: concat_compat
State: closed

Fastpath for all-matching-dtype case.

Labels:
name: Performance
description: Memory or execution speed performance
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode

Comments:
Thanks @jbrockmendel 
@jbrockmendel I think there might be typing failures here. Mind taking a look? https://github.com/pandas-dev/pandas/actions/runs/4578025660/jobs/8084134007
yep, will make a PR shortly
TYP: concat_compat
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode
name: Typing
description: type annotations, mypy/pyright type checking

Comments:
Thanks @jbrockmendel
BUG: DataFrame.corrwith raising for pyarrow-backed dtypes
State: closed

- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/v2.1.0.rst` file if fixing a bug or adding a new feature.


Fixes the following:

```
import pandas as pd
import numpy as np

data = np.random.randn(100, 10)
df = pd.DataFrame(data, dtype="float64[pyarrow]")

df.corrwith(df)
```

```
NotImplementedError: <class 'pandas.core.arrays.arrow.array.ArrowExtensionArray'> does not support reshape as backed by a 1D pyarrow.ChunkedArray.
```

Labels:
name: Bug
name: Arrow
description: pyarrow functionality

Comments:
Thanks @lukemanley
REF/PERF: pd.concat
State: closed

Sits on top of #52291.  Mostly a refactor making `Concatenator.__init__` less stateful, makes some of the annotations more accurate.  A bit of a performance improvement from avoiding a `_get_concat_axis` in some cases.  Makes it easier to spot in profiling where more overhead can be trimmed going forward.

Labels:
name: Performance
description: Memory or execution speed performance
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode
name: Typing
description: type annotations, mypy/pyright type checking

Comments:
Thanks @jbrockmendel
DOC: Clarify Timestamp(unit=) will not necessarily set unit
State: closed

- [ ] closes #52286 (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Docs
name: Non-Nano
description: datetime64/timedelta64 with non-nanosecond resolution
name: Timestamp
description: pd.Timestamp and associated methods

Comments:
thx @mroeschke
DOC: Update link in user guide/style
State: closed

- DOC: Fixes broken link in [https://pandas.pydata.org/docs/user_guide/style.html](https://pandas.pydata.org/docs/user_guide/style.html).
- [x] Closes #52157


Labels:
name: Docs
name: Web
description: pandas website

Comments:
/preview
Website preview of this PR available at: https://pandas.pydata.org/preview/52304/
Thanks @allisonkwan.
combine_first with mix tz test GH#26283
State: closed

- [ ] closes #26283 (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).


Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Timezones
description: Timezone data dtype

Comments:
Thanks @liang3zy22
PERF: concat
State: closed

xref #23362 (closes?  not sure how much more we can do)

```
import numpy as np
import pandas as pd

a = np.random.randint(2000, 2100, size=1000)
b = np.random.randint(2000, 2100, size=1000)

x = pd.core.arrays.period_array(a, freq='B')
y = pd.core.arrays.period_array(b, freq='B')

s = pd.Series(x)
t = pd.Series(y)


In [2]: %timeit pd.concat([s, t], ignore_index=True)
54.6 µs ± 2.41 µs per loop (mean ± std. dev. of 7 runs, 10,000 loops each)  # <- PR
81.5 µs ± 419 ns per loop (mean ± std. dev. of 7 runs, 10,000 loops each)  # <- main
```

cc @phofl do we need to do anything else refs-wise? when calling Manager.from_array?


Labels:
name: Performance
description: Memory or execution speed performance
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode

Comments:
I think we only get there when we have ``axis=0`` which creates a copy anyway, so no we are good
@mroeschke updated + green includes typing fix for main
Thanks @jbrockmendel
PERF: concat_same_type for PeriodDtype
State: closed

xref #23362

```
import numpy as np
import pandas as pd

a = np.random.randint(2000, 2100, size=1000)
b = np.random.randint(2000, 2100, size=1000)

x = pd.core.arrays.period_array(a, freq='B')
y = pd.core.arrays.period_array(b, freq='B')

%timeit x._concat_same_type([x, y])
6.86 µs ± 75.2 ns per loop (mean ± std. dev. of 7 runs, 100,000 loops each)  # <- main
4.24 µs ± 109 ns per loop (mean ± std. dev. of 7 runs, 100,000 loops each)  # <- PR
```

Labels:
name: Performance
description: Memory or execution speed performance
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode
name: Period
description: Period data type

Comments:
Thanks @jbrockmendel
PERF: needs_i8_conversion expect dtype object
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Performance
description: Memory or execution speed performance

Comments:
Thanks @jbrockmendel
DOC: Escape backslash in read_clipboard docstring
State: closed

- [x] closes #51868
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Docs
name: IO Data
description: IO issues that don't fit into a more specific label

Comments:
/preview
Website preview of this PR available at: https://pandas.pydata.org/preview/52285/
Thanks @wjandrea
BUG: mean/median with strings
State: closed

- [x] closes #36703 (Replace xxxx with the GitHub issue number)
- [x] closes #44008
- [x] closes #34671
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

I'm not wild about this multi-pass implementation.  Longer-term I think we need to re-write a lot of nanops to be single-pass (zero|few)-copy.

Labels:
name: Strings
description: String extension data type and string data
name: Reduction Operations
description: sum, mean, min, max, etc.

Comments:
As an aside, it feels like a non-explicit operation that strings are coerced to numeric before calling mean/median. Would be nice if it was deprecation in favor of the user explicitly casting to a numeric type
> As an aside, it feels like a non-explicit operation that strings are coerced to numeric before calling mean/median. Would be nice if it was deprecation in favor of the user explicitly casting to a numeric type

That could work for mean (though eg Decimal objects may be tricky until we can tell infer those to pyarrow decimal).  For  median i actually think it can work on object dtype without casting (though may be slower), saving that for another day.

Big picture I'd like to refactor a lot of nanops to be no-copy and share code with libgroupby.
Worth moving forward here or do we need to wait for a different approach?
Could the behavior of strings automatically being coerced to numbers during reductions be deprecated instead of supporting them?
> Could the behavior of strings automatically being coerced to numbers during reductions be deprecated instead of supporting them?

I'm not clear on how that is different from what this does (aside from calling it a bugfix instead of a deprecation)
Ah okay sorry. Could you mention in the whatsnew that this raises a `TypeError` now? 
Thanks @jbrockmendel
Doc: add ExtensionArray.map to reference/extensions.rst
State: closed

xref #52247 & #52263.

Labels:
name: Docs
name: ExtensionArray
description: Extending pandas with custom dtypes or arrays.
name: Stale

Comments:
This is the error that also failed last time

```
2023-03-29T20:13:24.9068667Z /home/runner/work/pandas/pandas/doc/source/reference/api/pandas.api.extensions.ExtensionArray.map.rst: WARNING: document isn't included in any toctree
```
Yeah, I’m currently at a loss why this fails. I’l look at this again to see if I can figure it out.
/preview
No preview found for PR #52280. Did the docs build complete?
Still as a loss why this fails. Error message still the same (`pandas/doc/source/reference/api/pandas.api.extensions.ExtensionArray.map.rst: WARNING: document isn't included in any toctree`).

Any suggestion welcome :-)
This pull request is stale because it has been open for thirty days with no activity. Please [update](https://pandas.pydata.org/pandas-docs/stable/development/contributing.html#updating-your-pull-request) and respond to this comment if you're still interested in working on this.
Looks like this PR is stale and the check is still failing so closing for now. Feel free to reopen when you want to get back to this PR
Set na_rep='=na()' as default value, issue #52258
State: closed

- [x] closes issue #52258 
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:

Comments:
Thanks for the pull request but we would need more discussion or buy in from the core team before implementing this feature, so closing for now but happy to have your input in the issue if you would like to see this implemented and will reopen if there more consensus.
DEPR: Passing a dictionary to SeriesGroupBy.agg
State: closed

- [x] closes #50684 (Replace xxxx with the GitHub issue number)
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Groupby
name: Deprecate
description: Functionality to remove in pandas
name: Apply
description: Apply, Aggregate, Transform, Map

Comments:
I don't see a  feasible way to use a single template for both SeriesGroupBy.agg and DataFrameGroupBy.agg, so I split it into two. Is there one, or should these be split?
Thanks @rhshadrach