BrandonFors/pandas-github-issues huggingface dataset:
Updated `pre-commit` hooks versions.
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Code Style
description: Code style, linting, code_checks

Comments:
Thanks @Sai-Suraj-27 (for future reference, we have a bot that automatically updates these) 
> Thanks @Sai-Suraj-27 (for future reference, we have a bot that automatically updates these)

I know sir, but I saw that its been almost 2-3 weeks since the new versions were released but they were not updated here, So I thought to just quickly make a PR.
Update pyproject.toml - replace `output_formatting` with `output-formatting`
State: closed

closes #55290

Simply said, none of this would work (dependencies from `output-formatting` would not be installed):

```
pip install pandas[output-formatting]
pip install pandas[output_formatting]
pip install pandas[output-formatting,xml]
```

When being distributed, underscores are replaces with dashes. So, in the packaged PKG-INFO file you won't find spots of `output_formatting`, but only `output-formatting`:

```
Requires-Dist: lxml>=4.8.0; extra == "xml"
Requires-Dist: jinja2>=3.1.2; extra == "output-formatting"
Requires-Dist: tabulate>=0.8.10; extra == "output-formatting"
```

But on the same time, this:

```
Provides-Extra: output_formatting
```

This fix addresses that.

Labels:
name: Build
description: Library building on various platforms

Comments:
Thanks could you add a whatsnew note in `2.1.2.rst`?
@mroeschke:

Done.
Could you also change the references in `doc/source/getting_started/install.rst` and `.github/workflows/package-checks.yml`?
@mroeschke:

Done.
Thanks @barseghyanartur
ASV: add GroupBy.sum benchmark with timedelta and integer types
State: closed

- [x] closes #20660
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests).
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added new benchmark to `GroupBy.sum` method with `timedelta64` and `int64` types.


Labels:
name: Benchmark
description: Performance (ASV) benchmarks

Comments:
Thanks @josemayer
This fix enables you to preserve the datetime precision when using the melt method.
State: closed



- [X ] closes #55254 (Replace xxxx with the GitHub issue number)
- [X ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ X] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ X] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode

Comments:
Nice work @paulreece
BUG: groupby.idxmax/idxmin consistently raise on unobserved categorical
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

Part of #10694, but doesn't close it fully.

 - When given a non-ordered category with unobserved categories, we currently raise about the unobserved categories. We should instead raise about the non-orderedness for consistency with min/max.
 - _python_apply_general does not return the correct dtype when there is a CategoricalIndex will NaN values.
 - _python_apply_general fails on a single grouping with unobserved categories even when we're call it from transform where the unobserved categories should have no impact.
 - _python_apply_general with an empty DataFrame with no numeric columns returns all the columns even when `numeric_only=True`.

All of these are fixed in #54234

Labels:
name: Groupby
name: Categorical
description: Categorical Data Type
name: Reduction Operations
description: sum, mean, min, max, etc.

Comments:
Friendly ping @WillAyd @jbrockmendel 
@WillAyd - are you good here?
TST: Load iris and types data in sql tests as needed
State: closed

Also factors out timezone data in a fixture to only apply to the single test where it's used

Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: IO SQL
description: to_sql, read_sql, read_sql_query

Comments:
This generally looks good  - thanks for cleaning these up
Thanks for the review @WillAyd. Going to merge this in and look at the data loading cleanups in a follow up PR
TYP: Misc changes for pandas-stubs; use Protocol to avoid str in Sequence
State: closed

The first commit addresses some type issues found by running mypy on the tests from pandas-stubs (one big issue is that Index/Series are not included in IndexLabel; there might be cases where that is correct but in many cases they should be included).

The second commit re-writes #47233 with the protocol version of Sequence.

Can split it in two PRs if one of them is controversial.

Labels:
name: Typing
description: type annotations, mypy/pyright type checking

Comments:
Thanks @twoertwein (you two are the typing authorities as far as I'm concerned :) )
CLN: Remove temp_setattr from groupby.transform
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

Using temp_setattr doesn't actually change anything. All of the objects that `observed` impacts are cached and have already been computed.

Labels:
name: Groupby
name: Clean

Comments:
nice!  LGTM pending green
Fix value_counts for sort=False parameter
State: closed

- [x] closes #55224 
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Stale

Comments:
It appears that `sort` is ignored for many dtypes. Two questions here:

 1. Are we in favor of implementing `sort` here across the board?
 2. Are we okay with adding a do-nothing `sort` argument to some dtypes and implementing fully in future PRs?

My answer to both is "yes", but wanted to get some other eyes on this.

cc @mroeschke @phofl
This pull request is stale because it has been open for thirty days with no activity. Please [update](https://pandas.pydata.org/pandas-docs/stable/development/contributing.html#updating-your-pull-request) and respond to this comment if you're still interested in working on this.
Hi @rhshadrach, Any suggestions for me to improve the PR?
@tirth-hihoriya - adding a `sort` argument that gets ignored for various array types isn't great. I think if these were implemented, we'd be good here.
Thanks for the pull request, but it appears to have gone stale. If interested in continuing, please merge in the main branch, address any review comments and/or failing tests, and we can reopen.
BUG: DatetimeIndex.union returning object dtype for indexes with the same tz but different units
State: closed

- [x] closes #55238
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/v2.2.0.rst` file if fixing a bug or adding a new feature.


Labels:
name: Bug
name: Dtype Conversions
description: Unexpected or buggy dtype conversions

Comments:
Thanks @lukemanley
BUG: Series[slc]=foo raising with IntervalIndex
State: closed

- [x] closes #54722 (Replace xxxx with the GitHub issue number)
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves
name: Interval
description: Interval data type

Comments:
Thanks @jbrockmendel
DEPR: Index.insert dtype-inference
State: closed

- [x] closes #51363 (Replace xxxx with the GitHub issue number)
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

Doesn't have a "do X to avoid this warning, not sure if this merits future option or temporary keyword.

Labels:
name: Deprecate
description: Functionality to remove in pandas

Comments:
Its kind of annoying that this warning gets issued (the PR currently suppress it) in `frame_or_series[new_key] = val`.  any preferences on whether to warn there?
This pull request is stale because it has been open for thirty days with no activity. Please [update](https://pandas.pydata.org/pandas-docs/stable/development/contributing.html#updating-your-pull-request) and respond to this comment if you're still interested in working on this.
@phofl thoughts on doing this?  i think there was some vague similarity to some of what you were working on.  main open question here is whether to continue suppressing warnings on setitem-with-expansion
thx @jbrockmendel 
Glad to get movement here, but i was hoping for some discussion of whether suppressing the warning in setitem-with-expansion was the way to go
I wouldn’t want to warn there, I don’t think that anyone cares about inference there and getting rid of it is a pain
ENH/PERF: add ExtensionArray.duplicated
State: closed

- [x] closes #27264
- [x] closes #48424
- [x] closes #48788
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/v2.2.0.rst` file if fixing a bug or adding a new feature.

[updated]

`asv continuous -f 1.1 upstream/main arrow-temporal-duplicated -b algorithms.Duplicated`

```
       before           after         ratio
-         101±2ms         51.6±3ms     0.51  algorithms.Duplicated.time_duplicated(False, False, 'string[pyarrow]')
-        98.5±3ms         47.3±4ms     0.48  algorithms.Duplicated.time_duplicated(False, 'first', 'string[pyarrow]')
-        96.0±4ms         45.5±4ms     0.47  algorithms.Duplicated.time_duplicated(False, 'last', 'string[pyarrow]')
-      1.57±0.03s       13.6±0.3ms     0.01  algorithms.Duplicated.time_duplicated(False, False, 'timestamp[ms][pyarrow]')
-      1.54±0.02s       13.1±0.7ms     0.01  algorithms.Duplicated.time_duplicated(False, 'first', 'timestamp[ms][pyarrow]')
-         1.55±0s       12.8±0.7ms     0.01  algorithms.Duplicated.time_duplicated(False, 'last', 'timestamp[ms][pyarrow]')

```

Labels:
name: Performance
description: Memory or execution speed performance
name: Arrow
description: pyarrow functionality

Comments:
Thanks @lukemanley 
This patch may have induced a performance regression. If it was a necessary behavior change, this may have been expected and everything is okay.

Please check the links below. If any ASVs are parameterized, the combinations of parameters that a regression has been detected for appear as subbullets.

 - [period.Algorithms.time_drop_duplicates](https://asv-runner.github.io/asv-collection/pandas/#period.Algorithms.time_drop_duplicates)
   - [typ='index'](https://asv-runner.github.io/asv-collection/pandas/#period.Algorithms.time_drop_duplicates?p-typ=%27index%27)

Subsequent benchmarks may have skipped some commits. The link below lists the commits that are between the two benchmark runs where the regression was identified.

https://github.com/pandas-dev/pandas/compare/6e6a6831aaa2d1606aa897b34f106e655180034b...3bf0f647ca42bf21de76f9e6027de38cb5589147
DEPR: 'A' for yearly frequency and YearEnd in favour of 'Y'
State: closed

xref #54275, #54061

deprecated string `'A'` for yearly frequency and YearEnd in favour of `'Y'`.

EDIT:
deprecated annual frequencies with various fiscal year ends:` "A-DEC", "A-JAN"`, etc. in favour of `"Y-DEC",  "Y-JAN"`, etc.

Labels:
name: Frequency
description: DateOffsets
name: Deprecate
description: Functionality to remove in pandas

Comments:
> in `offsets.pyx` I think you'll need to generalise this part
> 
> ```
>                 if is_period is False and name == "M":
>                     warnings.warn(
>                         "\'M\' will be deprecated, please use \'ME\' "
>                         "for \'month end\'",
>                         UserWarning,
>                         stacklevel=find_stack_level(),
>                     )
>                     name = "ME"
>                 if is_period is True and name == "ME":
>                     raise ValueError(
>                         r"for Period, please use \'M\' "
>                         "instead of \'ME\'"
>                     )
>                 elif is_period is True and name == "M":
>                     name = "ME"
> ```

Thank you for the comment.  I generalised it by adding a dictionary with frequencies deprecated for offsets. 
I deprecated annual frequencies with various fiscal year ends: `"A-DEC", "A-JAN"`, etc. in favour of `"Y-DEC",  "Y-JAN"`, etc., added tests for FutureWarning and notes in v2.2.0. @MarcoGorelli, could you please take a look at this PR?
Thank you for helping me with this PR.
BUG: incompatible dtype when creating string column with loc
State: closed

- [ ] closes #55025 (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves

Comments:
Thanks @MarcoGorelli
BUG: Rolling pd.date_range incorrect for unit='s' #55026
State: closed

- [X] closes #55026
- [X] [Tests added and passed]

This follows the #55173 and will be fixed by the same approach. The custom tests were added for the `max` function itself.

Labels:

Comments:
Thanks. Could you just include this test in https://github.com/pandas-dev/pandas/pull/55173? Closing in favor of moving this there
> Thanks. Could you just include this test in #55173? Closing in favor of moving this there

Of course :) Done
PERF: Add type-hints in tzconversion.pyx
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

Part of #55179. With this, I'm seeing time plummet in the benchmark

    tslibs.tz_convert.TimeTZConvert.time_tz_localize_to_utc(1000000, tzfile('/usr/share/zoneinfo/Asia/Tokyo'))

from 159ms (Cython 0.29) and 271ms (Cython 3.0.2) to ~5-6ms on both.

Labels:
name: Performance
description: Memory or execution speed performance
name: Timezones
description: Timezone data dtype

Comments:
Any suggestions on how to write a whatsnew note here? Perf improvement when localizing time to UTC?
This is great.  One potential future issue to be aware of though: this locks us in to having only 1D ndarrays passed to `_get_utc_bounds`.  This isn't a problem ATM since the calling function `tz_localize_to_utc` is already locked in.  But if we ever wanted to make `tz_localize_to_utc` handle unknown-ndim (which is on my radar bc it would allow removing the potentially-copying `@dtl.ravel_compat` from `DatetimeArray.tz_localize`) this would become an issue.  So no worries, just heads up.
In that case to keep perf we'd have to resort to fused types, is that right?
AFAIK there isn't a nice solution to the arbitrary-ndim use case.  In practice it is always either 1 or 2 (maybe 0 in some corner cases) so we could make a fused type for just those.  This won't be an issue anytime soon so this is just a "be aware of".  im kind of hoping cython adds support for dtype-declaring ndarrays without declaring the dimension
DOC: Fix doc for first_valid_index
State: closed

- [x] closes #55187
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Docs

Comments:
Thanks @jfadia
DEPR: Change FutureWarning for observed=False to DeprecationWarning
State: closed

- [ ] closes #54970 (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Groupby
name: Deprecate
description: Functionality to remove in pandas

Comments:
https://github.com/pandas-dev/pandas/issues/54970#issuecomment-1730686981 makes me think more that we should not revert.
Yeah I think this is a case where we should have been more careful in hindsight misjudging the "impact", but I don't think we should change to a `DeprecationWarning` due to the churn.