BrandonFors/pandas-github-issues huggingface dataset:
DOC: point out that only period frequencies are valid for to_period
State: closed

Updated docs for `to_period`. Point out that only `period` frequencies are valid for `to_period`. For example, offset frequency `'MS'` isn't valid for `to_period`.

```
>>> df = pd.DataFrame({"y": [1, 2]}, `index=pd.to_datetime(["2000-03-31` 00:00:00", "2000-05-31 00:00:00"]))
>>> df.index.to_period("MS")
AttributeError: 'pandas._libs.tslibs.offsets.MonthBegin' object has no attribute '_period_dtype_code'

```
Maybe it would be better to have 2 different tables, for offset aliases and period aliases, instead of this one for [offset aliases](https://pandas.pydata.org/docs/user_guide/timeseries.html#timeseries-offset-aliases)?

Labels:
name: Docs
name: Period
description: Period data type

Comments:
Thanks @natmokval 

> Maybe it would be better to have 2 different tables, for [offset aliases](https://pandas.pydata.org/docs/user_guide/timeseries.html#timeseries-offset-aliases) and period aliases, instead of this one for offset aliases?

Yeah I think this would be useful. Do you have the offsets on-hand that don't work with Periods?
> Thanks @natmokval
> 
> Yeah I think this would be useful. Do you have the offsets on-hand that don't work with Periods?

The aliases listed below don't work with period:

```
SM 'pandas._libs.tslibs.offsets.SemiMonthEnd' object has no attribute '_period_dtype_code'
BM 'pandas._libs.tslibs.offsets.BusinessMonthEnd' object has no attribute '_period_dtype_code'
CBM 'pandas._libs.tslibs.offsets.CustomBusinessMonthEnd' object has no attribute '_period_dtype_code'
MS 'pandas._libs.tslibs.offsets.MonthBegin' object has no attribute '_period_dtype_code'
SMS 'pandas._libs.tslibs.offsets.SemiMonthBegin' object has no attribute '_period_dtype_code'
BMS 'pandas._libs.tslibs.offsets.BusinessMonthBegin' object has no attribute '_period_dtype_code'
CBMS 'pandas._libs.tslibs.offsets.CustomBusinessMonthBeg' object has no attribute '_period_dtype_code'
BQ 'pandas._libs.tslibs.offsets.BQuarterEnd' object has no attribute '_period_dtype_code'
QS 'pandas._libs.tslibs.offsets.QuarterBegin' object has no attribute '_period_dtype_code'
BQS 'pandas._libs.tslibs.offsets.BQuarterBegin' object has no attribute '_period_dtype_code'
BA 'pandas._libs.tslibs.offsets.BYearEnd' object has no attribute '_period_dtype_code'
BY 'pandas._libs.tslibs.offsets.BYearEnd' object has no attribute '_period_dtype_code'
AS 'pandas._libs.tslibs.offsets.YearBegin' object has no attribute '_period_dtype_code'
YS 'pandas._libs.tslibs.offsets.YearBegin' object has no attribute '_period_dtype_code'
BAS 'pandas._libs.tslibs.offsets.BYearBegin' object has no attribute '_period_dtype_code'
BYS 'pandas._libs.tslibs.offsets.BYearBegin' object has no attribute '_period_dtype_code'
BH 'pandas._libs.tslibs.offsets.BusinessHour' object has no attribute '_period_dtype_code'

```
Use _values_for_factorize by default for hashing ExtensionArrays
State: closed

PR https://github.com/pandas-dev/pandas/pull/51319 added the `EA._hash_pandas_object` method to let ExtensionArrays override how they are hashed. But it also changed to no longer use the values returned by `EA._values_for_factorize()` by default for hashing, but change this to `EA.to_numpy()`.  The previous behaviour was documented, and changing this can cause regressions / changes in behaviour or performance (depending on the return values of those two methods). 

See https://github.com/pandas-dev/pandas/pull/51319/files#r1212106303 for some more details.

Labels:
name: ExtensionArray
description: Extending pandas with custom dtypes or arrays.

Comments:
@jbrockmendel are you OK with this?
No objections.
Owee, I'm MrMeeseeks, Look at me.

There seem to be a conflict, please backport manually. Here are approximate instructions:

1. Checkout backport branch and update it.

```
git checkout 2.0.x
git pull
```

2. Cherry pick the first parent branch of the this PR on top of the older branch:
```
git cherry-pick -x -m1 2a1d2da13f45c4ae38099a2f73e92308245e66f8
```

3. You will likely have some merge/cherry-pick conflict here, fix them and commit:

```
git commit -am 'Backport PR #53475: Use _values_for_factorize by default for hashing ExtensionArrays'
```

4. Push to a named branch:

```
git push YOURFORK 2.0.x:auto-backport-of-pr-53475-on-2.0.x
```

5. Create a PR against branch 2.0.x, I would have named this PR:

> "Backport PR #53475 on branch 2.0.x (Use _values_for_factorize by default for hashing ExtensionArrays)"

And apply the correct labels and milestones.

Congratulations — you did some good work! Hopefully your backport PR will be tested by the continuous integration and merged soon!

Remember to remove the `Still Needs Manual Backport` label once the PR gets merged.

If these instructions are inaccurate, feel free to [suggest an improvement](https://github.com/MeeseeksBox/MeeseeksDev).
                
Sorry for the backport noise, the original PR was only for 2.1, so of course this fixup only needs to target 2.1 as well
Does this need a whatsnew?

This looks like only a regression on main.
(If so, the whatsnew can be deleted)
Good catch, sorry! I only realized after merging it was only a change on main and thus didn't need to be backported, but forgot that I actually added a whatsnew ...
TST: fix import in test_hash_pandas_object extension test
State: closed

Calling `pd.core.util.hashing._default_hash_key` directly after import of pandas fails with an ImportError, but depending on whether you already did some other operations, the import will succeed. That gives intermittent failures in the test suite (I don't know if it happens in pandas itself, but at least we are seeing it in the geopandas test suite: https://github.com/geopandas/geopandas/issues/2911)

Labels:
name: Testing
description: pandas testing functions or related to the test suite

Comments:
Owee, I'm MrMeeseeks, Look at me.

There seem to be a conflict, please backport manually. Here are approximate instructions:

1. Checkout backport branch and update it.

```
git checkout 2.0.x
git pull
```

2. Cherry pick the first parent branch of the this PR on top of the older branch:
```
git cherry-pick -x -m1 d9067ba2922501560e699161ae1c5d8eb62d939c
```

3. You will likely have some merge/cherry-pick conflict here, fix them and commit:

```
git commit -am 'Backport PR #53474: TST: fix import in test_hash_pandas_object extension test'
```

4. Push to a named branch:

```
git push YOURFORK 2.0.x:auto-backport-of-pr-53474-on-2.0.x
```

5. Create a PR against branch 2.0.x, I would have named this PR:

> "Backport PR #53474 on branch 2.0.x (TST: fix import in test_hash_pandas_object extension test)"

And apply the correct labels and milestones.

Congratulations — you did some good work! Hopefully your backport PR will be tested by the continuous integration and merged soon!

Remember to remove the `Still Needs Manual Backport` label once the PR gets merged.

If these instructions are inaccurate, feel free to [suggest an improvement](https://github.com/MeeseeksBox/MeeseeksDev).
TYP: Fix typing issues, mainly Something => IndexLabel
State: closed

Several typing annotations in Pandas are not correct. This patch tries to fix some of them.

The internal names in Pandas are not straight forward. An `IndexLabel` is used for one or multiple columns.

 * `value_counts` also works for a single column
 * `stack`/`unstack` can work with multiple levels at the same time

I have the feeling that there might be more such problems in the code. There are also some incorrect annotations in pandas-stubs mainly related to one/multiple index levels (refer to https://github.com/pandas-dev/pandas-stubs/issues/703 - even though I haven't had time for a PR over there)

Labels:

Comments:
I have a general request: I would like to have better names in general.

`Axis` (singular) is `int` or a `Literal['columns', ...]`. So far so good.
`Axes` (plural) is a `ListLike` and is NOT used for multiple axis but instead for `columns: Axes` - a list of columns (or rows)
`IndexLabel` seems to be used for one or multiple columns (or rows) - it is a `Sequence[Hashable]|Hashable`. So that means it does not support a `pd.Index`? 

There is also `Level = Hashable`. When functions take one or multiple levels, that makes it `Level | Sequence[Level]` - or `IndexLabel` again. Is that a good name or should there be a `LevelOrLevels` alias? (I hate "Or" in names but cannot come up with something better that is as clear)

It is not always clear to me which type to use where and when.


Just a general comment here...

The types used in the pandas code are meant to help with keeping the pandas code consistent. 

The types in pandas-stubs are meant for public consumption.

There are some inconsistencies in naming between the two projects.  See https://github.com/pandas-dev/pandas-stubs/issues/128 and https://github.com/pandas-dev/pandas-stubs/issues/161 for some discussion.

I'm not going to look at the details of this PR, but if your goal is to have better naming for use in type checking user code that calls pandas, it might be a better contribution to do something in the pandas-stubs project instead of in pandas, or you could do both.  

> I'm not going to look at the details of this PR, but if your goal is to have better naming for use in type checking user code that calls pandas

No, my goal is to fix wrong type hints in Pandas, specifically:
 * `value_counts` also works for a single column but the current type hints don't cover that
 * `stack`/`unstack` can work with multiple levels at the same time but the current type hints don't cover that

The naming is just an additional issue which makes it more difficult than it should be. I shouldn't have raised that topic here.
> No, my goal is to fix wrong type hints in Pandas, specifically:

Wrong type hints for pandas users?  Or for pandas developers?

If it is the former, then you should also look at `pandas-stubs` .  If it is the latter, then that's fine. 

If you are using the pandas source to type check your user code, then it will be incomplete as compared to using `pandas-stubs` .

Thanks @behrenhoff!
STYLE: Add ruff check for multiple with statements
State: closed

- [ ] closes #53436 
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).



Labels:
name: Code Style
description: Code style, linting, code_checks

Comments:
Thanks for the pull request, but it appears to have gone stale. If interested in continuing, please merge in the main branch, address any review comments and/or failing tests, and we can reopen.
DOC: Fixing EX01 - Added examples
State: closed

- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).

Towards https://github.com/pandas-dev/pandas/issues/37875


Labels:
name: Docs

Comments:
Could you fix the merge conflict?
Thanks @DeaMariaLeon
BUG: merge_asof raising KeyError for extension dtypes
State: closed

- [x] closes #52904
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/v2.1.0.rst` file if fixing a bug or adding a new feature.


Labels:
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode
name: ExtensionArray
description: Extending pandas with custom dtypes or arrays.

Comments:
Thanks @lukemanley 
@lukemanley thanks for fixing. I was wondering, do you expect it to work for timezone aware pyarrow timestamps eg `pa.timestamp("ns", "UTC")`

@0x26res - yes, tz-aware pyarrow timestamps should work in pandas 2.1 (to be released later this month):

```
In [1]: import pandas as pd

In [2]: import pyarrow as pa

In [3]: dtype = pd.ArrowDtype(pa.timestamp('ns', 'UTC'))

In [4]: df1 = pd.DataFrame({"a": [1, 3]}, index=pd.Index([1, 3], dtype=dtype))

In [5]: df2 = pd.DataFrame({"b": [0, 2]}, index=pd.Index([0, 2], dtype=dtype))

In [6]: pd.merge_asof(df1, df2, left_index=True, right_index=True)
Out[6]: 
                                     a  b
1970-01-01 00:00:00.000000001+00:00  1  0
1970-01-01 00:00:00.000000003+00:00  3  2
```
CLN: Remove deprecated numba functions
State: closed

- [ ] closes #53449 (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

The replacement for ``generated_jit`` is ``numba.extending.overload``.

That shouldn't be necessary tho IMO. 
Since the user function is always part of another jitted numba function, we can just mark the function as ``jitable`` and have it inherit the ``nopython``, ``nogil``, etc. of the outer function.

Labels:
name: Compat
description: pandas objects compatability with Numpy or Python functions
name: Clean
name: numba
description: numba-accelerated operations

Comments:
Thanks @lithomas1
ENH: Allow numba aggregations to return non-float64 results
State: closed

- [ ] xref #44952 (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Groupby
name: Dtype Conversions
description: Unexpected or buggy dtype conversions
name: Window
description: rolling, ewma, expanding
name: numba
description: numba-accelerated operations

Comments:
Could you also add a whatsnew note for 2.1?
There seems to be some flakiness with the benchmarks I added.
(Wasn't able to get an error message unfortunately).

I'll let this sit for a couple of days then, but other than that it should be good to go.

EDIT: Root caused, it was a timeout in the benchs.
ENH: xz compression_args df.to_pickle(path, compression={'method':'xz','pr…
State: closed

…eset': level})
level = 9

- [x] closes #52979 (Replace xxxx with the GitHub issue number)
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: IO Data
description: IO issues that don't fit into a more specific label

Comments:
Thank you for the PR. You definitely will need the following:
- tests
- update the documentation https://github.com/pandas-dev/pandas/blob/8cf4ab491b06274f555d1ea305c172354208747a/pandas/core/shared_docs.py#L462 (and for decompression below that)
- and a whatsnew entry: doc/source/whatsnew/v2.1.0.rst
Thank you for advices and maintaining pandas! Seems done with checks and docs. Do I need add any new tests or it's ready?
> Thank you for advices and maintaining pandas! Seems done with checks and docs. Do I need add any new tests or it's ready?

Yes, you will need a test. Probably in https://github.com/pandas-dev/pandas/blob/main/pandas/tests/io/test_compression.py I'm not sure how to best test that the keywords are being forwarded: maybe one test with either to_pickle/read_csv and just pass a valid argument and then another test that passes an invalid keyword that raises an error in lzma to ensure the keyword is forwarded to it.

edit: should probably test both to_pickle and read_csv just to cover one read/write method.
Thank you! Done. Seems now everything is ready for review.
Thanks @segatrade
DOC: PDEP6 page format update
State: closed

- [x] closes #53414
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).


Labels:
name: Web
description: pandas website

Comments:
No preview found for PR #53442. Did the docs build complete?
No preview found for PR #53442. Did the docs build complete?
@MarcoGorelli, here is the link to preview the change:

https://pandas.pydata.org/preview/53442/pdeps/0006-ban-upcasting.html

[77d7834](https://github.com/pandas-dev/pandas/pull/53442/commits/77d7834f9fe02caa52995b405446ccae348f0f4e) updates the list format for the occurrence which was mentioned in the issue.
[ddf551a](https://github.com/pandas-dev/pandas/pull/53442/commits/ddf551a6fcdad4ae5205d57f975a6a8fcf9670b2) updates the entire document format where bulleted lists are suggested.
[b976bc5](https://github.com/pandas-dev/pandas/pull/53442/commits/b976bc5501f5cb21833a843b570d3e6fd96ce44c) fixes unintended deletion.
[478e2e9](https://github.com/pandas-dev/pandas/pull/53442/commits/478e2e91e5f4a133ebcf4508ace2b4ef965a4fda) includes a `'` for (`Series`' dtype) in line 12 reflecting grammar rules expressed in line 92.
[9f345e8](https://github.com/pandas-dev/pandas/pull/53442/commits/9f345e8d2f23aff2ba4c70538590dcf4a627b9ed) includes formatting for numbered list, and punctuation for list objects where appropriate.
[fa8cf3a](https://github.com/pandas-dev/pandas/pull/53442/commits/fa8cf3a34fd1b273b2cf2639eefdf6158fc48bfb) includes proper formatting for nested list (2) in detailed description section of PDEP6.
[7722182](https://github.com/pandas-dev/pandas/pull/53442/commits/772218239ffd081d2e6a421838021edc76722763) includes final formatting changes utilizing local build, resulting in accurate rendering.
REF: partially decouple Period from Tick
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Refactor
description: Internal refactoring of code
name: Period
description: Period data type

Comments:
Thanks @jbrockmendel
DEPR: Deprecate use of un-supported numpy dt64/td64 dtype for pandas.array
State: closed

- [X] closes #53058
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [X] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Deprecate
description: Functionality to remove in pandas

Comments:
@jbrockmendel Would you be able to take a look at this PR and let me know if its good to go?
Oh, i see.  #53058 isnt about deprecating all dt64/td64 dtypes, just the unsupported units.  we support s, ms, us, ns.
In pd.array, at the end where we have checks for is_supported_dtype, do an 'else' statement and issue a deprecation warning
@jbrockmendel I should add a deprecation warning rather than a TypeError? I was thinking that we wanted to [raise an error](https://github.com/pandas-dev/pandas/issues/53058#issuecomment-1566289538) rather than a deprecation warning?

Is the image below what you're looking for? If we use an else statement I think that we'll be throwing a deprecation warning for other supported dtypes before we hit the return statement at the bottom.

 I had my td64/dt64 check up top. Are we okay with me adding an elif statement at the bottom and checking for the unsupported types?

![image](https://github.com/pandas-dev/pandas/assets/45905457/3ce1b9b3-6794-499d-819a-ff4f77c74e68)

@jbrockmendel PR is ready for inspection. Failing test does not appear to be related to PR
@jbrockmendel PR is ready for inspection
@jbrockmendel Fixed PR based on recommendations and pinging on green!
@jbrockmendel PR is ready for inspection. 

Failing tests are unrelated to PR. Doctest and code checks are passing
DEPR: inplace in GroupBy.fillna
State: closed

- [x] closes #48792 (Replace xxxx with the GitHub issue number)
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Groupby
name: Missing-data
description: np.nan, pd.NaT, pd.NA, dropna, isnull, interpolate
name: Deprecate
description: Functionality to remove in pandas
name: Transformations
description: e.g. cumsum, diff, rank

Comments:
Closing in favor of #53496
REF/CLN: func in core.apply
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

Replaces the conditional global currying with local currying where necessary. Also renames variables to use `func` consistently throughout `core.apply`.

cc @topper-123 

Labels:
name: Refactor
description: Internal refactoring of code
name: Clean
name: Apply
description: Apply, Aggregate, Transform, Map

Comments:
Thanks @rhshadrach 
Yes, this easier to follow. Nice one.