BrandonFors/pandas-github-issues huggingface dataset:
BUG: describe not returning ArrowDtype
State: closed

- [ ] closes #52427 (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Dtype Conversions
description: Unexpected or buggy dtype conversions
name: Algos
description: Non-arithmetic algos: value_counts, factorize, sorting, isin, clip, shift, diff
name: Arrow
description: pyarrow functionality

Comments:
@meeseeksdev backport 2.0.x
Owee, I'm MrMeeseeks, Look at me.

There seem to be a conflict, please backport manually. Here are approximate instructions:

1. Checkout backport branch and update it.

```
git checkout 2.0.x
git pull
```

2. Cherry pick the first parent branch of the this PR on top of the older branch:
```
git cherry-pick -x -m1 4d64a8a057117a8b63d502e8da934a148af489cd
```

3. You will likely have some merge/cherry-pick conflict here, fix them and commit:

```
git commit -am 'Backport PR #52470: BUG: describe not returning ArrowDtype'
```

4. Push to a named branch:

```
git push YOURFORK 2.0.x:auto-backport-of-pr-52470-on-2.0.x
```

5. Create a PR against branch 2.0.x, I would have named this PR:

> "Backport PR #52470 on branch 2.0.x (BUG: describe not returning ArrowDtype)"

And apply the correct labels and milestones.

Congratulations — you did some good work! Hopefully your backport PR will be tested by the continuous integration and merged soon!

Remember to remove the `Still Needs Manual Backport` label once the PR gets merged.

If these instructions are inaccurate, feel free to [suggest an improvement](https://github.com/MeeseeksBox/MeeseeksDev).
BUG: segfault for null dtype in to_numpy
State: closed

- [x] closes #52443 (Replace xxxx with the GitHub issue number)
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Dtype Conversions
description: Unexpected or buggy dtype conversions
name: Arrow
description: pyarrow functionality

Comments:
Owee, I'm MrMeeseeks, Look at me.

There seem to be a conflict, please backport manually. Here are approximate instructions:

1. Checkout backport branch and update it.

```
git checkout 2.0.x
git pull
```

2. Cherry pick the first parent branch of the this PR on top of the older branch:
```
git cherry-pick -x -m1 7974ad0ba191e0a6b6bda5e0beb9669d0391b5af
```

3. You will likely have some merge/cherry-pick conflict here, fix them and commit:

```
git commit -am 'Backport PR #52466: BUG: segfault for null dtype in to_numpy'
```

4. Push to a named branch:

```
git push YOURFORK 2.0.x:auto-backport-of-pr-52466-on-2.0.x
```

5. Create a PR against branch 2.0.x, I would have named this PR:

> "Backport PR #52466 on branch 2.0.x (BUG: segfault for null dtype in to_numpy)"

And apply the correct labels and milestones.

Congratulations — you did some good work! Hopefully your backport PR will be tested by the continuous integration and merged soon!

Remember to remove the `Still Needs Manual Backport` label once the PR gets merged.

If these instructions are inaccurate, feel free to [suggest an improvement](https://github.com/MeeseeksBox/MeeseeksDev).
BUG: Series([ints], dtype=td[non-nano]) not respecting unit
State: closed

- [x] closes #48312 (Replace xxxx with the GitHub issue number)
- [x] closes #52457 (Replace xxxx with the GitHub issue number) 
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

I think this should be backported since non-nano support is a new feature in 2.0

Labels:
name: Timedelta
description: Timedelta data type
name: Mothballed
description: Temporarily-closed PR the author plans to return to
name: Non-Nano
description: datetime64/timedelta64 with non-nanosecond resolution

Comments:
So I've had a branch aimed at this issue sitting on the backburner for a while now and been indecisive about how to handle this.  In particular this makes `Series(values, dtype=dtype)` inconsistent with `Series(values.astype(object), dtype=dtype)` i think
I was thinking this qualified as a "bug fix" since we officially support non-nano units.
would it be helpful if i pushed the WIP branch i mentioned to see if there is anything there worth cannibalizing?
> would it be helpful if i pushed the WIP branch i mentioned to see if there is anything there worth cannibalizing?

Definitely! With the test failures maybe the scope of this change is larger than I expected
Not making much progress with this so closing for now
REF: remove freq arg from PeriodArray constructor
State: closed

Try to match the other constructors more closely

Labels:
name: Period
description: Period data type

Comments:
Does this need to be deprecated since it's somewhat public? https://pandas.pydata.org/docs/reference/api/pandas.arrays.PeriodArray.html


> Does this need to be deprecated since it's somewhat public?

OK, will update
updated as a deprecation + green
Thanks @jbrockmendel
ENH: NDFrame.axis_ops
State: closed

Refactor out NDFrame methods that operate only on axes into an `obj.axis_ops` namespace.  I'd want to do this in conjunction with a deprecation of the methods themselves xref #52110

Labels:
name: Refactor
description: Internal refactoring of code
name: Index
description: Related to the Index class or subclasses

Comments:
@phofl updated to include more underscores
> Could you also ensure the API reference page advertises e.g. DataFrame.axis_ops.foo instead of DataFrame.foo?

I'm not clear on how that works.  For e.g. swaplevel would I remove the entry for Series.swaplevel in source/reference/series.rst? I could add Series.axis_ops in the same place but will adding Series.axis_ops.swap_level render?

> I could add Series.axis_ops in the same place but will adding Series.axis_ops.swap_level render?

I think so. I think we have an `autosummary` template that renders our accessors (like Series.str.foo)
An alternative: just deprecate tz_localize, tz_convert, to_timestamp, to_period and leave the others as-is
Yeah actually that's a good point. if there's an `index.foo()` alternative might as well deprecate the `DataFrame.foo()` behavior
Closing for lack of interest, will do individual deprecations in the subset of cases where that is feasible
Reopening, will merge main and push shortly.
FWIW, while I understand the reason for adding something like this, I would not be very enthusiastic about an `axis_ops` accessor (from a user design point of view, it feels rather awkward)
DEPR: to_pydatetime return Index[object]
State: closed

- [x] closes #20306 (Replace xxxx with the GitHub issue number)
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Datetime
description: Datetime data dtype
name: Deprecate
description: Functionality to remove in pandas

Comments:
Thanks @jbrockmendel 
Should this have been done for `to_pytimedelta` too?
BUG: tm.assert_series_equalcheck_dtype=False) with different resos
State: closed

- [x] closes #52449 (Replace xxxx with the GitHub issue number)
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Non-Nano
description: datetime64/timedelta64 with non-nanosecond resolution

Comments:
thx @mroeschke 
Owee, I'm MrMeeseeks, Look at me.

There seem to be a conflict, please backport manually. Here are approximate instructions:

1. Checkout backport branch and update it.

```
git checkout 2.0.x
git pull
```

2. Cherry pick the first parent branch of the this PR on top of the older branch:
```
git cherry-pick -x -m1 b637ad2a7d75c85d39f4e042e960bb82c76e518a
```

3. You will likely have some merge/cherry-pick conflict here, fix them and commit:

```
git commit -am 'Backport PR #52458: BUG: tm.assert_series_equalcheck_dtype=False) with different resos'
```

4. Push to a named branch:

```
git push YOURFORK 2.0.x:auto-backport-of-pr-52458-on-2.0.x
```

5. Create a PR against branch 2.0.x, I would have named this PR:

> "Backport PR #52458 on branch 2.0.x (BUG: tm.assert_series_equalcheck_dtype=False) with different resos)"

And apply the correct labels and milestones.

Congratulations — you did some good work! Hopefully your backport PR will be tested by the continuous integration and merged soon!

Remember to remove the `Still Needs Manual Backport` label once the PR gets merged.

If these instructions are inaccurate, feel free to [suggest an improvement](https://github.com/MeeseeksBox/MeeseeksDev).
DOC: update pivot docs for version pandas 2.x
State: closed

- [x] closes #52436  (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

The doc for the `index` parameter of the `pivot` method was incorrect.


Labels:
name: Docs

Comments:
Thanks @srkds 
Thanks, @mroeschke 👍😀 for approving and merging the PR.
Added TypeAlias
State: closed

- [ ] closes #52448 
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).


Labels:
name: Code Style
description: Code style, linting, code_checks
name: Typing
description: type annotations, mypy/pyright type checking

Comments:
After running `pre-commit run ruff --all-files` , getting around 26 errors related to line formatting.  See below
![5C39E5C9-9647-45EC-B6CF-34A6962325EF](https://user-images.githubusercontent.com/110608231/230336864-01dd28e1-b19d-4b5e-96a2-9e66c5a51976.jpeg)

Do I need to fix this in this Issue itself, or is there going to be a seperate one.?
they'd need to be fixed as part of the same PR, CI need to be green before merging

what's causing the long lines now?
Not sure whats causing this, as lines are within limit of 88 chars. Breaking up the line solves this issue though.
if you push a commit I can take a look
I think this is a false-positive in ruff itself - I've reported it to them here: https://github.com/charliermarsh/ruff/issues/3902

let's put this on hold until they get back to us
> I think this is a false-positive in ruff itself - I've reported it to them here: [charliermarsh/ruff#3902](https://github.com/charliermarsh/ruff/issues/3902)
> 
> let's put this on hold until they get back to us

from https://github.com/charliermarsh/ruff/issues/3902 , it looks like we may just have to further split the lines
REF: move methods from base to Index
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Refactor
description: Internal refactoring of code
name: Index
description: Related to the Index class or subclasses
name: Series
description: Series data structure

Comments:
I've merged #52428, so I think this should be rebased to avoid merge issues here. 
Thanks @jbrockmendel
BUG: error when unstacking in DataFrameGroupby.apply
State: closed

- [x] closes #52444
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Bug
name: Groupby
name: Apply
description: Apply, Aggregate, Transform, Map

Comments:
Thanks @topper-123
This seems to solve the issue, although I am confused as to why there were two identical if and else statements before my commit. 
State: closed

- [x] closes #52435 (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:

Comments:
PR
#52405 Issue :Rename module from_dataframe to _from_dataframe
State: closed

- [x] closes #52405 (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:

Comments:
thanks @sampanacharya 

there's already a PR open, sorry, closing then. you're welcome to work on other issues
PERF: Series.to_numpy with float dtype and na_value=np.nan
State: closed

- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/v2.1.0.rst` file if fixing a bug or adding a new feature.

Avoids filling na values with `np.nan` if the array is already a numpy float dtype. (`DataFrame` has a similar optimization in `BlockManager.as_array`)

```
import pandas as pd
import numpy as np

ser = pd.Series(np.random.randn(10_000_000))

%timeit ser.to_numpy(dtype="float64", na_value=np.nan, copy=False)

# 22.5 ms ± 2.56 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)          -> main
# 1.97 µs ± 24.5 ns per loop (mean ± std. dev. of 7 runs, 1,000,000 loops each)   -> PR
```





ASV added:
```
       before           after         ratio
     [3ce07cb4]       [020fb14a]
     <main>           <series-to-numpy-float-nan>
-     1.02±0.02ms      2.12±0.03μs     0.00  series_methods.ToNumpy.time_to_numpy_float_with_nan

SOME BENCHMARKS HAVE CHANGED SIGNIFICANTLY.
PERFORMANCE INCREASED.
```

Labels:
name: Performance
description: Memory or execution speed performance
name: Series
description: Series data structure

Comments:
Actually, what about `FloatDtype(64|32)`?
> Actually, what about `FloatDtype(64|32)`?

EA dtypes dispatch to their own implementation of `to_numpy`. In the case of `FloatDtype(64|32)`, I'm not sure there is anything to be done. `BaseMaskedArray.to_numpy` needs to combine the data array and the mask array when NAs exist.
Thanks @lukemanley
REF: define reductions non-dynamically
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Refactor
description: Internal refactoring of code
name: DataFrame
description: DataFrame data structure
name: Series
description: Series data structure

Comments:
Thanks @jbrockmendel
API: Series/DataFrame from empty dict should have RangeIndex
State: closed

- [x] closes #52404
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Constructors
description: Series/DataFrame/Index/pd.array Constructors

Comments:
Small ping.
Thanks @topper-123
PERF: avoid is_dtype_equal
State: closed

This gets rid of all our non-test usages.

Labels:
name: Performance
description: Memory or execution speed performance

Comments:
Will need to wait until we bump our minimum numpy version, not sure when exactly numpy fixed `np.dtype.__eq__` to not raise
mothballing to clear the queue
Looks like this is now viable since we've bumped our min numpy version
Thanks @jbrockmendel
TYP/TST: tests/mypy complaints failing locally
State: closed

This un-xfails-on-mac a test that is passing locally.  Also suppresses some mypy complaints.  I'm still complaints:

```
pandas/io/formats/printing.py:266: error: Incompatible types in assignment (expression has type "Tuple[Type[Dict[Any, Any]]]", base class "BaseFormatter" defined the type as "Type[str]")  [assignment]
pandas/core/dtypes/dtypes.py:865: error: Incompatible types in assignment (expression has type "Callable[[PeriodDtypeBase], int]", base class "PandasExtensionDtype" defined the type as "Callable[[PandasExtensionDtype], int]")  [assignment]
```

that I haven't found a solution for

Labels:
name: Typing
description: type annotations, mypy/pyright type checking

Comments:
Thanks @jbrockmendel (can follow up with the other typings maybe if it makes things simpler)
PERF: lazify IO imports
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:

Comments:
generally in favour - do you have an estimate of how much this helps?
>  do you have an estimate of how much this helps?

The measurements with `python -X importtime -c "import pandas as pd"` are extremely noisy.  I eyeball it at roughly 511ms in main 472ms in this branch.

The next big chunk available would be to delay core.groupby import, which is 86ms.  Doing that requires some yak-shaving bc core.window imports it, so that needs to be made lazy (and docstrings moved xref #51823).  Also dask references pd.core.groupby.groupby.something_i_forget_what at import-time to do a docstring pinning, so we'd need to do something about that.

Following that is some smaller-but-easier stuff: stdlib imports like shutil and stuff in pd.compat.compressors that are relatively expensive, dateutil (xref #52659).

The big ones are pyarrow (87ms) and numpy (86ms).  making numpy import lazy is probably not worth the effort (though asking numpy to lazify parts of its own imports might be worthwhile).  I'd like to see the pyarrow import made lazy, especially if we don't make it required.
No clear interest here, closing.
Rename module from_dataframe to _from_dataframe to fix issue #52405
State: closed

- [x] closes #52405
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Closing Candidate
description: May be closeable, needs more eyeballs
name: Interchange
description: Dataframe Interchange Protocol

Comments:
closing then, but thanks for your PR, and you're more than welcomme to work on other issues