BrandonFors/pandas-github-issues huggingface dataset:
ENH/CoW: use lazy copy in set_axis method
State: closed

Copy-on-Write in `set_axis`
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
This PR is part of an effort to implement the Copy-on-Write implementation mentioned in #49473. More detail in this proposal https://docs.google.com/document/d/1ZCQ9mx3LBMy-nhwRl33_jgcvWo9IWdEfxDNQ2thyTb0/edit / with discussions in https://github.com/pandas-dev/pandas/issues/36195


Labels:
name: Copy / view semantics

Comments:
And you will also have to merge the main branch, to resolve the conflict from merging your other PR
@Th3nn3ss small reminder in case you have time to update for the minor comments above
sorry about the delay @jorisvandenbossche I was preoccupied with my research project. I have made some changes. Thanks for taking a look at my PR.
Thx @Th3nn3ss
PERF: df.groupby(categorical)
State: closed

- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/v2.0.0.rst` file if fixing a bug or adding a new feature.

Perf improvement in `Categorical.reorder_categories` with user facing performance improvements likely to be `DataFrame.groupby(categorical)`. 

```
import pandas as pd
import pandas._testing as tm

vals = pd.Series(tm.rands_array(10, 10**6), dtype="string")
df = pd.DataFrame({"cat": vals.astype("category")})

%timeit df.groupby("cat").size()

1.21 s ± 4.71 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)     <- main
15.1 ms ± 274 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)  <- PR
```


existing ASVs:
```
       before           after         ratio
     [d69e63d7]       [382245ad]
     <main>           <cat-reorder-categories>
-      3.91±0.1ms       3.08±0.1ms     0.79  groupby.Categories.time_groupby_nosort
-     3.54±0.04ms      2.31±0.02ms     0.65  groupby.Categories.time_groupby_extra_cat_nosort
-     3.85±0.05ms       2.39±0.1ms     0.62  groupby.Categories.time_groupby_ordered_nosort
-     2.05±0.01ms          514±6μs     0.25  groupby.Categories.time_groupby_extra_cat_sort
-      2.21±0.1ms          536±4μs     0.24  groupby.Categories.time_groupby_sort
-      2.23±0.1ms         523±30μs     0.23  groupby.Categories.time_groupby_ordered_sort
```

Labels:
name: Groupby
name: Performance
description: Memory or execution speed performance
name: Categorical
description: Categorical Data Type

Comments:
Wow, very nice!

One question, that's only semi-related to this: why do we even use `reorder_categories` in this groupby? it does seem unrelated especially as we're only using one dtype (if we were combining two dtypes, I would understand it).

Could we get an additional perf. improvement by avoiding `reorder_categories` in groupby in cases like this?
I'm going to guess the rationale for matching the original order of the categories is to keep the result categories dtype equivalent to the input categories dtype. I think we could add a short-circuit that checks to see if they happen to be equal already and if so skip `reorder_categories`. However, I just tried that locally and could not measure any noticeable improvement in performance. 
Ok. Thanks for trying it out. This performance improvement is already really great as is :-)
Thanks, @lukemanley. Great stuff.
API: Index(object_dtype_bool_ndarray) retain object dtype
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Dtype Conversions
description: Unexpected or buggy dtype conversions

Comments:
Just to solidify the "rule of thumb" where this is headed

```
pd.Index/Series(np.array([...], dtype=object), dtype=None) -> object
pd.Index/Series(np.array([...], dtype=object), dtype=other) == pd.Index/Series(np.array([...], dtype=object)).astype(other)
```

?
Thanks @jbrockmendel 
> Just to solidify the "rule of thumb" where this is headed [...]

Not quite.  In the `pd.Index/Series(ndarray_object)` case we'll continue to do inference on datetime, timedelta, interval, and period.

(and more generally id have no strong objection to doing inference on numeric/bool dtypes, just as long as we're consistent about it)
DEPR: Enforce DataFrame(list_with_categorical) deprecation
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Deprecate
description: Functionality to remove in pandas

Comments:
Merge conflict otherwise looks good
Thanks @jbrockmendel
REF: tighter typing in constructor functions
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Refactor
description: Internal refactoring of code

Comments:
Thanks @jbrockmendel
DEPR: Enforce deprecations
State: closed

Introduced in #43248, #33091, #45523

Labels:
name: Deprecate
description: Functionality to remove in pandas

Comments:
Failure is unrelated (and remaining check is the requirements-dev.txt install), so merging
This patch may have induced a potential regression. Please check the links below. If any ASVs are parameterized, the combinations of parameters that a regression has been detected appear as subbullets. This is a partially automated message.

Subsequent benchmarks may have skipped some commits. See the link below to see which commits are between the two benchmark runs where the regression was identified.

https://github.com/pandas-dev/pandas/compare/6122c7de128fce3a84d91ef91b9dc3a914531745...c0445543a10ae6c6abe5a3efc519fd9edcd2d276

 - [series_methods.ToFrame.time_to_frame](https://asv-runner.github.io/asv-collection/pandas/#series_methods.ToFrame.time_to_frame)
   - [dtype='Int64'; name=None](https://asv-runner.github.io/asv-collection/pandas/#series_methods.ToFrame.time_to_frame?p-dtype=%27Int64%27&p-name=None)
   - [dtype='category'; name=None](https://asv-runner.github.io/asv-collection/pandas/#series_methods.ToFrame.time_to_frame?p-dtype=%27category%27&p-name=None)
   - [dtype='datetime64[ns]'; name=None](https://asv-runner.github.io/asv-collection/pandas/#series_methods.ToFrame.time_to_frame?p-dtype=%27datetime64%5Bns%5D%27&p-name=None)
   - [dtype='int64'; name=None](https://asv-runner.github.io/asv-collection/pandas/#series_methods.ToFrame.time_to_frame?p-dtype=%27int64%27&p-name=None)

I think this looks fine, just the difference between `Index([name])` and `default_index(1)`, but wanted to be sure.
REF: simplify maybe_convert_objects
State: closed

Also avoids some memory allocations which is nice.

Labels:
name: Refactor
description: Internal refactoring of code

Comments:
Thanks @jbrockmendel 
I think this may have caused a slowdown here: https://asv-runner.github.io/asv-collection/pandas/#reshape.Explode.time_explode
what do you think the culprit is?
actually, looks to be https://github.com/pandas-dev/pandas/pull/49566 - and there is some discussion there about adding a fix for the perf
TYP:Replace union of subclasses with base class.
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [X] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

I am splitting this out of #49531 . In order to refactor the code in that PR, I had to change the return type of `pandasSQL_builder`, and that forced me to make additional changes in order to make mypy pass in CI:
* Line 581 (`pandas_sql.meta.reflect(bind=pandas_sql.connectable, only=[sql])`) was failing mypy, but it can be deleted since the table is reflected in `SQLDatabase.get_table`, when the `sqlalchemy.Table` object is created.
* I declared all required functions in `PandasSQL` and made the abstract.  I then made all the arguments agree between `PandasSQL` and subclasses.

Labels:
name: IO SQL
description: to_sql, read_sql, read_sql_query
name: Typing
description: type annotations, mypy/pyright type checking

Comments:
@mroeschke , is it ok to merge the PR's that I create out of #48576 one at a time, or do you want me to get all the PR's set up and ready to merge first?
Thank you @cdcadman !
API: stop special-casing SparseArray._quantile
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Sparse
description: Sparse Data Type
name: quantile
description: quantile method

Comments:
Thanks @jbrockmendel
BUG: Behaviour change in 1.5.0 when using Timedelta as Enum data type
State: closed

- [X] closes #48908
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Regression
description: Functionality that used to work in a prior pandas version
name: Timedelta
description: Timedelta data type

Comments:
pls add tests where appropriate
> pls add tests where appropriate

done
Can you add a note in the 1.5.2 file
thanks @krasch 
Oops, something went wrong applying the patch ... Please have a look at my logs.
PERF: MultiIndex.isin
State: closed

- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/v2.0.0.rst` file if fixing a bug or adding a new feature.

Improves perf for `MultiIndex.isin`.

```
       before           after         ratio
     [d47e0523]       [edd67b0a]
     <main>           <mi-isin> 
-     7.98±0.06ms      6.06±0.07ms     0.76  multiindex_object.Isin.time_isin_small('string')
-     5.70±0.07ms      4.02±0.03ms     0.71  multiindex_object.Isin.time_isin_small('int')
-     6.18±0.09ms      4.31±0.03ms     0.70  multiindex_object.Isin.time_isin_small('datetime')
-      42.7±0.5ms       6.73±0.2ms     0.16  multiindex_object.Isin.time_isin_large('string')
-      41.7±0.8ms       4.86±0.1ms     0.12  multiindex_object.Isin.time_isin_large('int')
-         136±2ms      4.87±0.07ms     0.04  multiindex_object.Isin.time_isin_large('datetime')
```

Labels:
name: Performance
description: Memory or execution speed performance
name: MultiIndex

Comments:
Nice find! Thanks @lukemanley
BUG: Series(index=[]) should have dtype=object
State: closed

- [X] closes #49573
- [X] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [X] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

No doc entry is needed, because this only fixes a corner case of #49342.

Labels:
name: Series
description: Series data structure
name: Constructors
description: Series/DataFrame/Index/pd.array Constructors

Comments:
The failed check looks unrelated.
Thanks for catching this. One question otherwise LGTM
Thanks @topper-123
STYLE: fix pylint consider-using-min-builtin warnings
State: closed

Related to https://github.com/pandas-dev/pandas/issues/48855
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Code Style
description: Code style, linting, code_checks

Comments:
Thanks @Moisan !
CLN: MultiIndex._values remove special casing for Timestamp.freq
State: closed

- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.

Following removal of deprecated `Timestamp.freq` in #49365, we can remove the special casing in `MultiIndex._values`.

Labels:
name: MultiIndex
name: Clean

Comments:
thx @lukemanley
BUG/PERF: MultiIndex.value_counts returning flat index
State: closed

- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/v2.0.0.rst` file if fixing a bug or adding a new feature.


I believe it is a bug that `MultiIndex.value_counts` returns a `Series` indexed by a flat index of tuples rather than a `MultiIndex`. By returning a flat index, we lose nullable dtypes, index names, etc.

This PR changes `MultiIndex.value_counts` to return a `Series` indexed by a `MultiIndex`. 

This also has some perf improvements:

```
import pandas as pd
import numpy as np

mi = pd.MultiIndex.from_product([np.arange(1000), np.arange(1000)])

%timeit mi.value_counts()

974 ms ± 28.9 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)  <- main
182 ms ± 10.2 ms per loop (mean ± std. dev. of 7 runs, 10 loops each) <- PR
```

Labels:
name: Performance
description: Memory or execution speed performance
name: MultiIndex

Comments:
> Two things:
> 
> Can we send this down a different path directly from MultiIndex.value_counts and what happens when bins is not None? Edit: Missed the normalise part at the end, so forget my comment about sending it down a different path

I also thought this path made sense since `algos.union_with_duplicates` calls `algos.value_counts` and will benefit from the performance improvements. The value_counts calls are currently bottlenecks when calling `union_with_duplicates` with MultiIndexes.

`bins` is documented to only work for numeric data. Passing anything other than None for a MultiIndex raises - this PR does not change that behavior. The error message could be improved as it simply raises when trying to construct a `Series` from a `MultiIndex`. 


Thx, could you open an issue about improving the error message?
> Thx, could you open an issue about improving the error message?

will do
thx @lukemanley
ENH/CoW: use lazy copy in set_index method
State: closed

Copy-on-Write in `set_index`
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
This PR is part of an effort to implement the Copy-on-Write implementation mentioned in #49473. More detail in this proposal https://docs.google.com/document/d/1ZCQ9mx3LBMy-nhwRl33_jgcvWo9IWdEfxDNQ2thyTb0/edit / with discussions in https://github.com/pandas-dev/pandas/issues/36195

Labels:
name: Copy / view semantics

Comments:
Thanks!
for gh-49508 changing Doc for DataFrame.astype
State: closed

added the series in input in the doc of DataFrame.astype

closes #49508 



Labels:
name: Docs

Comments:
> Also appears to be some code check failures

Thanks I'll fix them up
Done the changes sir
Looks to be some validation errors:

```
Error: /home/runner/work/pandas/pandas/pandas/core/generic.py:5999:PR06:pandas.DataFrame.astype:Parameter "dtype" type should use "str" instead of "string"
Error: /home/runner/work/pandas/pandas/pandas/core/generic.py:5999:PR06:pandas.Series.astype:Parameter "dtype" type should use "str" instead of "string"
```
> Looks to be some validation errors:
> 
> ```
> Error: /home/runner/work/pandas/pandas/pandas/core/generic.py:5999:PR06:pandas.DataFrame.astype:Parameter "dtype" type should use "str" instead of "string"
> Error: /home/runner/work/pandas/pandas/pandas/core/generic.py:5999:PR06:pandas.Series.astype:Parameter "dtype" type should use "str" instead of "string"
> ```

Sir I didn't understand how can a validation error come from line 5999 when I did not even change the line, seems strange.
This is a shared docstring so it's used by 2 other functions. I think `string` should be be `str` as mentioned in the message
> This is a shared docstring so it's used by 2 other functions. I think `string` should be be `str` as mentioned in the message

thanls sir done the changes
Thanks @ramvikrams 
> Thanks @ramvikrams

Thank you sir for your help all along
ENH/CoW: use lazy copy on replace method
State: closed

- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
This PR is part of an effort to implement the Copy-on-Write implementation mentioned in #49473. More detail in this proposal https://docs.google.com/document/d/1ZCQ9mx3LBMy-nhwRl33_jgcvWo9IWdEfxDNQ2thyTb0/edit / with discussions in https://github.com/pandas-dev/pandas/issues/36195

Labels:
name: Copy / view semantics
name: Stale

Comments:
@jorisvandenbossche I will need some clarification to help me handle CoW in `replace`. It feels like `replace` should return a copy.
`replace` is probably a difficult case, because it has a quite complex implementation (eg https://github.com/pandas-dev/pandas/blob/5a36b5f5273f1c19fd0c9b6ca13d985a25316649/pandas/core/internals/blocks.py#L526-L777, and that's only part of it). 
I think for replace there would be room for avoiding an actual column for those columns where no value actually gets replaced. But that will require going through the implementation to identify those cases. 

I was going to suggest that it might be easier to start with another method (and I can give some suggestions), but I see you already started with `set_index` as well! Will take a look at that.
This pull request is stale because it has been open for thirty days with no activity. Please [update](https://pandas.pydata.org/pandas-docs/stable/development/contributing.html#updating-your-pull-request) and respond to this comment if you're still interested in working on this.
Thanks for the pull request, but it appears to have gone stale. If interested in continuing, please merge in the main branch, address any review comments and/or failing tests, and we can reopen.