BrandonFors/pandas-github-issues huggingface dataset:
BUG: Series(nullable).tolist() -> numpy scalars
State: closed

- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/v2.0.0.rst` file if fixing a bug or adding a new feature.

`Series.tolist` documentation states: 
> These are each a scalar type, which is a Python scalar (for str, int, float) or a pandas scalar (for Timestamp/Timedelta/Interval/Period)

Nullable dtype behavior seems to be inconsistent with the docstring and other dtypes:

main:
```
import pandas as pd

type(pd.Series([1], dtype="Int64").tolist()[0])             # numpy.int64
type(pd.Series([1], dtype="int64").tolist()[0])             # int
type(pd.Series([1], dtype="int64[pyarrow]").tolist()[0])    # int
```

PR:
```
import pandas as pd

type(pd.Series([1], dtype="Int64").tolist()[0])             # int
type(pd.Series([1], dtype="int64").tolist()[0])             # int
type(pd.Series([1], dtype="int64[pyarrow]").tolist()[0])    # int
```

The fix also provides an improvement in performance:

```
import pandas as pd
import numpy as np

ser = pd.Series(np.arange(10**6), dtype="Int64")

%timeit ser.tolist()

# 45.5 ms ± 980 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)  <- main
# 28.5 ms ± 229 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)  <- PR
```





Labels:
name: Bug
name: ExtensionArray
description: Extending pandas with custom dtypes or arrays.

Comments:
hmm i seem to recall that we very specifically have python scalers in all of these list conversion methods (eg iter as well)

i would be surprised this doesn't break many things 

edit: i think i misread your post
you are making things consistent ok!
> Could you profile this for pyarrow as well and with NA?

pyarrow doesn't hit this method. It hits ExtensionArray.tolist. I have a separate PR in the works to make that faster.

Here is this PR profiled with NA:

```
import pandas as pd
import numpy as np

ser = pd.Series(np.arange(10**6), dtype="Int64")
ser.iloc[::2] = pd.NA

%timeit ser.tolist()

# 92.9 ms ± 1.72 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)  <- main
# 59.5 ms ± 4.72 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)  <- PR
```


> edit: i think i misread your post
you are making things consistent ok!

Yes, sorry my wording should have been more clear. This makes nullable return python scalars which is consistent with other dtypes
Thx, missed the file name. good that casting to object is still faster
Could you merge main? We got some deprecation warnings from numpy that are fixed on main
thx @lukemanley
CI: Remove deprecated numpy dtype aliases
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: CI
description: Continuous Integration
name: Dependencies
description: Required and optional dependencies

Comments:
Merging to get ci green
CLN: move coverage config to pyproject.toml
State: closed


Labels:
name: Clean

Comments:
is all of setup.cfg eventually going to move to pyproject.toml?
> is all of setup.cfg eventually going to move to pyproject.toml?

Yes.
> > is all of setup.cfg eventually going to move to pyproject.toml?
> 
> Yes.

flake8 might still be an exception for a while 😄
BUG: MultiIndex.join losing dtype
State: closed

- [x] closes #49830 (Replace xxxx with the GitHub issue number)
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.#

Sits on top of #49847


Labels:
name: Bug
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode
name: MultiIndex

Comments:
Added the test
Thanks @phofl
ENH: Add support for ea dtypes in merge
State: closed

- [x] closes #44240 (Replace xxxx with the GitHub issue number)
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Enhancement
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode
name: NA - MaskedArrays
description: Related to pd.NA and nullable extension arrays

Comments:
ASV failures might be real


```
2022-11-24T01:09:56.6735358Z [ 30.77%] ··· join_merge.MergeEA.time_merge                          2/10 failed
2022-11-24T01:09:56.6736230Z [ 30.77%] ··· ========= ==========
2022-11-24T01:09:56.6738310Z                 dtype             
2022-11-24T01:09:56.6738758Z               --------- ----------
2022-11-24T01:09:56.6739260Z                 Int64    12.8±0ms 
2022-11-24T01:09:56.6740223Z                 Int32    12.5±0ms 
2022-11-24T01:09:56.6740600Z                 Int16    11.4±0ms 
2022-11-24T01:09:56.6740850Z                  Int8     failed  
2022-11-24T01:09:56.6741171Z                 UInt64   11.9±0ms 
2022-11-24T01:09:56.6741481Z                 UInt32   12.8±0ms 
2022-11-24T01:09:56.6741774Z                 UInt16   11.6±0ms 
2022-11-24T01:09:56.6742021Z                 UInt8     failed  
2022-11-24T01:09:56.6742334Z                Float64   12.6±0ms 
2022-11-24T01:09:56.6742655Z                Float32   13.1±0ms 
2022-11-24T01:09:56.6742899Z               ========= ==========
```
Oh interesting, I’ll rerun them locally to check, they were passing at some point
overflow error, removed them since a useful test is not possible
```
       before           after         ratio
     [14dc069f]       [38d558c6]
-      5.32±0.2ms      2.44±0.04ms     0.46  join_merge.MergeEA.time_merge('UInt64')
-      5.23±0.3ms      2.26±0.06ms     0.43  join_merge.MergeEA.time_merge('Float64')
-     5.38±0.05ms      2.25±0.02ms     0.42  join_merge.MergeEA.time_merge('UInt32')
-      5.32±0.3ms      2.19±0.03ms     0.41  join_merge.MergeEA.time_merge('Float32')
-      5.55±0.5ms      2.28±0.08ms     0.41  join_merge.MergeEA.time_merge('Int64')
-     5.39±0.08ms      2.15±0.01ms     0.40  join_merge.MergeEA.time_merge('UInt16')
-      5.51±0.5ms      2.07±0.02ms     0.38  join_merge.MergeEA.time_merge('Int32')
-      5.70±0.5ms      2.06±0.04ms     0.36  join_merge.MergeEA.time_merge('Int16')
```
Thanks @phofl 
Thx, will open a pr to refactor tomorrow
BUG: constructors
State: closed

- [x] closes #49292 (Replace xxxx with the GitHub issue number)
- [x] closes #44917 (Replace xxxx with the GitHub issue number)
- [x] closes #49309 (Replace xxxx with the GitHub issue number)
- [x] closes #41805 (Replace xxxx with the GitHub issue number)
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Constructors
description: Series/DataFrame/Index/pd.array Constructors

Comments:
Thanks @jbrockmendel
CI/BENCH: Make CategoricalIndexIndexing benchmark less flaky
State: closed

Resizing `data_unique` to be the same length as `data` which maybe is some source of flakiness since `data_unique` was 3x `data`


Labels:
name: Unreliable Test
description: Unit tests that occasionally fail
name: Benchmark
description: Performance (ASV) benchmarks

Comments:
I'll merge in main maybe 2 more times just to confirm. 
might be quicker to just re-run the code-checks jobs

![image](https://user-images.githubusercontent.com/33491632/204044360-b5bbe815-2264-442a-b426-1a5c54779ac8.png)

I restarted the job and it still passed, so let's ship this, if it fails again we can revisit. Thanks @mroeschke !
API: series int-slicing always positional
State: closed

- [x] closes #49612 (Replace xxxx with the GitHub issue number)
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves

Comments:
One more merge conflict otherwise good to merge
Another conflict but feel free to merge when fixed
Discussed on the dev call yesterday, decided to just revert the deprecation in 1.5.3.  closing.
Backport PR #49864 on branch 1.5.x (REGR: DataFrameGroupBy.transform with reducer and as_index=False returns null values)
State: closed

Backport PR #49864: REGR: DataFrameGroupBy.transform with reducer and as_index=False returns null values

Labels:
name: Bug
name: Groupby
name: Regression
description: Functionality that used to work in a prior pandas version
name: Apply
description: Apply, Aggregate, Transform, Map

Comments:
cc @rhshadrach could you have a look?
REF factor out parse_pydatetime from array_to_datetime
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

Precursor to https://github.com/pandas-dev/pandas/issues/49298 , for which this function will be used in `strptime.pyx` too

Labels:
name: Datetime
description: Datetime data dtype
name: Clean

Comments:
thanks for your reviews and explanations!

merging then to take the next step towards https://github.com/pandas-dev/pandas/issues/49298
TYP: add missing `py.typed` marker
State: closed

I'd like to use `mypy` to check type hint in my project. But it's failed with following error (apache/iceberg#6254)
![image](https://user-images.githubusercontent.com/6848311/203572069-1bd0d889-e081-447a-b43a-a95a494d29e1.png)

`pandas` seem support for type hint already, however, because of missing [`py.typed` marker](https://mypy.readthedocs.io/en/stable/running_mypy.html#missing-imports), `mypy` can't analyze this package and report the error.

Labels:

Comments:
xref https://github.com/pandas-dev/pandas/issues/28142. cc @Dr-Irv on whether we're ready to add this or not yet.
We should not do this.  If people want to get type checking with pandas, they should install `pandas-stubs`
> We should not do this. If people want to get type checking with pandas, they should install `pandas-stubs`

thanks @Dr-Irv - closing then, but thanks @dungdm93 for your PR
Thanks @Dr-Irv, but is their any reason why we should not use builtin type checking?
> Thanks @Dr-Irv, but is their any reason why we should not use builtin type checking?

There are a few reasons:
1) The types inside of pandas are not complete.
2) The types inside of pandas are meant more for internal type checking of the code, and less for users of pandas. 
3) Many of the API functions are not typed in the pandas source.

That's why we created pandas-stubs.  It's meant to support type checking of user code, and we can do a LOT more checking there than is possible within the pandas code itself.  We also have a testing mechanism that tests the validity of those type stubs.

Install pandas-stubs and use it.  You'll be glad you did!
REGR: DataFrameGroupBy.transform with reducer and as_index=False returns null values
State: closed

- [x] closes #49834 (Replace xxxx with the GitHub issue number)
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Bug
name: Groupby
name: Regression
description: Functionality that used to work in a prior pandas version
name: Apply
description: Apply, Aggregate, Transform, Map

Comments:
Thanks @rhshadrach (ASV failure unrelated)
CLN Simplify tz / utc arguments
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

Noticed while looking into https://github.com/pandas-dev/pandas/issues/49298

There's quite a few places where `tz` can only be `'utc'` or `None`, which is confusing. Might as well just use the boolean `utc`

Labels:
name: Datetime
description: Datetime data dtype
name: Clean

Comments:
thanks @MarcoGorelli
DOC: Add release process to the docs
State: closed

This was in a branch in the release repo, and not fully updated. I expect to automate some of the tasks shortly, but having the current release process here while things are improved is still helpful.

The final testing before the release is not added here. I think that's the first thing to move to the CI.

CC: @lithomas1 @mroeschke 

Labels:
name: Docs

Comments:
Thanks @lithomas1 and @mroeschke, great feedback. All comments should be addressed now.
Awesome thanks @datapythonista
fixed bug in tz_localize to throw an error if the argument to the ambiguous parameter is not one of the 4 correct options
State: closed

- [x] closes #49565 
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:

Comments:
closing as per https://github.com/pandas-dev/pandas/issues/49565#issuecomment-1326611908, but thanks anyway for the pR
I believe the test cases were failing after I merged with the main branch to keep it up to date, but I fixed the issue and they should be passing now
PERF: BaseMaskedArray.__iter__
State: closed

- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/v2.0.0.rst` file if fixing a bug or adding a new feature.

Faster iteration through EA's backed by BaseMaskedArray.

ASV added for iterating through a `Series` backed by various dtypes:

```
       before           after         ratio
     [b6736b44]       [b9193932]
                      <masked-array-iter>
-      16.8±0.9ms      5.14±0.06ms     0.31  series_methods.Iter.time_iter('Float64')
-        17.2±1ms       5.15±0.1ms     0.30  series_methods.Iter.time_iter('Int64')
-        16.8±1ms      3.50±0.02ms     0.21  series_methods.Iter.time_iter('boolean')
```






Labels:
name: Performance
description: Memory or execution speed performance
name: ExtensionArray
description: Extending pandas with custom dtypes or arrays.

Comments:
thanks @lukemanley
48855 style fix pylint issues
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Code Style
description: Code style, linting, code_checks

Comments:
thanks @seanjedi  - as mentioned [here](https://github.com/pandas-dev/pandas/issues/48855#issuecomment-1324442339), this check seems to be all false-positive, so it'd be better to keep it turned off to not risk annoying contributors

closing then, but thanks anyway for your PR!
BUG: MultiIndex.putmask losing ea dtype
State: closed

- [x] xref #49830 (Replace xxxx with the GitHub issue number)
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

This is something we have to fix before we can address the issue

Its also significantly faster

```
       before           after         ratio
     [80527f48]       [b73beee0]
                      <49830>   
-        68.3±1ms      2.26±0.04ms     0.03  multiindex_object.Putmask.time_putmask_all_different
-      72.4±0.6ms      2.20±0.03ms     0.03  multiindex_object.Putmask.time_putmask
```

Labels:
name: MultiIndex

Comments:
failure unrelated
Thanks @phofl
STYLE: fix pylint redefined-outer-name warning for file
State: closed

Associated with [#49656](https://github.com/pandas-dev/pandas/issues/49656).
Fixes pylint warnings for `redefined-outer-name` on the following files:

`pandas/core/tools/datetimes.py`



- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).


Labels:
name: Code Style
description: Code style, linting, code_checks

Comments:
no need to keep updating the branch - this is probably fine, let's let the checks run and I'll merge later
You are welcome! Thank you for the opportunity to contribute!
DEPR/API: DataFrame.shift(axis=1, fill_value=inty)
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Algos
description: Non-arithmetic algos: value_counts, factorize, sorting, isin, clip, shift, diff
name: Deprecate
description: Functionality to remove in pandas

Comments:
Thanks @jbrockmendel
PERF: Fix performance regression in isin for empty values
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

Restores previous performance on my machine and gives us a speedup in other places
```
     [9f687cbc]       [17a3be81]
-     1.75±0.02μs        297±0.6ns     0.17  libs.InferDtype.time_infer_dtype_skipna('empty')
-      1.89±0.2μs        296±0.8ns     0.16  libs.InferDtype.time_infer_dtype('empty')
```

#49162

Labels:
name: Performance
description: Memory or execution speed performance
name: Regression
description: Functionality that used to work in a prior pandas version
name: Algos
description: Non-arithmetic algos: value_counts, factorize, sorting, isin, clip, shift, diff

Comments:
Cc @rhshadrach 
Thanks @phofl 
Oops, something went wrong applying the patch ... Please have a look at my logs.
@meeseeksdev backport 1.5.x