BrandonFors/pandas-github-issues huggingface dataset:
BUG/PERF: MaskedArray.__setitem__ validation
State: closed

- [x] closes #44172
- [x] closes #44186
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit) for how to run them
- [ ] whatsnew entry

Fixes a bunch of ways we currently allow invalid setting, but ATM does _not_ fix the non-scalar equivalents.

Perf example from #44172 OP
```
df = pd.DataFrame([[True, False, pd.NA] * 200] * 200, dtype = "object")
df2 = df.astype("boolean")

%timeit res = df2.apply(lambda row: row.count(), axis = 1)
3.9 s ± 192 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)  # <- main
886 ms ± 40.7 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)  # <- PR
```

Labels:
name: Performance
description: Memory or execution speed performance
name: NA - MaskedArrays
description: Related to pd.NA and nullable extension arrays

Comments:
what's the bug fix here?
> what's the bug fix here?

```
arr = pd.array([True, False])
arr[0] = pd.NaT  # <- fails to raise
```

> > what's the bug fix here?
> 
> ```
> arr = pd.array([True, False])
> arr[0] = pd.NaT  # <- fails to raise
> ```

kk prob worth a note in a followon
@jbrockmendel seems something related to setitem is breaking on master: https://dev.azure.com/pandas-dev/pandas/_build/results?buildId=72425&view=logs&j=e2350215-d973-503e-a41e-d287ff823797&t=751e993b-1954-5330-832a-a54bd1064f7d&l=44
yah just put up a PR that should fix it
REF: simplify Block.setitem
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [ ] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit) for how to run them
- [ ] whatsnew entry

Yak-shaving to get to a point where we can share Block.setitem with Block subclasses.

Labels:
name: Refactor
description: Internal refactoring of code

Comments:
https://github.com/pandas-dev/pandas/runs/4834282676?check_suite_focus=true look relevant
> https://github.com/pandas-dev/pandas/runs/4834282676?check_suite_focus=true look relevant

addressed + greenish
do we know what is wrong with the windows builds
> do we know what is wrong with the windows builds

no idea, seems like more than the usual number of timeouts today
DOC: whatsnew note for groupby.apply bugfix
State: closed

ref https://github.com/pandas-dev/pandas/issues/45315#issuecomment-1013655106
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit) for how to run them
- [x] whatsnew entry


Labels:
name: Docs

Comments:
thanks @rhshadrach 
@meeseeksdev backport 1.4.x
Something went wrong ... Please have a look at my logs.
I forgot to cc @jbrockmendel - wanted to make sure you're good with the description here. Can do a follow up.
this is a great idea
REG/REF: close file handles engine-independently in read_csv
State: closed

- [ ] closes #45384
- [ ] tests added / passed
- [ ] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit) for how to run them
- [ ] whatsnew entry

Open file handles in TextFileParser instead of in the read_csv engines. This makes it easier to close file handles in case of an  error.

It is probably better to merge #45388 (for 1.4.x) and then this PR for 1.5.

Labels:
name: IO CSV
description: read_csv, to_csv

Comments:
@twoertwein 

> It is probably better to merge #45388 (for 1.4.x) and then this PR for 1.5.

can we merge this to 1.4? i think its pretty comprehensive
Yes, can go in 1.4.0 Just wasn't sure whether there will be enough time for this.
thanks @twoertwein 
@meeseeksdev backport 1.4.x
Something went wrong ... Please have a look at my logs.
PERF: use groupby.transform fast path for DataFrame -> Series aggregations
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit) for how to run them
- [x] whatsnew entry


The groupby.transform fast path is currently only taken for DataFrame -> DataFrame operations. This PR makes use of the fast path for DataFrame -> Series operations. The performance impact is most observed for wide DataFrames. From what I can see, almost all benchmarks use tall DataFrames with only a few columns. I added an additional benchmark with a wide DataFrame to cover this case. 


```
import numpy as np
from pandas import DataFrame

n = 1000
df = DataFrame(
    np.random.randn(n, n),
    index=np.random.choice(range(10), n),
)

%timeit df.groupby(level=0).transform(lambda x: np.max(x, axis=0))

562 ms ± 7.93 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)   # <- main
76.2 ms ± 313 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)  # <- PR
```


Labels:
name: Groupby
name: Performance
description: Memory or execution speed performance

Comments:
thanks @lukemanley
Update the common.py file
State: closed

Solve the dtype issue in pandas
- [ ] closes #45345
- [ Update the pandas.dtype function at line no. 1779] whatsnew entry


Labels:
name: Dtype Conversions
description: Unexpected or buggy dtype conversions

Comments:
Hello @yashrajput0508! Thanks for opening this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:

* In the file [`pandas/core/dtypes/common.py`](https://github.com/pandas-dev/pandas/blob/5ab59b9eb4164969542f16fcf046043db18d85f9/pandas/core/dtypes/common.py):

> [Line 1778:1](https://github.com/pandas-dev/pandas/blob/5ab59b9eb4164969542f16fcf046043db18d85f9/pandas/core/dtypes/common.py#L1778): [W293](https://duckduckgo.com/?q=pep8%20W293) blank line contains whitespace
> [Line 1780:23](https://github.com/pandas-dev/pandas/blob/5ab59b9eb4164969542f16fcf046043db18d85f9/pandas/core/dtypes/common.py#L1780): [E225](https://duckduckgo.com/?q=pep8%20E225) missing whitespace around operator
> [Line 1780:38](https://github.com/pandas-dev/pandas/blob/5ab59b9eb4164969542f16fcf046043db18d85f9/pandas/core/dtypes/common.py#L1780): [E225](https://duckduckgo.com/?q=pep8%20E225) missing whitespace around operator
> [Line 1780:54](https://github.com/pandas-dev/pandas/blob/5ab59b9eb4164969542f16fcf046043db18d85f9/pandas/core/dtypes/common.py#L1780): [E225](https://duckduckgo.com/?q=pep8%20E225) missing whitespace around operator
> [Line 1780:70](https://github.com/pandas-dev/pandas/blob/5ab59b9eb4164969542f16fcf046043db18d85f9/pandas/core/dtypes/common.py#L1780): [E225](https://duckduckgo.com/?q=pep8%20E225) missing whitespace around operator
> [Line 1782:1](https://github.com/pandas-dev/pandas/blob/5ab59b9eb4164969542f16fcf046043db18d85f9/pandas/core/dtypes/common.py#L1782): [W293](https://duckduckgo.com/?q=pep8%20W293) blank line contains whitespace
> [Line 1783:1](https://github.com/pandas-dev/pandas/blob/5ab59b9eb4164969542f16fcf046043db18d85f9/pandas/core/dtypes/common.py#L1783): [W293](https://duckduckgo.com/?q=pep8%20W293) blank line contains whitespace


If this is supposed to fix a bug, it needs a test for that bug.  Also looks like it is breaking some existing tests.
Thanks for the PR @yashrajput0508. But the idea would be to add an example to the docstring that would be run as a doctest. If still interested in working on the docstring, please open a new PR after validating your change by running `ci/code_checks.sh doctests`
ENH: support 'out' keyword in Categorical.__array_ufunc__
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [ ] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit) for how to run them
- [ ] whatsnew entry


Labels:
name: ufuncs
description: __array_ufunc__ and __array_function__

Comments:
prob should add a whatsnew in a folllowup
REF: de-duplicate methods across nullable EA subclasses
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [ ] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit) for how to run them
- [ ] whatsnew entry


Labels:
name: Refactor
description: Internal refactoring of code
name: NA - MaskedArrays
description: Related to pd.NA and nullable extension arrays

Comments:
thanks!
REGR: DataFrame.reset_index with empty RangeIndex
State: closed

- [x] closes #45230
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit) for how to run them
- [ ] whatsnew entry


Labels:
name: Regression
description: Functionality that used to work in a prior pandas version

Comments:
@meeseeksdev backport 1.4.x
Something went wrong ... Please have a look at my logs.
PERF: avoid SQL MetaData reflection in init #45260
State: closed

- [x] closes #45260



Labels:
name: Performance
description: Memory or execution speed performance
name: IO SQL
description: to_sql, read_sql, read_sql_query

Comments:
Hello @fangchenli! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:



There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2022-01-15 20:36:30 UTC
> Thanks for the quick fix, looks good!
> 
> One comment: we also have a `tables` property on the `SQLDatabase`. It doesn't seem to be used internally in the `SQLDatabase` class itself, but to keep that property useful, we might need to reflect `meta` at that moment to actually get a list of tables? (no idea actually, just guessing)

Thanks for the comment. I'll take a look at the `tables` property later.
thanks @fangchenli very nice !
@meeseeksdev backport 1.4.x
Something went wrong ... Please have a look at my logs.
DOC: add version dropdown to the online doc site
State: closed

xref https://github.com/pandas-dev/pandas/issues/42722

This adds a version dropdown to our documentation site. 

Currently, it is added on the right side of the top nav bar (for the rest using the default styling from the upstream sphinx theme):

![Screenshot from 2022-01-14 17-28-30](https://user-images.githubusercontent.com/1020496/149550901-7680421d-2989-460c-9a78-0e4ecabfbf46.png)

(my mouse was hovering the "1.3.5 (stable") entry (not visible on the screenshot), which is why that entry has a different shade of grey)

An alternative location could be next to the logo (on the left side of the top nav bar)

The entries that are included are controlled by the `versions.json` file that is added.  
We will need to update that file after a new release. But, that file is not included in the docs itself, but lives in the `/web` for the main website, and since that is directly served from the main branch, we can always easily update that (so the update doesn't strictly need to happen before cutting the release to have it included). 

Labels:
name: Docs

Comments:
OK, I simplified it to show just the major.minor version, and to just use "dev" for the development version
https://github.com/pandas-dev/pandas/runs/4859259779?check_suite_focus=true looks failng
ok let's give this a try.
@meeseeksdev backport 1.4.x
Something went wrong ... Please have a look at my logs.
It's live -> https://pandas.pydata.org/docs/dev/

For now if you use the dropdown to go the another version, the dropdown disappears of course (only new releases will have the dropdown)
nice!
BUG: correctly instantiate subclassed DataFrame/Series in groupby apply
State: closed

closes #45314

- [ ] whatsnew entry

This reverts the "fastpath" construction in groupby.apply from https://github.com/pandas-dev/pandas/pull/40236 and restores the code from https://github.com/pandas-dev/pandas/pull/37461, and expands the metadata propagation that was added in that PR to more cases.

The problem is that subclasses can have logic in their `_constructor` that is not captured by the `_from_mgr` fastpath. Will check the performance impact now, and if needed we could also do this only for subclasses, and keep `_from_mgr` for base DataFrame.

Labels:
name: Bug
name: Groupby
name: Regression
description: Functionality that used to work in a prior pandas version

Comments:
Running the relevant benchmark:

```
$ asv continuous -f 1.01 upstream/main HEAD -b groupby.Apply
...
       before           after         ratio
     [5b2f4a53]       [7f9dfdec]
     <main>           <groupby-subclass-attr-regression>
+      9.33±0.5ms       13.6±0.2ms     1.46  groupby.ApplyDictReturn.time_groupby_apply_dict_return
+      13.0±0.7ms       18.5±0.2ms     1.43  groupby.Apply.time_scalar_function_single_col(4)
+        36.2±2ms       49.2±0.3ms     1.36  groupby.Apply.time_scalar_function_multi_col(4)
```

(I ran this several times, and it's quite consistent)

So the ones with a dummy function ("scalar_function") and many small groups ("(4)") show a slowdown. But that also means that the other ones (with a non-dummy function ("copy_function"), or with larger groups) didn't actually show a significant difference. I think that's the most relevant.
thanks @jorisvandenbossche 
@meeseeksdev backport 1.4.x
Something went wrong ... Please have a look at my logs.
@jorisvandenbossche can we have a brief release note for this change.
Yes, this PR was not really ready for merging, see also Brock's comments
Thanks @jorisvandenbossche. From the issue, it appears that the regression is from 1.3.x but the reproducer worked on 1.3.5? Does the test added here test the regression reported or perhaps some later change?
Yes, so the _pandas_ test that I added here in this PR only covers the regression in main (worked in 1.3.5). So for that we actually don't need a whatsnew notice. 

It also fixes the geopandas issue which was already broken in 1.3.5, but it's a bit complicated to add an exact test for it (but I added one on the geopandas side (https://github.com/geopandas/geopandas/pull/2298), and we test against pandas main).

Both issues are related to the instantiation of the sub-DataFrame (per group), and originally caused by no longer using `_constructor` and not properly calling `__finalize__`. But the reason that this only turned up in main (and was not yet broken in 1.3) for the pandas reproducer is because in main we removed the libreduction fastpath, which basically still masked the regression in the python path for the majority of cases in 1.3.x
@jorisvandenbossche if not ready pls make draft PRs
It was a review comment of Brock that required work ...
BUG: No error raised in to_stata() for -np.inf #45350
State: closed

* Add an absolute function when checking infinity, which includes both np.inf and -np.inf. Also changed the error message a bit.
* Add similar tests but replacing np.inf with -np.inf and change the correct error message accordingly

- [ ] closes #45350
- [ ] tests added / passed
- [ ] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit) for how to run them
- [ ] whatsnew entry


Labels:
name: Bug
name: IO Stata
description: read_stata, to_stata

Comments:
thanks @flcong
BUG: categorical-with-nas to float64 instead of object
State: closed

- [ ] closes #xxxx
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit) for how to run them
- [x] whatsnew entry

This makes us match what we do in Index._find_common_type_compat, after which we'll be able to refactor both into a helper.

Labels:
name: Categorical
description: Categorical Data Type

Comments:
great @jbrockmendel i believe you mentioned you can now refactor find_common_dtype, great
DEPR: DatetimeIndex.intersection with mixed timezones cast to UTC, not object
State: closed

- [x] closes #24362
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit) for how to run them
- [x] whatsnew entry

This matches the union behavior, which we deprecated in 1.3.0

(closing #24362 is pretty subjective, but i think this is the best we can do)

Labels:
name: Timezones
description: Timezone data dtype
name: Deprecate
description: Functionality to remove in pandas
name: setops
description: union, intersection, difference, symmetric_difference

Comments:
lgtm can merge on green
BUG: Series[Interval[int]][1] = np.nan incorrect coercion/raising
State: closed

- [ ] closes #xxxx
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit) for how to run them
- [x] whatsnew entry


Labels:
name: Mothballed
description: Temporarily-closed PR the author plans to return to

Comments:
Mothballing while troubleshooting.
CI: Ensure GHA uses major versions of actions
State: closed

* Change `@master` to `@<major version>` for more stability
* Change specifically pinned version to `@<major version>` to ensure bug & security changes are picked up
* Bump actions/github-script from `@v4` to `@v5`


Labels:
name: CI
description: Continuous Integration

Comments:
@mroeschke ok to backport?
@meeseeksdev backport 1.4.x
> @mroeschke ok to backport?

Yes should be good. Thank you.

https://github.com/pandas-dev/pandas/pull/45391 was also a followup to this but it only impacts the current repository state so will leave that up to you
TST: xfail test_arrowparquet_options for windows
State: closed

- [x] xref #45344
- [ ] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit) for how to run them

xfailing for now to get the CI to green

Labels:
name: Testing
description: pandas testing functions or related to the test suite

Comments:
See also the discussion in https://github.com/pandas-dev/pandas/issues/45344, updating the pyarrow pin to 2 in

https://github.com/pandas-dev/pandas/blob/5b2f4a531f38b8cc8a0b372aeae09c1d94fc99e8/ci/deps/azure-windows-38.yaml#L26


is probably the "proper" fix that doesn't need an xfail.
BUG: df.iloc[:, 0] = df.iloc[::-1, 0] not setting inplace for EAs
State: closed

- [x] closes #35417
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit) for how to run them
- [x] whatsnew entry


Labels:
name: Bug
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves

Comments:
@jreback gentle ping, this will let me get rid of some TODO/FIXMEs
Update cheatsheet
State: closed

This PR contributes the following items to the cheatsheet:
- List of frequently used options shown here: https://pandas.pydata.org/pandas-docs/stable/user_guide/options.html#frequently-used-options
- I/O: A whole section showing a variety of popular IO usage (CSV, Excel, SQL, HTML), and also output file formats (feather, parquet, HDF)
- An Apply Functions section
- pd.NA
Will make further additions based on review by maintainers. 

Labels:
name: Stale

Comments:
@MichaelTiemannOSC thank you for the feedback and sorry for the delay in getting back! Will address these asap. 
This pull request is stale because it has been open for thirty days with no activity. Please [update](https://pandas.pydata.org/pandas-docs/stable/development/contributing.html#updating-your-pull-request) and respond to this comment if you're still interested in working on this.
Thanks for the pull request, but this PR appears to be stale. Additionally it appears https://github.com/pandas-dev/pandas/pull/43036 must be addressed first before other cheat sheet changes so closing.

Happy to have other pull requests with the label `good first issue`
Ko ya346/update import doc
State: closed

- [x] DOCS: Update import statements in docstrings #21749
- [x] tests added / passed
- [ ] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit) for how to run them
- [ ] whatsnew entry


Labels:
name: Docs

Comments:
Hello @ko-ya346! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:





There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2022-01-13 14:34:34 UTC
thanks @ko-ya346
CI/DOC: replace master -> main branch
State: closed

Follow-up on https://github.com/pandas-dev/pandas/issues/39577

Labels:
name: Compat
description: pandas objects compatability with Numpy or Python functions

Comments:
This has failed a few times it appears. Not sure why this PR would influence this failure:

https://dev.azure.com/pandas-dev/pandas/_build/results?buildId=72190&view=logs&j=c4a134e2-1a49-5486-5bb3-f47b2d22370f&t=a5a6d308-4ac5-5e83-907a-cda493e43b7d&l=49
do we need to backport this? (yes?)
> do we need to backport this? (yes?)

I would think so.
thanks @jorisvandenbossche 
@meeseeksdev backport 1.4.x
Something went wrong ... Please have a look at my logs.
The backport is actually already done at https://github.com/pandas-dev/pandas/pull/45338 by meeseeksdev (not fully sure why he mentioned here that "something went wrong")
because @jreback triggered manually. The bot sometimes does not pick it up (although not noticed this lately)

yeah i always trigger (though i realiE i don't really need to)
> yeah i always trigger (though i realiE i don't really need to)

yeah probably safer to make sure nothing gets missed. If the PR has a release note then I check release notes are in sync so these won't get missed. If if fixes ci then again won't get missed as the ci would be failing on the backport branch. If it's a doc fix or some other cleanup then could potentially be missed if the bot does not pick it up and it is not triggered manually.
CI: Change master -> main
State: closed

Change needed to get GHA and Azure running again after the branch renaming.


Labels:
name: CI
description: Continuous Integration

Comments:
I also opened https://github.com/pandas-dev/pandas/pull/45336, which is a more complete replace (also updates the contributing docs). Now, if that needs more review (since it's bigger), it's also fine to merge this first quickly to get CI running again (it shouldn't give conflicts, as this PR should be a strict subset of the other)
> I also opened #45336, which is a more complete replace (also updates the contributing docs). Now, if that needs more review (since it's bigger), it's also fine to merge this first quickly to get CI running again (it shouldn't give conflicts, as this PR should be a strict subset of the other)

Ah sorry I didn't notice this PR. Closing this one.
DEPR: un-deprecate errors kwd reverts #44294
State: closed

reverts #44294

Labels:
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves
name: Deprecate
description: Functionality to remove in pandas

Comments:
@jreback undeprecation for 'where'
hmm the CI didn't run on this, can you rebase
FYI the CI won't run until https://github.com/pandas-dev/pandas/pull/45336 is in.
if you can rebase
> if you can rebase

Already up to date.
@meeseeksdev backport 1.4.x
Something went wrong ... Please have a look at my logs.