BrandonFors/pandas-github-issues huggingface dataset:
Read csv headers
State: closed

- [x] closes #36688 
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry


Labels:
name: IO CSV
description: read_csv, to_csv

Comments:
cc @twoertwein 
we have tons of fixtures eg s3_resources

pls try to not reinvent the wheel 

if u must mock then use monkeypatch

prefer not to mock at all
Hello @cdknox! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:









There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2020-12-13 23:17:56 UTC
I'm sure folks are busy; I just wanted to make sure no one is waiting on me for anything.  I think I've addressed the comments so far, but if I'm missing something or I need to do something else please let me know!
thanks @cdknox very nice
Glad to help as a long time pandas user! Just double checking, we're good without the change then from ``storage_options={"User-Agent": "custom_user_agent"}`` to ``storage_options={"headers": {"User-Agent": "custom_user_agent"}}``?  I'm good with it either way
BUG: Make DTI/TDI/PI argsort match their underlying arrays
State: closed

- [x] closes #37863
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry

The fix to keep resample working is a kludge I'd like to make unnecessary before merging; cc @mroeschke

Labels:
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode
name: ExtensionArray
description: Extending pandas with custom dtypes or arrays.
name: API - Consistency
description: Internal Consistency of API/Behavior

Comments:
@mroeschke will you have time to take a look at this in the not-too-distant future?  if not ill mothball, since its stuck in the mud.
lgtm. any user facing change? 
> lgtm. any user facing change?

Changes DTi/TDI/PI.argsort, that should be it.
BUG: Bug in setitem raising ValueError when setting more than one column via array
State: closed

- [x] xref #37954
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`

Did not add whatsnew, cause works on 1.1.4

Array to set has to have more or less rows than columns, otherwise bug won't occur. Wrong values would be set silently here.

This is only a regression on master, not related to any release

cc @jbrockmendel 


Labels:
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves
name: Regression
description: Functionality that used to work in a prior pandas version

Comments:
does this address the perf complaint in the OP from the linked issue?
small comments, LGTM
No, only the regression raising an error. 
LGTM cc @jreback (travis failure is unrelated)
thanks @phofl
BUG: DataFrame.to_html ignores formatters
State: closed

- [x] closes #36525
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry


Labels:
name: Output-Formatting
description: __repr__ of pandas objects, to_string
name: IO HTML
description: read_html, to_html, Styler.apply, Styler.applymap

Comments:
thanks @ma3da very nice
TST: add error message match for raise in test_datetimelike.py GH30999
State: closed

Reference #30999

- [ ] closes #xxxx
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Error Reporting
description: Incorrect or improved errors from pandas

Comments:
lgtm. @MarcoGorelli merge when ready.
Thanks @liaoaoyuan97 !
CLN: test_ewm_rolling_consistency.py
State: closed

Same changes as https://github.com/pandas-dev/pandas/pull/37944 and https://github.com/pandas-dev/pandas/pull/37946

Additionally allows removal of `moments/conftest.py` and `tests/window/common.py`


Labels:
name: Clean
name: Window
description: rolling, ewma, expanding

Comments:
thanks @mroeschke
DOC: Change warning for sort behavior in concat
State: closed

We should either add a ``versionchanged`` attribute, for when this behavior changed or remove the warning completly. Additionally I removed the ``sort=False`` input for the function, because this is no longer necessary.

Labels:
name: Docs
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode

Comments:
great thanks
CI: Fix ci flake error
State: closed

- [ ] xref #36305 
 came across it in #36305 


Labels:
name: CI
description: Continuous Integration

Comments:
hmm on master, ok thanks
@jreback I think an issue needs to be created to address this, it is the second time this happened: basically flake8 check doesn't detect `imported but unused` cases in feature PR, only detect the error once PR being merged to master.

Have seen it in #37923 and in #37703 , so ci checks ran successfully in those feature PRs, but actually they should have detected this unused import.

hmm interesting, cc @MarcoGorelli 
when ive seen this in the past it  has involved multiple PRs which independently are each correctly passed by flake8, only causing a problem when both are merged
> when ive seen this in the past it has involved multiple PRs which independently are each correctly passed by flake8, only causing a problem when both are merged

I think this is the issue - in #37584 I got this exact error during the PR CI

----

IMO this makes the case for always asking to merge master rather than restarting CI jobs manually
ehh, okay, i thought of this scenario as well, but just feel surprised to come across them twice recently

thanks @MarcoGorelli @jbrockmendel shall I close the issue?
I'll just dig through the logs to try to verify that that's actually what happened

Just for reference, the exact same command is run during CI for both PRs and builds from master:
```
pre-commit run --show-diff-on-failure --color=always --all-files
```
i closed #37940 

thanks for checking out! @MarcoGorelli 
OK, got it - 3 days ago, Brock opened #37872 and #37865, both of which removed an instance of `Label` from `pandas/core/indexes/period.py `.

#37872 was merged first. Then, a couple of days later, #37865 was merged, without it having been updated with changes from `master` - so its own CI was still green, but when it was merged, both instances of `Label` were gone and the CI check turned red
Anyway, this looks right and passes relevant checks - merging so the CI checks are fixed
ENH: make closed part of IntervalDtype
State: closed

- [x] closes #19371
- [ ] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry

cc @jschendel

Still need to figure out what to do for backwards compat for unpickled IntervalDtype objects; ATM this just sets closed=None.

This causes IntervalIndex.append to cast to object dtype instead of raising if we try to append a non-matching closed.  I think thats the right thing to do, but we haven't fully discussed that xref #37774

Labels:
name: Dtype Conversions
description: Unexpected or buggy dtype conversions
name: Interval
description: Interval data type

Comments:
updated to fix doctests, raise on mismatched 'closed'.  still not wild about how we're handling old pickles, and ATM still allowing closed=None to be passed to the constructor
mothballing to clear the queue while i figure out the pickle thing locally.
BUG: loc.setitem with expansion expanding rows
State: closed

- [ ] closes #xxxx
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry

discovered on branch that has dataframe always go through split_path

Labels:
name: Bug
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves

Comments:
@phofl do you have any other upcoming PRs that affect `_setitem_with_indexer_split_path`?  This and #37931 are my last two before im ready to start in on the always-split-path change, and id like to get any bugfixes+tests in place before that.
I think only #37749, but will have a closer look. At least nothing which is not already opened
is this a user facing change? (if so pls add a whatsnew), also merge master and ping on green.
ping
BUG: loc.setitem corner case
State: closed

- [ ] closes #xxxx
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry

cc @phofl if you've got any thoughts on how to handle this case

Labels:
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves

Comments:
rebased+green
CLN: tests/window/*
State: closed

* Renamed `test_window.py` to `test_win_type.py`
* Added `'count'` to `arithmetic_win_operators`
* Reused fixtures
* Used more pytest idioms in `test_moments_consistency_expanding.py`


Labels:
name: Clean
name: Window
description: rolling, ewma, expanding

Comments:
Hello @mroeschke! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:











There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2020-11-18 17:41:06 UTC
thanks @mroeschke
BUG: idxmax/min (and argmax/min) for Series with underlying ExtensionArray
State: closed

- [x] closes #32749, closes #33719, closes #36566
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry

Currently `pd.Series([1, 2, 3]).astype('Int8').idxmax()` raises an error.
This request proposes a fix.
My first time using Github and opening a pull request. Apologise if it's no good.

Labels:
name: ExtensionArray
description: Extending pandas with custom dtypes or arrays.

Comments:
Hello @tonyyyyip! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:







There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2020-12-31 01:56:35 UTC
Thanks @arw2019!

This should solve #33719.  Currently the code sample raises `TypeError: argmax() takes 1 positional argument but 2 were given`. This is because the implementation of argmax/min of ExtensionArray takes only 1 argument `self` but the "interface" passes an additional argument `None` to it.

Added test. Not sure if I have done right though. Thanks again. 
You also wanna look at the CI checks (in my experience easiest thing is to use the pre-commit tool locally)
Thanks @arw2019. I am completely new to testing. I will check that out.
This is possibly also related to https://github.com/pandas-dev/pandas/issues/35178 (although that speaks about argmin/argmax, and not idxmin/idxmax)
> This is possibly also related to #35178 (although that speaks about argmin/argmax, and not idxmin/idxmax)

Yes both the `argmax` and `idxmax` Series methods call the `argmax` method of the underlying `_values` with the additional argument `None`  (for the axis), which is normally fine when the `_values` is an nparray, but breaks when it's an EA, because the current EA implementation of `argmax` does not take any additional arguments. This PR fixes that.
> > This is possibly also related to #35178 (although that speaks about argmin/argmax, and not idxmin/idxmax)
> 
> Yes both the `argmax` and `idxmax` Series methods call the `argmax` method of the underlying `_values` with the additional argument `None` (for the axis), which is normally fine when the `_values` is an nparray, but breaks when it's an EA, because the current EA implementation of `argmax` does not take any additional arguments. This PR fixes that.

Can you add tests for argmax/argmin so we can decide if this resolves  #35178 (if yes great!)
Gentle ping to see if this is going anywhere.

Please let me know if my changes are not what what the requests wanted, as I might have misunderstood the requests.

Also I understand it was decided that argmax and idxmax should be separated because argmax is likely understated. I am happy to follow that philosophy here.
@jreback Added whatsnew entry and referenced issues. Thanks.
thanks @tonyyyyip very nice
> thanks @tonyyyyip very nice

Thanks.
TYP: __getitem__ method of EA (2nd pass)
State: closed

follow-on from #37898

remove casts and ignores added in #37898 (needed to use typevar in DatetimeLikeArrayMixin in conjunction with overload for slice to return correct DatetimeArray/PeriodArray)

another pass required to

more typevars for return type of type(self)
type \_\_getitem__ in concrete classes (Categorical, IntervalArray and SparseArray)
overloads with Any report overlapping overloads
NAType resolves to `Any` since implemented in cython without stub file


Labels:
name: ExtensionArray
description: Extending pandas with custom dtypes or arrays.
name: Typing
description: type annotations, mypy/pyright type checking
name: Stale

Comments:
lgtm. @jbrockmendel 
couple of comments, otherwise LGTM
removing this from the 1.2 milestone for now.

since #35259 is merged, can now proceed with the outstanding tasks in parallel. so plan is to get all the tasks started (to get review) and will then circle back to this.
This pull request is stale because it has been open for thirty days with no activity. Please update or respond to this comment if you're still interested in working on this.
closing while we discuss #39513
Deprecate inplace in Categorical.remove_unused_categories
State: closed

- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry

Related to #37643


Labels:
name: Categorical
description: Categorical Data Type
name: Deprecate
description: Functionality to remove in pandas

Comments:
This is a very good start.  For the checkbox at the top, since this doesn't handle all the relevant methods, it doesn't close the issue.  Instead write below the checkboxes "xref #37643"
@jbrockmendel Thanks for a quick review, I have updated this pull request according to your suggestions
Hello @OlehKSS! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:









There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2020-11-18 11:57:24 UTC
thanks @OlehKSS happy to have you do the rest ....
CI: fix mypy error
State: closed

cc @jbrockmendel 

Labels:
name: CI
description: Continuous Integration
name: Typing
description: type annotations, mypy/pyright type checking

Comments:
Neat, LGTM pending green
both #37898 and #37872 we independently Green. 

The only downside here is that Index is part of the public API, so using Generic here means that the backing array becomes part of the public type interface and we may want to hide this as an implementation detail.

I think that I'll open another PR that justs ignores the error for now while we give this more thought.

I'll close this, while we still have np.ndarray resolving to Any which masks errors, since it's not just Index._values but also __getitem__ still needs to be overloaded for the slice and use a typevar as a followon to #37898, so should do that first.
DOC: add two new examples to the transform docs
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [ ] passes `black pandas`
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry

This PR adds two examples to the transform user guide. The examples apply the transform method to a groupby object. This is from the PyData Global sprint, from [this issue](https://github.com/MarcoGorelli/PyDataGlobal2020-sprint/issues/5). The examples are taken from [here](https://stackoverflow.com/a/37189976/14583166) and [here](https://stackoverflow.com/a/54417351/14583166).

Labels:
name: Docs
name: Groupby

Comments:
Here is a preview of the new examples:
![Screen Shot 2020-11-17 at 17 20 15](https://user-images.githubusercontent.com/25107902/99424702-e80bfe80-28f9-11eb-82e1-9faa14d4ac1b.png)
![Screen Shot 2020-11-17 at 17 20 24](https://user-images.githubusercontent.com/25107902/99424726-ee01df80-28f9-11eb-9a88-1741f8e289aa.png)

@jreback I have merged with master and now all checks pass.
thanks @marktgraham
CLN: Add *.zip to .gitignore
State: closed

- [X] xref #37903
- [X] tests added / passed
- [X] passes `black pandas`
- [X] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: Compat
description: pandas objects compatability with Numpy or Python functions

Comments:
thanks @lucasrodes
BUG: use compression=None (again) to avoid inferring compression
State: closed

`read_fwf` was using `compression=None` to infer compression (in addition to `compression="infer"`). This is undocumented but was enforced in a test.

In #37639, I thought this was the expected behavior for all functions. That as wrong, `read_csv` explicitly states that `compression=None` does not infer any compression.


Labels:
name: IO CSV
description: read_csv, to_csv

Comments:
@jreback green'ish
thanks @twoertwein
CI: update pyupgrade to 2.7.4 in pre-commit
State: closed

Updating pyupgrade to 2.7.4 in pre-commit yaml. `pre-commit run pyupgrade --all-files` runs good :+1: 

- [x] closes #37891
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry
- [x] passes `pre-commit run pyupgrade --all-files`


Labels:
name: Code Style
description: Code style, linting, code_checks

Comments:
Thanks @BAlmeidaS !

@jreback the automated weekly update job from #37362 didn't work: https://github.com/pandas-dev/pandas/actions/runs/365510411 
 
If you added the `ACTION_TRIGGER_TOKEN` secret token then I don't know why it failed, the logs don't give much away - in which case, I could remove the job and do it manually (as here) when there's an update (which isn't too bad anyway)
BUG: groupby resample different results with .agg() vs .mean() 
State: closed

- [x] closes #33548
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry


Labels:
name: Bug
name: Groupby
name: Resample
description: resample method

Comments:
Hello @jalmaguer! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:





There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2020-12-21 05:42:45 UTC
some  CI Checks errors
@jreback I made some commits a few days ago since your last comments so I just wanted to bump this pull request so that it doesn't get lost and forgotten about. 
can you merge master (been lots of changes in groupby recently)
> can you merge master (been lots of changes in groupby recently)

@jreback Sure no problem.  Just merged it.  Looks like it should be good to go.
@jreback Just bumping this so it doesn't get forgotten
ok we had some patches which might have fixed this

can u merge master (and also test if your added tests pass w/o the patch)
@jreback Just merged master.  Looks like my added tests do not pass without the patch.
thanks @jalmaguer 
@meeseeksdev backport 1.2.x
CLN: Unused file 'path_to_file.zip'
State: closed

New file [path_to_file.zip](https://github.com/pandas-dev/pandas/blob/master/scripts/path_to_file.zip) was added in #37844.

I believe this is file is (accidentally?) generated when running [scripts/validate_docstrings.py](https://github.com/pandas-dev/pandas/blob/master/scripts/validate_docstrings.py) and probably is not needed in the repository.

Labels:
name: Clean

Comments:
I am not yet familiar with  [scripts/validate_docstrings.py](https://github.com/pandas-dev/pandas/blob/master/scripts/validate_docstrings.py), but is it possible that it executes the docstring example section?

Related to this, I found this line, where 'path_to_file.zip' appears:
https://github.com/pandas-dev/pandas/blob/1f89b82cc65f512e78b0038975425cc3d0772a05/pandas/io/excel/_base.py#L630
good point, thanks @lucasrodes 
> I am not yet familiar with [scripts/validate_docstrings.py](https://github.com/pandas-dev/pandas/blob/master/scripts/validate_docstrings.py), but is it possible that it executes the docstring example section?
> 
> Related to this, I found this line, where 'path_to_file.zip' appears:
> https://github.com/pandas-dev/pandas/blob/1f89b82cc65f512e78b0038975425cc3d0772a05/pandas/io/excel/_base.py#L630

we only execute this on the CI so nbd there (guess it was picked up when the PR was commited).

i suppose we should just add `*.zip` to our `.gitignore` (and other files that we generate just to be safe).

@lucasrodes would love a PR
TYP: __getitem__ method of EA
State: closed

as a precursor to https://github.com/pandas-dev/pandas/pull/35259#discussion_r522951940

there are more changes/additions needed but this PR might get too big to review and push through in a timely manor.

overload is required to avoid casts
Any is used for any type, whereas should use object.
only the abstract classes have been typed.
could create a scalar type alias in each of the EA classes similar to DTScalarOrNaT in DatetimeLikeArrayMixin
import and expose pandas._libs.missing.NAType in pandas._typing 

any of these could be done here or as follow-on

Labels:
name: ExtensionArray
description: Extending pandas with custom dtypes or arrays.
name: Typing
description: type annotations, mypy/pyright type checking

Comments:
lgtm any comments @jbrockmendel @jorisvandenbossche 
thanks @simonjayhawkins