BrandonFors/pandas-github-issues huggingface dataset:
TST: Assert msg with pytest raises in pandas/tests/extension/base
State: closed

- [ ] closes #xxxx
- [x] tests added / passed
- [x] passes `black pandas`
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry

This PR adds messages to the bare pytest raises in reduce.py and getitem.py in pandas/test/extension/base. This PR references #30999.

Labels:
name: Testing
description: pandas testing functions or related to the test suite

Comments:
Hello @marktgraham! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:





There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2020-12-11 16:27:05 UTC
The reason the tests are failing is because I don't know what causes the difference between these two error messages:

```
if ("numpy" not in str(type(arr))) | (
     arr.dtype.name not in ["string"]
):
     msg = "index 3 is out of bounds for axis 0 with size 3"
else:
     msg = "index 3 is out of bounds for size 3"
```

The error message is `AssertionError: Regex pattern 'index 3 is out of bounds for axis 0 with size 3' does not match 'index 3 is out of bounds for size 3'.` i.e. it should be selecting the second one but it selects the first one.

I've tried a couple of different things, but neither has worked. I thought it was something to do with the number of dimensions but that doesn't seem to work either.


Thanks @MarcoGorelli !
thanks @marktgraham
Rewrite dict literal for files in pandas/tests/frame/test_indexing and test_where 
State: closed

- [ ] closes #38138
- [ ] tests added / passed
- [ ] passes `black pandas`
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry
identify unnecessary dict call - rewrite as a literal
Rewrites as dictionary literals for the following files in two spots:
- pandas/tests/frame/test_indexing
- pandas/tests/frame/test_where 
based on @jreback comments:
https://github.com/pandas-dev/pandas/pull/38207#issuecomment-736960208

Labels:

Comments:
@jreback checks showing some issues: **pandas/tests/frame/indexing/test_indexing.py:77: error: "Set[Iterator[Tuple[Any, int]]]" has no attribute "keys"  [attr-defined]**

maybe its because replacing `dict()` with `{}` in this situation would cause python to mistake `{}` to be `set()`. Should we leave it as `dict()` for this case? 

```
l=[1,2,3]
a = dict(zip(l, range(len(l))))
type(a)
>> dict
```

```
b = {zip(l, range(len(l)))}; 
type(b)
>> set
```
The second change, which is for `pandas/tests/frame/test_where`, might also show a problem as using `dict()` gives us a dictionary of three items correctly
```
df = {1:2,2:3,3:4}
dict((c, s + 1)  for c, s in df.items())
> {1: 3, 2: 4, 3: 5}
```
but using `{}` gives us a set of 3 tuples, which might not be what we intended
```
{(c, s + 1)  for c, s in df.items()}
> {(1, 3), (2, 4), (3, 5)}
```

You could use dict comprehensions.
```python
l = [1,2,3]
x = {value: index for index, value in enumerate(l)}
x 
>> {1: 0, 2: 1, 3: 2}
```
ENH: context-manager for chunksize/iterator-reader
State: closed

- [ ] closes #xxxx
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry

Allows (and encourages) the following use:
```py
import pandas as pd

filename = "pandas/tests/io/data/csv/iris.csv"
chunksize = 2
with pd.read_csv(filename, chunksize=chunksize) as reader:
    for chunk in reader:
        # risky code that might raise
```
Same can be done for `read_json/sas` (I think these are all methods that support `chunksize`). If this PR should make it into 1.2, I can quickly add the changes for json/sas as well.

Are there more places to promote this new context manager?

Labels:
name: IO CSV
description: read_csv, to_csv

Comments:
this looks pretty neat. can you add a small note section in io.rst which shows this off. i think this is ok for 1.2, can add json/sas as followsup.
I don't understand why the documentation example is failing:

Exception in /home/runner/work/pandas/pandas/doc/source/user_guide/io.rst at block ending on line 1586
ParserError: Error tokenizing data. C error: out of memory

edit: the blank line was causing the issue, it was interpreted as the end of the `with`-block
@jreback green. I hope the whatsnew entry is good now
thanks @twoertwein very nice
BUG: DataFrame.drop raising TypeError for string label with non-unique MultiIndex and no level provided
State: closed

- [x] closes #36293
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry

As far as I know, isin needs a level input when only a string is given for a MultiIndex. If this should not change, we have to set level=0 in this case.

Labels:
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode
name: MultiIndex

Comments:
Hello @phofl! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:





There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2020-12-13 17:21:57 UTC
thanks @phofl
Update Dockerfile to use conda-forge compilers
State: closed

Additionally I switched to the miniforge3 image as this bases its installation on conda-forge
and I also have used `mamba` to install the environment as this massively saves time.

- [ ] closes #xxxx
- [ ] tests added / passed
- [ ] passes `black pandas`
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: Build
description: Library building on various platforms

Comments:
I'm not really sure where this is used / why this is useful but as it seems that it is broken and people seem to use it somehow, I gave it some love.
yeah this is fine. I don't think we are actively using this, assume you have tested it?
> yeah this is fine. I don't think we are actively using this, assume you have tested it?

Yes, tested and works.
thanks @xhochy 
just saw this, FWIW I use this, to get an isolated environment for working on pandas. Especially if I'm on Windows and want to use the Linux shell, thanks @xhochy for the update
Corrected the grammar in Exception
State: closed

Exception on different arrays' length corrected

- [ ] closes #xxxx
- [ ] tests added / passed
- [ ] passes `black pandas`
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: Error Reporting
description: Incorrect or improved errors from pandas

Comments:
@jreback , I have updated the message in 2 other files.
@jreback can you merge this MR?
Thanks @imaskm
TST: add cases for nearest_workday and before_nearest_workday
State: closed

- [ ] xref #38139
- [ ] tests added / passed
- [ ] passes `black pandas`
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry

Add cases for testing observance rules ``nearest_workday`` and ``before_nearest_workday`` for their full coverage.

Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Frequency
description: DateOffsets

Comments:
can you merge master, CI looks to be acting up.
Linux tests are red due to a failure to build environment (new pip).
Windows test failure is irrelevant.

```
    @tm.network
    @pytest.mark.single
    def test_round_trip_exception_(self):
        # GH 3867
        csv = "https://raw.github.com/hayd/lahman2012/master/csvs/Teams.csv"
>       df = pd.read_csv(csv)

```
> Linux tests are red due to a failure to build environment (new pip).
> Windows test failure is irrelevant.

re-run and green
thanks @ivanovmg
dict rewritten as literal for files in directory /pandas/core/arrays/
State: closed

- [x] xref #38138 
- [ ] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry: Identify unnecessary dict call - rewrite as a literal for directory /pandas/core/arrays/floating.py, interval.py, numpy_.py & base.py


Labels:
name: Code Style
description: Code style, linting, code_checks

Comments:
can you merge master and ping on green (the precommit checks are failing)
Thanks @vrsh - looks like there's some pre-commit failures, can you fix them up?
```
pre-commit run --files <files you committed>
```
Okay @MarcoGorelli and @jreback , I will take a look at the pre-commit checks and work on it
Could you fixing the merge conflict, @vrsh? 
@mroeschke Done
Did you resolve the conflicts right? It now shows 0 files changed
Yes @MacroGorelli the changes were done as part of commit d02348c4.

I guess someone else's PR on these files were accepted prior to mine hence closing this PR.

If this is not the case then please let me know how I can work on this further.
@vrsh sorry about that - lots of other open issues if you still want to contribute though
No problemo. I understand.
Looking forward to contribute on other pandas issues. 
@MarcoGorelli See you on another PR!
Backport PR #38057: PERF: fix regression in creation of resulting index in RollingGroupby
State: closed

Backport PR #38057

Labels:
name: Groupby
name: Performance
description: Memory or execution speed performance
name: Window
description: rolling, ewma, expanding

Comments:
@jorisvandenbossche there were several merge conflicts. If you can give this a quick once over before I merge.

location of tests are different `pandas/tests/window/test_groupby.py` vs `pandas/tests/window/test_grouper.py`
changes to `test_groupby_rolling_nan_included` not needed, added in #36842 and not on 1.1.x
class hierarchy has changed, changes to `_apply` in `RollingGroupby` instead of `BaseWindowGroupby`
Backport PR #38099: BUG: fix wrong error message in deprecated 2D indexing of Series with datetime values
State: closed

Backport PR #38099 on branch 1.1.x

Labels:

Comments:
@jorisvandenbossche the test failures are the same as master. was the change in https://github.com/pandas-dev/pandas/pull/38210/commits/e350b8215e2ec4a49c7ee2fb991ec898ee44e486 needed?

merge when ready.
>  was the change in e350b82 needed?

Yeah, the ``__getitem__`` behaviour of DatetimeArray has apparently quite changed on master, so it didn't work directly on 1.1.x. This does the conversion to a numpy array first, on the 1D EA, and only then does the 2D indexing operation. This seemed relatively safe to do
Thanks @jorisvandenbossche
Revert "ENH: Improve performance for df.__setitem__ with list-like indexers"
State: closed

Reverts pandas-dev/pandas#38148 xref https://github.com/pandas-dev/pandas/issues/37784#issuecomment-736200698

Labels:
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves
name: Performance
description: Memory or execution speed performance

Comments:
flaky test on the Travis push ci job
=========================== short test summary info ============================
FAILED pandas/tests/io/test_html.py::TestReadHtml::test_banklist_url_positional_match[bs4]
= 1 failed, 144300 passed, 3144 skipped, 1435 xfailed, 58 warnings in 2953.57s (0:49:13) =
Rewrite dict literal for files in pandas/tests/frame
State: closed

- [ ] xref #38138
- [ ] tests added / passed
- [ ] passes `black pandas`
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry
identify unnecessary dict call - rewrite as a literal
Rewrites as dictionary literals for the following files:

	modified:   pandas/tests/frame/indexing/test_indexing.py
	modified:   pandas/tests/frame/indexing/test_where.py
	modified:   pandas/tests/frame/methods/test_fillna.py
	modified:   pandas/tests/frame/methods/test_sort_values.py
	modified:   pandas/tests/frame/methods/test_to_csv.py
	modified:   pandas/tests/frame/methods/test_to_records.py
	modified:   pandas/tests/frame/test_arithmetic.py
	modified:   pandas/tests/frame/test_reductions.py

However, I did not change it to literal in the following two places, as they seem like valid dict call:
1. pandas/tests/frame/indexing/test_indexing.py
```
lambda l: dict(zip(l, range(len(l)))),   # line 76
lambda l: dict(zip(l, range(len(l)))).keys(),  # line 77
```
2. pandas/tests/frame/indexing/test_where.py

```
return DataFrame(dict((c, s + 1) if is_ok(s) else (c, s) for c, s in df.items()))  # line 32
```

Labels:
name: Code Style
description: Code style, linting, code_checks

Comments:
both of 1 and 2 can be dict-comprehensions. (can do separately).
TST: Added dataframe constructor tests confirming order is preserved with standard Python dicts
State: closed

- [ ] closes #38033 
- [ ] tests added / passed
- [ ] passes `black pandas`
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry

Adds additional test scenarios that confirm dataframes created with standard Python dicts preserve order as expected.

Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Constructors
description: Series/DataFrame/Index/pd.array Constructors

Comments:
thanks @JSextonn
Rewrite dict literal for files in tests/groupby & ./pandas/tests/dtypes 
State: closed

- [ ] xref #38138
- [ ] tests added / passed
- [ ] passes `black pandas`
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry
identify unnecessary dict call - rewrite as a literal
Rewrites as dictionary literals for the following files:

 ./pandas/tests/dtypes/test_inference.py
./pandas/tests/groupby/test_function.py
./pandas/tests/groupby/transform/test_transform.py



Labels:
name: Code Style
description: Code style, linting, code_checks

Comments:
Hello @Qbiwan! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:







There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2020-12-01 13:35:16 UTC
Thanks @Qbiwan - looks like there's some pre-commit failures, can you fix them up?
```
pre-commit run --files pandas/tests/dtypes/test_inference.py pandas/tests/groupby/test_function.py pandas/tests/groupby/transform/test_transform.py
```
@MarcoGorelli thanks, i think it is passing pre-commit now
Retain views with listlike indexers setitem
State: closed

- [x] closes #37954

xref #38148

Labels:
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves
name: Performance
description: Memory or execution speed performance

Comments:
@jbrockmendel #38148 has been reverted, so can you update this to close #37954 and add the release note in 1.2.0 instead.
lgtm. the perf enhancements hold here (I think so, just checking).?
> lgtm. the perf enhancements hold here (I think so, just checking).?

i timed it at slightly faster than the copying version (likely within the margin of error, dont recall)
great. ping on green (not sure what's going on with CI).
> not sure what's going on with CI

AFAICT its just backed up.  A lot got merged in a short time!
@jreback green
thanks @jbrockmendel 

I think we going to backport this @simonjayhawkins (not sure if the regression was on 1.1.5)?
no evidence this was a regression. maybe can do some digging post 1.2.0rc0 and maybe backport to 1.1.5 and move release note before 1.2.0

but for now, I would say this is not being backported.
Rewrite dict literal for files in ./pandas/tests/tools/ and  ./pandas/tests/resample/ 
State: closed

- [ ] xref #38138
- [ ] tests added / passed
- [ ] passes `black pandas`
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry

Rewrote dict literal for the following files: 

 ./pandas/tests/tools/test_to_numeric.py
 ./pandas/tests/resample/test_time_grouper.py
 ./pandas/tests/util/test_assert_series_equal.py
 ./pandas/tests/util/test_assert_interval_array_equal.py
 ./pandas/tests/util/test_assert_index_equal.py
 ./pandas/tests/util/test_assert_extension_array_equal.py
 ./pandas/tests/io/test_parquet.py


Labels:
name: Code Style
description: Code style, linting, code_checks

Comments:
thanks @AbdulMAbdi
TYP: @final in groupby
State: closed


Labels:
name: Groupby
name: Typing
description: type annotations, mypy/pyright type checking

Comments:
cc @simonjayhawkins 

Thanks @jbrockmendel no harm I guess if final is used where the method is not overridden or a class subclassed, but I think the intention of `@final` is to prevent overridding/subclassing. If this many finals are needed, is there something wrong with our class design? If this is kept to non-public code there is not an issue, since we can always remove the `@final` as needed, but we need to be careful when this is used on any subclasses of public classes.
> but I think the intention of @final is to prevent overridding/subclassing. If this many finals are needed, is there something wrong with our class design

In these cases I'm using it largely to indicate to the reader that a the method is not overridden, which can be a hassle to check manually bc there are a bunch of similarly-named class hierarchies (each one is individually reasonable) and a bunch of similarly-named methods.

yes, I can see how that helps. 

users that subclass, override methods and type check may get errors though. Only time will tell whether this will be an real issue for us to consider.
ok if you can rebase
rebased+green
Remove duplicate entry on series.rst
State: closed

- [x] closes #38198 
- [ ] tests passed. Nothing new fails. Few tails already were failing on `master`
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry (Intentionally ignored as insignificant)

Removed the duplicate entry of `Series.replace` from doc

Labels:
name: Docs

Comments:
thanks @vikramaditya91
BUG: name attr in RangeIndex.intersection
State: closed

- [ ] closes #xxxx
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry


Labels:
name: Refactor
description: Internal refactoring of code

Comments:
after #38111 will update the whatsnew from there to include this
@jreback did something break the npdev build again while i wasnt looking?
i wonder if this is the backport of the 2d indexing message (just merged to 1.1.5)


yah its just a complaint about a mismatched exception message in the 2d indexing case
can you rebase (and update the fixture name)
rebased with whatsnew
looks fine. waiting on CI.
greenish
re-run and green
BUG: Avoid duplicates in DatetimeIndex.intersection
State: closed

- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`

@jbrockmendel I think we should call ``_convert_can_do_setop`` here too, but https://github.com/pandas-dev/pandas/blob/56b9a80e1e47e123eb7a7c01448d1506400f8a5b/pandas/tests/indexes/datetimes/test_setops.py#L472 tests the opposite


Labels:
name: Refactor
description: Internal refactoring of code

Comments:
> I think we should call _convert_can_do_setop here too, but [test_intersection_list] tests the opposite

yah im pretty sure that test is wrong xref #38111
Fixed the test
thanks @phofl very nice
BUG: DataFrame.idxmin/idxmax with mixed dtypes
State: closed

- [ ] closes #xxxx
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry


Labels:
name: Bug
name: Reduction Operations
description: sum, mean, min, max, etc.

Comments:
looks good. merge master and ping on green (CI hanging)
@jreback green
thanks @jbrockmendel
DOC: Fixed to_pickle compression argument list 'gzip' > 'gz'
State: closed

Removed reference to gzip and replaced with correct extension of gz.

- [x] closes #35364


<img width="842" alt="Screenshot 2020-11-30 at 20 43 59" src="https://user-images.githubusercontent.com/74271903/100664097-17d1f200-334f-11eb-90e3-e37ef7ce7c66.png">


Labels:
name: Docs

Comments:
The compression string here is specifying the compression and is used by the code:

```
# GZ Compression
if compression == "gzip":
```

This is making the docstring incorrect.
@rhshadrach Outdated issue?
As I commented in the issue, it seems like a good idea to me to have the details that are currently in `DataFrame.to_csv` also in `DataFrame.to_pickle`. These details make clear why the argument is "gzip" and not "gz".
Added information as mentioned by @rhshadrach.
<img width="819" alt="Screenshot 2020-11-30 at 23 11 09" src="https://user-images.githubusercontent.com/74271903/100677048-13630480-3362-11eb-869e-a4d5e3935922.png">

it might be useful to share documentation across all function that support the same compression (csv/pickle/json/...), similar to #37953. I didn't check whether the docstrings match exactly.
thanks @ooojpeg 

@twoertwein yeah can you open issue(s) to share these doc-strings as you did before.
REF: IntervalIndex.intersection match pattern in other intersection methods
State: closed

Looking to share the boilerplate and have subclasses just implement _intersection.

Labels:
name: Refactor
description: Internal refactoring of code
name: Interval
description: Interval data type

Comments:
@jreback between @phofl and me there's a handful of FooIndex.intersection PRs that are becoming blockers: #38190, #38111, #38197, #38154, #38196 (and #38105 will allow for more de-duplication and fastpaths)
kk will look soon
 
can i indicate which are orthogonal? or any ordering here?
> or any ordering here?

Current plan in #38197 is to share the whatsnew note with #38111, but otherwise these should be all independent.

These should be all logically orthogonal, but the ones least likely to involve merge conflicts are #38105 and #38154.
I think all are merged (except one needs a rebase),
DOC: fix redirects to user guide in README
State: closed

Documentation pages were moved so this commit updates those links to
open more directly rather than getting a "This page has been moved
to..." message.

- [ ] closes #xxxx
- [ ] tests added / passed
- [ ] passes `black pandas`
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: Docs

Comments:
thanks @erictleung
Rewrite dict literal for files in tests/plotting/frame/test_frame_subplots.py
State: closed

- [ ] xref #38138
- [ ] tests added / passed
- [ ] passes `black pandas`
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry 
   identify unnecessary dict call - rewrite as a literal
Rewrites as dictionary literals for the following files:

    pandas/tests/plotting/frame/test_frame_subplots.py


Labels:
name: Code Style
description: Code style, linting, code_checks

Comments:
thanks, see my comments on your other PR; you can add more changes in a single PR. If we are getting near the end then pls indicate that (as we actually want to turn o nthe flake option).
BUG: groupby.rank for dt64tz, period dtypes
State: closed

- [ ] closes #xxxx
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry

This sits on top of #38162

Labels:
name: Groupby

Comments:
hmm, looks like CI is hanging, can you merge master and pin gon green.
> hmm, looks like CI is hanging, can you merge master and pin gon green.

not merged master, but re-run an green
Rewrite dict literal for files in tests/plotting/test_boxplot_method.py
State: closed

- [ ] xref #38138
- [ ] tests added / passed
- [ ] passes `black pandas`
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry

Rewrites as dictionary literals for the following files:
- pandas/tests/plotting/test_boxplot_method.py


Labels:
name: Code Style
description: Code style, linting, code_checks

Comments:
thanks @Qbiwan 

you can put *more* of these changes in a single PR (not *everything* of course), but can include many files.