BrandonFors/pandas-github-issues huggingface dataset:
WIP/TST/ENH: require freq match in assert_index_equal
State: closed

582 tests are failing locally, so there's some work yet to be done.

@mroeschke 3 of those 582 are in window tests.  Can you take a look and suggest where an edit should go?

Labels:
name: Mothballed
description: Temporarily-closed PR the author plans to return to

Comments:
@jbrockmendel sorry which tests in this PR should I look at that relate to the window test?
@mroeschke from the azure py37 build, the last ~4 failures are for window tests, e.g.:

```
________________________ test_rolling_window_as_string _________________________
[gw0] linux -- Python 3.7.6 /home/vsts/miniconda3/envs/pandas-dev/bin/python

    def test_rolling_window_as_string():
        # see gh-22590
        date_today = datetime.now()
        days = pd.date_range(date_today, date_today + timedelta(365), freq="D")
    
        npr = np.random.RandomState(seed=421)
    
        data = npr.randint(1, high=100, size=len(days))
        df = DataFrame({"DateCol": days, "metric": data})
    
        df.set_index("DateCol", inplace=True)
        result = df.rolling(window="21D", min_periods=2, closed="left")["metric"].agg("max")
E       [left]:  None
E       [right]: <Day>

pandas/tests/window/moments/test_moments_rolling.py:1412: AssertionError
```
@jbrockmendel after a cursory glance at the rolling failures, looks like the construction of the `expected` DataFrame/Series needs to include frequency. The `rolling` operation is adding `freq` to the index.
closing to clear the queue, will continue to chip away this locally
Make DTI.get_loc/get_value require tzawareness-compat for datetimes
State: closed

- [x] closes #30994 
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves

Comments:
> Can we first further discuss this at #30994 ?

Discuss away
closing to clear the queue, will revisit
BUG: DatetimeIndex.snap incorrectly setting freq
State: closed


Labels:
name: Bug
name: Frequency
description: DateOffsets

Comments:
Sounds good though I think we want to deprecate this function? #27098
> though I think we want to deprecate this function

Yah, i think the argument was that `round` can do what it snap does, but there was a comment somewhere that this isnt quite accurate.  Regardless, fixing the bug is worthwhile.
Should infer the frequency in case the result is actually regular? (to keep the correct behavior for those cases)
> Should infer the frequency in case the result is actually regular? (to keep the correct behavior for those cases)

Either way I guess.  I tend to prefer not inferring in most places.
thanks
BUG: IntegerArray.astype(boolean)
State: closed

Closes https://github.com/pandas-dev/pandas/issues/31102

Labels:
name: ExtensionArray
description: Extending pandas with custom dtypes or arrays.

Comments:
Added a test that IntegerArray.astype(boolean) works with non 0/1 values.
Thanks!
@MeeseeksDev backport to 1.0.x
BUG: Series/Frame invert dtypes
State: closed

Fixes NDFrame.__invert__ to preserve dtypes by applying things blockwise. Notably, this fixes `~Series[BooleanDtype]`.

This also provides a pattern for other unary ops, though I think that those are less pressing for 1.0, which should probably have a working `~Series[BooleanDtype]`

cc @jorisvandenbossche 


Labels:
name: Numeric Operations
description: Arithmetic, Comparison, and Logical operations
name: ExtensionArray
description: Extending pandas with custom dtypes or arrays.

Comments:
All green.
thanks @TomAugspurger 
@MeeseeksDev backport to 1.0.x
Owee, I'm MrMeeseeks, Look at me.

There seem to be a conflict, please backport manually. Here are approximate instructions:

1. Checkout backport branch and update it.

```
$ git checkout 1.0.x
$ git pull
```

2. Cherry pick the first parent branch of the this PR on top of the older branch:
```
$ git cherry-pick -m1 351760c0655b6c383e449cf857b9a718e3545229
```

3. You will likely have some merge/cherry-pick conflict here, fix them and commit:

```
$ git commit -am 'Backport PR #31183: BUG: Series/Frame invert dtypes'
```

4. Push to a named branch :

```
git push YOURFORK 1.0.x:auto-backport-of-pr-31183-on-1.0.x
```

5. Create a PR against branch 1.0.x, I would have named this PR:

> "Backport PR #31183 on branch 1.0.x"

And apply the correct labels and milestones.

Congratulation you did some good work ! Hopefully your backport PR will be tested by the continuous integration and merged soon!

If these instruction are inaccurate, feel free to [suggest an improvement](https://github.com/MeeseeksBox/MeeseeksDev).
TYP: annotations in indexes
State: closed


Labels:
name: Typing
description: type annotations, mypy/pyright type checking

Comments:
Hello @jbrockmendel! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:











There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2020-01-22 19:29:15 UTC
thanks!
BUG/CLN: use more-correct isinstance check
State: closed

this is only kind of a bug, since the isinstance check still returns True on the `type(obj)`

Labels:
name: Dtype Conversions
description: Unexpected or buggy dtype conversions

Comments:
Makes sense

> this is only kind of a bug, since the isinstance check still returns True on the `type(obj)`

So are there any code paths you could see hitting this?
Int64index.union(UInt64Index) I think goes through here, or the other way
around.

On Tue, Jan 21, 2020 at 11:55 AM William Ayd <notifications@github.com>
wrote:

> Makes sense
>
> this is only kind of a bug, since the isinstance check still returns True
> on the type(obj)
>
> So are there any code paths you could see hitting this?
>
> —
> You are receiving this because you authored the thread.
> Reply to this email directly, view it on GitHub
> <https://github.com/pandas-dev/pandas/pull/31180?email_source=notifications&email_token=AB5UM6FEVW5HRSHO3VXY2U3Q65HMTA5CNFSM4KJXPFG2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEJRBIVA#issuecomment-576853076>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AB5UM6H3P6WDMGMKJ2BRHWTQ65HMTANCNFSM4KJXPFGQ>
> .
>

does this any tests? 
> does this any tests?

no, it doesnt actually change any behavior
BUG: inconsistency between PeriodIndex.get_value vs get_loc
State: closed

ATM we have inconsistent treatment for indexing with strings where the string has a higher resolution than the PeriodIndex.  Luckily only one test checks for the inconsistent `PeriodIndex.__contains__` behavior, so fixing it isn't _too_ invasive.

```
pi = pd.period_range("2017-09-01", freq="D", periods=3)
ser = pd.Series(range(5, 8), index=pi)
key = "2017-09-01 00:00:01"

# master
>>> key in pi
True
>>> pi.get_loc(key)
0
>>> pi.get_value(ser, key)
KeyError: '2017-09-01 00:00:01'

# PR
>>> key in pi
False
>>> pi.get_loc(key)
KeyError: '2017-09-01 00:00:01'
>>> pi.get_value(ser, key)
KeyError: '2017-09-01 00:00:01'
```

Note: the diff is fairly big because we're mirroring the code in `get_value`.  Once this is fixed, we'll have get_value dispatch to get_loc and remove basically the same amount of code as we're adding here.

Labels:
name: Bug
name: Period
description: Period data type

Comments:
updated+green
thanks
REF: require scalar in IntervalIndex.get_loc, get_value
State: closed

cc @jschendel the casting of tuple->Interval on L819-820 seems sketchy, can you confirm if thats appropriate?

I'm also curious about the restriction on slice steps in _convert_slice_indexer, would be nice to have a comment there about why that is.

xref #31117.

Labels:
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves
name: Interval
description: Interval data type

Comments:
thanks @jbrockmendel and @jschendel
DOC: fix DataFrame.plot docs 
State: closed

- [x] fixes #29489 
- [ ] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry
 

Labels:
name: Docs

Comments:
@lithomas1 can you post a screenshot of how this looks after building?
![image](https://user-images.githubusercontent.com/47963215/72863198-9ae0e100-3c84-11ea-8afc-d645b3af1435.png)
cc @WillAyd 
Should I add tests for this too in order to prevent regression in the future?
Thanks. Let's merge this and see how they look when built.
TST: Add regression tests for fixed issues
State: closed

- [x] closes #14600
- [x] closes #14849
- [x] closes #14955
- [x] closes #14592
- [x] closes #12163
- [x] closes #12948
- [x] closes #15058
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`


Labels:
name: Testing
description: pandas testing functions or related to the test suite

Comments:
lgtm. merge on green.
ENH: Implement _from_sequence_of_strings for BooleanArray
State: closed

- [x] closes #31131
- [x] tests added / passed
- [x] passes `black pandas`
- [x] whatsnew entry

Labels:
name: Missing-data
description: np.nan, pd.NaT, pd.NA, dropna, isnull, interpolate
name: ExtensionArray
description: Extending pandas with custom dtypes or arrays.

Comments:
Hello @dsaxton! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:







There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2020-01-23 17:13:40 UTC
lgtm unless @jorisvandenbossche has more comments. (should make an issue to review performance of the string construction, e.g. an asv would asnwer if perf is an issue here)
LGTM aside from https://github.com/pandas-dev/pandas/pull/31159/files/29dbfa1165311260ffd54afbc6f8208ee27405c6#diff-20faa368ac513e325ac07ec72a8f93c7
There are actually also options in read_csv to specify the true/false strings. I suppose we are OK to ignore that? Or can we pass that through somehow?
I’m ok with ignoring that for now
> Can you also check what you get when you have an invalid string in your column? (if it is a decent error message)

Right now the error message that gets raised is `TypeError: Need to pass bool-like values` from inside `BooleanArray._from_sequence`. I think this should be a `ValueError`, maybe raised from `map_string` (instead of returning any string that doesn't match one of the true / false ones)?
>  I think this should be a ValueError, maybe raised from map_string (instead of returning any string that doesn't match one of the true / false ones)?

Yes, I think it is easy to raise a more informative ValueError from the last `else` clause in `map_string`
Remove possibly illegal test data
State: closed

Much of the test_html data looks like it was saved from real web pages, and not all of them look like the kind that are under a free software license.  (I couldn't actually check because only one of the three identifies the source site, and that site raises a security warning.)

This removes the questionable data, and adds tests that try to do something equivalent on known-free data.

If I am mistaken and these pages are under a free license, please instead document that.

This data was added in:
https://github.com/pandas-dev/pandas/commit/1d573b4421d5ff16fd68ea0cc6c63f5f1c7ebcac computer_sales_page (HP), for MultiIndex header and empty string handling in headers (should give empty not NaN)
https://github.com/pandas-dev/pandas/commit/e22fe1b25d1ff9d5c880f6c35bc91dfab7aa0b84 nyse_wsj, for thousands separator
https://github.com/pandas-dev/pandas/commit/92aa2774ca9aa89d0a2cca011707b80db995df92 macau, for MultiIndex header and thousands separator

Labels:
name: IO HTML
description: read_html, to_html, Styler.apply, Styler.applymap

Comments:
Looks like you just need to remove the `Index` import from `pandas/tests/io/test_html.py`
From the checks bot:

Check import format using isort
##[error]Skipped 2 files
Check import format using isort DONE
##[error]Process completed with exit code 1.

No other errors, so not sure what that's supposed to mean?

The build bots fail with: Git checkout failed with exit code: 128
@rebecca-palmer you should install the pre-commit hooks to help you on PRs, they automatically do black,isort,mypy, do ``pip instasll pre-commit``
Thanks @rebecca-palmer 
@Meeseeksdev backport to 1.0.x
Use https for links where available
State: closed

As some of our links are suggestions to install add-on software, insecure links potentially allow an attacker to replace this with their malware.

During the process of writing this, I also found and fixed some broken or semi-broken (e.g. redirected to the top level instead of the page we want) links.

The following (already) broken links remain:
http://nipunbatra.github.io/2015/06/timeseries/ + others from this site
http://collaboration.cmc.ec.gc.ca/science/rpn/biblio/ddj/Website/articles/CUJ/1992/9210/ross/ross.htm
http://mysite.verizon.net/aesir_research/date/jdalg2.htm

Labels:
name: Docs

Comments:
Hello @rebecca-palmer! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:





































There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2020-01-21 23:00:06 UTC
can you rebase on master. 

also if you can add a code_checks for this as well (can be a followup too)
ci/code_checks.sh looks like it checks everything (not just new changes), so while it would be easy to add "http://" to its banned patterns list, that would fail as long as we have _any_ http links.  As some of our existing links (the ones I didn't convert) don't work over https, this is probably not a good idea.

The PEP8 issue isn't new, and involves a link that is >80char by itself.
> ci/code_checks.sh looks like it checks everything (not just new changes), so while it would be easy to add "http://" to its banned patterns list, that would fail as long as we have _any_ http links. As some of our existing links (the ones I didn't convert) don't work over https, this is probably not a good idea.
> 
> The PEP8 issue isn't new, and involves a link that is >80char by itself.

@rebecca-palmer I *think* that appending ```# noqa``` should do the trick despite the fact that it's inside a comment, so it will look kida like this:

```python
# <comments> <comments>
# <VERY_LARGE_URL> # noqa
# <more comments> <more comments>
```
thanks !
Check for pyarrow not feather before pyarrow tests
State: closed

read_feather/to_feather now use pyarrow.feather, not top-level (feather-format) feather, but some of their tests were still looking for top-level feather.

Two of them that deliberately cause a file not found error were also looking for the wrong form of error message.  (Probably nobody noticed because the above was skipping them.)

Labels:
name: CI
description: Continuous Integration
name: Dependencies
description: Required and optional dependencies
name: IO Parquet
description: parquet, feather

Comments:
Hello @rebecca-palmer! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:



There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2020-01-21 22:13:53 UTC
The build failures were because my system quotes the filename in the error message and some of yours don't; should be fixed now.
lgtm. ping on green.
Versions of this error message from the failed CI runs:
"Failed to open local file 'foo/\~/does_not_exist.feather', error: 没有那个文件或目录"
"Failed to open local file '/home/vsts/work/1/s/pandas/tests/io/data/does_not_exist.feather', error: 没有那个文件或目录"
"Failed to open local file 'foo/\~/does_not_exist.feather', error: The system cannot find the path specified.\r\n"
"Failed to open local file 'D:/a/1/s/pandas/tests/io/data/does_not_exist.feather', error: The system cannot find the file specified.\r\n"
'Failed to open local file: foo/\~/does_not_exist.feather , error: File o directory non esistente'
'Failed to open local file: /home/vsts/work/1/s/pandas/tests/io/data/does_not_exist.feather , error: File o directory non esistente'
'Failed to open local file: /Users/runner/runners/2.164.3/work/1/s/pandas/tests/io/data/does_not_exist.feather , error: No such file or directory'
'Failed to open local file: foo/~/does_not_exist.feather , error: No such file or directory'
(My local error, that I first wrote msg8 to accept, was in the first one's format but in English.)

History: This test was added in https://github.com/pandas-dev/pandas/pull/16523 (because read_msgpack was returning garbage instead of an exception), initially checking only the exception type.  It began checking the error message in https://github.com/pandas-dev/pandas/pull/24676

My latest version changes the pattern to "Failed to open local file.+does_not_exist\.{fn_ext}.?, error: .*", i.e. gives up on matching the last part of the message.
green @jreback
thanks @rebecca-palmer very nice
BUG: in DataFrame.count not returning subclassed data types.
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry

Once #30945 is merged these tests will pass.

I also, didn't know if a PR without first submitting an issue was appropriate, but to cut down on workload I decided to give it a try. If I'm mistaken please let me know.

As for the test there are four logical branches in df.count() that needed to be address within the test:

1. Non-homogeneous data types (this is the partion of the test that is dependent on #30945 as it uses .sum()).
```
        df = tm.SubclassedDataFrame(
            {
                "Person": ["John", "Myla", "Lewis", "John", "Myla"],
                "Age": [24.0, np.nan, 21.0, 33, 26],
                "Single": [False, True, True, True, False],
            }
        )
        result = df.count()
        assert isinstance(result, tm.SubclassedSeries)
```
2. Homogeneous data.
```
        df = tm.SubclassedDataFrame({"A": [1, 0, 3], "B": [0, 5, 6], "C": [7, 8, 0]})
        result = df.count()
        assert isinstance(result, tm.SubclassedSeries)
```
3. MultiIndex with level kwarg.
```
        df = tm.SubclassedDataFrame(
            [[10, 11, 12, 13], [20, 21, 22, 23], [30, 31, 32, 33], [40, 41, 42, 43]],
            index=MultiIndex.from_tuples(
                list(zip(list("AABB"), list("cdcd"))), names=["aaa", "ccc"]
            ),
            columns=MultiIndex.from_tuples(
                list(zip(list("WWXX"), list("yzyz"))), names=["www", "yyy"]
            ),
        )
        result = df.count(level=1)
        assert isinstance(result, tm.SubclassedDataFrame)
```
4. Force length of axis to be 0. 
```
        df = tm.SubclassedDataFrame()
        result = df.count()
        assert isinstance(result, tm.SubclassedSeries)
```

Labels:
name: Numeric Operations
description: Arithmetic, Comparison, and Logical operations
name: API - Consistency
description: Internal Consistency of API/Behavior

Comments:
Hello @EmilianoJordan! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:





There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2020-01-19 20:23:00 UTC
you can include the other PR in this commit to get it passing.
TST:Disallow bare pytest raises issue 30999
State: closed

- [x] refers #30999 
- [ ] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry

This pull request is to add appropriate match arguments to bare pytest.raises listed in #30999 [Comment](https://github.com/pandas-dev/pandas/issues/30999#issuecomment-574339599)

Labels:
name: Testing
description: pandas testing functions or related to the test suite

Comments:
Hello @shubchat! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:







There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2020-01-21 19:45:51 UTC
Thanks @shubchat !
> Thanks @shubchat !

My pleasure @jorisvandenbossche
BUG: Break reference from grouping level to MI
State: closed

Closes https://github.com/pandas-dev/pandas/issues/31068

Labels:
name: Groupby
name: MultiIndex

Comments:
> is this just a regression in master? (so no note needed).

Yes, this is just a regression from 0.25.x
> > is this just a regression in master? (so no note needed).
> 
> Yes, this is just a regression from 0.25.x

your comments are conflicting, if this is from 0.25.x then we need a note
Sorry, I wasn't clear. by "from 0.25.x" I meant "after". This is fixing a regression that's only present on master, not 0.25.x.
thanks @TomAugspurger
TST/REF: Parameteratized tests
State: closed


Labels:
name: Testing
description: pandas testing functions or related to the test suite

Comments:
Thinking at this agian, I think that my main assumption that this test can be parameteraize is wrong, because it's __not__ different values that goes into the exact asserts, it's different values with different asserts.

I could go and parameteraize parts of it, but it won't help and in fact it will do worse, it will be harder to maintain (IMO).

---

Reopen this if you disagree.
TST: Implement external error raised helper function.
State: closed

- [x] ref #30999
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry

Opened PR for what @gfyoung wrote [here](https://github.com/pandas-dev/pandas/issues/30999#issuecomment-575426086)

Labels:
name: Enhancement
name: Testing
description: pandas testing functions or related to the test suite

Comments:
Restarting the tests, as it failed to fetch a URL.
gentle ping @gfyoung 
Sorry for the delay @MomIsBestFriend!  Thanks for the work!
@gfyoung Thank you, you are the one who wrote it ;)
TYP: DataFrame.(index|columns) and Series.index
State: closed

Makes mypy discover the ``.index`` and ``.columns`` attributes and that they're ``Index`` (sub-)classes.

This is done by manually adding the ``properties.AxisProperty`` to DataFrame/Series instead of doing it programatically, as mypy doesn't like adding attributes programatically very much.

Labels:
name: Typing
description: type annotations, mypy/pyright type checking

Comments:
Ping.
thanks!
CLN/BUG: Float64Index.get_loc
State: closed

- clean up unnecessary try/except
- the "bug" part is that ATM we incorrectly raise KeyError instead of TypeError on non-hashable.

Labels:
name: Bug
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves

Comments:
lgtm. ping on green. maybe add a whatsnew when you can.
BUG: concat not copying index and columns when copy=True
State: closed

- [x] closes #29879
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry


Labels:
name: Bug
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode

Comments:
Hello @fujiaxiang! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:







There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2020-01-21 02:12:15 UTC
ping
Thanks @fujiaxiang