BrandonFors/pandas-github-issues huggingface dataset:
DOC: Fix comment
State: closed

The code clears the O_NONBLOCK flag (that is, makes IO blocking), so make the comment match.

- [ ] closes #xxxx
- [ ] tests added / passed
- [ ] passes `black pandas`
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: CI
description: Continuous Integration

Comments:
thanks @CitizenB
REF: make BaseOffset a cdef class
State: closed

@jreback i could use help here with one last stubborn pickle/pytables test `test_read_py2_hdf_file_in_py3`

Within this legacy file we have the substring (sub-bytes?) 

`blob = b"ccopy_reg\n_reconstructor\np1\n(cpandas.tseries.offsets\nBusinessDay\np2\nc__builtin__\nobject\np3\nNtRp4\n(dp5\nS'normalize'\np6\nI00\nsS'n'\nI1\nsS'kwds'\np7\n(dp8\nsS'offset'\np9\ncdatetime\ntimedelta\np10\n(I0\nI0\nI0\ntRp11\nsb."`

AFAICT it is doing `pickle.loads(blob)` and getting:

```
>>> pickle.loads(blob)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/Cellar/python/3.7.7/Frameworks/Python.framework/Versions/3.7/lib/python3.7/copyreg.py", line 43, in _reconstructor
    obj = object.__new__(cls)
TypeError: object.__new__(BusinessDay) is not safe, use BusinessDay.__new__()
```

Then it is falling back and for reasons not clear to me is calling `to_offset(blob)` which raises `ValueError` which is what pytest shows us.

IIUC there is a way to register functions for unpickling, but _none_ of the functions in pd.compat.pickle_compat are getting called in this process, so I'm at a loss for how to do this.  Any ideas?

Labels:
name: Frequency
description: DateOffsets

Comments:
you can figure out what you need to patch by monkey patching the unpickler itself

and stepping thru

it’s going to be one of the dist instructions i think (i did this a number of years ago so don’t remember much more)
> you can figure out what you need to patch by monkey patching the unpickler itself

Very tentative, but it looks like we can make a context to temporarily patch pickle to use the Unpickler in compat.pickle_compat during calls to `HDFStore.get`
Booyah green
ok will have another look, pls rebase
updated; npdev failure is unrelated AFAICT
rebased+green
> ok i understand, can you add some comments here to that effect

updated+green

very nice
ENH: enable mul, div on Index by dispatching to Series
State: closed

One of the last remaining inconsistencies between Index and Series arithmetic is that division and multiplication is entirely disabled for object-dtype Index, while for Series we operate pointwise.  This PR makes Index behave the same way, and simplifies the code by using existing Series-op-wrapping code to do so.

- [ ] closes #xxxx
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry

I'm pretty sure there is at least one issue for this, will track it down.

Labels:
name: Enhancement
name: Numeric Operations
description: Arithmetic, Comparison, and Logical operations

Comments:
> are we actually testing these?

Yes.  In particular see the tests.arithmetic tests that are currently xfailed that this un-xfails.

> it seems we are enabling all numeric methods; do all of these even make sense on datetimelikes?

DTI/PI get these methods from DTA/PA, which define them with `make_invalid_op`, so these correctly raise.
ok seems fine to me. @jorisvandenbossche 
lgtm. do we need a release note for the additional things that Index can now do? (e.g. can you enumerate them). pls merge master.
updated+green
thanks @jbrockmendel
CI: Address linting errors in flake8 >= 3.8.1
State: closed

xref #34150

This partially addresses the above issue by:

- Ignoring E741
- Fixing other linting errors

The issue with flake8-rst is handled temporarily in #34151, and the build here will fail the flake8-rst part until that is merged in.

Labels:
name: Code Style
description: Code style, linting, code_checks

Comments:
great, does this complete the flake issues? e.g. can we unpin after this one?
> great, does this complete the flake issues? e.g. can we unpin after this one?

It fixes all the issues with flake8 in the code, but it unfortunately doesn't fix the issue with flake8-rst kataev/flake8-rst#22.  That's a problem with the `flake8-rst` tool itself.

So, no, until `flake-rst` is fixed, we can't unpin.
ok thanks.
@MeeseeksDev backport to 1.0.x

Owee, I'm MrMeeseeks, Look at me.

There seem to be a conflict, please backport manually. Here are approximate instructions:

1. Checkout backport branch and update it.

```
$ git checkout 1.0.x
$ git pull
```

2. Cherry pick the first parent branch of the this PR on top of the older branch:
```
$ git cherry-pick -m1 f9809fe9f206c96bda1e4be09ae17689fcd13adf
```

3. You will likely have some merge/cherry-pick conflict here, fix them and commit:

```
$ git commit -am 'Backport PR #34152: CI: Address linting errors in flake8 >= 3.8.1'
```

4. Push to a named branch :

```
git push YOURFORK 1.0.x:auto-backport-of-pr-34152-on-1.0.x
```

5. Create a PR against branch 1.0.x, I would have named this PR:

> "Backport PR #34152 on branch 1.0.x"

And apply the correct labels and milestones.

Congratulation you did some good work ! Hopefully your backport PR will be tested by the continuous integration and merged soon!

If these instruction are inaccurate, feel free to [suggest an improvement](https://github.com/MeeseeksBox/MeeseeksDev).
                
@jorisvandenbossche will start backporting this or could pin flake8 on 1.0.x? lmk
> @jorisvandenbossche will start backporting this or could pin flake8 on 1.0.x? lmk

I would personally not care about backporting style related issues. Just pinning flake8 seems easier (or letting the lint build fail ..)
> > @jorisvandenbossche will start backporting this or could pin flake8 on 1.0.x? lmk
> 
> I would personally not care about backporting style related issues. Just pinning flake8 seems easier (or letting the lint build fail ..)

yep. not backporting this at this time.
CI: Pin flake8 < 3.8.0
State: closed

xref #34150

This handles multiple issues:
- flake8 >= 3.8.1 introduces a new version of pycodestyle which introduces
  extra checking
- flake8-rst is broken with the newest version of flake8
    - See kataev/flake8-rst#22

A follow-up PR handling the first of these is forth-coming

Labels:
name: CI
description: Continuous Integration
name: Code Style
description: Code style, linting, code_checks

Comments:
thanks @mgmarino
REF: separate to_time, avoid runtime imports
State: closed


Labels:
name: Datetime
description: Datetime data dtype
name: Deprecate
description: Functionality to remove in pandas
name: Clean

Comments:
can you rebase
rebased+green
> also can you open an issue about exposing pd.to_time

https://github.com/pandas-dev/pandas/issues/11947
once more rebase
thanks!

can you edit the deprecations removal issue for this change as well (not sure if you did)
CLN: Splits Moments and MomentsConsistency tests into separate files
State: closed

Due to previous cleaning as well as the large size of those files, we could now split the `Moments` and `MomentsConsistency` test classes into two different files for `rolling` and `ewm`. And since current `test_moments_expanding` is actually MomentsConsistencyExpanding class, so rename it to `test_moments_consistency_expanding` to keep consistency.

And I think this will be the second last PR to close the original test clean issue #30486 . The next PR will be fixturizing the `_create_data` so that we could remove the Base entirely (which currently only creates those data). But since this `base` is also used in other several files like `test_api.py`, `test_ewm.py` etc in window tests, I am afraid the next PR will be quite big.

Please let me your thoughts on the plan (or if you think the change is quite big then no need to proceed since the original issue is to only clean the `moments`)? 

cc @jreback 



Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Window
description: rolling, ewma, expanding

Comments:
can you confirm the number of tests before / after is the same here?
yep, it's pure copy-paste and no code is missing (also adding/removing imports)
can you merge master and push again, not all checks ran.
@jreback FYI, i quickly run through tests: 
```
after this PR:
test_moments_ewm.py: 22
test_moments_consistency_ewm.py: 4310
Current master:
test_moments_ewm on master: 4332 

after this PR:
test_moments_rolling.py:253
test_moments_consistency_rolling.py: 10881
Current master:
test_moments_rolling.py: 11134
```
so i think it's all good, no test missing
looks good. let me re-run.
/azp run
<samp>
Azure Pipelines successfully started running 1 pipeline(s).<br>

</samp>
thanks @charlesdong1991
REF: move mixins, repr_attrs to liboffsets
State: closed


Labels:
name: Refactor
description: Internal refactoring of code
name: Frequency
description: DateOffsets

Comments:
no objection. I assume that this is a piece in moving vritually everthing to liboffsets.pyx
> I assume that this is a piece in moving vritually everthing to liboffsets.pyx

Yep.
Revert: REF: do length-checks in boilerplate decorator
State: closed

closes #34123 

As discussed in #34123 
let's revert this PR34081 first since it has potentially big impact on some cases in ops.

cc @jbrockmendel @jorisvandenbossche 

Labels:
name: Bug
name: MultiIndex

Comments:
Hello @charlesdong1991! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:































There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2020-05-15 09:02:38 UTC
small request, otherwise LGTM
Can you add test(s) for the behavior that was broken in #34081
emm, something might have changed alongside, now after this revert pr, 
```python
>>> pd.Index([('a', 'b'), ('b', 'c'), ('c', 'a')]) == ('c', 'a')
array([False, False, False])
```
it returns all False.

@jbrockmendel any idea which PR recently merged could influence this?
emm, not sure how on earth i could run get `array([False, False, True])` result that day, cannot replicate anymore. but seems since 1.0 or later this case returns `array([False, False, False])`  which i feel is wrong as well as a warning from numpy. i put some comments and todo there and pls let me know if it is appropriate

i could create a new issue to address this if you think is necessary.
@charlesdong1991 I was also quite sure that I saw the correct behaviour in one of my environments, but also don't know where anymore.

Now, looking at it on 1.0.3, the problem is that it is creating a MultiIndex, and not a plain index with tuples in it. Ensuring we get an Index, I actually get the correct behaviour:

```
In [14]: pd.Index([('a', 'b'), ('b', 'c'), ('c', 'a')], tupleize_cols=False) == ('c', 'a') 
Out[14]: array([False, False,  True])
```

(and this example also actually fails on master)
yeah, indeed, the issue i saw is that in such case, it automatically recognises as a MI, not Index. 

@jorisvandenbossche thanks very much, I will add `tupleize_cols=False` in the test to get the correct result and I will also create an issue to report this case.

LGTM
thanks @charlesdong1991 

can you open a new issue to re-implement this (e.g. essentially a cleanup to revert the revert with appropriate changes).
ref #34233 

cc. @jreback
TST: DataFrame.interpolate axis name for axis argument
State: closed

- [ ] closes #25190
- [ ] tests added / passed
- [ ] passes `black pandas`
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry

xref #31566


Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Missing-data
description: np.nan, pd.NaT, pd.NA, dropna, isnull, interpolate

Comments:
thanks @simonjayhawkins
BUG/ENH: Improve categorical construction when using the iterator in StataReader
State: closed

- [X] closes #31544
- [X] tests added / passed
- [X] passes `black pandas`
- [X] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [X] whatsnew entry


Labels:
name: Bug
name: Categorical
description: Categorical Data Type
name: IO Stata
description: read_stata, to_stata

Comments:
thanks @bashtage
Fix #23055 Change pd.to_datetime and pd.to_timedelta to always convert null-like argument into pd.NaT
State: closed

I've tried to address the bug in `to_datetime` and `to_timedelta` functions. Please let me know if there're some other places worth to be changed.

- [x] closes #23055
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry


Labels:
name: Datetime
description: Datetime data dtype
name: Missing-data
description: np.nan, pd.NaT, pd.NA, dropna, isnull, interpolate

Comments:
can you also add tests for null-likes in Timestamp / Timdelta as well
@dair-targ are you still working on this? can you address comments. ping if you need help.
closing as stale. ping to continue or open a new PR.
API: remove dropped multilevel index from levels (GH: #34097)
State: closed

- [x] closes #34097
- [ ] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: MultiIndex
name: API - Consistency
description: Internal Consistency of API/Behavior

Comments:
closing this. We do not automatically recalcualate the levels on indexing. This is an implementation details. The original issue is won't fix. All that said wouldn't be averse to perf enhancements generally.
TST/REF: Deprivate check_pairwise_moment and remove ConsistencyBase
State: closed

the `check_pairwise_moment` function will be the last function moving out of big `ConsistencyBase` class, and after this, `ConsistencyBase` class only has the `_create_data` function which creates some mock data, so we could entirely remove `ConsistencyBase` and use `Base` instead. 

cc @jreback 

Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Window
description: rolling, ewma, expanding

Comments:
thanks !
MAINT: Clarify supported Stata dta versions
State: closed

Test against old Stata versions and remove text indicating support
for versions which do not work reliably

closes #26667

- [x] closes #26667
- [X] tests added / passed
- [X] passes `black pandas`
- [X] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: IO Stata
description: read_stata, to_stata

Comments:
Not sure if a what's new is needed here.
lgtm. yeah I would add a whatsnew indicating which versions are being removed for unreliability :-> ping on green.
are these versions listed in the doc-string?
@jreback I have gone through and removed any references to the unsupported versions.  There were none in docstrings; they were only in a few places that compared the version read to a minimum version. 
thanks @bashtage
DOC: timezone warning for dates beyond TODAY
State: closed

introducing a suggestion discussed in PR #33863 :
Added a warning in the user guide that timezone conversion on future dates is inherently unreliable.

- [ ] closes #xxxx
- [ ] tests added / passed
- [ ] passes `black pandas`
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: Docs
name: Timezones
description: Timezone data dtype

Comments:
what is the status here?
LGTM. Just looks like master needs to be merged in
> what is the status here?

I don't have a working dev environment and no time to set it up. I also tried to figure out if this can be done on Github only (without using my local machine) but couldn't find a way to merge master into my GH repository online. Feel free to take over the PR and fix the one CI error it still throws. Or I'll get to it when I get to it -- that is sometime this year.
I merged in master but there are again CI failures unrelated to the docs. Why is it so hard to get a simple doc change in?
> LGTM. Just looks like master needs to be merged in

Have merged master, CI passes now - still happy with it @mroeschke ?
Thanks @joooeey !
TST: Mark groupby.nunique test as slow
State: closed

This test takes about 40 seconds to run for me while shrinking the size of the data brings the runtime down to 2 seconds, presumably without changing coverage (do we even need to parametrize over n and m here?).

Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Groupby

Comments:
I've also wondered why the parameters are so large. Maybe it'd be worth tracking down the original issue and adding a comment. 
> I've also wondered why the parameters are so large. Maybe it'd be worth tracking down the original issue and adding a comment.

Looks like it was added here: https://github.com/pandas-dev/pandas/pull/10894
thanks @dsaxton
CLN: Change old string formatting syntax into f-strings
State: closed

- [x] xref #29547
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: Code Style
description: Code style, linting, code_checks

Comments:
thanks
CLN, TYP: Factor out part of info
State: closed

Precursor to #31796, which still has a rather large diff. This should help make it easier to read.

On master, `cols` holds the DataFrame's columns. If we were to use this method on a Series, then it would hold an Index containing the Series' name (`Index([data.name])`). Is there a good name for a variable that could store either? At the moment I've called it `ids`

Labels:
name: Output-Formatting
description: __repr__ of pandas objects, to_string
name: Dtype Conversions
description: Unexpected or buggy dtype conversions
name: Clean

Comments:
>  ping on green.

@jreback ping
thanks @MarcoGorelli
Add regression test for nested meta path in json_normalize
State: closed

We want to add regression test for #27220 and make sure that this kind of issues will not be present in the future.

Thanks to @yanglinlee [code](https://github.com/pandas-dev/pandas/pull/27667) I created test that works with current `master` branch.

- [x] closes #27220
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry


Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: IO JSON
description: read_json, to_json, json_normalize

Comments:
thanks @LTe
ENH: make Tick comparisons match Timedelta behavior
State: closed


Labels:
name: Timedelta
description: Timedelta data type
name: Frequency
description: DateOffsets

Comments:
Can you indicate if (I assume so, given the title), and if so, what behaviour exactly changed?
> Can you indicate if (I assume so, given the title), and if so, what behaviour exactly changed?

`pd.offsets.Minute(3) < pd.Timedelta(4)` will now return `False` instead of raising `TypeError`
looks fine, can you add a whatsnew note
updated with whatsnew + green
CI: Fix botocore Error
State: closed

- [x] closes #34086

AWS Service Exceptions are not statically defined in boto3. Easiest to just catch a ClientError here.

Ref Docs:
https://boto3.amazonaws.com/v1/documentation/api/latest/guide/error-handling.html#parsing-error-responses-and-catching-exceptions-from-aws-services

Labels:
name: CI
description: Continuous Integration
name: IO Network
description: Local or Cloud (AWS, GCS, etc.) IO Issues
name: IO Parquet
description: parquet, feather

Comments:
Hello @alimcmaster1! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:



There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2020-05-09 11:48:53 UTC
also need to catch in test_write_s3_parquet_fails
https://github.com/pandas-dev/pandas/pull/34087/checks?check_run_id=658833815
Azure CI has run here^

Fixed up parquet test too and added boto docs on dynamic errors here

@jbrockmendel @jreback 
thanks @alimcmaster1  for the quick response
Simplify Reducer.get_result
State: closed

by using nditer this makes some progress towards #34014

Benchmarks for frame_apply looks as follows:

```sh
       before           after         ratio
     [3ed7dff4]       [5e77e1ed]
     <master>         <reduce-simplify> 
!      9.39±0.3ms           failed      n/a  frame_methods.Apply.time_apply_ref_by_name
-      8.52±0.2ms       2.87±0.2ms     0.34  frame_methods.Apply.time_apply_lambda_mean
-      9.32±0.2ms       2.96±0.3ms     0.32  frame_methods.Apply.time_apply_np_mean
```

I'm not quite clear yet why the one benchmark is failing. Running it over a thousands loops in the REPL seems fine, but maybe CI will help detect something

Labels:
name: Clean

Comments:
Closing as haven't had time to look but hope to reopen at a later date, since I think this is required for Cython 3
REF: share code for scalar validation in datetimelike array methods
State: closed

The user-facing effect here is in the exception messages we show to users, where we may want to be more verbose than this.  Thoughts @gfyoung?

Labels:
name: Error Reporting
description: Incorrect or improved errors from pandas

Comments:
> The user-facing effect here is in the exception messages we show to users, where we may want to be more verbose than this. Thoughts @gfyoung?

I agree that we would want to improve the user-friendliness of these error messages.  From that perspective, I prefer the error messages that were there before.
BUG: Fixed __join_multi always returning indexers (#34074)
State: closed

- [x] closes #34074
- [x] tests added / passed
- [ ] passes `black pandas`
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry


Labels:
name: Bug
name: MultiIndex

Comments:
Hello @CuylenE! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:





There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2020-05-11 15:56:26 UTC
thanks @CuylenE
add test for setitem from duplicate axis
State: closed

- [x] closes #34034
- [ ] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves

Comments:
Thanks @CloseChoice for the PR. lgtm. can you move the test to pandas\tests\indexing\test_loc.py
> Thanks @CloseChoice for the PR. lgtm. can you move the test to pandas\tests\indexing\test_loc.py

Done.
Thanks @CloseChoice!