BrandonFors/pandas-github-issues huggingface dataset:
BUG: Fixed strange behaviour of pd.DataFrame.drop() with inplace argu…
State: closed

closes  #30484

- [x] closes #xxxx
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry

This is my first PR, so pardon me if I did something wrong.

To fix this issue I started by creating a failing test based on the example given by the reporter. After, I fixed the test by removing the "inplace" special methods from add_special_arithmetic_methods, following what was suggested as a comment in the code.

Labels:
name: Bug
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode

Comments:
The code that you removed is used in many different files.  If you check the Travis CI logs you will see a list of failures.  Below are a few places where those methods are used:

pandas/tests/frame/test_operators.py
pandas/core/generic.py

You will probably need to either change the current implementation or reimplement those methods elsewhere.
Thank you guys! I've run some tests before the PR, so it turns out I was running the wrong ones. I'll take another look and try to find a better solution.
I've now fixed the problem by adding a new argument to NDFrame._maybe_update_cacher.

All the test are running fine, but the pipeline fails in building the documentation:
AttributeError: module 'IPython.utils.py3compat' has no attribute 'cast_bytes_py2'

Is this related to something I've done? Can you provide some guidance on how to solve this?
Looking further in the azure logs I see that the build is failing because of the following file:

doc/source/user_guide/enhancingperf.rst

I would look through that file and see if anything that you changed is referenced there.
Thanks for the remarks and for the review. I'll get back to it as soon as I have time
HI @rjfs - sorry to chase you up, just wanted to ask whether you're still working on this :)
Hi @MarcoGorelli, no problem!
I've been away during last days, but I'll get back to it during this week.
looks good. can you add a note in 1.1, reshaping bug fixes. ping on green.
> looks good. can you add a note in 1.1, reshaping bug fixes. ping on green.

It's done, thanks!
@rjfs can you rebase once again, azure pipelines are not triggering fro some reason; I think its fixed.
> @rjfs can you rebase once again, azure pipelines are not triggering fro some reason; I think its fixed.

still not triggering it seems...
/azp run
<samp>
Azure Pipelines successfully started running 1 pipeline(s).<br>

</samp>
lgtm. let's merge on green.
pls merge master and a lint issue: 
```
##[error]./pandas/tests/indexes/timedeltas/test_constructors.py:173:String has a space at the beginning instead of the end of the previous string.
```

ping on green.
Hello @rjfs! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:





There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2020-03-26 10:03:05 UTC
ping @jreback 
thanks @rjfs
DOC: Make pyplot import explicit in the 10 minutes to pandas page
State: closed

Beginners don't know what is `plt`, thus it is better to show how to import it explicitly.

This is a continuation of https://github.com/pandas-dev/pandas/pull/30274

- [ ] closes #xxxx
- [ ] tests added / passed
- [ ] passes `black pandas`
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: Docs

Comments:
@datapythonista Could you please merge this PR? Thank you.

> AttributeError: module 'IPython.utils.py3compat' has no attribute 'cast_bytes_py2'

pandas-dev.pandas (Web_and_Docs) is failing, but is it related to my change?

https://dev.azure.com/pandas-dev/pandas/_build/results?buildId=24127&view=logs&j=5291ec4e-9543-503b-43ce-895cf0e06ba3&t=ba0a2b35-e1f8-54df-fd66-2b9fde6f4b1f&l=5158
>pandas-dev.pandas (Web_and_Docs) is failing, but is it related to my change?

I also have the same problem. See #30282
@yuseitahara can you merge master into your branch please? It should fix the CI.
@jreback you're changes have been addressed, all green here.
thanks @yuseitahara
BLD: catch missing cython instead of tempita NameError
State: closed

- [x] closes #28836

I named this branch "bld-misc" in the optimistic hope I could close a bunch of BLD issues, but nope, just this one.

Labels:
name: Build
description: Library building on various platforms

Comments:
Looks good to me. I find the global `Cython` and `CYTHON_INSTALLED` variables a little strange, so this definitely helps

Anything in particular you are thinking we need to improve on with this?
> I find the global Cython and CYTHON_INSTALLED variables a little strange

Yah I think you're right that they are redundant.  I'd be on board with having just one of them.
> > I find the global Cython and CYTHON_INSTALLED variables a little strange
> 
> Yah I think you're right that they are redundant. I'd be on board with having just one of them.

yep agreed
Improve ISO Date Performance for JSON
State: closed

benchmarks below 

```sh
       before           after         ratio
     [9c6771c5]       [5c0f5682]
     <master>         <json-index-dates>
-        231±60ms          188±3ms     0.82  io.json.ToJSON.time_iso_format('split', 'df_td_int_ts')
-        496±20ms         218±40ms     0.44  io.json.ToJSON.time_iso_format('index', 'df_int_float_str')
-        486±20ms          207±2ms     0.42  io.json.ToJSON.time_iso_format('columns', 'df_int_float_str')
-         499±9ms          210±1ms     0.42  io.json.ToJSON.time_iso_format('index', 'df_date_idx')
-        503±20ms          206±2ms     0.41  io.json.ToJSON.time_iso_format('columns', 'df_int_floats')
-        515±20ms          210±1ms     0.41  io.json.ToJSON.time_iso_format('index', 'df')
-        524±80ms        209±0.5ms     0.40  io.json.ToJSON.time_iso_format('index', 'df_int_floats')
-        528±10ms          206±2ms     0.39  io.json.ToJSON.time_iso_format('columns', 'df_td_int_ts')
diff --git a/asv_bench/asv.conf.json b/asv_bench/asv.conf.json
index c04bbf53a..897de2f85 100644
--- a/asv_bench/asv.conf.json
+++ b/asv_bench/asv.conf.json
-        546±80ms        209±0.7ms     0.38  io.json.ToJSON.time_iso_format('index', 'df_td_int_ts')
-        568±60ms          208±3ms     0.37  io.json.ToJSON.time_iso_format('columns', 'df_date_idx')
-       598±100ms          206±2ms     0.35  io.json.ToJSON.time_iso_format('columns', 'df')

SOME BENCHMARKS HAVE CHANGED SIGNIFICANTLY.
PERFORMANCE INCREASED.
```

Note that this mostly improves on DTI which on 0.25.3 can't even be written as ISO format, so I didn't add a whatsnew. Timedelta is the big bottleneck remaining

Labels:
name: Datetime
description: Datetime data dtype
name: IO JSON
description: read_json, to_json, json_normalize

Comments:
CI notwithstanding, this looks like a nice cleanup+speedup
Hello @WillAyd! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:





There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2020-01-02 00:00:36 UTC
LGTM
thanks, might be worth a general whatsnew note for perf for json changes (e.g. list issues that impacted perf that you have done recently)
Yea no problem; will address in follow up
INT: Implement _set_freq for TDI/DTI
State: closed

- [x] closes #29843
- [ ] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: Refactor
description: Internal refactoring of code

Comments:
In #29604 I had to do the same methods twice for ``DateTimeIndex.difference`` and ``TimeDeltaIndex.difference``, because ``PeriodIndex.difference`` worked differently.

Could you see if it works to add a difference method to this MixIn and remove the other two?
> Could you see if it works to add a difference method to this MixIn and remove the other two?

Yes, in a follow-up I plan to see what else can be de-duplicated
Thanks, @jbrockmendel .
BUG: compound dtype ndarray passed to Series
State: closed

- [x] closes #13296
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: Dtype Conversions
description: Unexpected or buggy dtype conversions
name: Error Reporting
description: Incorrect or improved errors from pandas

Comments:
lgtm.
Thanks @jbrockmendel
BUG: The setting xrot=0 in DataFrame.hist() doesn't work with by and subplots #30288
State: closed

- [x] closes #30288
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: Bug
name: Visualization
description: plotting

Comments:
@jreback, I have updated the whatsnew entry. 
great thanks @souvik3333
DEPR: Deprecate pandas.datetime
State: closed

- [x] closes #30610
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry


Labels:
name: Deprecate
description: Functionality to remove in pandas

Comments:
Hello @ryankarlos! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:





















There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2020-01-02 02:08:34 UTC
having some issues with adding this Datetime class for py36 - was trying to keep it consistent with the way np  is handled - i get an error when trying to call pd.datetime in the test - can do pd.datetime.datetime although then  there is the problem of warning not been generated due to __getattr_ not being accessedd correctly.  

Also, original issue i had linked this PR with is now closed - do i need to open a new one for datetime ?
@ryankarlos 
The intention of the test_depr() function is to test that the __getattr__() function of the wrapper class(e.g. __numpy() or __datetime() in your case) raises a warning. 
> Also, original issue i had linked this PR with is now closed - do i need to open a new one for datetime ?


Yes please - and link the original issue. Also now that https://github.com/pandas-dev/pandas/pull/30386 has been merged the implementation could be simplified - similar to what we do for SparseSeries/SparseDataFrame.

Also can you merge master + take a look at test failures.

Thanks!
@alimcmaster1  @datapythonista all green now - think ive addressed the required changes.
very nice @ryankarlos thanks!
CLN/DOC: standardize wording on previous-removals
State: closed

In the 1.0.0 whatsnew we currently have 3 variants of "Removed :foo:X"

```
- Removed the previously deprecated :meth:`X`
- Removed previously deprecated :meth:`X`
- Removed :meth:`X`
```

I think we should standardize on one of these (lest readers think the difference is meaningful).  I don't particularly care which, but for this PR went with the last one as it is the least verbose

Labels:
name: Docs
name: Deprecate
description: Functionality to remove in pandas

Comments:
The next step in this process would be to standardize wording for "removed keyword X from Y"
thanks
PEP 3118 Buffer Protocol Support for Series
State: closed

Just a POC and not sure we want this but was curious nonetheless...

This extension module allows a Series to conform to the buffer protocol, which makes it readable / writeable through memoryviews. Right now in places like groupby we always in user code extract out the underlying array and pass that through Cython, which this in theory could centralize while helping third party code that may use this as well

Getting this to work with a DataFrame would be the bigger win but also takes more effort, so leaving that for another day

Labels:
name: Enhancement
name: Needs Discussion
description: Requires discussion from core team before further action
name: Series
description: Series data structure

Comments:
Wow, when I saw the title I thought this would be about correctly handling memoryviews in the constructor.

I'm not wild about the to_numpy usage.  In general we dont know if that is going to be lossy and/or expensive.

If there is a way to get our cython funcs to work directly on DTA/TDA/PA without casting to i8, that I could get excited about.
The datetimelike arrays still use numpy for strorage correct? Certainly open to alternate ways besides `to_numpy` but it needs to be low level storage and the advantage of numpy arrays currently is we can just dispatch to them as they already implement this
Surprised this cause build failures like:

https://dev.azure.com/pandas-dev/pandas/_build/results?buildId=23988&view=logs&j=a3a13ea8-7cf0-5bdb-71bb-6ac8830ae35c&t=fabfbac7-9f65-559f-bcbb-d243c750d1d9&l=244

Must be changing the semantics of assignment somehow if a Series on the rhs was previously not using the Buffer protocol but now is. Have to investigate further
> ValueError: cannot include dtype 'm' in a buffer

That would correspond to a timedelta64 dtype, which I dont think cython has a concept of.
Does this just make `Series.values` available, and the index is ignored?

What happens for EA-backed Series that don't / can't implement the buffer protocol? 
This current implementation just calls `to_numpy` and uses whatever buffer that exposes, but yea if we evolve and go another path in the future would have to think about how EAs could use, if at all. 

It does need to be a platform native representation of the data which maybe not all EAs can produce; if we switched off of `to_numpy` would probably need to raise where not applicable
Come back to this one later; might even need to do upstream. clearing queue for now
Bug groupby quantile listlike q and int columns
State: closed

When columns are integers, `df.groupby(label).quantile(<arraylike>)` fails.

- [x] closes #30289
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry


Labels:
name: Bug
name: Groupby
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode
name: MultiIndex

Comments:
@jreback I know this isn't exactly what you thought of per our discussion here (#30462). This is by far the cleanest implementation I can think of. Reason being:
1. I'm hoping to keep the function call to `_get_cythonized_result` the same whether q is scalar or not. Hence we would end of having a list of dataframes. They may have `Index` or `MultiIndex` with them so I feel it is not so clean to build index for them one-by-one. 
1. If we concatenate them first, using `reorder_levels` method seems the most natural way of doing things. There's no clean way (that I know of) to place the quantile index level inside before concatenating them. 

On a side note, I refactored the original for-loop which is slow and not as readable and used numpy to construct the same array (`indices` variable).
Let me know what you think, and I will adjust accordingly!
thanks @fujiaxiang nicely done! keep em coming!
DEPR. Change RangeIndex._start/_stop/_step to FutureWarning
State: closed

- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry


Labels:
name: Deprecate
description: Functionality to remove in pandas
name: Index
description: Related to the Index class or subclasses

Comments:
lgtm. can you rebase. (also make sure to add to the deprecation issue tracker)
lgtm pending rebase
Rebased and green
Thanks @mroeschke
CI: Fix GBQ Tests
State: closed

Believe these test only run on master on travis? So will keep xfail for now

Fixturize as per @jreback comment [here](https://github.com/pandas-dev/pandas/issues/30470#issuecomment-568942914).

closes #30470 

Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: CI
description: Continuous Integration

Comments:
I've configured travis to run against my fork with the relevant setup for gbq. (As per the instruction here https://github.com/pandas-dev/pandas/pull/15879/files). GBQ Tests are now running against these changes here:

https://travis-ci.org/alimcmaster1/pandas/jobs/629696201?utm_medium=notification&utm_source=email
These tests are running against my fork [here](https://travis-ci.org/alimcmaster1/pandas/builds/629812975?utm_medium=notification&utm_source=email) - all green and they don't appear in the skipped test section in logs


thanks @alimcmaster1 
cross fingers
> thanks @alimcmaster1
> cross fingers

Thanks @jreback I will keep an eye on travis..
Happy days all green - https://travis-ci.org/pandas-dev/pandas/builds/630051550?utm_medium=notification&utm_source=email
yes thanks @alimcmaster1 
cc @alimcmaster1 see for example https://travis-ci.org/pandas-dev/pandas/jobs/631599036

we need a try/except around the close part i think and/or a unique dataset name for each test as we are confliciting
> cc @alimcmaster1 see for example https://travis-ci.org/pandas-dev/pandas/jobs/631599036
> 
> we need a try/except around the close part i think and/or a unique dataset name for each test as we are confliciting

Aaah. I’ll be able to take a look at this tomorrow eve will revert back on here
thanks @alimcmaster1
CLN: assorted cleanups
State: closed


Labels:
name: Clean

Comments:
Hello @jbrockmendel! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:

















There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2019-12-26 00:10:11 UTC
lgtm. ping on green.
ping
thanks
TST: Xfailing broken GBQ test
State: closed

xref #30470

Test recently introduced in #30447 seems to be failing in master. Xfailing it for now.

CC: @jreback 

Labels:
name: Testing
description: pandas testing functions or related to the test suite

Comments:
thanks @datapythonista 
thanks @datapythonista 
hmm https://travis-ci.org/pandas-dev/pandas/jobs/629507853 this must be called multiple times.
@jreback I think the problem is the test is now passing.

```
_ TestToGBQIntegrationWithServiceAccountKeyPath.test_gbq_if_exists[append-300] _
[gw1] linux -- Python 3.6.6 /home/travis/miniconda3/envs/pandas-dev/bin/python
[XPASS(strict)] Test breaking master
```

https://travis-ci.org/pandas-dev/pandas/jobs/629502988

and we have `xfail_strict = True` set in our setup.cfg. 

Ill turn strict off for this test
CLN: OrderedDict -> Dict
State: closed

- [x] ref #30469

Labels:
name: Code Style
description: Code style, linting, code_checks
name: Clean

Comments:
lgtm ping when ready. don't have to do them all in a single PR
> lgtm ping when ready. don't have to do them all in a single PR

Thanks ready - will address the renaming cases + code check in a follow up. 
thanks @alimcmaster1
CLN: Orderdict -> defaultdict(list)
State: closed

- [x] ref #30469
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: Code Style
description: Code style, linting, code_checks
name: Clean

Comments:
> lgtm, thanks for cleaning this @MomIsBestFriend

Thanks for the detailed explanation, and the patience ;)
thanks @MomIsBestFriend
CI: Add test case for unwanted patterns
State: closed

- [x] closes #30454
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry

Can be merged, after merging #30464

Labels:
name: CI
description: Continuous Integration

Comments:
This is really nice, thanks @MomIsBestFriend.  I was _not_ looking forward to figuring out the `ast` stuff.

@datapythonista is there someone on one of the flake/lint projects we might want to talk to about upstreaming this?
This is really nice, thanks @MomIsBestFriend.  I was _not_ looking forward to figuring out the `ast` stuff.

@datapythonista is there someone on one of the flake/lint projects we might want to talk to about upstreaming this?
@datapythonista It's looking like the running version of python in github's action is ```2.7.15```, is it possible to increase it?

> @datapythonista It's looking like the running version of python in github's action is `2.7.15`, is it possible to increase it?

Where does it say this out of interest? It must be at least 3.6 as we run `validate_docstring.py` in github actions?
> > @datapythonista It's looking like the running version of python in github's action is `2.7.15`, is it possible to increase it?
> 
> Where does it say this out of interest? It must be at least 3.6 as we run `validate_docstring.py` in github actions?

Line 52 in the github actions under ```Looking for unwanted patterns``` section.

![python_version_github_actions](https://user-images.githubusercontent.com/50263213/71488539-c6111580-2829-11ea-9d46-480b2744d315.png)

(Screenshot because I squashed the original commit)
We never use the system python, we install conda in every build, and create an environment, you'll have to activate it.

I'd yield the values as a namedtuple. 
> and create an environment, you'll have to activate it.


The problem is, the activation of conda is happening after it's running the unwanted patterns check. Do you have any suggestions?
Better move the string thing in the linting, which is a good idea anyway. 
Can be merged after merging #30579
Merged #30579, you can update your branch, so the CI passes
Thank you for the very close guidance @datapythonista 
thanks @MomIsBestFriend