BrandonFors/pandas-github-issues huggingface dataset:
ENH: Allow plotting backend to be an option
State: closed

- [x] closes #28619
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry


Labels:
name: Visualization
description: plotting

Comments:
Hello @jsignell! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:





There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2019-11-05 15:39:27 UTC
Hmm I haven't added any docs. I suppose there should at least be a mention in the docstring.
I think this is ready.
@jsignell can you rebase
Thanks @jsignell!
DEPR: Deprecate Index.set_value
State: closed

- [ ] closes #xxxx
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry

Deprecates ``Index.set_value``. This is a very little used and confusing method IMO.

For a given index ``idx``, array ``arr``,  value ``idx_val`` in ``idx`` and a new value of ``val``, ``idx.set_value(arr, idx_val, val)``  is equivalent to ``arr[idx.get_loc(idx_val) = val``, which is more standard and should be used instead.


Labels:
name: Deprecate
description: Functionality to remove in pandas
name: Index
description: Related to the Index class or subclasses

Comments:
There has been some discussion before about `Index.get_value`: https://github.com/pandas-dev/pandas/issues/19728, where the thought was that this might be relied upon.  
But in contrast to get_value, set_value is apparently something we don't use internally ourselves?

cc @shoyer
Yeah that's right, ``Index.set_value`` is not used by pandas itself.
I've updated the PR according to comments.

In addition, I've added ``"set_value"`` to ``Index._deprecations``.
Thanks!
Backport PR #28614 on branch 0.25.x (CI: troubleshoot s3fs failures)
State: closed

Backport PR #28614: CI: troubleshoot s3fs failures

Labels:

Comments:
Not sure what is up with py36 failure. If someone has time to debug would certainly be welcome
Failure is in read_parquet, traceback goes to a .pxi file in pyarrow and ends with "???" which usually means its in C.  cc @jorisvandenbossche 
That's a know problem for certain versions of fastparquet, we skip those tests on master, but should probably backport that skip: https://github.com/pandas-dev/pandas/pull/28357
Triggered backport: https://github.com/pandas-dev/pandas/pull/28627

I restarted the failed builds, and now all green, so merging
CI: troubleshoot s3fs failures
State: closed

xref #28612.  Recent failures used 0.3.4, the last passing build I see used 0.2.0

Labels:
name: CI
description: Continuous Integration

Comments:
Won't be able to take a look today. Feel free to merge whenever.

cc @martindurant.
Thanks @jbrockmendel 
Locally, with pandas 0.25.0, s3fs master and fsspec master, a dataframe round-trips to S3 just fine.
BUG: value_counts can handle the case even with empty groups (#28479)
State: closed

* If applying rep to recons_labels go fail, use ids which has no
  consecutive duplicates instead. 

- [x] closes #28479 
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry

xuancong84 found that value_counts() crashes if `groupby` object contains empty groups.
However, even though I made the construction of DataFrame don't skip empty rows, it still crashed.
Till then, I already tried in many ways though, in this time I tried to correct the callee `self.grouper.recons_labels`. After several tests, I found that If freq of `Grouper` is too long so that it has empty groups in some periods then it crashes. And also have found that this is solved by using `ids` which has no consecutive duplicates instead of `self.grouper.recons_labels`.

Labels:

Comments:
Why did they go fail..?? I have never changed any contents of test_categorical.py though... and apart from me, he also has the same problem. https://github.com/pandas-dev/pandas/pull/28595

Is there any problem with CI now?
I closed this issue because of CI issues
BUG: value_counts can handle the case even with empty groups (#28479)
State: closed

* If applying rep to recons_labels go fail, use ids which has no
  consecutive duplicates instead. 

- [x] closes #28479 
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry

xuancong84 found that value_counts() crashes if `groupby` object contains empty groups.
However, even though I made the construction of DataFrame don't skip empty rows, it still crashed.
Till then, I already tried in many ways though, in this time I tried to correct the callee `self.grouper.recons_labels`. After several tests, I found that If freq of `Grouper` is too long so that it has empty groups in some periods then it crashes. And also have found that this is solved by using `ids` which has no consecutive duplicates instead of `self.grouper.recons_labels`.

Labels:

Comments:
Hello @0xF4D3C0D3! Thanks for opening this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:



* In the file [`pandas/tests/groupby/test_value_counts.py`](https://github.com/pandas-dev/pandas/blob/15635d4e1cc6e36f052ba5d614fe657da03785f7/pandas/tests/groupby/test_value_counts.py):

> [Line 83:1](https://github.com/pandas-dev/pandas/blob/15635d4e1cc6e36f052ba5d614fe657da03785f7/pandas/tests/groupby/test_value_counts.py#L83): [E302](https://duckduckgo.com/?q=pep8%20E302) expected 2 blank lines, found 1
> [Line 106:1](https://github.com/pandas-dev/pandas/blob/15635d4e1cc6e36f052ba5d614fe657da03785f7/pandas/tests/groupby/test_value_counts.py#L106): [W391](https://duckduckgo.com/?q=pep8%20W391) blank line at end of file


I'm sorry. I'll issue PR later again because I made some mistakes
fix unnecessary sort in pd.read_json and orient="index"
State: closed

- [x] closes #28557 
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry


Labels:
name: IO JSON
description: read_json, to_json, json_normalize
name: Compat
description: pandas objects compatability with Numpy or Python functions

Comments:
Looks simpler - nice job! Can you also add a whatsnew for v1.0.0?
Of course.
Looks like an isort failure is causing CI to go red, but outside of that this change lgtm
Looks like you still have an isort error. Can you run `isort pandas/io/json/_json.py` locally and push up the result?
Thanks @KimDoubleB - great PR and congrats on first contribution!
Thanks so much. @WillAyd 
I’m really grateful for your help.
OPS: Remove mask_cmp_op fallback behavior
State: closed

- [x] closes #28079 
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry

xref #28050. 

To make the Series vs DataFrame behavior consistent, the two main options are a) change the DataFrame behavior (this PR) or b) change the Series behavior.  The latter is complicated by the fact that for object dtypes the Series comparisons go through comp_method_OBJECT_ARRAY instead of the numpy op, so we would still have to change the complex-case test changed here.

Labels:
name: API Design

Comments:
lgtm. I think this needs a note (you can just put in bug fixes would be fine); as something that 'worked' before now will raise (correctly)
note added
thanks minor comment for followup
@jbrockmendel you mentioned that you changed the behaviour of the flex method for Series or DataFrame (~~the example in the issue also shows one without complex dtype~~). Shouldn't that be reflected in the whatsnew note?
Updated question: this was only for the comparison *methods* and not the comparison *operation*, correct? If so, I would be clearer on that in the whatsnew note.
> you mentioned that you changed the behaviour of the flex method for Series or DataFrame

`DataFrame`'s `lt, le, gt, ge` methods are affected.  AFAICT the only case where it really makes a difference is with complex-dtypes that used to get cast to object-dtype for masking fallback, because numpy (for some reason) does complex-dtype comparisons differently from python.

Which is a long way of saying I don't understand the question.  Do you have a suggested edit for the whatsnew note?

> this was only for the comparison methods and not the comparison operation, correct? I

I don't understand the distinction.  Can you give an example?
> DataFrame's lt, le, gt, ge methods are affected

As you say, that are methods. And with the comparison operators I mean >, <, .. An essential difference in this case, and if you just say "comparison" most people will think about the second IMO
How would you word the note?  Adding the word "flex" is a start.
REF/TST: Corner cases for op(DataFrame, Series)
State: closed

We have two special cases in `_combine_series_frame` that are never tested ATM.  This adds tests for them, then notes that the special case handling code is unnecessary and this can now fall through to the general case code.  In a follow-up, we'll be able to simplify _combine_series_frame further, but for now I want to keep the changed logic obvious.

Also note that the removed
```
         if not len(other):
            return self * np.nan 
```

is actually wrong in the non-float test case in L681-685 in the test file.

Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Compat
description: pandas objects compatability with Numpy or Python functions

Comments:
gentle ping.  The follow-ups to this are both a) really nice code-removals and b) orthogonal to the rest of the current ops stuff.
great thanks
Fix docstring error
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [ ] passes `black pandas`
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: Docs

Comments:
Hello Marc @datapythonista .
Kindly confirm if this is as discussed.
Thank you
Do we rather want to update the validation script to allow this? (because now we are duplicating an explanation that almost nobody needs)
@jorisvandenbossche . That will be faster especially for the *args and **kwargs. Some of the functions with PR02 errors have totally different parameters from their docstrings. 
I don't know how to go about updating the validation script. If you have the time, you can put me through.
Agree with Joris. Any other of the errors in the parameters can be fixed in
the docstrings, but having args and kwargs together is a false positive of
the validation, we should update the script.

I think I opened an issue for this, and not sure if someone was already
working on it. Can you have a look and see if you can find that issue, not
sure if in the mentoring repo or in pandas itself. I'm in my phone with
poor internet connection right now, but will help you later if you can't
find it.

On Wed, 25 Sep 2019, 13:29 Wuraola Oyewusi, <notifications@github.com>
wrote:

> @jorisvandenbossche <https://github.com/jorisvandenbossche> . That will
> be faster especially for the *args and **kwargs. Some of the functions with
> PR02 errors have totally different parameters from their docstrings.
> I don't know how to go about updating the validation script. If you have
> the time, you can put me through.
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <https://github.com/pandas-dev/pandas/pull/28598?email_source=notifications&email_token=ACMXUAC7LEBH6XOYOU2LG5LQLNKRBA5CNFSM4I2CJ5E2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD7RWXXA#issuecomment-534997980>,
> or mute the thread
> <https://github.com/notifications/unsubscribe-auth/ACMXUAD3KWZIR4BLG2HRE3DQLNKRBANCNFSM4I2CJ5EQ>
> .
>

Ok marc. I will check 
#28811 was just merged and should allow for this, so I don't think this PR is applicable any more.

Thanks of course for the submission @WuraolaOyewusi - there are plenty more out there would welcome contributions on
DOC: Add into about empty results in sample
State: closed

As referred to in https://github.com/pandas-dev/pandas/issues/28495.

- [x] closes #28495
- [x] tests added / passed
- [ ] passes `black pandas`
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: Docs

Comments:
Hello @roelhogervorst! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:



There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2019-09-24 06:44:10 UTC
Can you provide something more descriptive? This as is is rather ambiguous
Closing as stale but @roelhogervorst ping if you'd like to pick back up
CLN: unify __finalize__ treatment for Series ops
State: closed

ATM we have three slightly different usages for the arithmetic, comparison and logical ops, with no clear reason for the differences.  This changes that to make the arithmetic the one true version, which has the added benefit of letting us share _construct_result code across these three functions.


Labels:
name: Clean

Comments:
any additional tests needed?
> any additional tests needed?

I'm inclined towards "no" because the behavior that is affected is stuff we don't particularly care about.
Just to verify, we don't have any cases where we pass a DataFrame to `Series.__finalize__` right?

On my disallow-duplicates I noticed some strange behavior with DataFrame's having a `name` column, since we did `self.name = getattr(other, 'name', None)`, which tried to set `Series.name` to the `name` column in the DataFrame.
> Just to verify, we don't have any cases where we pass a DataFrame to Series.__finalize__ right?

Not in this file, no.
LGTM then.

On Tue, Sep 24, 2019 at 11:10 AM jbrockmendel <notifications@github.com>
wrote:

> Just to verify, we don't have any cases where we pass a DataFrame to
> Series.*finalize* right?
>
> Not in this file, no.
>
> —
> You are receiving this because you commented.
> Reply to this email directly, view it on GitHub
> <https://github.com/pandas-dev/pandas/pull/28590?email_source=notifications&email_token=AAKAOIUHL3G7RCOKTJFBBX3QLI3YNA5CNFSM4IZXYXJKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD7O5DPQ#issuecomment-534630846>,
> or mute the thread
> <https://github.com/notifications/unsubscribe-auth/AAKAOIXHGA2V44FSS3LV66DQLI3YNANCNFSM4IZXYXJA>
> .
>

thanks!
Fix typo in class DataFrame
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [ ] passes `black pandas`
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry

This is basically a version of #28579 for the master branch.

Labels:
name: Docs

Comments:
Thanks @felixdivo 
You're welcome :)
PERF: Implement DataFrame-with-scalar ops block-wise
State: closed

One of four cases we'll need to implement (the others being Series-align-index, Series-align-columns, and DataFrame).

~670x speedup on the fastest ops, ~8x on the slower end.
```
In [3]: arr = np.arange(10**5).reshape(100, 1000)                                                               
In [4]: df = pd.DataFrame(arr)                                                                                  
In [5]: %timeit df + 1                                              
198 ms ± 2.17 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)         # <-- master                                            
294 µs ± 3.25 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)  # <-- PR

In [8]: ts = pd.Timestamp.now("UTC")                                                                            
In [9]: df2 = pd.DataFrame(arr.view("timedelta64[ns]"))                                                         
In [10]: %timeit ts - df2                                                                                       
319 ms ± 2.49 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)      # <-- master
40.2 ms ± 622 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)   # <-- PR
```

The 2D DTA and TDAs that get created exist only briefly, and the changes to these classes are about as minimal as I can make them.

Labels:
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode

Comments:
Hello @jbrockmendel! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:



















There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2019-10-09 23:58:19 UTC
There are three more cases after this: Series[axis="rows"], Series[axis="columns], and DataFrame.  The axis="rows" case can share most of the implementation here.  The other two cases not so much.

Question: do we want to do this in 4ish steps, or iterate in this PR until we get them all implemented?
will have a look soon.
Closing to clear the queue; I'll be opening something similar (but implemented largely in internals) before long.
Fix typo
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [ ] passes `black pandas`
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: Docs

Comments:
thanks @felixdivo 
@felixdivo this was applied to the 0.25.x branch. If you want it to be fixed for all versions going forward you'll need to make a PR against master.
grr didn't notice this was not master :-<
The GitHub UI doesn't make it obvious :/ Only saw it in my email.
Okay sure, I didn't get your branching model. I'll apply it to master as well, that's the more important one. :smile:
DOC: Add scaling to large datasets section
State: closed

Closes https://github.com/pandas-dev/pandas/issues/28315

Labels:
name: Docs

Comments:
Thanks for addressing comments. Generally looks good and I think a welcome addition to docs
And thank you for the review :)

FYI, this adds ~30s to the doc build on my machine (haven't checked the slowdown on CI, but presumably its longer). Is that unacceptable? I can make the examples smaller if needed.
FYI, rendered version:

![FireShot Capture 007 - Scaling to large datasets — pandas 0 25 0+418 ga7fb97f44a dirty documen_ - ](https://user-images.githubusercontent.com/1312546/65462754-0f615b80-de1c-11e9-88c1-a939e6fb8aab.png)

Any other thoughts here?
> Not for this PR, but if we have an additional level of navigation, we could put this together with the current enhancingperf.rst in a "performance" section?

I slightly prefer smaller doc pages that cross reference eachother.

I've removed the section comparing the perf of read_csv and read_parquet. That muddies the purpose of this document, since it's talking about speed, while the rest of the document focuses primarily on memory usage.

I've updated the "subset of columns" example to use the `%memit` magic from memory_profiler.
flake8-rst didn't like jupyter magics, so I've had to remove it.
> > Not for this PR, but if we have an additional level of navigation, we could put this together with the current enhancingperf.rst in a "performance" section?
>
> I slightly prefer smaller doc pages that cross reference eachother.

Sorry, I was not very clear :) I didn't mean to suggest to put them in a single file. Because I also prefer smaller doc pages, I want to split some of the existing ones, and create an extra level of hierarchy in the user guide. And in that light, this one could fit together with the existing enhancingperf (which could be splitted for the cython stuff and the querying) in a general "Performance" section (that has multiple sub pages). But that's for the future, not now, so nothing to care about for this PR.

> I've removed the section comparing the perf of read_csv and read_parquet. That muddies the purpose of this document, since it's talking about speed, while the rest of the document focuses primarily on memory usage.

I actually think that section could still be useful. read_csv vs read_parquet is not only for speed, also memory wise I would think that read_parquet is quite a bit better? (I heard about read_csv needing quite some more memory than the original file or final dataframe) So maybe if using `%memit` instead of `%timeit` would illustrate that?
lgtm, needs a rebase though.
All green here.
Thanks.
Not that it needed to be in this PR (could be a follow-up), but an opinion on the usefulness of what I remarked in https://github.com/pandas-dev/pandas/pull/28577#issuecomment-535467605 (the second paragraph on reading files)
[WIP] Annotate DataFrame (Part 3)
State: closed

- [x] part of #26792 
- [ ] tests added / passed
- [x] passes `black pandas`
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: Typing
description: type annotations, mypy/pyright type checking

Comments:
closed as stale. @vaibhavhrt ping if you wish to continue.
Sorry been really busy last few weeks. Will be busy next few days as well,
but I do wish to continue and complete it, just give me a few days I will
reopen the PR.

On Mon, 21 Oct, 2019, 1:54 PM Simon Hawkins, <notifications@github.com>
wrote:

> closed as stale. @vaibhavhrt <https://github.com/vaibhavhrt> ping if you
> wish to continue.
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <https://github.com/pandas-dev/pandas/pull/28575?email_source=notifications&email_token=AFMJMLAM5YV36L7DKU5MNITQPVRNPA5CNFSM4IZHCCMKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEBZPMKA#issuecomment-544405032>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AFMJMLE5JRIIZWAABGYVX6LQPVRNPANCNFSM4IZHCCMA>
> .
>

removing from draft, otherwise I keep forgetting about it. Will try it get it done fast now.
closing as stale, ping if you want to continue. 
@WillAyd are we working on this (and https://github.com/pandas-dev/pandas/issues/26792)?
I can help out here, I really want to see type checking with a df.
@stangirala this PR specifically is stale; if you have contributions you'd like to make for that feel free to push a separate PR
@WillAyd can I use this PR as a template?
Add default parameter for get_group
State: closed

- [x] closes #9299
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:

Comments:
This is for #9299, but there was a previous attempt in #5452 where it was suggested in the final comments that perhaps it should be similar to how `#get()` works.

My initial attempt simply returns the `default` when not `None`. The `#get()`actually catches exceptions and returns the default instead.

https://github.com/pandas-dev/pandas/blob/12859382a4a5325e6fd19a209b128bab44a3a37a/pandas/core/generic.py#L3818-L3821

Let me know if you think that would be a better approach, and I will update it :+1: 

Thanks!
Bruno
Thinking through this the code itself is good I guess I'm just wondering why this is generally useful? Is there a real use case for this that you can think of?
>Thinking through this the code itself is good I guess I'm just wondering why this is generally useful? Is there a real use case for this that you can think of?

This issue never affected me. The use case for this appears to be related to code parallelization, reported in #9299
Read through the history of that and I guess same thing where it's unclear how this is practical. I see the consistency with `.get` being a motivating factor but I don't understand why returning another group would be more useful than returning `None`
Thanks for the PR but I don't think this is really useful, so closing. Ping if you disagree and can pick back up
No problems @WillAyd !  Cheers :+1: