BrandonFors/pandas-github-issues huggingface dataset:
DOC: Remove `# doctest: +SKIP`
State: closed

- [x] closes #32528
- [ ] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry

---

I want to be sure that the output of describe is deterministic, I will rebase regularly and if it passes all the times, I will mark it as ready for review. 

Labels:
name: Docs
name: Clean

Comments:
Is this still WIP or is it good to go?
> Is this still WIP or is it good to go?

@jbrockmendel still WIP, could not figure this out, I have thought about opening an issue to pytest. I can't see why they output is not deterministric on pytest, but it is on pandas.
closing to clear the queue, can reopen when you're ready
ERR: Better error message for missing columns in aggregate
State: closed

More descriptive SpecificationError message that reports to user non-existing columns causing error

- [x] closes #32755
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry


Labels:
name: Error Reporting
description: Incorrect or improved errors from pandas
name: Apply
description: Apply, Aggregate, Transform, Map

Comments:
merge master to fix the docs issue
@jreback Made changes and green
thanks @JDTruj2018
added cut as method to Series
State: closed

- [x] closes #28925 
- [ ] tests added / passed
- [ ] passes `black pandas`
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry

Not sure if we need to totally re-implement cut. Please feel free to give any advice, I am new to contributing.

Labels:
name: Enhancement
name: Series
description: Series data structure
name: cut
description: cut, qcut

Comments:
Hello @R4HMATT! Thanks for opening this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:

* In the file [`pandas/core/series.py`](https://github.com/pandas-dev/pandas/blob/a1a2dbb0c5f846028515dccf6b1db8748e3a5244/pandas/core/series.py):

> [Line 4529:5](https://github.com/pandas-dev/pandas/blob/a1a2dbb0c5f846028515dccf6b1db8748e3a5244/pandas/core/series.py#L4529): [E303](https://duckduckgo.com/?q=pep8%20E303) too many blank lines (2)
> [Line 4539:49](https://github.com/pandas-dev/pandas/blob/a1a2dbb0c5f846028515dccf6b1db8748e3a5244/pandas/core/series.py#L4539): [W291](https://duckduckgo.com/?q=pep8%20W291) trailing whitespace
> [Line 4540:1](https://github.com/pandas-dev/pandas/blob/a1a2dbb0c5f846028515dccf6b1db8748e3a5244/pandas/core/series.py#L4540): [W293](https://duckduckgo.com/?q=pep8%20W293) blank line contains whitespace
> [Line 4550:13](https://github.com/pandas-dev/pandas/blob/a1a2dbb0c5f846028515dccf6b1db8748e3a5244/pandas/core/series.py#L4550): [E123](https://duckduckgo.com/?q=pep8%20E123) closing bracket does not match indentation of opening bracket's line


Thanks for the PR. Why do you think this is a good idea though?
Echoing @WillAyd and my https://github.com/pandas-dev/pandas/issues/28925#issuecomment-541977217, I am still not convinced adding cut as a Series method makes sense.
Not sure if you are referring to the implementation or for the feature itself.

For the feature itself, the original issue expressed better method chaining for series like so:
`df.some_series.fillna(0).cut(10).mean()`.

At the moment, the above can only be done like this:
```
filled_series = df.some_series.fillna(0)
pd.cut(filled_series, 10).mean()
```
Series is only 1 possible output from `cut` that can be chained with other methods.

`cut` can otherwise produce a `Categorical`, `tuple`, or `ndarray` that cannot be chained.
@mroeschke Would it make sense (or even possible) to force the Series method implementation of cut to only return another Series objects then?
Possibly, but making the output behavior of `pd.cut` inconsistent with `Series.cut` would be odd from an API perspective.

Also for chaining, one can always call `Series.apply(pd.cut)`
Thanks for your interest in contributing @R4HMATT. I think there needs to be more discussion from the community plus core developers on the original issue before starting a PR for this feature. 

Feel free to contribute to many of the other issues!
DOC: FutureWarning in Sphinx build when calling read_parquet
State: closed

- [x] closes #32832
- [ ] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry

Labels:
name: Docs

Comments:
@datapythonista Is there a reason to limit the line characters length to 80, in ```rst``` files?

---

I don't mean disable the check, but 88 characters seems to me more reasonable rather than 80.
Even if ```auto_mkdir=True``` is passed, it still produces the warning on the CI, but on my local machine passing ```auto_mkdir=True``` does solve this issue.
@MomIsBestFriend can you just merge master? I just merged another PR that fixes the theme name error you now get
CI is green :)
Thanks for fixing this @MomIsBestFriend
BUG: ExtensionBlock.set not setting values inplace
State: closed

In trying to figure out the difference between Block.set vs Block.setitem I found that ExtensionBlock.set is not inplace like it is supposed to be.  Traced this back to a problem in CategoricalBlock.should_store, which this fixes+tests.

In separate passes I would like to
- rename set and setitem to something like "setitem_inplace" and "setitem_newobj"
- ATM setitem is _sometimes_ inplace; I'd like to make that consistent.

Labels:
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves
name: Internals
description: Related to non-user accessible pandas implementation

Comments:
does this need a whatsnew as your example IS user facing? if so, can you add in a followon
> does this need a whatsnew as your example IS user facing? if so, can you add in a followon

yes, will do.

@jreback two questions on `Block.setitem` behavior (AFAICT you wrote at least one of the two original implementations)

In Block.setitem we have a check

```
        elif (
            exact_match
            and is_categorical_dtype(arr_value.dtype)
            and not is_categorical_dtype(values)
        ):
            # GH25495 - If the current dtype is not categorical,
            # we need to create a new categorical block
            values[indexer] = value
            return self.make_block(Categorical(self.values, dtype=arr_value.dtype))
```

It isn't clear why we need `exact_match` here.  If we remove that, there is one test that fails because it expects to retain the non-Categorical dtype when setting only 2 of the 3 values with a length-2 Categorical.  Is this important?  (not having this restriction would make it easier to simplify this method)

Second, the next check in Block.setitem is:

```
        # if we are an exact match (ex-broadcasting),
        # then use the resultant dtype
        elif exact_match:
            # We are setting _all_ of the array's values, so can cast to new dtype
            values[indexer] = value
            values = values.astype(arr_value.dtype, copy=False)
```

The non-obvious thing here is why we are over-writing `values` instead of just using `value` (which would also save an `astype`!).  CoW semantics are _hard_, and it seems really easy for some of these to be careful and intentional and others not to be.
@jbrockmendel this PR also had the consequence that `__getitem__` modifies the values in place, which IMO is not the desired behaviour:

```
In [16]: cat = pd.Categorical(["A", "B", "C"]) 
    ...: df = pd.DataFrame({'cat': cat, 'int': [1, 2, 3]})  

In [17]: df['cat'] = cat[::-1]      

In [18]: cat     
Out[18]: 
[C, B, A]
Categories (3, object): [A, B, C]
```

Setting a single column with getitem with extension arrays should IMO simply override the existing column / values.


@jorisvandenbossche did you mean `__setitem__` modifies in-place?  If you really meant `__getitem__` then that is definitely bad
Indeed, `__setitem__` of course ;) But you already have seen the issue as well, so let's discuss it there.
STY: Correct whitespace placement
State: closed

- [x] ref #30755
- [ ] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: Code Style
description: Code style, linting, code_checks

Comments:
Thanks @MomIsBestFriend
CLN: Update docstring decorator from Appender to doc
State: closed

- [ ] xref #31942
- [ ] tests added / passed
- [ ] passes `black pandas`
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: Docs

Comments:
Attached are the screenshots of build docs

<img width="1256" alt="Screen Shot 2020-03-19 at 15 50 54" src="https://user-images.githubusercontent.com/34237447/77092398-b442a700-6a01-11ea-98ee-91e60acfb76a.png">

<img width="1254" alt="Screen Shot 2020-03-19 at 16 38 09" src="https://user-images.githubusercontent.com/34237447/77092667-e6540900-6a01-11ea-95a4-3f0cdcb1a804.png">

<img width="1248" alt="Screen Shot 2020-03-19 at 16 38 32" src="https://user-images.githubusercontent.com/34237447/77092688-eb18bd00-6a01-11ea-9c71-4f4d2f975309.png">

<img width="1246" alt="Screen Shot 2020-03-19 at 16 39 14" src="https://user-images.githubusercontent.com/34237447/77092716-f1a73480-6a01-11ea-928a-30d59a673755.png">

<img width="1252" alt="Screen Shot 2020-03-19 at 16 39 44" src="https://user-images.githubusercontent.com/34237447/77092741-f835ac00-6a01-11ea-8810-b092ad7b18c0.png">

Thanks @RaisaDZ, nice clean up.
DOC: Fixed contributors for bugfix releases
State: closed

Previously we showed just contributors who manually backported commits
to the maintenance branch.

```
Contributors
============

A total of 8 people contributed patches to this release.  People with a
"+" by their names contributed a patch for the first time.

* Daniel Saxton
* Joris Van den Bossche
* MeeseeksMachine
* MomIsBestFriend
* Pandas Development Team
* Simon Hawkins
* Tom Augspurger
* jbrockmendel
```

Fixed

```
Contributors
============

A total of 22 people contributed patches to this release.  People with a
"+" by their names contributed a patch for the first time.

* Anna Daglis +
* Daniel Saxton
* Irv Lustig
* Jan Škoda +
* Joris Van den Bossche
* Justin Zheng +
* Kaiqi Dong
* Kendall Masse +
* Marco Gorelli
* Matthew Roeschke +
* Pedro Reys +
* Prakhar Pandey +
* Robert de Vries +
* Rushabh Vasani +
* Simon Hawkins +
* Stijn Van Hoey +
* Terji Petersen +
* Tom Augspurger
* William Ayd
* alimcmaster1
* gfyoung +
* jbrockmendel
```

Closes https://github.com/pandas-dev/pandas/issues/31717

Labels:
name: Docs

Comments:
I guess this will need some more thought. As written, it only works when both the current and previous releases were bugfix releases.

I think we'll want to do both approaches (old and new) and then take the union.
New version (for v1.0.1..v1.0.2)

```
Contributors
============

A total of 25 people contributed patches to this release.  People with a
"+" by their names contributed a patch for the first time.

* Anna Daglis +
* Daniel Saxton
* Irv Lustig
* Jan Škoda
* Joris Van den Bossche
* Justin Zheng
* Kaiqi Dong
* Kendall Masse
* Marco Gorelli
* Matthew Roeschke
* MeeseeksMachine
* MomIsBestFriend
* Pandas Development Team
* Pedro Reys +
* Prakhar Pandey
* Robert de Vries +
* Rushabh Vasani
* Simon Hawkins
* Stijn Van Hoey
* Terji Petersen
* Tom Augspurger
* William Ayd
* alimcmaster1
* gfyoung
* jbrockmendel
```
> #31215 was one where the user specifically asked about this, though I don't see them on the list. Any idea what we may be missing there?

I posted the output for 1.0.1..1.0.2, that commit was in 1.0 (but backported during the RC period IIRC). I've confirmed that their name is in the output for v0.25.3..v1.0.0.
sure. you tagged for 1.04 ?
Yeah, Joris did.
So @pandas-dev/pandas-core when you merge PRs made by the backport bot, make sure to keep the text "Co-Authored by ...". Then the original author will be listed in the "Contributors" section of the 1.0.x docs.
PERF: skip non-consolidatable blocks when checking consolidation
State: closed

This skips the non-consolidatable blocks to determine if the blocks are consolidated. So meaning that if you have multiple EA columns with the same dtype, we do not do an unnecessary consolidation.

From investigating https://github.com/pandas-dev/pandas/issues/32196#issuecomment-600824238

@rth this should give another speed-up to your benchmark

Labels:
name: Performance
description: Memory or execution speed performance

Comments:
With:

```
arrays = [pd.arrays.SparseArray(np.random.randint(0, 2, 1000), dtype="float64") for _ in range(10000)]
index = pd.Index(range(len(arrays[0])))  
columns = pd.Index(range(len(arrays)))    
```

it gives

```
In [4]: %timeit pd.DataFrame._from_arrays(arrays, index=index, columns=columns)  
122 ms ± 2.92 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)   <-- PR
249 ms ± 8.5 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)   <-- master
```
Very nice @jorisvandenbossche , thanks! 

I can confirm that when cherry-picked on top of https://github.com/pandas-dev/pandas/pull/32825 this gives another 2x speed-up,
```
[100.00%] ··· sparse.SparseDataFrameConstructor.time_from_scipy                                                               115±0.5ms
       before           after         ratio
     [dbd7a5d3]       [c2476380]
     <master>         <tmp-sparse>
-       115±0.5ms       14.1±0.2ms     0.12  sparse.SparseDataFrameConstructor.time_from_scipy
```
So around 14x speed-up for these 3 PRs.
Looks like a nice speedup.  Couple of questions, non-blockers:

- are the improvements specific to SparseArray?
- if we exclude non-consolidateable, can we check dtype instead of ftype?
- does caching ftype make a difference?
Not specific to SparseArray, it's for all non-consolidatable blocks (which is now just ExtensionBlock I think?) 
It's just that creating a DataFrame from sparse matrix is a case where you only have extension blocks, and thus were this is relatively more costly.

I suppose that we can get rid of `ftype` entirely, as indeed, that was only used in the past to differentiate dense and sparse dtypes, but now sparse dtypes are ExtensionDtypes, and not numpy dtypes, so we don't need that anymore.

@jbrockmendel updated with removing `ftype`, thanks for the questions!
thanks @jorisvandenbossche 

yeah we had removed ftypes in 1.0, so this was a leftover i think.
DOC: Updating capitalization in folder doc/source/reference
State: closed

- [X] xref #32550


Labels:
name: Docs
name: Clean

Comments:
@datapythonista 
Ok I will work on that!


> Le 19 mars 2020 à 16:20, Marc Garcia <notifications@github.com> a écrit :
> 
> ﻿
> @datapythonista approved this pull request.
> 
> Thanks for working on this @cleconte987, looks great.
> 
> What would be very useful is to add the files that are fixed to https://github.com/pandas-dev/pandas/blob/master/ci/code_checks.sh#L330, so no more errors are being introduced in those fails (the CI will fail and let us know if any PR introduces titles capitalized incorrectly).
> 
> If you can add it to this PR that would be great.
> 
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub, or unsubscribe.

Thanks @cleconte987, changes look great. But looks like there are still several errors in that directory. Most of them need to be added here: https://github.com/pandas-dev/pandas/blob/master/scripts/validate_rst_title_capitalization.py#L20

Do you mind adding them, and fixing the cases that need it, if any? Thanks!
Hello @cleconte987! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:



There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2020-03-27 07:48:21 UTC
Ok.. again failures in the PR sorry for that. I know now why I putted "GroupBy" as an exception. Because it is written like that in title in file doc/source/reference/groupby.rst
I will re-update the PR. Should I add it to exceptions or not? I think it should
Thank you.

Yes I understand now for the case of "groupby/GroupBy", but yeah it was not passing the tests then. Ok :)
@cleconte987 can you merge upstream master to resolve conflicts
I guess everything that this PR was proposing has already been fixed in master, since no changes appear in the diff. Closing, let me know if that's not correct @cleconte987, happy to reopen this is the problem is another one.
No everything was updated, indeed!
DOC: Partial fix SA04 errors in docstrings #28792 (feedback needed)
State: closed

- [x] xref #28792

Would like feedback on a few issues before continuing to work on this issue :
- File pandas/core/accessor.py . How to handle multiline text? Couldn't find anywhere good recomendations.
- File pandas/core/ops/docstrings.py . Changed logic of templates. Is this okay? Also new lines with formatted text are too long, would like a reccomended sollution for this.
- File pandas/core/window/rolling.py and some others in regard to window functions have broken links that can be fixed by adding pandas. prefix to them(as in changed code). Is this right thing to do and should I make this changes whenever i see this?

Sorry for obvious mistakes, just starting to contribute. Thanks.

Labels:
name: Docs

Comments:
Hello @dilex42! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:

















There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2020-03-29 07:54:22 UTC
@datapythonista Thanks for the feedback. Will do the changes.
Before merging this changes there is still a problem with docs to window functions. The references doesn't work without pandas prefix but validation script gives errors if prefix present. I guess there is some inconsistency either in doc generator or validator.
Also a general rule how to deal with multiline text would be appreciated.
Can you open an issue for the prefix on the validation. Leave them the way they render correctly. If the validation is breaking the CI, remove the validation of that specific error in `ci/code_checks.sh` please.

For the multiline, I think the best is to use whatever makes code more readable. I think in most cases would be a multiline string, but a `\n` can be better for simple things. But it's more a mix of common sense and personal preference than a rule. Always keeping readability in mind.
Thanks
Should be good now.
Thanks @dilex42
PERF: fix SparseArray._simple_new object initialization
State: closed

Apart from this being more idiomatic, it also avoids creating a SparseArray through the normal machinery (including validation of the input etc) for the empty list.

With this PR:
```
In [1]: data = np.array([1, 2, 3], dtype=float)  

In [2]: index = pd.core.arrays.sparse.IntIndex(5, np.array([0, 2, 4]))  

In [3]: dtype = pd.SparseDtype("float64", 0)      

In [4]: pd.arrays.SparseArray._simple_new(data, index, dtype)  
Out[4]: 
[1.0, 0, 2.0, 0, 3.0]
Fill: 0
IntIndex
Indices: array([0, 2, 4], dtype=int32)

In [5]: %timeit pd.arrays.SparseArray._simple_new(data, index, dtype)    
381 ns ± 4.83 ns per loop (mean ± std. dev. of 7 runs, 1000000 loops each)
```

while on released version this gives around 50µs (100x slower)

Noticed while investigating https://github.com/pandas-dev/pandas/issues/32196

Labels:
name: Performance
description: Memory or execution speed performance
name: Sparse
description: Sparse Data Type

Comments:
Thanks @jorisvandenbossche! Quick benchmark result below when running `pd.DataFrame.sparse.from_spmatrix` on a random sparse CSR array of given `n_samples, n_features` with a `density=0.01`,
```
label                 master (s)    PR (s)
n_samples n_features                   
100       100000         14.5374   10.1247
10000     10000           1.5599    1.1037
100000    100             0.0180    0.0134
```

so overall this makes that method around 30% faster.
@rth thanks for the timings! Yes, it was indeed a large part of the original slow `from_spmatrix`, the snippet in the issue does most of the rest
so we have asvs for this? also add a whats new note
Note added in https://github.com/pandas-dev/pandas/pull/32825 that should work for both PRs I think.
TST: bare pytest raises
State: closed

GH30999

- [ ] ref #30999
- [ ] tests added / passed
- [ ] passes `black pandas`
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: Testing
description: pandas testing functions or related to the test suite

Comments:
Thanks @quangngd
TST: Avoid bare pytest.raises in multiple files
State: closed

* [x]  ref #30999
 
* [x]  tests added / passed
 
* [x]  passes `black pandas`

* [x]  passes `git diff upstream/master -u -- "*.py" | flake8 --diff`

Labels:
name: Testing
description: pandas testing functions or related to the test suite

Comments:
Thanks @Vlek
PERF: Using Numpy C-API for left-join calls
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry

---

#### Benchmarks:

This is the setup:

```python
import pandas as pd

df1 = pd.DataFrame(
    {
        "Customer_id": pd.Series([1, 2, 3, 4, 5, 6]),
        "Product": pd.Series(
            ["Oven", "Oven", "Oven", "Television", "Television", "Television"]
        ),
    }
)

df2 = pd.DataFrame(
    {
        "Customer_id": pd.Series([2, 4, 6]),
        "State": pd.Series(["California", "California", "Texas"]),
    }
)
```

And the ```%timeit```:

```python
pd.merge(df1, df2, on="Customer_id", how="left")   
```

---

```
In [4]: %timeit pd.merge(df1, df2, on="Customer_id", how="left")
1.54 ms ± 102 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each) # Master
1.46 ms ± 21.2 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each) # PR

In [5]: %timeit pd.merge(df1, df2, on="Customer_id", how="left")
1.47 ms ± 790 ns per loop (mean ± std. dev. of 7 runs, 1000 loops each) # Master
1.45 ms ± 2.59 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each) # PR

In [6]: %timeit pd.merge(df1, df2, on="Customer_id", how="left")
1.47 ms ± 1.99 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each) # Master
1.48 ms ± 65 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each) # PR
```
Since the results were so small I ran the same test several times:

Labels:
name: Performance
description: Memory or execution speed performance

Comments:
again I'd rather not do this unless this is a huge perf gain. can you time.
Closing this as there is no performance gain.

I will open a separate PR addressing comments that were mentioned here
ERR: Better error message
State: closed

When invalid value for pd.to_datetime or pd.to_timedelta is passed

- [x] closes #10720
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry

---

Resurrection of #31118


Labels:

Comments:
Reopening when this gets a bit more shape.