BrandonFors/pandas-github-issues huggingface dataset:
CLN/PERF: group quantile
State: closed

Inspired by idea from @jbrockmendel in https://github.com/pandas-dev/pandas/pull/43489#discussion_r705868510.

Didn't end up seeing a huge perf improvement though, for example results on a wide frame:
```
arr = np.random.randn(10 ** 5, 100)
df = pd.DataFrame(arr)
gb = df.groupby(df.index % 3)
gb.quantile(0.5)
# this pr: 1.18 s ± 9.58 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
# master: 1.55 s ± 10.1 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
```
Based on a quick profile, almost all time is spent in `lexsort` on master or with this pr - seems lexsorting the entire block at once has comparable performance to going column by column (I guess more improvement would show up on a _really_ wide frame).

Regardless, though, I think moving this out of cython is an improvement (since gives no perf benefit for them to be in cython when just numpy functions, also saves ~80kb on compiled code).

Labels:
name: Groupby
name: Algos
description: Non-arithmetic algos: value_counts, factorize, sorting, isin, clip, shift, diff
name: Clean

Comments:
thanks @mzeitlin11
REF: de-duplicate operate-column-wise code, pct_change
State: closed


Labels:
name: Groupby
name: Clean

Comments:
hmm can u show perf changes here

is pct_change just dispatched now?
> hmm can u show perf changes here

ill figure something out, not sure if we have any asvs that go through these paths

> is pct_change just dispatched now?

ATM the subclass implementation is effectively identical to the base class implementation, so this is just ripping out the unecessary one
REF: remove libreduction.SeriesGrouper
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [ ] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit) for how to run them
- [ ] whatsnew entry

cc @bashtage per comment https://github.com/pandas-dev/pandas/pull/37468#discussion_r706100798 I'm taking your word for it that the change this makes in the crosstab test is correct.  Does this represent a bugfix?

Labels:
name: Groupby
name: Clean

Comments:
I agree that this looks like a bug fix too. 
@bashtage can i pawn off the writing of a whatsnew on you?  i have zero experience with crosstab
What's new in #43548
ENH: consistency of input args for boundaries (pd.date_range)
State: closed

closes #43394 

xref #40245
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit) for how to run them
- [x] whatsnew entry


Labels:
name: Enhancement
name: Datetime
description: Datetime data dtype
name: Deprecate
description: Functionality to remove in pandas

Comments:
is this *also* closing https://github.com/pandas-dev/pandas/issues/43394 ? e.g. a bug fix, if so this needs a separate note 
> is this _also_ closing #43394 ? e.g. a bug fix, if so this needs a separate note

What do you mean by separate note by the way?
go ahead and rebase this
https://github.com/pandas-dev/pandas/pull/43504/checks?check_run_id=3612446736

couple of tests that need addressing (you can either assert the future warning or just change closed -> inclusive in the test)
some comments & can you merge master. ping on green.
thanks @zyc09
BUG: GroupBy.apply with Grouper and NaT
State: closed

- [ ] closes #xxxx
- [x] tests added / passed
- [ ] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit) for how to run them
- [x] whatsnew entry

Inching towards getting rid of redundant index-like attributes

Labels:
name: Bug
name: Groupby
name: Missing-data
description: np.nan, pd.NaT, pd.NA, dropna, isnull, interpolate

Comments:
hmm some failing tests
fixed ex windows timeout
DOC: sync release notes
State: closed

sync release notes on 1.3.x with master to reduce chances of merge conflicts.

Labels:
name: Docs

Comments:
@meeseeksdev backport 1.3.x
duh, this was direct already :->
Can't Dooooo.... It seem like this is already backported (commit is empty).I won't do anything. MrMeeseeks out.
Backport PR #43468: CI: split Windows Azure tests in half
State: closed

Backport #43468 to 1.3.x

Labels:
name: CI
description: Continuous Integration

Comments:
https://dev.azure.com/pandas-dev/pandas/_build/results?buildId=65945&view=logs&j=7ae4cfe6-c794-57c3-450a-5b4ec49e7feb&t=b053f402-64af-50ba-ae37-4304bce0dbf0, missing some changes.
> https://dev.azure.com/pandas-dev/pandas/_build/results?buildId=65945&view=logs&j=7ae4cfe6-c794-57c3-450a-5b4ec49e7feb&t=b053f402-64af-50ba-ae37-4304bce0dbf0, missing some changes.

Actually - the new version has 3.9 tests, but 1.3.x doesn't have them, so had to remove them from the  merge.
thanks @Dr-Irv very nice!
ENH Remove import time warning for missing lzma
State: closed

Also adds a test that lzma is not imported by default with `import pandas`.

- [x] closes #43461
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit) for how to run them
- [x] whatsnew entry: I'm not sure if a what's new entry is necessairy, and if yes, under which section of v1.4.0 should I put it?


cc @TomAugspurger


Labels:
name: Dependencies
description: Required and optional dependencies

Comments:
Hello @rth! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:













There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2021-11-28 21:37:44 UTC
I think the code check at https://github.com/pandas-dev/pandas/blob/3467469e6311561183bfb3981ae9fd6dad2bc1d8/ci/code_checks.sh#L48-L50 could be updated to add lzma, rather than the test you added.
can you merge master and addresss comments
Thanks for the reviews (and for the reminder)! I addressed the comments.  
pls merge master as well
this is failing here: https://github.com/pandas-dev/pandas/runs/4125318814?check_suite_focus=true
@jreback Yes, sorry for slow iterations. I'll look into it tomorrow. 
needs to merge master
Thanks for your patience @jreback . I merged master and removed lzma from the blocklist import (which fails) due to an optional import of lzma in numpy https://github.com/numpy/numpy/pull/18278 . Generally, since loading `shutil` will optionally load lzma (including in dependencies) it's hard do verify that it's never imported. 

I think the current version does  solve the initial issue of an unnecessary warning.
thanks @rth very nice!
BUG: fillna not replacing missing values when duplicate colnames
State: closed

- [x] closes #43476
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit) for how to run them
- [x] whatsnew entry


Labels:
name: Bug
name: Missing-data
description: np.nan, pd.NaT, pd.NA, dropna, isnull, interpolate

Comments:
thx @mzeitlin11
BUG: df.sparse.to_coo() raising on duplicate colnames
State: closed

- [x] closes #29564
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit) for how to run them
- [x] whatsnew entry


Labels:
name: Bug
name: Sparse
description: Sparse Data Type

Comments:
Added an ASV since frame case wasn't previously covered, no change
thanks @mzeitlin11
REF: Groupby.pad/backfill operate blockwise
State: closed

We get a decent-but-not-whopping perf boost, with a much bigger perf boost in the follow-up that de-duplicates an argsort call that we're currently doing repeatedly inside the cython function.

More importantly, we're getting close to having everything operate blockwise, at which point we can get rid of _get_cythonized_result and send everything through the _cython_operation path.

```
import pandas as pd
import numpy as np

np.random.seed(23446365)
arr = np.random.randn(10**5, 10)
mask = arr < -1
arr[mask] = np.nan

df = pd.DataFrame(arr)

gb = df.groupby(df.index % 7)

%timeit res = gb.pad()
28.7 ms ± 113 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)  # <- master
24 ms ± 1.06 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)  # <- PR
```

Labels:
name: Refactor
description: Internal refactoring of code
name: Groupby
name: Performance
description: Memory or execution speed performance

Comments:
lgtm
rebased + green
Backport PR #43428 on branch 1.3.x (Regression in __getitem__ raising for slice DatetimeIndex)
State: closed

Backport PR #43428: Regression in __getitem__ raising for slice DatetimeIndex

Labels:
name: Datetime
description: Datetime data dtype
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves
name: Regression
description: Functionality that used to work in a prior pandas version

Comments:
is the failing tests spurious (3.7)?
Looks unrelated, but rerunning to be sure
merge on greenish
thanks @phofl
Backport PR #43455 on branch 1.3.x (REGR: fillna on f32 column raising for f64)
State: closed

Backport PR #43455: REGR: fillna on f32 column raising for f64

Labels:
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves
name: Missing-data
description: np.nan, pd.NaT, pd.NA, dropna, isnull, interpolate
name: Dtype Conversions
description: Unexpected or buggy dtype conversions
name: Regression
description: Functionality that used to work in a prior pandas version

Comments:
cc @mzeitlin11 
@mzeitlin11 i think need to update the fixture here: https://dev.azure.com/pandas-dev/pandas/_build/results?buildId=65861&view=logs&j=f39ddbca-85f6-5283-3bfe-05832bcf5208&t=60a5c9af-5075-529a-e870-f4d886d8e7ce

as recently changed the names :-< (note I think you *can* just push commits to this PR)
> @mzeitlin11 i think need to update the fixture here: https://dev.azure.com/pandas-dev/pandas/_build/results?buildId=65861&view=logs&j=f39ddbca-85f6-5283-3bfe-05832bcf5208&t=60a5c9af-5075-529a-e870-f4d886d8e7ce
> 
> as recently changed the names :-< (note I think you _can_ just push commits to this PR)

Thanks, will do
thx @mzeitlin11
PERF: groupby.quantile
State: closed

Nearly 10x

```
import pandas as pd
import numpy as np

np.random.seed(536445246)
arr = np.random.randn(10**5, 5)
df = pd.DataFrame(arr, columns=["A", "B", "C", "D", "E"])

gb = df.groupby(df.index % 7)

qs = np.arange(0, 1, 0.1)

%timeit res = gb.quantile(qs)
771 ms ± 34 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)  # <- master
81.2 ms ± 8.63 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)  # <- PR
```

Labels:
name: Performance
description: Memory or execution speed performance
name: Algos
description: Non-arithmetic algos: value_counts, factorize, sorting, isin, clip, shift, diff

Comments:
prob worth a whatsnew note as large change in perf
whatsnew added + green
TST: add test to read empty array
State: closed

- [ ] closes #41241
- [ ] tests added / passed
- [ ] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [ ] whatsnew entry


Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: IO Parquet
description: parquet, feather

Comments:
@jreback 
I get a timeout on the `pandas-dev.pandas` test, could you tell me why?
If you know the solution, I'd like to hear it.
> @jreback I get a timeout on the `pandas-dev.pandas` test, could you tell me why? If you know the solution, I'd like to hear it.

you can merge master and see if that fixes, but sometimes this does timeout unrelated to your change
thanks @nakatomotoi very nice.

if you would like to do a PR with an empty category would be great as well (llikley just need to construct it properly)
DOC: 'if_sheet_exists' only on openpyxl in stead of 'append'
State: closed

- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [ ] whatsnew entry

The `if_sheet_exists` parameter works also in write mode of openpyxl. e.g.:
```python
import pandas as pd
with pd.ExcelWriter("f.xlsx",engine="openpyxl") as writer:
    pd.DataFrame([1,2]).to_excel(writer)
    pd.DataFrame([3,4]).to_excel(writer)
```
will raise an error, because the default `if_sheet_exists` is set to "error". This could be fixed by this simple docs change, or changing the default of `if_sheet_exists` to "overlay". However, if that is the case, it could also very well be implemented for other engines.

Labels:

Comments:
oops, I was wrong, closing
TST: add test to check numeric precision GH33234
State: closed

test to check numeric precision after .groupby and .agg

- [X] closes #33234
- [X] tests added / passed
- [X] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [X] whatsnew entry


Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Groupby
name: Dtype Conversions
description: Unexpected or buggy dtype conversions
name: Stale

Comments:
This pull request is stale because it has been open for thirty days with no activity. Please update or respond to this comment if you're still interested in working on this.
Thanks for the PR, but it appears to have gone stale. If interested in continuing, please merge in master and address the comments and we can reopen.
@mroeschke , can I continue the PR? I want to do some contribution.
Sure, go for it
REGR: fillna on f32 column raising for f64
State: closed

- [x] closes #43424
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [x] whatsnew entry

For now have put whatsnew in 1.3.3, but might be better for 1.4 since regression is later (worked in 1.1.5) and the `can_hold_element` change causes subtle indexing behavior differences (but that are now more consistent between int and float)

Labels:
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves
name: Missing-data
description: np.nan, pd.NaT, pd.NA, dropna, isnull, interpolate
name: Dtype Conversions
description: Unexpected or buggy dtype conversions
name: Regression
description: Functionality that used to work in a prior pandas version

Comments:
does this break any current tests?
> does this break any current tests?

Think the failures are flaky
Have changed the approach to be less invasive, so backport I think safer now. 

I think the `can_hold_element` changes were in the right direction, but was unsure of the exact contract `can_hold_element` is meant to uphold and what subtle changes might be caused by what I changed.

EDIT: more reasoning for changing approach was not being able satisfactorily answer https://github.com/pandas-dev/pandas/pull/43455#discussion_r704293229 and other failing test cases I discovered that weren't handled by just the itemsize patch. The test now covers all numpy int/floating types
lgtm. cc @jbrockmendel @phofl 
thanks @mzeitlin11 
@meeseeksdev backport 1.3.x
Something went wrong ... Please have  a look at my logs.
Regression in loc setitem raising ValueError when setting array as cell value
State: closed

- [x] closes #43422
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [x] whatsnew entry


Labels:
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves
name: Regression
description: Functionality that used to work in a prior pandas version

Comments:
thanks @phofl 
@meeseeksdev backport 1.3.x
Something went wrong ... Please have  a look at my logs.
REGR: SpooledTemporaryFile support in read_csv
State: closed

- [ ] closes #43439
- [ ] tests added / passed
- [ ] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [ ] whatsnew entry


Probably should throw a warning, if we cannot wrap the handle with `io.TextIOWrapper`.

To avoid the try-except, we could check whether the handle has the `readable` attribute, but there might be other attribute errors that might occur inside `TextIOWrapper`.

Labels:
name: IO CSV
description: read_csv, to_csv
name: Compat
description: pandas objects compatability with Numpy or Python functions

Comments:
Hello @twoertwein! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:





There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2021-09-09 21:59:11 UTC
Thanks @twoertwein. Is this targeting 1.3.x?
> Thanks @twoertwein. Is this targeting 1.3.x?

It should be, I'll probably have time this evening to finish it up.

I'm not yet sure whether try-except is a good solution here. Another option (maybe more for 1.4), would be to add a new keyword to read_csv to set the mode (to_csv has that currently). Then a user could overwrite the binary/text auto-detection by specifying `mode="rt"` or `mode="rb"`.
Instead of using a try-except block, I added `tempfile.SpooledTemporaryFile` to the list of classes that should always be treated as text handles.

I'm not sure whether #43439 is a regression: yes, it worked previously, but it only worked because of inconsistencies between the c-engine and python-engine. This PR allows this inconsistency to continue (I'm not a fan of this). It is great that the c-engine is not only faster than the python-engine, but also doesn't seem to need the`io.TextIOWrapper` (skipping the wrapper for the c-engine might actually give a slight performance improvement).

I would be slightly in favor of not fixing #43439 for 1.3.3 and instead adding a new keyword argument ("mode", same as in `to_csv`) in 1.4 for `read_csv` which allows the user to over-write the text/binary auto-detection. (And a PR for 1.4 to make the c-engine "dumber": if it is told to get strings, but it gets bytes, it should fail.).

If it is preferred to fix #43439 for 1.3.3, I'm happy to make this happen (I think this PR should be ready).
oh new commit way nicer
very nice @twoertwein 
@meeseeksdev backport 1.3.x
Something went wrong ... Please have  a look at my logs.
ENH: Improve IOError handling on missing parent directory for Series/DataFrame write methods #24306
State: closed

- [x] closes #24306 
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [x] whatsnew entry


Labels:
name: IO Data
description: IO issues that don't fit into a more specific label
name: Error Reporting
description: Incorrect or improved errors from pandas

Comments:
alright tests are good here except for the timed-out one - which seems to occur in other PRs as well
@twoertwein good here?
should I resolve the merge conflict?
Yup. Thanks for the PR! We'll get it merged afterwards.
Alright done
thanks @leonarduschen very nice!
REF: Groupby._get_cythonized_result operate blockwise in axis==1 case
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [ ] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [ ] whatsnew entry


Labels:
name: Missing-data
description: np.nan, pd.NaT, pd.NA, dropna, isnull, interpolate
name: Performance
description: Memory or execution speed performance

Comments:
needs a rebase, b ut i think you have changed a bunch of this code
rebased + green
is this a bug fix? or just perf on a case?
> is this a bug fix? or just perf on a case?

Neither, just going from 2 paths to 1 path.
TST: Test for MultiIndex merge with CategoricalIndex (#36973)
State: closed

- [x] closes #36973 
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them


Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode
name: MultiIndex
name: Categorical
description: Categorical Data Type

Comments:
thanks @tim-tran
DOC: Modify pandas.ExcelWriter default engine in docstring (#43359)
State: closed

* engine for xlsx : xlsxwriter instead of openpyxl
* correction to pass the doctest

- [x] closes #43359
- [x] tests added / passed : script/validate_docstrings.py detected 3 errors before my pull request. It's now resolved
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them : black pandas didn't change anything
- [x] whatsnew entry : one word changed in docstring


Script validate_docstings report : [validate_docstrings.txt](https://github.com/pandas-dev/pandas/files/7117843/validate_docstrings.txt)


Labels:
name: Docs
name: IO Excel
description: read_excel, to_excel

Comments:
Thanks @LiquidMika
Backport PR #43403: Revert Cythonized Kendall implementation and improve test case to prevent regressions
State: closed

- [ ] closes #43401
- [ ] tests added / passed
- [ ] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [ ] whatsnew entry


Labels:
name: Regression
description: Functionality that used to work in a prior pandas version
name: Algos
description: Non-arithmetic algos: value_counts, factorize, sorting, isin, clip, shift, diff

Comments:
thanks @zrait
Regression in __getitem__ raising for slice DatetimeIndex
State: closed

- [x] closes #43223
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [x] whatsnew entry

@jbrockmendel Any idea why we used ``maybe_indices_to_slice`` instead of take like in loc?


Labels:
name: Datetime
description: Datetime data dtype
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves
name: Regression
description: Functionality that used to work in a prior pandas version

Comments:
> Any idea why we used maybe_indices_to_slice instead of take like in loc?

Best guess: if we have e.g. `[0, 2, 4, 6]` that we can change to `slice(0, 8, 2)`, then a) the indexing is no-copy and b) we preserve e.g. DTI.freq and RangeIndex.
Hm good point, should we try to convert to slice and only if it fails, e.g. stays an ndarray, call take?
> Hm good point, should we try to convert to slice and only if it fails, e.g. stays an ndarray, call take?

i think thats what we're already doing isnt it?
I've removed the cast. Added it and we only call take if we can not convert to slice