BrandonFors/pandas-github-issues huggingface dataset:
BUG: Groupby quantiles incorrect bins #33200
State: closed

Maintain the order of the bins in group_quantile. Updated tests #33200

- [x] closes #33200, closes #33569
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry


Labels:
name: Groupby

Comments:
Updated the PR. I did not notice the other PR. It was not referenced in the original issue #33200 .
BTW this should also close https://github.com/pandas-dev/pandas/issues/33569
xref #33300 to track.


@mabelvj This PR seems to be timing out on Windows. is this related to the changes here?
@mabelvj Can you merge master? The quantile tests were moved to /tests/groupby/test_quantile.py.
thanks @mabelvj for the patch. 
Owee, I'm MrMeeseeks, Look at me.

There seem to be a conflict, please backport manually. Here are approximate instructions:

1. Checkout backport branch and update it.

```
$ git checkout 1.0.x
$ git pull
```

2. Cherry pick the first parent branch of the this PR on top of the older branch:
```
$ git cherry-pick -m1 2740fb44e5e2ff64087ca7de5999eb30352722f0
```

3. You will likely have some merge/cherry-pick conflict here, fix them and commit:

```
$ git commit -am 'Backport PR #33644: BUG: Groupby quantiles incorrect bins #33200'
```

4. Push to a named branch :

```
git push YOURFORK 1.0.x:auto-backport-of-pr-33644-on-1.0.x
```

5. Create a PR against branch 1.0.x, I would have named this PR:

> "Backport PR #33644 on branch 1.0.x"

And apply the correct labels and milestones.

Congratulation you did some good work ! Hopefully your backport PR will be tested by the continuous integration and merged soon!

If these instruction are inaccurate, feel free to [suggest an improvement](https://github.com/MeeseeksBox/MeeseeksDev).
REF: dispatch Series.__setitem__ to loc/iloc, remove redundant helpers
State: closed

This will entail some extra overhead, but we get a) de-duplication and b) the structure of `__setitem__` comes to parallel the structure of `__getitem__`, which i find helpful grok-wise.

Labels:
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves
name: Clean

Comments:
prefer cleaner code to small perf diff always, thanks!
TST: groupby-reindex on DTI
State: closed

Add test to check whether reindexing works correctly.

- [ ] closes #26209
- [ ] tests added / passed
- [ ] passes `black pandas`
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Datetime
description: Datetime data dtype
name: Groupby

Comments:
I think need to run black on the change to get CI to pass
> I think need to run black on the change to get CI to pass

done and now CI-linting passes
Hello @CloseChoice! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:



There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2020-04-23 22:17:39 UTC
thanks @CloseChoice
REF: _AXIS_TO_AXIS_NUMBER to simplify axis access
State: closed

This adds a dict called ``_AXIS_TO_AXIS_NUMBER `` to NDFrame/DataFrame where the keys are the allowed parameter values for the ``axis`` parameter in various ndframe methods and the dict values are the related axis number. This makes getting to the correct axis more straight forward, see for example the new ``_get_axis_number``, and makes adding type hints to the ``axis`` parameter easier.

Labels:
name: Refactor
description: Internal refactoring of code

Comments:
This is not adding things, it's moving things around and simplifying.

A nice side effect will that we can now see the allowed values for ``axis`` in as the keys in the new attribute, where previosly it had to be done in THREE(!) locations (``_AXIS_NAMES``, ``_AXIS_NUMBERS`` and ``_AXIS_ALIASES``), which is relatively insane, when you think about it...

I've found out that this PR requires a change to the recipe on setting aliases in cookbook.rst.

I quite dislike encouraging setting aliases in the official pandas docs. If anyone actually uses that pattern in real code, is would be just confusing without being helpful IMO. But I've just changed it to the new style, for now, but would prefer to delete the section on aliases if there's no objections. 
>  ex my doc-comment above.

Could you expand, I don't understand this comment (and maybe the first comment too).
I think Jeff just mean to say to remove the cookbook example, alltogether, as you did (and indeed, that is not something to put in our docs I think)
Ok.
thanks @topper-123 very nice
can you add the deprecations to issue for removal: https://github.com/pandas-dev/pandas/issues/30228
CLN: replaced Appender with doc
State: closed

- [x] ref #31942
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] replaced the Appender/Substitute decorators with doc


Labels:
name: Docs
name: Clean

Comments:
Can you fix merge conflicts?
Can you fix up merge conflict again? If so can merge
@smartvinnetou can you fix conflicts?
Will do.

On Thu, 10 Sep 2020, 20:04 William Ayd, <notifications@github.com> wrote:

> @smartvinnetou <https://github.com/smartvinnetou> can you fix conflicts?
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <https://github.com/pandas-dev/pandas/pull/33633#issuecomment-690629913>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AOSDPMQR23NHNI3MBUUW6L3SFEPMHANCNFSM4MLNGF7A>
> .
>

one more rebase pls
Hello @smartvinnetou! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:



There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2020-09-13 19:38:05 UTC
Rebase done. Still some glitches in the doctests. Will keep fixing.
thanks @smartvinnetou  keep em coming
IO: Fix parquet read from s3 directory
State: closed

- [x] closes https://github.com/pandas-dev/pandas/issues/26388 
- [x] closes https://github.com/pandas-dev/pandas/issues/27596
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry

(Seems to have also fixed the xfailing test in https://github.com/pandas-dev/pandas/issues/33077)

NOTE: lets merge https://github.com/pandas-dev/pandas/pull/33645 first - since that fixes up a crucial bit of error handling around this functionality.

Labels:
name: IO Parquet
description: parquet, feather

Comments:
can you rebase
> can you rebase

sure merged + green
Hello @alimcmaster1! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:











There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2020-04-26 20:58:49 UTC
thanks @alimcmaster1
BUG: DataFrameGroupby std/sem modify grouped column when as_index=False
State: closed

- [x] closes #10355
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry

When working on this, I noticed an unrelated line of code that could be moved inside an if-block and made the change. Should unrelated cleanups like this be left to a separate PR? Can revert if that's the case.

Labels:
name: Groupby

Comments:
use .get_indexer in the columbus 
and then use .iloc to avoid all issues
@jreback tests now passing
@WillAyd thanks for the comments, changes made.
@jreback 

> can you reproduce this bug independently of this PR? please open a separate issue, this is a problem that needs fixing.

Yes, I confirmed on master using this MWE:

````
idx = pd.MultiIndex.from_tuples((("a", "a"), ("a", "a")))
df = pd.DataFrame([[1, 1]], columns=idx)
df.iloc[:, [0, 1]] = df.iloc[:, [0, 1]]
````

This is issue #22046.  It is also mentioned in the docstring of iLocIndexer._setitem_with_indexer:

````
This method is currently broken when dealing with non-unique Indexes,
since it goes from positional indexers back to labels when calling
BlockManager methods, see GH#12991, GH#22046, GH#15686.
````
@jreback Changes made, and mostly green. One Travis build failed because of a broken connection. Should I force push in this situation to have the check run again?
@jreback Merged to fix conflicts, all green now.
thanks @rhshadrach  very nice
BUG: Fix Categorical use_inf_as_na bug
State: closed

- [x] closes #33594
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry


Labels:
name: Bug
name: Missing-data
description: np.nan, pd.NaT, pd.NA, dropna, isnull, interpolate
name: Categorical
description: Categorical Data Type

Comments:
thanks @dsaxton 

if you can push the followon when you can to consolidate code here would be great
xref #33300 to track.
inspect-safety for DataFrame._constructor_expanddim
State: closed

- [x] closes #31549, closes #31474
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: Compat
description: pandas objects compatability with Numpy or Python functions

Comments:
Ah, interesting, it's  failing *without* jinja2, since `df.style` returns a Styler, which requires jinja2 to initialize.

I'm fine with just ignoring this for now, and essentially requiring jinaj2 for `inspect.getmembers` to work.
> I'm fine with just ignoring this for now, and essentially requiring jinaj2 for inspect.getmembers to work.

agreed
thanks
how about a follow-up for Series._constructor_sliced
TST: Groupby first/last/nth nan column test
State: closed

- [X] xref https://github.com/pandas-dev/pandas/issues/33591#issuecomment-615365647
- [X] tests added / passed
- [X] passes `black pandas`
- [X] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry

The first two asserts (`first()`/`last()`) fail on `1.0.3`, as expected.

Labels:
name: Testing
description: pandas testing functions or related to the test suite

Comments:
> The first two asserts (`first()`/`last()`) fail on `1.0.3`, as expected

this was changed in #32949

be86b6583f97a49ded14bdd4290ccf721bf1f645 is the first new commit
commit be86b6583f97a49ded14bdd4290ccf721bf1f645
Author: jbrockmendel <jbrockmendel@gmail.com>
Date:   Fri Apr 10 09:18:06 2020 -0700

    CLN: remove unnecessary Series._convert calls (#32949)

> Thanks @jamescobonkerr for the PR. does this test follow the same code path as the code sample in the issue OP?, see [#33591 (comment)](https://github.com/pandas-dev/pandas/issues/33591#issuecomment-616164785)

Ah, you're right — pardon the oversight.



> > The first two asserts (`first()`/`last()`) fail on `1.0.3`, as expected
> 
> this was changed in #32949
> 
> [be86b65](https://github.com/pandas-dev/pandas/commit/be86b6583f97a49ded14bdd4290ccf721bf1f645) is the first new commit
> commit [be86b65](https://github.com/pandas-dev/pandas/commit/be86b6583f97a49ded14bdd4290ccf721bf1f645)
> Author: jbrockmendel [jbrockmendel@gmail.com](mailto:jbrockmendel@gmail.com)
> Date: Fri Apr 10 09:18:06 2020 -0700
> 
> ```
> CLN: remove unnecessary Series._convert calls (#32949)
> ```

Removing the `_convert` call from `core.groupby.generic.DataFrameGroupBy._wrap_aggregated_output` seems to fix the reported issue, however, breaks a range of other tests. I don't have a great understanding of how `GroupBy` works. I'll close this PR.
@jamescobonkerr Thanks for working on this.

The test you have added here does have value since it is testing a change in behaviour against regression.

As you pointed out this test fails on 1.0.3 and is fixed on master. AFAICT this is working since #32949

I'll reopen this PR, but once merged will not close #33591 (rather than open a new issue at this point, i'll reference you code sample in #33591)
thanks @jamescobonkerr always happy to take these
BUG: pd.Series.replace does not preserve the original dtype
State: closed

- [ ] closes #33484
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: Dtype Conversions
description: Unexpected or buggy dtype conversions
name: Strings
description: String extension data type and string data
name: replace
description: replace method
name: Stale

Comments:
Hello @kotamatsuoka! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:





There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2020-06-06 10:18:53 UTC
Hi @kotamatsuoka - is this still active? Looks like there's some test failures and linting errors (from `black`), if you fix them up we'll take another look
@kotamatsuoka closing as stale. please comment if you want to continue and the PR will be reopened.
REF: Make numba function cache globally accessible
State: closed

- [x] tests passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`

For all the numba accessible operations, we use a cache to store compiled JIT'd functions for performance. 

This `_numba_func_cache` used to be an attribute of `Rolling` and `Groupby` objects before, but while working on https://github.com/pandas-dev/pandas/pull/33388, I cannot easily rely on having access to those objects.

This PR moves `_numba_func_cache` to  `pandas/core/util/numba_.py`. Since this cache is now globally shared, the key has changed from `_numba_func_cache[func] = compiled_func` to `_numba_func_cache[(func, op)] = compiled_func`


Labels:
name: Refactor
description: Internal refactoring of code

Comments:
Does this cache ever get invalidated?
>Does this cache ever get invalidated?

No. Nothing immediately comes to mind why we need to invalidate it at a certain point. Open to thoughts.
thanks.
CLN: Cython warnings for type declartions
State: closed

```
warning: pandas/_libs/internals.pyx:144:29: Strings should no longer be used for type declarations. Use 'cython.int' etc. directly.
warning: pandas/_libs/internals.pyx:147:32: Strings should no longer be used for type declarations. Use 'cython.int' etc. directly.
warning: pandas/_libs/internals.pyx:188:28: Strings should no longer be used for type declarations. Use 'cython.int' etc. directly.
```
(Against cython master)



Labels:
name: Clean
name: Typing
description: type annotations, mypy/pyright type checking

Comments:
/azp run
<samp>
Azure Pipelines successfully started running 1 pipeline(s).<br>

</samp>
Can you merge master again; doesn't look like all the checks have run. Chance this might not work with Py36 but will see in CI (maybe PEP 563 is a pre-cursor)
@WillAyd thanks - all checks have run now 
lgtm. I suppose this differs slightly from Python annotation requirements before PEP 563 was implemented but I don't have any objection unless @scoder knows any better
Hmm … yeah, that's PEP-563. I think it's really best to move forward with this and use real types in annotations instead of string names.
thanks @alimcmaster1
CLN: avoid catching AssertionError, AttributeError in NDFrame methods
State: closed

@WillAyd when i run `mypy` manually it passes, but when run via the pre-commit hook I get

```
pandas/core/internals/blocks.py:2739: error: unused 'type: ignore' comment
pandas/core/generic.py:4034: error: unused 'type: ignore' comment
pandas/core/generic.py:4093: error: unused 'type: ignore' comment
Found 3 errors in 2 files (checked 2 source files)
```

Something similar is happening with most branches these days.  Have you seen this?

Labels:
name: Clean

Comments:
looks ok, needs a rebase
updated+green
I think @xhochy has some experience with that pre-commit / mypy issue
Fixed bug. Added in check for ufunc and evaluates inner expression be…
State: closed

…fore evaluating outer expression

- [x] closes  #24670 
- [x] passes `black pandas`


Labels:
name: Numeric Operations
description: Arithmetic, Comparison, and Logical operations

Comments:
Hello @SurajH1! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:





There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2020-05-16 00:21:35 UTC
Is this ready to be accepted @maintainters @jbrockmendel 
any suggestions on why the failures are only occurring on some systems? @jreback 
I tested this branch locally using a function that makes multiple arguments, and it isn't working properly either. It isn't parsing the multiple arguments correctly. 

```python
import pandas as pd
from  numpy import *

print(pd.__version__)

r = pd.eval("floor(1+5)")
print(r)

r = pd.eval("minimum(1 + 5, 4)")  # This one fails
print(r)
```
> any suggestions on why the failures are only occurring on some systems? @jreback

Looking at the Azure CI output logs, it looks like you are failing on the environments that are testing the minimum versions, in particular, the lowest version of `numexpr`. There seems to be an `if` branch that specifically tries to handle the `floor` function. You could try written the test with a different function that isn't `floor`, or fix the if statement causing this error. (`pandas/core/computation/ops.py:602: ValueError`)

The Travis CI failures look like they are on environments that have `numba` installed. Here I am totally guessing, but I imagine that the failing operations use the `eval` capability somewhere internally, and that your changes are causing them to now fail. 
can you merge master and get this passing and will have a look
closing as stale, if you'd like continue, pls ping.
DOC: Remove extra backtick in example in documentation.
State: closed

:meth:`~Series.idxmax``  --> :meth:`~Series.idxmax`

- [x] closes #xxxx
- [ ] tests added / passed
- [ ] passes `black pandas`
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: Docs

Comments:
Thanks @pkirlin
BUG: set_levels set wrong order levels for MutiIndex
State: closed

- [ ] closes #33420
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry

Added parameter that allows user to reset codes

Labels:
name: MultiIndex

Comments:
If there's a better way to update the codes, please let me know!
@jreback Thanks for getting back to me! Sorry if I misunderstood the issue, but I originally believed that set_levels should effectively reassign the index automatically. I do realize now that adding a parameter to the method isn't necessary since the user can reassign the index themselves. Please let me know what your thoughts are on my original understanding!
closing as this is an internal implementation detail.
TYP: type NDFrame.(_get_axis|_get_axis_name|_get_axis_number)
State: closed

Gives return types to ``NDFrame._get_axis`` etc.

Labels:
name: Error Reporting
description: Incorrect or improved errors from pandas
name: Typing
description: type annotations, mypy/pyright type checking

Comments:
LGTM; @simonjayhawkins might have an opinion about the "type: ignore"
Updated.
DOC/CLN: Clean/Fix documentation for Window module
State: closed

- xref #31661
- xref #28792

- Added '.' before dataframe/series reference to fix missing link problem. Don't know if it's the best approach.
- Changed base template that is used by some functions to return better See Also section. Hope it's ok.


- Also question about Window class. It doesn't has its counterpart from Series/DF modules, its doc containing examples from Rolling part and there're only 4 functions. Is it internal base class for all windows and maybe shoul be excluded from docs or is it used somewhere maybe for creating custom windows?

Labels:
name: Docs

Comments:
Thanks @dilex42!
BUG: support count function for custom BaseIndexer rolling windows
State: closed

- [X] xref #32865 
- [X] 1 tests added / 1 passed
- [X] passes `black pandas`
- [X] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [X] whatsnew entry

## Scope of PR
This PR makes sure that when we call `count` with a `BaseIndexer` subclass, custom `start` and `end` arrays get calculated, and the algorithm in the `aggregations.pyx` gets called instead of the `_Window.create_blocks`.

Turns out that we were sending the calculation into `aggregations.pyx` only for frequency-based windows, and custom `BaseIndexer` subclasses got ignored. Fixed it and cleaned the code up a bit.

## Background on the wider issue
We currently don't support several rolling window functions when building a rolling window object using a custom class descended from `pandas.api.indexers.Baseindexer`. The implementations were written with backward-looking windows in mind, and this led to these functions breaking.
Currently, using these functions returns a `NotImplemented` error thanks to #33057, but ideally we want to update the implementations, so that they will work without a performance hit. This is what I aim to do over a series of PRs.

## Perf notes
No changes to the algorithms necessary.


Labels:
name: Window
description: rolling, ewma, expanding

Comments:
@mroeschke @jreback 
This PR implements `count` support for custom rolling windows.

All green, details above.
@jreback 
All green, please take a look if it's what you had in mind.
thanks @AlexKirko
BUG: DatetimeIndex.intersection losing freq and tz
State: closed

- [ ] closes #xxxx
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry


Labels:
name: Frequency
description: DateOffsets

Comments:
Does this somehow solve https://github.com/pandas-dev/pandas/issues/24362?
> Does this somehow solve #24362?

No, but ill make a PR for that in the AM.
.format changed to f string
State: closed

.format changed to f string 
https://github.com/pandas-dev/pandas/issues/29547
- [ ] closes #xxxx
- [x] tests added / passed
- [ ] passes `black pandas`
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: Code Style
description: Code style, linting, code_checks

Comments:
thanks @csaoma