BrandonFors/pandas-github-issues huggingface dataset:
BUG: wrong df.groupby().groups when grouping with [Grouper(freq=), ...] (GH33132)
State: closed

- [ ] closes #33132 
- [ ] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:

Comments:
The proposed change breaks the equivalence of `df.groupby(grouper)` and `df.groupby([grouper])`, captured by test https://github.com/pandas-dev/pandas/blob/8415f9ba2f5f3df6b10975c49d0e3af257149e64/pandas/tests/groupby/test_timegrouper.py#L740
Currently executing this test results in: 
(**The correct behavior is the one in `result`, and not in `expected`.**)

```python

import pandas as pd
import pandas._testing as tm

data_frame = {
    "location": ["shanghai", "beijing", "shanghai"],
    "time": pd.Series(
        ["2017-08-09 13:32:23", "2017-08-11 23:23:15", "2017-08-11 22:23:15"],
        dtype="datetime64[ns]",
    ),
    "value": [1, 2, 3],
}
data_frame = pd.DataFrame(data_frame).set_index("time")
grouper = pd.Grouper(freq="D")

grouped = data_frame.groupby(grouper)
result = grouped.count()
grouped = data_frame.groupby([grouper])
expected = grouped.count()

print(result)
#             location  value
# time                       
# 2017-08-09         1      1
# 2017-08-10         0      0
# 2017-08-11         2      2
print(expected)
#             location  value
# time                       
# 2017-08-09         1      1
# 2017-08-11         2      2
```

The problem occurs when we have something like `df.groupby([pd.Grouper(freq=)])` (i.e., a list with one element of type `TimeGrouper`).

One way to fix this is by capturing the `keys` arg in https://github.com/pandas-dev/pandas/blob/8305a856c150126ef899544091395104ac6141e2/pandas/core/groupby/groupby.py#L356-L363 and, if it is a `[pd.Grouper]`, we transform it to a `pd.Grouper`.

Does this modification make sense?
please merge master and see if you can resolve the issues.
Closing this PR as stale, but feel free to ping if you'd like to continue working on it.
BUG: DataFrame._item_cache not cleared on on .copy()
State: closed

- [x] closes #31784
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry


Labels:
name: Bug
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode

Comments:
lgtm. merging pending doc-build merge (so nothing to do @neilkg )
thanks @neilkg
REGR: Fix bug when replacing categorical value with self
State: closed

- [ ] closes #33288
- [ ] tests added / passed
- [ ] passes `black pandas`
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: Regression
description: Functionality that used to work in a prior pandas version
name: Categorical
description: Categorical Data Type

Comments:
LGTM pending green
@dsaxton If you rebase it should fix the Travis issue
@dsaxton I've tagged this as 1.0.4 for now since #33288 is a regression.

I don't think need to move whatsnew yet, see https://github.com/pandas-dev/pandas/pull/33157#issuecomment-606742451

opened #33300 to help keep track and discuss
@dsaxton if other comments are addressed, can you re-push?  the docbuild should be OK now
thanks @dsaxton
DEPR: Index.is_mixed
State: closed

- [x] closes #32922
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry


Labels:
name: Deprecate
description: Functionality to remove in pandas

Comments:
thanks, pls add to the deprecation issue tracker
CLN: Add/refine type hints to some functions in core.dtypes.cast
State: closed


- [ ] closes #xxxx
- [ ] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry

Some cleanup resulting from PR #32894. In order to get SeriesGroupBy._transform_fast to realize self.obj is a Series, it seemed I needed to add a generic type to GroupBy and _GroupBy. I don't know if there is a better way to do this. Here, one cannot use FrameOrSeries as a Series may return a DataFrame and vice-versa, so I added _GroupByT.

Labels:
name: Typing
description: type annotations, mypy/pyright type checking

Comments:
> In order to get SeriesGroupBy._transform_fast to realize self.obj is a Series, it seemed I needed to add a generic type to GroupBy and _GroupBy. I don't know if there is a better way to do this. Here, one cannot use FrameOrSeries as a Series may return a DataFrame and vice-versa, so I added _GroupByT.

see https://github.com/pandas-dev/pandas/pull/32302#issuecomment-594541073
thanks @rhshadrach very nice!
BUG: fix boolean array skipna=False for .any() and .all()
State: closed

- [x] closes #33253
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry

I added "not self._mask.any()" to check if there is no missing value for both .any() and .all() when skipna is False. It looks fine for the example you provided.
![any_all](https://user-images.githubusercontent.com/43714531/78453697-65cc1400-7661-11ea-9666-2fef60f2e81e.png)


Labels:
name: Bug
name: Missing-data
description: np.nan, pd.NaT, pd.NA, dropna, isnull, interpolate

Comments:
Thanks for the PR - does this close a particular issue? Can you add tests?
@WillAyd I think yes, and I intend to close the issue here [#33253](https://github.com/pandas-dev/pandas/issues/33253). It is my first time in open source so I am not familiar with PR.

I am not familiar with testing, but I can have a try. Any suggestions before I get started
Don't worry about Travis that is failing (that's a general issue, there is already another PR trying to fix that)
The any/all tests are here:

https://github.com/pandas-dev/pandas/blob/164064d4d4e79c75a7c2068a76deb5685cd87794/pandas/tests/arrays/boolean/test_reduction.py#L15-L39

so you can add to that test some new cases for the all-True and all-False cases from the issue.
Thanks @linxiaow !
DOC: Fix EX01 in DataFrame.drop_duplicates
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry

Related to #27977. 

```
################################################################################
################################## Validation ##################################
################################################################################

Labels:
name: Docs

Comments:
thanks @farhanreynaldo
DOC: Fix error in Series.clip and DataFrame.clip
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry

Related to #27977. 

output of `python scripts/validate_docstrings.py pandas.Series.clip and pandas.DataFrame.clip`:
```
################################################################################
################################## Validation ##################################
################################################################################
```



Labels:
name: Docs

Comments:
Thanks @farhanreynaldo
ENH: Optimize nrows in read_excel
State: closed

- [x] closes #32727
- [x] tests passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] If header, skiprows and nrows are integers, rows that will be skipped are not loaded in get_sheet_data function. 


Labels:
name: Performance
description: Memory or execution speed performance
name: IO Excel
description: read_excel, to_excel

Comments:
Tests which are not connected to that PR fail.
But tests for readers already exist and include header, skiprows and nrows variables. Should I add something more?
Mind fixing up the code checks @mproszewska and I agree with @mroeschke comments we should find a way to get rid of duplicate code 
The CI is failing because of an unused import of _validate_integer
Hello @mproszewska! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:















There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2020-06-08 17:08:14 UTC
I added suggested changes, but one test is still not passing and I'm not sure how to fix it.
I removed skipping rows in openpyxl, because it seems like there is a problem with files leaking. It might be connected to [this](https://bitbucket.org/openpyxl/openpyxl/issues/673/workbook-context-manager#comment-36565768).
Can you add a benchmark for this to show performance improvement?
@mproszewska can you post the ASV results here as a comment?
```
[ 75.00%] ··· io.excel.ReadExcel.time_read_excel_nrows                                   ok
[ 75.00%] ··· ========== ===========
                engine              
              ---------- -----------
                 xlrd     1.92±0.1s 
               openpyxl   4.69±0.5s 
                 odf       11.3±1s  
              ========== ===========

[ 75.00%] · For pandas commit f0a4e8e5 <master> (round 2/2):
[ 75.00%] ·· Building for conda-py3.6-Cython0.29.16-matplotlib-numba-numexpr-numpy-odfpy-openpyxl-pytables-pytest-scipy-sqlalchemy-xlrd-xlsxwriter-xlwt...
[ 75.00%] ·· Benchmarking conda-py3.6-Cython0.29.16-matplotlib-numba-numexpr-numpy-odfpy-openpyxl-pytables-pytest-scipy-sqlalchemy-xlrd-xlsxwriter-xlwt
[100.00%] ··· Setting up io/excel.py:62                                                  ok
[100.00%] ··· io.excel.ReadExcel.time_read_excel_nrows                                   ok
[100.00%] ··· ========== ============
                engine               
              ---------- ------------
                 xlrd     2.11±0.08s 
               openpyxl   4.65±0.5s  
                 odf      14.5±0.1s  
              ========== ============
```
Can you fix merge master and fix merge conflicts? That may also fix CI

Can you post the output of a continuous asv run instead of just the snippet? It should provide results across multiple runs at the end
@mproszewska can you address comments?
Closing as I think this is stale but @mproszewska ping if you'd like to pick back up and can fix merge conflicts
Troubleshoot Travis
State: closed

Reverts #33228, which appears to be the first build that started failing on Travis.

Labels:

Comments:
@WillAyd looks like Travis passed (though it still shows yellow here, need to click through).  any idea why the categorical test would affect the numexpr test?
On phone so haven’t checked in detail but seemingly unrelated?
Any objection to merging this to get the CI fixed, then re-implementing #33228 in pieces to track down the issue?

cc @jreback
nope, do it when green
> nope, do it when green

Hello @jreback any idea why CI test always fail for numexpr test?
> nope, do it when green

If you click through to Travis, it is green there (and has been for 16 hours), just hasnt de-yellowed here.
cool, seems we have seen this when  travis status not updating before oh well
API/CLN: simplify CategoricalBlock.replace
State: closed

- [x] closes #33272 
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry

Makes DataFrame.replace on Categorical behave like Series.replace.

Labels:
name: Internals
description: Related to non-user accessible pandas implementation
name: Categorical
description: Categorical Data Type

Comments:
The follow-up to this lets us get the `filter` kwarg out of `BlockManager.apply`
ENH: Added schema kwarg to get_schema method
State: closed

- [x] closes #28486
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry

Labels:
name: IO SQL
description: to_sql, read_sql, read_sql_query
name: Stale

Comments:
Please can you merge master
Seems like there are some test failures - mind taking a look? 
@alimcmaster1 

I'm not sure what the issue was, it was not my test that was breaking, I rebased from master again and it seems to be working now. 
@alimcmaster1 
I have completed the requests.
This looks good to me @gregorylivschitz mind merging master? 
@gregorylivschitz can you merge master
@jreback 
It is done.
@gregorylivschitz could you please merge master again? This looks like it's close to being merged
@gregorylivschitz whatsnew entry will now be v1.2

Almost there
@alimcmaster1 @MarcoGorelli 

I have merged it in. Thanks for reminding me!
Hello @gregorylivschitz! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:





There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2020-08-28 03:32:20 UTC
@gregorylivschitz I think this looks good but has a merge conflict and a CI failure - can you take a look? Should be able to get this in soon
Closing in favor of #38146
CLN 31942/replace appender with doc 3
State: closed

- [x] ref #31942
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] replaces `Appender` decorator with `doc`


Labels:
name: Docs

Comments:
Hello @smartvinnetou! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:



















There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2020-06-14 20:34:47 UTC
Can someone help with reviewing and merging this PR?
Looks like some merge conflicts in - can you fix that up?
> Looks like some merge conflicts in - can you fix that up?

done
> @smartvinnetou for future reference would be nice if you can make smaller PRs. Something with this many changes is much harder to review at once, so typically better to have multiple PRs with smaller changes.

@WillAyd I appreciate that. That's why I stopped making more changes. I started other branches but soon found that there was a non trivial overhead on maintaining them and keeping them current. I hit a pause button until PRs start to be merged.
@smartvinnetou can you please resolve the conflicts?

As said, smaller PRs are faster to review, get merged faster, and conflicts are less likely.
> As said, smaller PRs are faster to review, get merged faster, and conflicts are less likely.

@datapythonista, point taken. Will keep future PRs smaller.

Conflicts are now reslved.
@smartvinnetou pls merge master again.

@jorisvandenbossche @TomAugspurger @datapythonista can you give a quick look again and merge if good.
> @smartvinnetou pls merge master again.

Conflict resolved.

> 
> @jorisvandenbossche @TomAugspurger @datapythonista can you give a quick look again and merge if good.

Thanks @smartvinnetou, very nice clean up.

Once the development documentation is updated, can you please have a quick look at the updated pages, and see if everything is rendering as expected.

Thanks!
> Thanks @smartvinnetou, very nice clean up.
> 
> Once the development documentation is updated, can you please have a quick look at the updated pages, and see if everything is rendering as expected.
> 
> Thanks!

@datapythonista, where do I find this documentation?

https://pandas.pydata.org/docs/dev/
REF: dispatch TDBlock.to_native_types to TDA._format_native_types
State: closed


Labels:
name: Refactor
description: Internal refactoring of code
name: Timedelta
description: Timedelta data type

Comments:
If we don't want to change the DataFrame/to_csv behavior, we could add the "0 days" back in to the TDI/TDA behavior.  That only requires changing 2 tests (both of which are directly testing Timedelta64Formatter)
i think that would be better to add back to TDIFormatter
updated
nice!
TYP: Fixed type annotaions in `scripts/validate_rst_title_capitalization`
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: Typing
description: type annotations, mypy/pyright type checking

Comments:
Thanks @MomIsBestFriend
TST: add DataFrame test for construct from tuple case from GH-32776
State: closed

- [x] closes #32776 
- [x] tests added / passed
- [x] passes `black pandas`
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: Testing
description: pandas testing functions or related to the test suite

Comments:
Thanks for the PR! I think this is already covered here though?

https://github.com/pandas-dev/pandas/blob/37b14e50a1db5ee901df6cb7e57ecb9dff7fa358/pandas/tests/frame/test_constructors.py#L1342

If not, probably should just add to the parametrization there instead of creating a new test
> Thanks for the PR! I think this is already covered here though?
> 
> https://github.com/pandas-dev/pandas/blob/37b14e50a1db5ee901df6cb7e57ecb9dff7fa358/pandas/tests/frame/test_constructors.py#L1342
> 
> If not, probably should just add to the parametrization there instead of creating a new test

It is not covered. Will add to the param list instead
Great thanks @BenjaminLiuPenrose 
glad that I can help contribute to this amazing project ;)
TST: add date range test for reindex case from GH-32740 
State: closed

- [x] closes #32740 
- [x] tests added / passed
- [x] passes `black pandas`
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: Testing
description: pandas testing functions or related to the test suite

Comments:
Hello @BenjaminLiuPenrose! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:



There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2020-04-04 17:19:32 UTC
@mroeschke hi, could you point me out why the CI PR test always fails?
The Travis failure is unrelated, currently affecting all PRs.
Thanks @BenjaminLiuPenrose!
DOC: Fixed examples in `pandas/core/aggregation.py`
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: Docs

Comments:
> Probably better to split the changes to the examples from the changes to the typing. I guess they're unrelated.

@datapythonista by "split" you mean, separate PRs?
Yes, I mean separate PRs
> Probably better to split the changes to the examples from the changes to the typing. I guess they're unrelated.

IMO they are related. see https://github.com/pandas-dev/pandas/pull/33263#discussion_r403133623
Thanks @MomIsBestFriend
PERF: masked ops for reductions (min/max)
State: closed

Follow-up on https://github.com/pandas-dev/pandas/pull/30982, adding similar mask reduction but now for min/max.

Labels:
name: Performance
description: Memory or execution speed performance
name: NA - MaskedArrays
description: Related to pd.NA and nullable extension arrays

Comments:
thanks @jorisvandenbossche
DOC: Fixed examples in `pandas/core/accessor.py`
State: closed

Since this changes require an interactive Ipython shell (I think), we can't run check them from pytest.

This is what the docs looks like:

Before:

![Screenshot_20200403_170159](https://user-images.githubusercontent.com/50263213/78368644-9c0b8400-75cc-11ea-9061-24652f2bdfca.png)

After:
![register_acc](https://user-images.githubusercontent.com/50263213/78368530-71b9c680-75cc-11ea-8ab5-d2ee2eadc727.png)

Labels:
name: Docs

Comments:
Thanks @momisbestfriend
DOC: Added an example for each series.dt field accessor
State: closed

- [x] xref https://github.com/pandas-dev/pandas/pull/33208#discussion_r402189839
- [ ] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry

---

I can also add examples that would cover, ````TimedeltaProperties``` and ```PeriodProperties```, but I think that would be over verbose, wdyt?

Labels:
name: Docs

Comments:
> I can also add examples that would cover, ````TimedeltaProperties`and`PeriodProperties```, but I think that would be over verbose, wdyt?

agreed.
thanks @MomIsBestFriend
CLN: Reorgenized doctests order
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry

---

This does not changes anything, just moving things around, to make it more manageable (IMO)


Labels:
name: Clean

Comments:
Thanks @MomIsBestFriend
Display Timedelta nanoseconds
State: closed

- [x] closes #9309
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry

This allows us to make TimedeltaBlock.to_native_types dispatch to TimedeltaArray._format_native_types.  ATM TDBlock.to_native_types has a comment

```
        # FIXME:
        # should use the formats.format.Timedelta64Formatter here
        # to figure what format to pass to the Timedelta
        # e.g. to not show the decimals say
```

Labels:
name: Timedelta
description: Timedelta data type

Comments:
Seems reasonable. I suppose could use a whatsnew
ping
Thanks @jbrockmendel
BUG: Fix droped result column in groupby with as_index False
State: closed

- [x] closes #32240
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry

The aggregate function *lost* the results of the aggregate operation, if the ```as_index=False``` flag was set and one of the result columns is relabeled. The index was reset at first, adding columns to the DataFrame. The relabeling operation only took the columns before resetting the index as input. So the weird error occurred. I changed the order of the index reset and the relabeling. This fixes the issue.

Labels:
name: Bug
name: Groupby
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode

Comments:
One comment otherwise LGTM
Done @mroeschke 
Merge conflict in whats new.
Looks good. Looks like there are some failing code checks
> 
> 
> Looks good. Looks like there are some failing code checks

Missed that one, is green now.
let's rebase this on top of #34012 (or could wait, likey to merge that one tomorrow). this migth be fine but will want to re-look.
@jreback Merged master after the other pull request was merged. Seems to be fine.
@jreback  merged master
@jreback Changed the docs, test failure seems unrelated
looks good, can you rebase to fix the conflict (the other fail is ok), ping when ready.
@jreback fixed conflict, failure unrelated. Can be merged
can you merge master and ping when green
@phofl if you can merge master will have another look
Sorry, was a Bit busy the last few days. Will merge master tomorrow and ping when ready 
thanks @phofl
CLN: Clean nanops.get_corr_func
State: closed

Small cleaning (instead of creating a dictionary of functions and then returning only one, just return the one)

Labels:
name: Numeric Operations
description: Arithmetic, Comparison, and Logical operations
name: Clean

Comments:
> do we have tests for invaid method names?

There's testing at least for DataFrame.corr (test_corr_invalid_method in /tests/frame/methods/test_cov_corr.py) but not this function directly AFAIK
thanks
DOC: Contributing - escaping backslash.
State: closed

Non-escaped backslash disappeared in resulting doc.

- [ ] closes #xxxx
- [ ] tests added / passed
- [ ] passes `black pandas`
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: Docs

Comments:
Great thanks @MarianD