BUG: add reset logic for Grouper if new obj is passed in (#26564)
State: closed

- [ ] closes #26564
- [ ] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: Bug
name: Groupby

Comments:
needs tests that the relevant bug is fixed
this PR is likely duplicating this one: https://github.com/pandas-dev/pandas/pull/29131


> this PR is likely duplicating this one: #29131

Maybe I'll add my test here and see if it passes with #29131 .
@jreback I wrote a test [here](https://github.com/pandas-dev/pandas/pull/29800/files#diff-6f020a6c8d0f692c7f6e87cfe0775fe7R280) and ran it against #29131 which doesn't pass the test. So this is still an outstanding issue:

```
=========================================================== test session starts ============================================================
platform linux -- Python 3.7.3, pytest-5.3.0, py-1.8.0, pluggy-0.13.0
rootdir: /home/ali/repo/pydatascience/pandas-dev2, inifile: setup.cfg
plugins: cov-2.8.1, xdist-1.30.0, hypothesis-4.47.1, forked-1.1.2
collected 1 item                                                                                                                           

pandas/tests/resample/test_resampler_grouper.py F                                                                                    [100%]

================================================================= FAILURES =================================================================
__________________________________________________ test_same_grouper_on_different_frames ___________________________________________________

    def test_same_grouper_on_different_frames():
    
        df1 = pd.DataFrame(
            [["a", 1, 2], ["a", 4, 5], ["b", 2, 3]], columns=["type", "num1", "num2"],
        )
        df1["date"] = pd.to_datetime(["05/29/2019", "05/28/2019", "05/27/2019"])
    
        df2 = pd.DataFrame([["c", 6, 7], ["d", 8, 9]], columns=["type", "num1", "num2"],)
        df2["date"] = pd.to_datetime(["02/12/2018", "03/13/2018"])
    
        groupbys = ["type", pd.Grouper(key="date", freq="1D")]
    
        df1.groupby(groupbys).sum()
>       df2.groupby(groupbys).count()

pandas/tests/resample/test_resampler_grouper.py:294: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
pandas/core/groupby/generic.py:1674: in count
    ids, _, ngroups = self.grouper.group_info
pandas/_libs/properties.pyx:34: in pandas._libs.properties.CachedProperty.__get__
    val = self.func(obj)
pandas/core/groupby/ops.py:299: in group_info
    comp_ids, obs_group_ids = self._get_compressed_labels()
pandas/core/groupby/ops.py:315: in _get_compressed_labels
    all_labels = [ping.labels for ping in self.groupings]
pandas/core/groupby/ops.py:315: in <listcomp>
    all_labels = [ping.labels for ping in self.groupings]
pandas/core/groupby/grouper.py:405: in labels
    self._make_labels()
pandas/core/groupby/grouper.py:424: in _make_labels
    labels = self.grouper.label_info
pandas/_libs/properties.pyx:34: in pandas._libs.properties.CachedProperty.__get__
    val = self.func(obj)
pandas/core/groupby/ops.py:310: in label_info
    sorter = np.lexsort((labels, self.indexer))
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

args = ((array([ 0, 29]), array([2, 1, 0])),), kwargs = {}, relevant_args = (array([ 0, 29]), array([2, 1, 0]))

>   ???
E   ValueError: all keys need to be the same shape

<__array_function__ internals>:6: ValueError
============================================================ 1 failed in 0.17s =============================================================
```
> @jreback I wrote a test [here](https://github.com/pandas-dev/pandas/pull/29800/files#diff-6f020a6c8d0f692c7f6e87cfe0775fe7R280) and ran it against #29131 which doesn't pass the test. So this is still an outstanding issue:
> 
> ```
> =========================================================== test session starts ============================================================
> platform linux -- Python 3.7.3, pytest-5.3.0, py-1.8.0, pluggy-0.13.0
> rootdir: /home/ali/repo/pydatascience/pandas-dev2, inifile: setup.cfg
> plugins: cov-2.8.1, xdist-1.30.0, hypothesis-4.47.1, forked-1.1.2
> collected 1 item                                                                                                                           
> 
> pandas/tests/resample/test_resampler_grouper.py F                                                                                    [100%]
> 
> ================================================================= FAILURES =================================================================
> __________________________________________________ test_same_grouper_on_different_frames ___________________________________________________
> 
>     def test_same_grouper_on_different_frames():
>     
>         df1 = pd.DataFrame(
>             [["a", 1, 2], ["a", 4, 5], ["b", 2, 3]], columns=["type", "num1", "num2"],
>         )
>         df1["date"] = pd.to_datetime(["05/29/2019", "05/28/2019", "05/27/2019"])
>     
>         df2 = pd.DataFrame([["c", 6, 7], ["d", 8, 9]], columns=["type", "num1", "num2"],)
>         df2["date"] = pd.to_datetime(["02/12/2018", "03/13/2018"])
>     
>         groupbys = ["type", pd.Grouper(key="date", freq="1D")]
>     
>         df1.groupby(groupbys).sum()
> >       df2.groupby(groupbys).count()
> 
> pandas/tests/resample/test_resampler_grouper.py:294: 
> _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
> pandas/core/groupby/generic.py:1674: in count
>     ids, _, ngroups = self.grouper.group_info
> pandas/_libs/properties.pyx:34: in pandas._libs.properties.CachedProperty.__get__
>     val = self.func(obj)
> pandas/core/groupby/ops.py:299: in group_info
>     comp_ids, obs_group_ids = self._get_compressed_labels()
> pandas/core/groupby/ops.py:315: in _get_compressed_labels
>     all_labels = [ping.labels for ping in self.groupings]
> pandas/core/groupby/ops.py:315: in <listcomp>
>     all_labels = [ping.labels for ping in self.groupings]
> pandas/core/groupby/grouper.py:405: in labels
>     self._make_labels()
> pandas/core/groupby/grouper.py:424: in _make_labels
>     labels = self.grouper.label_info
> pandas/_libs/properties.pyx:34: in pandas._libs.properties.CachedProperty.__get__
>     val = self.func(obj)
> pandas/core/groupby/ops.py:310: in label_info
>     sorter = np.lexsort((labels, self.indexer))
> _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
> 
> args = ((array([ 0, 29]), array([2, 1, 0])),), kwargs = {}, relevant_args = (array([ 0, 29]), array([2, 1, 0]))
> 
> >   ???
> E   ValueError: all keys need to be the same shape
> 
> <__array_function__ internals>:6: ValueError
> ============================================================ 1 failed in 0.17s =============================================================
> ```

@alichaudry that PR is not merged, it *may* or may not pass the test you created; its not clear if this is a different issue or not. happy for you to investigate.
> @alichaudry that PR is not merged, it _may_ or may not pass the test you created; its not clear if this is a different issue or not. happy for you to investigate.

@jreback So what I meant is I took the following steps:

1. cloned #29131 locally (the PR you mentioned)
2. built pandas from scratch in a separate conda env (given that PR author's code)
3. put my test into the local version of #29131 and ran it, which failed

My point is, from what I can tell [my test](https://github.com/pandas-dev/pandas/pull/29800/files#diff-6f020a6c8d0f692c7f6e87cfe0775fe7R280) doesn't work on the code/changes in #29131, which means those changes don't address the bug that I found.
@alichaudry well i suspect that that PR solves some of the problem

happy to take a partial or full patch
@alichaudry pls merge master
@alichaudry looks like CI is red - can you investigate and try to get passing?
Hello @alichaudry! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:





There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2020-02-07 17:41:07 UTC
@WillAyd At this point I'm not sure how to proceed with this PR. I know what the issue is, and I've written a test in this PR which has the correct pass/fail behavior. I also put in a hacky solution for `pd.Grouper` in this PR which passes my test but fails a number of other tests. I don't have the bandwidth right now to re-work the `pd.Grouper` class to fix the root cause so I'm not sure where to go with this.

This issue as I described it in #26564 comes from the fact that I use `pd.Grouper` in a stateless way -- but when I use the grouper in a group-by, say `df_first.groupby(my_grouper)`, and then use the same grouper in another group-by: `df_second.groupby(my_grouper)` it fails as `my_grouper` was mutated by the first group-by and the internal state of the grouper object (and its attributes) makes it fail on the latter one.
@alichaudry Will may have some more specific thoughts, but worst-case scenario you can make a PR with just the test and xfail it, so we'll know if this gets fixed down the road.
@jbrockmendel followed your advice; updated the PR to include only the test and marked it as xfail. I am seeing some unrelated test errors (they seem to have to do with pyarrow/feather functionality) so I was wondering if you have any insight into this. I will still wait for feedback from @WillAyd.
The pyarrow/feather failures are unrelated, you can ignore them for now.
@WillAyd If you get a chance I'd love to hear your thoughts on this PR. It's passed all tests as expected (only failing unrelated pyarrow/feather tests).
The OP says that this closes #26564, is that still accurate?
@alichaudry if you can merge master to get CI green can take another look
closing as stale if you want to continue, please open a new PR.