BUG/TST: fix and test for timezone drop in GroupBy.shift/bfill/ffill
State: closed

closes #19995

Timezone info is dropped in GroupBy.shift, bfill, and ffill
since index calculated by Cythonized functions are applied to
NumPy representation of values without timezone.

Could you please find and review this fix?

Many thanks,

-noritada

```
In [1]: import pandas as pd 
   ...: pd.__version__ 
Out[1]: '0.24.2'

In [2]: df = pd.DataFrame([ 
   ...:     ['a', pd.Timestamp('2019-01-01 00:00:00+09:00')], 
   ...:     ['b', pd.Timestamp('2019-01-01 00:05:00+09:00')], 
   ...:     ['a', None], 
   ...:     ['b', None], 
   ...:     ['a', pd.Timestamp('2019-01-01 00:20:00+09:00')], 
   ...:     ['b', pd.Timestamp('2019-01-01 00:25:00+09:00')], 
   ...: ], columns=['id', 'time']) 

In [3]: df['time'].shift(1) 
Out[3]: 
0                         NaT
1   2019-01-01 00:00:00+09:00
2   2019-01-01 00:05:00+09:00
3                         NaT
4                         NaT
5   2019-01-01 00:20:00+09:00
Name: time, dtype: datetime64[ns, pytz.FixedOffset(540)]

In [4]: df.groupby('id')['time'].shift(1) 
Out[4]: 
0                   NaT
1                   NaT
2   2018-12-31 15:00:00
3   2018-12-31 15:05:00
4                   NaT
5                   NaT
Name: time, dtype: datetime64[ns]

In [5]: df.groupby('id')['time'].bfill() 
Out[5]: 
0   2018-12-31 15:00:00
1   2018-12-31 15:05:00
2   2018-12-31 15:20:00
3   2018-12-31 15:25:00
4   2018-12-31 15:20:00
5   2018-12-31 15:25:00
Name: time, dtype: datetime64[ns]

In [6]: df.groupby('id')['time'].ffill() 
Out[6]: 
0   2018-12-31 15:00:00
1   2018-12-31 15:05:00
2   2018-12-31 15:00:00
3   2018-12-31 15:05:00
4   2018-12-31 15:20:00
5   2018-12-31 15:25:00
Name: time, dtype: datetime64[ns]
```



- [ ] closes #xxxx
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry


Labels:
name: Bug
name: Groupby
name: Timezones
description: Timezone data dtype

Comments:
@mroeschke 
Thank you very much for your reviews.
I've updated as per your suggestions.
Could you please check again?

Many thanks,

-noritada
@mroeschke
Thank you so much again.
I should have found this function.
Test code updated again.

Many thanks,

-noritada

Can you update whatsnew for 0.25.2 as well?
1.0.0, unless this is a regression in 0.25.x.

On Fri, Aug 23, 2019 at 8:52 AM William Ayd <notifications@github.com>
wrote:

> *@WillAyd* requested changes on this pull request.
> ------------------------------
>
> In pandas/core/groupby/groupby.py
> <https://github.com/pandas-dev/pandas/pull/27992#discussion_r317141603>:
>
> > @@ -2220,26 +2220,28 @@ def _get_cythonized_result(
>          base_func = getattr(libgroupby, how)
>
>          for name, obj in self._iterate_slices():
> +            values = obj._data._values
>
> Hmm I'm not sure relying on underlying block values is the best way to go
> about this. Is it possible to just work with the obj here instead?
>
> —
> You are receiving this because you are subscribed to this thread.
> Reply to this email directly, view it on GitHub
> <https://github.com/pandas-dev/pandas/pull/27992?email_source=notifications&email_token=AAKAOITRKMZJLG3H2KELYODQF7TQNA5CNFSM4IMSLLYKYY3PNVWWK3TUL52HS4DFWFIHK3DMKJSXC5LFON2FEZLWNFSXPKTDN5WW2ZLOORPWSZGOCCQWFKQ#pullrequestreview-279012010>,
> or mute the thread
> <https://github.com/notifications/unsubscribe-auth/AAKAOIXQRK4RRN4LSTZ5RR3QF7TQNANCNFSM4IMSLLYA>
> .
>

also pls merge master
@WillAyd @TomAugspurger @jreback
Thanks for comments and sorry to be absent for 3 weeks.
I've merged master, move a whatsnew entry to v0.25.2 and document #19995 .

Could you please check this again?

Many thanks,

-noritada
thanks @noritada 
Owee, I'm MrMeeseeks, Look at me.

There seem to be a conflict, please backport manually. Here are approximate instructions:

1. Checkout backport branch and update it.

```
$ git checkout 0.25.x
$ git pull
```

2. Cherry pick the first parent branch of the this PR on top of the older branch:
```
$ git cherry-pick -m1 e0c63b4cfaa821dfe310f4a8a1f84929ced5f5bd
```

3. You will likely have some merge/cherry-pick conflict here, fix them and commit:

```
$ git commit -am 'Backport PR #27992: BUG/TST: fix and test for timezone drop in GroupBy.shift/bfill/ffill'
```

4. Push to a named branch :

```
git push YOURFORK 0.25.x:auto-backport-of-pr-27992-on-0.25.x
```

5. Create a PR against branch 0.25.x, I would have named this PR:

> "Backport PR #27992 on branch 0.25.x"

And apply the correct labels and milestones.

Congratulation you did some good work ! Hopefully your backport PR will be tested by the continuous integration and merged soon!

If these instruction are inaccurate, feel free to [suggest an improvement](https://github.com/MeeseeksBox/MeeseeksDev).
                
@jreback I think the plan was for 0.25.2 to just be Python 3.8 compat and 0.25 regressions: https://github.com/pandas-dev/pandas/issues/28110. I don't believe this was a regression in 0.25.
ok sure np

alright then prob just need to move the actual release note