Repository: pandas-dev/pandas
Pull request: BUG/TST: fix and test for timezone drop in GroupBy.shift/bfill/ffill
closes #19995

Timezone info is dropped in GroupBy.shift, bfill, and ffill
since index calculated by Cythonized functions are applied to
NumPy representation of values without timezone.

Could you please find and review this fix?

Many thanks,

-noritada

```
In [1]: import pandas as pd 
   ...: pd.__version__ 
Out[1]: '0.24.2'

In [2]: df = pd.DataFrame([ 
   ...:     ['a', pd.Timestamp('2019-01-01 00:00:00+09:00')], 
   ...:     ['b', pd.Timestamp('2019-01-01 00:05:00+09:00')], 
   ...:     ['a', None], 
   ...:     ['b', None], 
   ...:     ['a', pd.Timestamp('2019-01-01 00:20:00+09:00')], 
   ...:     ['b', pd.Timestamp('2019-01-01 00:25:00+09:00')], 
   ...: ], columns=['id', 'time']) 

In [3]: df['time'].shift(1) 
Out[3]: 
0                         NaT
1   2019-01-01 00:00:00+09:00
2   2019-01-01 00:05:00+09:00
3                         NaT
4                         NaT
5   2019-01-01 00:20:00+09:00
Name: time, dtype: datetime64[ns, pytz.FixedOffset(540)]

In [4]: df.groupby('id')['time'].shift(1) 
Out[4]: 
0                   NaT
1                   NaT
2   2018-12-31 15:00:00
3   2018-12-31 15:05:00
4                   NaT
5                   NaT
Name: time, dtype: datetime64[ns]

In [5]: df.groupby('id')['time'].bfill() 
Out[5]: 
0   2018-12-31 15:00:00
1   2018-12-31 15:05:00
2   2018-12-31 15:20:00
3   2018-12-31 15:25:00
4   2018-12-31 15:20:00
5   2018-12-31 15:25:00
Name: time, dtype: datetime64[ns]

In [6]: df.groupby('id')['time'].ffill() 
Out[6]: 
0   2018-12-31 15:00:00
1   2018-12-31 15:05:00
2   2018-12-31 15:00:00
3   2018-12-31 15:05:00
4   2018-12-31 15:20:00
5   2018-12-31 15:25:00
Name: time, dtype: datetime64[ns]
```



- [ ] closes #xxxx
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry


Problem statement:
Shifting a datetime column with timezone after groupby loses the timezone.
#### Code Sample, a copy-pastable example if possible

```python
weeks = pd.Series(pd.date_range('2018-01', '2018-02', freq='7D', tz='America/New_York'))
week_ago = weeks.groupby([1, 1, 1, 1, 1]).shift()
print(week_ago)
```
Outputs:
```
0                   NaT
1   2018-01-01 05:00:00
2   2018-01-08 05:00:00
3   2018-01-15 05:00:00
4   2018-01-22 05:00:00
dtype: datetime64[ns]
```

#### Problem description

Shifting a groupby'd datetime column removes the timezone. It's not mentioned in the documentation so either the timezone should be preserved or the documentation should mention that timezone is removed (by converting to UTC it seems).

I had a quick look and couldn't find another issue for this. I glanced at the GroupBy shift implementation and my guess is that it's due to cython/numpy not supporting timezones.

I'm happy to work on a fix if someone says what the preferred solution is.

#### Expected Output

```
0                         NaT
1   2018-01-01 00:00:00-05:00
2   2018-01-08 00:00:00-05:00
3   2018-01-15 00:00:00-05:00
4   2018-01-22 00:00:00-05:00
dtype: datetime64[ns, America/New_York]
```

#### Output of ``pd.show_versions()``

<details>

INSTALLED VERSIONS
------------------
commit: None
python: 3.6.3.final.0
python-bits: 64
OS: Linux
OS-release: 3.10.0-693.el7.x86_64
machine: x86_64
processor: x86_64
byteorder: little
LC_ALL: en_US.UTF-8
LANG: en_US.UTF-8
LOCALE: en_US.UTF-8

pandas: 0.21.0
pytest: 3.4.1
pip: 9.0.1
setuptools: 38.5.1
Cython: None
numpy: 1.13.3
scipy: 1.0.0
pyarrow: 0.8.0
xarray: None
IPython: 6.2.1
sphinx: None
patsy: 0.5.0
dateutil: 2.6.1
pytz: 2018.3
blosc: None
bottleneck: 1.2.1
tables: None
numexpr: 2.6.4
feather: 0.4.0
matplotlib: 2.1.2
openpyxl: None
xlrd: None
xlwt: None
xlsxwriter: None
lxml: None
bs4: None
html5lib: 0.9999999
sqlalchemy: 1.2.1
pymysql: None
psycopg2: None
jinja2: 2.10
s3fs: None
fastparquet: 0.1.4
pandas_gbq: None
pandas_datareader: None

</details>

Hints from issue discussion:
duplicate of #15884 
@jreback the linked issue is closed as fixed but `shift` still has this behaviour (only `first` and `last` were fixed), can this be reopened?
Thanks @akdor1154.

Commit message: BUG/TST: fix and test for timezone drop in GroupBy.shift/bfill/ffill

Patch to review:
diff --git a/doc/source/whatsnew/v0.25.1.rst b/doc/source/whatsnew/v0.25.1.rst
index 34b149a6b8261..863780610fee5 100644
--- a/doc/source/whatsnew/v0.25.1.rst
+++ b/doc/source/whatsnew/v0.25.1.rst
@@ -120,6 +120,7 @@ Groupby/resample/rolling
 - Bug in :meth:`pandas.core.groupby.DataFrameGroupBy.transform` where applying a timezone conversion lambda function would drop timezone information (:issue:`27496`)
 - Bug in windowing over read-only arrays (:issue:`27766`)
 - Fixed segfault in `pandas.core.groupby.DataFrameGroupBy.quantile` when an invalid quantile was passed (:issue:`27470`)
+- Bug in :meth:`pandas.core.groupby.GroupBy.shift`, :meth:`pandas.core.groupby.GroupBy.bfill` and :meth:`pandas.core.groupby.GroupBy.ffill` where timezone information would be dropped (:issue:`xxxxx`)
 -
 
 Reshaping
diff --git a/pandas/core/groupby/groupby.py b/pandas/core/groupby/groupby.py
index c5e81e21e9fd5..05005d2ca94b1 100644
--- a/pandas/core/groupby/groupby.py
+++ b/pandas/core/groupby/groupby.py
@@ -2220,26 +2220,28 @@ def _get_cythonized_result(
         base_func = getattr(libgroupby, how)
 
         for name, obj in self._iterate_slices():
+            values = obj._data._values
+
             if aggregate:
                 result_sz = ngroups
             else:
-                result_sz = len(obj.values)
+                result_sz = len(values)
 
             if not cython_dtype:
-                cython_dtype = obj.values.dtype
+                cython_dtype = values.dtype
 
             result = np.zeros(result_sz, dtype=cython_dtype)
             func = partial(base_func, result, labels)
             inferences = None
 
             if needs_values:
-                vals = obj.values
+                vals = values
                 if pre_processing:
                     vals, inferences = pre_processing(vals)
                 func = partial(func, vals)
 
             if needs_mask:
-                mask = isna(obj.values).view(np.uint8)
+                mask = isna(values).view(np.uint8)
                 func = partial(func, mask)
 
             if needs_ngroups:
@@ -2248,7 +2250,7 @@ def _get_cythonized_result(
             func(**kwargs)  # Call func to modify indexer values in place
 
             if result_is_index:
-                result = algorithms.take_nd(obj.values, result)
+                result = algorithms.take_nd(values, result)
 
             if post_processing:
                 result = post_processing(result, inferences)
diff --git a/pandas/tests/groupby/test_timezone.py b/pandas/tests/groupby/test_timezone.py
new file mode 100644
index 0000000000000..caa9f9323a664
--- /dev/null
+++ b/pandas/tests/groupby/test_timezone.py
@@ -0,0 +1,112 @@
+import pytest
+
+from pandas import DataFrame, Timestamp
+from pandas.util.testing import assert_frame_equal
+
+
+@pytest.mark.parametrize(
+    "data, expected_shift, expected_bfill, expected_ffill",
+    [
+        (
+            {
+                "id": ["A", "B", "A", "B", "A", "B"],
+                "time": [
+                    Timestamp("2019-01-01 12:00:00"),
+                    Timestamp("2019-01-01 12:30:00"),
+                    None,
+                    None,
+                    Timestamp("2019-01-01 14:00:00"),
+                    Timestamp("2019-01-01 14:30:00"),
+                ],
+            },
+            {
+                "time": [
+                    None,
+                    None,
+                    Timestamp("2019-01-01 12:00:00"),
+                    Timestamp("2019-01-01 12:30:00"),
+                    None,
+                    None,
+                ]
+            },
+            {
+                "time": [
+                    Timestamp("2019-01-01 12:00:00"),
+                    Timestamp("2019-01-01 12:30:00"),
+                    Timestamp("2019-01-01 14:00:00"),
+                    Timestamp("2019-01-01 14:30:00"),
+                    Timestamp("2019-01-01 14:00:00"),
+                    Timestamp("2019-01-01 14:30:00"),
+                ]
+            },
+            {
+                "time": [
+                    Timestamp("2019-01-01 12:00:00"),
+                    Timestamp("2019-01-01 12:30:00"),
+                    Timestamp("2019-01-01 12:00:00"),
+                    Timestamp("2019-01-01 12:30:00"),
+                    Timestamp("2019-01-01 14:00:00"),
+                    Timestamp("2019-01-01 14:30:00"),
+                ]
+            },
+        ),
+        (
+            {
+                "id": ["A", "B", "A", "B", "A", "B"],
+                "time": [
+                    Timestamp("2019-01-01 12:00:00", tz="Asia/Tokyo"),
+                    Timestamp("2019-01-01 12:30:00", tz="Asia/Tokyo"),
+                    None,
+                    None,
+                    Timestamp("2019-01-01 14:00:00", tz="Asia/Tokyo"),
+                    Timestamp("2019-01-01 14:30:00", tz="Asia/Tokyo"),
+                ],
+            },
+            {
+                "time": [
+                    None,
+                    None,
+                    Timestamp("2019-01-01 12:00:00", tz="Asia/Tokyo"),
+                    Timestamp("2019-01-01 12:30:00", tz="Asia/Tokyo"),
+                    None,
+                    None,
+                ]
+            },
+            {
+                "time": [
+                    Timestamp("2019-01-01 12:00:00", tz="Asia/Tokyo"),
+                    Timestamp("2019-01-01 12:30:00", tz="Asia/Tokyo"),
+                    Timestamp("2019-01-01 14:00:00", tz="Asia/Tokyo"),
+                    Timestamp("2019-01-01 14:30:00", tz="Asia/Tokyo"),
+                    Timestamp("2019-01-01 14:00:00", tz="Asia/Tokyo"),
+                    Timestamp("2019-01-01 14:30:00", tz="Asia/Tokyo"),
+                ]
+            },
+            {
+                "time": [
+                    Timestamp("2019-01-01 12:00:00", tz="Asia/Tokyo"),
+                    Timestamp("2019-01-01 12:30:00", tz="Asia/Tokyo"),
+                    Timestamp("2019-01-01 12:00:00", tz="Asia/Tokyo"),
+                    Timestamp("2019-01-01 12:30:00", tz="Asia/Tokyo"),
+                    Timestamp("2019-01-01 14:00:00", tz="Asia/Tokyo"),
+                    Timestamp("2019-01-01 14:30:00", tz="Asia/Tokyo"),
+                ]
+            },
+        ),
+    ],
+)
+def test_shift_bfill_ffill_tz(data, expected_shift, expected_bfill, expected_ffill):
+    # GHxxxxx: Check that timezone does not drop in shift, bfill, and ffill
+    df = DataFrame(data)
+
+    result = df.groupby("id").shift()
+    expected = DataFrame(expected_shift)
+    assert_frame_equal(result, expected)
+
+    result = df.groupby("id").bfill()
+    expected = DataFrame(expected_bfill)
+    assert_frame_equal(result, expected)
+
+    result = df.groupby("id").ffill()
+    expected = DataFrame(expected_ffill)
+    assert_frame_equal(result, expected)


Code review comments:
diff_hunk: @@ -120,6 +120,7 @@ Groupby/resample/rolling
 - Bug in :meth:`pandas.core.groupby.DataFrameGroupBy.transform` where applying a timezone conversion lambda function would drop timezone information (:issue:`27496`)
 - Bug in windowing over read-only arrays (:issue:`27766`)
 - Fixed segfault in `pandas.core.groupby.DataFrameGroupBy.quantile` when an invalid quantile was passed (:issue:`27470`)
+- Bug in :meth:`pandas.core.groupby.GroupBy.shift`, :meth:`pandas.core.groupby.GroupBy.bfill` and :meth:`pandas.core.groupby.GroupBy.ffill` where timezone information would be dropped (:issue:`xxxxx`)
path: doc/source/whatsnew/v0.25.1.rst
text: @user1:
If there is not an existing issue for this bug, use the number of this PR.