Repository: pandas-dev/pandas
Pull request: Series.describe returns first and last for tz-aware datetimes
GH issue 21328

- [x] closes #21328 
- [x] tests added / passed
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Problem statement:
describe() does not report first and last on tz-aware datetime Series
#### Code Sample, a copy-pastable example if possible

```python
s = pd.Series(pd.date_range("2018-01-01","2018-01-05"))
s.describe() # includes first and last
s.dt.tz_localize('US/Eastern').describe() # does not include first and last
```
#### Problem description

The `describe` method on a timezone-naive datetime Series reports the "first" (min) and "last" (max) values. I would expect the same to be true for a timezone-aware datetime Series, since `min` and `max` are defined there as well.

#### Output of ``pd.show_versions()``

<details>

INSTALLED VERSIONS
------------------
commit: None
python: 3.6.4.final.0
python-bits: 64
OS: Linux
OS-release: 4.13.0-43-generic
machine: x86_64
processor: x86_64
byteorder: little
LC_ALL: None
LANG: en_US.UTF-8
LOCALE: en_US.UTF-8

pandas: 0.23.0
pytest: 3.4.2
pip: 9.0.1
setuptools: 38.4.0
Cython: None
numpy: 1.14.3
scipy: 1.1.0
pyarrow: 0.9.0
xarray: None
IPython: 6.2.1
sphinx: 1.7.3
patsy: 0.5.0
dateutil: 2.7.3
pytz: 2018.4
blosc: None
bottleneck: None
tables: None
numexpr: None
feather: 0.4.0
matplotlib: 2.1.2
openpyxl: None
xlrd: None
xlwt: None
xlsxwriter: None
lxml: None
bs4: None
html5lib: 1.0.1
sqlalchemy: 1.2.7
pymysql: None
psycopg2: 2.7.4 (dt dec pq3 ext lo64)
jinja2: 2.10
s3fs: None
fastparquet: None
pandas_gbq: None
pandas_datareader: None

</details>

Hints from issue discussion:
Thanks for the report; that definitely sounds reasonable. This block below needs some additional logic to handle timezone aware datetimes:

https://github.com/pandas-dev/pandas/blob/7dc6f7023ce439e8ef10ef336f314b10192911c5/pandas/core/generic.py#L8533-L8538

Pull requests are welcome!
Right, looks like replacing that check with [`is_datetime64_any_dtype`](https://github.com/pandas-dev/pandas/blob/7dc6f7023ce439e8ef10ef336f314b10192911c5/pandas/core/dtypes/common.py#L1043) would catch the tz-aware ones.

Thanks, I'll try to get a PR in soon.

Commit message: fix formatting in whatsnew and add issue number to tests

Patch to review:
diff --git a/doc/source/whatsnew/v0.24.0.txt b/doc/source/whatsnew/v0.24.0.txt
index 691345ad26e58..f0d62627462bb 100644
--- a/doc/source/whatsnew/v0.24.0.txt
+++ b/doc/source/whatsnew/v0.24.0.txt
@@ -295,6 +295,8 @@ Datetimelike
 ^^^^^^^^^^^^
 
 - Fixed bug where two :class:`DateOffset` objects with different ``normalize`` attributes could evaluate as equal (:issue:`21404`)
+- Bug in :class:`Index` with ``datetime64[ns, tz]`` dtype that did not localize integer data correctly (:issue:`20964`)
+- Fixed bug where :meth:`DataFrame.describe` and :meth:`Series.describe` on tz-aware datetimes did not show `first` and `last` result (:issue:`21328`)
 
 Timedelta
 ^^^^^^^^^
diff --git a/pandas/core/generic.py b/pandas/core/generic.py
index 818dd1b408518..65ca467a05840 100644
--- a/pandas/core/generic.py
+++ b/pandas/core/generic.py
@@ -20,7 +20,7 @@
     is_bool_dtype,
     is_categorical_dtype,
     is_numeric_dtype,
-    is_datetime64_dtype,
+    is_datetime64_any_dtype,
     is_timedelta64_dtype,
     is_datetime64tz_dtype,
     is_list_like,
@@ -8531,12 +8531,13 @@ def describe_categorical_1d(data):
             if result[1] > 0:
                 top, freq = objcounts.index[0], objcounts.iloc[0]
 
-                if is_datetime64_dtype(data):
+                if is_datetime64_any_dtype(data):
+                    tz = data.dt.tz
                     asint = data.dropna().values.view('i8')
                     names += ['top', 'freq', 'first', 'last']
-                    result += [tslib.Timestamp(top), freq,
-                               tslib.Timestamp(asint.min()),
-                               tslib.Timestamp(asint.max())]
+                    result += [tslib.Timestamp(top, tz=tz), freq,
+                               tslib.Timestamp(asint.min(), tz=tz),
+                               tslib.Timestamp(asint.max(), tz=tz)]
                 else:
                     names += ['top', 'freq']
                     result += [top, freq]
diff --git a/pandas/tests/frame/test_analytics.py b/pandas/tests/frame/test_analytics.py
index d357208813dd8..c0e9b89c1877f 100644
--- a/pandas/tests/frame/test_analytics.py
+++ b/pandas/tests/frame/test_analytics.py
@@ -417,6 +417,28 @@ def test_describe_timedelta_values(self):
                     "max           5 days 00:00:00         0 days 05:00:00")
         assert repr(res) == exp_repr
 
+    def test_describe_tz_values(self, tz_naive_fixture):
+        # GH 21332
+        tz = tz_naive_fixture
+        s1 = Series(range(5))
+        start = Timestamp(2018, 1, 1)
+        end = Timestamp(2018, 1, 5)
+        s2 = Series(date_range(start, end, tz=tz))
+        df = pd.DataFrame({'s1': s1, 's2': s2})
+
+        expected = DataFrame({'s1': [5, np.nan, np.nan, np.nan, np.nan, np.nan,
+                                     2, 1.581139, 0, 1, 2, 3, 4],
+                              's2': [5, 5, s2.value_counts().index[0], 1,
+                                     start.tz_localize(tz),
+                                     end.tz_localize(tz), np.nan, np.nan,
+                                     np.nan, np.nan, np.nan, np.nan, np.nan]},
+                             index=['count', 'unique', 'top', 'freq', 'first',
+                                    'last', 'mean', 'std', 'min', '25%', '50%',
+                                    '75%', 'max']
+                             )
+        res = df.describe(include='all')
+        tm.assert_frame_equal(res, expected)
+
     def test_reduce_mixed_frame(self):
         # GH 6806
         df = DataFrame({
diff --git a/pandas/tests/series/test_analytics.py b/pandas/tests/series/test_analytics.py
index fcfaff9b11002..b574b6dce930c 100644
--- a/pandas/tests/series/test_analytics.py
+++ b/pandas/tests/series/test_analytics.py
@@ -336,6 +336,23 @@ def test_describe(self):
                           index=['count', 'unique', 'top', 'freq'])
         tm.assert_series_equal(result, expected)
 
+    def test_describe_with_tz(self, tz_naive_fixture):
+        # GH 21332
+        tz = tz_naive_fixture
+        name = tz_naive_fixture
+        start = Timestamp(2018, 1, 1)
+        end = Timestamp(2018, 1, 5)
+        s = Series(date_range(start, end, tz=tz), name=name)
+        result = s.describe()
+        expected = Series(
+            [5, 5, s.value_counts().index[0], 1, start.tz_localize(tz),
+             end.tz_localize(tz)
+             ],
+            name=name,
+            index=['count', 'unique', 'top', 'freq', 'first', 'last']
+        )
+        tm.assert_series_equal(result, expected)
+
     def test_argsort(self):
         self._check_accum_op('argsort', check_dtype=False)
         argsorted = self.ts.argsort()


Code review comments:
diff_hunk: @@ -295,6 +295,8 @@ Datetimelike
 ^^^^^^^^^^^^
 
 - Fixed bug where two :class:`DateOffset` objects with different ``normalize`` attributes could evaluate as equal (:issue:`21404`)
+- Bug in :class:`Index` with ``datetime64[ns, tz]`` dtype that did not localize integer data correctly (:issue:`20964`)
+- Fixed bug where :meth:`DataFrame.describe` and :meth:`Series.describe` on tz-aware datetimes did not show `first` and `last` result (:issue:`21328`)
path: doc/source/whatsnew/v0.24.0.txt
text: @user1:
One more nit: Could you move this entry to the timezones section below?

Also the entry above this one exists in the timezone section already i.e. you can remove it.

Thanks for you patience. 

@author:
Done, no problem.