Repository: pandas-dev/pandas
Pull request: ENH: show percentiles in timestamp describe (#30164)
closes https://github.com/pandas-dev/pandas/issues/30164

When creating a pandas Series of timestamps and calling `describe` on it, it will not show the percentiles of the data, even if these are specified as arguments (`s.describe(percentiles = [0.25, 0.5, 0.75])`).

This PR solves the issue by introducing a new describe logic for timestamps that would:
* Add the percentiles in the same way as for numeric.
* Rename `first` and `last` to `min` and `max` in order to match numeric types.
* Add the mean of the series.
* ~~Show datetime columns alongside numerics by default in the describe method for DataFrames.~~ (just realized there were other issues in which this was deemed to be the desirable behavior)

After this, it more or less matches the functionality of R's `summary` on `POSIXct` and `Date` types.

Why I think this is a good idea: I oftentimes find myself trying to inspect tables of mixed types, and want to get a quick overview of which kind of ranges and variations are there in each column, including timestamps. Currently this is very inconvenient in pandas, and they are not displayed alongside numeric columns by default and don't match the names if passing `include = "all"`, which is what I want to see when I call `DataFrame.describe`.

Examples:
```python
import numpy as np, pandas as pd
dt_list = [
    "2019-01-01 00:01:00",
    "2019-01-01 00:02:00",
    "2019-01-01 00:02:10",
    "2019-01-01 00:10:01",
    "2019-01-01 00:11:00"
]
s = pd.to_datetime(pd.Series(dt_list))
s.describe()
```
```python
pd.DataFrame({
    "col1" : s,
    "col2" : np.arange(s.shape[0])
}).describe(include = "all")
```

Update: corrected the coding style to comply with `black` and pass the automatic test.

Problem statement:
Request: show datetime percentiles in 'describe' method
Would be nice if `pandas.Series.describe` would show percentiles of `datetime` dtypes in the same way it does for other numeric types:
```python
import pandas as pd
dt_list = [
    "2019-01-01 00:01:00",
    "2019-01-01 00:02:00",
    "2019-01-01 00:02:10",
    "2019-01-01 00:10:01",
    "2019-01-01 00:11:00"
]
s = pd.to_datetime(pd.Series(dt_list))
s.describe()
```
```
count                       5
unique                      5
top       2019-01-01 00:02:00
freq                        1
first     2019-01-01 00:01:00
last      2019-01-01 00:11:00
dtype: object
```
```python
s.describe(percentiles = [0.25, 0.5, 0.75])
### doesn't show percentiles
```

Compare to R:
```r
dt <- c(
	"2019-01-01 00:01:00",
	"2019-01-01 00:02:00",
	"2019-01-01 00:02:10",
	"2019-01-01 00:10:01",
	"2019-01-01 00:11:00"
)
summary(as.POSIXct(dt))
```
```
                 Min.               1st Qu.                Median                  Mean               3rd Qu.                  Max. 
"2019-01-01 00:01:00" "2019-01-01 00:02:00" "2019-01-01 00:02:10" "2019-01-01 00:05:14" "2019-01-01 00:10:01" "2019-01-01 00:11:00"
```

Hints from issue discussion:
Sure that sounds reasonable. PR's welcome!

Commit message: ENH: show percentiles in timestamp describe (#30164)

Patch to review:
diff --git a/pandas/core/generic.py b/pandas/core/generic.py
index 6332ff45c59d0..325651aa81954 100644
--- a/pandas/core/generic.py
+++ b/pandas/core/generic.py
@@ -30,7 +30,7 @@
 
 from pandas._config import config
 
-from pandas._libs import Timestamp, iNaT, lib, properties
+from pandas._libs import NaT, Timestamp, iNaT, lib, properties
 from pandas._typing import (
     Axis,
     Dtype,
@@ -9626,26 +9626,8 @@ def describe_categorical_1d(data):
             dtype = None
             if result[1] > 0:
                 top, freq = objcounts.index[0], objcounts.iloc[0]
-
-                if is_datetime64_any_dtype(data):
-                    tz = data.dt.tz
-                    asint = data.dropna().values.view("i8")
-                    top = Timestamp(top)
-                    if top.tzinfo is not None and tz is not None:
-                        # Don't tz_localize(None) if key is already tz-aware
-                        top = top.tz_convert(tz)
-                    else:
-                        top = top.tz_localize(tz)
-                    names += ["top", "freq", "first", "last"]
-                    result += [
-                        top,
-                        freq,
-                        Timestamp(asint.min(), tz=tz),
-                        Timestamp(asint.max(), tz=tz),
-                    ]
-                else:
-                    names += ["top", "freq"]
-                    result += [top, freq]
+                names += ["top", "freq"]
+                result += [top, freq]
 
             # If the DataFrame is empty, set 'top' and 'freq' to None
             # to maintain output shape consistency
@@ -9656,11 +9638,30 @@ def describe_categorical_1d(data):
 
             return pd.Series(result, index=names, name=data.name, dtype=dtype)
 
+        def describe_timestamp_1d(data):
+            # GH-30164
+            tz = data.dt.tz
+            asint = data.dropna().values.view("i8")
+            is_empty = asint.shape[0] == 0
+            stat_index = ["count", "mean", "min"] + formatted_percentiles + ["max"]
+            d = (
+                [
+                    asint.shape[0],
+                    Timestamp(asint.mean(), tz=tz) if not is_empty else NaT,
+                    Timestamp(asint.min(), tz=tz) if not is_empty else NaT,
+                ]
+                + data.quantile(percentiles).tolist()
+                + [Timestamp(asint.max(), tz=tz) if not is_empty else NaT]
+            )
+            return pd.Series(d, index=stat_index, name=data.name)
+
         def describe_1d(data):
             if is_bool_dtype(data):
                 return describe_categorical_1d(data)
             elif is_numeric_dtype(data):
                 return describe_numeric_1d(data)
+            elif is_datetime64_any_dtype(data):
+                return describe_timestamp_1d(data)
             elif is_timedelta64_dtype(data):
                 return describe_numeric_1d(data)
             else:
diff --git a/pandas/tests/frame/methods/test_describe.py b/pandas/tests/frame/methods/test_describe.py
index 251563e51e15a..127233ed2713e 100644
--- a/pandas/tests/frame/methods/test_describe.py
+++ b/pandas/tests/frame/methods/test_describe.py
@@ -253,52 +253,19 @@ def test_describe_tz_values(self, tz_naive_fixture):
 
         expected = DataFrame(
             {
-                "s1": [
-                    5,
-                    np.nan,
-                    np.nan,
-                    np.nan,
-                    np.nan,
-                    np.nan,
-                    2,
-                    1.581139,
-                    0,
-                    1,
-                    2,
-                    3,
-                    4,
-                ],
+                "s1": [5, 2, 0, 1, 2, 3, 4, 1.581139],
                 "s2": [
                     5,
-                    5,
-                    s2.value_counts().index[0],
-                    1,
+                    Timestamp(2018, 1, 3).tz_localize(tz),
                     start.tz_localize(tz),
+                    s2[1],
+                    s2[2],
+                    s2[3],
                     end.tz_localize(tz),
                     np.nan,
-                    np.nan,
-                    np.nan,
-                    np.nan,
-                    np.nan,
-                    np.nan,
-                    np.nan,
                 ],
             },
-            index=[
-                "count",
-                "unique",
-                "top",
-                "freq",
-                "first",
-                "last",
-                "mean",
-                "std",
-                "min",
-                "25%",
-                "50%",
-                "75%",
-                "max",
-            ],
+            index=["count", "mean", "min", "25%", "50%", "75%", "max", "std"],
         )
         result = df.describe(include="all")
         tm.assert_frame_equal(result, expected)
diff --git a/pandas/tests/series/methods/test_describe.py b/pandas/tests/series/methods/test_describe.py
index b147a04b11090..483863bd7b93b 100644
--- a/pandas/tests/series/methods/test_describe.py
+++ b/pandas/tests/series/methods/test_describe.py
@@ -57,13 +57,14 @@ def test_describe_with_tz(self, tz_naive_fixture):
         expected = Series(
             [
                 5,
-                5,
-                s.value_counts().index[0],
-                1,
+                Timestamp(2018, 1, 3).tz_localize(tz),
                 start.tz_localize(tz),
+                s[1],
+                s[2],
+                s[3],
                 end.tz_localize(tz),
             ],
             name=name,
-            index=["count", "unique", "top", "freq", "first", "last"],
+            index=["count", "mean", "min", "25%", "50%", "75%", "max"],
         )
         tm.assert_series_equal(result, expected)


Code review comments:
diff_hunk: @@ -9656,11 +9638,30 @@ def describe_categorical_1d(data):
 
             return pd.Series(result, index=names, name=data.name, dtype=dtype)
 
+        def describe_timestamp_1d(data):
+            # GH-30164
+            tz = data.dt.tz
+            asint = data.dropna().values.view("i8")
path: pandas/core/generic.py
text: @user1:
you don't need the .values.view, just use` len(data.dropna())` for the shape check

data.min() and so on all work and return NaT correctly, none of these gymnastics are needed

@user1:
I know that's what the current code does, but let's simplify