Repository: pandas-dev/pandas
Pull request: BUG: Fix bug in quantile() for resample and groupby with Timedelta
- [x] closes #29485
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry


Problem statement:
groupby.quantile() fails on Timedelta columns
#### Code Sample, groupby quantile example

```python
# groupby quantile example
import pandas as pd

print(f"Pandas {pd.__version__}")

df = pd.DataFrame({
    'idx': [1, 1, 2],
    'start_date': ['2019-10-02', '2019-10-04', '2019-10-05'],
    'arrival_date': ['2019-11-02', '2019-10-14', '2019-10-15']})

for c in ['start_date', 'arrival_date']:
    df[c] = pd.to_datetime(df[c])

df['delta'] = df['arrival_date'] - df['start_date']

print(df.groupby('idx')[['delta']].quantile(.9))
```
#### Problem description

The code example used to work in 0.21.1.
In 0.25 I obtain a "TypeError: No matching signature found"

#### Expected Output
Pandas 0.21.1

               delta
idx
1   28 days 21:36:00
2   10 days 00:00:00


#### Output of ``pd.show_versions()``

<details>


INSTALLED VERSIONS
------------------
commit           : None
python           : 3.6.8.final.0
python-bits      : 64
OS               : Linux
OS-release       : 3.10.0-693.el7.x86_64
machine          : x86_64
processor        : x86_64
byteorder        : little
LC_ALL           : None
LANG             : en_US.UTF-8
LOCALE           : en_US.UTF-8

pandas           : 0.25.3
numpy            : 1.17.3
pytz             : 2019.3
dateutil         : 2.8.1
pip              : 18.1
setuptools       : 40.8.0.post20191016
Cython           : 0.29.14
pytest           : None
hypothesis       : None
sphinx           : None
blosc            : None
feather          : None
xlsxwriter       : None
lxml.etree       : None
html5lib         : None
pymysql          : None
psycopg2         : None
jinja2           : None
IPython          : None
pandas_datareader: None
bs4              : None
bottleneck       : None
fastparquet      : None
gcsfs            : None
lxml.etree       : None
matplotlib       : None
numexpr          : None
odfpy            : None
openpyxl         : None
pandas_gbq       : None
pyarrow          : None
pytables         : None
s3fs             : None
scipy            : None
sqlalchemy       : None
tables           : None
xarray           : None
xlrd             : None
xlwt             : None
xlsxwriter       : None

</details>

Hints from issue discussion:
The method was overhauled in 0.25 to push it down from the Python space to C. If you want to poke around the implementation is in pandas.core.groupby.groupby
a related version (different error on master)
```
In [8]: x = pd.DataFrame({'time': pd.date_range('20200101', periods=10, tz='UTC'), 'value': pd.to_timedelta(np.arange(10), unit='s')}).set_index('time')             

In [9]: x                                                                                                                                                            
Out[9]: 
                                    value
time                                     
2020-01-01 00:00:00+00:00 0 days 00:00:00
2020-01-02 00:00:00+00:00 0 days 00:00:01
2020-01-03 00:00:00+00:00 0 days 00:00:02
2020-01-04 00:00:00+00:00 0 days 00:00:03
2020-01-05 00:00:00+00:00 0 days 00:00:04
2020-01-06 00:00:00+00:00 0 days 00:00:05
2020-01-07 00:00:00+00:00 0 days 00:00:06
2020-01-08 00:00:00+00:00 0 days 00:00:07
2020-01-09 00:00:00+00:00 0 days 00:00:08
2020-01-10 00:00:00+00:00 0 days 00:00:09

In [10]: x.resample('2D').quantile(0.99)                                                                                                                             
TypeError: Cannot cast TimedeltaArray to dtype float64

```

Commit message: BUG: Fix bug in quantile() for resample and groupby with Timedelta

Patch to review:
diff --git a/doc/source/whatsnew/v1.2.0.rst b/doc/source/whatsnew/v1.2.0.rst
index 9fc094330fb36..6f7c0a514ef23 100644
--- a/doc/source/whatsnew/v1.2.0.rst
+++ b/doc/source/whatsnew/v1.2.0.rst
@@ -462,6 +462,7 @@ Groupby/resample/rolling
 - Bug in :meth:`RollingGroupby.count` where a ``ValueError`` was raised when specifying the ``closed`` parameter (:issue:`35869`)
 - Bug in :meth:`DataFrame.groupby.rolling` returning wrong values with partial centered window (:issue:`36040`).
 - Bug in :meth:`DataFrameGroupBy.rolling` returned wrong values with timeaware window containing ``NaN``. Raises ``ValueError`` because windows are not monotonic now (:issue:`34617`)
+- Bug in :meth:`DataFrameGroupBy.quantile` and :meth:`DataFrame.resample().quantile()` raised ``TypeError`` when values to calculate quantile over where ``Timedelta`` (:issue:`29485`)
 
 Reshaping
 ^^^^^^^^^
diff --git a/pandas/core/groupby/groupby.py b/pandas/core/groupby/groupby.py
index 3938adce6736a..0364092ff5d76 100644
--- a/pandas/core/groupby/groupby.py
+++ b/pandas/core/groupby/groupby.py
@@ -60,6 +60,7 @@ class providing the base-class of operations.
     is_numeric_dtype,
     is_object_dtype,
     is_scalar,
+    is_timedelta64_dtype,
 )
 from pandas.core.dtypes.missing import isna, notna
 
@@ -2151,6 +2152,9 @@ def pre_processor(vals: np.ndarray) -> Tuple[np.ndarray, Optional[Type]]:
             elif is_datetime64_dtype(vals.dtype):
                 inference = "datetime64[ns]"
                 vals = np.asarray(vals).astype(float)
+            elif is_timedelta64_dtype(vals.dtype):
+                inference = "timedelta64[ns]"
+                vals = np.asarray(vals).astype(float)
 
             return vals, inference
 
diff --git a/pandas/tests/groupby/test_quantile.py b/pandas/tests/groupby/test_quantile.py
index 9338742195bfe..d56f1fbea456b 100644
--- a/pandas/tests/groupby/test_quantile.py
+++ b/pandas/tests/groupby/test_quantile.py
@@ -240,3 +240,21 @@ def test_groupby_quantile_skips_invalid_dtype(q):
     result = df.groupby("a").quantile(q)
     expected = df.groupby("a")[["b"]].quantile(q)
     tm.assert_frame_equal(result, expected)
+
+
+def test_groupby_timedelta_quantile():
+    # GH: 29485
+    df = pd.DataFrame(
+        {"value": pd.to_timedelta(np.arange(4), unit="s"), "group": [1, 1, 2, 2]}
+    )
+    result = df.groupby("group").quantile(0.99)
+    expected = pd.DataFrame(
+        {
+            "value": [
+                pd.Timedelta("0 days 00:00:00.990000"),
+                pd.Timedelta("0 days 00:00:02.990000"),
+            ]
+        },
+        index=pd.Index([1, 2], name="group"),
+    )
+    tm.assert_frame_equal(result, expected)
diff --git a/pandas/tests/resample/test_timedelta.py b/pandas/tests/resample/test_timedelta.py
index 3fa85e62d028c..51860a04dce7b 100644
--- a/pandas/tests/resample/test_timedelta.py
+++ b/pandas/tests/resample/test_timedelta.py
@@ -165,3 +165,22 @@ def test_resample_with_timedelta_yields_no_empty_groups():
         index=pd.timedelta_range(start="1s", periods=13, freq="3s"),
     )
     tm.assert_frame_equal(result, expected)
+
+
+def test_resample_quantile_timedelta():
+    # GH: 29485
+    df = pd.DataFrame(
+        {"value": pd.to_timedelta(np.arange(4), unit="s")},
+        index=pd.date_range("20200101", periods=4, tz="UTC"),
+    )
+    result = df.resample("2D").quantile(0.99)
+    expected = pd.DataFrame(
+        {
+            "value": [
+                pd.Timedelta("0 days 00:00:00.990000"),
+                pd.Timedelta("0 days 00:00:02.990000"),
+            ]
+        },
+        index=pd.date_range("20200101", periods=2, tz="UTC", freq="2D"),
+    )
+    tm.assert_frame_equal(result, expected)


Code review comments:
diff_hunk: @@ -462,6 +462,7 @@ Groupby/resample/rolling
 - Bug in :meth:`RollingGroupby.count` where a ``ValueError`` was raised when specifying the ``closed`` parameter (:issue:`35869`)
 - Bug in :meth:`DataFrame.groupby.rolling` returning wrong values with partial centered window (:issue:`36040`).
 - Bug in :meth:`DataFrameGroupBy.rolling` returned wrong values with timeaware window containing ``NaN``. Raises ``ValueError`` because windows are not monotonic now (:issue:`34617`)
+- Bug in :meth:`DataFrameGroupBy.quantile` and :meth:`DataFrame.resample().quantile()` raised ``TypeError`` when values to calculate quantile over where ``Timedelta`` (:issue:`29485`)
path: doc/source/whatsnew/v1.2.0.rst
text: @user3:
I know all the other entries do the same (so no blame at all! ;)), but both `` :meth:`DataFrameGroupBy.quantile` `` and `` :meth:`DataFrame.resample().quantile()` `` are not working to create API doc links (see https://pandas.pydata.org/docs/dev/whatsnew/v1.2.0.html#groupby-resample-rolling, everything bold pink is a link that doesn't work)

So either we simply keep it as code snippet, and then you can remove the ``:meth:`` part. Eg something like (either with `df` or `DataFrame`, not sure what is clearest):

```
Bug in ``df.groupby(..).quantile()` and `DataFrame.resample(..).quantile()` raised ...
```

or either we need to create the proper link according to the actual objects:

```
Bug in :meth:`pandas.core.groupby.DataFrameGroupBy.quantile` and :meth:`pandas.core.resample.Resampler.quantile` raised ...
```

Now, personally I don't think we should show those full paths, as a user doesn't need to care about the fact that the groupby quantile methods lives in `pandas.core.groupby.DataFrameGroupBy` ... So then you can also do a combination of both above, and give a custom url link text:

```
Bug in :meth:`df.groupby(..).quantile() <pandas.core.groupby.DataFrameGroupBy.quantile>` ...
```


cc @user1 @user2 (since this was mentioned on gitter)

@user2:
@user3 Can we use a tilde to hide the full path? That is if we do :meth:\`~pandas.core.groupby.DataFrameGroupBy.quantile\` then I think it will only show the method at the end but still provide a link. Then could use the surrounding text to provide more context of which quantile method we're talking about.

@user3:
Indeed, that's yet another option ;) So as you say, since the link with only show `quantile()`, then the surrounding text needs to clarify what it is about, like "groupby `quantile()`" or "`quantile()` with groupby" or something alike. 

Personally I don't have a big preference for either solution (as long as we are not showing "(core.groupby.)DataFrameGroupBy" as we are doing now). I think a link text like `df.groupby(..).quantile()` is also quite clear in what it is about. Probably depends a bit on the full sentence which might fit best in which case. 

@user1:
The last option @user3 gave:

```
:meth:`df.groupby(..).quantile() <pandas.core.groupby.DataFrameGroupBy.quantile>`
```

seems to be the best of all worlds to me.

@author:
Yeah, I think thats probably the best solution. Is a bit more to write for us, but clearer to the reader.