Repository: pandas-dev/pandas (Python)

Issue:
BUG:dataframe.groupby('some_column').timedelta.sum() results wrong when timedelta contains NaT (pandas=1.3.0)
- [y] I have checked that this issue has not already been reported.

- [y] I have confirmed this bug exists on the latest version of pandas.

- [ ] (optional) I have confirmed this bug exists on the master branch of pandas.

---

**Note**: Please read [this guide](https://matthewrocklin.com/blog/work/2018/02/28/minimal-bug-reports) detailing how to provide the necessary information for us to reproduce your bug.

#### Code Sample, a copy-pastable example
this is a tiny example
```python
tmp = pd.DataFrame({'id': ['1624477460271-3908654213', '1624477460271-3908654213'], 'dt':[np.nan, pd.to_timedelta('0:0:2')]})
tmp
Out[6]: 
                         id              dt
0  1624477460271-3908654213             NaT
1  1624477460271-3908654213 0 days 00:00:02
```
#### Problem description

dataframe.groupby('some_column').timedelta.sum() results wrong when timedelta contains NaT
```python
tmp.groupby('id').dt.sum()
Out[7]: 
id
1624477460271-3908654213   -106752 days +00:12:45.145224192
Name: dt, dtype: timedelta64[ns]
```

#### Expected Output
```python
tmp.dt.sum()
Out[8]: Timedelta('0 days 00:00:02')
```
#### Output of ``pd.show_versions()``

<details>

[pandas 1.3.0]

</details>



Discussion:
Please povide something which is copy and pasteable (see https://matthewrocklin.com/blog/work/2018/02/28/minimal-bug-reports)

also please format your code
> Please povide something which is copy and pasteable (see https://matthewrocklin.com/blog/work/2018/02/28/minimal-bug-reports)
> 
> also please format your code

updated
I also encountered this problem, it seems quite a major bug. 

@phofl  Here is a full copy-pastable example :

```python
import pandas as pd

df = pd.DataFrame({
    'a': [1, 1, 2, 2],
    'b': [pd.Timedelta('1d'), pd.Timedelta('2d'), pd.Timedelta('3d'), pd.NaT],
})

print('\ndf:')
print(df)

print('\ndf.dtypes:')
print(df.dtypes)

print('\ndf.groupby(\'a\').b.sum():')
print(df.groupby('a').b.sum())
```

![326DD059-EB73-4C3A-9AE3-41E8A8AD75B1](https://user-images.githubusercontent.com/20555592/127364967-55d28059-1346-40d0-9d63-8f5d864d357f.png)


#### Output of ``pd.show_versions()``

<details>

INSTALLED VERSIONS
------------------
commit           : c7f7443c1bad8262358114d5e88cd9c8a308e8aa
python           : 3.9.5.final.0
python-bits      : 64
OS               : Darwin
OS-release       : 20.4.0
Version          : Darwin Kernel Version 20.4.0: Thu Apr 22 21:46:47 PDT 2021; root:xnu-7195.101.2~1/RELEASE_X86_64
machine          : x86_64
processor        : i386
byteorder        : little
LC_ALL           : en_US.UTF-8
LANG             : en_US.UTF-8
LOCALE           : en_US.UTF-8

pandas           : 1.3.1
numpy            : 1.21.1
pytz             : 2021.1
dateutil         : 2.8.1
pip              : 21.2.1
setuptools       : 49.6.0
Cython           : 0.29.24
pytest           : 6.2.3
hypothesis       : None
sphinx           : 3.5.4
blosc            : None
feather          : None
xlsxwriter       : 1.4.4
lxml.etree       : 4.6.3
html5lib         : None
pymysql          : 1.0.2
psycopg2         : 2.9.1 (dt dec pq3 ext lo64)
jinja2           : 2.11.3
IPython          : 7.25.0
pandas_datareader: None
bs4              : 4.9.3
bottleneck       : 1.3.2
fsspec           : None
fastparquet      : None
gcsfs            : None
matplotlib       : 3.4.2
numexpr          : None
odfpy            : None
openpyxl         : 3.0.7
pandas_gbq       : None
pyarrow          : None
pyxlsb           : None
s3fs             : None
scipy            : 1.7.0
sqlalchemy       : 1.4.22
tables           : None
tabulate         : 0.8.9
xarray           : None
xlrd             : 2.0.1
xlwt             : None
numba            : None

</details>

first bad commit: [fa8f68e9a788901bc6459acb228c7cc8412791d3] BUG/API: SeriesGroupBy reduction with numeric_only=True (#41342)

cc @jbrockmendel 
changing milestone to 1.3.5


@phofl @simonjayhawkins, thanks for handling this request. 

Do you think this problem will be fixed, or is it considered to be a bad practice to use groupby & mean/sum on `pd.Timedelta` values?
A PR addressing this would be welcome.
good to fix but not necessary for 1.3.x
Best guess is a `is_datetimelike` arg in `group_add` is needed



Patch:
diff --git a/doc/source/whatsnew/v1.3.5.rst b/doc/source/whatsnew/v1.3.5.rst
index dabd9a650f45b..048cd978c4478 100644
--- a/doc/source/whatsnew/v1.3.5.rst
+++ b/doc/source/whatsnew/v1.3.5.rst
@@ -19,6 +19,7 @@ Fixed regressions
 - Fixed regression in creating a :class:`DataFrame` from a timezone-aware :class:`Timestamp` scalar near a Daylight Savings Time transition (:issue:`42505`)
 - Fixed performance regression in :func:`read_csv` (:issue:`44106`)
 - Fixed regression in :meth:`Series.duplicated` and :meth:`Series.drop_duplicates` when Series has :class:`Categorical` dtype with boolean categories (:issue:`44351`)
+- Fixed regression in :meth:`.GroupBy.sum` with ``timedelta64[ns]`` dtype containing ``NaT`` failing to treat that value as NA (:issue:`42659`)
 -
 
 .. ---------------------------------------------------------------------------
diff --git a/pandas/_libs/groupby.pyi b/pandas/_libs/groupby.pyi
index 0450a3483d346..8eccd0eec8a1c 100644
--- a/pandas/_libs/groupby.pyi
+++ b/pandas/_libs/groupby.pyi
@@ -56,6 +56,7 @@ def group_add(
     values: np.ndarray,  # ndarray[complexfloating_t, ndim=2]
     labels: np.ndarray,  # const intp_t[:]
     min_count: int = ...,
+    datetimelike: bool = ...,
 ) -> None: ...
 def group_prod(
     out: np.ndarray,  # floating[:, ::1]
diff --git a/pandas/_libs/groupby.pyx b/pandas/_libs/groupby.pyx
index 078cb8e02e824..c0d1405e92518 100644
--- a/pandas/_libs/groupby.pyx
+++ b/pandas/_libs/groupby.pyx
@@ -495,7 +495,8 @@ def group_add(add_t[:, ::1] out,
               int64_t[::1] counts,
               ndarray[add_t, ndim=2] values,
               const intp_t[::1] labels,
-              Py_ssize_t min_count=0) -> None:
+              Py_ssize_t min_count=0,
+              bint datetimelike=False) -> None:
     """
     Only aggregates on axis=0 using Kahan summation
     """
@@ -557,7 +558,14 @@ def group_add(add_t[:, ::1] out,
                     val = values[i, j]
 
                     # not nan
-                    if val == val:
+                    # With dt64/td64 values, values have been cast to float64
+                    #  instead if int64 for group_add, but the logic
+                    #  is otherwise the same as in _treat_as_na
+                    if val == val and not (
+                        add_t is float64_t
+                        and datetimelike
+                        and val == <float64_t>NPY_NAT
+                    ):
                         nobs[lab, j] += 1
                         y = val - compensation[lab, j]
                         t = sumx[lab, j] + y
diff --git a/pandas/core/groupby/ops.py b/pandas/core/groupby/ops.py
index 8223a04883738..7915e107afae6 100644
--- a/pandas/core/groupby/ops.py
+++ b/pandas/core/groupby/ops.py
@@ -531,6 +531,16 @@ def _call_cython_op(
                     result_mask=result_mask,
                     is_datetimelike=is_datetimelike,
                 )
+            elif self.how in ["add"]:
+                # We support datetimelike
+                func(
+                    result,
+                    counts,
+                    values,
+                    comp_ids,
+                    min_count,
+                    datetimelike=is_datetimelike,
+                )
             else:
                 func(result, counts, values, comp_ids, min_count)
         else:


Test patch:
diff --git a/pandas/tests/groupby/test_function.py b/pandas/tests/groupby/test_function.py
index c462db526b36d..39a3e82fc2d98 100644
--- a/pandas/tests/groupby/test_function.py
+++ b/pandas/tests/groupby/test_function.py
@@ -1162,3 +1162,27 @@ def test_mean_on_timedelta():
         pd.to_timedelta([4, 5]), name="time", index=Index(["A", "B"], name="cat")
     )
     tm.assert_series_equal(result, expected)
+
+
+def test_groupby_sum_timedelta_with_nat():
+    # GH#42659
+    df = DataFrame(
+        {
+            "a": [1, 1, 2, 2],
+            "b": [pd.Timedelta("1d"), pd.Timedelta("2d"), pd.Timedelta("3d"), pd.NaT],
+        }
+    )
+    td3 = pd.Timedelta(days=3)
+
+    gb = df.groupby("a")
+
+    res = gb.sum()
+    expected = DataFrame({"b": [td3, td3]}, index=Index([1, 2], name="a"))
+    tm.assert_frame_equal(res, expected)
+
+    res = gb["b"].sum()
+    tm.assert_series_equal(res, expected["b"])
+
+    res = gb["b"].sum(min_count=2)
+    expected = Series([td3, pd.NaT], dtype="m8[ns]", name="b", index=expected.index)
+    tm.assert_series_equal(res, expected)