Repository: pandas-dev/pandas

Issue:
BUG(?): Var of Timedelta with empty / NA
*edit*: it seems like the bug is that var returns numeric values for the operation, instead of timedeltas like `.std` (https://github.com/pandas-dev/pandas/issues/18880#issuecomment-353164903)

---

Working on the empty / all-NA stuff. This is related, but separate.

Currently for an all-NA series of timedelta's, we return `NaT` for `.var()` on an empty series (see Out[44]). I think this should be `NaN` since the reduction is returning numeric values, not timedeltas.

```python
In [40]: s = pd.Series(pd.timedelta_range(0, periods=12, freq='S'))

In [41]: s[0] = np.nan

In [42]: s.var()
Out[42]: 1.1e+19

In [43]: s.var(skipna=False)
Out[43]: nan

In [44]: s[:0].var()
Out[44]: NaT
```

I've just noticed some other buggy behavior with aggregations on timedeltas, so maybe this can become a meta-issue about timedeltas and numeric aggregations:

```python
In [57]: s.sum(skipna=False)
Out[57]: Timedelta('-106752 days +00:13:49.145223')
```

I'm guessing we improperly add the timedelta min here.


Discussion:
Ahh interesting, `.std` returns the units in timedelta's so maybe that's the bug that `.var` isn't handling dtypes correctly:

```python
In [11]: s.std()
Out[11]: Timedelta('0 days 00:00:03.316624')

In [12]: s[:0].std()
Out[12]: NaT
```
what would this be numeric? any reduction should stay the same dtype (except int-> float). 
yeah it prob doesn't does handle the ufunc propertly (sqrt), e.g.

This works, but prob not what its doing.
```
In [7]: pd.Timedelta(np.sqrt(s.std().value))
Out[7]: Timedelta('0 days 00:00:00.000057')
```
Yeah, I was just saying numeric because the non-NA versions were numeric, but that's the real bug. I'll edit the original bug report.
Should this be closed? @TomAugspurger 
I don't think so, do you? `Series[timedelta].var()` is still numeric for me, rather than a Timedelta.
Shouldnt `Series[timedelta].var()` raise?  `std()` makes sense.
> Shouldnt `Series[timedelta].var()` raise? `std()` makes sense.

Can you clarify this? The meaning of std & var for timedeltas is a bit hazy to me, but why would one make sense and not the other?
> Can you clarify this? The meaning of std & var for timedeltas is a bit hazy to me, but why would one make sense and not the other?

The issue is the units.  `Var(td64ns)` would have to have unit `ns^2`, which we don't have a way to represent.
One more question, why do we not say that `Var(td64ns) <=>
Timedelta(Var(td64ns.asi8))`? IOW

```python
In [14]: values = [1000, 2000, 3000]

In [15]: pd.Series(values).std()
Out[15]: 1000.0

In [16]: pd.Series(values).var()
Out[16]: 1000000.0

In [18]: pd.Series(pd.to_timedelta(values)).std()
Out[18]: Timedelta('0 days 00:00:00.000001')  # = Timedelta(values.std())

In [19]: pd.Series(pd.to_timedelta(values)).std().value
Out[19]: 1000
```

I agreed that ns^2 isn't something we can represent, but what about
Timedelta(i8^2)? Like I said, this is hazy to me, so it may be meaningless.


On Fri, Jun 7, 2019 at 8:49 AM jbrockmendel <notifications@github.com>
wrote:

> Can you clarify this? The meaning of std & var for timedeltas is a bit
> hazy to me, but why would one make sense and not the other?
>
> The issue is the units. Var(td64ns) would have to have unit ns^2, which
> we don't have a way to represent.
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <https://github.com/pandas-dev/pandas/issues/18880?email_source=notifications&email_token=AAKAOIRHOOKY5NU7V3B5HWDPZJRPTA5CNFSM4EJDQG6KYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODXF366Q#issuecomment-499892090>,
> or mute the thread
> <https://github.com/notifications/unsubscribe-auth/AAKAOIVU5JZAMMXITAREZ7TPZJRPTANCNFSM4EJDQG6A>
> .
>



```
i8values = np.array([1e9, 2e9, 3e9], dtype='i8')
tdi = pd.to_timedelta(i8values)
ser = pd.Series(tdi)

>>> ser.std()
Timedelta('0 days 00:00:01')
```

I think we're on the same page that `ser.std()` here makes sense.  If we _could_ do `ser.var()`, it would have to obey the identity `ser.var() == (ser.std())**2`.  But the right hand side we can't do.

Does this help?  Any way you do it, you need to get a ns^2 in there.

Resolving patch:
diff --git a/pandas/core/nanops.py b/pandas/core/nanops.py
--- a/pandas/core/nanops.py
+++ b/pandas/core/nanops.py
@@ -704,11 +704,14 @@ def nanstd(values, axis=None, skipna=True, ddof=1, mask=None):
     >>> nanops.nanstd(s)
     1.0
     """
+    orig_dtype = values.dtype
+    values, mask, dtype, dtype_max, fill_value = _get_values(values, skipna, mask=mask)
+
     result = np.sqrt(nanvar(values, axis=axis, skipna=skipna, ddof=ddof, mask=mask))
-    return _wrap_results(result, values.dtype)
+    return _wrap_results(result, orig_dtype)
 
 
-@disallow("M8")
+@disallow("M8", "m8")
 @bottleneck_switch(ddof=1)
 def nanvar(values, axis=None, skipna=True, ddof=1, mask=None):
     """
diff --git a/pandas/core/series.py b/pandas/core/series.py
--- a/pandas/core/series.py
+++ b/pandas/core/series.py
@@ -3988,6 +3988,7 @@ def _reduce(
         If we have an ndarray as a value, then simply perform the operation,
         otherwise delegate to the object.
         """
+
         delegate = self._values
 
         if axis is not None:


Test patch:
diff --git a/pandas/tests/reductions/test_reductions.py b/pandas/tests/reductions/test_reductions.py
--- a/pandas/tests/reductions/test_reductions.py
+++ b/pandas/tests/reductions/test_reductions.py
@@ -300,11 +300,14 @@ def test_timedelta_ops(self):
         assert result[0] == expected
 
         # invalid ops
-        for op in ["skew", "kurt", "sem", "prod"]:
+        for op in ["skew", "kurt", "sem", "prod", "var"]:
             msg = "reduction operation '{}' not allowed for this dtype"
             with pytest.raises(TypeError, match=msg.format(op)):
                 getattr(td, op)()
 
+            with pytest.raises(TypeError, match=msg.format(op)):
+                getattr(td.to_frame(), op)(numeric_only=False)
+
         # GH#10040
         # make sure NaT is properly handled by median()
         s = Series([Timestamp("2015-02-03"), Timestamp("2015-02-07")])
@@ -636,8 +639,13 @@ def test_ops_consistency_on_empty(self, method):
         assert pd.isna(result)
 
         # timedelta64[ns]
-        result = getattr(Series(dtype="m8[ns]"), method)()
-        assert result is pd.NaT
+        tdser = Series([], dtype="m8[ns]")
+        if method == "var":
+            with pytest.raises(TypeError, match="operation 'var' not allowed"):
+                getattr(tdser, method)()
+        else:
+            result = getattr(tdser, method)()
+            assert result is pd.NaT
 
     def test_nansum_buglet(self):
         ser = Series([1.0, np.nan], index=[0, 1])