BUG: GH11349 where Series.apply and Series.map did not box timedelta64
State: closed

closes #11349 
closes #11925 


Labels:
name: Bug
name: Timedelta
description: Timedelta data type

Comments:
One test failing `test_timedelta64_conversions (pandas.tests.test_series.TestSeries)`

OK:

```
import pandas as pd
import numpy as np
startdate = pd.Series(pd.date_range('2013-01-01', '2013-01-03'))
enddate = pd.Series(pd.date_range('2013-03-01', '2013-03-03'))
s1 = enddate - startdate
s1[2] = np.nan
s1.apply(lambda x: x / pd.Timedelta(1, 'D'))
```

Fail:

```
s1.apply(lambda x: x / np.timedelta64(1, 'D'))
```

I think because 
`pd.NaT/np.timedelta64(1, 'D')` fails

Solutions I can think of all seem quite intrusive:
1) define `__truediv__` for `Timestamp` (maybe too small a use case to justify the change IMO)
2) define `__truediv__` for `NaTType` (confusing because time series operations might succeed or fail depending on whether it is all `NaT`)

Besides this, is there any need to have `NaT` support other `np.timedelta` or `pd.Timedelta` operations?

why would this be failing?

```
In [23]: s1
Out[23]: 
0   59 days
1   59 days
2       NaT
dtype: timedelta64[ns]

In [22]: s1.apply(lambda x: x / pd.Timedelta(1, 'D'))
Out[22]: 
0    59
1    59
2   NaN
dtype: float64

In [25]: s1.apply(lambda x: x / np.timedelta64(1, 'D'))
Out[25]: 
0    59
1    59
2   NaN
dtype: float64
```

I meant if you are to box `timedelta64` on `apply`. 
It would fail for the same reason this fails in master

```
In [9]: s1[2]/np.timedelta64(1, 'D')
TypeError: ufunc true_divide cannot use operands with types dtype('O') and dtype('<m8[D]')
```

@kawochen I c.

yes, I think that `NaT` should have these methods (`__div__/__floordiv__/__truediv__/__mul__`), and either raise `TypeError` (e.g. if the rhs is a `datetime64[ns]/Timestamp`) or return a `NaT` if its a `timedelta64[ns]/Timedelta` 

```
In [17]: s1[2]/np.timedelta64(1,'s')
TypeError: ufunc divide cannot use operands with types dtype('O') and dtype('<m8[s]')

In [18]: s1[2]/pd.Timedelta(np.timedelta64(1,'s'))
Out[18]: NaT
```

I think what happens now is that the reversed methods are called and for example `Timedelta` knows how to handle `NaT` so it works, but of course `timedeltat64[ns]` has no clue.

Addtl cases which should all be the same (e.g. should all raise a `TypeError`). I think [20] is wrong. We don't allow division for `Timestamps` at all. But this is incorrectly converted to `Timedelta` division. 

Ideally also would have more uniform / better error messages (e.g. [21]) is hard to understand and [22] is deferring to numpy and even more tricky to understand.

```
In [20]: Timestamp('nat')/Timedelta('1s')
Out[20]: NaT

In [21]: Timestamp('nat')/Timedelta('nat')
TypeError: unsupported operand type(s) for /: 'NaTType' and 'NaTType'

In [22]: Timestamp('nat')/np.timedelta64(1,'s')
TypeError: ufunc divide cannot use operands with types dtype('O') and dtype('<m8[s]')
```

Now [20] actually may be impossible to fix as we have the **same** `NaT` for both `Timestamp/Timedelta`.

@kawochen lmk progress on this.

@kawochen can you update

will update soon.  but I think `__mul__` should never succeed, and the divisions should return `nan` if `Timedelta`/`timedelta64`?

@kawochen these interactions are already defined in `core/ops.py/_Timeop`, which handle Series interactions. So these should all be uniform (prob best to look at the tests for this, mostly in `tseries/tests_timeseries.py` and `test_series.py`

seems to depend on whether `NaT` is the divisor or the dividend.

```
#timedelta/NaT -> float
In [19]: pandas.Series(pandas.Timedelta('0 days 00:00:04'))/pandas.NaT
Out[19]:
0   NaN
dtype: float64

#NaT/Timedelta -> NaT
In [20]: pandas.NaT/pandas.Series(pandas.Timedelta('0 days 00:00:04'))
Out[20]:
0   NaT
dtype: timedelta64[ns]

#Timedelta/Timedelta -> float
In [25]: pandas.Series(pandas.Timedelta('0 days 00:00:08'))/pandas.Series(pandas.Timedelta('0 days 00:00:04'))
Out[25]:
0    2
dtype: float64

#Timedelta/NaT -> float
In [26]: pandas.Series(pandas.Timedelta('0 days 00:00:04'))/pandas.Series(pandas.NaT)
Out[26]:
0   NaN
dtype: float64

In [27]: pandas.Series(pandas.NaT) / pandas.Timedelta('0 days 00:00:02')
TypeError: can only operate on a datetime with a rhs of a timedelta/DateOffset for addition and subtraction, but the operator [__truediv__]
was passed 
```

Maybe this is better, since a `Timedelta` divided by a `Timedelta` is a `float` now.

| / | `Timedelta` | `Timestamp` | `NaT` |
| --- | --- | --- | --- |
| `Timedelta` | `float` | `TypeError` | `nan` |
| `Timestamp` | `TypeError` | `TypeError` | `TypeError` |
| `NaT` | `nan` | `TypeError` | ? |

Since `NaT` plays the dual role of `Timestamp` null and `Timedelta` null, it's not clear whether `NaT/NaT` should error or not.

`NaT / NaT` should prob be `nan` or ok with error as well. its a bit inconsitent but not should can avoid that.

OK.  How about these? Should all of them be `NaN`?

```
#all current behavior
In [83]: Timedelta('1s')/NaT
Out[83]: NaT

In [84]: NaT/Timedelta('1s')
Out[84]: NaT

In [85]: Series([Timedelta('1s')])/Series([NaT])
Out[85]:
0   NaN
dtype: float64

In [86]: Series([NaT])/Series([Timedelta('1s')])
TypeError: ...
```

So if we have timedelta/timedelta it returns a float, otherwise tries to preserve type.

```
In [2]: pd.Timedelta('1s')/10
Out[2]: Timedelta('0 days 00:00:00.100000')

In [3]: pd.Timedelta('1s')/np.nan
Out[3]: NaT

In [5]: Timedelta('1s')/Timedelta('2s')
Out[5]: 0.5
```

I think would be ok to make [3] return `np.nan` though (and all of your examples). Generally if that is intermixed with other `Timedeltas` it would be eventually coerced as well, e.g.

```
In [18]: pandas.Series([pandas.Timedelta('1s')])/2
Out[18]: 
0   00:00:00.500000
dtype: timedelta64[ns]

In [19]: pandas.Series([pandas.Timedelta('1s')])/2.0
TypeError: ...
```

Should `[19]` work?  

Should `Series([pd.NaT])*2` , `Series([pd.NaT])*2.0`, `Series([pd.NaT])/2.0` work? (I think they should, since the analogous `numpy` ops work)

yes in all

OK.  Lots to change :smile: 

Just realized `pd.Timedelta('1s')/2.0` doesn't work.  Should it?

yep suppose that dividing by floats should also work 

So a naked `pd.NaT` or `np.nan` is always the same as `Series([pd.NaT], dtype='timedelta64[ns]')`?

@kawochen can you add a whatsnew note detailing any changes in the API here (might need a sub-section).

@sinhrks can you have a look

updated

@kawochen really just some minor comments. looks really good! lots of edge cases you figured out!

@kawochen can you check this:

This is on master

```
In [12]: s2
Out[12]:
0   1 days
1   2 days
2   3 days
dtype: timedelta64[ns]

In [9]: pd.Timestamp('2012-01-01') + s2
Out[9]:
0   2012-01-02
1   2012-01-03
2   2012-01-04
dtype: datetime64[ns]

In [10]: pd.Timestamp('2012-01-01') - s2
TypeError: can only operate on a timedelta/DateOffset and a datetime for addition, but the operator [__rsub__] was passed

In [11]: pd.Timestamp('2012-01-01') + (-s2)
Out[11]:
0   2011-12-31
1   2011-12-30
2   2011-12-29
dtype: datetime64[ns]
```

this last fails, xref https://github.com/pydata/pandas/issues/11925

updated

@kawochen very minor comment. ping when pushed.

@kawochen awesome fixes! This really cleaned lots of edge cases.

keep em coming!

http://pandas-docs.github.io/pandas-docs-travis/whatsnew.html#nat-and-timedelta-operations

There is this naked exception.

```
In [3]: pd.Timestamp('1990315') + pd.Timestamp('19900315')
ValueError:
```

can you raise an informative message here?

yes sorry I made a mistake in the example.  will get to it tonight.