Repository: pandas-dev/pandas
Pull request:  BUG: Unwanted conversion from timedelta to float (#18493)
- [ ] closes #18493
- [ ] tests added / passed
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry

Problem statement:
Unwanted conversion from timedelta to float (but not datetime)
In the code below, masked assignment of a timedelta series converts the series values to float64. The same code running against a datetime correctly keeps the series dtype.

    In [1]: import pandas as pd

    In [2]: dseries=pd.Series(pd.date_range('1/1/2011', periods=5, freq='H')); dseries=dseries-dseries[0]; dseries
    Out[2]: 
    0   00:00:00
    1   01:00:00
    2   02:00:00
    3   03:00:00
    4   04:00:00
    dtype: timedelta64[ns]

    In [3]: dseries[dseries==dseries[0]]=None; dseries
    Out[3]: 
    0   -9.223372e+18
    1    3.600000e+12
    2    7.200000e+12
    3    1.080000e+13
    4    1.440000e+13
    dtype: float64

    In [4]: dseries=pd.Series(pd.date_range('1/1/2011', periods=5, freq='H')); dseries=dseries; dseries
    Out[4]: 
    0   2011-01-01 00:00:00
    1   2011-01-01 01:00:00
    2   2011-01-01 02:00:00
    3   2011-01-01 03:00:00
    4   2011-01-01 04:00:00
    dtype: datetime64[ns]

    In [5]: dseries[dseries==dseries[0]]=None; dseries
    Out[5]: 
    0                   NaT
    1   2011-01-01 01:00:00
    2   2011-01-01 02:00:00
    3   2011-01-01 03:00:00
    4   2011-01-01 04:00:00
    dtype: datetime64[ns]

Hints from issue discussion:
show your version of pandas. 
this was broken in master as well.

This will patch, can you do a PR with appropriate tests?
```
(pandas) bash-3.2$ git diff
diff --git a/pandas/core/internals.py b/pandas/core/internals.py
index e537cb2..7a5877d 100644
--- a/pandas/core/internals.py
+++ b/pandas/core/internals.py
@@ -1956,7 +1956,8 @@ class TimeDeltaBlock(DatetimeLikeBlockMixin, IntBlock):
         tipo = maybe_infer_dtype_type(element)
         if tipo is not None:
             return issubclass(tipo.type, np.timedelta64)
-        return isinstance(element, (timedelta, np.timedelta64))
+        return is_integer(element) or isinstance(
+            element, (timedelta, np.timedelta64))
 
     def fillna(self, value, **kwargs):
``` 

tests can go
``pandas/tests/indexing/test_timedelta.py``
add tests setting with each of ``[None, np.nan, pd.NaT]``

Thanks for the fast response! I'm not sure I'll be able to get a dev environment up and running soon. There are various versioning conflicts with Cython that make this more than a five minute job. :-(
> There are various versioning conflicts with Cython that make this more than a five minute job. :-(

really?

http://pandas.pydata.org/pandas-docs/stable/contributing.html#

```
(pandas) bash-3.2$ conda create -n foo --file ci/requirements_dev.txt  --dry
Fetching package metadata .............
Solving package specifications: .

Package plan for installation in environment /Users/jreback/miniconda3/envs/foo:

The following NEW packages will be INSTALLED:

    alabaster:                0.7.10-py35hb692fe1_0    defaults   
    asn1crypto:               0.22.0-py35ha010153_1    defaults   
    babel:                    2.5.0-py35h6f05a75_0     defaults   
    boto:                     2.48.0-py35h10f7326_1    defaults   
    boto3:                    1.4.7-py35hd462e4d_0     defaults   
    botocore:                 1.7.20-py35h01c6a2e_0    defaults   
    ca-certificates:          2017.08.26-ha1e5d58_0    defaults   
    certifi:                  2017.11.5-py35hd00889a_0 defaults   
    cffi:                     1.10.0-py35hca2225f_1    defaults   
    chardet:                  3.0.4-py35h16a84c2_1     defaults   
    click:                    6.7-py35hcc65ea6_0       defaults   
    cookies:                  2.2.1-py35_0             defaults   
    cryptography:             2.0.3-py35hde935b2_1     defaults   
....
```
Hey guys!

I think the selection and the comparison which is done like ...
> dseries[dseries==dseries[0]]

... produces a unwanted transformation of the series dtype. A good way to transform the series back is the from float or int is the following way:
> pd.to_timedelta(dseries[dseries < '30 sec'])

I'm not sure if the transformation is a bug, but I hope that I could help :) 

This is also something I found:
<img width="1105" alt="screen shot 2017-11-30 at 08 44 23" src="https://user-images.githubusercontent.com/12597572/33419480-e15dd3ba-d5aa-11e7-8239-ee19ac055b82.png">

I think there's a transformation in the series which transforms a pandas.timedelta into numpy.timedelta..
@SDzinklandi the patch is above if you want to submit a PR

Commit message:  BUG: Unwanted conversion from timedelta to float (#18493)

Patch to review:
diff --git a/doc/source/whatsnew/v0.22.0.txt b/doc/source/whatsnew/v0.22.0.txt
index d43d5bec7175f..f22f5a9763e54 100644
--- a/doc/source/whatsnew/v0.22.0.txt
+++ b/doc/source/whatsnew/v0.22.0.txt
@@ -151,7 +151,7 @@ Indexing
 - Bug in :class:`IntervalIndex` where empty and purely NA data was constructed inconsistently depending on the construction method (:issue:`18421`)
 - Bug in ``IntervalIndex.symmetric_difference()`` where the symmetric difference with a non-``IntervalIndex`` did not raise (:issue:`18475`)
 - Bug in indexing a datetimelike ``Index`` that raised ``ValueError`` instead of ``IndexError`` (:issue:`18386`).
-
+- Bug in ``TimeDeltaBlock._can_hold_element()`` where masked assignment of a timedelta series converts the series values to float64 (:issue:`18493`)
 
 I/O
 ^^^
diff --git a/pandas/core/internals.py b/pandas/core/internals.py
index 4f25a19d437ca..1d1d71be16c00 100644
--- a/pandas/core/internals.py
+++ b/pandas/core/internals.py
@@ -1956,7 +1956,8 @@ def _can_hold_element(self, element):
         tipo = maybe_infer_dtype_type(element)
         if tipo is not None:
             return issubclass(tipo.type, np.timedelta64)
-        return isinstance(element, (timedelta, np.timedelta64))
+        return is_integer(element) or isinstance(
+            element, (timedelta, np.timedelta64))
 
     def fillna(self, value, **kwargs):
 
diff --git a/pandas/tests/indexing/test_timedelta.py b/pandas/tests/indexing/test_timedelta.py
index 32609362e49af..3533035520e72 100644
--- a/pandas/tests/indexing/test_timedelta.py
+++ b/pandas/tests/indexing/test_timedelta.py
@@ -2,6 +2,7 @@
 
 import pandas as pd
 from pandas.util import testing as tm
+from numpy import nan
 
 
 class TestTimedeltaIndexing(object):
@@ -47,3 +48,14 @@ def test_string_indexing(self):
         expected = df.iloc[0]
         sliced = df.loc['0 days']
         tm.assert_series_equal(sliced, expected)
+
+    @pytest.mark.parametrize(
+        "value, expected",
+        [(None, [pd.NaT, 1, 2]),
+         (pd.NaT, [pd.NaT, 1, 2]),
+         (nan, [pd.NaT, 1, 2])])
+    def test_nan(self, value, expected):
+        series = pd.Series([0, 1, 2], dtype='timedelta64[ns]')
+        series[series == series[0]] = value
+        expected = pd.Series([pd.NaT, 1, 2], dtype='timedelta64[ns]')
+        tm.assert_series_equal(series, expected)


Code review comments:
diff_hunk: @@ -2,6 +2,7 @@
 
 import pandas as pd
 from pandas.util import testing as tm
+from numpy import nan
path: pandas/tests/indexing/test_timedelta.py
text: @user1:
don't import this, use ``np.nan``

diff_hunk: @@ -47,3 +48,14 @@ def test_string_indexing(self):
         expected = df.iloc[0]
         sliced = df.loc['0 days']
         tm.assert_series_equal(sliced, expected)
+
+    @pytest.mark.parametrize(
+        "value, expected",
+        [(None, [pd.NaT, 1, 2]),
+         (pd.NaT, [pd.NaT, 1, 2]),
+         (nan, [pd.NaT, 1, 2])])
+    def test_nan(self, value, expected):
path: pandas/tests/indexing/test_timedelta.py
text: @user1:
name this like: test_masked_setitem

diff_hunk: @@ -151,7 +151,7 @@ Indexing
 - Bug in :class:`IntervalIndex` where empty and purely NA data was constructed inconsistently depending on the construction method (:issue:`18421`)
 - Bug in ``IntervalIndex.symmetric_difference()`` where the symmetric difference with a non-``IntervalIndex`` did not raise (:issue:`18475`)
 - Bug in indexing a datetimelike ``Index`` that raised ``ValueError`` instead of ``IndexError`` (:issue:`18386`).
-
+- Bug in ``TimeDeltaBlock._can_hold_element()`` where masked assignment of a timedelta series converts the series values to float64 (:issue:`18493`)
path: doc/source/whatsnew/v0.22.0.txt
text: @user1:
you can move to 0.21.1

don't use internal language here, these are user-level notes. So say something like

Bug in masked assignment of a ``timedelta64[ns]`` dtype ``Series`` with a ``Timedelta`` incorrectly coerced to float.

diff_hunk: @@ -47,3 +48,14 @@ def test_string_indexing(self):
         expected = df.iloc[0]
         sliced = df.loc['0 days']
         tm.assert_series_equal(sliced, expected)
+
+    @pytest.mark.parametrize(
+        "value, expected",
+        [(None, [pd.NaT, 1, 2]),
path: pandas/tests/indexing/test_timedelta.py
text: @user1:
you don't need to pass expected here as they are the same, just define in the body