Repository: pandas-dev/pandas

Issue:
Stripped timedelta[s] series represented as timedelta[ns]
```
In [2]: s = pd.Series(range(61)).astype('timedelta64[s]')

In [3]: s.tail(1)
Out[3]: 
60   00:01:00
dtype: timedelta64[s]

In [4]: str(s).splitlines()[-2]
Out[4]: '60   00:00:00.000000'
```

... because the snipped representation interprets the content of the series as nanoseconds rather than milliseconds. The same happens with `timedelta[ms]` and probably any other resolution (if there are).

```
In [5]: pd.show_versions()

INSTALLED VERSIONS
------------------
commit: 404819358e90da57c8025a259ab58cd75426069f
python: 3.4.3.final.0
python-bits: 64
OS: Linux
OS-release: 4.3.0-1-amd64
machine: x86_64
processor: 
byteorder: little
LC_ALL: None
LANG: it_IT.utf8

pandas: 0.18.0rc1+35.g4048193
nose: 1.3.6
pip: 1.5.6
setuptools: 20.1.1
Cython: 0.23.2
numpy: 1.10.0.post2
scipy: 0.16.0
statsmodels: 0.8.0.dev0+755fa81
xarray: None
IPython: 4.1.1
sphinx: 1.3.1
patsy: 0.3.0-dev
dateutil: 2.4.2
pytz: 2015.6
blosc: None
bottleneck: None
tables: 3.2.2
numexpr: 2.4.3
matplotlib: 1.5.dev1
openpyxl: None
xlrd: 0.9.4
xlwt: None
xlsxwriter: 0.7.3
lxml: None
bs4: 4.4.0
html5lib: 0.999
httplib2: 0.9.1
apiclient: None
sqlalchemy: 1.0.11
pymysql: None
psycopg2: None
jinja2: 2.8
```



Discussion:
The same example, but shown in another way:

```
In [27]: s = pd.Series(range(5)).astype('timedelta64[s]')

In [28]: s
Out[28]:
0   00:00:00
1   00:00:01
2   00:00:02
3   00:00:03
4   00:00:04
dtype: timedelta64[s]

In [29]: pd.options.display.max_rows = 4

In [30]: s
Out[30]:
0          00:00:00
1   00:00:00.000000
          ...
3   00:00:00.000000
4   00:00:00.000000
dtype: timedelta64[ns]
```

Possibly related with https://github.com/pydata/pandas/issues/11594

The problem actually lies in (the usage of) `concat`:

```
In [38]: pd.concat([s[0:2], s[-2:]])
Out[38]:
0          00:00:00
1   00:00:00.000000
3   00:00:00.000000
4   00:00:00.000000
dtype: timedelta64[ns]
```

well this is a different issue.

`.astype('timedelta64[s]')` is intepreting the `timedelta64[s]` units as `ns` so there needs to be a conversion done in ``core.common._possibly_convert_to_datetimelike` or maybe in the astype itself. So i'll mark this bug as one like that. Note this feature is not technically supported.

e.g. we reject things like this (so we can either reject, or better yet accept both). We do this for `datetime64`, IOW, we accept multiple dtypes (and then convert).

```
In [1]: Series(np.arange(61),dtype='m8[s]')
TypeError: cannot convert timedeltalike to dtype [timedelta64[s]]
```

Separately is the bug @jorisvandenbossche notes, #11594 which is different (and not a problem with `.concat`), rather with preserving the dtype on slicing. I'll put an example there

@jreback : ... but there is no problem in the `.tail()` (or `.iloc[-1]`, actually) version of the same `Series`. How can it be the fault of `.astype('timedelta64[s]')`?!

These are not stored correctly. We only store as `timedelta64[ns]`

```
In [1]: s = pd.Series(range(61)).astype('timedelta64[s]')

In [2]: s.values
Out[2]: 
array([ 0,  1,  2,  3,  4,  5,  6,  7,  8,  9, 10, 11, 12, 13, 14, 15, 16,
       17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33,
       34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50,
       51, 52, 53, 54, 55, 56, 57, 58, 59, 60], dtype='timedelta64[s]')

In [3]: Series(pd.to_timedelta(range(61),unit='s')).values
Out[3]: 
array([          0,  1000000000,  2000000000,  3000000000,  4000000000,
        5000000000,  6000000000,  7000000000,  8000000000,  9000000000,
       10000000000, 11000000000, 12000000000, 13000000000, 14000000000,
       15000000000, 16000000000, 17000000000, 18000000000, 19000000000,
       20000000000, 21000000000, 22000000000, 23000000000, 24000000000,
       25000000000, 26000000000, 27000000000, 28000000000, 29000000000,
       30000000000, 31000000000, 32000000000, 33000000000, 34000000000,
       35000000000, 36000000000, 37000000000, 38000000000, 39000000000,
       40000000000, 41000000000, 42000000000, 43000000000, 44000000000,
       45000000000, 46000000000, 47000000000, 48000000000, 49000000000,
       50000000000, 51000000000, 52000000000, 53000000000, 54000000000,
       55000000000, 56000000000, 57000000000, 58000000000, 59000000000,
       60000000000], dtype='timedelta64[ns]')
```

OK... but the ordinary `__repr__` is aware of this, so it behaves fine. If the snipped version also behaved well, and we just stated "all `timedelta64[*]` are stored as nanoseconds", wouldn't everybody be happy?

(that is: am I missing an official definition of `timedelta64[s]` which we should comply with?)

no the repr is wrong which is indicative of the internal represenation is wrong. This needs to be fixed in the astyping. Putting something in the docs is the very last thing to do. It is a bug and should be fixed (though technicaly this is not supported, but prob should be)

Does the following (on `datetime` rather than `timedelta`) also reflect a bug?

```
In [2]: str(pd.Series(range(100)).astype('datetime64[ms]')).splitlines()[-1]
Out[2]: 'dtype: datetime64[ns]'
```

why are you doing this stringifying thing? that doesn't make any sense.

yes, this is properly converted to `datetime64[ns]`

```
In [10]: pd.Series(range(100)).astype('datetime64[ms]').dt.microsecond.head()
Out[10]: 
0       0
1    1000
2    2000
3    3000
4    4000
dtype: int64
```

```
In [13]: pd.to_datetime(pd.Series(range(100)), unit='ms').dt.microsecond.head()
Out[13]: 
0       0
1    1000
2    2000
3    3000
4    4000
dtype: int64
```

note, no `millisecond` attribute because its not compat with `datetime`.

@jreback The stringifying is another way to show what the repr looks like. And this does makes sense in eg the case of timedelta, as the dtype in the repr did not correspond with the actual dtype of the series

But in case of datetime64, these values are stored correctly (so @toobaz what you showed in your last comment is not a bug, but the correct behaviour). When doing `astype('datetime64[s]')`, the values are correctly interpreted as seconds, and subsequently converted for `datetime64[ns]` to store it in the series:

```

In [22]: pd.Series(range(5)).astype('datetime64[s]')
Out[22]:
0   1970-01-01 00:00:00
1   1970-01-01 00:00:01
2   1970-01-01 00:00:02
3   1970-01-01 00:00:03
4   1970-01-01 00:00:04
dtype: datetime64[ns]

In [23]: pd.Series(range(5)).astype('datetime64[s]').values
Out[23]:
array(['1970-01-01T01:00:00.000000000+0100',
       '1970-01-01T01:00:01.000000000+0100',
       '1970-01-01T01:00:02.000000000+0100',
       '1970-01-01T01:00:03.000000000+0100',
       '1970-01-01T01:00:04.000000000+0100'], dtype='datetime64[ns]')
```

Probably the same should happen for `astype('timedelta[s]')`?

```
In [29]: pd.Series(range(5)).astype('timedelta64[s]')
Out[29]:
0   00:00:00
1   00:00:01
2   00:00:02
3   00:00:03
4   00:00:04
dtype: timedelta64[s]

In [30]: pd.Series(range(5)).astype('timedelta64[s]').values
Out[30]: array([0, 1, 2, 3, 4], dtype='timedelta64[s]')
```

@jorisvandenbossche exactly, that's what I noted above. `datetime64` are all coerced pretty well. it can be used as a model on how/where to do a similar coercion (which is done already for the constructor and most places, obviously not in astyping).


Resolving patch:
diff --git a/pandas/core/dtypes/cast.py b/pandas/core/dtypes/cast.py
--- a/pandas/core/dtypes/cast.py
+++ b/pandas/core/dtypes/cast.py
@@ -656,11 +656,15 @@ def astype_nansafe(arr, dtype, copy=True):
             return tslib.ints_to_pydatetime(arr.view(np.int64))
         elif dtype == np.int64:
             return arr.view(dtype)
-        elif dtype != _NS_DTYPE:
-            raise TypeError("cannot astype a datetimelike from [{from_dtype}] "
-                            "to [{to_dtype}]".format(from_dtype=arr.dtype,
-                                                     to_dtype=dtype))
-        return arr.astype(_NS_DTYPE)
+
+        # allow frequency conversions
+        if dtype.kind == 'M':
+            return arr.astype(dtype)
+
+        raise TypeError("cannot astype a datetimelike from [{from_dtype}] "
+                        "to [{to_dtype}]".format(from_dtype=arr.dtype,
+                                                 to_dtype=dtype))
+
     elif is_timedelta64_dtype(arr):
         if dtype == np.int64:
             return arr.view(dtype)
@@ -668,21 +672,23 @@ def astype_nansafe(arr, dtype, copy=True):
             return tslib.ints_to_pytimedelta(arr.view(np.int64))
 
         # in py3, timedelta64[ns] are int64
-        elif ((PY3 and dtype not in [_INT64_DTYPE, _TD_DTYPE]) or
-              (not PY3 and dtype != _TD_DTYPE)):
+        if ((PY3 and dtype not in [_INT64_DTYPE, _TD_DTYPE]) or
+                (not PY3 and dtype != _TD_DTYPE)):
 
             # allow frequency conversions
+            # we return a float here!
             if dtype.kind == 'm':
                 mask = isna(arr)
                 result = arr.astype(dtype).astype(np.float64)
                 result[mask] = np.nan
                 return result
+        elif dtype == _TD_DTYPE:
+            return arr.astype(_TD_DTYPE, copy=copy)
 
-            raise TypeError("cannot astype a timedelta from [{from_dtype}] "
-                            "to [{to_dtype}]".format(from_dtype=arr.dtype,
-                                                     to_dtype=dtype))
+        raise TypeError("cannot astype a timedelta from [{from_dtype}] "
+                        "to [{to_dtype}]".format(from_dtype=arr.dtype,
+                                                 to_dtype=dtype))
 
-        return arr.astype(_TD_DTYPE)
     elif (np.issubdtype(arr.dtype, np.floating) and
           np.issubdtype(dtype, np.integer)):
 
@@ -704,19 +710,7 @@ def astype_nansafe(arr, dtype, copy=True):
 
     if copy:
 
-        if arr.dtype == dtype:
-            return arr.copy()
-
-        # we handle datetimelikes with pandas machinery
-        # to be robust to the input type
-        elif is_datetime64_dtype(dtype):
-            from pandas import to_datetime
-            return to_datetime(arr).values
-        elif is_timedelta64_dtype(dtype):
-            from pandas import to_timedelta
-            return to_timedelta(arr).values
-
-        return arr.astype(dtype)
+        return arr.astype(dtype, copy=True)
     return arr.view(dtype)
 
 
diff --git a/pandas/core/dtypes/common.py b/pandas/core/dtypes/common.py
--- a/pandas/core/dtypes/common.py
+++ b/pandas/core/dtypes/common.py
@@ -4,6 +4,7 @@
 from pandas.compat import (string_types, text_type, binary_type,
                            PY3, PY36)
 from pandas._libs import algos, lib
+from pandas._libs.tslibs import conversion
 from .dtypes import (CategoricalDtype, CategoricalDtypeType,
                      DatetimeTZDtype, DatetimeTZDtypeType,
                      PeriodDtype, PeriodDtypeType,
@@ -21,8 +22,8 @@
                              for t in ['O', 'int8', 'uint8', 'int16', 'uint16',
                                        'int32', 'uint32', 'int64', 'uint64']])
 
-_NS_DTYPE = np.dtype('M8[ns]')
-_TD_DTYPE = np.dtype('m8[ns]')
+_NS_DTYPE = conversion.NS_DTYPE
+_TD_DTYPE = conversion.TD_DTYPE
 _INT64_DTYPE = np.dtype(np.int64)
 
 # oh the troubles to reduce import time
@@ -31,6 +32,9 @@
 _ensure_float64 = algos.ensure_float64
 _ensure_float32 = algos.ensure_float32
 
+_ensure_datetime64ns = conversion.ensure_datetime64ns
+_ensure_timedelta64ns = conversion.ensure_timedelta64ns
+
 
 def _ensure_float(arr):
     """
diff --git a/pandas/core/internals.py b/pandas/core/internals.py
--- a/pandas/core/internals.py
+++ b/pandas/core/internals.py
@@ -631,7 +631,7 @@ def _astype(self, dtype, copy=False, errors='raise', values=None,
                 values = astype_nansafe(values.ravel(), dtype, copy=True)
                 values = values.reshape(self.shape)
 
-            newb = make_block(values, placement=self.mgr_locs, dtype=dtype,
+            newb = make_block(values, placement=self.mgr_locs,
                               klass=klass)
         except:
             if errors == 'raise':
@@ -1954,6 +1954,13 @@ class TimeDeltaBlock(DatetimeLikeBlockMixin, IntBlock):
     _can_hold_na = True
     is_numeric = False
 
+    def __init__(self, values, placement, fastpath=False, **kwargs):
+        if values.dtype != _TD_DTYPE:
+            values = conversion.ensure_timedelta64ns(values)
+
+        super(TimeDeltaBlock, self).__init__(values, fastpath=True,
+                                             placement=placement, **kwargs)
+
     @property
     def _box_func(self):
         return lambda x: tslib.Timedelta(x, unit='ns')


Test patch:
diff --git a/pandas/tests/frame/test_dtypes.py b/pandas/tests/frame/test_dtypes.py
--- a/pandas/tests/frame/test_dtypes.py
+++ b/pandas/tests/frame/test_dtypes.py
@@ -640,6 +640,71 @@ def test_astype_categoricaldtype_class_raises(self, cls):
         with tm.assert_raises_regex(TypeError, xpr):
             df['A'].astype(cls)
 
+    @pytest.mark.parametrize("arr_dtype", [np.int64, np.float64])
+    @pytest.mark.parametrize("dtype", ["M8", "m8"])
+    @pytest.mark.parametrize("unit", ['ns', 'us', 'ms', 's', 'h', 'm', 'D'])
+    def test_astype_to_datetimelike_unit(self, arr_dtype, dtype, unit):
+        # tests all units from numeric origination
+        # gh-19223 / gh-12425
+        dtype = "{}[{}]".format(dtype, unit)
+        arr = np.array([[1, 2, 3]], dtype=arr_dtype)
+        df = DataFrame(arr)
+        result = df.astype(dtype)
+        expected = DataFrame(arr.astype(dtype))
+
+        tm.assert_frame_equal(result, expected)
+
+    @pytest.mark.parametrize("unit", ['ns', 'us', 'ms', 's', 'h', 'm', 'D'])
+    def test_astype_to_datetime_unit(self, unit):
+        # tests all units from datetime origination
+        # gh-19223
+        dtype = "M8[{}]".format(unit)
+        arr = np.array([[1, 2, 3]], dtype=dtype)
+        df = DataFrame(arr)
+        result = df.astype(dtype)
+        expected = DataFrame(arr.astype(dtype))
+
+        tm.assert_frame_equal(result, expected)
+
+    @pytest.mark.parametrize("unit", ['ns'])
+    def test_astype_to_timedelta_unit_ns(self, unit):
+        # preserver the timedelta conversion
+        # gh-19223
+        dtype = "m8[{}]".format(unit)
+        arr = np.array([[1, 2, 3]], dtype=dtype)
+        df = DataFrame(arr)
+        result = df.astype(dtype)
+        expected = DataFrame(arr.astype(dtype))
+
+        tm.assert_frame_equal(result, expected)
+
+    @pytest.mark.parametrize("unit", ['us', 'ms', 's', 'h', 'm', 'D'])
+    def test_astype_to_timedelta_unit(self, unit):
+        # coerce to float
+        # gh-19223
+        dtype = "m8[{}]".format(unit)
+        arr = np.array([[1, 2, 3]], dtype=dtype)
+        df = DataFrame(arr)
+        result = df.astype(dtype)
+        expected = DataFrame(df.values.astype(dtype).astype(float))
+
+        tm.assert_frame_equal(result, expected)
+
+    @pytest.mark.parametrize("unit", ['ns', 'us', 'ms', 's', 'h', 'm', 'D'])
+    def test_astype_to_incorrect_datetimelike(self, unit):
+        # trying to astype a m to a M, or vice-versa
+        # gh-19224
+        dtype = "M8[{}]".format(unit)
+        other = "m8[{}]".format(unit)
+
+        df = DataFrame(np.array([[1, 2, 3]], dtype=dtype))
+        with pytest.raises(TypeError):
+            df.astype(other)
+
+        df = DataFrame(np.array([[1, 2, 3]], dtype=other))
+        with pytest.raises(TypeError):
+            df.astype(dtype)
+
     def test_timedeltas(self):
         df = DataFrame(dict(A=Series(date_range('2012-1-1', periods=3,
                                                 freq='D')),
diff --git a/pandas/tests/reshape/merge/test_merge.py b/pandas/tests/reshape/merge/test_merge.py
--- a/pandas/tests/reshape/merge/test_merge.py
+++ b/pandas/tests/reshape/merge/test_merge.py
@@ -523,25 +523,23 @@ def test_other_datetime_unit(self):
                                columns=['entity_id', 'days'])
             tm.assert_frame_equal(result, exp)
 
-    def test_other_timedelta_unit(self):
+    @pytest.mark.parametrize("unit", ['D', 'h', 'm', 's', 'ms', 'us', 'ns'])
+    def test_other_timedelta_unit(self, unit):
         # GH 13389
         df1 = pd.DataFrame({'entity_id': [101, 102]})
         s = pd.Series([None, None], index=[101, 102], name='days')
 
-        for dtype in ['timedelta64[D]', 'timedelta64[h]', 'timedelta64[m]',
-                      'timedelta64[s]', 'timedelta64[ms]', 'timedelta64[us]',
-                      'timedelta64[ns]']:
+        dtype = "m8[{}]".format(unit)
+        df2 = s.astype(dtype).to_frame('days')
+        assert df2['days'].dtype == 'm8[ns]'
 
-            df2 = s.astype(dtype).to_frame('days')
-            assert df2['days'].dtype == dtype
-
-            result = df1.merge(df2, left_on='entity_id', right_index=True)
+        result = df1.merge(df2, left_on='entity_id', right_index=True)
 
-            exp = pd.DataFrame({'entity_id': [101, 102],
-                                'days': np.array(['nat', 'nat'],
-                                                 dtype=dtype)},
-                               columns=['entity_id', 'days'])
-            tm.assert_frame_equal(result, exp)
+        exp = pd.DataFrame({'entity_id': [101, 102],
+                            'days': np.array(['nat', 'nat'],
+                                             dtype=dtype)},
+                           columns=['entity_id', 'days'])
+        tm.assert_frame_equal(result, exp)
 
     def test_overlapping_columns_error_message(self):
         df = DataFrame({'key': [1, 2, 3],
diff --git a/pandas/tests/series/test_constructors.py b/pandas/tests/series/test_constructors.py
--- a/pandas/tests/series/test_constructors.py
+++ b/pandas/tests/series/test_constructors.py
@@ -552,10 +552,6 @@ def test_constructor_dtype_datetime64(self):
         s.iloc[0] = np.nan
         assert s.dtype == 'M8[ns]'
 
-        # invalid astypes
-        for t in ['s', 'D', 'us', 'ms']:
-            pytest.raises(TypeError, s.astype, 'M8[%s]' % t)
-
         # GH3414 related
         pytest.raises(TypeError, lambda x: Series(
             Series(dates).astype('int') / 1000000, dtype='M8[ms]'))
@@ -707,6 +703,20 @@ def test_constructor_with_datetime_tz(self):
         expected = Series(pd.DatetimeIndex(['NaT', 'NaT'], tz='US/Eastern'))
         assert_series_equal(s, expected)
 
+    @pytest.mark.parametrize("arr_dtype", [np.int64, np.float64])
+    @pytest.mark.parametrize("dtype", ["M8", "m8"])
+    @pytest.mark.parametrize("unit", ['ns', 'us', 'ms', 's', 'h', 'm', 'D'])
+    def test_construction_to_datetimelike_unit(self, arr_dtype, dtype, unit):
+        # tests all units
+        # gh-19223
+        dtype = "{}[{}]".format(dtype, unit)
+        arr = np.array([1, 2, 3], dtype=arr_dtype)
+        s = Series(arr)
+        result = s.astype(dtype)
+        expected = Series(arr.astype(dtype))
+
+        tm.assert_series_equal(result, expected)
+
     @pytest.mark.parametrize('arg',
                              ['2013-01-01 00:00:00', pd.NaT, np.nan, None])
     def test_constructor_with_naive_string_and_datetimetz_dtype(self, arg):
diff --git a/pandas/tests/series/test_operators.py b/pandas/tests/series/test_operators.py
--- a/pandas/tests/series/test_operators.py
+++ b/pandas/tests/series/test_operators.py
@@ -1649,32 +1649,26 @@ def test_invalid_ops(self):
         pytest.raises(Exception, self.objSeries.__sub__,
                       np.array(1, dtype=np.int64))
 
-    def test_timedelta64_conversions(self):
+    @pytest.mark.parametrize("m", [1, 3, 10])
+    @pytest.mark.parametrize("unit", ['D', 'h', 'm', 's', 'ms', 'us', 'ns'])
+    def test_timedelta64_conversions(self, m, unit):
+
         startdate = Series(date_range('2013-01-01', '2013-01-03'))
         enddate = Series(date_range('2013-03-01', '2013-03-03'))
 
         s1 = enddate - startdate
         s1[2] = np.nan
 
-        for m in [1, 3, 10]:
-            for unit in ['D', 'h', 'm', 's', 'ms', 'us', 'ns']:
-
-                # op
-                expected = s1.apply(lambda x: x / np.timedelta64(m, unit))
-                result = s1 / np.timedelta64(m, unit)
-                assert_series_equal(result, expected)
-
-                if m == 1 and unit != 'ns':
-
-                    # astype
-                    result = s1.astype("timedelta64[{0}]".format(unit))
-                    assert_series_equal(result, expected)
+        # op
+        expected = s1.apply(lambda x: x / np.timedelta64(m, unit))
+        result = s1 / np.timedelta64(m, unit)
+        assert_series_equal(result, expected)
 
-                # reverse op
-                expected = s1.apply(
-                    lambda x: Timedelta(np.timedelta64(m, unit)) / x)
-                result = np.timedelta64(m, unit) / s1
-                assert_series_equal(result, expected)
+        # reverse op
+        expected = s1.apply(
+            lambda x: Timedelta(np.timedelta64(m, unit)) / x)
+        result = np.timedelta64(m, unit) / s1
+        assert_series_equal(result, expected)
 
         # astype
         s = Series(date_range('20130101', periods=3))