Repository: pandas-dev/pandas

Issue:
Numpy dtype weirdness in TimedeltaIndex._simple_new
I was poking around in tdi.py with pdb and I noticed this code at [line 255 of current master](https://github.com/pydata/pandas/blob/0efd4b3c8d2b0308441a6960327a2447bd419a1b/pandas/tseries/tdi.py#L255):

``` python
    @classmethod
    def _simple_new(cls, values, name=None, freq=None, **kwargs):
-->     if not getattr(values,'dtype',None):   # This doesn't do what it looks like it does
            values = np.array(values,copy=False)
        if values.dtype == np.object_:
            values = tslib.array_to_timedelta64(values)
        if values.dtype != _TD_DTYPE:
            values = com._ensure_int64(values).view(_TD_DTYPE)

        result = object.__new__(cls)
        result._data = values
        result.name = name
        result.freq = freq
        result._reset_identity()
        return result
```

This looks to me like it's testing whether `values` is already an `ndarray`, and if it isn't, it's taking a view into its values.  But that's not what it's doing at all.  If `values` is a `Series`, it has a dtype, so the view-making code should be skipped!  Except that it isn't:

``` python
In [1]: import numpy as np

In [2]: map(np.dtype, [None, 'i8', 'i4,i4', ('m8[ns]', (2, 2)), ('S', 4),
   ...:                [('name', 'u1', (2, 2))],
   ...:                {'names': ['name'], 'formats': ['f8']},
   ...:                ('i4,i4', (2, 2))])
Out[2]: 
[dtype('float64'),                                # note that dtype(None) is float!
 dtype('int64'),                                  # fixed dtype
 dtype([('f0', '<i4'), ('f1', '<i4')]),           # record dtype
 dtype(('<m8[ns]', (2, 2))),                      # array dtype
 dtype('S4'),                                     # flexible dtype
 dtype([('name', 'u1', (2, 2))]),                 # record dtype with array fields
 dtype([('name', '<f8')]),                        # another record dtype
 dtype(([('f0', '<i4'), ('f1', '<i4')], (2, 2)))] # array dtype with record fields

In [3]: map(bool, _)
Out[3]: [False, False, True, False, False, True, True, False]   # madness
```

My tentative guess is that if the outermost layer of the dtype is a record type, then it is `True`y, and otherwise it's `False`y.  This behavior seems too pathological to depend on.

Unfortunately, that's exactly what's happening when constructing a `TimedeltaIndex` from a `Series` of `Timedelta`s:

``` python
In [1]: import pandas as pd

In [2]: import pdb

In [3]: pd.Series([pd.Timedelta(1, 'h')])
Out[3]: 
0   01:00:00
dtype: timedelta64[ns]

In [4]: pdb.run('pd.Index(_)')
> <string>(1)<module>()
(Pdb) b pd.TimedeltaIndex._simple_new
Breakpoint 1 at /Users/afni/ActivityMonitorProcessing/sojourns/source/lib/python2.7/site-packages/pandas/tseries/tdi.py:253
(Pdb) c
> /Users/afni/ActivityMonitorProcessing/sojourns/source/lib/python2.7/site-packages/pandas/tseries/tdi.py(255)_simple_new()
-> if not getattr(values,'dtype',None):
(Pdb) p values
0   01:00:00
dtype: timedelta64[ns]   # it's still a Series
(Pdb) n
> /Users/afni/ActivityMonitorProcessing/sojourns/source/lib/python2.7/site-packages/pandas/tseries/tdi.py(256)_simple_new()
-> values = np.array(values,copy=False)
(Pdb) n
> /Users/afni/ActivityMonitorProcessing/sojourns/source/lib/python2.7/site-packages/pandas/tseries/tdi.py(257)_simple_new()
-> if values.dtype == np.object_:
(Pdb) p values
array([3600000000000], dtype='timedelta64[ns]')   # now it's an ndarray
```

Everything seems to _work_, but what's actually going on is so different from what I would expect to be going on that it seems like a time bomb.



Discussion:
this is just to make sure it's' m8[ns] dtyped as the data has to be s view (and not actual objects)
this is only if an index is constructed and is not hit with a series 

so not sure what your issue is about

the np.array(,...) coerces the series to its base numpy repr

So `not getattr(values,'dtype',None)` is expected to always be `True`?

this is an internal routine
record types are meaningless here
this is only a simple dtype or scalar iirc

> record types are meaningless here

...which means that `bool(dtype)` is always `False`, because numpy is nuts.

> this is only a simple dtype or scalar iirc

The pdb session in the OP shows that calling `Index` on a `Series` of `Timedelta`s ends up calling this internal routine with the input `Series` as the values.

It looks like `copy=False` is a problem too:

``` python
In [1]: import numpy as np

In [2]: import pandas as pd

In [3]: idx = pd.Index([pd.Timedelta(1, 'h')])

In [4]: idx[0] *= 2
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-4-7a30aa64d2ec> in <module>()
----> 1 idx[0] *= 2

/Users/afni/ActivityMonitorProcessing/sojourns/source/lib/python2.7/site-packages/pandas/core/index.pyc in __setitem__(self, key, value)
    894 
    895     def __setitem__(self, key, value):
--> 896         raise TypeError("Indexes does not support mutable operations")
    897 
    898     def __getitem__(self, key):

TypeError: Indexes does not support mutable operations

In [5]: val = idx.values

In [6]: idx2 = pd.Index(val)

In [7]: val
Out[7]: array([3600000000000], dtype='timedelta64[ns]')

In [8]: idx
Out[8]: 
<class 'pandas.tseries.tdi.TimedeltaIndex'>
['01:00:00']
Length: 1, Freq: None

In [9]: idx2
Out[9]: 
<class 'pandas.tseries.tdi.TimedeltaIndex'>
['01:00:00']
Length: 1, Freq: None

In [10]: val[:] *= 2

In [11]: idx
Out[11]: 
<class 'pandas.tseries.tdi.TimedeltaIndex'>
['02:00:00']   # This was supposed to be immutable!
Length: 1, Freq: None

In [12]: idx2
Out[12]: 
<class 'pandas.tseries.tdi.TimedeltaIndex'>
['02:00:00']
Length: 1, Freq: None
```

The `Index` class is not meant to prevent what you are doing here. If the user grabs an internal accessor (`.values`) and then does stuff with it nothing can be done. In fact this property is used internally. It doesn't support mutable ops thru its accessors.

I am happy that you are looking at this class. Feel free to propose some fixes. I did rip off some code from `tseries/index.py` (so some of the same idioms are prob there). 

There are certainly some edge cases that are not tested / caught.


Resolving patch:
diff --git a/pandas/core/arrays/timedeltas.py b/pandas/core/arrays/timedeltas.py
--- a/pandas/core/arrays/timedeltas.py
+++ b/pandas/core/arrays/timedeltas.py
@@ -11,7 +11,7 @@
 from pandas import compat
 
 from pandas.core.dtypes.common import (
-    _TD_DTYPE, ensure_int64, is_timedelta64_dtype, is_list_like)
+    _TD_DTYPE, is_list_like)
 from pandas.core.dtypes.generic import ABCSeries
 from pandas.core.dtypes.missing import isna
 
@@ -111,16 +111,16 @@ def dtype(self):
     _attributes = ["freq"]
 
     @classmethod
-    def _simple_new(cls, values, freq=None, **kwargs):
-        values = np.array(values, copy=False)
-        if values.dtype == np.object_:
-            values = array_to_timedelta64(values)
-        if values.dtype != _TD_DTYPE:
-            if is_timedelta64_dtype(values):
-                # non-nano unit
-                values = values.astype(_TD_DTYPE)
-            else:
-                values = ensure_int64(values).view(_TD_DTYPE)
+    def _simple_new(cls, values, freq=None, dtype=_TD_DTYPE):
+        # `dtype` is passed by _shallow_copy in corner cases, should always
+        #  be timedelta64[ns] if present
+        assert dtype == _TD_DTYPE
+        assert isinstance(values, np.ndarray), type(values)
+
+        if values.dtype == 'i8':
+            values = values.view('m8[ns]')
+
+        assert values.dtype == 'm8[ns]'
 
         result = object.__new__(cls)
         result._data = values
@@ -131,6 +131,10 @@ def __new__(cls, values, freq=None):
 
         freq, freq_infer = dtl.maybe_infer_freq(freq)
 
+        values = np.array(values, copy=False)
+        if values.dtype == np.object_:
+            values = array_to_timedelta64(values)
+
         result = cls._simple_new(values, freq=freq)
         if freq_infer:
             inferred = result.inferred_freq
@@ -166,17 +170,15 @@ def _generate_range(cls, start, end, periods, freq, closed=None):
 
         if freq is not None:
             index = _generate_regular_range(start, end, periods, freq)
-            index = cls._simple_new(index, freq=freq)
         else:
             index = np.linspace(start.value, end.value, periods).astype('i8')
-            index = cls._simple_new(index, freq=freq)
 
         if not left_closed:
             index = index[1:]
         if not right_closed:
             index = index[:-1]
 
-        return index
+        return cls._simple_new(index, freq=freq)
 
     # ----------------------------------------------------------------
     # Arithmetic Methods
diff --git a/pandas/core/indexes/timedeltas.py b/pandas/core/indexes/timedeltas.py
--- a/pandas/core/indexes/timedeltas.py
+++ b/pandas/core/indexes/timedeltas.py
@@ -35,6 +35,7 @@
     to_timedelta, _coerce_scalar_to_timedelta_type)
 from pandas._libs import (lib, index as libindex,
                           join as libjoin, Timedelta, NaT)
+from pandas._libs.tslibs.timedeltas import array_to_timedelta64
 
 
 class TimedeltaIndex(TimedeltaArrayMixin, DatetimeIndexOpsMixin,
@@ -166,6 +167,19 @@ def __new__(cls, data=None, unit=None, freq=None, start=None, end=None,
         elif copy:
             data = np.array(data, copy=True)
 
+        data = np.array(data, copy=False)
+        if data.dtype == np.object_:
+            data = array_to_timedelta64(data)
+        if data.dtype != _TD_DTYPE:
+            if is_timedelta64_dtype(data):
+                # non-nano unit
+                # TODO: watch out for overflows
+                data = data.astype(_TD_DTYPE)
+            else:
+                data = ensure_int64(data).view(_TD_DTYPE)
+
+        assert data.dtype == 'm8[ns]', data.dtype
+
         subarr = cls._simple_new(data, name=name, freq=freq)
         # check that we are matching freqs
         if verify_integrity and len(subarr) > 0:
@@ -180,12 +194,23 @@ def __new__(cls, data=None, unit=None, freq=None, start=None, end=None,
         return subarr
 
     @classmethod
-    def _simple_new(cls, values, name=None, freq=None, **kwargs):
-        result = super(TimedeltaIndex, cls)._simple_new(values, freq, **kwargs)
+    def _simple_new(cls, values, name=None, freq=None, dtype=_TD_DTYPE):
+        # `dtype` is passed by _shallow_copy in corner cases, should always
+        #  be timedelta64[ns] if present
+        assert dtype == _TD_DTYPE
+
+        assert isinstance(values, np.ndarray), type(values)
+        if values.dtype == 'i8':
+            values = values.view('m8[ns]')
+        assert values.dtype == 'm8[ns]', values.dtype
+
+        result = super(TimedeltaIndex, cls)._simple_new(values, freq)
         result.name = name
         result._reset_identity()
         return result
 
+    _shallow_copy = Index._shallow_copy
+
     @property
     def _formatter_func(self):
         from pandas.io.formats.format import _get_format_timedelta64


Test patch: