Repository: pandas-dev/pandas

Issue:
BUG: tz-aware DatetimeIndex + array(timedelta) gives incorrect result
#### Code Sample, a copy-pastable example if possible

```python

In [1]: import numpy as np, pandas as pd

In [2]: dt = pd.DatetimeIndex([pd.Timestamp("2017/01/01")], tz='US/Eastern')  # dtype=datetime64[ns, US/Eastern]

In [3]: td_np = np.array([np.timedelta64(1, 'ns')])  # dtype="timedelta64[ns]"

In [4]: (dt + td_np).values  # Bad, "applies" TZ twice to get 10:00 instead of 05:00!
Out[4]: array(['2017-01-01T10:00:00.000000001'], dtype='datetime64[ns]')

In [5]: (dt + td_np[0]).values
Out[5]: array(['2017-01-01T05:00:00.000000001'], dtype='datetime64[ns]')

In [6]: (dt + pd.Series(td_np)).values
Out[6]: array(['2017-01-01T05:00:00.000000001'], dtype='datetime64[ns]')

In [7]: (dt.tz_convert(None) + td_np).values
Out[7]: array(['2017-01-01T05:00:00.000000001'], dtype='datetime64[ns]')

In [8]: (dt.values + td_np)
Out[8]: array(['2017-01-01T05:00:00.000000001'], dtype='datetime64[ns]')

In [9]: (dt + pd.TimedeltaIndex(td_np))
TypeError: data type not understood
```
#### Problem description

(Above is run on 0.20.3) When adding a tz-aware `DatetimeIndex` and a numpy array of timedeltas, the result incorrectly has the timezone applied twice. There are several related issues on tz-aware `DatetimeIndex` and `timedelta` sums on the issue tracker, for example #14022. However, as far as I've read, those issues appear to have been mostly resolved as of ~0.19; this case just slipped through the cracks. I haven't found any other variants of this bug, except maybe that tz-aware `DatetimeIndex + TimedeltaIndex` raises an exception.

(Just for context, I am not relying on this behavior, just happened to come across it while updating some tests to use pandas 0.20.3, but it could lead to someone silently getting incorrect results.)

I think all that needs to be fixed here is to add another case in `pd.DatetimeIndex[OpsMixin].__add__` to handle any array-like with `dtype=timedelta64[..]`. (Maybe there's a more general upstream solution?) If someone confirms I can submit a patch.

#### Expected Output
```python

In [4]: (dt + td_np).values
Out[5]: array(['2017-01-01T05:00:00.000000001'], dtype='datetime64[ns]')
```

#### Output of ``pd.show_versions()``

<details>
INSTALLED VERSIONS
------------------
commit: None
python: 2.7.6.final.0
python-bits: 64
OS: Linux
OS-release: 4.4.75-el6.x86_64.lime.1
machine: x86_64
processor: x86_64
byteorder: little
LC_ALL: None
LANG: en_US.UTF-8
LOCALE: None.None

pandas: 0.20.3
pytest: 2.9.1
pip: 7.1.0
setuptools: 19.4
Cython: 0.24
numpy: 1.13.1
scipy: 0.17.0
xarray: None
IPython: 5.1.0
sphinx: 1.4.5
patsy: 0.4.1
dateutil: 2.2
pytz: 2015.2
blosc: None
bottleneck: None
tables: None
numexpr: None
feather: None
matplotlib: 1.5.3
openpyxl: 1.8.6
xlrd: 0.9.3
xlwt: None
xlsxwriter: None
lxml: 3.6.0
bs4: 4.4.0
html5lib: None
sqlalchemy: 1.0.12
pymysql: 0.6.3.None
psycopg2: None
jinja2: 2.9.6
s3fs: None
pandas_gbq: None
pandas_datareader: None
</details>



Discussion:
This does look buggy. Though looking at the result in numpy space is pretty useless. Numpy does really odd things with tz, IOW it display in *local* tz (so this makes it really hard to read your example).

```
In [18]: dt
Out[18]: DatetimeIndex(['2017-01-01 00:00:00-05:00'], dtype='datetime64[ns, US/Eastern]', freq=None)

In [19]: dt + td_np[0]
Out[19]: DatetimeIndex(['2017-01-01 00:00:00.000000001-05:00'], dtype='datetime64[ns, US/Eastern]', freq=None)

In [20]: dt + td_np
Out[20]: DatetimeIndex(['2017-01-01 05:00:00.000000001-05:00'], dtype='datetime64[ns, US/Eastern]', freq=None)
```

[19] and [20] should be the same

Further, generally it is a good idea to be very explict, IOW

```
In [21]: dt = pd.DatetimeIndex([pd.Timestamp("2017/01/01")]).tz_localize('US/Eastern')

In [22]: dt
Out[22]: DatetimeIndex(['2017-01-01 00:00:00-05:00'], dtype='datetime64[ns, US/Eastern]', freq=None)
```

This is way more explicit that we are localizing.

But this is a small bug, happy to have a PR to fix.

Resolving patch:
diff --git a/pandas/core/indexes/datetimelike.py b/pandas/core/indexes/datetimelike.py
--- a/pandas/core/indexes/datetimelike.py
+++ b/pandas/core/indexes/datetimelike.py
@@ -14,7 +14,7 @@
     is_integer, is_float,
     is_bool_dtype, _ensure_int64,
     is_scalar, is_dtype_equal,
-    is_list_like)
+    is_list_like, is_timedelta64_dtype)
 from pandas.core.dtypes.generic import (
     ABCIndex, ABCSeries,
     ABCPeriodIndex, ABCIndexClass)
@@ -651,14 +651,14 @@ def __add__(self, other):
             from pandas.core.index import Index
             from pandas.core.indexes.timedeltas import TimedeltaIndex
             from pandas.tseries.offsets import DateOffset
-            if isinstance(other, TimedeltaIndex):
+            if is_timedelta64_dtype(other):
                 return self._add_delta(other)
             elif isinstance(self, TimedeltaIndex) and isinstance(other, Index):
                 if hasattr(other, '_add_delta'):
                     return other._add_delta(self)
                 raise TypeError("cannot add TimedeltaIndex and {typ}"
                                 .format(typ=type(other)))
-            elif isinstance(other, (DateOffset, timedelta, np.timedelta64)):
+            elif isinstance(other, (DateOffset, timedelta)):
                 return self._add_delta(other)
             elif is_integer(other):
                 return self.shift(other)
@@ -674,7 +674,7 @@ def __sub__(self, other):
             from pandas.core.indexes.datetimes import DatetimeIndex
             from pandas.core.indexes.timedeltas import TimedeltaIndex
             from pandas.tseries.offsets import DateOffset
-            if isinstance(other, TimedeltaIndex):
+            if is_timedelta64_dtype(other):
                 return self._add_delta(-other)
             elif isinstance(self, TimedeltaIndex) and isinstance(other, Index):
                 if not isinstance(other, TimedeltaIndex):
@@ -687,7 +687,7 @@ def __sub__(self, other):
                 raise TypeError("cannot subtract {typ1} and {typ2}"
                                 .format(typ1=type(self).__name__,
                                         typ2=type(other).__name__))
-            elif isinstance(other, (DateOffset, timedelta, np.timedelta64)):
+            elif isinstance(other, (DateOffset, timedelta)):
                 return self._add_delta(-other)
             elif is_integer(other):
                 return self.shift(-other)
@@ -736,7 +736,7 @@ def _add_delta_tdi(self, other):
         if self.hasnans or other.hasnans:
             mask = (self._isnan) | (other._isnan)
             new_values[mask] = iNaT
-        return new_values.view(self.dtype)
+        return new_values.view('i8')
 
     def isin(self, values):
         """
diff --git a/pandas/core/indexes/datetimes.py b/pandas/core/indexes/datetimes.py
--- a/pandas/core/indexes/datetimes.py
+++ b/pandas/core/indexes/datetimes.py
@@ -13,6 +13,7 @@
     _NS_DTYPE, _INT64_DTYPE,
     is_object_dtype, is_datetime64_dtype,
     is_datetimetz, is_dtype_equal,
+    is_timedelta64_dtype,
     is_integer, is_float,
     is_integer_dtype,
     is_datetime64_ns_dtype,
@@ -858,10 +859,13 @@ def _add_delta(self, delta):
 
         if isinstance(delta, (Tick, timedelta, np.timedelta64)):
             new_values = self._add_delta_td(delta)
-        elif isinstance(delta, TimedeltaIndex):
+        elif is_timedelta64_dtype(delta):
+            if not isinstance(delta, TimedeltaIndex):
+                delta = TimedeltaIndex(delta)
+            else:
+                # update name when delta is Index
+                name = com._maybe_match_name(self, delta)
             new_values = self._add_delta_tdi(delta)
-            # update name when delta is Index
-            name = com._maybe_match_name(self, delta)
         elif isinstance(delta, DateOffset):
             new_values = self._add_offset(delta).asi8
         else:


Test patch:
diff --git a/pandas/tests/indexes/datetimes/test_arithmetic.py b/pandas/tests/indexes/datetimes/test_arithmetic.py
--- a/pandas/tests/indexes/datetimes/test_arithmetic.py
+++ b/pandas/tests/indexes/datetimes/test_arithmetic.py
@@ -121,6 +121,99 @@ def test_dti_isub_timedeltalike(self, tz, delta):
         rng -= delta
         tm.assert_index_equal(rng, expected)
 
+    # -------------------------------------------------------------
+    # Binary operations DatetimeIndex and TimedeltaIndex/array
+    def test_dti_add_tdi(self, tz):
+        # GH 17558
+        dti = DatetimeIndex([Timestamp('2017-01-01', tz=tz)] * 10)
+        tdi = pd.timedelta_range('0 days', periods=10)
+        expected = pd.date_range('2017-01-01', periods=10, tz=tz)
+
+        # add with TimdeltaIndex
+        result = dti + tdi
+        tm.assert_index_equal(result, expected)
+
+        result = tdi + dti
+        tm.assert_index_equal(result, expected)
+
+        # add with timedelta64 array
+        result = dti + tdi.values
+        tm.assert_index_equal(result, expected)
+
+        result = tdi.values + dti
+        tm.assert_index_equal(result, expected)
+
+    def test_dti_iadd_tdi(self, tz):
+        # GH 17558
+        dti = DatetimeIndex([Timestamp('2017-01-01', tz=tz)] * 10)
+        tdi = pd.timedelta_range('0 days', periods=10)
+        expected = pd.date_range('2017-01-01', periods=10, tz=tz)
+
+        # iadd with TimdeltaIndex
+        result = DatetimeIndex([Timestamp('2017-01-01', tz=tz)] * 10)
+        result += tdi
+        tm.assert_index_equal(result, expected)
+
+        result = pd.timedelta_range('0 days', periods=10)
+        result += dti
+        tm.assert_index_equal(result, expected)
+
+        # iadd with timedelta64 array
+        result = DatetimeIndex([Timestamp('2017-01-01', tz=tz)] * 10)
+        result += tdi.values
+        tm.assert_index_equal(result, expected)
+
+        result = pd.timedelta_range('0 days', periods=10)
+        result += dti
+        tm.assert_index_equal(result, expected)
+
+    def test_dti_sub_tdi(self, tz):
+        # GH 17558
+        dti = DatetimeIndex([Timestamp('2017-01-01', tz=tz)] * 10)
+        tdi = pd.timedelta_range('0 days', periods=10)
+        expected = pd.date_range('2017-01-01', periods=10, tz=tz, freq='-1D')
+
+        # sub with TimedeltaIndex
+        result = dti - tdi
+        tm.assert_index_equal(result, expected)
+
+        msg = 'cannot subtract TimedeltaIndex and DatetimeIndex'
+        with tm.assert_raises_regex(TypeError, msg):
+            tdi - dti
+
+        # sub with timedelta64 array
+        result = dti - tdi.values
+        tm.assert_index_equal(result, expected)
+
+        msg = 'cannot perform __neg__ with this index type:'
+        with tm.assert_raises_regex(TypeError, msg):
+            tdi.values - dti
+
+    def test_dti_isub_tdi(self, tz):
+        # GH 17558
+        dti = DatetimeIndex([Timestamp('2017-01-01', tz=tz)] * 10)
+        tdi = pd.timedelta_range('0 days', periods=10)
+        expected = pd.date_range('2017-01-01', periods=10, tz=tz, freq='-1D')
+
+        # isub with TimedeltaIndex
+        result = DatetimeIndex([Timestamp('2017-01-01', tz=tz)] * 10)
+        result -= tdi
+        tm.assert_index_equal(result, expected)
+
+        msg = 'cannot subtract TimedeltaIndex and DatetimeIndex'
+        with tm.assert_raises_regex(TypeError, msg):
+            tdi -= dti
+
+        # isub with timedelta64 array
+        result = DatetimeIndex([Timestamp('2017-01-01', tz=tz)] * 10)
+        result -= tdi.values
+        tm.assert_index_equal(result, expected)
+
+        msg = '|'.join(['cannot perform __neg__ with this index type:',
+                        'ufunc subtract cannot use operands with types'])
+        with tm.assert_raises_regex(TypeError, msg):
+            tdi.values -= dti
+
     # -------------------------------------------------------------
     # Binary Operations DatetimeIndex and datetime-like
     # TODO: A couple other tests belong in this section.  Move them in