Repository: pandas-dev/pandas
Pull request: BUG: silent overflow in DateTimeArray subtraction
- [x] closes #22492
- [x] tests added / passed
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry


Problem statement:
Incorrect timedelta computation of pd.Series datetime64[ns] if timedelta is too large
#### Problem description

Below I present two methods to calculate the time difference between two `pd.Timestamp`'s. The first way is the native python `datetime` subtraction, whereas the second way wraps things in a `pd.Series`. They should be equivalent. In example 1 the output is correct; but in example 2 the resulting timedelta is negative. This seems to be the case when the time difference  is greater than ~300 years. 

#### Code Sample
Example 1:
```python
t0 = pd.Timestamp('1723-08-23 00:00:00')
t1 = pd.Timestamp('2013-08-23 00:00:00')
print( 'Expected', (t1 - t0).total_seconds() )
print( 'Got\t', (pd.Series([t1]) - pd.Series([t0]))[0].total_seconds() )
```

Output 1:
```
Expected 9151574400.0
Got	 9151574400.0
```

Whereas Example 2:
```python
t0 = pd.Timestamp('1713-08-23 00:00:00')
t1 = pd.Timestamp('2013-08-23 00:00:00')
print( 'Expected', (t1 - t0).total_seconds() )
print( 'Got\t', (pd.Series([t1]) - pd.Series([t0]))[0].total_seconds() )
```
Output 2:
```
Expected 9467107200.0
Got	 -8979636873.709553
```


#### Output of ``pd.show_versions()``

<details>

INSTALLED VERSIONS
------------------
commit: None
python: 3.6.5.final.0
python-bits: 64
OS: Linux
OS-release: 4.4.0-130-generic
machine: x86_64
processor: x86_64
byteorder: little
LC_ALL: None
LANG: en_US.UTF-8
LOCALE: en_US.UTF-8

pandas: 0.23.4
pytest: None
pip: 18.0
setuptools: 40.2.0
Cython: None
numpy: 1.15.1
scipy: 1.1.0
pyarrow: None
xarray: None
IPython: 6.5.0
sphinx: None
patsy: None
dateutil: 2.7.2
pytz: 2018.5
blosc: None
bottleneck: None
tables: None
numexpr: None
feather: None
matplotlib: 2.2.3
openpyxl: None
xlrd: 1.1.0
xlwt: None
xlsxwriter: None
lxml: None
bs4: None
html5lib: 0.9999999
sqlalchemy: None
pymysql: None
psycopg2: None
jinja2: 2.10
s3fs: None
fastparquet: None
pandas_gbq: None
pandas_datareader: None

</details>

Hints from issue discussion:
I suspect there is some overflow happening here behind the scenes.

cc @jbrockmendel 
Yep, the issue is in `core.arrays.datetimes._sub_datelike_dti` where instead of `new_values = self_i8 - other_i8` we should be using `checked_add_with_arr`. @shengpu1126 want to try a PR?

Commit message: updated

Patch to review:
diff --git a/doc/source/whatsnew/v0.24.0.txt b/doc/source/whatsnew/v0.24.0.txt
index 3e22084d98234..c94751c8e7d9c 100644
--- a/doc/source/whatsnew/v0.24.0.txt
+++ b/doc/source/whatsnew/v0.24.0.txt
@@ -582,6 +582,7 @@ Datetimelike
 - Bug in :class:`DataFrame` comparisons against ``Timestamp``-like objects failing to raise ``TypeError`` for inequality checks with mismatched types (:issue:`8932`,:issue:`22163`)
 - Bug in :class:`DataFrame` with mixed dtypes including ``datetime64[ns]`` incorrectly raising ``TypeError`` on equality comparisons (:issue:`13128`,:issue:`22163`)
 - Bug in :meth:`DataFrame.eq` comparison against ``NaT`` incorrectly returning ``True`` or ``NaN`` (:issue:`15697`,:issue:`22163`)
+- Bug in :class:`DatetimeIndex` subtraction that incorrectly failed to raise `OverflowError` when the difference between :class:`Timestamp`s (a :class:`Timedelta` object) exceeds `int64` limit (:issue:`22492`, :issue:`22508`)
 
 Timedelta
 ^^^^^^^^^
diff --git a/pandas/core/arrays/datetimes.py b/pandas/core/arrays/datetimes.py
index 1dd34cdf73ab5..484eb430c82b1 100644
--- a/pandas/core/arrays/datetimes.py
+++ b/pandas/core/arrays/datetimes.py
@@ -459,7 +459,8 @@ def _sub_datelike_dti(self, other):
 
         self_i8 = self.asi8
         other_i8 = other.asi8
-        new_values = self_i8 - other_i8
+        new_values = checked_add_with_arr(self_i8, -other_i8,
+                                          arr_mask=self._isnan)
         if self.hasnans or other.hasnans:
             mask = (self._isnan) | (other._isnan)
             new_values[mask] = iNaT
diff --git a/pandas/tests/arithmetic/test_datetime64.py b/pandas/tests/arithmetic/test_datetime64.py
index 879a4e1b4af1a..b7c0d9a540655 100644
--- a/pandas/tests/arithmetic/test_datetime64.py
+++ b/pandas/tests/arithmetic/test_datetime64.py
@@ -1570,6 +1570,38 @@ def test_datetimeindex_sub_timestamp_overflow(self):
             with pytest.raises(OverflowError):
                 dtimin - variant
 
+    def test_datetimeindex_sub_datetimeindex_overflow(self):
+        # GH#22492, GH#22508
+        dtimax = pd.to_datetime(['now', pd.Timestamp.max])
+        dtimin = pd.to_datetime(['now', pd.Timestamp.min])
+
+        ts_neg = pd.to_datetime(['1950-01-01', '1950-01-01'])
+        ts_pos = pd.to_datetime(['1980-01-01', '1980-01-01'])
+
+        with pytest.raises(OverflowError):
+            dtimax - ts_neg
+
+        expected = pd.Timestamp.max.value - ts_pos[1].value
+        res = dtimax - ts_pos
+        assert res[1].value == expected
+
+        expected = pd.Timestamp.min.value - ts_neg[1].value
+        res = dtimin - ts_neg
+        assert res[1].value == expected
+
+        with pytest.raises(OverflowError):
+            dtimin - ts_pos
+
+        tmin = pd.to_datetime([pd.Timestamp.min])
+        t1 = tmin + pd.Timedelta.max + pd.Timedelta('1us')
+        with pytest.raises(OverflowError):
+            t1 - tmin
+
+        tmax = pd.to_datetime([pd.Timestamp.max])
+        t2 = tmax + pd.Timedelta.min - pd.Timedelta('1us')
+        with pytest.raises(OverflowError):
+            tmax - t2
+
     @pytest.mark.parametrize('names', [('foo', None, None),
                                        ('baz', 'bar', None),
                                        ('bar', 'bar', 'bar')])


Code review comments:
diff_hunk: @@ -582,6 +582,7 @@ Datetimelike
 - Bug in :class:`DataFrame` comparisons against ``Timestamp``-like objects failing to raise ``TypeError`` for inequality checks with mismatched types (:issue:`8932`,:issue:`22163`)
 - Bug in :class:`DataFrame` with mixed dtypes including ``datetime64[ns]`` incorrectly raising ``TypeError`` on equality comparisons (:issue:`13128`,:issue:`22163`)
 - Bug in :meth:`DataFrame.eq` comparison against ``NaT`` incorrectly returning ``True`` or ``NaN`` (:issue:`15697`,:issue:`22163`)
+- Bug in :class:`DatetimeIndex` subtraction that incorrectly failed to raise `OverflowError` when the difference between :class:`Timestamp`s (a :class:`Timedelta` object) exceeds `int64` limit (:issue:`22492`, :issue:`22508`)
path: doc/source/whatsnew/v0.24.0.txt
text: @user1:
this is now confusing. let's go back to the prior one;

@author:
sorry about that! 
for the record, it was mentioned [here](https://pandas.pydata.org/pandas-docs/stable/timedeltas.html#timedelta-limitations) that `pd.Timedelta`s are represented as nanoseconds using 64 bit integers, and I believe that's the source of potential overflow.

diff_hunk: @@ -1570,6 +1570,38 @@ def test_datetimeindex_sub_timestamp_overflow(self):
             with pytest.raises(OverflowError):
                 dtimin - variant
 
+    def test_datetimeindex_sub_datetimeindex_overflow(self):
+        # GH#22492, GH#22508
+        dtimax = pd.to_datetime(['now', pd.Timestamp.max])
+        dtimin = pd.to_datetime(['now', pd.Timestamp.min])
+
+        ts_neg = pd.to_datetime(['1950-01-01', '1950-01-01'])
+        ts_pos = pd.to_datetime(['1980-01-01', '1980-01-01'])
+
+        with pytest.raises(OverflowError):
+            dtimax - ts_neg
+
+        expected = pd.Timestamp.max.value - ts_pos[1].value
+        res = dtimax - ts_pos
path: pandas/tests/arithmetic/test_datetime64.py
text: @user1:
use result (rather than res) in all occurrences