Repository: pandas-dev/pandas

Issue:
read_sas OverflowError: int too big to convert with '31DEC9999'd formatted as DATE9. 
```python
import os
import pandas as pd

sasDS_works = 'highDate_works.sas7bdat'
sasDS_fails = 'highDate_fails.sas7bdat'
path = r'C:\temp'

print(sasDS_works)
df = pd.read_sas(os.path.join(path, sasDS_works))
print(df)

print(sasDS_fails)
df = pd.read_sas(os.path.join(path, sasDS_fails))
print(df)
```

```sas
data temp.highDate_works ;
	highDate_num = '31DEC9999'd;
	highDate_date = '31DEC9999'd; 
run;
data temp.highDate_fails ;
	highDate_num = '31DEC9999'd;
	highDate_date = '31DEC9999'd; format highDate_date date9.;
run;
```
The sas7bdat files are zipped and uploaded.
[highdate_works.zip](https://github.com/pandas-dev/pandas/files/1967842/highdate_works.zip)


#### Problem description
Traceback (most recent call last):
  File "pandas/_libs/tslib.pyx", line 2075, in pandas._libs.tslib.array_with_unit_to_datetime
  File "pandas/_libs/tslibs/timedeltas.pyx", line 96, in pandas._libs.tslibs.timedeltas.cast_from_unit
OverflowError: int too big to convert

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:/xxxx/aargh_sasData_31DEC9999.py", line 27, in <module>
    df = pd.read_sas(os.path.join(path, sasDS_fails))
  File "C:\xxxx\AppData\Local\Continuum\Anaconda3\envs\ratabasePython\lib\site-packages\pandas\io\sas\sasreader.py", line 68, in read_sas
    data = reader.read()
  File "C:\xxx\AppData\Local\Continuum\Anaconda3\envs\ratabasePython\lib\site-packages\pandas\io\sas\sas7bdat.py", line 614, in read
    rslt = self._chunk_to_dataframe()
  File "C:\xxxx\AppData\Local\Continuum\Anaconda3\envs\ratabasePython\lib\site-packages\pandas\io\sas\sas7bdat.py", line 666, in _chunk_to_dataframe
    origin="1960-01-01")
  File "C:\xxxx\AppData\Local\Continuum\Anaconda3\envs\ratabasePython\lib\site-packages\pandas\core\tools\datetimes.py", line 373, in to_datetime
    values = _convert_listlike(arg._values, True, format)
  File "C:\xxxx\AppData\Local\Continuum\Anaconda3\envs\ratabasePython\lib\site-packages\pandas\core\tools\datetimes.py", line 229, in _convert_listlike
    errors=errors)
  File "pandas/_libs/tslib.pyx", line 2001, in pandas._libs.tslib.array_with_unit_to_datetime
  File "pandas/_libs/tslib.pyx", line 2078, in pandas._libs.tslib.array_with_unit_to_datetime
pandas._libs.tslib.OutOfBoundsDatetime: cannot convert input 2932894.0 with the unit 'd'

#### Expected Output




#### Output of ``pd.show_versions()``
<details>

INSTALLED VERSIONS
------------------
commit: None
python: 3.6.4.final.0
python-bits: 64
OS: Windows
OS-release: 7
machine: AMD64
processor: Intel64 Family 6 Model 78 Stepping 3, GenuineIntel
byteorder: little
LC_ALL: None
LANG: None
LOCALE: None.None
pandas: 0.22.0
pytest: 3.3.2
pip: 9.0.1
setuptools: 38.4.0
Cython: 0.27.3
numpy: 1.14.2
scipy: 1.0.0
pyarrow: None
xarray: None
IPython: 6.2.1
sphinx: 1.6.6
patsy: 0.5.0
dateutil: 2.6.1
pytz: 2018.4
blosc: None
bottleneck: 1.2.1
tables: 3.4.2
numexpr: 2.6.4
feather: None
matplotlib: 2.1.2
openpyxl: 2.4.10
xlrd: 1.1.0
xlwt: 1.3.0
xlsxwriter: 1.0.2
lxml: 4.2.1
bs4: 4.6.0
html5lib: 1.0.1
sqlalchemy: 1.2.1
pymysql: None
psycopg2: 2.7.3.2 (dt dec pq3 ext lo64)
jinja2: 2.10
s3fs: None
fastparquet: None
pandas_gbq: None
pandas_datareader: None
</details>



Discussion:
I've dug a bit more into this.

The exception throws from 
File "pandas/_libs/tslibs/timedeltas.pyx", line 237, in pandas._libs.tslibs.timedeltas.cast_from_unit
which tries to cast the number of days from SAS epoch (midnight 1960-01-01) to 9999-12-31 to nanoseconds and return it as an int64_t
(as an aside the correct number of days is 2932896 but 2932894 is passed in - I think this is a follow-through error
from SAS7BDAT which pandas uses as direct use of SAS7BDAT converts 9999-12-31 to 9999-12-29 - i.e. the same 2-day diff)
Anyhow, 2932894 * 1000000000L * 86400 is too large for an int64_t => exception "OverflowError: int too big to convert"

question is - why convert a datediff in days to nanosecs?
answer is likely that the function _libs.tslibs.timedeltas.cast_from_unit is generic and not just for importing data from SAS

according to http://support.sas.com/documentation/cdl/en/etsug/63939/HTML/default/viewer.htm#etsug_tsdata_sect006.htm
SAS datetimes are the number of seconds the time is from midnight, 1 January 1960.  
All numerics in SAS are floating point so milliseconds in a SAS datetime can be stored in the fractional part but 
there is insufficient precision to store nanoseconds in a SAS datetime.
(see https://support.sas.com/resources/papers/proceedings16/SAS2802-2016.pdf for a suggested workaround)
i.e. nanosecond granularity is never needed for converting SAS dates or datetimes

solutions?
a) return an int128_t from pandas._libs.tslibs.timedeltas.cast_from_unit
b) have a separate conversion for SAS dates - maybe just use what SAS7BDAT gives ? (and fix the out-by-two-days error in SAS7BDAT rather than pandas)

PS - Not sure the Timeseries label is appropriate for this
Oops - didn't mean to close.
The SAS import issue is NOT related to the format of the very high dates - i.e. 31DEC9999 or 9999-12-31 as a SAS format both fail.  
The root problem is the conversion via nanoseconds in _libs.tslibs.timedeltas.cast_from_unit which is used by to_datetime()
Thanks to comment from @jschendel on #21972 a possible resolution to this may to be catch the pandas._libs.tslib.OutOfBoundsDatetime and convert larger dates to Periods https://pandas.pydata.org/pandas-docs/stable/timeseries.html#timeseries-oob
On second thoughts, I think a better approach is exposing the convert_dates=bool parameter from the underlying SAS7BDATReader up to read_sas().  
This retains current functionality but also allows catching the exception for very high datetimes/dates and re-reading with the dates/datetimes as SAS native float64 values which can then be converted/cleansed/capped as required.

I've forked, altered and filed a pull-request - first time I've done this so any feedback is most welcome.
Hi @paul-lilley 
Thanks for your work here.
I am currently facing this issue. Did you solve the problem?
Hi @sunilkalmadi 
Sorry for the long delay replying, I've now developed a fix and will create a pull-request for it.
The fix converts dates > 2262-04-11 to datetime.date rather than np.datetime64 so that they can be read in.

Resolving patch:
diff --git a/pandas/io/sas/sas7bdat.py b/pandas/io/sas/sas7bdat.py
--- a/pandas/io/sas/sas7bdat.py
+++ b/pandas/io/sas/sas7bdat.py
@@ -14,12 +14,12 @@
   http://collaboration.cmc.ec.gc.ca/science/rpn/biblio/ddj/Website/articles/CUJ/1992/9210/ross/ross.htm
 """
 from collections import abc
-from datetime import datetime
+from datetime import datetime, timedelta
 import struct
 
 import numpy as np
 
-from pandas.errors import EmptyDataError
+from pandas.errors import EmptyDataError, OutOfBoundsDatetime
 
 import pandas as pd
 
@@ -29,6 +29,39 @@
 from pandas.io.sas.sasreader import ReaderBase
 
 
+def _convert_datetimes(sas_datetimes: pd.Series, unit: str) -> pd.Series:
+    """
+    Convert to Timestamp if possible, otherwise to datetime.datetime.
+    SAS float64 lacks precision for more than ms resolution so the fit
+    to datetime.datetime is ok.
+
+    Parameters
+    ----------
+    sas_datetimes : {Series, Sequence[float]}
+       Dates or datetimes in SAS
+    unit : {str}
+       "d" if the floats represent dates, "s" for datetimes
+
+    Returns
+    -------
+    Series
+       Series of datetime64 dtype or datetime.datetime.
+    """
+    try:
+        return pd.to_datetime(sas_datetimes, unit=unit, origin="1960-01-01")
+    except OutOfBoundsDatetime:
+        if unit == "s":
+            return sas_datetimes.apply(
+                lambda sas_float: datetime(1960, 1, 1) + timedelta(seconds=sas_float)
+            )
+        elif unit == "d":
+            return sas_datetimes.apply(
+                lambda sas_float: datetime(1960, 1, 1) + timedelta(days=sas_float)
+            )
+        else:
+            raise ValueError("unit must be 'd' or 's'")
+
+
 class _subheader_pointer:
     pass
 
@@ -706,15 +739,10 @@ def _chunk_to_dataframe(self):
                 rslt[name] = self._byte_chunk[jb, :].view(dtype=self.byte_order + "d")
                 rslt[name] = np.asarray(rslt[name], dtype=np.float64)
                 if self.convert_dates:
-                    unit = None
                     if self.column_formats[j] in const.sas_date_formats:
-                        unit = "d"
+                        rslt[name] = _convert_datetimes(rslt[name], "d")
                     elif self.column_formats[j] in const.sas_datetime_formats:
-                        unit = "s"
-                    if unit:
-                        rslt[name] = pd.to_datetime(
-                            rslt[name], unit=unit, origin="1960-01-01"
-                        )
+                        rslt[name] = _convert_datetimes(rslt[name], "s")
                 jb += 1
             elif self._column_types[j] == b"s":
                 rslt[name] = self._string_chunk[js, :]


Test patch:
diff --git a/pandas/tests/io/sas/data/max_sas_date.sas7bdat b/pandas/tests/io/sas/data/max_sas_date.sas7bdat
new file mode 100644
Binary files /dev/null and b/pandas/tests/io/sas/data/max_sas_date.sas7bdat differ
diff --git a/pandas/tests/io/sas/test_sas7bdat.py b/pandas/tests/io/sas/test_sas7bdat.py
--- a/pandas/tests/io/sas/test_sas7bdat.py
+++ b/pandas/tests/io/sas/test_sas7bdat.py
@@ -3,6 +3,7 @@
 import os
 from pathlib import Path
 
+import dateutil.parser
 import numpy as np
 import pytest
 
@@ -214,3 +215,94 @@ def test_zero_variables(datapath):
     fname = datapath("io", "sas", "data", "zero_variables.sas7bdat")
     with pytest.raises(EmptyDataError):
         pd.read_sas(fname)
+
+
+def round_datetime_to_ms(ts):
+    if isinstance(ts, datetime):
+        return ts.replace(microsecond=int(round(ts.microsecond, -3) / 1000) * 1000)
+    elif isinstance(ts, str):
+        _ts = dateutil.parser.parse(timestr=ts)
+        return _ts.replace(microsecond=int(round(_ts.microsecond, -3) / 1000) * 1000)
+    else:
+        return ts
+
+
+def test_max_sas_date(datapath):
+    # GH 20927
+    # NB. max datetime in SAS dataset is 31DEC9999:23:59:59.999
+    #    but this is read as 29DEC9999:23:59:59.998993 by a buggy
+    #    sas7bdat module
+    fname = datapath("io", "sas", "data", "max_sas_date.sas7bdat")
+    df = pd.read_sas(fname, encoding="iso-8859-1")
+
+    # SAS likes to left pad strings with spaces - lstrip before comparing
+    df = df.applymap(lambda x: x.lstrip() if isinstance(x, str) else x)
+    # GH 19732: Timestamps imported from sas will incur floating point errors
+    try:
+        df["dt_as_dt"] = df["dt_as_dt"].dt.round("us")
+    except pd._libs.tslibs.np_datetime.OutOfBoundsDatetime:
+        df = df.applymap(round_datetime_to_ms)
+    except AttributeError:
+        df["dt_as_dt"] = df["dt_as_dt"].apply(round_datetime_to_ms)
+    # if there are any date/times > pandas.Timestamp.max then ALL in that chunk
+    # are returned as datetime.datetime
+    expected = pd.DataFrame(
+        {
+            "text": ["max", "normal"],
+            "dt_as_float": [253717747199.999, 1880323199.999],
+            "dt_as_dt": [
+                datetime(9999, 12, 29, 23, 59, 59, 999000),
+                datetime(2019, 8, 1, 23, 59, 59, 999000),
+            ],
+            "date_as_float": [2936547.0, 21762.0],
+            "date_as_date": [datetime(9999, 12, 29), datetime(2019, 8, 1)],
+        },
+        columns=["text", "dt_as_float", "dt_as_dt", "date_as_float", "date_as_date"],
+    )
+    tm.assert_frame_equal(df, expected)
+
+
+def test_max_sas_date_iterator(datapath):
+    # GH 20927
+    # when called as an iterator, only those chunks with a date > pd.Timestamp.max
+    # are returned as datetime.datetime, if this happens that whole chunk is returned
+    # as datetime.datetime
+    col_order = ["text", "dt_as_float", "dt_as_dt", "date_as_float", "date_as_date"]
+    fname = datapath("io", "sas", "data", "max_sas_date.sas7bdat")
+    results = []
+    for df in pd.read_sas(fname, encoding="iso-8859-1", chunksize=1):
+        # SAS likes to left pad strings with spaces - lstrip before comparing
+        df = df.applymap(lambda x: x.lstrip() if isinstance(x, str) else x)
+        # GH 19732: Timestamps imported from sas will incur floating point errors
+        try:
+            df["dt_as_dt"] = df["dt_as_dt"].dt.round("us")
+        except pd._libs.tslibs.np_datetime.OutOfBoundsDatetime:
+            df = df.applymap(round_datetime_to_ms)
+        except AttributeError:
+            df["dt_as_dt"] = df["dt_as_dt"].apply(round_datetime_to_ms)
+        df.reset_index(inplace=True, drop=True)
+        results.append(df)
+    expected = [
+        pd.DataFrame(
+            {
+                "text": ["max"],
+                "dt_as_float": [253717747199.999],
+                "dt_as_dt": [datetime(9999, 12, 29, 23, 59, 59, 999000)],
+                "date_as_float": [2936547.0],
+                "date_as_date": [datetime(9999, 12, 29)],
+            },
+            columns=col_order,
+        ),
+        pd.DataFrame(
+            {
+                "text": ["normal"],
+                "dt_as_float": [1880323199.999],
+                "dt_as_dt": [np.datetime64("2019-08-01 23:59:59.999")],
+                "date_as_float": [21762.0],
+                "date_as_date": [np.datetime64("2019-08-01")],
+            },
+            columns=col_order,
+        ),
+    ]
+    for result, expected in zip(results, expected):
+        tm.assert_frame_equal(result, expected)