Repository: pandas-dev/pandas

Issue:
Failing to parse date given as integers from a (MS)SQL query
#### Code Sample
```python
SQL = """
SELECT TOP 10
submitted_date_id
FROM my_table
WHERE submitted_date_id BETWEEN %(mindate)s AND %(maxdate)s
"""

df = pd.read_sql(
    SQL, 
    con, 
    params={'mindate': 20171008, 'maxdate': '20171010'},
    parse_dates={'submitted_date_id': '%Y%m%d'})

df
```

The underlaying database is MSSQL and I'm using `pymssql` to build the connection `con`.

#### Problem description

The column `submitted_date_id` has integers representing the date `YYYYMMDD`. The code above is expected to parse the column and populate the resulting column in the dataframe with `Timestamp` objects. However, this code yields an error:

```
ValueError: cannot cast unit %Y%m%d
```

The current behavior is counter intuitive and seems to assume that the date column contains stings. A work around is to cast the column in the query to `VARCHAR`, but this is cumbersome. Moreover, it adds unneeded complexity to the query.

#### Expected Output

When providing the format of the dates to `parse_dates`, it is expected that the column will hold Timestamp objects.

#### Output of ``pd.show_versions()``

<details>

INSTALLED VERSIONS
------------------
commit: None
python: 3.6.2.final.0
python-bits: 64
OS: Darwin
OS-release: 16.7.0
machine: x86_64
processor: i386
byteorder: little
LC_ALL: en_US.UTF-8
LANG: en_US.UTF-8
LOCALE: en_US.UTF-8

pandas: 0.20.3
pytest: 3.2.1
pip: 9.0.1
setuptools: 36.3.0
Cython: None
numpy: 1.13.1
scipy: 0.19.1
xarray: None
IPython: 6.2.0
sphinx: 1.6.4
patsy: None
dateutil: 2.6.1
pytz: 2017.2
blosc: None
bottleneck: None
tables: None
numexpr: None
feather: None
matplotlib: 2.0.2
openpyxl: None
xlrd: None
xlwt: None
xlsxwriter: None
lxml: None
bs4: None
html5lib: 0.999999999
sqlalchemy: None
pymysql: None
psycopg2: None
jinja2: 2.9.6
s3fs: None
pandas_gbq: None
pandas_datareader: None

</details>



Discussion:
w/o a reproducible example you would have to debug this and see where things are going wrong.
as we handle this case already. (not 100% sure that the implementation actually passes things thru properly though).
```
In [10]: pd.to_datetime(20170101, format='%Y%m%d')
Out[10]: Timestamp('2017-01-01 00:00:00')

In [11]: pd.to_datetime('20170101', format='%Y%m%d')
Out[11]: Timestamp('2017-01-01 00:00:00')
```
Here's a reproducible example:

```python
import sqlalchemy
import pymysql
import pandas as pd

eng = sqlalchemy.create_engine('mysql+pymysql://sql2199180:kI1*yG4%@sql2.freemysqlhosting.net/sql2199180')
con = eng.connect()

pd.read_sql('select d from foo', con) # Works
pd.read_sql('select d from foo', con, parse_dates={'d': '%Y%m%d'}) # Fails
pd.read_sql('select CONVERT(d, CHAR(10)) as d from foo', con, parse_dates={'d': '%Y%m%d'}) # Works
```
So this is the current code to handle the `parse_dates` keyword:

https://github.com/pandas-dev/pandas/blob/c277cd76416d4e930b1f05da873b9eaf101139da/pandas/io/sql.py#L102-L119

where it explicitly checks for numeric column, and then passes `format` as the `unit`, which then fails.

I agree this is a bit inconvenient. 
As a workaround for now, going from the code (didn't try), I *think* you can do `parse_dates={'d':dict(format='%Y%m%d')}` to force the use of the correct keyword.

An easy fix could also be to check whether '%' is contained in the format string, and in that case not pass it to `unit` (cleaner fix would be to not make this check on the data type, and always pass the format to `format` if it is not in the list of known units, but still parse it with `unit='s'` when no format is passed. I *think* this should not break any code).

@drorata Would you like to do a PR?

I am missing something. Why `to_datetime` is not used directly and if there's a formatting available it should be passed and used by `to_datetime`? The code @jorisvandenbossche referred to is changing the expected behavior of converting a Series to `Timestamps`s using `to_datetime`.
@jorisvandenbossche I just tried to run the test suite (`pytest` from the root directory of the repository) on my checked out copy and I get the following:

<details>
```
../../anaconda3/envs/pandas_dev/lib/python3.6/site-packages/_pytest/config.py:342: in _getconftestmodules
    return self._path2confmods[path]
E   KeyError: local('/Users/username/Dropbox/dev/pandas/pandas')

During handling of the above exception, another exception occurred:
../../anaconda3/envs/pandas_dev/lib/python3.6/site-packages/_pytest/config.py:373: in _importconftest
    return self._conftestpath2mod[conftestpath]
E   KeyError: local('/Users/username/Dropbox/dev/pandas/pandas/conftest.py')

During handling of the above exception, another exception occurred:
../../anaconda3/envs/pandas_dev/lib/python3.6/site-packages/_pytest/config.py:379: in _importconftest
    mod = conftestpath.pyimport()
../../anaconda3/envs/pandas_dev/lib/python3.6/site-packages/py/_path/local.py:662: in pyimport
    __import__(modname)
pandas/pandas/__init__.py:40: in <module>
    import pandas.core.config_init
pandas/pandas/core/config_init.py:12: in <module>
    import pandas.core.config as cf
pandas/pandas/core/config.py:820: in <module>
    is_text = is_instance_factory((str, bytes))
pandas/pandas/core/config.py:781: in is_instance_factory
    from pandas.io.formats.printing import pprint_thing
pandas/pandas/io/formats/printing.py:6: in <module>
    from pandas.core.dtypes.inference import is_sequence
pandas/pandas/core/dtypes/inference.py:24: in <module>
    is_interval = lib.is_interval
E   AttributeError: module 'pandas._libs.lib' has no attribute 'is_interval'

During handling of the above exception, another exception occurred:
../../anaconda3/envs/pandas_dev/lib/python3.6/site-packages/py/_path/common.py:372: in visit
    for x in Visitor(fil, rec, ignore, bf, sort).gen(self):
../../anaconda3/envs/pandas_dev/lib/python3.6/site-packages/py/_path/common.py:410: in gen
    dirs = self.optsort([p for p in entries
../../anaconda3/envs/pandas_dev/lib/python3.6/site-packages/py/_path/common.py:411: in <listcomp>
    if p.check(dir=1) and (rec is None or rec(p))])
../../anaconda3/envs/pandas_dev/lib/python3.6/site-packages/_pytest/main.py:728: in _recurse
    ihook = self.gethookproxy(path)
../../anaconda3/envs/pandas_dev/lib/python3.6/site-packages/_pytest/main.py:632: in gethookproxy
    my_conftestmodules = pm._getconftestmodules(fspath)
../../anaconda3/envs/pandas_dev/lib/python3.6/site-packages/_pytest/config.py:356: in _getconftestmodules
    mod = self._importconftest(conftestpath)
../../anaconda3/envs/pandas_dev/lib/python3.6/site-packages/_pytest/config.py:381: in _importconftest
    raise ConftestImportFailure(conftestpath, sys.exc_info())
E   _pytest.config.ConftestImportFailure: AttributeError("module 'pandas._libs.lib' has no attribute 'is_interval'",)
E     File "/Users/username/Dropbox/dev/pandas/pandas/__init__.py", line 40, in <module>
E       import pandas.core.config_init
E     File "/Users/username/Dropbox/dev/pandas/pandas/core/config_init.py", line 12, in <module>
E       import pandas.core.config as cf
E     File "/Users/username/Dropbox/dev/pandas/pandas/core/config.py", line 820, in <module>
E       is_text = is_instance_factory((str, bytes))
E     File "/Users/username/Dropbox/dev/pandas/pandas/core/config.py", line 781, in is_instance_factory
E       from pandas.io.formats.printing import pprint_thing
E     File "/Users/username/Dropbox/dev/pandas/pandas/io/formats/printing.py", line 6, in <module>
E       from pandas.core.dtypes.inference import is_sequence
E     File "/Users/username/Dropbox/dev/pandas/pandas/core/dtypes/inference.py", line 24, in <module>
E       is_interval = lib.is_interval
```
</details>

If this is not a trivial problem, I'll start a separate issue.

Resolving patch:
diff --git a/pandas/io/sql.py b/pandas/io/sql.py
--- a/pandas/io/sql.py
+++ b/pandas/io/sql.py
@@ -103,12 +103,12 @@ def _handle_date_column(col, utc=None, format=None):
     if isinstance(format, dict):
         return to_datetime(col, errors='ignore', **format)
     else:
-        if format in ['D', 's', 'ms', 'us', 'ns']:
-            return to_datetime(col, errors='coerce', unit=format, utc=utc)
-        elif (issubclass(col.dtype.type, np.floating) or
-              issubclass(col.dtype.type, np.integer)):
-            # parse dates as timestamp
-            format = 's' if format is None else format
+        # Allow passing of formatting string for integers
+        # GH17855
+        if format is None and (issubclass(col.dtype.type, np.floating) or
+                               issubclass(col.dtype.type, np.integer)):
+            format = 's'
+        if format in ['D', 'd', 'h', 'm', 's', 'ms', 'us', 'ns']:
             return to_datetime(col, errors='coerce', unit=format, utc=utc)
         elif is_datetime64tz_dtype(col):
             # coerce to UTC timezone


Test patch:
diff --git a/pandas/tests/io/test_sql.py b/pandas/tests/io/test_sql.py
--- a/pandas/tests/io/test_sql.py
+++ b/pandas/tests/io/test_sql.py
@@ -88,6 +88,7 @@
                     "TextCol" TEXT,
                     "DateCol" TEXT,
                     "IntDateCol" INTEGER,
+                    "IntDateOnlyCol" INTEGER,
                     "FloatCol" REAL,
                     "IntCol" INTEGER,
                     "BoolCol" INTEGER,
@@ -98,6 +99,7 @@
                     `TextCol` TEXT,
                     `DateCol` DATETIME,
                     `IntDateCol` INTEGER,
+                    `IntDateOnlyCol` INTEGER,
                     `FloatCol` DOUBLE,
                     `IntCol` INTEGER,
                     `BoolCol` BOOLEAN,
@@ -109,6 +111,7 @@
                     "DateCol" TIMESTAMP,
                     "DateColWithTz" TIMESTAMP WITH TIME ZONE,
                     "IntDateCol" INTEGER,
+                    "IntDateOnlyCol" INTEGER,
                     "FloatCol" DOUBLE PRECISION,
                     "IntCol" INTEGER,
                     "BoolCol" BOOLEAN,
@@ -120,31 +123,33 @@
         'sqlite': {
             'query': """
                 INSERT INTO types_test_data
-                VALUES(?, ?, ?, ?, ?, ?, ?, ?)
+                VALUES(?, ?, ?, ?, ?, ?, ?, ?, ?)
                 """,
             'fields': (
-                'TextCol', 'DateCol', 'IntDateCol', 'FloatCol',
-                'IntCol', 'BoolCol', 'IntColWithNull', 'BoolColWithNull'
+                'TextCol', 'DateCol', 'IntDateCol', 'IntDateOnlyCol',
+                'FloatCol', 'IntCol', 'BoolCol', 'IntColWithNull',
+                'BoolColWithNull'
             )
         },
         'mysql': {
             'query': """
                 INSERT INTO types_test_data
-                VALUES("%s", %s, %s, %s, %s, %s, %s, %s)
+                VALUES("%s", %s, %s, %s, %s, %s, %s, %s, %s)
                 """,
             'fields': (
-                'TextCol', 'DateCol', 'IntDateCol', 'FloatCol',
-                'IntCol', 'BoolCol', 'IntColWithNull', 'BoolColWithNull'
+                'TextCol', 'DateCol', 'IntDateCol', 'IntDateOnlyCol',
+                'FloatCol', 'IntCol', 'BoolCol', 'IntColWithNull',
+                'BoolColWithNull'
             )
         },
         'postgresql': {
             'query': """
                 INSERT INTO types_test_data
-                VALUES(%s, %s, %s, %s, %s, %s, %s, %s, %s)
+                VALUES(%s, %s, %s, %s, %s, %s, %s, %s, %s, %s)
                 """,
             'fields': (
                 'TextCol', 'DateCol', 'DateColWithTz',
-                'IntDateCol', 'FloatCol',
+                'IntDateCol', 'IntDateOnlyCol', 'FloatCol',
                 'IntCol', 'BoolCol', 'IntColWithNull', 'BoolColWithNull'
             )
         },
@@ -313,13 +318,13 @@ def _load_raw_sql(self):
         self.drop_table('types_test_data')
         self._get_exec().execute(SQL_STRINGS['create_test_types'][self.flavor])
         ins = SQL_STRINGS['insert_test_types'][self.flavor]
-
         data = [
             {
                 'TextCol': 'first',
                 'DateCol': '2000-01-03 00:00:00',
                 'DateColWithTz': '2000-01-01 00:00:00-08:00',
                 'IntDateCol': 535852800,
+                'IntDateOnlyCol': 20101010,
                 'FloatCol': 10.10,
                 'IntCol': 1,
                 'BoolCol': False,
@@ -331,6 +336,7 @@ def _load_raw_sql(self):
                 'DateCol': '2000-01-04 00:00:00',
                 'DateColWithTz': '2000-06-01 00:00:00-07:00',
                 'IntDateCol': 1356998400,
+                'IntDateOnlyCol': 20101212,
                 'FloatCol': 10.10,
                 'IntCol': 1,
                 'BoolCol': False,
@@ -610,20 +616,42 @@ def test_date_parsing(self):
         df = sql.read_sql_query("SELECT * FROM types_test_data", self.conn,
                                 parse_dates=['DateCol'])
         assert issubclass(df.DateCol.dtype.type, np.datetime64)
+        assert df.DateCol.tolist() == [
+            pd.Timestamp(2000, 1, 3, 0, 0, 0),
+            pd.Timestamp(2000, 1, 4, 0, 0, 0)
+        ]
 
         df = sql.read_sql_query("SELECT * FROM types_test_data", self.conn,
                                 parse_dates={'DateCol': '%Y-%m-%d %H:%M:%S'})
         assert issubclass(df.DateCol.dtype.type, np.datetime64)
+        assert df.DateCol.tolist() == [
+            pd.Timestamp(2000, 1, 3, 0, 0, 0),
+            pd.Timestamp(2000, 1, 4, 0, 0, 0)
+        ]
 
         df = sql.read_sql_query("SELECT * FROM types_test_data", self.conn,
                                 parse_dates=['IntDateCol'])
-
         assert issubclass(df.IntDateCol.dtype.type, np.datetime64)
+        assert df.IntDateCol.tolist() == [
+            pd.Timestamp(1986, 12, 25, 0, 0, 0),
+            pd.Timestamp(2013, 1, 1, 0, 0, 0)
+        ]
 
         df = sql.read_sql_query("SELECT * FROM types_test_data", self.conn,
                                 parse_dates={'IntDateCol': 's'})
-
         assert issubclass(df.IntDateCol.dtype.type, np.datetime64)
+        assert df.IntDateCol.tolist() == [
+            pd.Timestamp(1986, 12, 25, 0, 0, 0),
+            pd.Timestamp(2013, 1, 1, 0, 0, 0)
+        ]
+
+        df = sql.read_sql_query("SELECT * FROM types_test_data", self.conn,
+                                parse_dates={'IntDateOnlyCol': '%Y%m%d'})
+        assert issubclass(df.IntDateOnlyCol.dtype.type, np.datetime64)
+        assert df.IntDateOnlyCol.tolist() == [
+            pd.Timestamp('2010-10-10'),
+            pd.Timestamp('2010-12-12')
+        ]
 
     def test_date_and_index(self):
         # Test case where same column appears in parse_date and index_col