Repository: pandas-dev/pandas

Issue:
Inconsistent parsing for timestamp with timezone with read_sql_query
#### Code Sample, a copy-pastable example if possible
I'm creating a test table in postgresql as follows:
```sql
CREATE TABLE test(date timestamptz);

INSERT INTO test VALUES('2014-10-16 07:50:49-04'),('2014-11-06 07:42:06-05')
```
Notice that the dates are across the dst date, so that they have different offsets to utc.

```python
df1 = pd.read_sql_query("SELECT date FROM test", engine,
                        index_col='date')

df2 = pd.read_sql_query("SELECT date FROM test WHERE date<='2014-10-17'",
                        engine,
                        index_col='date')

df2bis = pd.read_sql_query("SELECT date FROM test WHERE date<='2014-10-17'",
                           engine,
                           index_col='date',
                           parse_dates=['date'])
df3 = pd.read_sql_query("SELECT date FROM test", engine,
                        index_col='date',
                        parse_dates=['date'])

```
#### Problem description
```
In [159]: df1.index
Out[159]: 
Index([2014-10-16 07:50:49-04:00, 2014-11-06 07:42:06-05:00], dtype='object', name='date')

In [160]: df2.index
Out[160]: 
DatetimeIndex(['2014-10-16 11:50:49+00:00'], dtype='datetime64[ns, UTC]', name='date', freq=None)

In [162]: df2bis.index
Out[162]: 
DatetimeIndex(['2014-10-16 11:50:49+00:00'], dtype='datetime64[ns, UTC]', name='date', freq=None)

In [163]: df3.index
Out[163]: 
DatetimeIndex(['2014-10-16 11:50:49', '2014-11-06 12:42:06'], dtype='datetime64[ns]', name='date', freq=None)
```
#### Expected Output
The current behavior if I understand correctly is that if there is only one timezone offset: timestamps are parsed automatically to timezone aware timestamps.
If dates are across different offsets, they are not parsed automatically. We can force parsing using parse_dates, but then they are not timezone aware. I can make the conversion using tz_localize, but it's not very consistent. I would expect to have everything parsed automatically to timezone aware timestamps.

This is related to issue #7364. If there is a consensus on what should be done, I can try to make the required changes to make it more regular.

#### Output of ``pd.show_versions()``

<details>
INSTALLED VERSIONS
------------------
commit: None
python: 3.6.0.final.0
python-bits: 64
OS: Linux
OS-release: 4.8.13-1-ARCH
machine: x86_64
processor: 
byteorder: little
LC_ALL: None
LANG: en_US.utf8
LOCALE: en_US.UTF-8

pandas: 0.19.2
nose: 1.3.7
pip: 9.0.1
setuptools: 33.0.0
Cython: 0.25.2
numpy: 1.11.2
scipy: 0.18.1
statsmodels: 0.6.1
xarray: None
IPython: 5.1.0
sphinx: 1.5.1
patsy: 0.4.1
dateutil: 2.6.0
pytz: 2016.10
blosc: None
bottleneck: None
tables: 3.3.0
numexpr: 2.6.1
matplotlib: 2.0.0rc2
openpyxl: 2.4.1
xlrd: 1.0.0
xlwt: None
xlsxwriter: None
lxml: 3.7.2
bs4: 4.5.3
html5lib: None
httplib2: 0.9.2
apiclient: 1.5.5
sqlalchemy: 1.1.4
pymysql: 0.7.9.None
psycopg2: 2.6.2 (dt dec pq3 ext lo64)
jinja2: 2.9.4
boto: None
pandas_datareader: None

</details>

Inconsistent parsing for timestamp with timezone with read_sql_query
#### Code Sample, a copy-pastable example if possible
I'm creating a test table in postgresql as follows:
```sql
CREATE TABLE test(date timestamptz);

INSERT INTO test VALUES('2014-10-16 07:50:49-04'),('2014-11-06 07:42:06-05')
```
Notice that the dates are across the dst date, so that they have different offsets to utc.

```python
df1 = pd.read_sql_query("SELECT date FROM test", engine,
                        index_col='date')

df2 = pd.read_sql_query("SELECT date FROM test WHERE date<='2014-10-17'",
                        engine,
                        index_col='date')

df2bis = pd.read_sql_query("SELECT date FROM test WHERE date<='2014-10-17'",
                           engine,
                           index_col='date',
                           parse_dates=['date'])
df3 = pd.read_sql_query("SELECT date FROM test", engine,
                        index_col='date',
                        parse_dates=['date'])

```
#### Problem description
```
In [159]: df1.index
Out[159]: 
Index([2014-10-16 07:50:49-04:00, 2014-11-06 07:42:06-05:00], dtype='object', name='date')

In [160]: df2.index
Out[160]: 
DatetimeIndex(['2014-10-16 11:50:49+00:00'], dtype='datetime64[ns, UTC]', name='date', freq=None)

In [162]: df2bis.index
Out[162]: 
DatetimeIndex(['2014-10-16 11:50:49+00:00'], dtype='datetime64[ns, UTC]', name='date', freq=None)

In [163]: df3.index
Out[163]: 
DatetimeIndex(['2014-10-16 11:50:49', '2014-11-06 12:42:06'], dtype='datetime64[ns]', name='date', freq=None)
```
#### Expected Output
The current behavior if I understand correctly is that if there is only one timezone offset: timestamps are parsed automatically to timezone aware timestamps.
If dates are across different offsets, they are not parsed automatically. We can force parsing using parse_dates, but then they are not timezone aware. I can make the conversion using tz_localize, but it's not very consistent. I would expect to have everything parsed automatically to timezone aware timestamps.

This is related to issue #7364. If there is a consensus on what should be done, I can try to make the required changes to make it more regular.

#### Output of ``pd.show_versions()``

<details>
INSTALLED VERSIONS
------------------
commit: None
python: 3.6.0.final.0
python-bits: 64
OS: Linux
OS-release: 4.8.13-1-ARCH
machine: x86_64
processor: 
byteorder: little
LC_ALL: None
LANG: en_US.utf8
LOCALE: en_US.UTF-8

pandas: 0.19.2
nose: 1.3.7
pip: 9.0.1
setuptools: 33.0.0
Cython: 0.25.2
numpy: 1.11.2
scipy: 0.18.1
statsmodels: 0.6.1
xarray: None
IPython: 5.1.0
sphinx: 1.5.1
patsy: 0.4.1
dateutil: 2.6.0
pytz: 2016.10
blosc: None
bottleneck: None
tables: 3.3.0
numexpr: 2.6.1
matplotlib: 2.0.0rc2
openpyxl: 2.4.1
xlrd: 1.0.0
xlwt: None
xlsxwriter: None
lxml: 3.7.2
bs4: 4.5.3
html5lib: None
httplib2: 0.9.2
apiclient: 1.5.5
sqlalchemy: 1.1.4
pymysql: 0.7.9.None
psycopg2: 2.6.2 (dt dec pq3 ext lo64)
jinja2: 2.9.4
boto: None
pandas_datareader: None

</details>



Discussion:
can you create a test using the existing test infrastructure (and either push a commit or paste here)
Working on it. I'll try to come up with something today.
Turns out there is already a test for it, but it's too lenient on purpose: someone obviously thought about this before. Should I try to fix this? I think it should always return datetime64tz, but maybe it's the best that can be done at the moment.

```diff
diff --git a/pandas/io/tests/test_sql.py b/pandas/io/tests/test_sql.py
index cb08944e8..d8eff4ccf 100644
--- a/pandas/io/tests/test_sql.py
+++ b/pandas/io/tests/test_sql.py
@@ -1310,9 +1310,7 @@ class _TestSQLAlchemy(SQLAlchemyMixIn, PandasSQLTest):
         # even with the same versions of psycopg2 & sqlalchemy, possibly a
         # Postgrsql server version difference
         col = df.DateColWithTz
-        self.assertTrue(is_object_dtype(col.dtype) or
-                        is_datetime64_dtype(col.dtype) or
-                        is_datetime64tz_dtype(col.dtype),
+        self.assertTrue(is_datetime64tz_dtype(col.dtype),
                         "DateCol loaded with incorrect type -> {0}"
                         .format(col.dtype))
```
@thrasibule well sometimes we seemed to get object dtypes back and sometimes it was correctly returned as datetime64, so this is lenient on purpose. never could narrow down why sometimes it didn't work..
@thrasibule See also the discussion in https://github.com/pandas-dev/pandas/pull/11216#issuecomment-145132291.

This is a complex topic, so I should dive into it again, but I *think* we want to coerce all tz aware datetimes to UTC (so to keep the tz-aware part, but to have a consistent output). Based on the discussions in #7364 and #11216, I would also think it is this way, but apparently not (at least not in your case).

Certainly welcome to look into it!
I encountered similar problem but on the to_sql() side of parsing

```
import pandas as pd
import sqlite3
import pytz
from datetime import datetime

tz=pytz.timezone('US/Eastern')
# tz=None

conn = sqlite3.connect(':memory:')

# Timezone aware DatetimeIndex raise error
df = pd.DataFrame(
    index=pd.DatetimeIndex(tz=tz, freq='D', start='2017-1-1', periods=0)
    )
df.to_sql('test',conn)

# Timezone aware data in column raise error
df = pd.DataFrame([datetime.now(tz)])
df.to_sql('test',conn)

conn.close()
```
Both section of the above code will raise exception if timezone-aware index/data is used. 

```
Traceback (most recent call last):
  File "/home/ubuntu/project/playground.py", line 16, in <module>
    df.to_sql('test',conn)
  File "/home/ubuntu/project/venv/lib/python3.5/site-packages/pandas/core/generic.py", line 1362, in to_sql
    chunksize=chunksize, dtype=dtype)
  File "/home/ubuntu/project/venv/lib/python3.5/site-packages/pandas/io/sql.py", line 471, in to_sql
    chunksize=chunksize, dtype=dtype)
  File "/home/ubuntu/project/venv/lib/python3.5/site-packages/pandas/io/sql.py", line 1506, in to_sql
    table.insert(chunksize)
  File "/home/ubuntu/project/venv/lib/python3.5/site-packages/pandas/io/sql.py", line 644, in insert
    keys, data_list = self.insert_data()
  File "/home/ubuntu/project/venv/lib/python3.5/site-packages/pandas/io/sql.py", line 625, in insert_data
    d = b.values.astype('M8[us]').astype(object)
  File "/home/ubuntu/project/venv/lib/python3.5/site-packages/pandas/core/indexes/datetimes.py", line 864, in astype
    raise ValueError('Cannot cast DatetimeIndex to dtype %s' % dtype)
ValueError: Cannot cast DatetimeIndex to dtype datetime64[us]
```

Based on #7364 I also agree the correct behavior is to coerce to UTC.
This is also supported by [Sqlite documentation](https://sqlite.org/lang_datefunc.html)

> The date and time functions use UTC or "zulu" time internally, and so the "Z" suffix is a no-op. Any non-zero "HH:MM" suffix is subtracted from the indicated date and time in order to compute zulu time. For example, all of the following time strings are equivalent:
> 
> 2013-10-07 08:23:19.120
> 2013-10-07T08:23:19.120Z
> 2013-10-07 04:23:19.120-04:00
> 2456572.84952685 
can you create a test using the existing test infrastructure (and either push a commit or paste here)
Working on it. I'll try to come up with something today.
Turns out there is already a test for it, but it's too lenient on purpose: someone obviously thought about this before. Should I try to fix this? I think it should always return datetime64tz, but maybe it's the best that can be done at the moment.

```diff
diff --git a/pandas/io/tests/test_sql.py b/pandas/io/tests/test_sql.py
index cb08944e8..d8eff4ccf 100644
--- a/pandas/io/tests/test_sql.py
+++ b/pandas/io/tests/test_sql.py
@@ -1310,9 +1310,7 @@ class _TestSQLAlchemy(SQLAlchemyMixIn, PandasSQLTest):
         # even with the same versions of psycopg2 & sqlalchemy, possibly a
         # Postgrsql server version difference
         col = df.DateColWithTz
-        self.assertTrue(is_object_dtype(col.dtype) or
-                        is_datetime64_dtype(col.dtype) or
-                        is_datetime64tz_dtype(col.dtype),
+        self.assertTrue(is_datetime64tz_dtype(col.dtype),
                         "DateCol loaded with incorrect type -> {0}"
                         .format(col.dtype))
```
@thrasibule well sometimes we seemed to get object dtypes back and sometimes it was correctly returned as datetime64, so this is lenient on purpose. never could narrow down why sometimes it didn't work..
@thrasibule See also the discussion in https://github.com/pandas-dev/pandas/pull/11216#issuecomment-145132291.

This is a complex topic, so I should dive into it again, but I *think* we want to coerce all tz aware datetimes to UTC (so to keep the tz-aware part, but to have a consistent output). Based on the discussions in #7364 and #11216, I would also think it is this way, but apparently not (at least not in your case).

Certainly welcome to look into it!
I encountered similar problem but on the to_sql() side of parsing

```
import pandas as pd
import sqlite3
import pytz
from datetime import datetime

tz=pytz.timezone('US/Eastern')
# tz=None

conn = sqlite3.connect(':memory:')

# Timezone aware DatetimeIndex raise error
df = pd.DataFrame(
    index=pd.DatetimeIndex(tz=tz, freq='D', start='2017-1-1', periods=0)
    )
df.to_sql('test',conn)

# Timezone aware data in column raise error
df = pd.DataFrame([datetime.now(tz)])
df.to_sql('test',conn)

conn.close()
```
Both section of the above code will raise exception if timezone-aware index/data is used. 

```
Traceback (most recent call last):
  File "/home/ubuntu/project/playground.py", line 16, in <module>
    df.to_sql('test',conn)
  File "/home/ubuntu/project/venv/lib/python3.5/site-packages/pandas/core/generic.py", line 1362, in to_sql
    chunksize=chunksize, dtype=dtype)
  File "/home/ubuntu/project/venv/lib/python3.5/site-packages/pandas/io/sql.py", line 471, in to_sql
    chunksize=chunksize, dtype=dtype)
  File "/home/ubuntu/project/venv/lib/python3.5/site-packages/pandas/io/sql.py", line 1506, in to_sql
    table.insert(chunksize)
  File "/home/ubuntu/project/venv/lib/python3.5/site-packages/pandas/io/sql.py", line 644, in insert
    keys, data_list = self.insert_data()
  File "/home/ubuntu/project/venv/lib/python3.5/site-packages/pandas/io/sql.py", line 625, in insert_data
    d = b.values.astype('M8[us]').astype(object)
  File "/home/ubuntu/project/venv/lib/python3.5/site-packages/pandas/core/indexes/datetimes.py", line 864, in astype
    raise ValueError('Cannot cast DatetimeIndex to dtype %s' % dtype)
ValueError: Cannot cast DatetimeIndex to dtype datetime64[us]
```

Based on #7364 I also agree the correct behavior is to coerce to UTC.
This is also supported by [Sqlite documentation](https://sqlite.org/lang_datefunc.html)

> The date and time functions use UTC or "zulu" time internally, and so the "Z" suffix is a no-op. Any non-zero "HH:MM" suffix is subtracted from the indicated date and time in order to compute zulu time. For example, all of the following time strings are equivalent:
> 
> 2013-10-07 08:23:19.120
> 2013-10-07T08:23:19.120Z
> 2013-10-07 04:23:19.120-04:00
> 2456572.84952685 

Resolving patch:
diff --git a/pandas/io/sql.py b/pandas/io/sql.py
--- a/pandas/io/sql.py
+++ b/pandas/io/sql.py
@@ -82,6 +82,17 @@ def _convert_params(sql, params):
     return args
 
 
+def _process_parse_dates_argument(parse_dates):
+    """Process parse_dates argument for read_sql functions"""
+    # handle non-list entries for parse_dates gracefully
+    if parse_dates is True or parse_dates is None or parse_dates is False:
+        parse_dates = []
+
+    elif not hasattr(parse_dates, '__iter__'):
+        parse_dates = [parse_dates]
+    return parse_dates
+
+
 def _handle_date_column(col, utc=None, format=None):
     if isinstance(format, dict):
         return to_datetime(col, errors='ignore', **format)
@@ -96,8 +107,7 @@ def _handle_date_column(col, utc=None, format=None):
         elif is_datetime64tz_dtype(col):
             # coerce to UTC timezone
             # GH11216
-            return (to_datetime(col, errors='coerce')
-                    .astype('datetime64[ns, UTC]'))
+            return to_datetime(col, utc=True)
         else:
             return to_datetime(col, errors='coerce', format=format, utc=utc)
 
@@ -107,27 +117,18 @@ def _parse_date_columns(data_frame, parse_dates):
     Force non-datetime columns to be read as such.
     Supports both string formatted and integer timestamp columns.
     """
-    # handle non-list entries for parse_dates gracefully
-    if parse_dates is True or parse_dates is None or parse_dates is False:
-        parse_dates = []
-
-    if not hasattr(parse_dates, '__iter__'):
-        parse_dates = [parse_dates]
-
-    for col_name in parse_dates:
-        df_col = data_frame[col_name]
-        try:
-            fmt = parse_dates[col_name]
-        except TypeError:
-            fmt = None
-        data_frame[col_name] = _handle_date_column(df_col, format=fmt)
+    parse_dates = _process_parse_dates_argument(parse_dates)
 
-    # we want to coerce datetime64_tz dtypes for now
+    # we want to coerce datetime64_tz dtypes for now to UTC
     # we could in theory do a 'nice' conversion from a FixedOffset tz
     # GH11216
     for col_name, df_col in data_frame.iteritems():
-        if is_datetime64tz_dtype(df_col):
-            data_frame[col_name] = _handle_date_column(df_col)
+        if is_datetime64tz_dtype(df_col) or col_name in parse_dates:
+            try:
+                fmt = parse_dates[col_name]
+            except TypeError:
+                fmt = None
+            data_frame[col_name] = _handle_date_column(df_col, format=fmt)
 
     return data_frame
 
@@ -139,7 +140,7 @@ def _wrap_result(data, columns, index_col=None, coerce_float=True,
     frame = DataFrame.from_records(data, columns=columns,
                                    coerce_float=coerce_float)
 
-    _parse_date_columns(frame, parse_dates)
+    frame = _parse_date_columns(frame, parse_dates)
 
     if index_col is not None:
         frame.set_index(index_col, inplace=True)
@@ -818,17 +819,24 @@ def _harmonize_columns(self, parse_dates=None):
         Datetimes should already be converted to np.datetime64 if supported,
         but here we also force conversion if required.
         """
-        # handle non-list entries for parse_dates gracefully
-        if parse_dates is True or parse_dates is None or parse_dates is False:
-            parse_dates = []
-
-        if not hasattr(parse_dates, '__iter__'):
-            parse_dates = [parse_dates]
+        parse_dates = _process_parse_dates_argument(parse_dates)
 
         for sql_col in self.table.columns:
             col_name = sql_col.name
             try:
                 df_col = self.frame[col_name]
+
+                # Handle date parsing upfront; don't try to convert columns
+                # twice
+                if col_name in parse_dates:
+                    try:
+                        fmt = parse_dates[col_name]
+                    except TypeError:
+                        fmt = None
+                    self.frame[col_name] = _handle_date_column(
+                        df_col, format=fmt)
+                    continue
+
                 # the type the dataframe column should have
                 col_type = self._get_dtype(sql_col.type)
 
@@ -846,16 +854,6 @@ def _harmonize_columns(self, parse_dates=None):
                     if col_type is np.dtype('int64') or col_type is bool:
                         self.frame[col_name] = df_col.astype(
                             col_type, copy=False)
-
-                # Handle date parsing
-                if col_name in parse_dates:
-                    try:
-                        fmt = parse_dates[col_name]
-                    except TypeError:
-                        fmt = None
-                    self.frame[col_name] = _handle_date_column(
-                        df_col, format=fmt)
-
             except KeyError:
                 pass  # this column not in results
 


Test patch:
diff --git a/pandas/tests/io/test_sql.py b/pandas/tests/io/test_sql.py
--- a/pandas/tests/io/test_sql.py
+++ b/pandas/tests/io/test_sql.py
@@ -28,9 +28,8 @@
 
 from datetime import datetime, date, time
 
-from pandas.core.dtypes.common import (
-    is_object_dtype, is_datetime64_dtype,
-    is_datetime64tz_dtype)
+from pandas.core.dtypes.common import (is_datetime64_dtype,
+                                       is_datetime64tz_dtype)
 from pandas import DataFrame, Series, Index, MultiIndex, isna, concat
 from pandas import date_range, to_datetime, to_timedelta, Timestamp
 import pandas.compat as compat
@@ -1356,9 +1355,7 @@ def check(col):
         # even with the same versions of psycopg2 & sqlalchemy, possibly a
         # Postgrsql server version difference
         col = df.DateColWithTz
-        assert (is_object_dtype(col.dtype) or
-                is_datetime64_dtype(col.dtype) or
-                is_datetime64tz_dtype(col.dtype))
+        assert is_datetime64tz_dtype(col.dtype)
 
         df = pd.read_sql_query("select * from types_test_data",
                                self.conn, parse_dates=['DateColWithTz'])