Repository: pandas-dev/pandas

Issue:
BUG: error in handling a sqlalchemy type with arguments (instantiated type, not class)
I'm trying to use DataFrame().to_sql to input a time aware dataframe series. Here is an example of my code.

```
times = ['201412120154', '201412110254']

df = pd.DataFrame()
df['time'] = pd.to_datetime(times, utc=True)

df.time.to_sql('test', engine, dtype={'time': sqlalchemy.TIMESTAMP(timezone=True)})
```

The error I recieve is:

```
TypeError: issubclass() arg 1 must be a class
```

The following code works but obviously results in a postgresql column that is not timezone aware.

```
times = ['201412120154', '201412110254']

df = pd.DataFrame()
df['time'] = pd.to_datetime(times, utc=True)

df.time.to_sql('test', engine, dtype={'time': sqlalchemy.TIMESTAMP})
```

I'm using python 2.7, pandas 0.15.2, postsgresql 9.3 and SQLAlchemy 0.9.7. This same issue also occurs with sqlalchemy.DATETIME(timezone=True) vs sqlalchemy.DATETIME

Full Traceback:

```
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-161-ec79a553e6d0> in <module>()
----> 1 df.time.to_sql('test', w.engine, dtype={'time': sqlalchemy.TIMESTAMP(timezone=True)})

/home/stew/.pyenv/versions/p276/lib/python2.7/site-packages/pandas/core/generic.pyc in to_sql(self, name, con, flavor, schema, if_exists, index, index_label, chunksize, dtype)
    964             self, name, con, flavor=flavor, schema=schema, if_exists=if_exists,   
    965             index=index, index_label=index_label, chunksize=chunksize,
--> 966             dtype=dtype)
    967 
    968     def to_pickle(self, path):

/home/stew/.pyenv/versions/p276/lib/python2.7/site-packages/pandas/io/sql.pyc in to_sql(frame, name, con, flavor, schema, if_exists, index, index_label, chunksize, dtype)
    536     pandas_sql.to_sql(frame, name, if_exists=if_exists, index=index,
    537                       index_label=index_label, schema=schema,
--> 538                       chunksize=chunksize, dtype=dtype)
    539 
    540 

/home/stew/.pyenv/versions/p276/lib/python2.7/site-packages/pandas/io/sql.pyc in to_sql(self, frame, name, if_exists, index, index_label, schema, chunksize, dtype)
   1162             import sqlalchemy.sql.type_api as type_api
   1163             for col, my_type in dtype.items():
-> 1164                 if not issubclass(my_type, type_api.TypeEngine):
   1165                     raise ValueError('The type of %s is not a SQLAlchemy '
   1166                                      'type ' % col)

TypeError: issubclass() arg 1 must be a class
```



Discussion:
@pilotstew Thanks for the report!

There appear to be a few problems:
- first, the fact that the `dtype` argument does not work is an oversight. This feature was recently introduced (#8926), and only tested with sqlalchemy types (as `sqlalchemy.TIMESTAMP`) without providing more arguments to it (as `sqlalchemy.TIMESTAMP(timezone=True)`). So this should be fixed.
- second, this should work out of the box I think. There is even code for this (https://github.com/pydata/pandas/blob/v0.15.2/pandas/io/sql.py#L915), but this is not working because when you have a timezone, pandas now treats it as an object column of Timestamp objects (converted to `TEXT`), because `datetime64` dtype does not support timezones.

@pilotstew You can solve it for now using this monkey-patch:

```
from pandas.io.sql import SQLTable

def to_sql(self, frame, name, if_exists='fail', index=True,
           index_label=None, schema=None, chunksize=None, dtype=None):
    """
    patched version of https://github.com/pydata/pandas/blob/v0.15.2/pandas/io/sql.py#L1129
    """
    if dtype is not None:
        from sqlalchemy.types import to_instance, TypeEngine
        for col, my_type in dtype.items():
            if not isinstance(to_instance(my_type), TypeEngine):
                raise ValueError('The type of %s is not a SQLAlchemy '
                                 'type ' % col)

    table = SQLTable(name, self, frame=frame, index=index,
                     if_exists=if_exists, index_label=index_label,
                     schema=schema, dtype=dtype)
    table.create()
    table.insert(chunksize)
    # check for potentially case sensitivity issues (GH7815)
    if name not in self.engine.table_names(schema=schema or self.meta.schema):
        warnings.warn("The provided table name '{0}' is not found exactly "
                      "as such in the database after writing the table, "
                      "possibly due to case sensitivity issues. Consider "
                      "using lower case table names.".format(name), UserWarning)

pd.io.sql.SQLDatabase.to_sql = to_sql
```

Can you try that? This should let you be able to do `df.time.to_sql('test', engine, dtype={'time': sqlalchemy.TIMESTAMP(timezone=True)})`

Worked Perfectly!

@pilotstew Good to hear! 
Did you also get the correct time zone in the database? (or are you in the same timezone as your data?) see #9086

Actually, no I didn't. In pandas all timestamps were UTC and converted to local timezone in postgres. Since they are all uniform and converted correctly it's not a huge issue. I can convert back to UTC on the read_sql. 

OK, yes that is what I expected and described in #9086. So this is then another enhacement to do in 'real' timezone support!

I opened a couple of other issues that I encountered while looking into this:
- #9084: ENH: support index=True for SQL io.sql.get_schema Enhancement Good as first PR SQL
- #9085: BUG: Timestamps in object column not written as DateTime data in to_sql SQL
- #9086: ENH/SQL: support writing timestamps with timezone in to_sql Enhancement SQL Timezones
- #9087: SQL: test with oldest supported sqlalchemy SQL Testing

Lets keep this issue for the actual `dtype` problem of only accepting classes, and not instances.


Resolving patch:
diff --git a/pandas/io/sql.py b/pandas/io/sql.py
--- a/pandas/io/sql.py
+++ b/pandas/io/sql.py
@@ -1159,9 +1159,9 @@ def to_sql(self, frame, name, if_exists='fail', index=True,
 
         """
         if dtype is not None:
-            import sqlalchemy.sql.type_api as type_api
+            from sqlalchemy.types import to_instance, TypeEngine
             for col, my_type in dtype.items():
-                if not issubclass(my_type, type_api.TypeEngine):
+                if not isinstance(to_instance(my_type), TypeEngine):
                     raise ValueError('The type of %s is not a SQLAlchemy '
                                      'type ' % col)
 


Test patch:
diff --git a/pandas/io/tests/test_sql.py b/pandas/io/tests/test_sql.py
--- a/pandas/io/tests/test_sql.py
+++ b/pandas/io/tests/test_sql.py
@@ -1229,6 +1229,14 @@ def test_dtype(self):
         self.assertRaises(ValueError, df.to_sql,
                           'error', self.conn, dtype={'B': str})
 
+        # GH9083
+        df.to_sql('dtype_test3', self.conn, dtype={'B': sqlalchemy.String(10)})
+        meta.reflect()
+        sqltype = meta.tables['dtype_test3'].columns['B'].type
+        print(sqltype)
+        self.assertTrue(isinstance(sqltype, sqlalchemy.String))
+        self.assertEqual(sqltype.length, 10)
+
     def test_notnull_dtype(self):
         cols = {'Bool': Series([True,None]),
                 'Date': Series([datetime(2012, 5, 1), None]),