Repository: pandas-dev/pandas

Issue:
problem with to_sql with NA
i am experiencing issues with writing NA values for a column of boolean with missing values.

``` python
data = [True, None, False]
df = pd.DataFrame(a, columns=['test'])

df.info()

<class 'pandas.core.frame.DataFrame'>
Int64Index: 3 entries, 0 to 2
Data columns (total 1 columns):
test    2 non-null object
dtypes: object(1)
memory usage: 48.0 bytes
```

I am reading data from a mssql server and am trying to write it unchanged in another table.
As the data contains NULL values, pandas changes the dtype to object.  when calling to_sql, object fields seem to be mapped to a "text" column type, and the export then fails:

``` python

df.to_sql('test_table', engine_local, if_exists='append', index=True)

DataError: (DataError) ('22018', '[22018] [Microsoft][ODBC SQL Server Driver][SQL Server]Operand type clash: bit is incompatible with text (206) (SQLExecDirectW); [42000] [Microsoft][ODBC SQL Server Driver][SQL Server]Statement(s) could not be prepared. (8180)') u'INSERT INTO test_table ([index], test) VALUES (?, ?)' ((0L, True), (1L, None), (2L, False))
```

'object' dtypes are tricky to handle systematically I guess, but maybe we could add the ability to tell pandas how to handle it on a column by column basis?.

something like:

``` python
df.to_sql('test_table', engine_local, if_exists='append', index=True, col_types={'test': bool})
```

could do the trick?



Discussion:
I think this would be a reasonable addition to the api (a `dtype` kwarg in `to_sql`). A way to override the default sqlalchemy type that is used per column. That would enable to workaround such cornercases as above.

comment from @artemyk in #8926:

>  What if instead of having to specify the types as in this PR, we iterated over the columns with dtype==object, and did pandas.core.common._infer_dtype_from_scalar on their first non-null entry?

I think the problem with that is that the series can also be really mixed (and not only because of the nan's), and then the checking on one scalar can give wrong results. 
But maybe it would be interesting to have some kind of `infer` function that does not take into account NaNs to determine if something is mixed or not. So that the following would return 'boolean' instead of 'mixed':

```
In [61]: s = pd.Series([False, True, np.nan])

In [62]: pd.lib.infer_dtype(s)
Out[62]: 'mixed'
```

@jorisvandenbossche Yes, I think that would be very useful (though perhaps the _real_ solution would be to have typed NA values --- I suppose that would require some big structural changes though).  I can see how having the `dtypes` parameter can be good for corner cases, and for solving this issue now, but it would be great if (at least in most cases) NAs could be inserted w.o having to specify dtypes.

@jorisvandenbossche I looked into how `DataFrame` infers types and it seems like it uses `numpy.array` inference for the most part.  We could use something similar, or, perhaps more robustly,

```
def infer_notnull_dtype(series):
    if series.dtype != 'object':
        return series.dtype
    else:
        s = pd.Series(series[~series.isnull()].tolist())
        return s.dtype
```

Not sure how slow this would be for large dataframes.

The `tolist` will certainly be not that good for large dataframes I think ... 
But you have the `lib.infer_dtype` for this, and maybe just providing series[~series.isnull()] to that can already do the trick?

```
In [1]: s = pd.Series([False, True, np.nan])

In [2]: pd.lib.infer_dtype(s)
Out[2]: 'mixed'

In [3]: pd.lib.infer_dtype(s[s.notnull()])
Out[3]: 'boolean'
```

Hmm, this seems certainly like something that can be added to the type interference just before the last step when for object `Text` is returned.

@jorisvandenbossche yep, much better. In fact the only series you really need to infer is a mixed-type boolean I think. (e.g. if its object, then infer it; check if its mixed, then remove nans and reinfer).

@jorisvandenbossche Oh, nice!  Didn't realize that `lib.infer_dtype` is something that already exists.  I could probably do the patch, though would be easier to wait until #8926 is merged.

A little weird though since `infer_dtype` returns different named types than numpy types (the ones we test for in `sql.py`). E.g. 'boolean' vs 'bool', 'integer' vs 'int', etc..  Is there an existing way to standardize these?  Or just setup checks for both variants?

@artemyk these are 'specific' to pandas and not real 'types'. These are to try to figure out things when given unknown input. These are purely internal.

Going to keep this open for now, given the discussion above


Resolving patch:
diff --git a/pandas/io/sql.py b/pandas/io/sql.py
--- a/pandas/io/sql.py
+++ b/pandas/io/sql.py
@@ -885,37 +885,56 @@ def _harmonize_columns(self, parse_dates=None):
             except KeyError:
                 pass  # this column not in results
 
+    def _get_notnull_col_dtype(self, col):
+        """
+        Infer datatype of the Series col.  In case the dtype of col is 'object' 
+        and it contains NA values, this infers the datatype of the not-NA 
+        values.  Needed for inserting typed data containing NULLs, GH8778.
+        """
+        col_for_inference = col
+        if col.dtype == 'object':
+            notnulldata = col[~isnull(col)]
+            if len(notnulldata):
+                col_for_inference = notnulldata
+
+        return lib.infer_dtype(col_for_inference)
+
     def _sqlalchemy_type(self, col):
-        from sqlalchemy.types import (BigInteger, Float, Text, Boolean,
-            DateTime, Date, Time)
 
         dtype = self.dtype or {}
         if col.name in dtype:
             return self.dtype[col.name]
 
-        if com.is_datetime64_dtype(col):
+        col_type = self._get_notnull_col_dtype(col)
+
+        from sqlalchemy.types import (BigInteger, Float, Text, Boolean,
+            DateTime, Date, Time)
+
+        if col_type == 'datetime64': 
             try:
                 tz = col.tzinfo
                 return DateTime(timezone=True)
             except:
                 return DateTime
-        if com.is_timedelta64_dtype(col):
+        if col_type == 'timedelta64':
             warnings.warn("the 'timedelta' type is not supported, and will be "
                           "written as integer values (ns frequency) to the "
                           "database.", UserWarning)
             return BigInteger
-        elif com.is_float_dtype(col):
+        elif col_type == 'floating':
             return Float
-        elif com.is_integer_dtype(col):
+        elif col_type == 'integer':
             # TODO: Refine integer size.
             return BigInteger
-        elif com.is_bool_dtype(col):
+        elif col_type == 'boolean':
             return Boolean
-        inferred = lib.infer_dtype(com._ensure_object(col))
-        if inferred == 'date':
+        elif col_type == 'date':
             return Date
-        if inferred == 'time':
+        elif col_type == 'time':
             return Time
+        elif col_type == 'complex':
+            raise ValueError('Complex datatypes not supported')
+
         return Text
 
     def _numpy_type(self, sqltype):
@@ -1187,15 +1206,15 @@ def _create_sql_schema(self, frame, table_name, keys=None):
 # SQLAlchemy installed
 # SQL type convertions for each DB
 _SQL_TYPES = {
-    'text': {
+    'string': {
         'mysql': 'VARCHAR (63)',
         'sqlite': 'TEXT',
     },
-    'float': {
+    'floating': {
         'mysql': 'FLOAT',
         'sqlite': 'REAL',
     },
-    'int': {
+    'integer': {
         'mysql': 'BIGINT',
         'sqlite': 'INTEGER',
     },
@@ -1211,12 +1230,13 @@ def _create_sql_schema(self, frame, table_name, keys=None):
         'mysql': 'TIME',
         'sqlite': 'TIME',
     },
-    'bool': {
+    'boolean': {
         'mysql': 'BOOLEAN',
         'sqlite': 'INTEGER',
     }
 }
 
+
 # SQL enquote and wildcard symbols
 _SQL_SYMB = {
     'mysql': {
@@ -1291,8 +1311,8 @@ def _create_table_setup(self):
         br_l = _SQL_SYMB[flv]['br_l']  # left val quote char
         br_r = _SQL_SYMB[flv]['br_r']  # right val quote char
 
-        create_tbl_stmts = [(br_l + '%s' + br_r + ' %s') % (cname, ctype)
-                            for cname, ctype, _ in column_names_and_types]
+        create_tbl_stmts = [(br_l + '%s' + br_r + ' %s') % (cname, col_type)
+                            for cname, col_type, _ in column_names_and_types]
         if self.keys is not None and len(self.keys):
             cnames_br = ",".join([br_l + c + br_r for c in self.keys])
             create_tbl_stmts.append(
@@ -1317,30 +1337,27 @@ def _sql_type_name(self, col):
         dtype = self.dtype or {}
         if col.name in dtype:
             return dtype[col.name]
-        pytype = col.dtype.type
-        pytype_name = "text"
-        if issubclass(pytype, np.floating):
-            pytype_name = "float"
-        elif com.is_timedelta64_dtype(pytype):
+
+        col_type = self._get_notnull_col_dtype(col)
+        if col_type == 'timedelta64':
             warnings.warn("the 'timedelta' type is not supported, and will be "
                           "written as integer values (ns frequency) to the "
                           "database.", UserWarning)
-            pytype_name = "int"
-        elif issubclass(pytype, np.integer):
-            pytype_name = "int"
-        elif issubclass(pytype, np.datetime64) or pytype is datetime:
-            # Caution: np.datetime64 is also a subclass of np.number.
-            pytype_name = "datetime"
-        elif issubclass(pytype, np.bool_):
-            pytype_name = "bool"
-        elif issubclass(pytype, np.object):
-            pytype = lib.infer_dtype(com._ensure_object(col))
-            if pytype == "date":
-                pytype_name = "date"
-            elif pytype == "time":
-                pytype_name = "time"
-
-        return _SQL_TYPES[pytype_name][self.pd_sql.flavor]
+            col_type = "integer"
+
+        elif col_type == "datetime64":
+            col_type = "datetime"
+
+        elif col_type == "empty":
+            col_type = "string"
+
+        elif col_type == "complex":
+            raise ValueError('Complex datatypes not supported')
+            
+        if col_type not in _SQL_TYPES:
+            col_type = "string"
+
+        return _SQL_TYPES[col_type][self.pd_sql.flavor]
 
 
 class SQLiteDatabase(PandasSQL):


Test patch:
diff --git a/pandas/io/tests/test_sql.py b/pandas/io/tests/test_sql.py
--- a/pandas/io/tests/test_sql.py
+++ b/pandas/io/tests/test_sql.py
@@ -560,6 +560,11 @@ def test_timedelta(self):
         result = sql.read_sql_query('SELECT * FROM test_timedelta', self.conn)
         tm.assert_series_equal(result['foo'], df['foo'].astype('int64'))
 
+    def test_complex(self):
+        df = DataFrame({'a':[1+1j, 2j]})
+        # Complex data type should raise error
+        self.assertRaises(ValueError, df.to_sql, 'test_complex', self.conn)
+
     def test_to_sql_index_label(self):
         temp_frame = DataFrame({'col1': range(4)})
 
@@ -1175,19 +1180,38 @@ def test_dtype(self):
                 (0.9, None)]
         df = DataFrame(data, columns=cols)
         df.to_sql('dtype_test', self.conn)
-        df.to_sql('dtype_test2', self.conn, dtype={'B': sqlalchemy.Boolean})
+        df.to_sql('dtype_test2', self.conn, dtype={'B': sqlalchemy.TEXT})
+        meta = sqlalchemy.schema.MetaData(bind=self.conn)
+        meta.reflect()
+        sqltype = meta.tables['dtype_test2'].columns['B'].type
+        self.assertTrue(isinstance(sqltype, sqlalchemy.TEXT))
+        self.assertRaises(ValueError, df.to_sql,
+                          'error', self.conn, dtype={'B': str})
+
+    def test_notnull_dtype(self):
+        cols = {'Bool': Series([True,None]),
+                'Date': Series([datetime(2012, 5, 1), None]),
+                'Int' : Series([1, None], dtype='object'),
+                'Float': Series([1.1, None])
+               }
+        df = DataFrame(cols)
+
+        tbl = 'notnull_dtype_test'
+        df.to_sql(tbl, self.conn)
+        returned_df = sql.read_sql_table(tbl, self.conn)
         meta = sqlalchemy.schema.MetaData(bind=self.conn)
         meta.reflect()
-        self.assertTrue(isinstance(meta.tables['dtype_test'].columns['B'].type,
-                                   sqltypes.TEXT))
         if self.flavor == 'mysql':
             my_type = sqltypes.Integer
         else:
             my_type = sqltypes.Boolean
-        self.assertTrue(isinstance(meta.tables['dtype_test2'].columns['B'].type,
-                                   my_type))
-        self.assertRaises(ValueError, df.to_sql,
-                          'error', self.conn, dtype={'B': bool})
+
+        col_dict = meta.tables[tbl].columns
+
+        self.assertTrue(isinstance(col_dict['Bool'].type, my_type))
+        self.assertTrue(isinstance(col_dict['Date'].type, sqltypes.DateTime))
+        self.assertTrue(isinstance(col_dict['Int'].type, sqltypes.Integer))
+        self.assertTrue(isinstance(col_dict['Float'].type, sqltypes.Float))
 
 
 class TestSQLiteAlchemy(_TestSQLAlchemy):
@@ -1507,6 +1531,13 @@ def test_to_sql_save_index(self):
     def test_transactions(self):
         self._transaction_test()
 
+    def _get_sqlite_column_type(self, table, column):
+        recs = self.conn.execute('PRAGMA table_info(%s)' % table)
+        for cid, name, ctype, not_null, default, pk in recs:
+            if name == column:
+                return ctype
+        raise ValueError('Table %s, column %s not found' % (table, column))
+
     def test_dtype(self):
         if self.flavor == 'mysql':
             raise nose.SkipTest('Not applicable to MySQL legacy')
@@ -1515,20 +1546,35 @@ def test_dtype(self):
                 (0.9, None)]
         df = DataFrame(data, columns=cols)
         df.to_sql('dtype_test', self.conn)
-        df.to_sql('dtype_test2', self.conn, dtype={'B': 'bool'})
+        df.to_sql('dtype_test2', self.conn, dtype={'B': 'STRING'})
 
-        def get_column_type(table, column):
-            recs = self.conn.execute('PRAGMA table_info(%s)' % table)
-            for cid, name, ctype, not_null, default, pk in recs:
-                if name == column:
-                    return ctype
-            raise ValueError('Table %s, column %s not found' % (table, column))
-
-        self.assertEqual(get_column_type('dtype_test', 'B'), 'TEXT')
-        self.assertEqual(get_column_type('dtype_test2', 'B'), 'bool')
+        # sqlite stores Boolean values as INTEGER
+        self.assertEqual(self._get_sqlite_column_type('dtype_test', 'B'), 'INTEGER')
+        
+        self.assertEqual(self._get_sqlite_column_type('dtype_test2', 'B'), 'STRING')
         self.assertRaises(ValueError, df.to_sql,
                           'error', self.conn, dtype={'B': bool})
 
+    def test_notnull_dtype(self):
+        if self.flavor == 'mysql':
+            raise nose.SkipTest('Not applicable to MySQL legacy')
+            
+        cols = {'Bool': Series([True,None]),
+                'Date': Series([datetime(2012, 5, 1), None]),
+                'Int' : Series([1, None], dtype='object'),
+                'Float': Series([1.1, None])
+               }
+        df = DataFrame(cols)
+
+        tbl = 'notnull_dtype_test'
+        df.to_sql(tbl, self.conn)
+
+        self.assertEqual(self._get_sqlite_column_type(tbl, 'Bool'), 'INTEGER')
+        self.assertEqual(self._get_sqlite_column_type(tbl, 'Date'), 'TIMESTAMP')
+        self.assertEqual(self._get_sqlite_column_type(tbl, 'Int'), 'INTEGER')
+        self.assertEqual(self._get_sqlite_column_type(tbl, 'Float'), 'REAL')
+
+
 class TestMySQLLegacy(TestSQLiteFallback):
     """
     Test the legacy mode against a MySQL database.