Repository: pandas-dev/pandas

Issue:
BUG: SQL writing can lead to data loss
`to_sql` uses `iterrows` which leads to data conversion, and in the case of mixed data types, can lead to data loss.

``` python
s1 = pd.Series([0],dtype=np.float32)
s2 = pd.Series([2**27 + 1],dtype=np.int32)
df = pd.DataFrame({'float':s1, 'int':s2})

for row in df.iterrows():
    print row[1][1] - (2**27 + 1)
```

(The same issue applies when using `df.to_sql`)

I found the same bug in `to_stata` and have submitted a PR on it.  

`to_sql` is the only other location in pandas the uses `iterrows`, which should probably be avoided in all non-user code.

It should be an easy fix using something like `itertuple` - I don't use the SQL code so I'm not in a good position to fix this problem.



Discussion:
cc @mangecoeur, @hayd, @jorisvandenbossche 

I can see that using `itertuples` instead of `iterrows` can maybe be the better solution, but I don't see how this can lead to data loss. Because the iterrows is used internally in the sql code, so between using iterrows and inserting the data in an sql table, no operations are performed on the data.

It is a fact that the dtype info is lost (and data are maybe unnecessarily converted), but the column type info to write the data to sql is constructed from the column types of the dataframe itself (https://github.com/pydata/pandas/blob/master/pandas/io/sql.py#L484). 
But it is true that the dtypes are not conserved between writing and reading to/from eg postgresql. So eg the example from your tests for to_stata will not work for sql:

```
s1 = pd.Series(2**9, dtype=np.int16)
s2 = pd.Series(2**17, dtype=np.int32)
s3 = pd.Series(2**33, dtype=np.int64)
original = pd.DataFrame({'int16': s1, 'int32': s2, 'int64': s3})
original.index.name = 'index'

formatted = original
formatted['int64'] = formatted['int64'].astype(np.float64)

original.to_sql("test_read_write2", engine)
written_and_read_again = sql.read_table("test_read_write", engine)
```

This will not work for sql, as the resulting types are int64 instead of int16/int32. But this is due to our mapping of the dtypes to sqlalchemy types, where just all int types are mapped to Integer (no distinction between smallint/int/bigint).

But I am not that familiar with the sql conversion code. @mangecoeur, can you take a look at this?

One scenario where this can occur is when a DataFrame contains both `int32` and `float32`.  When you use `iterrows` pandas decides to cast these to `float32` which means that the integer no longer has 32 bits of precision (it now has about 25).

You cannot generate this by mixing `int` data - only by mixing `int` with `float`, and then with either `float32` and `int32` or `float64` and `int64`.

Lets take this for a spin:

``` python
import pandas as pd
import numpy as np

s1 = pd.Series(2**25 + 1,dtype=np.int32)
s2 = pd.Series(0.0,dtype=np.float32)
df = pd.DataFrame({'s1': s1, 's2': s2})

for r in df.iterrows():
    iterrows = list(r[1])

for t in df.df.itertuples():
    itertuples = list(t[1:])

for v in zip(iterrows ,itertuples):
    print v[0] - v[1]
```

The output is

```
-1.0
0.0
```

Which occurs cein the last bit in the `int32` is lost when it is _upcast( sidecast??)_ to `float32`.  pandas should always upcast any `float32` mixed with any `int32` to `float64` which would preserve fidelity.  But the same can happen with `float64` and `int64` -- fixing this would require `float128`, which isn't on offer.

FWIW even storing as `float64` doesn't work since the data-loss occurs when `iterrows` is called.

yep..only shows up on 'bigger' values of ints when mixed....IIRC @bashtage you caught this in to_stata......

easy fix is to user `itertuples` (which doesn't coerce to a uniform series)

other way is to use a `dtype='object'` Series (maybe as a kw to `iterrows`). though I think `itertuples` might be faster anyhow

the problem with `iterrows` generally is that it calls `self.values` which by definition tries to cast everything to a compatible type (where compatible doesn't take into account precision), just a type that will 'hold' everything

@bashtage Thanks for the clarification! Now I see, and I quickly tried using itertuples and this fixes this indeed (not the int32->int64 type conversion, but the data loss)

I can do a PR. The other question remains, whether we should try to preserve more of the dtypes (using smallint/int/bigint).

I tried to fix this on my side, also in the hope of more efficient inserts - see https://github.com/mangecoeur/pandas/tree/sql-dataloss-6509 but I'm getting a lot of test failiures for some reason, something to do with indexes being included or not, is anyone else seeing this after changing to itertuples?


Resolving patch:
diff --git a/pandas/io/sql.py b/pandas/io/sql.py
--- a/pandas/io/sql.py
+++ b/pandas/io/sql.py
@@ -423,16 +423,17 @@ def insert(self):
         ins = self.insert_statement()
         data_list = []
         # to avoid if check for every row
+        keys = self.frame.columns
         if self.index is not None:
-            for t in self.frame.iterrows():
+            for t in self.frame.itertuples():
                 data = dict((k, self.maybe_asscalar(v))
-                            for k, v in t[1].iteritems())
+                            for k, v in zip(keys, t[1:]))
                 data[self.index] = self.maybe_asscalar(t[0])
                 data_list.append(data)
         else:
-            for t in self.frame.iterrows():
+            for t in self.frame.itertuples():
                 data = dict((k, self.maybe_asscalar(v))
-                            for k, v in t[1].iteritems())
+                            for k, v in zip(keys, t[1:]))
                 data_list.append(data)
         self.pd_sql.execute(ins, data_list)
 
@@ -758,8 +759,8 @@ def insert_statement(self):
     def insert(self):
         ins = self.insert_statement()
         cur = self.pd_sql.con.cursor()
-        for r in self.frame.iterrows():
-            data = [self.maybe_asscalar(v) for v in r[1].values]
+        for r in self.frame.itertuples():
+            data = [self.maybe_asscalar(v) for v in r[1:]]
             if self.index is not None:
                 data.insert(0, self.maybe_asscalar(r[0]))
             cur.execute(ins, tuple(data))


Test patch:
diff --git a/pandas/io/tests/test_sql.py b/pandas/io/tests/test_sql.py
--- a/pandas/io/tests/test_sql.py
+++ b/pandas/io/tests/test_sql.py
@@ -7,7 +7,7 @@
 import nose
 import numpy as np
 
-from pandas import DataFrame
+from pandas import DataFrame, Series
 from pandas.compat import range, lrange, iteritems
 #from pandas.core.datetools import format as date_format
 
@@ -554,6 +554,18 @@ def test_date_parsing(self):
         self.assertTrue(issubclass(df.IntDateCol.dtype.type, np.datetime64),
                         "IntDateCol loaded with incorrect type")
 
+    def test_mixed_dtype_insert(self):
+        # see GH6509
+        s1 = Series(2**25 + 1,dtype=np.int32)
+        s2 = Series(0.0,dtype=np.float32)
+        df = DataFrame({'s1': s1, 's2': s2})
+
+        # write and read again
+        df.to_sql("test_read_write", self.conn, index=False)
+        df2 = sql.read_table("test_read_write", self.conn)
+
+        tm.assert_frame_equal(df, df2, check_dtype=False, check_exact=True)
+
 
 class TestSQLAlchemy(_TestSQLAlchemy):
     """
diff --git a/pandas/util/testing.py b/pandas/util/testing.py
--- a/pandas/util/testing.py
+++ b/pandas/util/testing.py
@@ -499,12 +499,18 @@ def is_sorted(seq):
 def assert_series_equal(left, right, check_dtype=True,
                         check_index_type=False,
                         check_series_type=False,
-                        check_less_precise=False):
+                        check_less_precise=False,
+                        check_exact=False):
     if check_series_type:
         assert_isinstance(left, type(right))
     if check_dtype:
         assert_attr_equal('dtype', left, right)
-    assert_almost_equal(left.values, right.values, check_less_precise)
+    if check_exact:
+        if not np.array_equal(left.values, right.values):
+            raise AssertionError('{0} is not equal to {1}.'.format(left.values,
+                                                                   right.values))
+    else:
+        assert_almost_equal(left.values, right.values, check_less_precise)
     if check_less_precise:
         assert_almost_equal(
             left.index.values, right.index.values, check_less_precise)
@@ -522,7 +528,8 @@ def assert_frame_equal(left, right, check_dtype=True,
                        check_frame_type=False,
                        check_less_precise=False,
                        check_names=True,
-                       by_blocks=False):
+                       by_blocks=False,
+                       check_exact=False):
     if check_frame_type:
         assert_isinstance(left, type(right))
     assert_isinstance(left, DataFrame)
@@ -555,7 +562,8 @@ def assert_frame_equal(left, right, check_dtype=True,
             assert_series_equal(lcol, rcol,
                                 check_dtype=check_dtype,
                                 check_index_type=check_index_type,
-                                check_less_precise=check_less_precise)
+                                check_less_precise=check_less_precise,
+                                check_exact=check_exact)
 
     if check_index_type:
         assert_isinstance(left.index, type(right.index))