Repository: pandas-dev/pandas

Issue:
Choose correct datatype when creating new dataframe column from old ones
I am converting a Stata dataset to a dataframe, then multiply two columns to create a third one. The chosen datatype for the two columns is int8 and for some reason the new column blindly follows that.

For instance, this code

``` Python
df = pd.read_stata(file)
df['w_age_educ'] = df['w_age'] * df['weduc']
print(df[['w_age', 'weduc', 'w_age_educ']].dtypes)
print(df[['w_age', 'weduc', 'w_age_educ']][:3])
```

would give me

``` Python
w_age         int8
weduc         int8
w_age_educ    int8
dtype: object
   w_age  weduc  w_age_educ
0     44     14         104
1     34     13         -70
2     33     18          82
```

Is this a bug or intended behavior? If the latter, how can I get my desired product column?



Discussion:
can u show df.info() right after the read from stata

pd.show_versions() as well

``` Python
df = pd.read_stata(file)
df.info()
pd.show_versions()
df['w_age_educ'] = df['w_age'] * df['weduc']
print(df[['w_age', 'weduc', 'w_age_educ']].dtypes)
print(df[['w_age', 'weduc', 'w_age_educ']][:3])
```

yields

``` Python
<class 'pandas.core.frame.DataFrame'>
Int64Index: 4066 entries, 0 to 4065
Data columns (total 62 columns):
state                4066 non-null int8
hhnum                4066 non-null int32
cbsa_size            4066 non-null int8
famnum               4066 non-null int8
himm                 4066 non-null int8
hgen2                4066 non-null int8
heduc                4066 non-null int8
htwage               3789 non-null float32
hannhrs              4066 non-null int16
hfamearn             4066 non-null int32
h_age                4066 non-null int8
hrace                4066 non-null int8
hwagesal             4066 non-null int32
hwage_ogr            3281 non-null float32
hselfinc             4066 non-null int32
hfarminc             4066 non-null int32
hclassly             4066 non-null int8
hhealth              4066 non-null int8
hweeksly             4066 non-null int8
hhrswkly             4066 non-null int8
hogrflag             4066 non-null int8
hrgroup              4066 non-null int8
ownkidsu6            4066 non-null int8
ownkidsu18           4066 non-null int8
wimm                 4066 non-null int8
wgen2                4066 non-null int8
weduc                4066 non-null int8
wtwage               3265 non-null float32
wannhrs              4066 non-null int16
wfamearn             4066 non-null int32
w_age                4066 non-null int8
wrace                4066 non-null int8
wwagesal             4066 non-null int32
wwage_ogr            2906 non-null float32
wselfinc             4066 non-null int32
wfarminc             4066 non-null int32
wclassly             4066 non-null int8
wweeksly             4066 non-null int8
whrswkly             4066 non-null int8
whealth              4066 non-null int8
wogrflag             4066 non-null int8
wrgroup              4066 non-null int8
mergestat            4066 non-null object
wifework             4066 non-null int8
husbandwork          4066 non-null int8
age_gap              4066 non-null int8
wagesalgap           4066 non-null int32
educgap              4066 non-null int8
hlogwage             3789 non-null float32
wlogwage             3265 non-null float32
hexp                 4066 non-null int8
hexp2                4066 non-null float32
hexp3                4066 non-null float32
hblack               4066 non-null int8
wblack               4066 non-null int8
hhispanic            4066 non-null int8
whispanic            4066 non-null int8
wife_wage_ogr        4066 non-null int8
husband_wage_ogr     4066 non-null int8
missing_wwage_ogr    4066 non-null int8
wlogwage_ogr         2906 non-null float32
any_under6           4066 non-null int8
dtypes: float32(9), int16(2), int32(10), int8(40), object(1)
INSTALLED VERSIONS
------------------
commit: None
python: 3.3.2.final.0
python-bits: 32
OS: Linux
OS-release: 3.16.3-200.fc20.i686
machine: i686
processor: i686
byteorder: little
LC_ALL: None
LANG: en_US.utf8

pandas: 0.14.1
nose: 1.3.4
Cython: 0.20.2
numpy: 1.9.0
scipy: 0.12.1
statsmodels: 0.5.0
IPython: 2.3.0
sphinx: 1.2.3
patsy: 0.3.0
scikits.timeseries: None
dateutil: 2.2
pytz: 2014.7
bottleneck: None
tables: None
numexpr: None
matplotlib: 1.3.1
openpyxl: None
xlrd: None
xlwt: None
xlsxwriter: None
lxml: None
bs4: None
html5lib: None
httplib2: None
apiclient: None
rpy2: 2.4.4
sqlalchemy: None
pymysql: None
psycopg2: None
w_age         int8
weduc         int8
w_age_educ    int8
dtype: object
   w_age  weduc  w_age_educ
0     44     14         104
1     34     13         -70
2     33     18          82
```

cc @bashtage

do u explicitly coerce stata types ?

Yes, it attempts to preserve Stata data types for two reasons.  First if a Stats data set is large, blowing up int8 to int64 is expensive.   Second, preserving types helps ensure data fidelity under round trips (except under some conditions, e.g. no missing values)

On Oct 10, 2014 6:20 AM, jreback notifications@github.com wrote:

cc @bashtagehttps://github.com/bashtage

do u explicitly coerce stata types ?

—
Reply to this email directly or view it on GitHubhttps://github.com/pydata/pandas/issues/8527#issuecomment-58637559.

@bashtage thanks - I do recall that from the docs

For clarity there is no coercion since the raw data is directly read into a structured array with the Stata data types.  The old version would first create Int64 and then convert to smaller types, which meant that the memory requirement could be temporarily high.  This is no longer the case.

On Oct 10, 2014 6:36 AM, jreback notifications@github.com wrote:

@bashtagehttps://github.com/bashtage thanks - I do recall that from the docs

—
Reply to this email directly or view it on GitHubhttps://github.com/pydata/pandas/issues/8527#issuecomment-58639033.

unfortunately this is fundamental a numpy issue (and no fidling with `np.seterr` could make this raise overflow).

```
In [3]: np.array([1,2,3],dtype='int8')*100
Out[3]: array([100, -56,  44], dtype=int8)

In [4]: np.array([1,2,3],dtype='int16')*100
Out[4]: array([100, 200, 300], dtype=int16)
```

so couple of options:
- easy workaround is to simply coerce to pandas standard float64/int64:

```
In [22]:  df = DataFrame({'A' : np.array([1,2,3],dtype='int8'), 'B' : np.array([1,2,3],dtype='float32'), 'C' : 'foo'})

In [23]: df
Out[23]: 
   A  B    C
0  1  1  foo
1  2  2  foo
2  3  3  foo

In [24]: df.dtypes
Out[24]: 
A       int8
B    float32
C     object
dtype: object

In [26]: columns = df.select_dtypes(['integer']).columns

In [27]: df[columns] = df[columns].astype('int64')

In [28]: columns = df.select_dtypes(['floating']).columns

In [30]: df[columns] = df[columns].astype('float64')

In [31]: df.dtypes
Out[31]: 
A      int64
B    float64
C     object
dtype: object
```
- maybe add something like this to `.astype('upcast')` or something which does the above more automatically
- add an option to `read_stata` to `convert_dtypes` (and I would make this True by default I think)

Calling it a numpy issue is a little unfair to numpy.  It is a hardware issue since int addition doesn't check for overflow, -- to do so would be a real performance killer.

Your second comment indicates that there is nothing like `pd.to_safest_dtypes` which will upcast all ints to `int64` and all floats to `float64`?

ok not numpy's fault per-se. agreed overflow is an issue (but I thought I could get it check somehow). In any event.

Normally it is not necessary to coerce to 'safe' types as things are simply created that way. That said, I get why you are passing the stata types thru. Thus it is passed to the user.

I think if we allow `.astype('safe')` or `.astype('upcast')` makes the most sense.

FYI their is another issue out theri to add dict support to `.astype({ column_a : 'float64', column_b : 'int64' })` so this would work with that as well.

@djrobust @bashtage either of you interested in doing this?

I think its only `integer/floating` types that need this kind of conversion.

What do other io tools do that read binary data with a standard type?  For example, `read_sql`?  

The one I know is `read_hdf` which preserves fidelity (i.e. `int8` out is `int8` in).

Agreed `read_hdf` does preserver fidelity. `read_sql` does to the extent that the underlying db support it (which I don't think they really do). `read_csv` needs user interaction.

I think `read_stata` is fine, maybe just a more stringent warningin the docs.

And usually `read_hdf` is used by people storing/retriving using pandas, so the dtypes start at full-precision (unless explicity changed by the user).

stata a bit different as its an imported file.


Resolving patch:
diff --git a/pandas/io/stata.py b/pandas/io/stata.py
--- a/pandas/io/stata.py
+++ b/pandas/io/stata.py
@@ -29,7 +29,7 @@
 
 def read_stata(filepath_or_buffer, convert_dates=True,
                convert_categoricals=True, encoding=None, index=None,
-               convert_missing=False):
+               convert_missing=False, preserve_dtypes=True):
     """
     Read Stata file into DataFrame
 
@@ -52,13 +52,14 @@ def read_stata(filepath_or_buffer, convert_dates=True,
         If True, columns containing missing values are returned with
         object data types and missing values are represented by
         StataMissingValue objects.
+    preserve_dtypes : boolean, defaults to True
+        Preserve Stata datatypes. If False, numeric data are upcast to pandas
+        default types for foreign data (float64 or int64)
     """
     reader = StataReader(filepath_or_buffer, encoding)
 
-    return reader.data(convert_dates,
-                       convert_categoricals,
-                       index,
-                       convert_missing)
+    return reader.data(convert_dates, convert_categoricals, index,
+                       convert_missing, preserve_dtypes)
 
 _date_formats = ["%tc", "%tC", "%td", "%d", "%tw", "%tm", "%tq", "%th", "%ty"]
 
@@ -976,7 +977,7 @@ def _read_strls(self):
             self.path_or_buf.read(1)  # zero-termination
 
     def data(self, convert_dates=True, convert_categoricals=True, index=None,
-             convert_missing=False):
+             convert_missing=False, preserve_dtypes=True):
         """
         Reads observations from Stata file, converting them into a dataframe
 
@@ -995,7 +996,9 @@ def data(self, convert_dates=True, convert_categoricals=True, index=None,
             nans.  If True, columns containing missing values are returned with
             object data types and missing values are represented by
             StataMissingValue objects.
-
+        preserve_dtypes : boolean, defaults to True
+            Preserve Stata datatypes. If False, numeric data are upcast to
+            pandas default types for foreign data (float64 or int64)
         Returns
         -------
         y : DataFrame instance
@@ -1107,6 +1110,21 @@ def data(self, convert_dates=True, convert_categoricals=True, index=None,
                     labeled_data[(data[col] == k).values] = v
                 data[col] = Categorical.from_array(labeled_data)
 
+        if not preserve_dtypes:
+            retyped_data = []
+            convert = False
+            for col in data:
+                dtype = data[col].dtype
+                if dtype in (np.float16, np.float32):
+                    dtype = np.float64
+                    convert = True
+                elif dtype in (np.int8, np.int16, np.int32):
+                    dtype = np.int64
+                    convert = True
+                retyped_data.append((col, data[col].astype(dtype)))
+            if convert:
+                data = DataFrame.from_items(retyped_data)
+
         return data
 
     def data_label(self):


Test patch:
diff --git a/pandas/io/tests/test_stata.py b/pandas/io/tests/test_stata.py
--- a/pandas/io/tests/test_stata.py
+++ b/pandas/io/tests/test_stata.py
@@ -83,6 +83,7 @@ def setUp(self):
 
 
     def read_dta(self, file):
+        # Legacy default reader configuration
         return read_stata(file, convert_dates=True)
 
     def read_csv(self, file):
@@ -694,6 +695,35 @@ def test_big_dates(self):
             tm.assert_frame_equal(written_and_read_again.set_index('index'),
                                   expected)
 
+    def test_dtype_conversion(self):
+        expected = self.read_csv(self.csv15)
+        expected['byte_'] = expected['byte_'].astype(np.int8)
+        expected['int_'] = expected['int_'].astype(np.int16)
+        expected['long_'] = expected['long_'].astype(np.int32)
+        expected['float_'] = expected['float_'].astype(np.float32)
+        expected['double_'] = expected['double_'].astype(np.float64)
+        expected['date_td'] = expected['date_td'].apply(datetime.strptime,
+                                                        args=('%Y-%m-%d',))
+
+        no_conversion = read_stata(self.dta15_117,
+                                   convert_dates=True)
+        tm.assert_frame_equal(expected, no_conversion)
+
+        conversion = read_stata(self.dta15_117,
+                                convert_dates=True,
+                                preserve_dtypes=False)
+
+        # read_csv types are the same
+        expected = self.read_csv(self.csv15)
+        expected['date_td'] = expected['date_td'].apply(datetime.strptime,
+                                                        args=('%Y-%m-%d',))
+
+        tm.assert_frame_equal(expected, conversion)
+
+
+
+
+