Repository: pandas-dev/pandas

Issue:
ENH: read_excel dtypes and converts
Dear all,

Here from the mailing list: https://groups.google.com/forum/#!topic/pydata/jKiPOvYUQ1c

I have an excel table about family ages like this

| Family | People | Mean size [cm] |
| --- | --- | --- |
| Foo | 5 | 173.0 |
| Bar | 3 | 189.0 |

and I would like to use read_excel to parse it into Python. I would like "People" to be read as an integer, "Mean size [cm]" as a float. (And "Family" as a string, but that might be a different issue.) Now:
- if I set convert_float=True, the last column reads as int
- if I set convert_float=False, the second column reads as float

Neither one is correct, for a stupid reason: there happen to be those .0 in all sizes! So I would like to specify something like:

``` python
convert_float = ['People']
```

so only that column gets converted. An even better solution would be to be explicit about types of some columns, letting pandas perform the automagic for the others, such as:

``` python
read_excel('foo.xlsx', types={'People': np.uint8, 'Family': 'S3'})
```

but this changes the signature of the function more significantly.

Are you folks in favour of any of this? If yes, I can get a look and try to code it in.



Discussion:
you could add a `dtypes=` parmeter to coerce similarly to how `read_csv`. I would leave `convert_float` alone (its a bit non-standard; maybe even deprecate once `dtypes` is in place)

@jtratner ?

Yes I think this is a good idea to handle coercion like that. Related #8272

Hi,

I finally had some time to look into this. It is evident from our test suites (pandas/io/tests/test_excel.py, function 'test_reader_special_dtypes') that this is actually already implemented!

The way it works is via the 'converters' keyword argument to pandas.read_excel. This is passed down all the way to the parser:

read_excel --> ExcelFile._parse_excel --> TextParser --> TextFileReader._clean_options/_make_engine

So we just need to document this. If that's ok with you folks, I'll just make a minimal commit containing another parameter in the docstring(s) and make a pull request.
Or do you guys prefer to use 'dtypes' as a kwarg instead of 'converters'? numpy.loadtxt and numpy.genfromtxt support 'dtype' and 'converters', but not 'dtypes', so I'd be for keeping 'converters'.

`dtypes` is the standard parameter to specify dtype mappings (numpy text loaders are much less robust, much slower and not used nearly as much as `read_csv` so it is the de-facto standard now adays)

`converters` is somewhat duplicative but that's a different argument
`convert_float` is used in several places in pandas as well.

I think the `converters` are used, because ultimately the `PythonParser` which is called when this is actually parsed doesn't support `dtypes`, but does support `converters` (not implemented).

I think for consistency would like to add `dtypes` argument and fix the parser to support it.

In the meantime if you are wanting to document `converers` yes? then that is ok (copy doc-string from `read_csv`)

Actually, I tested `converters` a bit more and it has issues with missing values. For instance, if you write `converters={'mycol': int}` and a cell is empty, the parser defaults to empty string so when it tries to call `int(mycell)` it raises

``` python
ValueError: invalid literal for int() with base 10: ''
```

This happens in `_convert_to_ndarrays` in pandas.io.parsers.py, while calling pandas.lib.map_infer. This is in my opinion a plain bug. The parser should _first_ check for missing values, and _then_ convert only cells that do not contain missing values, but now it's the opposite: it first tries to convert everything to int, fails with an exception, and never gets to see that there was nothing to convert!

So my suggestion: I close this bug by reversing the order of the operations:

check for missing <--> convert dtype

and then I add a docstring to read_excel for `converters`. The `dtypes` change can come in a second step.

if a convert raises it's on the user
pandas can't guess what to do
you would have to have a try except in that converter 

pls submit a pr and I'll take a look

Hi,
If the user specifies converters --> int, pandas needs not guess anything. It just needs to convert to int, keeping empty cells as NaNs (or masked). I am fixing it and will submit a pull request ASAP, but just to give you the idea, if your table is:

| name | age |
| --- | --- |
| Alice |  |
| Bob | 30 |

with alice's age missing, pandas does this:
- `read_excel('mytable.xlsx')['age']` --> `float64`
- `read_excel('mytable.xlsx', converters={'age': int})['age']` --> `ValueError`!
- `read_excel('mytable.xlsx', converters={'age': (lambda x: int(x) if x else 0)}` --> correct (except you don't get masked)

You will agree that this behavious is not very friendly... but let me code it and then you can have a look.

Thanks for the clear explanation @iosonofabio :-)


Resolving patch:
diff --git a/pandas/io/excel.py b/pandas/io/excel.py
--- a/pandas/io/excel.py
+++ b/pandas/io/excel.py
@@ -87,6 +87,14 @@
     either be integers or column labels, values are functions that take one
     input argument, the Excel cell content, and return the transformed
     content.
+dtype : Type name or dict of column -> type, default None
+    Data type for data or columns. E.g. {'a': np.float64, 'b': np.int32}
+    Use `str` or `object` to preserve and not interpret dtype.
+    If converters are specified, they will be applied INSTEAD
+    of dtype conversion.
+
+    .. versionadded:: 0.20.0
+
 true_values : list, default None
     Values to consider as True
 
@@ -184,8 +192,8 @@ def read_excel(io, sheetname=0, header=0, skiprows=None, skip_footer=0,
                index_col=None, names=None, parse_cols=None, parse_dates=False,
                date_parser=None, na_values=None, thousands=None,
                convert_float=True, has_index_names=None, converters=None,
-               true_values=None, false_values=None, engine=None, squeeze=False,
-               **kwds):
+               dtype=None, true_values=None, false_values=None, engine=None,
+               squeeze=False, **kwds):
 
     if not isinstance(io, ExcelFile):
         io = ExcelFile(io, engine=engine)
@@ -195,7 +203,7 @@ def read_excel(io, sheetname=0, header=0, skiprows=None, skip_footer=0,
         index_col=index_col, parse_cols=parse_cols, parse_dates=parse_dates,
         date_parser=date_parser, na_values=na_values, thousands=thousands,
         convert_float=convert_float, has_index_names=has_index_names,
-        skip_footer=skip_footer, converters=converters,
+        skip_footer=skip_footer, converters=converters, dtype=dtype,
         true_values=true_values, false_values=false_values, squeeze=squeeze,
         **kwds)
 
@@ -318,7 +326,7 @@ def _parse_excel(self, sheetname=0, header=0, skiprows=None, names=None,
                      parse_cols=None, parse_dates=False, date_parser=None,
                      na_values=None, thousands=None, convert_float=True,
                      true_values=None, false_values=None, verbose=False,
-                     squeeze=False, **kwds):
+                     dtype=None, squeeze=False, **kwds):
 
         skipfooter = kwds.pop('skipfooter', None)
         if skipfooter is not None:
@@ -501,6 +509,7 @@ def _parse_cell(cell_contents, cell_typ):
                                     skiprows=skiprows,
                                     skipfooter=skip_footer,
                                     squeeze=squeeze,
+                                    dtype=dtype,
                                     **kwds)
 
                 output[asheetname] = parser.read()
diff --git a/pandas/io/parsers.py b/pandas/io/parsers.py
--- a/pandas/io/parsers.py
+++ b/pandas/io/parsers.py
@@ -18,7 +18,7 @@
 from pandas.types.common import (is_integer, _ensure_object,
                                  is_list_like, is_integer_dtype,
                                  is_float, is_dtype_equal,
-                                 is_object_dtype,
+                                 is_object_dtype, is_string_dtype,
                                  is_scalar, is_categorical_dtype)
 from pandas.types.missing import isnull
 from pandas.types.cast import _astype_nansafe
@@ -1329,7 +1329,7 @@ def _convert_to_ndarrays(self, dct, na_values, na_fvalues, verbose=False,
                     try_num_bool=False)
             else:
                 # skip inference if specified dtype is object
-                try_num_bool = not (cast_type and is_object_dtype(cast_type))
+                try_num_bool = not (cast_type and is_string_dtype(cast_type))
 
                 # general type inference and conversion
                 cvals, na_count = self._infer_types(


Test patch:
diff --git a/pandas/io/tests/data/testdtype.xls b/pandas/io/tests/data/testdtype.xls
new file mode 100644
Binary files /dev/null and b/pandas/io/tests/data/testdtype.xls differ
diff --git a/pandas/io/tests/data/testdtype.xlsm b/pandas/io/tests/data/testdtype.xlsm
new file mode 100644
Binary files /dev/null and b/pandas/io/tests/data/testdtype.xlsm differ
diff --git a/pandas/io/tests/data/testdtype.xlsx b/pandas/io/tests/data/testdtype.xlsx
new file mode 100644
Binary files /dev/null and b/pandas/io/tests/data/testdtype.xlsx differ
diff --git a/pandas/io/tests/test_excel.py b/pandas/io/tests/test_excel.py
--- a/pandas/io/tests/test_excel.py
+++ b/pandas/io/tests/test_excel.py
@@ -373,6 +373,33 @@ def test_reader_converters(self):
         actual = self.get_exceldf(basename, 'Sheet1', converters=converters)
         tm.assert_frame_equal(actual, expected)
 
+    def test_reader_dtype(self):
+        # GH 8212
+        basename = 'testdtype'
+        actual = self.get_exceldf(basename)
+
+        expected = DataFrame({
+            'a': [1, 2, 3, 4],
+            'b': [2.5, 3.5, 4.5, 5.5],
+            'c': [1, 2, 3, 4],
+            'd': [1.0, 2.0, np.nan, 4.0]}).reindex(
+                columns=['a', 'b', 'c', 'd'])
+
+        tm.assert_frame_equal(actual, expected)
+
+        actual = self.get_exceldf(basename,
+                                  dtype={'a': 'float64',
+                                         'b': 'float32',
+                                         'c': str})
+
+        expected['a'] = expected['a'].astype('float64')
+        expected['b'] = expected['b'].astype('float32')
+        expected['c'] = ['001', '002', '003', '004']
+        tm.assert_frame_equal(actual, expected)
+
+        with tm.assertRaises(ValueError):
+            actual = self.get_exceldf(basename, dtype={'d': 'int64'})
+
     def test_reading_all_sheets(self):
         # Test reading all sheetnames by setting sheetname to None,
         # Ensure a dict is returned.