Repository: pandas-dev/pandas (Python)

Issue:
index_col in read_csv and read_table ignores dtype argument
xref #11728 for the multi-index case 
xref https://github.com/pydata/pandas/issues/14379 for converters

``` python
import pandas as pd
import numpy as np

from pandas.compat import StringIO

data = """Internets,Spaceships
01,a
02,b
03,c
04,d
05,e
06,f
"""

# No leading zeroes in the index because it interprets the column as numeric
print pd.read_csv(StringIO.StringIO(data), index_col="Internets")

# Expected to see trailing zeroes in the index still, got 1,2,3,4 instead. Index is int64.
print pd.read_csv(StringIO.StringIO(data), index_col="Internets", dtype={"Internets": np.object})

# Trailing zeroes now, index is object.
print pd.read_csv(StringIO.StringIO(data), dtype={"Internets": np.object}).set_index("Internets")

```

Version:

```
In [1]: pd.show_versions()

INSTALLED VERSIONS
------------------
commit: None
python: 2.7.9.final.0
python-bits: 64
OS: Darwin
OS-release: 13.4.0
machine: x86_64
processor: i386
byteorder: little
LC_ALL: None
LANG: en_US.UTF-8

pandas: 0.15.2
nose: 1.3.4
Cython: None
numpy: 1.9.1
scipy: 0.15.1
statsmodels: 0.6.1
IPython: 2.1.0
sphinx: None
patsy: 0.3.0
dateutil: 2.4.0
pytz: 2011c
bottleneck: None
tables: None
numexpr: 2.4
matplotlib: 1.3.1
openpyxl: 2.0.4
xlrd: 0.9.3
xlwt: 0.7.5
xlsxwriter: None
lxml: None
bs4: 4.3.2
html5lib: 0.999
httplib2: None
apiclient: None
rpy2: None
sqlalchemy: 0.9.6
pymysql: None
psycopg2: None
```



Discussion:
iIIRC this is already fixed in master
can u give a try

@jreback Still seems to happen in version 0.15.2-163-g671b384 commit 671b384c27b11a2caf0ac1c9b3f8ce2396f554c2 (from today). :( Sorry!

I'm guessing the reason for this is that the index assignment happens separately from the dtype conversion somehow - I dug around in parsers.py trying to see if I could find the root but no luck.

The values that are passed to the index constructor are object (e.g. the `01,02`....etc), but the `Index` constructor will coerce these if possible (and here it IS possible), unless a dtype is passed. So to fix you need to pass the dtype if it is passed in the dtype spec to `read_csv`. want to do a pull-request?

My first pandas pull request! I'll give it a shot tomorrow or over the long weekend. Thanks!

Notes to self:

https://github.com/pydata/pandas/blob/master/pandas/io/parsers.py#L721
https://github.com/pydata/pandas/blob/master/pandas/core/index.py#L126
https://github.com/pydata/pandas/blob/master/pandas/io/parsers.py#L246
https://github.com/pydata/pandas/blob/master/pandas/io/parsers.py#L558
- What is dtypes / vs converters ?
- Stick this in cparserwrapper / python parser or higher up?

This bug still existed in 0.17.

@Eastsun doesnt the open tag on the issue make this clear?

Is there also a quick workaround for the following case? I write multi-indexed columns to csv and then read them as follows:

```
import pandas as pd
from io import StringIO

data=""",col11,col21
,col12,col22
uuid,,
0001,1,1
0002,2,2
"""

print(pd.read_csv(StringIO(data), header = [0,1], index_col=0, dtype={'uuid':object))
```

Omitting the index_col messes up the columns. Rebuilding the correct structure is a bit of a pain...

not sure how 'quick' this is....

```
In [35]: result = pd.read_csv(StringIO.StringIO(data),dtype={'uuid':object},skiprows=2).set_index('uuid')

In [36]: result.columns = pd.MultiIndex.from_tuples(pd.read_csv(StringIO.StringIO(data),header=[0,1],index_col=0,nrows=0).columns)

In [37]: result
Out[37]: 
     col11 col21
     col12 col22
uuid            
0001     1     1
0002     2     2
```

Nice minimal example here:

``` python
>>> from pandas import read_csv
>>> from pandas.compat import StringIO
>>>
>>> data = 'a,b\n01,2'
>>> read_csv(StringIO(data), index_col='a', dtype={'a': object})
   b
a   
1  2
```

The reason for this is because we have this awkward double conversion for indices.  First, the column will be converted <a href="https://github.com/pydata/pandas/blob/185fcbebd8c65706a9e3fea2d1ae28aa2d8baa93/pandas/parser.pyx#L1069">here</a>.  After this conversion, the column is correctly outputted.  However, a second conversion will take place <a href="https://github.com/pydata/pandas/blob/185fcbebd8c65706a9e3fea2d1ae28aa2d8baa93/pandas/io/parsers.py#L1255">here</a>, and it is this conversion that screws up everything.

The reason for this awkward double conversion is because for some reason or another, some functionality does depend on that second conversion.  For example, if a converter is passed in for the column, the first conversion step is skipped, and the second one is relied on.

This is yet another indication of the much needed refactoring in the `read_csv` world.

Note that this bug also affects `read_fwf`.
Just checking, are both of these issues still open on 0.20+? Is makmanalp's resolved but johanneshk's case still open (the latter is still wrong on 0.20.3). Do we perhaps need to open a separate issue?

Also, in Python 3, `import StringIO` -> `import io` ... and instantiate `io.StringIO`
@smcinerney the OP's (@makmanalp) hasn't been addressed as of 0.21.1.
Still open?

On 0.23.4:

```python
import io
pd.read_csv(io.StringIO('''
version,downloads
1.1,100
1.2,1000
1.3,10000
'''), dtype={'version': str}, index_col='version').index.dtype
# dtype('float64')
```
@gwerbin : Indeed, this is an open issue.
I see.

> The reason for this awkward double conversion is because for some reason or another, some functionality does depend on that second conversion. For example, if a converter is passed in for the column, the first conversion step is skipped, and the second one is relied on.

That bites. Is test coverage good enough in this area that this can be fixed without a big refactor?
> Is test coverage good enough in this area that this can be fixed without a big refactor?

It should be.  The problem is that there are a bunch of failures if you try to "correct" the behavior.
Ran into this today, seems still open.

Any preferred workaround in the meantime?



Patch:
diff --git a/doc/source/whatsnew/v1.4.0.rst b/doc/source/whatsnew/v1.4.0.rst
index 39e3894f86302..4b67ab33bbbc6 100644
--- a/doc/source/whatsnew/v1.4.0.rst
+++ b/doc/source/whatsnew/v1.4.0.rst
@@ -658,6 +658,7 @@ I/O
 - Bug in :func:`json_normalize` where multi-character ``sep`` parameter is incorrectly prefixed to every key (:issue:`43831`)
 - Bug in :func:`json_normalize` where reading data with missing multi-level metadata would not respect errors="ignore" (:issue:`44312`)
 - Bug in :func:`read_csv` with :code:`float_precision="round_trip"` which did not skip initial/trailing whitespace (:issue:`43713`)
+- Bug in :func:`read_csv` not applying dtype for ``index_col`` (:issue:`9435`)
 - Bug in dumping/loading a :class:`DataFrame` with ``yaml.dump(frame)`` (:issue:`42748`)
 - Bug in :class:`ExcelWriter`, where ``engine_kwargs`` were not passed through to all engines (:issue:`43442`)
 - Bug in :func:`read_csv` raising ``ValueError`` when ``parse_dates`` was used with ``MultiIndex`` columns (:issue:`8991`)
diff --git a/pandas/io/parsers/base_parser.py b/pandas/io/parsers/base_parser.py
index 25a89d1c57006..f06950e3450a7 100644
--- a/pandas/io/parsers/base_parser.py
+++ b/pandas/io/parsers/base_parser.py
@@ -1,6 +1,7 @@
 from __future__ import annotations
 
 from collections import defaultdict
+from copy import copy
 import csv
 import datetime
 from enum import Enum
@@ -148,6 +149,8 @@ def __init__(self, kwds):
         self.na_filter = kwds.get("na_filter", False)
         self.keep_default_na = kwds.get("keep_default_na", True)
 
+        self.dtype = copy(kwds.get("dtype", None))
+
         self.true_values = kwds.get("true_values")
         self.false_values = kwds.get("false_values")
         self.mangle_dupe_cols = kwds.get("mangle_dupe_cols", True)
@@ -498,6 +501,19 @@ def _get_name(icol):
 
         return index
 
+    def _clean_mapping(self, mapping):
+        """converts col numbers to names"""
+        if not isinstance(mapping, dict):
+            return mapping
+        clean = {}
+        for col, v in mapping.items():
+            # for mypy
+            assert self.orig_names is not None
+            if isinstance(col, int) and col not in self.orig_names:
+                col = self.orig_names[col]
+            clean[col] = v
+        return clean
+
     @final
     def _agg_index(self, index, try_parse_dates: bool = True) -> Index:
         arrays = []
@@ -522,7 +538,17 @@ def _agg_index(self, index, try_parse_dates: bool = True) -> Index:
                         col_name, self.na_values, self.na_fvalues, self.keep_default_na
                     )
 
-            arr, _ = self._infer_types(arr, col_na_values | col_na_fvalues)
+            clean_dtypes = self._clean_mapping(self.dtype)
+
+            cast_type = None
+            if isinstance(clean_dtypes, dict) and self.index_names is not None:
+                cast_type = clean_dtypes.get(self.index_names[i], None)
+
+            try_num_bool = not (cast_type and is_string_dtype(cast_type))
+
+            arr, _ = self._infer_types(
+                arr, col_na_values | col_na_fvalues, try_num_bool
+            )
             arrays.append(arr)
 
         names = self.index_names
diff --git a/pandas/io/parsers/python_parser.py b/pandas/io/parsers/python_parser.py
index f5420618c0235..6800fca508adc 100644
--- a/pandas/io/parsers/python_parser.py
+++ b/pandas/io/parsers/python_parser.py
@@ -4,7 +4,6 @@
     abc,
     defaultdict,
 )
-from copy import copy
 import csv
 from io import StringIO
 import re
@@ -89,7 +88,6 @@ def __init__(
         self.verbose = kwds["verbose"]
         self.converters = kwds["converters"]
 
-        self.dtype = copy(kwds["dtype"])
         self.thousands = kwds["thousands"]
         self.decimal = kwds["decimal"]
 
@@ -308,21 +306,8 @@ def get_chunk(self, size=None):
 
     def _convert_data(self, data):
         # apply converters
-        def _clean_mapping(mapping):
-            """converts col numbers to names"""
-            clean = {}
-            for col, v in mapping.items():
-                if isinstance(col, int) and col not in self.orig_names:
-                    col = self.orig_names[col]
-                clean[col] = v
-            return clean
-
-        clean_conv = _clean_mapping(self.converters)
-        if not isinstance(self.dtype, dict):
-            # handles single dtype applied to all columns
-            clean_dtypes = self.dtype
-        else:
-            clean_dtypes = _clean_mapping(self.dtype)
+        clean_conv = self._clean_mapping(self.converters)
+        clean_dtypes = self._clean_mapping(self.dtype)
 
         # Apply NA values.
         clean_na_values = {}


Test patch:
diff --git a/pandas/tests/io/parser/test_index_col.py b/pandas/tests/io/parser/test_index_col.py
index 7315dcc0c4c07..58b5eebbec344 100644
--- a/pandas/tests/io/parser/test_index_col.py
+++ b/pandas/tests/io/parser/test_index_col.py
@@ -321,3 +321,14 @@ def test_infer_types_boolean_sum(all_parsers):
     # index column of dtype 'object', and the Python parser will return a
     # index column of dtype 'int64'.
     tm.assert_frame_equal(result, expected, check_index_type=False)
+
+
+@skip_pyarrow
+@pytest.mark.parametrize("dtype, val", [(object, "01"), ("int64", 1)])
+def test_specify_dtype_for_index_col(all_parsers, dtype, val):
+    # GH#9435
+    data = "a,b\n01,2"
+    parser = all_parsers
+    result = parser.read_csv(StringIO(data), index_col="a", dtype={"a": dtype})
+    expected = DataFrame({"b": [2]}, index=Index([val], name="a"))
+    tm.assert_frame_equal(result, expected)