Repository: pandas-dev/pandas

Issue:
read_csv issues with dict for na_values
Basically, I can't get a dictionary of `na_values` to work properly for me, no matter what I try. Pandas version is 0.22.0.
hack.csv contains:
```
113125,"blah","/blaha",kjsdkj,412.166,225.874,214.008
729639,"qwer","",asdfkj,466.681,,252.373
```

Here are two variants of my code - the one with the list does what I expect, but the dict version doesn't:
```python
df = pd.read_csv("hack.csv", header=None, keep_default_na=False, na_values=[214.008,'',"blah"])
df.head()
```
output from list version

0 | 1 | 2 | 3 | 4 | 5 | 6
-- | -- | -- | -- | -- | -- | --
113125 | NaN | /blaha | kjsdkj | 412.166 | 225.874 | NaN
729639 | qwer | NaN | asdfkj | 466.681 | NaN | 252.373

looks correct, but the dict version
```python
df = pd.read_csv("hack.csv", header=None, 
                 keep_default_na=False, na_values={2:"",6:"214.008",1:"blah",0:'113125'})
df.head()
```
although clearly paying attention to the columns I specify, is simply refusing to create any NaNs in those columns:

0 | 1 | 2 | 3 | 4 | 5 | 6
-- | -- | -- | -- | -- | -- | --
113125 | blah | /blaha | kjsdkj | 412.166 | 225.874 | 214.008
729639 | qwer |   | asdfkj | 466.681 | NaN | 252.373

So... I'm stuck. Any suggestions? I really want to have column-specific NaN handling so I need the dict.

Additionally the dict version _does_ create NaNs in columns I didn't specify in the dict, which also totally goes against my expectations for the combination of `keep_default_na=False` and an explicit value for `na_values`. Maybe I'm misreading the docs on that point.

Finally, you may notice that I used "214.008" (and other quoted numeric values) in the dict above. This is because I get a "not iterable" error when I provide unquoted numbers. This is despite that having been flagged as an issue and fixed a while back. This feels like another buglet to me.

Btw: to be picky, another doc-related quibble: I think the docs for `keep_default_na` are a bit misleading, in that they imply that `keep_default_na=True` should have no effect unless `na_values` is supplied (but in fact there is an effect even when `na_values` isn't supplied). It might be over-pedantic of me to care, but I feel that primary documentation really ought to be unambiguous. If anybody agrees with this pedantry I would be happy to propose a tweak ;-)

This issue was raised after I recently commented on two other issues with the problems described above, and @gfyoung suggested I ought to raise a new issue (comment links below).
https://github.com/pandas-dev/pandas/issues/1657#issuecomment-355833116
https://github.com/pandas-dev/pandas/issues/12224#issuecomment-355960984

#### Output of ``pd.show_versions()``
NB: I quite likely have some "old" modules in the pile below, but I believe that I've updated pandas itself correctly, so any dependencies ought to have been updated too. If my errors aren't reproducible by others, it might indicate that there's a hidden version dependency(?) but I'm too much of a pandas noob to know how likely that is.
<details>

INSTALLED VERSIONS
------------------
commit: None
python: 3.5.2.final.0
python-bits: 64
OS: Windows
OS-release: 10
machine: AMD64
processor: Intel64 Family 6 Model 158 Stepping 9, GenuineIntel
byteorder: little
LC_ALL: None
LANG: None
LOCALE: None.None

pandas: 0.22.0
pytest: 2.9.2
pip: 8.1.2
setuptools: 27.2.0
Cython: 0.24.1
numpy: 1.13.3
scipy: 1.0.0
pyarrow: None
xarray: None
IPython: 5.1.0
sphinx: 1.4.6
patsy: 0.4.1
dateutil: 2.5.3
pytz: 2016.6.1
blosc: None
bottleneck: 1.1.0
tables: 3.2.2
numexpr: 2.6.4
feather: None
matplotlib: 2.0.2
openpyxl: 2.3.2
xlrd: 1.0.0
xlwt: 1.1.2
xlsxwriter: 0.9.3
lxml: 3.6.4
bs4: 4.5.1
html5lib: None
sqlalchemy: 1.0.13
pymysql: None
psycopg2: None
jinja2: 2.8
s3fs: None
fastparquet: None
pandas_gbq: None
pandas_datareader: None
</details>



Discussion:
Thanks for the report. 
Can you check what happens if you use column names (and so can use strings instead of integers in the dict)? Just to check whether that works or whether it is a general problem with `na_values`
@jorisvandenbossche : So many bugs...that's all I can say. 😂 Luckily, the patch for them isn't so bad 😄 (PR coming for them soon).

Resolving patch:
diff --git a/pandas/io/parsers.py b/pandas/io/parsers.py
--- a/pandas/io/parsers.py
+++ b/pandas/io/parsers.py
@@ -149,8 +149,20 @@
     NaN: '""" + fill("', '".join(sorted(_NA_VALUES)),
                      70, subsequent_indent="    ") + """'.
 keep_default_na : bool, default True
-    If na_values are specified and keep_default_na is False the default NaN
-    values are overridden, otherwise they're appended to.
+    Whether or not to include the default NaN values when parsing the data.
+    Depending on whether `na_values` is passed in, the behavior is as follows:
+
+    * If `keep_default_na` is True, and `na_values` are specified, `na_values`
+      is appended to the default NaN values used for parsing.
+    * If `keep_default_na` is True, and `na_values` are not specified, only
+      the default NaN values are used for parsing.
+    * If `keep_default_na` is False, and `na_values` are specified, only
+      the NaN values specified `na_values` are used for parsing.
+    * If `keep_default_na` is False, and `na_values` are not specified, no
+      strings will be parsed as NaN.
+
+    Note that if `na_filter` is passed in as False, the `keep_default_na` and
+    `na_values` parameters will be ignored.
 na_filter : boolean, default True
     Detect missing value markers (empty strings and the value of na_values). In
     data without any NAs, passing na_filter=False can improve the performance
@@ -910,9 +922,6 @@ def _clean_options(self, options, engine):
         na_values = options['na_values']
         skiprows = options['skiprows']
 
-        # really delete this one
-        keep_default_na = result.pop('keep_default_na')
-
         _validate_header_arg(options['header'])
 
         depr_warning = ''
@@ -957,6 +966,7 @@ def _clean_options(self, options, engine):
             converters = {}
 
         # Converting values to NA
+        keep_default_na = options['keep_default_na']
         na_values, na_fvalues = _clean_na_values(na_values, keep_default_na)
 
         # handle skiprows; this is internally handled by the
@@ -1225,6 +1235,7 @@ def __init__(self, kwds):
         self.na_values = kwds.get('na_values')
         self.na_fvalues = kwds.get('na_fvalues')
         self.na_filter = kwds.get('na_filter', False)
+        self.keep_default_na = kwds.get('keep_default_na', True)
 
         self.true_values = kwds.get('true_values')
         self.false_values = kwds.get('false_values')
@@ -1487,7 +1498,8 @@ def _agg_index(self, index, try_parse_dates=True):
                 col_name = self.index_names[i]
                 if col_name is not None:
                     col_na_values, col_na_fvalues = _get_na_values(
-                        col_name, self.na_values, self.na_fvalues)
+                        col_name, self.na_values, self.na_fvalues,
+                        self.keep_default_na)
 
             arr, _ = self._infer_types(arr, col_na_values | col_na_fvalues)
             arrays.append(arr)
@@ -1510,7 +1522,7 @@ def _convert_to_ndarrays(self, dct, na_values, na_fvalues, verbose=False,
 
             if self.na_filter:
                 col_na_values, col_na_fvalues = _get_na_values(
-                    c, na_values, na_fvalues)
+                    c, na_values, na_fvalues, self.keep_default_na)
             else:
                 col_na_values, col_na_fvalues = set(), set()
 
@@ -3097,16 +3109,23 @@ def _clean_na_values(na_values, keep_default_na=True):
             na_values = set()
         na_fvalues = set()
     elif isinstance(na_values, dict):
-        na_values = na_values.copy()  # Prevent aliasing.
-        if keep_default_na:
-            for k, v in compat.iteritems(na_values):
-                if not is_list_like(v):
-                    v = [v]
+        old_na_values = na_values.copy()
+        na_values = {}  # Prevent aliasing.
+
+        # Convert the values in the na_values dictionary
+        # into array-likes for further use. This is also
+        # where we append the default NaN values, provided
+        # that `keep_default_na=True`.
+        for k, v in compat.iteritems(old_na_values):
+            if not is_list_like(v):
+                v = [v]
+
+            if keep_default_na:
                 v = set(v) | _NA_VALUES
-                na_values[k] = v
-        na_fvalues = dict(
-            (k, _floatify_na_values(v)) for k, v in na_values.items()  # noqa
-        )
+
+            na_values[k] = v
+        na_fvalues = dict((k, _floatify_na_values(v))
+                          for k, v in na_values.items())
     else:
         if not is_list_like(na_values):
             na_values = [na_values]
@@ -3225,12 +3244,38 @@ def _stringify_na_values(na_values):
     return set(result)
 
 
-def _get_na_values(col, na_values, na_fvalues):
+def _get_na_values(col, na_values, na_fvalues, keep_default_na):
+    """
+    Get the NaN values for a given column.
+
+    Parameters
+    ----------
+    col : str
+        The name of the column.
+    na_values : array-like, dict
+        The object listing the NaN values as strings.
+    na_fvalues : array-like, dict
+        The object listing the NaN values as floats.
+    keep_default_na : bool
+        If `na_values` is a dict, and the column is not mapped in the
+        dictionary, whether to return the default NaN values or the empty set.
+
+    Returns
+    -------
+    nan_tuple : A length-two tuple composed of
+
+        1) na_values : the string NaN values for that column.
+        2) na_fvalues : the float NaN values for that column.
+    """
+
     if isinstance(na_values, dict):
         if col in na_values:
             return na_values[col], na_fvalues[col]
         else:
-            return _NA_VALUES, set()
+            if keep_default_na:
+                return _NA_VALUES, set()
+
+            return set(), set()
     else:
         return na_values, na_fvalues
 


Test patch:
diff --git a/pandas/tests/io/parser/na_values.py b/pandas/tests/io/parser/na_values.py
--- a/pandas/tests/io/parser/na_values.py
+++ b/pandas/tests/io/parser/na_values.py
@@ -224,6 +224,45 @@ def test_na_values_keep_default(self):
                                   'seven']})
         tm.assert_frame_equal(xp.reindex(columns=df.columns), df)
 
+    def test_no_keep_default_na_dict_na_values(self):
+        # see gh-19227
+        data = "a,b\n,2"
+
+        df = self.read_csv(StringIO(data), na_values={"b": ["2"]},
+                           keep_default_na=False)
+        expected = DataFrame({"a": [""], "b": [np.nan]})
+        tm.assert_frame_equal(df, expected)
+
+        # Scalar values shouldn't cause the parsing to crash or fail.
+        data = "a,b\n1,2"
+
+        df = self.read_csv(StringIO(data), na_values={"b": 2},
+                           keep_default_na=False)
+        expected = DataFrame({"a": [1], "b": [np.nan]})
+        tm.assert_frame_equal(df, expected)
+
+        data = """\
+113125,"blah","/blaha",kjsdkj,412.166,225.874,214.008
+729639,"qwer","",asdfkj,466.681,,252.373
+"""
+        expected = DataFrame({0: [np.nan, 729639.0],
+                              1: [np.nan, "qwer"],
+                              2: ["/blaha", np.nan],
+                              3: ["kjsdkj", "asdfkj"],
+                              4: [412.166, 466.681],
+                              5: ["225.874", ""],
+                              6: [np.nan, 252.373]})
+
+        df = self.read_csv(StringIO(data), header=None, keep_default_na=False,
+                           na_values={2: "", 6: "214.008",
+                                      1: "blah", 0: 113125})
+        tm.assert_frame_equal(df, expected)
+
+        df = self.read_csv(StringIO(data), header=None, keep_default_na=False,
+                           na_values={2: "", 6: "214.008",
+                                      1: "blah", 0: "113125"})
+        tm.assert_frame_equal(df, expected)
+
     def test_na_values_na_filter_override(self):
         data = """\
 A,B