Repository: pandas-dev/pandas

Issue:
na_filter=False ignored when index_col set
Given the following CSV file:

```
u1,u2,u3,d1,d2,d3,d4
Good Things,C,,1,1,1,1
Good Things,R,,1,1,1,1
Bad Things,C,,1,1,1,1
Bad Things,T,,1,1,1,1
Okay Things,N,B,1,1,1,1
Okay Things,N,D,1,1,1,1
Okay Things,B,,1,1,1,1
Okay Things,D,,1,1,1,1
```

First I parse with `na_filter=True`:

```
In [13]: pd.read_csv("/home/alex/nan_issue.csv", na_filter=True)
Out[13]: 
            u1 u2   u3  d1  d2  d3  d4
0  Good Things  C  NaN   1   1   1   1
1  Good Things  R  NaN   1   1   1   1
2   Bad Things  C  NaN   1   1   1   1
3   Bad Things  T  NaN   1   1   1   1
4  Okay Things  N    B   1   1   1   1
5  Okay Things  N    D   1   1   1   1
6  Okay Things  B  NaN   1   1   1   1
7  Okay Things  D  NaN   1   1   1   1
```

then I parse with `na_filter=False`:

```
In [12]: pd.read_csv("/home/alex/nan_issue.csv", na_filter=False)
Out[12]: 
            u1 u2 u3  d1  d2  d3  d4
0  Good Things  C      1   1   1   1
1  Good Things  R      1   1   1   1
2   Bad Things  C      1   1   1   1
3   Bad Things  T      1   1   1   1
4  Okay Things  N  B   1   1   1   1
5  Okay Things  N  D   1   1   1   1
6  Okay Things  B      1   1   1   1
7  Okay Things  D      1   1   1   1
```

then `index_cols` set:

```
In [11]: pd.read_csv("/home/alex/nan_issue.csv", na_filter=False,index_col=[0,1,2],)
Out[11]: 
                    d1  d2  d3  d4
u1          u2 u3                 
Good Things C  NaN   1   1   1   1
            R  NaN   1   1   1   1
Bad Things  C  NaN   1   1   1   1
            T  NaN   1   1   1   1
Okay Things N  B     1   1   1   1
               D     1   1   1   1
            B  NaN   1   1   1   1
            D  NaN   1   1   1   1
```

Finally setting `na_values=[], keep_default_na=False` seems to fix the issue:

```
In [14]: pd.read_csv("/home/alex/nan_issue.csv", na_filter=False,index_col=[0,1,2],na_values=[], keep_default_na=False)
Out[14]: 
                   d1  d2  d3  d4
u1          u2 u3                
Good Things C       1   1   1   1
            R       1   1   1   1
Bad Things  C       1   1   1   1
            T       1   1   1   1
Okay Things N  B    1   1   1   1
               D    1   1   1   1
            B       1   1   1   1
            D       1   1   1   1
```



Discussion:
There are very limited tests with `na_filter=False` and its a pretty silly parameter, so moving to low-priority. You are welcome to do a PR if you'd like.

TBI I can work around for now. My observation is that there probably too many parameters on the method having to do with handling of nans.It would be great to clean this up.
`keep_default_na`, `na_filter`, `na_values`

There is also interaction between these parameters. 

sure.....though to be honest, easiest just to drop `na_filter`...but if you come up with a better API, gr8


Resolving patch:
diff --git a/pandas/io/parsers.py b/pandas/io/parsers.py
--- a/pandas/io/parsers.py
+++ b/pandas/io/parsers.py
@@ -1249,6 +1249,8 @@ def __init__(self, kwds):
 
         self.na_values = kwds.get('na_values')
         self.na_fvalues = kwds.get('na_fvalues')
+        self.na_filter = kwds.get('na_filter', False)
+
         self.true_values = kwds.get('true_values')
         self.false_values = kwds.get('false_values')
         self.as_recarray = kwds.get('as_recarray', False)
@@ -1424,7 +1426,6 @@ def _make_index(self, data, alldata, columns, indexnamerow=False):
         elif not self._has_complex_date_col:
             index = self._get_simple_index(alldata, columns)
             index = self._agg_index(index)
-
         elif self._has_complex_date_col:
             if not self._name_processed:
                 (self.index_names, _,
@@ -1504,8 +1505,12 @@ def _agg_index(self, index, try_parse_dates=True):
             if try_parse_dates and self._should_parse_dates(i):
                 arr = self._date_conv(arr)
 
-            col_na_values = self.na_values
-            col_na_fvalues = self.na_fvalues
+            if self.na_filter:
+                col_na_values = self.na_values
+                col_na_fvalues = self.na_fvalues
+            else:
+                col_na_values = set()
+                col_na_fvalues = set()
 
             if isinstance(self.na_values, dict):
                 col_name = self.index_names[i]
@@ -2060,8 +2065,6 @@ def __init__(self, f, **kwds):
 
         self.names_passed = kwds['names'] or None
 
-        self.na_filter = kwds['na_filter']
-
         self.has_index_names = False
         if 'has_index_names' in kwds:
             self.has_index_names = kwds['has_index_names']


Test patch:
diff --git a/pandas/tests/io/parser/na_values.py b/pandas/tests/io/parser/na_values.py
--- a/pandas/tests/io/parser/na_values.py
+++ b/pandas/tests/io/parser/na_values.py
@@ -312,3 +312,21 @@ def test_empty_na_values_no_default_with_index(self):
         out = self.read_csv(StringIO(data), keep_default_na=False, index_col=0)
 
         tm.assert_frame_equal(out, expected)
+
+    def test_no_na_filter_on_index(self):
+        # see gh-5239
+        data = "a,b,c\n1,,3\n4,5,6"
+
+        # Don't parse NA-values in index when na_filter=False.
+        out = self.read_csv(StringIO(data), index_col=[1], na_filter=False)
+
+        expected = DataFrame({"a": [1, 4], "c": [3, 6]},
+                             index=Index(["", "5"], name="b"))
+        tm.assert_frame_equal(out, expected)
+
+        # Parse NA-values in index when na_filter=True.
+        out = self.read_csv(StringIO(data), index_col=[1], na_filter=True)
+
+        expected = DataFrame({"a": [1, 4], "c": [3, 6]},
+                             index=Index([np.nan, 5.0], name="b"))
+        tm.assert_frame_equal(out, expected)