Repository: pandas-dev/pandas

Issue:
DOC/BUG: NA string in data gets read as NaN
better docs for `keep_default_na` and `na_values` when used together

I have a field set where "NA" is meaningful, but not meaning "not applicable" (it refers to the name of my school's Homeroom). There's no obvious way to handle this edge case...
On 0.11.0



Discussion:
pass `keep-default_na=False` to `read_csv` then it will only recognize NA's that you explicity pass in `na_values` (none by default)

Ah. The doc for that isn't clear:
        If na_values are specified and keep_default_na is False the default NaN
        values are overridden, otherwise they're appended to

so I think if you want NO values converted to NA's

do `keep_default_na=False, na_values=[]`.....otherwise put what you want in na_values
(e.g. you need to have `na_values` be not None)

maybe should put an example....(and/or make a bit clearer in docs)

let us know if that works

Definitely worked. I just think this might be a common problem, the docs don't make it clear at all about the behaviour, really convoluted. How about this:
To customize what constitutes a null value, set keep_na_values to false and pass the list of values that are to be interpreted as null to na_values. Or, if more values than the default are to be interpreted as null, keep_na_values as True (the default) and pass the additional values to na_values.

great
want to so a PR for this? (prob doc string and io.rst), maybe even add an example of doing this 

Sure, happy to contribute. 

(if you use `keep_default_na=False`, I don't think you need to add `na_values=[]`... the default behaviour is to have no NA values.)

actually...just tested with with `keep_default_na=False` and not passing `na_values` which defaults it to `None`

unfortunately

`na_values = set(['None', None])`

which prob works, but seems wrong....

(just needs a small change in `pandas.io.parsers._clean_na_values` to fix this

Good spot! (I had only checked for "NA", which is NaN'd normally but left with keep_default_na=False.)


Resolving patch:
diff --git a/pandas/io/parsers.py b/pandas/io/parsers.py
--- a/pandas/io/parsers.py
+++ b/pandas/io/parsers.py
@@ -1774,8 +1774,11 @@ def _try_convert_dates(parser, colspec, data_dict, columns):
 
 def _clean_na_values(na_values, keep_default_na=True):
 
-    if na_values is None and keep_default_na:
-        na_values = _NA_VALUES
+    if na_values is None:
+        if keep_default_na:
+            na_values = _NA_VALUES
+        else:
+            na_values = []
         na_fvalues = set()
     elif isinstance(na_values, dict):
         if keep_default_na:


Test patch:
diff --git a/pandas/io/tests/test_parsers.py b/pandas/io/tests/test_parsers.py
--- a/pandas/io/tests/test_parsers.py
+++ b/pandas/io/tests/test_parsers.py
@@ -108,6 +108,27 @@ def test_empty_string(self):
                                   np.nan, 'seven']})
         tm.assert_frame_equal(xp.reindex(columns=df.columns), df)
 
+
+        # GH4318, passing na_values=None and keep_default_na=False yields 'None' as a na_value
+        data = """\
+One,Two,Three
+a,1,None
+b,2,two
+,3,None
+d,4,nan
+e,5,five
+nan,6,
+g,7,seven
+"""
+        df = self.read_csv(
+            StringIO(data), keep_default_na=False)
+        xp = DataFrame({'One': ['a', 'b', '', 'd', 'e', 'nan', 'g'],
+                        'Two': [1, 2, 3, 4, 5, 6, 7],
+                        'Three': ['None', 'two', 'None', 'nan', 'five', '',
+                                  'seven']})
+        tm.assert_frame_equal(xp.reindex(columns=df.columns), df)
+
+
     def test_read_csv(self):
         if not compat.PY3:
             if 'win' in sys.platform: