Repository: pandas-dev/pandas

Issue:
 Odd behavior from read_csv with na_values set to non-string values
read_csv behaves oddly when na_values is set to non-string values. Sometimes
it correctly replaces the assigned number with NaN, and sometimes it doesn't. Here are some examples. Note in particular the different behavior of the last two statements:
# Create file

df = DataFrame({'A' : [-999, 2, 3], 'B' : [1.2, -999, 4.5]})
df.to_csv('test2.csv', sep=' ', index=False)

print read_csv('test2.csv', sep= ' ', header=0, na_values=[-999])

---

A B
0 NaN 1.2
1 2 -999.0
## 2 3 4.5

print read_csv('test2.csv', sep= ' ', header=0, na_values=[-999.0])

---

A B
0 -999 1.2
1 2 NaN
## 2 3 4.5

print read_csv('test2.csv', sep= ' ', header=0, na_values=[-999.0,-999])

---

A B
0 -999 1.2
1 2 NaN
## 2 3 4.5

print read_csv('test2.csv', sep= ' ', header=0, na_values=[-999,-999.0])

---

A B
0 NaN 1.2
1 2 -999.0
## 2 3 4.5



Discussion:
@rhstanton give a try with master, this should be fixed (via #3615)

Thanks! The behavior in the last two cases is now consistent.

However, using 999 does not register as na when the input value is (float) 999.0, and vice versa. I’m not really sure what should happen here, or even if this should work at all instead of flagging an error (should this insist on string values?). If it is supposed to work with numeric instead of string values (which I’d be quite happy with), wouldn’t it then make sense for (say) 999 to work with both integer and float inputs equal to 999?  After all, if I set i = 999, then both

i == 999

and

i == 999.0

return True.

From: jreback [mailto:notifications@github.com]
Sent: Wednesday, May 15, 2013 6:02 PM
To: pydata/pandas
Cc: Richard Stanton
Subject: Re: [pandas] Odd behavior from read_csv with na_values set to non-string values (#3611)

@rhstantonhttps://github.com/rhstanton give a try with master, this should be fixed (via #3615https://github.com/pydata/pandas/issues/3615)

—
Reply to this email directly or view it on GitHubhttps://github.com/pydata/pandas/issues/3611#issuecomment-17975374.

999 and 999.0 are different strings which is how this matched

u can different values per column as well (pass a dict)

‘999’ and ‘999.0’ are different strings, but in these examples we’re passing numbers, not strings, and

999 == 999.0

returns True. If it’s going to work at all passing numbers, I think my expectation would be for the results to be based on a numerical comparison, not a string comparison.

Thanks for the dict tip. That could be useful.

From: jreback [mailto:notifications@github.com]
Sent: Wednesday, May 15, 2013 9:01 PM
To: pydata/pandas
Cc: Richard Stanton
Subject: Re: [pandas] Odd behavior from read_csv with na_values set to non-string values (#3611)

999 and 999.0 are different strings which is how this matched

u can different values per column as well (pass a dict)

—
Reply to this email directly or view it on GitHubhttps://github.com/pydata/pandas/issues/3611#issuecomment-17980206.

keep in mind that this is an exact string match, these don't have dtypes at this point. that said, you have a valid point, so you are proposing adding the float versions of ints and vice versa if specified in na_vaues, pls open an enhancement issue for this

note you can do this:

read in with no na match for 999,999.0

```
df[df==999] == np.nan
```

will work for ints and floats (you can of course column restrict)

This is a problem for me as well, despite having a very recent build of Pandas. In my dataset, `88` is used as a missing value. Hence, I used the following:

```
dataset = data_file.parse("Sheet1", index_col=0, na_values=[88])
```

however, this value was not picked up as being NaN:

```
dataset.diarrhea.value_counts()

0     282
1     197
88      8
dtype: int64
```

I have tried using a float and an integer, but neither worked. Also, specifying the columns for replacement with a dict does not work.

[Here is the dataset](https://dl.dropboxusercontent.com/u/233041/test.xlsx) that I am using.

Also, the solution you suggest above does not work if even a single column is of a different type:

```
dataset[dataset==88] = np.nan

ValueError: Cannot do boolean setting on mixed-type frame
```

So, it appears that the only solution is to manually check which columns contain the value, and replace them in a loop.

If you pass `na_values=['88']` does it work?

Unfortunately not.

This was a bug (introduced when I fixed this originally)

also, slight API change, na_values will now take a value like 88 (as a int), and match '88',88,88.0 (string/numeric dtypes) in whatever the resulting dtype of the column

@fonnesbeck it seems natural to read this dataset with `index_col=None`

@fonnesbeck pls try this out if you can (via the PR)

```
In [5]: pd.read_excel('test.xlsx',"Sheet1", index_col=0, na_values=['88'])
Out[5]: 
<class 'pandas.core.frame.DataFrame'>
Index: 490 entries, 1.0 to 1.0
Data columns (total 36 columns):
race         490  non-null values
ethn         484  non-null values
stcol        484  non-null values
rota         393  non-null values
blood        487  non-null values
saliva       484  non-null values
diarrhea     479  non-null values
qod          160  non-null values
abdpain      333  non-null values
vomit        483  non-null values
dos          478  non-null values
travel       16  non-null values
sickcon      231  non-null values
noc          486  non-null values
type         490  non-null values
abx          28  non-null values
dtp          337  non-null values
fever        472  non-null values
feverval     112  non-null values
pathogens    160  non-null values
futuse       488  non-null values
ecoli        486  non-null values
amc          490  non-null values
am           490  non-null values
cz           490  non-null values
fep          490  non-null values
caz          490  non-null values
cro          490  non-null values
cxm          490  non-null values
cip          490  non-null values
gm           490  non-null values
mem          490  non-null values
tzp          490  non-null values
stx          490  non-null values
case         490  non-null values
month        490  non-null values
dtypes: float64(36)

In [6]: pd.read_excel('test.xlsx',"Sheet1", index_col=0, na_values=['88.0']).head()
Out[6]: 
     race  ethn  stcol  rota  blood  saliva  diarrhea  qod  abdpain  vomit  dos  travel  sickcon  noc  type  abx  dtp  fever  feverval  pathogens  futuse  ecoli  amc  am  cz  fep  caz  cro  cxm  cip  gm  mem  tzp  stx  case  month
sex                                                                                                                                                                                                                                   
1       1     1      1     0      0       1         1  NaN        0      0    4     NaN      NaN    1     0  NaN    8      0       NaN        NaN       1      1    1   2   1    1    1    1    1    1   1    1    1    1     1     12
0       2     2      1     0      0       1         0  NaN      NaN      1    3     NaN        1    0     0  NaN  NaN      1       NaN        NaN       0      0    0   0   0    0    0    0    0    0   0    0    0    0     1     12
0       2     2      0   NaN      0       2         0  NaN        1      1    4     NaN      NaN    0     0  NaN  NaN      0       NaN        NaN       1      0    0   0   0    0    0    0    0    0   0    0    0    0     1      0
1       3     2      1     0      0       1         1    0      NaN      1    4     NaN        1    0     0  NaN  NaN      0       NaN        NaN       1      0    0   0   0    0    0    0    0    0   0    0    0    0     1     12
1       1     1      1     0      0       1         0  NaN        1      1    2     NaN        0    0     0  NaN    3      1       NaN        NaN       0      0    0   0   0    0    0    0    0    0   0    0    0    0     1     12
```

Sorry, the original file included a patient ID in column 0, which I had to remove before sharing. I will try the fix. 

It is still failing for me, unfortunately:

```
In[11]: pd.read_excel("Ecoli1.xlsx", "Sheet1", na_values=[88]).travel.value_counts()

Out[11]:
88    471
0      14
1       2
dtype: int64
```

did you try this PR? (its not in master)

I just merged it...try master now

It sure helps when I build the right code. Sorry. Works for me now, yes.

great! 
thanks for catching these bugs!
reclosing

I like the sound of this edit, but it ether doesn't quite work or I'm misinterpreting what its supposed to do. Here's an example, where I want -9988 to denote missing data, and where the data file, test.csv, contains both '-9988' and '-9988.0': 

---

Field1,Field2,Field3
-9988,-9988.0,5.6

---

The command 

read_csv('data/test.csv', header=0,  sep=',', na_values=[-9988])

yields the output

```
Field1  Field2  Field3
```

0   NaN        -9988            5.6

Are you on latest master?

```
In [9]: pd.__version__
Out[9]: '0.11.1.dev-2b7efc3'

In [4]: path = 'test.csv'

In [5]: df = DataFrame({'A' : [-999, 2, 3], 'B' : [1.2, -999, 4.5]})

In [10]: df
Out[10]: 
     A      B
0 -999    1.2
1    2 -999.0
2    3    4.5

In [6]: df.to_csv(path, sep=' ', index=False)

In [7]: read_csv(path, sep= ' ', header=0, na_values=['-999.0','-999'])
Out[7]: 
    A    B
0 NaN  1.2
1   2  NaN
2   3  4.5

In [8]: read_csv(path, sep= ' ', header=0, na_values=[-999,-999.0])
Out[8]: 
    A    B
0 NaN  1.2
1   2  NaN
2   3  4.5
```

I downloaded the latest master this morning, so it should be up to date. I may just be misunderstanding how this is supposed to work. I only passed the integer version of the missing data indicator (999 in your example), assuming this would also match against the float value (999.0) in my csv file (e.g., 999.0), but it doesn't seem to. How is it supposed to work?

From: jreback <notifications@github.com<mailto:notifications@github.com>>
Reply-To: pydata/pandas <reply@reply.github.com<mailto:reply@reply.github.com>>
Date: Monday, June 10, 2013 11:07 AM
To: pydata/pandas <pandas@noreply.github.com<mailto:pandas@noreply.github.com>>
Cc: Richard Stanton <stanton@haas.berkeley.edu<mailto:stanton@haas.berkeley.edu>>
Subject: Re: [pandas] Odd behavior from read_csv with na_values set to non-string values (#3611)

Are you on latest master?

In [9]: pd.**version**
Out[9]: '0.11.1.dev-2b7efc3'

In [4]: path = 'test.csv'

In [5]: df = DataFrame({'A' : [-999, 2, 3], 'B' : [1.2, -999, 4.5]})

In [10]: df
Out[10]:
     A      B
0 -999    1.2
1    2 -999.0
2    3    4.5

In [6]: df.to_csv(path, sep=' ', index=False)

In [7]: read_csv(path, sep= ' ', header=0, na_values=['-999.0','-999'])
Out[7]:
    A    B
0 NaN  1.2
1   2  NaN
2   3  4.5

In [8]: read_csv(path, sep= ' ', header=0, na_values=[-999,-999.0])
Out[8]:
    A    B
0 NaN  1.2
1   2  NaN
2   3  4.5

—
Reply to this email directly or view it on GitHubhttps://github.com/pydata/pandas/issues/3611#issuecomment-19216006.

hmmm....I think this is not working like it should....


Resolving patch:
diff --git a/pandas/io/parsers.py b/pandas/io/parsers.py
--- a/pandas/io/parsers.py
+++ b/pandas/io/parsers.py
@@ -297,6 +297,7 @@ def parser_f(filepath_or_buffer,
                  skipfooter=None,
                  skip_footer=0,
                  na_values=None,
+                 na_fvalues=None,
                  true_values=None,
                  false_values=None,
                  delimiter=None,
@@ -359,6 +360,7 @@ def parser_f(filepath_or_buffer,
                     prefix=prefix,
                     skiprows=skiprows,
                     na_values=na_values,
+                    na_fvalues=na_fvalues,
                     true_values=true_values,
                     false_values=false_values,
                     keep_default_na=keep_default_na,
@@ -554,7 +556,7 @@ def _clean_options(self, options, engine):
             converters = {}
 
         # Converting values to NA
-        na_values = _clean_na_values(na_values, keep_default_na)
+        na_values, na_fvalues = _clean_na_values(na_values, keep_default_na)
 
         if com.is_integer(skiprows):
             skiprows = range(skiprows)
@@ -565,6 +567,7 @@ def _clean_options(self, options, engine):
         result['names'] = names
         result['converters'] = converters
         result['na_values'] = na_values
+        result['na_fvalues'] = na_fvalues
         result['skiprows'] = skiprows
 
         return result, engine
@@ -644,6 +647,7 @@ def __init__(self, kwds):
         self.keep_date_col = kwds.pop('keep_date_col', False)
 
         self.na_values = kwds.get('na_values')
+        self.na_fvalues = kwds.get('na_fvalues')
         self.true_values = kwds.get('true_values')
         self.false_values = kwds.get('false_values')
         self.tupleize_cols = kwds.get('tupleize_cols',True)
@@ -837,31 +841,34 @@ def _agg_index(self, index, try_parse_dates=True):
                 arr = self._date_conv(arr)
 
             col_na_values = self.na_values
+            col_na_fvalues = self.na_fvalues
 
             if isinstance(self.na_values, dict):
                 col_name = self.index_names[i]
                 if col_name is not None:
-                    col_na_values = _get_na_values(col_name,
-                                                   self.na_values)
-
-            arr, _ = self._convert_types(arr, col_na_values)
+                    col_na_values, col_na_fvalues = _get_na_values(col_name,
+                                                                   self.na_values,
+                                                                   self.na_fvalues)
+                    
+            arr, _ = self._convert_types(arr, col_na_values | col_na_fvalues)
             arrays.append(arr)
 
         index = MultiIndex.from_arrays(arrays, names=self.index_names)
 
         return index
 
-    def _convert_to_ndarrays(self, dct, na_values, verbose=False,
+    def _convert_to_ndarrays(self, dct, na_values, na_fvalues, verbose=False,
                              converters=None):
         result = {}
         for c, values in dct.iteritems():
             conv_f = None if converters is None else converters.get(c, None)
-            col_na_values = _get_na_values(c, na_values)
+            col_na_values, col_na_fvalues = _get_na_values(c, na_values, na_fvalues)
             coerce_type = True
             if conv_f is not None:
                 values = lib.map_infer(values, conv_f)
                 coerce_type = False
-            cvals, na_count = self._convert_types(values, col_na_values,
+            cvals, na_count = self._convert_types(values,
+                                                  set(col_na_values) | col_na_fvalues,
                                                   coerce_type)
             result[c] = cvals
             if verbose and na_count:
@@ -1370,7 +1377,7 @@ def _convert_data(self, data):
                 col = self.orig_names[col]
             clean_conv[col] = f
 
-        return self._convert_to_ndarrays(data, self.na_values, self.verbose,
+        return self._convert_to_ndarrays(data, self.na_values, self.na_fvalues, self.verbose,
                                          clean_conv)
 
     def _infer_columns(self):
@@ -1754,37 +1761,26 @@ def _try_convert_dates(parser, colspec, data_dict, columns):
 
 
 def _clean_na_values(na_values, keep_default_na=True):
+
     if na_values is None and keep_default_na:
         na_values = _NA_VALUES
+        na_fvalues = set()
     elif isinstance(na_values, dict):
         if keep_default_na:
             for k, v in na_values.iteritems():
                 v = set(list(v)) | _NA_VALUES
                 na_values[k] = v
+        na_fvalues = dict([ (k, _floatify_na_values(v)) for k, v in na_values.items() ])
     else:
         if not com.is_list_like(na_values):
             na_values = [na_values]
-        na_values = set(_stringify_na_values(na_values))
+        na_values = _stringify_na_values(na_values)
         if keep_default_na:
             na_values = na_values | _NA_VALUES
 
-    return na_values
+        na_fvalues = _floatify_na_values(na_values)
 
-def _stringify_na_values(na_values):
-    """ return a stringified and numeric for these values """
-    result = []
-    for x in na_values:
-        result.append(str(x))
-        result.append(x)
-        try:
-            result.append(float(x))
-        except:
-            pass
-        try:
-            result.append(int(x))
-        except:
-            pass
-    return result
+    return na_values, na_fvalues
 
 def _clean_index_names(columns, index_col):
     if not _is_index_col(index_col):
@@ -1832,14 +1828,52 @@ def _get_empty_meta(columns, index_col, index_names):
     return index, columns, {}
 
 
-def _get_na_values(col, na_values):
+def _floatify_na_values(na_values):
+    # create float versions of the na_values
+    result = set()
+    for v in na_values:
+        try:
+            v = float(v)
+            if not np.isnan(v):
+                result.add(v)
+        except:
+            pass
+    return result
+
+def _stringify_na_values(na_values):
+    """ return a stringified and numeric for these values """
+    result = []
+    for x in na_values:
+        result.append(str(x))
+        result.append(x)
+        try:
+            v = float(x)
+
+            # we are like 999 here
+            if v == int(v):
+                v = int(v)
+                result.append("%s.0" % v)
+                result.append(str(v))
+
+            result.append(v)
+        except:
+            pass
+        try:
+            result.append(int(x))
+        except:
+            pass
+    return set(result)
+
+def _get_na_values(col, na_values, na_fvalues):
     if isinstance(na_values, dict):
         if col in na_values:
-            return set(_stringify_na_values(list(na_values[col])))
+            values = na_values[col]
+            fvalues = na_fvalues[col]
+            return na_values[col], na_fvalues[col]
         else:
-            return _NA_VALUES
+            return _NA_VALUES, set()
     else:
-        return na_values
+        return na_values, na_fvalues
 
 
 def _get_col_names(colspec, columns):


Test patch:
diff --git a/pandas/io/tests/test_parsers.py b/pandas/io/tests/test_parsers.py
--- a/pandas/io/tests/test_parsers.py
+++ b/pandas/io/tests/test_parsers.py
@@ -540,6 +540,36 @@ def test_non_string_na_values(self):
             tm.assert_frame_equal(result1,result2)
             tm.assert_frame_equal(result2,result3)
 
+            result4 = read_csv(path, sep= ' ', header=0, na_values=['-999.0'])
+            result5 = read_csv(path, sep= ' ', header=0, na_values=['-999'])
+            result6 = read_csv(path, sep= ' ', header=0, na_values=[-999.0])
+            result7 = read_csv(path, sep= ' ', header=0, na_values=[-999])
+            tm.assert_frame_equal(result4,result3)
+            tm.assert_frame_equal(result5,result3)
+            tm.assert_frame_equal(result6,result3)
+            tm.assert_frame_equal(result7,result3)
+
+            good_compare = result3
+
+            # with an odd float format, so we can't match the string 999.0 exactly,
+            # but need float matching
+            df.to_csv(path, sep=' ', index=False, float_format = '%.3f')
+            result1 = read_csv(path, sep= ' ', header=0, na_values=['-999.0','-999'])
+            result2 = read_csv(path, sep= ' ', header=0, na_values=[-999,-999.0])
+            result3 = read_csv(path, sep= ' ', header=0, na_values=[-999.0,-999])
+            tm.assert_frame_equal(result1,good_compare)
+            tm.assert_frame_equal(result2,good_compare)
+            tm.assert_frame_equal(result3,good_compare)
+
+            result4 = read_csv(path, sep= ' ', header=0, na_values=['-999.0'])
+            result5 = read_csv(path, sep= ' ', header=0, na_values=['-999'])
+            result6 = read_csv(path, sep= ' ', header=0, na_values=[-999.0])
+            result7 = read_csv(path, sep= ' ', header=0, na_values=[-999])
+            tm.assert_frame_equal(result4,good_compare)
+            tm.assert_frame_equal(result5,good_compare)
+            tm.assert_frame_equal(result6,good_compare)
+            tm.assert_frame_equal(result7,good_compare)
+
     def test_custom_na_values(self):
         data = """A,B,C
 ignore,this,row