Repository: pandas-dev/pandas

Issue:
read_csv character encoding bug?
This is a weird one from [StackOverflow](http://stackoverflow.com/questions/14488697/pandas-read-csv-failing-on-columns-with-null-characters), this file has some `\x00`s which seem to be ignored when printing but confuse `read_csv`:

```
x = 'x,y\n \x00\x00\x00,Reg\n \x00\x00\x00,Reg\nI,Swp\nI,Swp\n'
X = StringIO(x)

In [3]: pd.read_csv(X)
Out[3]: 
     x    y
0          
1  NaN  NaN
2    I  Swp
3    I  Swp

In [4]: print x
x,y
 ,Reg
 ,Reg
I,Swp
I,Swp
```



Discussion:
Yes. The tokenizer uses null terminators in a couple of places as a marker, I'll have to look to see exactly why this is failing. 

In complete fairness the `csv` module doesn't handle NULL bytes:

```
In [6]: import csv; f = csv.reader(StringIO(x))

In [7]: next(f)
Out[7]: ['x', 'y']

In [8]: next(f)
---------------------------------------------------------------------------
Error                                     Traceback (most recent call last)
<ipython-input-8-468f0afdf1b9> in <module>()
----> 1 next(f)

Error: line contains NULL byte
```

Pushing this issue to some other day (not 0.11)

I know this is an old issue, but I'd like to give it a little bump. I have customers giving us a dump of some of their production database tables. The files we are getting are tab delimited and values that were null in the database are null bytes in the text file. I don't know how to get pandas to read this without having to do some manual munging first.

I'm sorry this never got fixed! I have also seen null bytes used to code nulls. @jreback @jorisvandenbossche let's slate this for 0.18?

I'd like to make some changes to the CSV tokenizer to hopefully improve its performance, that will also allow us to get rid of the null terminators that are complicating issues like this, but it would make most sense to understand for 1.0 / libpandas. Question is whether there is a quick fix for this particular issue with the existing tokenization strategy

I don't believe this is an issue with the C engine anymore:

``` python
>>> from pandas.compat import StringIO
>>> from pandas import read_csv
>>> data = 'x,y\n\x00\x00\x00,Reg\n\x00\x00\x00,Reg\nI,Swp\nI,Swp\n'
>>> read_csv(StringIO(data), engine='c')
     x    y
0  NaN  Reg
1  NaN  Reg
2    I  Swp
3    I  Swp
```

Unfortunately, as @wesm pointed out <a href="https://github.com/pydata/pandas/issues/2741#issuecomment-15628473">here</a>, it still does fail with the Python engine:

``` python
>>> read_csv(StringIO(data), engine='python')
...
_csv.Error: line contains NULL byte
```

This issue however, seems beyond our control, so I'm not sure if we should still classify this as a BUG on the `pandas` end if the issue is originating in Python's `csv` module.

if u want to put up tests for the c engine and s nice error message Python engine then can close


Resolving patch:
diff --git a/pandas/io/parsers.py b/pandas/io/parsers.py
--- a/pandas/io/parsers.py
+++ b/pandas/io/parsers.py
@@ -2190,7 +2190,17 @@ def _next_line(self):
                 next(self.data)
 
             while True:
-                orig_line = next(self.data)
+                try:
+                    orig_line = next(self.data)
+                except csv.Error as e:
+                    if 'NULL byte' in str(e):
+                        raise csv.Error(
+                            'NULL byte detected. This byte '
+                            'cannot be processed in Python\'s '
+                            'native csv library at the moment, '
+                            'so please pass in engine=\'c\' instead.')
+                    else:
+                        raise
                 line = self._check_comments([orig_line])[0]
                 self.pos += 1
                 if (not self.skip_blank_lines and


Test patch:
diff --git a/pandas/io/tests/parser/common.py b/pandas/io/tests/parser/common.py
--- a/pandas/io/tests/parser/common.py
+++ b/pandas/io/tests/parser/common.py
@@ -1503,3 +1503,19 @@ def test_memory_map(self):
 
         out = self.read_csv(mmap_file, memory_map=True)
         tm.assert_frame_equal(out, expected)
+
+    def test_null_byte_char(self):
+        # see gh-2741
+        data = '\x00,foo'
+        cols = ['a', 'b']
+
+        expected = DataFrame([[np.nan, 'foo']],
+                             columns=cols)
+
+        if self.engine == 'c':
+            out = self.read_csv(StringIO(data), names=cols)
+            tm.assert_frame_equal(out, expected)
+        else:
+            msg = "NULL byte detected"
+            with tm.assertRaisesRegexp(csv.Error, msg):
+                self.read_csv(StringIO(data), names=cols)