Repository: pandas-dev/pandas

Issue:
Bug: read_csv does not work with chr(254) as quotechar parameter



Discussion:
pls show a copy-pastable example

I can reproduce:

```
>>> pd.read_csv(io.StringIO(chr(127)+"a, "+chr(127)+",b"), quotechar=chr(127), header=None)

     0  1
0  a,   b

>>> pd.read_csv(io.StringIO(chr(254)+"a, "+chr(254)+",b"), quotechar=chr(254), header=None)

    0   1  2
0  þa   þ  b
```

The problem sets in at `chr(128)`, which just so happens to be the first value when `len(chr(x).encode("utf-8")) > 1`, and we're using only `char quotechar` in parser.pyx.  

I suspect this would be a bit of a headache to fix, and it would only really be useful for people who decided to use a small thorn as a separator, and who were _also_ using every other viable separator character so that a pass to replace the thorns wouldn't work, _and_ who can't use the python engine for some reason.

```
>>> pd.read_csv(io.StringIO(chr(254)+"a, "+chr(254)+",b"), quotechar=chr(254), 
header=None, engine='python')

     0  1
0  a,   b
```

I like the idea of being able to use anything anywhere, but in terms of developer time, maybe we should just raise NotimplementedError if you're using the C engine in these circumstances..

thanks @dsm054 for the example.

@jreback : We have a similar issue with `sep` in which we cannot use multi-char `sep` with the C engine.  I'm inclined to agree with @joelschw and raise `NotImplementedError` (or some kind of error) for such a `quotechar` for the time being.  Thoughts?
multi-char sep for sure should be NotImplementedError (when c-engine is given). so the same here is fine.

Resolving patch:
diff --git a/pandas/io/parsers.py b/pandas/io/parsers.py
--- a/pandas/io/parsers.py
+++ b/pandas/io/parsers.py
@@ -841,6 +841,17 @@ def _clean_options(self, options, engine):
                                       encoding=encoding)
                 engine = 'python'
 
+        quotechar = options['quotechar']
+        if (quotechar is not None and
+                isinstance(quotechar, (str, compat.text_type, bytes))):
+            if (len(quotechar) == 1 and ord(quotechar) > 127 and
+                    engine not in ('python', 'python-fwf')):
+                fallback_reason = ("ord(quotechar) > 127, meaning the "
+                                   "quotechar is larger than one byte, "
+                                   "and the 'c' engine does not support "
+                                   "such quotechars")
+                engine = 'python'
+
         if fallback_reason and engine_specified:
             raise ValueError(fallback_reason)
 


Test patch:
diff --git a/pandas/io/tests/parser/quoting.py b/pandas/io/tests/parser/quoting.py
--- a/pandas/io/tests/parser/quoting.py
+++ b/pandas/io/tests/parser/quoting.py
@@ -149,5 +149,5 @@ def test_quotechar_unicode(self):
 
         # Compared to Python 3.x, Python 2.x does not handle unicode well.
         if PY3:
-            result = self.read_csv(StringIO(data), quotechar=u('\u0394'))
+            result = self.read_csv(StringIO(data), quotechar=u('\u0001'))
             tm.assert_frame_equal(result, expected)
diff --git a/pandas/io/tests/parser/test_unsupported.py b/pandas/io/tests/parser/test_unsupported.py
--- a/pandas/io/tests/parser/test_unsupported.py
+++ b/pandas/io/tests/parser/test_unsupported.py
@@ -50,12 +50,16 @@ def test_c_engine(self):
                        sep=None, delim_whitespace=False)
         with tm.assertRaisesRegexp(ValueError, msg):
             read_table(StringIO(data), engine='c', sep=r'\s')
+        with tm.assertRaisesRegexp(ValueError, msg):
+            read_table(StringIO(data), engine='c', quotechar=chr(128))
         with tm.assertRaisesRegexp(ValueError, msg):
             read_table(StringIO(data), engine='c', skipfooter=1)
 
         # specify C-unsupported options without python-unsupported options
         with tm.assert_produces_warning(parsers.ParserWarning):
             read_table(StringIO(data), sep=None, delim_whitespace=False)
+        with tm.assert_produces_warning(parsers.ParserWarning):
+            read_table(StringIO(data), quotechar=chr(128))
         with tm.assert_produces_warning(parsers.ParserWarning):
             read_table(StringIO(data), sep=r'\s')
         with tm.assert_produces_warning(parsers.ParserWarning):