Repository: pandas-dev/pandas

Issue:
BUG: Python parser breaks with quotes and multi-char sep 
On `master` (<a href="https://github.com/pydata/pandas/commit/b722222f5ea760a3f3df4d063309949eb4956674">b722222</a>):

``` python
>>> data = 'a,,b\n1,,a\n2,,"2,,b"'
>>> read_csv(StringIO(data), sep=',,', engine='python')
...
ValueError: Expected 2 fields in line 3, saw 3
```

I expect this command to work, but because no parsing is done on quoted fields as can be seen <a href="https://github.com/pydata/pandas/blob/master/pandas/io/parsers.py#L1847">here</a>, an extra field is produced, breaking the parser.  Note that this does not affect the C parser because multi-char delimiters are not supported.  Similar to what we saw in #10911 and #12775, but unless we want to write the `tokenizer.c` code in Python, a similar fix does not seem trivial.



Discussion:
isn't a multi-char sep turned into a regex? (in python engine)

Yes, but that's besides the point in the grand scheme of things.  It's just that the support multi-char or regex isn't really there in the Python engine if it breaks in simple cases like this.

does it work when no quotes are there?

Should work AFAICT without quotes.  I only stumbled across it when I was trying to figure out a clever way to get `regex` or multi-char `sep` to work on the C engine by doing some pre-processing in Python layer, but this is an obstacle to that.

ok, why don't we just raise NotImplementedError if quoting is not None and python engine is selected (onlly for multi-char sep?)

I suspect its not worth it to actually implement, but an error would be fine.

The extra field will be created irregardless of the quoting style.  It's because of the faulty splitting that the error will be raised, not the quoting style.  The quotes in this example just serve to illustrate why the splitting is faulty.

The alternative to make it "work" is to `regex` replace the `sep` with a more conventional one like `,` but of course that means loss of data in the cases of quoted or commented data.  So my first instinct is to say that we actually don't support `regex` or multi-char `sep` for either engine.  How about that?


Resolving patch:
diff --git a/pandas/io/parsers.py b/pandas/io/parsers.py
--- a/pandas/io/parsers.py
+++ b/pandas/io/parsers.py
@@ -2515,6 +2515,11 @@ def _rows_to_cols(self, content):
 
             msg = ('Expected %d fields in line %d, saw %d' %
                    (col_len, row_num + 1, zip_len))
+            if len(self.delimiter) > 1 and self.quoting != csv.QUOTE_NONE:
+                # see gh-13374
+                reason = ('Error could possibly be due to quotes being '
+                          'ignored when a multi-char delimiter is used.')
+                msg += '. ' + reason
             raise ValueError(msg)
 
         if self.usecols:


Test patch:
diff --git a/pandas/io/tests/parser/python_parser_only.py b/pandas/io/tests/parser/python_parser_only.py
--- a/pandas/io/tests/parser/python_parser_only.py
+++ b/pandas/io/tests/parser/python_parser_only.py
@@ -7,6 +7,7 @@
 arguments when parsing.
 """
 
+import csv
 import sys
 import nose
 
@@ -204,3 +205,19 @@ def test_encoding_non_utf8_multichar_sep(self):
                                        sep=sep, names=['a', 'b'],
                                        encoding=encoding)
                 tm.assert_frame_equal(result, expected)
+
+    def test_multi_char_sep_quotes(self):
+        # see gh-13374
+
+        data = 'a,,b\n1,,a\n2,,"2,,b"'
+        msg = 'ignored when a multi-char delimiter is used'
+
+        with tm.assertRaisesRegexp(ValueError, msg):
+            self.read_csv(StringIO(data), sep=',,')
+
+        # We expect no match, so there should be an assertion
+        # error out of the inner context manager.
+        with tm.assertRaises(AssertionError):
+            with tm.assertRaisesRegexp(ValueError, msg):
+                self.read_csv(StringIO(data), sep=',,',
+                              quoting=csv.QUOTE_NONE)