Repository: pandas-dev/pandas

Issue:
BUG: Python parser breaks with quotes and multi-char sep 
On `master` (<a href="https://github.com/pydata/pandas/commit/b722222f5ea760a3f3df4d063309949eb4956674">b722222</a>):

``` python
>>> data = 'a,,b\n1,,a\n2,,"2,,b"'
>>> read_csv(StringIO(data), sep=',,', engine='python')
...
ValueError: Expected 2 fields in line 3, saw 3
```

I expect this command to work, but because no parsing is done on quoted fields as can be seen <a href="https://github.com/pydata/pandas/blob/master/pandas/io/parsers.py#L1847">here</a>, an extra field is produced, breaking the parser.  Note that this does not affect the C parser because multi-char delimiters are not supported.  Similar to what we saw in #10911 and #12775, but unless we want to write the `tokenizer.c` code in Python, a similar fix does not seem trivial.



Discussion:
isn't a multi-char sep turned into a regex? (in python engine)

Yes, but that's besides the point in the grand scheme of things.  It's just that the support multi-char or regex isn't really there in the Python engine if it breaks in simple cases like this.

does it work when no quotes are there?

Should work AFAICT without quotes.  I only stumbled across it when I was trying to figure out a clever way to get `regex` or multi-char `sep` to work on the C engine by doing some pre-processing in Python layer, but this is an obstacle to that.

ok, why don't we just raise NotImplementedError if quoting is not None and python engine is selected (onlly for multi-char sep?)

I suspect its not worth it to actually implement, but an error would be fine.

The extra field will be created irregardless of the quoting style.  It's because of the faulty splitting that the error will be raised, not the quoting style.  The quotes in this example just serve to illustrate why the splitting is faulty.

The alternative to make it "work" is to `regex` replace the `sep` with a more conventional one like `,` but of course that means loss of data in the cases of quoted or commented data.  So my first instinct is to say that we actually don't support `regex` or multi-char `sep` for either engine.  How about that?

I actually get an error when I have a bad line and don't specify a delimiter (using the python engine), due to the following code:
```python
                if len(self.delimiter) > 1 and self.quoting != csv.QUOTE_NONE:
                    # see gh-13374
                    reason = ('Error could possibly be due to quotes being '
                              'ignored when a multi-char delimiter is used.')
                    msg += '. ' + reason
```
A `TypeError` is thrown because `self.delimiter` which is `NoneType` doesn't have a `len()` operator.

It should be relatively simple to fix, just change the statement to 
```python
    if self.delimiter and len(self.delimiter) > 1 and self.quoting != csv.QUOTE_NONE:
``` 
Can you reproduce this?

*edit*
I've confirmed I can reproduce the issue, I can provide sample data but you can just use a simple csv with a header and three columns, one with an extra element. Here's some sample code:

```python
from pandas import read_csv
from StringIO import StringIO

csv_str = "a,b,c\n0,1,2\n3,4,5,6\n7,8,9"

chunks = read_csv(StringIO(csv_str), header=0, chunksize=2048, sep=None,
                  error_bad_lines=False, warn_bad_lines=True,
                  engine='python', iterator=True,
                  tupleize_cols=True)
for chunk in chunks:
    print chunk
```

*edit simplified sample*
@matthax : Preferable would if you could provide a CSV in string form for the rest of us to reproduce e.g.:

~~~python
data = ...
read_csv(StringIO(data), engine='python',...)
~~~

Resolving patch:
diff --git a/pandas/io/parsers.py b/pandas/io/parsers.py
--- a/pandas/io/parsers.py
+++ b/pandas/io/parsers.py
@@ -2836,7 +2836,9 @@ def _rows_to_cols(self, content):
             for row_num, actual_len in bad_lines:
                 msg = ('Expected %d fields in line %d, saw %d' %
                        (col_len, row_num + 1, actual_len))
-                if len(self.delimiter) > 1 and self.quoting != csv.QUOTE_NONE:
+                if (self.delimiter and
+                        len(self.delimiter) > 1 and
+                        self.quoting != csv.QUOTE_NONE):
                     # see gh-13374
                     reason = ('Error could possibly be due to quotes being '
                               'ignored when a multi-char delimiter is used.')


Test patch:
diff --git a/pandas/tests/io/parser/python_parser_only.py b/pandas/tests/io/parser/python_parser_only.py
--- a/pandas/tests/io/parser/python_parser_only.py
+++ b/pandas/tests/io/parser/python_parser_only.py
@@ -218,6 +218,25 @@ def test_multi_char_sep_quotes(self):
                 self.read_csv(StringIO(data), sep=',,',
                               quoting=csv.QUOTE_NONE)
 
+    def test_none_delimiter(self):
+        # see gh-13374 and gh-17465
+
+        data = "a,b,c\n0,1,2\n3,4,5,6\n7,8,9"
+        expected = DataFrame({'a': [0, 7],
+                              'b': [1, 8],
+                              'c': [2, 9]})
+
+        # We expect the third line in the data to be
+        # skipped because it is malformed,
+        # but we do not expect any errors to occur.
+        result = self.read_csv(StringIO(data), header=0,
+                               sep=None,
+                               error_bad_lines=False,
+                               warn_bad_lines=True,
+                               engine='python',
+                               tupleize_cols=True)
+        tm.assert_frame_equal(result, expected)
+
     def test_skipfooter_bad_row(self):
         # see gh-13879
         # see gh-15910