Repository: pandas-dev/pandas
Pull request: TST: Add the default separator test for PythonParser
- [x] closes #17333
- [x] tests added / passed
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Problem statement:
Problem detecting separator (sep=None) with a DataFrame with a single column
#### Code Sample

```
from io import StringIO

import pandas as pd

TESTDATA=StringIO("""col1
    val1
    val2d
    val3
    val4
    """)

df = pd.read_csv(TESTDATA, sep=None, engine='python')
print(df)
```
returns:

```
           c  l1
0       val1 NaN
1      val2d NaN
2       val3 NaN
3       val4 NaN
```

#### Problem

I would expect the input to be detected as a DataFrame with one column (with no seperator) which is not the case here...

#### Expected Output

```
        col1
0       val1
1      val2d
2       val3
3       val4
```

#### Output of ``pd.show_versions()``

```
INSTALLED VERSIONS
------------------
commit: None
python: 3.5.2.final.0
python-bits: 64
OS: Linux
OS-release: 4.4.0-92-generic
machine: x86_64
processor: x86_64
byteorder: little
LC_ALL: None
LANG: en_US.UTF-8
LOCALE: en_US.UTF-8

pandas: 0.20.3
pytest: None
pip: 9.0.1
setuptools: 35.0.2
Cython: 0.23.4
numpy: 1.13.1
scipy: 0.19.0
xarray: None
IPython: 6.0.0
sphinx: 1.5.5
patsy: None
dateutil: 2.6.1
pytz: 2017.2
blosc: None
bottleneck: None
tables: 3.2.2
numexpr: 2.4.3
feather: None
matplotlib: 1.5.1
openpyxl: None
xlrd: 1.0.0
xlwt: None
xlsxwriter: 0.7.3
lxml: None
bs4: None
html5lib: 0.999999999
sqlalchemy: 1.1.9
pymysql: None
psycopg2: 2.7.1 (dt dec pq3 ext lo64)
jinja2: 2.9.6
s3fs: None
pandas_gbq: None
pandas_datareader: None
```

Hints from issue discussion:
@Leobouloc : Thanks for reporting this!  A little odd, especially since you somehow drop an "o" from `col1`.  As a workaround, you should pass in `delim_whitespace=True`, and you should get the table that you expect.  I can't tell at the moment whether this error is stemming from Python's native `csv` parsing or our own wrapper around it.

Investigation (and PR) are welcome!  
@gfyoung This seems right behavior.
When `sep=None`, sniff the first line and check what the separator is.
https://github.com/pandas-dev/pandas/blob/master/pandas/io/parsers.py#L2167

The `csv` library treats `'o'` as separator.
https://docs.python.org/2/library/csv.html#csv.Sniffer

Here is the example.
### Input
```python
import csv

sniffed = csv.Sniffer().sniff('col1')
sniffed.delimiter
```
### Output
```python
'o'
```
@Licht-T : Ah, interesting...this issue is beyond our control then since the sniffing behavior is on Python.  Do you mind adding this example as a test to the Python engine?

@Leobouloc : We will close this issue after the test is added because the behavior causing this output is outside of our control.  Feel free to ping the Python devs about this if you think this is sufficiently bothersome to address.
As this issue can be resolved by simply adding a test, I would want to add this addition to the release.  If not resolved before `0.21.0`, ping, and we'll get it added in time.
@gfyoung Okay! I will write this test asap.

Commit message: TST: Add the default separator test for PythonParser

Patch to review:
diff --git a/pandas/tests/io/parser/python_parser_only.py b/pandas/tests/io/parser/python_parser_only.py
index a0784d3aeae2d..220825782ae26 100644
--- a/pandas/tests/io/parser/python_parser_only.py
+++ b/pandas/tests/io/parser/python_parser_only.py
@@ -19,6 +19,15 @@
 
 class PythonParserTests(object):
 
+    def test_default_separator(self):
+        # GH17333
+        text = 'aob\n1o2\n3o4'
+        expected = DataFrame({'a': [1, 3], 'b': [2, 4]})
+
+        result = self.read_csv(StringIO(text), sep=None)
+
+        tm.assert_frame_equal(result, expected)
+
     def test_invalid_skipfooter(self):
         text = "a\n1\n2"
 


Code review comments:
diff_hunk: @@ -21,6 +21,7 @@ class PythonParserTests(object):
 
     def test_default_separator(self):
         # GH17333
+        # The csv parser in Python treats 'o' as separator.
path: pandas/tests/io/parser/python_parser_only.py
text: @user1:
Not "parser", actually "sniffer"