Repository: pandas-dev/pandas
Pull request: Raise error in read_csv when arguments header and prefix both are not None
- [x] closes #27394
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry


Problem statement:
pd.read_csv prefix parameter seems do not works
#### Code Sample, a copy-pastable example if possible
```test.csv
field,data,values
text1,34,hh,fail
text2,76,tt,fail2
```
```python
import pandas as pd

if __name__ == "__main__":
    data_list = pd.read_csv('test.csv', prefix="X")
    print(data_list)

```
#### Problem description
A documentation said: 
```
prefix : str, optional
    Prefix to add to column numbers when no header, e.g. ‘X’ for X0, X1, …
```
If I understood correctly this documentation I should get this

#### Expected Output
```
field    data    values  X0
text1     34    hh        fail
text2     76    tt         fail2
```
But I got this anyway:

#### Current Output
```
          field  data    values
text1     34    hh      fail
text2     76    tt        fail2
```
#### Output of ``pd.show_versions()``

<details>

[paste the output of ``pd.show_versions()`` here below this line]
INSTALLED VERSIONS
------------------
commit: None
python: 3.6.3.final.0
python-bits: 64
OS: Windows
OS-release: 8.1
machine: AMD64
processor: Intel64 Family 6 Model 42 Stepping 7, GenuineIntel
byteorder: little
LC_ALL: None
LANG: None
LOCALE: None.None

pandas: 0.24.2
pytest: None
pip: 19.1.1
setuptools: 40.1.0
Cython: None
numpy: 1.15.0
scipy: None
pyarrow: None
xarray: None
IPython: None
sphinx: None
patsy: None
dateutil: 2.7.5
pytz: 2018.5
blosc: None
bottleneck: None
tables: None
numexpr: None
feather: None
matplotlib: 2.2.3
openpyxl: 2.5.5
xlrd: None
xlwt: None
xlsxwriter: None
lxml.etree: 4.1.1
bs4: 4.6.0
html5lib: None
sqlalchemy: 1.2.19
pymysql: None
psycopg2: None
jinja2: 2.10
s3fs: None
fastparquet: None
pandas_gbq: None
pandas_datareader: None
gcsfs: None
None


</details>

Hints from issue discussion:
I think you misunderstand. That's only used when there's no header and the default `range(n)` header is used.

```python
In [9]: pd.read_csv(io.StringIO('0,1\n2,3'), header=None, prefix="X_")
Out[9]:
   X_0  X_1
0    0    1
1    2    3

In [10]: pd.read_csv(io.StringIO('0,1\n2,3'), prefix="X_")
Out[10]:
   0  1
0  2  3
```

We should probably raise when `header` and `prefix` are both not None.
Ah. That's it. "Prefix to add to column numbers when no header **at all**, e.g. ‘X’ for X0, X1, …"
So I consider it as NOTABUG

I look around for function to drop columns **without a header**. df.dropna() did not help? Could I create a feature request?
> I look around for function to drop columns without a header

I don't understand. Every column in a DataFrame always has a name. You can't have a "column without a header".
@TomAugspurger , what do you mean by

> We should probably raise when `header` and `prefix` are both not None.

Could you explain me the issue. I would like to take it up.

@sameshl great!

If you look at https://github.com/pandas-dev/pandas/issues/27394#issuecomment-511374439, `In[10]` should raise a `TypeError` saying that `prefix` can only be used when `header=None`.

@WillAyd @simonjayhawkins does that sound right to you?
Makes sense to me
@TomAugspurger I see. Thanks for the info. Will work on it!
> > I look around for function to drop columns without a header
> 
> I don't understand. Every column in a DataFrame always has a name. You can't have a "column without a header".

Yes. I totally agree with you. But some csv files have columns without headers. Peoples make mistakes... Would be nice if pandas.read_csv method have an option to cut off such columns (without header).

ex. https://stackoverflow.com/questions/50301544/how-to-delete-columns-without-headers-in-python-pandas-read-csv
The answers there seem reasonable to me.

On Mon, Jul 15, 2019 at 10:55 AM Gutsycat <notifications@github.com> wrote:

> I look around for function to drop columns without a header
>
> I don't understand. Every column in a DataFrame always has a name. You
> can't have a "column without a header".
>
> Yes. I totally agree with you. But some csv files have columns without
> headers. Peoples make mistakes... Would be nice if pandas.read_csv method
> have an option to cut off such columns (without header).
>
> ex.
> https://stackoverflow.com/questions/50301544/how-to-delete-columns-without-headers-in-python-pandas-read-csv
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <https://github.com/pandas-dev/pandas/issues/27394?email_source=notifications&email_token=AAKAOIVQWY2TZETQARIYWLLP7SMVPA5CNFSM4IDUF6IKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODZ6EKHA#issuecomment-511460636>,
> or mute the thread
> <https://github.com/notifications/unsubscribe-auth/AAKAOITP76VTC46XUF4HKVDP7SMVPANCNFSM4IDUF6IA>
> .
>

> The answers there seem reasonable to me.
> […](#)

Yes. But it is not the same that I suggest: 

Now pandas.read_csv has this

```
error_bad_lines : bool, default True
Lines with too many fields (e.g. a csv line with too many commas) will by default cause an exception to be raised, and no DataFrame will be returned. If False, then these “bad lines” will dropped from
the DataFrame that is returned.
```

May be do the same for columns?

```
error_bad_columns : bool, default True
Columns with empty header and/or [other "damages"] will by default cause an exception to be raised, and no DataFrame will be returned. If False, then these **“bad columns” will dropped from the DataFrame that is returned.**
```

What do you think?

Commit message: remove extra quotes

Patch to review:
diff --git a/pandas/io/parsers.py b/pandas/io/parsers.py
index 84a8b5b2a94fe..8be131e589a7c 100755
--- a/pandas/io/parsers.py
+++ b/pandas/io/parsers.py
@@ -1908,6 +1908,10 @@ def __init__(self, src, **kwds):
                 ]
             else:
                 self.names = list(range(self._reader.table_width))
+        elif self.prefix:
+            raise ValueError(
+                "Argument prefix must be None if argument header is not None"
+            )
 
         # gh-9755
         #
diff --git a/pandas/tests/io/parser/test_header.py b/pandas/tests/io/parser/test_header.py
index 7dc106ef0c186..34d4b05555005 100644
--- a/pandas/tests/io/parser/test_header.py
+++ b/pandas/tests/io/parser/test_header.py
@@ -24,6 +24,14 @@ def test_read_with_bad_header(all_parsers):
         parser.read_csv(s, header=[10])
 
 
+def test_read_raises_on_header_prefix(all_parsers):
+    parser = all_parsers
+    msg = "Argument prefix must be None if argument header is not None"
+    with pytest.raises(ValueError, match=msg):
+        s = StringIO("0,1\n2,3")
+        parser.read_csv(s, header=0, prefix="_X")
+
+
 def test_negative_header(all_parsers):
     # see gh-27779
     parser = all_parsers


Code review comments:
diff_hunk: @@ -24,6 +24,14 @@ def test_read_with_bad_header(all_parsers):
         parser.read_csv(s, header=[10])
 
 
+def test_read_raises_on_header_prefix(all_parsers):
+    parser = all_parsers
+    msg = "Argument prefix must be None if argument header is not None"
+    with pytest.raises(ValueError, match=msg):
+        s = StringIO("0,1\n2,3")
path: pandas/tests/io/parser/test_header.py
text: @user1:
move outside the context manager