Repository: pandas-dev/pandas
Pull request: TST: Raise ValueError and suggestion to use header=None if header=-1 is pa…
…ssed

- [x] closes #27779
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry


Problem statement:
Give more precise warning when passing `header=-1` in `pandas.read_csv`, or don't throw error
#### Code Sample, a copy-pastable example if possible

```python
from io import StringIO
import pandas as pd

print(f"Using pandas version {pd.__version__}")
pd.read_csv(StringIO('spam'), header=-1)
``` 
> Using pandas version 0.25.0
>
>---------------------------------------------------------------------------
>OverflowError  
#### Problem description

In previous versions of pandas (e.g. 0.24.2), this would have worked fine. Instead, now, one has to pass `header=None`. However, when working with old code after having updated pandas, it may be unclear where the problem lies.

#### Expected Output
```
      0
0  spam
```


#### Output of ``pd.show_versions()``

<details>

[paste the output of ``pd.show_versions()`` here below this line]
INSTALLED VERSIONS
------------------
commit           : None
python           : 3.7.3.final.0
python-bits      : 64
OS               : Linux
OS-release       : 4.15.0-55-generic
machine          : x86_64
processor        : x86_64
byteorder        : little
LC_ALL           : None
LANG             : en_GB.UTF-8
LOCALE           : en_GB.UTF-8

pandas           : 0.25.0
numpy            : 1.16.4
pytz             : 2019.1
dateutil         : 2.8.0
pip              : 19.1.1
setuptools       : 41.0.1
Cython           : 0.29.12
pytest           : 4.5.0
hypothesis       : None
sphinx           : None
blosc            : None
feather          : None
xlsxwriter       : None
lxml.etree       : None
html5lib         : None
pymysql          : None
psycopg2         : None
jinja2           : 2.10.1
IPython          : 7.7.0
pandas_datareader: None
bs4              : None
bottleneck       : None
fastparquet      : None
gcsfs            : None
lxml.etree       : None
matplotlib       : 3.1.0
numexpr          : None
odfpy            : None
openpyxl         : None
pandas_gbq       : None
pyarrow          : 0.13.0
pytables         : None
s3fs             : 0.2.1
scipy            : 1.3.0
sqlalchemy       : 1.3.5
tables           : None
xarray           : None
xlrd             : None
xlwt             : None
xlsxwriter       : None

</details>

Hints from issue discussion:
@MarcoGorelli are you interested in submitting a pull request?
If this would be of interest to you, then yes, sure! I'll get to work

On Fri, 9 Aug 2019, 19:18 Tom Augspurger, <notifications@github.com> wrote:

> @MarcoGorelli <https://github.com/MarcoGorelli> are you interested in
> submitting a pull request?
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <https://github.com/pandas-dev/pandas/issues/27779?email_source=notifications&email_token=AH7QVMG4FEQS62QA4FLRZF3QDWYIFA5CNFSM4IJWWNK2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD37NBBI#issuecomment-520016005>,
> or mute the thread
> <https://github.com/notifications/unsubscribe-auth/AH7QVMCECIDPIO55Z5N6TVLQDWYIFANCNFSM4IJWWNKQ>
> .
>

I've attempted to fix this by modifying `validate_header` from ` pandas/io/common.py` to check that `header=-1` wasn't passed and to raise an error advising user to pass `header=None` instead if their data has no header, along with an accompanying test.

Commit message: Add issue number to test as comment

Patch to review:
diff --git a/doc/source/whatsnew/v0.25.1.rst b/doc/source/whatsnew/v0.25.1.rst
index dfa216b1db56e..823064949859f 100644
--- a/doc/source/whatsnew/v0.25.1.rst
+++ b/doc/source/whatsnew/v0.25.1.rst
@@ -105,7 +105,7 @@ I/O
 ^^^
 
 - Avoid calling ``S3File.s3`` when reading parquet, as this was removed in s3fs version 0.3.0 (:issue:`27756`)
--
+- Better error message when a negative header is passed in :func:`pandas.read_csv` (:issue:`27779`)
 -
 
 Plotting
diff --git a/pandas/io/common.py b/pandas/io/common.py
index e01e473047b88..94bacf3816696 100644
--- a/pandas/io/common.py
+++ b/pandas/io/common.py
@@ -125,6 +125,12 @@ def _validate_header_arg(header) -> None:
             "the row(s) making up the column names"
         )
 
+    # In version 0.24, this would have worked.
+    if header == -1:
+        raise ValueError(
+            "Passing -1 to header is invalid. For no header, use header=None instead"
+        )
+
 
 def _stringify_path(
     filepath_or_buffer: FilePathOrBuffer[AnyStr]
diff --git a/pandas/tests/io/parser/test_header.py b/pandas/tests/io/parser/test_header.py
index 99e0181741998..aae0c9c0c2011 100644
--- a/pandas/tests/io/parser/test_header.py
+++ b/pandas/tests/io/parser/test_header.py
@@ -24,6 +24,21 @@ def test_read_with_bad_header(all_parsers):
         parser.read_csv(s, header=[10])
 
 
+def test_negative_header(all_parsers):
+    # see gh-27779
+    parser = all_parsers
+    data = """1,2,3,4,5
+6,7,8,9,10
+11,12,13,14,15
+"""
+    with pytest.raises(
+        ValueError,
+        match="Passing -1 to header is invalid. "
+        "For no header, use header=None instead",
+    ):
+        parser.read_csv(StringIO(data), header=-1)
+
+
 @pytest.mark.parametrize("header", [True, False])
 def test_bool_header_arg(all_parsers, header):
     # see gh-6114


Code review comments:
diff_hunk: @@ -125,6 +125,12 @@ def _validate_header_arg(header) -> None:
             "the row(s) making up the column names"
         )
 
+    # In version 0.24, this would have worked.
+    if header == -1:
path: pandas/io/common.py
text: @user1:
this should only validate if header is an int (or it could be an array-like of ints); these should raise if < 0 right?

@user2:
@user1 : I agree that this check needs to be expanded as you described

@user2:
The test that was added will of course need to be updated as well

@author:
Sure - thank you all for having taken the time to review my work!