Repository: pandas-dev/pandas
Pull request: ENH: Raise ParserWarning when length of names does not match length of data
- [x] closes #21768
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry

@gfyoung 

Raising ParserWarning now. Could change to FutureWarning, if we would like to deprecate for 2.0

As long as we are only raising a ParserWarning I am inclined to raise for trailing commas too.

Problem statement:
pd.read_table: Using space as delimiter on file with trailing space gives cryptic error

```python
import numpy as np
import os
import pandas as pd

# put test.csv in same folder as script
mydir = os.path.dirname(os.path.abspath(__file__))
csv_path = os.path.join(mydir, "test.csv")

df = pd.read_table(csv_path, sep=' ',
                   comment='#',
                   header=None,
                   skip_blank_lines=True,
                   names=["A", "B", "C", "D", "E", "F", "G"],
                   dtype={"A": np.int32,
                       "B": np.int32,
                       "C": np.float64,
                       "D": np.float64,
                       "E": np.float64,
                       "F": np.float64,
                       "G": np.int32})
```
`test.csv`:
```csv
2270433 3 21322.889 11924.667 5228.753 1.0 -1 
2270432 3 21322.297 11924.667 5228.605 1.0 2270433 
```

#### Problem description

Attempting to load test.csv with pd.read_table() results in the following errors: 
`TypeError: Cannot cast array from dtype('float64') to dtype('int32') according to the rule 'safe'`

and

`ValueError: cannot safely convert passed user dtype of int32 for float64 dtyped data in column 2`

Expected behavior:

Either trailing whitespace is ignored by Pandas, or throw a more informative error than "cannot safely convert passed user dtype of int32 for float64". It took me a really long time to figure out that this was caused by trailing spaces in the csv.

Hints from issue discussion:
How weird!  Investigation and PR are certainly welcome!
I'd argue this is behavior as expected.

Consider it was a comma separated file instead, the provided csv example would look like
```
2270433,3,21322.889,11924.667,5228.753,1.0,-1,
2270432,3,21322.297,11924.667,5228.605,1.0,2270433,
```
which would indicate that this csv file has 8 columns. (the last column being nulls). Since 7 columns are passed in the `names` parameter, it is forced to try and set the first column as the index and column `G` is a column of `np.nan`s. But since the dtype was specified to be `int32`, it raises the error which would be as per expectations as there's no way to convert a `nan` to an int.
@asishm : Ah, that's a very good point.  Nevertheless, it's not immediately obvious that that was the case IMO.  Might you be able to pinpoint where we perform that kind of logic?  I can see two ways to fix this:

* Add this to the documentation that this how `names` behaves
* Change the handling of what we do when `len(names)` is not equal to the length the parsed columns in the table (especially when `index_col=None` by default in this case).
@gfyoung I believe the documentation for `index_col` does specify this specific case:

> index_col : int or sequence or False, default None
    Column to use as the row labels of the DataFrame. If a sequence is given, a
    MultiIndex is used. **If you have a malformed file with delimiters at the end
    of each line, you might consider index_col=False to force pandas to _not_
    use the first column as the index (row names)**

I would agree that for the case `index_col=None` and where `names` is specified, it should be noted in the documentation that if `len(names) < len(actual_columns)`, then the first `x` (difference) would be treated as the index. I can't point exactly where this happens due to my unfamiliarity with pandas internals, but see below for examples:

```py
In [113]: s
Out[113]: '2270433 3 21322.889 11924.667 5228.753 1.0 -1 \n2270432 3 21322.297 11924.667 5228.605 1.0 2270433 '

In [114]: pd.read_csv(io.StringIO(s), header=None, sep=' ', names=list('abcd'))
Out[114]:
                                      a    b        c   d
2270433 3 21322.889 11924.667  5228.753  1.0       -1 NaN
2270432 3 21322.297 11924.667  5228.605  1.0  2270433 NaN

In [115]: pd.read_csv(io.StringIO(s), header=None, sep=' ', names=list('abcdef'))
Out[115]:
                   a          b         c    d        e   f
2270433 3  21322.889  11924.667  5228.753  1.0       -1 NaN
2270432 3  21322.297  11924.667  5228.605  1.0  2270433 NaN

In [116]: pd.read_csv(io.StringIO(s), header=None, sep=' ', names=list('abcdefg'))
Out[116]:
         a          b          c         d    e        f   g
2270433  3  21322.889  11924.667  5228.753  1.0       -1 NaN
2270432  3  21322.297  11924.667  5228.605  1.0  2270433 NaN

In [117]: pd.read_csv(io.StringIO(s), header=None, sep=' ', names=list('abcdefgh'))
Out[117]:
         a  b          c          d         e    f        g   h
0  2270433  3  21322.889  11924.667  5228.753  1.0       -1 NaN
1  2270432  3  21322.297  11924.667  5228.605  1.0  2270433 NaN

In [118]: pd.read_csv(io.StringIO(s), header=None, sep=' ', names=list('abcdefg'), index_col=False)
Out[118]:
         a  b          c          d         e    f        g
0  2270433  3  21322.889  11924.667  5228.753  1.0       -1
1  2270432  3  21322.297  11924.667  5228.605  1.0  2270433

In [119]: pd.read_csv(io.StringIO(s), header=None, sep=' ', names=list('abcdef'), index_col=False) # error - List index out of range

In [121]: pd.read_csv(io.StringIO(s), header=None, sep=' ', names=list('abcdef'), index_col=False, engine='python')
Out[121]:
         a  b          c          d         e    f
0  2270433  3  21322.889  11924.667  5228.753  1.0
1  2270432  3  21322.297  11924.667  5228.605  1.0
```

Something that is curious here is the difference between the python and C engine parsers when `index_col=False` as demonstrated in the last 2 examples. C parser raises the list index out of range, but python parser just discards the extra columns at the end.
@asishm : Thanks for pointing out.  I think the docs could be clarified further to make this more explicit.  That discrepancy that you found is a bug in the Python engine IMO.  Malformed CSV shouldn't just drop columns silently like that.
@gfyoung This behavior seems to have changed in the past. The c engine does not raise an error now either. Should we change this to raise errors instead of dropping the data for both engines?
IMO if data loss is imminent, then we should error, warn at least if we don't want to make an API change.
Thx for the response.

I looked into this a bit, We have a few cases covering exactly this behavior (#26218 for example). So warning would be appropriate here before raising probably?
Let's go with warning for now.

Commit message: Remove warning

Patch to review:
diff --git a/doc/source/whatsnew/v1.3.0.rst b/doc/source/whatsnew/v1.3.0.rst
index 622029adf357f..bfbaa46170a28 100644
--- a/doc/source/whatsnew/v1.3.0.rst
+++ b/doc/source/whatsnew/v1.3.0.rst
@@ -194,6 +194,7 @@ Other enhancements
 - Improve error message when ``usecols`` and ``names`` do not match for :func:`read_csv` and ``engine="c"`` (:issue:`29042`)
 - Improved consistency of error message when passing an invalid ``win_type`` argument in :class:`Window` (:issue:`15969`)
 - :func:`pandas.read_sql_query` now accepts a ``dtype`` argument to cast the columnar data from the SQL database based on user input (:issue:`10285`)
+- :func:`read_csv` now raising ``ParserWarning`` if length of header or given names does not match length of data when ``usecols`` is not specified (:issue:`21768`)
 - Improved integer type mapping from pandas to SQLAlchemy when using :meth:`DataFrame.to_sql` (:issue:`35076`)
 - :func:`to_numeric` now supports downcasting of nullable ``ExtensionDtype`` objects (:issue:`33013`)
 - Add support for dict-like names in :class:`MultiIndex.set_names` and :class:`MultiIndex.rename` (:issue:`20421`)
diff --git a/pandas/io/parsers/base_parser.py b/pandas/io/parsers/base_parser.py
index 49eb7ea9f26e5..a37d45e28a2fb 100644
--- a/pandas/io/parsers/base_parser.py
+++ b/pandas/io/parsers/base_parser.py
@@ -26,6 +26,7 @@
 from pandas._libs.parsers import STR_NA_VALUES
 from pandas._libs.tslibs import parsing
 from pandas._typing import (
+    ArrayLike,
     DtypeArg,
     FilePathOrBuffer,
     final,
@@ -798,6 +799,26 @@ def _do_date_conversions(self, names, data):
 
         return names, data
 
+    def _check_data_length(self, columns: List[str], data: List[ArrayLike]):
+        """Checks if length of data is equal to length of column names. One set of
+        trailing commas is allowed.
+        Parameters
+        ----------
+        columns: list of column names
+        data: list of array-likes containing the data column-wise
+        """
+        if not self.index_col and len(columns) != len(data) and columns:
+            if len(columns) == len(data) - 1 and np.all(
+                (is_object_dtype(data[-1]) and data[-1] == "") | isna(data[-1])
+            ):
+                return
+            warnings.warn(
+                "Length of header or names does not match length of data. This leads "
+                "to a loss of data with index_col=False.",
+                ParserWarning,
+                stacklevel=6,
+            )
+
     def _evaluate_usecols(self, usecols, names):
         """
         Check whether or not the 'usecols' parameter
diff --git a/pandas/io/parsers/c_parser_wrapper.py b/pandas/io/parsers/c_parser_wrapper.py
index 7a0e704d2fbc4..f7ba9213071fc 100644
--- a/pandas/io/parsers/c_parser_wrapper.py
+++ b/pandas/io/parsers/c_parser_wrapper.py
@@ -292,6 +292,8 @@ def read(self, nrows=None):
 
             # columns as list
             alldata = [x[1] for x in data_tups]
+            if self.usecols is None:
+                self._check_data_length(names, alldata)
 
             data = {k: v for k, (i, v) in zip(names, data_tups)}
 
diff --git a/pandas/io/parsers/python_parser.py b/pandas/io/parsers/python_parser.py
index 9082e41698913..e94cd5f597996 100644
--- a/pandas/io/parsers/python_parser.py
+++ b/pandas/io/parsers/python_parser.py
@@ -300,6 +300,8 @@ def _exclude_implicit_index(self, alldata):
             offset = len(self.index_col)  # type: ignore[has-type]
 
         len_alldata = len(alldata)
+        self._check_data_length(names, alldata)
+
         return {
             name: alldata[i + offset] for i, name in enumerate(names) if i < len_alldata
         }, names
diff --git a/pandas/tests/io/parser/common/__init__.py b/pandas/tests/io/parser/common/__init__.py
new file mode 100644
index 0000000000000..e69de29bb2d1d
diff --git a/pandas/tests/io/parser/common/test_chunksize.py b/pandas/tests/io/parser/common/test_chunksize.py
index ceb770ce72b78..5b7df02603b74 100644
--- a/pandas/tests/io/parser/common/test_chunksize.py
+++ b/pandas/tests/io/parser/common/test_chunksize.py
@@ -143,10 +143,7 @@ def test_read_chunksize_jagged_names(all_parsers):
     parser = all_parsers
     data = "\n".join(["0"] * 7 + [",".join(["0"] * 10)])
 
-    # error: List item 0 has incompatible type "float"; expected "int"
-    expected = DataFrame(
-        [[0] + [np.nan] * 9] * 7 + [[0] * 10]  # type: ignore[list-item]
-    )
+    expected = DataFrame([[0] + [np.nan] * 9] * 7 + [[0] * 10])
     with parser.read_csv(StringIO(data), names=range(10), chunksize=4) as reader:
         result = concat(reader)
     tm.assert_frame_equal(result, expected)
diff --git a/pandas/tests/io/parser/common/test_common_basic.py b/pandas/tests/io/parser/common/test_common_basic.py
index ed395df53432e..fbefb8d8c7d7c 100644
--- a/pandas/tests/io/parser/common/test_common_basic.py
+++ b/pandas/tests/io/parser/common/test_common_basic.py
@@ -15,6 +15,7 @@
 from pandas.errors import (
     EmptyDataError,
     ParserError,
+    ParserWarning,
 )
 
 from pandas import (
@@ -685,7 +686,8 @@ def test_no_header_two_extra_columns(all_parsers):
     ref = DataFrame([["foo", "bar", "baz"]], columns=column_names)
     stream = StringIO("foo,bar,baz,bam,blah")
     parser = all_parsers
-    df = parser.read_csv(stream, header=None, names=column_names, index_col=False)
+    with tm.assert_produces_warning(ParserWarning):
+        df = parser.read_csv(stream, header=None, names=column_names, index_col=False)
     tm.assert_frame_equal(df, ref)
 
 
diff --git a/pandas/tests/io/parser/usecols/test_usecols_basic.py b/pandas/tests/io/parser/usecols/test_usecols_basic.py
index b86dc5ef85fc6..16649be5b8a58 100644
--- a/pandas/tests/io/parser/usecols/test_usecols_basic.py
+++ b/pandas/tests/io/parser/usecols/test_usecols_basic.py
@@ -383,7 +383,9 @@ def test_usecols_indices_out_of_bounds(all_parsers, names):
 a,b
 1,2
     """
-    with tm.assert_produces_warning(FutureWarning, check_stacklevel=False):
+    with tm.assert_produces_warning(
+        FutureWarning, check_stacklevel=False, raise_on_extra_warnings=False
+    ):
         result = parser.read_csv(StringIO(data), usecols=[0, 2], names=names, header=0)
     expected = DataFrame({"a": [1], "b": [None]})
     if names is None and parser.engine == "python":


Code review comments:
diff_hunk: @@ -798,6 +799,26 @@ def _do_date_conversions(self, names, data):
 
         return names, data
 
+    def _check_data_length(self, columns: List[str], data: List[ArrayLike]):
path: pandas/io/parsers/base_parser.py
text: @user1:
-> None for return type

@author:
Done

diff_hunk: @@ -798,6 +799,26 @@ def _do_date_conversions(self, names, data):
 
         return names, data
 
+    def _check_data_length(self, columns: List[str], data: List[ArrayLike]):
+        """Checks if length of data is equal to length of column names. One set of
+        trailing commas is allowed.
path: pandas/io/parsers/base_parser.py
text: @user1:
Short summary should be one line. Second sentence can be moved to extended summary I think.

@author:
Done