Repository: pandas-dev/pandas

Issue:
csv_reader with limited number of columns should should completely disregard the unused fields
xref #6710

I have a CSV whose lines may have 11 or 18 fields. I only need to read the first 6 fields, so I use "usecols=range(6)". Even with the limited number of columns, I get the exception:

ValueError: Expected 11 fields in line 776483, saw 18

The csv_reader should completely disregard the unused fields.

Small test case:

```
csv = '19,29,39\n'*2 + '10,20,30,40\n'
df = pd.read_csv(io.StringIO(csv), engine='python', header=None, usecols=list(range(3)))
```

It also affects the C engine.

Discussed at the users mailing list at https://groups.google.com/d/topic/pydata/vjhFpHtgnvw/discussion



Discussion:
this is like #6710. Usecols will only select from the valid columns. It is inferring that you have 3, so it is a bit contradictory here.

Soln is to use names (or `names=range(3)`) to just get those first 3

```
In [21]: pd.read_csv(io.StringIO(csv), header=None, names=range(4))
Out[21]: 
    0   1   2   3
0  19  29  39 NaN
1  19  29  39 NaN
2  10  20  30  40
```

not sure if this is a bug or not; i'll mark it same as the other one.

I don't agree that they are the same bug.

Bug #6710 is about the ability to infer whether the next lines will have more fields or not.

In this case, _all_ rows have _all_ fields of interest ( [0:6] ). The remaining of the line will not be used anyway and should not be considered at all.

@cordeiro I said they are alike, that's why its a separate issue :)

Oups. :)

np. if you do have interested in looking at this would be appreciated.

sounds easy ill try to make a pr today for this


Resolving patch:
diff --git a/pandas/io/parsers.py b/pandas/io/parsers.py
--- a/pandas/io/parsers.py
+++ b/pandas/io/parsers.py
@@ -2295,11 +2295,12 @@ def _infer_columns(self):
                     columns = [lrange(ncols)]
                 columns = self._handle_usecols(columns, columns[0])
             else:
-                if self.usecols is None or len(names) == num_original_columns:
+                if self.usecols is None or len(names) >= num_original_columns:
                     columns = self._handle_usecols([names], names)
                     num_original_columns = len(names)
                 else:
-                    if self.usecols and len(names) != len(self.usecols):
+                    if (not callable(self.usecols) and
+                            len(names) != len(self.usecols)):
                         raise ValueError(
                             'Number of passed names did not match number of '
                             'header fields in the file'


Test patch:
diff --git a/pandas/io/tests/parser/usecols.py b/pandas/io/tests/parser/usecols.py
--- a/pandas/io/tests/parser/usecols.py
+++ b/pandas/io/tests/parser/usecols.py
@@ -440,3 +440,28 @@ def test_callable_usecols(self):
         expected = DataFrame()
         df = self.read_csv(StringIO(s), usecols=lambda x: False)
         tm.assert_frame_equal(df, expected)
+
+    def test_incomplete_first_row(self):
+        # see gh-6710
+        data = '1,2\n1,2,3'
+        names = ['a', 'b', 'c']
+        expected = DataFrame({'a': [1, 1],
+                              'c': [np.nan, 3]})
+
+        usecols = ['a', 'c']
+        df = self.read_csv(StringIO(data), names=names, usecols=usecols)
+        tm.assert_frame_equal(df, expected)
+
+        usecols = lambda x: x in ['a', 'c']
+        df = self.read_csv(StringIO(data), names=names, usecols=usecols)
+        tm.assert_frame_equal(df, expected)
+
+    def test_uneven_length_cols(self):
+        # see gh-8985
+        usecols = [0, 1, 2]
+        data = '19,29,39\n' * 2 + '10,20,30,40'
+        expected = DataFrame([[19, 29, 39],
+                              [19, 29, 39],
+                              [10, 20, 30]])
+        df = self.read_csv(StringIO(data), header=None, usecols=usecols)
+        tm.assert_frame_equal(df, expected)