BUG: Patch missing data handling with usecols
State: closed

Patch handling of cases when the first row of a CSV is incomplete and `usecols` is specified.

Closes #6710.
Closes #8985.

xref #14782.

Labels:
name: Bug
name: IO CSV
description: read_csv, to_csv

Comments:
## [Current coverage](https://codecov.io/gh/pandas-dev/pandas/pull/15066?src=pr) is 84.75% (diff: 100%)
> Merging [#15066](https://codecov.io/gh/pandas-dev/pandas/pull/15066?src=pr) into [master](https://codecov.io/gh/pandas-dev/pandas/branch/master?src=pr) will decrease coverage by **<.01%**

```diff
@@             master     #15066   diff @@
==========================================
  Files           145        145          
  Lines         51220      51220          
  Methods           0          0          
  Messages          0          0          
  Branches          0          0          
==========================================
- Hits          43415      43413     -2   
- Misses         7805       7807     +2   
  Partials          0          0          
```

> Powered by [Codecov](https://codecov.io?src=pr). Last update [c71f214...6bb558d](https://codecov.io/gh/pandas-dev/pandas/compare/c71f214a7024a3a8d679f8c2eafdd4e870d7c216...6bb558d6c118f1426211dc072e0095bcce0ed4dc?src=pr)
@jorisvandenbossche can you have a look.
Regarding the 'malformed' lines (rows with too many values), we eg also have `error_bad_lines` and `warn_bad_lines`. Although this will 'drop' the full bad line instead of dropping only the excessive values on the line. 
Slightly related, there is also https://github.com/pandas-dev/pandas/issues/9729 which asks for similar handling for bad lines with too few values. So maybe we should have a more general discussion about how to handle bad lines (where the options are raise/drop full line/'process' combined with a warning or not in case of not raising)
@jorisvandenbossche : Let me address your comments separately:

1) #8985 is a bug, not an enhancement.  The `usecols` parameter was 100% valid in the test case presented from the issue.  I fail to see what new thing needs to be documented here.

2) #9729 is tangentially related to this issue because it has to deal with an inconsistent number of fields across rows.  `usecols` should not be involved in that discussion.  I don't believe this discussion needs include this PR.
> #8985 is a bug, not an enhancement. The usecols parameter was 100% valid in the test case presented from the issue. I fail to see what new thing needs to be documented here.

Can you point me to documentation and/or tests that specifies this behaviour of `usecols` ?
The usecols parameter is specifying three columns.  Every row has three columns, so the parameter is perfectly valid.  We should be able to handle it just fine.
> We should be able to handle it just fine.

I am not saying that we necessarily shouldn't do this (the use case is certainly useful). My request above was just, if we fix this, to document this as a feature of `usecols`
BTW, the example case for #8985 actually already seems to work on latest master?

```
In [21]: usecols = [0, 1, 2]

In [22]: data = '19,29,39\n' * 2 + '10,20,30,40'

In [25]: pd.read_csv(StringIO(data), header=None, usecols=usecols)
Out[25]: 
    0   1   2
0  19  29  39
1  19  29  39
2  10  20  30

In [26]: pd.read_csv(StringIO(data), header=None, usecols=usecols, engine='python')
Out[26]: 
    0   1   2
0  19  29  39
1  19  29  39
2  10  20  30

In [27]: pd.__version__
Out[27]: '0.19.0+317.g41170d4'

```
Handling missing data is independent of usecols IMO.  That's why this isn't a "feature" for me.
It's rather about too many fields rather than missing fields. But to the point :-), we have some documentation about how read_csv handles malformed files (http://pandas.pydata.org/pandas-docs/stable/io.html#handling-bad-lines + docstring on those keywords), which is: raising an error when there are lines with more fields than the number of columns (or skip the line altogether if you specify a keyword). So IMO it would be nice to explain there that you can also make use of `usecols` to ignore the excessive data (instead of raising or dropping full line).

BTW, it seems already worked in at least 0.18.1, so it's indeed not a bug 'fix' :-) 
But a confirmation of the behaviour in tests (which is good!)
@jorisvandenbossche : 

1) Ah, indeed #8985 was already a non-issue, but 100% agree that confirmation is good.

2) Fair enough regarding `usecols`.  I'll add an example about that.
@jorisvandenbossche : Added example in the docs regarding `usecols` for handling malformed rows.  Ready to merge if there are no other concerns.
@gfyoung Thanks a lot for the addition!