Deprecated usecols with out of bounds indices in read_csv
State: closed

- [x] xref #25623
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [x] whatsnew entry

This currently raises 

<details>
Traceback (most recent call last):
  File "/home/developer/.config/JetBrains/PyCharm2021.1/scratches/scratch_4.py", line 539, in <module>
    pd.read_csv('test.csv', header=0, usecols=[0, 10], engine='python')
  File "/home/developer/PycharmProjects/pandas/pandas/io/parsers/readers.py", line 552, in read_csv
    return _read(filepath_or_buffer, kwds)
  File "/home/developer/PycharmProjects/pandas/pandas/io/parsers/readers.py", line 471, in _read
    return parser.read(nrows)
  File "/home/developer/PycharmProjects/pandas/pandas/io/parsers/readers.py", line 998, in read
    index, columns, col_dict = self._engine.read(nrows)
  File "/home/developer/PycharmProjects/pandas/pandas/io/parsers/python_parser.py", line 285, in read
    data, columns = self._exclude_implicit_index(alldata)
  File "/home/developer/PycharmProjects/pandas/pandas/io/parsers/python_parser.py", line 303, in _exclude_implicit_index
    names = [names[i] for i in sorted(self._col_indices)]
  File "/home/developer/PycharmProjects/pandas/pandas/io/parsers/python_parser.py", line 303, in <listcomp>
    names = [names[i] for i in sorted(self._col_indices)]
</details>
IndexError: list index out of range

on master but unfortunately works on 1.2.4, so we can either raise ParserError with 1.2.4 or fix and deprecate then to remove in 2.0, related to #41129

I think fixing and deprecating would be more sensible, but only realised that this works on 1.2.4 after finishing this, so wanted to put up for discussion at least :)

cc @gfyoung 


Labels:
name: IO CSV
description: read_csv, to_csv
name: Deprecate
description: Functionality to remove in pandas

Comments:
> unfortunately works on 1.2.4,

what does this mean?
This is a regression on master compared to 1.2.x series. So we should probably fix and then deprecate to not change behavior in 1.3

Unfortunately means, if this would not have worked on 1.2.x we could immediately start raising a ParserError without worrying about backwarts compatibility
IMO okay with making this change immediately (without deprecation) because `ParserError` subclasses `ValueError`.
@gfyoung was not sure, because this is working on 1.2.x without raising an error, it is simply ignoring the indexes out of range, but I would be fine with doin this immediately
> @gfyoung was not sure, because this is working on 1.2.x without raising an error, it is simply ignoring the indexes out of range, but I would be fine with doin this immediately

oh maybe i misuderstood. so if this was working on 1.2.x then we should deprecate first
> because this is working on 1.2.x without raising an error, it is simply ignoring the indexes out of range, but I would be fine with doin this immediately

Oh, I see!  The w/o raising an error part puts me in agree with @jreback then.

I would also then advocate for deprecation.
Yeah same on my side. Will mark this as draft until I have fixed the error on master. Then we can switch ParserError with FutureWarning
After #41244 was merged we can deprecate now for both engines
Done. should we use from __future__ import annotations in a follow up?
> Done. should we use from **future** import annotations in a follow up?

yes for sure, as that's being done elsewhere in the codebase.


thanks @phofl