Repository: pandas-dev/pandas

Issue:
ENH/BUG: usecols does not raise an exception when col index is out of bounds.
```python
import pandas as pd

df = pd.DataFrame({'col1': [1, 2], 'col2': [3, 4]})
df.to_csv('test.csv', index=False)

# Should raise ValueError, since there is no col at index 10
pd.read_csv('test.csv', header=0, usecols=[0, 10], engine='python')

print('Silently ran to completion.')
```
<hr>

Output

```python
Silently ran to completion.
```

#### Problem description
Silently passing incorrect column indices (result of some arithmetic) could potentially lead to a debugging nightmare. 

A possible solution is to check if the largest value in `usecols` is larger than the number of columns at any level.

```python
# Something along the lines of
if any(max(col_indices) >= len(column) for column in columns):
    raise ValueError('...')
columns = [[n for i, n in enumerate(column) if i in col_indices]
           for column in columns]
self._col_indices = col_indices
```

If the idea is to accommodate different levels having different number of columns, maybe it would be better if `usecols` allowed a sequence of sequences (one for each level).

**Note**: 
* A similar problem exists with the C engine, although the result is an unexpected column full of _nulls_. 

* The issue seems to be handled correctly when `index` is written to file.

**Edit**:

* The issue occurs when the length of `usecols` equals the number of columns in dataframe.

#### Expected Output
``ValueError``


#### Output of ``pd.show_versions()``

<details>

INSTALLED VERSIONS
------------------
commit: None
python: 3.7.1.final.0
python-bits: 64
OS: Linux
OS-release: 4.18.0-16-generic
machine: x86_64
processor: x86_64
byteorder: little
LC_ALL: None
LANG: en_US.UTF-8
LOCALE: en_US.UTF-8

pandas: 0.23.4
pytest: 4.0.2
pip: 18.1
setuptools: 40.6.3
Cython: 0.29.2
numpy: 1.15.4
scipy: 1.1.0
pyarrow: None
xarray: None
IPython: 7.2.0
sphinx: 1.8.2
patsy: 0.5.1
dateutil: 2.7.5
pytz: 2018.7
blosc: None
bottleneck: 1.2.1
tables: 3.4.4
numexpr: 2.6.8
feather: None
matplotlib: 3.0.2
openpyxl: 2.5.12
xlrd: 1.2.0
xlwt: 1.3.0
xlsxwriter: 1.1.2
lxml: 4.2.5
bs4: 4.6.3
html5lib: 1.0.1
sqlalchemy: 1.2.15
pymysql: None
psycopg2: None
jinja2: 2.10
s3fs: None
fastparquet: None
pandas_gbq: None
pandas_datareader: None

</details>

ENH/BUG: usecols does not raise an exception when col index is out of bounds.
```python
import pandas as pd

df = pd.DataFrame({'col1': [1, 2], 'col2': [3, 4]})
df.to_csv('test.csv', index=False)

# Should raise ValueError, since there is no col at index 10
pd.read_csv('test.csv', header=0, usecols=[0, 10], engine='python')

print('Silently ran to completion.')
```
<hr>

Output

```python
Silently ran to completion.
```

#### Problem description
Silently passing incorrect column indices (result of some arithmetic) could potentially lead to a debugging nightmare. 

A possible solution is to check if the largest value in `usecols` is larger than the number of columns at any level.

```python
# Something along the lines of
if any(max(col_indices) >= len(column) for column in columns):
    raise ValueError('...')
columns = [[n for i, n in enumerate(column) if i in col_indices]
           for column in columns]
self._col_indices = col_indices
```

If the idea is to accommodate different levels having different number of columns, maybe it would be better if `usecols` allowed a sequence of sequences (one for each level).

**Note**: 
* A similar problem exists with the C engine, although the result is an unexpected column full of _nulls_. 

* The issue seems to be handled correctly when `index` is written to file.

**Edit**:

* The issue occurs when the length of `usecols` equals the number of columns in dataframe.

#### Expected Output
``ValueError``


#### Output of ``pd.show_versions()``

<details>

INSTALLED VERSIONS
------------------
commit: None
python: 3.7.1.final.0
python-bits: 64
OS: Linux
OS-release: 4.18.0-16-generic
machine: x86_64
processor: x86_64
byteorder: little
LC_ALL: None
LANG: en_US.UTF-8
LOCALE: en_US.UTF-8

pandas: 0.23.4
pytest: 4.0.2
pip: 18.1
setuptools: 40.6.3
Cython: 0.29.2
numpy: 1.15.4
scipy: 1.1.0
pyarrow: None
xarray: None
IPython: 7.2.0
sphinx: 1.8.2
patsy: 0.5.1
dateutil: 2.7.5
pytz: 2018.7
blosc: None
bottleneck: 1.2.1
tables: 3.4.4
numexpr: 2.6.8
feather: None
matplotlib: 3.0.2
openpyxl: 2.5.12
xlrd: 1.2.0
xlwt: 1.3.0
xlsxwriter: 1.1.2
lxml: 4.2.5
bs4: 4.6.3
html5lib: 1.0.1
sqlalchemy: 1.2.15
pymysql: None
psycopg2: None
jinja2: 2.10
s3fs: None
fastparquet: None
pandas_gbq: None
pandas_datareader: None

</details>



Discussion:
I suppose this is related to #17301 though that only covers labels and not indices.

I think it makes sense to raise on this; investigation and PRs would certainly be welcome
I suppose this is related to #17301 though that only covers labels and not indices.

I think it makes sense to raise on this; investigation and PRs would certainly be welcome

Resolving patch:
diff --git a/pandas/io/parsers.py b/pandas/io/parsers.py
--- a/pandas/io/parsers.py
+++ b/pandas/io/parsers.py
@@ -1901,6 +1901,12 @@ def __init__(self, src, **kwds):
                     not set(usecols).issubset(self.orig_names)):
                 _validate_usecols_names(usecols, self.orig_names)
 
+            # GH 25623
+            # validate that column indices in usecols are not out of bounds
+            elif self.usecols_dtype == 'integer':
+                indices = lrange(self._reader.table_width)
+                _validate_usecols_names(usecols, indices)
+
             if len(self.names) > len(usecols):
                 self.names = [n for i, n in enumerate(self.names)
                               if (i in usecols or n in usecols)]
@@ -2204,7 +2210,8 @@ def __init__(self, f, **kwds):
         self.skipinitialspace = kwds['skipinitialspace']
         self.lineterminator = kwds['lineterminator']
         self.quoting = kwds['quoting']
-        self.usecols, _ = _validate_usecols_arg(kwds['usecols'])
+        self.usecols, self.usecols_dtype = _validate_usecols_arg(
+            kwds['usecols'])
         self.skip_blank_lines = kwds['skip_blank_lines']
 
         self.warn_bad_lines = kwds['warn_bad_lines']
@@ -2597,6 +2604,13 @@ def _infer_columns(self):
             if clear_buffer:
                 self._clear_buffer()
 
+            # GH 25623
+            # validate that column indices in usecols are not out of bounds
+            if self.usecols_dtype == 'integer':
+                for col in columns:
+                    indices = lrange(len(col))
+                    _validate_usecols_names(self.usecols, indices)
+
             if names is not None:
                 if ((self.usecols is not None and
                      len(names) != len(self.usecols)) or
@@ -2632,6 +2646,11 @@ def _infer_columns(self):
             ncols = len(line)
             num_original_columns = ncols
 
+            # GH 25623
+            # validate that column indices in usecols are not out of bounds
+            if self.usecols_dtype == 'integer':
+                _validate_usecols_names(self.usecols, lrange(ncols))
+
             if not names:
                 if self.prefix:
                     columns = [['{prefix}{idx}'.format(


Test patch:
diff --git a/pandas/tests/io/parser/test_usecols.py b/pandas/tests/io/parser/test_usecols.py
--- a/pandas/tests/io/parser/test_usecols.py
+++ b/pandas/tests/io/parser/test_usecols.py
@@ -21,6 +21,22 @@
                                "expected but not found: {0}")
 
 
+@pytest.mark.parametrize("names,usecols,missing", [
+    (None, [0, 3], r"\[3\]"),
+    (["a", "b", "c"], [0, -1, 2], r"\[-1\]"),
+    (None, [3], r"\[3\]"),
+    (["a"], [3], r"\[3\]")
+])
+def test_usecols_out_of_bounds(all_parsers, names, usecols, missing):
+    # See gh-25623
+    data = "a,b,c\n1,2,3\n4,5,6"
+    parser = all_parsers
+
+    mssg = _msg_validate_usecols_names.format(missing)
+    with pytest.raises(ValueError, match=mssg):
+        parser.read_csv(StringIO(data), usecols=usecols, names=names)
+
+
 def test_raise_on_mixed_dtype_usecols(all_parsers):
     # See gh-12678
     data = """a,b,c