Repository: pandas-dev/pandas
Pull request: REGR: read_csv raising when dtypes is specified with usecols
- [ ] closes #54868 (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Problem statement:
read_fwf: `IndexError: list index out of range` when using `dtype` and `usecols` parameters
Reproducer:
```python
import pandas as pd
from io import BytesIO

pd.read_fwf(BytesIO(b"a       b           c          d\r\nid8141  360.242940  149.910199 11950.7\r\nid1594  444.953632  166.985655 11788.4\r\n"), usecols=["a", "b", "d"], dtype={"a": str})  # <- IndexError: list index out of range
```

pandas 2.1.0rc0

```bash
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "Miniconda3\envs\modin\lib\site-packages\pandas\io\parsers\readers.py", line 1393, in read_fwf
    return _read(filepath_or_buffer, kwds)
  File "Miniconda3\envs\modin\lib\site-packages\pandas\io\parsers\readers.py", line 611, in _read
    parser = TextFileReader(filepath_or_buffer, **kwds)
  File "Miniconda3\envs\modin\lib\site-packages\pandas\io\parsers\readers.py", line 1448, in __init__
    self._engine = self._make_engine(f, self.engine)
  File "Miniconda3\envs\modin\lib\site-packages\pandas\io\parsers\readers.py", line 1723, in _make_engine
    return mapping[engine](f, **self.options)
  File "Miniconda3\envs\modin\lib\site-packages\pandas\io\parsers\python_parser.py", line 1324, in __init__
    PythonParser.__init__(self, f, **kwds)
  File "Miniconda3\envs\modin\lib\site-packages\pandas\io\parsers\python_parser.py", line 162, in __init__
    self._no_thousands_columns = self._set_no_thousand_columns()
  File "Miniconda3\envs\modin\lib\site-packages\pandas\io\parsers\python_parser.py", line 1186, in _set_no_thousand_columns
    and self.columns[i] in self.dtype
IndexError: list index out of range
```

Hints from issue discussion:
Upgraded to the just 2.1.0 and a prior working  pd.read_excel exhibits the same problem.

Commit message: REGR: read_csv raising when dtypes is specified with usecols

Patch to review:
diff --git a/pandas/io/parsers/python_parser.py b/pandas/io/parsers/python_parser.py
index 520d2193e1c04..66a46d7040ab4 100644
--- a/pandas/io/parsers/python_parser.py
+++ b/pandas/io/parsers/python_parser.py
@@ -1176,7 +1176,7 @@ def _set_no_thousand_columns(self) -> set[int]:
             )
         if self.columns and self.dtype:
             assert self._col_indices is not None
-            for i in self._col_indices:
+            for i, idx in enumerate(self._col_indices):
                 if not isinstance(self.dtype, dict) and not is_numeric_dtype(
                     self.dtype
                 ):
diff --git a/pandas/tests/io/parser/dtypes/test_dtypes_basic.py b/pandas/tests/io/parser/dtypes/test_dtypes_basic.py
index a578b2a402e93..97a32ad79a67c 100644
--- a/pandas/tests/io/parser/dtypes/test_dtypes_basic.py
+++ b/pandas/tests/io/parser/dtypes/test_dtypes_basic.py
@@ -575,3 +575,20 @@ def test_accurate_parsing_of_large_integers(all_parsers):
     assert len(orders.loc[orders["ID_DEAL"] == 2023552585717263359, "ID_DEAL"]) == 1
     assert len(orders.loc[orders["ID_DEAL"] == 2023552585717263360, "ID_DEAL"]) == 2
     assert len(orders.loc[orders["ID_DEAL"] == 2023552585717263361, "ID_DEAL"]) == 2
+
+
+def test_dtypes_with_usecols(all_parsers):
+    # GH#54868
+
+    parser = all_parsers
+    data = """a,b,c
+1,2,3
+4,5,6"""
+
+    result = parser.read_csv(StringIO(data), usecols=["a", "c"], dtype={"a": object})
+    if parser.engine == "pyarrow":
+        values = [1, 4]
+    else:
+        values = ["1", "4"]
+    expected = DataFrame({"a": pd.Series(values, dtype=object), "c": [3, 6]})
+    tm.assert_frame_equal(result, expected)


Code review comments:
diff_hunk: @@ -1176,7 +1176,7 @@ def _set_no_thousand_columns(self) -> set[int]:
             )
         if self.columns and self.dtype:
             assert self._col_indices is not None
-            for i in self._col_indices:
+            for i, idx in enumerate(self._col_indices):
path: pandas/io/parsers/python_parser.py
text: @user1:
```suggestion
            for i, col in zip(self._col_indices, self.columns):
```

and then `self.columns[i]` can just be replaced with `col`?

@author:
yep thx