Repository: pandas-dev/pandas
Pull request: dataframe.astype() exception message to include column name
- [X] closes #47571(Replace xxxx with the Github issue number)
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [X] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

**Description:**
Added column related info as part of `astype` exceptions

| Code  | Old Error | New Error |
| ------------- | ------------- | ----------|
| pd.DataFrame({'a':[1,2,"text"], 'b':[1,2,3]}).astype({'a': int}) | ValueError: invalid literal for int() with base 10: 'text' | ValueError: invalid literal for int() with base 10: 'text': Error while type casting for column 'a' |


Problem statement:
ENH: dataframe.astype() exception message to include column name
#### Is your feature request related to a problem?

When using dataframe.astype() and there is a column missing or there are some values that cannot be correctly converted, there is an exception raised. But there is no way to know which column is missing or which column has the wrong value.

#### Describe the solution you'd like

The exception message of the dataframe.astype() should contain the column name responsible for the exception (missing column(s) or column with wrong values).

#### API breaking implications

This does not introduce any breaking changes

#### Additional context

Some examples of the current exceptions raised by astype():

```
KeyError: 'Only a column name can be used for the key in a dtype mappings argument.'

TypeError: object cannot be converted to an IntegerDtype

ValueError: invalid literal for int() with base 10: 'test'
```

Hints from issue discussion:
Note -

**issue # 1 -** KeyError: 'Only a column name can be used for the key in a dtype mappings argument.'
has already been addressed by #44487

am working on addressing issue # 3 with PR #48321

Commit message: Update doc/source/whatsnew/v1.6.0.rst

Co-authored-by: Matthew Roeschke <10647082+mroeschke@users.noreply.github.com>

Patch to review:
diff --git a/doc/source/whatsnew/v1.6.0.rst b/doc/source/whatsnew/v1.6.0.rst
index d726f69286469..de4f7969dbf7d 100644
--- a/doc/source/whatsnew/v1.6.0.rst
+++ b/doc/source/whatsnew/v1.6.0.rst
@@ -35,7 +35,7 @@ Other enhancements
 - Added ``index`` parameter to :meth:`DataFrame.to_dict` (:issue:`46398`)
 - Added metadata propagation for binary operators on :class:`DataFrame` (:issue:`28283`)
 - :class:`.CategoricalConversionWarning`, :class:`.InvalidComparison`, :class:`.InvalidVersion`, :class:`.LossySetitemError`, and :class:`.NoBufferPresent` are now exposed in ``pandas.errors`` (:issue:`27656`)
--
+- :func:`DataFrame.astype` exception message thrown improved to include column name when type conversion is not possible. (:issue:`47571`)
 
 .. ---------------------------------------------------------------------------
 .. _whatsnew_160.notable_bug_fixes:
diff --git a/pandas/core/generic.py b/pandas/core/generic.py
index b520aab2cfa5b..06536bcc9e38a 100644
--- a/pandas/core/generic.py
+++ b/pandas/core/generic.py
@@ -6261,7 +6261,13 @@ def astype(
                 if isna(cdt):
                     res_col = col.copy() if copy else col
                 else:
-                    res_col = col.astype(dtype=cdt, copy=copy, errors=errors)
+                    try:
+                        res_col = col.astype(dtype=cdt, copy=copy, errors=errors)
+                    except ValueError as ex:
+                        ex.args = (
+                            f"Error during type conversion for column {col_name}: {ex}",
+                        )
+                        raise
                 results.append(res_col)
 
         elif is_extension_array_dtype(dtype) and self.ndim > 1:
diff --git a/pandas/tests/frame/methods/test_astype.py b/pandas/tests/frame/methods/test_astype.py
index 6d343de9f5d3a..c491fb6434cf2 100644
--- a/pandas/tests/frame/methods/test_astype.py
+++ b/pandas/tests/frame/methods/test_astype.py
@@ -490,6 +490,18 @@ def test_astype_arg_for_errors(self):
 
         df.astype(np.int8, errors="ignore")
 
+    def test_astype_invalid_conversion(self):
+        # GH#47571
+        df = DataFrame({"a": [1, 2, "text"], "b": [1, 2, 3]})
+
+        msg = (
+            "Error during type conversion for column a: "
+            "invalid literal for int() with base 10: 'text'"
+        )
+
+        with pytest.raises(ValueError, match=re.escape(msg)):
+            df.astype({"a": int})
+
     def test_astype_arg_for_errors_dictlist(self):
         # GH#25905
         df = DataFrame(


Code review comments:
diff_hunk: @@ -6261,7 +6261,13 @@ def astype(
                 if isna(cdt):
                     res_col = col.copy() if copy else col
                 else:
-                    res_col = col.astype(dtype=cdt, copy=copy, errors=errors)
+                    try:
+                        res_col = col.astype(dtype=cdt, copy=copy, errors=errors)
+                    except ValueError as ex:
+                        ex.args = (
+                            f"Error during type conversion for column {col_name}: {ex}",
path: pandas/core/generic.py
text: @user1:
Actually could you move the `Error during type conversion for column {col_name}:` after the original exception message? Might help users that are trying to catch the original message and it expect it to be first.

@author:
sure, that makes sense. Will update the PR

@author:
done