Repository: feature-engine/feature_engine
Pull request #686 (2023-07-22 15:49:49)

Raise informative error on duplicated column names
This PR adds an informative error message in cases in which the user supplies inputs having duplicated column names, which otherwise manifest in hard-to-track errors (e.g. https://github.com/feature-engine/feature_engine/pull/681).

david-cortes: Moved the check to `check_X` instead as suggested above.

Regarding errors from `pandas`, it does throw errors sometimes when there are duplicates, but only under some particular situations:
https://pandas.pydata.org/pandas-docs/stable/user_guide/duplicates.html

Also checked the mechanism towards the attribute `is_unique` as it seems that's what they recommend in their guide.
david-cortes: Moved the tests to `test_dataframe_checks.py`.
david-cortes: Added a check on the error message.
solegalli: Hey @david-cortes 

I made a PR to your repo: https://github.com/david-cortes/feature_engine/pull/4

Where I rebase main and add this contribution to the changelog.

Would you have time to merge over there, so it updates here and I can merge and close?

Thanks a lot!
david-cortes: > Hey @david-cortes
> 
> I made a PR to your repo: [david-cortes#4](https://github.com/david-cortes/feature_engine/pull/4)
> 
> Where I rebase main and add this contribution to the changelog.
> 
> Would you have time to merge over there, so it updates here and I can merge and close?
> 
> Thanks a lot!

Thanks, although I think you should also be able to push changes to the branch directly.
codecov[bot]: ## [Codecov](https://app.codecov.io/gh/feature-engine/feature_engine/pull/686?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=feature-engine) Report
> Merging [#686](https://app.codecov.io/gh/feature-engine/feature_engine/pull/686?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=feature-engine) (a53a7bd) into [main](https://app.codecov.io/gh/feature-engine/feature_engine/commit/3343305a01d1acfeff846b65d33a5686c6e8c84f?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=feature-engine) (3343305) will **increase** coverage by `0.00%`.
> Report is 1 commits behind head on main.
> The diff coverage is `100.00%`.

```diff
@@           Coverage Diff           @@
##             main     #686   +/-   ##
=======================================
  Coverage   97.99%   97.99%           
=======================================
  Files         100      100           
  Lines        3843     3849    +6     
  Branches      754      752    -2     
=======================================
+ Hits         3766     3772    +6     
  Misses         28       28           
  Partials       49       49           
```


| [Files Changed](https://app.codecov.io/gh/feature-engine/feature_engine/pull/686?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=feature-engine) | Coverage Δ | |
|---|---|---|
| [feature\_engine/creation/math\_features.py](https://app.codecov.io/gh/feature-engine/feature_engine/pull/686?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=feature-engine#diff-ZmVhdHVyZV9lbmdpbmUvY3JlYXRpb24vbWF0aF9mZWF0dXJlcy5weQ==) | `97.77% <ø> (ø)` | |
| [feature\_engine/dataframe\_checks.py](https://app.codecov.io/gh/feature-engine/feature_engine/pull/686?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=feature-engine#diff-ZmVhdHVyZV9lbmdpbmUvZGF0YWZyYW1lX2NoZWNrcy5weQ==) | `97.05% <100.00%> (+0.08%)` | :arrow_up: |
| [feature\_engine/datetime/datetime.py](https://app.codecov.io/gh/feature-engine/feature_engine/pull/686?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=feature-engine#diff-ZmVhdHVyZV9lbmdpbmUvZGF0ZXRpbWUvZGF0ZXRpbWUucHk=) | `100.00% <100.00%> (ø)` | |
| [feature\_engine/datetime/datetime\_subtraction.py](https://app.codecov.io/gh/feature-engine/feature_engine/pull/686?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=feature-engine#diff-ZmVhdHVyZV9lbmdpbmUvZGF0ZXRpbWUvZGF0ZXRpbWVfc3VidHJhY3Rpb24ucHk=) | `94.73% <100.00%> (+0.07%)` | :arrow_up: |
| [feature\_engine/encoding/base\_encoder.py](https://app.codecov.io/gh/feature-engine/feature_engine/pull/686?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=feature-engine#diff-ZmVhdHVyZV9lbmdpbmUvZW5jb2RpbmcvYmFzZV9lbmNvZGVyLnB5) | `100.00% <100.00%> (ø)` | |
| [feature\_engine/encoding/one\_hot.py](https://app.codecov.io/gh/feature-engine/feature_engine/pull/686?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=feature-engine#diff-ZmVhdHVyZV9lbmdpbmUvZW5jb2Rpbmcvb25lX2hvdC5weQ==) | `100.00% <100.00%> (ø)` | |
| [feature\_engine/encoding/rare\_label.py](https://app.codecov.io/gh/feature-engine/feature_engine/pull/686?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=feature-engine#diff-ZmVhdHVyZV9lbmdpbmUvZW5jb2RpbmcvcmFyZV9sYWJlbC5weQ==) | `100.00% <100.00%> (ø)` | |
| [feature\_engine/imputation/categorical.py](https://app.codecov.io/gh/feature-engine/feature_engine/pull/686?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=feature-engine#diff-ZmVhdHVyZV9lbmdpbmUvaW1wdXRhdGlvbi9jYXRlZ29yaWNhbC5weQ==) | `95.31% <100.00%> (ø)` | |
| [feature\_engine/selection/shuffle\_features.py](https://app.codecov.io/gh/feature-engine/feature_engine/pull/686?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=feature-engine#diff-ZmVhdHVyZV9lbmdpbmUvc2VsZWN0aW9uL3NodWZmbGVfZmVhdHVyZXMucHk=) | `100.00% <100.00%> (ø)` | |
| [feature\_engine/transformation/yeojohnson.py](https://app.codecov.io/gh/feature-engine/feature_engine/pull/686?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=feature-engine#diff-ZmVhdHVyZV9lbmdpbmUvdHJhbnNmb3JtYXRpb24veWVvam9obnNvbi5weQ==) | `100.00% <100.00%> (ø)` | |
| ... and [1 more](https://app.codecov.io/gh/feature-engine/feature_engine/pull/686?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=feature-engine) | |

:mega: We’re building smart automated test selection to slash your CI/CD build times. [Learn more](https://about.codecov.io/iterative-testing/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=feature-engine)


Patch:
diff --git a/docs/whats_new/v_160.rst b/docs/whats_new/v_160.rst
index b7af7f9..d457e29 100644
--- a/docs/whats_new/v_160.rst
+++ b/docs/whats_new/v_160.rst
@@ -36,6 +36,7 @@ Code improvements
 
 - Routine in `DatetimeFeatures()` does not enter into our check for `utc=True` when working with different timezones any more (`Soledad Galli <https://github.com/solegalli>`_)
 - Improve performance in `OneHotEncoder()` (`Soledad Galli <https://github.com/solegalli>`_)
+- Add check for dupicated variable names in dataframe  (`David Cortes <https://github.com/david-cortes>`_)
 
 
 Version 1.6.1
diff --git a/feature_engine/dataframe_checks.py b/feature_engine/dataframe_checks.py
index 95ae2e8..41346d5 100644
--- a/feature_engine/dataframe_checks.py
+++ b/feature_engine/dataframe_checks.py
@@ -47,6 +47,8 @@ def check_X(X: Union[np.generic, np.ndarray, pd.DataFrame]) -> pd.DataFrame:
         A copy of original DataFrame or a converted Numpy array.
     """
     if isinstance(X, pd.DataFrame):
+        if not X.columns.is_unique:
+            raise ValueError("Input data contains duplicated variable names.")
         X = X.copy()
 
     elif isinstance(X, (np.generic, np.ndarray)):


Test patch:
diff --git a/tests/test_dataframe_checks.py b/tests/test_dataframe_checks.py
index d48c7ff..c6cbec2 100644
--- a/tests/test_dataframe_checks.py
+++ b/tests/test_dataframe_checks.py
@@ -170,3 +170,19 @@ def test_contains_inf(df_na):
     df_na.fillna(np.inf, inplace=True)
     with pytest.raises(ValueError):
         assert _check_contains_inf(df_na, ["Age", "Marks"])
+
+
+def test_check_X_raises_error_on_duplicated_column_names():
+    df = pd.DataFrame(
+        {
+            "col1": [1, 2, 3],
+            "col2": ["a", "b", "c"],
+            "col3": pd.date_range("2023-01-01", periods=3),
+        }
+    )
+    df.columns = ["same", "unique", "same"]
+
+    with pytest.raises(ValueError) as err_txt:
+        check_X(df)
+
+    assert err_txt.match("Input data contains duplicated variable names.")