Repository: scikit-learn/scikit-learn
Pull request: FIX handle column names renaming in ColumnTransformer
closes #28260

The adapters in charge of renaming columns with `set_output` to a DataFrame-like object was failing in case that the original DataFrame-like as duplicated columns (that is possible in `pandas`.

We inverse the stacking and and renaming to avoid potential column duplicate that should not happen.

Problem statement:
ColumnTransformer output unexpected prefixed feature names from FunctionTransformer() step
### Describe the bug

The following code demonstrates that when `FunctionTransformer` is present as a step in `ColumnTransformer`, the feature names output are all prefixed with the name from the last step '**C__**'.  For example, column '**x1**' is output as '**C__x1**' for 3 times. 
When 'FunctionTransferformer' is _not_ present as a step, the feature names are corrected prefixed by the name in each steps. For example, column '**x1**' is output as '**A__x1**' and '**C__x1**' respectively, as expected. 

### Steps/Code to Reproduce

```python
import pandas as pd
from sklearn.preprocessing import FunctionTransformer, StandardScaler
from sklearn.compose import ColumnTransformer
from sklearn import set_config
set_config(transform_output='pandas') 

df = pd.DataFrame({
        'x1' : [1, 2, 3],
        'x2' : [10, 20, 30],
        'x3' : [100, 200, 300]})

my_coltransformer = ColumnTransformer([
    ('A', 'passthrough', ['x1', 'x2', 'x3']),
    ('B', FunctionTransformer(lambda x: x**2), ['x1', 'x2', 'x3']),
    ('C', StandardScaler(), ['x1', 'x2', 'x3'])])
transformed_df = my_coltransformer.fit_transform(df)
print(transformed_df)

my_coltransformer = ColumnTransformer([
    ('A', 'passthrough', ['x1', 'x2', 'x3']),
    #('B', FunctionTransformer(lambda x: x**2), ['x1', 'x2', 'x3']),
    ('C', StandardScaler(), ['x1', 'x2', 'x3'])])
transformed_df = my_coltransformer.fit_transform(df)
print(transformed_df)
```

### Expected Results

```
   A__x1  A__x2  A__x3  B__x1  B__x2  B__x3     C__x1     C__x2     C__x3
0      1     10    100      1    100  10000 -1.224745 -1.224745 -1.224745
1      2     20    200      4    400  40000  0.000000  0.000000  0.000000
2      3     30    300      9    900  90000  1.224745  1.224745  1.224745
```

### Actual Results

```
   C__x1  C__x2  C__x3  C__x1  C__x2  C__x3     C__x1     C__x2     C__x3
0      1     10    100      1    100  10000 -1.224745 -1.224745 -1.224745
1      2     20    200      4    400  40000  0.000000  0.000000  0.000000
2      3     30    300      9    900  90000  1.224745  1.224745  1.224745
```

### Versions

```shell
1.4.0
I tested this in 1.3.0 and did not observe the unexpected results.
```

Hints from issue discussion:
I can reproduce the issue. I'll investigate the reason.
As a temporary workaround, setting `feature_names_out="one-to-one"` provide the expected results.
We indeed refactor and use a dictionary to rename the columns: https://github.com/scikit-learn/scikit-learn/blob/main/sklearn/utils/_set_output.py#L125

But we overlooked that one could reuse the same column and then `x1` get remapped to the latest transformer. We should therefore pass a list instead.
I'll submit a patch.

/take

Commit message: TST check that we raise the proper error message

Patch to review:
diff --git a/doc/whats_new/v1.4.rst b/doc/whats_new/v1.4.rst
index 4b5a822108f4b..855c64ba7e053 100644
--- a/doc/whats_new/v1.4.rst
+++ b/doc/whats_new/v1.4.rst
@@ -44,6 +44,11 @@ Changes impacting all modules
 
   :pr:`28235` by :user:`Julien Jerphanion <jjerphan>`.
 
+- |Fix| The adapters handling the creation of DataFrame now rename properly the
+  column names by handling duplicates. It notably happens when using the
+  :class:`~compose.ColumnTransformer`.
+  :pr:`28262` by :user:`Guillaume Lemaitre <glemaitre>`.
+
 Changelog
 ---------
 
diff --git a/sklearn/compose/_column_transformer.py b/sklearn/compose/_column_transformer.py
index ee1ee88635516..07387a2a2ef46 100644
--- a/sklearn/compose/_column_transformer.py
+++ b/sklearn/compose/_column_transformer.py
@@ -1056,24 +1056,8 @@ def _hstack(self, Xs):
             Xs = [f.toarray() if sparse.issparse(f) else f for f in Xs]
             adapter = _get_container_adapter("transform", self)
             if adapter and all(adapter.is_supported_container(X) for X in Xs):
-                output = adapter.hstack(Xs)
-
-                output_samples = output.shape[0]
-                if any(_num_samples(X) != output_samples for X in Xs):
-                    raise ValueError(
-                        "Concatenating DataFrames from the transformer's output lead to"
-                        " an inconsistent number of samples. The output may have Pandas"
-                        " Indexes that do not match."
-                    )
-
-                # If all transformers define `get_feature_names_out`, then transform
-                # will adjust the column names to be consistent with
-                # verbose_feature_names_out. Here we prefix the feature names if
-                # verbose_feature_names_out=True.
-
-                if not self.verbose_feature_names_out:
-                    return output
-
+                # rename before stacking as it avoids to error on temporary duplicated
+                # columns
                 transformer_names = [
                     t[0]
                     for t in self._iter(
@@ -1083,13 +1067,69 @@ def _hstack(self, Xs):
                         skip_empty_columns=True,
                     )
                 ]
-                # Selection of columns might be empty.
-                # Hence feature names are filtered for non-emptiness.
                 feature_names_outs = [X.columns for X in Xs if X.shape[1] != 0]
-                names_out = self._add_prefix_for_feature_names_out(
-                    list(zip(transformer_names, feature_names_outs))
-                )
-                return adapter.rename_columns(output, names_out)
+                if self.verbose_feature_names_out:
+                    # `_add_prefix_for_feature_names_out` takes care about raising
+                    # an error if there are duplicated columns.
+                    feature_names_outs = self._add_prefix_for_feature_names_out(
+                        list(zip(transformer_names, feature_names_outs))
+                    )
+                else:
+                    # check for duplicated columns and raise if any
+                    feature_names_outs = list(chain.from_iterable(feature_names_outs))
+                    feature_names_count = Counter(feature_names_outs)
+                    if any(count > 1 for count in feature_names_count.values()):
+                        duplicated_feature_names = [
+                            name
+                            for name, count in feature_names_count.items()
+                            if count > 1
+                        ]
+                        err_msg = (
+                            "Duplicated feature names found before to concatenate the"
+                            " outputs of the transformers:"
+                            f" {duplicated_feature_names}.\n"
+                        )
+                        # find which transformers are responsible for the duplicated
+                        # columns
+                        for transformer_name, X in zip(transformer_names, Xs):
+                            if X.shape[1] == 0:
+                                continue
+                            dup_cols_in_transformer = sorted(
+                                set(X.columns).intersection(duplicated_feature_names)
+                            )
+                            if len(dup_cols_in_transformer):
+                                err_msg += (
+                                    f"Transformer {transformer_name} has conflicting "
+                                    f"columns names: {dup_cols_in_transformer}.\n"
+                                )
+                        raise ValueError(
+                            err_msg
+                            + "Either make sure that the transformers named above "
+                            "do not generate any duplicated columns or set "
+                            "verbose_feature_names_out=True to automatically "
+                            "prefix to the output feature names with the name "
+                            "of the transformer to prevent any conflicting "
+                            "names."
+                        )
+
+                idx = 0
+                for X in Xs:
+                    if X.shape[1] == 0:
+                        continue
+                    names_out = feature_names_outs[idx : idx + X.shape[1]]
+                    adapter.rename_columns(X, names_out)
+                    idx += X.shape[1]
+
+                output = adapter.hstack(Xs)
+                output_samples = output.shape[0]
+                if any(_num_samples(X) != output_samples for X in Xs):
+                    raise ValueError(
+                        "Concatenating DataFrames from the transformer's output lead to"
+                        " an inconsistent number of samples. The output may have Pandas"
+                        " Indexes that do not match."
+                    )
+
+                return output
 
             return np.hstack(Xs)
 
diff --git a/sklearn/compose/tests/test_column_transformer.py b/sklearn/compose/tests/test_column_transformer.py
index fe417e8575e81..61c6d38e61d7b 100644
--- a/sklearn/compose/tests/test_column_transformer.py
+++ b/sklearn/compose/tests/test_column_transformer.py
@@ -2354,6 +2354,65 @@ def test_column_transformer__getitem__():
         ct["does_not_exist"]
 
 
+@pytest.mark.parametrize("dataframe_lib", ["pandas", "polars"])
+def test_column_transformer_column_renaming(dataframe_lib):
+    """Check that we properly rename columns when using `ColumnTransformer` and
+    selected columns are redundant between transformers.
+
+    Non-regression test for:
+    https://github.com/scikit-learn/scikit-learn/issues/28260
+    """
+    lib = pytest.importorskip(dataframe_lib)
+
+    df = lib.DataFrame({"x1": [1, 2, 3], "x2": [10, 20, 30], "x3": [100, 200, 300]})
+
+    transformer = ColumnTransformer(
+        transformers=[
+            ("A", "passthrough", ["x1", "x2", "x3"]),
+            ("B", FunctionTransformer(), ["x1", "x2"]),
+            ("C", StandardScaler(), ["x1", "x3"]),
+        ],
+        verbose_feature_names_out=True,
+    ).set_output(transform=dataframe_lib)
+    df_trans = transformer.fit_transform(df)
+    assert list(df_trans.columns) == [
+        "A__x1",
+        "A__x2",
+        "A__x3",
+        "B__x1",
+        "B__x2",
+        "C__x1",
+        "C__x3",
+    ]
+
+
+@pytest.mark.parametrize("dataframe_lib", ["pandas", "polars"])
+def test_column_transformer_error_with_duplicated_columns(dataframe_lib):
+    """Check that we raise an error when using `ColumnTransformer` and
+    the columns names are duplicated between transformers."""
+    lib = pytest.importorskip(dataframe_lib)
+
+    df = lib.DataFrame({"x1": [1, 2, 3], "x2": [10, 20, 30], "x3": [100, 200, 300]})
+
+    transformer = ColumnTransformer(
+        transformers=[
+            ("A", "passthrough", ["x1", "x2", "x3"]),
+            ("B", FunctionTransformer(), ["x1", "x2"]),
+            ("C", StandardScaler(), ["x1", "x3"]),
+        ],
+        verbose_feature_names_out=False,
+    ).set_output(transform=dataframe_lib)
+    err_msg = re.escape(
+        "Duplicated feature names found before to concatenate the outputs of the "
+        "transformers: ['x1', 'x2', 'x3'].\n"
+        "Transformer A has conflicting columns names: ['x1', 'x2', 'x3'].\n"
+        "Transformer B has conflicting columns names: ['x1', 'x2'].\n"
+        "Transformer C has conflicting columns names: ['x1', 'x3'].\n"
+    )
+    with pytest.raises(ValueError, match=err_msg):
+        transformer.fit_transform(df)
+
+
 # Metadata Routing Tests
 # ======================
 
diff --git a/sklearn/utils/_set_output.py b/sklearn/utils/_set_output.py
index e1792fe369348..d73f8abc7841c 100644
--- a/sklearn/utils/_set_output.py
+++ b/sklearn/utils/_set_output.py
@@ -122,7 +122,10 @@ def is_supported_container(self, X):
         return isinstance(X, pd.DataFrame)
 
     def rename_columns(self, X, columns):
-        return X.rename(columns=dict(zip(X.columns, columns)))
+        # we cannot use `rename` since it takes a dictionary and at this stage we have
+        # potentially duplicate column names in `X`
+        X.columns = columns
+        return X
 
     def hstack(self, Xs):
         pd = check_library_installed("pandas")
@@ -151,7 +154,10 @@ def is_supported_container(self, X):
         return isinstance(X, pl.DataFrame)
 
     def rename_columns(self, X, columns):
-        return X.rename(dict(zip(X.columns, columns)))
+        # we cannot use `rename` since it takes a dictionary and at this stage we have
+        # potentially duplicate column names in `X`
+        X.columns = columns
+        return X
 
     def hstack(self, Xs):
         pl = check_library_installed("polars")
diff --git a/sklearn/utils/tests/test_set_output.py b/sklearn/utils/tests/test_set_output.py
index c94aef3448b29..ebf8089b1e4f4 100644
--- a/sklearn/utils/tests/test_set_output.py
+++ b/sklearn/utils/tests/test_set_output.py
@@ -58,6 +58,14 @@ def test_pandas_adapter():
     )
     pd.testing.assert_frame_equal(X_stacked, expected_df)
 
+    # check that we update properly the columns even with duplicate column names
+    # this use-case potentially happen when using ColumnTransformer
+    # non-regression test for gh-28260
+    X_df = pd.DataFrame([[1, 2], [1, 3]], columns=["a", "a"])
+    new_columns = np.array(["x__a", "y__a"], dtype=object)
+    new_df = adapter.rename_columns(X_df, new_columns)
+    assert_array_equal(new_df.columns, new_columns)
+
 
 def test_polars_adapter():
     """Check Polars adapter has expected behavior."""


Code review comments:
diff_hunk: @@ -1083,13 +1067,69 @@ def _hstack(self, Xs):
                         skip_empty_columns=True,
                     )
                 ]
-                # Selection of columns might be empty.
-                # Hence feature names are filtered for non-emptiness.
                 feature_names_outs = [X.columns for X in Xs if X.shape[1] != 0]
-                names_out = self._add_prefix_for_feature_names_out(
-                    list(zip(transformer_names, feature_names_outs))
-                )
-                return adapter.rename_columns(output, names_out)
+                if self.verbose_feature_names_out:
+                    # `_add_prefix_for_feature_names_out` takes care about raising
+                    # an error if there are duplicated columns.
+                    feature_names_outs = self._add_prefix_for_feature_names_out(
+                        list(zip(transformer_names, feature_names_outs))
+                    )
+                else:
+                    # check for duplicated columns and raise if any
+                    feature_names_outs = list(chain.from_iterable(feature_names_outs))
+                    feature_names_count = Counter(feature_names_outs)
+                    if any(count > 1 for count in feature_names_count.values()):
+                        duplicated_feature_names = [
+                            name
+                            for name, count in feature_names_count.items()
+                            if count > 1
+                        ]
+                        err_msg = (
+                            "Duplicated feature names found before to concatenate the"
+                            " outputs of the transformers:"
+                            f" {duplicated_feature_names}.\n"
+                        )
+                        # find which transformers are responsible for the duplicated
+                        # columns
+                        for transformer_name, X in zip(transformer_names, Xs):
+                            if X.shape[1] == 0:
+                                continue
+                            dup_cols_in_transformer = sorted(
+                                set(X.columns).intersection(duplicated_feature_names)
+                            )
+                            if len(dup_cols_in_transformer):
+                                err_msg += (
+                                    f"Transformer {transformer_name} has conflicting "
+                                    f"columns names: {dup_cols_in_transformer}.\n"
+                                )
+                        raise ValueError(
+                            err_msg
+                            + "Either make sure that the transformers named above "
+                            "do not generate any duplicated columns or set "
path: sklearn/compose/_column_transformer.py
text: @user1:
```suggestion
                            "do not generate columns with conflicting names or set "
```

diff_hunk: @@ -2354,6 +2354,65 @@ def test_column_transformer__getitem__():
         ct["does_not_exist"]
 
 
+@pytest.mark.parametrize("dataframe_lib", ["pandas", "polars"])
+def test_column_transformer_column_renaming(dataframe_lib):
+    """Check that we properly rename columns when using `ColumnTransformer` and
+    selected columns are redundant between transformers.
+
+    Non-regression test for:
+    https://github.com/scikit-learn/scikit-learn/issues/28260
+    """
+    lib = pytest.importorskip(dataframe_lib)
+
+    df = lib.DataFrame({"x1": [1, 2, 3], "x2": [10, 20, 30], "x3": [100, 200, 300]})
+
+    transformer = ColumnTransformer(
+        transformers=[
+            ("A", "passthrough", ["x1", "x2", "x3"]),
+            ("B", FunctionTransformer(), ["x1", "x2"]),
+            ("C", StandardScaler(), ["x1", "x3"]),
+        ],
+        verbose_feature_names_out=True,
+    ).set_output(transform=dataframe_lib)
+    df_trans = transformer.fit_transform(df)
+    assert list(df_trans.columns) == [
+        "A__x1",
+        "A__x2",
+        "A__x3",
+        "B__x1",
+        "B__x2",
+        "C__x1",
+        "C__x3",
+    ]
+
+
+@pytest.mark.parametrize("dataframe_lib", ["pandas", "polars"])
+def test_column_transformer_error_with_duplicated_columns(dataframe_lib):
+    """Check that we raise an error when using `ColumnTransformer` and
+    the columns names are duplicated between transformers."""
+    lib = pytest.importorskip(dataframe_lib)
+
+    df = lib.DataFrame({"x1": [1, 2, 3], "x2": [10, 20, 30], "x3": [100, 200, 300]})
+
+    transformer = ColumnTransformer(
+        transformers=[
+            ("A", "passthrough", ["x1", "x2", "x3"]),
+            ("B", FunctionTransformer(), ["x1", "x2"]),
+            ("C", StandardScaler(), ["x1", "x3"]),
+        ],
+        verbose_feature_names_out=False,
+    ).set_output(transform=dataframe_lib)
+    err_msg = re.escape(
+        "Duplicated feature names found before to concatenate the outputs of the "
path: sklearn/compose/tests/test_column_transformer.py
text: @user1:
```suggestion
        "Duplicated feature names found before concatenating the outputs of the "
```

diff_hunk: @@ -1083,13 +1067,69 @@ def _hstack(self, Xs):
                         skip_empty_columns=True,
                     )
                 ]
-                # Selection of columns might be empty.
-                # Hence feature names are filtered for non-emptiness.
                 feature_names_outs = [X.columns for X in Xs if X.shape[1] != 0]
-                names_out = self._add_prefix_for_feature_names_out(
-                    list(zip(transformer_names, feature_names_outs))
-                )
-                return adapter.rename_columns(output, names_out)
+                if self.verbose_feature_names_out:
+                    # `_add_prefix_for_feature_names_out` takes care about raising
+                    # an error if there are duplicated columns.
+                    feature_names_outs = self._add_prefix_for_feature_names_out(
+                        list(zip(transformer_names, feature_names_outs))
+                    )
+                else:
+                    # check for duplicated columns and raise if any
+                    feature_names_outs = list(chain.from_iterable(feature_names_outs))
+                    feature_names_count = Counter(feature_names_outs)
+                    if any(count > 1 for count in feature_names_count.values()):
+                        duplicated_feature_names = [
+                            name
+                            for name, count in feature_names_count.items()
+                            if count > 1
+                        ]
+                        err_msg = (
+                            "Duplicated feature names found before to concatenate the"
path: sklearn/compose/_column_transformer.py
text: @user1:
```suggestion
                            "Duplicated feature names found before concatenating the"
```