Repository: scikit-learn/scikit-learn
Pull request: FIX Improve ColumnTransformer error when indexes are different
<!--
Thanks for contributing a pull request! Please ensure you have taken a look at
the contribution guidelines: https://github.com/scikit-learn/scikit-learn/blob/main/CONTRIBUTING.md
-->

#### Reference Issues/PRs
<!--
Example: Fixes #1234. See also #3456.
Please use keywords (e.g., Fixes) to create link to the issues or pull requests
you resolved, so that they will automatically be closed when your pull request
is merged. See https://github.com/blog/1506-closing-issues-via-pull-requests
-->
Fixes #26210


#### What does this implement/fix? Explain your changes.
This PR adds a check to check that the concatenated DataFrame has a `n_samples` that is consistent with any of the inner transformer's output. The check is quick and gives a informative error compared to the current behavior.

<!--
Please be aware that we are a loose team of volunteers so patience is
necessary; assistance handling other issues is very welcome. We value
all user contributions, no matter how minor they are. If we are slow to
review, either the pull request needs some benchmarking, tinkering,
convincing, etc. or more likely the reviewers are simply busy. In either
case, we ask for your understanding during the review process.
For more information, see our FAQ on this topic:
http://scikit-learn.org/dev/faq.html#why-is-my-pull-request-not-getting-any-attention.

Thanks for contributing!
-->


Problem statement:
confusing fit failure when using set_config(transform_output="pandas")
### Describe the bug

Using `set_config(transform_output="pandas")` in combination with `ColumnTransformer` may result in a failed fit, since indices do not align as expected. The error will not be especially helpful: "ValueError: Found input variables with inconsistent numbers of samples: [705, 527]" (Error appears to be related to merging DataFrames with non-aligned indices, as evident by NaNs and change in shape).

### Steps/Code to Reproduce

```
import pandas as pd
from sklearn.ensemble import GradientBoostingRegressor
from sklearn.pipeline import Pipeline
from sklearn.decomposition import TruncatedSVD
from sklearn import set_config
from sklearn.feature_extraction.text import CountVectorizer

set_config(transform_output="pandas")

d = pd.DataFrame({
    'txt': ['cat eats milk', 'tree says hi', 'bone eats dog'],
}, index=[8, 5, 3])
d


Pipeline([('cv', CountVectorizer()), ('tsvd', TruncatedSVD(n_components=1))]).fit_transform(d.txt)


```

### Expected Results

expected index: [8, 5, 3]

expected behavior: works with ColumnTransformer

### Actual Results

actual index: [0, 1, 2]

actual behavior: doesn't work with ColumnTransformer if other transformers maintain aligned indices

### Versions

```shell
System:
    python: 3.8.16 | packaged by conda-forge | (default, Feb  1 2023, 16:01:55)  [GCC 11.3.0]
executable: /opt/conda/envs/py38/bin/python3.8
   machine: Linux-4.19.0-23-cloud-amd64-x86_64-with-glibc2.10

Python dependencies:
      sklearn: 1.2.2
          pip: 23.0.1
   setuptools: 67.6.1
        numpy: 1.23.5
        scipy: 1.10.1
       Cython: None
       pandas: 1.5.3
   matplotlib: 3.7.1
       joblib: 1.2.0
threadpoolctl: 3.1.0

Built with OpenMP: True

threadpoolctl info:
       user_api: blas
   internal_api: openblas
         prefix: libopenblas
       filepath: /opt/conda/envs/py38/lib/python3.8/site-packages/numpy.libs/libopenblas64_p-r0-742d56dc.3.20.so
        version: 0.3.20
threading_layer: pthreads
   architecture: Haswell
    num_threads: 4

       user_api: openmp
   internal_api: openmp
         prefix: libgomp
       filepath: /opt/conda/envs/py38/lib/python3.8/site-packages/scikit_learn.libs/libgomp-a34b3233.so.1.0.0
        version: None
    num_threads: 4

       user_api: blas
   internal_api: openblas
         prefix: libopenblas
       filepath: /opt/conda/envs/py38/lib/python3.8/site-packages/scipy.libs/libopenblasp-r0-41284840.3.18.so
        version: 0.3.18
threading_layer: pthreads
   architecture: Haswell
    num_threads: 4
```

Hints from issue discussion:
> doesn't work with ColumnTransformer if other transformers maintain aligned indices

Could you please provide a reproducer snippet that triggers the problem with `ColumnTransformer` and include the full traceback?

I assume the problem is related to the use of non-positional indices in the dataframe but it would be helpful to get the original problem.

As off now, scikit-learn does not attempt to preserve dataframe indices. But maybe this is causing a bug in conjunction with the user of `ColumnTransformer`?
No problem. 

(You can also remove the SimpleImputer step to get a different error related to the NaNs, which are visible if just transforming with the ColumnTransformer like `ct.fit_transform(X)`)

This shows the misalignment caused by different indices.

```
import pandas as pd
from sklearn.ensemble import GradientBoostingRegressor
from sklearn.pipeline import Pipeline, make_pipeline
from sklearn.decomposition import TruncatedSVD
from sklearn import set_config
from sklearn.impute import SimpleImputer
from sklearn.feature_extraction.text import CountVectorizer
from sklearn.compose import ColumnTransformer
from sklearn.preprocessing import OneHotEncoder

# call to fit will work if this is commented out
set_config(transform_output="pandas")

X = pd.DataFrame({
    'txt': ['cat eats milk', 'tree says hi', 'bone eats dog'],
}, index=[8, 5, 3])

ct = ColumnTransformer([
    ('first', Pipeline([('cv', CountVectorizer()), ('tsvd', TruncatedSVD(n_components=1))]), 0),
    ('ohe', OneHotEncoder(sparse_output=False), [0]),
])

pipe = make_pipeline(ct, SimpleImputer(), GradientBoostingRegressor())

pipe.fit(X, [1, 2, 3])
```
I'd like to try my hand here, even though I'm not sure if I can solve that. By Friday I'll be able to tell. 
Is this okay @ogrisel?
Thanks for giving me the opportunity. Unfortunately I wasn't able to put enough time aside to investigate this and now I'll be busy with other deadlined tasks for the next 5 days at least. Thus, I cannot post a 'take'.
Note that Scikit-learn transformers will retain the index if there is one:

```python
from sklearn.preprocessing import StandardScaler
import pandas as pd
import sklearn

sklearn.set_config(transform_output="pandas")
X = pd.DataFrame({"a": [1, 2, 3]}, index=[4, 1, 2])

ss = StandardScaler()
X_trans = ss.fit_transform(X)
print(X_trans.index)
# Index([4, 1, 2], dtype='int64')
```

There are two issues with the `ColumnTransformer` snippet in https://github.com/scikit-learn/scikit-learn/issues/26210#issuecomment-1514692919

1. It's possible for transformers in the `ColumnTransformer` to output different indexes. Internally, ColumnTransformer will call `pd.concat` which cares about the indices by default. I prefer raising an error in this case, but checking the indexes add more overhead.
2. `CountVectorizer` outputs a sparse matrix, which should have failed with `set_config(transform_output="pandas")` The `set_output` API does not support pandas output. Since the vectorizer's output was a sparse matrix, the index was not passed to the `TruncatedSVD`. I opened #26253 to raise an error for Vectorizers.

Commit message: Apply suggestions from code review

Co-authored-by: Christian Lorentzen <lorentzen.ch@gmail.com>

Patch to review:
diff --git a/doc/whats_new/v1.3.rst b/doc/whats_new/v1.3.rst
index bb245aa466152..739ced82f9ec9 100644
--- a/doc/whats_new/v1.3.rst
+++ b/doc/whats_new/v1.3.rst
@@ -184,6 +184,13 @@ Changelog
   :user:`Jérémie du Boisberranger <jeremiedbb>`,
   :user:`Guillaume Lemaitre <glemaitre>`.
 
+:mod:`sklearn.compose`
+......................
+
+- |Fix| `compose.ColumnTransformer` raises an informative error when the individual transformers of `ColumnTransformer`
+  output pandas dataframes with indexes that are not consistent with each other and the output is configured
+  to be pandas. :pr:`26286` by `Thomas Fan`_.
+
 :mod:`sklearn.covariance`
 .........................
 
diff --git a/sklearn/compose/_column_transformer.py b/sklearn/compose/_column_transformer.py
index 90718d86d2d89..4eddd12b1baea 100644
--- a/sklearn/compose/_column_transformer.py
+++ b/sklearn/compose/_column_transformer.py
@@ -25,6 +25,7 @@
 from ..utils import check_pandas_support
 from ..utils.metaestimators import _BaseComposition
 from ..utils.validation import check_array, check_is_fitted, _check_feature_names_in
+from ..utils.validation import _num_samples
 from ..utils.parallel import delayed, Parallel
 
 
@@ -854,6 +855,14 @@ def _hstack(self, Xs):
                 pd = check_pandas_support("transform")
                 output = pd.concat(Xs, axis=1)
 
+                output_samples = output.shape[0]
+                if any(_num_samples(X) != output_samples for X in Xs):
+                    raise ValueError(
+                        "Concatenating DataFrames from the transformer's output lead to"
+                        " an inconsistent number of samples. The output may have Pandas"
+                        " Indexes that do not match."
+                    )
+
                 # If all transformers define `get_feature_names_out`, then transform
                 # will adjust the column names to be consistent with
                 # verbose_feature_names_out. Here we prefix the feature names if
diff --git a/sklearn/compose/tests/test_column_transformer.py b/sklearn/compose/tests/test_column_transformer.py
index a2d1cef0412ba..6d1822b387d88 100644
--- a/sklearn/compose/tests/test_column_transformer.py
+++ b/sklearn/compose/tests/test_column_transformer.py
@@ -2157,3 +2157,31 @@ def test_empty_selection_pandas_output(empty_selection):
     ct.set_params(verbose_feature_names_out=False)
     X_out = ct.fit_transform(X)
     assert_array_equal(X_out.columns, ["a", "b"])
+
+
+def test_raise_error_if_index_not_aligned():
+    """Check column transformer raises error if indeices are not aligned.
+
+    Non-regression test for gh-26210.
+    """
+    pd = pytest.importorskip("pandas")
+
+    X = pd.DataFrame([[1.0, 2.2], [3.0, 1.0]], columns=["a", "b"], index=[8, 3])
+    reset_index_transformer = FunctionTransformer(
+        lambda x: x.reset_index(drop=True), feature_names_out="one-to-one"
+    )
+
+    ct = ColumnTransformer(
+        [
+            ("num1", "passthrough", ["a"]),
+            ("num2", reset_index_transformer, ["b"]),
+        ],
+    )
+    ct.set_output(transform="pandas")
+    msg = (
+        "Concatenating DataFrames from the transformer's output lead to"
+        " an inconsistent number of samples. The output may have Pandas"
+        " Indexes that do not match."
+    )
+    with pytest.raises(ValueError, match=msg):
+        ct.fit_transform(X)


Code review comments:
diff_hunk: @@ -2157,3 +2157,31 @@ def test_empty_selection_pandas_output(empty_selection):
     ct.set_params(verbose_feature_names_out=False)
     X_out = ct.fit_transform(X)
     assert_array_equal(X_out.columns, ["a", "b"])
+
+
+def test_raise_error_if_index_not_aligned():
+    """Check column transformer raises error if indeices are not aligned.
path: sklearn/compose/tests/test_column_transformer.py
text: @user1:
```suggestion
    """Check column transformer raises error if indices are not aligned.
```