Error with set_output(transform='pandas') in ColumnTransformer when using OneHotEncoder with sparse output in intermediate steps

### Describe the bug

**Explanation**

Using the ColumnTransformer with set_output(transform='pandas') raises an error when there is a sparse intermediate output, even if the final output is dense. The error suggests setting sparse_output=False in OneHotEncoder, even though the intermediate sparse output should not impact the final dense output after transformations like TruncatedSVD.


The transformer raises this error even though the final output is dense due to the use of TruncatedSVD, which converts the intermediate sparse output to a dense matrix. The requirement to specify sparse_output=False for OneHotEncoder should not be enforced here, as the final output does not contain sparse data.

**Suggested Fix**

This check should be modified to allow cases where the final output is dense, regardless of intermediate sparse representations.




### Steps/Code to Reproduce

```python
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import OneHotEncoder, MinMaxScaler
from sklearn.decomposition import TruncatedSVD
from sklearn.compose import ColumnTransformer
from sklearn.datasets import load_diabetes
import pandas as pd

ds = load_diabetes()
df = pd.DataFrame(ds['data'], columns=ds['feature_names'])

ct = ColumnTransformer([
    ('ohe_tsvd', make_pipeline(OneHotEncoder(), TruncatedSVD()), ['sex']),
    ('mm', MinMaxScaler(), ['age', 'bmi', 'bp', 's1', 's2', 's3', 's4', 's5', 's6']),
]).set_output(transform='pandas')

ct.fit_transform(df)
```


### Expected Results

pandas DataFrame as follow


ohe_mm__truncatedsvd0 | ohe_mm__truncatedsvd1 | mm__age | mm__bmi | mm__bp | mm__s1 | mm__s2 | mm__s3 | mm__s4 | mm__s5 | mm__s6
-- | -- | -- | -- | -- | -- | -- | -- | -- | -- | --
0.0 | 1.0 | 0.666667 | 0.582645 | 0.549296 | 0.294118 | 0.256972 | 0.207792 | 0.282087 | 0.562217 | 0.439394
1.0 | 0.0 | 0.483333 | 0.148760 | 0.352113 | 0.421569 | 0.306773 | 0.623377 | 0.141044 | 0.222437 | 0.166667
0.0 | 1.0 | 0.883333 | 0.516529 | 0.436620 | 0.289216 | 0.258964 | 0.246753 | 0.282087 | 0.496578 | 0.409091
1.0 | 0.0 | 0.083333 | 0.301653 | 0.309859 | 0.495098 | 0.447211 | 0.233766 | 0.423131 | 0.572923 | 0.469697
1.0 | 0.0 | 0.516667 | 0.206612 | 0.549296 | 0.465686 | 0.417331 | 0.389610 | 0.282087 | 0.362385 | 0.333333

### Actual Results

```pytb
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
Cell In[19], line 10
      5 df = pd.DataFrame(ds['data'], columns=ds['feature_names'])
      6 ct = ColumnTransformer([
      7     ('ohe_mm', make_pipeline(OneHotEncoder(), TruncatedSVD()), ['sex']),
      8     ('mm', MinMaxScaler(), ['age', 'bmi', 'bp', 's1', 's2', 's3', 's4', 's5', 's6']),
      9 ]).set_output(transform='pandas')
---> 10 ct.fit_transform(df)

File [~/python312/lib/python3.12/site-packages/sklearn/utils/_set_output.py:316](http://26.2.133.182:8888/lab/tree/jnote/sunkusun9/kaggle/PGS4_ep11/result/python312/lib/python3.12/site-packages/sklearn/utils/_set_output.py#line=315), in _wrap_method_output.<locals>.wrapped(self, X, *args, **kwargs)
    314 @wraps(f)
    315 def wrapped(self, X, *args, **kwargs):
--> 316     data_to_wrap = f(self, X, *args, **kwargs)
    317     if isinstance(data_to_wrap, tuple):
    318         # only wrap the first output for cross decomposition
    319         return_tuple = (
    320             _wrap_data_with_container(method, data_to_wrap[0], X, self),
    321             *data_to_wrap[1:],
    322         )

File [~/python312/lib/python3.12/site-packages/sklearn/base.py:1473](http://26.2.133.182:8888/lab/tree/jnote/sunkusun9/kaggle/PGS4_ep11/result/python312/lib/python3.12/site-packages/sklearn/base.py#line=1472), in _fit_context.<locals>.decorator.<locals>.wrapper(estimator, *args, **kwargs)
   1466     estimator._validate_params()
   1468 with config_context(
   1469     skip_parameter_validation=(
   1470         prefer_skip_nested_validation or global_skip_validation
   1471     )
   1472 ):
-> 1473     return fit_method(estimator, *args, **kwargs)

File [~/python312/lib/python3.12/site-packages/sklearn/compose/_column_transformer.py:976](http://26.2.133.182:8888/lab/tree/jnote/sunkusun9/kaggle/PGS4_ep11/result/python312/lib/python3.12/site-packages/sklearn/compose/_column_transformer.py#line=975), in ColumnTransformer.fit_transform(self, X, y, **params)
    973 else:
    974     routed_params = self._get_empty_routing()
--> 976 result = self._call_func_on_transformers(
    977     X,
    978     y,
    979     _fit_transform_one,
    980     column_as_labels=False,
    981     routed_params=routed_params,
    982 )
    984 if not result:
    985     self._update_fitted_transformers([])

File [~/python312/lib/python3.12/site-packages/sklearn/compose/_column_transformer.py:885](http://26.2.133.182:8888/lab/tree/jnote/sunkusun9/kaggle/PGS4_ep11/result/python312/lib/python3.12/site-packages/sklearn/compose/_column_transformer.py#line=884), in ColumnTransformer._call_func_on_transformers(self, X, y, func, column_as_labels, routed_params)
    873             extra_args = {}
    874         jobs.append(
    875             delayed(func)(
    876                 transformer=clone(trans) if not fitted else trans,
   (...)
    882             )
    883         )
--> 885     return Parallel(n_jobs=self.n_jobs)(jobs)
    887 except ValueError as e:
    888     if "Expected 2D array, got 1D array instead" in str(e):

File [~/python312/lib/python3.12/site-packages/sklearn/utils/parallel.py:74](http://26.2.133.182:8888/lab/tree/jnote/sunkusun9/kaggle/PGS4_ep11/result/python312/lib/python3.12/site-packages/sklearn/utils/parallel.py#line=73), in Parallel.__call__(self, iterable)
     69 config = get_config()
     70 iterable_with_config = (
     71     (_with_config(delayed_func, config), args, kwargs)
     72     for delayed_func, args, kwargs in iterable
     73 )
---> 74 return super().__call__(iterable_with_config)

File [~/python312/lib/python3.12/site-packages/joblib/parallel.py:1918](http://26.2.133.182:8888/lab/tree/jnote/sunkusun9/kaggle/PGS4_ep11/result/python312/lib/python3.12/site-packages/joblib/parallel.py#line=1917), in Parallel.__call__(self, iterable)
   1916     output = self._get_sequential_output(iterable)
   1917     next(output)
-> 1918     return output if self.return_generator else list(output)
   1920 # Let's create an ID that uniquely identifies the current call. If the
   1921 # call is interrupted early and that the same instance is immediately
   1922 # re-used, this id will be used to prevent workers that were
   1923 # concurrently finalizing a task from the previous call to run the
   1924 # callback.
   1925 with self._lock:

File [~/python312/lib/python3.12/site-packages/joblib/parallel.py:1847](http://26.2.133.182:8888/lab/tree/jnote/sunkusun9/kaggle/PGS4_ep11/result/python312/lib/python3.12/site-packages/joblib/parallel.py#line=1846), in Parallel._get_sequential_output(self, iterable)
   1845 self.n_dispatched_batches += 1
   1846 self.n_dispatched_tasks += 1
-> 1847 res = func(*args, **kwargs)
   1848 self.n_completed_tasks += 1
   1849 self.print_progress()

File [~/python312/lib/python3.12/site-packages/sklearn/utils/parallel.py:136](http://26.2.133.182:8888/lab/tree/jnote/sunkusun9/kaggle/PGS4_ep11/result/python312/lib/python3.12/site-packages/sklearn/utils/parallel.py#line=135), in _FuncWrapper.__call__(self, *args, **kwargs)
    134     config = {}
    135 with config_context(**config):
--> 136     return self.function(*args, **kwargs)

File [~/python312/lib/python3.12/site-packages/sklearn/pipeline.py:1310](http://26.2.133.182:8888/lab/tree/jnote/sunkusun9/kaggle/PGS4_ep11/result/python312/lib/python3.12/site-packages/sklearn/pipeline.py#line=1309), in _fit_transform_one(transformer, X, y, weight, message_clsname, message, params)
   1308 with _print_elapsed_time(message_clsname, message):
   1309     if hasattr(transformer, "fit_transform"):
-> 1310         res = transformer.fit_transform(X, y, **params.get("fit_transform", {}))
   1311     else:
   1312         res = transformer.fit(X, y, **params.get("fit", {})).transform(
   1313             X, **params.get("transform", {})
   1314         )

File [~/python312/lib/python3.12/site-packages/sklearn/base.py:1473](http://26.2.133.182:8888/lab/tree/jnote/sunkusun9/kaggle/PGS4_ep11/result/python312/lib/python3.12/site-packages/sklearn/base.py#line=1472), in _fit_context.<locals>.decorator.<locals>.wrapper(estimator, *args, **kwargs)
   1466     estimator._validate_params()
   1468 with config_context(
   1469     skip_parameter_validation=(
   1470         prefer_skip_nested_validation or global_skip_validation
   1471     )
   1472 ):
-> 1473     return fit_method(estimator, *args, **kwargs)

File [~/python312/lib/python3.12/site-packages/sklearn/pipeline.py:533](http://26.2.133.182:8888/lab/tree/jnote/sunkusun9/kaggle/PGS4_ep11/result/python312/lib/python3.12/site-packages/sklearn/pipeline.py#line=532), in Pipeline.fit_transform(self, X, y, **params)
    490 """Fit the model and transform with the final estimator.
    491 
    492 Fit all the transformers one after the other and sequentially transform
   (...)
    530     Transformed samples.
    531 """
    532 routed_params = self._check_method_params(method="fit_transform", props=params)
--> 533 Xt = self._fit(X, y, routed_params)
    535 last_step = self._final_estimator
    536 with _print_elapsed_time("Pipeline", self._log_message(len(self.steps) - 1)):

File [~/python312/lib/python3.12/site-packages/sklearn/pipeline.py:406](http://26.2.133.182:8888/lab/tree/jnote/sunkusun9/kaggle/PGS4_ep11/result/python312/lib/python3.12/site-packages/sklearn/pipeline.py#line=405), in Pipeline._fit(self, X, y, routed_params)
    404     cloned_transformer = clone(transformer)
    405 # Fit or load from cache the current transformer
--> 406 X, fitted_transformer = fit_transform_one_cached(
    407     cloned_transformer,
    408     X,
    409     y,
    410     None,
    411     message_clsname="Pipeline",
    412     message=self._log_message(step_idx),
    413     params=routed_params[name],
    414 )
    415 # Replace the transformer of the step with the fitted
    416 # transformer. This is necessary when loading the transformer
    417 # from the cache.
    418 self.steps[step_idx] = (name, fitted_transformer)

File [~/python312/lib/python3.12/site-packages/joblib/memory.py:312](http://26.2.133.182:8888/lab/tree/jnote/sunkusun9/kaggle/PGS4_ep11/result/python312/lib/python3.12/site-packages/joblib/memory.py#line=311), in NotMemorizedFunc.__call__(self, *args, **kwargs)
    311 def __call__(self, *args, **kwargs):
--> 312     return self.func(*args, **kwargs)

File [~/python312/lib/python3.12/site-packages/sklearn/pipeline.py:1310](http://26.2.133.182:8888/lab/tree/jnote/sunkusun9/kaggle/PGS4_ep11/result/python312/lib/python3.12/site-packages/sklearn/pipeline.py#line=1309), in _fit_transform_one(transformer, X, y, weight, message_clsname, message, params)
   1308 with _print_elapsed_time(message_clsname, message):
   1309     if hasattr(transformer, "fit_transform"):
-> 1310         res = transformer.fit_transform(X, y, **params.get("fit_transform", {}))
   1311     else:
   1312         res = transformer.fit(X, y, **params.get("fit", {})).transform(
   1313             X, **params.get("transform", {})
   1314         )

File [~/python312/lib/python3.12/site-packages/sklearn/utils/_set_output.py:316](http://26.2.133.182:8888/lab/tree/jnote/sunkusun9/kaggle/PGS4_ep11/result/python312/lib/python3.12/site-packages/sklearn/utils/_set_output.py#line=315), in _wrap_method_output.<locals>.wrapped(self, X, *args, **kwargs)
    314 @wraps(f)
    315 def wrapped(self, X, *args, **kwargs):
--> 316     data_to_wrap = f(self, X, *args, **kwargs)
    317     if isinstance(data_to_wrap, tuple):
    318         # only wrap the first output for cross decomposition
    319         return_tuple = (
    320             _wrap_data_with_container(method, data_to_wrap[0], X, self),
    321             *data_to_wrap[1:],
    322         )

File [~/python312/lib/python3.12/site-packages/sklearn/base.py:1098](http://26.2.133.182:8888/lab/tree/jnote/sunkusun9/kaggle/PGS4_ep11/result/python312/lib/python3.12/site-packages/sklearn/base.py#line=1097), in TransformerMixin.fit_transform(self, X, y, **fit_params)
   1083         warnings.warn(
   1084             (
   1085                 f"This object ({self.__class__.__name__}) has a `transform`"
   (...)
   1093             UserWarning,
   1094         )
   1096 if y is None:
   1097     # fit method of arity 1 (unsupervised transformation)
-> 1098     return self.fit(X, **fit_params).transform(X)
   1099 else:
   1100     # fit method of arity 2 (supervised transformation)
   1101     return self.fit(X, y, **fit_params).transform(X)

File [~/python312/lib/python3.12/site-packages/sklearn/utils/_set_output.py:316](http://26.2.133.182:8888/lab/tree/jnote/sunkusun9/kaggle/PGS4_ep11/result/python312/lib/python3.12/site-packages/sklearn/utils/_set_output.py#line=315), in _wrap_method_output.<locals>.wrapped(self, X, *args, **kwargs)
    314 @wraps(f)
    315 def wrapped(self, X, *args, **kwargs):
--> 316     data_to_wrap = f(self, X, *args, **kwargs)
    317     if isinstance(data_to_wrap, tuple):
    318         # only wrap the first output for cross decomposition
    319         return_tuple = (
    320             _wrap_data_with_container(method, data_to_wrap[0], X, self),
    321             *data_to_wrap[1:],
    322         )

File [~/python312/lib/python3.12/site-packages/sklearn/preprocessing/_encoders.py:1012](http://26.2.133.182:8888/lab/tree/jnote/sunkusun9/kaggle/PGS4_ep11/result/python312/lib/python3.12/site-packages/sklearn/preprocessing/_encoders.py#line=1011), in OneHotEncoder.transform(self, X)
   1010 if transform_output != "default" and self.sparse_output:
   1011     capitalize_transform_output = transform_output.capitalize()
-> 1012     raise ValueError(
   1013         f"{capitalize_transform_output} output does not support sparse data."
   1014         f" Set sparse_output=False to output {transform_output} dataframes or"
   1015         f" disable {capitalize_transform_output} output via"
   1016         '` ohe.set_output(transform="default").'
   1017     )
   1019 # validation of X happens in _check_X called by _transform
   1020 warn_on_unknown = self.drop is not None and self.handle_unknown in {
   1021     "ignore",
   1022     "infrequent_if_exist",
   1023 }

ValueError: Pandas output does not support sparse data. Set sparse_output=False to output pandas dataframes or disable Pandas output via` ohe.set_output(transform="default").
```
### Versions

```shell
System:
    python: 3.12.6 (main, Sep 30 2024, 02:19:13) [GCC 9.4.0]
executable: /home/sun9sun9/python312/bin/python3.12
   machine: Linux-5.4.0-196-generic-x86_64-with-glibc2.31

Python dependencies:
      sklearn: 1.5.2
          pip: 24.3.1
   setuptools: 75.2.0
        numpy: 1.26.4
        scipy: 1.12.0
       Cython: None
       pandas: 2.2.3
   matplotlib: 3.8.4
       joblib: 1.4.2
threadpoolctl: 3.5.0

Built with OpenMP: True

threadpoolctl info:
       user_api: blas
   internal_api: openblas
    num_threads: 8
         prefix: libopenblas
       filepath: /home/sun9sun9/python312/lib/python3.12/site-packages/numpy.libs/libopenblas64_p-r0-0cf96a72.3.23.dev.so
        version: 0.3.23.dev
threading_layer: pthreads
   architecture: Haswell

       user_api: blas
   internal_api: openblas
    num_threads: 8
         prefix: libopenblas
       filepath: /home/sun9sun9/python312/lib/python3.12/site-packages/scipy.libs/libopenblasp-r0-23e5df77.3.21.dev.so
        version: 0.3.21.dev
threading_layer: pthreads
   architecture: Haswell

       user_api: openmp
   internal_api: openmp
    num_threads: 8
         prefix: libgomp
       filepath: /home/sun9sun9/python312/lib/python3.12/site-packages/scikit_learn.libs/libgomp-a34b3233.so.1.0.0
        version: None
```

COMMENT:
Thank you for your thoughtful response. I completely understand and appreciate the consideration about the compatibility of using OneHotEncoder with dimensionality reduction techniques like PCA or TruncatedSVD. My example was purely illustrative to demonstrate the issue in a simple and straightforward way.

As an avid user of scikit-learn, I often rely on its Pipeline and Transformer functionalities for quickly iterating through ideas during the data wrangling phase. The ability to output directly in pandas format is incredibly convenient, particularly for seamless integration with tools like CatBoost, where specifying categorical features using cat_features benefits from pandas-based workflows. I’m genuinely thankful for these powerful features provided by scikit-learn.

Regarding the specific behavior of transform='pandas', while I haven't reviewed the exact implementation, I imagine that the pandas container is likely applied at the final stage of processing rather than being used throughout the transformation pipeline. Upon further reflection, I suspect that this error stems from a well-intentioned attempt to notify users about potential issues with sparse outputs when pandas output is requested—perhaps to preemptively avoid runtime errors or surprises after long-running computations.

That said, I feel there might be ways to enhance the user experience here. For example:

If transform='pandas' is specified and a sparse output arises mid-pipeline, a warning could be raised, and the output could be automatically converted to dense.
Alternatively, the error could be deferred to the end of the pipeline and raised only if the final output cannot be converted to a pandas DataFrame.
Another approach could involve lowering the severity from an Error to a Warning in this specific scenario, allowing users to decide how to proceed.
Finally, the most flexible solution might be to allow users to configure this behavior explicitly through a setting or parameter.
I hope this feedback is helpful and provides some insights from a user’s perspective. Thank you again for all the hard work on scikit-learn—it is truly an indispensable tool for so many of us!

Labels: Bug, Enhancement