Add Sparse DataFrame support

### Describe the workflow you want to enable

Hello !

It seems that we can configure ColumnTransformer to return a DataFrame via the set_output function.
On the other hand, this requires deactivating sparsity on each underlying transformer which has a negative impact on memory consumption (as explained among others here https://github.com/scikit-learn/scikit-learn/issues/26515).

However, we can in principle use a Sparse DataFrame to store sparse data, right?
![image](https://github.com/scikit-learn/scikit-learn/assets/41157653/340f69b3-11e0-4925-bbe7-552ec4112c4a)

### Describe your proposed solution

Here is a quick draft with a custom ColumnTransformer (only working for Pandas DataFrame).

```python
import pandas as pd
from sklearn.preprocessing import LabelEncoder, OneHotEncoder, OrdinalEncoder
from sklearn.compose import ColumnTransformer

class PandasColumnTransformer(ColumnTransformer):
    """
    > Auto-handle Pandas DataFrame : take DataFrame as input and return Dataframe as output after transform/fit_transform
    > Handle Sparse DataFrame
    """
    
    def fit(self, X, y=None):
        if type(X) is not pd.DataFrame:
            raise Exception('PandasColumnTransformer is designed for Pandas DataFrame only')
        
        return super().fit(X, y)

    def transform(self, X, **params):
        if type(X) is not pd.DataFrame:
            raise Exception('PandasColumnTransformer is designed for Pandas DataFrame only')
        
        transformed=super().transform(X, **params)
    
        #### After applying transform, we ensure that we return a DataFrame / sparse DataFrame
        return self.to_pandas(transformed, X.index)

    # .fit_transform doesnt use .transform, we need to override it too
    def fit_transform(self, X, y=None, **params):
        if type(X) is not pd.DataFrame:
            raise Exception('PandasColumnTransformer is designed for Pandas DataFrame only')
        
        transformed=super().fit_transform(X, y=y, **params)
        
        #### After applying transform, we ensure that we return a DataFrame / sparse DataFrame
        return self.to_pandas(transformed, X.index)

    def to_pandas(self, transformed, index):
        columns=self.get_feature_names_out()
        # if Transformer produces sparse matrix, try to create a sparse DataFrame
        if hasattr(transformed, 'tocsc'):
            print('from_spmatrix')
            return pd.DataFrame.sparse.from_spmatrix(transformed, columns=columns, index=index)
        else:
            print('not sparse')
            return pd.DataFrame(transformed, columns=columns, index=index)
    

df = pd.DataFrame({
    'department': ['operations', 'operations', 'support', 'logistics', 'sales'],
    'review': [0.577569, 0.751900, 0.722548, 0.675158, 0.676203],
    'projects': [3, 3, 3, 4, 3],
    'salary': ['low', 'medium', 'medium', 'low', 'high'],
    'satisfaction': [0.626759, 0.751900, 0.722548, 0.675158, 0.676203],
    'bonus': [0, 0, 0, 0, 1],
    'avg_hrs_month': [180.866070, 182.708149, 184.416084, 188.707545, 179.821083],
    'left': [0, 0, 1, 0, 0]
})
ord_features = ["salary"]
cat_features = ["department"]

ordinal_transformer = OrdinalEncoder()

categorical_transformer = OneHotEncoder(
    sparse_output=True, # <--------
    handle_unknown="ignore")

ct = PandasColumnTransformer(transformers=[
    ("ord", ordinal_transformer, ord_features),
    ("cat", categorical_transformer, cat_features )],
    sparse_threshold=1 # if data sparsity > x% after gathering, make it dense
)

transformed=ct.fit_transform(df)
display(transformed, type(transformed), transformed.dtypes)
```

### Describe alternatives you've considered, if relevant

Is there any alternative to this solution ?

More generally, is there any benefit to switching to a Sparse Dataframe? Or is it useless? (and if so, why ?)

### Additional context

_No response_

COMMENT:
Hello,

Thank you very much for your feedback which answers all my questions. So I understand better why such a feature is not integrated.

Out of curiosity, is there somewhere I could find details on this kind of implementation choice?

Finally, more generally, even without Sparse DataFrame, there is no benefit to using the Pandas format internally in Pipelines (except for convenience when it comes to writing custom Transformers) because this will always require rendering the dense data, leading to excess memory consumption.

Thanks again and have a nice day!

Labels: New Feature, Needs Triage