Add Sparse DataFrame support

### Describe the workflow you want to enable

Hello !

It seems that we can configure ColumnTransformer to return a DataFrame via the set_output function.
On the other hand, this requires deactivating sparsity on each underlying transformer which has a negative impact on memory consumption (as explained among others here https://github.com/scikit-learn/scikit-learn/issues/26515).

However, we can in principle use a Sparse DataFrame to store sparse data, right?
![image](https://github.com/scikit-learn/scikit-learn/assets/41157653/340f69b3-11e0-4925-bbe7-552ec4112c4a)

### Describe your proposed solution

Here is a quick draft with a custom ColumnTransformer (only working for Pandas DataFrame).

```python
import pandas as pd
from sklearn.preprocessing import LabelEncoder, OneHotEncoder, OrdinalEncoder
from sklearn.compose import ColumnTransformer

class PandasColumnTransformer(ColumnTransformer):
    """
    > Auto-handle Pandas DataFrame : take DataFrame as input and return Dataframe as output after transform/fit_transform
    > Handle Sparse DataFrame
    """
    
    def fit(self, X, y=None):
        if type(X) is not pd.DataFrame:
            raise Exception('PandasColumnTransformer is designed for Pandas DataFrame only')
        
        return super().fit(X, y)

    def transform(self, X, **params):
        if type(X) is not pd.DataFrame:
            raise Exception('PandasColumnTransformer is designed for Pandas DataFrame only')
        
        transformed=super().transform(X, **params)
    
        #### After applying transform, we ensure that we return a DataFrame / sparse DataFrame
        return self.to_pandas(transformed, X.index)

    # .fit_transform doesnt use .transform, we need to override it too
    def fit_transform(self, X, y=None, **params):
        if type(X) is not pd.DataFrame:
            raise Exception('PandasColumnTransformer is designed for Pandas DataFrame only')
        
        transformed=super().fit_transform(X, y=y, **params)
        
        #### After applying transform, we ensure that we return a DataFrame / sparse DataFrame
        return self.to_pandas(transformed, X.index)

    def to_pandas(self, transformed, index):
        columns=self.get_feature_names_out()
        # if Transformer produces sparse matrix, try to create a sparse DataFrame
        if hasattr(transformed, 'tocsc'):
            print('from_spmatrix')
            return pd.DataFrame.sparse.from_spmatrix(transformed, columns=columns, index=index)
        else:
            print('not sparse')
            return pd.DataFrame(transformed, columns=columns, index=index)
    

df = pd.DataFrame({
    'department': ['operations', 'operations', 'support', 'logistics', 'sales'],
    'review': [0.577569, 0.751900, 0.722548, 0.675158, 0.676203],
    'projects': [3, 3, 3, 4, 3],
    'salary': ['low', 'medium', 'medium', 'low', 'high'],
    'satisfaction': [0.626759, 0.751900, 0.722548, 0.675158, 0.676203],
    'bonus': [0, 0, 0, 0, 1],
    'avg_hrs_month': [180.866070, 182.708149, 184.416084, 188.707545, 179.821083],
    'left': [0, 0, 1, 0, 0]
})
ord_features = ["salary"]
cat_features = ["department"]

ordinal_transformer = OrdinalEncoder()

categorical_transformer = OneHotEncoder(
    sparse_output=True, # <--------
    handle_unknown="ignore")

ct = PandasColumnTransformer(transformers=[
    ("ord", ordinal_transformer, ord_features),
    ("cat", categorical_transformer, cat_features )],
    sparse_threshold=1 # if data sparsity > x% after gathering, make it dense
)

transformed=ct.fit_transform(df)
display(transformed, type(transformed), transformed.dtypes)
```

### Describe alternatives you've considered, if relevant

Is there any alternative to this solution ?

More generally, is there any benefit to switching to a Sparse Dataframe? Or is it useless? (and if so, why ?)

### Additional context

_No response_

COMMENT:
In scikit-learn, we work with SciPy CSR or CSC matrices, while Pandas dataframes are represented as Sparse extension arrays. The conversion between SciPy CSR matricies to Pandas sparse dataframes will create a copy.  The problem gets bigger because there can be multiple copies. For examples:

```python
pipe = make_pipeline(OneHotEncoder(), RandomForestClassifier())
```

If `OntHotEncoder.transform` returns a pandas dataframes with sparse arrays: `OneHotEncoder` will copy a SciPy CSR matrix into pandas. Then `RandomForestClassifier` will make another copy going from a pandas DataFrame back into a SciPy CSR matrix.

When developing `set_output`, I benchmarked Pandas Sparse DataFrame support, but it ended up adding a lot of overhead, even with one copy. It's technically possible, but it's hard for me to recommend because of the performance regression compared to a SciPy sparse matrix.

Labels: New Feature, Needs Triage