Add Sparse DataFrame support

### Describe the workflow you want to enable

Hello !

It seems that we can configure ColumnTransformer to return a DataFrame via the set_output function.
On the other hand, this requires deactivating sparsity on each underlying transformer which has a negative impact on memory consumption (as explained among others here https://github.com/scikit-learn/scikit-learn/issues/26515).

However, we can in principle use a Sparse DataFrame to store sparse data, right?
![image](https://github.com/scikit-learn/scikit-learn/assets/41157653/340f69b3-11e0-4925-bbe7-552ec4112c4a)

### Describe your proposed solution

Here is a quick draft with a custom ColumnTransformer (only working for Pandas DataFrame).

```python
import pandas as pd
from sklearn.preprocessing import LabelEncoder, OneHotEncoder, OrdinalEncoder
from sklearn.compose import ColumnTransformer

class PandasColumnTransformer(ColumnTransformer):
    """
    > Auto-handle Pandas DataFrame : take DataFrame as input and return Dataframe as output after transform/fit_transform
    > Handle Sparse DataFrame
    """
    
    def fit(self, X, y=None):
        if type(X) is not pd.DataFrame:
            raise Exception('PandasColumnTransformer is designed for Pandas DataFrame only')
        
        return super().fit(X, y)

    def transform(self, X, **params):
        if type(X) is not pd.DataFrame:
            raise Exception('PandasColumnTransformer is designed for Pandas DataFrame only')
        
        transformed=super().transform(X, **params)
    
        #### After applying transform, we ensure that we return a DataFrame / sparse DataFrame
        return self.to_pandas(transformed, X.index)

    # .fit_transform doesnt use .transform, we need to override it too
    def fit_transform(self, X, y=None, **params):
        if type(X) is not pd.DataFrame:
            raise Exception('PandasColumnTransformer is designed for Pandas DataFrame only')
        
        transformed=super().fit_transform(X, y=y, **params)
        
        #### After applying transform, we ensure that we return a DataFrame / sparse DataFrame
        return self.to_pandas(transformed, X.index)

    def to_pandas(self, transformed, index):
        columns=self.get_feature_names_out()
        # if Transformer produces sparse matrix, try to create a sparse DataFrame
        if hasattr(transformed, 'tocsc'):
            print('from_spmatrix')
            return pd.DataFrame.sparse.from_spmatrix(transformed, columns=columns, index=index)
        else:
            print('not sparse')
            return pd.DataFrame(transformed, columns=columns, index=index)
    

df = pd.DataFrame({
    'department': ['operations', 'operations', 'support', 'logistics', 'sales'],
    'review': [0.577569, 0.751900, 0.722548, 0.675158, 0.676203],
    'projects': [3, 3, 3, 4, 3],
    'salary': ['low', 'medium', 'medium', 'low', 'high'],
    'satisfaction': [0.626759, 0.751900, 0.722548, 0.675158, 0.676203],
    'bonus': [0, 0, 0, 0, 1],
    'avg_hrs_month': [180.866070, 182.708149, 184.416084, 188.707545, 179.821083],
    'left': [0, 0, 1, 0, 0]
})
ord_features = ["salary"]
cat_features = ["department"]

ordinal_transformer = OrdinalEncoder()

categorical_transformer = OneHotEncoder(
    sparse_output=True, # <--------
    handle_unknown="ignore")

ct = PandasColumnTransformer(transformers=[
    ("ord", ordinal_transformer, ord_features),
    ("cat", categorical_transformer, cat_features )],
    sparse_threshold=1 # if data sparsity > x% after gathering, make it dense
)

transformed=ct.fit_transform(df)
display(transformed, type(transformed), transformed.dtypes)
```

### Describe alternatives you've considered, if relevant

Is there any alternative to this solution ?

More generally, is there any benefit to switching to a Sparse Dataframe? Or is it useless? (and if so, why ?)

### Additional context

_No response_

COMMENT:
Sparse pandas dataframes have many issues which have made them outside the scope and we intentionally have not included them.

@thomasjpfan might remember more about this.

I'm not sure what polars is doing regarding sparse though.

So closing this as wontfix as much as it goes for pandas sparse dataframes.

Labels: New Feature, Needs Triage