Performance Degradation in FeatureUnion with String Columns when concatenate the outputs of the transformers

### Describe the bug

I am experiencing significant performance degradation when using FeatureUnion in a Pipeline with DataFrames that include string columns set to be concatenated in the passthrough, the execution time is notably slower.

### Steps/Code to Reproduce

```
import numpy as np
import pandas as pd
from sklearn.base import BaseEstimator, TransformerMixin
from sklearn.datasets import make_classification
from sklearn.pipeline import FeatureUnion, Pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.compose import ColumnTransformer
from joblib import parallel_backend


class CustomStandardScaler(BaseEstimator, TransformerMixin):
    def __init__(self, columns=None):
        self.columns = columns
        self.scaler = None

    def fit(self, X, y=None):
        self.scaler = StandardScaler()
        self.scaler.fit(X[self.columns])
        return self

    def transform(self, X):
        X_scaled = X[self.columns].copy()
        X_scaled = self.scaler.transform(X_scaled)
        return X_scaled

    def fit_transform(self, X, y=None):
        self.fit(X[self.columns], y)
        X_scaled = self.scaler.transform(X[self.columns])
        return X_scaled

    def set_output(self, *, transform=None):
        pass


def benchmark_feature_union(n_samples, n_steps, n_features, n_string_features=0):
    X, y = make_classification(n_samples=n_samples, n_features=n_features, random_state=42)
    X = pd.DataFrame(X)
    X.columns = [f'feature_{i}' for i in range(n_features)]
    num_cols = X.columns.values

    if n_string_features > 0:
        string_data = np.array([f"text_{i}" for i in range(n_samples)])
        string_columns = np.tile(string_data[:, np.newaxis], (1, n_string_features))
        X_string = pd.DataFrame(string_columns, columns=[f'text_feature_{i}' for i in range(n_string_features)])
        X = pd.concat([X, X_string], axis=1)
    steps = []
    for i in range(n_steps):
        step_name = f'scaler_{i + 1}'
        steps.append((step_name, CustomStandardScaler(columns=num_cols)))
    steps.append(('pass', 'passthrough'))

    feature_union = FeatureUnion(steps, verbose=True)

    pipeline = Pipeline([
        ('union', feature_union),
    ], verbose=True)
    pipeline.set_output(transform='pandas')
    with parallel_backend("threading", n_jobs=5):
        transformed_data = pipeline.fit_transform(X, y)

    return transformed_data

# Benchmark with only numerical features
benchmark_feature_union(n_samples=2_000_000, n_steps=5, n_features=150)
# Benchmark with additional string features
benchmark_feature_union(n_samples=2_000_000, n_steps=5, n_features=150, n_string_features=10)
``` 

### Expected Results

The performance should not degrade substantially when string columns are included in the passthrough.

### Actual Results

The performance with string columns is ~5 times slower (~21s vs 1.5min)

[FeatureUnion] ...... (step 2 of 6) Processing scaler_2, total=  11.2s
[FeatureUnion] .......... (step 6 of 6) Processing pass, total=   0.0s
[FeatureUnion] ...... (step 3 of 6) Processing scaler_3, total=  11.3s
[FeatureUnion] ...... (step 4 of 6) Processing scaler_4, total=  11.3s
[FeatureUnion] ...... (step 5 of 6) Processing scaler_5, total=  11.9s
[FeatureUnion] ...... (step 1 of 6) Processing scaler_1, total=  11.9s
[Pipeline] ............. (step 1 of 1) Processing union, total=  21.7s


[FeatureUnion] ...... (step 3 of 6) Processing scaler_3, total=   9.7s[FeatureUnion] ...... (step 5 of 6) Processing scaler_5, total=   9.7s
[FeatureUnion] .......... (step 6 of 6) Processing pass, total=   0.0s

[FeatureUnion] ...... (step 4 of 6) Processing scaler_4, total=   9.7s
[FeatureUnion] ...... (step 2 of 6) Processing scaler_2, total=   9.7s
[FeatureUnion] ...... (step 1 of 6) Processing scaler_1, total=   9.9s
[Pipeline] ............. (step 1 of 1) Processing union, total= 1.5min

Process finished with exit code 0

### Versions

```shell
System:
    python: 3.10.13 (main, Mar 13 2024, 14:52:58) [GCC 11.4.1 20230605 (Red Hat 11.4.1-2)]
   machine: Linux-5.14.0-427.22.1.el9_4.x86_64-x86_64-with-glibc2.34

Python dependencies:
      sklearn: 1.5.0
          pip: 24.0
   setuptools: 71.0.2
        numpy: 1.26.4
        scipy: 1.14.0
       Cython: None
       pandas: 2.2.2
   matplotlib: 3.9.1
       joblib: 1.4.2
threadpoolctl: 3.5.0

Built with OpenMP: True

threadpoolctl info:
       user_api: blas
   internal_api: openblas
    num_threads: 32
         prefix: libopenblas
        version: 0.3.23.dev
threading_layer: pthreads
   architecture: SkylakeX

       user_api: blas
   internal_api: openblas
    num_threads: 32
         prefix: libscipy_openblas
        version: 0.3.27.dev
threading_layer: pthreads
   architecture: SkylakeX

       user_api: openmp
   internal_api: openmp
    num_threads: 32
         prefix: libgomp
        version: None
```

COMMENT:
Hi, I observed the same degradation of performance in my code when mixing in string features. I kinda understand it given the poor support for string type in numpy but I've sadly observed an identical degradation when using  `category` type from pandas. Given the fact that `category` type consists in a basic ordinal encoding more or less behind the scenes, would it makes sense to leverage this pandas type in sklearn? In other terms, would it be an ok enhancement or does it not fit the package?

PS: I already made sure that passing an OrdinalEncoder to encode the string features was indeed improving time performance

Thanks for your help!

Labels: Bug, Needs Triage