set_output API do not preserve original dtypes for pandas

### Describe the bug

Following issue #24182,

When using the set_output with expected output to be a pandas' data frame, while converting tougher columns with different dtypes the output does not preserve the original dtype but the "common type" by numpy.

Possible workaround is to create transformer for each column, but it doesn't feel like the right approach.

In collaboration with @BenEfrati

### Steps/Code to Reproduce

```python
import numpy as np
import pandas as pd
from sklearn.impute import SimpleImputer



X = pd.DataFrame({
                  "age": [1, 2, 4],
                  "float": [1.5, 2.5, None],
                  })
X["age"] = X["age"].astype(np.uint8)
X.dtypes

# age        uint8
# float    float64
# dtype: object

simple_imputer = SimpleImputer(strategy='constant', fill_value=1).set_output(transform="pandas")
X_trans_partial = simple_imputer.fit_transform(X)
X_trans_partial.dtypes

# age      float64
# float    float64
# dtype: object
```

### Expected Results

We'd expect the age column to keep being an unsigned 8-bit integer.

### Actual Results

the age column is now a 64-bit floating point.

### Versions

```shell
System:
    python: 3.9.12 (main, Apr  4 2022, 05:22:27) [MSC v.1916 64 bit (AMD64)]
executable: C:\New folder\venv\Scripts\python.exe
   machine: Windows-10-10.0.19044-SP0

Python dependencies:
      sklearn: 1.2.1
          pip: 23.0
   setuptools: 58.1.0
        numpy: 1.24.2
        scipy: 1.10.0
       Cython: None
       pandas: 1.5.3
   matplotlib: None
       joblib: 1.2.0
threadpoolctl: 3.1.0

Built with OpenMP: True

threadpoolctl info:
       user_api: blas
   internal_api: openblas
         prefix: libopenblas
       filepath: C:\New folder\venv\Lib\site-packages\numpy\.libs\libopenblas64__v0.3.21-gcc_10_3_0.dll
        version: 0.3.21
threading_layer: pthreads
   architecture: Haswell
    num_threads: 12

       user_api: openmp
   internal_api: openmp
         prefix: vcomp
       filepath: C:\New folder\venv\Lib\site-packages\sklearn\.libs\vcomp140.dll
        version: None
    num_threads: 12

       user_api: blas
   internal_api: openblas
         prefix: libopenblas
       filepath: C:\New folder\venv\Lib\site-packages\scipy.libs\libopenblas-802f9ed1179cb9c9b03d67ff79f48187.dll
        version: 0.3.18
threading_layer: pthreads
   architecture: Haswell
    num_threads: 12
```

COMMENT:
Just to add another use case where I've hit this issue: The [sklearn interface for xgboost](https://xgboost.readthedocs.io/en/stable/tutorials/categorical.html#training-with-scikit-learn-interface) uses the pandas "category" type to infer that a column should be treated as categorical. I'm using a pipeline that is set to output pandas and the category transformer is set to return category type, but I get the whole dataframe back as object type.

```
Pipeline(
        [
            (
                "processing",
                ColumnTransformer(
                    transformers=[
                        (
                            "num",
                            StandardScaler(),
                            numerical_features,
                        ),
                        (
                            "cat",
                            OrdinalEncoder(dtype=pd.CategoricalDtype),
                            categorical_features,
                        ),
                    ],
                    remainder="passthrough",
                ),
            ),
            (
                "model",
                DummyClassifier(),
            ),
        ]
    )
```

Labels: Bug, module:impute, Needs Decision - Include Feature