set_output API do not preserve original dtypes for pandas

### Describe the bug

Following issue #24182,

When using the set_output with expected output to be a pandas' data frame, while converting tougher columns with different dtypes the output does not preserve the original dtype but the "common type" by numpy.

Possible workaround is to create transformer for each column, but it doesn't feel like the right approach.

In collaboration with @BenEfrati

### Steps/Code to Reproduce

```python
import numpy as np
import pandas as pd
from sklearn.impute import SimpleImputer



X = pd.DataFrame({
                  "age": [1, 2, 4],
                  "float": [1.5, 2.5, None],
                  })
X["age"] = X["age"].astype(np.uint8)
X.dtypes

# age        uint8
# float    float64
# dtype: object

simple_imputer = SimpleImputer(strategy='constant', fill_value=1).set_output(transform="pandas")
X_trans_partial = simple_imputer.fit_transform(X)
X_trans_partial.dtypes

# age      float64
# float    float64
# dtype: object
```

### Expected Results

We'd expect the age column to keep being an unsigned 8-bit integer.

### Actual Results

the age column is now a 64-bit floating point.

### Versions

```shell
System:
    python: 3.9.12 (main, Apr  4 2022, 05:22:27) [MSC v.1916 64 bit (AMD64)]
executable: C:\New folder\venv\Scripts\python.exe
   machine: Windows-10-10.0.19044-SP0

Python dependencies:
      sklearn: 1.2.1
          pip: 23.0
   setuptools: 58.1.0
        numpy: 1.24.2
        scipy: 1.10.0
       Cython: None
       pandas: 1.5.3
   matplotlib: None
       joblib: 1.2.0
threadpoolctl: 3.1.0

Built with OpenMP: True

threadpoolctl info:
       user_api: blas
   internal_api: openblas
         prefix: libopenblas
       filepath: C:\New folder\venv\Lib\site-packages\numpy\.libs\libopenblas64__v0.3.21-gcc_10_3_0.dll
        version: 0.3.21
threading_layer: pthreads
   architecture: Haswell
    num_threads: 12

       user_api: openmp
   internal_api: openmp
         prefix: vcomp
       filepath: C:\New folder\venv\Lib\site-packages\sklearn\.libs\vcomp140.dll
        version: None
    num_threads: 12

       user_api: blas
   internal_api: openblas
         prefix: libopenblas
       filepath: C:\New folder\venv\Lib\site-packages\scipy.libs\libopenblas-802f9ed1179cb9c9b03d67ff79f48187.dll
        version: 0.3.18
threading_layer: pthreads
   architecture: Haswell
    num_threads: 12
```

COMMENT:
Hi I'm trying to look at understandable issues so I can improve my knowledge of scikit-learn, with a view of contributing (other than the 'good first issue' type issues), so spent a bit of time looking at this issue. So don't take what i say too seriously!

Might there be some useful discussion about the impact of this in #24860, which raise some points about how ensuring the pandas dtypes is preserved for all transformers can lead to problems. However, in the case of SimpleImputer, a lot of those points are not relevant, so is this an example of where we could do something similar to #25102 ?

P.S. If it's decided to do some work on this, I'd love to help if possible :)

Labels: Bug, module:impute, Needs Decision - Include Feature