set_config(transform_output="pandas") does not act on inverse_transform

### Describe the bug

The new [`set_config(transform_output='pandas')` functionality](https://blog.scikit-learn.org/technical/pandas-dataframe-output-for-sklearn-transformer/) is very useful, but unfortunately it is only taking an effect on the `transform` method of a transformer and not on the `inverse_transform`.

### Steps/Code to Reproduce

```python
import pandas as pd
from sklearn import set_config
from sklearn.datasets import load_diabetes
from sklearn.preprocessing import StandardScaler

set_config(transform_output="pandas")

# Load the diabetes dataset
X, _ = load_diabetes(return_X_y=True)
X = pd.DataFrame(X)
scaler = StandardScaler()
Xt = scaler.fit_transform(X)
assert isinstance(Xt, pd.DataFrame)  # OK!

X = scaler.inverse_transform(Xt)
assert isinstance(X, pd.DataFrame)  # AssertionError!
```

### Expected Results

Output of `StandardScale.inverse_transform()` to be a pandas DataFrame.

### Actual Results

---------------------------------------------------------------------------                                                                                 
AssertionError                            Traceback (most recent call last)
Cell In[25], line 16
     13 assert isinstance(Xt, pd.DataFrame)
     15 X = scaler.inverse_transform(Xt)
---> 16 assert isinstance(X, pd.DataFrame)

AssertionError: 

### Versions

```shell
System:
    python: 3.10.6 (main, Nov 14 2022, 16:10:14) [GCC 11.3.0]
executable: /usr/bin/python3.10
   machine: Linux-5.14.0-1038-oem-x86_64-with-glibc2.35

Python dependencies:
      sklearn: 1.2.1
          pip: 23.0
   setuptools: 65.3.0
        numpy: 1.22.4
        scipy: 1.9.0
       Cython: 0.29.30
       pandas: 1.5.1
   matplotlib: 3.5.3
       joblib: 1.2.0
threadpoolctl: 3.1.0

Built with OpenMP: True

threadpoolctl info:
       user_api: blas
   internal_api: openblas
         prefix: libopenblas
       filepath: /home/afr/.local/lib/python3.10/site-packages/numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so
        version: 0.3.18
threading_layer: pthreads
   architecture: Prescott
    num_threads: 16

       user_api: blas
   internal_api: openblas
         prefix: libopenblas
       filepath: /home/afr/.local/lib/python3.10/site-packages/scipy.libs/libopenblasp-r0-9f9f5dbc.3.18.so
        version: 0.3.18
threading_layer: pthreads
   architecture: Prescott
    num_threads: 16

       user_api: openmp
   internal_api: openmp
         prefix: libgomp
       filepath: /usr/lib/x86_64-linux-gnu/libgomp.so.1.0.0
        version: None
    num_threads: 16

       user_api: openmp
   internal_api: openmp
         prefix: libgomp
       filepath: /home/afr/.local/lib/python3.10/site-packages/scikit_learn.libs/libgomp-a34b3233.so.1.0.0
        version: None
    num_threads: 16

       user_api: openmp
   internal_api: openmp
         prefix: libgomp
       filepath: /home/afr/.local/lib/python3.10/site-packages/torch/lib/libgomp-a34b3233.so.1
        version: None
    num_threads: 8
```

COMMENT:
Yea, the set_output API only works on `transform`. Any other methods such as `inverse_transform`, `predict`, or `predict_proba` has not been added to the API yet. (I purposely reduce the scope of `set_output` to `transform`, so the feature can get released without waiting on the other methods)

If we want to add `inverse_transform` and have a consistent API, it will likely be `set_output(inverse_transform_output="pandas")`.

In any case, this is feature request and we need to decide if we want to add it to the API.

Labels: New Feature, Needs Decision - Include Feature