memory leak for QuantileTransformer

### Describe the bug

There is a doubling of the memory footprint when QuantileTransformer is called on a dataframe and old references to the dataframe are discarded. See repro.


### Steps/Code to Reproduce


```python
import sys, os, gc, psutil
import numpy as np
import pandas as pd
from sklearn.preprocessing import QuantileTransformer
from sklearn.compose import ColumnTransformer


df_train = pd.DataFrame(np.random.randn(1000000, 500), columns = ["C"+str(x) for x in np.arange(500)], ).astype('float32')
ordered_columns = df_train.columns
N, p = df_train.shape
gc.collect()


def current_mem():
    process = psutil.Process(os.getpid())
    rssgb = process.memory_info().rss / 2 ** 30
    print(rssgb)
    return rssgb


def fit_apply_scaler(df_train, columns=ordered_columns):
    if not isinstance(df_train, pd.DataFrame):
        df_train = pd.DataFrame(df_train, columns=columns)
    ordered_columns = df_train.columns
    current_mem()
    columns_to_scale = ["C1", "C2", "C3", "C4", "C73", "C77" , "C10", "C20"]
    scaler = ColumnTransformer([('qts', QuantileTransformer(n_quantiles=20, output_distribution="normal", subsample=N, copy=False, random_state=0), columns_to_scale)], remainder='passthrough', n_jobs=None, verbose=False, verbose_feature_names_out=False).set_output(transform='pandas')
    df_train = scaler.fit_transform(df_train)[ordered_columns].astype('float32').values
    gc.collect()
    current_mem()
    return scaler, df_train


def outerfunc(df_train, ordered_columns=ordered_columns):
    current_mem()
    scaler, df_train = fit_apply_scaler(df_train)
    print(sys.getsizeof(df_train))
    current_mem()
    gc.collect()
    return df_train


for i in range(5):
    df_train = outerfunc(df_train)
    gc.collect()
    current_mem()


sys.getsizeof(df_train)
```

### Expected Results

Memory footprint should not exceed 4GB

### Actual Results

Used memory grows to 6GB upon repetition. Df_train is replaced within the outerfunc by the scaled and transformed array version, but the memory usage does not reflect this until the function is exited. The final df_train is only 128 bytes but memory is not cleared from process.
```python

2.087127685546875
2.087127685546875
3.9517784118652344
128
3.9517784118652344
2.089130401611328
2.089130401611328
2.089130401611328
3.981578826904297
128
3.981578826904297
2.1189308166503906
2.1189308166503906
2.1189308166503906
3.9816246032714844
128
3.9816246032714844
2.118976593017578
2.118976593017578
2.118976593017578
3.9815940856933594
128
3.9815940856933594
2.118946075439453
2.118946075439453
2.118946075439453
3.981700897216797
128
3.981700897216797
2.1190528869628906


sys.getsizeof(df_train)
128

sys.getrefcount(df_train)
5
```

### Versions

```shell
System:
    python: 3.13.5 | packaged by Anaconda, Inc. | (main, Jun 12 2025, 16:09:02) [GCC 11.2.0]
executable: /home/user/anaconda3/bin/python
   machine: Linux-6.6.87.2-microsoft-standard-WSL2-x86_64-with-glibc2.31

Python dependencies:
      sklearn: 1.6.1
          pip: 25.1
   setuptools: 72.1.0
        numpy: 2.1.3
        scipy: 1.15.3
       Cython: None
       pandas: 2.2.3
   matplotlib: 3.10.0
       joblib: 1.4.2
threadpoolctl: 3.5.0

Built with OpenMP: True
```

COMMENT:
Thanks for raising this. The observed memory bloat stems from Python's garbage collector delaying the cleanup of intermediate objects within `QuantileTransformer` and `ColumnTransformer`. When `set_output(transform='pandas')` is used, it constructs a full DataFrame, which is then immediately converted to a NumPy array, leading to a temporary memory footprint nearly 2x the size of the dataset.

Although `df_train` is reassigned inside `outerfunc`, intermediate references remain in scope until the function exits, preventing timely GC reclamation. This is expected behavior in CPython due to lingering stack frame references.

To mitigate:
- Avoid `.set_output(transform='pandas')` if the output is to be immediately cast to NumPy.
- Delete large intermediate variables explicitly and invoke `gc.collect()` after heavy transformations.
- Consider using `transform='default'` to skip redundant DataFrame construction.

This isn’t a memory leak per se, but an artifact of memory handling during large transformation pipelines. If tight memory constraints are critical, minimizing function nesting and avoiding DataFrame-to-NumPy conversions mid-pipeline helps reduce peak memory usage.

**Suggestion:**
```py
def fit_apply_scaler(df_train, columns=ordered_columns):
    columns_to_scale = ["C1", "C2", "C3", "C4", "C73", "C77", "C10", "C20"]
    scaler = ColumnTransformer(
        [('qts', QuantileTransformer(n_quantiles=20, output_distribution="normal", subsample=N, copy=False, random_state=0), columns_to_scale)],
        remainder='passthrough',
        verbose=False
    ).set_output(transform='default')  # <- Output stays as NumPy

    transformed = scaler.fit_transform(df_train)
    del df_train  # Explicitly free memory
    gc.collect()
    return scaler, transformed.astype('float32')
```

Labels: Bug, Performance, module:preprocessing, Needs Investigation