Multiple GMMs with variable training samples lead to memory leak?

### Describe the bug

When i use sklearn GaussianMixture multiple times, while varying the the number of training pixels, it creates a memory leak.

### Steps/Code to Reproduce

```python
import os, psutil
from sklearn import mixture
import numpy as np
import matplotlib.pyplot as plt

process = psutil.Process()

def get_size():
    process = psutil.Process()
    return process.memory_info().rss  / 100000 

def no_leak():
    sizes = []
    for _ in range(100):
        training_pixels = np.random.random((500000,3))
        gmm = mixture.GaussianMixture(
            n_components=5, covariance_type="full", random_state=40
        ).fit(training_pixels)

        sizes.append(get_size())
    return sizes

def leak():
    sizes = []
    for _ in range(100):
        training_pixels = np.random.random((np.random.randint(200000,500000),3))
        gmm = mixture.GaussianMixture(
            n_components=5, covariance_type="full", random_state=40
        ).fit(training_pixels)

        sizes.append(get_size())
    return sizes

#plt.plot(no_leak())
plt.plot(leak())

```

### Expected Results

The "leak" function should not have a memory leak. it's memory footprint should be bounded by the sizes it needs when using the highest possible number of pixels, i.e 500000, which is the number used in the "no_leak" function

### Actual Results

Here is the plot for the "no_leak" function:
![leak1](https://github.com/scikit-learn/scikit-learn/assets/21261959/3fe3599d-ff34-427c-9590-5d13ebc19f81)

It oscillates, but stays below 2100Mb used

Here is the plot for the "leak" function:
![leak2](https://github.com/scikit-learn/scikit-learn/assets/21261959/40b6d189-1cce-4b00-bec2-c0e3776cfeeb)

We can clearly see the memory leak, and it goes much higher than the previous 2100Mb, while having a less or equal number of training examples.

### Versions

```shell
System:
    python: 3.8.10 (default, Jun 22 2022, 20:18:18)  [GCC 9.4.0]
executable: /usr/bin/python3
   machine: Linux-5.4.0-166-generic-x86_64-with-glibc2.29

Python dependencies:
      sklearn: 1.3.2
          pip: 23.3.1
   setuptools: 65.3.0
        numpy: 1.23.2
        scipy: 1.10.1
       Cython: None
       pandas: 2.0.3
   matplotlib: 3.7.3
       joblib: 1.3.2
threadpoolctl: 3.2.0

Built with OpenMP: True

threadpoolctl info:
       user_api: openmp
   internal_api: openmp
    num_threads: 20
         prefix: libgomp
       filepath: /usr/local/lib/python3.8/dist-packages/scikit_learn.libs/libgomp-a34b3233.so.1.0.0
        version: None

       user_api: blas
   internal_api: openblas
    num_threads: 20
         prefix: libopenblas
       filepath: /usr/local/lib/python3.8/dist-packages/numpy.libs/libopenblas64_p-r0-742d56dc.3.20.so
        version: 0.3.20
threading_layer: pthreads
   architecture: Haswell

       user_api: blas
   internal_api: openblas
    num_threads: 20
         prefix: libopenblas
       filepath: /usr/local/lib/python3.8/dist-packages/scipy.libs/libopenblasp-r0-41284840.3.18.so
        version: 0.3.18
threading_layer: pthreads
   architecture: Haswell
```

COMMENT:
Can't reproduce with python=3.11 and latest versions of other deps.

Will reopen if the issue persists with latest versions.

Labels: Bug, Needs Triage