MemoryLeak in `LogisticRession`

### Describe the bug

repro
* Repeatedly call `LogisticRegression().fit(X, y)` on same size feature matrix. Or run the attached repro-script.

expected
* The max memory allocated is steady regardless the number of train calls.

actual
* Memory usage grows linear with the number of calls suggesting a memory leak.

notes
* I'm using `scikit-learn==1.2.2`, and `Python 3.10.11`
* Repro steps works on OSX (M1 Macbook) and on a AWS t3.large running the "Amazon Linux" OS.
* Based on [this thread](https://stackoverflow.com/questions/11195395/scikit-learn-logistic-regression-memory-error) at SO, I tried using `SGDClassifier` instead of `LogisticRegression`. The memory usage behaves as expected when using this solver. I therefore suspect this has to do libsvm not releasing memory correctly after completed training. 
* This ticket seems related to https://github.com/scikit-learn/scikit-learn/issues/217


### Steps/Code to Reproduce

```python
import tracemalloc
import warnings

import numpy as np
from sklearn.linear_model import LogisticRegression, SGDClassifier


def do_train(train_type: str) -> None:
    np.random.seed(42)
    X = np.random.rand(sample_count, feature_dim)
    y = np.random.randint(2, size=sample_count)
    if train_type == "LR":
        clf = LogisticRegression(max_iter=20)
    elif train_type == "SGD":
        clf = SGDClassifier(loss="log_loss", max_iter=20)
    clf.fit(X, y)


def run(reps, train_type):
    tracemalloc.start()
    with warnings.catch_warnings():
        warnings.simplefilter("ignore")
        for _ in range(reps):
            do_train(train_type)

    base, peak = tracemalloc.get_traced_memory()
    tracemalloc.stop()
    print(f"| {reps} | {train_type} | {base/ 1024**2:.2f} | {peak/ 1024**2:.2f} | {(peak-base)/ 1024**2:.2f}")


sample_count = 5000
feature_dim = 200

expected_memory_usage_features = sample_count * feature_dim * 8
expected_memory_usage_classifier = 2 * feature_dim * 8
expected_memory_usage_mb = (expected_memory_usage_features + expected_memory_usage_classifier) / 1024**2

if __name__ == "__main__":
    print(f"Expected memory usage: {expected_memory_usage_mb:.2f} MB")
    print("| # trainings | solver | base (MB) | peak (MB) | diff (MB) |")
    print("| ---- | ---- | ---- | ---- | ---- |")

    run(1, "LR")
    run(2, "LR")
    run(4, "LR")
    run(1, "SGD")
    run(2, "SGD")
    run(4, "SGD")
```

### Expected Results

Expected memory usage: 7.63 MB
| # trainings | solver | base (MB) | peak (MB) | diff (MB) |
| ---- | ---- | ---- | ---- | ---- |
| 1 | LR | 0.02 | 7.83 | 7.81
| 2 | LR | 0.00 | 7.82 | 7.81
| 4 | LR | 0.00 | 7.82 | 7.81
| 1 | SGD | 0.02 | 7.83 | 7.81
| 2 | SGD | 0.00 | 7.82 | 7.81
| 4 | SGD | 0.00 | 7.82 | 7.81

### Actual Results

Expected memory usage: 7.63 MB
| # trainings | solver | base (MB) | peak (MB) | diff (MB) |
| ---- | ---- | ---- | ---- | ---- |
| 1 | LR | 7.83 | 8.06 | 0.23
| 2 | LR | 15.45 | 15.69 | 0.24
| 4 | LR | 30.88 | 31.12 | 0.24
| 1 | SGD | 0.02 | 7.83 | 7.81
| 2 | SGD | 0.00 | 7.82 | 7.81
| 4 | SGD | 0.00 | 7.82 | 7.81

### Versions

```shell
System:
    python: 3.10.11 (v3.10.11:7d4cc5aa85, Apr  4 2023, 19:05:19) [Clang 13.0.0 (clang-1300.0.29.30)]
executable: /Users/beijbom/.virtualenvs/trainer/bin/python
   machine: macOS-14.1.2-arm64-arm-64bit

Python dependencies:
      sklearn: 1.2.2
          pip: 24.0
   setuptools: 68.0.0
        numpy: 1.25.1
        scipy: 1.11.1
       Cython: None
       pandas: 2.1.0
   matplotlib: None
       joblib: 1.3.1
threadpoolctl: 3.2.0

Built with OpenMP: True

threadpoolctl info:
       user_api: openmp
   internal_api: openmp
    num_threads: 8
         prefix: libomp
       filepath: /Users/beijbom/.virtualenvs/trainer/lib/python3.10/site-packages/sklearn/.dylibs/libomp.dylib
        version: None

       user_api: blas
   internal_api: openblas
    num_threads: 8
         prefix: libopenblas
       filepath: /Users/beijbom/.virtualenvs/trainer/lib/python3.10/site-packages/numpy/.dylibs/libopenblas64_.0.dylib
        version: 0.3.23
threading_layer: pthreads
   architecture: armv8

       user_api: blas
   internal_api: openblas
    num_threads: 8
         prefix: libopenblas
       filepath: /Users/beijbom/.virtualenvs/trainer/lib/python3.10/site-packages/scipy/.dylibs/libopenblas.0.dylib
        version: 0.3.21.dev
threading_layer: pthreads
   architecture: armv8
```

COMMENT:
> So this is not a real memory leak, but more of a delayed garbage collection. This can indeed be caused by cyclic references involving data structures that also hold references to large numpy arrays.

I am almost sure that the issue is happening when calling the `optimize.minimize` function by looking at the number if putting a `gc.collect()` before or after the call. So my guess would be to look at the C-function `loss_gradient` to be sure that we don't have something going on there.

Labels: Bug