MemoryLeak in `LogisticRession`

### Describe the bug

repro
* Repeatedly call `LogisticRegression().fit(X, y)` on same size feature matrix. Or run the attached repro-script.

expected
* The max memory allocated is steady regardless the number of train calls.

actual
* Memory usage grows linear with the number of calls suggesting a memory leak.

notes
* I'm using `scikit-learn==1.2.2`, and `Python 3.10.11`
* Repro steps works on OSX (M1 Macbook) and on a AWS t3.large running the "Amazon Linux" OS.
* Based on [this thread](https://stackoverflow.com/questions/11195395/scikit-learn-logistic-regression-memory-error) at SO, I tried using `SGDClassifier` instead of `LogisticRegression`. The memory usage behaves as expected when using this solver. I therefore suspect this has to do libsvm not releasing memory correctly after completed training. 
* This ticket seems related to https://github.com/scikit-learn/scikit-learn/issues/217


### Steps/Code to Reproduce

```python
import tracemalloc
import warnings

import numpy as np
from sklearn.linear_model import LogisticRegression, SGDClassifier


def do_train(train_type: str) -> None:
    np.random.seed(42)
    X = np.random.rand(sample_count, feature_dim)
    y = np.random.randint(2, size=sample_count)
    if train_type == "LR":
        clf = LogisticRegression(max_iter=20)
    elif train_type == "SGD":
        clf = SGDClassifier(loss="log_loss", max_iter=20)
    clf.fit(X, y)


def run(reps, train_type):
    tracemalloc.start()
    with warnings.catch_warnings():
        warnings.simplefilter("ignore")
        for _ in range(reps):
            do_train(train_type)

    base, peak = tracemalloc.get_traced_memory()
    tracemalloc.stop()
    print(f"| {reps} | {train_type} | {base/ 1024**2:.2f} | {peak/ 1024**2:.2f} | {(peak-base)/ 1024**2:.2f}")


sample_count = 5000
feature_dim = 200

expected_memory_usage_features = sample_count * feature_dim * 8
expected_memory_usage_classifier = 2 * feature_dim * 8
expected_memory_usage_mb = (expected_memory_usage_features + expected_memory_usage_classifier) / 1024**2

if __name__ == "__main__":
    print(f"Expected memory usage: {expected_memory_usage_mb:.2f} MB")
    print("| # trainings | solver | base (MB) | peak (MB) | diff (MB) |")
    print("| ---- | ---- | ---- | ---- | ---- |")

    run(1, "LR")
    run(2, "LR")
    run(4, "LR")
    run(1, "SGD")
    run(2, "SGD")
    run(4, "SGD")
```

### Expected Results

Expected memory usage: 7.63 MB
| # trainings | solver | base (MB) | peak (MB) | diff (MB) |
| ---- | ---- | ---- | ---- | ---- |
| 1 | LR | 0.02 | 7.83 | 7.81
| 2 | LR | 0.00 | 7.82 | 7.81
| 4 | LR | 0.00 | 7.82 | 7.81
| 1 | SGD | 0.02 | 7.83 | 7.81
| 2 | SGD | 0.00 | 7.82 | 7.81
| 4 | SGD | 0.00 | 7.82 | 7.81

### Actual Results

Expected memory usage: 7.63 MB
| # trainings | solver | base (MB) | peak (MB) | diff (MB) |
| ---- | ---- | ---- | ---- | ---- |
| 1 | LR | 7.83 | 8.06 | 0.23
| 2 | LR | 15.45 | 15.69 | 0.24
| 4 | LR | 30.88 | 31.12 | 0.24
| 1 | SGD | 0.02 | 7.83 | 7.81
| 2 | SGD | 0.00 | 7.82 | 7.81
| 4 | SGD | 0.00 | 7.82 | 7.81

### Versions

```shell
System:
    python: 3.10.11 (v3.10.11:7d4cc5aa85, Apr  4 2023, 19:05:19) [Clang 13.0.0 (clang-1300.0.29.30)]
executable: /Users/beijbom/.virtualenvs/trainer/bin/python
   machine: macOS-14.1.2-arm64-arm-64bit

Python dependencies:
      sklearn: 1.2.2
          pip: 24.0
   setuptools: 68.0.0
        numpy: 1.25.1
        scipy: 1.11.1
       Cython: None
       pandas: 2.1.0
   matplotlib: None
       joblib: 1.3.1
threadpoolctl: 3.2.0

Built with OpenMP: True

threadpoolctl info:
       user_api: openmp
   internal_api: openmp
    num_threads: 8
         prefix: libomp
       filepath: /Users/beijbom/.virtualenvs/trainer/lib/python3.10/site-packages/sklearn/.dylibs/libomp.dylib
        version: None

       user_api: blas
   internal_api: openblas
    num_threads: 8
         prefix: libopenblas
       filepath: /Users/beijbom/.virtualenvs/trainer/lib/python3.10/site-packages/numpy/.dylibs/libopenblas64_.0.dylib
        version: 0.3.23
threading_layer: pthreads
   architecture: armv8

       user_api: blas
   internal_api: openblas
    num_threads: 8
         prefix: libopenblas
       filepath: /Users/beijbom/.virtualenvs/trainer/lib/python3.10/site-packages/scipy/.dylibs/libopenblas.0.dylib
        version: 0.3.21.dev
threading_layer: pthreads
   architecture: armv8
```

COMMENT:
Extreme RAM spike and OOM after upgrade (0.24.2 → 1.1.1) for ultra‑wide sparse L1 Logistic Regression (liblinear & saga) – baseline 680 GB → >1.4 TB (ref #28993)



Summary

After upgrading scikit‑learn and Python, fitting an L1 Logistic Regression on a very large, sparse, high‑dimensional dataset now triggers a massive memory spike (baseline ~680 GB → allocates >1.4 TB → OOM). Previously (with 0.24.2) peak memory stabilized around 750–770 GB for the same data shape. Switching solvers (liblinear → saga) and using SGDClassifier did not resolve the issue.



Workload Scale

Samples: ~7,500,000

Features: ~550,000

Binary classification, L1 regularization

Input matrix: sparse (CSR)



Environments

Stable (old):

Python 3.9

scikit-learn 0.24.2

numpy 1.23.5

scipy 1.7.1

Peak during fit: ~750–770 GB



Problematic (new):

Python 3.10

scikit-learn 1.1.1

scipy 1.14.1

numpy 1.23.5

Baseline resident memory before calling .fit: ~680 GB (holding data & process)

During .fit: climbs rapidly past ~1.4 TB → OOM



Models Tried

Original:

Python

LogisticRegression(

    solver='liblinear',

    C=1.0,

    penalty='l1',

    intercept_scaling=3.0,

    random_state=123,

)



Alternative:

Python

SGDClassifier(

    loss='log_loss',

    penalty='l1',

    alpha=1.0,

    random_state=123,

)



Attempt with saga (still spikes):

Python

LogisticRegression(

    solver='saga',

    C=1.0,

    penalty='l1',

    max_iter=1000,

    n_jobs=-1,

    random_state=123,

)



New Observation

Memory right before training (after data load / preprocessing) sits around 680 GB.

Training triggers a single large jump rather than gradual leakage across multiple fits.

This suggests a large dense working allocation or multiple parallel buffers rather than slow accumulation.



Questions

Are there known regressions or algorithmic changes between 0.24.2 and 1.1.1 that increase memory for sparse, high‑feature L1 logistic regression (liblinear or saga)?
Does liblinear (L1) or saga under certain conditions implicitly densify parts of X at this scale?
Could n_jobs=-1 for saga (or OpenMP threads) multiply large temporary buffers? Is single-thread execution recommended for memory control here?
Is intercept_scaling=3.0 relevant for any hidden copy or augmentation step that becomes costly at extreme dimensionality?
Are there scipy 1.14.1 sparse changes that alter memory layout or workspace usage compared to what 0.24.2 relied on?
Recommended solver / pipeline for this regime (7.5M × 550k, very sparse, L1): saga vs SGD vs feature hashing + SGD vs dimensionality reduction first?
Would upgrading to a newer scikit-learn (1.5.x / 1.6.dev) likely help (any memory PRs touching saga/liblinear)?

Labels: Bug