RFC Memory usage of tests

I recently tried to run the scikit-learn test suite with the [pytest-memray](https://github.com/bloomberg/pytest-memray) plugin. Here are the tests that allocate the most memory.

Here is are the top 10 worst offenders on my local machine (with CPython 3.11):

```
pytest --memray --most-allocations=10
```

<details>

```
==================================================================== MEMRAY REPORT =====================================================================
Allocations results for sklearn/tests/test_random_projection.py::test_random_projection_numerical_consistency[GaussianRandomProjection]

         📦 Total memory allocated: 348.7MiB
         📏 Total allocations: 208
         📊 Histogram of allocation sizes: |▃ ▁ ▁ ▂ █|
         🥇 Biggest allocating functions:
                - within_tol:/Users/ogrisel/mambaforge/envs/dev/lib/python3.11/site-packages/numpy/core/numeric.py:2361 -> 134.2MiB
                - assert_array_compare:/Users/ogrisel/mambaforge/envs/dev/lib/python3.11/site-packages/numpy/testing/_private/utils.py:799 -> 94.7MiB
                - _gaussian_random_matrix:/Users/ogrisel/code/scikit-learn/sklearn/random_projection.py:196 -> 63.1MiB
                - fit:/Users/ogrisel/code/scikit-learn/sklearn/random_projection.py:417 -> 31.6MiB
                - isclose:/Users/ogrisel/mambaforge/envs/dev/lib/python3.11/site-packages/numpy/core/numeric.py:2378 -> 7.9MiB


Allocations results for sklearn/ensemble/tests/test_iforest.py::test_iforest_with_n_jobs_does_not_segfault

         📦 Total memory allocated: 324.9MiB
         📏 Total allocations: 1052
         📊 Histogram of allocation sizes: |▆       █|
         🥇 Biggest allocating functions:
                - __init__:/Users/ogrisel/mambaforge/envs/dev/lib/python3.11/site-packages/scipy/sparse/_coo.py:190 -> 129.7MiB
                - _array_indexing:/Users/ogrisel/code/scikit-learn/sklearn/utils/__init__.py:186 -> 65.5MiB
                - __init__:/Users/ogrisel/mambaforge/envs/dev/lib/python3.11/site-packages/scipy/sparse/_coo.py:191 -> 64.8MiB
                - _check:/Users/ogrisel/mambaforge/envs/dev/lib/python3.11/site-packages/scipy/sparse/_coo.py:279 -> 32.4MiB
                - _check:/Users/ogrisel/mambaforge/envs/dev/lib/python3.11/site-packages/scipy/sparse/_coo.py:278 -> 32.4MiB


Allocations results for sklearn/tests/test_kernel_approximation.py::test_polynomial_count_sketch[0-3-5000-0.1]

         📦 Total memory allocated: 150.0MiB
         📏 Total allocations: 3118
         📊 Histogram of allocation sizes: |▂ ▅ █ ▅ █|
         🥇 Biggest allocating functions:
                - c2c:/Users/ogrisel/mambaforge/envs/dev/lib/python3.11/site-packages/scipy/fft/_pocketfft/basic.py:31 -> 68.7MiB
                - transform:/Users/ogrisel/code/scikit-learn/sklearn/kernel_approximation.py:217 -> 34.3MiB
                - _wrapreduction:/Users/ogrisel/mambaforge/envs/dev/lib/python3.11/site-packages/numpy/core/fromnumeric.py:86 -> 22.9MiB
                - _wrapreduction:/Users/ogrisel/mambaforge/envs/dev/lib/python3.11/site-packages/numpy/core/fromnumeric.py:86 -> 22.9MiB
                - safe_sparse_dot:/Users/ogrisel/code/scikit-learn/sklearn/utils/extmath.py:193 -> 703.1KiB


Allocations results for sklearn/tests/test_kernel_approximation.py::test_polynomial_count_sketch[2.5-3-5000-2.5]

         📦 Total memory allocated: 149.9MiB
         📏 Total allocations: 3136
         📊 Histogram of allocation sizes: |▂ ▅ ▂ ▅ █|
         🥇 Biggest allocating functions:
                - c2c:/Users/ogrisel/mambaforge/envs/dev/lib/python3.11/site-packages/scipy/fft/_pocketfft/basic.py:31 -> 68.7MiB
                - transform:/Users/ogrisel/code/scikit-learn/sklearn/kernel_approximation.py:217 -> 34.3MiB
                - _wrapreduction:/Users/ogrisel/mambaforge/envs/dev/lib/python3.11/site-packages/numpy/core/fromnumeric.py:86 -> 22.9MiB
                - _wrapreduction:/Users/ogrisel/mambaforge/envs/dev/lib/python3.11/site-packages/numpy/core/fromnumeric.py:86 -> 22.9MiB
                - safe_sparse_dot:/Users/ogrisel/code/scikit-learn/sklearn/utils/extmath.py:193 -> 703.1KiB


Allocations results for sklearn/tests/test_kernel_approximation.py::test_polynomial_count_sketch[2.5-3-5000-1]

         📦 Total memory allocated: 149.9MiB
         📏 Total allocations: 3136
         📊 Histogram of allocation sizes: |▂ ▅ ▂ ▅ █|
         🥇 Biggest allocating functions:
                - c2c:/Users/ogrisel/mambaforge/envs/dev/lib/python3.11/site-packages/scipy/fft/_pocketfft/basic.py:31 -> 68.7MiB
                - transform:/Users/ogrisel/code/scikit-learn/sklearn/kernel_approximation.py:217 -> 34.3MiB
                - _wrapreduction:/Users/ogrisel/mambaforge/envs/dev/lib/python3.11/site-packages/numpy/core/fromnumeric.py:86 -> 22.9MiB
                - _wrapreduction:/Users/ogrisel/mambaforge/envs/dev/lib/python3.11/site-packages/numpy/core/fromnumeric.py:86 -> 22.9MiB
                - safe_sparse_dot:/Users/ogrisel/code/scikit-learn/sklearn/utils/extmath.py:193 -> 703.1KiB


Allocations results for sklearn/tests/test_kernel_approximation.py::test_polynomial_count_sketch[2.5-3-5000-0.1]

         📦 Total memory allocated: 149.9MiB
         📏 Total allocations: 3136
         📊 Histogram of allocation sizes: |▂ ▅ ▂ ▅ █|
         🥇 Biggest allocating functions:
                - c2c:/Users/ogrisel/mambaforge/envs/dev/lib/python3.11/site-packages/scipy/fft/_pocketfft/basic.py:31 -> 68.7MiB
                - transform:/Users/ogrisel/code/scikit-learn/sklearn/kernel_approximation.py:217 -> 34.3MiB
                - _wrapreduction:/Users/ogrisel/mambaforge/envs/dev/lib/python3.11/site-packages/numpy/core/fromnumeric.py:86 -> 22.9MiB
                - _wrapreduction:/Users/ogrisel/mambaforge/envs/dev/lib/python3.11/site-packages/numpy/core/fromnumeric.py:86 -> 22.9MiB
                - safe_sparse_dot:/Users/ogrisel/code/scikit-learn/sklearn/utils/extmath.py:193 -> 703.1KiB


Allocations results for sklearn/tests/test_kernel_approximation.py::test_polynomial_count_sketch[0-3-5000-1]

         📦 Total memory allocated: 149.9MiB
         📏 Total allocations: 3108
         📊 Histogram of allocation sizes: |▂ ▅ ▂ ▅ █|
         🥇 Biggest allocating functions:
                - c2c:/Users/ogrisel/mambaforge/envs/dev/lib/python3.11/site-packages/scipy/fft/_pocketfft/basic.py:31 -> 68.7MiB
                - transform:/Users/ogrisel/code/scikit-learn/sklearn/kernel_approximation.py:217 -> 34.3MiB
                - _wrapreduction:/Users/ogrisel/mambaforge/envs/dev/lib/python3.11/site-packages/numpy/core/fromnumeric.py:86 -> 22.9MiB
                - _wrapreduction:/Users/ogrisel/mambaforge/envs/dev/lib/python3.11/site-packages/numpy/core/fromnumeric.py:86 -> 22.9MiB
                - safe_sparse_dot:/Users/ogrisel/code/scikit-learn/sklearn/utils/extmath.py:193 -> 703.1KiB


Allocations results for sklearn/tests/test_kernel_approximation.py::test_polynomial_count_sketch[0-3-5000-2.5]

         📦 Total memory allocated: 149.9MiB
         📏 Total allocations: 3108
         📊 Histogram of allocation sizes: |▂ ▅ ▂ ▅ █|
         🥇 Biggest allocating functions:
                - c2c:/Users/ogrisel/mambaforge/envs/dev/lib/python3.11/site-packages/scipy/fft/_pocketfft/basic.py:31 -> 68.7MiB
                - transform:/Users/ogrisel/code/scikit-learn/sklearn/kernel_approximation.py:217 -> 34.3MiB
                - _wrapreduction:/Users/ogrisel/mambaforge/envs/dev/lib/python3.11/site-packages/numpy/core/fromnumeric.py:86 -> 22.9MiB
                - _wrapreduction:/Users/ogrisel/mambaforge/envs/dev/lib/python3.11/site-packages/numpy/core/fromnumeric.py:86 -> 22.9MiB
                - safe_sparse_dot:/Users/ogrisel/code/scikit-learn/sklearn/utils/extmath.py:193 -> 703.1KiB


Allocations results for sklearn/tests/test_kernel_approximation.py::test_skewed_chi2_sampler

         📦 Total memory allocated: 103.2MiB
         📏 Total allocations: 217
         📊 Histogram of allocation sizes: |▄ █    |
         🥇 Biggest allocating functions:
                - test_skewed_chi2_sampler:/Users/ogrisel/code/scikit-learn/sklearn/tests/test_kernel_approximation.py:179 -> 103.0MiB
                - test_skewed_chi2_sampler:/Users/ogrisel/code/scikit-learn/sklearn/tests/test_kernel_approximation.py:174 -> 117.2KiB
                - test_skewed_chi2_sampler:/Users/ogrisel/code/scikit-learn/sklearn/tests/test_kernel_approximation.py:173 -> 117.2KiB
                - wrapper:/Users/ogrisel/mambaforge/envs/dev/lib/python3.11/site-packages/pytest_memray/plugin.py:157 -> 2.1KiB


Allocations results for sklearn/ensemble/_hist_gradient_boosting/tests/test_gradient_boosting.py::test_poisson

         📦 Total memory allocated: 99.4MiB
         📏 Total allocations: 327946
         📊 Histogram of allocation sizes: |█ ▅ ▄ ▂  |
         🥇 Biggest allocating functions:
                - split_next:/Users/ogrisel/code/scikit-learn/sklearn/ensemble/_hist_gradient_boosting/grower.py:583 -> 1.5MiB
                - split_next:/Users/ogrisel/code/scikit-learn/sklearn/ensemble/_hist_gradient_boosting/grower.py:587 -> 1006.7KiB
                - _array_indexing:/Users/ogrisel/code/scikit-learn/sklearn/utils/__init__.py:186 -> 472.7KiB
                - dot:<__array_function__ internals>:200 -> 468.8KiB
```

</details>


It seems to confirm that high memory usage in some tests can cause the PyPy CI to crash: #25117.

Questions:

- should we define an official memory limit for individual tests (e.g. 50 MB or less) and then work towards rewriting the offending tests to work within that budget?
- alernatively shall we mark some tests (or test parametrizations) explicitly as high memory tests and use a flag to skip those tests on sensitive CI configs such as PyPy?
- should we use `pytest-memray` on some CI runs? It seems to be compatible with our use of `pytest-xdist` and the overhead seems low enough in quick experiments (to be confirmed on our CI in a PR).

COMMENT:
reading your post, I was also thinking that we shouldn't have a hard limit, but I'm happy to have a limit by default (and we can test that in a CI maybe?), and if it requires a lot of memory, it needs to really have to, otherwise reduce the memory consumption.

Labels: Build / CI, RFC