Segmentation fault in sklearn.metrics.pairwise_distances with OpenBLAS 0.3.28 (only pthreads variant)

```
mamba create -n testenv scikit-learn python=3.12 libopenblas=0.3.28 -y
conda activate testenv
PYTHONFAULTHANDLER=1 python /tmp/test_openblas.py
```

```py
# /tmp/test_openblas.py
import numpy as np

from joblib import Parallel, delayed
from threadpoolctl import threadpool_limits

from sklearn.metrics.pairwise import pairwise_distances


X = np.ones((1000, 10))


def blas_threaded_func(i):
    X.T @ X


# Needs to be there and before Parallel
threadpool_limits(10)

Parallel(n_jobs=2)(delayed(blas_threaded_func)(i) for i in range(10))

for _ in range(10):
    distances = pairwise_distances(X, metric="l2", n_jobs=2)
```

This happens with OpenBLAS 0.3.28 but not 0.3.27. Setting the `OPENBLAS_NUM_THREADS` or `OMP_NUM_THREADS` environment variable also make the issue disappear.
 
This is somewhat reminiscent of https://github.com/scipy/scipy/issues/21479 so there may be something in OpenBLAS 0.3.28 [^1] that doesn't like `threapool_limits` followed by `Parallel`? No idea how to test this hypothesis ... this could well be OS-dependent since https://github.com/scipy/scipy/issues/21479 only happens on Linux.

[^1]: OpenBLAS 0.3.28 is used in numpy development wheel and OpenBLAS 0.3.27 is used in numpy latest release 2.1.2 at the time of writing

<details>

<summary>Python traceback</summary>

```
Fatal Python error: Segmentation fault

Thread 0x00007c7907e006c0 (most recent call first):
  File "/home/lesteve/micromamba/envs/testenv/lib/python3.12/multiprocessing/pool.py", line 579 in _handle_results
  File "/home/lesteve/micromamba/envs/testenv/lib/python3.12/threading.py", line 1012 in run
  File "/home/lesteve/micromamba/envs/testenv/lib/python3.12/threading.py", line 1075 in _bootstrap_inner
  File "/home/lesteve/micromamba/envs/testenv/lib/python3.12/threading.py", line 1032 in _bootstrap

Thread 0x00007c790d2006c0 (most recent call first):
  File "/home/lesteve/micromamba/envs/testenv/lib/python3.12/multiprocessing/pool.py", line 531 in _handle_tasks
  File "/home/lesteve/micromamba/envs/testenv/lib/python3.12/threading.py", line 1012 in run
  File "/home/lesteve/micromamba/envs/testenv/lib/python3.12/threading.py", line 1075 in _bootstrap_inner
  File "/home/lesteve/micromamba/envs/testenv/lib/python3.12/threading.py", line 1032 in _bootstrap

Thread 0x00007c790dc006c0 (most recent call first):
  File "/home/lesteve/micromamba/envs/testenv/lib/python3.12/selectors.py", line 415 in select
  File "/home/lesteve/micromamba/envs/testenv/lib/python3.12/multiprocessing/connection.py", line 1136 in wait
  File "/home/lesteve/micromamba/envs/testenv/lib/python3.12/multiprocessing/pool.py", line 502 in _wait_for_updates
  File "/home/lesteve/micromamba/envs/testenv/lib/python3.12/multiprocessing/pool.py", line 522 in _handle_workers
  File "/home/lesteve/micromamba/envs/testenv/lib/python3.12/threading.py", line 1012 in run
  File "/home/lesteve/micromamba/envs/testenv/lib/python3.12/threading.py", line 1075 in _bootstrap_inner
  File "/home/lesteve/micromamba/envs/testenv/lib/python3.12/threading.py", line 1032 in _bootstrap

Thread 0x00007c79146006c0 (most recent call first):
  File "/home/lesteve/micromamba/envs/testenv/lib/python3.12/site-packages/sklearn/utils/extmath.py", line 205 in safe_sparse_dot
  File "/home/lesteve/micromamba/envs/testenv/lib/python3.12/site-packages/sklearn/metrics/pairwise.py", line 407 in _euclidean_distances
  File "/home/lesteve/micromamba/envs/testenv/lib/python3.12/site-packages/sklearn/metrics/pairwise.py", line 372 in euclidean_distances
  File "/home/lesteve/micromamba/envs/testenv/lib/python3.12/site-packages/sklearn/utils/_param_validation.py", line 186 in wrapper
  File "/home/lesteve/micromamba/envs/testenv/lib/python3.12/site-packages/sklearn/metrics/pairwise.py", line 1881 in _dist_wrapper
  File "/home/lesteve/micromamba/envs/testenv/lib/python3.12/site-packages/sklearn/utils/parallel.py", line 136 in __call__
  File "/home/lesteve/micromamba/envs/testenv/lib/python3.12/site-packages/joblib/parallel.py", line 598 in __call__
  File "/home/lesteve/micromamba/envs/testenv/lib/python3.12/site-packages/joblib/_utils.py", line 72 in __call__
  File "/home/lesteve/micromamba/envs/testenv/lib/python3.12/multiprocessing/pool.py", line 125 in worker
  File "/home/lesteve/micromamba/envs/testenv/lib/python3.12/threading.py", line 1012 in run
  File "/home/lesteve/micromamba/envs/testenv/lib/python3.12/threading.py", line 1075 in _bootstrap_inner
  File "/home/lesteve/micromamba/envs/testenv/lib/python3.12/threading.py", line 1032 in _bootstrap

Thread 0x00007c7913c006c0 (most recent call first):
  File "/home/lesteve/micromamba/envs/testenv/lib/python3.12/site-packages/sklearn/utils/extmath.py", line 205 in safe_sparse_dot
  File "/home/lesteve/micromamba/envs/testenv/lib/python3.12/site-packages/sklearn/metrics/pairwise.py", line 407 in _euclidean_distances
  File "/home/lesteve/micromamba/envs/testenv/lib/python3.12/site-packages/sklearn/metrics/pairwise.py", line 372 in euclidean_distances
  File "/home/lesteve/micromamba/envs/testenv/lib/python3.12/site-packages/sklearn/utils/_param_validation.py", line 186 in wrapper
  File "/home/lesteve/micromamba/envs/testenv/lib/python3.12/site-packages/sklearn/metrics/pairwise.py", line 1881 in _dist_wrapper
  File "/home/lesteve/micromamba/envs/testenv/lib/python3.12/site-packages/sklearn/utils/parallel.py", line 136 in __call__
  File "/home/lesteve/micromamba/envs/testenv/lib/python3.12/site-packages/joblib/parallel.py", line 598 in __call__
  File "/home/lesteve/micromamba/envs/testenv/lib/python3.12/site-packages/joblib/_utils.py", line 72 in __call__
  File "/home/lesteve/micromamba/envs/testenv/lib/python3.12/multiprocessing/pool.py", line 125 in worker
  File "/home/lesteve/micromamba/envs/testenv/lib/python3.12/threading.py", line 1012 in run
  File "/home/lesteve/micromamba/envs/testenv/lib/python3.12/threading.py", line 1075 in _bootstrap_inner
  File "/home/lesteve/micromamba/envs/testenv/lib/python3.12/threading.py", line 1032 in _bootstrap

Thread 0x00007c79132006c0 (most recent call first):
  File "/home/lesteve/micromamba/envs/testenv/lib/python3.12/threading.py", line 355 in wait
  File "/home/lesteve/micromamba/envs/testenv/lib/python3.12/site-packages/joblib/externals/loky/backend/queues.py", line 147 in _feed
  File "/home/lesteve/micromamba/envs/testenv/lib/python3.12/threading.py", line 1012 in run
  File "/home/lesteve/micromamba/envs/testenv/lib/python3.12/threading.py", line 1075 in _bootstrap_inner
  File "/home/lesteve/micromamba/envs/testenv/lib/python3.12/threading.py", line 1032 in _bootstrap

Thread 0x00007c79128006c0 (most recent call first):
  File "/home/lesteve/micromamba/envs/testenv/lib/python3.12/selectors.py", line 415 in select
  File "/home/lesteve/micromamba/envs/testenv/lib/python3.12/multiprocessing/connection.py", line 1136 in wait
  File "/home/lesteve/micromamba/envs/testenv/lib/python3.12/site-packages/joblib/externals/loky/process_executor.py", line 654 in wait_result_broken_or_wakeup
  File "/home/lesteve/micromamba/envs/testenv/lib/python3.12/site-packages/joblib/externals/loky/process_executor.py", line 596 in run
  File "/home/lesteve/micromamba/envs/testenv/lib/python3.12/threading.py", line 1075 in _bootstrap_inner
  File "/home/lesteve/micromamba/envs/testenv/lib/python3.12/threading.py", line 1032 in _bootstrap

Thread 0x00007c797c527480 (most recent call first):
  File "/home/lesteve/micromamba/envs/testenv/lib/python3.12/site-packages/joblib/parallel.py", line 1762 in _retrieve
  File "/home/lesteve/micromamba/envs/testenv/lib/python3.12/site-packages/joblib/parallel.py", line 1650 in _get_outputs
  File "/home/lesteve/micromamba/envs/testenv/lib/python3.12/site-packages/joblib/parallel.py", line 2007 in __call__
  File "/home/lesteve/micromamba/envs/testenv/lib/python3.12/site-packages/sklearn/utils/parallel.py", line 74 in __call__
  File "/home/lesteve/micromamba/envs/testenv/lib/python3.12/site-packages/sklearn/metrics/pairwise.py", line 1898 in _parallel_pairwise
  File "/home/lesteve/micromamba/envs/testenv/lib/python3.12/site-packages/sklearn/metrics/pairwise.py", line 2375 in pairwise_distances
  File "/home/lesteve/micromamba/envs/testenv/lib/python3.12/site-packages/sklearn/utils/_param_validation.py", line 213 in wrapper
  File "/tmp/test_openblas.py", line 21 in <module>
```

</details>


<details>

<summary>Version info</summary>

```
System:
    python: 3.12.7 | packaged by conda-forge | (main, Oct  4 2024, 16:05:46) [GCC 13.3.0]
executable: /home/lesteve/micromamba/envs/testenv/bin/python
   machine: Linux-6.10.10-arch1-1-x86_64-with-glibc2.40

Python dependencies:
      sklearn: 1.5.2
          pip: 24.2
   setuptools: 75.1.0
        numpy: 2.1.2
        scipy: 1.14.1
       Cython: None
       pandas: None
   matplotlib: None
       joblib: 1.4.2
threadpoolctl: 3.5.0

Built with OpenMP: True

threadpoolctl info:
       user_api: blas
   internal_api: openblas
    num_threads: 12
         prefix: libopenblas
       filepath: /home/lesteve/micromamba/envs/testenv/lib/libopenblasp-r0.3.28.so
        version: 0.3.28
threading_layer: pthreads
   architecture: Haswell

       user_api: openmp
   internal_api: openmp
    num_threads: 12
         prefix: libgomp
       filepath: /home/lesteve/micromamba/envs/testenv/lib/libgomp.so.1.0.0
        version: None
```

</details>

<details>

<summary>mamba list output</summary>

```
 ❯ mamba list                                                            
List of packages in environment: "/home/lesteve/micromamba/envs/testenv"

  Name              Version    Build                Channel                                                          
───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
  _libgcc_mutex     0.1        conda_forge          conda-forge/linux-64/_libgcc_mutex-0.1-conda_forge.tar.bz2       
  _openmp_mutex     4.5        2_gnu                conda-forge/linux-64/_openmp_mutex-4.5-2_gnu.tar.bz2             
  bzip2             1.0.8      h4bc722e_7           conda-forge/linux-64/bzip2-1.0.8-h4bc722e_7.conda                
  ca-certificates   2024.8.30  hbcca054_0           conda-forge/linux-64/ca-certificates-2024.8.30-hbcca054_0.conda  
  joblib            1.4.2      pyhd8ed1ab_0         conda-forge/noarch/joblib-1.4.2-pyhd8ed1ab_0.conda               
  ld_impl_linux-64  2.43       h712a8e2_2           conda-forge/linux-64/ld_impl_linux-64-2.43-h712a8e2_2.conda      
  libblas           3.9.0      25_linux64_openblas  conda-forge/linux-64/libblas-3.9.0-25_linux64_openblas.conda     
  libcblas          3.9.0      25_linux64_openblas  conda-forge/linux-64/libcblas-3.9.0-25_linux64_openblas.conda    
  libexpat          2.6.3      h5888daf_0           conda-forge/linux-64/libexpat-2.6.3-h5888daf_0.conda             
  libffi            3.4.2      h7f98852_5           conda-forge/linux-64/libffi-3.4.2-h7f98852_5.tar.bz2             
  libgcc            14.2.0     h77fa898_1           conda-forge/linux-64/libgcc-14.2.0-h77fa898_1.conda              
  libgcc-ng         14.2.0     h69a702a_1           conda-forge/linux-64/libgcc-ng-14.2.0-h69a702a_1.conda           
  libgfortran       14.2.0     h69a702a_1           conda-forge/linux-64/libgfortran-14.2.0-h69a702a_1.conda         
  libgfortran-ng    14.2.0     h69a702a_1           conda-forge/linux-64/libgfortran-ng-14.2.0-h69a702a_1.conda      
  libgfortran5      14.2.0     hd5240d6_1           conda-forge/linux-64/libgfortran5-14.2.0-hd5240d6_1.conda        
  libgomp           14.2.0     h77fa898_1           conda-forge/linux-64/libgomp-14.2.0-h77fa898_1.conda             
  liblapack         3.9.0      25_linux64_openblas  conda-forge/linux-64/liblapack-3.9.0-25_linux64_openblas.conda   
  libnsl            2.0.1      hd590300_0           conda-forge/linux-64/libnsl-2.0.1-hd590300_0.conda               
  libopenblas       0.3.28     pthreads_h94d23a6_0  conda-forge/linux-64/libopenblas-0.3.28-pthreads_h94d23a6_0.conda
  libsqlite         3.47.0     hadc24fc_0           conda-forge/linux-64/libsqlite-3.47.0-hadc24fc_0.conda           
  libstdcxx         14.2.0     hc0a3c3a_1           conda-forge/linux-64/libstdcxx-14.2.0-hc0a3c3a_1.conda           
  libuuid           2.38.1     h0b41bf4_0           conda-forge/linux-64/libuuid-2.38.1-h0b41bf4_0.conda             
  libxcrypt         4.4.36     hd590300_1           conda-forge/linux-64/libxcrypt-4.4.36-hd590300_1.conda           
  libzlib           1.3.1      hb9d3cd8_2           conda-forge/linux-64/libzlib-1.3.1-hb9d3cd8_2.conda              
  ncurses           6.5        he02047a_1           conda-forge/linux-64/ncurses-6.5-he02047a_1.conda                
  numpy             2.1.2      py312h58c1407_0      conda-forge/linux-64/numpy-2.1.2-py312h58c1407_0.conda           
  openssl           3.3.2      hb9d3cd8_0           conda-forge/linux-64/openssl-3.3.2-hb9d3cd8_0.conda              
  pip               24.2       pyh8b19718_1         conda-forge/noarch/pip-24.2-pyh8b19718_1.conda                   
  python            3.12.7     hc5c86c4_0_cpython   conda-forge/linux-64/python-3.12.7-hc5c86c4_0_cpython.conda      
  python_abi        3.12       5_cp312              conda-forge/linux-64/python_abi-3.12-5_cp312.conda               
  readline          8.2        h8228510_1           conda-forge/linux-64/readline-8.2-h8228510_1.conda               
  scikit-learn      1.5.2      py312h7a48858_1      conda-forge/linux-64/scikit-learn-1.5.2-py312h7a48858_1.conda    
  scipy             1.14.1     py312h62794b6_1      conda-forge/linux-64/scipy-1.14.1-py312h62794b6_1.conda          
  setuptools        75.1.0     pyhd8ed1ab_0         conda-forge/noarch/setuptools-75.1.0-pyhd8ed1ab_0.conda          
  threadpoolctl     3.5.0      pyhc1e730c_0         conda-forge/noarch/threadpoolctl-3.5.0-pyhc1e730c_0.conda        
  tk                8.6.13     noxft_h4845f30_101   conda-forge/linux-64/tk-8.6.13-noxft_h4845f30_101.conda          
  tzdata            2024b      hc8b5060_0           conda-forge/noarch/tzdata-2024b-hc8b5060_0.conda                 
  wheel             0.44.0     pyhd8ed1ab_0         conda-forge/noarch/wheel-0.44.0-pyhd8ed1ab_0.conda               
  xz                5.2.6      h166bdaf_0           conda-forge/linux-64/xz-5.2.6-h166bdaf_0.tar.bz2                 
```

</details>

I saw this segmentation fault locally when running the scikit-learn tests:
```
pytest --pyargs sklearn.metrics.tests
```

which can be reduced a bit more:
```
pytest --pyargs sklearn.metrics.tests.test_classification sklearn.metrics.tests.test_pairwise -k 'classification and nan_valid and scoring0 or test_parallel_pairwise_distances_diagonal and float64' -v
```

If I had to guess the reason that it has not been seen in the CI is that it doesn't test advanced parallelism very thoroughly: in most CI builds there are 2 cores and pytest-xdist is used so neither BLAS nor OpenMP parallelism. We have a CI build without pytest-xdist but since we have two cores, we probably only use BLAS parallelism or OpenMP parallelism, not both at the same time.

COMMENT:
Note that the linux wheel packages from pypi.org still ship openblas 0.3.27 (for numpy) and 0.3.27.dev (for scipy) at the time of writing.

I suppose that once they upgrade to openblas 0.3.28 this problem will impact many more users.

Labels: Bug