Pairwise distances (single precision) throwing seg fault on AWS c6i.metal instances

### Describe the bug

## Pairwise distances (single precision) throwing seg fault on AWS c6i.metal instances

### The Issue

Applying pairwise (Euclidean) distances on a matrix of size 5000x5000.

```python
import numpy as np
from sklearn.metrics import pairwise_distances

samples = 5000
features = 5000
x = np.random.RandomState(0).random_sample((samples,features))
x = x.astype(np.float32, copy=False)  # to convert from fl64 -> fl32
dist = pairwise_distances(x, metric='euclidean', n_jobs=-1)
```
The instance c6i.metal has 128 vCPUs.

When `n_jobs=1` the script runs fine.  But when `n_jobs` is larger than 67, I get a warning saying

```
OpenBLAS warning: precompiled NUM_THREADS exceeded, adding auxiliary array for thread metadata.
To avoid this warning, please rebuild your copy of OpenBLAS with a larger NUM_THREADS setting
or set the environment variable OPENBLAS_NUM_THREADS to 128 or lower
```

(for float64, same warning is thrown only when `n_jobs` is larger than 68).

When `n_jobs=-1` (or 128), I get the same warning for float64. But for float32, the script does not exit successfully and throws a segmentation fault along with a larger error message (note that there is no seg fault for double precision float).
```
OpenBLAS warning: precompiled NUM_THREADS exceeded, adding auxiliary array for thread metadata.
To avoid this warning, please rebuild your copy of OpenBLAS with a larger NUM_THREADS setting
or set the environment variable OPENBLAS_NUM_THREADS to 128 or lower
OpenBLAS : Program is Terminated. Because you tried to allocate too many memory regions.
This library was built to support a maximum of 128 threads - either rebuild OpenBLAS
with a larger NUM_THREADS value or set the environment variable OPENBLAS_NUM_THREADS to
a sufficiently small number. This error typically occurs when the software that relies on
OpenBLAS calls BLAS functions from many threads in parallel, or when your computer has more
cpu cores than what OpenBLAS was configured to handle.
```


Setting `OPENBLAS_NUM_THREADS` to 128, as mentioned in the error message has no effect. For very low values of `OPENBLAS_NUM_THREADS`, the execution gets complete but throws the `NUM_THREADS` exceeded message, unless it is set to 1.


### Environment Used

- **Machine**: I am using AWS c6i.metal instances, which use Intel Xeon Platinum 8375 processors.
There are 2 sockets each containing 64 vCPUs. 

- **OS**: Ubuntu 22.04.2 LTS

- **Python**: 3.10.12

`scikit-learn` was installed using pip (v22.0.2)  on a freshly created virtual
envirnonment created using `venv` module.

- **scikit-learn**: 1.3.2

- **numpy**: 1.26.2

- **scipy**: 1.11.4


### How to replicate

Run the script given above on a c6i.metal instance (or some other intel CPU with
larger number of cores maybe?). Before running:

1. Create virtual environment: `python3 -m venv new-env`.
2. Activate the environment: `source ./new-env/bin/activate`.
3. Install scikit-learn: `pip install scikit-learn`.

### Steps/Code to Reproduce

```python
import numpy as np
from sklearn.metrics import pairwise_distances

samples = 5000
features = 5000
x = np.random.RandomState(0).random_sample((samples,features))
x = x.astype(np.float32, copy=False)  # to convert from fl64 -> fl32
dist = pairwise_distances(x, metric='euclidean', n_jobs=-1)
```

### Expected Results

No error, no warnings or segmentation faults.

### Actual Results

```
OpenBLAS warning: precompiled NUM_THREADS exceeded, adding auxiliary array for thread metadata.
To avoid this warning, please rebuild your copy of OpenBLAS with a larger NUM_THREADS setting
or set the environment variable OPENBLAS_NUM_THREADS to 128 or lower
OpenBLAS : Program is Terminated. Because you tried to allocate too many memory regions.
This library was built to support a maximum of 128 threads - either rebuild OpenBLAS
with a larger NUM_THREADS value or set the environment variable OPENBLAS_NUM_THREADS to
a sufficiently small number. This error typically occurs when the software that relies on
OpenBLAS calls BLAS functions from many threads in parallel, or when your computer has more
cpu cores than what OpenBLAS was configured to handle.
Segmentation fault (core dumped)
```

### Versions

```shell
Python dependencies:
      sklearn: 1.3.2
          pip: 22.0.2
   setuptools: 59.6.0
        numpy: 1.26.2
        scipy: 1.11.4
       Cython: None
       pandas: None
   matplotlib: None
       joblib: 1.3.2
threadpoolctl: 3.2.0

Built with OpenMP: True

threadpoolctl info:
       user_api: openmp
   internal_api: openmp
    num_threads: 128
         prefix: libgomp
       filepath: /home/dhanus/Projects/issues-fujitsu/pairwise-distances/new/lib/python3.10/site-packages/scikit_learn.libs/libgomp-a34b3233.so.1.0.0
        version: None

       user_api: blas
   internal_api: openblas
    num_threads: 64
         prefix: libopenblas
       filepath: /home/dhanus/Projects/issues-fujitsu/pairwise-distances/new/lib/python3.10/site-packages/numpy.libs/libopenblas64_p-r0-0cf96a72.3.23.dev.so
        version: 0.3.23.dev
threading_layer: pthreads
   architecture: SkylakeX

       user_api: blas
   internal_api: openblas
    num_threads: 64
         prefix: libopenblas
       filepath: /home/dhanus/Projects/issues-fujitsu/pairwise-distances/new/lib/python3.10/site-packages/scipy.libs/libopenblasp-r0-23e5df77.3.21.dev.so
        version: 0.3.21.dev
threading_layer: pthreads
   architecture: SkylakeX
```

COMMENT:
The issue is that there are 2 nested levels of parallelism here: 1 from scikit-learn and 1 from OpenBLAS. By setting ``n_jobs=-1``, scikit-learn will spawn 128 threads and for each OpenBLAS will try to spawn 128 itself. But the limit on the number of OpenBLAS threads that you see in the warning message is a global limit. That's why setting `OPENBLAS_NUM_THREADS` to any value other than 1 triggers the warning.

When using the euclidean metric (or cosine, or any metric relying on BLAS), I'd suggest to set ``n_jobs=1`` and let the BLAS handle the multithreading.

When using a metric that does not rely on BLAS but computes the distances one pair of vectors at a time, then you can try to increase ``n_jobs``, up to a point where you'll start getting diminishing returns.

We should improve the doc of ``pairwise_distances`` to tell which metrics will trigger oversubscription and which won't.

Labels: Bug