Talip7/scikit-learn-issues-multilabel huggingface dataset:
V1.5 randomly hangs on import

### Describe the bug

Since upgrading to version 1.5.0 we are seeing issues where the process stalls on import. Through a core dump, it appears to be locking up [here](https://github.com/scikit-learn/scikit-learn/blob/main/sklearn/__init__.py#L154). We run hundreds of processes in parallel and it will hang on only a handful (less than 10).

### Steps/Code to Reproduce

```
import sklearn
```

### Expected Results

it imports

### Actual Results

process hangs indefinitely.

### Versions

```shell
System:
    python: 3.11.3 (main, May 15 2023, 15:45:52) [GCC 11.2.0]
executable: /home/jcoder/git/neo/neo/pyenv/bin/python
   machine: Linux-6.1.87-x86_64-with-glibc2.31

Python dependencies:
      sklearn: 1.5.0
          pip: 24.0
   setuptools: 70.0.0
        numpy: 1.24.4
        scipy: 1.11.1
       Cython: 0.29.35
       pandas: 2.0.2
   matplotlib: 3.7.2
       joblib: 1.3.0
threadpoolctl: 3.1.0

Built with OpenMP: True

threadpoolctl info:
       user_api: blas
   internal_api: openblas
         prefix: libopenblas
       filepath: /opt/anaconda/2023/envs/pandas2/lib/libopenblasp-r0.3.23.so
        version: 0.3.23
threading_layer: pthreads
   architecture: Zen
    num_threads: 64

       user_api: openmp
   internal_api: openmp
         prefix: libgomp
       filepath: /home/~~~~~~/lib/python3.11/site-packages/scikit_learn.libs/libgomp-a34b3233.so.1.0.0
        version: None
    num_threads: 64
```

COMMENT:
I just saw the same issue when importing `sklearn` from a grpc RPC handler thread for the first time. Importing `sklearn` in the main thread first removes the issue in my tests.

Labels: Bug, Needs Reproducible Code
V1.5 randomly hangs on import

### Describe the bug

Since upgrading to version 1.5.0 we are seeing issues where the process stalls on import. Through a core dump, it appears to be locking up [here](https://github.com/scikit-learn/scikit-learn/blob/main/sklearn/__init__.py#L154). We run hundreds of processes in parallel and it will hang on only a handful (less than 10).

### Steps/Code to Reproduce

```
import sklearn
```

### Expected Results

it imports

### Actual Results

process hangs indefinitely.

### Versions

```shell
System:
    python: 3.11.3 (main, May 15 2023, 15:45:52) [GCC 11.2.0]
executable: /home/jcoder/git/neo/neo/pyenv/bin/python
   machine: Linux-6.1.87-x86_64-with-glibc2.31

Python dependencies:
      sklearn: 1.5.0
          pip: 24.0
   setuptools: 70.0.0
        numpy: 1.24.4
        scipy: 1.11.1
       Cython: 0.29.35
       pandas: 2.0.2
   matplotlib: 3.7.2
       joblib: 1.3.0
threadpoolctl: 3.1.0

Built with OpenMP: True

threadpoolctl info:
       user_api: blas
   internal_api: openblas
         prefix: libopenblas
       filepath: /opt/anaconda/2023/envs/pandas2/lib/libopenblasp-r0.3.23.so
        version: 0.3.23
threading_layer: pthreads
   architecture: Zen
    num_threads: 64

       user_api: openmp
   internal_api: openmp
         prefix: libgomp
       filepath: /home/~~~~~~/lib/python3.11/site-packages/scikit_learn.libs/libgomp-a34b3233.so.1.0.0
        version: None
    num_threads: 64
```

COMMENT:
@mabl hard to tell whether this is the same issue without more information, if you could help provide a stand-alone snippet reproducing the issue that would be super helpful :pray:

In case of a hang, using `faulthandler.dump_traceback_later` could provide useful information see https://docs.python.org/3/library/faulthandler.html#dumping-the-tracebacks-after-a-timeout

Labels: Bug, Needs Reproducible Code
V1.5 randomly hangs on import

### Describe the bug

Since upgrading to version 1.5.0 we are seeing issues where the process stalls on import. Through a core dump, it appears to be locking up [here](https://github.com/scikit-learn/scikit-learn/blob/main/sklearn/__init__.py#L154). We run hundreds of processes in parallel and it will hang on only a handful (less than 10).

### Steps/Code to Reproduce

```
import sklearn
```

### Expected Results

it imports

### Actual Results

process hangs indefinitely.

### Versions

```shell
System:
    python: 3.11.3 (main, May 15 2023, 15:45:52) [GCC 11.2.0]
executable: /home/jcoder/git/neo/neo/pyenv/bin/python
   machine: Linux-6.1.87-x86_64-with-glibc2.31

Python dependencies:
      sklearn: 1.5.0
          pip: 24.0
   setuptools: 70.0.0
        numpy: 1.24.4
        scipy: 1.11.1
       Cython: 0.29.35
       pandas: 2.0.2
   matplotlib: 3.7.2
       joblib: 1.3.0
threadpoolctl: 3.1.0

Built with OpenMP: True

threadpoolctl info:
       user_api: blas
   internal_api: openblas
         prefix: libopenblas
       filepath: /opt/anaconda/2023/envs/pandas2/lib/libopenblasp-r0.3.23.so
        version: 0.3.23
threading_layer: pthreads
   architecture: Zen
    num_threads: 64

       user_api: openmp
   internal_api: openmp
         prefix: libgomp
       filepath: /home/~~~~~~/lib/python3.11/site-packages/scikit_learn.libs/libgomp-a34b3233.so.1.0.0
        version: None
    num_threads: 64
```

COMMENT:
I opened #29235 which restores the pre 1.5.0 behavior and should prevent this kind of deadlock. It would be very helpful if you could confirm that this branch solves the issue for your setup.

Labels: Bug, Needs Reproducible Code
V1.5 randomly hangs on import

### Describe the bug

Since upgrading to version 1.5.0 we are seeing issues where the process stalls on import. Through a core dump, it appears to be locking up [here](https://github.com/scikit-learn/scikit-learn/blob/main/sklearn/__init__.py#L154). We run hundreds of processes in parallel and it will hang on only a handful (less than 10).

### Steps/Code to Reproduce

```
import sklearn
```

### Expected Results

it imports

### Actual Results

process hangs indefinitely.

### Versions

```shell
System:
    python: 3.11.3 (main, May 15 2023, 15:45:52) [GCC 11.2.0]
executable: /home/jcoder/git/neo/neo/pyenv/bin/python
   machine: Linux-6.1.87-x86_64-with-glibc2.31

Python dependencies:
      sklearn: 1.5.0
          pip: 24.0
   setuptools: 70.0.0
        numpy: 1.24.4
        scipy: 1.11.1
       Cython: 0.29.35
       pandas: 2.0.2
   matplotlib: 3.7.2
       joblib: 1.3.0
threadpoolctl: 3.1.0

Built with OpenMP: True

threadpoolctl info:
       user_api: blas
   internal_api: openblas
         prefix: libopenblas
       filepath: /opt/anaconda/2023/envs/pandas2/lib/libopenblasp-r0.3.23.so
        version: 0.3.23
threading_layer: pthreads
   architecture: Zen
    num_threads: 64

       user_api: openmp
   internal_api: openmp
         prefix: libgomp
       filepath: /home/~~~~~~/lib/python3.11/site-packages/scikit_learn.libs/libgomp-a34b3233.so.1.0.0
        version: None
    num_threads: 64
```

COMMENT:
Hi @jeremiedbb I did a 
`pip install git+https://github.com/jeremiedbb/scikit-learn@lazy-threadpool-controller` and then ran the tests I have been using to troubleshoot this and everything worked. When running with v1.5.0 the same tests hung on 14 of the 600 processes so in my case, this seems to fix it.

Labels: Bug, Needs Reproducible Code
V1.5 randomly hangs on import

### Describe the bug

Since upgrading to version 1.5.0 we are seeing issues where the process stalls on import. Through a core dump, it appears to be locking up [here](https://github.com/scikit-learn/scikit-learn/blob/main/sklearn/__init__.py#L154). We run hundreds of processes in parallel and it will hang on only a handful (less than 10).

### Steps/Code to Reproduce

```
import sklearn
```

### Expected Results

it imports

### Actual Results

process hangs indefinitely.

### Versions

```shell
System:
    python: 3.11.3 (main, May 15 2023, 15:45:52) [GCC 11.2.0]
executable: /home/jcoder/git/neo/neo/pyenv/bin/python
   machine: Linux-6.1.87-x86_64-with-glibc2.31

Python dependencies:
      sklearn: 1.5.0
          pip: 24.0
   setuptools: 70.0.0
        numpy: 1.24.4
        scipy: 1.11.1
       Cython: 0.29.35
       pandas: 2.0.2
   matplotlib: 3.7.2
       joblib: 1.3.0
threadpoolctl: 3.1.0

Built with OpenMP: True

threadpoolctl info:
       user_api: blas
   internal_api: openblas
         prefix: libopenblas
       filepath: /opt/anaconda/2023/envs/pandas2/lib/libopenblasp-r0.3.23.so
        version: 0.3.23
threading_layer: pthreads
   architecture: Zen
    num_threads: 64

       user_api: openmp
   internal_api: openmp
         prefix: libgomp
       filepath: /home/~~~~~~/lib/python3.11/site-packages/scikit_learn.libs/libgomp-a34b3233.so.1.0.0
        version: None
    num_threads: 64
```

COMMENT:
> I got this error while doing `pip install git+https://github.com/jeremiedbb/scikit-learn@lazy-threadpool-controller`

@HydraDragonAntivirus In principle there is no need to do this anymore, the issue has been fixed in scikit-learn 1.5.1 and you can do `pip install scikit-learn>=1.5.1`.

Labels: Bug, Needs Reproducible Code
V1.5 randomly hangs on import

### Describe the bug

Since upgrading to version 1.5.0 we are seeing issues where the process stalls on import. Through a core dump, it appears to be locking up [here](https://github.com/scikit-learn/scikit-learn/blob/main/sklearn/__init__.py#L154). We run hundreds of processes in parallel and it will hang on only a handful (less than 10).

### Steps/Code to Reproduce

```
import sklearn
```

### Expected Results

it imports

### Actual Results

process hangs indefinitely.

### Versions

```shell
System:
    python: 3.11.3 (main, May 15 2023, 15:45:52) [GCC 11.2.0]
executable: /home/jcoder/git/neo/neo/pyenv/bin/python
   machine: Linux-6.1.87-x86_64-with-glibc2.31

Python dependencies:
      sklearn: 1.5.0
          pip: 24.0
   setuptools: 70.0.0
        numpy: 1.24.4
        scipy: 1.11.1
       Cython: 0.29.35
       pandas: 2.0.2
   matplotlib: 3.7.2
       joblib: 1.3.0
threadpoolctl: 3.1.0

Built with OpenMP: True

threadpoolctl info:
       user_api: blas
   internal_api: openblas
         prefix: libopenblas
       filepath: /opt/anaconda/2023/envs/pandas2/lib/libopenblasp-r0.3.23.so
        version: 0.3.23
threading_layer: pthreads
   architecture: Zen
    num_threads: 64

       user_api: openmp
   internal_api: openmp
         prefix: libgomp
       filepath: /home/~~~~~~/lib/python3.11/site-packages/scikit_learn.libs/libgomp-a34b3233.so.1.0.0
        version: None
    num_threads: 64
```

COMMENT:
> > I got this error while doing `pip install git+https://github.com/jeremiedbb/scikit-learn@lazy-threadpool-controller`
> 
> @HydraDragonAntivirus In principle there is no need to do this anymore, the issue has been fixed in scikit-learn 1.5.1 and you can do `pip install scikit-learn>=1.5.1`.

I tried 1.5.1 and same thing happened to me but if you wait too much time (probably 30mins) it loads.

Labels: Bug, Needs Reproducible Code
V1.5 randomly hangs on import

### Describe the bug

Since upgrading to version 1.5.0 we are seeing issues where the process stalls on import. Through a core dump, it appears to be locking up [here](https://github.com/scikit-learn/scikit-learn/blob/main/sklearn/__init__.py#L154). We run hundreds of processes in parallel and it will hang on only a handful (less than 10).

### Steps/Code to Reproduce

```
import sklearn
```

### Expected Results

it imports

### Actual Results

process hangs indefinitely.

### Versions

```shell
System:
    python: 3.11.3 (main, May 15 2023, 15:45:52) [GCC 11.2.0]
executable: /home/jcoder/git/neo/neo/pyenv/bin/python
   machine: Linux-6.1.87-x86_64-with-glibc2.31

Python dependencies:
      sklearn: 1.5.0
          pip: 24.0
   setuptools: 70.0.0
        numpy: 1.24.4
        scipy: 1.11.1
       Cython: 0.29.35
       pandas: 2.0.2
   matplotlib: 3.7.2
       joblib: 1.3.0
threadpoolctl: 3.1.0

Built with OpenMP: True

threadpoolctl info:
       user_api: blas
   internal_api: openblas
         prefix: libopenblas
       filepath: /opt/anaconda/2023/envs/pandas2/lib/libopenblasp-r0.3.23.so
        version: 0.3.23
threading_layer: pthreads
   architecture: Zen
    num_threads: 64

       user_api: openmp
   internal_api: openmp
         prefix: libgomp
       filepath: /home/~~~~~~/lib/python3.11/site-packages/scikit_learn.libs/libgomp-a34b3233.so.1.0.0
        version: None
    num_threads: 64
```

COMMENT:
Can you open a separate issue please with more details about your problem?

In particular, please include the following information in your new issue:
- is this a regression in scikit-learn 1.5, i.e. did your code work scikit-learn 1.4?
- ideally a stand-alone snippet that reproduces the behaviour on your machine and that others can try to run. Without a stand-alone snippet to reproduce, there is very little we can do to help ...

Labels: Bug, Needs Reproducible Code
FEAT Allow the vector-form representation of symetric distance matrices as input

### Describe the workflow you want to enable

I would like to calculate the upper triangle of a distance from `scipy.spatial.distance.pdist` instead of the redundant and memory intensive version from `sklearn.metrics.pairwise_distances` which has $0.5 * (N^2 - N)$ values and use this as input to `metric="precompute"`

### Describe your proposed solution

```python
from scipy.spatial.distance import pdist
X = # Data
dist = pdist(X, metric="jaccard") # Or any other but I just jaccard a lot for my boolean datasets.  Just didn't want to use euclidean/cosine 
clusterer = HDBSCAN(metric="precomputed_triangle")
clusterer.fit(dist)
```

In addition to the original functionality (or ideally replacing):

```python
from scipy.spatial.distance import pdist, squareform
X = # Data
dist = squareform(pdist(X, metric="jaccard")) # Or any other but I just jaccard a lot for my boolean datasets.  Just didn't want to use euclidean/cosine 
clusterer = HDBSCAN(metric="precomputed")
clusterer.fit(dist)
```

### Describe alternatives you've considered, if relevant

Using the redundant square form but this requires a lot more memory that isn't necessary. 

### Additional context

It may be worthwhile creating a `DistanceMatrix` object like https://scikit.bio/docs/latest/generated/skbio.stats.distance.DistanceMatrix.html#skbio.stats.distance.DistanceMatrix

COMMENT:
This seems like a nice idea. WDYT @jjerphan , @Micky774 , @jeremiedbb ?

Labels: New Feature, Performance, help wanted, Hard
FEAT Allow the vector-form representation of symetric distance matrices as input

### Describe the workflow you want to enable

I would like to calculate the upper triangle of a distance from `scipy.spatial.distance.pdist` instead of the redundant and memory intensive version from `sklearn.metrics.pairwise_distances` which has $0.5 * (N^2 - N)$ values and use this as input to `metric="precompute"`

### Describe your proposed solution

```python
from scipy.spatial.distance import pdist
X = # Data
dist = pdist(X, metric="jaccard") # Or any other but I just jaccard a lot for my boolean datasets.  Just didn't want to use euclidean/cosine 
clusterer = HDBSCAN(metric="precomputed_triangle")
clusterer.fit(dist)
```

In addition to the original functionality (or ideally replacing):

```python
from scipy.spatial.distance import pdist, squareform
X = # Data
dist = squareform(pdist(X, metric="jaccard")) # Or any other but I just jaccard a lot for my boolean datasets.  Just didn't want to use euclidean/cosine 
clusterer = HDBSCAN(metric="precomputed")
clusterer.fit(dist)
```

### Describe alternatives you've considered, if relevant

Using the redundant square form but this requires a lot more memory that isn't necessary. 

### Additional context

It may be worthwhile creating a `DistanceMatrix` object like https://scikit.bio/docs/latest/generated/skbio.stats.distance.DistanceMatrix.html#skbio.stats.distance.DistanceMatrix

COMMENT:
Indeed, this might be a good contribution to introduce as a strategy after #26983 gets in.

I do not think creating a structure like [`skbio.stats.distance.DistanceMatrix`](https://scikit.bio/docs/latest/generated/skbio.stats.distance.DistanceMatrix.html#skbio.stats.distance.DistanceMatrix) would be helpful, at least from a memory footprint perspective as [it seems it stores an entire dense matrix](https://github.com/scikit-bio/scikit-bio/blob/0.6.0/skbio/stats/distance/_base.py#L1043).

We could have a vector-form distance such as the one returned by [`scipy.spatial.distance.squareform`](https://docs.scipy.org/doc/scipy/reference/generated/scipy.spatial.distance.squareform.html) (i.e. which would store $\frac{n (n - 1)}{2}$ elements using a flat 1-D arrays) which would have an interface allowing accessing the distance between $x_i$ and $x_i$, $\forall (i,j) \in [0 .. n-1]^2$.

---

Edit: I actually did not understand your request at first. I just have changed the title of this issue so that it is more consistent. Ideally, we should not materialize this square form representation of the distance matrix or any other ones but have the algorithm support the boolean distance metrics (like Jaccard's) directly.

Out of curiosity, is there something blocking you from using:

```python
X = # Data
clusterer = HDBSCAN(metric="jaccard")
clusterer.fit(X)
```

?

Labels: New Feature, Performance, help wanted, Hard
FEAT Allow the vector-form representation of symetric distance matrices as input

### Describe the workflow you want to enable

I would like to calculate the upper triangle of a distance from `scipy.spatial.distance.pdist` instead of the redundant and memory intensive version from `sklearn.metrics.pairwise_distances` which has $0.5 * (N^2 - N)$ values and use this as input to `metric="precompute"`

### Describe your proposed solution

```python
from scipy.spatial.distance import pdist
X = # Data
dist = pdist(X, metric="jaccard") # Or any other but I just jaccard a lot for my boolean datasets.  Just didn't want to use euclidean/cosine 
clusterer = HDBSCAN(metric="precomputed_triangle")
clusterer.fit(dist)
```

In addition to the original functionality (or ideally replacing):

```python
from scipy.spatial.distance import pdist, squareform
X = # Data
dist = squareform(pdist(X, metric="jaccard")) # Or any other but I just jaccard a lot for my boolean datasets.  Just didn't want to use euclidean/cosine 
clusterer = HDBSCAN(metric="precomputed")
clusterer.fit(dist)
```

### Describe alternatives you've considered, if relevant

Using the redundant square form but this requires a lot more memory that isn't necessary. 

### Additional context

It may be worthwhile creating a `DistanceMatrix` object like https://scikit.bio/docs/latest/generated/skbio.stats.distance.DistanceMatrix.html#skbio.stats.distance.DistanceMatrix

COMMENT:
@jjerphan at the time of writing this I was benchmarking `HDBSCAN` by comparing to some "gold-standard" labels for my dataset which includes >50k observations.  It just seemed like a waste to recompute the distances each time so I wanted to use the precompute option and then I realized that it was using the redundant square form.  When I took a deep dive into the other classes on sklearn, it looked like most required the precomputed distance to be in the redundant square form. 

**Is the square form the redundant form or the 1d non-redundant form?** I thought it was the former but the change in title makes me think otherwise.

Labels: New Feature, Performance, help wanted, Hard
FEAT Allow the vector-form representation of symetric distance matrices as input

### Describe the workflow you want to enable

I would like to calculate the upper triangle of a distance from `scipy.spatial.distance.pdist` instead of the redundant and memory intensive version from `sklearn.metrics.pairwise_distances` which has $0.5 * (N^2 - N)$ values and use this as input to `metric="precompute"`

### Describe your proposed solution

```python
from scipy.spatial.distance import pdist
X = # Data
dist = pdist(X, metric="jaccard") # Or any other but I just jaccard a lot for my boolean datasets.  Just didn't want to use euclidean/cosine 
clusterer = HDBSCAN(metric="precomputed_triangle")
clusterer.fit(dist)
```

In addition to the original functionality (or ideally replacing):

```python
from scipy.spatial.distance import pdist, squareform
X = # Data
dist = squareform(pdist(X, metric="jaccard")) # Or any other but I just jaccard a lot for my boolean datasets.  Just didn't want to use euclidean/cosine 
clusterer = HDBSCAN(metric="precomputed")
clusterer.fit(dist)
```

### Describe alternatives you've considered, if relevant

Using the redundant square form but this requires a lot more memory that isn't necessary. 

### Additional context

It may be worthwhile creating a `DistanceMatrix` object like https://scikit.bio/docs/latest/generated/skbio.stats.distance.DistanceMatrix.html#skbio.stats.distance.DistanceMatrix

COMMENT:
I meant vector-form and adapted the title accordingly again. 

I was confused by [`scipy.spatial.distance.squareform`](https://docs.scipy.org/doc/scipy/reference/generated/scipy.spatial.distance.squareform.html) which converts the one in the other and vice versa.

Labels: New Feature, Performance, help wanted, Hard
FEAT Allow the vector-form representation of symetric distance matrices as input

### Describe the workflow you want to enable

I would like to calculate the upper triangle of a distance from `scipy.spatial.distance.pdist` instead of the redundant and memory intensive version from `sklearn.metrics.pairwise_distances` which has $0.5 * (N^2 - N)$ values and use this as input to `metric="precompute"`

### Describe your proposed solution

```python
from scipy.spatial.distance import pdist
X = # Data
dist = pdist(X, metric="jaccard") # Or any other but I just jaccard a lot for my boolean datasets.  Just didn't want to use euclidean/cosine 
clusterer = HDBSCAN(metric="precomputed_triangle")
clusterer.fit(dist)
```

In addition to the original functionality (or ideally replacing):

```python
from scipy.spatial.distance import pdist, squareform
X = # Data
dist = squareform(pdist(X, metric="jaccard")) # Or any other but I just jaccard a lot for my boolean datasets.  Just didn't want to use euclidean/cosine 
clusterer = HDBSCAN(metric="precomputed")
clusterer.fit(dist)
```

### Describe alternatives you've considered, if relevant

Using the redundant square form but this requires a lot more memory that isn't necessary. 

### Additional context

It may be worthwhile creating a `DistanceMatrix` object like https://scikit.bio/docs/latest/generated/skbio.stats.distance.DistanceMatrix.html#skbio.stats.distance.DistanceMatrix

COMMENT:
This sounds worth doing, however I agree that perhaps the best way forwards is through #26983, and maybe a buffer wrapper so that we can have e.g. `FullDistance` to wrap the vector-form distance, allowing existing infrastructure to treat `FullDistance` as a redundant squareform matrix without materializing extra memory. This would come at the expense of some performance due to indirection, but IMO a slightly lower-performance, lower-memory option here sounds like a good addition. I'm curious what folks think.

Labels: New Feature, Performance, help wanted, Hard