Talip7/scikit-learn-issues-multilabel huggingface dataset:
OPTICS not detecting last data as outlier

This needs investigation.

Thanks for reporting it @yagao7411


### Discussed in https://github.com/scikit-learn/scikit-learn/discussions/26304

<div type='discussions-op-text'>

<sup>Originally posted by **yagao7411** April 30, 2023</sup>
Hi, I am trying to use sklearn.cluster.OPTICS, but found an issue:

I use 2 examples with exactly the same data but different orders. They give different results:

1) 1st example
```py
from sklearn.cluster import OPTICS
import pandas as pd
import numpy as np
X = np.array([[1], [2], [3],[1],[8], [8], [7],
               [100]
              ])

clust  = OPTICS(min_samples=3, metric='euclidean',).fit(X)
clust.labels_
////////////////////////////////////////////
output: array([0, 0, 0, 0, 1, 1, 1, 1])
```

2) 2nd example
```py
from sklearn.cluster import OPTICS
import pandas as pd
import numpy as np
X = np.array([[1], [2], [3],[8], [8], [7],
               [100],[1]
              ])

clust  = OPTICS(min_samples=3, metric='euclidean',).fit(X)
clust.labels_
////////////////////////////////////////////
output:
array([ 0,  0,  0,  1,  1,  1, -1,  0])
```

We can see X has the same data but different orders. The 2nd output is supposed to be correct as [100] should be an outlier.  But oddly, if we change the order of the data, the model gave wrong results.

Can anyone help?

thanks

Ya</div>

COMMENT:
@stevezhang1999 the existing tests, as well as the examples which use OPTICS should be a good indicator.

Labels: Bug, Moderate, help wanted
OPTICS not detecting last data as outlier

This needs investigation.

Thanks for reporting it @yagao7411


### Discussed in https://github.com/scikit-learn/scikit-learn/discussions/26304

<div type='discussions-op-text'>

<sup>Originally posted by **yagao7411** April 30, 2023</sup>
Hi, I am trying to use sklearn.cluster.OPTICS, but found an issue:

I use 2 examples with exactly the same data but different orders. They give different results:

1) 1st example
```py
from sklearn.cluster import OPTICS
import pandas as pd
import numpy as np
X = np.array([[1], [2], [3],[1],[8], [8], [7],
               [100]
              ])

clust  = OPTICS(min_samples=3, metric='euclidean',).fit(X)
clust.labels_
////////////////////////////////////////////
output: array([0, 0, 0, 0, 1, 1, 1, 1])
```

2) 2nd example
```py
from sklearn.cluster import OPTICS
import pandas as pd
import numpy as np
X = np.array([[1], [2], [3],[8], [8], [7],
               [100],[1]
              ])

clust  = OPTICS(min_samples=3, metric='euclidean',).fit(X)
clust.labels_
////////////////////////////////////////////
output:
array([ 0,  0,  0,  1,  1,  1, -1,  0])
```

We can see X has the same data but different orders. The 2nd output is supposed to be correct as [100] should be an outlier.  But oddly, if we change the order of the data, the model gave wrong results.

Can anyone help?

thanks

Ya</div>

COMMENT:
@adrinjalali I think the fix won't break existing tests. However, order of data somehow matters in this algorithm (especially when the dataset size is very small) - so we cannot expect that every order will give the same results.

Labels: Bug, Moderate, help wanted
Duality gap computation in covariance.GraphicalLasso yields negative values.

### Describe the bug

The computation of the duality gap in `_dual_gap(emp_cov, precision_, alpha)` of `GraphicalLasso` uses the definition from `Duchi et al., 2012`. 
However, their duality gap is expressed given a _feasible_ dual variable. In the implementation, it is applied to a primal variable that doesn't necessarily satisfy the dual variable's feasibility constraints.
This results in potentially negative values for the duality gap.

The primal problem reads

$$P(\Theta) = \min_\Theta \varphi(\Theta) + \nu(\Theta)$$
with
$$\varphi(\Theta) =  -\log \det (\Theta)  + \langle S, \Theta \rangle \text{ and } \nu(\Theta) =  \Vert \Lambda \odot \Theta \Vert_1$$
The Fenchel-Rockafellar dual problem is then given by

$$D(W) = \min_{W}  \varphi^{\star} (W) + \nu^{\star} (-W)$$
$$= \min_{W} -\log \det(S - W) - p \quad \text{s.t. } \quad \vert W_{ij} \vert < \lambda_{ij}$$
We can then compute the duality gap as follows:
$$G(\Theta, W) = P(\Theta) + D(W)$$
Only for a *feasible* $W$ can we actually take $\theta = (S - W)^{-1}$ and thus simplify the duality gap expression and fall back on the implemented formula:
$$G(\Theta) = \langle S, \Theta \rangle + \Vert \Lambda \odot \Theta \Vert_1 - p$$ 
@mathurinm 

### Steps/Code to Reproduce

```python
import numpy as np
from sklearn.covariance import GraphicalLasso

np.random.seed(0)
X = np.random.randn(100, 50)
emp_cov = X.T @ X / len(X)

clf = GraphicalLasso(verbose=True, tol=1e-20).fit(emp_cov)
```

### Expected Results

The expected results would be positive values for the duality gap at all iterations.

### Actual Results

Here are the duality gaps for the first 4 iterations : 
```
[graphical_lasso] Iteration   0, cost  5.24e+01, dual gap 6.745e-01
[graphical_lasso] Iteration   1, cost  5.24e+01, dual gap -3.808e-05
[graphical_lasso] Iteration   2, cost  5.24e+01, dual gap -4.379e-09
[graphical_lasso] Iteration   3, cost  5.24e+01, dual gap 5.049e-09

```

### Versions

```shell
System:
    python: 3.10.6 (main, Oct 24 2022, 16:07:47) [GCC 11.2.0]
executable: /home/cpouliquen/anaconda3/envs/these-can/bin/python3.10
   machine: Linux-5.4.0-148-generic-x86_64-with-glibc2.27

Python dependencies:
          pip: 22.2.2
   setuptools: 64.0.2
      sklearn: 1.0.2
        numpy: 1.22.3
        scipy: 1.8.1
       Cython: 0.29.33
       pandas: 1.5.1
   matplotlib: 3.5.3
       joblib: 1.2.0
threadpoolctl: 3.1.0

Built with OpenMP: True
```
```

COMMENT:
Alternatively we could just stop the objective function change lower than `tol` (with some scaling related to `y`) as everybody else does, optionally in conjunction on a check on the diff of coefficients between consecutive iterations. It might be the pragmatic solution.

Labels: Bug, module:covariance
SimpleImputer.strategy = 'random'

### Describe the workflow you want to enable

SimpleImputer.strategy = 'random' will randomly choose a non-NaN value.

### Describe your proposed solution

Find a list of non-NaN indices. Randomly pick one index for each NaN.

### Describe alternatives you've considered, if relevant

Pandas:

```
# Impute missing NaN values by randomly selecting a non-NaN value
for column in df.columns:
    nan_indexes = df[df[column].isna()].index
    non_nan_values = df[column].dropna().tolist()
    imputed_values = [random.choice(non_nan_values) for _ in range(len(nan_indexes))]
    df.loc[nan_indexes, column] = imputed_values
```

### Additional context

Let's say you're in the following setting:

GOAL: Iteratively tighten a hyperparameter grid.
* You have a broad hyperparameter grid
* Some hyperparameter choices disable (i.e. make NaN) other hyperparameter choices.
* You do random selection of grid points and get the loss.
* You make _isnan a value, true or false, for params that can be NaN.
* You impute the missing values for NaNs.
* You use decision tree regression to figure out what hyperparameter choices are best.
You tighten your grid iteratively using the above.

However, one downside is that some of the splits will be misleading, if they involved imputed NaNs. (The split on this hyperparameter value could suggest a particular threshold OR that the governing hyperparameter that created this NaN is important.) A way to reduce this confusion is using random choice when imputing.

COMMENT:
Thank you for opening the issue. As stated in our [Triaging guidelines](https://scikit-learn.org/dev/developers/bug_triaging.html#triaging-operations-for-members-of-the-core-and-contributor-experience-teams), I am closing this issue as a duplicate of https://github.com/scikit-learn/scikit-learn/issues/11209. You are welcome to continue in the linked issue.

Labels: New Feature, Needs Triage
Use scipy.stats.yeojohnson PowerTransformer

Inside `PowerTransformer`, we should use [`scipy.stats.yeojohnson`](https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.yeojohnson.html#scipy-stats-yeojohnson) instead of our own implementation.

`scipy.stats.yeojohnson` was release with scipy 1.2.0. With PR #24665, we now have even 1.5.0 as minimum.

Edit: Note that https://github.com/scipy/scipy/issues/18389 to be release in scipy 1.12 will also resolve #23319.

COMMENT:
@lsorber are you interested?

Labels: Moderate, help wanted, module:preprocessing, Refactor
Use scipy.stats.yeojohnson PowerTransformer

Inside `PowerTransformer`, we should use [`scipy.stats.yeojohnson`](https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.yeojohnson.html#scipy-stats-yeojohnson) instead of our own implementation.

`scipy.stats.yeojohnson` was release with scipy 1.2.0. With PR #24665, we now have even 1.5.0 as minimum.

Edit: Note that https://github.com/scipy/scipy/issues/18389 to be release in scipy 1.12 will also resolve #23319.

COMMENT:
Sure, but I'd first like to see the outcome of https://github.com/scipy/scipy/issues/18389 before contributing a PR.

Labels: Moderate, help wanted, module:preprocessing, Refactor
Use scipy.stats.yeojohnson PowerTransformer

Inside `PowerTransformer`, we should use [`scipy.stats.yeojohnson`](https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.yeojohnson.html#scipy-stats-yeojohnson) instead of our own implementation.

`scipy.stats.yeojohnson` was release with scipy 1.2.0. With PR #24665, we now have even 1.5.0 as minimum.

Edit: Note that https://github.com/scipy/scipy/issues/18389 to be release in scipy 1.12 will also resolve #23319.

COMMENT:
Hello, I'm a first timer and almost new to open source,
Can I take up this issue ?

Labels: Moderate, help wanted, module:preprocessing, Refactor
Use scipy.stats.yeojohnson PowerTransformer

Inside `PowerTransformer`, we should use [`scipy.stats.yeojohnson`](https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.yeojohnson.html#scipy-stats-yeojohnson) instead of our own implementation.

`scipy.stats.yeojohnson` was release with scipy 1.2.0. With PR #24665, we now have even 1.5.0 as minimum.

Edit: Note that https://github.com/scipy/scipy/issues/18389 to be release in scipy 1.12 will also resolve #23319.

COMMENT:
@lsorber kind of brought up this issue and already indicated his interest.

Labels: Moderate, help wanted, module:preprocessing, Refactor
Use scipy.stats.yeojohnson PowerTransformer

Inside `PowerTransformer`, we should use [`scipy.stats.yeojohnson`](https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.yeojohnson.html#scipy-stats-yeojohnson) instead of our own implementation.

`scipy.stats.yeojohnson` was release with scipy 1.2.0. With PR #24665, we now have even 1.5.0 as minimum.

Edit: Note that https://github.com/scipy/scipy/issues/18389 to be release in scipy 1.12 will also resolve #23319.

COMMENT:
PR #27818 uses `scipy.stats.yeojohnson` instead of our own implementation in `PowerTransformer`. However, we need to bump the minimum version of scipy from 1.5.0 to 1.9.0 to resolve this issue #14959.

Labels: Moderate, help wanted, module:preprocessing, Refactor
KNeighborsClassifier OpenBLAS warning: precompiled NUM_THREADS exceeded, adding auxiliary array for thread metadata

### Describe the bug

when i run flask app on 56 core cpus,it show this warning and app exit, when i change \site-packages\joblib\externals\loky\backend\context.py can solved

```py
os_cpu_count = min(os.cpu_count() or 1,12)

cpu_count_user = min(_cpu_count_user(os_cpu_count),12)
```

### Steps/Code to Reproduce

```py
#knn is KNeighborsClassifier 
knn= joblib.load('knn.model')
knn.predict(data)
```

### Expected Results

KNeighborsClassifier.predict no bug on 56 cpus or more cpus

### Actual Results

OpenBLAS warning: precompiled NUM_THREADS exceeded, adding auxiliary array for thread metadata; flask app exit,no Error show.

### Versions

```shell
>>> import sklearn
>>> sklearn.show_versions()

System:
    python: 3.10.11 (tags/v3.10.11:7d4cc5a, Apr  5 2023, 00:38:17) [MSC v.1929 64 bit (AMD64)]
executable: E:\Program Files\Python310\python.exe
   machine: Windows-10-10.0.14393-SP0

Python dependencies:
      sklearn: 1.2.2
          pip: 23.1.1
   setuptools: 65.5.0
        numpy: 1.24.3
        scipy: 1.10.1
       Cython: None
       pandas: 1.4.3
   matplotlib: 3.7.1
       joblib: 1.2.0
threadpoolctl: 3.1.0

Built with OpenMP: True

threadpoolctl info:
       user_api: openmp
   internal_api: openmp
         prefix: vcomp
       filepath: E:\Program Files\Python310\Lib\site-packages\sklearn\.libs\vcomp140.dll
        version: None
    num_threads: 112

       user_api: blas
   internal_api: openblas
         prefix: libopenblas
       filepath: E:\Program Files\Python310\Lib\site-packages\numpy\.libs\libopenblas64__v0.3.21-gcc_10_3_0.dll
        version: 0.3.21
threading_layer: pthreads
   architecture: SkylakeX
    num_threads: 24

       user_api: blas
   internal_api: openblas
         prefix: libopenblas
       filepath: E:\Program Files\Python310\Lib\site-packages\scipy.libs\libopenblas-802f9ed1179cb9c9b03d67ff79f48187.dll
        version: 0.3.18
threading_layer: pthreads
   architecture: SkylakeX
    num_threads: 24
```

COMMENT:
I think this is related to a similar issue discussed in 
https://github.com/xianyi/OpenBLAS/issues/3321.

There was a proposal to make it possible to detect the problem here:

https://github.com/xianyi/OpenBLAS/issues/3321#issuecomment-902606930

and a subsequent pull request that was merged in OpenBLAS but I did not have time to follow-up.

It would be great to check if new OpenBLAS versions provide a way to deal with this situation better in scikit-learn.

Labels: Bug, module:neighbors, Needs Reproducible Code, Needs Investigation, upstream bug
KNeighborsClassifier OpenBLAS warning: precompiled NUM_THREADS exceeded, adding auxiliary array for thread metadata

### Describe the bug

when i run flask app on 56 core cpus,it show this warning and app exit, when i change \site-packages\joblib\externals\loky\backend\context.py can solved

```py
os_cpu_count = min(os.cpu_count() or 1,12)

cpu_count_user = min(_cpu_count_user(os_cpu_count),12)
```

### Steps/Code to Reproduce

```py
#knn is KNeighborsClassifier 
knn= joblib.load('knn.model')
knn.predict(data)
```

### Expected Results

KNeighborsClassifier.predict no bug on 56 cpus or more cpus

### Actual Results

OpenBLAS warning: precompiled NUM_THREADS exceeded, adding auxiliary array for thread metadata; flask app exit,no Error show.

### Versions

```shell
>>> import sklearn
>>> sklearn.show_versions()

System:
    python: 3.10.11 (tags/v3.10.11:7d4cc5a, Apr  5 2023, 00:38:17) [MSC v.1929 64 bit (AMD64)]
executable: E:\Program Files\Python310\python.exe
   machine: Windows-10-10.0.14393-SP0

Python dependencies:
      sklearn: 1.2.2
          pip: 23.1.1
   setuptools: 65.5.0
        numpy: 1.24.3
        scipy: 1.10.1
       Cython: None
       pandas: 1.4.3
   matplotlib: 3.7.1
       joblib: 1.2.0
threadpoolctl: 3.1.0

Built with OpenMP: True

threadpoolctl info:
       user_api: openmp
   internal_api: openmp
         prefix: vcomp
       filepath: E:\Program Files\Python310\Lib\site-packages\sklearn\.libs\vcomp140.dll
        version: None
    num_threads: 112

       user_api: blas
   internal_api: openblas
         prefix: libopenblas
       filepath: E:\Program Files\Python310\Lib\site-packages\numpy\.libs\libopenblas64__v0.3.21-gcc_10_3_0.dll
        version: 0.3.21
threading_layer: pthreads
   architecture: SkylakeX
    num_threads: 24

       user_api: blas
   internal_api: openblas
         prefix: libopenblas
       filepath: E:\Program Files\Python310\Lib\site-packages\scipy.libs\libopenblas-802f9ed1179cb9c9b03d67ff79f48187.dll
        version: 0.3.18
threading_layer: pthreads
   architecture: SkylakeX
    num_threads: 24
```

COMMENT:
Note that we should not change `loky.cpu_count` that works as expected but instead change the way we use it in scikit-learn when dealing with mixed BLAS / OpenMP calls, typically in Cython code, probably in https://github.com/scikit-learn/scikit-learn/blob/6392148d80e9f14a9524c137ac5cfa04f2274d48/sklearn/utils/_openmp_helpers.pyx#L21.

Labels: Bug, module:neighbors, Needs Reproducible Code, Needs Investigation, upstream bug
KNeighborsClassifier OpenBLAS warning: precompiled NUM_THREADS exceeded, adding auxiliary array for thread metadata

### Describe the bug

when i run flask app on 56 core cpus,it show this warning and app exit, when i change \site-packages\joblib\externals\loky\backend\context.py can solved

```py
os_cpu_count = min(os.cpu_count() or 1,12)

cpu_count_user = min(_cpu_count_user(os_cpu_count),12)
```

### Steps/Code to Reproduce

```py
#knn is KNeighborsClassifier 
knn= joblib.load('knn.model')
knn.predict(data)
```

### Expected Results

KNeighborsClassifier.predict no bug on 56 cpus or more cpus

### Actual Results

OpenBLAS warning: precompiled NUM_THREADS exceeded, adding auxiliary array for thread metadata; flask app exit,no Error show.

### Versions

```shell
>>> import sklearn
>>> sklearn.show_versions()

System:
    python: 3.10.11 (tags/v3.10.11:7d4cc5a, Apr  5 2023, 00:38:17) [MSC v.1929 64 bit (AMD64)]
executable: E:\Program Files\Python310\python.exe
   machine: Windows-10-10.0.14393-SP0

Python dependencies:
      sklearn: 1.2.2
          pip: 23.1.1
   setuptools: 65.5.0
        numpy: 1.24.3
        scipy: 1.10.1
       Cython: None
       pandas: 1.4.3
   matplotlib: 3.7.1
       joblib: 1.2.0
threadpoolctl: 3.1.0

Built with OpenMP: True

threadpoolctl info:
       user_api: openmp
   internal_api: openmp
         prefix: vcomp
       filepath: E:\Program Files\Python310\Lib\site-packages\sklearn\.libs\vcomp140.dll
        version: None
    num_threads: 112

       user_api: blas
   internal_api: openblas
         prefix: libopenblas
       filepath: E:\Program Files\Python310\Lib\site-packages\numpy\.libs\libopenblas64__v0.3.21-gcc_10_3_0.dll
        version: 0.3.21
threading_layer: pthreads
   architecture: SkylakeX
    num_threads: 24

       user_api: blas
   internal_api: openblas
         prefix: libopenblas
       filepath: E:\Program Files\Python310\Lib\site-packages\scipy.libs\libopenblas-802f9ed1179cb9c9b03d67ff79f48187.dll
        version: 0.3.18
threading_layer: pthreads
   architecture: SkylakeX
    num_threads: 24
```

COMMENT:
Actually, when reading https://github.com/xianyi/OpenBLAS/pull/3352/files, I do not see why the program would exit. It should work instead. This might be a bug in OpenBLAS.

Labels: Bug, module:neighbors, Needs Reproducible Code, Needs Investigation, upstream bug
KNeighborsClassifier OpenBLAS warning: precompiled NUM_THREADS exceeded, adding auxiliary array for thread metadata

### Describe the bug

when i run flask app on 56 core cpus,it show this warning and app exit, when i change \site-packages\joblib\externals\loky\backend\context.py can solved

```py
os_cpu_count = min(os.cpu_count() or 1,12)

cpu_count_user = min(_cpu_count_user(os_cpu_count),12)
```

### Steps/Code to Reproduce

```py
#knn is KNeighborsClassifier 
knn= joblib.load('knn.model')
knn.predict(data)
```

### Expected Results

KNeighborsClassifier.predict no bug on 56 cpus or more cpus

### Actual Results

OpenBLAS warning: precompiled NUM_THREADS exceeded, adding auxiliary array for thread metadata; flask app exit,no Error show.

### Versions

```shell
>>> import sklearn
>>> sklearn.show_versions()

System:
    python: 3.10.11 (tags/v3.10.11:7d4cc5a, Apr  5 2023, 00:38:17) [MSC v.1929 64 bit (AMD64)]
executable: E:\Program Files\Python310\python.exe
   machine: Windows-10-10.0.14393-SP0

Python dependencies:
      sklearn: 1.2.2
          pip: 23.1.1
   setuptools: 65.5.0
        numpy: 1.24.3
        scipy: 1.10.1
       Cython: None
       pandas: 1.4.3
   matplotlib: 3.7.1
       joblib: 1.2.0
threadpoolctl: 3.1.0

Built with OpenMP: True

threadpoolctl info:
       user_api: openmp
   internal_api: openmp
         prefix: vcomp
       filepath: E:\Program Files\Python310\Lib\site-packages\sklearn\.libs\vcomp140.dll
        version: None
    num_threads: 112

       user_api: blas
   internal_api: openblas
         prefix: libopenblas
       filepath: E:\Program Files\Python310\Lib\site-packages\numpy\.libs\libopenblas64__v0.3.21-gcc_10_3_0.dll
        version: 0.3.21
threading_layer: pthreads
   architecture: SkylakeX
    num_threads: 24

       user_api: blas
   internal_api: openblas
         prefix: libopenblas
       filepath: E:\Program Files\Python310\Lib\site-packages\scipy.libs\libopenblas-802f9ed1179cb9c9b03d67ff79f48187.dll
        version: 0.3.18
threading_layer: pthreads
   architecture: SkylakeX
    num_threads: 24
```

COMMENT:
Is the cause for this issue the same as in https://github.com/scikit-learn/scikit-learn/issues/27948?

Labels: Bug, module:neighbors, Needs Reproducible Code, Needs Investigation, upstream bug