Talip7/scikit-learn-issues-multilabel huggingface dataset:
LinearSVC crashes with no errors when `max_iter` is hit

### Describe the bug

LinearSVC seems to fail with no warnings.
I think  it is when the number of iterations hit `max_iter` .
I've set `verbose=1` and `max_iter=1000`, and the exact moment it printed `iter 1000` the program crashed


### Steps/Code to Reproduce

The data consists of character-ngrams of various short texts by using `CountVectorizer` i.e `CountVectroizer(analyer="char_wb",ngram_range=(3,5), binary = True)`.

A dictionary consisting of the data can be found [here](https://github.com/Jakobhenningjensen/public/blob/main/sklearn_bugs/LinearSVC/linearSVC_data_dict.pkl)

```python
from sklearn.svm import LinearSVC
import pickle

with open("linearSVC_data_dict.pkl","rb") as f:
    data = pickle.load(f)

X_train = data["train"] # sparse matrix
y_train = data["target"]

model = LinearSVC(dual = True)
model.fit(X_train,y_train)
```


### Expected Results

The program does not crash or some error function atleast is raised

### Actual Results

There is no traceback. 
If I run the script directly as `python train.py` the script suddenly is just "done".

Doing it in an interactive window, the python-console just crashes with a non-zero exit status and the message `0xC0000005`

Running the training in Google Vertex I get at SIGSEV(139) error.

Note, I have used a machine with 664 GB RAM, which I assume should be enough. The error occurs if `n_jobs=-1`, `n_jobs=5`, `n_jobs=1` thus I don't think it's a memory issue

### Versions

```shell
System:
    python: 3.10.0 (tags/v3.10.0:b494f59, Oct  4 2021, 19:00:18) [MSC v.1929 64 bit (AMD64)]
executable: C:\Users\Hennings\AppData\Local\pypoetry\Cache\virtualenvs\transtovat-2_ekYx-D-py3.10\Scripts\python.exe
   machine: Windows-10-10.0.22621-SP0
Python dependencies:
      sklearn: 1.2.2
          pip: 22.3.1
   setuptools: 67.7.2
        numpy: 1.24.3
        scipy: 1.10.1
       Cython: None
       pandas: 1.5.3
   matplotlib: 3.7.1
       joblib: 1.2.0
threadpoolctl: 3.1.0
Built with OpenMP: True
threadpoolctl info:
       user_api: blas
   internal_api: openblas
         prefix: libopenblas
       filepath: C:\Users\Hennings\AppData\Local\pypoetry\Cache\virtualenvs\transtovat-2_ekYx-D-py3.10\Lib\site-packages\numpy\.libs\libopenblas64__v0.3.21-gcc_10_3_0.dll
        version: 0.3.21
threading_layer: pthreads
   architecture: SkylakeX
    num_threads: 16
       user_api: blas
   internal_api: openblas
         prefix: libopenblas
       filepath: C:\Users\Hennings\AppData\Local\pypoetry\Cache\virtualenvs\transtovat-2_ekYx-D-py3.10\Lib\site-packages\scipy.libs\libopenblas-802f9ed1179cb9c9b03d67ff79f48187.dll
        version: 0.3.18
threading_layer: pthreads
   architecture: Prescott
    num_threads: 16
       user_api: openmp
   internal_api: openmp
         prefix: vcomp
       filepath: C:\Users\Hennings\AppData\Local\pypoetry\Cache\virtualenvs\transtovat-2_ekYx-D-py3.10\Lib\site-packages\sklearn\.libs\vcomp140.dll
        version: None
    num_threads: 16
```

COMMENT:
I can confirm that using a subset works fine

Labels: Needs Reproducible Code
LinearSVC crashes with no errors when `max_iter` is hit

### Describe the bug

LinearSVC seems to fail with no warnings.
I think  it is when the number of iterations hit `max_iter` .
I've set `verbose=1` and `max_iter=1000`, and the exact moment it printed `iter 1000` the program crashed


### Steps/Code to Reproduce

The data consists of character-ngrams of various short texts by using `CountVectorizer` i.e `CountVectroizer(analyer="char_wb",ngram_range=(3,5), binary = True)`.

A dictionary consisting of the data can be found [here](https://github.com/Jakobhenningjensen/public/blob/main/sklearn_bugs/LinearSVC/linearSVC_data_dict.pkl)

```python
from sklearn.svm import LinearSVC
import pickle

with open("linearSVC_data_dict.pkl","rb") as f:
    data = pickle.load(f)

X_train = data["train"] # sparse matrix
y_train = data["target"]

model = LinearSVC(dual = True)
model.fit(X_train,y_train)
```


### Expected Results

The program does not crash or some error function atleast is raised

### Actual Results

There is no traceback. 
If I run the script directly as `python train.py` the script suddenly is just "done".

Doing it in an interactive window, the python-console just crashes with a non-zero exit status and the message `0xC0000005`

Running the training in Google Vertex I get at SIGSEV(139) error.

Note, I have used a machine with 664 GB RAM, which I assume should be enough. The error occurs if `n_jobs=-1`, `n_jobs=5`, `n_jobs=1` thus I don't think it's a memory issue

### Versions

```shell
System:
    python: 3.10.0 (tags/v3.10.0:b494f59, Oct  4 2021, 19:00:18) [MSC v.1929 64 bit (AMD64)]
executable: C:\Users\Hennings\AppData\Local\pypoetry\Cache\virtualenvs\transtovat-2_ekYx-D-py3.10\Scripts\python.exe
   machine: Windows-10-10.0.22621-SP0
Python dependencies:
      sklearn: 1.2.2
          pip: 22.3.1
   setuptools: 67.7.2
        numpy: 1.24.3
        scipy: 1.10.1
       Cython: None
       pandas: 1.5.3
   matplotlib: 3.7.1
       joblib: 1.2.0
threadpoolctl: 3.1.0
Built with OpenMP: True
threadpoolctl info:
       user_api: blas
   internal_api: openblas
         prefix: libopenblas
       filepath: C:\Users\Hennings\AppData\Local\pypoetry\Cache\virtualenvs\transtovat-2_ekYx-D-py3.10\Lib\site-packages\numpy\.libs\libopenblas64__v0.3.21-gcc_10_3_0.dll
        version: 0.3.21
threading_layer: pthreads
   architecture: SkylakeX
    num_threads: 16
       user_api: blas
   internal_api: openblas
         prefix: libopenblas
       filepath: C:\Users\Hennings\AppData\Local\pypoetry\Cache\virtualenvs\transtovat-2_ekYx-D-py3.10\Lib\site-packages\scipy.libs\libopenblas-802f9ed1179cb9c9b03d67ff79f48187.dll
        version: 0.3.18
threading_layer: pthreads
   architecture: Prescott
    num_threads: 16
       user_api: openmp
   internal_api: openmp
         prefix: vcomp
       filepath: C:\Users\Hennings\AppData\Local\pypoetry\Cache\virtualenvs\transtovat-2_ekYx-D-py3.10\Lib\site-packages\sklearn\.libs\vcomp140.dll
        version: None
    num_threads: 16
```

COMMENT:
Closing as no reproducible provided.

Labels: Needs Reproducible Code
Cross validation error of a Gaussian process with noisy target

### Describe the bug

Hi,

I'm trying to use RandomizedSearchCV with GP with a vector of alpha's. 
It seems that with the cross validation the alpha are not being split into train/test sets because I get the following error:
ValueError: alpha must be a scalar or an array with same number of entries as y. (100 != 80)

Is there any workaround, or am I missing something?

Thanks!

### Steps/Code to Reproduce

```py
import numpy as np
from sklearn.gaussian_process import GaussianProcessRegressor
from sklearn.gaussian_process.kernels import RBF
from sklearn.model_selection import RandomizedSearchCV

X = np.random.rand(100, 2)
y = np.sin(X[:, 0]) + np.cos(X[:, 1])
alpha = 0.1 * np.ones_like(y)

kernel = RBF(length_scale=1.)

param_dist = {
    'kernel__length_scale': (1e-3, 1e3),
}


gp = GaussianProcessRegressor(kernel=kernel, alpha=alpha)
rs = RandomizedSearchCV(gp, param_distributions=param_dist, cv=5, n_iter=50)

rs.fit(X, y)

print("Best hyperparameters: ", rs.best_params_)
print("Best score: ", rs.best_score_)
```

### Expected Results

No error is thrown and estimation is based on the values and target uncertainty

### Actual Results

```py
ValueError: alpha must be a scalar or an array with same number of entries as y. (100 != 80)
```

### Versions

```shell
System:
    python: 3.10.10 | packaged by conda-forge | (main, Mar 24 2023, 20:12:31) [Clang 14.0.6 ]
   machine: macOS-13.3.1-arm64-arm-64bit

Python dependencies:
      sklearn: 1.2.2
          pip: 23.1.2
   setuptools: 67.7.2
        numpy: 1.23.5
        scipy: 1.10.0
       Cython: None
       pandas: 1.5.2
   matplotlib: 3.6.2
       joblib: 1.2.0
threadpoolctl: 3.1.0

Built with OpenMP: True
```

COMMENT:
Yeah, I checked the code, parameters, alpha here, of the estimator are not splited into k-folder form. It is easy to cope this case, but what I wander is there a general way to cope with all estimator with arguments need to be splited?

Labels: Bug, Needs Triage
Cross validation error of a Gaussian process with noisy target

### Describe the bug

Hi,

I'm trying to use RandomizedSearchCV with GP with a vector of alpha's. 
It seems that with the cross validation the alpha are not being split into train/test sets because I get the following error:
ValueError: alpha must be a scalar or an array with same number of entries as y. (100 != 80)

Is there any workaround, or am I missing something?

Thanks!

### Steps/Code to Reproduce

```py
import numpy as np
from sklearn.gaussian_process import GaussianProcessRegressor
from sklearn.gaussian_process.kernels import RBF
from sklearn.model_selection import RandomizedSearchCV

X = np.random.rand(100, 2)
y = np.sin(X[:, 0]) + np.cos(X[:, 1])
alpha = 0.1 * np.ones_like(y)

kernel = RBF(length_scale=1.)

param_dist = {
    'kernel__length_scale': (1e-3, 1e3),
}


gp = GaussianProcessRegressor(kernel=kernel, alpha=alpha)
rs = RandomizedSearchCV(gp, param_distributions=param_dist, cv=5, n_iter=50)

rs.fit(X, y)

print("Best hyperparameters: ", rs.best_params_)
print("Best score: ", rs.best_score_)
```

### Expected Results

No error is thrown and estimation is based on the values and target uncertainty

### Actual Results

```py
ValueError: alpha must be a scalar or an array with same number of entries as y. (100 != 80)
```

### Versions

```shell
System:
    python: 3.10.10 | packaged by conda-forge | (main, Mar 24 2023, 20:12:31) [Clang 14.0.6 ]
   machine: macOS-13.3.1-arm64-arm-64bit

Python dependencies:
      sklearn: 1.2.2
          pip: 23.1.2
   setuptools: 67.7.2
        numpy: 1.23.5
        scipy: 1.10.0
       Cython: None
       pandas: 1.5.2
   matplotlib: 3.6.2
       joblib: 1.2.0
threadpoolctl: 3.1.0

Built with OpenMP: True
```

COMMENT:
That's because while doing a CV search, the data is split into train/test, but your `alpha` is still of length 100.

You can instead pass a single scalar since you're adding a constant there anyway. Once SLEP006 #24027  is merged we could think of a better API for constructor args which are sample aligned (they're metadata in a way).

For now using a scalar works.

Labels: Bug, Needs Triage
Add a facility that allows random forest classifiers to be combined after training

### Describe the workflow you want to enable

In a federated environment, I have federation elements that build private random forest classifiers, which I would like to combine after the fact into a single random forest.  



### Describe your proposed solution

See the "alternatives" section.  

### Describe alternatives you've considered, if relevant

Stacking might suffice as a work-around, although I'd like to avoid that.  

As a throw-away experiment, simply concatenating all the constituent decision tree estimators into a common estimators_ array (and adjusting the count) seems to work superficially, but clearly isn't good practice.   

In addition, this approach can fail, if, say, we try to combine random forest instance #1 which has classes_ of [dog, cat] and forest #2 which has classes_ of [cow, dog, cat].    To address that concern, I looked at forcing the union of all possible classes (over all the forests) into the resultant combined forest, and the underlying trees.   This appears to work at some level, but doesn't handle misshapen oob_decision_function_ which is shaped according n_classes_.    

Another approach to dealing with classes_ heterogeneity is to make sure  each federation forest is exposed to the full gamut of potential classes during training.  (Even then, one worries about the order of the elements found in classes_ : [dog,cat] vs [cat,dog]).    It appears that classes_ is constructed before any bootstrap sampling, so, assuming we can rely on that implementation detail, and we expose each federation to a consistently ordered and specially constructed "gamut" pre-pended to their X, we can (hopefully) expect all forest instances to have identical classes_ with the same elements in the same order.  That, in turn, would allow easier combining of the forests.     Ensuring complete exposure via the "gamut" might also impact accuracy.    (The training "gamut" is a minimal set of X records that produce all possible y categorical values). 
 

### Additional context

R provides a combine() operator.

COMMENT:
For features like this it would be good to find a/several papers or other examples "out in the wild" to see how people are solving this problem. Scikit-learn doesn't really want to be in the position of inventing solutions, instead we like to "follow the consensus".

Do you know of papers (or other sources) describing how to solve this problem? In addition there are fairly stringent requirements for a new thing to be added https://scikit-learn.org/dev/faq.html#selectiveness

Labels: New Feature, module:ensemble, Needs Decision - Include Feature
OPTICS not detecting last data as outlier

This needs investigation.

Thanks for reporting it @yagao7411


### Discussed in https://github.com/scikit-learn/scikit-learn/discussions/26304

<div type='discussions-op-text'>

<sup>Originally posted by **yagao7411** April 30, 2023</sup>
Hi, I am trying to use sklearn.cluster.OPTICS, but found an issue:

I use 2 examples with exactly the same data but different orders. They give different results:

1) 1st example
```py
from sklearn.cluster import OPTICS
import pandas as pd
import numpy as np
X = np.array([[1], [2], [3],[1],[8], [8], [7],
               [100]
              ])

clust  = OPTICS(min_samples=3, metric='euclidean',).fit(X)
clust.labels_
////////////////////////////////////////////
output: array([0, 0, 0, 0, 1, 1, 1, 1])
```

2) 2nd example
```py
from sklearn.cluster import OPTICS
import pandas as pd
import numpy as np
X = np.array([[1], [2], [3],[8], [8], [7],
               [100],[1]
              ])

clust  = OPTICS(min_samples=3, metric='euclidean',).fit(X)
clust.labels_
////////////////////////////////////////////
output:
array([ 0,  0,  0,  1,  1,  1, -1,  0])
```

We can see X has the same data but different orders. The 2nd output is supposed to be correct as [100] should be an outlier.  But oddly, if we change the order of the data, the model gave wrong results.

Can anyone help?

thanks

Ya</div>

COMMENT:
I have reproduced the bug in the dev version and I think the behavior is interesting. I would like to take a closer look at it tomorrow.

Labels: Bug, Moderate, help wanted
OPTICS not detecting last data as outlier

This needs investigation.

Thanks for reporting it @yagao7411


### Discussed in https://github.com/scikit-learn/scikit-learn/discussions/26304

<div type='discussions-op-text'>

<sup>Originally posted by **yagao7411** April 30, 2023</sup>
Hi, I am trying to use sklearn.cluster.OPTICS, but found an issue:

I use 2 examples with exactly the same data but different orders. They give different results:

1) 1st example
```py
from sklearn.cluster import OPTICS
import pandas as pd
import numpy as np
X = np.array([[1], [2], [3],[1],[8], [8], [7],
               [100]
              ])

clust  = OPTICS(min_samples=3, metric='euclidean',).fit(X)
clust.labels_
////////////////////////////////////////////
output: array([0, 0, 0, 0, 1, 1, 1, 1])
```

2) 2nd example
```py
from sklearn.cluster import OPTICS
import pandas as pd
import numpy as np
X = np.array([[1], [2], [3],[8], [8], [7],
               [100],[1]
              ])

clust  = OPTICS(min_samples=3, metric='euclidean',).fit(X)
clust.labels_
////////////////////////////////////////////
output:
array([ 0,  0,  0,  1,  1,  1, -1,  0])
```

We can see X has the same data but different orders. The 2nd output is supposed to be correct as [100] should be an outlier.  But oddly, if we change the order of the data, the model gave wrong results.

Can anyone help?

thanks

Ya</div>

COMMENT:
I think `self.ordering_`,  `self.core_distances_` and `self.reachability_` are identical (after reordering) among two different input. The only thing that breaks the symmetry is `_correct_predecessor` in `_optics.py`

I'd like to double confirm what is `self.predecessor_`. 
From the doc, it is "Point that a sample was reached from, indexed by object order. Seed points have a predecessor of -1."
My understanding is that `self.predecessor_[0]` refers to the predecessor of X[0], which is X[self.predecessor_[0]]

If my understanding is correct,
then
```
p_e = ordering[predecessor_plot[e]]
```
should be changed to

```
p_e = predecessor_plot[e]
```
where `p_e` is "Predecessor of object at position i" according to the paper


because `predecessor_plot[e]` already points to an element in X and it doen't points to an order.

Changing this will give identical results (where 100 won't be considered outliner by either case)
Previously, in two cases, what `p_e` and `ordering[i]` points to in each step of the function is not identical. Now they will be pointing to the same element in X.

Would like to know whether you think this make sense. Then I can continue investigating what's happening here.

Labels: Bug, Moderate, help wanted
OPTICS not detecting last data as outlier

This needs investigation.

Thanks for reporting it @yagao7411


### Discussed in https://github.com/scikit-learn/scikit-learn/discussions/26304

<div type='discussions-op-text'>

<sup>Originally posted by **yagao7411** April 30, 2023</sup>
Hi, I am trying to use sklearn.cluster.OPTICS, but found an issue:

I use 2 examples with exactly the same data but different orders. They give different results:

1) 1st example
```py
from sklearn.cluster import OPTICS
import pandas as pd
import numpy as np
X = np.array([[1], [2], [3],[1],[8], [8], [7],
               [100]
              ])

clust  = OPTICS(min_samples=3, metric='euclidean',).fit(X)
clust.labels_
////////////////////////////////////////////
output: array([0, 0, 0, 0, 1, 1, 1, 1])
```

2) 2nd example
```py
from sklearn.cluster import OPTICS
import pandas as pd
import numpy as np
X = np.array([[1], [2], [3],[8], [8], [7],
               [100],[1]
              ])

clust  = OPTICS(min_samples=3, metric='euclidean',).fit(X)
clust.labels_
////////////////////////////////////////////
output:
array([ 0,  0,  0,  1,  1,  1, -1,  0])
```

We can see X has the same data but different orders. The 2nd output is supposed to be correct as [100] should be an outlier.  But oddly, if we change the order of the data, the model gave wrong results.

Can anyone help?

thanks

Ya</div>

COMMENT:
Hi Steve,

Thank you for the investigation. I think what you said makes sense (I am not familiar with how the codes work though). As the end user of the package, I would be very happy as long as the function gives consistent results (regardless of the data order).

Looking forward to getting the issue resolved.

Thanks again

Ya

Labels: Bug, Moderate, help wanted
OPTICS not detecting last data as outlier

This needs investigation.

Thanks for reporting it @yagao7411


### Discussed in https://github.com/scikit-learn/scikit-learn/discussions/26304

<div type='discussions-op-text'>

<sup>Originally posted by **yagao7411** April 30, 2023</sup>
Hi, I am trying to use sklearn.cluster.OPTICS, but found an issue:

I use 2 examples with exactly the same data but different orders. They give different results:

1) 1st example
```py
from sklearn.cluster import OPTICS
import pandas as pd
import numpy as np
X = np.array([[1], [2], [3],[1],[8], [8], [7],
               [100]
              ])

clust  = OPTICS(min_samples=3, metric='euclidean',).fit(X)
clust.labels_
////////////////////////////////////////////
output: array([0, 0, 0, 0, 1, 1, 1, 1])
```

2) 2nd example
```py
from sklearn.cluster import OPTICS
import pandas as pd
import numpy as np
X = np.array([[1], [2], [3],[8], [8], [7],
               [100],[1]
              ])

clust  = OPTICS(min_samples=3, metric='euclidean',).fit(X)
clust.labels_
////////////////////////////////////////////
output:
array([ 0,  0,  0,  1,  1,  1, -1,  0])
```

We can see X has the same data but different orders. The 2nd output is supposed to be correct as [100] should be an outlier.  But oddly, if we change the order of the data, the model gave wrong results.

Can anyone help?

thanks

Ya</div>

COMMENT:
@stevezhang1999 I remember the issue you mention when I implemented it, but I remember doing some tests where your fix would make them break.

It'd be nice if you could send a PR with your fix, and we do more tests on the PR to see if it fixes the issue. Thanks for the very nice investigation.

Labels: Bug, Moderate, help wanted
OPTICS not detecting last data as outlier

This needs investigation.

Thanks for reporting it @yagao7411


### Discussed in https://github.com/scikit-learn/scikit-learn/discussions/26304

<div type='discussions-op-text'>

<sup>Originally posted by **yagao7411** April 30, 2023</sup>
Hi, I am trying to use sklearn.cluster.OPTICS, but found an issue:

I use 2 examples with exactly the same data but different orders. They give different results:

1) 1st example
```py
from sklearn.cluster import OPTICS
import pandas as pd
import numpy as np
X = np.array([[1], [2], [3],[1],[8], [8], [7],
               [100]
              ])

clust  = OPTICS(min_samples=3, metric='euclidean',).fit(X)
clust.labels_
////////////////////////////////////////////
output: array([0, 0, 0, 0, 1, 1, 1, 1])
```

2) 2nd example
```py
from sklearn.cluster import OPTICS
import pandas as pd
import numpy as np
X = np.array([[1], [2], [3],[8], [8], [7],
               [100],[1]
              ])

clust  = OPTICS(min_samples=3, metric='euclidean',).fit(X)
clust.labels_
////////////////////////////////////////////
output:
array([ 0,  0,  0,  1,  1,  1, -1,  0])
```

We can see X has the same data but different orders. The 2nd output is supposed to be correct as [100] should be an outlier.  But oddly, if we change the order of the data, the model gave wrong results.

Can anyone help?

thanks

Ya</div>

COMMENT:
> @stevezhang1999 I remember the issue you mention when I implemented it, but I remember doing some tests where your fix would make them break.
> 
> It'd be nice if you could send a PR with your fix, and we do more tests on the PR to see if it fixes the issue. Thanks for the very nice investigation.

@adrinjalali Would like to know more about how will the tests break - I'm planning to  doing some tests with shuffling the same dataset. Are there some other things I need to pay attention to?

Labels: Bug, Moderate, help wanted