Talip7/scikit-learn-issues-multilabel huggingface dataset:
CI MacOS job pylatest_conda_mkl_no_openmp install fails

see https://dev.azure.com/scikit-learn/scikit-learn/_build/results?buildId=40226&view=logs&jobId=e6d5b7c0-0dfd-5ddf-13d5-c71bebf56ce2&j=e6d5b7c0-0dfd-5ddf-13d5-c71bebf56ce2&t=83107d01-18db-5293-bb0f-49ac2bf2f625 for instance

Fails because of
```
CondaVerificationError: The package for python located at /usr/local/miniconda/pkgs/python-3.10.3-hdfd78df_5
appears to be corrupted. The path 'lib/python3.1'
specified in the package manifest cannot be found.
```
This is similar to this old issue https://github.com/conda/conda/issues/10969, but for default channel.

COMMENT:
I restarted the job and it looks like the install is working again.

Labels: Build / CI
CI MacOS job pylatest_conda_mkl_no_openmp install fails

see https://dev.azure.com/scikit-learn/scikit-learn/_build/results?buildId=40226&view=logs&jobId=e6d5b7c0-0dfd-5ddf-13d5-c71bebf56ce2&j=e6d5b7c0-0dfd-5ddf-13d5-c71bebf56ce2&t=83107d01-18db-5293-bb0f-49ac2bf2f625 for instance

Fails because of
```
CondaVerificationError: The package for python located at /usr/local/miniconda/pkgs/python-3.10.3-hdfd78df_5
appears to be corrupted. The path 'lib/python3.1'
specified in the package manifest cannot be found.
```
This is similar to this old issue https://github.com/conda/conda/issues/10969, but for default channel.

COMMENT:
Weird. It failed twice in a row. It also failed in another PR. Let's close but keep an eye open :)

Labels: Build / CI
Why does GridSearchCV cause the computer to overheat when n_job=-1?

In GridsSarchCV, when n_job=-1, the computer overheats and spyder disconnects.
I use TimeSeriesSplit together, but does using TimeSeriesSplit together cause this problem?


`tscv = TimeSeriesSplit(n_splits=3)
lstm = GridSearchCV(estimator=model, param_grid=parameter_space, scoring='r2', n_jobs=-1, cv=tscv)`

COMMENT:
Setting `n_jobs=-1` specifies that `GridSearchCV` should use _all_ available processors. Depending on your hardware and set up this could lead to your system slowing/struggling, and indeed heating up since all your processors will be working on an intensive task. This is especially the case if you have a laptop. 

You should set `n_jobs` to however many processors you want to dedicate to the task instead of setting it to use all via `n_jobs=-1`.

Labels: Needs Triage
Why does GridSearchCV cause the computer to overheat when n_job=-1?

In GridsSarchCV, when n_job=-1, the computer overheats and spyder disconnects.
I use TimeSeriesSplit together, but does using TimeSeriesSplit together cause this problem?


`tscv = TimeSeriesSplit(n_splits=3)
lstm = GridSearchCV(estimator=model, param_grid=parameter_space, scoring='r2', n_jobs=-1, cv=tscv)`

COMMENT:
Thank you for your comment. However, this did not happen when I previously set more parameters for GridSearchCV, and my computer has 48 cores. Is there any other way to configure multiple cores without using n_jobs?

Labels: Needs Triage
Why does GridSearchCV cause the computer to overheat when n_job=-1?

In GridsSarchCV, when n_job=-1, the computer overheats and spyder disconnects.
I use TimeSeriesSplit together, but does using TimeSeriesSplit together cause this problem?


`tscv = TimeSeriesSplit(n_splits=3)
lstm = GridSearchCV(estimator=model, param_grid=parameter_space, scoring='r2', n_jobs=-1, cv=tscv)`

COMMENT:
You can always limit the number of workers to launch with `n_jobs=10` for instance.
Closing the issue since there is not really a bug here.

> this did not happen when I previously set more parameters for GridSearchCV,

You can still provide us the parameters grid, maybe we can help to understand what happen there.

Labels: Needs Triage
Out of Fold score calculation for Cross validation

### Describe the workflow you want to enable

What roughly happens in cross-validation as of now: 

```

# k-fold cross validation
scores = list()
kfold = KFold(n_splits=10, shuffle=True)
# enumerate splits
for train_ix, test_ix in kfold.split(X):
	# get data
	train_X, test_X = X[train_ix], X[test_ix]
	train_y, test_y = y[train_ix], y[test_ix]
	# fit model
	model = KNeighborsClassifier()
	model.fit(train_X, train_y)
	# evaluate model
	yhat = model.predict(test_X)
	acc = accuracy_score(test_y, yhat)
	# store score

        #------------- NOTE THIS ---------------------
	scores.append(acc)
	print('> ', acc)
# summarize model performance

#------------- AND ALSO NOTE THIS ---------------------
mean_s, std_s = mean(scores), std(scores)
print('Mean: %.3f, Standard Deviation: %.3f' % (mean_s, std_s))
```

Now, this for sure satisfies the purpose. But there should be an option to calculate the OOF score as well. (Described in the solution section) 

### Describe your proposed solution

What OOF would do:

```
data_y, data_yhat = list(), list()
kfold = KFold(n_splits=10, shuffle=True)
# enumerate splits
for train_ix, test_ix in kfold.split(X):
	# get data
	train_X, test_X = X[train_ix], X[test_ix]
	train_y, test_y = y[train_ix], y[test_ix]
	# fit model
	model = KNeighborsClassifier()
	model.fit(train_X, train_y)
	# make predictions
	yhat = model.predict(test_X)
	# store
        #------------- NOTE THIS ---------------------
	data_y.extend(test_y)
	data_yhat.extend(yhat)
# evaluate the model

#------------- AND ALSO NOTE THIS ---------------------
acc = accuracy_score(data_y, data_yhat)
print('Accuracy: %.3f' % (acc))
```

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

An article on OOF predictions and score - https://machinelearningmastery.com/out-of-fold-predictions-in-machine-learning/

COMMENT:
@amueller

Labels: New Feature, module:model_selection
Out of Fold score calculation for Cross validation

### Describe the workflow you want to enable

What roughly happens in cross-validation as of now: 

```

# k-fold cross validation
scores = list()
kfold = KFold(n_splits=10, shuffle=True)
# enumerate splits
for train_ix, test_ix in kfold.split(X):
	# get data
	train_X, test_X = X[train_ix], X[test_ix]
	train_y, test_y = y[train_ix], y[test_ix]
	# fit model
	model = KNeighborsClassifier()
	model.fit(train_X, train_y)
	# evaluate model
	yhat = model.predict(test_X)
	acc = accuracy_score(test_y, yhat)
	# store score

        #------------- NOTE THIS ---------------------
	scores.append(acc)
	print('> ', acc)
# summarize model performance

#------------- AND ALSO NOTE THIS ---------------------
mean_s, std_s = mean(scores), std(scores)
print('Mean: %.3f, Standard Deviation: %.3f' % (mean_s, std_s))
```

Now, this for sure satisfies the purpose. But there should be an option to calculate the OOF score as well. (Described in the solution section) 

### Describe your proposed solution

What OOF would do:

```
data_y, data_yhat = list(), list()
kfold = KFold(n_splits=10, shuffle=True)
# enumerate splits
for train_ix, test_ix in kfold.split(X):
	# get data
	train_X, test_X = X[train_ix], X[test_ix]
	train_y, test_y = y[train_ix], y[test_ix]
	# fit model
	model = KNeighborsClassifier()
	model.fit(train_X, train_y)
	# make predictions
	yhat = model.predict(test_X)
	# store
        #------------- NOTE THIS ---------------------
	data_y.extend(test_y)
	data_yhat.extend(yhat)
# evaluate the model

#------------- AND ALSO NOTE THIS ---------------------
acc = accuracy_score(data_y, data_yhat)
print('Accuracy: %.3f' % (acc))
```

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

An article on OOF predictions and score - https://machinelearningmastery.com/out-of-fold-predictions-in-machine-learning/

COMMENT:
Thank you for opening the issue! The request feature can currently be done with `cross_val_predict`:

```python
from sklearn.datasets import make_blobs
from sklearn.neighbors import KNeighborsClassifier
from sklearn.model_selection import cross_val_predict
from sklearn.model_selection import KFold
from sklearn.metrics import accuracy_score

X, y = make_blobs(n_samples=1000, centers=2, n_features=100,
                  cluster_std=20, random_state=0)

knc = KNeighborsClassifier()
cv = KFold(n_splits=10, shuffle=True, random_state=0)

# Here is how to get the out of fold predictions:
y_predict_cv = cross_val_predict(knc, X, y, cv=cv)
print(accuracy_score(y, y_predict_cv))
# 0.933
```

Since this can already be done with using `cross_val_predict`, I am -1 for adding another parameter to `cross_validate` or `cross_val_score`.

Labels: New Feature, module:model_selection
Out of Fold score calculation for Cross validation

### Describe the workflow you want to enable

What roughly happens in cross-validation as of now: 

```

# k-fold cross validation
scores = list()
kfold = KFold(n_splits=10, shuffle=True)
# enumerate splits
for train_ix, test_ix in kfold.split(X):
	# get data
	train_X, test_X = X[train_ix], X[test_ix]
	train_y, test_y = y[train_ix], y[test_ix]
	# fit model
	model = KNeighborsClassifier()
	model.fit(train_X, train_y)
	# evaluate model
	yhat = model.predict(test_X)
	acc = accuracy_score(test_y, yhat)
	# store score

        #------------- NOTE THIS ---------------------
	scores.append(acc)
	print('> ', acc)
# summarize model performance

#------------- AND ALSO NOTE THIS ---------------------
mean_s, std_s = mean(scores), std(scores)
print('Mean: %.3f, Standard Deviation: %.3f' % (mean_s, std_s))
```

Now, this for sure satisfies the purpose. But there should be an option to calculate the OOF score as well. (Described in the solution section) 

### Describe your proposed solution

What OOF would do:

```
data_y, data_yhat = list(), list()
kfold = KFold(n_splits=10, shuffle=True)
# enumerate splits
for train_ix, test_ix in kfold.split(X):
	# get data
	train_X, test_X = X[train_ix], X[test_ix]
	train_y, test_y = y[train_ix], y[test_ix]
	# fit model
	model = KNeighborsClassifier()
	model.fit(train_X, train_y)
	# make predictions
	yhat = model.predict(test_X)
	# store
        #------------- NOTE THIS ---------------------
	data_y.extend(test_y)
	data_yhat.extend(yhat)
# evaluate the model

#------------- AND ALSO NOTE THIS ---------------------
acc = accuracy_score(data_y, data_yhat)
print('Accuracy: %.3f' % (acc))
```

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

An article on OOF predictions and score - https://machinelearningmastery.com/out-of-fold-predictions-in-machine-learning/

COMMENT:
Hi @thomasjpfan 
I was not aware of the `cross_val_predict` function. But thanks to you, now I know.
One small doubt - In the docs, it mentions 
> Passing these predictions into an evaluation metric may not be a valid way to measure generalization performance.

Could you please elaborate on that? And this:

> Results can differ from [cross_validate](https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.cross_validate.html#sklearn.model_selection.cross_validate) and [cross_val_score](https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.cross_val_score.html#sklearn.model_selection.cross_val_score) unless all tests sets have equal size and the metric decomposes over samples.

If you could answer it in context of the above two code snippets that I put, it would be really great.

Labels: New Feature, module:model_selection
Out of Fold score calculation for Cross validation

### Describe the workflow you want to enable

What roughly happens in cross-validation as of now: 

```

# k-fold cross validation
scores = list()
kfold = KFold(n_splits=10, shuffle=True)
# enumerate splits
for train_ix, test_ix in kfold.split(X):
	# get data
	train_X, test_X = X[train_ix], X[test_ix]
	train_y, test_y = y[train_ix], y[test_ix]
	# fit model
	model = KNeighborsClassifier()
	model.fit(train_X, train_y)
	# evaluate model
	yhat = model.predict(test_X)
	acc = accuracy_score(test_y, yhat)
	# store score

        #------------- NOTE THIS ---------------------
	scores.append(acc)
	print('> ', acc)
# summarize model performance

#------------- AND ALSO NOTE THIS ---------------------
mean_s, std_s = mean(scores), std(scores)
print('Mean: %.3f, Standard Deviation: %.3f' % (mean_s, std_s))
```

Now, this for sure satisfies the purpose. But there should be an option to calculate the OOF score as well. (Described in the solution section) 

### Describe your proposed solution

What OOF would do:

```
data_y, data_yhat = list(), list()
kfold = KFold(n_splits=10, shuffle=True)
# enumerate splits
for train_ix, test_ix in kfold.split(X):
	# get data
	train_X, test_X = X[train_ix], X[test_ix]
	train_y, test_y = y[train_ix], y[test_ix]
	# fit model
	model = KNeighborsClassifier()
	model.fit(train_X, train_y)
	# make predictions
	yhat = model.predict(test_X)
	# store
        #------------- NOTE THIS ---------------------
	data_y.extend(test_y)
	data_yhat.extend(yhat)
# evaluate the model

#------------- AND ALSO NOTE THIS ---------------------
acc = accuracy_score(data_y, data_yhat)
print('Accuracy: %.3f' % (acc))
```

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

An article on OOF predictions and score - https://machinelearningmastery.com/out-of-fold-predictions-in-machine-learning/

COMMENT:
> Could you please elaborate on that? And this:

`cross_val_predict` provides a prediction of the size of `X`. However, it comes from the different models trained on a different part of `X`. Therefore, computing the metric between `y_true` and the `y_pred` provided by `cross_val_predict` is not a valid statistical performance estimate.

Labels: New Feature, module:model_selection
Out of Fold score calculation for Cross validation

### Describe the workflow you want to enable

What roughly happens in cross-validation as of now: 

```

# k-fold cross validation
scores = list()
kfold = KFold(n_splits=10, shuffle=True)
# enumerate splits
for train_ix, test_ix in kfold.split(X):
	# get data
	train_X, test_X = X[train_ix], X[test_ix]
	train_y, test_y = y[train_ix], y[test_ix]
	# fit model
	model = KNeighborsClassifier()
	model.fit(train_X, train_y)
	# evaluate model
	yhat = model.predict(test_X)
	acc = accuracy_score(test_y, yhat)
	# store score

        #------------- NOTE THIS ---------------------
	scores.append(acc)
	print('> ', acc)
# summarize model performance

#------------- AND ALSO NOTE THIS ---------------------
mean_s, std_s = mean(scores), std(scores)
print('Mean: %.3f, Standard Deviation: %.3f' % (mean_s, std_s))
```

Now, this for sure satisfies the purpose. But there should be an option to calculate the OOF score as well. (Described in the solution section) 

### Describe your proposed solution

What OOF would do:

```
data_y, data_yhat = list(), list()
kfold = KFold(n_splits=10, shuffle=True)
# enumerate splits
for train_ix, test_ix in kfold.split(X):
	# get data
	train_X, test_X = X[train_ix], X[test_ix]
	train_y, test_y = y[train_ix], y[test_ix]
	# fit model
	model = KNeighborsClassifier()
	model.fit(train_X, train_y)
	# make predictions
	yhat = model.predict(test_X)
	# store
        #------------- NOTE THIS ---------------------
	data_y.extend(test_y)
	data_yhat.extend(yhat)
# evaluate the model

#------------- AND ALSO NOTE THIS ---------------------
acc = accuracy_score(data_y, data_yhat)
print('Accuracy: %.3f' % (acc))
```

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

An article on OOF predictions and score - https://machinelearningmastery.com/out-of-fold-predictions-in-machine-learning/

COMMENT:
> I am -1 for adding another parameter to cross_validate

@thomasjpfan I might be interesting to be able to return the predictions in `cross_validate`. Indeed, it might be useful in plotting displays if we provide the `cv_results` in some cases. However, it should not be returned by default and should as well come with a big warning about when to use it.

Labels: New Feature, module:model_selection
Out of Fold score calculation for Cross validation

### Describe the workflow you want to enable

What roughly happens in cross-validation as of now: 

```

# k-fold cross validation
scores = list()
kfold = KFold(n_splits=10, shuffle=True)
# enumerate splits
for train_ix, test_ix in kfold.split(X):
	# get data
	train_X, test_X = X[train_ix], X[test_ix]
	train_y, test_y = y[train_ix], y[test_ix]
	# fit model
	model = KNeighborsClassifier()
	model.fit(train_X, train_y)
	# evaluate model
	yhat = model.predict(test_X)
	acc = accuracy_score(test_y, yhat)
	# store score

        #------------- NOTE THIS ---------------------
	scores.append(acc)
	print('> ', acc)
# summarize model performance

#------------- AND ALSO NOTE THIS ---------------------
mean_s, std_s = mean(scores), std(scores)
print('Mean: %.3f, Standard Deviation: %.3f' % (mean_s, std_s))
```

Now, this for sure satisfies the purpose. But there should be an option to calculate the OOF score as well. (Described in the solution section) 

### Describe your proposed solution

What OOF would do:

```
data_y, data_yhat = list(), list()
kfold = KFold(n_splits=10, shuffle=True)
# enumerate splits
for train_ix, test_ix in kfold.split(X):
	# get data
	train_X, test_X = X[train_ix], X[test_ix]
	train_y, test_y = y[train_ix], y[test_ix]
	# fit model
	model = KNeighborsClassifier()
	model.fit(train_X, train_y)
	# make predictions
	yhat = model.predict(test_X)
	# store
        #------------- NOTE THIS ---------------------
	data_y.extend(test_y)
	data_yhat.extend(yhat)
# evaluate the model

#------------- AND ALSO NOTE THIS ---------------------
acc = accuracy_score(data_y, data_yhat)
print('Accuracy: %.3f' % (acc))
```

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

An article on OOF predictions and score - https://machinelearningmastery.com/out-of-fold-predictions-in-machine-learning/

COMMENT:
> One small doubt - In the docs, it mentions

`cross_val_predict` does mostly what the second code snippet does. It trains different models on a training set, predicts on a test set, and combines the predictions. Afterwards a single metric is evaluated on the combined predictions.

The documentation says that the method in the second code snippet, which includes using `cross_val_predict` to get generalized error, is not a appropriate method to measure generalized error.

Labels: New Feature, module:model_selection
Out of Fold score calculation for Cross validation

### Describe the workflow you want to enable

What roughly happens in cross-validation as of now: 

```

# k-fold cross validation
scores = list()
kfold = KFold(n_splits=10, shuffle=True)
# enumerate splits
for train_ix, test_ix in kfold.split(X):
	# get data
	train_X, test_X = X[train_ix], X[test_ix]
	train_y, test_y = y[train_ix], y[test_ix]
	# fit model
	model = KNeighborsClassifier()
	model.fit(train_X, train_y)
	# evaluate model
	yhat = model.predict(test_X)
	acc = accuracy_score(test_y, yhat)
	# store score

        #------------- NOTE THIS ---------------------
	scores.append(acc)
	print('> ', acc)
# summarize model performance

#------------- AND ALSO NOTE THIS ---------------------
mean_s, std_s = mean(scores), std(scores)
print('Mean: %.3f, Standard Deviation: %.3f' % (mean_s, std_s))
```

Now, this for sure satisfies the purpose. But there should be an option to calculate the OOF score as well. (Described in the solution section) 

### Describe your proposed solution

What OOF would do:

```
data_y, data_yhat = list(), list()
kfold = KFold(n_splits=10, shuffle=True)
# enumerate splits
for train_ix, test_ix in kfold.split(X):
	# get data
	train_X, test_X = X[train_ix], X[test_ix]
	train_y, test_y = y[train_ix], y[test_ix]
	# fit model
	model = KNeighborsClassifier()
	model.fit(train_X, train_y)
	# make predictions
	yhat = model.predict(test_X)
	# store
        #------------- NOTE THIS ---------------------
	data_y.extend(test_y)
	data_yhat.extend(yhat)
# evaluate the model

#------------- AND ALSO NOTE THIS ---------------------
acc = accuracy_score(data_y, data_yhat)
print('Accuracy: %.3f' % (acc))
```

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

An article on OOF predictions and score - https://machinelearningmastery.com/out-of-fold-predictions-in-machine-learning/

COMMENT:
@glemaitre 
> `cross_val_predict` provides a prediction of the size of `X`. However, it comes from the different models trained on a different part of `X`. Therefore, computing the metric between `y_true` and the `y_pred` provided by `cross_val_predict` is not a valid statistical performance estimate.

You mean "it comes from the ~different~ same models trained on a different part of `X`". And if yes, do you still hold on to the claim that " computing the metric between y_true and the y_pred provided by cross_val_predict is not a valid statistical performance estimate"
Also, what if the different parts of `X` are equal in size? 

I have seen a lot of Kaggle notebooks where they report the OOF score.  So just wanted to clarify things.

Labels: New Feature, module:model_selection
Out of Fold score calculation for Cross validation

### Describe the workflow you want to enable

What roughly happens in cross-validation as of now: 

```

# k-fold cross validation
scores = list()
kfold = KFold(n_splits=10, shuffle=True)
# enumerate splits
for train_ix, test_ix in kfold.split(X):
	# get data
	train_X, test_X = X[train_ix], X[test_ix]
	train_y, test_y = y[train_ix], y[test_ix]
	# fit model
	model = KNeighborsClassifier()
	model.fit(train_X, train_y)
	# evaluate model
	yhat = model.predict(test_X)
	acc = accuracy_score(test_y, yhat)
	# store score

        #------------- NOTE THIS ---------------------
	scores.append(acc)
	print('> ', acc)
# summarize model performance

#------------- AND ALSO NOTE THIS ---------------------
mean_s, std_s = mean(scores), std(scores)
print('Mean: %.3f, Standard Deviation: %.3f' % (mean_s, std_s))
```

Now, this for sure satisfies the purpose. But there should be an option to calculate the OOF score as well. (Described in the solution section) 

### Describe your proposed solution

What OOF would do:

```
data_y, data_yhat = list(), list()
kfold = KFold(n_splits=10, shuffle=True)
# enumerate splits
for train_ix, test_ix in kfold.split(X):
	# get data
	train_X, test_X = X[train_ix], X[test_ix]
	train_y, test_y = y[train_ix], y[test_ix]
	# fit model
	model = KNeighborsClassifier()
	model.fit(train_X, train_y)
	# make predictions
	yhat = model.predict(test_X)
	# store
        #------------- NOTE THIS ---------------------
	data_y.extend(test_y)
	data_yhat.extend(yhat)
# evaluate the model

#------------- AND ALSO NOTE THIS ---------------------
acc = accuracy_score(data_y, data_yhat)
print('Accuracy: %.3f' % (acc))
```

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

An article on OOF predictions and score - https://machinelearningmastery.com/out-of-fold-predictions-in-machine-learning/

COMMENT:
@thomasjpfan @glemaitre

Labels: New Feature, module:model_selection