Talip7/scikit-learn-issues-multilabel huggingface dataset:
Python 3.11 wheels

### Describe the workflow you want to enable
Installing scikit-learn for Python 3.11

### Describe your proposed solution
Add Python 3.11 `python: 311` to cibuildwheel CI job matrix

### Additional context
Likely you'll want to wait for scipy/scipy#16851 to land

COMMENT:
I have started to work on this on my [fork](https://github.com/cmarmo/scikit-learn/tree/wheel-python-311), to avoid using scikit-learn gh-action time.
- The two MacOS wheels are failing during the build with the same error (see eg the [x86_64 build](https://github.com/cmarmo/scikit-learn/actions/runs/3050969258/jobs/4918659784)): I don't know why, if someone has any idea about that, I would be happy to hear it... :).
- The [windows wheel is failing](https://github.com/cmarmo/scikit-learn/actions/runs/3050969258/jobs/4918658974) during testing with the error
  ```
  manifest for winamd64/python:3.11-windowsservercore not found: manifest unknown: manifest unknown
  ```
  This is because for now only the image [winamd64/python:3.11-rc-windowsservercore](https://hub.docker.com/layers/python/winamd64/python/3.11-rc-windowsservercore/images/sha256-9614329eb16dff52ef92a7ea399d306e97d5c0dac8ad2b95f29e3abfecc61763?context=explore) is available... I think I can manage this.
- The linux build [fails during the tests](https://github.com/cmarmo/scikit-learn/actions/runs/3050969258/jobs/4918659445#step:4:5532): the failing test is `test_grid_search_failing_classifier` in `sklearn/model_selection/tests/test_search.py`.

I have two questions:
- scikit-learn CI does not test against python 3.11, perhaps is time to add 3.11 to the CI?
- should I prioritize the wheel generation and do not test them? Or it is better to make all the tests (or at least those that already pass against previous python version) pass with 3.11?

Thanks!

Labels: New Feature, Build / CI, Packaging
Python 3.11 wheels

### Describe the workflow you want to enable
Installing scikit-learn for Python 3.11

### Describe your proposed solution
Add Python 3.11 `python: 311` to cibuildwheel CI job matrix

### Additional context
Likely you'll want to wait for scipy/scipy#16851 to land

COMMENT:
> scikit-learn CI does not test against python 3.11, perhaps is time to add 3.11 to the CI?

As far as I know, there is no released version of our dependencies that targets 3.11-rc for now. Let's wait for stable releases to be published before bumping up the Python version in our `-latest` CI entries.

I think we should not make development versions part of the default CI jobs running in pull requests to avoid annoying contributors with failures that might come from changes in the unreleased dev branches but we should definitely run our "scipy-dev" scheduled run on a nightly basis on Python dev (using the latest nightly builds of all our dependencies, including CPython itself) if possible.

Labels: New Feature, Build / CI, Packaging
Python 3.11 wheels

### Describe the workflow you want to enable
Installing scikit-learn for Python 3.11

### Describe your proposed solution
Add Python 3.11 `python: 311` to cibuildwheel CI job matrix

### Additional context
Likely you'll want to wait for scipy/scipy#16851 to land

COMMENT:
> should I prioritize the wheel generation and do not test them? Or it is better to make all the tests (or at least those that already pass against previous python version) pass with 3.11?

I would rather not publish wheels with broken tests, even for nightly builds. Let's fix them along the way in dedicated PRs.

Labels: New Feature, Build / CI, Packaging
Python 3.11 wheels

### Describe the workflow you want to enable
Installing scikit-learn for Python 3.11

### Describe your proposed solution
Add Python 3.11 `python: 311` to cibuildwheel CI job matrix

### Additional context
Likely you'll want to wait for scipy/scipy#16851 to land

COMMENT:
For the failing assertion on rank, replacing:

```python
assert ranks[0] <= 2 and ranks[1] <= 2
```

by:

```python
assert ranks[0] <= 2 and ranks[1] <= 2, ranks
```

should give us more information on what's happening and allow us to report an issue upstream if this is a regression in numpy or Python.

Labels: New Feature, Build / CI, Packaging
Python 3.11 wheels

### Describe the workflow you want to enable
Installing scikit-learn for Python 3.11

### Describe your proposed solution
Add Python 3.11 `python: 311` to cibuildwheel CI job matrix

### Additional context
Likely you'll want to wait for scipy/scipy#16851 to land

COMMENT:
@jeremiedbb I am milestoning this issue for 1.2 because we will need to fix the build on 3.11 before releasing. This might even be considered a blocker for the 1.2 release.

Labels: New Feature, Build / CI, Packaging
Python 3.11 wheels

### Describe the workflow you want to enable
Installing scikit-learn for Python 3.11

### Describe your proposed solution
Add Python 3.11 `python: 311` to cibuildwheel CI job matrix

### Additional context
Likely you'll want to wait for scipy/scipy#16851 to land

COMMENT:
I am having an issue with pandas: it is a dependency for the tests, but there are no pandas wheels for python 3.11.
On linux the source is downloaded and compiled, on Windows the source is downloaded but the compilation fails with
```
Error: Microsoft Visual C++ 14.0 or greater is required. Get it with "Microsoft C++ Build Tools": https://visualstudio.microsoft.com/visual-cpp-build-tools/
```
May I suggest to skip tests depending on pandas for now? I don't feel like it is worth it to reinstall a compiler into the window docker image for a border line situation. Also, perhaps if nicely asked, pandas people would be willing to embrace the movement and produce a wheel for python 3.11?

Labels: New Feature, Build / CI, Packaging
Python 3.11 wheels

### Describe the workflow you want to enable
Installing scikit-learn for Python 3.11

### Describe your proposed solution
Add Python 3.11 `python: 311` to cibuildwheel CI job matrix

### Additional context
Likely you'll want to wait for scipy/scipy#16851 to land

COMMENT:
+1 for skipping the tests depending on pandas (or other optional dependencies) on the dev builds while they do not provide nightly builds for CPython dev.

Labels: New Feature, Build / CI, Packaging
Python 3.11 wheels

### Describe the workflow you want to enable
Installing scikit-learn for Python 3.11

### Describe your proposed solution
Add Python 3.11 `python: 311` to cibuildwheel CI job matrix

### Additional context
Likely you'll want to wait for scipy/scipy#16851 to land

COMMENT:
About the other tests failures with numpy / scipy dev, our bot has automatically open this issue to track them: #24424

Labels: New Feature, Build / CI, Packaging
⚠️ CI failed on Linux_Nightly.pylatest_pip_scipy_dev ⚠️

**CI is still failing on [Linux_Nightly.pylatest_pip_scipy_dev](https://dev.azure.com/scikit-learn/scikit-learn/_build/results?buildId=47587&view=logs&j=dfe99b15-50db-5d7b-b1e9-4105c42527cf)** (Oct 13, 2022)
- test_searchcv_raise_warning_with_non_finite_score[GridSearchCV-specialized_params0-True]
- test_searchcv_raise_warning_with_non_finite_score[RandomizedSearchCV-specialized_params1-True]

COMMENT:
One of the issue is a DeprecationWarning because `scipy.misc.face` has moved to `scipy.datasets.face`
https://scipy.github.io/devdocs/reference/generated/scipy.misc.face.html
![image](https://user-images.githubusercontent.com/1680079/189590901-d59d2941-dd20-4f26-8a53-c6bdaa0fe172.png)

I guess we need a `utils.fixes` for this. There may be other issues, not 100% sure.

Labels: Bug
⚠️ CI failed on Linux_Nightly.pylatest_pip_scipy_dev ⚠️

**CI is still failing on [Linux_Nightly.pylatest_pip_scipy_dev](https://dev.azure.com/scikit-learn/scikit-learn/_build/results?buildId=47587&view=logs&j=dfe99b15-50db-5d7b-b1e9-4105c42527cf)** (Oct 13, 2022)
- test_searchcv_raise_warning_with_non_finite_score[GridSearchCV-specialized_params0-True]
- test_searchcv_raise_warning_with_non_finite_score[RandomizedSearchCV-specialized_params1-True]

COMMENT:
@lesteve for `test_grid_search_failing_classifier` could the reason be https://github.com/scipy/scipy/pull/16406 merged after your fix in #24141?

Labels: Bug
⚠️ CI failed on Linux_Nightly.pylatest_pip_scipy_dev ⚠️

**CI is still failing on [Linux_Nightly.pylatest_pip_scipy_dev](https://dev.azure.com/scikit-learn/scikit-learn/_build/results?buildId=47587&view=logs&j=dfe99b15-50db-5d7b-b1e9-4105c42527cf)** (Oct 13, 2022)
- test_searchcv_raise_warning_with_non_finite_score[GridSearchCV-specialized_params0-True]
- test_searchcv_raise_warning_with_non_finite_score[RandomizedSearchCV-specialized_params1-True]

COMMENT:
Yep, it seems like we need to replace NaNs by `np.inf` before calling `scipy.stats.rankdata`.

In scipy 1.10.dev, now `scipy.stats.rankdata` returns only NaNs as soon as there is a single NaN:

```py
In [1]: import numpy as np
   ...: 
   ...: from scipy.stats import rankdata
   ...: 
   ...: print(rankdata([1, 2, np.nan]))
[nan nan nan]
```

Labels: Bug
⚠️ CI failed on Linux_Nightly.pylatest_pip_scipy_dev ⚠️

**CI is still failing on [Linux_Nightly.pylatest_pip_scipy_dev](https://dev.azure.com/scikit-learn/scikit-learn/_build/results?buildId=47587&view=logs&j=dfe99b15-50db-5d7b-b1e9-4105c42527cf)** (Oct 13, 2022)
- test_searchcv_raise_warning_with_non_finite_score[GridSearchCV-specialized_params0-True]
- test_searchcv_raise_warning_with_non_finite_score[RandomizedSearchCV-specialized_params1-True]

COMMENT:
Why not just call `nan_policy="omit"` explicitly instead of the new default `nan_policy="propagate"`?

To support older versions of scipy without the `nan_policy` argument, we could have `sklearn.util.fixes` that implements the previous behavior.

Labels: Bug
⚠️ CI failed on Linux_Nightly.pylatest_pip_scipy_dev ⚠️

**CI is still failing on [Linux_Nightly.pylatest_pip_scipy_dev](https://dev.azure.com/scikit-learn/scikit-learn/_build/results?buildId=47587&view=logs&j=dfe99b15-50db-5d7b-b1e9-4105c42527cf)** (Oct 13, 2022)
- test_searchcv_raise_warning_with_non_finite_score[GridSearchCV-specialized_params0-True]
- test_searchcv_raise_warning_with_non_finite_score[RandomizedSearchCV-specialized_params1-True]

COMMENT:
It feels like the simplest option is a local fix but :shrug:.

I quickly looked and I think `SearchCV` objects is the only place in scikit-learn where `scipy.stats.rankdata` is called with NaNs. The only other place that `rankdata` is used is `sklearn.metrics.label_ranking_average_precision_score` and this uses `check_array` so there won't be any NaNs there.

Note that scipy 1.9 does something different than scipy 1.10dev with either `nan_policy="omit"` or `nan_policy="propagate"`, so we need some special logic to get the previous behaviour (NaNs get maximum rank).

Labels: Bug
⚠️ CI failed on Linux_Nightly.pylatest_pip_scipy_dev ⚠️

**CI is still failing on [Linux_Nightly.pylatest_pip_scipy_dev](https://dev.azure.com/scikit-learn/scikit-learn/_build/results?buildId=47587&view=logs&j=dfe99b15-50db-5d7b-b1e9-4105c42527cf)** (Oct 13, 2022)
- test_searchcv_raise_warning_with_non_finite_score[GridSearchCV-specialized_params0-True]
- test_searchcv_raise_warning_with_non_finite_score[RandomizedSearchCV-specialized_params1-True]

COMMENT:
For linking purposes: https://github.com/scikit-learn/scikit-learn/pull/24543 should fix the remaining issue with `rankdata`.

Labels: Bug
MLPRegressor - Validation score wrongly defined

### Describe the bug

In MLPRegressor, if the option early_stopping is set as True, the model will monitor the loss calculated on the validation set in stead of the training set, using the same loss formulation which is the mean squared error. However, as implemented in the line 719 in the source code:
```
self.validation_scores_.append(self.score(X_val, y_val))
```
The function "score", which returns (to confirm) the coefficient of determination, is used.  This is not correct. It should be something like:
```
self.validation_scores_.append(mean_squared_error(self.predict(X_val), y_val))
```


### Steps/Code to Reproduce

Sorry, I don't have time to write a simple code. But the error is quite clear.

### Expected Results

The validation score must be mean squared error.

### Actual Results

Coefficient of determination

### Versions

```shell
1.1.1
```

COMMENT:
Hi @ducvinh9 , thank you for reporting the issue. 

The coefficient of determination is calculated as `1 - u / v`, where `u` is sum of square errors `((y_true - y_pred) **2).sum()` and `v` is the total sum of squares `((y_true - y_true.mean()) **2).sum()` .

Therefore a larger coefficient of determination is equivalent to smaller mean squared error.

Labels: Bug, Enhancement, Needs Decision, module:neural_network
MLPRegressor - Validation score wrongly defined

### Describe the bug

In MLPRegressor, if the option early_stopping is set as True, the model will monitor the loss calculated on the validation set in stead of the training set, using the same loss formulation which is the mean squared error. However, as implemented in the line 719 in the source code:
```
self.validation_scores_.append(self.score(X_val, y_val))
```
The function "score", which returns (to confirm) the coefficient of determination, is used.  This is not correct. It should be something like:
```
self.validation_scores_.append(mean_squared_error(self.predict(X_val), y_val))
```


### Steps/Code to Reproduce

Sorry, I don't have time to write a simple code. But the error is quite clear.

### Expected Results

The validation score must be mean squared error.

### Actual Results

Coefficient of determination

### Versions

```shell
1.1.1
```

COMMENT:
Hi @MaxwellLZH, 

Thank for your reply. It is true that a larger coefficient of determination is equivalent to smaller mean squared error, but I think we should not use them interchangeably for the the following reasons:

- With neural networks, one wants to model a non linear model. And R-squared is not a valid metric in this case. See the following links for more details: [link1](https://statisticsbyjim.com/regression/r-squared-invalid-nonlinear-regression/), [link2](https://www.ncbi.nlm.nih.gov/pmc/articles/PMC2892436/)

- The mean squared error has a unit which is the square of the unit of the output and the R_squared has no unit. Or the self.tol variable is an absolute value which is compared to the absolute change of the monitored metric. So when one use a validation set, he must rethink about self.tol, which is not intuitive and easy to be falsely ignored.

- In almost all ML courses, one monitors and plots the learning curve and the validation curve in function of the number of epochs on the same plot, with the same axis, and bases on it to conclude on underfitting and overfitting probleme. I do not understand why we must do the other way and make a counter-intuitive change here, and not just use the mean squared error for the validation.

Juste my thinking. MLPRegressor is an awesome tool which performs faster on simple NN models than Tensorflow on CPU. I heard someone says people stop supporting additional features for the MLP module, which is sad.

Labels: Bug, Enhancement, Needs Decision, module:neural_network
MLPRegressor - Validation score wrongly defined

### Describe the bug

In MLPRegressor, if the option early_stopping is set as True, the model will monitor the loss calculated on the validation set in stead of the training set, using the same loss formulation which is the mean squared error. However, as implemented in the line 719 in the source code:
```
self.validation_scores_.append(self.score(X_val, y_val))
```
The function "score", which returns (to confirm) the coefficient of determination, is used.  This is not correct. It should be something like:
```
self.validation_scores_.append(mean_squared_error(self.predict(X_val), y_val))
```


### Steps/Code to Reproduce

Sorry, I don't have time to write a simple code. But the error is quite clear.

### Expected Results

The validation score must be mean squared error.

### Actual Results

Coefficient of determination

### Versions

```shell
1.1.1
```

COMMENT:
I agree that usually, we should monitor the loss instead of the final metric. This is indeed the default in other estimators such as gradient boosting. We could think of adding a `scoring` keyword allowing us to switch to another score and by default, it uses the loss as in `HistGradientBoosting`.

Labels: Bug, Enhancement, Needs Decision, module:neural_network
MLPRegressor - Validation score wrongly defined

### Describe the bug

In MLPRegressor, if the option early_stopping is set as True, the model will monitor the loss calculated on the validation set in stead of the training set, using the same loss formulation which is the mean squared error. However, as implemented in the line 719 in the source code:
```
self.validation_scores_.append(self.score(X_val, y_val))
```
The function "score", which returns (to confirm) the coefficient of determination, is used.  This is not correct. It should be something like:
```
self.validation_scores_.append(mean_squared_error(self.predict(X_val), y_val))
```


### Steps/Code to Reproduce

Sorry, I don't have time to write a simple code. But the error is quite clear.

### Expected Results

The validation score must be mean squared error.

### Actual Results

Coefficient of determination

### Versions

```shell
1.1.1
```

COMMENT:
Note that in scikit-learn we have the naming convention that "score" always mean "higher is better" and "loss" mean that "lower is better" when we speak about performance metrics.

I think our early stopping API is rather poor and not consistent across estimators. One way to make them consistent without introducing too much code duplication would be to go through the future callback API currently being designed by @jeremiedbb in #22000.

Labels: Bug, Enhancement, Needs Decision, module:neural_network
MLPRegressor - Validation score wrongly defined

### Describe the bug

In MLPRegressor, if the option early_stopping is set as True, the model will monitor the loss calculated on the validation set in stead of the training set, using the same loss formulation which is the mean squared error. However, as implemented in the line 719 in the source code:
```
self.validation_scores_.append(self.score(X_val, y_val))
```
The function "score", which returns (to confirm) the coefficient of determination, is used.  This is not correct. It should be something like:
```
self.validation_scores_.append(mean_squared_error(self.predict(X_val), y_val))
```


### Steps/Code to Reproduce

Sorry, I don't have time to write a simple code. But the error is quite clear.

### Expected Results

The validation score must be mean squared error.

### Actual Results

Coefficient of determination

### Versions

```shell
1.1.1
```

COMMENT:
Note that in `HistGradientBoostingClassifier/Regressor` we do allow for custom metrics but we use the negative loss by default in an attribute named `validation_score_`...

This is not very intuitive either (but it's consistent with the scikit-learn loss/score naming convention).

We need to rethink this. The callback API should allow us to both use one specific loss or scoring to decide when to stop but also allow us to compute many metric values both on the training and validation data at the end of each iteration in `fit`.

Labels: Bug, Enhancement, Needs Decision, module:neural_network
RFC bump up dependencies for 1.2

This is an issue to discuss what will be the min versions of our dependencies for the 1.2 release, targeted in november.

Here's the list of our current min versions
```
   current min version  |  latest version
                        |
      python = 3.8      |  3.11 (by then)
                        |
 (pypy) numpy = 1.19.2  |  1.23.2
        numpy = 1.17.3  |  1.23.2
        scipy = 1.3.2   |  1.9.1
       joblib = 1.0.0   |  1.1.0
threadpoolctl = 2.0.0   |  3.1.0
       cython = 0.29.24 |  0.29.32
                        |
       pytest = 5.0.1   |  7.1.3
   matplotlib = 3.1.2   |  3.5.3
 scikit-image = 0.16.2  |  0.19.3
       pandas = 1.0.5   |  1.4.4
      seaborn = 0.9.0   |  0.12.0
```

Following [NEP 29](https://numpy.org/neps/nep-0029-deprecation_policy.html), we should keep support for python 3.8 for this release, and drop support for numpy 1.19.

Looking at the ``fixes`` module, we could benefit from
- threadpoolctl 3.0+ released on october 2021.
- scipy 1.4+, released on december 2019, (related to https://github.com/scikit-learn/scikit-learn/issues/23699)

Please share other needs here

COMMENT:
Hi @jeremiedbb ,
I'd love to contribute for this. I think individual PRs to update each requirement should be the way to go.

Would it be possible to already start or should we wait for the full list?

Labels: RFC, Packaging
RFC bump up dependencies for 1.2

This is an issue to discuss what will be the min versions of our dependencies for the 1.2 release, targeted in november.

Here's the list of our current min versions
```
   current min version  |  latest version
                        |
      python = 3.8      |  3.11 (by then)
                        |
 (pypy) numpy = 1.19.2  |  1.23.2
        numpy = 1.17.3  |  1.23.2
        scipy = 1.3.2   |  1.9.1
       joblib = 1.0.0   |  1.1.0
threadpoolctl = 2.0.0   |  3.1.0
       cython = 0.29.24 |  0.29.32
                        |
       pytest = 5.0.1   |  7.1.3
   matplotlib = 3.1.2   |  3.5.3
 scikit-image = 0.16.2  |  0.19.3
       pandas = 1.0.5   |  1.4.4
      seaborn = 0.9.0   |  0.12.0
```

Following [NEP 29](https://numpy.org/neps/nep-0029-deprecation_policy.html), we should keep support for python 3.8 for this release, and drop support for numpy 1.19.

Looking at the ``fixes`` module, we could benefit from
- threadpoolctl 3.0+ released on october 2021.
- scipy 1.4+, released on december 2019, (related to https://github.com/scikit-learn/scikit-learn/issues/23699)

Please share other needs here

COMMENT:
@Kshitij68 we first need to agree on the min versions. For instance, we don't always strictly follow NEP29.

Labels: RFC, Packaging