Talip7/scikit-learn-issues-multilabel huggingface dataset:
Fix version warning banner on the stable documentation page

### Describe the issue linked to the documentation

Currently https://scikit-learn.org/stable/index.html shows the version warning banner ("This are the docs for an unstable version").
<img width="1258" alt="Screenshot 2024-05-22 at 09 08 02" src="https://github.com/scikit-learn/scikit-learn/assets/1448859/2b1e7309-b11f-4e92-8b80-1a6b96743d9c">

I think this is happening because we haven't updated https://scikit-learn.org/dev/_static/versions.json which lists all the available versions and declares which is the stable version.

This file is generated by `build_tools/circle/list_versions.py` which should run as part of the CI on `main`. My guess as to why the file hasn't been updated is that we have not merged a PR since releasing v1.5.0. If I run the script locally it generates the correct content for the `.json` file. 



### Suggest a potential alternative/fix

I think the fix is to merge any PR into `main` as this will regenerate the `versions.json` file.

This is because all versions of the documentation read the versions from the same URL, which is based on `/dev/`.

COMMENT:
For the next release we need to figure out how we perform the update of `versions.json` without having to merge a "random" PR.

Labels: Documentation
Fix version warning banner on the stable documentation page

### Describe the issue linked to the documentation

Currently https://scikit-learn.org/stable/index.html shows the version warning banner ("This are the docs for an unstable version").
<img width="1258" alt="Screenshot 2024-05-22 at 09 08 02" src="https://github.com/scikit-learn/scikit-learn/assets/1448859/2b1e7309-b11f-4e92-8b80-1a6b96743d9c">

I think this is happening because we haven't updated https://scikit-learn.org/dev/_static/versions.json which lists all the available versions and declares which is the stable version.

This file is generated by `build_tools/circle/list_versions.py` which should run as part of the CI on `main`. My guess as to why the file hasn't been updated is that we have not merged a PR since releasing v1.5.0. If I run the script locally it generates the correct content for the `.json` file. 



### Suggest a potential alternative/fix

I think the fix is to merge any PR into `main` as this will regenerate the `versions.json` file.

This is because all versions of the documentation read the versions from the same URL, which is based on `/dev/`.

COMMENT:
This seems fixed indeed, thanks for noticing and opening an issue!

Labels: Documentation
Fix version warning banner on the stable documentation page

### Describe the issue linked to the documentation

Currently https://scikit-learn.org/stable/index.html shows the version warning banner ("This are the docs for an unstable version").
<img width="1258" alt="Screenshot 2024-05-22 at 09 08 02" src="https://github.com/scikit-learn/scikit-learn/assets/1448859/2b1e7309-b11f-4e92-8b80-1a6b96743d9c">

I think this is happening because we haven't updated https://scikit-learn.org/dev/_static/versions.json which lists all the available versions and declares which is the stable version.

This file is generated by `build_tools/circle/list_versions.py` which should run as part of the CI on `main`. My guess as to why the file hasn't been updated is that we have not merged a PR since releasing v1.5.0. If I run the script locally it generates the correct content for the `.json` file. 



### Suggest a potential alternative/fix

I think the fix is to merge any PR into `main` as this will regenerate the `versions.json` file.

This is because all versions of the documentation read the versions from the same URL, which is based on `/dev/`.

COMMENT:
cc @Charlie-XIAO

Labels: Documentation
Fix version warning banner on the stable documentation page

### Describe the issue linked to the documentation

Currently https://scikit-learn.org/stable/index.html shows the version warning banner ("This are the docs for an unstable version").
<img width="1258" alt="Screenshot 2024-05-22 at 09 08 02" src="https://github.com/scikit-learn/scikit-learn/assets/1448859/2b1e7309-b11f-4e92-8b80-1a6b96743d9c">

I think this is happening because we haven't updated https://scikit-learn.org/dev/_static/versions.json which lists all the available versions and declares which is the stable version.

This file is generated by `build_tools/circle/list_versions.py` which should run as part of the CI on `main`. My guess as to why the file hasn't been updated is that we have not merged a PR since releasing v1.5.0. If I run the script locally it generates the correct content for the `.json` file. 



### Suggest a potential alternative/fix

I think the fix is to merge any PR into `main` as this will regenerate the `versions.json` file.

This is because all versions of the documentation read the versions from the same URL, which is based on `/dev/`.

COMMENT:
> For the next release we need to figure out how we perform the update of `versions.json` without having to merge a "random" PR.

What about #29077 instead of a "random" PR? If I'm not mistaken updating `SECURITY.md` is the last step of a release process and should be done except for RC. Such a PR should also be quick to review and merge.

Labels: Documentation
Sphinx search summary disappeared from 1.5 website

Looks like the sphinx search summary has gone from 1.5 website. Not crucial, but showing the context of the match is handy to decide which link is more likely to have the information we want when we use the doc search bar.

Maybe due to pydata-sphinx-theme switch, maybe something else ...

More context: https://github.com/scikit-learn/scikit-learn/pull/27797

Screenshot for 1.4 https://scikit-learn.org/1.4/search.html?q=ridge
![image](https://github.com/scikit-learn/scikit-learn/assets/1680079/c83809ef-ff3c-4a4d-b9ad-92ba707d3b2f)

Screenshot for 1.5 https://scikit-learn.org/1.5/search.html?q=ridge
![image](https://github.com/scikit-learn/scikit-learn/assets/1680079/7c0efb7e-3b28-486c-b562-5661e46eda1a)

COMMENT:
+1 for re-enabling the sphinx search result previews with the new theme if possible while waiting for a possible revival of the Algolia alternative discussed at #28478.

/cc @Charlie-XIAO.

Labels: Documentation
Sphinx search summary disappeared from 1.5 website

Looks like the sphinx search summary has gone from 1.5 website. Not crucial, but showing the context of the match is handy to decide which link is more likely to have the information we want when we use the doc search bar.

Maybe due to pydata-sphinx-theme switch, maybe something else ...

More context: https://github.com/scikit-learn/scikit-learn/pull/27797

Screenshot for 1.4 https://scikit-learn.org/1.4/search.html?q=ridge
![image](https://github.com/scikit-learn/scikit-learn/assets/1680079/c83809ef-ff3c-4a4d-b9ad-92ba707d3b2f)

Screenshot for 1.5 https://scikit-learn.org/1.5/search.html?q=ridge
![image](https://github.com/scikit-learn/scikit-learn/assets/1680079/7c0efb7e-3b28-486c-b562-5661e46eda1a)

COMMENT:
This I believe is some upstream bug but I can't remember clearly. Let me try to find it out.

Labels: Documentation
Sphinx search summary disappeared from 1.5 website

Looks like the sphinx search summary has gone from 1.5 website. Not crucial, but showing the context of the match is handy to decide which link is more likely to have the information we want when we use the doc search bar.

Maybe due to pydata-sphinx-theme switch, maybe something else ...

More context: https://github.com/scikit-learn/scikit-learn/pull/27797

Screenshot for 1.4 https://scikit-learn.org/1.4/search.html?q=ridge
![image](https://github.com/scikit-learn/scikit-learn/assets/1680079/c83809ef-ff3c-4a4d-b9ad-92ba707d3b2f)

Screenshot for 1.5 https://scikit-learn.org/1.5/search.html?q=ridge
![image](https://github.com/scikit-learn/scikit-learn/assets/1680079/7c0efb7e-3b28-486c-b562-5661e46eda1a)

COMMENT:
It seems to be because `pydata-sphinx-theme` removes `role="main"` which sphinx search relies on for fetching the search summary. It is restored in `0.15.3rc1` but I'm not sure when `0.15.3` will be finally released (it seems to be blocked by some issue, according to https://github.com/pydata/pydata-sphinx-theme/issues/1797). Should we somehow force the doc build to use `pydata-sphinx-theme==0.15.3rc1`?

Labels: Documentation
Sphinx search summary disappeared from 1.5 website

Looks like the sphinx search summary has gone from 1.5 website. Not crucial, but showing the context of the match is handy to decide which link is more likely to have the information we want when we use the doc search bar.

Maybe due to pydata-sphinx-theme switch, maybe something else ...

More context: https://github.com/scikit-learn/scikit-learn/pull/27797

Screenshot for 1.4 https://scikit-learn.org/1.4/search.html?q=ridge
![image](https://github.com/scikit-learn/scikit-learn/assets/1680079/c83809ef-ff3c-4a4d-b9ad-92ba707d3b2f)

Screenshot for 1.5 https://scikit-learn.org/1.5/search.html?q=ridge
![image](https://github.com/scikit-learn/scikit-learn/assets/1680079/7c0efb7e-3b28-486c-b562-5661e46eda1a)

COMMENT:
I think that we can temporary await for the final release since it should be coming soon. Also, I'll start to look again at the Algolia search engine.

Labels: Documentation
Sphinx search summary disappeared from 1.5 website

Looks like the sphinx search summary has gone from 1.5 website. Not crucial, but showing the context of the match is handy to decide which link is more likely to have the information we want when we use the doc search bar.

Maybe due to pydata-sphinx-theme switch, maybe something else ...

More context: https://github.com/scikit-learn/scikit-learn/pull/27797

Screenshot for 1.4 https://scikit-learn.org/1.4/search.html?q=ridge
![image](https://github.com/scikit-learn/scikit-learn/assets/1680079/c83809ef-ff3c-4a4d-b9ad-92ba707d3b2f)

Screenshot for 1.5 https://scikit-learn.org/1.5/search.html?q=ridge
![image](https://github.com/scikit-learn/scikit-learn/assets/1680079/7c0efb7e-3b28-486c-b562-5661e46eda1a)

COMMENT:
I'm wondering if this is not a problem of configuration because I see that Jupyter has the right output with the latest release 0.15.2:

<img width="1336" alt="image" src="https://github.com/scikit-learn/scikit-learn/assets/7454015/ca6185e3-1cbe-499d-889c-1ab38094cb7d">

Labels: Documentation
Sphinx search summary disappeared from 1.5 website

Looks like the sphinx search summary has gone from 1.5 website. Not crucial, but showing the context of the match is handy to decide which link is more likely to have the information we want when we use the doc search bar.

Maybe due to pydata-sphinx-theme switch, maybe something else ...

More context: https://github.com/scikit-learn/scikit-learn/pull/27797

Screenshot for 1.4 https://scikit-learn.org/1.4/search.html?q=ridge
![image](https://github.com/scikit-learn/scikit-learn/assets/1680079/c83809ef-ff3c-4a4d-b9ad-92ba707d3b2f)

Screenshot for 1.5 https://scikit-learn.org/1.5/search.html?q=ridge
![image](https://github.com/scikit-learn/scikit-learn/assets/1680079/7c0efb7e-3b28-486c-b562-5661e46eda1a)

COMMENT:
Let me try `sphinx==7.2.6` with `pydata-sphinx-theme==0.15.2`. We are currently using `sphinx==7.3.7`; perhaps `sphinx` has changed their way of generating search summary.

Labels: Documentation
Sphinx search summary disappeared from 1.5 website

Looks like the sphinx search summary has gone from 1.5 website. Not crucial, but showing the context of the match is handy to decide which link is more likely to have the information we want when we use the doc search bar.

Maybe due to pydata-sphinx-theme switch, maybe something else ...

More context: https://github.com/scikit-learn/scikit-learn/pull/27797

Screenshot for 1.4 https://scikit-learn.org/1.4/search.html?q=ridge
![image](https://github.com/scikit-learn/scikit-learn/assets/1680079/c83809ef-ff3c-4a4d-b9ad-92ba707d3b2f)

Screenshot for 1.5 https://scikit-learn.org/1.5/search.html?q=ridge
![image](https://github.com/scikit-learn/scikit-learn/assets/1680079/7c0efb7e-3b28-486c-b562-5661e46eda1a)

COMMENT:
I have actually tried `sphinx==7.3.7` with `pydata-sphinx-theme==0.15.3rc1` and at least on my local machine this combination works.

Labels: Documentation
Sphinx search summary disappeared from 1.5 website

Looks like the sphinx search summary has gone from 1.5 website. Not crucial, but showing the context of the match is handy to decide which link is more likely to have the information we want when we use the doc search bar.

Maybe due to pydata-sphinx-theme switch, maybe something else ...

More context: https://github.com/scikit-learn/scikit-learn/pull/27797

Screenshot for 1.4 https://scikit-learn.org/1.4/search.html?q=ridge
![image](https://github.com/scikit-learn/scikit-learn/assets/1680079/c83809ef-ff3c-4a4d-b9ad-92ba707d3b2f)

Screenshot for 1.5 https://scikit-learn.org/1.5/search.html?q=ridge
![image](https://github.com/scikit-learn/scikit-learn/assets/1680079/7c0efb7e-3b28-486c-b562-5661e46eda1a)

COMMENT:
OK so let's just wait.

Labels: Documentation
Sphinx search summary disappeared from 1.5 website

Looks like the sphinx search summary has gone from 1.5 website. Not crucial, but showing the context of the match is handy to decide which link is more likely to have the information we want when we use the doc search bar.

Maybe due to pydata-sphinx-theme switch, maybe something else ...

More context: https://github.com/scikit-learn/scikit-learn/pull/27797

Screenshot for 1.4 https://scikit-learn.org/1.4/search.html?q=ridge
![image](https://github.com/scikit-learn/scikit-learn/assets/1680079/c83809ef-ff3c-4a4d-b9ad-92ba707d3b2f)

Screenshot for 1.5 https://scikit-learn.org/1.5/search.html?q=ridge
![image](https://github.com/scikit-learn/scikit-learn/assets/1680079/7c0efb7e-3b28-486c-b562-5661e46eda1a)

COMMENT:
So strange 🤣

| Working | Link | `sphinx` | `pydata-sphinx-theme` |
| :-----: | :--: | :------: | :-------------------: |
| 👎 | [numpy](https://numpy.org/devdocs/search.html?q=documentation#) | `7.2.6` | `0.15.2` |
| 👎 | [scipy](https://scipy.github.io/devdocs/search.html?q=documentation#) | `7.3.7` | `0.15.2` |
| 👎 | [polars](https://docs.pola.rs/docs/python/dev/search.html?q=documentation#) | `7.2.4` | `0.14.1` |
| 👍 | [pandas](https://pandas.pydata.org/docs/dev/search.html?q=documentation#) | `7.3.7` | `0.14.4` |
| 👍 | [jupyter](https://docs.jupyter.org/en/latest/search.html?q=documentation) | `7.2.6` | `0.15.2` |

But I found a very simple fix: we can overwrite `searchtools.js` by vendoring from `sphinx` and changing every occurrence of `[role="main"]` to `#main-content`. After `pydata-sphinx-theme==0.15.3` is release we can simply deleted this file.

Labels: Documentation
Sphinx search summary disappeared from 1.5 website

Looks like the sphinx search summary has gone from 1.5 website. Not crucial, but showing the context of the match is handy to decide which link is more likely to have the information we want when we use the doc search bar.

Maybe due to pydata-sphinx-theme switch, maybe something else ...

More context: https://github.com/scikit-learn/scikit-learn/pull/27797

Screenshot for 1.4 https://scikit-learn.org/1.4/search.html?q=ridge
![image](https://github.com/scikit-learn/scikit-learn/assets/1680079/c83809ef-ff3c-4a4d-b9ad-92ba707d3b2f)

Screenshot for 1.5 https://scikit-learn.org/1.5/search.html?q=ridge
![image](https://github.com/scikit-learn/scikit-learn/assets/1680079/7c0efb7e-3b28-486c-b562-5661e46eda1a)

COMMENT:
#29090

Labels: Documentation
Sphinx search summary disappeared from 1.5 website

Looks like the sphinx search summary has gone from 1.5 website. Not crucial, but showing the context of the match is handy to decide which link is more likely to have the information we want when we use the doc search bar.

Maybe due to pydata-sphinx-theme switch, maybe something else ...

More context: https://github.com/scikit-learn/scikit-learn/pull/27797

Screenshot for 1.4 https://scikit-learn.org/1.4/search.html?q=ridge
![image](https://github.com/scikit-learn/scikit-learn/assets/1680079/c83809ef-ff3c-4a4d-b9ad-92ba707d3b2f)

Screenshot for 1.5 https://scikit-learn.org/1.5/search.html?q=ridge
![image](https://github.com/scikit-learn/scikit-learn/assets/1680079/7c0efb7e-3b28-486c-b562-5661e46eda1a)

COMMENT:
The version 0.15.3 is available ;)

Labels: Documentation
Broken link at the 1.5.0 release page

### Describe the issue linked to the documentation

At the [release page](https://github.com/scikit-learn/scikit-learn/releases/tag/1.5.0)

> We're happy to announce the 1.5.0 release.
> 
> You can read the release highlights under https://scikit-learn.org/stable/auto_examples/release_highlights/plot_release_highlights_1_5_0.html and the long version of the change log under https://scikit-learn.org/stable/whats_new/v1.5.html
> 
> This version supports Python versions 3.9 to 3.12.

The [first link](https://scikit-learn.org/stable/auto_examples/release_highlights/plot_release_highlights_1_5_0.html) points to a missing page. Maybe it is not online yet

### Suggest a potential alternative/fix

_No response_

COMMENT:
The link is up now! Closing

Labels: Documentation, Needs Triage
GridSearchCV.score: support multiple scoring metrics

### Describe the workflow you want to enable

`GridSearchCV` supports multiple scoring metrics using the `scoring` parameter. However, this only applies to `fit`, not to `score`. I would like to use these same scoring metrics to evaluate on the test set as well.

### Describe your proposed solution

This change would require changing the return type of `score` depending on whether `scoring` is a list of metrics. Other than that, it should be easy to iterate over `scorer_` and compute the test accuracy on the same set of metrics used during scoring.

### Describe alternatives you've considered, if relevant

The only alternative right now is to manually loop over `scorer_` and compute the metrics yourself using `predict` instead of `score`.

### Additional context

As an example, the last line in the following code block should return scores for all `scoring` metrics:
```python
from sklearn.datasets import load_iris
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import GridSearchCV, train_test_split

X, y = load_iris(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(X, y)

estimator = LogisticRegression(n_jobs=-1)
param_grid = {"C": [10, 1.0, 0.1]}
score = ["accuracy", "balanced_accuracy", "precision_macro", "recall_macro", "f1_macro"]
grid_search = GridSearchCV(estimator, param_grid, scoring=score, refit="accuracy")
grid_search.fit(X_train, y_train)
grid_search.score(X_test, y_test)
```

COMMENT:
I'm going to close and redirect you to the recent discussion in https://github.com/scikit-learn/scikit-learn/issues/29065.

Here, you are discussing the API of `GridSearchCV.score` that as the same API than the general `Estimator.score`. Since #29065 is looking at the potential changes to do on the API, proposing an extension of the API by supporting multiple scores would be in the scope of the discussion (whether or not this is accepted).

@adamjstewart Would you mind to make your point in https://github.com/scikit-learn/scikit-learn/issues/29065

Labels: New Feature, Needs Triage
GridSearchCV.score: support multiple scoring metrics

### Describe the workflow you want to enable

`GridSearchCV` supports multiple scoring metrics using the `scoring` parameter. However, this only applies to `fit`, not to `score`. I would like to use these same scoring metrics to evaluate on the test set as well.

### Describe your proposed solution

This change would require changing the return type of `score` depending on whether `scoring` is a list of metrics. Other than that, it should be easy to iterate over `scorer_` and compute the test accuracy on the same set of metrics used during scoring.

### Describe alternatives you've considered, if relevant

The only alternative right now is to manually loop over `scorer_` and compute the metrics yourself using `predict` instead of `score`.

### Additional context

As an example, the last line in the following code block should return scores for all `scoring` metrics:
```python
from sklearn.datasets import load_iris
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import GridSearchCV, train_test_split

X, y = load_iris(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(X, y)

estimator = LogisticRegression(n_jobs=-1)
param_grid = {"C": [10, 1.0, 0.1]}
score = ["accuracy", "balanced_accuracy", "precision_macro", "recall_macro", "f1_macro"]
grid_search = GridSearchCV(estimator, param_grid, scoring=score, refit="accuracy")
grid_search.fit(X_train, y_train)
grid_search.score(X_test, y_test)
```

COMMENT:
I think you commented the wrong issue number.

Labels: New Feature, Needs Triage
GridSearchCV.score: support multiple scoring metrics

### Describe the workflow you want to enable

`GridSearchCV` supports multiple scoring metrics using the `scoring` parameter. However, this only applies to `fit`, not to `score`. I would like to use these same scoring metrics to evaluate on the test set as well.

### Describe your proposed solution

This change would require changing the return type of `score` depending on whether `scoring` is a list of metrics. Other than that, it should be easy to iterate over `scorer_` and compute the test accuracy on the same set of metrics used during scoring.

### Describe alternatives you've considered, if relevant

The only alternative right now is to manually loop over `scorer_` and compute the metrics yourself using `predict` instead of `score`.

### Additional context

As an example, the last line in the following code block should return scores for all `scoring` metrics:
```python
from sklearn.datasets import load_iris
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import GridSearchCV, train_test_split

X, y = load_iris(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(X, y)

estimator = LogisticRegression(n_jobs=-1)
param_grid = {"C": [10, 1.0, 0.1]}
score = ["accuracy", "balanced_accuracy", "precision_macro", "recall_macro", "f1_macro"]
grid_search = GridSearchCV(estimator, param_grid, scoring=score, refit="accuracy")
grid_search.fit(X_train, y_train)
grid_search.score(X_test, y_test)
```

COMMENT:
Indeed, I was referring to https://github.com/scikit-learn/scikit-learn/issues/28995

Labels: New Feature, Needs Triage