Talip7/scikit-learn-issues-multilabel huggingface dataset:
Enhance "Choosing the Right Estimator" Graphic (scikit-learn algorithm cheat sheet)

### Describe the issue linked to the documentation

In its user guide, scikit-learn offers a [Choosing the right estimator](https://scikit-learn.org/stable/machine_learning_map.html) which is an interactive scikit-learn algorithm cheat sheet that is great.


When thinking about new features for [skore](https://github.com/probabl-ai/skore), I thought of enhancing the user guide and have a pedagogical table which, for each estimator, says:
- if it needs to be scaled,
- if it can handle categorical features,
- if it can handle missing data,
- if it holds some randomness (and where / why),
- if it can be paralleled,
- etc (full proper list to be determined).

EDIT:
- The scikit-learn graph / map is great, but not sufficient IMHO because I would like to have, for each estimator, if I need to normalize the data or not, etc -> guidelines for each estimator
- I would like a table that is separate from the map, this is also a cheat sheet but not to appear on the map, maybe at the bottom of the map on the same user guide page

When discussing this with @jeromedockes and @Vincent-Maladiere, they told me about scikit-learn's [estimator tags](https://scikit-learn.org/dev/developers/develop.html#estimator-tags) such as [`is_regressor`](https://scikit-learn.org/dev/modules/generated/sklearn.base.is_regressor.html#sklearn.base.is_regressor). It seems that that knowledge is already partially in the tags.

### Suggest a potential alternative/fix

- Maybe scikit-learn could have a table in the user guide with guidelines for each estimator?
- Maybe scikit-learn could hold more tags? And the table could be built from those tags?

COMMENT:
Also about having doc that uses estimator tags, we kind of already do it in some places e.g.  [Estimators handling NaNs](https://scikit-learn.org/dev/modules/impute.html#estimators-that-handle-nan-values) is actually generated with a [sphinx directive](https://github.com/scikit-learn/scikit-learn/blob/main/doc/sphinxext/allow_nan_estimators.py) using estimator tags.

Labels: Documentation
Enhance "Choosing the Right Estimator" Graphic (scikit-learn algorithm cheat sheet)

### Describe the issue linked to the documentation

In its user guide, scikit-learn offers a [Choosing the right estimator](https://scikit-learn.org/stable/machine_learning_map.html) which is an interactive scikit-learn algorithm cheat sheet that is great.


When thinking about new features for [skore](https://github.com/probabl-ai/skore), I thought of enhancing the user guide and have a pedagogical table which, for each estimator, says:
- if it needs to be scaled,
- if it can handle categorical features,
- if it can handle missing data,
- if it holds some randomness (and where / why),
- if it can be paralleled,
- etc (full proper list to be determined).

EDIT:
- The scikit-learn graph / map is great, but not sufficient IMHO because I would like to have, for each estimator, if I need to normalize the data or not, etc -> guidelines for each estimator
- I would like a table that is separate from the map, this is also a cheat sheet but not to appear on the map, maybe at the bottom of the map on the same user guide page

When discussing this with @jeromedockes and @Vincent-Maladiere, they told me about scikit-learn's [estimator tags](https://scikit-learn.org/dev/developers/develop.html#estimator-tags) such as [`is_regressor`](https://scikit-learn.org/dev/modules/generated/sklearn.base.is_regressor.html#sklearn.base.is_regressor). It seems that that knowledge is already partially in the tags.

### Suggest a potential alternative/fix

- Maybe scikit-learn could have a table in the user guide with guidelines for each estimator?
- Maybe scikit-learn could hold more tags? And the table could be built from those tags?

COMMENT:
Sorry, let me try to clarify my point (I also edited my initial issue):
- The scikit-learn graph / map is great, but not sufficient IMHO because I would like to have, for each estimator, if I need to normalize the data or not, etc -> guidelines for each estimator
- I would like a table that is separate from the map, this is also a cheat sheet but not to appear on the map, maybe at the bottom of the map on the same user guide page
- This table could be generated automatically from tags

Thanks for your feedback! Understood, will try to break down into more digestible tasks and will look into your provided links

Labels: Documentation
Revisit the "chance level" for the different displays

@e-pet commented on different PRs & issues some interesting fact. I take the opportunity to consolidate some of those comments here.

First, we use the term "chance" that is ambiguous depending of the displays. The term "baseline" would probably be better. In addition, I checked and I think we should make an extra effort on the definition of the baseline for each of the type of plot: for ROC curve, the baseline is "a random classifier assigning the positive class with probability p and the negative class with probability 1 − p" [1] while for the PR curve, the baseline is derived from the "always-positive classifier" where any recall or precision under π should be discarded [1].

It leads to a second where in the PR curve, we plot the horizontal line derived from the always-positive classifier but we don't discard when recall < π. In this case, as mentioned by @e-pet, it might make sense to show the hyperbolic line of the always-positive classifier instead (cf. Fig. 2 in [1]).

@e-pet feel free to add any other points that you wanted to discuss. Here, I wanted to focus on the one that looks critical and could be addressed.

[1] [Flach, P., & Kull, M. (2015). Precision-recall-gain curves: PR analysis done right. Advances in neural information processing systems, 28.](https://papers.nips.cc/paper_files/paper/2015/file/33e8075e9970de0cfea955afd4644bb2-Paper.pdf)

COMMENT:
For the ROC curve, what we currently call "chance level" (the diagonal ROC curve) is any non-informative baseline/predictor: a predictor whose predictions do not depend on X. They can be any constant predict_proba predictions: 0, 1, the fraction of the majority class in the training set or any arbitrary constant prediction. Or even a classifier that outputs a random predicted probability at each test point would also have a ROC Curve lying on the diagonal in the limit of a large prediction set.

Personally, I don't like the "chance-level" naming because it is a bit fuzzy, and it's not intuitive how a classifier that constantly predicts 1 or 0 can be related to "chance". I would rather name this "non-informative predictor" or "non-informative baseline" (or event "constant predictor").

Labels: Documentation, API
Revisit the "chance level" for the different displays

@e-pet commented on different PRs & issues some interesting fact. I take the opportunity to consolidate some of those comments here.

First, we use the term "chance" that is ambiguous depending of the displays. The term "baseline" would probably be better. In addition, I checked and I think we should make an extra effort on the definition of the baseline for each of the type of plot: for ROC curve, the baseline is "a random classifier assigning the positive class with probability p and the negative class with probability 1 − p" [1] while for the PR curve, the baseline is derived from the "always-positive classifier" where any recall or precision under π should be discarded [1].

It leads to a second where in the PR curve, we plot the horizontal line derived from the always-positive classifier but we don't discard when recall < π. In this case, as mentioned by @e-pet, it might make sense to show the hyperbolic line of the always-positive classifier instead (cf. Fig. 2 in [1]).

@e-pet feel free to add any other points that you wanted to discuss. Here, I wanted to focus on the one that looks critical and could be addressed.

[1] [Flach, P., & Kull, M. (2015). Precision-recall-gain curves: PR analysis done right. Advances in neural information processing systems, 28.](https://papers.nips.cc/paper_files/paper/2015/file/33e8075e9970de0cfea955afd4644bb2-Paper.pdf)

COMMENT:
I’m taking this on and already have a working patch locally - will open a PR shortly.

Labels: Documentation, API
Revisit the "chance level" for the different displays

@e-pet commented on different PRs & issues some interesting fact. I take the opportunity to consolidate some of those comments here.

First, we use the term "chance" that is ambiguous depending of the displays. The term "baseline" would probably be better. In addition, I checked and I think we should make an extra effort on the definition of the baseline for each of the type of plot: for ROC curve, the baseline is "a random classifier assigning the positive class with probability p and the negative class with probability 1 − p" [1] while for the PR curve, the baseline is derived from the "always-positive classifier" where any recall or precision under π should be discarded [1].

It leads to a second where in the PR curve, we plot the horizontal line derived from the always-positive classifier but we don't discard when recall < π. In this case, as mentioned by @e-pet, it might make sense to show the hyperbolic line of the always-positive classifier instead (cf. Fig. 2 in [1]).

@e-pet feel free to add any other points that you wanted to discuss. Here, I wanted to focus on the one that looks critical and could be addressed.

[1] [Flach, P., & Kull, M. (2015). Precision-recall-gain curves: PR analysis done right. Advances in neural information processing systems, 28.](https://papers.nips.cc/paper_files/paper/2015/file/33e8075e9970de0cfea955afd4644bb2-Paper.pdf)

COMMENT:
Hi! I'm Nithish Reddy, an aspiring contributor interested in ML and Data Engineering. I'm exploring this issue and would love to understand and contribute more. I've already cloned the repo and gone through the linked paper by Flach & Kull. Looking forward to helping on this or similar tasks. 🙌

GitHub: https://github.com/nithish-74

Labels: Documentation, API
Revisit the "chance level" for the different displays

@e-pet commented on different PRs & issues some interesting fact. I take the opportunity to consolidate some of those comments here.

First, we use the term "chance" that is ambiguous depending of the displays. The term "baseline" would probably be better. In addition, I checked and I think we should make an extra effort on the definition of the baseline for each of the type of plot: for ROC curve, the baseline is "a random classifier assigning the positive class with probability p and the negative class with probability 1 − p" [1] while for the PR curve, the baseline is derived from the "always-positive classifier" where any recall or precision under π should be discarded [1].

It leads to a second where in the PR curve, we plot the horizontal line derived from the always-positive classifier but we don't discard when recall < π. In this case, as mentioned by @e-pet, it might make sense to show the hyperbolic line of the always-positive classifier instead (cf. Fig. 2 in [1]).

@e-pet feel free to add any other points that you wanted to discuss. Here, I wanted to focus on the one that looks critical and could be addressed.

[1] [Flach, P., & Kull, M. (2015). Precision-recall-gain curves: PR analysis done right. Advances in neural information processing systems, 28.](https://papers.nips.cc/paper_files/paper/2015/file/33e8075e9970de0cfea955afd4644bb2-Paper.pdf)

COMMENT:
Hey, I'd like to work on this. I have a fix ready where I've updated the label to "Non-informative baseline" in RocCurveDisplay. I will be creating a Pull Request shortly.

Labels: Documentation, API
Revisit the "chance level" for the different displays

@e-pet commented on different PRs & issues some interesting fact. I take the opportunity to consolidate some of those comments here.

First, we use the term "chance" that is ambiguous depending of the displays. The term "baseline" would probably be better. In addition, I checked and I think we should make an extra effort on the definition of the baseline for each of the type of plot: for ROC curve, the baseline is "a random classifier assigning the positive class with probability p and the negative class with probability 1 − p" [1] while for the PR curve, the baseline is derived from the "always-positive classifier" where any recall or precision under π should be discarded [1].

It leads to a second where in the PR curve, we plot the horizontal line derived from the always-positive classifier but we don't discard when recall < π. In this case, as mentioned by @e-pet, it might make sense to show the hyperbolic line of the always-positive classifier instead (cf. Fig. 2 in [1]).

@e-pet feel free to add any other points that you wanted to discuss. Here, I wanted to focus on the one that looks critical and could be addressed.

[1] [Flach, P., & Kull, M. (2015). Precision-recall-gain curves: PR analysis done right. Advances in neural information processing systems, 28.](https://papers.nips.cc/paper_files/paper/2015/file/33e8075e9970de0cfea955afd4644bb2-Paper.pdf)

COMMENT:
Hi @glemaitre,
I’d like to work on this issue — updating “chance level” to “non-informative baseline” and improving baseline handling/documentation in the ROC & PR displays.
I’ve read the Flach & Kull paper and reviewed the display code, so I’m ready to start right away. 
Could you please assign this to me?

Labels: Documentation, API
Revisit the "chance level" for the different displays

@e-pet commented on different PRs & issues some interesting fact. I take the opportunity to consolidate some of those comments here.

First, we use the term "chance" that is ambiguous depending of the displays. The term "baseline" would probably be better. In addition, I checked and I think we should make an extra effort on the definition of the baseline for each of the type of plot: for ROC curve, the baseline is "a random classifier assigning the positive class with probability p and the negative class with probability 1 − p" [1] while for the PR curve, the baseline is derived from the "always-positive classifier" where any recall or precision under π should be discarded [1].

It leads to a second where in the PR curve, we plot the horizontal line derived from the always-positive classifier but we don't discard when recall < π. In this case, as mentioned by @e-pet, it might make sense to show the hyperbolic line of the always-positive classifier instead (cf. Fig. 2 in [1]).

@e-pet feel free to add any other points that you wanted to discuss. Here, I wanted to focus on the one that looks critical and could be addressed.

[1] [Flach, P., & Kull, M. (2015). Precision-recall-gain curves: PR analysis done right. Advances in neural information processing systems, 28.](https://papers.nips.cc/paper_files/paper/2015/file/33e8075e9970de0cfea955afd4644bb2-Paper.pdf)

COMMENT:
/take

Labels: Documentation, API
Revisit the "chance level" for the different displays

@e-pet commented on different PRs & issues some interesting fact. I take the opportunity to consolidate some of those comments here.

First, we use the term "chance" that is ambiguous depending of the displays. The term "baseline" would probably be better. In addition, I checked and I think we should make an extra effort on the definition of the baseline for each of the type of plot: for ROC curve, the baseline is "a random classifier assigning the positive class with probability p and the negative class with probability 1 − p" [1] while for the PR curve, the baseline is derived from the "always-positive classifier" where any recall or precision under π should be discarded [1].

It leads to a second where in the PR curve, we plot the horizontal line derived from the always-positive classifier but we don't discard when recall < π. In this case, as mentioned by @e-pet, it might make sense to show the hyperbolic line of the always-positive classifier instead (cf. Fig. 2 in [1]).

@e-pet feel free to add any other points that you wanted to discuss. Here, I wanted to focus on the one that looks critical and could be addressed.

[1] [Flach, P., & Kull, M. (2015). Precision-recall-gain curves: PR analysis done right. Advances in neural information processing systems, 28.](https://papers.nips.cc/paper_files/paper/2015/file/33e8075e9970de0cfea955afd4644bb2-Paper.pdf)

COMMENT:
I am taking it

Labels: Documentation, API
Revisit the "chance level" for the different displays

@e-pet commented on different PRs & issues some interesting fact. I take the opportunity to consolidate some of those comments here.

First, we use the term "chance" that is ambiguous depending of the displays. The term "baseline" would probably be better. In addition, I checked and I think we should make an extra effort on the definition of the baseline for each of the type of plot: for ROC curve, the baseline is "a random classifier assigning the positive class with probability p and the negative class with probability 1 − p" [1] while for the PR curve, the baseline is derived from the "always-positive classifier" where any recall or precision under π should be discarded [1].

It leads to a second where in the PR curve, we plot the horizontal line derived from the always-positive classifier but we don't discard when recall < π. In this case, as mentioned by @e-pet, it might make sense to show the hyperbolic line of the always-positive classifier instead (cf. Fig. 2 in [1]).

@e-pet feel free to add any other points that you wanted to discuss. Here, I wanted to focus on the one that looks critical and could be addressed.

[1] [Flach, P., & Kull, M. (2015). Precision-recall-gain curves: PR analysis done right. Advances in neural information processing systems, 28.](https://papers.nips.cc/paper_files/paper/2015/file/33e8075e9970de0cfea955afd4644bb2-Paper.pdf)

COMMENT:
I'm taking it on

Labels: Documentation, API
Revisit the "chance level" for the different displays

@e-pet commented on different PRs & issues some interesting fact. I take the opportunity to consolidate some of those comments here.

First, we use the term "chance" that is ambiguous depending of the displays. The term "baseline" would probably be better. In addition, I checked and I think we should make an extra effort on the definition of the baseline for each of the type of plot: for ROC curve, the baseline is "a random classifier assigning the positive class with probability p and the negative class with probability 1 − p" [1] while for the PR curve, the baseline is derived from the "always-positive classifier" where any recall or precision under π should be discarded [1].

It leads to a second where in the PR curve, we plot the horizontal line derived from the always-positive classifier but we don't discard when recall < π. In this case, as mentioned by @e-pet, it might make sense to show the hyperbolic line of the always-positive classifier instead (cf. Fig. 2 in [1]).

@e-pet feel free to add any other points that you wanted to discuss. Here, I wanted to focus on the one that looks critical and could be addressed.

[1] [Flach, P., & Kull, M. (2015). Precision-recall-gain curves: PR analysis done right. Advances in neural information processing systems, 28.](https://papers.nips.cc/paper_files/paper/2015/file/33e8075e9970de0cfea955afd4644bb2-Paper.pdf)

COMMENT:
I've removed the good first issue tag as judging by the 4 PRs which all address this issue in a different way, I don't think we have reached consensus on how best to address this issue.

I think we agree that 'chance' is not an ideal term.
@glemaitre has suggested "baseline" while @ogrisel has suggested "non-informative predictor" or "non-informative baseline" (or event "constant predictor"). I think it would be nice if the ROC curve and PR curve "chance"/"baseline"/"non-informative"... lines both had names that were somewhat consistent. Let's get consensus on what to change it to before we start.

@e-pet (https://github.com/scikit-learn/scikit-learn/issues/30352#issuecomment-2509732896) do you think we should re-think which 'baseline' line we include in our PR curve? Or if we should include more than one? Or that we should mention the others in documentation?

Labels: Documentation, API
Revisit the "chance level" for the different displays

@e-pet commented on different PRs & issues some interesting fact. I take the opportunity to consolidate some of those comments here.

First, we use the term "chance" that is ambiguous depending of the displays. The term "baseline" would probably be better. In addition, I checked and I think we should make an extra effort on the definition of the baseline for each of the type of plot: for ROC curve, the baseline is "a random classifier assigning the positive class with probability p and the negative class with probability 1 − p" [1] while for the PR curve, the baseline is derived from the "always-positive classifier" where any recall or precision under π should be discarded [1].

It leads to a second where in the PR curve, we plot the horizontal line derived from the always-positive classifier but we don't discard when recall < π. In this case, as mentioned by @e-pet, it might make sense to show the hyperbolic line of the always-positive classifier instead (cf. Fig. 2 in [1]).

@e-pet feel free to add any other points that you wanted to discuss. Here, I wanted to focus on the one that looks critical and could be addressed.

[1] [Flach, P., & Kull, M. (2015). Precision-recall-gain curves: PR analysis done right. Advances in neural information processing systems, 28.](https://papers.nips.cc/paper_files/paper/2015/file/33e8075e9970de0cfea955afd4644bb2-Paper.pdf)

COMMENT:
Hi all, thank you for continuing to work on this! 

A few thoughts from my side on the discussion:
- "Chance" vs. "Non-informative": I like the former because it is "constructive" in the sense that it also tells the reader how exactly that baseline can actually be obtained. But I also get @ogrisel on the fact that other non-informative baselines (eg constant predictors) yield the same line. So maybe "Non-informative" + some elaboration on this in the docs would be the best solution?
- For the PR curve, I would personally advocate for the possibility to optionally also display the two other 'baselines':
  1) "F_1 score of always-positive classifier"
  2) "Unreachable region" - this would ideally be not just a line but some kind of shading of the region below it

I would actually prefer to show all relevant baselines by default, but my understanding was that this is unwanted because of backwards stability.

Labels: Documentation, API