Talip7/scikit-learn-issues-multilabel huggingface dataset:
API to predict multiple quantiles at once

Classifiers have a `predict_proba` method that makes it possible to quantify probabilistic ally the certainty in the predictions for a given input `X_i`.

Currently most regressors in scikit-learn only predict a conditional expectile E[Y|X], and some have a `return_std` option that makes it also possible to estimate sqrt(VAR[Y|X]), which can be used to quantify the certainty when assuming a Gaussian predictive distribution (typically for Gaussian processes which estimate a Gaussian predictive posterior distribution).

We do have pointwise quantile estimators (linear models, gradient boosting, hist gradient boosting) where the `predict` method returns a single point estimate for target quantile passed as an hyper-parameter instead of estimating an expectile.

Several people have expressed the need to have more generic API that can return an array of quantile estimates for a given input `X_i`.

The goal of this issue is to centralize the discussion of an API extension to be able to do this more uniformly in scikit-learn, either via a meta-estimator that wraps an array of point-wise quantile estimator to turn it into a quantile-array estimator or to directly have the base estimators able to do this directly (and sometimes more efficiently).

Some non-exhausitive list of related PRs and issues (feel free to add or suggest new ones):

- https://github.com/scikit-learn/scikit-learn/issues/19851
- https://github.com/scikit-learn/scikit-learn/pull/19754
- https://github.com/scikit-learn/scikit-learn/issues/20964

Also related:

- conformal predictions: https://github.com/scikit-learn-contrib/MAPIE
- XGBoostLSS (location, scale and shape) https://github.com/StatMixedML/XGBoostLSS

Furthermore, models like Poisson regression that make a specific assumption about the conditional Y|X distribution, it would be possible to estimates of the inverse-CDF values of the estimated Y|X for instance. Those could probably also benefit from an expanded API.

If we do this, then we have the side question of how to evaluate such multi-quantile models. We could probably extend the pinball_loss scorer to average the pinball scores for an array of quantiles for instance.

/cc @GaelVaroquaux @amueller @lorentzenchr

COMMENT:
> The specific multiple quantiles interface that is proposed - and widely in use with sktime already - is the `predict_quantiles` interface which is shared by skpro and sktime.

I‘m not 100% convinced of introducing such a method. For quantile regressors, `predict` already predicts a (single) quantile.

@ogrisel wrote in the initial statement:
> The goal of this issue is to centralize the discussion of an API extension to be able to do this more uniformly in scikit-learn, either via a meta-estimator that wraps an array of point-wise quantile estimator to turn it into a quantile-array estimator or to directly have the base estimators able to do this directly (and sometimes more efficiently).

@scikit-learn/core-devs ping for API discussion.

Labels: API, Needs Decision
API to predict multiple quantiles at once

Classifiers have a `predict_proba` method that makes it possible to quantify probabilistic ally the certainty in the predictions for a given input `X_i`.

Currently most regressors in scikit-learn only predict a conditional expectile E[Y|X], and some have a `return_std` option that makes it also possible to estimate sqrt(VAR[Y|X]), which can be used to quantify the certainty when assuming a Gaussian predictive distribution (typically for Gaussian processes which estimate a Gaussian predictive posterior distribution).

We do have pointwise quantile estimators (linear models, gradient boosting, hist gradient boosting) where the `predict` method returns a single point estimate for target quantile passed as an hyper-parameter instead of estimating an expectile.

Several people have expressed the need to have more generic API that can return an array of quantile estimates for a given input `X_i`.

The goal of this issue is to centralize the discussion of an API extension to be able to do this more uniformly in scikit-learn, either via a meta-estimator that wraps an array of point-wise quantile estimator to turn it into a quantile-array estimator or to directly have the base estimators able to do this directly (and sometimes more efficiently).

Some non-exhausitive list of related PRs and issues (feel free to add or suggest new ones):

- https://github.com/scikit-learn/scikit-learn/issues/19851
- https://github.com/scikit-learn/scikit-learn/pull/19754
- https://github.com/scikit-learn/scikit-learn/issues/20964

Also related:

- conformal predictions: https://github.com/scikit-learn-contrib/MAPIE
- XGBoostLSS (location, scale and shape) https://github.com/StatMixedML/XGBoostLSS

Furthermore, models like Poisson regression that make a specific assumption about the conditional Y|X distribution, it would be possible to estimates of the inverse-CDF values of the estimated Y|X for instance. Those could probably also benefit from an expanded API.

If we do this, then we have the side question of how to evaluate such multi-quantile models. We could probably extend the pinball_loss scorer to average the pinball scores for an array of quantiles for instance.

/cc @GaelVaroquaux @amueller @lorentzenchr

COMMENT:
> @scikit-learn/core-devs ping for API discussion.

I'm +1 with the API proposed here: https://github.com/scikit-learn/scikit-learn/issues/23334#issuecomment-1124808687. Now that we have metadata routing, we can properly support `predict(X, quantile_levels=[0.05, 0.5, 0.95])`.

Labels: API, Needs Decision
API to predict multiple quantiles at once

Classifiers have a `predict_proba` method that makes it possible to quantify probabilistic ally the certainty in the predictions for a given input `X_i`.

Currently most regressors in scikit-learn only predict a conditional expectile E[Y|X], and some have a `return_std` option that makes it also possible to estimate sqrt(VAR[Y|X]), which can be used to quantify the certainty when assuming a Gaussian predictive distribution (typically for Gaussian processes which estimate a Gaussian predictive posterior distribution).

We do have pointwise quantile estimators (linear models, gradient boosting, hist gradient boosting) where the `predict` method returns a single point estimate for target quantile passed as an hyper-parameter instead of estimating an expectile.

Several people have expressed the need to have more generic API that can return an array of quantile estimates for a given input `X_i`.

The goal of this issue is to centralize the discussion of an API extension to be able to do this more uniformly in scikit-learn, either via a meta-estimator that wraps an array of point-wise quantile estimator to turn it into a quantile-array estimator or to directly have the base estimators able to do this directly (and sometimes more efficiently).

Some non-exhausitive list of related PRs and issues (feel free to add or suggest new ones):

- https://github.com/scikit-learn/scikit-learn/issues/19851
- https://github.com/scikit-learn/scikit-learn/pull/19754
- https://github.com/scikit-learn/scikit-learn/issues/20964

Also related:

- conformal predictions: https://github.com/scikit-learn-contrib/MAPIE
- XGBoostLSS (location, scale and shape) https://github.com/StatMixedML/XGBoostLSS

Furthermore, models like Poisson regression that make a specific assumption about the conditional Y|X distribution, it would be possible to estimates of the inverse-CDF values of the estimated Y|X for instance. Those could probably also benefit from an expanded API.

If we do this, then we have the side question of how to evaluate such multi-quantile models. We could probably extend the pinball_loss scorer to average the pinball scores for an array of quantiles for instance.

/cc @GaelVaroquaux @amueller @lorentzenchr

COMMENT:
#32100 and https://github.com/scikit-learn/scikit-learn/pull/32100#pullrequestreview-3377679931 in particular would make it possible to implement this feature efficiently for traditional regressor trees and bagging/forest ensembles.

Labels: API, Needs Decision
Consistent formulae for metrics in the user guide

### Describe the issue linked to the documentation

### Description
https://scikit-learn.org/stable/modules/model_evaluation.html lists the formulae of many metrics / scores. Most of them sum over a sample (incomplete list):
- accuracy
- multinomial / multiclass log loss
- Brier score
- all regression metrics

But some do not (complete list, hoping not to have missed one):
- Hinge loss
- Binary log loss
- zero-one loss (we should mention that this is 1-accuracy!)

### Proposition
Always sum over data points.


### Suggest a potential alternative/fix

_No response_

COMMENT:
I am +1 on summing over the data points.

Labels: Easy, Documentation
Consistent formulae for metrics in the user guide

### Describe the issue linked to the documentation

### Description
https://scikit-learn.org/stable/modules/model_evaluation.html lists the formulae of many metrics / scores. Most of them sum over a sample (incomplete list):
- accuracy
- multinomial / multiclass log loss
- Brier score
- all regression metrics

But some do not (complete list, hoping not to have missed one):
- Hinge loss
- Binary log loss
- zero-one loss (we should mention that this is 1-accuracy!)

### Proposition
Always sum over data points.


### Suggest a potential alternative/fix

_No response_

COMMENT:
In the Log Loss section, that first "binary log loss" formula seems intended to help guide the reader in explaining the overall "log loss" formula. It specifically calls out that it's for a single sample. In the following paragraph, the author explains how to generalize the single-sample binary log-loss formula to both handle:

1. The Multi-class case
2. Multiple samples

To me, it seems like that first binary log loss formula should stay as is, since it's being used as a building block to explain the concept. The multi-class case in the following paragraph does contain `N`.

Pardon me jumping in here, it's my first time contributing!

Labels: Easy, Documentation
Consistent formulae for metrics in the user guide

### Describe the issue linked to the documentation

### Description
https://scikit-learn.org/stable/modules/model_evaluation.html lists the formulae of many metrics / scores. Most of them sum over a sample (incomplete list):
- accuracy
- multinomial / multiclass log loss
- Brier score
- all regression metrics

But some do not (complete list, hoping not to have missed one):
- Hinge loss
- Binary log loss
- zero-one loss (we should mention that this is 1-accuracy!)

### Proposition
Always sum over data points.


### Suggest a potential alternative/fix

_No response_

COMMENT:
Additionally missing:
- Hamming
- Jaccard

Labels: Easy, Documentation
Consistent formulae for metrics in the user guide

### Describe the issue linked to the documentation

### Description
https://scikit-learn.org/stable/modules/model_evaluation.html lists the formulae of many metrics / scores. Most of them sum over a sample (incomplete list):
- accuracy
- multinomial / multiclass log loss
- Brier score
- all regression metrics

But some do not (complete list, hoping not to have missed one):
- Hinge loss
- Binary log loss
- zero-one loss (we should mention that this is 1-accuracy!)

### Proposition
Always sum over data points.


### Suggest a potential alternative/fix

_No response_

COMMENT:
Hi everyone, I'm new to open source and willing to add my first contribution. Can I work on this issue? How can I get started?

Labels: Easy, Documentation
Consistent formulae for metrics in the user guide

### Describe the issue linked to the documentation

### Description
https://scikit-learn.org/stable/modules/model_evaluation.html lists the formulae of many metrics / scores. Most of them sum over a sample (incomplete list):
- accuracy
- multinomial / multiclass log loss
- Brier score
- all regression metrics

But some do not (complete list, hoping not to have missed one):
- Hinge loss
- Binary log loss
- zero-one loss (we should mention that this is 1-accuracy!)

### Proposition
Always sum over data points.


### Suggest a potential alternative/fix

_No response_

COMMENT:
Hello all, I am getting started in Open Source contribution. Can I solve this issue?

Labels: Easy, Documentation
Consistent formulae for metrics in the user guide

### Describe the issue linked to the documentation

### Description
https://scikit-learn.org/stable/modules/model_evaluation.html lists the formulae of many metrics / scores. Most of them sum over a sample (incomplete list):
- accuracy
- multinomial / multiclass log loss
- Brier score
- all regression metrics

But some do not (complete list, hoping not to have missed one):
- Hinge loss
- Binary log loss
- zero-one loss (we should mention that this is 1-accuracy!)

### Proposition
Always sum over data points.


### Suggest a potential alternative/fix

_No response_

COMMENT:
I'm working on this.

Labels: Easy, Documentation
Consistent formulae for metrics in the user guide

### Describe the issue linked to the documentation

### Description
https://scikit-learn.org/stable/modules/model_evaluation.html lists the formulae of many metrics / scores. Most of them sum over a sample (incomplete list):
- accuracy
- multinomial / multiclass log loss
- Brier score
- all regression metrics

But some do not (complete list, hoping not to have missed one):
- Hinge loss
- Binary log loss
- zero-one loss (we should mention that this is 1-accuracy!)

### Proposition
Always sum over data points.


### Suggest a potential alternative/fix

_No response_

COMMENT:
/take

Labels: Easy, Documentation
Support for ordinal multi-classification

### Describe the workflow you want to enable

Encode response variable ordering with every scikit learn classifier according to the method introduced in this [frequently cited paper](https://link.springer.com/chapter/10.1007/3-540-44795-4_13).

### Describe your proposed solution

Implement `OrdinalClassifier`, a classifier that takes another scikit learn classifier as input and encodes the ordinality assumption into the classifier. It's quite simple, and there are also blog posts on it, for example by @M46F [here](https://towardsdatascience.com/simple-trick-to-train-an-ordinal-regression-with-any-classifier-6911183d2a3c). Here's a slightly more advanced version that can also handle arbitrary labels that I'd be willing to contribute as a PR: https://gist.github.com/lorenzwalthert/51371894225c7b530b66bdabfad60327

A lot of classification problems are inherently ordinal in nature and this trick has proven to work well in various settings, the code is very little to add to the project, adds zero dependencies and is a much cited approach (600+).

### Describe alternatives you've considered, if relevant

Using that code outside of scikit learn, but I think ti's generally useful to others.

### Additional context

_No response_

COMMENT:
How does it compare with NOCATS such as started in this pull request: https://github.com/scikit-learn/scikit-learn/pull/12866

Labels: New Feature, Needs Decision - Include Feature
Support for ordinal multi-classification

### Describe the workflow you want to enable

Encode response variable ordering with every scikit learn classifier according to the method introduced in this [frequently cited paper](https://link.springer.com/chapter/10.1007/3-540-44795-4_13).

### Describe your proposed solution

Implement `OrdinalClassifier`, a classifier that takes another scikit learn classifier as input and encodes the ordinality assumption into the classifier. It's quite simple, and there are also blog posts on it, for example by @M46F [here](https://towardsdatascience.com/simple-trick-to-train-an-ordinal-regression-with-any-classifier-6911183d2a3c). Here's a slightly more advanced version that can also handle arbitrary labels that I'd be willing to contribute as a PR: https://gist.github.com/lorenzwalthert/51371894225c7b530b66bdabfad60327

A lot of classification problems are inherently ordinal in nature and this trick has proven to work well in various settings, the code is very little to add to the project, adds zero dependencies and is a much cited approach (600+).

### Describe alternatives you've considered, if relevant

Using that code outside of scikit learn, but I think ti's generally useful to others.

### Additional context

_No response_

COMMENT:
From my understanding of NOCATS, the differences are:

* NOCATS approaches handle categorical variables in the predictor, ordinal classification adresses the response variable.
* NOCATS try to get rid of the ordinality of features, or allows it to be unknown, while ordinal classification assumes the user knows the order. In more detail: There are two classical ways to treat categorical features. WIth dummy-encoding, the split is one group vs rest (which requires a lot of nodes). With integer-encoding, an order of the categories is assumed at the splitpoint (because all observations below the cut-off are sent left, the rest sent right). The NOCATS approach outlined in https://github.com/jblackburne uses random splits to generate the grouping at the split (and hopes to find useful ones). 
* NOCATS and similar efforts have a bigger scope in the sense that they seem to be relatively big changes in terms of line count, require compiled code etc. 
* NOCATS implementations like #12866 are specific for one (base) learner like trees, while the learner proposed in this issue can turn any classifier into a classifier that respects the order of the y values. In that sense, the scope of this isuse is bigger than NOCATS.

Labels: New Feature, Needs Decision - Include Feature
Support for ordinal multi-classification

### Describe the workflow you want to enable

Encode response variable ordering with every scikit learn classifier according to the method introduced in this [frequently cited paper](https://link.springer.com/chapter/10.1007/3-540-44795-4_13).

### Describe your proposed solution

Implement `OrdinalClassifier`, a classifier that takes another scikit learn classifier as input and encodes the ordinality assumption into the classifier. It's quite simple, and there are also blog posts on it, for example by @M46F [here](https://towardsdatascience.com/simple-trick-to-train-an-ordinal-regression-with-any-classifier-6911183d2a3c). Here's a slightly more advanced version that can also handle arbitrary labels that I'd be willing to contribute as a PR: https://gist.github.com/lorenzwalthert/51371894225c7b530b66bdabfad60327

A lot of classification problems are inherently ordinal in nature and this trick has proven to work well in various settings, the code is very little to add to the project, adds zero dependencies and is a much cited approach (600+).

### Describe alternatives you've considered, if relevant

Using that code outside of scikit learn, but I think ti's generally useful to others.

### Additional context

_No response_

COMMENT:
I completely missed that it was referring to encoding the target and not the data. So this is not related to NOCATS.

Labels: New Feature, Needs Decision - Include Feature
Support for ordinal multi-classification

### Describe the workflow you want to enable

Encode response variable ordering with every scikit learn classifier according to the method introduced in this [frequently cited paper](https://link.springer.com/chapter/10.1007/3-540-44795-4_13).

### Describe your proposed solution

Implement `OrdinalClassifier`, a classifier that takes another scikit learn classifier as input and encodes the ordinality assumption into the classifier. It's quite simple, and there are also blog posts on it, for example by @M46F [here](https://towardsdatascience.com/simple-trick-to-train-an-ordinal-regression-with-any-classifier-6911183d2a3c). Here's a slightly more advanced version that can also handle arbitrary labels that I'd be willing to contribute as a PR: https://gist.github.com/lorenzwalthert/51371894225c7b530b66bdabfad60327

A lot of classification problems are inherently ordinal in nature and this trick has proven to work well in various settings, the code is very little to add to the project, adds zero dependencies and is a much cited approach (600+).

### Describe alternatives you've considered, if relevant

Using that code outside of scikit learn, but I think ti's generally useful to others.

### Additional context

_No response_

COMMENT:
So this is something that could be adequate for the `sklearn.multiclass` module in addition to OvO and OvR classifier.

Labels: New Feature, Needs Decision - Include Feature
Support for ordinal multi-classification

### Describe the workflow you want to enable

Encode response variable ordering with every scikit learn classifier according to the method introduced in this [frequently cited paper](https://link.springer.com/chapter/10.1007/3-540-44795-4_13).

### Describe your proposed solution

Implement `OrdinalClassifier`, a classifier that takes another scikit learn classifier as input and encodes the ordinality assumption into the classifier. It's quite simple, and there are also blog posts on it, for example by @M46F [here](https://towardsdatascience.com/simple-trick-to-train-an-ordinal-regression-with-any-classifier-6911183d2a3c). Here's a slightly more advanced version that can also handle arbitrary labels that I'd be willing to contribute as a PR: https://gist.github.com/lorenzwalthert/51371894225c7b530b66bdabfad60327

A lot of classification problems are inherently ordinal in nature and this trick has proven to work well in various settings, the code is very little to add to the project, adds zero dependencies and is a much cited approach (600+).

### Describe alternatives you've considered, if relevant

Using that code outside of scikit learn, but I think ti's generally useful to others.

### Additional context

_No response_

COMMENT:
Hello
Thanks for mentioning me I'm the writer of the article mentioned above.
I think it's important to be aware that there is a major flaws in the implementation that I wrote in that article: since each classifier are independent from each other, the method won’t give the true probability (it won’t sum up to 1)

this article by [Christopher Coffee](https://medium.com/@deathswithbenefits?source=post_page-----ec9aa0915d40--------------------------------)  might help deal with the problem mentioned above
https://medium.com/@deathswithbenefits/heres-the-actual-recipe-ec9aa0915d40

Labels: New Feature, Needs Decision - Include Feature