Talip7/scikit-learn-issues-multilabel huggingface dataset:
⚠️ CI failed on Linux_nogil.pylatest_pip_nogil (last failure: Jun 10, 2024) ⚠️

**CI is still failing on [Linux_nogil.pylatest_pip_nogil](https://dev.azure.com/scikit-learn/scikit-learn/_build/results?buildId=67402&view=logs&j=67fbb25f-e417-50be-be55-3b1e9637fce5)** (Jun 10, 2024)
Unable to find junit file. Please see link for details.

COMMENT:
> once there is a Scipy wheel, it is up to us to add a CI build for CPython 3.13 free-threaded 😉.

@lesteve SciPy nightly wheels are available now. Our team for free-threaded CPython at Quansight Labs has bandwidth to help with this kind of task for scikit-learn (wheels, debugging, etc.) over the next months. If you want us to work on adding this new CI job, or contribute to thread-safety fixes or whatever, please don't hesitate to ask.

Labels: Needs Triage
⚠️ CI failed on Linux_nogil.pylatest_pip_nogil (last failure: Jun 10, 2024) ⚠️

**CI is still failing on [Linux_nogil.pylatest_pip_nogil](https://dev.azure.com/scikit-learn/scikit-learn/_build/results?buildId=67402&view=logs&j=67fbb25f-e417-50be-be55-3b1e9637fce5)** (Jun 10, 2024)
Unable to find junit file. Please see link for details.

COMMENT:
Yeah I have seen this, this is great! I think it will be more efficient if I add the CI job, because I am familiar with scikit-learn CI quirks, but I definitely will ask you or Nathan when needed.

FYI I opened a CPython 3.13 free-threaded WIP PR in joblib as well recently https://github.com/joblib/joblib/pull/1589

Labels: Needs Triage
RFC make response / inverse link / activation function official

With questions like #29163 and with the private loss functions #15123 (almost everywhere) in place, I would like to discuss to make the inverse link function public.

Models like LogisticRegression or HistGradientBoostingRegressor(loss="poisson") have predictions like `inverse_link(raw_prediction(X))` where `raw_prediction(X)` is the prediction in "link space", e.g. linear predictor ("eta") for linear models.

**In line with the most recent nomenclature of HistGradientBoosting\*, I propose the following public API for regressors and classifiers:**
- `raw_predict(X)`
- `response_function(y_raw)` or `activation_function(y_raw)`
- `link_function(y_obs)`

**Alternatives:**
1. `estimator.link` is a link object which has 2 methods named like above.
2. 1-to-1 with the actual implementation: `estimator.loss.link` and then as alternative 1. This would also expose the loss function (object), see also #28169.

**Further considerations**
- This would also make easier/solve #18309
- Does this necessitate a SLEP?

@scikit-learn/communication-team @scikit-learn/contributor-experience-team @scikit-learn/core-devs @scikit-learn/documentation-team ping

COMMENT:
isn't `raw_predictions` the same as `decision_function` in effect?

`activation_function` sounds NN like, so maybe not appropriate.

Overall, I don't think I like the idea of adding these at the estimator level. I personally don't even understand what `link_function` or `response_function` would do. However, I'd be happy to have `estimator.link` documented properly, and then have methods of `estimator.link` as public methods for people who really want to dig in.

Labels: API, RFC
RFC make response / inverse link / activation function official

With questions like #29163 and with the private loss functions #15123 (almost everywhere) in place, I would like to discuss to make the inverse link function public.

Models like LogisticRegression or HistGradientBoostingRegressor(loss="poisson") have predictions like `inverse_link(raw_prediction(X))` where `raw_prediction(X)` is the prediction in "link space", e.g. linear predictor ("eta") for linear models.

**In line with the most recent nomenclature of HistGradientBoosting\*, I propose the following public API for regressors and classifiers:**
- `raw_predict(X)`
- `response_function(y_raw)` or `activation_function(y_raw)`
- `link_function(y_obs)`

**Alternatives:**
1. `estimator.link` is a link object which has 2 methods named like above.
2. 1-to-1 with the actual implementation: `estimator.loss.link` and then as alternative 1. This would also expose the loss function (object), see also #28169.

**Further considerations**
- This would also make easier/solve #18309
- Does this necessitate a SLEP?

@scikit-learn/communication-team @scikit-learn/contributor-experience-team @scikit-learn/core-devs @scikit-learn/documentation-team ping

COMMENT:
The link function is the inverse of what you usually apply in reality:
- Binary Logistic: `return of predict_proba = expit(raw)` and link function is `logit`
- (Log-)-Poisson regression: `return of predict = exp(raw)` and link function is `log`

> isn't raw_predictions the same as decision_function in effect?

For classifiers, in effect, yes. But not API-wise. Also, `decision_function` is one of the greates misnomers of our API, IMO. The existance of decision_function is also not guaranteed from what I read in our [docs](https://scikit-learn.org/dev/developers/develop.html):
> Classification algorithms usually also offer a way to quantify certainty of a prediction, either using decision_function or predict_proba

Labels: API, RFC
RFC make response / inverse link / activation function official

With questions like #29163 and with the private loss functions #15123 (almost everywhere) in place, I would like to discuss to make the inverse link function public.

Models like LogisticRegression or HistGradientBoostingRegressor(loss="poisson") have predictions like `inverse_link(raw_prediction(X))` where `raw_prediction(X)` is the prediction in "link space", e.g. linear predictor ("eta") for linear models.

**In line with the most recent nomenclature of HistGradientBoosting\*, I propose the following public API for regressors and classifiers:**
- `raw_predict(X)`
- `response_function(y_raw)` or `activation_function(y_raw)`
- `link_function(y_obs)`

**Alternatives:**
1. `estimator.link` is a link object which has 2 methods named like above.
2. 1-to-1 with the actual implementation: `estimator.loss.link` and then as alternative 1. This would also expose the loss function (object), see also #28169.

**Further considerations**
- This would also make easier/solve #18309
- Does this necessitate a SLEP?

@scikit-learn/communication-team @scikit-learn/contributor-experience-team @scikit-learn/core-devs @scikit-learn/documentation-team ping

COMMENT:
Site note: _I guess that for the softmax inverse link used for the multiclass log-loss, we would expose `softmax` as inverse link and `log` as (forward) link since softmax is not bijective._

Main comment to the core of the RFC:

I think would intuitively favor making `estimator.loss.link` public instead of introducing new methods on the estimator itself, but we would still need to expose a `raw_predict` method on the estimator to make that useful.

I agree that `decision_function` is a really bad name in retrospect, especially now that we started introducing cost-sensitive learning tools such as `TunedThresholdClassifierCV` where the "decision" would better represent the thresholded `predict_proba` or `decision_function` value of a binary classifier, for instance.

So we could introduce `raw_predict` (at least for classifiers, regressors, not sure about clustering algorithms).

Then we could soft deprecate (e.g. stop using and hide in the doc any occurrence of `decision_function` in favor of `raw_predict` for classifiers, but keep the `decision_function` as a pure backward compat alias to avoid breaking user code).

We would definitely a SLEP for such an impacting public API change though.

Labels: API, RFC
RFC make response / inverse link / activation function official

With questions like #29163 and with the private loss functions #15123 (almost everywhere) in place, I would like to discuss to make the inverse link function public.

Models like LogisticRegression or HistGradientBoostingRegressor(loss="poisson") have predictions like `inverse_link(raw_prediction(X))` where `raw_prediction(X)` is the prediction in "link space", e.g. linear predictor ("eta") for linear models.

**In line with the most recent nomenclature of HistGradientBoosting\*, I propose the following public API for regressors and classifiers:**
- `raw_predict(X)`
- `response_function(y_raw)` or `activation_function(y_raw)`
- `link_function(y_obs)`

**Alternatives:**
1. `estimator.link` is a link object which has 2 methods named like above.
2. 1-to-1 with the actual implementation: `estimator.loss.link` and then as alternative 1. This would also expose the loss function (object), see also #28169.

**Further considerations**
- This would also make easier/solve #18309
- Does this necessitate a SLEP?

@scikit-learn/communication-team @scikit-learn/contributor-experience-team @scikit-learn/core-devs @scikit-learn/documentation-team ping

COMMENT:
Note that the question in #29163 should first be addressed by updating our user guide (and maybe cross-reference it from the docstring for the `predict` (for regressors) and `predict_proba` (for classifiers) methods.

Labels: API, RFC
RFC make response / inverse link / activation function official

With questions like #29163 and with the private loss functions #15123 (almost everywhere) in place, I would like to discuss to make the inverse link function public.

Models like LogisticRegression or HistGradientBoostingRegressor(loss="poisson") have predictions like `inverse_link(raw_prediction(X))` where `raw_prediction(X)` is the prediction in "link space", e.g. linear predictor ("eta") for linear models.

**In line with the most recent nomenclature of HistGradientBoosting\*, I propose the following public API for regressors and classifiers:**
- `raw_predict(X)`
- `response_function(y_raw)` or `activation_function(y_raw)`
- `link_function(y_obs)`

**Alternatives:**
1. `estimator.link` is a link object which has 2 methods named like above.
2. 1-to-1 with the actual implementation: `estimator.loss.link` and then as alternative 1. This would also expose the loss function (object), see also #28169.

**Further considerations**
- This would also make easier/solve #18309
- Does this necessitate a SLEP?

@scikit-learn/communication-team @scikit-learn/contributor-experience-team @scikit-learn/core-devs @scikit-learn/documentation-team ping

COMMENT:
I think `raw_predict` makes sense.

I'm unsure of adding expose the loss objects through `estimator.loss`. Can the link functions be a regular functions and in the docstring of `raw_predict` we state how to generate `predict` or `predict_proba`?

Labels: API, RFC
RFC make response / inverse link / activation function official

With questions like #29163 and with the private loss functions #15123 (almost everywhere) in place, I would like to discuss to make the inverse link function public.

Models like LogisticRegression or HistGradientBoostingRegressor(loss="poisson") have predictions like `inverse_link(raw_prediction(X))` where `raw_prediction(X)` is the prediction in "link space", e.g. linear predictor ("eta") for linear models.

**In line with the most recent nomenclature of HistGradientBoosting\*, I propose the following public API for regressors and classifiers:**
- `raw_predict(X)`
- `response_function(y_raw)` or `activation_function(y_raw)`
- `link_function(y_obs)`

**Alternatives:**
1. `estimator.link` is a link object which has 2 methods named like above.
2. 1-to-1 with the actual implementation: `estimator.loss.link` and then as alternative 1. This would also expose the loss function (object), see also #28169.

**Further considerations**
- This would also make easier/solve #18309
- Does this necessitate a SLEP?

@scikit-learn/communication-team @scikit-learn/contributor-experience-team @scikit-learn/core-devs @scikit-learn/documentation-team ping

COMMENT:
In principle, loss and link function are independent. It's only an efficient implementation that needs to merge them together.
As written in the RFC, having link and inverse link function available on the estimator would help several use cases and reduce confusion.

Labels: API, RFC
RFC make response / inverse link / activation function official

With questions like #29163 and with the private loss functions #15123 (almost everywhere) in place, I would like to discuss to make the inverse link function public.

Models like LogisticRegression or HistGradientBoostingRegressor(loss="poisson") have predictions like `inverse_link(raw_prediction(X))` where `raw_prediction(X)` is the prediction in "link space", e.g. linear predictor ("eta") for linear models.

**In line with the most recent nomenclature of HistGradientBoosting\*, I propose the following public API for regressors and classifiers:**
- `raw_predict(X)`
- `response_function(y_raw)` or `activation_function(y_raw)`
- `link_function(y_obs)`

**Alternatives:**
1. `estimator.link` is a link object which has 2 methods named like above.
2. 1-to-1 with the actual implementation: `estimator.loss.link` and then as alternative 1. This would also expose the loss function (object), see also #28169.

**Further considerations**
- This would also make easier/solve #18309
- Does this necessitate a SLEP?

@scikit-learn/communication-team @scikit-learn/contributor-experience-team @scikit-learn/core-devs @scikit-learn/documentation-team ping

COMMENT:
I think adding three new methods estimators is pretty big API update. I'm trying to find a path to get some momentum, like standardizing on `raw_predict`.

After `raw_predict` is out, then there is a standard way to do this:

```python
log_reg = LogisticRegression().fit(X, y)
raw_pred = log_reg.raw_predict(X)

# If they want to get the actual predctions, they need to know the inverse link:
y_pred = expit(raw_pred)
```

I think this is already a net win.

Afterwards, if we want to go further, we can extend the API:

```python
y_pred = log_reg.link.inverse(raw_pred)

raw_again = log_reg.link(y_pred)
```

Labels: API, RFC
RFC make response / inverse link / activation function official

With questions like #29163 and with the private loss functions #15123 (almost everywhere) in place, I would like to discuss to make the inverse link function public.

Models like LogisticRegression or HistGradientBoostingRegressor(loss="poisson") have predictions like `inverse_link(raw_prediction(X))` where `raw_prediction(X)` is the prediction in "link space", e.g. linear predictor ("eta") for linear models.

**In line with the most recent nomenclature of HistGradientBoosting\*, I propose the following public API for regressors and classifiers:**
- `raw_predict(X)`
- `response_function(y_raw)` or `activation_function(y_raw)`
- `link_function(y_obs)`

**Alternatives:**
1. `estimator.link` is a link object which has 2 methods named like above.
2. 1-to-1 with the actual implementation: `estimator.loss.link` and then as alternative 1. This would also expose the loss function (object), see also #28169.

**Further considerations**
- This would also make easier/solve #18309
- Does this necessitate a SLEP?

@scikit-learn/communication-team @scikit-learn/contributor-experience-team @scikit-learn/core-devs @scikit-learn/documentation-team ping

COMMENT:
But `raw_predict` without something like `response_function`/`inverse` is quite meaningless. Maybe, we don't need the `link` function itself, only it's inverse.

EDIT: Ok, starting with only `raw_predict` makes sense and already enable new things.

> I think adding three new methods estimators is pretty big API update.

Yes, it is. Again "but" those are missing from the beginnings as they are quite fundamental (for any loss except the squared error and quantile loss).

Labels: API, RFC
RFC make response / inverse link / activation function official

With questions like #29163 and with the private loss functions #15123 (almost everywhere) in place, I would like to discuss to make the inverse link function public.

Models like LogisticRegression or HistGradientBoostingRegressor(loss="poisson") have predictions like `inverse_link(raw_prediction(X))` where `raw_prediction(X)` is the prediction in "link space", e.g. linear predictor ("eta") for linear models.

**In line with the most recent nomenclature of HistGradientBoosting\*, I propose the following public API for regressors and classifiers:**
- `raw_predict(X)`
- `response_function(y_raw)` or `activation_function(y_raw)`
- `link_function(y_obs)`

**Alternatives:**
1. `estimator.link` is a link object which has 2 methods named like above.
2. 1-to-1 with the actual implementation: `estimator.loss.link` and then as alternative 1. This would also expose the loss function (object), see also #28169.

**Further considerations**
- This would also make easier/solve #18309
- Does this necessitate a SLEP?

@scikit-learn/communication-team @scikit-learn/contributor-experience-team @scikit-learn/core-devs @scikit-learn/documentation-team ping

COMMENT:
In the short term, I'm thinking of having the inverse link function in the docstring, so the `raw_predict` is meaningful:

```python
def raw_predict(self, X):
    """Return the raw prediction.

    `predict`'s output is the same as `inverse_link(raw_prediction)`, where 
    the inverse link function is `scipy.special.expit`.
    """
```

For estimators that can adjust their `loss`, we'll list out the inverse link functions.

Labels: API, RFC
RFC make response / inverse link / activation function official

With questions like #29163 and with the private loss functions #15123 (almost everywhere) in place, I would like to discuss to make the inverse link function public.

Models like LogisticRegression or HistGradientBoostingRegressor(loss="poisson") have predictions like `inverse_link(raw_prediction(X))` where `raw_prediction(X)` is the prediction in "link space", e.g. linear predictor ("eta") for linear models.

**In line with the most recent nomenclature of HistGradientBoosting\*, I propose the following public API for regressors and classifiers:**
- `raw_predict(X)`
- `response_function(y_raw)` or `activation_function(y_raw)`
- `link_function(y_obs)`

**Alternatives:**
1. `estimator.link` is a link object which has 2 methods named like above.
2. 1-to-1 with the actual implementation: `estimator.loss.link` and then as alternative 1. This would also expose the loss function (object), see also #28169.

**Further considerations**
- This would also make easier/solve #18309
- Does this necessitate a SLEP?

@scikit-learn/communication-team @scikit-learn/contributor-experience-team @scikit-learn/core-devs @scikit-learn/documentation-team ping

COMMENT:
Another use case for `raw_predict` is #28574 which needs access to logits.

Labels: API, RFC
RFC make response / inverse link / activation function official

With questions like #29163 and with the private loss functions #15123 (almost everywhere) in place, I would like to discuss to make the inverse link function public.

Models like LogisticRegression or HistGradientBoostingRegressor(loss="poisson") have predictions like `inverse_link(raw_prediction(X))` where `raw_prediction(X)` is the prediction in "link space", e.g. linear predictor ("eta") for linear models.

**In line with the most recent nomenclature of HistGradientBoosting\*, I propose the following public API for regressors and classifiers:**
- `raw_predict(X)`
- `response_function(y_raw)` or `activation_function(y_raw)`
- `link_function(y_obs)`

**Alternatives:**
1. `estimator.link` is a link object which has 2 methods named like above.
2. 1-to-1 with the actual implementation: `estimator.loss.link` and then as alternative 1. This would also expose the loss function (object), see also #28169.

**Further considerations**
- This would also make easier/solve #18309
- Does this necessitate a SLEP?

@scikit-learn/communication-team @scikit-learn/contributor-experience-team @scikit-learn/core-devs @scikit-learn/documentation-team ping

COMMENT:
I think we would need to clarify:

- if we want to expose `raw_predict` for all supervised estimators in scikit-learn or only for models with an explicit link function (GLMs, MLPs, Gradient Boosting...);
- how it is defined for non-probabilistic classifiers such as SVC (and variants): is it always mapped to `decision_function` for a classifier?

Labels: API, RFC
RFC make response / inverse link / activation function official

With questions like #29163 and with the private loss functions #15123 (almost everywhere) in place, I would like to discuss to make the inverse link function public.

Models like LogisticRegression or HistGradientBoostingRegressor(loss="poisson") have predictions like `inverse_link(raw_prediction(X))` where `raw_prediction(X)` is the prediction in "link space", e.g. linear predictor ("eta") for linear models.

**In line with the most recent nomenclature of HistGradientBoosting\*, I propose the following public API for regressors and classifiers:**
- `raw_predict(X)`
- `response_function(y_raw)` or `activation_function(y_raw)`
- `link_function(y_obs)`

**Alternatives:**
1. `estimator.link` is a link object which has 2 methods named like above.
2. 1-to-1 with the actual implementation: `estimator.loss.link` and then as alternative 1. This would also expose the loss function (object), see also #28169.

**Further considerations**
- This would also make easier/solve #18309
- Does this necessitate a SLEP?

@scikit-learn/communication-team @scikit-learn/contributor-experience-team @scikit-learn/core-devs @scikit-learn/documentation-team ping

COMMENT:
We discussed it in the monthly meeting https://github.com/scikit-learn/administrative/blob/master/monthly_meetings/2025-02-24.md and agreed on having a `raw_predict`, officially.

And yes, for non-probabilistic classifiers, it returns the same as `decision_function`. We also discussed deprecating `decision_function` (with a long cycle). No hard objections were mentioned. I would be +1 for that.

Labels: API, RFC