Talip7/scikit-learn-issues-multilabel huggingface dataset:
Add Accumulated local effects (ALE) to inspection

### Describe the workflow you want to enable

I'd love to push the inspection module further by adding Accumulated local effects (ALE) from Apley 2020. A great description can be found in Christoph's online book https://christophm.github.io/interpretable-ml-book/ale.html

ALE fix the problem of partial dependence that they force the model to be evaluated on impossible or rare feature combinations. ALE are defined for numeric features that can be binned. From both bin edges, the slope of the partial dependence is calculated locally, i.e., only using observations in the bin. The slopes from all bins are cumsummed and vertically centered to the average response or prediction.

### Reference

Apley, Daniel W., and Jingyu Zhu. 2020. Visualizing the Effects of Predictor Variables in Black Box Supervised Learning Models. Journal of the Royal Statistical Society Series B: Statistical Methodology, 82 (4): 1059–1086. doi:10.1111/rssb.12377.

### Describe your proposed solution

Pseudo code to calculate ALE for one feature:

``` py
pd_slopes = np.zeros_like(bins)

for i, bin in enumerate(bins):
  X_bin = pick n_per_bin = 200 rows from data in bin
  if X_bin is empty:
    next
  pd = partial_dependence_brute(model, X_bin, grid = [lower bin edge, upper bin edge], sample_weights)
  pd_slopes[i] = pd[1] - pd[0]

return np.cumsum(pd_slopes)
```

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_

COMMENT:
@mayer79 What do you think is the best strategy for computing the quantiles?

Labels: New Feature, Needs Decision
Changelog check on towncrier false positive case

Observed on this PR: https://github.com/scikit-learn/scikit-learn/pull/30209
This run: https://github.com/scikit-learn/scikit-learn/actions/runs/11681055082/job/32525320042?pr=30209

The PR needs to add PR number to existing changelog, and changes another affected changelog, therefore there are 3 changelog files affected in the PR. However, the changelog checker complains with:

```
 Not all changelog file number(s) match this pull request number (30209):
doc/whats_new/upcoming_changes/sklearn.calibration/30171.api.rst
doc/whats_new/upcoming_changes/sklearn.frozen/29705.major-feature.rst
doc/whats_new/upcoming_changes/sklearn.frozen/30209.major-feature.rst
```

Which I'd say is a false positive.

cc @lesteve

COMMENT:
https://github.com/scientific-python/action-towncrier-changelog checks the PR number is correct and wants all the fragment files to have the PR number in it:
https://github.com/scientific-python/action-towncrier-changelog/blob/1d7332022f76e36fe8ce2d716b851f3f98063c62/check_changelog.py#L261-L268

In the majority of cases I would say that you add only one changelog file and that it makes sense to check the PR number is correct. I would be enclined to use the "No Changelog needed" for now for this kind of egde cases so that we have a better view of how much this happens and whether we want to do something about it.

Labels: Bug, Build / CI
RFC Remove top level indentation from changelog entry files after towncrier

I just saw a PR where the changelog entry didn't have the top level indentation, i.e., it looks like this:

```
This is my multi line
change log
By Author
```

instead of 

```
- This is my multi line
  change log
  By Author
```

And made me wonder, do we really need that top level indentation in every single file? We could make our scripts / towncrier simply add that instead, couldn't we?

cc @lesteve @glemaitre

COMMENT:
I'll check how it renders or if we can configure `towncrier` to have it.

Labels: Documentation, Build / CI
RFC Remove top level indentation from changelog entry files after towncrier

I just saw a PR where the changelog entry didn't have the top level indentation, i.e., it looks like this:

```
This is my multi line
change log
By Author
```

instead of 

```
- This is my multi line
  change log
  By Author
```

And made me wonder, do we really need that top level indentation in every single file? We could make our scripts / towncrier simply add that instead, couldn't we?

cc @lesteve @glemaitre

COMMENT:
I think I thought about something like this at one point, but IIRC it was not straightforward to do this in the towncrier template.

There will be some special cases as well: for custom-top-level section you do not want to add a bullet point.

Labels: Documentation, Build / CI
Add drawings to demonstrate Pipeline, ColumnTransformer, and FeatureUnion

### Describe the issue linked to the documentation

Several classes allow one to build a complete pipeline, namely Pipeline, ColumnTransformer and FeatureUnion. Those are documented at https://scikit-learn.org/stable/modules/compose.html, mostly described individually (as opposed to "compared to each others"). 

While [the first paragraph](https://scikit-learn.org/stable/modules/compose.html#pipelines-and-composite-estimators) briefly describes the use of each of them, it is rather short and more of an introduction rather than a descriptive comparison of use.

I found that the differences between those 3 classes, Pipeline, ColumnTransformer and FeatureUnion, can be hard to grasp in terms of "are the steps applied sequentially ?" or "are the transformed columns concatenated ?". I've seen here and there people writing blog posts stating for example that a ColumnTranformer applies steps sequentially : "a Column Transformer, [like a pipeline], will first apply transformer_1 to Column1, then apply transformer_2 to the transformed version of Column1, and so on". 


### Suggest a potential alternative/fix

Maybe adding some kind of drawings (either using generic "feature1", "feature2" - or using maybe a toy dataset) to demonstrate how they handle features and corresponding steps, making visually explicit that the whole input dataset in passed and applied sequentially in a Pipeline, versus the way features and transformers are mapped in ColumnTransformer and FeatureUnion.

I've jotted down a proof of concept of drawings (using [excalidraw](https://excalidraw.com/) for instance) that I find usefull as a support for explaining their differences.

<img width="492" alt="image" src="https://github.com/user-attachments/assets/66f6f692-bef4-4735-96f1-99b5c1afc1ec">
<img width="479" alt="image" src="https://github.com/user-attachments/assets/9e107be0-e6f9-4687-82a4-4b5d438ba0f7">
<img width="572" alt="image" src="https://github.com/user-attachments/assets/fc571481-dcfa-4dce-96e2-aba8f7d76b8c">

They are defnitely not perfect but it gives the idea.

@glemaitre mentionned some like-drawings were also done for the [scikit-learn MOOC](https://inria.github.io/scikit-learn-mooc/python_scripts/03_categorical_pipeline_column_transformer.html#dispatch-columns-to-a-specific-processor): 

![image](https://github.com/user-attachments/assets/69c7cc83-d742-42d1-beca-2a20f0268d9f)

Another approach could be to just display 3 examples as their html representation side-by-side.

COMMENT:
I think this is great and would love to see a pull request improving our user guides, and links to the guide in the docstrings of those classes wherever appropriate.

Labels: Documentation
Tuning `alpha` in `GaussianProcessRegressor`

### Describe the workflow you want to enable

In the [GaussianProcessRegressor](https://scikit-learn.org/stable/modules/generated/sklearn.gaussian_process.GaussianProcessRegressor.html), `alpha` stands for the likelihood variance of the targets given the inputs: $Y = f(X) + \sqrt{\alpha}\xi$, where $f(X)$ is the GP and $\xi\sim N(\cdot|0, I)$. It is an important hyper-parameter, as it represents the measurement error in the target labels.

Currently, the `fit` does not tune $\alpha$. It only tunes the parameters of the kernel. 

### Describe your proposed solution

Would it be possible to enable the tuning of $\alpha$ when calling `fit`?

### Describe alternatives you've considered, if relevant

Note: one could try to work around the problem by adding `WhiteKernel`, but this is not equivalent to tuning $\alpha$ and retains a different interpretation - see [here](https://gaussianprocess.org/gpml/chapters/RW.pdf), eq. (2.25) and (2.26). Here, $\sigma_n^2$ plays the role of $\alpha$.

### Additional context

_No response_

COMMENT:
@snath-xoc @antoinebaker could you please have a look at this one?

Labels: New Feature, module:gaussian_process, Needs Investigation
Tuning `alpha` in `GaussianProcessRegressor`

### Describe the workflow you want to enable

In the [GaussianProcessRegressor](https://scikit-learn.org/stable/modules/generated/sklearn.gaussian_process.GaussianProcessRegressor.html), `alpha` stands for the likelihood variance of the targets given the inputs: $Y = f(X) + \sqrt{\alpha}\xi$, where $f(X)$ is the GP and $\xi\sim N(\cdot|0, I)$. It is an important hyper-parameter, as it represents the measurement error in the target labels.

Currently, the `fit` does not tune $\alpha$. It only tunes the parameters of the kernel. 

### Describe your proposed solution

Would it be possible to enable the tuning of $\alpha$ when calling `fit`?

### Describe alternatives you've considered, if relevant

Note: one could try to work around the problem by adding `WhiteKernel`, but this is not equivalent to tuning $\alpha$ and retains a different interpretation - see [here](https://gaussianprocess.org/gpml/chapters/RW.pdf), eq. (2.25) and (2.26). Here, $\sigma_n^2$ plays the role of $\alpha$.

### Additional context

_No response_

COMMENT:
> Note: one could try to work around the problem by adding `WhiteKernel`, but this is not equivalent to tuning α and retains a different interpretation - see [here](https://gaussianprocess.org/gpml/chapters/RW.pdf), eq. (2.25) and (2.26). Here, σ n 2 plays the role of α .

Hello @gianlucadetommaso could you clarify why the tuning of $\alpha$ would not be equivalent to using a `WhiteKernel` ?

If I'm not mistaken, the optimal $\alpha$ (whether you put it in the observation noise or the `WhiteKernel`) would be the same, as well as the posterior mean, the only difference would be the posterior variance (with $\alpha$ added using the WhiteKernel). I agree that the interpretation is not the same (what is considered signal vs noise) but in the end you should get the same results by either manually correcting the predicted std `y_std = np.sqrt(y_std**2 - fitted_alpha)`, or if you want to keep the sklearn API, you could reinstantiate a `GaussianProcessRegressor` by passing:
- `kernel` = fitted kernel (without the `WhiteKernel` part)
- `alpha` = fitted `noise_level` of the WhiteKernel
-  refit the GP using `optimizer=None` (to not change the optimal hyperparameters)

Does the workaround seem reasonable to you ?

Labels: New Feature, module:gaussian_process, Needs Investigation
Tuning `alpha` in `GaussianProcessRegressor`

### Describe the workflow you want to enable

In the [GaussianProcessRegressor](https://scikit-learn.org/stable/modules/generated/sklearn.gaussian_process.GaussianProcessRegressor.html), `alpha` stands for the likelihood variance of the targets given the inputs: $Y = f(X) + \sqrt{\alpha}\xi$, where $f(X)$ is the GP and $\xi\sim N(\cdot|0, I)$. It is an important hyper-parameter, as it represents the measurement error in the target labels.

Currently, the `fit` does not tune $\alpha$. It only tunes the parameters of the kernel. 

### Describe your proposed solution

Would it be possible to enable the tuning of $\alpha$ when calling `fit`?

### Describe alternatives you've considered, if relevant

Note: one could try to work around the problem by adding `WhiteKernel`, but this is not equivalent to tuning $\alpha$ and retains a different interpretation - see [here](https://gaussianprocess.org/gpml/chapters/RW.pdf), eq. (2.25) and (2.26). Here, $\sigma_n^2$ plays the role of $\alpha$.

### Additional context

_No response_

COMMENT:
So let's take the posterior predictive mean from eq. (2.23): $f^\*=K(X^\*,X)(K(X, X) + \sigma_n^2I)^{-1}y$. 

If $X=X^\*$ and we added a `WhiteKernel` with noise level $w$, we would get $f=(K(X,X) + w)(K(X, X) + (\sigma_n^2 + w)I)^{-1}y$. If instead we added $w$ to $\sigma_n^2$, we would get $f^\*=K(X^\*,X)(K(X, X) + (\sigma_n^2+w)I)^{-1}y$.

This means that using a `WhiteKernel` or tuning the `alpha` parameter (in this case called $\sigma_n^2$) is not equivalent. 


In line with what you say, one could first fit a GP with `alpha=1e-10` (default), then set $\alpha:=\mathbb{E}[(Y-f(X))^2]$, where $f(X)$ is the posterior predictive mean, then re-fit a GP using the new $\alpha$ parameter instead. This is sensible, however it is not equivalent to optimise $\alpha$ because the posterior mean also depends on $\alpha$! I guess it is possible to show that, iteratively, the approach converges, but it seems like a unjustified workaround. Why not tuning `alpha` together with the other hyper-parameters?

Labels: New Feature, module:gaussian_process, Needs Investigation
Tuning `alpha` in `GaussianProcessRegressor`

### Describe the workflow you want to enable

In the [GaussianProcessRegressor](https://scikit-learn.org/stable/modules/generated/sklearn.gaussian_process.GaussianProcessRegressor.html), `alpha` stands for the likelihood variance of the targets given the inputs: $Y = f(X) + \sqrt{\alpha}\xi$, where $f(X)$ is the GP and $\xi\sim N(\cdot|0, I)$. It is an important hyper-parameter, as it represents the measurement error in the target labels.

Currently, the `fit` does not tune $\alpha$. It only tunes the parameters of the kernel. 

### Describe your proposed solution

Would it be possible to enable the tuning of $\alpha$ when calling `fit`?

### Describe alternatives you've considered, if relevant

Note: one could try to work around the problem by adding `WhiteKernel`, but this is not equivalent to tuning $\alpha$ and retains a different interpretation - see [here](https://gaussianprocess.org/gpml/chapters/RW.pdf), eq. (2.25) and (2.26). Here, $\sigma_n^2$ plays the role of $\alpha$.

### Additional context

_No response_

COMMENT:
It would be great if you could provide some more details on your derivations @gianlucadetommaso . When I ran through it I came up with this:

Adding the `WhiteKernel` to any other kernel we get an overall kernel of $$k(x_i, y_j) + \sigma^2_n \delta(x_i, y_j)$$

The way this is then evaluated, the value added to the diagonal $\alpha$ is only added to the K(X, X) kernel matrix [on line 2 of Algorithm 2.1.](https://github.com/scikit-learn/scikit-learn/blob/7387c267e4c57de2ffd118ca9df85576b01df11c/sklearn/gaussian_process/_gpr.py#L342-L344)
```python
        # Alg. 2.1, page 19, line 2 -> L = cholesky(K + sigma^2 I)
        K = self.kernel_(self.X_train_)
        K[np.diag_indices_from(K)] += self.alpha
```

In essence, the joint distribution takes the form
![joint_gp_2](https://github.com/user-attachments/assets/c56c7de9-c12d-4880-bb56-16157657a4dd)


where $(X, y)$ are the training data and $(X_{\*}, f_{\*})$ the evaluation predictions. 

The predictive equations become

![predictive_gp](https://github.com/user-attachments/assets/16a59bd3-d474-40ef-a2eb-85a120d01326)

I would thus tend to agree that other than a manual correction of the variance, these two approaches should be the same.

Labels: New Feature, module:gaussian_process, Needs Investigation
Tuning `alpha` in `GaussianProcessRegressor`

### Describe the workflow you want to enable

In the [GaussianProcessRegressor](https://scikit-learn.org/stable/modules/generated/sklearn.gaussian_process.GaussianProcessRegressor.html), `alpha` stands for the likelihood variance of the targets given the inputs: $Y = f(X) + \sqrt{\alpha}\xi$, where $f(X)$ is the GP and $\xi\sim N(\cdot|0, I)$. It is an important hyper-parameter, as it represents the measurement error in the target labels.

Currently, the `fit` does not tune $\alpha$. It only tunes the parameters of the kernel. 

### Describe your proposed solution

Would it be possible to enable the tuning of $\alpha$ when calling `fit`?

### Describe alternatives you've considered, if relevant

Note: one could try to work around the problem by adding `WhiteKernel`, but this is not equivalent to tuning $\alpha$ and retains a different interpretation - see [here](https://gaussianprocess.org/gpml/chapters/RW.pdf), eq. (2.25) and (2.26). Here, $\sigma_n^2$ plays the role of $\alpha$.

### Additional context

_No response_

COMMENT:
@JakobBull I agree with your equations, but I think there is a term $\alpha\delta_{X_\*, X}$ added to $K(X_\*, X)$ missing. What happens in the case $X_*=X$? 

If we have a kernel $K$ and add a white kernel, we get a kernel $\tilde{K}(X_*, X) = K(X_\*, X) + \alpha\delta_{X_\*, X}$.

Therefore, the posterior predictive mean $\bar{f}_\*$ using the kernel $\tilde{K}$ would be

$$  \tilde{K}(X_\*, X)[\tilde{K}(X, X) + \sigma_n^2I]^{-1}y = (K(X_\*, X)+\alpha\delta_{X_\*, X})[K(X, X) + (\sigma_n^2+\alpha)I]^{-1}y $$

On the other hand, if we used just the kernel $K$ but we used a likelihood variance $\tilde{\sigma}_n^2=\sigma_n^2+\alpha$, we would get

$$  K(X_\*, X)[K(X, X) + \tilde{\sigma}^2_n I]^{-1}y = K(X_\*, X)[K(X, X) + (\sigma_n^2+\alpha)I]^{-1}y $$

These are not exactly the same - there is a term $\alpha\delta_{X_\*, X}$ appearing only in one case. The same can be argued for the posterior predictive covariance.


However, from a practical perspective, for a continuous input distribution it's true that nothing changes because the case $X=X^*$ never happens with positive probability. But if the inputs were taken on a grid, I still think the two approaches don't give the same quantity.

Labels: New Feature, module:gaussian_process, Needs Investigation
Tuning `alpha` in `GaussianProcessRegressor`

### Describe the workflow you want to enable

In the [GaussianProcessRegressor](https://scikit-learn.org/stable/modules/generated/sklearn.gaussian_process.GaussianProcessRegressor.html), `alpha` stands for the likelihood variance of the targets given the inputs: $Y = f(X) + \sqrt{\alpha}\xi$, where $f(X)$ is the GP and $\xi\sim N(\cdot|0, I)$. It is an important hyper-parameter, as it represents the measurement error in the target labels.

Currently, the `fit` does not tune $\alpha$. It only tunes the parameters of the kernel. 

### Describe your proposed solution

Would it be possible to enable the tuning of $\alpha$ when calling `fit`?

### Describe alternatives you've considered, if relevant

Note: one could try to work around the problem by adding `WhiteKernel`, but this is not equivalent to tuning $\alpha$ and retains a different interpretation - see [here](https://gaussianprocess.org/gpml/chapters/RW.pdf), eq. (2.25) and (2.26). Here, $\sigma_n^2$ plays the role of $\alpha$.

### Additional context

_No response_

COMMENT:
The tuning of the hyperparameters is done by maximizing the log marginal likelihood (2.30).

Let me clarify my answer, sorry if it was confusing, I am considering two possible `GaussianProcessRegressor` instances:
1. base kernel `k` + `WhiteKernel` with `noise_level` = $\sigma^2$ but `alpha` = 0.
2. only base kernel `k` but `alpha` = $\sigma^2$

where the "base" kernel `k` stands for the kernel used to model the prior covariance (eg a RBF etc...).

In your original question, I think you want version 2 with the possibility to tune $\sigma^2$ (as well as the kernel `k` hyperparameters).

Notice that versions 1 and 2 lead to the exact same log marginal likelihood (2.30) [where $K$ comes from the base kernel `k` and the $\sigma^2 I$ term comes from the `WhiteKernel` in version 1 and from `alpha` in version 2] and therefore leads to the same optimal values, thus the tuning of hyperparameters is equivalent if I'm not mistaken.

Then the proposed workaround should work: tune the hyperparameters using version 1 (because on this one you can optimize over `noise_level` and gives the same optimal hyperparameters in the end), and reinstantiate version 2 with the fitted hyperparameters (because this one gives you the GP you actually want, with the correct  posterior mean (2.25) and variance (2.26)).

Labels: New Feature, module:gaussian_process, Needs Investigation
Tuning `alpha` in `GaussianProcessRegressor`

### Describe the workflow you want to enable

In the [GaussianProcessRegressor](https://scikit-learn.org/stable/modules/generated/sklearn.gaussian_process.GaussianProcessRegressor.html), `alpha` stands for the likelihood variance of the targets given the inputs: $Y = f(X) + \sqrt{\alpha}\xi$, where $f(X)$ is the GP and $\xi\sim N(\cdot|0, I)$. It is an important hyper-parameter, as it represents the measurement error in the target labels.

Currently, the `fit` does not tune $\alpha$. It only tunes the parameters of the kernel. 

### Describe your proposed solution

Would it be possible to enable the tuning of $\alpha$ when calling `fit`?

### Describe alternatives you've considered, if relevant

Note: one could try to work around the problem by adding `WhiteKernel`, but this is not equivalent to tuning $\alpha$ and retains a different interpretation - see [here](https://gaussianprocess.org/gpml/chapters/RW.pdf), eq. (2.25) and (2.26). Here, $\sigma_n^2$ plays the role of $\alpha$.

### Additional context

_No response_

COMMENT:
For the differences in the posterior mean for versions 1 and 2 you're correct @gianlucadetommaso that there is, _mathematically_ speaking, a difference only when the test point $x^*$ is exactly equal to the training point $x_i$,  the version 1 posterior mean actually yields the observation $y_i$ (and version 1 with `alpha`=0 is indeed an interpolation setting).

But you're correct @JakobBull that _numerically_ speaking, in the sklearn implementation version 1 and version 2 `predict` methods gives the exact same posterior mean.

As you mentioned @gianlucadetommaso  this means that the posterior mean in version 1 is mathematically incorrect when $x^* = x_i$, but the `predict` method cannot sensibly handle such cases (two real numbers being exactly equal is not possible to test in a floating point arithmetic system). In the fixed grid designs you mentioned you must indeed be careful, but then the workaround still work.

Labels: New Feature, module:gaussian_process, Needs Investigation
Tuning `alpha` in `GaussianProcessRegressor`

### Describe the workflow you want to enable

In the [GaussianProcessRegressor](https://scikit-learn.org/stable/modules/generated/sklearn.gaussian_process.GaussianProcessRegressor.html), `alpha` stands for the likelihood variance of the targets given the inputs: $Y = f(X) + \sqrt{\alpha}\xi$, where $f(X)$ is the GP and $\xi\sim N(\cdot|0, I)$. It is an important hyper-parameter, as it represents the measurement error in the target labels.

Currently, the `fit` does not tune $\alpha$. It only tunes the parameters of the kernel. 

### Describe your proposed solution

Would it be possible to enable the tuning of $\alpha$ when calling `fit`?

### Describe alternatives you've considered, if relevant

Note: one could try to work around the problem by adding `WhiteKernel`, but this is not equivalent to tuning $\alpha$ and retains a different interpretation - see [here](https://gaussianprocess.org/gpml/chapters/RW.pdf), eq. (2.25) and (2.26). Here, $\sigma_n^2$ plays the role of $\alpha$.

### Additional context

_No response_

COMMENT:
Thanks for the discussion here. As I read through, it seems we're okay in our implementation. So I think we can close this issue. However, this is the kind of material which we might also include in our documentation / user guides. Would be nice if one of you would add a summary of this discussion to the docs for future users.

Labels: New Feature, module:gaussian_process, Needs Investigation