Talip7/scikit-learn-issues-multilabel huggingface dataset:
Add Accumulated local effects (ALE) to inspection

### Describe the workflow you want to enable

I'd love to push the inspection module further by adding Accumulated local effects (ALE) from Apley 2020. A great description can be found in Christoph's online book https://christophm.github.io/interpretable-ml-book/ale.html

ALE fix the problem of partial dependence that they force the model to be evaluated on impossible or rare feature combinations. ALE are defined for numeric features that can be binned. From both bin edges, the slope of the partial dependence is calculated locally, i.e., only using observations in the bin. The slopes from all bins are cumsummed and vertically centered to the average response or prediction.

### Reference

Apley, Daniel W., and Jingyu Zhu. 2020. Visualizing the Effects of Predictor Variables in Black Box Supervised Learning Models. Journal of the Royal Statistical Society Series B: Statistical Methodology, 82 (4): 1059–1086. doi:10.1111/rssb.12377.

### Describe your proposed solution

Pseudo code to calculate ALE for one feature:

``` py
pd_slopes = np.zeros_like(bins)

for i, bin in enumerate(bins):
  X_bin = pick n_per_bin = 200 rows from data in bin
  if X_bin is empty:
    next
  pd = partial_dependence_brute(model, X_bin, grid = [lower bin edge, upper bin edge], sample_weights)
  pd_slopes[i] = pd[1] - pd[0]

return np.cumsum(pd_slopes)
```

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_

COMMENT:
I am also interested in adding this function to scikit-learn.
I have time to help you with a pull request or I can create one if the decision is positive to include this new functionality.

Labels: New Feature, Needs Decision
Add Accumulated local effects (ALE) to inspection

### Describe the workflow you want to enable

I'd love to push the inspection module further by adding Accumulated local effects (ALE) from Apley 2020. A great description can be found in Christoph's online book https://christophm.github.io/interpretable-ml-book/ale.html

ALE fix the problem of partial dependence that they force the model to be evaluated on impossible or rare feature combinations. ALE are defined for numeric features that can be binned. From both bin edges, the slope of the partial dependence is calculated locally, i.e., only using observations in the bin. The slopes from all bins are cumsummed and vertically centered to the average response or prediction.

### Reference

Apley, Daniel W., and Jingyu Zhu. 2020. Visualizing the Effects of Predictor Variables in Black Box Supervised Learning Models. Journal of the Royal Statistical Society Series B: Statistical Methodology, 82 (4): 1059–1086. doi:10.1111/rssb.12377.

### Describe your proposed solution

Pseudo code to calculate ALE for one feature:

``` py
pd_slopes = np.zeros_like(bins)

for i, bin in enumerate(bins):
  X_bin = pick n_per_bin = 200 rows from data in bin
  if X_bin is empty:
    next
  pd = partial_dependence_brute(model, X_bin, grid = [lower bin edge, upper bin edge], sample_weights)
  pd_slopes[i] = pd[1] - pd[0]

return np.cumsum(pd_slopes)
```

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_

COMMENT:
I have not started yet as I wanted to see this related PR https://github.com/scikit-learn/scikit-learn/pull/28375 on Friedman's H^2 statistic being merged first.

Still, here some aspects the implementation would need to take care of:

- multivariate output (easy)
- pandas/polars/numpy input with and without column names 
- with or without plots (I'd tend to "without")
- optional case weights (easy)

Labels: New Feature, Needs Decision
Add Accumulated local effects (ALE) to inspection

### Describe the workflow you want to enable

I'd love to push the inspection module further by adding Accumulated local effects (ALE) from Apley 2020. A great description can be found in Christoph's online book https://christophm.github.io/interpretable-ml-book/ale.html

ALE fix the problem of partial dependence that they force the model to be evaluated on impossible or rare feature combinations. ALE are defined for numeric features that can be binned. From both bin edges, the slope of the partial dependence is calculated locally, i.e., only using observations in the bin. The slopes from all bins are cumsummed and vertically centered to the average response or prediction.

### Reference

Apley, Daniel W., and Jingyu Zhu. 2020. Visualizing the Effects of Predictor Variables in Black Box Supervised Learning Models. Journal of the Royal Statistical Society Series B: Statistical Methodology, 82 (4): 1059–1086. doi:10.1111/rssb.12377.

### Describe your proposed solution

Pseudo code to calculate ALE for one feature:

``` py
pd_slopes = np.zeros_like(bins)

for i, bin in enumerate(bins):
  X_bin = pick n_per_bin = 200 rows from data in bin
  if X_bin is empty:
    next
  pd = partial_dependence_brute(model, X_bin, grid = [lower bin edge, upper bin edge], sample_weights)
  pd_slopes[i] = pd[1] - pd[0]

return np.cumsum(pd_slopes)
```

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_

COMMENT:
I have started a first version of the implementation of ALE. I still need to improve it and I need to include the plot functions, tests and examples. 
You can find it here: https://github.com/lionelkusch/scikit-learn/tree/PR_ALE

I have based my implementation on the code [ALEpython](https://github.com/blent-ai/ALEPython/) and the implementation of partial dependence function.

Before continuing, I have some questions:
- Do we include the ALE at the second order or do we stay only with the first order?
- I don't understand the way to use weights in the context of ALE. If you can provide me an example of it, I will include it. 
- From the technical part, there will be some parts of the code shared with the partial_dependence function. For the moment, I duplicate the code, but I don't think it's the best solution. The only proposition that I have for the moment is to provide a function for the common initialisation between the two functions, but I don't really like. If you have some ideas, I will be happy to hear them.

Labels: New Feature, Needs Decision
Add Accumulated local effects (ALE) to inspection

### Describe the workflow you want to enable

I'd love to push the inspection module further by adding Accumulated local effects (ALE) from Apley 2020. A great description can be found in Christoph's online book https://christophm.github.io/interpretable-ml-book/ale.html

ALE fix the problem of partial dependence that they force the model to be evaluated on impossible or rare feature combinations. ALE are defined for numeric features that can be binned. From both bin edges, the slope of the partial dependence is calculated locally, i.e., only using observations in the bin. The slopes from all bins are cumsummed and vertically centered to the average response or prediction.

### Reference

Apley, Daniel W., and Jingyu Zhu. 2020. Visualizing the Effects of Predictor Variables in Black Box Supervised Learning Models. Journal of the Royal Statistical Society Series B: Statistical Methodology, 82 (4): 1059–1086. doi:10.1111/rssb.12377.

### Describe your proposed solution

Pseudo code to calculate ALE for one feature:

``` py
pd_slopes = np.zeros_like(bins)

for i, bin in enumerate(bins):
  X_bin = pick n_per_bin = 200 rows from data in bin
  if X_bin is empty:
    next
  pd = partial_dependence_brute(model, X_bin, grid = [lower bin edge, upper bin edge], sample_weights)
  pd_slopes[i] = pd[1] - pd[0]

return np.cumsum(pd_slopes)
```

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_

COMMENT:
Thanks for the initial work! Let's hear what the Scikit-Learn developers mean. My focus is still to first have a solution for https://github.com/scikit-learn/scikit-learn/pull/28375 and then care about ALE. 

We need to be patient with the inspection module.

Labels: New Feature, Needs Decision
Add Accumulated local effects (ALE) to inspection

### Describe the workflow you want to enable

I'd love to push the inspection module further by adding Accumulated local effects (ALE) from Apley 2020. A great description can be found in Christoph's online book https://christophm.github.io/interpretable-ml-book/ale.html

ALE fix the problem of partial dependence that they force the model to be evaluated on impossible or rare feature combinations. ALE are defined for numeric features that can be binned. From both bin edges, the slope of the partial dependence is calculated locally, i.e., only using observations in the bin. The slopes from all bins are cumsummed and vertically centered to the average response or prediction.

### Reference

Apley, Daniel W., and Jingyu Zhu. 2020. Visualizing the Effects of Predictor Variables in Black Box Supervised Learning Models. Journal of the Royal Statistical Society Series B: Statistical Methodology, 82 (4): 1059–1086. doi:10.1111/rssb.12377.

### Describe your proposed solution

Pseudo code to calculate ALE for one feature:

``` py
pd_slopes = np.zeros_like(bins)

for i, bin in enumerate(bins):
  X_bin = pick n_per_bin = 200 rows from data in bin
  if X_bin is empty:
    next
  pd = partial_dependence_brute(model, X_bin, grid = [lower bin edge, upper bin edge], sample_weights)
  pd_slopes[i] = pd[1] - pd[0]

return np.cumsum(pd_slopes)
```

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_

COMMENT:
In my opinion, starting with a version for one continuous feature and without any plot would be sufficient. E.g., `permutation_importance()` only calculates things without plots. The output could be two np.arrays: one with bin breaks, one with corresponding ALE values.

Labels: New Feature, Needs Decision
Add Accumulated local effects (ALE) to inspection

### Describe the workflow you want to enable

I'd love to push the inspection module further by adding Accumulated local effects (ALE) from Apley 2020. A great description can be found in Christoph's online book https://christophm.github.io/interpretable-ml-book/ale.html

ALE fix the problem of partial dependence that they force the model to be evaluated on impossible or rare feature combinations. ALE are defined for numeric features that can be binned. From both bin edges, the slope of the partial dependence is calculated locally, i.e., only using observations in the bin. The slopes from all bins are cumsummed and vertically centered to the average response or prediction.

### Reference

Apley, Daniel W., and Jingyu Zhu. 2020. Visualizing the Effects of Predictor Variables in Black Box Supervised Learning Models. Journal of the Royal Statistical Society Series B: Statistical Methodology, 82 (4): 1059–1086. doi:10.1111/rssb.12377.

### Describe your proposed solution

Pseudo code to calculate ALE for one feature:

``` py
pd_slopes = np.zeros_like(bins)

for i, bin in enumerate(bins):
  X_bin = pick n_per_bin = 200 rows from data in bin
  if X_bin is empty:
    next
  pd = partial_dependence_brute(model, X_bin, grid = [lower bin edge, upper bin edge], sample_weights)
  pd_slopes[i] = pd[1] - pd[0]

return np.cumsum(pd_slopes)
```

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_

COMMENT:
This is a draft of an (almost...) Scikit-Learn compatible implementation:

https://github.com/mayer79/accumulated_local_effects

```py
import matplotlib.pyplot as plt
import numpy as np
import pandas as pd
from ale import accumulated_local_effects
from sklearn.compose import ColumnTransformer
from sklearn.linear_model import LinearRegression
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import SplineTransformer
rng = np.random.default_rng(1)

age = rng.uniform(0, 40, size=1000)
living_area = rng.uniform(40, 150, size=1000)
X = pd.DataFrame({"age": age, "living_area": living_area})

rent = 20 * living_area - 20 * age + (age - 20) ** 2 + rng.normal(0, 10, size=1000)

preprocess_glm = ColumnTransformer(
    transformers=[
        ("spline", SplineTransformer(include_bias=False, knots="quantile"), ["age"]),
        ("linear", "passthrough", ["living_area"]),
    ],
    verbose_feature_names_out=False,
).set_output(transform="pandas")

model = Pipeline(
    steps=[
        ("preprocessor", preprocess_glm),
        ("model", LinearRegression()),
    ]
)
model.fit(X, y=rent)

fig, axes = plt.subplots(1, 2, figsize=(12, 5))

for feature, ax in zip(("age", "living_area"), axes):
    ale = accumulated_local_effects(X, model.predict, feature=feature)
    ax.plot(ale["breaks"], ale["ale"])
    ax.set_title(f"{feature}")
plt.show()
```

![Image](https://github.com/user-attachments/assets/aa8ac734-ce6c-4473-a0eb-0a6610620c64)

Labels: New Feature, Needs Decision
Add Accumulated local effects (ALE) to inspection

### Describe the workflow you want to enable

I'd love to push the inspection module further by adding Accumulated local effects (ALE) from Apley 2020. A great description can be found in Christoph's online book https://christophm.github.io/interpretable-ml-book/ale.html

ALE fix the problem of partial dependence that they force the model to be evaluated on impossible or rare feature combinations. ALE are defined for numeric features that can be binned. From both bin edges, the slope of the partial dependence is calculated locally, i.e., only using observations in the bin. The slopes from all bins are cumsummed and vertically centered to the average response or prediction.

### Reference

Apley, Daniel W., and Jingyu Zhu. 2020. Visualizing the Effects of Predictor Variables in Black Box Supervised Learning Models. Journal of the Royal Statistical Society Series B: Statistical Methodology, 82 (4): 1059–1086. doi:10.1111/rssb.12377.

### Describe your proposed solution

Pseudo code to calculate ALE for one feature:

``` py
pd_slopes = np.zeros_like(bins)

for i, bin in enumerate(bins):
  X_bin = pick n_per_bin = 200 rows from data in bin
  if X_bin is empty:
    next
  pd = partial_dependence_brute(model, X_bin, grid = [lower bin edge, upper bin edge], sample_weights)
  pd_slopes[i] = pd[1] - pd[0]

return np.cumsum(pd_slopes)
```

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_

COMMENT:
Can you compare the result with this code?
https://github.com/lionelkusch/scikit-learn/blob/PR_ALE/sklearn/inspection/_accumulated_local_effects.py

Labels: New Feature, Needs Decision
Add Accumulated local effects (ALE) to inspection

### Describe the workflow you want to enable

I'd love to push the inspection module further by adding Accumulated local effects (ALE) from Apley 2020. A great description can be found in Christoph's online book https://christophm.github.io/interpretable-ml-book/ale.html

ALE fix the problem of partial dependence that they force the model to be evaluated on impossible or rare feature combinations. ALE are defined for numeric features that can be binned. From both bin edges, the slope of the partial dependence is calculated locally, i.e., only using observations in the bin. The slopes from all bins are cumsummed and vertically centered to the average response or prediction.

### Reference

Apley, Daniel W., and Jingyu Zhu. 2020. Visualizing the Effects of Predictor Variables in Black Box Supervised Learning Models. Journal of the Royal Statistical Society Series B: Statistical Methodology, 82 (4): 1059–1086. doi:10.1111/rssb.12377.

### Describe your proposed solution

Pseudo code to calculate ALE for one feature:

``` py
pd_slopes = np.zeros_like(bins)

for i, bin in enumerate(bins):
  X_bin = pick n_per_bin = 200 rows from data in bin
  if X_bin is empty:
    next
  pd = partial_dependence_brute(model, X_bin, grid = [lower bin edge, upper bin edge], sample_weights)
  pd_slopes[i] = pd[1] - pd[0]

return np.cumsum(pd_slopes)
```

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_

COMMENT:
No, there was a bug in the implementation. I am sorry for it. 
I just fix it.

I get the same result:

![Image](https://github.com/user-attachments/assets/288f3ca1-5182-4672-9115-f5ab5fff25bb)

Labels: New Feature, Needs Decision
Add Accumulated local effects (ALE) to inspection

### Describe the workflow you want to enable

I'd love to push the inspection module further by adding Accumulated local effects (ALE) from Apley 2020. A great description can be found in Christoph's online book https://christophm.github.io/interpretable-ml-book/ale.html

ALE fix the problem of partial dependence that they force the model to be evaluated on impossible or rare feature combinations. ALE are defined for numeric features that can be binned. From both bin edges, the slope of the partial dependence is calculated locally, i.e., only using observations in the bin. The slopes from all bins are cumsummed and vertically centered to the average response or prediction.

### Reference

Apley, Daniel W., and Jingyu Zhu. 2020. Visualizing the Effects of Predictor Variables in Black Box Supervised Learning Models. Journal of the Royal Statistical Society Series B: Statistical Methodology, 82 (4): 1059–1086. doi:10.1111/rssb.12377.

### Describe your proposed solution

Pseudo code to calculate ALE for one feature:

``` py
pd_slopes = np.zeros_like(bins)

for i, bin in enumerate(bins):
  X_bin = pick n_per_bin = 200 rows from data in bin
  if X_bin is empty:
    next
  pd = partial_dependence_brute(model, X_bin, grid = [lower bin edge, upper bin edge], sample_weights)
  pd_slopes[i] = pd[1] - pd[0]

return np.cumsum(pd_slopes)
```

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_

COMMENT:
I was playing a bit with your implementation, and sometimes stumble over unequal sized quantiles and ale:

```py
import matplotlib.pyplot as plt
import numpy as np
from sklearn.datasets import load_diabetes
from sklearn.linear_model import LinearRegression

X, y = load_diabetes(return_X_y=True, as_frame=True)

est = LinearRegression().fit(X, y)

ale = accumulated_local_effects2(est, X, features="bmi")
plt.plot(np.asarray(ale["center_quantile"]).flatten(), np.asarray(ale["ale"]).flatten())
```

Gives `ValueError: x and y must have same first dimension, but have shapes (99,) and (91,)`

Labels: New Feature, Needs Decision
Add Accumulated local effects (ALE) to inspection

### Describe the workflow you want to enable

I'd love to push the inspection module further by adding Accumulated local effects (ALE) from Apley 2020. A great description can be found in Christoph's online book https://christophm.github.io/interpretable-ml-book/ale.html

ALE fix the problem of partial dependence that they force the model to be evaluated on impossible or rare feature combinations. ALE are defined for numeric features that can be binned. From both bin edges, the slope of the partial dependence is calculated locally, i.e., only using observations in the bin. The slopes from all bins are cumsummed and vertically centered to the average response or prediction.

### Reference

Apley, Daniel W., and Jingyu Zhu. 2020. Visualizing the Effects of Predictor Variables in Black Box Supervised Learning Models. Journal of the Royal Statistical Society Series B: Statistical Methodology, 82 (4): 1059–1086. doi:10.1111/rssb.12377.

### Describe your proposed solution

Pseudo code to calculate ALE for one feature:

``` py
pd_slopes = np.zeros_like(bins)

for i, bin in enumerate(bins):
  X_bin = pick n_per_bin = 200 rows from data in bin
  if X_bin is empty:
    next
  pd = partial_dependence_brute(model, X_bin, grid = [lower bin edge, upper bin edge], sample_weights)
  pd_slopes[i] = pd[1] - pd[0]

return np.cumsum(pd_slopes)
```

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_

COMMENT:
The problem is when there is any point in some quantiles.
What value do you think that we should provide? Or Do we remove these intervals from the result?

Labels: New Feature, Needs Decision
Add Accumulated local effects (ALE) to inspection

### Describe the workflow you want to enable

I'd love to push the inspection module further by adding Accumulated local effects (ALE) from Apley 2020. A great description can be found in Christoph's online book https://christophm.github.io/interpretable-ml-book/ale.html

ALE fix the problem of partial dependence that they force the model to be evaluated on impossible or rare feature combinations. ALE are defined for numeric features that can be binned. From both bin edges, the slope of the partial dependence is calculated locally, i.e., only using observations in the bin. The slopes from all bins are cumsummed and vertically centered to the average response or prediction.

### Reference

Apley, Daniel W., and Jingyu Zhu. 2020. Visualizing the Effects of Predictor Variables in Black Box Supervised Learning Models. Journal of the Royal Statistical Society Series B: Statistical Methodology, 82 (4): 1059–1086. doi:10.1111/rssb.12377.

### Describe your proposed solution

Pseudo code to calculate ALE for one feature:

``` py
pd_slopes = np.zeros_like(bins)

for i, bin in enumerate(bins):
  X_bin = pick n_per_bin = 200 rows from data in bin
  if X_bin is empty:
    next
  pd = partial_dependence_brute(model, X_bin, grid = [lower bin edge, upper bin edge], sample_weights)
  pd_slopes[i] = pd[1] - pd[0]

return np.cumsum(pd_slopes)
```

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_

COMMENT:
The bins are empty?

Labels: New Feature, Needs Decision
Add Accumulated local effects (ALE) to inspection

### Describe the workflow you want to enable

I'd love to push the inspection module further by adding Accumulated local effects (ALE) from Apley 2020. A great description can be found in Christoph's online book https://christophm.github.io/interpretable-ml-book/ale.html

ALE fix the problem of partial dependence that they force the model to be evaluated on impossible or rare feature combinations. ALE are defined for numeric features that can be binned. From both bin edges, the slope of the partial dependence is calculated locally, i.e., only using observations in the bin. The slopes from all bins are cumsummed and vertically centered to the average response or prediction.

### Reference

Apley, Daniel W., and Jingyu Zhu. 2020. Visualizing the Effects of Predictor Variables in Black Box Supervised Learning Models. Journal of the Royal Statistical Society Series B: Statistical Methodology, 82 (4): 1059–1086. doi:10.1111/rssb.12377.

### Describe your proposed solution

Pseudo code to calculate ALE for one feature:

``` py
pd_slopes = np.zeros_like(bins)

for i, bin in enumerate(bins):
  X_bin = pick n_per_bin = 200 rows from data in bin
  if X_bin is empty:
    next
  pd = partial_dependence_brute(model, X_bin, grid = [lower bin edge, upper bin edge], sample_weights)
  pd_slopes[i] = pd[1] - pd[0]

return np.cumsum(pd_slopes)
```

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_

COMMENT:
PyALE and you are using [quantile](https://numpy.org/doc/stable/reference/generated/numpy.quantile.html) from numpy. This function provides the value of quantiles from the data. In this case, using function `unique`, you remove all the empty quantiles and reduce the number of bins.
In my case, I use [mquantiles](https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.mstats.mquantiles.html) for calculating the quantiles. This function provides an estimation of the quantile. I can't use the function `unique` for removing empty ones because my quantile is not based on the data. However, I returned the number of bins expected.

Labels: New Feature, Needs Decision