Talip7/scikit-learn-issues-multilabel huggingface dataset:
Clarification on Kruskal Stress as an Optimization Target in Metric and Non-metric MDS

### Describe the issue linked to the documentation

I am working on research involving the optimization targets used in metric and non-metric MDS, and I have some questions regarding how scikit-learn's implementation of MDS defines and calculates stress, particularly Kruskal Stress. While reviewing the official documentation, I noticed that specific formulas for stress calculations are not explicitly provided, and I would appreciate some clarification.

Non-metric MDS: My understanding is that non-metric MDS typically minimizes Kruskal Stress, defined as:

![屏幕截图 2024-11-08 163808](https://github.com/user-attachments/assets/d289f4f0-afa0-4c80-a982-c2186c103e27)

in the reduced space. Could you confirm if scikit-learn's non-metric MDS implementation uses this definition, or if it employs an alternative method?

Metric MDS: Does metric MDS in scikit-learn also optimize for Kruskal Stress, or does it use a different stress formula? If a different approach is used, would it be possible to provide some insight or references on the stress function applied here?


### Suggest a potential alternative/fix


Documentation Clarification: It would be incredibly helpful if the documentation could include specific details on the stress formulas used in both metric and non-metric MDS. This addition would help researchers and users better understand the theoretical underpinnings of the algorithm in scikit-learn.
Thank you very much for your guidance and clarification on these points. Your insights would be instrumental in my work with MDS.

COMMENT:
> My understanding is that non-metric MDS typically minimizes Kruskal Stress

No -- the stress formula is optimized by *metric MDS*! Note that the square root and the normalization by the summed squared original distances do not change the loss minimum, so essentially the loss is simply squared error between high-dim and low-dim distances. That's minimized by metric MDS.

Non-metric MDS allows arbitrary monotonic transformation of low-dim distances, please see here: https://en.wikipedia.org/wiki/Multidimensional_scaling#Non-metric_multidimensional_scaling_(NMDS).

Note that I think non-metric MDS in sklearn is broken: https://github.com/scikit-learn/scikit-learn/issues/27028

Labels: Documentation, Needs Triage
Clarification on Kruskal Stress as an Optimization Target in Metric and Non-metric MDS

### Describe the issue linked to the documentation

I am working on research involving the optimization targets used in metric and non-metric MDS, and I have some questions regarding how scikit-learn's implementation of MDS defines and calculates stress, particularly Kruskal Stress. While reviewing the official documentation, I noticed that specific formulas for stress calculations are not explicitly provided, and I would appreciate some clarification.

Non-metric MDS: My understanding is that non-metric MDS typically minimizes Kruskal Stress, defined as:

![屏幕截图 2024-11-08 163808](https://github.com/user-attachments/assets/d289f4f0-afa0-4c80-a982-c2186c103e27)

in the reduced space. Could you confirm if scikit-learn's non-metric MDS implementation uses this definition, or if it employs an alternative method?

Metric MDS: Does metric MDS in scikit-learn also optimize for Kruskal Stress, or does it use a different stress formula? If a different approach is used, would it be possible to provide some insight or references on the stress function applied here?


### Suggest a potential alternative/fix


Documentation Clarification: It would be incredibly helpful if the documentation could include specific details on the stress formulas used in both metric and non-metric MDS. This addition would help researchers and users better understand the theoretical underpinnings of the algorithm in scikit-learn.
Thank you very much for your guidance and clarification on these points. Your insights would be instrumental in my work with MDS.

COMMENT:
Wikipedia's formula is the same as the formulas in Kruskal 1964 papers, but the notation is different.

In Kruskal's papers, $d_{ij}$ are the embedding distances, and $\hat d_{ij}$ are some monotonic trasformation of the original distances which he calls $\delta_{ij}$. The optimization happens over embedding coordinates AND ALSO over all possible monotonic functions.

In Wikipedia, $\hat d_{ij}$ are the embedding distances, and $f(d_{ij})$ is the monotonic transformation of the original distances $d_{ij}$.

Either way, this is non-metric MDS.

If you leave out the monotonic transformation, you get "metric MDS". But I don't think this is explicitly called like that in Kruskal 1964 papers. In fact, I don't know who first coined the term "metric MDS".

Labels: Documentation, Needs Triage
Clarification on Kruskal Stress as an Optimization Target in Metric and Non-metric MDS

### Describe the issue linked to the documentation

I am working on research involving the optimization targets used in metric and non-metric MDS, and I have some questions regarding how scikit-learn's implementation of MDS defines and calculates stress, particularly Kruskal Stress. While reviewing the official documentation, I noticed that specific formulas for stress calculations are not explicitly provided, and I would appreciate some clarification.

Non-metric MDS: My understanding is that non-metric MDS typically minimizes Kruskal Stress, defined as:

![屏幕截图 2024-11-08 163808](https://github.com/user-attachments/assets/d289f4f0-afa0-4c80-a982-c2186c103e27)

in the reduced space. Could you confirm if scikit-learn's non-metric MDS implementation uses this definition, or if it employs an alternative method?

Metric MDS: Does metric MDS in scikit-learn also optimize for Kruskal Stress, or does it use a different stress formula? If a different approach is used, would it be possible to provide some insight or references on the stress function applied here?


### Suggest a potential alternative/fix


Documentation Clarification: It would be incredibly helpful if the documentation could include specific details on the stress formulas used in both metric and non-metric MDS. This addition would help researchers and users better understand the theoretical underpinnings of the algorithm in scikit-learn.
Thank you very much for your guidance and clarification on these points. Your insights would be instrumental in my work with MDS.

COMMENT:
> 维基百科的公式与 Kruskal 1964 论文中的公式相同，但符号不同。
> 
> 在 Kruskal 的论文中， d 我 j 是嵌入距离，而 d ^ 我 j 是原始距离的一些单调转换，他称之为 δ 我 j .优化发生在嵌入坐标上，也发生在所有可能的单调函数上。
> 
> 在维基百科中， d ^ 我 j 是嵌入距离，而 f ( d 我 j ) 是原始距离的单调变换 d 我 j .
> 
> 无论哪种方式，这都是非公制 MDS。
> 
> 如果省略单调变换，则会得到“metric MDS”。但我不认为这在 Kruskal 1964 年的论文中是这样明确称呼的。事实上，我不知道是谁首先创造了“公制 MDS”这个词。

Thank you so much for your detailed explanation! ！It has significantly clarified my understanding.

I have one further question: When we refer to "Kruskal Stress" today, does it implicitly assume the use of the monotonic transformations required for non-metric MDS? Or can the term "Kruskal Stress" now broadly refer to the squared error between the original high-dimensional distances and the low-dimensional embedding distances, regardless of whether the original distances undergo monotonic transformation?

I deeply appreciate your insights and apologize for my limited understanding. Your clarification is invaluable to me!

Labels: Documentation, Needs Triage
Clarification on Kruskal Stress as an Optimization Target in Metric and Non-metric MDS

### Describe the issue linked to the documentation

I am working on research involving the optimization targets used in metric and non-metric MDS, and I have some questions regarding how scikit-learn's implementation of MDS defines and calculates stress, particularly Kruskal Stress. While reviewing the official documentation, I noticed that specific formulas for stress calculations are not explicitly provided, and I would appreciate some clarification.

Non-metric MDS: My understanding is that non-metric MDS typically minimizes Kruskal Stress, defined as:

![屏幕截图 2024-11-08 163808](https://github.com/user-attachments/assets/d289f4f0-afa0-4c80-a982-c2186c103e27)

in the reduced space. Could you confirm if scikit-learn's non-metric MDS implementation uses this definition, or if it employs an alternative method?

Metric MDS: Does metric MDS in scikit-learn also optimize for Kruskal Stress, or does it use a different stress formula? If a different approach is used, would it be possible to provide some insight or references on the stress function applied here?


### Suggest a potential alternative/fix


Documentation Clarification: It would be incredibly helpful if the documentation could include specific details on the stress formulas used in both metric and non-metric MDS. This addition would help researchers and users better understand the theoretical underpinnings of the algorithm in scikit-learn.
Thank you very much for your guidance and clarification on these points. Your insights would be instrumental in my work with MDS.

COMMENT:
I am not sure. I would say that "stress" (without Kruskal's name attached to it) often refers to the loss without monotonic transformation, i.e. metric MDS loss. I guess if you say "Kruskal's stress", then maybe it assumes the monotonic transformation in there? In general, I find that terminology surrounding MDS can be very confusing/ambiguous, so I would recommend to be very precise and give a formula or explain in words what you mean exactly.

Labels: Documentation, Needs Triage
Clarification on Kruskal Stress as an Optimization Target in Metric and Non-metric MDS

### Describe the issue linked to the documentation

I am working on research involving the optimization targets used in metric and non-metric MDS, and I have some questions regarding how scikit-learn's implementation of MDS defines and calculates stress, particularly Kruskal Stress. While reviewing the official documentation, I noticed that specific formulas for stress calculations are not explicitly provided, and I would appreciate some clarification.

Non-metric MDS: My understanding is that non-metric MDS typically minimizes Kruskal Stress, defined as:

![屏幕截图 2024-11-08 163808](https://github.com/user-attachments/assets/d289f4f0-afa0-4c80-a982-c2186c103e27)

in the reduced space. Could you confirm if scikit-learn's non-metric MDS implementation uses this definition, or if it employs an alternative method?

Metric MDS: Does metric MDS in scikit-learn also optimize for Kruskal Stress, or does it use a different stress formula? If a different approach is used, would it be possible to provide some insight or references on the stress function applied here?


### Suggest a potential alternative/fix


Documentation Clarification: It would be incredibly helpful if the documentation could include specific details on the stress formulas used in both metric and non-metric MDS. This addition would help researchers and users better understand the theoretical underpinnings of the algorithm in scikit-learn.
Thank you very much for your guidance and clarification on these points. Your insights would be instrumental in my work with MDS.

COMMENT:
> I am not sure. I would say that "stress" (without Kruskal's name attached to it) often refers to the loss without monotonic transformation, i.e. metric MDS loss. I guess if you say "Kruskal's stress", then maybe it assumes the monotonic transformation in there? In general, I find that terminology surrounding MDS can be very confusing/ambiguous, so I would recommend to be very precise and give a formula or explain in words what you mean exactly.

Thank you for your clarification and the detailed explanation.
Actually，I am currently working on reproducing a paper where the authors mentioned using Kruskal's stress as the optimization criterion for MDS by R language. The formula they provided matches the one shared earlier in this issue. 
![image](https://github.com/user-attachments/assets/84405be7-157e-4d25-948e-1c05e31156cb)
However, they did not specify whether they used metric or non-metric MDS. Initially, I assumed Kruskal's stress was exclusively for non-metric MDS, but surprisingly, the metric MDS results from scikit-learn align better with the paper's outcomes.
This led me to ask this question to clarify which stress formula is used in scikit-learn's two MDS methods.

Labels: Documentation, Needs Triage
Missing format string arguments

This assertion error string is not properly formatted as the 2 format arguments `y_pred.shape` and `y.shape` are missing:

https://github.com/scikit-learn/scikit-learn/blob/551d56c254197c4b6ad63974d749824ed2c7bc58/sklearn/utils/estimator_checks.py#L2139

```python
assert y_pred.shape == y.shape, (
    "The shape of the prediction for multioutput data is incorrect."
    " Expected {}, got {}."
)
```

should become

```python
assert y_pred.shape == y.shape, (
    "The shape of the prediction for multioutput data is incorrect."
    " Expected {}, got {}.".format(y_pred.shape, y.shape)
)
```

COMMENT:
Please feel free to directly submit a PR with the fix in the future in such cases :)

Labels: Bug, Easy, help wanted
Have a common test in check_estimator checking for the right order of mixin inheritance

xref: https://github.com/scikit-learn/scikit-learn/pull/30227#pullrequestreview-2416421266

We should make sure it goes the right way, so that tags are set correctly, and to avoid other potential issues.

COMMENT:
I guess the main thought is that tags, and certain class attributes will get overwritten depending on the sequence of the inheritance?

In general though, our Mixin classes are relatively "mutually exclusive" right? 

This would only be an issue if for some reason, someone inherited first from Mixin, and then `BaseEstimator`, or some other `Base*`, where some attributes/tags get overwritten?

Labels: Developer API
Have a common test in check_estimator checking for the right order of mixin inheritance

xref: https://github.com/scikit-learn/scikit-learn/pull/30227#pullrequestreview-2416421266

We should make sure it goes the right way, so that tags are set correctly, and to avoid other potential issues.

COMMENT:
Yeah the main concern is `BaseEstimator` needing to be the last one of our mixins in MRO. I'll write up a test, that should make it clear, and if we need to add more constraints, we can do that

Labels: Developer API
Label Encoder example typo

### Describe the issue linked to the documentation

In the [Label Encoder](https://scikit-learn.org/dev/modules/generated/sklearn.preprocessing.LabelEncoder.html) documentation, the example uses an array to demonstrate the functioning of label encoding. But the arrays used in fit and transform operations are different. For the fit operation, it uses [1,2,2,6] whereas for the transform operation it uses [1,1,2,6] resulting in inconsistency. Just attaching the screen grab of the example code in the documentation.

![Screenshot 2024-11-05 094129](https://github.com/user-attachments/assets/9e560c82-8875-4cb3-924e-8bc18bb1fdd7)


### Suggest a potential alternative/fix

Change to array in `le.fit()` to [1,1,2,6]

COMMENT:
TL;DR: I believe the example in the [Label Encoder documentation](https://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.LabelEncoder.html) works as intended; and applying the suggested fix would likely cause more confusion than clarification.

The `.fit()` method is designed to train the Label Encoder, allowing it to learn the mapping between labels and encoded values; while the `.transform()` method applies this learned mapping to new or repeated data (mutatis mutandis for other scikit-learn estimators). The documentation should illustrate this by demonstrating that you can `.fit()` on one dataset and then `.transform()` on other datasets, whether similar or not.

 In practice, `.fit()`  would typically be called on a training dataset, and `.transform()` on a test or validation dataset, which naturally won’t be identical.

Labels: Documentation, Needs Triage
Label Encoder example typo

### Describe the issue linked to the documentation

In the [Label Encoder](https://scikit-learn.org/dev/modules/generated/sklearn.preprocessing.LabelEncoder.html) documentation, the example uses an array to demonstrate the functioning of label encoding. But the arrays used in fit and transform operations are different. For the fit operation, it uses [1,2,2,6] whereas for the transform operation it uses [1,1,2,6] resulting in inconsistency. Just attaching the screen grab of the example code in the documentation.

![Screenshot 2024-11-05 094129](https://github.com/user-attachments/assets/9e560c82-8875-4cb3-924e-8bc18bb1fdd7)


### Suggest a potential alternative/fix

Change to array in `le.fit()` to [1,1,2,6]

COMMENT:
As @virchan says, that's intended. The calls to `fit` and `transform` don't have to be the same, and the example shows that.

Labels: Documentation, Needs Triage
Label Encoder example typo

### Describe the issue linked to the documentation

In the [Label Encoder](https://scikit-learn.org/dev/modules/generated/sklearn.preprocessing.LabelEncoder.html) documentation, the example uses an array to demonstrate the functioning of label encoding. But the arrays used in fit and transform operations are different. For the fit operation, it uses [1,2,2,6] whereas for the transform operation it uses [1,1,2,6] resulting in inconsistency. Just attaching the screen grab of the example code in the documentation.

![Screenshot 2024-11-05 094129](https://github.com/user-attachments/assets/9e560c82-8875-4cb3-924e-8bc18bb1fdd7)


### Suggest a potential alternative/fix

Change to array in `le.fit()` to [1,1,2,6]

COMMENT:
Thanks for clarifying, @virchan and @adrinjalali!

Labels: Documentation, Needs Triage
Add Accumulated local effects (ALE) to inspection

### Describe the workflow you want to enable

I'd love to push the inspection module further by adding Accumulated local effects (ALE) from Apley 2020. A great description can be found in Christoph's online book https://christophm.github.io/interpretable-ml-book/ale.html

ALE fix the problem of partial dependence that they force the model to be evaluated on impossible or rare feature combinations. ALE are defined for numeric features that can be binned. From both bin edges, the slope of the partial dependence is calculated locally, i.e., only using observations in the bin. The slopes from all bins are cumsummed and vertically centered to the average response or prediction.

### Reference

Apley, Daniel W., and Jingyu Zhu. 2020. Visualizing the Effects of Predictor Variables in Black Box Supervised Learning Models. Journal of the Royal Statistical Society Series B: Statistical Methodology, 82 (4): 1059–1086. doi:10.1111/rssb.12377.

### Describe your proposed solution

Pseudo code to calculate ALE for one feature:

``` py
pd_slopes = np.zeros_like(bins)

for i, bin in enumerate(bins):
  X_bin = pick n_per_bin = 200 rows from data in bin
  if X_bin is empty:
    next
  pd = partial_dependence_brute(model, X_bin, grid = [lower bin edge, upper bin edge], sample_weights)
  pd_slopes[i] = pd[1] - pd[0]

return np.cumsum(pd_slopes)
```

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_

COMMENT:
@glemaitre or @thomasjpfan maybe have an opinion on the inclusion criteria here.

Labels: New Feature, Needs Decision
Add Accumulated local effects (ALE) to inspection

### Describe the workflow you want to enable

I'd love to push the inspection module further by adding Accumulated local effects (ALE) from Apley 2020. A great description can be found in Christoph's online book https://christophm.github.io/interpretable-ml-book/ale.html

ALE fix the problem of partial dependence that they force the model to be evaluated on impossible or rare feature combinations. ALE are defined for numeric features that can be binned. From both bin edges, the slope of the partial dependence is calculated locally, i.e., only using observations in the bin. The slopes from all bins are cumsummed and vertically centered to the average response or prediction.

### Reference

Apley, Daniel W., and Jingyu Zhu. 2020. Visualizing the Effects of Predictor Variables in Black Box Supervised Learning Models. Journal of the Royal Statistical Society Series B: Statistical Methodology, 82 (4): 1059–1086. doi:10.1111/rssb.12377.

### Describe your proposed solution

Pseudo code to calculate ALE for one feature:

``` py
pd_slopes = np.zeros_like(bins)

for i, bin in enumerate(bins):
  X_bin = pick n_per_bin = 200 rows from data in bin
  if X_bin is empty:
    next
  pd = partial_dependence_brute(model, X_bin, grid = [lower bin edge, upper bin edge], sample_weights)
  pd_slopes[i] = pd[1] - pd[0]

return np.cumsum(pd_slopes)
```

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_

COMMENT:
I think that it could be a good addition to the module. The reasoning of what it solves make sense. Regarding the pseudo code, it seems that it would be not a huge amount of code to maintain.

I have to read more in details the literature to be sure thought.

@meyer89 Something that I'm thinking looking at your description: does the recursion method of the partial dependence would already lead to not using impossible combination of feature, isn't it?

Labels: New Feature, Needs Decision
Add Accumulated local effects (ALE) to inspection

### Describe the workflow you want to enable

I'd love to push the inspection module further by adding Accumulated local effects (ALE) from Apley 2020. A great description can be found in Christoph's online book https://christophm.github.io/interpretable-ml-book/ale.html

ALE fix the problem of partial dependence that they force the model to be evaluated on impossible or rare feature combinations. ALE are defined for numeric features that can be binned. From both bin edges, the slope of the partial dependence is calculated locally, i.e., only using observations in the bin. The slopes from all bins are cumsummed and vertically centered to the average response or prediction.

### Reference

Apley, Daniel W., and Jingyu Zhu. 2020. Visualizing the Effects of Predictor Variables in Black Box Supervised Learning Models. Journal of the Royal Statistical Society Series B: Statistical Methodology, 82 (4): 1059–1086. doi:10.1111/rssb.12377.

### Describe your proposed solution

Pseudo code to calculate ALE for one feature:

``` py
pd_slopes = np.zeros_like(bins)

for i, bin in enumerate(bins):
  X_bin = pick n_per_bin = 200 rows from data in bin
  if X_bin is empty:
    next
  pd = partial_dependence_brute(model, X_bin, grid = [lower bin edge, upper bin edge], sample_weights)
  pd_slopes[i] = pd[1] - pd[0]

return np.cumsum(pd_slopes)
```

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_

COMMENT:
> Something that I'm thinking looking at your description: does the recursion method of the partial dependence would already lead to not using impossible combination of feature, isn't it?

Good question! I don't think it is better (if the gender node is switched off, then also men will hit the post-menopausal node etc.). But I'd say that the problem is generally smaller for tree-based models than for other types of models.

Labels: New Feature, Needs Decision