Talip7/scikit-learn-issues-multilabel huggingface dataset:
Macro vs micro-averaging switched up in user guide

### Describe the issue linked to the documentation

Hi guys,
In the "ROC curve using micro-averaged OvR" part of the doc (https://scikit-learn.org/stable/auto_examples/model_selection/plot_roc.html#roc-curve-using-micro-averaged-ovr)

it says:
"In a multi-class classification setup with highly imbalanced classes, micro-averaging is preferable over macro-averaging. In such cases, one can alternatively use a weighted macro-averaging, not demoed here."

I believe it should say: 
In a multi-class classification setup with highly imbalanced classes, **macro**-averaging is preferable over **micro**-averaging. In such cases, one can alternatively use a weighted macro-averaging, not demoed here.

If correct, I believe it could spare users some confusion. Thanks for all your work, Im just trying to help :) !!!

### Suggest a potential alternative/fix

I believe it should say: 
In a multi-class classification setup with highly imbalanced classes, **macro**-averaging is preferable over **micro**-averaging. In such cases, one can alternatively use a weighted macro-averaging, not demoed here.

COMMENT:
>  I would like to emphasize the value of providing guidance, particularly regarding handling imbalanced datasets.

Agreed. In that case the recommendation is:
- Choose a metric as close as possible to the business/use case outcome you hope to achieve.
- To compare (classification) models and measure predictive performance, use consistent scoring rules. They are not negatively affected by imbalanced classes, nor do you need to choose beween micro vs macro averaging.

Labels: Documentation
Macro vs micro-averaging switched up in user guide

### Describe the issue linked to the documentation

Hi guys,
In the "ROC curve using micro-averaged OvR" part of the doc (https://scikit-learn.org/stable/auto_examples/model_selection/plot_roc.html#roc-curve-using-micro-averaged-ovr)

it says:
"In a multi-class classification setup with highly imbalanced classes, micro-averaging is preferable over macro-averaging. In such cases, one can alternatively use a weighted macro-averaging, not demoed here."

I believe it should say: 
In a multi-class classification setup with highly imbalanced classes, **macro**-averaging is preferable over **micro**-averaging. In such cases, one can alternatively use a weighted macro-averaging, not demoed here.

If correct, I believe it could spare users some confusion. Thanks for all your work, Im just trying to help :) !!!

### Suggest a potential alternative/fix

I believe it should say: 
In a multi-class classification setup with highly imbalanced classes, **macro**-averaging is preferable over **micro**-averaging. In such cases, one can alternatively use a weighted macro-averaging, not demoed here.

COMMENT:
PR welcome to fix the documentation.

Labels: Documentation
Macro vs micro-averaging switched up in user guide

### Describe the issue linked to the documentation

Hi guys,
In the "ROC curve using micro-averaged OvR" part of the doc (https://scikit-learn.org/stable/auto_examples/model_selection/plot_roc.html#roc-curve-using-micro-averaged-ovr)

it says:
"In a multi-class classification setup with highly imbalanced classes, micro-averaging is preferable over macro-averaging. In such cases, one can alternatively use a weighted macro-averaging, not demoed here."

I believe it should say: 
In a multi-class classification setup with highly imbalanced classes, **macro**-averaging is preferable over **micro**-averaging. In such cases, one can alternatively use a weighted macro-averaging, not demoed here.

If correct, I believe it could spare users some confusion. Thanks for all your work, Im just trying to help :) !!!

### Suggest a potential alternative/fix

I believe it should say: 
In a multi-class classification setup with highly imbalanced classes, **macro**-averaging is preferable over **micro**-averaging. In such cases, one can alternatively use a weighted macro-averaging, not demoed here.

COMMENT:
i have been catching up on this issue and would like to contribute to it. i am thinking that remarks on the pros and cons of micro vs macro averaging can be placed after macro-averaging is introduced, or perhaps near the end where there is already a bit of discussion on the topic. also, a latex like formula for macro notation can be written, like what was done for micro. i think this would help illustrate the point. 

 i notice that the data currently being worked on is not the best in this case, since the classes are perfectly balanced (25 examples of each if im not mistaken). so describing general ideas might be more appropriate over giving concrete examples. perhaps this should be noted as well?

im not too sure what is meant by your second bullet point @lorentzenchr on consistent scoring rules. could you please elaborate?

Labels: Documentation
Macro vs micro-averaging switched up in user guide

### Describe the issue linked to the documentation

Hi guys,
In the "ROC curve using micro-averaged OvR" part of the doc (https://scikit-learn.org/stable/auto_examples/model_selection/plot_roc.html#roc-curve-using-micro-averaged-ovr)

it says:
"In a multi-class classification setup with highly imbalanced classes, micro-averaging is preferable over macro-averaging. In such cases, one can alternatively use a weighted macro-averaging, not demoed here."

I believe it should say: 
In a multi-class classification setup with highly imbalanced classes, **macro**-averaging is preferable over **micro**-averaging. In such cases, one can alternatively use a weighted macro-averaging, not demoed here.

If correct, I believe it could spare users some confusion. Thanks for all your work, Im just trying to help :) !!!

### Suggest a potential alternative/fix

I believe it should say: 
In a multi-class classification setup with highly imbalanced classes, **macro**-averaging is preferable over **micro**-averaging. In such cases, one can alternatively use a weighted macro-averaging, not demoed here.

COMMENT:
/take

Labels: Documentation
RFECV docstring does not state how the `cv_results_` attribute is ordered by

### Describe the issue linked to the documentation

[This StackOverflow post](https://stackoverflow.com/questions/78111803/how-is-scikit-learns-rfecv-cv-results-attribute-ordered-by) has more details regarding this small issue.

In essence, I noticed that the documentation for [RFECV](https://scikit-learn.org/stable/modules/generated/sklearn.feature_selection.RFECV.html) does not state how the `cv_results_` attribute is ordered by.

Given that the process is *recursive*, some users (myself included) may assume that the dictionary is sorted in descending order (i.e., the first element corresponds to the models that used ALL features, then one step less, then two steps less, etc.). However, it seems to me that the dictionary is sorted in ascending order.

### Suggest a potential alternative/fix

From my perspective, the easiest fix would be to add a few lines to the docstring. Something along the lines of:
> This dictionary is sorted by the number of features in ascending order (i.e., the first element represents the models that use the least number of features, while the last element represents the models that use all available features).

As an alternative, the resulting dictionary could have an additional key named `n_features` (or something along those lines) that states how many features each element in the dictionary represents.

COMMENT:
Thanks for the report. Feel free to open a PR to improve the docstring.

Labels: Documentation
RFECV docstring does not state how the `cv_results_` attribute is ordered by

### Describe the issue linked to the documentation

[This StackOverflow post](https://stackoverflow.com/questions/78111803/how-is-scikit-learns-rfecv-cv-results-attribute-ordered-by) has more details regarding this small issue.

In essence, I noticed that the documentation for [RFECV](https://scikit-learn.org/stable/modules/generated/sklearn.feature_selection.RFECV.html) does not state how the `cv_results_` attribute is ordered by.

Given that the process is *recursive*, some users (myself included) may assume that the dictionary is sorted in descending order (i.e., the first element corresponds to the models that used ALL features, then one step less, then two steps less, etc.). However, it seems to me that the dictionary is sorted in ascending order.

### Suggest a potential alternative/fix

From my perspective, the easiest fix would be to add a few lines to the docstring. Something along the lines of:
> This dictionary is sorted by the number of features in ascending order (i.e., the first element represents the models that use the least number of features, while the last element represents the models that use all available features).

As an alternative, the resulting dictionary could have an additional key named `n_features` (or something along those lines) that states how many features each element in the dictionary represents.

COMMENT:
@ArturoSbr are you working on this? Would be interested to contribute.

Labels: Documentation
RFECV docstring does not state how the `cv_results_` attribute is ordered by

### Describe the issue linked to the documentation

[This StackOverflow post](https://stackoverflow.com/questions/78111803/how-is-scikit-learns-rfecv-cv-results-attribute-ordered-by) has more details regarding this small issue.

In essence, I noticed that the documentation for [RFECV](https://scikit-learn.org/stable/modules/generated/sklearn.feature_selection.RFECV.html) does not state how the `cv_results_` attribute is ordered by.

Given that the process is *recursive*, some users (myself included) may assume that the dictionary is sorted in descending order (i.e., the first element corresponds to the models that used ALL features, then one step less, then two steps less, etc.). However, it seems to me that the dictionary is sorted in ascending order.

### Suggest a potential alternative/fix

From my perspective, the easiest fix would be to add a few lines to the docstring. Something along the lines of:
> This dictionary is sorted by the number of features in ascending order (i.e., the first element represents the models that use the least number of features, while the last element represents the models that use all available features).

As an alternative, the resulting dictionary could have an additional key named `n_features` (or something along those lines) that states how many features each element in the dictionary represents.

COMMENT:
@ArturoSbr also suggested the following:

> As an alternative, the resulting dictionary could have an additional key named n_features (or something along those lines) that states how many features each element in the dictionary represents.

I think this would further help usability of this attribute (e.g. by loading it in a dataframe to plot `n_features` vs `mean_test_score` with error bars.

We could then update the following example to make it simpler by using `pd.DataFrame(rfecv.cv_results_)` and not having to magically recompute the x values of the plot with `range(min_features_to_select, n_scores + min_features_to_select):

https://scikit-learn.org/stable/auto_examples/feature_selection/plot_rfe_with_cross_validation.html

@miguelcsilva would you be interested in working in separate PR to implement this?

Labels: Documentation
RFECV docstring does not state how the `cv_results_` attribute is ordered by

### Describe the issue linked to the documentation

[This StackOverflow post](https://stackoverflow.com/questions/78111803/how-is-scikit-learns-rfecv-cv-results-attribute-ordered-by) has more details regarding this small issue.

In essence, I noticed that the documentation for [RFECV](https://scikit-learn.org/stable/modules/generated/sklearn.feature_selection.RFECV.html) does not state how the `cv_results_` attribute is ordered by.

Given that the process is *recursive*, some users (myself included) may assume that the dictionary is sorted in descending order (i.e., the first element corresponds to the models that used ALL features, then one step less, then two steps less, etc.). However, it seems to me that the dictionary is sorted in ascending order.

### Suggest a potential alternative/fix

From my perspective, the easiest fix would be to add a few lines to the docstring. Something along the lines of:
> This dictionary is sorted by the number of features in ascending order (i.e., the first element represents the models that use the least number of features, while the last element represents the models that use all available features).

As an alternative, the resulting dictionary could have an additional key named `n_features` (or something along those lines) that states how many features each element in the dictionary represents.

COMMENT:
> @miguelcsilva would you be interested in working in separate PR to implement this?

Yes, thanks for that. I'll get started working on a separate PR for this.

Labels: Documentation
RFECV docstring does not state how the `cv_results_` attribute is ordered by

### Describe the issue linked to the documentation

[This StackOverflow post](https://stackoverflow.com/questions/78111803/how-is-scikit-learns-rfecv-cv-results-attribute-ordered-by) has more details regarding this small issue.

In essence, I noticed that the documentation for [RFECV](https://scikit-learn.org/stable/modules/generated/sklearn.feature_selection.RFECV.html) does not state how the `cv_results_` attribute is ordered by.

Given that the process is *recursive*, some users (myself included) may assume that the dictionary is sorted in descending order (i.e., the first element corresponds to the models that used ALL features, then one step less, then two steps less, etc.). However, it seems to me that the dictionary is sorted in ascending order.

### Suggest a potential alternative/fix

From my perspective, the easiest fix would be to add a few lines to the docstring. Something along the lines of:
> This dictionary is sorted by the number of features in ascending order (i.e., the first element represents the models that use the least number of features, while the last element represents the models that use all available features).

As an alternative, the resulting dictionary could have an additional key named `n_features` (or something along those lines) that states how many features each element in the dictionary represents.

COMMENT:
@ogrisel while working on this I noticed the following unexpected behavior. If I initialize the `RFECV` with a `min_features_to_select` larger than the number of features that I pass to the `fit` method, I do not get an error (as I expected), but instead a result is returned. See the minimal example below:

```python
from sklearn.feature_selection import RFECV
from sklearn.linear_model import LogisticRegression
from sklearn.datasets import make_classification
from pandas import DataFrame

X, y = make_classification(n_samples=1000, n_features=20, n_redundant=0, n_classes=2, random_state=0)

rfecv = RFECV(
    estimator=LogisticRegression(random_state=0),
    min_features_to_select=21,
    cv=2,
    step=2,
    scoring="precision"
)
rfecv.fit(X=X, y=y)

print(DataFrame(rfecv.cv_results_))
```
Although I do not know the code base too well, my suspicion is that we start from an array with the value for `min_features_to_select` already present and append the remaining steps until we reach the maximum number of features. This however means that we get the unexpected behavior described above.

One option would be to raise an `InvalidParameterError` at the start of the `fit` method like this:
```python
        if self.min_features_to_select > n_features:
            raise InvalidParameterError(
                "Minimum number of features to select cannot exceed maximum number of available features."
            )
```
Let me know if you think it is worth throwing an error here or if you have any other ideas? And also if you think this is better treated in the same PR I'm currently working on, or in a new one?

Labels: Documentation
RFECV docstring does not state how the `cv_results_` attribute is ordered by

### Describe the issue linked to the documentation

[This StackOverflow post](https://stackoverflow.com/questions/78111803/how-is-scikit-learns-rfecv-cv-results-attribute-ordered-by) has more details regarding this small issue.

In essence, I noticed that the documentation for [RFECV](https://scikit-learn.org/stable/modules/generated/sklearn.feature_selection.RFECV.html) does not state how the `cv_results_` attribute is ordered by.

Given that the process is *recursive*, some users (myself included) may assume that the dictionary is sorted in descending order (i.e., the first element corresponds to the models that used ALL features, then one step less, then two steps less, etc.). However, it seems to me that the dictionary is sorted in ascending order.

### Suggest a potential alternative/fix

From my perspective, the easiest fix would be to add a few lines to the docstring. Something along the lines of:
> This dictionary is sorted by the number of features in ascending order (i.e., the first element represents the models that use the least number of features, while the last element represents the models that use all available features).

As an alternative, the resulting dictionary could have an additional key named `n_features` (or something along those lines) that states how many features each element in the dictionary represents.

COMMENT:
Let's open a dedicated PR for this!

Labels: Documentation
How to solve the AttributeError: 'LabelPowerset' object has no attribute 'classes_'?

When I try to use LabelPowerset in scikit-multilearn, the code runs with GrieSearchCV with an error
```python
from skmultilearn.problem_transform import LabelPowerset
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import GridSearchCV
from sklearn.metrics import make_scorer, hamming_loss
from sklearn.datasets import make_multilabel_classification
X, Y = make_multilabel_classification()
classifier = LabelPowerset(classifier=RandomForestClassifier())
parameters = {
'classifier__n_estimators': [10, 50, 100],
'classifier__max_depth': [None, 5, 10]
}
scorer = make_scorer(hamming_loss, greater_is_better=False)
grid_search = GridSearchCV(classifier, parameters, scoring=scorer, cv=5)
grid_search.fit(X, Y)
``` 
The above code reports an error
```python
Traceback (most recent call last):
  File "/home/zhangkai/.conda/envs/classifier/lib/python3.9/site-packages/sklearn/model_selection/_validation.py", line 813, in _score
    scores = scorer(estimator, X_test, y_test)
  File "/home/zhangkai/.conda/envs/classifier/lib/python3.9/site-packages/sklearn/metrics/_scorer.py", line 266, in __call__
    return self._score(partial(_cached_call, None), estimator, X, y_true, **_kwargs)
  File "/home/zhangkai/.conda/envs/classifier/lib/python3.9/site-packages/sklearn/metrics/_scorer.py", line 353, in _score
    y_pred = method_caller(estimator, "predict", X)
  File "/home/zhangkai/.conda/envs/classifier/lib/python3.9/site-packages/sklearn/metrics/_scorer.py", line 86, in _cached_call
    result, _ = _get_response_values(
  File "/home/zhangkai/.conda/envs/classifier/lib/python3.9/site-packages/sklearn/utils/_response.py", line 74, in _get_response_values
    classes = estimator.classes_
AttributeError: 'LabelPowerset' object has no attribute 'classes
``` 
I tried the example code given on the official website(http://scikit.ml/api/skmultilearn.problem_transform.lp.html#skmultilearn.problem_transform.LabelPowerset) and still reported the same error.

How to solve this problem?

COMMENT:
This seems to be a `scikit-multilearn` problem, but it doesn't seem to be actively maintained currently, so it is not fully compatible with the latest few versions of scikit-learn. If you want the code above to work you may need to downgrade to scikit-learn 1.2.2.

Labels: Needs Triage
How to solve the AttributeError: 'LabelPowerset' object has no attribute 'classes_'?

When I try to use LabelPowerset in scikit-multilearn, the code runs with GrieSearchCV with an error
```python
from skmultilearn.problem_transform import LabelPowerset
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import GridSearchCV
from sklearn.metrics import make_scorer, hamming_loss
from sklearn.datasets import make_multilabel_classification
X, Y = make_multilabel_classification()
classifier = LabelPowerset(classifier=RandomForestClassifier())
parameters = {
'classifier__n_estimators': [10, 50, 100],
'classifier__max_depth': [None, 5, 10]
}
scorer = make_scorer(hamming_loss, greater_is_better=False)
grid_search = GridSearchCV(classifier, parameters, scoring=scorer, cv=5)
grid_search.fit(X, Y)
``` 
The above code reports an error
```python
Traceback (most recent call last):
  File "/home/zhangkai/.conda/envs/classifier/lib/python3.9/site-packages/sklearn/model_selection/_validation.py", line 813, in _score
    scores = scorer(estimator, X_test, y_test)
  File "/home/zhangkai/.conda/envs/classifier/lib/python3.9/site-packages/sklearn/metrics/_scorer.py", line 266, in __call__
    return self._score(partial(_cached_call, None), estimator, X, y_true, **_kwargs)
  File "/home/zhangkai/.conda/envs/classifier/lib/python3.9/site-packages/sklearn/metrics/_scorer.py", line 353, in _score
    y_pred = method_caller(estimator, "predict", X)
  File "/home/zhangkai/.conda/envs/classifier/lib/python3.9/site-packages/sklearn/metrics/_scorer.py", line 86, in _cached_call
    result, _ = _get_response_values(
  File "/home/zhangkai/.conda/envs/classifier/lib/python3.9/site-packages/sklearn/utils/_response.py", line 74, in _get_response_values
    classes = estimator.classes_
AttributeError: 'LabelPowerset' object has no attribute 'classes
``` 
I tried the example code given on the official website(http://scikit.ml/api/skmultilearn.problem_transform.lp.html#skmultilearn.problem_transform.LabelPowerset) and still reported the same error.

How to solve this problem?

COMMENT:
> This seems to be a `scikit-multilearn` problem, but it doesn't seem to be actively maintained currently, so it is not fully compatible with the latest few versions of scikit-learn. If you want the code above to work you may need to downgrade to scikit-learn 1.2.2.

Thank you for your answer and I will try your solution as soon as possible!

Labels: Needs Triage