Talip7/scikit-learn-issues-multilabel huggingface dataset:
ENH partial_dependece plot for HistGradientBoosting estimator fitted with `sample_weight`

### Describe the workflow you want to enable

As partial dependence of a model at a point [is defined as an expectation](https://scikit-learn.org/stable/modules/partial_dependence.html#mathematical-definition), it should respect `sample_weight` if someone wishes to use it (for instance, when you know your `X` does not follow the distribution you are interested in).

#25209 tries to solve this for `method='brute'` when you have new `X`. For older tree-based models trained with sample_weights, [`method='recursion'`](https://nicolas-hug.com/blog/pdps) keeps track of the training `sample_weight` and calculates the `partial_dependece` with that into consideration.

But, as discussed during the implementation of `sample_weight` on the HistGradientBoosting estimators (https://github.com/scikit-learn/scikit-learn/pull/14696#issuecomment-548295813), these models stores an attribute `_fitted_with_sw` and when `partial_dependece` with recursion is asked, it throws an error:

https://github.com/scikit-learn/scikit-learn/blob/205f3b76ef8500c90c346c6c6fb6f4e589368278/sklearn/ensemble/_hist_gradient_boosting/gradient_boosting.py#L1142-L1148


### Describe your proposed solution

As discussed in https://github.com/scikit-learn/scikit-learn/issues/24872#issuecomment-1352354690, the big difference between other tree-based algorithms and HistGradientBoosting is that HistGradientBoosting does not save the `weighted_n_node_samples` when building the tree.

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_

COMMENT:
Hi @vitaliset, as you can see from my draft pull request, I have added support for using the user-inputted sample weights to compute and keep track of the `weighted_n_node_samples` when growing the tree. Currently, I am attempting to add some tests to reflect these changes, specifically in `test_grower.py` and possibly `test_predictor.py` and `test_gradient_boosting.py`, but I am struggling with what parameters to use for `sample_weight` and `min_weight_leaf`. Any assistance would be highly appreciated. In the meantime, I am working on documenting the performance and memory footprint of the updated `HistGradientBoosting` model.

Labels: New Feature, module:ensemble
ENH partial_dependece plot for HistGradientBoosting estimator fitted with `sample_weight`

### Describe the workflow you want to enable

As partial dependence of a model at a point [is defined as an expectation](https://scikit-learn.org/stable/modules/partial_dependence.html#mathematical-definition), it should respect `sample_weight` if someone wishes to use it (for instance, when you know your `X` does not follow the distribution you are interested in).

#25209 tries to solve this for `method='brute'` when you have new `X`. For older tree-based models trained with sample_weights, [`method='recursion'`](https://nicolas-hug.com/blog/pdps) keeps track of the training `sample_weight` and calculates the `partial_dependece` with that into consideration.

But, as discussed during the implementation of `sample_weight` on the HistGradientBoosting estimators (https://github.com/scikit-learn/scikit-learn/pull/14696#issuecomment-548295813), these models stores an attribute `_fitted_with_sw` and when `partial_dependece` with recursion is asked, it throws an error:

https://github.com/scikit-learn/scikit-learn/blob/205f3b76ef8500c90c346c6c6fb6f4e589368278/sklearn/ensemble/_hist_gradient_boosting/gradient_boosting.py#L1142-L1148


### Describe your proposed solution

As discussed in https://github.com/scikit-learn/scikit-learn/issues/24872#issuecomment-1352354690, the big difference between other tree-based algorithms and HistGradientBoosting is that HistGradientBoosting does not save the `weighted_n_node_samples` when building the tree.

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_

COMMENT:
Hi @vitaliset, thanks for your suggestions! I have incorporated them into my branch, cleaned it up, and made it available for review. Out of curiosity, how do I request for a reviewer for a pull request?

Labels: New Feature, module:ensemble
ENH partial_dependece plot for HistGradientBoosting estimator fitted with `sample_weight`

### Describe the workflow you want to enable

As partial dependence of a model at a point [is defined as an expectation](https://scikit-learn.org/stable/modules/partial_dependence.html#mathematical-definition), it should respect `sample_weight` if someone wishes to use it (for instance, when you know your `X` does not follow the distribution you are interested in).

#25209 tries to solve this for `method='brute'` when you have new `X`. For older tree-based models trained with sample_weights, [`method='recursion'`](https://nicolas-hug.com/blog/pdps) keeps track of the training `sample_weight` and calculates the `partial_dependece` with that into consideration.

But, as discussed during the implementation of `sample_weight` on the HistGradientBoosting estimators (https://github.com/scikit-learn/scikit-learn/pull/14696#issuecomment-548295813), these models stores an attribute `_fitted_with_sw` and when `partial_dependece` with recursion is asked, it throws an error:

https://github.com/scikit-learn/scikit-learn/blob/205f3b76ef8500c90c346c6c6fb6f4e589368278/sklearn/ensemble/_hist_gradient_boosting/gradient_boosting.py#L1142-L1148


### Describe your proposed solution

As discussed in https://github.com/scikit-learn/scikit-learn/issues/24872#issuecomment-1352354690, the big difference between other tree-based algorithms and HistGradientBoosting is that HistGradientBoosting does not save the `weighted_n_node_samples` when building the tree.

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_

COMMENT:
Hi @vitaliset. For the pull request to incorporate the `weighted_n_node_samples` into the tree, it is currently on hold due to the significant performance degradation. I have tried different methods to speed up the code but unfortunately they are still too slow. Do you have any suggestions for improving the performance or finding another, less costly, approach to achieve the same effects?

Labels: New Feature, module:ensemble
ENH partial_dependece plot for HistGradientBoosting estimator fitted with `sample_weight`

### Describe the workflow you want to enable

As partial dependence of a model at a point [is defined as an expectation](https://scikit-learn.org/stable/modules/partial_dependence.html#mathematical-definition), it should respect `sample_weight` if someone wishes to use it (for instance, when you know your `X` does not follow the distribution you are interested in).

#25209 tries to solve this for `method='brute'` when you have new `X`. For older tree-based models trained with sample_weights, [`method='recursion'`](https://nicolas-hug.com/blog/pdps) keeps track of the training `sample_weight` and calculates the `partial_dependece` with that into consideration.

But, as discussed during the implementation of `sample_weight` on the HistGradientBoosting estimators (https://github.com/scikit-learn/scikit-learn/pull/14696#issuecomment-548295813), these models stores an attribute `_fitted_with_sw` and when `partial_dependece` with recursion is asked, it throws an error:

https://github.com/scikit-learn/scikit-learn/blob/205f3b76ef8500c90c346c6c6fb6f4e589368278/sklearn/ensemble/_hist_gradient_boosting/gradient_boosting.py#L1142-L1148


### Describe your proposed solution

As discussed in https://github.com/scikit-learn/scikit-learn/issues/24872#issuecomment-1352354690, the big difference between other tree-based algorithms and HistGradientBoosting is that HistGradientBoosting does not save the `weighted_n_node_samples` when building the tree.

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_

COMMENT:
Hi @Andrew-Wang-IB45. Simplifying and taking a smaller step might be a way. I suggested on your PR not to do extra calculations beyond the essential ones to store the `weighted_n_node_samples`. I hope this works!

Labels: New Feature, module:ensemble
ENH partial_dependece plot for HistGradientBoosting estimator fitted with `sample_weight`

### Describe the workflow you want to enable

As partial dependence of a model at a point [is defined as an expectation](https://scikit-learn.org/stable/modules/partial_dependence.html#mathematical-definition), it should respect `sample_weight` if someone wishes to use it (for instance, when you know your `X` does not follow the distribution you are interested in).

#25209 tries to solve this for `method='brute'` when you have new `X`. For older tree-based models trained with sample_weights, [`method='recursion'`](https://nicolas-hug.com/blog/pdps) keeps track of the training `sample_weight` and calculates the `partial_dependece` with that into consideration.

But, as discussed during the implementation of `sample_weight` on the HistGradientBoosting estimators (https://github.com/scikit-learn/scikit-learn/pull/14696#issuecomment-548295813), these models stores an attribute `_fitted_with_sw` and when `partial_dependece` with recursion is asked, it throws an error:

https://github.com/scikit-learn/scikit-learn/blob/205f3b76ef8500c90c346c6c6fb6f4e589368278/sklearn/ensemble/_hist_gradient_boosting/gradient_boosting.py#L1142-L1148


### Describe your proposed solution

As discussed in https://github.com/scikit-learn/scikit-learn/issues/24872#issuecomment-1352354690, the big difference between other tree-based algorithms and HistGradientBoosting is that HistGradientBoosting does not save the `weighted_n_node_samples` when building the tree.

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_

COMMENT:
Hi @vitaliset, thanks for your suggestions. I am currently removing the extraneous code so that it only computes the `weighted_n_node_samples` along the way. I will let you how that goes.

Labels: New Feature, module:ensemble
ENH partial_dependece plot for HistGradientBoosting estimator fitted with `sample_weight`

### Describe the workflow you want to enable

As partial dependence of a model at a point [is defined as an expectation](https://scikit-learn.org/stable/modules/partial_dependence.html#mathematical-definition), it should respect `sample_weight` if someone wishes to use it (for instance, when you know your `X` does not follow the distribution you are interested in).

#25209 tries to solve this for `method='brute'` when you have new `X`. For older tree-based models trained with sample_weights, [`method='recursion'`](https://nicolas-hug.com/blog/pdps) keeps track of the training `sample_weight` and calculates the `partial_dependece` with that into consideration.

But, as discussed during the implementation of `sample_weight` on the HistGradientBoosting estimators (https://github.com/scikit-learn/scikit-learn/pull/14696#issuecomment-548295813), these models stores an attribute `_fitted_with_sw` and when `partial_dependece` with recursion is asked, it throws an error:

https://github.com/scikit-learn/scikit-learn/blob/205f3b76ef8500c90c346c6c6fb6f4e589368278/sklearn/ensemble/_hist_gradient_boosting/gradient_boosting.py#L1142-L1148


### Describe your proposed solution

As discussed in https://github.com/scikit-learn/scikit-learn/issues/24872#issuecomment-1352354690, the big difference between other tree-based algorithms and HistGradientBoosting is that HistGradientBoosting does not save the `weighted_n_node_samples` when building the tree.

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_

COMMENT:
The conclusion of #25431 is that it is too costly to add sample weigths per default. I opened #26128 to discuss a possible way forward.

Labels: New Feature, module:ensemble
Unclear train_score_ attribute description for GradientBoostingClassifier

### Describe the issue linked to the documentation

I recently trained a binary gradient boosting model using the GradientBoostingClassifier. I wanted to plot the intermediate losses at each iteration on the test set while evaluating my results and stumbled upon the `train_score_` attribute which is described as `The i-th score train_score_[i] is the deviance (= loss) of the model at iteration i on the in-bag sample.`. 
However, when plotting my test losses (calculated using normal log_loss using probabilities based on the `staged_predict_proba(X)` function) the training scores were not in the expected [0,1) interval and the resulting plot did not make a lot of sense (I would love a test set with the loss curve :D). Following the plot:

![loss_without_scalled_test_loss](https://user-images.githubusercontent.com/44102842/208318601-d3c3d372-f91e-4694-9a14-e51889a90e85.png)


After some digging, I found out that, at least for binomial log_loss, deviance is scaled by a factor of 2 using the following formula:

```
-2 * np.mean((y * raw_predictions) - np.logaddexp(0, raw_predictions))
```


Knowing this, the fix for the problem is very easy but at least for me took quite some time to find the scaling factor. For completion, the resulting plot makes much more sense if we scale the test loss by the same factor of 2:

![loss_with_scaled_test_loss](https://user-images.githubusercontent.com/44102842/208318607-d48c0d63-635a-49b6-934f-19a3069c9f5d.png)


### Suggest a potential alternative/fix

I would suggest adding a hint to the `train_scores_` attribute that the deviance is not necessarily the loss but could be a scaled version of the loss and dependent on the loss function employed. In addition, I would somehow change the `deviance (= loss)` part of the current documentation as the equality sign leaves room for interpretation. 

I did not open a pull request with a proposed change since my problem is quite specific to the binomial log loss/deviance and might not be important enough for a dedicated hint.

COMMENT:
We have a similar example here: https://scikit-learn.org/stable/auto_examples/ensemble/plot_gradient_boosting_oob.html#sphx-glr-auto-examples-ensemble-plot-gradient-boosting-oob-py

The issue is more that usually all the losses that have a factor `x2` will be neglected and we take only the half-loss. I would rather try to update the documentation of the loss parameter to be more explicit since it is related to the loss itself and not what we are logging.

Labels: Documentation, module:ensemble
Unclear train_score_ attribute description for GradientBoostingClassifier

### Describe the issue linked to the documentation

I recently trained a binary gradient boosting model using the GradientBoostingClassifier. I wanted to plot the intermediate losses at each iteration on the test set while evaluating my results and stumbled upon the `train_score_` attribute which is described as `The i-th score train_score_[i] is the deviance (= loss) of the model at iteration i on the in-bag sample.`. 
However, when plotting my test losses (calculated using normal log_loss using probabilities based on the `staged_predict_proba(X)` function) the training scores were not in the expected [0,1) interval and the resulting plot did not make a lot of sense (I would love a test set with the loss curve :D). Following the plot:

![loss_without_scalled_test_loss](https://user-images.githubusercontent.com/44102842/208318601-d3c3d372-f91e-4694-9a14-e51889a90e85.png)


After some digging, I found out that, at least for binomial log_loss, deviance is scaled by a factor of 2 using the following formula:

```
-2 * np.mean((y * raw_predictions) - np.logaddexp(0, raw_predictions))
```


Knowing this, the fix for the problem is very easy but at least for me took quite some time to find the scaling factor. For completion, the resulting plot makes much more sense if we scale the test loss by the same factor of 2:

![loss_with_scaled_test_loss](https://user-images.githubusercontent.com/44102842/208318607-d48c0d63-635a-49b6-934f-19a3069c9f5d.png)


### Suggest a potential alternative/fix

I would suggest adding a hint to the `train_scores_` attribute that the deviance is not necessarily the loss but could be a scaled version of the loss and dependent on the loss function employed. In addition, I would somehow change the `deviance (= loss)` part of the current documentation as the equality sign leaves room for interpretation. 

I did not open a pull request with a proposed change since my problem is quite specific to the binomial log loss/deviance and might not be important enough for a dedicated hint.

COMMENT:
Hello i'll take care of this if possible (:

Labels: Documentation, module:ensemble
Unclear train_score_ attribute description for GradientBoostingClassifier

### Describe the issue linked to the documentation

I recently trained a binary gradient boosting model using the GradientBoostingClassifier. I wanted to plot the intermediate losses at each iteration on the test set while evaluating my results and stumbled upon the `train_score_` attribute which is described as `The i-th score train_score_[i] is the deviance (= loss) of the model at iteration i on the in-bag sample.`. 
However, when plotting my test losses (calculated using normal log_loss using probabilities based on the `staged_predict_proba(X)` function) the training scores were not in the expected [0,1) interval and the resulting plot did not make a lot of sense (I would love a test set with the loss curve :D). Following the plot:

![loss_without_scalled_test_loss](https://user-images.githubusercontent.com/44102842/208318601-d3c3d372-f91e-4694-9a14-e51889a90e85.png)


After some digging, I found out that, at least for binomial log_loss, deviance is scaled by a factor of 2 using the following formula:

```
-2 * np.mean((y * raw_predictions) - np.logaddexp(0, raw_predictions))
```


Knowing this, the fix for the problem is very easy but at least for me took quite some time to find the scaling factor. For completion, the resulting plot makes much more sense if we scale the test loss by the same factor of 2:

![loss_with_scaled_test_loss](https://user-images.githubusercontent.com/44102842/208318607-d48c0d63-635a-49b6-934f-19a3069c9f5d.png)


### Suggest a potential alternative/fix

I would suggest adding a hint to the `train_scores_` attribute that the deviance is not necessarily the loss but could be a scaled version of the loss and dependent on the loss function employed. In addition, I would somehow change the `deviance (= loss)` part of the current documentation as the equality sign leaves room for interpretation. 

I did not open a pull request with a proposed change since my problem is quite specific to the binomial log loss/deviance and might not be important enough for a dedicated hint.

COMMENT:
Future contributors should have a look at the discussions here: https://github.com/scikit-learn/scikit-learn/pull/25383

Labels: Documentation, module:ensemble
⚠️ CI failed on Linux_Nightly.pylatest_pip_scipy_dev ⚠️

**CI is still failing on [Linux_Nightly.pylatest_pip_scipy_dev](https://dev.azure.com/scikit-learn/scikit-learn/_build/results?buildId=51438&view=logs&j=dfe99b15-50db-5d7b-b1e9-4105c42527cf)** (Jan 27, 2023)
- test_estimators[FeatureAgglomeration()-check_parameters_default_constructible]

COMMENT:
I will try to take care of this one. I assume that we should only be using "kulsinski1" instead. People should have got the deprecation warning raised by SciPy since 2 versions.

Labels: Build / CI
⚠️ CI failed on Linux_Nightly.pylatest_pip_scipy_dev ⚠️

**CI is still failing on [Linux_Nightly.pylatest_pip_scipy_dev](https://dev.azure.com/scikit-learn/scikit-learn/_build/results?buildId=51438&view=logs&j=dfe99b15-50db-5d7b-b1e9-4105c42527cf)** (Jan 27, 2023)
- test_estimators[FeatureAgglomeration()-check_parameters_default_constructible]

COMMENT:
https://github.com/scikit-learn/scikit-learn/pull/25425 has been opened to fix some failures.

Labels: Build / CI
StratifiedKFold and StratifiedGroupKFold for multilabel classification

### Describe the workflow you want to enable

Currently, these two functionalities only support binary/multiclass classification. I am looking for similar functionalities (maybe not even k fold, a train_test_split also suffices) in multilabel classification. 

### Describe your proposed solution

For the multilabel version of stratification, [this](https://github.com/trent-b/iterative-stratification) and [this](http://scikit.ml/api/skmultilearn.model_selection.iterative_stratification.html#module-skmultilearn.model_selection.iterative_stratification) libraries seem already provided a solution. Adding grouping to them should be similar to how `StratifiedGroupKFold` adds to `StratifiedKFold`.

### Describe alternatives you've considered, if relevant

See above for existing libraries developed based upon Sklearn.

### Additional context

_No response_

COMMENT:
Het, I would love to work on this!

Labels: New Feature, module:model_selection
StratifiedKFold and StratifiedGroupKFold for multilabel classification

### Describe the workflow you want to enable

Currently, these two functionalities only support binary/multiclass classification. I am looking for similar functionalities (maybe not even k fold, a train_test_split also suffices) in multilabel classification. 

### Describe your proposed solution

For the multilabel version of stratification, [this](https://github.com/trent-b/iterative-stratification) and [this](http://scikit.ml/api/skmultilearn.model_selection.iterative_stratification.html#module-skmultilearn.model_selection.iterative_stratification) libraries seem already provided a solution. Adding grouping to them should be similar to how `StratifiedGroupKFold` adds to `StratifiedKFold`.

### Describe alternatives you've considered, if relevant

See above for existing libraries developed based upon Sklearn.

### Additional context

_No response_

COMMENT:
Since the paper is already quite cited and there is some implementation of reference, I think that we could add this feature to our current implementation. It could be handy.
@MDK8888 Feel free to go ahead.

Labels: New Feature, module:model_selection
StratifiedKFold and StratifiedGroupKFold for multilabel classification

### Describe the workflow you want to enable

Currently, these two functionalities only support binary/multiclass classification. I am looking for similar functionalities (maybe not even k fold, a train_test_split also suffices) in multilabel classification. 

### Describe your proposed solution

For the multilabel version of stratification, [this](https://github.com/trent-b/iterative-stratification) and [this](http://scikit.ml/api/skmultilearn.model_selection.iterative_stratification.html#module-skmultilearn.model_selection.iterative_stratification) libraries seem already provided a solution. Adding grouping to them should be similar to how `StratifiedGroupKFold` adds to `StratifiedKFold`.

### Describe alternatives you've considered, if relevant

See above for existing libraries developed based upon Sklearn.

### Additional context

_No response_

COMMENT:
Any update on this? Would love to use this! I think both library references do not provide a method for doing grouped stratified folds, rather just regular ones?

Labels: New Feature, module:model_selection