Talip7/scikit-learn-issues-multilabel huggingface dataset:
[RFC] Leaf Level Variance in Multi Output Decision Trees

### Describe the workflow you want to enable

For single output RFR trained with the squared error criterion the impurity of the leaves can be used as a crude but useful estimate of the aleatoric uncertainty. In the multi output case the impurity is the sum over outputs hence this is no longer possible to estimate.

### Describe your proposed solution

Currently we can access the node values from `estimator.tree_.values` that stores the leaf means, I propose adding another data store or `estimator.tree_.square_means` that would store the square means of the leaves. Using these it is possible to work out the variance within the leaf that can then be used as a crude but useful estimate of the aleatoric uncertainty. The added benefit of doing this would be that this could be calculated for any choice of criterion rather than just the squared error. The principal issue is that it would double the amount of data stored.

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

This measure of the aleatoric uncertainty is widely used in the work of frank hutter in more recent versions of SMAC - https://ml.informatik.uni-freiburg.de/wp-content/uploads/papers/11-LION5-SMAC.pdf

COMMENT:
I think we need a clean story and API to implement prediction uncertainty modeling in scikit-learn).

What you suggest might be cheap but not necessarily as principled as fitting trees with a quantile criterion (possibly for several quantile level at once). Their has been past discussions, let me cross link:

- https://github.com/scikit-learn/scikit-learn/issues/4768
- https://github.com/scikit-learn/scikit-learn/issues/18540
- https://github.com/scikit-learn/scikit-learn/issues/19710

We probably need a meta-issue to discuss uncertainty inpredictions in scikit-learn that surveys the different methods and possible API suggestions.

Labels: New Feature, Needs Decision, RFC
[RFC] Leaf Level Variance in Multi Output Decision Trees

### Describe the workflow you want to enable

For single output RFR trained with the squared error criterion the impurity of the leaves can be used as a crude but useful estimate of the aleatoric uncertainty. In the multi output case the impurity is the sum over outputs hence this is no longer possible to estimate.

### Describe your proposed solution

Currently we can access the node values from `estimator.tree_.values` that stores the leaf means, I propose adding another data store or `estimator.tree_.square_means` that would store the square means of the leaves. Using these it is possible to work out the variance within the leaf that can then be used as a crude but useful estimate of the aleatoric uncertainty. The added benefit of doing this would be that this could be calculated for any choice of criterion rather than just the squared error. The principal issue is that it would double the amount of data stored.

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

This measure of the aleatoric uncertainty is widely used in the work of frank hutter in more recent versions of SMAC - https://ml.informatik.uni-freiburg.de/wp-content/uploads/papers/11-LION5-SMAC.pdf

COMMENT:
I am not fully suggesting that we add this to the scikit-learn API yet but more that accumulating more information on the nodes could allow for wrapping the base models to allow end users to do this. There are several papers claiming more principled and corrected uncertainty estimates than those I am looking at but I haven't seen empirical evidence that the corrections really are justified. 

It's fairly easy to hack around the current RFR model to extract the variance over tree's by just editing the `_accumulate_prediction` method to include a running total of the prediction squared allowing the variance to be determined later. I can provide a snippet showing this.

The extra step which is only really relevant if we don't grown trees to max depth or set `min_samples` is the variance within each leaf. If we train with `squared_error` criterion on a single output task we can also grab the leaf variance with a call to the impurity here but that's much more restrictive than keeping the square_means/sums/counts within each node. Particularly given the impurity in the multi output case is only stored having summed over all outputs (https://github.com/scikit-learn/scikit-learn/issues/27882). 

I think quantile regression forests are a philosophically different approach to the one proposed here but in general welcome them and can think about any implementation synergy.

Labels: New Feature, Needs Decision, RFC
[RFC] Leaf Level Variance in Multi Output Decision Trees

### Describe the workflow you want to enable

For single output RFR trained with the squared error criterion the impurity of the leaves can be used as a crude but useful estimate of the aleatoric uncertainty. In the multi output case the impurity is the sum over outputs hence this is no longer possible to estimate.

### Describe your proposed solution

Currently we can access the node values from `estimator.tree_.values` that stores the leaf means, I propose adding another data store or `estimator.tree_.square_means` that would store the square means of the leaves. Using these it is possible to work out the variance within the leaf that can then be used as a crude but useful estimate of the aleatoric uncertainty. The added benefit of doing this would be that this could be calculated for any choice of criterion rather than just the squared error. The principal issue is that it would double the amount of data stored.

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

This measure of the aleatoric uncertainty is widely used in the work of frank hutter in more recent versions of SMAC - https://ml.informatik.uni-freiburg.de/wp-content/uploads/papers/11-LION5-SMAC.pdf

COMMENT:
> For single output RFR trained with the squared error criterion the impurity of the leaves can be used as a crude but useful estimate of the aleatoric uncertainty.

Very crude, indeed.

> I propose adding another data store or estimator.tree_.square_means

Our trees are made efficient (memory and cpu) for the target functional being estimated, i.e. the mean or the median. Any added attribute per node increases the memory footprint. They are not made for uncertainty quantification of the estimates themselves.

With `tree.decision_path` and `tree.apply`, it is possible the say for each training sample in which node it ended up, see https://scikit-learn.org/stable/auto_examples/tree/plot_unveil_tree_structure.html#decision-path. Based on that, it should be possible to calculate the variance ex post.

Labels: New Feature, Needs Decision, RFC
[RFC] Leaf Level Variance in Multi Output Decision Trees

### Describe the workflow you want to enable

For single output RFR trained with the squared error criterion the impurity of the leaves can be used as a crude but useful estimate of the aleatoric uncertainty. In the multi output case the impurity is the sum over outputs hence this is no longer possible to estimate.

### Describe your proposed solution

Currently we can access the node values from `estimator.tree_.values` that stores the leaf means, I propose adding another data store or `estimator.tree_.square_means` that would store the square means of the leaves. Using these it is possible to work out the variance within the leaf that can then be used as a crude but useful estimate of the aleatoric uncertainty. The added benefit of doing this would be that this could be calculated for any choice of criterion rather than just the squared error. The principal issue is that it would double the amount of data stored.

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

This measure of the aleatoric uncertainty is widely used in the work of frank hutter in more recent versions of SMAC - https://ml.informatik.uni-freiburg.de/wp-content/uploads/papers/11-LION5-SMAC.pdf

COMMENT:
This issue is interesting and it is very related to [quantile-forests](https://github.com/zillow/quantile-forest/tree/main) where the training samples are stored in the leaf nodes. That definitely should not be turned on by default, cuz it can result in huge RAM requirement and disc requirement if the model is pickled.

However, for small datasets, where the user is interested in analyzing the leaf nodes, it could make sense to optionally store the training samples like quantile forests does.

I've played around with allowing this in [scikit-tree](https://docs.neurodata.io/scikit-tree/dev/generated/sktree.RandomForestClassifier.html#sktree.RandomForestClassifier.get_leaf_node_samples) optionally. Perhaps @CompRhys this might meet your needs.

What does scikit-learn think of adding the "option to store data samples in the leaf nodes", so the resultant quantile/uncertainty analyses are made easier for the end user? Since it is off by default, it would not impact RAM footprint.

Labels: New Feature, Needs Decision, RFC
[RFC] Leaf Level Variance in Multi Output Decision Trees

### Describe the workflow you want to enable

For single output RFR trained with the squared error criterion the impurity of the leaves can be used as a crude but useful estimate of the aleatoric uncertainty. In the multi output case the impurity is the sum over outputs hence this is no longer possible to estimate.

### Describe your proposed solution

Currently we can access the node values from `estimator.tree_.values` that stores the leaf means, I propose adding another data store or `estimator.tree_.square_means` that would store the square means of the leaves. Using these it is possible to work out the variance within the leaf that can then be used as a crude but useful estimate of the aleatoric uncertainty. The added benefit of doing this would be that this could be calculated for any choice of criterion rather than just the squared error. The principal issue is that it would double the amount of data stored.

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

This measure of the aleatoric uncertainty is widely used in the work of frank hutter in more recent versions of SMAC - https://ml.informatik.uni-freiburg.de/wp-content/uploads/papers/11-LION5-SMAC.pdf

COMMENT:
What does scikit-learn think of adding the "option to store data samples in the leaf nodes",

That’s a highly complex C/Cython surgery. I prefer to add such an uncertainty estimate with existing functionality (post fit) to an existing example.

Labels: New Feature, Needs Decision, RFC
[RFC] Leaf Level Variance in Multi Output Decision Trees

### Describe the workflow you want to enable

For single output RFR trained with the squared error criterion the impurity of the leaves can be used as a crude but useful estimate of the aleatoric uncertainty. In the multi output case the impurity is the sum over outputs hence this is no longer possible to estimate.

### Describe your proposed solution

Currently we can access the node values from `estimator.tree_.values` that stores the leaf means, I propose adding another data store or `estimator.tree_.square_means` that would store the square means of the leaves. Using these it is possible to work out the variance within the leaf that can then be used as a crude but useful estimate of the aleatoric uncertainty. The added benefit of doing this would be that this could be calculated for any choice of criterion rather than just the squared error. The principal issue is that it would double the amount of data stored.

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

This measure of the aleatoric uncertainty is widely used in the work of frank hutter in more recent versions of SMAC - https://ml.informatik.uni-freiburg.de/wp-content/uploads/papers/11-LION5-SMAC.pdf

COMMENT:
> That’s a highly complex C/Cython surgery. I prefer to add such an uncertainty estimate with existing functionality (post fit) to an existing example.

Is the idea that we store it somehow in a Python data structure by just sending samples down the leaf node?

I like this idea since it bypasses anything in Cython, and leveraging the data samples stored in Cython is potentially unnecessary for any of the applications of estimating quantiles and confidence intervals. 

To beat the dead bush again tho: Optionally storing data samples in leaf nodes enables more robust features for trees (when RAM is not an issue). And it is in-line w/ some of the scikit-learn pushes to enable more quantile regression. Rn using the MAE or related quantile criterion is one way to "do quantile-like fitting", but the more flexible way is just using Gini/MSE and then computing quantiles leveraging the data samples in the leaf nodes. This is essentially the proposed feature in this GH issue more-or-less.

Labels: New Feature, Needs Decision, RFC
DOC replace MAPE in lagged features example

A few improvements could be made on the new example of #25350:
- Mean absolute percentage error (MAPE) is used quite a lot. I propose to replace it, in particular if predicting/forecasting the mean value. Note that MAPE is optimized by the median of a distribution with pdf propotional to $\frac{f(y)}{y}$, where $f(y)$ is the pdf of the true distribution of the data.

- The `pinball_loss_50` is the same as `1/2 MAE`, this redundancy could be removed.

- A residual vs predicted does note really make sense for 5%- and 95%-quantile prediction.
  A reliability diagram for quantiles might be a  good replacement, see [model-diagnostics plot_reliability_diagram](https://lorentzenchr.github.io/model-diagnostics/reference/model_diagnostics/calibration/plots/#model_diagnostics.calibration.plots.plot_reliability_diagram). Note that this is not possible within current scikit-learn. Maybe the best next action is to add a little more explanation to the graphs.

COMMENT:
Hi @lorentzenchr, I would like to contribute to this issue.

> * Mean absolute percentage error (MAPE) is used quite a lot. I propose to replace it, in particular if predicting/forecasting the mean value. Note that MAPE is optimized by the median of a distribution with pdf propotional to $\frac{f(y)}{y}$, where $f(y)$ is the pdf of the true distribution of the data.

With respect to this, what alternate evaluation metric would you recommend replacing MAPE with? Also, do you suggest that we should replace MAPE with another metric within the `scoring` dictionary below?

https://github.com/scikit-learn/scikit-learn/blob/f59c16de7b0ca7b9db59113eb2112f03842bfde1/examples/applications/plot_time_series_lagged_features.py#L193-L200

> * The `pinball_loss_50` is the same as `1/2 MAE`, this redundancy could be removed.
> * A residual vs predicted does note really make sense for 5%- and 95%-quantile prediction.
A reliability diagram for quantiles might be a good replacement, see [model-diagnostics plot_reliability_diagram](https://lorentzenchr.github.io/model-diagnostics/reference/model_diagnostics/calibration/plots/#model_diagnostics.calibration.plots.plot_reliability_diagram). Note that this is not possible within current scikit-learn. Maybe the best next action is to add a little more explanation to the graphs.

With regards to these suggestions, I will work on incorporating the required changes to the example.

Labels: Documentation
DOC replace MAPE in lagged features example

A few improvements could be made on the new example of #25350:
- Mean absolute percentage error (MAPE) is used quite a lot. I propose to replace it, in particular if predicting/forecasting the mean value. Note that MAPE is optimized by the median of a distribution with pdf propotional to $\frac{f(y)}{y}$, where $f(y)$ is the pdf of the true distribution of the data.

- The `pinball_loss_50` is the same as `1/2 MAE`, this redundancy could be removed.

- A residual vs predicted does note really make sense for 5%- and 95%-quantile prediction.
  A reliability diagram for quantiles might be a  good replacement, see [model-diagnostics plot_reliability_diagram](https://lorentzenchr.github.io/model-diagnostics/reference/model_diagnostics/calibration/plots/#model_diagnostics.calibration.plots.plot_reliability_diagram). Note that this is not possible within current scikit-learn. Maybe the best next action is to add a little more explanation to the graphs.

COMMENT:
Just remove MAPE.

Labels: Documentation
HDBSCAN: Remove centroids_ attribute from API documentation

### Describe the issue linked to the documentation

The API documentation of `HDBSCAN` on the [scikit-learn website](https://scikit-learn.org/stable/modules/generated/sklearn.cluster.HDBSCAN.html#sklearn.cluster.HDBSCAN) lists `centroids_` as an attribute. However, this is not a valid attribute for `HDBSCAN` nor its parent classes `ClusterMixin` and `BaseEstimator` (scikit-learn Version 1.3.0). 

### Suggest a potential alternative/fix

`centroids_` should be removed from the attributes entry of the [scikit-learn website](https://scikit-learn.org/stable/modules/generated/sklearn.cluster.HDBSCAN.html#sklearn.cluster.HDBSCAN).

COMMENT:
The centroids are stored only when `store_centers` is set properly as documented:

```python
from sklearn.cluster import HDBSCAN
from sklearn.datasets import load_digits
X, _ = load_digits(return_X_y=True)
HDBSCAN(min_cluster_size=20, store_centers="centroid").fit(X).centroids_
```

Labels: Documentation, Needs Triage
⚠️ CI failed on Wheel builder ⚠️

**CI failed on [Wheel builder](https://github.com/scikit-learn/scikit-learn/actions/runs/7027741686)** (Nov 29, 2023)

COMMENT:
So apparently we have some failures with NumPy 2 here.
@ogrisel is it something known from the PR that have been open by @seberg?
I did not follow those unfortunately.

Labels: Needs Triage
⚠️ CI failed on Wheel builder ⚠️

**CI failed on [Wheel builder](https://github.com/scikit-learn/scikit-learn/actions/runs/7027741686)** (Nov 29, 2023)

COMMENT:
Grrrrrrrr, this is a new thing, indirectly related to bumping maxdims.  I also bumped MAXARGS, which is ABI compatible but changes the size of 1 or 2 objects.
I had somehow thought the fact that we had `checksize ignore` in:
```
    ctypedef class numpy.broadcast [object PyArrayMultiIterObject, check_size ignore]:
```
would ignore this, but apparently it only ignores it for growing structs.

Trying to think of how to best fix this, especially since I am not sure that fixing cython is viable as it would enforce Cython 3 more strictly potentially... In the public API, we could just make the struct end with `[]` (unspecified length).  That way, the actual size is always larger, and that is ignored.  Makes `sizeof(MultiIterObject)` wrong in a sense, but there is no point I can see for using that.

Labels: Needs Triage
⚠️ CI failed on Wheel builder ⚠️

**CI failed on [Wheel builder](https://github.com/scikit-learn/scikit-learn/actions/runs/7027741686)** (Nov 29, 2023)

COMMENT:
## CI is no longer failing! ✅

[Successful run](https://github.com/scikit-learn/scikit-learn/actions/runs/7041734769) on Nov 30, 2023

Labels: Needs Triage
⚠️ CI failed on Ubuntu_Atlas.ubuntu_atlas (last failure: Aug 28, 2025) ⚠️

**CI is still failing on [Ubuntu_Atlas.ubuntu_atlas](https://dev.azure.com/scikit-learn/scikit-learn/_build/results?buildId=79396&view=logs&j=689a1c8f-ff4e-5689-1a1a-6fa551ae9eba)** (Aug 28, 2025)
- test_float_precision[33-MiniBatchKMeans-dense]

COMMENT:
## CI is no longer failing! ✅

[Successful run](https://dev.azure.com/scikit-learn/scikit-learn/_build/results?buildId=79541&view=logs&j=689a1c8f-ff4e-5689-1a1a-6fa551ae9eba) on Sep 02, 2025

Labels: Build / CI
⚠️ CI failed on Ubuntu_Atlas.ubuntu_atlas (last failure: Aug 28, 2025) ⚠️

**CI is still failing on [Ubuntu_Atlas.ubuntu_atlas](https://dev.azure.com/scikit-learn/scikit-learn/_build/results?buildId=79396&view=logs&j=689a1c8f-ff4e-5689-1a1a-6fa551ae9eba)** (Aug 28, 2025)
- test_float_precision[33-MiniBatchKMeans-dense]

COMMENT:
For the record:

```python
../1/s/sklearn/cluster/tests/test_k_means.py:851
[...]
E       AssertionError: 
E       Not equal to tolerance rtol=0.0001, atol=0
E       
E       Mismatched elements: 1 / 1 (100%)
E       Max absolute difference: 0.04080632
E       Max relative difference: 0.00011865
E        x: array(343.971558)
E        y: array(343.930751)

actual     = array(343.97155762)
atol       = 0.0
desired    = array(343.9307513)
dtypes     = [dtype('float64'), dtype('float64')]
equal_nan  = True
err_msg    = ''
rtol       = 0.0001
verbose    = True
```

I assume we can just increase the `rtol` value for this test but this should be re-run with all random seeds on an atlas environment to find a suitable `rtol` value that is strict enough but not too much.

Labels: Build / CI
⚠️ CI failed on Ubuntu_Atlas.ubuntu_atlas (last failure: Aug 28, 2025) ⚠️

**CI is still failing on [Ubuntu_Atlas.ubuntu_atlas](https://dev.azure.com/scikit-learn/scikit-learn/_build/results?buildId=79396&view=logs&j=689a1c8f-ff4e-5689-1a1a-6fa551ae9eba)** (Aug 28, 2025)
- test_float_precision[33-MiniBatchKMeans-dense]

COMMENT:
I will push a fix for this.

Labels: Build / CI