Talip7/scikit-learn-issues-multilabel huggingface dataset:
Huber Loss for HistGradientBoostingRegressor

### Describe the workflow you want to enable

Huber loss is available as an option for `GradientBoostingRegressor` and works great when training on data with frequent outliers (thank you!). `HistGradientBoostingRegressor` however does not support Huber loss, which may be required when scaling to larger datasets. 

### Describe your proposed solution

Add HuberLoss as an option for the `HistGradientBoostingRegressor` class. 

### Describe alternatives you've considered, if relevant

Possibly allow custom loss functions for the `HistGradientBoostingRegressor`

### Additional context

_No response_

COMMENT:
@adrinjalali I suppose customisable loss-function is already implemented unless I am mistaken? I do see the value of adding this if it is already in the GradientBoostingRegressor not sure of any reasons against it?

Labels: New Feature, help wanted, Hard
Huber Loss for HistGradientBoostingRegressor

### Describe the workflow you want to enable

Huber loss is available as an option for `GradientBoostingRegressor` and works great when training on data with frequent outliers (thank you!). `HistGradientBoostingRegressor` however does not support Huber loss, which may be required when scaling to larger datasets. 

### Describe your proposed solution

Add HuberLoss as an option for the `HistGradientBoostingRegressor` class. 

### Describe alternatives you've considered, if relevant

Possibly allow custom loss functions for the `HistGradientBoostingRegressor`

### Additional context

_No response_

COMMENT:
> reasons against it

It adds an additional step of estimating the Huber loss parameter `delta` at each iteration.
What is the advantage of HGBT over GBT?
- Use of Hessian (faster fit)
  But: Similar to pinball/quantile loss, Huber is not smooth, so no Hessian.
- Feature binning (faster fit, maybe worse model performance)
  If outliers are your problem, do you have large enough datasets such that HGBT is worth it?

I’m not saying no. I just want to understand the modeler’s needs before blindly adding it.

Labels: New Feature, help wanted, Hard
Huber Loss for HistGradientBoostingRegressor

### Describe the workflow you want to enable

Huber loss is available as an option for `GradientBoostingRegressor` and works great when training on data with frequent outliers (thank you!). `HistGradientBoostingRegressor` however does not support Huber loss, which may be required when scaling to larger datasets. 

### Describe your proposed solution

Add HuberLoss as an option for the `HistGradientBoostingRegressor` class. 

### Describe alternatives you've considered, if relevant

Possibly allow custom loss functions for the `HistGradientBoostingRegressor`

### Additional context

_No response_

COMMENT:
> How many samples (rows)? 

Several hundred thousand samples. But each sample has a measurement error (expressed through sampling variance) that is 10-
100x the true variance in the labels; so the effective sample size is much smaller.

> How many features (columns)?

In the 40-80 range.

>  When you say L1 and L2 evaluation metrics are often better with a model minimizing Huber loss, is it on a separate hold-out/test set or on the training set? 

The L1/L2 scoring are applied to holdout/test set, where the loss is weighted by the inverse noise of each label (a metric proportional to the label's sampling variance-- the labels are themselves statistical estimates). 

> If you do cross validation, are the L1/L2 CV scores within the uncertainty/std_deviation of the other model (e.g. GradientBoostingRegressor with Huber and one with absolute error).

It does but using quantile loss narrows this gap.

Labels: New Feature, help wanted, Hard
Huber Loss for HistGradientBoostingRegressor

### Describe the workflow you want to enable

Huber loss is available as an option for `GradientBoostingRegressor` and works great when training on data with frequent outliers (thank you!). `HistGradientBoostingRegressor` however does not support Huber loss, which may be required when scaling to larger datasets. 

### Describe your proposed solution

Add HuberLoss as an option for the `HistGradientBoostingRegressor` class. 

### Describe alternatives you've considered, if relevant

Possibly allow custom loss functions for the `HistGradientBoostingRegressor`

### Additional context

_No response_

COMMENT:
L1L2 (Pseudo Huber) or LogCosh losses are smooth alternatives to huber loss, maybe we can look into them?

Xgboost have pseudo huber without additional parameters and it is stable and kind of easy to implement.

LogCosh is used in ANNs, and in my experience have some problems when calculating updates (the distribution of Hessians has a very high probability mass around 0, so your updates will "explode").

There is also Fair Loss which is a smooth alternative to L1 loss used in LGBM and it works flawlessly.

Labels: New Feature, help wanted, Hard
Huber Loss for HistGradientBoostingRegressor

### Describe the workflow you want to enable

Huber loss is available as an option for `GradientBoostingRegressor` and works great when training on data with frequent outliers (thank you!). `HistGradientBoostingRegressor` however does not support Huber loss, which may be required when scaling to larger datasets. 

### Describe your proposed solution

Add HuberLoss as an option for the `HistGradientBoostingRegressor` class. 

### Describe alternatives you've considered, if relevant

Possibly allow custom loss functions for the `HistGradientBoostingRegressor`

### Additional context

_No response_

COMMENT:
>> When you say L1 and L2 evaluation metrics are often better with a model minimizing Huber loss, is it on a separate hold-out/test set or on the training set?
>
>The L1/L2 scoring are applied to holdout/test set, where the loss is weighted by the inverse noise of each label (a metric proportional to the label's sampling variance-- the labels are themselves statistical estimates).

Could you provide unweighted test set estimates of L1 and L2 error? (Or compare them and report back)

Labels: New Feature, help wanted, Hard
Huber Loss for HistGradientBoostingRegressor

### Describe the workflow you want to enable

Huber loss is available as an option for `GradientBoostingRegressor` and works great when training on data with frequent outliers (thank you!). `HistGradientBoostingRegressor` however does not support Huber loss, which may be required when scaling to larger datasets. 

### Describe your proposed solution

Add HuberLoss as an option for the `HistGradientBoostingRegressor` class. 

### Describe alternatives you've considered, if relevant

Possibly allow custom loss functions for the `HistGradientBoostingRegressor`

### Additional context

_No response_

COMMENT:
I guess we could add Huber loss. As was already said, it would make HGBT more consistent with GBT. @scikit-learn/core-devs What do you think about inclusion?

Still, I would personally be interested to see a dataset where it is beneficial.

Labels: New Feature, help wanted, Hard
Huber Loss for HistGradientBoostingRegressor

### Describe the workflow you want to enable

Huber loss is available as an option for `GradientBoostingRegressor` and works great when training on data with frequent outliers (thank you!). `HistGradientBoostingRegressor` however does not support Huber loss, which may be required when scaling to larger datasets. 

### Describe your proposed solution

Add HuberLoss as an option for the `HistGradientBoostingRegressor` class. 

### Describe alternatives you've considered, if relevant

Possibly allow custom loss functions for the `HistGradientBoostingRegressor`

### Additional context

_No response_

COMMENT:
The inclusion of the Huber loss is sensible to me.

Labels: New Feature, help wanted, Hard
Huber Loss for HistGradientBoostingRegressor

### Describe the workflow you want to enable

Huber loss is available as an option for `GradientBoostingRegressor` and works great when training on data with frequent outliers (thank you!). `HistGradientBoostingRegressor` however does not support Huber loss, which may be required when scaling to larger datasets. 

### Describe your proposed solution

Add HuberLoss as an option for the `HistGradientBoostingRegressor` class. 

### Describe alternatives you've considered, if relevant

Possibly allow custom loss functions for the `HistGradientBoostingRegressor`

### Additional context

_No response_

COMMENT:
> I guess we could add Huber loss. As was already said, it would make HGBT more consistent with GBT. @scikit-learn/core-devs What do you think about inclusion?

+1 for Huber loss on HGBT. It's a classic loss

Labels: New Feature, help wanted, Hard
Huber Loss for HistGradientBoostingRegressor

### Describe the workflow you want to enable

Huber loss is available as an option for `GradientBoostingRegressor` and works great when training on data with frequent outliers (thank you!). `HistGradientBoostingRegressor` however does not support Huber loss, which may be required when scaling to larger datasets. 

### Describe your proposed solution

Add HuberLoss as an option for the `HistGradientBoostingRegressor` class. 

### Describe alternatives you've considered, if relevant

Possibly allow custom loss functions for the `HistGradientBoostingRegressor`

### Additional context

_No response_

COMMENT:
I think Huber loss would be a good addition. Achieves closer feature parity, and is a classic that is commonly used.

Labels: New Feature, help wanted, Hard
Huber Loss for HistGradientBoostingRegressor

### Describe the workflow you want to enable

Huber loss is available as an option for `GradientBoostingRegressor` and works great when training on data with frequent outliers (thank you!). `HistGradientBoostingRegressor` however does not support Huber loss, which may be required when scaling to larger datasets. 

### Describe your proposed solution

Add HuberLoss as an option for the `HistGradientBoostingRegressor` class. 

### Describe alternatives you've considered, if relevant

Possibly allow custom loss functions for the `HistGradientBoostingRegressor`

### Additional context

_No response_

COMMENT:
How big is the downside of not having hessians? Asking because maybe we should implement one of the losses @glevv mentioned instead and add some documentation telling people that if they are looking for huber loss they should use L1L2 (Pseudo Huber) or LogCosh (which ever we implement).

Having functions which are basically "huber loss" as well as smooth sounds like "best of both worlds" - which makes me think we should explicitly reject it/discuss it.

Labels: New Feature, help wanted, Hard
Huber Loss for HistGradientBoostingRegressor

### Describe the workflow you want to enable

Huber loss is available as an option for `GradientBoostingRegressor` and works great when training on data with frequent outliers (thank you!). `HistGradientBoostingRegressor` however does not support Huber loss, which may be required when scaling to larger datasets. 

### Describe your proposed solution

Add HuberLoss as an option for the `HistGradientBoostingRegressor` class. 

### Describe alternatives you've considered, if relevant

Possibly allow custom loss functions for the `HistGradientBoostingRegressor`

### Additional context

_No response_

COMMENT:
If the problem is having hessians, is there any issue with implementing the pseudo-Huber loss? 

Having that seems like it would be useful. 

I don't believe we have that implemented within HGBT. https://scikit-learn.org/stable/modules/generated/sklearn.ensemble.HistGradientBoostingRegressor.html

Labels: New Feature, help wanted, Hard
Huber Loss for HistGradientBoostingRegressor

### Describe the workflow you want to enable

Huber loss is available as an option for `GradientBoostingRegressor` and works great when training on data with frequent outliers (thank you!). `HistGradientBoostingRegressor` however does not support Huber loss, which may be required when scaling to larger datasets. 

### Describe your proposed solution

Add HuberLoss as an option for the `HistGradientBoostingRegressor` class. 

### Describe alternatives you've considered, if relevant

Possibly allow custom loss functions for the `HistGradientBoostingRegressor`

### Additional context

_No response_

COMMENT:
> How big is the downside of not having hessians?

In HGBT, we already have quantile/pinball loss which is not smooth. Not having hessians means one needs an explicit line search step which is (already) done in https://github.com/scikit-learn/scikit-learn/blob/bd60ea7219defd223d26173637984a52c0734536/sklearn/ensemble/_hist_gradient_boosting/gradient_boosting.py#L66

As I already explained, for Huber loss we need an additional step of estimating the Huber loss parameter in each iteration (see Freedman's Greedy Function Approximation). In GBT it is done here https://github.com/scikit-learn/scikit-learn/blob/bd60ea7219defd223d26173637984a52c0734536/sklearn/ensemble/_gb.py#L266

Labels: New Feature, help wanted, Hard
Huber Loss for HistGradientBoostingRegressor

### Describe the workflow you want to enable

Huber loss is available as an option for `GradientBoostingRegressor` and works great when training on data with frequent outliers (thank you!). `HistGradientBoostingRegressor` however does not support Huber loss, which may be required when scaling to larger datasets. 

### Describe your proposed solution

Add HuberLoss as an option for the `HistGradientBoostingRegressor` class. 

### Describe alternatives you've considered, if relevant

Possibly allow custom loss functions for the `HistGradientBoostingRegressor`

### Additional context

_No response_

COMMENT:
Is it a "big deal" in terms of performance to have to do the line search and parameter estimation? As in, would Huber loss be significantly slower than one of the other "not quite Huber but similar" losses? I unfortunately have no idea how to estimate this :(

Labels: New Feature, help wanted, Hard
Huber Loss for HistGradientBoostingRegressor

### Describe the workflow you want to enable

Huber loss is available as an option for `GradientBoostingRegressor` and works great when training on data with frequent outliers (thank you!). `HistGradientBoostingRegressor` however does not support Huber loss, which may be required when scaling to larger datasets. 

### Describe your proposed solution

Add HuberLoss as an option for the `HistGradientBoostingRegressor` class. 

### Describe alternatives you've considered, if relevant

Possibly allow custom loss functions for the `HistGradientBoostingRegressor`

### Additional context

_No response_

COMMENT:
I did no dedicated benchmarking of timings. Usually, estimating quantiles takes longer, noticable for users.

I am really critical of smoothed versions of losses. A loss function in (statistical) estimation is a foundational object with much more severe consequences than other hyperparameters. The loss specifies what you actually predict. Is it a mean or a median or something else? If it's something else and nobody can tell what, I personally would avoid it (unless it is clearly motivated by a business metric). Huber is well investigated (well, in ML it is not that much investigated to be honest).

Conclusion: Let's stay with Huber as `GradientBoostingRegressor` and (linear model) `HuberRegressor`.

Labels: New Feature, help wanted, Hard
Huber Loss for HistGradientBoostingRegressor

### Describe the workflow you want to enable

Huber loss is available as an option for `GradientBoostingRegressor` and works great when training on data with frequent outliers (thank you!). `HistGradientBoostingRegressor` however does not support Huber loss, which may be required when scaling to larger datasets. 

### Describe your proposed solution

Add HuberLoss as an option for the `HistGradientBoostingRegressor` class. 

### Describe alternatives you've considered, if relevant

Possibly allow custom loss functions for the `HistGradientBoostingRegressor`

### Additional context

_No response_

COMMENT:
I see @lorentzenchr points with computing the Hessian as well as maintaining as unbiased of a loss estimator as possible. @AndrewYRoyal you say that the ```HuberLoss``` could be important when scaling to larger datasets, perhaps you have one in mind that we could take a look at and try some tests to see if it really is beneficial?

Labels: New Feature, help wanted, Hard
Make `sklearn.metrics._scorer._MultimetricScorer` part of the public API

### Describe the workflow you want to enable

This tool is great to run multiple scorers on a single estimator thanks to the caching mechanism. It is a bummer that it is not part of the public API.

### Describe your proposed solution

Make it part of the public API

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_

COMMENT:
What exactly is your usecase @MarcBresson ? We don't necessarily have to make the class public. We can allow `make_scorer` to accept a dictionary / list of scorers and return the object instead, for a nicer API maybe. It can probably even simplify a few places inside scikit-learn itself.

Labels: Enhancement, API, Needs Decision
Make `sklearn.metrics._scorer._MultimetricScorer` part of the public API

### Describe the workflow you want to enable

This tool is great to run multiple scorers on a single estimator thanks to the caching mechanism. It is a bummer that it is not part of the public API.

### Describe your proposed solution

Make it part of the public API

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_

COMMENT:
Hello, thank you for your patience :)

- I believe people rarely computes only one metric, and always deal with multiple at a time
- our use case is exactly that, we are building (in a way) an autoML wrapper around Scikit learn (a bit like what skrub is doing). In that sense, my very selfish goal is to decrease the maintenance required for our repos, and hence, stop dealing with what the scikit learn team has already spent time developing. I spent countless hours exploring the source code of scikit learn, and it lead to this discovery.

Labels: Enhancement, API, Needs Decision
当selector = VarianceThreshold(threshold=0.1)和selector = VarianceThreshold()输出的结果不一样

### Describe the bug

import numpy as np
X = np.arange(30,dtype=float).reshape((10, 3))
X[:,1] = 1
from sklearn.feature_selection import VarianceThreshold
vt = VarianceThreshold(threshold=0.01)
xt = vt.fit_transform(X)
# 未设置阈值时，可能未实际计算方差
vt1 = VarianceThreshold()
vt1.fit(X)                # 先调用fit方法
print(vt1.variances_)     # 现在可以安全访问

# 设置阈值后强制计算
vt2 = VarianceThreshold(threshold=0.01)
vt2.fit(X)  # 实际执行计算
print(vt2.variances_)     # 输出正确值
vt = VarianceThreshold(threshold=0.01)
vt.fit(X)  # 确保实际计算
print(vt.variances_)
# 检查方差计算一致性
manual_var = np.var(X, axis=0, ddof=0)
sklearn_var = vt.variances_
if not np.allclose(manual_var, sklearn_var):
    print(f"警告：方差计算不一致！手动:{manual_var}，sklearn:{sklearn_var}")
    # 确保使用最新稳定版
import sklearn
print("scikit-learn版本:", sklearn.__version__)  # 应 ≥ 1.0

[27.  0. 27.]
[74.25  0.   74.25]
[74.25  0.   74.25]
scikit-learn版本: 1.7.0

### Steps/Code to Reproduce

[27.  0. 27.]
[74.25  0.   74.25]
[74.25  0.   74.25]
scikit-learn版本: 1.7.0

### Expected Results

[27.  0. 27.]
[74.25  0.   74.25]
[74.25  0.   74.25]
scikit-learn版本: 1.7.0

### Actual Results

[27.  0. 27.]
[74.25  0.   74.25]
[74.25  0.   74.25]
scikit-learn版本: 1.7.0

### Versions

```shell
System:
    python: 3.12.1 (tags/v3.12.1:2305ca5, Dec  7 2023, 22:03:25) [MSC v.1937 64 bit (AMD64)]
executable: c:\Users\wp\Desktop\python312\python.exe
   machine: Windows-10-10.0.19045-SP0

Python dependencies:
      sklearn: 1.7.0
          pip: 25.1.1
   setuptools: 78.1.0
        numpy: 1.26.0
        scipy: 1.13.1
       Cython: None
       pandas: 2.2.3
   matplotlib: 3.10.1
       joblib: 1.4.2
threadpoolctl: 3.6.0

Built with OpenMP: True

threadpoolctl info:
       user_api: blas
   internal_api: openblas
    num_threads: 4
         prefix: libopenblas
...
       filepath: C:\Users\wp\Desktop\python312\Lib\site-packages\scipy.libs\libopenblas_v0.3.27--3aa239bc726cfb0bd8e5330d8d4c15c6.dll
        version: 0.3.27
threading_layer: pthreads
   architecture: Haswell
```

COMMENT:
Our working language is English. I am going to close this one, but feel free to reopen when you have translated your comment into English.

Labels: Bug, Needs Triage
当selector = VarianceThreshold(threshold=0.1)和selector = VarianceThreshold()输出的结果不一样

### Describe the bug

import numpy as np
X = np.arange(30,dtype=float).reshape((10, 3))
X[:,1] = 1
from sklearn.feature_selection import VarianceThreshold
vt = VarianceThreshold(threshold=0.01)
xt = vt.fit_transform(X)
# 未设置阈值时，可能未实际计算方差
vt1 = VarianceThreshold()
vt1.fit(X)                # 先调用fit方法
print(vt1.variances_)     # 现在可以安全访问

# 设置阈值后强制计算
vt2 = VarianceThreshold(threshold=0.01)
vt2.fit(X)  # 实际执行计算
print(vt2.variances_)     # 输出正确值
vt = VarianceThreshold(threshold=0.01)
vt.fit(X)  # 确保实际计算
print(vt.variances_)
# 检查方差计算一致性
manual_var = np.var(X, axis=0, ddof=0)
sklearn_var = vt.variances_
if not np.allclose(manual_var, sklearn_var):
    print(f"警告：方差计算不一致！手动:{manual_var}，sklearn:{sklearn_var}")
    # 确保使用最新稳定版
import sklearn
print("scikit-learn版本:", sklearn.__version__)  # 应 ≥ 1.0

[27.  0. 27.]
[74.25  0.   74.25]
[74.25  0.   74.25]
scikit-learn版本: 1.7.0

### Steps/Code to Reproduce

[27.  0. 27.]
[74.25  0.   74.25]
[74.25  0.   74.25]
scikit-learn版本: 1.7.0

### Expected Results

[27.  0. 27.]
[74.25  0.   74.25]
[74.25  0.   74.25]
scikit-learn版本: 1.7.0

### Actual Results

[27.  0. 27.]
[74.25  0.   74.25]
[74.25  0.   74.25]
scikit-learn版本: 1.7.0

### Versions

```shell
System:
    python: 3.12.1 (tags/v3.12.1:2305ca5, Dec  7 2023, 22:03:25) [MSC v.1937 64 bit (AMD64)]
executable: c:\Users\wp\Desktop\python312\python.exe
   machine: Windows-10-10.0.19045-SP0

Python dependencies:
      sklearn: 1.7.0
          pip: 25.1.1
   setuptools: 78.1.0
        numpy: 1.26.0
        scipy: 1.13.1
       Cython: None
       pandas: 2.2.3
   matplotlib: 3.10.1
       joblib: 1.4.2
threadpoolctl: 3.6.0

Built with OpenMP: True

threadpoolctl info:
       user_api: blas
   internal_api: openblas
    num_threads: 4
         prefix: libopenblas
...
       filepath: C:\Users\wp\Desktop\python312\Lib\site-packages\scipy.libs\libopenblas_v0.3.27--3aa239bc726cfb0bd8e5330d8d4c15c6.dll
        version: 0.3.27
threading_layer: pthreads
   architecture: Haswell
```

COMMENT:
>>>import numpy as np
>>>import sklearn
>>>print（“scikit-learn version：”， sklearn.version） # 

>>>X = np.arange（30，dtype=float）.reshape（（10， 3））
>>>X[：，1] = 1
>>>from sklearn.feature_selection import VarianceThreshold
>>>vt = VarianceThreshold（threshold=0.01）

>>>xt = vt.fit_transform（X）
>>>vt1 = VarianceThreshold（）
>>>vt1.fit（X）
>>>print（vt1.variances_） 

>>>vt2 = VarianceThreshold（threshold=0.01）
>>>vt2.fit（X）
>>>print（vt2.variances_）
>>>vt = VarianceThreshold（threshold=0.01）
>>>vt.fit（X）
>>>print（vt.variances_）


scikit-learn version： 1.7.0
[27. 0.27.]
[74.25 0. 74.25]
[74.25 0. 74.25]

why  vt1 = VarianceThreshold（）and  vt2 = VarianceThreshold（threshold=0.01）output  different

Labels: Bug, Needs Triage