Talip7/scikit-learn-issues-multilabel huggingface dataset:
Is the check of strict convergence in KMeans too expensive for the benefits ?

### Describe the bug

In `KMeans` scikit-learn defines [`strict_convergence`](https://github.com/scikit-learn/scikit-learn/blob/main/sklearn/cluster/_kmeans.py#L701) as the event of producing the same label assignments at two successive iterations.

When this happens, it means convergence for both labels and centroids (set aside possible oscillations due to numerical instability, were the iterations to continue).

But checking for strict convergence seems to be somewhat expensive (one loop over the last two label assignments of each sample per iteration), and if the user properly set `tol` it doesn't seem necessary at all ?

Checking for strict convergence seems to really help when `tol==0`. With `tol==0` I've seen cases of endless oscillations around 0 because of numerical instability, but never reaching 0, and finally terminating at `max_iter` iterations.

For the general case, isn't it detrimental to performance though ? one can expect the performance cost to be significant for small dimensions of data, for which an additional pass on a column is marginally more expensive.

So I would maybe suggest the following improvements:

- [ ] enable automatically the strict convergence checks only if `tol==0` (or if `tol` is "very small")
- [ ] maybe expose to the user the choice of enabling strict convergence at each iteration ?


### Versions

```shell
1.3
```

COMMENT:
> enable automatically the strict convergence checks only if tol==0 (or if tol is "very small")

I am worried about oscillations for very small yet non-zero tols but then harcoding a threshold for "very small" would feel too magic / implicit.

> maybe expose to the user the choice of enabling strict convergence at each iteration ?

If there is a measurable performance impact, then I think I would be in favor of this option. But I would rather avoid exposing too many options to the users if there is no clear performance benefit in doing so.

Labels: Bug, module:cluster, Needs Investigation
Is the check of strict convergence in KMeans too expensive for the benefits ?

### Describe the bug

In `KMeans` scikit-learn defines [`strict_convergence`](https://github.com/scikit-learn/scikit-learn/blob/main/sklearn/cluster/_kmeans.py#L701) as the event of producing the same label assignments at two successive iterations.

When this happens, it means convergence for both labels and centroids (set aside possible oscillations due to numerical instability, were the iterations to continue).

But checking for strict convergence seems to be somewhat expensive (one loop over the last two label assignments of each sample per iteration), and if the user properly set `tol` it doesn't seem necessary at all ?

Checking for strict convergence seems to really help when `tol==0`. With `tol==0` I've seen cases of endless oscillations around 0 because of numerical instability, but never reaching 0, and finally terminating at `max_iter` iterations.

For the general case, isn't it detrimental to performance though ? one can expect the performance cost to be significant for small dimensions of data, for which an additional pass on a column is marginally more expensive.

So I would maybe suggest the following improvements:

- [ ] enable automatically the strict convergence checks only if `tol==0` (or if `tol` is "very small")
- [ ] maybe expose to the user the choice of enabling strict convergence at each iteration ?


### Versions

```shell
1.3
```

COMMENT:
/cc @jeremiedbb

Labels: Bug, module:cluster, Needs Investigation
Is the check of strict convergence in KMeans too expensive for the benefits ?

### Describe the bug

In `KMeans` scikit-learn defines [`strict_convergence`](https://github.com/scikit-learn/scikit-learn/blob/main/sklearn/cluster/_kmeans.py#L701) as the event of producing the same label assignments at two successive iterations.

When this happens, it means convergence for both labels and centroids (set aside possible oscillations due to numerical instability, were the iterations to continue).

But checking for strict convergence seems to be somewhat expensive (one loop over the last two label assignments of each sample per iteration), and if the user properly set `tol` it doesn't seem necessary at all ?

Checking for strict convergence seems to really help when `tol==0`. With `tol==0` I've seen cases of endless oscillations around 0 because of numerical instability, but never reaching 0, and finally terminating at `max_iter` iterations.

For the general case, isn't it detrimental to performance though ? one can expect the performance cost to be significant for small dimensions of data, for which an additional pass on a column is marginally more expensive.

So I would maybe suggest the following improvements:

- [ ] enable automatically the strict convergence checks only if `tol==0` (or if `tol` is "very small")
- [ ] maybe expose to the user the choice of enabling strict convergence at each iteration ?


### Versions

```shell
1.3
```

COMMENT:
IIRC when adding this strict convergnce policy, I checked that it did not slow down the iterations.
I just checked again and did not see a measurable difference. Maybe on a machine with many cores, for a problem with small n_features and n_clusters and large n_samples.

Here's a measurement of np.array_equal:
```py
import numpy as np
a = np.arange(100000)
b = np.arange(100000)
%timeit np.array_equal(a, b)
# 35.4 μs ± 313 ns
```

Labels: Bug, module:cluster, Needs Investigation
Please add **fit_params for SequentialFeatureSelector.fit method

### Describe the workflow you want to enable

I need to use `StratifiedGroupKFold` on my data.

Unlike `cross_validate`, `SequentialFeatureSelector.fit` method has no `**fit_params` or `groups` parameter, so I can't use group split.



### Describe your proposed solution

- Adding `**fit_params` or `groups` parameter to `SequentialFeatureSelector.fit` method.


### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_

COMMENT:
Thank you for opening the issue. According to the [triaging workflow](https://scikit-learn.org/dev/developers/bug_triaging.html#triaging-operations-for-members-of-the-core-and-contributor-experience-teams), I am closing this issue because it is a duplicate of #25236. You are welcome to continue the conversation there.

Labels: New Feature, Needs Triage
ENH HistGradientBoosting estimators should have `.feature_importances_` attribute

### Describe the workflow you want to enable

The practicality of using the `.feature_importances_` attribute to superficially analyze global explanations of your classifier is really useful, IMHO.

Right after evaluating the model, running something like
```python
pd.DataFrame(model.feature_importances_, index=X_train.columns, columns=["feat_imp"]).sort_values("feat_imp", ascending=False)
```
is the first thing I do.

Furthermore, some variable selection strategies assume that your estimator has this attribute, and it seems suboptimal to me to deprive ourselves of using `HistGradientBoosting` with these techniques.
```python
from sklearn.feature_selection import SelectFromModel
from sklearn.ensemble import HistGradientBoostingClassifier
from lightgbm import LGBMClassifier

X = [[ 0.87, -1.34,  0.31 ],
     [-2.79, -0.02, -0.85 ],
     [-1.34, -0.48, -2.55 ],
     [ 1.92,  1.48,  0.65 ]]
y = [0, 1, 0, 1]
selector = SelectFromModel(estimator=LGBMClassifier()).fit(X, y)
selector.get_support()
>>> array([ True,  True,  True])

selector = SelectFromModel(estimator=HistGradientBoostingClassifier()).fit(X, y)
selector.get_support()
>>> ValueError: when `importance_getter=='auto'`, the underlying estimator HistGradientBoostingClassifier should have `coef_` or `feature_importances_` attribute. Either pass a fitted estimator to feature selector or call fit before calling transform.
```

I understand the concerns raised by @ogrisel's [comment in the original PR that implemented HistGradientBoosting](https://github.com/scikit-learn/scikit-learn/pull/12807#issuecomment-462802273), but I believe we have more points in favor of its implementation than its omission. I'd love to hear your thoughts. :)

### Describe your proposed solution

_No response_

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_

COMMENT:
I think it could useful to find a way to compute the mean decrease in impurity (MDI) but the `feature_importances_` API is too limiting because it forces us to only consider training set statistics.

So let's find a way to implement/fix #20059 in general (and in particular how to compute MDI on test data https://github.com/scikit-learn/scikit-learn/issues/20059#issuecomment-1013387157) and then we can discuss how to make this available for HistGradientBoosting* models.

Labels: New Feature, API, Needs Decision
⚠️ CI failed on Wheel builder ⚠️

**CI failed on [Wheel builder](https://github.com/scikit-learn/scikit-learn/actions/runs/4268187511)** (Feb 25, 2023)

COMMENT:
## CI is no longer failing! ✅

[Successful run](https://github.com/scikit-learn/scikit-learn/actions/runs/4273200865) on Feb 26, 2023

Labels: Needs Triage
CalibratedClassifierCV fails on lgbm fit_params

Hi,

I'm trying to use CalibratedClassifierCV to calibrate the probabilities from a LGBM model. The issue is that when I try CalibratedClassifierCV with eval_set, I get an error ValueError: Found input variables with inconsistent numbers of samples: [43364, 1] which is caused by check_consistent_length function in validation.py The input to eval set is [X_valid,Y_valid] where both X_valid,Y_valid are arrays with different shape. Since this format is a requirement for LGBM eval set https://lightgbm.readthedocs.io/en/latest/pythonapi/lightgbm.LGBMClassifier.html, I am not sure how will I make the check_consistent_length pass in my scenario. Full code is given below:

```python
import lightgbm as lgbm
from sklearn.calibration import CalibratedClassifierCV

def _train_lgbm_model():
    model = lgbm.LGBMClassifier(**parameters.lgbm_params)
    fit_params = {
        "eval_set": [(X_valid, Y_valid)],
        "eval_names": ["train", "valid"],
        "verbose": 0,
    }
    return model, fit_params

model = CalibratedClassifierCV(model, method='isotonic')
trained_model = model.fit(X_train, Y_train, **fit_param)

Error: ValueError: Found input variables with inconsistent numbers of samples: [43364, 1]

``` 
X_train.shape = (43364, 152)
X_valid.shape = (43364,)
Y_train.shape = (43364, 152)
Y_valid.shape = (43364,)

COMMENT:
Once we have metadata routing with would have an explicit way to tell how to broadcast such parameters to the base estimators. However as far as I know, the current state of SLEP6 does not have a way to tell whether or not we want to apply cross-validation indexing and therefore disable the length consistency check for the things that are not meant to be the sample-aligned fit params.

Maybe @adrinjalali knows if this topic was already addressed in SLEP6 or not.

Labels: Bug
CalibratedClassifierCV fails on lgbm fit_params

Hi,

I'm trying to use CalibratedClassifierCV to calibrate the probabilities from a LGBM model. The issue is that when I try CalibratedClassifierCV with eval_set, I get an error ValueError: Found input variables with inconsistent numbers of samples: [43364, 1] which is caused by check_consistent_length function in validation.py The input to eval set is [X_valid,Y_valid] where both X_valid,Y_valid are arrays with different shape. Since this format is a requirement for LGBM eval set https://lightgbm.readthedocs.io/en/latest/pythonapi/lightgbm.LGBMClassifier.html, I am not sure how will I make the check_consistent_length pass in my scenario. Full code is given below:

```python
import lightgbm as lgbm
from sklearn.calibration import CalibratedClassifierCV

def _train_lgbm_model():
    model = lgbm.LGBMClassifier(**parameters.lgbm_params)
    fit_params = {
        "eval_set": [(X_valid, Y_valid)],
        "eval_names": ["train", "valid"],
        "verbose": 0,
    }
    return model, fit_params

model = CalibratedClassifierCV(model, method='isotonic')
trained_model = model.fit(X_train, Y_train, **fit_param)

Error: ValueError: Found input variables with inconsistent numbers of samples: [43364, 1]

``` 
X_train.shape = (43364, 152)
X_valid.shape = (43364,)
Y_train.shape = (43364, 152)
Y_valid.shape = (43364,)

COMMENT:
I think this is orthogonal to SLEP006, and a bug in `CalibratedClassifierCV`, which has these lines:

https://github.com/scikit-learn/scikit-learn/blob/c991e30c96ace1565604b429de22e36ed6b1e7bd/sklearn/calibration.py#L311-L312

I'm not sure why this check is there. IMO the logic should be like `GridSearchCV`, where we split the data where the length is the same as `y`, otherwise pass unchanged.

cc @glemaitre

Labels: Bug
CalibratedClassifierCV fails on lgbm fit_params

Hi,

I'm trying to use CalibratedClassifierCV to calibrate the probabilities from a LGBM model. The issue is that when I try CalibratedClassifierCV with eval_set, I get an error ValueError: Found input variables with inconsistent numbers of samples: [43364, 1] which is caused by check_consistent_length function in validation.py The input to eval set is [X_valid,Y_valid] where both X_valid,Y_valid are arrays with different shape. Since this format is a requirement for LGBM eval set https://lightgbm.readthedocs.io/en/latest/pythonapi/lightgbm.LGBMClassifier.html, I am not sure how will I make the check_consistent_length pass in my scenario. Full code is given below:

```python
import lightgbm as lgbm
from sklearn.calibration import CalibratedClassifierCV

def _train_lgbm_model():
    model = lgbm.LGBMClassifier(**parameters.lgbm_params)
    fit_params = {
        "eval_set": [(X_valid, Y_valid)],
        "eval_names": ["train", "valid"],
        "verbose": 0,
    }
    return model, fit_params

model = CalibratedClassifierCV(model, method='isotonic')
trained_model = model.fit(X_train, Y_train, **fit_param)

Error: ValueError: Found input variables with inconsistent numbers of samples: [43364, 1]

``` 
X_train.shape = (43364, 152)
X_valid.shape = (43364,)
Y_train.shape = (43364, 152)
Y_valid.shape = (43364,)

COMMENT:
@adrinjalali - Thanks for your response Adrin, this was my conclusion as well. What would be the path forward to resolve this bug?

Labels: Bug
CalibratedClassifierCV fails on lgbm fit_params

Hi,

I'm trying to use CalibratedClassifierCV to calibrate the probabilities from a LGBM model. The issue is that when I try CalibratedClassifierCV with eval_set, I get an error ValueError: Found input variables with inconsistent numbers of samples: [43364, 1] which is caused by check_consistent_length function in validation.py The input to eval set is [X_valid,Y_valid] where both X_valid,Y_valid are arrays with different shape. Since this format is a requirement for LGBM eval set https://lightgbm.readthedocs.io/en/latest/pythonapi/lightgbm.LGBMClassifier.html, I am not sure how will I make the check_consistent_length pass in my scenario. Full code is given below:

```python
import lightgbm as lgbm
from sklearn.calibration import CalibratedClassifierCV

def _train_lgbm_model():
    model = lgbm.LGBMClassifier(**parameters.lgbm_params)
    fit_params = {
        "eval_set": [(X_valid, Y_valid)],
        "eval_names": ["train", "valid"],
        "verbose": 0,
    }
    return model, fit_params

model = CalibratedClassifierCV(model, method='isotonic')
trained_model = model.fit(X_train, Y_train, **fit_param)

Error: ValueError: Found input variables with inconsistent numbers of samples: [43364, 1]

``` 
X_train.shape = (43364, 152)
X_valid.shape = (43364,)
Y_train.shape = (43364, 152)
Y_valid.shape = (43364,)

COMMENT:
I'll open a PR to propose a fix @Sidshroff

Labels: Bug
CalibratedClassifierCV fails on lgbm fit_params

Hi,

I'm trying to use CalibratedClassifierCV to calibrate the probabilities from a LGBM model. The issue is that when I try CalibratedClassifierCV with eval_set, I get an error ValueError: Found input variables with inconsistent numbers of samples: [43364, 1] which is caused by check_consistent_length function in validation.py The input to eval set is [X_valid,Y_valid] where both X_valid,Y_valid are arrays with different shape. Since this format is a requirement for LGBM eval set https://lightgbm.readthedocs.io/en/latest/pythonapi/lightgbm.LGBMClassifier.html, I am not sure how will I make the check_consistent_length pass in my scenario. Full code is given below:

```python
import lightgbm as lgbm
from sklearn.calibration import CalibratedClassifierCV

def _train_lgbm_model():
    model = lgbm.LGBMClassifier(**parameters.lgbm_params)
    fit_params = {
        "eval_set": [(X_valid, Y_valid)],
        "eval_names": ["train", "valid"],
        "verbose": 0,
    }
    return model, fit_params

model = CalibratedClassifierCV(model, method='isotonic')
trained_model = model.fit(X_train, Y_train, **fit_param)

Error: ValueError: Found input variables with inconsistent numbers of samples: [43364, 1]

``` 
X_train.shape = (43364, 152)
X_valid.shape = (43364,)
Y_train.shape = (43364, 152)
Y_valid.shape = (43364,)

COMMENT:
I looked at other estimators and indeed we never check that `fit_params` are sample-aligned but only propagate. I could not find the reasoning in the original PR, we were a bit sloppy regarding this aspect. I think a PR removing this check is fine.

Labels: Bug
CalibratedClassifierCV fails on lgbm fit_params

Hi,

I'm trying to use CalibratedClassifierCV to calibrate the probabilities from a LGBM model. The issue is that when I try CalibratedClassifierCV with eval_set, I get an error ValueError: Found input variables with inconsistent numbers of samples: [43364, 1] which is caused by check_consistent_length function in validation.py The input to eval set is [X_valid,Y_valid] where both X_valid,Y_valid are arrays with different shape. Since this format is a requirement for LGBM eval set https://lightgbm.readthedocs.io/en/latest/pythonapi/lightgbm.LGBMClassifier.html, I am not sure how will I make the check_consistent_length pass in my scenario. Full code is given below:

```python
import lightgbm as lgbm
from sklearn.calibration import CalibratedClassifierCV

def _train_lgbm_model():
    model = lgbm.LGBMClassifier(**parameters.lgbm_params)
    fit_params = {
        "eval_set": [(X_valid, Y_valid)],
        "eval_names": ["train", "valid"],
        "verbose": 0,
    }
    return model, fit_params

model = CalibratedClassifierCV(model, method='isotonic')
trained_model = model.fit(X_train, Y_train, **fit_param)

Error: ValueError: Found input variables with inconsistent numbers of samples: [43364, 1]

``` 
X_train.shape = (43364, 152)
X_valid.shape = (43364,)
Y_train.shape = (43364, 152)
Y_valid.shape = (43364,)

COMMENT:
Thanks @adrinjalali for fixing the issue. Which version of sklearn would this new fix be available from?

Labels: Bug
CalibratedClassifierCV fails on lgbm fit_params

Hi,

I'm trying to use CalibratedClassifierCV to calibrate the probabilities from a LGBM model. The issue is that when I try CalibratedClassifierCV with eval_set, I get an error ValueError: Found input variables with inconsistent numbers of samples: [43364, 1] which is caused by check_consistent_length function in validation.py The input to eval set is [X_valid,Y_valid] where both X_valid,Y_valid are arrays with different shape. Since this format is a requirement for LGBM eval set https://lightgbm.readthedocs.io/en/latest/pythonapi/lightgbm.LGBMClassifier.html, I am not sure how will I make the check_consistent_length pass in my scenario. Full code is given below:

```python
import lightgbm as lgbm
from sklearn.calibration import CalibratedClassifierCV

def _train_lgbm_model():
    model = lgbm.LGBMClassifier(**parameters.lgbm_params)
    fit_params = {
        "eval_set": [(X_valid, Y_valid)],
        "eval_names": ["train", "valid"],
        "verbose": 0,
    }
    return model, fit_params

model = CalibratedClassifierCV(model, method='isotonic')
trained_model = model.fit(X_train, Y_train, **fit_param)

Error: ValueError: Found input variables with inconsistent numbers of samples: [43364, 1]

``` 
X_train.shape = (43364, 152)
X_valid.shape = (43364,)
Y_train.shape = (43364, 152)
Y_valid.shape = (43364,)

COMMENT:
1.3 @Sidshroff , which should be out in 1-2 months. In the meantime you can use it by installing our nightly builds: https://scikit-learn.org/stable/developers/advanced_installation.html#installing-nightly-builds

Labels: Bug
Decision Tree Regressor giving different result for scaled and unscaled data

The decision tree regressor outputs different tree for scaled and unscaled data. This is true for the random forest regressor also. As per my understanding scaling data should not change the results for decision tree or random forest regressor. I am attaching the ipynb file of my analysis. In the analysis I have generated a synthetic dataset where the input data is in different scales. The decision tree is then made using the unscaled data, minmax scaled data and arbitrarily scaled data. The arbitrary scaling simply multiplies each input with a suitable value to bring all the inputs in the same order of magnitude. It is seen that the tree formed by using the the minmax scaled and arbitrarily scaled data is same but the one formed by the unscaled data is different. The same results are reflected when using random forest. I am unable to understand this result. Therefore, an explanation would be of help. The trees for unscaled, minmax scaled and arbitrary scaled data are also attached with names test_unscaled.png, test_scaled.png and test_arscaled.png respectively.
[test_tree.zip](https://github.com/scikit-learn/scikit-learn/files/10826456/test_tree.zip)

COMMENT:
I assume this is only due to numerical error to the scaling as already discussed in https://github.com/scikit-learn/scikit-learn/issues/25495

Labels: Needs Triage
⚠️ CI failed on Linux_Docker.debian_atlas_32bit (last failure: Sep 03, 2024) ⚠️

**CI is still failing on [Linux_Docker.debian_atlas_32bit](https://dev.azure.com/scikit-learn/scikit-learn/_build/results?buildId=69809&view=logs&j=aabdcdc3-bb64-5414-b357-ed024fe8659e)** (Sep 03, 2024)
- test_spectral_clustering_with_arpack_amg_solvers
- test_img_to_graph
- test_imputation_mean_median[csc_matrix]
- test_imputation_mean_median[csc_array]
- test_robust_scale_axis1

COMMENT:
## CI is no longer failing! ✅

[Successful run](https://dev.azure.com/scikit-learn/scikit-learn/_build/results?buildId=69841&view=logs&j=aabdcdc3-bb64-5414-b357-ed024fe8659e) on Sep 04, 2024

Labels: Build / CI, Numerical Stability
⚠️ CI failed on Linux_Docker.debian_atlas_32bit (last failure: Sep 03, 2024) ⚠️

**CI is still failing on [Linux_Docker.debian_atlas_32bit](https://dev.azure.com/scikit-learn/scikit-learn/_build/results?buildId=69809&view=logs&j=aabdcdc3-bb64-5414-b357-ed024fe8659e)** (Sep 03, 2024)
- test_spectral_clustering_with_arpack_amg_solvers
- test_img_to_graph
- test_imputation_mean_median[csc_matrix]
- test_imputation_mean_median[csc_array]
- test_robust_scale_axis1

COMMENT:
This was an real failure with 32bit Linux, when `SKLEARN_TESTS_GLOBAL_RANDOM_SEED="50"`.

Labels: Build / CI, Numerical Stability
⚠️ CI failed on Linux_Docker.debian_atlas_32bit (last failure: Sep 03, 2024) ⚠️

**CI is still failing on [Linux_Docker.debian_atlas_32bit](https://dev.azure.com/scikit-learn/scikit-learn/_build/results?buildId=69809&view=logs&j=aabdcdc3-bb64-5414-b357-ed024fe8659e)** (Sep 03, 2024)
- test_spectral_clustering_with_arpack_amg_solvers
- test_img_to_graph
- test_imputation_mean_median[csc_matrix]
- test_imputation_mean_median[csc_array]
- test_robust_scale_axis1

COMMENT:
It seems that we still have numerical stability problems in either `FastICA.fit_transform` or in `FastICA.inverse_transform`.

Labels: Build / CI, Numerical Stability