Talip7/scikit-learn-issues-multilabel huggingface dataset:
Standard "Total Variance" Scaler

### Desired feature

A preprocessor that removes the mean for each feature, and then scales the total variance of the dataset, rather than the variance of each feature, to 1.

### Proposed Solution

A new preprocessor that operates like StandardScaler but automatically scales total-variance instead of the variance of individual feature

### Possible Alternatives

A new input parameter for StandardScaler that allows the user to set the variance of each feature, or which allows the user to identify groups of features to be considered individual "macro" features

### Additional context

Intended use case for situations where more than one feature (column) is associated with the same data-concept (like when including multiple points in space for sea surface temperature in the Pacific and also multiple points in space for sea level pressure in the Atlantic)

COMMENT:
What @adrinjalali means, I guess, is
1. Add a parameter (naming to be defined) to `StandardScaler`.
2. Use it in combination with a `ColumnTransformer` like this
  ```python
  col_trans = ColumnTransformer(
      [
          ("group1", StandardScaler(total_variance=True), ["col_1", "col_2", "col_3"]),
          ("group2", StandardScaler(total_variance=True), ["col_3", "col_4", "col_5"]),
          ("std_scaled", StandardScaler(total_variance=False), ["col_5", "col_6", "col_7"]),
      ]
  )
  ```

Labels: New Feature
Standard "Total Variance" Scaler

### Desired feature

A preprocessor that removes the mean for each feature, and then scales the total variance of the dataset, rather than the variance of each feature, to 1.

### Proposed Solution

A new preprocessor that operates like StandardScaler but automatically scales total-variance instead of the variance of individual feature

### Possible Alternatives

A new input parameter for StandardScaler that allows the user to set the variance of each feature, or which allows the user to identify groups of features to be considered individual "macro" features

### Additional context

Intended use case for situations where more than one feature (column) is associated with the same data-concept (like when including multiple points in space for sea surface temperature in the Pacific and also multiple points in space for sea level pressure in the Atlantic)

COMMENT:
That would definitely work, if ColumnTransformer is also given `inverse_transform` functionality and, I hope, also a way to either pass in a subset of the columns for `transform` and `inverse_transform`, or to subset the ColumnTransformer object itself it by column/transform name. I see that, if a dataframe with variable names is used, one can pass in data for transformation with more columns than were used in the fit. 

It would be really wonderful to be able to do the same with fewer columns, and even when a dataframe is not used. In this case, the user could pass in the names of the columns, or identify which named transformations will be provided with data.

Labels: New Feature
Standard "Total Variance" Scaler

### Desired feature

A preprocessor that removes the mean for each feature, and then scales the total variance of the dataset, rather than the variance of each feature, to 1.

### Proposed Solution

A new preprocessor that operates like StandardScaler but automatically scales total-variance instead of the variance of individual feature

### Possible Alternatives

A new input parameter for StandardScaler that allows the user to set the variance of each feature, or which allows the user to identify groups of features to be considered individual "macro" features

### Additional context

Intended use case for situations where more than one feature (column) is associated with the same data-concept (like when including multiple points in space for sea surface temperature in the Pacific and also multiple points in space for sea level pressure in the Atlantic)

COMMENT:
Should I make a new thread for my proposed changes to ColumnTransformer, or is it fine here?

Labels: New Feature
Standard "Total Variance" Scaler

### Desired feature

A preprocessor that removes the mean for each feature, and then scales the total variance of the dataset, rather than the variance of each feature, to 1.

### Proposed Solution

A new preprocessor that operates like StandardScaler but automatically scales total-variance instead of the variance of individual feature

### Possible Alternatives

A new input parameter for StandardScaler that allows the user to set the variance of each feature, or which allows the user to identify groups of features to be considered individual "macro" features

### Additional context

Intended use case for situations where more than one feature (column) is associated with the same data-concept (like when including multiple points in space for sea surface temperature in the Pacific and also multiple points in space for sea level pressure in the Atlantic)

COMMENT:
Different topic => difference/new issue.

Labels: New Feature
Standard "Total Variance" Scaler

### Desired feature

A preprocessor that removes the mean for each feature, and then scales the total variance of the dataset, rather than the variance of each feature, to 1.

### Proposed Solution

A new preprocessor that operates like StandardScaler but automatically scales total-variance instead of the variance of individual feature

### Possible Alternatives

A new input parameter for StandardScaler that allows the user to set the variance of each feature, or which allows the user to identify groups of features to be considered individual "macro" features

### Additional context

Intended use case for situations where more than one feature (column) is associated with the same data-concept (like when including multiple points in space for sea surface temperature in the Pacific and also multiple points in space for sea level pressure in the Atlantic)

COMMENT:
ok!
Then, in this issue, it should be decided whether to add a new Scaler, or to add an argument `total_variance` to the existing StandardScaler (and perhaps other similar preprocessors as well). 
For continued conversation on Expanded ColumnTransformer functionality, see #28130

Labels: New Feature
Standard "Total Variance" Scaler

### Desired feature

A preprocessor that removes the mean for each feature, and then scales the total variance of the dataset, rather than the variance of each feature, to 1.

### Proposed Solution

A new preprocessor that operates like StandardScaler but automatically scales total-variance instead of the variance of individual feature

### Possible Alternatives

A new input parameter for StandardScaler that allows the user to set the variance of each feature, or which allows the user to identify groups of features to be considered individual "macro" features

### Additional context

Intended use case for situations where more than one feature (column) is associated with the same data-concept (like when including multiple points in space for sea surface temperature in the Pacific and also multiple points in space for sea level pressure in the Atlantic)

COMMENT:
Already answered in https://github.com/scikit-learn/scikit-learn/issues/27957#issuecomment-1877068489:
> I'm suggesting to add a constructor argument to StandardScaler to calculate statistics from the whole data, that would solve the issue.

Labels: New Feature
Standard "Total Variance" Scaler

### Desired feature

A preprocessor that removes the mean for each feature, and then scales the total variance of the dataset, rather than the variance of each feature, to 1.

### Proposed Solution

A new preprocessor that operates like StandardScaler but automatically scales total-variance instead of the variance of individual feature

### Possible Alternatives

A new input parameter for StandardScaler that allows the user to set the variance of each feature, or which allows the user to identify groups of features to be considered individual "macro" features

### Additional context

Intended use case for situations where more than one feature (column) is associated with the same data-concept (like when including multiple points in space for sea surface temperature in the Pacific and also multiple points in space for sea level pressure in the Atlantic)

COMMENT:
OK! Sorry I wasn't sure if that was a suggestion or decision. 
Do we want me to make a pull request for it, or is someone else interested in implementing it?

Labels: New Feature
Standard "Total Variance" Scaler

### Desired feature

A preprocessor that removes the mean for each feature, and then scales the total variance of the dataset, rather than the variance of each feature, to 1.

### Proposed Solution

A new preprocessor that operates like StandardScaler but automatically scales total-variance instead of the variance of individual feature

### Possible Alternatives

A new input parameter for StandardScaler that allows the user to set the variance of each feature, or which allows the user to identify groups of features to be considered individual "macro" features

### Additional context

Intended use case for situations where more than one feature (column) is associated with the same data-concept (like when including multiple points in space for sea surface temperature in the Pacific and also multiple points in space for sea level pressure in the Atlantic)

COMMENT:
I understand that it is so easy to conclude that there is a decision. @adrinjalali suggested and I gave my 👍 to it which results in 2 core devs agreeing. That is already a strong hint. Similar for the question about ColumnTransformer, several core devs had concerns about that.

@rebeccaherman1 If you want to contribute a PR that would be nice. I'm not aware of anybody else planning to do so.

Labels: New Feature
Standard "Total Variance" Scaler

### Desired feature

A preprocessor that removes the mean for each feature, and then scales the total variance of the dataset, rather than the variance of each feature, to 1.

### Proposed Solution

A new preprocessor that operates like StandardScaler but automatically scales total-variance instead of the variance of individual feature

### Possible Alternatives

A new input parameter for StandardScaler that allows the user to set the variance of each feature, or which allows the user to identify groups of features to be considered individual "macro" features

### Additional context

Intended use case for situations where more than one feature (column) is associated with the same data-concept (like when including multiple points in space for sea surface temperature in the Pacific and also multiple points in space for sea level pressure in the Atlantic)

COMMENT:
ok @lorentzenchr. Which other preprocessors would we also want to have this functionality?

Labels: New Feature
Standard "Total Variance" Scaler

### Desired feature

A preprocessor that removes the mean for each feature, and then scales the total variance of the dataset, rather than the variance of each feature, to 1.

### Proposed Solution

A new preprocessor that operates like StandardScaler but automatically scales total-variance instead of the variance of individual feature

### Possible Alternatives

A new input parameter for StandardScaler that allows the user to set the variance of each feature, or which allows the user to identify groups of features to be considered individual "macro" features

### Additional context

Intended use case for situations where more than one feature (column) is associated with the same data-concept (like when including multiple points in space for sea surface temperature in the Pacific and also multiple points in space for sea level pressure in the Atlantic)

COMMENT:
@rebeccaherman1 I don't follow. If you want an option for total variance in `StandardScaler`, then a PR for that is welcome. No other preprocessors / estimators are discussed here.

Labels: New Feature
Standard "Total Variance" Scaler

### Desired feature

A preprocessor that removes the mean for each feature, and then scales the total variance of the dataset, rather than the variance of each feature, to 1.

### Proposed Solution

A new preprocessor that operates like StandardScaler but automatically scales total-variance instead of the variance of individual feature

### Possible Alternatives

A new input parameter for StandardScaler that allows the user to set the variance of each feature, or which allows the user to identify groups of features to be considered individual "macro" features

### Additional context

Intended use case for situations where more than one feature (column) is associated with the same data-concept (like when including multiple points in space for sea surface temperature in the Pacific and also multiple points in space for sea level pressure in the Atlantic)

COMMENT:
@lorentzenchr for instance, @adrinjalali mentioned `RubustScaler` above

Labels: New Feature
Standard "Total Variance" Scaler

### Desired feature

A preprocessor that removes the mean for each feature, and then scales the total variance of the dataset, rather than the variance of each feature, to 1.

### Proposed Solution

A new preprocessor that operates like StandardScaler but automatically scales total-variance instead of the variance of individual feature

### Possible Alternatives

A new input parameter for StandardScaler that allows the user to set the variance of each feature, or which allows the user to identify groups of features to be considered individual "macro" features

### Additional context

Intended use case for situations where more than one feature (column) is associated with the same data-concept (like when including multiple points in space for sea surface temperature in the Pacific and also multiple points in space for sea level pressure in the Atlantic)

COMMENT:
@rebeccaherman1 you can start a PR with `StandardScaler` and when that gets to a conclusion, we can do the same for `RobustScaler`

Labels: New Feature
Standard "Total Variance" Scaler

### Desired feature

A preprocessor that removes the mean for each feature, and then scales the total variance of the dataset, rather than the variance of each feature, to 1.

### Proposed Solution

A new preprocessor that operates like StandardScaler but automatically scales total-variance instead of the variance of individual feature

### Possible Alternatives

A new input parameter for StandardScaler that allows the user to set the variance of each feature, or which allows the user to identify groups of features to be considered individual "macro" features

### Additional context

Intended use case for situations where more than one feature (column) is associated with the same data-concept (like when including multiple points in space for sea surface temperature in the Pacific and also multiple points in space for sea level pressure in the Atlantic)

COMMENT:
Hey @rebeccaherman1, thanks for proposing! That is a useful feature, I'm also interested in it. 
I see that there is a "green light" to implement it, but I don't see any development branches linked to it. 
So I wonder if there is some code already, maybe you could wrap it as a PR? In a meanwhile, I'll check if there is a simple way to  make it done - please reach me out if you would like to contribute to the solution.

Labels: New Feature
Unable to control warning logs generated by GridSearchCV fit method when setting n_jobs to >1 for parallel processing

### Describe the workflow you want to enable

I am running GridSearchCV with n_jobs set to value which is > 1. The grid search is writing log of convergence and other warnings to the console. I want to control those logs so that I can write them to a logfile instead of spamming console. I tried tweaking with joblib but it didn't worked. Can you provide us feature to control logs generated by sklearn gridsearch in case of using parallelism.

### Describe your proposed solution

I don't have any solution at this moment

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_

COMMENT:
This is kind of duplicate of https://github.com/scikit-learn/scikit-learn/issues/27676 where we want to provide callback to be able what you request.

I am closing this issue then.

Labels: New Feature, Needs Triage
CalibratedClassifierCV gives a NotFittedError when accessing the underlying XGBoostClassifier feature_importances property

### Describe the bug

I am using CalibratedClassifierCV and XGBoost in a Pipeline and was able to train the model and use it to make predictions, etc. But I cannot access the underlying property of the XGBoost model. 


### Steps/Code to Reproduce

```
from sklearn.pipeline import Pipeline
import xgboost as xgb
from sklearn.calibration import CalibratedClassifierCV
from sklearn.datasets import make_classification

X, y = make_classification(n_features=2, n_redundant=0, n_informative=2, random_state=1, n_clusters_per_class=1)

xgb_model = xgb.XGBClassifier(n_jobs = -1,
                              objective = "binary:logistic", 
                              eval_metric = 'auc')

calibrated_xgb = CalibratedClassifierCV(base_estimator=xgb_model, cv=5, method='isotonic')

calibrated_xgb_pipeline = Pipeline(steps=[('model', calibrated_xgb)])

calibrated_xgb_pipeline.fit(X, y)

calibrated_xgb_pipeline.named_steps["model"].base_estimator.feature_importances_
```

### Expected Results

Feature importance should be returned

### Actual Results

```
---------------------------------------------------------------------------
NotFittedError                            Traceback (most recent call last)
Cell In[206], line 21
     17 calibrated_xgb_pipeline = Pipeline(steps=[('model', calibrated_xgb)])
     19 calibrated_xgb_pipeline.fit(X, y)
---> 21 calibrated_xgb_pipeline.named_steps["model"].base_estimator.feature_importances_

File ~/anaconda3/envs/python3/lib/python3.10/site-packages/xgboost/sklearn.py:1278, in XGBModel.feature_importances_(self)
   1263 @property
   1264 def feature_importances_(self) -> np.ndarray:
   1265     """Feature importances property, return depends on `importance_type`
   1266     parameter. When model trained with multi-class/multi-label/multi-target dataset,
   1267     the feature importance is "averaged" over all targets. The "average" is defined
   (...)
   1276 
   1277     """
-> 1278     b: Booster = self.get_booster()
   1280     def dft() -> str:
   1281         return "weight" if self.booster == "gblinear" else "gain"

File ~/anaconda3/envs/python3/lib/python3.10/site-packages/xgboost/sklearn.py:649, in XGBModel.get_booster(self)
    646 if not self.__sklearn_is_fitted__():
    647     from sklearn.exceptions import NotFittedError
--> 649     raise NotFittedError("need to call fit or load_model beforehand")
    650 return self._Booster

NotFittedError: need to call fit or load_model beforehand
```

### Versions

```shell
I am using sklearn 1.3.2 and xgboost 1.7.6.
```

COMMENT:
There was a typo (missing s) in my code which I just fixed :sweat_smile:

Labels: Bug, Needs Triage
CalibratedClassifierCV gives a NotFittedError when accessing the underlying XGBoostClassifier feature_importances property

### Describe the bug

I am using CalibratedClassifierCV and XGBoost in a Pipeline and was able to train the model and use it to make predictions, etc. But I cannot access the underlying property of the XGBoost model. 


### Steps/Code to Reproduce

```
from sklearn.pipeline import Pipeline
import xgboost as xgb
from sklearn.calibration import CalibratedClassifierCV
from sklearn.datasets import make_classification

X, y = make_classification(n_features=2, n_redundant=0, n_informative=2, random_state=1, n_clusters_per_class=1)

xgb_model = xgb.XGBClassifier(n_jobs = -1,
                              objective = "binary:logistic", 
                              eval_metric = 'auc')

calibrated_xgb = CalibratedClassifierCV(base_estimator=xgb_model, cv=5, method='isotonic')

calibrated_xgb_pipeline = Pipeline(steps=[('model', calibrated_xgb)])

calibrated_xgb_pipeline.fit(X, y)

calibrated_xgb_pipeline.named_steps["model"].base_estimator.feature_importances_
```

### Expected Results

Feature importance should be returned

### Actual Results

```
---------------------------------------------------------------------------
NotFittedError                            Traceback (most recent call last)
Cell In[206], line 21
     17 calibrated_xgb_pipeline = Pipeline(steps=[('model', calibrated_xgb)])
     19 calibrated_xgb_pipeline.fit(X, y)
---> 21 calibrated_xgb_pipeline.named_steps["model"].base_estimator.feature_importances_

File ~/anaconda3/envs/python3/lib/python3.10/site-packages/xgboost/sklearn.py:1278, in XGBModel.feature_importances_(self)
   1263 @property
   1264 def feature_importances_(self) -> np.ndarray:
   1265     """Feature importances property, return depends on `importance_type`
   1266     parameter. When model trained with multi-class/multi-label/multi-target dataset,
   1267     the feature importance is "averaged" over all targets. The "average" is defined
   (...)
   1276 
   1277     """
-> 1278     b: Booster = self.get_booster()
   1280     def dft() -> str:
   1281         return "weight" if self.booster == "gblinear" else "gain"

File ~/anaconda3/envs/python3/lib/python3.10/site-packages/xgboost/sklearn.py:649, in XGBModel.get_booster(self)
    646 if not self.__sklearn_is_fitted__():
    647     from sklearn.exceptions import NotFittedError
--> 649     raise NotFittedError("need to call fit or load_model beforehand")
    650 return self._Booster

NotFittedError: need to call fit or load_model beforehand
```

### Versions

```shell
I am using sklearn 1.3.2 and xgboost 1.7.6.
```

COMMENT:
Oy! I see. Didn't even notice. Thank you! It now works.

Labels: Bug, Needs Triage
Allowing to group infrequent categories in `HistGradientBoosting`

### Describe the workflow you want to enable

`HistGradientBoostingClassifier` and `HistGradientBoostingRegressor` have built-in support for categorical features and use an `OrdinalEncoder` to encode them. Each feature must have less than `max_bins` (255) categories and if there are more we get a `ValueError`.

Would it be useful to add a parameter allowing to group together the least frequent categories when there are too many (and thus avoid the error)? This amounts to setting `max_categories=self.max_bins` in the internal `OrdinalEncoder`.

The workflow I would like to enable is fitting a HGB estimator when a categorical feature has a few too many categories, without needing to encode them myself beforehand.

### Describe your proposed solution

the HGB estimators would have a parameter (maybe something like `group_infrequent_categories`) to control whether they should display the current behavior (raise an error) or group together rare categories (with the OrdinalEncoder's `max_categories`) when there are more than `max_bins` categories

### Describe alternatives you've considered, if relevant

 It is relatively easy to apply an OrdinalEncoder before but (i) it is more verbose, and we have to use a pipeline and probably a ColumnTransformer, (ii) we lose the dtypes of pandas  dataframe columns so we have to specify the categorical columns manually instead of using `categorical_features="from_dtype"`, (iii) we end up doing the categorical feature encoding twice, once before and once inside the estimator

### Additional context

_No response_

COMMENT:
I thought we already do this (group infrequent categories when there are more than 255). Maybe we only do this in some other part of scikit-learn though.

-> I think this would be a useful feature to have.

It would be nice if we could find a way to build this kind of "infrequent category handling" once and then use it in all the places where we deal with categories. Instead of having to build it several times. I'm not sure how realistic it is to do this, but it would be interesting to explore I think.

Labels: New Feature