Talip7/scikit-learn-issues-multilabel huggingface dataset:
Implement p-value splitting criterion for Decision Trees

### Describe the workflow you want to enable

The current list of valid criterions for Decision Trees are:

{“squared_error”, “friedman_mse”, “absolute_error”, “poisson”}

With regard to regression problems, I have run into numerous situations where I would very much like to make splits on the data based on a t-test statistics comparing the means between the two split subsamples.

The primary issue with using MSE or Friedman MSE is that the quality of a split is dependent on both the marginal impact of splitting on a particularly variable, as well as the frequency in which that split is triggered. It's not uncommon where I want to identify splits that have highly different means, even though that splitting rule would come into effect rarely.

See this post (I am not the author) for more information on why splitting on p-values could be valuable: http://erikerlandson.github.io/blog/2016/05/26/measuring-decision-tree-split-quality-with-test-statistic-p-values/

### Describe your proposed solution

The solution would be to add a new splitting criterion called "p-value" or "t-test" or something similar. The quality of a split would be measured based on either the p-value or the magnitude of the t-statistic.

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_

COMMENT:
> It seems an elegant way to give awareness of samples size during the decision to split or not. I am curious to know how it works on several tasks. Someone has more info (bench, papers) ? And/or intention to implement it ?

Additionally, p-values could be used as a regularisation parameter. For example, if there exists no split where p < 0.01, then we end building the tree there and mark it as a leaf node.

Labels: New Feature, module:tree, Needs Decision - Include Feature
Implement p-value splitting criterion for Decision Trees

### Describe the workflow you want to enable

The current list of valid criterions for Decision Trees are:

{“squared_error”, “friedman_mse”, “absolute_error”, “poisson”}

With regard to regression problems, I have run into numerous situations where I would very much like to make splits on the data based on a t-test statistics comparing the means between the two split subsamples.

The primary issue with using MSE or Friedman MSE is that the quality of a split is dependent on both the marginal impact of splitting on a particularly variable, as well as the frequency in which that split is triggered. It's not uncommon where I want to identify splits that have highly different means, even though that splitting rule would come into effect rarely.

See this post (I am not the author) for more information on why splitting on p-values could be valuable: http://erikerlandson.github.io/blog/2016/05/26/measuring-decision-tree-split-quality-with-test-statistic-p-values/

### Describe your proposed solution

The solution would be to add a new splitting criterion called "p-value" or "t-test" or something similar. The quality of a split would be measured based on either the p-value or the magnitude of the t-statistic.

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_

COMMENT:
You mention regression metrics. Are you sure it is not more suitable for classification one ? (or both ?)

Labels: New Feature, module:tree, Needs Decision - Include Feature
Implement p-value splitting criterion for Decision Trees

### Describe the workflow you want to enable

The current list of valid criterions for Decision Trees are:

{“squared_error”, “friedman_mse”, “absolute_error”, “poisson”}

With regard to regression problems, I have run into numerous situations where I would very much like to make splits on the data based on a t-test statistics comparing the means between the two split subsamples.

The primary issue with using MSE or Friedman MSE is that the quality of a split is dependent on both the marginal impact of splitting on a particularly variable, as well as the frequency in which that split is triggered. It's not uncommon where I want to identify splits that have highly different means, even though that splitting rule would come into effect rarely.

See this post (I am not the author) for more information on why splitting on p-values could be valuable: http://erikerlandson.github.io/blog/2016/05/26/measuring-decision-tree-split-quality-with-test-statistic-p-values/

### Describe your proposed solution

The solution would be to add a new splitting criterion called "p-value" or "t-test" or something similar. The quality of a split would be measured based on either the p-value or the magnitude of the t-statistic.

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_

COMMENT:
> You mention regression metrics. Are you sure it is not more suitable for classification one ? (or both ?)

I would say it's most intuitive and suitable for regression problems, but it would also be applicable to binary classification. Multi-class classification probably doesn't make sense I think, at least, not with a t-test. Perhaps another test could be used.

Labels: New Feature, module:tree, Needs Decision - Include Feature
Implement p-value splitting criterion for Decision Trees

### Describe the workflow you want to enable

The current list of valid criterions for Decision Trees are:

{“squared_error”, “friedman_mse”, “absolute_error”, “poisson”}

With regard to regression problems, I have run into numerous situations where I would very much like to make splits on the data based on a t-test statistics comparing the means between the two split subsamples.

The primary issue with using MSE or Friedman MSE is that the quality of a split is dependent on both the marginal impact of splitting on a particularly variable, as well as the frequency in which that split is triggered. It's not uncommon where I want to identify splits that have highly different means, even though that splitting rule would come into effect rarely.

See this post (I am not the author) for more information on why splitting on p-values could be valuable: http://erikerlandson.github.io/blog/2016/05/26/measuring-decision-tree-split-quality-with-test-statistic-p-values/

### Describe your proposed solution

The solution would be to add a new splitting criterion called "p-value" or "t-test" or something similar. The quality of a split would be measured based on either the p-value or the magnitude of the t-statistic.

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_

COMMENT:
So how do we get the ball rolling on implementing this update?

Labels: New Feature, module:tree, Needs Decision - Include Feature
Implement p-value splitting criterion for Decision Trees

### Describe the workflow you want to enable

The current list of valid criterions for Decision Trees are:

{“squared_error”, “friedman_mse”, “absolute_error”, “poisson”}

With regard to regression problems, I have run into numerous situations where I would very much like to make splits on the data based on a t-test statistics comparing the means between the two split subsamples.

The primary issue with using MSE or Friedman MSE is that the quality of a split is dependent on both the marginal impact of splitting on a particularly variable, as well as the frequency in which that split is triggered. It's not uncommon where I want to identify splits that have highly different means, even though that splitting rule would come into effect rarely.

See this post (I am not the author) for more information on why splitting on p-values could be valuable: http://erikerlandson.github.io/blog/2016/05/26/measuring-decision-tree-split-quality-with-test-statistic-p-values/

### Describe your proposed solution

The solution would be to add a new splitting criterion called "p-value" or "t-test" or something similar. The quality of a split would be measured based on either the p-value or the magnitude of the t-statistic.

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_

COMMENT:
I think you need to search for an article with a lot of citations to convince scikit-learn core contributors it should be added. And more, the implementation should be distant enough from what is already inside.
I understand this is an elegant splitting criterion on paper, but we have already mechanisms in decision trees to avoid overfitting (max number of levels, min number of samples to split a node ...). Those simple thresholds have been proven statistically robust and computing efficient.

Labels: New Feature, module:tree, Needs Decision - Include Feature
Implement p-value splitting criterion for Decision Trees

### Describe the workflow you want to enable

The current list of valid criterions for Decision Trees are:

{“squared_error”, “friedman_mse”, “absolute_error”, “poisson”}

With regard to regression problems, I have run into numerous situations where I would very much like to make splits on the data based on a t-test statistics comparing the means between the two split subsamples.

The primary issue with using MSE or Friedman MSE is that the quality of a split is dependent on both the marginal impact of splitting on a particularly variable, as well as the frequency in which that split is triggered. It's not uncommon where I want to identify splits that have highly different means, even though that splitting rule would come into effect rarely.

See this post (I am not the author) for more information on why splitting on p-values could be valuable: http://erikerlandson.github.io/blog/2016/05/26/measuring-decision-tree-split-quality-with-test-statistic-p-values/

### Describe your proposed solution

The solution would be to add a new splitting criterion called "p-value" or "t-test" or something similar. The quality of a split would be measured based on either the p-value or the magnitude of the t-statistic.

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_

COMMENT:
As already mentioned, please have a look at the inclusion criteria https://scikit-learn.org/stable/faq.html#what-are-the-inclusion-criteria-for-new-algorithms.

> The primary issue with using MSE or Friedman MSE is that the quality of a split is dependent on both the marginal impact of splitting on a particularly variable, as well as the frequency in which that split is triggered. It's not uncommon where I want to identify splits that have highly different means, even though that splitting rule would come into effect rarely.

From this description, I do not fully understand what the issue with MSE is. Note that MSE as splitting criterion reduces to the sample variances of left and right split.

Labels: New Feature, module:tree, Needs Decision - Include Feature
Implement p-value splitting criterion for Decision Trees

### Describe the workflow you want to enable

The current list of valid criterions for Decision Trees are:

{“squared_error”, “friedman_mse”, “absolute_error”, “poisson”}

With regard to regression problems, I have run into numerous situations where I would very much like to make splits on the data based on a t-test statistics comparing the means between the two split subsamples.

The primary issue with using MSE or Friedman MSE is that the quality of a split is dependent on both the marginal impact of splitting on a particularly variable, as well as the frequency in which that split is triggered. It's not uncommon where I want to identify splits that have highly different means, even though that splitting rule would come into effect rarely.

See this post (I am not the author) for more information on why splitting on p-values could be valuable: http://erikerlandson.github.io/blog/2016/05/26/measuring-decision-tree-split-quality-with-test-statistic-p-values/

### Describe your proposed solution

The solution would be to add a new splitting criterion called "p-value" or "t-test" or something similar. The quality of a split would be measured based on either the p-value or the magnitude of the t-statistic.

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_

COMMENT:
I close as the discussion got silent. Feel free to reopen in case there are new arguments.

Labels: New Feature, module:tree, Needs Decision - Include Feature
Please change Confusion Matrix Heatmap cmap.

The current **cmap** is `viridis` is un-readable. This really needs to be changed to a much better, and readable cmap.

COMMENT:
When using `metrics.ConfusionMatrixDisplay` you can directly set the `cmap` to whatever you prefer through the `cmap` parameter passed to any of its plotting methods. `viridis` is a strong default since it is one of the more accessible and colorblindness-friendly cmaps.

Labels: Needs Triage
Different dataset for different models when using StackingClassifier

### Describe the workflow you want to enable

Since stacking is a combination of different models, the different models might have different features i.e is is rather often that the same dataset does not fit different models (or it is a sub-optimal dataset for one of the models).

E.g a "LightGBM" classifier can handle categorical values by it self, where we in LogisticRegression have to handle that in the dataset, which then (might) lead to two different datasets, and we can't just parse one dataset to `StackingClassifier` for the training. Or, an SVM would have it's data normalized, where a Decision Tree doesn't care,

We can use the `cv ="prefit"` to sort of work around this but being able to parse multiple datasets  could be such a great option

### Describe your proposed solution

Something like

```python
from sklearn.linear_model import LogsiticRegression
from sklearn.ensemble import StackingClassifier
from lightgbm import LGBMClassifier
from utils import get_data_lgbm,get_data_logreg
data_lgbm = get_data_lgbm()
data_logreg= get_data_logreg()

lgbm = LGBMClassifier()
logreg = LogisticRegression()
stacking_model = StackingClassifier(estimators = [lgbm,logreg], 
                                                   datasets = [data_lgbm,data_logreg] #Parsing the datasets here
                                                      )
stacking_model .fit()
```



### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_

COMMENT:
I don't think that we need this feature.

For the categorical features, `LogisiticRegression` will generally require a one-hot encoding of the features while `LightGBM` (or equivalently `HistGradientBoostingClassifier`) would require ordinally encoded features to internally deal with them. Therefore, it only boils down to creating a different pipeline for each model and passing it to the model.

Effectively, they will be trained on different datasets if we consider the input dataset for each model. The code would look something like this:

```python

hgbdt = make_pipeline(
    make_column_transformer(
        (OrdinalEncoder(), categorical_columns),
        remainder="passthrough",
    ),
    HistGradientBoostingClassifier(),
)
lr = make_pipeline(
    make_column_transformer(
        (OneHotEncoder(), categorical_columns),
        (StandardScaler(), numerical_columns),
    ),
    LogisticRegression(),
)

stacker = StackingClassifier(
    estimators=[("hgbdt", hgbdt), ("lr", lr)]
).fit(X, y)
```

Labels: New Feature, Needs Triage
StackingRegressor does not allow classifiers as estimators (and vice versa)

### Describe the workflow you want to enable

I'm facing an ordinal regression problem. In these kind of problems, both a classifier or a regressor can be used as a naive approach.

I'm trying to stack both of them, however sklearn does not allow me to do that. The error is raised from the next line:

https://github.com/scikit-learn/scikit-learn/blob/36958fb240fbe435673a9e3c52e769f01f36bec0/sklearn/ensemble/_base.py#L277

### Describe your proposed solution

I think that behaviour is a bit strict and limits some use cases like mine. A simple solution would be raising a warning instead of an error given that everything can work properly even if one of the estimators in a stack is not of the same kind of the stack itself.

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_

COMMENT:
I agree we could mix estimators in stacking (but not in voting). The implementation is rather straightforward, with a different `_validate_estimators` in `_BaseStacking` and `_BaseVoting`. PR welcome.

In the meantime, one could use:
```py
from sklearn.ensemble import StackingRegressor

class MyStackingRegressor(StackingRegressor):

    # overwrite the inherited method without the check
    def _validate_estimators(self):
        if self.estimators is None or len(self.estimators) == 0:
            raise ValueError(
                "Invalid 'estimators' attribute, 'estimators' should be a list"
                " of (string, estimator) tuples."
            )
        names, estimators = zip(*self.estimators)
        self._validate_names(names)

        has_estimator = any(est != "drop" for est in estimators)
        if not has_estimator:
            raise ValueError(
                "All estimators are dropped. At least one is required "
                "to be an estimator."
            )
        return names, estimators
```

Labels: New Feature
StackingRegressor does not allow classifiers as estimators (and vice versa)

### Describe the workflow you want to enable

I'm facing an ordinal regression problem. In these kind of problems, both a classifier or a regressor can be used as a naive approach.

I'm trying to stack both of them, however sklearn does not allow me to do that. The error is raised from the next line:

https://github.com/scikit-learn/scikit-learn/blob/36958fb240fbe435673a9e3c52e769f01f36bec0/sklearn/ensemble/_base.py#L277

### Describe your proposed solution

I think that behaviour is a bit strict and limits some use cases like mine. A simple solution would be raising a warning instead of an error given that everything can work properly even if one of the estimators in a stack is not of the same kind of the stack itself.

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_

COMMENT:
@TomDLT I sent a PR, the idea is implemented but I have some doubts that I commented there

Labels: New Feature
"StackingClassifier" throws ValueError when cv = "prefit"

### Describe the bug

When I use the `StackingClassifier`, I get the error "ValueError: Expected cv as an integer, cross-validation object (from sklearn.model_selection) or an iterable. Got prefit."  when calling the `.fit` method

### Steps/Code to Reproduce

```python

from sklearn.svm import LinearSVC
from sklearn.ensemble import BaggingClassifier, StackingClassifier
from sklearn.linear_model import LogisticRegression

from sklearn.datasets import make_classification

X, y = make_classification(n_samples=100_000)

logreg_base = LogisticRegression()
logreg_bagging = BaggingClassifier(
    logreg_base, n_estimators=20, max_samples=0.05, n_jobs=14)
logreg_bagging.fit(X, y)

svm_base = LinearSVC()
svm_bagging = BaggingClassifier(
    svm_base, n_estimators=20, max_samples=0.05, n_jobs=14)
svm_bagging.fit(X, y)


meta_model = StackingClassifier(estimators=[("logreg", logreg_bagging),
                                            ("svm", svm_bagging)],
                                            cv = "prefit")
meta_model.fit(X,y) # Expected cv as an integer, cross-validation object (from sklearn.model_selection) or an iterable. 
                                #Got prefit.

```

### Expected Results

Expected to fit the model. According to the [docs](https://scikit-learn.org/stable/modules/generated/sklearn.ensemble.StackingClassifier.html) the `cv="prefit"` is a valid argument

### Actual Results

It throws the error "ValueError: Expected cv as an integer, cross-validation object (from sklearn.model_selection) or an iterable. Got prefit."

### Versions

```shell
System:
    python: 3.10.2 (tags/v3.10.2:a58ebcc, Jan 17 2022, 14:12:15) [MSC v.1929 64 bit (AMD64)]
executable: c:\Users\user\AppData\Local\Programs\Python\Python310\python.exe
   machine: Windows-10-10.0.22000-SP0

Python dependencies:
          pip: 22.2.2
   setuptools: 59.8.0
      sklearn: 1.0.2
        numpy: 1.23.2
        scipy: 1.8.0
       Cython: 0.29.28
       pandas: 1.4.0
   matplotlib: 3.5.1
       joblib: 1.1.0
threadpoolctl: 3.1.0

Built with OpenMP: True
```

COMMENT:
Thank you for opening the issue. As noted in the docs:

> New in version 1.1: The ‘prefit’ option was added in 1.1

I see you are using 1.0.2. Please update to scikit-learn 1.1 to get the new feature.

Labels: Bug, Needs Triage
Python 3.11 wheels

### Describe the workflow you want to enable
Installing scikit-learn for Python 3.11

### Describe your proposed solution
Add Python 3.11 `python: 311` to cibuildwheel CI job matrix

### Additional context
Likely you'll want to wait for scipy/scipy#16851 to land

COMMENT:
@cmarmo: would you be interested to work on this? :slightly_smiling_face:

Labels: New Feature, Build / CI, Packaging
Python 3.11 wheels

### Describe the workflow you want to enable
Installing scikit-learn for Python 3.11

### Describe your proposed solution
Add Python 3.11 `python: 311` to cibuildwheel CI job matrix

### Additional context
Likely you'll want to wait for scipy/scipy#16851 to land

COMMENT:
Cross-referencing a related discussion on scientific-python.org for inter-project coordination:

- https://discuss.scientific-python.org/t/spec-4-nightly-wheels/474/3

Labels: New Feature, Build / CI, Packaging
Python 3.11 wheels

### Describe the workflow you want to enable
Installing scikit-learn for Python 3.11

### Describe your proposed solution
Add Python 3.11 `python: 311` to cibuildwheel CI job matrix

### Additional context
Likely you'll want to wait for scipy/scipy#16851 to land

COMMENT:
> @cmarmo: would you be interested to work on this? slightly_smiling_face

@jjerphan: yes, if I am allowed to experiment a bit, but please don't expect a quick PR, it seems to me that some details should still be fixed in a number of dependencies.

Labels: New Feature, Build / CI, Packaging