Talip7/scikit-learn-issues-multilabel huggingface dataset:
RFC drop support for python 3.8 ? bump min dependencies ?

The 1.3 release being scheduled for the first half of june, it's time to think about bumping the min dependencies.

Up to now we've been trying to roughly follow [NEP29](https://numpy.org/neps/nep-0029-deprecation_policy.html) regarding supported Python versions. According to this timeline we should drop support for python 3.8 in april 2023, hence for the 1.3 release. @scikit-learn/core-devs do we agree on this ?

Dropping support for python3.8 means bumping numpy min version to 1.19.3 (currently 1.17.3) according to [oldest-supported-numpy](https://github.com/scipy/oldest-supported-numpy/blob/main/setup.cfg)

Scipy min version has already been bumped in https://github.com/scikit-learn/scikit-learn/pull/24665.
Pytest has also already been bumped  in https://github.com/scikit-learn/scikit-learn/pull/26373. 
Joblib min version is just 1 release before the latest so we probably don't want to bump it yet.

Any objection or request ?

COMMENT:
To make my previous comment https://github.com/scikit-learn/scikit-learn/issues/26438#issuecomment-1563565309 more precise: I am fine dropping Python 3.8 but I do not think there are strong reasons to.

Labels: RFC
RFC drop support for python 3.8 ? bump min dependencies ?

The 1.3 release being scheduled for the first half of june, it's time to think about bumping the min dependencies.

Up to now we've been trying to roughly follow [NEP29](https://numpy.org/neps/nep-0029-deprecation_policy.html) regarding supported Python versions. According to this timeline we should drop support for python 3.8 in april 2023, hence for the 1.3 release. @scikit-learn/core-devs do we agree on this ?

Dropping support for python3.8 means bumping numpy min version to 1.19.3 (currently 1.17.3) according to [oldest-supported-numpy](https://github.com/scipy/oldest-supported-numpy/blob/main/setup.cfg)

Scipy min version has already been bumped in https://github.com/scikit-learn/scikit-learn/pull/24665.
Pytest has also already been bumped  in https://github.com/scikit-learn/scikit-learn/pull/26373. 
Joblib min version is just 1 release before the latest so we probably don't want to bump it yet.

Any objection or request ?

COMMENT:
+1 for dropping support of 3.8 once 3.12 is out, probably for scikit-learn 1.4.

Labels: RFC
RFC drop support for python 3.8 ? bump min dependencies ?

The 1.3 release being scheduled for the first half of june, it's time to think about bumping the min dependencies.

Up to now we've been trying to roughly follow [NEP29](https://numpy.org/neps/nep-0029-deprecation_policy.html) regarding supported Python versions. According to this timeline we should drop support for python 3.8 in april 2023, hence for the 1.3 release. @scikit-learn/core-devs do we agree on this ?

Dropping support for python3.8 means bumping numpy min version to 1.19.3 (currently 1.17.3) according to [oldest-supported-numpy](https://github.com/scipy/oldest-supported-numpy/blob/main/setup.cfg)

Scipy min version has already been bumped in https://github.com/scikit-learn/scikit-learn/pull/24665.
Pytest has also already been bumped  in https://github.com/scikit-learn/scikit-learn/pull/26373. 
Joblib min version is just 1 release before the latest so we probably don't want to bump it yet.

Any objection or request ?

COMMENT:
Another argument raised in an irl discussion for not keeping a too large range of supported versions is because it requires more CI resources. That being said, keeping 3.8 means that we support 4 python versions which is acceptable. Let's say we try to not go beyond that.

Since supporting py38 for one more release doesn't provide major benefits and doesn't make our life more difficult, let's keep it for now and drop it for 1.4.

Labels: RFC
RFC drop support for python 3.8 ? bump min dependencies ?

The 1.3 release being scheduled for the first half of june, it's time to think about bumping the min dependencies.

Up to now we've been trying to roughly follow [NEP29](https://numpy.org/neps/nep-0029-deprecation_policy.html) regarding supported Python versions. According to this timeline we should drop support for python 3.8 in april 2023, hence for the 1.3 release. @scikit-learn/core-devs do we agree on this ?

Dropping support for python3.8 means bumping numpy min version to 1.19.3 (currently 1.17.3) according to [oldest-supported-numpy](https://github.com/scipy/oldest-supported-numpy/blob/main/setup.cfg)

Scipy min version has already been bumped in https://github.com/scikit-learn/scikit-learn/pull/24665.
Pytest has also already been bumped  in https://github.com/scikit-learn/scikit-learn/pull/26373. 
Joblib min version is just 1 release before the latest so we probably don't want to bump it yet.

Any objection or request ?

COMMENT:
[NEP 29](https://numpy.org/neps/nep-0029-deprecation_policy.html) was superseded by the scientific python ecosystem coordination guideline [SPEC 0 — Minimum Supported Versions](https://scientific-python.org/specs/spec-0000/). Where NEP had a 42 month support windows (3.5 years) for each Python version, SPEC 0 recommends a 3 year support window.

If we were to follow SPEC 0, than Python 3.9 support can already be dropped, since it was released October 5th, 2020.

Labels: RFC
RFC drop support for python 3.8 ? bump min dependencies ?

The 1.3 release being scheduled for the first half of june, it's time to think about bumping the min dependencies.

Up to now we've been trying to roughly follow [NEP29](https://numpy.org/neps/nep-0029-deprecation_policy.html) regarding supported Python versions. According to this timeline we should drop support for python 3.8 in april 2023, hence for the 1.3 release. @scikit-learn/core-devs do we agree on this ?

Dropping support for python3.8 means bumping numpy min version to 1.19.3 (currently 1.17.3) according to [oldest-supported-numpy](https://github.com/scipy/oldest-supported-numpy/blob/main/setup.cfg)

Scipy min version has already been bumped in https://github.com/scikit-learn/scikit-learn/pull/24665.
Pytest has also already been bumped  in https://github.com/scikit-learn/scikit-learn/pull/26373. 
Joblib min version is just 1 release before the latest so we probably don't want to bump it yet.

Any objection or request ?

COMMENT:
I would consider to be pragmatic: if at the time our own release, the latest version of our release (NumPy, SciPy, pandas?) are still supporting Python 3.9, then it is safer to not drop support until this is not becoming a maintenance burden for some reason.

Labels: RFC
Conformal inference

### Describe the workflow you want to enable

Conformal prediction (CP) is a statistical technique for producing prediction sets without assumptions on the predictive algorithm (often a machine learning system) and only assuming exchangeability of the data. Given this method is assumption lean and thereby very defendable for causal inference, it would be nice if there was out-of-the-box support. 

### Describe your proposed solution

Some example code can be found [here](https://github.com/aangelopoulos/conformal-prediction). There would need to be at least 2 implementations:
1. Categorical: https://github.com/aangelopoulos/conformal-prediction/blob/main/notebooks/imagenet-aps-randomized.ipynb
2. Numeric: https://github.com/aangelopoulos/conformal-prediction/blob/main/notebooks/meps-cqr.ipynb
[Seminal paper that proves coverage (2017)](https://www.stat.cmu.edu/~ryantibs/papers/conformal.pdf)

### Describe alternatives you've considered, if relevant

The current options for using conformal prediction with an sklearn model are:
1. Write your own code
2. Import from [MAPIE](https://mapie.readthedocs.io/en/latest/index.html) or another library

### Additional context

I'm happy to implement these changes, but before filing a PR I wanted to get feedback if this aligns with sklearn's structure.

COMMENT:
I have the feeling that the research on CP is still very active and that's it's too early to consider for inclusion of such methods in scikit-learn.

The fact that MAPIE exists, is well maintained and has very good compatibility with scikit-learn models is a good solution in the mean time.

We might discuss again this point in a few years to see if CP research has settled down on strong baselines that offer a very robust speed / resolution / calibration tradeoff that stays stable over years.

CQR might be one such method for regression, but let's wait a bit to see if a cheaper alternative gets published in the mean time.

Labels: New Feature, Needs Decision - Include Feature
Conformal inference

### Describe the workflow you want to enable

Conformal prediction (CP) is a statistical technique for producing prediction sets without assumptions on the predictive algorithm (often a machine learning system) and only assuming exchangeability of the data. Given this method is assumption lean and thereby very defendable for causal inference, it would be nice if there was out-of-the-box support. 

### Describe your proposed solution

Some example code can be found [here](https://github.com/aangelopoulos/conformal-prediction). There would need to be at least 2 implementations:
1. Categorical: https://github.com/aangelopoulos/conformal-prediction/blob/main/notebooks/imagenet-aps-randomized.ipynb
2. Numeric: https://github.com/aangelopoulos/conformal-prediction/blob/main/notebooks/meps-cqr.ipynb
[Seminal paper that proves coverage (2017)](https://www.stat.cmu.edu/~ryantibs/papers/conformal.pdf)

### Describe alternatives you've considered, if relevant

The current options for using conformal prediction with an sklearn model are:
1. Write your own code
2. Import from [MAPIE](https://mapie.readthedocs.io/en/latest/index.html) or another library

### Additional context

I'm happy to implement these changes, but before filing a PR I wanted to get feedback if this aligns with sklearn's structure.

COMMENT:
> CQR might be one such method for regression, but let's wait a bit to see if a cheaper alternative gets published in the mean time.

In particular, I wonder if there exist cases where CQR can significantly beat a strong pair of quantile regression pipelines (e.g. linear quantile regressor on spline / features + a nystroem kernel, or quantile random forests with honesty sampling or histogram gradient boosting quantile regressors) when all hyper-parameters of the pair of regressors have been tuned with cross-validation on the proper scoring rule for quantile regression (the pinball loss in this case).

Labels: New Feature, Needs Decision - Include Feature
Conformal inference

### Describe the workflow you want to enable

Conformal prediction (CP) is a statistical technique for producing prediction sets without assumptions on the predictive algorithm (often a machine learning system) and only assuming exchangeability of the data. Given this method is assumption lean and thereby very defendable for causal inference, it would be nice if there was out-of-the-box support. 

### Describe your proposed solution

Some example code can be found [here](https://github.com/aangelopoulos/conformal-prediction). There would need to be at least 2 implementations:
1. Categorical: https://github.com/aangelopoulos/conformal-prediction/blob/main/notebooks/imagenet-aps-randomized.ipynb
2. Numeric: https://github.com/aangelopoulos/conformal-prediction/blob/main/notebooks/meps-cqr.ipynb
[Seminal paper that proves coverage (2017)](https://www.stat.cmu.edu/~ryantibs/papers/conformal.pdf)

### Describe alternatives you've considered, if relevant

The current options for using conformal prediction with an sklearn model are:
1. Write your own code
2. Import from [MAPIE](https://mapie.readthedocs.io/en/latest/index.html) or another library

### Additional context

I'm happy to implement these changes, but before filing a PR I wanted to get feedback if this aligns with sklearn's structure.

COMMENT:
Got it. Your points about MAPIE and in-progress research makes sense. 

From your perspective, is there any way to build in more support for CP into scikit-learn, maybe through examples/docs? Given CP is the most widely-accepted assumption-lean approach, making more users aware could improve causal inference as a whole. Just a thought 🙂

Labels: New Feature, Needs Decision - Include Feature
⚠️ CI failed on linux_arm64_wheel ⚠️

**CI failed on [linux_arm64_wheel](https://cirrus-ci.com/build/4941425045929984)** (May 25, 2023)

COMMENT:
## CI is no longer failing! ✅

[Successful run](https://cirrus-ci.com/build/5048220582150144) on May 26, 2023

Labels: Needs Triage
`HistGradientBoosting` should shuffle features when exploring for new splits

### Summary

Our implementation of `HistGradientBoosting` does not shuffle the feature at each node to find the best split. Note that our `GradientBoosting`, `RandomForest`, and `DecisionTree` use a Fisher-Yates permutation.

Not permuting features introduces a bias that can have some impact. For instance, it will impact the inspection and could lead to misinterpretation if one does not know about this implementation detail.

Below, I show an example where correlated features will have close importance in the case of a `GradientBoosting`. For `HistGradientBoosting`, a single feature (in an arbitrary manner) will be used and thus only a single feature is reported as important. 

### Reproducer

```python
# %%
import numpy as np

rng = np.random.default_rng(42)

n_samples, n_features = 100, 3
X = rng.normal(size=(n_samples, n_features))
X = np.concatenate([X, X], axis=1)  # create correlated features
y = X[:, 0] + X[:, 1] + rng.normal(scale=0.1, size=n_samples)

# %%
from sklearn.ensemble import GradientBoostingRegressor, HistGradientBoostingRegressor

gbdt = GradientBoostingRegressor(n_estimators=100, random_state=42).fit(X, y)
hgbdt = HistGradientBoostingRegressor(max_iter=100, random_state=42).fit(X, y)

# %%
import pandas as pd
from sklearn.inspection import permutation_importance

importances = {}
for name, model in zip(["gbdt", "hgbdt"], [gbdt, hgbdt]):
    result = permutation_importance(model, X, y, n_repeats=100, random_state=42, n_jobs=-1)
    importances[name] = pd.DataFrame(
        result.importances.T, columns=[f"Features #{i}" for i in range(X.shape[1])]
    )

# %%
import matplotlib.pyplot as plt

fig, axes = plt.subplots(ncols=2, figsize=(10, 4))
for ax, (name, importance) in zip(axes, importances.items()):
    importance.plot.box(ax=ax, vert=False, whis=100)
    ax.set_title(name)
    ax.set_xlabel("Decrease in R2 score")
fig.tight_layout()
```

![image](https://github.com/scikit-learn/scikit-learn/assets/7454015/acd0956f-fc87-4df9-bc3a-448becb108d3)

### Resolution

We can use a shuffling in `_find_node_split` to alleviate this bias and have a similar implementation as in the current tree implementation.

COMMENT:
As discussed IRL with @glemaitre, there might be a similar issue with `Lasso(selection="cyclic")` vs `Lasso(selection="random")` and `selection="cyclic"` is the default.

Labels: Enhancement, module:ensemble
`HistGradientBoosting` should shuffle features when exploring for new splits

### Summary

Our implementation of `HistGradientBoosting` does not shuffle the feature at each node to find the best split. Note that our `GradientBoosting`, `RandomForest`, and `DecisionTree` use a Fisher-Yates permutation.

Not permuting features introduces a bias that can have some impact. For instance, it will impact the inspection and could lead to misinterpretation if one does not know about this implementation detail.

Below, I show an example where correlated features will have close importance in the case of a `GradientBoosting`. For `HistGradientBoosting`, a single feature (in an arbitrary manner) will be used and thus only a single feature is reported as important. 

### Reproducer

```python
# %%
import numpy as np

rng = np.random.default_rng(42)

n_samples, n_features = 100, 3
X = rng.normal(size=(n_samples, n_features))
X = np.concatenate([X, X], axis=1)  # create correlated features
y = X[:, 0] + X[:, 1] + rng.normal(scale=0.1, size=n_samples)

# %%
from sklearn.ensemble import GradientBoostingRegressor, HistGradientBoostingRegressor

gbdt = GradientBoostingRegressor(n_estimators=100, random_state=42).fit(X, y)
hgbdt = HistGradientBoostingRegressor(max_iter=100, random_state=42).fit(X, y)

# %%
import pandas as pd
from sklearn.inspection import permutation_importance

importances = {}
for name, model in zip(["gbdt", "hgbdt"], [gbdt, hgbdt]):
    result = permutation_importance(model, X, y, n_repeats=100, random_state=42, n_jobs=-1)
    importances[name] = pd.DataFrame(
        result.importances.T, columns=[f"Features #{i}" for i in range(X.shape[1])]
    )

# %%
import matplotlib.pyplot as plt

fig, axes = plt.subplots(ncols=2, figsize=(10, 4))
for ax, (name, importance) in zip(axes, importances.items()):
    importance.plot.box(ax=ax, vert=False, whis=100)
    ax.set_title(name)
    ax.set_xlabel("Decrease in R2 score")
fig.tight_layout()
```

![image](https://github.com/scikit-learn/scikit-learn/assets/7454015/acd0956f-fc87-4df9-bc3a-448becb108d3)

### Resolution

We can use a shuffling in `_find_node_split` to alleviate this bias and have a similar implementation as in the current tree implementation.

COMMENT:
I‘m not 100% convinced that this needs to be fixed. The model does nothing wrong and even the inspection shows nothing wrong.
A predictive model is not primarily meant for (statistical) inference. So drawing the conclusion that some feature has no impact/correlation to the target based on feature importance of an ML model is just wrong.

Labels: Enhancement, module:ensemble
`HistGradientBoosting` should shuffle features when exploring for new splits

### Summary

Our implementation of `HistGradientBoosting` does not shuffle the feature at each node to find the best split. Note that our `GradientBoosting`, `RandomForest`, and `DecisionTree` use a Fisher-Yates permutation.

Not permuting features introduces a bias that can have some impact. For instance, it will impact the inspection and could lead to misinterpretation if one does not know about this implementation detail.

Below, I show an example where correlated features will have close importance in the case of a `GradientBoosting`. For `HistGradientBoosting`, a single feature (in an arbitrary manner) will be used and thus only a single feature is reported as important. 

### Reproducer

```python
# %%
import numpy as np

rng = np.random.default_rng(42)

n_samples, n_features = 100, 3
X = rng.normal(size=(n_samples, n_features))
X = np.concatenate([X, X], axis=1)  # create correlated features
y = X[:, 0] + X[:, 1] + rng.normal(scale=0.1, size=n_samples)

# %%
from sklearn.ensemble import GradientBoostingRegressor, HistGradientBoostingRegressor

gbdt = GradientBoostingRegressor(n_estimators=100, random_state=42).fit(X, y)
hgbdt = HistGradientBoostingRegressor(max_iter=100, random_state=42).fit(X, y)

# %%
import pandas as pd
from sklearn.inspection import permutation_importance

importances = {}
for name, model in zip(["gbdt", "hgbdt"], [gbdt, hgbdt]):
    result = permutation_importance(model, X, y, n_repeats=100, random_state=42, n_jobs=-1)
    importances[name] = pd.DataFrame(
        result.importances.T, columns=[f"Features #{i}" for i in range(X.shape[1])]
    )

# %%
import matplotlib.pyplot as plt

fig, axes = plt.subplots(ncols=2, figsize=(10, 4))
for ax, (name, importance) in zip(axes, importances.items()):
    importance.plot.box(ax=ax, vert=False, whis=100)
    ax.set_title(name)
    ax.set_xlabel("Decrease in R2 score")
fig.tight_layout()
```

![image](https://github.com/scikit-learn/scikit-learn/assets/7454015/acd0956f-fc87-4df9-bc3a-448becb108d3)

### Resolution

We can use a shuffling in `_find_node_split` to alleviate this bias and have a similar implementation as in the current tree implementation.

COMMENT:
Hello, 
If this issue is still open, can I work on it ?

Labels: Enhancement, module:ensemble
`HistGradientBoosting` should shuffle features when exploring for new splits

### Summary

Our implementation of `HistGradientBoosting` does not shuffle the feature at each node to find the best split. Note that our `GradientBoosting`, `RandomForest`, and `DecisionTree` use a Fisher-Yates permutation.

Not permuting features introduces a bias that can have some impact. For instance, it will impact the inspection and could lead to misinterpretation if one does not know about this implementation detail.

Below, I show an example where correlated features will have close importance in the case of a `GradientBoosting`. For `HistGradientBoosting`, a single feature (in an arbitrary manner) will be used and thus only a single feature is reported as important. 

### Reproducer

```python
# %%
import numpy as np

rng = np.random.default_rng(42)

n_samples, n_features = 100, 3
X = rng.normal(size=(n_samples, n_features))
X = np.concatenate([X, X], axis=1)  # create correlated features
y = X[:, 0] + X[:, 1] + rng.normal(scale=0.1, size=n_samples)

# %%
from sklearn.ensemble import GradientBoostingRegressor, HistGradientBoostingRegressor

gbdt = GradientBoostingRegressor(n_estimators=100, random_state=42).fit(X, y)
hgbdt = HistGradientBoostingRegressor(max_iter=100, random_state=42).fit(X, y)

# %%
import pandas as pd
from sklearn.inspection import permutation_importance

importances = {}
for name, model in zip(["gbdt", "hgbdt"], [gbdt, hgbdt]):
    result = permutation_importance(model, X, y, n_repeats=100, random_state=42, n_jobs=-1)
    importances[name] = pd.DataFrame(
        result.importances.T, columns=[f"Features #{i}" for i in range(X.shape[1])]
    )

# %%
import matplotlib.pyplot as plt

fig, axes = plt.subplots(ncols=2, figsize=(10, 4))
for ax, (name, importance) in zip(axes, importances.items()):
    importance.plot.box(ax=ax, vert=False, whis=100)
    ax.set_title(name)
    ax.set_xlabel("Decrease in R2 score")
fig.tight_layout()
```

![image](https://github.com/scikit-learn/scikit-learn/assets/7454015/acd0956f-fc87-4df9-bc3a-448becb108d3)

### Resolution

We can use a shuffling in `_find_node_split` to alleviate this bias and have a similar implementation as in the current tree implementation.

COMMENT:
I am going to close this one. As I said, neither strategy (shuffling or not) is better. It depends on the use case. If I want to detect features that I can drop, the HistGradientBoosting* estimators do a much better job.

On top, meanwhile we have feature subsampling for HGBT since #27139. By modifying the above code with the addition of `max_features=0.5`, I get:

<img width="1000" height="400" alt="Image" src="https://github.com/user-attachments/assets/a8092839-bfe6-4c02-aded-8ea4a0d4bc11" />

Labels: Enhancement, module:ensemble
⚠️ CI failed on linux_arm64_wheel ⚠️

**CI failed on [linux_arm64_wheel](https://cirrus-ci.com/build/6424992615759872)** (May 24, 2023)

COMMENT:
## CI is no longer failing! ✅

[Successful run](https://cirrus-ci.com/build/6424992615759872) on May 24, 2023

Labels: Needs Triage