Talip7/scikit-learn-issues-multilabel huggingface dataset:
HalvingRandomSearchCV error

### Describe the bug

https://github.com/scikit-learn/scikit-learn/blob/main/sklearn/model_selection/_search_successive_halving.py#L220
when resources is the hyperparameter of estimator, the best performance is not always at its maximum, right?. so in my opinion, in this case, we should select the best in whole iter instead of the best in last iter.

### Steps/Code to Reproduce

```py
from sklearn.datasets import load_iris
from sklearn.ensemble import RandomForestClassifier
from sklearn.experimental import enable_halving_search_cv
from sklearn.model_selection import  HalvingRandomSearchCV, KFold

X, y = load_iris(return_X_y=True)
clf = RandomForestClassifier(n_estimators=10, random_state=0)  

kf = KFold(shuffle=False)
param_distributions = {"max_depth": [2, 3, 4], "min_samples_split": list(range(2, 12))}
search = HalvingRandomSearchCV(clf, param_distributions, min_resources=15, max_resources=150,factor=3,
                            random_state=0, n_jobs=1,cv=kf, **resource='n_estimators'**
                            ).fit(X, y)

print(search.best_estimator_)
```

### Expected Results

```
RandomForestClassifier(max_depth=3, min_samples_split=5, n_estimators=15,
                       random_state=0)
```

### Actual Results

```
RandomForestClassifier(max_depth=4, min_samples_split=4, **n_estimators=135**,random_state=0)
```

### Versions

```shell
System:
    python: 3.8.13 | packaged by conda-forge | (default, Mar 25 2022, 05:59:45) [MSC v.1929 64 bit (AMD64)]
executable: C:\software\anaconda\python.exe
   machine: Windows-10-10.0.19043-SP0

Python dependencies:
          pip: 22.0.4
   setuptools: 62.1.0
      sklearn: 1.0.2
        numpy: 1.21.5
        scipy: 1.8.0
       Cython: 0.29.28
       pandas: 1.4.2
   matplotlib: 3.5.1
       joblib: 1.1.0
threadpoolctl: 3.1.0
```

COMMENT:
Working on this

Labels: Documentation, module:model_selection
Proposal for creating a SwitchCase

### Discussed in https://github.com/scikit-learn/scikit-learn/discussions/24619

<div type='discussions-op-text'>

<sup>Originally posted by **JaimeArboleda** October 10, 2022</sup>
New operator in `sklearn.compose` for creating *branches* or *switch-cases* in the whole pipeline. 

I think an operator like this one would be handy for hyperparameter tuning:

```python
sc = SwitchCase(
    cases=[
        (
            "option_1",
            Pipeline([
                ("scale", StandardScaler()), 
                ("reduce_dims", PCA(n_components=5)),
                ("clf", LogisticRegression())
            ])
        ),
        (
            "option_2",
            Pipeline([
                ("scale", StandardScaler()), 
                ("reduce_dims", SelectKBest(mutual_info_classif, k=5)),
                ("clf", RandomForestClassifier())
            ])
        )
    ],
    switch="option_1")
```
In particular, it would make it easier for defining complex search spaces where you want to assess which option among a discrete set is best (while playing with inner hyperparameter as well). In this example, I could play with the *switch* while, at the same time, I could move the inner parameters using the double underscore convention. For instance, this kind of assignment will work: 

```python
sc.set_params(**{
    "switch": "option_2",
    "option_2__reduce_dims__k": 2
})
```

I have created the `SwitchCase` class for myself and checked that it works properly. If there is interest, I could try to do a PR. I just want to check it first because I don't want to spend a lot of time polishing the PR if in the end it won't be merged. </div>

COMMENT:
You can already do this with `GridSearchCV` and other search estimators:


``` py
from sklearn.datasets import make_classification
from sklearn.ensemble import RandomForestClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import GridSearchCV
from sklearn.pipeline import Pipeline

X, y = make_classification()

est = Pipeline([("clf", None)])

grid = [
    {"clf": [LogisticRegression()], "clf__C": [0.1, 1, 10]},
    {"clf": [RandomForestClassifier()], "clf__max_depth": [2, 5, 10]},
]
gs = GridSearchCV(est, param_grid=grid, scoring='roc_auc').fit(X, y)
print(gs.best_estimator_)
```

which outputs:

```
Pipeline(steps=[('clf', RandomForestClassifier(max_depth=10))])
```

For more info: https://scikit-learn.org/stable/modules/compose.html#usage

Closing, happy to re-open if this doesn't satisfy the usecase you're seeking to solve.

Labels: Needs Triage
Proposal for creating a SwitchCase

### Discussed in https://github.com/scikit-learn/scikit-learn/discussions/24619

<div type='discussions-op-text'>

<sup>Originally posted by **JaimeArboleda** October 10, 2022</sup>
New operator in `sklearn.compose` for creating *branches* or *switch-cases* in the whole pipeline. 

I think an operator like this one would be handy for hyperparameter tuning:

```python
sc = SwitchCase(
    cases=[
        (
            "option_1",
            Pipeline([
                ("scale", StandardScaler()), 
                ("reduce_dims", PCA(n_components=5)),
                ("clf", LogisticRegression())
            ])
        ),
        (
            "option_2",
            Pipeline([
                ("scale", StandardScaler()), 
                ("reduce_dims", SelectKBest(mutual_info_classif, k=5)),
                ("clf", RandomForestClassifier())
            ])
        )
    ],
    switch="option_1")
```
In particular, it would make it easier for defining complex search spaces where you want to assess which option among a discrete set is best (while playing with inner hyperparameter as well). In this example, I could play with the *switch* while, at the same time, I could move the inner parameters using the double underscore convention. For instance, this kind of assignment will work: 

```python
sc.set_params(**{
    "switch": "option_2",
    "option_2__reduce_dims__k": 2
})
```

I have created the `SwitchCase` class for myself and checked that it works properly. If there is interest, I could try to do a PR. I just want to check it first because I don't want to spend a lot of time polishing the PR if in the end it won't be merged. </div>

COMMENT:
Given the existing tools can handle most of these cases, and the maintenance burden of adding new tools, I think this would be more suitable if a third party package and not inside sklearn.

Labels: Needs Triage
Proposal for creating a SwitchCase

### Discussed in https://github.com/scikit-learn/scikit-learn/discussions/24619

<div type='discussions-op-text'>

<sup>Originally posted by **JaimeArboleda** October 10, 2022</sup>
New operator in `sklearn.compose` for creating *branches* or *switch-cases* in the whole pipeline. 

I think an operator like this one would be handy for hyperparameter tuning:

```python
sc = SwitchCase(
    cases=[
        (
            "option_1",
            Pipeline([
                ("scale", StandardScaler()), 
                ("reduce_dims", PCA(n_components=5)),
                ("clf", LogisticRegression())
            ])
        ),
        (
            "option_2",
            Pipeline([
                ("scale", StandardScaler()), 
                ("reduce_dims", SelectKBest(mutual_info_classif, k=5)),
                ("clf", RandomForestClassifier())
            ])
        )
    ],
    switch="option_1")
```
In particular, it would make it easier for defining complex search spaces where you want to assess which option among a discrete set is best (while playing with inner hyperparameter as well). In this example, I could play with the *switch* while, at the same time, I could move the inner parameters using the double underscore convention. For instance, this kind of assignment will work: 

```python
sc.set_params(**{
    "switch": "option_2",
    "option_2__reduce_dims__k": 2
})
```

I have created the `SwitchCase` class for myself and checked that it works properly. If there is interest, I could try to do a PR. I just want to check it first because I don't want to spend a lot of time polishing the PR if in the end it won't be merged. </div>

COMMENT:
Ok, I understand. Thank you for your time!

Labels: Needs Triage
GLM doesn't have an offset option

### Describe the workflow you want to enable

As described at #24155, GLM should also support the offset option.

### Describe your proposed solution

For example
```python
from sklearn.linear_model import PoissonRegressor

glm = PoissonRegressor(
                        alpha=0,
                        fit_intercept=False, 
                        max_iter=300,
                        offset=A # offset option
                      )
glm.fit(X_train, y_train)
```

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_

COMMENT:
If I am not wrong I think that we can easily get this behaviour using the `TransformedTargetRegressor`:

```python
import numpy as np
from sklearn.datasets import make_regression
from sklearn.linear_model import Ridge
from sklearn.compose import TransformedTargetRegressor

X, y = make_regression(
    n_samples=100, n_features=1, random_state=0, noise=30
)
# force the value to be positive to easily decide of the offset
X += np.abs(X.min(axis=0))

offset = -150
model = TransformedTargetRegressor(
    regressor=Ridge(fit_intercept=False),
    func=lambda y: y - offset,
    inverse_func=lambda y: y + offset,
)
model.fit(X, y)
```

```python
import matplotlib.pyplot as plt

plt.scatter(X, y)
plt.plot(X, model.predict(X), "--k")
```

<img width="394" alt="image" src="https://user-images.githubusercontent.com/7454015/197731632-54beafcf-e701-41f5-9cbf-c45c6eddc6db.png">

We can therefore see that the intercept corresponds to the offset imposed when transforming the target.

Labels: New Feature, Needs Decision, module:linear_model, Needs Decision - Include Feature
GLM doesn't have an offset option

### Describe the workflow you want to enable

As described at #24155, GLM should also support the offset option.

### Describe your proposed solution

For example
```python
from sklearn.linear_model import PoissonRegressor

glm = PoissonRegressor(
                        alpha=0,
                        fit_intercept=False, 
                        max_iter=300,
                        offset=A # offset option
                      )
glm.fit(X_train, y_train)
```

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_

COMMENT:
It could be worth potentially adding an example in the gallery.
@ogrisel @lorentzenchr @betatim do you think that this is worth an example?

Labels: New Feature, Needs Decision, module:linear_model, Needs Decision - Include Feature
GLM doesn't have an offset option

### Describe the workflow you want to enable

As described at #24155, GLM should also support the offset option.

### Describe your proposed solution

For example
```python
from sklearn.linear_model import PoissonRegressor

glm = PoissonRegressor(
                        alpha=0,
                        fit_intercept=False, 
                        max_iter=300,
                        offset=A # offset option
                      )
glm.fit(X_train, y_train)
```

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_

COMMENT:
@kanxu526 @rehanganapathy please feel invited to join the discussion here.

Labels: New Feature, Needs Decision, module:linear_model, Needs Decision - Include Feature
GLM doesn't have an offset option

### Describe the workflow you want to enable

As described at #24155, GLM should also support the offset option.

### Describe your proposed solution

For example
```python
from sklearn.linear_model import PoissonRegressor

glm = PoissonRegressor(
                        alpha=0,
                        fit_intercept=False, 
                        max_iter=300,
                        offset=A # offset option
                      )
glm.fit(X_train, y_train)
```

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_

COMMENT:
I was going to comment on #24155 but I will put the message here. For `LogisticRegression`, it might be a bit more complex. `LogisticRegression` will be seen as a classifier in scikit-learn. I expect the `TransformedTargetRegressor` to error while passing this model.

Edit: Actually, `TransformedTargetRegressor` does not error, but we internally binarize the target to 0/1 and therefore cancel the manual intercept manually applied with the transformer.

Labels: New Feature, Needs Decision, module:linear_model, Needs Decision - Include Feature
GLM doesn't have an offset option

### Describe the workflow you want to enable

As described at #24155, GLM should also support the offset option.

### Describe your proposed solution

For example
```python
from sklearn.linear_model import PoissonRegressor

glm = PoissonRegressor(
                        alpha=0,
                        fit_intercept=False, 
                        max_iter=300,
                        offset=A # offset option
                      )
glm.fit(X_train, y_train)
```

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_

COMMENT:
> @ogrisel @lorentzenchr @betatim do you think that this is worth an example?

Yes, because it's not obvious to me what are the use cases for such an option, besides manually constraining the intercept value when used in conjunction with `fit_intercept=False`.

Labels: New Feature, Needs Decision, module:linear_model, Needs Decision - Include Feature
GLM doesn't have an offset option

### Describe the workflow you want to enable

As described at #24155, GLM should also support the offset option.

### Describe your proposed solution

For example
```python
from sklearn.linear_model import PoissonRegressor

glm = PoissonRegressor(
                        alpha=0,
                        fit_intercept=False, 
                        max_iter=300,
                        offset=A # offset option
                      )
glm.fit(X_train, y_train)
```

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_

COMMENT:
@glemaitre Nice idea with the TargetTransformer. For `PoissonRegressor`, it would more likely read
```python
model = TransformedTargetRegressor(
    regressor=Ridge(fit_intercept=False),
    func=lambda y: y / offset,
    inverse_func=lambda y: y * offset,
)
```
The problem is with `LogisticRegression` is that an offset can't be singled out: `prediction = expit(X@coef + offset)` and expit/sigmoid does not allow to separate the terms.

Labels: New Feature, Needs Decision, module:linear_model, Needs Decision - Include Feature
GLM doesn't have an offset option

### Describe the workflow you want to enable

As described at #24155, GLM should also support the offset option.

### Describe your proposed solution

For example
```python
from sklearn.linear_model import PoissonRegressor

glm = PoissonRegressor(
                        alpha=0,
                        fit_intercept=False, 
                        max_iter=300,
                        offset=A # offset option
                      )
glm.fit(X_train, y_train)
```

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_

COMMENT:
> it's not obvious to me what are the use cases for such an option

I know about 2 use cases:

1. It's a way to deal with "exposure". But this can also (or better) be accomplished with sample weights (model y=count/w with sample weight w, same is true for logistic regression).
2. One may have a priori knowledge of the effect of a certain feature or certain features, or one wants to fix it (for other reasons).

> besides manually constraining the intercept

It's usually not the intercept, but a certain (amount of) coefficient(s) that one wants to fix.

Labels: New Feature, Needs Decision, module:linear_model, Needs Decision - Include Feature
GLM doesn't have an offset option

### Describe the workflow you want to enable

As described at #24155, GLM should also support the offset option.

### Describe your proposed solution

For example
```python
from sklearn.linear_model import PoissonRegressor

glm = PoissonRegressor(
                        alpha=0,
                        fit_intercept=False, 
                        max_iter=300,
                        offset=A # offset option
                      )
glm.fit(X_train, y_train)
```

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_

COMMENT:
> One may have a priori knowledge of the effect of a certain feature or certain features, or one wants to fix it (for other reasons).


> It's usually not the intercept, but a certain (amount of) coefficient(s) that one wants to fix.

Hum, this does not match the description of the issue. One would need to be able to specify a mapping from feature names (or positional indices) to fixed coefficients in the constructor, right? Calling this "offset" would be confusing. Calling it `frozen_coefs` might be more meaningful (and the fixed offset/intercept would be a special case).

Labels: New Feature, Needs Decision, module:linear_model, Needs Decision - Include Feature
GLM doesn't have an offset option

### Describe the workflow you want to enable

As described at #24155, GLM should also support the offset option.

### Describe your proposed solution

For example
```python
from sklearn.linear_model import PoissonRegressor

glm = PoissonRegressor(
                        alpha=0,
                        fit_intercept=False, 
                        max_iter=300,
                        offset=A # offset option
                      )
glm.fit(X_train, y_train)
```

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_

COMMENT:
In the GLM literatur, offsets have a clear meaning. This is reflected by implementations of GLMs in [R](https://stat.ethz.ch/R-manual/R-devel/library/stats/html/glm.html) where  `offset` can be part of the model formula:
```R
## an example with offsets from Venables & Ripley (2002, p.189)
utils::data(anorexia, package = "MASS")

anorex.1 <- glm(Postwt ~ Prewt + Treat + offset(Prewt),
                family = gaussian, data = anorexia)
summary(anorex.1)
```
[statsmodels](https://www.statsmodels.org/stable/generated/statsmodels.genmod.generalized_linear_model.GLM.html#statsmodels.genmod.generalized_linear_model.GLM), [Julia](https://juliastats.org/GLM.jl/stable/api/#GLM.glm).

My main concern is that it does not fit in our API.

Labels: New Feature, Needs Decision, module:linear_model, Needs Decision - Include Feature
GLM doesn't have an offset option

### Describe the workflow you want to enable

As described at #24155, GLM should also support the offset option.

### Describe your proposed solution

For example
```python
from sklearn.linear_model import PoissonRegressor

glm = PoissonRegressor(
                        alpha=0,
                        fit_intercept=False, 
                        max_iter=300,
                        offset=A # offset option
                      )
glm.fit(X_train, y_train)
```

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_

COMMENT:
Hi all, are there any news on this feature? Do you think that there are any possibilities that it will be implemented in a near future? 😊

Labels: New Feature, Needs Decision, module:linear_model, Needs Decision - Include Feature
GLM doesn't have an offset option

### Describe the workflow you want to enable

As described at #24155, GLM should also support the offset option.

### Describe your proposed solution

For example
```python
from sklearn.linear_model import PoissonRegressor

glm = PoissonRegressor(
                        alpha=0,
                        fit_intercept=False, 
                        max_iter=300,
                        offset=A # offset option
                      )
glm.fit(X_train, y_train)
```

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_

COMMENT:
Hi, with @manudarmi and our team we are working on adding offsets to the GLMs.

We have a working implementation which passes the offset as a kwarg of the `fit` method.

It would look like this:

```python
from sklearn.linear_model import GammaRegressor

reg = GammaRegressor(...)
reg.fit(X, y, sample_weight=sample_weight, offset=offset)
```

With `offset` being an array-like with the same shape as `sample_weight`.

The main problem is that whenever offsets are not known a priori (e.g. computed during a previous step in a pipeline), it would be impossible to pass them to the GLM without creating a custom wrapper.

For example:

```python
from sklearn.linear_model import GammaRegressor
from sklearn.pipeline import Pipeline

pipeline = Pipeline([
    ("create_the_offsets", CustomTransformation(...)),
    ("regressor": GammaRegressor(...)),
])
# no way to pass the offset from `CustomTransformation` output into the GLM
```

A solution could be to track whether the GLM expects offsets or not as a constructor argument, and then expect them to be either in the first or last column of X. This would make the snippet above work properly, at the cost of being less explicit.

Would look like this:

```python
import numpy
from sklearn.linear_model import GammaRegressor

X = numpy.concatenate([X, offset], axis=1) # offset in the last column
reg = GammaRegressor(use_offset=True, **other_params)
reg.fit(X, y, sample_weight=sample_weight)
```

If you could give me some feedback I could start working towards a PR in either direction 😄

Labels: New Feature, Needs Decision, module:linear_model, Needs Decision - Include Feature
GLM doesn't have an offset option

### Describe the workflow you want to enable

As described at #24155, GLM should also support the offset option.

### Describe your proposed solution

For example
```python
from sklearn.linear_model import PoissonRegressor

glm = PoissonRegressor(
                        alpha=0,
                        fit_intercept=False, 
                        max_iter=300,
                        offset=A # offset option
                      )
glm.fit(X_train, y_train)
```

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_

COMMENT:
API design is often hard. I don't think that passing offset as an array is a good API, neither in `fit` nor in `predict`. That array must come from somewhere, in particular at prediction time!

*edited*
Currently, I  have ~~2~~ 3 ideas in mind:
1. Pass a model in init, `GLM(offset=model)`. Internally, `model.predict(X)` is called to get the offset in fit and in predict.
  Problem: Difficult for the GLM to ignore a feature that is set/fixed/used as offset.
2. Pass a dict with key=column, value=coefficint value in init, `GLM(offset={"col_age": 1.2})`. For those columns, the coefficients are then fixed, non will be fitted on top.
   Problem: Sometime one wants to fit the offset feature on top of the offset. So the values of the dict could be tuples (coefficient, True/False).
3. Pass offsets as array with shape n_samples to `fit` and `predict`.
  This gives the flexibility to not provide offsets in predict (there are use cases!) and leave it to the user to provide those arrays. With the metadata routing, it becomes possible to pass those along a pipeline (but not from within).

Once API is fixed, implementing it is much easier.

Labels: New Feature, Needs Decision, module:linear_model, Needs Decision - Include Feature