Proposal for creating a SwitchCase

### Discussed in https://github.com/scikit-learn/scikit-learn/discussions/24619

<div type='discussions-op-text'>

<sup>Originally posted by **JaimeArboleda** October 10, 2022</sup>
New operator in `sklearn.compose` for creating *branches* or *switch-cases* in the whole pipeline. 

I think an operator like this one would be handy for hyperparameter tuning:

```python
sc = SwitchCase(
    cases=[
        (
            "option_1",
            Pipeline([
                ("scale", StandardScaler()), 
                ("reduce_dims", PCA(n_components=5)),
                ("clf", LogisticRegression())
            ])
        ),
        (
            "option_2",
            Pipeline([
                ("scale", StandardScaler()), 
                ("reduce_dims", SelectKBest(mutual_info_classif, k=5)),
                ("clf", RandomForestClassifier())
            ])
        )
    ],
    switch="option_1")
```
In particular, it would make it easier for defining complex search spaces where you want to assess which option among a discrete set is best (while playing with inner hyperparameter as well). In this example, I could play with the *switch* while, at the same time, I could move the inner parameters using the double underscore convention. For instance, this kind of assignment will work: 

```python
sc.set_params(**{
    "switch": "option_2",
    "option_2__reduce_dims__k": 2
})
```

I have created the `SwitchCase` class for myself and checked that it works properly. If there is interest, I could try to do a PR. I just want to check it first because I don't want to spend a lot of time polishing the PR if in the end it won't be merged. </div>

COMMENT:
Hello, thanks for your answer. I believe you are right in the sense that the use cases I have in mind can be covered with the existing tools. However, I think the SwitchCase makes the code more modular, and the hyperparameter tuning can be done more easily. I will give an example that is not so simple, so that I hope the case will be clearer. 

Let's say one wish to test several methods for feature generation, preprocessing and models, and perform a hyperparameter search to optimize for every combination of them. Then, instead of defining different pipelines (which will end up in a combinatorial explosion if there are lots of options for every step), it will be possible to specify a single *meta-classifier* encoding every possibility, and then the search space could be defined also in one pass (this is quite handy if instead of grid or random search one wants to try bayesian search for example, because the underlying gaussian process needs to know about every parameter). 

The code for defining the model could be like that (it's an made up example, not very useful in practice probably)

```python
feature_generation = SwitchCase(
    cases=[
        (
            "gen_1",
            PolynomialFeatures()
        ),
        (
            "gen_2",
            FunctionTransformer()
        )
    ],
    switch="gen_1")


preprocessor = SwitchCase(
    cases=[
        (
            "prep_1",
            Pipeline([
                ("scale", StandardScaler()), 
                ("reduce_dims", PCA(n_components=5))
            ])
        ),
        (
            "prep_2",
            Pipeline([
                ("scale", StandardScaler()), 
                ("reduce_dims", SelectKBest(mutual_info_classif, k=5)),
            ])
        )
    ],
    switch="prep_1")

model = SwitchCase(
    cases=[
        (
            "model_1",
            LogisticRegression()
        ),
        (
            "model_2",
            RandomForestClassifier()
        )
    ],
    switch="model_1")

clf = Pipeline(
    [("generator", feature_generation), ("preprocessor", preprocessor), ("model", model)]
)
```

And then, given this example, a possible search space for hyperparameter tuning -using the `skopt` schema- could be:
 
```python
search_space =  [
    Categorical(["gen_1", "gen_2"], name="generator__switch"),
    Categorical(["prep_1", "prep_2"], name="preprocessor__switch"),
    Categorical(["model_1", "model_2"], name="model__switch"),
    Integer(3, 10, name="preprocessor__prep_1__reduce_dims__n_components"),
    Integer(3, 10, name="preprocessor__prep_2__reduce_dims__k"),
    Integer(1, 3, name="generator__gen_1__degree"),
    Categorical(["l1", "l2"], name="model__model_1__penalty"), 
    Integer(5, 15, name="model__model_2__max_depth"),
]
```

Again, I am not trying to present an example that makes a lot of sense from an ML point of view. What I try to say is that, in a situation like this, a SwitchCase can make the code and the search space much modular. Without it, instead of defining the options isolated, one would need to manually *chain* every posibility...

I hope it is more clear now. As I said before, if you or other core developer of sklearn think that this is an interesting addition, I would be happy to work on a PR.

Labels: Needs Triage