RFC SLEP006: verbose vs non-verbose declaration in meta-estimator

As the proposal and the implementation of meta-estimator routing (SLEP006) stands, if the user wants to use `sample_weight`, they need to be quite verbose in how they declare the estimators. Taking `AdaBoostClassifier` as an example, and imagining if `AdaBoostClassifier` would use the sub-estimator's `score` method, the user would have to write:

``` python
est = (
    AdaBoostClassifier(LogisticRegression().set_fit_request(sample_weight=True)
    .set_score_request(sample_weight=True))
    .fit(X, y, sample_weight=sw)
)
```

which is quite more verbose than the current code users need to write:

``` python
est = AdaBoostClassifier(LogisticRegression()).fit(X, y, sample_weight=sw)
```

There have been concerns about making users write quite verbose code in cases where the current pattern seems quite reasonable.

Without changing everything related to SLEP006, there are three paths we can take:

#### Option 1: Helper function
We can introduce helper functions to make the above code simpler. For instance, a `weighted` function could request `sample_weight` on all methods which accept `sample_weight` for a given estimator. Then the above code would look like:

``` python
est = AdaBoostClassifier(weighted(LogisticRegression())).fit(X, y, sample_weight=sw)
```

and if the sub-estimator is a pipeline:

``` python
est = AdaBoostClassifier(
    make_pipeline(weighted(StandardScaler()), weighted(LogisticRegression())))
).fit(X, y, sample_weight=sw)
```

Implementing `weighted` for a `Pipeline` (or other meta-estimators) would be tricky since `set_fit_request` is only available for consumers and not non-consumer routers; therefore the user needs to repeat the `weighted` call for all sub-estimators.

#### Option 2: Different meta-estimators
Have two classes of meta-estimators (or routers to be specific).

In this scenario, we divide meta-estimators into two classes, _simple_ and _complex_. Simple routers are the ones which simply forward `**kwargs` to sub-estimators, and by default the assume sub-estimators have requested those metadata. This simplifies the users' code and makes the existing code for simple meta-estimators to keep working, but it raises a few issues.

First is that there will be two classes of meta-estimators, and the user would need to know which estimator is of which class. It's also not clear what we should do if the user explicitly sets request values for metadata (we can probably respect those if present).

Another issue is that if a meta-estimator changes behavior, it needs to become a _complex_ meta-estimator if we want to keep backward compatibility for it. This doesn't seem like a good pattern.

#### Option 3: Keep as is
Do nothing, things are as is.

I'm in favor or option 1 because:
- with the helper function the user code doesn't look too verbose
- using metadata is not a beginner kinda thing and therefore this API is not hampering beginners' experience with the library
- it keeps consistency among meta-estimators/consumers

xref: https://github.com/scikit-learn/scikit-learn/pull/22986#pullrequestreview-1005144559

COMMENT:
> 1. What if `LogisticRegression` doesn't support `sample_weight` in version `x`, and starts supporting it in version `x+1`? Then the same code, would have to different behaviors silently. That's now what we do in scikit-learn.

I'm not sure. In version `x` you could not use it in a pipeline or other setup where you pass `sample_weight` to `fit`. Because I think having a mix of "some of my steps in a pipeline support weights and others don't" is a bug. Here "support" means "can do something when weights are passed", which might be "ignore them" where that is appropriate For example `MinMaxScaler`'s behaviour is probably the same whether there are weights or not, so it "supports" using them, but ignores them.

This means in version `x` you can only have a setup that does not support passing in sample weights. In version `x+1` it becomes possible to pass in sample weights (assuming `LogisticRegression` was the only thing not supporting them). However this is not a silent change as the user will have to change their code to pass in weights.

> 2. How is `AdaBoost` supposed to know `LogisticRegression` supports `sample_weight`? Is it because its `fit` accepts the argument? What if instead of `LogisticRegression`, I pass something which is not a simple consumer?

`AdaBoost` would find out by interrogating the thing passed to it. Maybe by inspecting the signature of `fit` or because the thing passed in has an annotation that lets you find out or a method to call. I'm not sure I have an opinion on how exactly to implement it. The thing I care about is that there is a default, so that as a user I don't have to set it explicitly.

What I'm imagining is something like the following inside say `fit` of `Pipeline`:
```
@property
def supports_weights(self):
  return all(step.supports_weights for step in self.steps)

@property
def supports_no_weights(self):
  return all(step.supports_no_weights for step in self.steps)

def fit(..., sample_weights=None):
  if sample_weights is not None and not all(step.supports_weights for step in self.steps):
    raise RuntimeError("You passed weights but not all steps support them.")
  # For completeness. Though I'm not sure I can imagine a case where this would happen
  if sample_weights is None and not all(step.supports_no_weights for step in self.steps):
    raise RuntimeError("You did not pass weights but at least one step requires them.")
  ...
  # if weights are passed in we pass them to every step
  if sample_weights is not None:
    for step in self.steps:
      step.fit(..., sample_weights=sample_weights)
  ...
```
Each step of the pipeline either knows what the value of `self.supports_{no_}weights` is or interrogates its components to compute it.

The logic for `supports_weights` should consider "everything" that happens inside the particular estimator, so there might be a few where it has to be handcrafted for that estimator.

This is also a backwards incompatible change and would require third party developers to make changes. I think we can introduce it without a breaking change though by warning for two releases that your third party estimator has no `supports_weights` and will stop working soon. Similarly with users that use `step_name__sample_weights=blah`.

For some estimators, metrics, etc both `supports_weights` and `supports_no_weights` can be true. For example an accuracy metric can work with weights and without. Same with `LogisticRegressionCV`, unless the user instantiated it with a metric that does not support weights, in that case `supports_no_weights` would be false.

Finally, maybe the method for implementing the mechanism of `supports_weights` and over-riding the default values is the metadata routing that SLEP6 proposes. I don't know.

> 3.

I think this is covered by what I wrote above. The pipeline needs to inspect its steps and each step needs to come up with an answer.

> 4. `AdaBoost` basically follows two strategies based on whether the sub-estimator supports `sample_weight` or not, and the user needs to know that. Otherwise we're not encouraging them to do good machine learning.

This I disagree with. A lot of users could probably not explain the Ada Boost (or gradient boosting or probably even random forest) algorithm without making a mistake. That is Ok. I personally can't explain to you how to multiply two matrices together, I rely on Numpy knowing and having an efficient implementation. Delegating to experts is super efficient. That is why I think it is fine for users to not know how Ada Boost works or that it works differently when I pass weights/no weights. As long as scikit-learn gives me an error when I request something inconsistent (e.g. passing weights to `LogisticRegressionCV.fit` but using a metric that does not support weights). The vast majority of users are better off trusting that scikit-learn does the right thing and that they could find out what it is that it does via documentation if they ever have to.

Labels: API, RFC