RFC: Breaking Changes for Version 2

A while ago we talked about the possibility of a version 2 with breaking changes. Specifically, this came up in the context of SLEP006: metadata routing, but there are other things we have wanted to break which we can do all in a single release. In this issue we can talk about the possibility of such a release, and maybe a few things we'd like to include in it.

I'll try to summarize the challenges of keeping backward compatibility (BC) in the context of SLEP006:

- `MetaEstimator(Estimator()).fit(X, y, sample_weight)` will raise according to SLEP006 since `sample_weight` is not requested by any object. For backward compatibility, during the transition period, it'll only warn and assume `sample_weight` is requested by `Estimator()`.
- What should the behavior of `GridSearchCV(Estimator(), scorer=scorer).fit(X, y, sample_weight)` be during the transition period? If we keep backward compatibility, it should warn (whatever the warning be), route `sample_weight` to `Estimator`, but not to the `scorer`. And that's only the case because we're keeping BC. From the user's perspective, it's very weird.
- `Pipeline` ATM in certain methods like `fit`, `fit_transform`, `predict`, routes `*_params` to the `fit/etc` of the last estimator only. And `decision_function` and `transform`, `score_samples` have no routing at all. Keeping this BC, proves to be challenging, and a bit nasty. ref: #24270
- In quite a few meta-estimators, we check if the sub-estimator has `sample_weight` in the signature of `fit`, and we pass the given sample weight to it if that's the case, and not if that's not the case. With routing, we would have a much better idea of when to pass sample weight and when not, but if we're keeping BC, it's really challenging to see if we should forward sample weights or not. Both AdaBoost (#24026) and Bagging (#24250) have been very challenging, to the point that we might end up having to check if a sub-estimator is a meta-estimator itself or not, and check signature of sub-estimator's `fit`, and check the given routing info to keep BC.

I'm pretty sure there are other issues we haven't foreseen, and they'll come up as we work on this project. We thought it'd be easy once we get to this point, but BC has proven to make it very tricky and has stalled the progress.

All of the above is adding substantial complexity which is probably error prone, and buggy. We have had to spend quite a bit of time every time we go back to review those bits to understand why we've done what we've done. Even as a temporary solution they're not very maintainable.

My proposal here is to have a version 2 which is not BC with previous releases in certain ways, and target a few breaking changes for it. We can aim to have the release in October 2023 or April/May 2024 as the version 2.

cc @scikit-learn/core-devs

COMMENT:
I can't make it to the dedicated meeting today (🛩️). After some thinking and discussing I think my main worry is about making the "I don't know either scikit-learn, please just make it happen, you are the expert after all" user story harder for little gain. To personify the project for a moment: scikit-learn is an expert in machine-learning, people do and should delegate decisions to it. Delegation is convenient and for the vast majority of people the outcomes are better (day trading vs passive index investing, wiring your own house vs a professional electrician doing it, using Python to do science vs C++).

For me this means that as a user I've expressed everything that there is to express with
```python
est = AdaBoostClassifier(LogisticRegression())
est.fit(X, y, sample_weight=sw)
```
Most likely the average user doesn't even know that there are options and what the trade-offs between them are. As an expert who does know, most of the time I am not using this in "expert mode" either, so it is inconvenient to have to make choices.

I also believe that if there are sample weights involved then everything in the "chain" (be that a `Pipeline` or an estimator that does internal CV or an estimator that uses a metric to do early stopping) has to support weights. Here "supports" means "does something sensible when weights are involved". This sensible thing could be "take them into account when re-sampling samples" (say for some kind of stratified sampler) or "ignore the weights" (say for something like `MinMaxScaler`).

Something to avoid are cases like https://github.com/scikit-learn/scikit-learn/issues/25906. To me this is a bug. If a user passes sample weights in the metric inside `LogisticRegressionCV` has to support that. Otherwise the user selected an invalid combination and should receive an exception.

This does not mean we should forbid/make it impossible for experts to re-jig things so that they can use `LogisticRegressionCV` with weights for the estimator but ignore them during metric calculation. However this is the case where extra effort from the user should be required.

I wrote more about this in https://github.com/scikit-learn/scikit-learn/issues/23928#issuecomment-1491443254

It feels like introducing "your chain of tools has to be consistent when it comes to using/not using sample weights" is something that can be done in a backwards incompatible with deprecation cycle way. For additional sample aligned properties like protected group membership I think we have less constraints from current user's code because AFAIK it is virtually impossible to do this now anyway. This means I am less worried about requiring explicit routing setup for these use-cases. It would still be nice if there was a way to make the "happy path" work without much extra code from the user.

Most users have no sample weights, a few less have sample weights, even fewer have additional sample aligned properties and nearly no users are experts-doing-expert things. The defaults and convenience of using scikit-learn should reflect that (least amount of extra code for no weights, most extra code and "know what you are doing" for the experts doing expert things).

Labels: RFC, Breaking Change