RFC: Breaking Changes for Version 2

A while ago we talked about the possibility of a version 2 with breaking changes. Specifically, this came up in the context of SLEP006: metadata routing, but there are other things we have wanted to break which we can do all in a single release. In this issue we can talk about the possibility of such a release, and maybe a few things we'd like to include in it.

I'll try to summarize the challenges of keeping backward compatibility (BC) in the context of SLEP006:

- `MetaEstimator(Estimator()).fit(X, y, sample_weight)` will raise according to SLEP006 since `sample_weight` is not requested by any object. For backward compatibility, during the transition period, it'll only warn and assume `sample_weight` is requested by `Estimator()`.
- What should the behavior of `GridSearchCV(Estimator(), scorer=scorer).fit(X, y, sample_weight)` be during the transition period? If we keep backward compatibility, it should warn (whatever the warning be), route `sample_weight` to `Estimator`, but not to the `scorer`. And that's only the case because we're keeping BC. From the user's perspective, it's very weird.
- `Pipeline` ATM in certain methods like `fit`, `fit_transform`, `predict`, routes `*_params` to the `fit/etc` of the last estimator only. And `decision_function` and `transform`, `score_samples` have no routing at all. Keeping this BC, proves to be challenging, and a bit nasty. ref: #24270
- In quite a few meta-estimators, we check if the sub-estimator has `sample_weight` in the signature of `fit`, and we pass the given sample weight to it if that's the case, and not if that's not the case. With routing, we would have a much better idea of when to pass sample weight and when not, but if we're keeping BC, it's really challenging to see if we should forward sample weights or not. Both AdaBoost (#24026) and Bagging (#24250) have been very challenging, to the point that we might end up having to check if a sub-estimator is a meta-estimator itself or not, and check signature of sub-estimator's `fit`, and check the given routing info to keep BC.

I'm pretty sure there are other issues we haven't foreseen, and they'll come up as we work on this project. We thought it'd be easy once we get to this point, but BC has proven to make it very tricky and has stalled the progress.

All of the above is adding substantial complexity which is probably error prone, and buggy. We have had to spend quite a bit of time every time we go back to review those bits to understand why we've done what we've done. Even as a temporary solution they're not very maintainable.

My proposal here is to have a version 2 which is not BC with previous releases in certain ways, and target a few breaking changes for it. We can aim to have the release in October 2023 or April/May 2024 as the version 2.

cc @scikit-learn/core-devs

COMMENT:
> > .... `from sklearn.experimental import sleep6` ...
> 
> I really think y'all are underestimating the complexity this brings.

I'd put it the other way around: I might be overestimating the complexity of the change, and as a result thinking that doing something complicated like modifying code through an import statement is "a good trade-off". The reason I am thinking that this change is a complex one, both in terms of code and for users, is that I've read the SLEP, some of the discussions, peaked at the related PRs and left confused/thinking "not sure I can wrap my head around this".

> Also, I do understand the challenges for the users when it comes to breaking changes, but I don't really see a feasible way forward at this point which doesn't involve that, and this is keeping us behind. I like @lorentzenchr 's expression of "5 years too late".
> 
> On a personal level, I've been working on this topic for over 4 years now, and I have been under the impression that we care about this. and I think it'd be a pity to shelf this feature.

It is hard, especially because so much work and thought has gone into it. For me as a bystander it is amazing for how long this work was under discussion and then how long people have worked on it. It also means that as an outsider I look at it as a project with a huge cost and unclear benefits. But I think a large part of the unclearness is because I've not been involved a lot. There are also plenty of things that I've bet against in the past that turned out to be winners, so what do I know about the future.

The reason I wanted to explicitly bring up the idea of "stop working on this" is that I see the same warning signs I've learnt to look for when out and about in the mountains. They include things like "we've invested so much" (e.g. we spent a day travelling here, then paid for a hotel, permits, ...), "we are nearly there, so let's push" (we've done 95% of the ascent and there are only 50 meters left), "let's make an exception just this one time" (don't ski slopes over 30 degrees when the avalanche risk is Considerable, but today is different!), lots of people in the group going quiet ("I had a bad feeling, but no one else said anything, so I didn't say anything either"), etc. In a lot of cases where people end up in trouble, it is a factor like those that lead to it. The only way I know to help deal with this "get-there-itis" is that you explicitly talk about it. Even when you don't think you are in a situation where you are effected by it. To make it a normal thing that you always talk about. People don't get in trouble 100m from the start, people get in trouble 100m from the top.

So my goal is not to suggest to stop this, but to have it as an explicit option that is discussed and then decided upon.

Labels: RFC, Breaking Change