Making the extension contract stable through version upgrades

### Describe the workflow you want to enable

Currently, every time `scikit-learn` releases a new minor version - e.g., 1.5.0, 1.6.0, 1.7.0 - compliant extensions, e.g., custom transformers, classifiers, etc, break - specifically, referring to the API conformance as tested through `check_estimator` or `parametrize_with_checks`.

These repeated breakages in the "extender contract" contrast the stability of the usage contract, which is stable and professionally managed.

For a package like `scikit-learn` which means to be a standard not just for ML algorithms but also an API standard that everyone uses, this is not a good state to be in - "do not break user code" is the maxim that gets broken for power users writing extensions.

Of course maintaining downwards compatibility is not always possible, but nothing should break without a proper warning.

### Describe your proposed solution

The main reason imo why this keeps happening is that `scikit-learn` is not using a proper pattern that ensures stability of the extension contract - and also no secondary deprecation patterns in relation to it.

A simple pattern that could improve a lot would be the "template pattern", in a specific form to separate likely changing parts such as the boilerplate (e.g., `validate_data` vs `_validate_data` and such) from the extension locus.
Reference: https://refactoring.guru/design-patterns/template-method

Examples of how this can be used to improve stability:

* `sktime`, for a different API, has a separation between `fit` calling an internal `_fit`, where change-prone boilerplate is sandwiched between a stable user contract (`fit`) and a stable extender contract (`_fit`); similarly `predict` and `_predict`
* `feature-engine` overrides the `BaseTransformer` `scikit-learn` extension contract with a similar pattern using `super()` calls in `fit` etc.

In particular the `fit`/`_fit` pairing that combines strategy and template pattern can be introduced easily via pure internal refactoring - I would go with that, and also add a layer of tests that checks the stability of the extension contract.

As a side effect, this would likely reduce the amount of copy-pasted boilerplate among the core estimators in `scikit-learn` proper as well.

### Describe alternatives you've considered, if relevant

The obvious alternative is "do nothing" - which will lead to packages with extensions producing their own version of mitigating patterns for the unstable status quo.

The problem with this is that the extending packages are mutually incompatible with their extender pattern. It would be better if there is a single stable extension pattern prescribed - or at least recommended - by `scikit-learn`.

### Additional context

_No response_

COMMENT:
> It feels like the likely trajectory is that this input is going to get ignored, even though there seems to be a serious problem.
> 
> Is there a way of engaging or escalating that the suggestion gets attention? Is there a procedure to follow? Or is it more an unstructured random process, by which maintainers may or may not ultimately reply?

Aggressive and passive aggressive comments like this are certainly not helpful, and we reserve our rights to block individuals if it repeats @fkiraly .

As for the developer API, historically, we haven't had any, and in the past year it's been one of our major milestones to improve on that front. Since we haven't had any developer API kinda thing, we'd change any private functionality w/o any warning or deprecation, which is expected.

However, it's been hard to develop scikit-learn compatible extensions w/o using private API, which is why we've worked on that front. That means there's a lot more in the public space, either simply as publicly exposed, or as a `__sklearn_....__` method (such as `__sklearn_clone__`).

As a result of these changes, developing scikit-learn extensions has become a lot more predictable and stable going forward, but it has broken some existing extensions, depending on how much of the private API they'd use. That's why we developed `sklearn-compat`.

https://github.com/orgs/scikit-learn/projects/13 keeps track of relevant work on this front, and as @jeremiedbb mentioned, there's slightly more to come. However, the idea is that if you vendor or depend on `sklearn-compat`, you can write easily backward compatible code.

Any library breaks things for advanced users, the same way we continuously have CI issues regarding our dependencies (`numpy`, `scipy`, etc), and the way to make sure users are not affected, is to make sure you have a CI testing against the nightly build, and report issues as soon as they come about, for us to fix it either inside sklearn or sklearn-compat.

So in a sense, I'm not sure if there's anything to do specifically in this issue, and I'm also in favor of a `_fit` / `fit` separation which we can implement in a non-breaking way for people developing future extensions. Happy to discuss that in a separate dedicated issue.

As for this one, I think we can close since otherwise I don't see any actionables here. But happy to reopen if something comes up here.

Labels: New Feature, Developer API