Making the extension contract stable through version upgrades

### Describe the workflow you want to enable

Currently, every time `scikit-learn` releases a new minor version - e.g., 1.5.0, 1.6.0, 1.7.0 - compliant extensions, e.g., custom transformers, classifiers, etc, break - specifically, referring to the API conformance as tested through `check_estimator` or `parametrize_with_checks`.

These repeated breakages in the "extender contract" contrast the stability of the usage contract, which is stable and professionally managed.

For a package like `scikit-learn` which means to be a standard not just for ML algorithms but also an API standard that everyone uses, this is not a good state to be in - "do not break user code" is the maxim that gets broken for power users writing extensions.

Of course maintaining downwards compatibility is not always possible, but nothing should break without a proper warning.

### Describe your proposed solution

The main reason imo why this keeps happening is that `scikit-learn` is not using a proper pattern that ensures stability of the extension contract - and also no secondary deprecation patterns in relation to it.

A simple pattern that could improve a lot would be the "template pattern", in a specific form to separate likely changing parts such as the boilerplate (e.g., `validate_data` vs `_validate_data` and such) from the extension locus.
Reference: https://refactoring.guru/design-patterns/template-method

Examples of how this can be used to improve stability:

* `sktime`, for a different API, has a separation between `fit` calling an internal `_fit`, where change-prone boilerplate is sandwiched between a stable user contract (`fit`) and a stable extender contract (`_fit`); similarly `predict` and `_predict`
* `feature-engine` overrides the `BaseTransformer` `scikit-learn` extension contract with a similar pattern using `super()` calls in `fit` etc.

In particular the `fit`/`_fit` pairing that combines strategy and template pattern can be introduced easily via pure internal refactoring - I would go with that, and also add a layer of tests that checks the stability of the extension contract.

As a side effect, this would likely reduce the amount of copy-pasted boilerplate among the core estimators in `scikit-learn` proper as well.

### Describe alternatives you've considered, if relevant

The obvious alternative is "do nothing" - which will lead to packages with extensions producing their own version of mitigating patterns for the unstable status quo.

The problem with this is that the extending packages are mutually incompatible with their extender pattern. It would be better if there is a single stable extension pattern prescribed - or at least recommended - by `scikit-learn`.

### Additional context

_No response_

COMMENT:
- Regarding downwards compatibility, things should be better from now on since we moved a bunch of private stuff part of a developer API (meaning that they won't change without prior warning). In particular, compliance checks with `check_estimator` should be a lot more stable now. Note that there's still a plan to split these checks into pure compliance checks and sklearn specific checks, to avoid forcing third-party developers rely on all our private helpers.

  I see 2 ways where you (third-party developers) can help us make life easier for you:
  - By testing against the development version of scikit-learn in your CI and open issues in our issue tracker when we break stuff. This way we can fix it before releasing or work together to adapt your lib with the new changes, without breaking your users' code.
  - By opening issues to request us to make public some, currently private, functions or classes that you use. It would help the developer API to grow in the right direction.
  
  ping @adrinjalali @glemaitre who were involved a lot in the developer API in case I forgot something or if you want to add something.

- Regarding `fit/_fit`, I think the best way to move forward is to open a dedicated issue if you're willing to. As said I'm open to that and I think @adrinjalali as well iirc from previous discussions. In that issue we can discuss precisely the API we want, maybe involving more third-party developers to make sure that we define something that everyone can benefit, and the plan to implement it through the whole code base.

>  Is there a way of engaging or escalating that the suggestion gets attention? Is there a procedure to follow? Or is it more an unstructured random process, by which maintainers may or may not ultimately reply?

I'd say that usually it's more the latter. I pinged a couple of maintainers that have been working on related concerns.

Labels: New Feature, Developer API