Stable extender contract via `fit` / `_fit` resp `predict` / `_predict` separation

### Describe the workflow you want to enable

tl;dr, I am suggestion to refactor `scikit-learn` internals to a layer separation with boilerplate between `fit` and `_fit` resp `predict` and `_predict` methods, to make extender interfaces more stable. Also see https://github.com/scikit-learn/scikit-learn/issues/31728

More background: Currently, every time `scikit-learn` releases a new minor version - e.g., 1.5.0, 1.6.0, 1.7.0 - compliant extensions, e.g., custom transformers, classifiers, etc, break - specifically, referring to the API conformance as tested through `check_estimator` or `parametrize_with_checks`.

These repeated breakages in the "extender contract" contrast the stability of the usage contract, which is stable and professionally managed.

For a package like `scikit-learn` which means to be a standard not just for ML algorithms but also an API standard that everyone uses, this is not a good state to be in - "do not break user code" is the maxim that gets broken for power users writing extensions.

Of course maintaining downwards compatibility is not always possible, but nothing should break without a proper warning.

### Describe your proposed solution

The `fit`/`_fit` separation would ensure stability of the extension contract - and would also allow to build secondary deprecation patterns in relation to it.

The (oop) pattern this would implement is the so-called "template pattern".

It would allow to remove likely changing parts such as the boilerplate (e.g., `validate_data` vs `_validate_data` and such) from the extension locus, and thus completely prevent breakage in relation to boilerplate changes.
Reference: https://refactoring.guru/design-patterns/template-method

Examples of how this can be used to improve stability:

* `sktime`, for a different API, has a separation between `fit` calling an internal `_fit`, where change-prone boilerplate is sandwiched between a stable user contract (`fit`) and a stable extender contract (`_fit`); similarly `predict` and `_predict`
* `feature-engine` overrides the `BaseTransformer` `scikit-learn` extension contract with a similar pattern using `super()` calls in `fit` etc.

The proposed `fit`/`_fit` pairing that combines strategy and template pattern can be introduced easily via pure internal refactoring - this also add a layer of tests that checks the stability of the extension contract.

As a side effect, this would likely reduce the amount of copy-pasted boilerplate among the core estimators in `scikit-learn` proper as well.

### Describe alternatives you've considered, if relevant

The obvious alternative is "do nothing" - which will lead to packages with extensions producing their own version of mitigating patterns for the unstable status quo.

The problem with this is that the extending packages are mutually incompatible with their extender pattern. It would be better if there is a single stable extension pattern prescribed - or at least recommended - by `scikit-learn`.

### Additional context

_No response_

COMMENT:
Doing a bit of the leg work here to extract the relevant information from the things linked in https://github.com/scikit-learn/scikit-learn/issues/31912#issuecomment-3182305064. Concrete cases that I could find:

* `_check_method_params` from `sklearn.utils.validation` was used. This is part of the parameter validation "framework" that scikit-learn uses
* `self._validate_data` on an estimator was used, which is now replaced by `validate_data` from `sklearn.utils.validation`
* the change in the tag system needed to be reflected
* `check_array` was being used, now replaced with `validate_data`

This means all the changes were results of changes in scikit-learn that had the goal of taking the undefined "developer API" towards a more well defined (with the possibility of stability promises) "developer API".

The only exception is the parameter validation framework - it isn't clear to me why a third party has to use it.

I think it is true that for all the above cases there was never a need to use the functions and methods from scikit-learn to be able to comply with the common tests. It was convenient to use them, and many packages do use them, but from a strict "API contract" point of view nothing unusual happened. They are private, they change without notice. It isn't clear to me how a different kind of API contract would solve this.

It is important to note that despite being private and technically "free to change without notice" the changes were only made as part of making these parts part of the developer API that can have stability promises.

For me the challenges are:
1. in practice third party packages rely on utilities in scikit-learn that are marked as "can change without notice"
2. there is only so much good will scikit-learn has with developers of third party packages (they want to work on their package, not on keeping up with sickit-learn)
3. evolving to a situation where more utilities are available under a well defined developer API is not easy/quick work

I agree with Gael about his points made with respect to software engineering practices.

As an aside, there is a lot of software engineering, Python programming and interacting with domain experts experience in the scikit-learn maintainer group. This means that when well known software patterns are not used it is because of choice. For example, perceived repetition is on purpose, not as a result of an oversight.

Labels: RFC, Developer API