Stable extender contract via `fit` / `_fit` resp `predict` / `_predict` separation

### Describe the workflow you want to enable

tl;dr, I am suggestion to refactor `scikit-learn` internals to a layer separation with boilerplate between `fit` and `_fit` resp `predict` and `_predict` methods, to make extender interfaces more stable. Also see https://github.com/scikit-learn/scikit-learn/issues/31728

More background: Currently, every time `scikit-learn` releases a new minor version - e.g., 1.5.0, 1.6.0, 1.7.0 - compliant extensions, e.g., custom transformers, classifiers, etc, break - specifically, referring to the API conformance as tested through `check_estimator` or `parametrize_with_checks`.

These repeated breakages in the "extender contract" contrast the stability of the usage contract, which is stable and professionally managed.

For a package like `scikit-learn` which means to be a standard not just for ML algorithms but also an API standard that everyone uses, this is not a good state to be in - "do not break user code" is the maxim that gets broken for power users writing extensions.

Of course maintaining downwards compatibility is not always possible, but nothing should break without a proper warning.

### Describe your proposed solution

The `fit`/`_fit` separation would ensure stability of the extension contract - and would also allow to build secondary deprecation patterns in relation to it.

The (oop) pattern this would implement is the so-called "template pattern".

It would allow to remove likely changing parts such as the boilerplate (e.g., `validate_data` vs `_validate_data` and such) from the extension locus, and thus completely prevent breakage in relation to boilerplate changes.
Reference: https://refactoring.guru/design-patterns/template-method

Examples of how this can be used to improve stability:

* `sktime`, for a different API, has a separation between `fit` calling an internal `_fit`, where change-prone boilerplate is sandwiched between a stable user contract (`fit`) and a stable extender contract (`_fit`); similarly `predict` and `_predict`
* `feature-engine` overrides the `BaseTransformer` `scikit-learn` extension contract with a similar pattern using `super()` calls in `fit` etc.

The proposed `fit`/`_fit` pairing that combines strategy and template pattern can be introduced easily via pure internal refactoring - this also add a layer of tests that checks the stability of the extension contract.

As a side effect, this would likely reduce the amount of copy-pasted boilerplate among the core estimators in `scikit-learn` proper as well.

### Describe alternatives you've considered, if relevant

The obvious alternative is "do nothing" - which will lead to packages with extensions producing their own version of mitigating patterns for the unstable status quo.

The problem with this is that the extending packages are mutually incompatible with their extender pattern. It would be better if there is a single stable extension pattern prescribed - or at least recommended - by `scikit-learn`.

### Additional context

_No response_

COMMENT:
Replying in sequence to the above.

> @adrinjalali: I can't think of a previous case where this would have helped

I can think of more previous cases:

* introduction of array API - non-conformant estimators can be "patched" through the boilerplate by coercion to inner supported type
* broadcasting for multioutput - every estimator can be made multioutput compatible by broadcasting self in the boilerplate
* changes to data validation and check, this can essentially move to boilerplate entirely

> @adrinjalali: it is possible that we can find a backward compatible way to implement the change which would avoid going through a SLEP.

This would actually be my standard assumption!

Since `fit` is overridden currently in all concrete classes, a change in base default will not affect any current concrete class.

Concrete classes can be moved individuall to `_fit`, while those that have not moved will continue working - at leats as long as divergence in the public `fit` contract is avoided entirely.

> @thomasjpfan: This means we'll also need to include extension points for child estimators to customize the validation logic.

In `sktime`, custom validation logic is steered by tags, e.g., you can say which internal types are supported, see
https://www.sktime.net/en/latest/api_reference/auto_generated/sktime.registry._tags.x_inner_mtype.html

An alternative is to also template the validation logic, e.g., have a `_fit_validate` method that has a sensible default but can be extended.

> @adrinjalali: As for validation logic, some of it can go into tags, and the rest we can start by letting developers implement themselves in their classes to start with.

Exactly, that is what we do in `sktime` (see above).

> @betatim: It sounds like the problem this issue is trying to address is the challenge of creating a third-party estimator that is considered compatible/conformant with several versions of scikit-learn at the same time.

That is one problem that is addressed but not the only one.
The template design also makes the extension contract a "public" and "stable" interface; and it makes it easier to conform with.

> @betatim: which of those were "one off" and which aren't.

I think everything was "one off", the point is not that one locus causes breakage multiple times, but that at every version increment something else breaks. The template pattern would address all of these, past and future, by moving the break locus in the boilerplate layer, thus not affecting extenders.

> @adrinjalali: Note that we should be careful considering changing private methods as breaking change. It'd be impossible to develop the library if we can't change our private tooling.

I agree!

The "generalized" stance would be considering your "contract surfaces" as breaking sensitive.

The "default" in python is to consider all public objects as your "contract surface" and nothing more. In case of a public application of the template pattern, your "contract surface" would extend to `_fit`, `_predict`, `_transform`, and a handful other methods you specify and which you can curate in a list if you desire. Any other private methods would not be part of a public contract and can still be touched without deprecation.

From a software engineering perspective, you would need to define which "contracts" are meant to be public or not. The extender contracts have been used widely as - and are actually also widely (at least implicitly) advertised as - public.

> Which private methods/attributes/functions do 3rd-party libraries usually depend on to develop scikit-learn compatible estimators?

* tag related methods, e.g., `_get_tags` ( or `get_tags`)
* `_check_sth`, e.g., `_check_method_params`
* `_validate_sth`, e.g., `_validate_data`
* for unknown and mysterious reasons, recently also a method names `_deprecate_Xt_in_inverse_transform` became necessary

Labels: RFC, Developer API