SLEP006 - Metadata Routing task list

This issue is to track the work we need to do before we can merge `sample-props` branch into `main`:

- [x] Based on this prototype: https://github.com/scikit-learn/scikit-learn/pull/16079
- [x] Merge https://github.com/scikit-learn/scikit-learn/pull/22083 into `sample-props`. This PR only touches `BaseEstimator` and hence _consumers_. It does NOT touch meta-estimators, scorers or cv splitters.
- [x] Work on splitters and merge that into `sample-props`: https://github.com/scikit-learn/scikit-learn/pull/22765
- [x] feature flag: https://github.com/scikit-learn/scikit-learn/issues/26045: https://github.com/scikit-learn/scikit-learn/pull/26103
- [x] Add new terms to the glossary: router, consumer, metadata: https://github.com/scikit-learn/scikit-learn/pull/26685
- [x] Work on scorers and merge that into `sample-props` ~(note that this involves an ongoing discussion on whether we'd like to mutate a scorer or not)~: https://github.com/scikit-learn/scikit-learn/pull/22757
- [x] Get backward compatibility mechanisms in place for meta-estimators which already route given metadata: https://github.com/scikit-learn/scikit-learn/pull/22986
- [ ] Work on meta-estimators: this involves writing `get_metadata_routing` in easy cases, and a whole lot more in cases where we'd like to keep backward compatibility in parsing input args suchs as `estimator__param` in `Pipeline`
- [x] Once all the above is merged into `sample-props`, do a few tests with third party estimators to see if they'd work out of the box. Note that _consumer_ estimators should work out of the box as long as they inherit from `BaseEstimator` and their `fit` accepts metadata as explicit arguments rather than `**kwargs`.
  - Merged w/o completion of meta-estimators, they got done progressively.
- [x] Check whether a library such as cuML could vendor `_metadata_requests.py` and work with scikit-learn meta-estimators w/o depending on the library.
  - dropped the idea of supporting vendoring, they'd need to inherit from `BaseEstimator`
- [x] Refactor tests: https://github.com/scikit-learn/scikit-learn/issues/23918
- [x] Merge `sample-props` into `main`: https://github.com/scikit-learn/scikit-learn/pull/24027

## Enhancements:
- [x] https://github.com/scikit-learn/scikit-learn/pull/24023
- [ ] https://github.com/scikit-learn/scikit-learn/issues/18936
- [x] https://github.com/scikit-learn/scikit-learn/issues/28467

## Cleanup
- [x] https://github.com/scikit-learn/scikit-learn/issues/26505
	- [x] https://github.com/scikit-learn/scikit-learn/pull/26503

## Open issues:
- [x] https://github.com/scikit-learn/scikit-learn/issues/22987
	- [x] https://github.com/scikit-learn/scikit-learn/pull/26506
- [x] https://github.com/scikit-learn/scikit-learn/issues/22988: https://github.com/scikit-learn/scikit-learn/pull/23342
- [ ] https://github.com/scikit-learn/scikit-learn/issues/23920
- [ ] https://github.com/scikit-learn/scikit-learn/issues/23928
	- [ ] global flags: https://github.com/scikit-learn/scikit-learn/issues/26050
	- [ ] auto-routing: https://github.com/scikit-learn/scikit-learn/issues/26179
- [x] https://github.com/scikit-learn/scikit-learn/issues/23933
	- [x] https://github.com/scikit-learn/scikit-learn/pull/26911
- [x] https://github.com/scikit-learn/scikit-learn/issues/25776: https://github.com/scikit-learn/scikit-learn/pull/25852 - resolution is to use the feature flag instead of breaking change.
- [ ] https://github.com/scikit-learn/scikit-learn/issues/27977

Our plan is to hopefully have this feature in 1.1, which we should be releasing in late April/early May.

cc @jnothman @thomasjpfan @lorentzenchr 

## Misc:

- https://github.com/scikit-learn/scikit-learn/pull/25851
- https://github.com/scikit-learn/scikit-learn/pull/27389

-------------------------------------------------------------
From #24027 

Once we're finished with merging PRs to `sample-props` branch, we should merge this.

I've opened this for us to have an idea of what the whole project is touching.

We probably should rebase and merge this instead of squash and merge.

This closes and fixes the following issues and PRs:

- Routing issues:
Enables #6322, 

- AdaBoost:
Fixes #2630, 
Fixes #21706

- ClassifierChain:
Fixes  #11429, 

- FeatureUnion:
Fixes #7136

- LogisticRegressionCV:
Fixes  #8950,

- Multioutput*
Fixes #15953

- OneVsRestClassifier
Fixes #10882

- RFE/RFECV:
Fixes #7308

- *SearchCV:
Fixes #8127, 
Fixes #8158,

- cross_val*:
Fixes #13432, 
Fixes #4632, 
Fixes #7646, 
Fixes #20349

- Stacking*
Fixes #18028,

- Voting*
Fixes #20167

COMMENT:
I spent a few days trying to write common tests for meta-estimators but that didn't go anywhere. After being stuck for a while, @thomasjpfan and I spent some time last week together and we decided to start with simple individual tests, starting for one meta-estimator, and then refactor the tests later when we find recurring patterns.

Right now I'm working on multioutput meta-estimators and should have a PR coming today.

Labels: Hard, Meta-issue, Metadata Routing