Talip7/scikit-learn-issues-multilabel huggingface dataset:
Describe `set_{method}_request()` API, expose `_MetadataRequester`, or expose `_BaseScorer`

### Describe the issue linked to the documentation

TL;DR: Metadata routing for scoring could either use a base class or documentation of how to write `set_score_request()`.

Currently the [Metadata Estimator Dev Guide](https://scikit-learn.org/stable/auto_examples/miscellaneous/plot_metadata_routing.html#metadata-routing) has examples of a metadata-consuming estimator and a metadata-routing estimator.  However, the metadata routing is also designed for scorers and CV splitters which may or may not be estimators.   Fortunately, `sklearn.model_selection` exposes `BaseCrossValidator`, which like `BaseEstimator`, subclasses `_MetadataRequester`.  Unfortunately, ~there's no base class for scorers.~ the base class for scorers, `_BaseScorer`, is not public.

I don't understand how to string together the relevant methods that should be a part of `set_score_params`, The current workaround is to simply subclass `BaseEstimator`, even if I'm not making an estimator, or to subclass `_MetadataRequester`, even though its not part of the public API.  ~Or use `make_scorer` to pin the kwargs when instantiating the meta-estimator, rather than in `fit()`~

My use case is for scoring a time-series model where the data generating mechanism is known to the experiment, but not the model, and I need to compare the fit model to the true data generating mechanism.  I understand how to use a custom scorer in `RandomizedSearchCV`, and the [metadata API](https://scikit-learn.org/stable/metadata_routing.html#api-interface) explains how meta estimators like `RandomizedSearchCV` can pass additional arguments to my custom scorer.

### Suggest a potential alternative/fix

* publicly exposing `_MetadataRequester`
* publicly exposing `_BaseScorer`
* Document how to write the `set_{method}_request()` methods.  It looks like `_MetadataRequester` uses a descriptor `RequestMethod`, which relies on an instance having a `_get_metadata_request` method and a `_metadata_request` attribute (which it doesn't really describe).

EDIT: Found out about `_BaseScorer`.  Also, removed `make_scorer` workaround, as it doesn't result in passing the fitted estimator to the `_score_func`

COMMENT:
Two ancillary confusions I've run into trying to understand metadata
1)  the documentation for [`MetadataRouter`](https://scikit-learn.org/stable/modules/generated/sklearn.utils.metadata_routing.MetadataRouter.html) says that it "Stores and handles metadata routing for a router object".  So a `MetadataRouter` instance isn't a router object?  Or it's designed recursively?
2) Since I'm making a pure consumer, I looked at [`MetadataRequest`](https://scikit-learn.org/stable/modules/generated/sklearn.utils.metadata_routing.MetadataRequest.html) docs to understand the role of `get_metadata_routing()`.  It says,  "Consumer-only classes such as simple estimators return a serialized version of this class as the output of `get_metadata_routing()`"  What is a serialized version of this class?  a named tuple of attributes?  Or the output of `pickle.dumps()`?

Labels: Documentation, Needs Investigation, Metadata Routing
Describe `set_{method}_request()` API, expose `_MetadataRequester`, or expose `_BaseScorer`

### Describe the issue linked to the documentation

TL;DR: Metadata routing for scoring could either use a base class or documentation of how to write `set_score_request()`.

Currently the [Metadata Estimator Dev Guide](https://scikit-learn.org/stable/auto_examples/miscellaneous/plot_metadata_routing.html#metadata-routing) has examples of a metadata-consuming estimator and a metadata-routing estimator.  However, the metadata routing is also designed for scorers and CV splitters which may or may not be estimators.   Fortunately, `sklearn.model_selection` exposes `BaseCrossValidator`, which like `BaseEstimator`, subclasses `_MetadataRequester`.  Unfortunately, ~there's no base class for scorers.~ the base class for scorers, `_BaseScorer`, is not public.

I don't understand how to string together the relevant methods that should be a part of `set_score_params`, The current workaround is to simply subclass `BaseEstimator`, even if I'm not making an estimator, or to subclass `_MetadataRequester`, even though its not part of the public API.  ~Or use `make_scorer` to pin the kwargs when instantiating the meta-estimator, rather than in `fit()`~

My use case is for scoring a time-series model where the data generating mechanism is known to the experiment, but not the model, and I need to compare the fit model to the true data generating mechanism.  I understand how to use a custom scorer in `RandomizedSearchCV`, and the [metadata API](https://scikit-learn.org/stable/metadata_routing.html#api-interface) explains how meta estimators like `RandomizedSearchCV` can pass additional arguments to my custom scorer.

### Suggest a potential alternative/fix

* publicly exposing `_MetadataRequester`
* publicly exposing `_BaseScorer`
* Document how to write the `set_{method}_request()` methods.  It looks like `_MetadataRequester` uses a descriptor `RequestMethod`, which relies on an instance having a `_get_metadata_request` method and a `_metadata_request` attribute (which it doesn't really describe).

EDIT: Found out about `_BaseScorer`.  Also, removed `make_scorer` workaround, as it doesn't result in passing the fitted estimator to the `_score_func`

COMMENT:
As a side note about AI/automatically generated work/comments for scikit-learn https://scikit-learn.org/stable/developers/contributing.html#automated-contributions-policy Please review and thinking about the content before posting it. That way we can make sure it is helpful and doesn't lead to a lot of spam.

Labels: Documentation, Needs Investigation, Metadata Routing
Describe `set_{method}_request()` API, expose `_MetadataRequester`, or expose `_BaseScorer`

### Describe the issue linked to the documentation

TL;DR: Metadata routing for scoring could either use a base class or documentation of how to write `set_score_request()`.

Currently the [Metadata Estimator Dev Guide](https://scikit-learn.org/stable/auto_examples/miscellaneous/plot_metadata_routing.html#metadata-routing) has examples of a metadata-consuming estimator and a metadata-routing estimator.  However, the metadata routing is also designed for scorers and CV splitters which may or may not be estimators.   Fortunately, `sklearn.model_selection` exposes `BaseCrossValidator`, which like `BaseEstimator`, subclasses `_MetadataRequester`.  Unfortunately, ~there's no base class for scorers.~ the base class for scorers, `_BaseScorer`, is not public.

I don't understand how to string together the relevant methods that should be a part of `set_score_params`, The current workaround is to simply subclass `BaseEstimator`, even if I'm not making an estimator, or to subclass `_MetadataRequester`, even though its not part of the public API.  ~Or use `make_scorer` to pin the kwargs when instantiating the meta-estimator, rather than in `fit()`~

My use case is for scoring a time-series model where the data generating mechanism is known to the experiment, but not the model, and I need to compare the fit model to the true data generating mechanism.  I understand how to use a custom scorer in `RandomizedSearchCV`, and the [metadata API](https://scikit-learn.org/stable/metadata_routing.html#api-interface) explains how meta estimators like `RandomizedSearchCV` can pass additional arguments to my custom scorer.

### Suggest a potential alternative/fix

* publicly exposing `_MetadataRequester`
* publicly exposing `_BaseScorer`
* Document how to write the `set_{method}_request()` methods.  It looks like `_MetadataRequester` uses a descriptor `RequestMethod`, which relies on an instance having a `_get_metadata_request` method and a `_metadata_request` attribute (which it doesn't really describe).

EDIT: Found out about `_BaseScorer`.  Also, removed `make_scorer` workaround, as it doesn't result in passing the fitted estimator to the `_score_func`

COMMENT:
cc @adrinjalali in case you have time to have a closer look.

Labels: Documentation, Needs Investigation, Metadata Routing
Describe `set_{method}_request()` API, expose `_MetadataRequester`, or expose `_BaseScorer`

### Describe the issue linked to the documentation

TL;DR: Metadata routing for scoring could either use a base class or documentation of how to write `set_score_request()`.

Currently the [Metadata Estimator Dev Guide](https://scikit-learn.org/stable/auto_examples/miscellaneous/plot_metadata_routing.html#metadata-routing) has examples of a metadata-consuming estimator and a metadata-routing estimator.  However, the metadata routing is also designed for scorers and CV splitters which may or may not be estimators.   Fortunately, `sklearn.model_selection` exposes `BaseCrossValidator`, which like `BaseEstimator`, subclasses `_MetadataRequester`.  Unfortunately, ~there's no base class for scorers.~ the base class for scorers, `_BaseScorer`, is not public.

I don't understand how to string together the relevant methods that should be a part of `set_score_params`, The current workaround is to simply subclass `BaseEstimator`, even if I'm not making an estimator, or to subclass `_MetadataRequester`, even though its not part of the public API.  ~Or use `make_scorer` to pin the kwargs when instantiating the meta-estimator, rather than in `fit()`~

My use case is for scoring a time-series model where the data generating mechanism is known to the experiment, but not the model, and I need to compare the fit model to the true data generating mechanism.  I understand how to use a custom scorer in `RandomizedSearchCV`, and the [metadata API](https://scikit-learn.org/stable/metadata_routing.html#api-interface) explains how meta estimators like `RandomizedSearchCV` can pass additional arguments to my custom scorer.

### Suggest a potential alternative/fix

* publicly exposing `_MetadataRequester`
* publicly exposing `_BaseScorer`
* Document how to write the `set_{method}_request()` methods.  It looks like `_MetadataRequester` uses a descriptor `RequestMethod`, which relies on an instance having a `_get_metadata_request` method and a `_metadata_request` attribute (which it doesn't really describe).

EDIT: Found out about `_BaseScorer`.  Also, removed `make_scorer` workaround, as it doesn't result in passing the fitted estimator to the `_score_func`

COMMENT:
Sorry for the noise generated by @MagicDake , the user is now banned from the org.

I agree `_BaseScorer` could be public and document how to write a scorer, which should solve the issue here.

I don't think we need to make `_MetadataRequester` public since I haven't seen a case where using it directly would make sense (assuming `_BaseScorer` is public.

What I've seen happening in a few places, is the need to implement a `set_{method}_request`, both in `fairlearn` and in `skorch`. So documenting that and having a way to implement it nicely probably makes very much sense.

cc @thomasjpfan @glemaitre on making `_BaseScorer` public, ideas?

Labels: Documentation, Needs Investigation, Metadata Routing
Describe `set_{method}_request()` API, expose `_MetadataRequester`, or expose `_BaseScorer`

### Describe the issue linked to the documentation

TL;DR: Metadata routing for scoring could either use a base class or documentation of how to write `set_score_request()`.

Currently the [Metadata Estimator Dev Guide](https://scikit-learn.org/stable/auto_examples/miscellaneous/plot_metadata_routing.html#metadata-routing) has examples of a metadata-consuming estimator and a metadata-routing estimator.  However, the metadata routing is also designed for scorers and CV splitters which may or may not be estimators.   Fortunately, `sklearn.model_selection` exposes `BaseCrossValidator`, which like `BaseEstimator`, subclasses `_MetadataRequester`.  Unfortunately, ~there's no base class for scorers.~ the base class for scorers, `_BaseScorer`, is not public.

I don't understand how to string together the relevant methods that should be a part of `set_score_params`, The current workaround is to simply subclass `BaseEstimator`, even if I'm not making an estimator, or to subclass `_MetadataRequester`, even though its not part of the public API.  ~Or use `make_scorer` to pin the kwargs when instantiating the meta-estimator, rather than in `fit()`~

My use case is for scoring a time-series model where the data generating mechanism is known to the experiment, but not the model, and I need to compare the fit model to the true data generating mechanism.  I understand how to use a custom scorer in `RandomizedSearchCV`, and the [metadata API](https://scikit-learn.org/stable/metadata_routing.html#api-interface) explains how meta estimators like `RandomizedSearchCV` can pass additional arguments to my custom scorer.

### Suggest a potential alternative/fix

* publicly exposing `_MetadataRequester`
* publicly exposing `_BaseScorer`
* Document how to write the `set_{method}_request()` methods.  It looks like `_MetadataRequester` uses a descriptor `RequestMethod`, which relies on an instance having a `_get_metadata_request` method and a `_metadata_request` attribute (which it doesn't really describe).

EDIT: Found out about `_BaseScorer`.  Also, removed `make_scorer` workaround, as it doesn't result in passing the fitted estimator to the `_score_func`

COMMENT:
I think that I would not mind to have 3 elements public: `BaseEstimator`, `_BaseScorer`, and `BaseCrossValidator`. In my mind, it is the triptych that I could use to extend scikit-learn and can make me answer to many additional use-cases.

Labels: Documentation, Needs Investigation, Metadata Routing
Describe `set_{method}_request()` API, expose `_MetadataRequester`, or expose `_BaseScorer`

### Describe the issue linked to the documentation

TL;DR: Metadata routing for scoring could either use a base class or documentation of how to write `set_score_request()`.

Currently the [Metadata Estimator Dev Guide](https://scikit-learn.org/stable/auto_examples/miscellaneous/plot_metadata_routing.html#metadata-routing) has examples of a metadata-consuming estimator and a metadata-routing estimator.  However, the metadata routing is also designed for scorers and CV splitters which may or may not be estimators.   Fortunately, `sklearn.model_selection` exposes `BaseCrossValidator`, which like `BaseEstimator`, subclasses `_MetadataRequester`.  Unfortunately, ~there's no base class for scorers.~ the base class for scorers, `_BaseScorer`, is not public.

I don't understand how to string together the relevant methods that should be a part of `set_score_params`, The current workaround is to simply subclass `BaseEstimator`, even if I'm not making an estimator, or to subclass `_MetadataRequester`, even though its not part of the public API.  ~Or use `make_scorer` to pin the kwargs when instantiating the meta-estimator, rather than in `fit()`~

My use case is for scoring a time-series model where the data generating mechanism is known to the experiment, but not the model, and I need to compare the fit model to the true data generating mechanism.  I understand how to use a custom scorer in `RandomizedSearchCV`, and the [metadata API](https://scikit-learn.org/stable/metadata_routing.html#api-interface) explains how meta estimators like `RandomizedSearchCV` can pass additional arguments to my custom scorer.

### Suggest a potential alternative/fix

* publicly exposing `_MetadataRequester`
* publicly exposing `_BaseScorer`
* Document how to write the `set_{method}_request()` methods.  It looks like `_MetadataRequester` uses a descriptor `RequestMethod`, which relies on an instance having a `_get_metadata_request` method and a `_metadata_request` attribute (which it doesn't really describe).

EDIT: Found out about `_BaseScorer`.  Also, removed `make_scorer` workaround, as it doesn't result in passing the fitted estimator to the `_score_func`

COMMENT:
Yup I agree, it’s time to make `_BaseScorer` public.

> What I've seen happening in a few places, is the need to implement a set_{method}_request, both in fairlearn and in skorch

Is this for scorers or estimators?

Labels: Documentation, Needs Investigation, Metadata Routing
Describe `set_{method}_request()` API, expose `_MetadataRequester`, or expose `_BaseScorer`

### Describe the issue linked to the documentation

TL;DR: Metadata routing for scoring could either use a base class or documentation of how to write `set_score_request()`.

Currently the [Metadata Estimator Dev Guide](https://scikit-learn.org/stable/auto_examples/miscellaneous/plot_metadata_routing.html#metadata-routing) has examples of a metadata-consuming estimator and a metadata-routing estimator.  However, the metadata routing is also designed for scorers and CV splitters which may or may not be estimators.   Fortunately, `sklearn.model_selection` exposes `BaseCrossValidator`, which like `BaseEstimator`, subclasses `_MetadataRequester`.  Unfortunately, ~there's no base class for scorers.~ the base class for scorers, `_BaseScorer`, is not public.

I don't understand how to string together the relevant methods that should be a part of `set_score_params`, The current workaround is to simply subclass `BaseEstimator`, even if I'm not making an estimator, or to subclass `_MetadataRequester`, even though its not part of the public API.  ~Or use `make_scorer` to pin the kwargs when instantiating the meta-estimator, rather than in `fit()`~

My use case is for scoring a time-series model where the data generating mechanism is known to the experiment, but not the model, and I need to compare the fit model to the true data generating mechanism.  I understand how to use a custom scorer in `RandomizedSearchCV`, and the [metadata API](https://scikit-learn.org/stable/metadata_routing.html#api-interface) explains how meta estimators like `RandomizedSearchCV` can pass additional arguments to my custom scorer.

### Suggest a potential alternative/fix

* publicly exposing `_MetadataRequester`
* publicly exposing `_BaseScorer`
* Document how to write the `set_{method}_request()` methods.  It looks like `_MetadataRequester` uses a descriptor `RequestMethod`, which relies on an instance having a `_get_metadata_request` method and a `_metadata_request` attribute (which it doesn't really describe).

EDIT: Found out about `_BaseScorer`.  Also, removed `make_scorer` workaround, as it doesn't result in passing the fitted estimator to the `_score_func`

COMMENT:
> Is this for scorers or estimators?

Both. Scorers usually don't have the metadata in the signature, so a generic "accepting all kwarg" is used there (same in sklearn itself). And for estimators where metadata can be dynamic or pretty much anything, the same thing happens.

Labels: Documentation, Needs Investigation, Metadata Routing
Describe `set_{method}_request()` API, expose `_MetadataRequester`, or expose `_BaseScorer`

### Describe the issue linked to the documentation

TL;DR: Metadata routing for scoring could either use a base class or documentation of how to write `set_score_request()`.

Currently the [Metadata Estimator Dev Guide](https://scikit-learn.org/stable/auto_examples/miscellaneous/plot_metadata_routing.html#metadata-routing) has examples of a metadata-consuming estimator and a metadata-routing estimator.  However, the metadata routing is also designed for scorers and CV splitters which may or may not be estimators.   Fortunately, `sklearn.model_selection` exposes `BaseCrossValidator`, which like `BaseEstimator`, subclasses `_MetadataRequester`.  Unfortunately, ~there's no base class for scorers.~ the base class for scorers, `_BaseScorer`, is not public.

I don't understand how to string together the relevant methods that should be a part of `set_score_params`, The current workaround is to simply subclass `BaseEstimator`, even if I'm not making an estimator, or to subclass `_MetadataRequester`, even though its not part of the public API.  ~Or use `make_scorer` to pin the kwargs when instantiating the meta-estimator, rather than in `fit()`~

My use case is for scoring a time-series model where the data generating mechanism is known to the experiment, but not the model, and I need to compare the fit model to the true data generating mechanism.  I understand how to use a custom scorer in `RandomizedSearchCV`, and the [metadata API](https://scikit-learn.org/stable/metadata_routing.html#api-interface) explains how meta estimators like `RandomizedSearchCV` can pass additional arguments to my custom scorer.

### Suggest a potential alternative/fix

* publicly exposing `_MetadataRequester`
* publicly exposing `_BaseScorer`
* Document how to write the `set_{method}_request()` methods.  It looks like `_MetadataRequester` uses a descriptor `RequestMethod`, which relies on an instance having a `_get_metadata_request` method and a `_metadata_request` attribute (which it doesn't really describe).

EDIT: Found out about `_BaseScorer`.  Also, removed `make_scorer` workaround, as it doesn't result in passing the fitted estimator to the `_score_func`

COMMENT:
It looks like `_BaseScorer` and `_Scorer` both do some useful things for a base class, and both have certain assumptions that a user might want to change.  E.g. in my case, I wouldn't have `self._score_func` that compares `y_pred` and `y_true`.  While `_Scorer._score` makes predictions and calls `_score_func` with a particular syntax, `_score_func` is an attribute of `_BaseScorer`.  OTOH, I probably *would* want the call to `self._warn_overlap` in case kwargs overlapped between metadata and directly-passed args, and that's currently in `_Scorer._score`.

Would you mind if I took a stab at a PR?  ~Only thing I'm unsure of now is how `_BaseScorer._response_method` gets set.~ EDIT: nvm it looks like that's already publicly documented in `sklearn.metrics.make_scorer()`.  I'll try to make a PR

Labels: Documentation, Needs Investigation, Metadata Routing