Describe `set_{method}_request()` API, expose `_MetadataRequester`, or expose `_BaseScorer`

### Describe the issue linked to the documentation

TL;DR: Metadata routing for scoring could either use a base class or documentation of how to write `set_score_request()`.

Currently the [Metadata Estimator Dev Guide](https://scikit-learn.org/stable/auto_examples/miscellaneous/plot_metadata_routing.html#metadata-routing) has examples of a metadata-consuming estimator and a metadata-routing estimator.  However, the metadata routing is also designed for scorers and CV splitters which may or may not be estimators.   Fortunately, `sklearn.model_selection` exposes `BaseCrossValidator`, which like `BaseEstimator`, subclasses `_MetadataRequester`.  Unfortunately, ~there's no base class for scorers.~ the base class for scorers, `_BaseScorer`, is not public.

I don't understand how to string together the relevant methods that should be a part of `set_score_params`, The current workaround is to simply subclass `BaseEstimator`, even if I'm not making an estimator, or to subclass `_MetadataRequester`, even though its not part of the public API.  ~Or use `make_scorer` to pin the kwargs when instantiating the meta-estimator, rather than in `fit()`~

My use case is for scoring a time-series model where the data generating mechanism is known to the experiment, but not the model, and I need to compare the fit model to the true data generating mechanism.  I understand how to use a custom scorer in `RandomizedSearchCV`, and the [metadata API](https://scikit-learn.org/stable/metadata_routing.html#api-interface) explains how meta estimators like `RandomizedSearchCV` can pass additional arguments to my custom scorer.

### Suggest a potential alternative/fix

* publicly exposing `_MetadataRequester`
* publicly exposing `_BaseScorer`
* Document how to write the `set_{method}_request()` methods.  It looks like `_MetadataRequester` uses a descriptor `RequestMethod`, which relies on an instance having a `_get_metadata_request` method and a `_metadata_request` attribute (which it doesn't really describe).

EDIT: Found out about `_BaseScorer`.  Also, removed `make_scorer` workaround, as it doesn't result in passing the fitted estimator to the `_score_func`

COMMENT:
Okay, based on our deep dive into the source code, here's a more informed and detailed English reply you can post for `betatim`:

---

Hi @betatim ,

Thanks for the guidance and for encouraging a thorough discussion before implementation. I completely agree that reaching a consensus on the approach is the best way forward.

Having looked into the relevant parts of the codebase (notably `sklearn/metrics/_scorer.py` and `sklearn/utils/_metadata_requests.py`), my understanding of the issue and potential solutions has been refined:

**Understanding of the Issue:**

The core challenge is to provide a clear, public, and user-friendly way for custom scorers to participate in scikit-learn's metadata routing mechanism. Users, like Jacob-Stevens-Haas, find it difficult to make their scorers request and consume metadata (e.g., `sample_weight`, or other custom parameters) effectively.

My key takeaways from the source code are:
1.  The internal `_BaseScorer` class (in `sklearn/metrics/_scorer.py`) **already inherits from `_MetadataRequester`**. This is a crucial piece of existing infrastructure, meaning `_BaseScorer` instances (and classes inheriting from it) inherently possess the `set_{method}_request()` methods, including `set_score_request()`.
2.  Scorers created by `make_scorer` are instances of `_Scorer`, which itself inherits from `_BaseScorer`. Therefore, these scorer objects *also* have the `set_score_request()` capability.
3.  The main gaps seem to be:
    *   **Visibility/API:** `_BaseScorer` is internal.
    *   **Documentation/Discoverability:** The fact that `make_scorer` objects can have their metadata requests configured via `set_score_request()` is not well-documented or obvious.
    *   **Ease of Use for Complex Scorers:** While `make_scorer` is convenient for simple functions, creating more complex, stateful scorers that properly handle metadata routing by inheriting from an internal class is not a viable public pattern.

**Proposed Solutions (Informed by Source Code):**

Given that `_BaseScorer` already has the foundational link to `_MetadataRequester`, the path forward might be less about building from scratch and more about exposing and refining existing mechanisms:

1.  **Short-Term - Documentation & Clarification (Leveraging Existing Functionality):**
    *   **Action:** Immediately improve the documentation to clearly state that scorer objects returned by `make_scorer` possess `set_score_request()` (and other `set_{method}_request()` methods) due to their inheritance chain (`_Scorer` -> `_BaseScorer` -> `_MetadataRequester`).
    *   **Benefit:** This would provide an immediate, albeit perhaps not perfectly idiomatic, way for users to make their `make_scorer`-generated scorers metadata-aware. This directly addresses part of "Describe set\_{method}\_request() API".
    *   **Example to add to docs:**
        ```python
        from sklearn.metrics import make_scorer, f1_score
        # Assume my_complex_score_func uses sample_weight and custom_meta
        scorer = make_scorer(my_complex_score_func)
        scorer.set_score_request(sample_weight=True, custom_meta=True)
        # ... then use scorer in GridSearchCV etc.
        ```

2.  **Medium-Term - Enhance `make_scorer` (Improved Convenience):**
    *   **Action:** Add an optional parameter to `make_scorer`, perhaps `score_requests: dict = None`. If provided, `make_scorer` would internally call `set_score_request(**score_requests)` on the created scorer object.
    *   **Benefit:** This makes the common case more straightforward without requiring a separate call after `make_scorer`.

3.  **Longer-Term - Public `BaseScorer` (Robust Solution for Custom Scorers):**
    *   **Action:** Refactor and publicize `sklearn.metrics._scorer._BaseScorer` into something like `sklearn.metrics.BaseScorer`.
        *   It would continue to inherit from `sklearn.utils.metadata_routing.MetadataRequester` (which is publicly available as `sklearn.utils.metadata_routing.MetadataRequester`).
        *   Review and refine its `__init__` method and the way it handles the `score_func` and `_score` method for easier subclassing. The goal is to allow users to primarily override an internal scoring method (like `_score`) and use the inherited `set_score_request()` for metadata.
    *   **Documentation:** Provide clear examples of subclassing this public `BaseScorer`, implementing the scoring logic, and declaring metadata requirements.
    *   **Benefit:** This directly addresses the "make a base class for Scorers" part of the issue title and provides a solid, object-oriented way for users to build complex, metadata-aware scorers. It also naturally explains how `set_score_request()` works in this context.
    *   *Regarding "expose `_MetadataRequester`"*: `_MetadataRequester`'s functionality is best exposed *through* classes that inherit it (like `BaseEstimator` and this proposed public `BaseScorer`), rather than exposing it directly for all users, as it's more of a mixin/building block.

**Path Forward:**

I believe a combination of these, perhaps starting with improved documentation (1) and then working towards a public `BaseScorer` (3), would be a robust way to resolve this issue. Enhancing `make_scorer` (2) could be a valuable addition for convenience along the way.

The key is that much of the underlying machinery (thanks to `_BaseScorer` inheriting `_MetadataRequester`) seems to be in place, and the effort might be more focused on API design, public exposure, and documentation.

I'm keen to discuss these points and any other perspectives the team has. My aim is to contribute to a solution that is both powerful and user-friendly.

Best regards,
MagicDake

Labels: Documentation, Needs Investigation, Metadata Routing