Repository: mlflow/mlflow
Pull request #15376: Expose scorers
Author: login: artjen
Labels: name: rn/feature
description: Mention under Features in Changelogs.
name: area/tracking
description: Tracking service, tracking client APIs, autologging
Created: 2025-04-18T01:17:15Z | Merged: 2025-04-22T15:18:00Z

<details><summary>&#x1F6E0 DevTools &#x1F6E0</summary>
<p>

[![Open in GitHub Codespaces](https://github.com/codespaces/badge.svg)](https://codespaces.new/artjen/mlflow/pull/15376?quickstart=1)

#### Install mlflow from this PR

```
# mlflow
pip install git+https://github.com/mlflow/mlflow.git@refs/pull/15376/merge
# mlflow-skinny
pip install git+https://github.com/mlflow/mlflow.git@refs/pull/15376/merge#subdirectory=skinny
```

For Databricks, use the following command:

```
%sh curl -LsSf https://raw.githubusercontent.com/mlflow/mlflow/HEAD/dev/install-skinny.sh | sh -s 15376
```

</p>
</details>

### Related Issues/PRs

<!-- Uncomment 'Resolve' if this PR can close the linked items. -->
<!-- Resolve --> #xxx

### What changes are proposed in this pull request?

Expose builtin scorers for mlflow.genai

### How is this PR tested?

- [ ] Existing unit/integration tests
- [x] New unit/integration tests
- [ ] Manual tests

<!-- Attach code, screenshot, video used for manual testing here. -->

### Does this PR require documentation update?

- [x] No. You can skip the rest of this section.
- [ ] Yes. I've updated:
  - [ ] Examples
  - [ ] API references
  - [ ] Instructions

### Release Notes

#### Is this a user-facing change?

- [ ] No. You can skip the rest of this section.
- [ ] Yes. Give a description of this change to be included in the release notes for MLflow users.

<!-- Details in 1-2 sentences. You can just refer to another PR with a description if this PR is part of a larger change. -->

#### What component(s), interfaces, languages, and integrations does this PR affect?

Components

- [ ] `area/artifacts`: Artifact stores and artifact logging
- [ ] `area/build`: Build and test infrastructure for MLflow
- [ ] `area/deployments`: MLflow Deployments client APIs, server, and third-party Deployments integrations
- [ ] `area/docs`: MLflow documentation pages
- [ ] `area/examples`: Example code
- [ ] `area/model-registry`: Model Registry service, APIs, and the fluent client calls for Model Registry
- [ ] `area/models`: MLmodel format, model serialization/deserialization, flavors
- [ ] `area/recipes`: Recipes, Recipe APIs, Recipe configs, Recipe Templates
- [ ] `area/projects`: MLproject format, project running backends
- [ ] `area/scoring`: MLflow Model server, model deployment tools, Spark UDFs
- [ ] `area/server-infra`: MLflow Tracking server backend
- [x] `area/tracking`: Tracking Service, tracking client APIs, autologging

Interface

- [ ] `area/uiux`: Front-end, user experience, plotting, JavaScript, JavaScript dev server
- [ ] `area/docker`: Docker use across MLflow's components, such as MLflow Projects and MLflow Models
- [ ] `area/sqlalchemy`: Use of SQLAlchemy in the Tracking Service or Model Registry
- [ ] `area/windows`: Windows support

Language

- [ ] `language/r`: R APIs and clients
- [ ] `language/java`: Java APIs and clients
- [ ] `language/new`: Proposals for new client languages

Integrations

- [ ] `integrations/azure`: Azure and Azure ML integrations
- [ ] `integrations/sagemaker`: SageMaker integrations
- [ ] `integrations/databricks`: Databricks integrations

<!--
Insert an empty named anchor here to allow jumping to this section with a fragment URL
(e.g. https://github.com/mlflow/mlflow/pull/123#user-content-release-note-category).
Note that GitHub prefixes anchor names in markdown with "user-content-".
-->

<a name="release-note-category"></a>

#### How should the PR be classified in the release notes? Choose one:

- [ ] `rn/none` - No description will be included. The PR will be mentioned only by the PR number in the "Small Bugfixes and Documentation Updates" section
- [ ] `rn/breaking-change` - The PR will be mentioned in the "Breaking Changes" section
- [x] `rn/feature` - A new user-facing feature worth mentioning in the release notes
- [ ] `rn/bug-fix` - A user-facing bug fix worth mentioning in the release notes
- [ ] `rn/documentation` - A user-facing documentation change worth mentioning in the release notes

#### Should this PR be included in the next patch release?

`Yes` should be selected for bug fixes, documentation updates, and other small changes. `No` should be selected for new features and larger changes. If you're unsure about the release classification of this PR, leave this unchecked to let the maintainers decide.

<details>
<summary>What is a minor/patch release?</summary>

- Minor release: a release that increments the second part of the version number (e.g., 1.2.0 -> 1.3.0).
  Bug fixes, doc updates and new features usually go into minor releases.
- Patch release: a release that increments the third part of the version number (e.g., 1.2.0 -> 1.2.1).
  Bug fixes and doc updates usually go into patch releases.

</details>

<!-- patch -->

- [ ] Yes (this PR will be cherry-picked and included in the next patch release)
- [x] No (this PR will be included in the next minor release)



Diff:
diff --git a/mlflow/genai/scorers/__init__.py b/mlflow/genai/scorers/__init__.py
index 80437bc19df5e..9deffdb7862e8 100644
--- a/mlflow/genai/scorers/__init__.py
+++ b/mlflow/genai/scorers/__init__.py
@@ -1,3 +1,27 @@
 from mlflow.genai.scorers.base import BuiltInScorer, Scorer, scorer
+from mlflow.genai.scorers.builtin_scorers import (
+    chunk_relevance,
+    context_sufficiency,
+    document_recall,
+    global_guideline_adherence,
+    groundedness,
+    guideline_adherence,
+    rag_scorers,
+    relevance_to_query,
+    safety,
+)
 
-__all__ = ["BuiltInScorer", "Scorer", "scorer"]
+__all__ = [
+    "BuiltInScorer",
+    "Scorer",
+    "scorer",
+    "chunk_relevance",
+    "context_sufficiency",
+    "document_recall",
+    "global_guideline_adherence",
+    "groundedness",
+    "guideline_adherence",
+    "rag_scorers",
+    "relevance_to_query",
+    "safety",
+]
diff --git a/mlflow/genai/scorers/builtin_scorers.py b/mlflow/genai/scorers/builtin_scorers.py
new file mode 100644
index 0000000000000..e573caee40311
--- /dev/null
+++ b/mlflow/genai/scorers/builtin_scorers.py
@@ -0,0 +1,134 @@
+from copy import deepcopy
+
+from mlflow.genai.scorers import BuiltInScorer
+
+
+class _BaseBuiltInScorer(BuiltInScorer):
+    """
+    Base class for built-in scorers that share a common implementation. All built-in scorers should
+    inherit from this class.
+    """
+
+    def update_evaluation_config(self, evaluation_config) -> dict:
+        config = deepcopy(evaluation_config)
+        metrics = config.setdefault("databricks-agents", {}).setdefault("metrics", [])
+        if self.name not in metrics:
+            metrics.append(self.name)
+        return config
+
+
+# === Builtin Scorers ===
+class _ChunkRelevance(_BaseBuiltInScorer):
+    name: str = "chunk_relevance"
+
+
+def chunk_relevance():
+    """
+    Chunk relevance measures whether each chunk is relevant to the input request.
+    """
+    return _ChunkRelevance()
+
+
+class _ContextSufficiency(_BaseBuiltInScorer):
+    name: str = "context_sufficiency"
+
+
+def context_sufficiency():
+    """
+    Context sufficiency evaluates whether the retrieved documents provide all necessary
+    information to generate the expected response.
+    """
+    return _ContextSufficiency()
+
+
+class _DocumentRecall(_BaseBuiltInScorer):
+    name: str = "document_recall"
+
+
+def document_recall():
+    """
+    Document recall measures the proportion of ground truth relevant documents that were
+    retrieved compared to the total number of relevant documents in ground truth.
+    """
+    return _DocumentRecall()
+
+
+class _GlobalGuidelineAdherence(_BaseBuiltInScorer):
+    name: str = "global_guideline_adherence"
+    global_guideline: str
+
+    def update_evaluation_config(self, evaluation_config) -> dict:
+        config = super().update_evaluation_config(evaluation_config)
+        config["databricks-agents"]["global_guideline"] = self.global_guideline
+        return config
+
+
+def global_guideline_adherence(guideline: str):
+    """
+    Guideline adherence evaluates whether the agent’s response follows specific constraints or
+    instructions provided in the guidelines.
+    """
+    return _GlobalGuidelineAdherence(global_guideline=guideline)
+
+
+class _Groundedness(_BaseBuiltInScorer):
+    name: str = "groundedness"
+
+
+def groundedness():
+    """
+    Groundedness assesses whether the agent’s response is aligned with the information provided
+    in the retrieved context.
+    """
+    return _Groundedness()
+
+
+class _GuidelineAdherence(_BaseBuiltInScorer):
+    name: str = "guideline_adherence"
+
+
+def guideline_adherence():
+    """
+    Guideline adherence evaluates whether the agent’s response follows specific constraints
+    or instructions provided in the guidelines.
+    """
+    return _GuidelineAdherence()
+
+
+class _RelevanceToQuery(_BaseBuiltInScorer):
+    name: str = "relevance_to_query"
+
+
+def relevance_to_query():
+    """
+    Relevance ensures that the agent’s response directly addresses the user’s input without
+    deviating into unrelated topics.
+    """
+    return _RelevanceToQuery()
+
+
+class _Safety(_BaseBuiltInScorer):
+    name: str = "safety"
+
+
+def safety():
+    """
+    Safety ensures that the agent’s responses do not contain harmful, offensive, or toxic content.
+    """
+    return _Safety()
+
+
+# === Shorthand for all builtin RAG scorers ===
+def rag_scorers() -> list[BuiltInScorer]:
+    """
+    Returns a list of built-in scorers for evaluating RAG models. Contains scorers
+    chunk_relevance, context_sufficiency, document_recall, global_guideline_adherence,
+    groundedness, and relevance_to_query.
+    """
+    return [
+        chunk_relevance(),
+        context_sufficiency(),
+        document_recall(),
+        groundedness(),
+        relevance_to_query(),
+    ]
diff --git a/tests/genai/scorers/test_builtin_scorers.py b/tests/genai/scorers/test_builtin_scorers.py
new file mode 100644
index 0000000000000..16082a8f8b285
--- /dev/null
+++ b/tests/genai/scorers/test_builtin_scorers.py
@@ -0,0 +1,66 @@
+import pytest
+
+from mlflow.genai.scorers import (
+    chunk_relevance,
+    context_sufficiency,
+    document_recall,
+    global_guideline_adherence,
+    groundedness,
+    guideline_adherence,
+    rag_scorers,
+    relevance_to_query,
+    safety,
+)
+
+
+def normalize_config(config):
+    config = config.copy()
+    metrics = config.get("databricks-agents", {}).get("metrics", [])
+    config.setdefault("databricks-agents", {})["metrics"] = sorted(metrics)
+    return config
+
+
+ALL_SCORERS = [
+    chunk_relevance(),
+    context_sufficiency(),
+    document_recall(),
+    global_guideline_adherence("Be polite"),
+    groundedness(),
+    guideline_adherence(),
+    relevance_to_query(),
+    safety(),
+]
+
+
+@pytest.mark.parametrize(
+    "scorers",
+    [
+        ALL_SCORERS,
+        [*ALL_SCORERS] + [*ALL_SCORERS],  # duplicate scorers
+        rag_scorers() + [global_guideline_adherence("Be polite"), guideline_adherence(), safety()],
+        [*rag_scorers()]
+        + [global_guideline_adherence("Be polite"), guideline_adherence(), safety()],
+    ],
+)
+def test_scorers_and_rag_scorers_config(scorers):
+    evaluation_config = {}
+    for scorer in scorers:
+        evaluation_config = scorer.update_evaluation_config(evaluation_config)
+
+    expected = {
+        "databricks-agents": {
+            "metrics": [
+                "chunk_relevance",
+                "context_sufficiency",
+                "document_recall",
+                "global_guideline_adherence",
+                "groundedness",
+                "guideline_adherence",
+                "relevance_to_query",
+                "safety",
+            ],
+            "global_guideline": "Be polite",
+        }
+    }
+
+    assert normalize_config(evaluation_config) == normalize_config(expected)


Reviews:
user: login: copilot-pull-request-reviewer[bot]
body: ## Pull Request Overview

This PR exposes built‐in scorers for mlflow.genai and updates several components to support new scorer functionality and tracing improvements. Key changes include new test cases for built‐in evaluators, the addition of new scorer implementations (e.g. global_guideline_adherence and a dummy built-in scorer), and slight modifications to tracing utilities.

### Reviewed Changes

Copilot reviewed 13 out of 13 changed files in this pull request and generated 1 comment.

<details>
<summary>Show a summary per file</summary>

| File | Description |
| ---- | ----------- |
| tests/tracing/test_fluent.py | Added tests for tracing behavior and introduced functions for OpenAI and dummy predict functions. |
| tests/genai/scorers/test_builtin_evaluators.py | Introduced tests to ensure evaluation configuration equivalence for all scorers and rag evaluators. |
| mlflow/tracing/fluent.py | Added an is_traced() function which patches _get_tracer to determine if tracing is activated. |
| mlflow/ml-package-versions.yml | Minor update to include spacing adjustments. |
| mlflow/genai/scorers/global_guideline_adherence.py | Added and exposed the global_guideline_adherence scorer implementation. |
| mlflow/genai/scorers/dummy_builtin_scorer.py | Introduced a dummy built-in scorer; review required for safe handling of evaluation configuration keys. |
| mlflow/genai/scorers/builtin_scorers.py | Added built-in scorers with clear naming and scorer implementations. |
| mlflow/genai/scorers/base.py | Defined a base scorer and scoring decorator ensuring proper input/output validation. |
| mlflow/genai/scorers/__init__.py | Updated module exports to include all scorer implementations. |
| mlflow/genai/evaluation/utils.py | Added functions to support legacy evaluation set conversion and scorer to legacy metric conversion. |
| mlflow/genai/evaluation/base.py | Updated the evaluate function to incorporate built-in scorers and enhanced tracing checks. |
| mlflow/genai/evaluation/__init__.py | Exposed the evaluate function. |
| mlflow/genai/__init__.py | Updated module export to expose evaluate. |
</details>





state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-18T01:17:48Z
user: login: copilot-pull-request-reviewer[bot]
body: ## Pull Request Overview

This PR exposes built-in scorers for mlflow.genai and adds corresponding tests. Key changes include:
- Introducing several built-in scorers in mlflow/genai/scorers/builtin_scorers.py.
- Updating the __init__.py export list to include the newly added scorers.
- Adding tests in tests/genai/scorers/test_builtin_scorers.py to verify the scorers’ evaluation configuration.

### Reviewed Changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 1 comment.

| File                                                  | Description                                                                          |
| ----------------------------------------------------- | ------------------------------------------------------------------------------------ |
| tests/genai/scorers/test_builtin_scorers.py           | Added tests to validate the evaluation configuration produced by the scorers.        |
| mlflow/genai/scorers/builtin_scorers.py                 | Introduced various built-in scorer classes and a shorthand factory (rag_evaluators).  |
| mlflow/genai/scorers/__init__.py                        | Updated exports (__all__) to include the new built-in scorers.                       |
| mlflow/genai/evaluation/base.py                        | Adjusted the return type annotation of the evaluate function.                        |




state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-18T16:30:24Z
user: login: B-Step62
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-04-21T08:26:59Z
user: login: copilot-pull-request-reviewer[bot]
body: ## Pull Request Overview

This PR exposes built-in scorers for mlflow.genai by adding implementations, corresponding tests, and updating the module exports.  
- Added implementations for various builtin scorers in mlflow/genai/scorers/builtin_scorers.py  
- Introduced tests for scorer configuration in tests/genai/scorers/test_builtin_scorers.py  
- Updated mlflow/genai/scorers/__init__.py to export the new scorers

### Reviewed Changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated no comments.

| File | Description |
| ---- | ----------- |
| tests/genai/scorers/test_builtin_scorers.py | Adds tests to ensure scorers correctly update the evaluation configuration without duplicates. |
| mlflow/genai/scorers/builtin_scorers.py | Implements various built-in scorers with shared behavior via a base scorer class and a dedicated method for RAG scorers. |
| mlflow/genai/scorers/__init__.py | Updates the public API to include the newly implemented scorers. |




state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-21T17:52:52Z
user: login: B-Step62
body: LGTM w/ removal of global guideline adherence from the rag scorers
state: APPROVED
author_association: COLLABORATOR
submitted_at: 2025-04-22T05:56:32Z
user: login: B-Step62
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-04-22T05:57:06Z
user: login: B-Step62
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-04-22T05:57:15Z
user: login: B-Step62
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-04-22T05:58:18Z
user: login: B-Step62
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-04-22T09:13:02Z
user: login: B-Step62
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-04-22T14:27:22Z

Review comments:
diff_hunk: @@ -0,0 +1,11 @@
+import copy
+
+from mlflow.genai.scorers import BuiltInScorer
+
+
+# TODO: ML-52304 example builtin scorer implementation
+class DummyBuiltInScorer(BuiltInScorer):
+    def update_evaluation_config(evaluation_config) -> dict:
+        updated_config = copy.deepcopy(evaluation_config)
+        updated_config["databricks-agents"]["metrics"].append("dummy_metric")
path: mlflow/genai/scorers/dummy_builtin_scorer.py
user: login: Copilot
body: The update_evaluation_config implementation in DummyBuiltInScorer assumes that the 'metrics' key already exists under 'databricks-agents'. Consider initializing it using setdefault (e.g., updated_config.setdefault("databricks-agents", {}).setdefault("metrics", [])) to avoid potential KeyError.
```suggestion
        updated_config.setdefault("databricks-agents", {}).setdefault("metrics", []).append("dummy_metric")
```
created_at: 2025-04-18T01:17:47Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,94 @@
+from copy import deepcopy
+from typing import List
+
+from mlflow.genai.scorers import BuiltInScorer
+
+
+class _BaseBuiltInScorer(BuiltInScorer):
+    """
+    Base class for built-in scorers that share a common implementation. All built-in scorers should
+    inherit from this class.
+    """
+
+    def update_evaluation_config(self, evaluation_config) -> dict:
+        config = deepcopy(evaluation_config)
+        config.setdefault("databricks-agents", {}).setdefault("metrics", []).append(self.name)
+        return config
+
path: mlflow/genai/scorers/builtin_scorers.py
user: login: Copilot
body: The update_evaluation_config method appends the scorer name to the metrics list without checking for duplicates. If update_evaluation_config is called multiple times for the same scorer, this could result in duplicate metric entries; consider checking whether self.name is already present before appending.
```suggestion
        metrics = config.setdefault("databricks-agents", {}).setdefault("metrics", [])
        if self.name not in metrics:
            metrics.append(self.name)
```
created_at: 2025-04-18T16:30:24Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,94 @@
+from copy import deepcopy
+from typing import List
+
+from mlflow.genai.scorers import BuiltInScorer
+
+
+class _BaseBuiltInScorer(BuiltInScorer):
+    """
+    Base class for built-in scorers that share a common implementation. All built-in scorers should
+    inherit from this class.
+    """
+
+    def update_evaluation_config(self, evaluation_config) -> dict:
+        config = deepcopy(evaluation_config)
+        config.setdefault("databricks-agents", {}).setdefault("metrics", []).append(self.name)
+        return config
+
path: mlflow/genai/scorers/builtin_scorers.py
user: login: B-Step62
body: nit: Let's follow this recommendation - there can be a case where duplicated judge names are provided by users.
created_at: 2025-04-21T08:21:17Z
author_association: COLLABORATOR
diff_hunk: @@ -0,0 +1,93 @@
+from copy import deepcopy
+
+from mlflow.genai.scorers import BuiltInScorer
+
+
+class _BaseBuiltInScorer(BuiltInScorer):
+    """
+    Base class for built-in scorers that share a common implementation. All built-in scorers should
+    inherit from this class.
+    """
+
+    def update_evaluation_config(self, evaluation_config) -> dict:
+        config = deepcopy(evaluation_config)
+        config.setdefault("databricks-agents", {}).setdefault("metrics", []).append(self.name)
+        return config
+
+
+# === Builtin Scorers ===
+class _ChunkRelevance(_BaseBuiltInScorer):
+    name: str = "chunk_relevance"
+
+
+chunk_relevance = _ChunkRelevance()
path: mlflow/genai/scorers/builtin_scorers.py
user: login: B-Step62
body: Can we define `chunk_relevance` as function? To reasons:
1. We will likely add a few configurations for built-in judge, e.g., aggregations to compute, which will looks like `chunk_relevance(aggregations=["mean", "p99"])`. Same applies to others.
2. We can add docstring to auto-document them in API docs.
created_at: 2025-04-21T08:23:10Z
author_association: COLLABORATOR
diff_hunk: @@ -0,0 +1,93 @@
+from copy import deepcopy
+
+from mlflow.genai.scorers import BuiltInScorer
+
+
+class _BaseBuiltInScorer(BuiltInScorer):
+    """
+    Base class for built-in scorers that share a common implementation. All built-in scorers should
+    inherit from this class.
+    """
+
+    def update_evaluation_config(self, evaluation_config) -> dict:
+        config = deepcopy(evaluation_config)
+        config.setdefault("databricks-agents", {}).setdefault("metrics", []).append(self.name)
+        return config
+
+
+# === Builtin Scorers ===
+class _ChunkRelevance(_BaseBuiltInScorer):
+    name: str = "chunk_relevance"
+
+
+chunk_relevance = _ChunkRelevance()
+
+
+class _ContextSufficiency(_BaseBuiltInScorer):
+    name: str = "context_sufficiency"
+
+
+context_sufficiency = _ContextSufficiency()
+
+
+class _DocumentRecall(_BaseBuiltInScorer):
+    name: str = "document_recall"
+
+
+document_recall = _DocumentRecall()
+
+
+class _GlobalGuidelineAdherence(_BaseBuiltInScorer):
+    name: str = "global_guideline_adherence"
+    global_guideline: str
+
+    def update_evaluation_config(self, evaluation_config) -> dict:
+        config = super().update_evaluation_config(evaluation_config)
+        config["databricks-agents"]["global_guideline"] = self.global_guideline
+        return config
+
+
+def global_guideline_adherence(guideline: str):
+    return _GlobalGuidelineAdherence(global_guideline=guideline)
+
+
+class _Groundedness(_BaseBuiltInScorer):
+    name: str = "groundedness"
+
+
+groundedness = _Groundedness()
+
+
+class _GuidelineAdherence(_BaseBuiltInScorer):
+    name: str = "guideline_adherence"
+
+
+guideline_adherence = _GuidelineAdherence()
+
+
+class _RelevanceToQuery(_BaseBuiltInScorer):
+    name: str = "relevance_to_query"
+
+
+relevance_to_query = _RelevanceToQuery()
+
+
+class _Safety(_BaseBuiltInScorer):
+    name: str = "safety"
+
+
+safety = _Safety()
+
+
+# === Shorthand for all builtin scorers ===
+def rag_evaluators(global_guideline: str) -> list[BuiltInScorer]:
path: mlflow/genai/scorers/builtin_scorers.py
user: login: B-Step62
body: ```suggestion
def rag_scorers(global_guideline: str) -> list[BuiltInScorer]:
```

Also we want to keep `rag_scorers` only includes RAG related metrics, i.e. can we move guideline adherence and safety outside the list?
created_at: 2025-04-21T08:25:17Z
author_association: COLLABORATOR
diff_hunk: @@ -0,0 +1,135 @@
+from copy import deepcopy
+
+from mlflow.genai.scorers import BuiltInScorer
+
+
+class _BaseBuiltInScorer(BuiltInScorer):
+    """
+    Base class for built-in scorers that share a common implementation. All built-in scorers should
+    inherit from this class.
+    """
+
+    def update_evaluation_config(self, evaluation_config) -> dict:
+        config = deepcopy(evaluation_config)
+        metrics = config.setdefault("databricks-agents", {}).setdefault("metrics", [])
+        if self.name not in metrics:
+            metrics.append(self.name)
+        return config
+
+
+# === Builtin Scorers ===
+class _ChunkRelevance(_BaseBuiltInScorer):
+    name: str = "chunk_relevance"
+
+
+def chunk_relevance():
+    """
+    Chunk relevance measures whether each chunk is relevant to the input request.
+    """
+    return _ChunkRelevance()
+
+
+class _ContextSufficiency(_BaseBuiltInScorer):
+    name: str = "context_sufficiency"
+
+
+def context_sufficiency():
+    """
+    Context sufficiency evaluates whether the retrieved documents provide all necessary
+    information to generate the expected response.
+    """
+    return _ContextSufficiency()
+
+
+class _DocumentRecall(_BaseBuiltInScorer):
+    name: str = "document_recall"
+
+
+def document_recall():
+    """
+    Document recall measures the proportion of ground truth relevant documents that were
+    retrieved compared to the total number of relevant documents in ground truth.
+    """
+    return _DocumentRecall()
+
+
+class _GlobalGuidelineAdherence(_BaseBuiltInScorer):
+    name: str = "global_guideline_adherence"
+    global_guideline: str
+
+    def update_evaluation_config(self, evaluation_config) -> dict:
+        config = super().update_evaluation_config(evaluation_config)
+        config["databricks-agents"]["global_guideline"] = self.global_guideline
+        return config
+
+
+def global_guideline_adherence(guideline: str):
+    """
+    Guideline adherence evaluates whether the agent’s response follows specific constraints or
+    instructions provided in the guidelines.
+    """
+    return _GlobalGuidelineAdherence(global_guideline=guideline)
+
+
+class _Groundedness(_BaseBuiltInScorer):
+    name: str = "groundedness"
+
+
+def groundedness():
+    """
+    Groundedness assesses whether the agent’s response is aligned with the information provided
+    in the retrieved context.
+    """
+    return _Groundedness()
+
+
+class _GuidelineAdherence(_BaseBuiltInScorer):
+    name: str = "guideline_adherence"
+
+
+def guideline_adherence():
+    """
+    Guideline adherence evaluates whether the agent’s response follows specific constraints
+    or instructions provided in the guidelines.
+    """
+    return _GuidelineAdherence()
+
+
+class _RelevanceToQuery(_BaseBuiltInScorer):
+    name: str = "relevance_to_query"
+
+
+def relevance_to_query():
+    """
+    Relevance ensures that the agent’s response directly addresses the user’s input without
+    deviating into unrelated topics.
+    """
+    return _RelevanceToQuery()
+
+
+class _Safety(_BaseBuiltInScorer):
+    name: str = "safety"
+
+
+def safety():
+    """
+    Safety ensures that the agent’s responses do not contain harmful, offensive, or toxic content.
+    """
+    return _Safety()
+
+
+# === Shorthand for all builtin RAG scorers ===
+def rag_scorers(global_guideline: str) -> list[BuiltInScorer]:
path: mlflow/genai/scorers/builtin_scorers.py
user: login: B-Step62
body: ```suggestion
def rag_scorers() -> list[BuiltInScorer]:
```
created_at: 2025-04-22T05:57:05Z
author_association: COLLABORATOR
diff_hunk: @@ -0,0 +1,135 @@
+from copy import deepcopy
+
+from mlflow.genai.scorers import BuiltInScorer
+
+
+class _BaseBuiltInScorer(BuiltInScorer):
+    """
+    Base class for built-in scorers that share a common implementation. All built-in scorers should
+    inherit from this class.
+    """
+
+    def update_evaluation_config(self, evaluation_config) -> dict:
+        config = deepcopy(evaluation_config)
+        metrics = config.setdefault("databricks-agents", {}).setdefault("metrics", [])
+        if self.name not in metrics:
+            metrics.append(self.name)
+        return config
+
+
+# === Builtin Scorers ===
+class _ChunkRelevance(_BaseBuiltInScorer):
+    name: str = "chunk_relevance"
+
+
+def chunk_relevance():
+    """
+    Chunk relevance measures whether each chunk is relevant to the input request.
+    """
+    return _ChunkRelevance()
+
+
+class _ContextSufficiency(_BaseBuiltInScorer):
+    name: str = "context_sufficiency"
+
+
+def context_sufficiency():
+    """
+    Context sufficiency evaluates whether the retrieved documents provide all necessary
+    information to generate the expected response.
+    """
+    return _ContextSufficiency()
+
+
+class _DocumentRecall(_BaseBuiltInScorer):
+    name: str = "document_recall"
+
+
+def document_recall():
+    """
+    Document recall measures the proportion of ground truth relevant documents that were
+    retrieved compared to the total number of relevant documents in ground truth.
+    """
+    return _DocumentRecall()
+
+
+class _GlobalGuidelineAdherence(_BaseBuiltInScorer):
+    name: str = "global_guideline_adherence"
+    global_guideline: str
+
+    def update_evaluation_config(self, evaluation_config) -> dict:
+        config = super().update_evaluation_config(evaluation_config)
+        config["databricks-agents"]["global_guideline"] = self.global_guideline
+        return config
+
+
+def global_guideline_adherence(guideline: str):
+    """
+    Guideline adherence evaluates whether the agent’s response follows specific constraints or
+    instructions provided in the guidelines.
+    """
+    return _GlobalGuidelineAdherence(global_guideline=guideline)
+
+
+class _Groundedness(_BaseBuiltInScorer):
+    name: str = "groundedness"
+
+
+def groundedness():
+    """
+    Groundedness assesses whether the agent’s response is aligned with the information provided
+    in the retrieved context.
+    """
+    return _Groundedness()
+
+
+class _GuidelineAdherence(_BaseBuiltInScorer):
+    name: str = "guideline_adherence"
+
+
+def guideline_adherence():
+    """
+    Guideline adherence evaluates whether the agent’s response follows specific constraints
+    or instructions provided in the guidelines.
+    """
+    return _GuidelineAdherence()
+
+
+class _RelevanceToQuery(_BaseBuiltInScorer):
+    name: str = "relevance_to_query"
+
+
+def relevance_to_query():
+    """
+    Relevance ensures that the agent’s response directly addresses the user’s input without
+    deviating into unrelated topics.
+    """
+    return _RelevanceToQuery()
+
+
+class _Safety(_BaseBuiltInScorer):
+    name: str = "safety"
+
+
+def safety():
+    """
+    Safety ensures that the agent’s responses do not contain harmful, offensive, or toxic content.
+    """
+    return _Safety()
+
+
+# === Shorthand for all builtin RAG scorers ===
+def rag_scorers(global_guideline: str) -> list[BuiltInScorer]:
+    """
+    Returns a list of built-in scorers for evaluating RAG models. Contains scorers
+    chunk_relevance, context_sufficiency, document_recall, global_guideline_adherence,
+    groundedness, and relevance_to_query.
+    """
+    return [
+        chunk_relevance(),
+        context_sufficiency(),
+        document_recall(),
+        global_guideline_adherence(global_guideline),
path: mlflow/genai/scorers/builtin_scorers.py
user: login: B-Step62
body: ```suggestion
```
created_at: 2025-04-22T05:57:15Z
author_association: COLLABORATOR
diff_hunk: @@ -0,0 +1,65 @@
+import pytest
+
+from mlflow.genai.scorers import (
+    chunk_relevance,
+    context_sufficiency,
+    document_recall,
+    global_guideline_adherence,
+    groundedness,
+    guideline_adherence,
+    rag_scorers,
+    relevance_to_query,
+    safety,
+)
+
+
+def normalize_config(config):
+    config = config.copy()
+    metrics = config.get("databricks-agents", {}).get("metrics", [])
+    config.setdefault("databricks-agents", {})["metrics"] = sorted(metrics)
+    return config
+
+
+ALL_SCORERS = [
+    chunk_relevance(),
+    context_sufficiency(),
+    document_recall(),
+    global_guideline_adherence("Be polite"),
+    groundedness(),
+    guideline_adherence(),
+    relevance_to_query(),
+    safety(),
+]
+
+
+@pytest.mark.parametrize(
+    "scorers",
+    [
+        ALL_SCORERS,
+        [*ALL_SCORERS] + [*ALL_SCORERS],  # duplicate scorers
+        rag_scorers(global_guideline="Be polite") + [guideline_adherence(), safety()],
+        [*rag_scorers(global_guideline="Be polite")] + [guideline_adherence(), safety()],
path: tests/genai/scorers/test_builtin_scorers.py
user: login: B-Step62
body: ```suggestion
        rag_scorers() + [global_guideline_adherence("By polite"), guideline_adherence(), safety()],
        [*rag_scorers()] + [global_guideline_adherence("By polite"), guideline_adherence(), safety()],
```
created_at: 2025-04-22T05:58:18Z
author_association: COLLABORATOR
diff_hunk: @@ -0,0 +1,65 @@
+import pytest
+
+from mlflow.genai.scorers import (
+    chunk_relevance,
+    context_sufficiency,
+    document_recall,
+    global_guideline_adherence,
+    groundedness,
+    guideline_adherence,
+    rag_scorers,
+    relevance_to_query,
+    safety,
+)
+
+
+def normalize_config(config):
+    config = config.copy()
+    metrics = config.get("databricks-agents", {}).get("metrics", [])
+    config.setdefault("databricks-agents", {})["metrics"] = sorted(metrics)
+    return config
+
+
+ALL_SCORERS = [
+    chunk_relevance(),
+    context_sufficiency(),
+    document_recall(),
+    global_guideline_adherence("Be polite"),
+    groundedness(),
+    guideline_adherence(),
+    relevance_to_query(),
+    safety(),
+]
+
+
+@pytest.mark.parametrize(
+    "scorers",
+    [
+        ALL_SCORERS,
+        [*ALL_SCORERS] + [*ALL_SCORERS],  # duplicate scorers
+        rag_scorers() + [global_guideline_adherence("By polite"), guideline_adherence(), safety()],
+        [*rag_scorers()] + [global_guideline_adherence("By polite"), guideline_adherence(), safety()],
path: tests/genai/scorers/test_builtin_scorers.py
user: login: B-Step62
body: ```suggestion
        [*rag_scorers()]
        + [global_guideline_adherence("By polite"), guideline_adherence(), safety()],
```
created_at: 2025-04-22T09:13:02Z
author_association: COLLABORATOR
diff_hunk: @@ -0,0 +1,66 @@
+import pytest
+
+from mlflow.genai.scorers import (
+    chunk_relevance,
+    context_sufficiency,
+    document_recall,
+    global_guideline_adherence,
+    groundedness,
+    guideline_adherence,
+    rag_scorers,
+    relevance_to_query,
+    safety,
+)
+
+
+def normalize_config(config):
+    config = config.copy()
+    metrics = config.get("databricks-agents", {}).get("metrics", [])
+    config.setdefault("databricks-agents", {})["metrics"] = sorted(metrics)
+    return config
+
+
+ALL_SCORERS = [
+    chunk_relevance(),
+    context_sufficiency(),
+    document_recall(),
+    global_guideline_adherence("Be polite"),
+    groundedness(),
+    guideline_adherence(),
+    relevance_to_query(),
+    safety(),
+]
+
+
+@pytest.mark.parametrize(
+    "scorers",
+    [
+        ALL_SCORERS,
+        [*ALL_SCORERS] + [*ALL_SCORERS],  # duplicate scorers
+        rag_scorers() + [global_guideline_adherence("By polite"), guideline_adherence(), safety()],
+        [*rag_scorers()]
+        + [global_guideline_adherence("By polite"), guideline_adherence(), safety()],
path: tests/genai/scorers/test_builtin_scorers.py
user: login: B-Step62
body: ```suggestion
        rag_scorers() + [global_guideline_adherence("Be polite"), guideline_adherence(), safety()],
        [*rag_scorers()]
        + [global_guideline_adherence("Be polite"), guideline_adherence(), safety()],
```
created_at: 2025-04-22T14:27:22Z
author_association: COLLABORATOR

PR comments:
user: login: github-actions[bot]
created_at: 2025-04-18T01:17:56Z
body: <!-- documentation preview -->


Documentation preview for 338379a60b544debb7499b5280ee6433a1ba1553 will be available when [this CircleCI job](https://circleci.com/gh/mlflow/mlflow/154885)
completes successfully. You may encounter a `{"message":"not found"}` error when reloading
a page. If so, add `/index.html` to the URL.

- [Top page](https://output.circle-artifacts.com/output/job/162965dd-3858-40a1-8e94-698baa674ebc/artifacts/0/docs/build/latest/index.html)
- [Changed pages](https://output.circle-artifacts.com/output/job/162965dd-3858-40a1-8e94-698baa674ebc/artifacts/0/docs/build/latest/diff.html) (⚠️ only MDX file changes are detected ⚠️)

<details>
<summary>More info</summary>

- Ignore this comment if this PR does not change the documentation.
- It takes a few minutes for the preview to be available.
- The preview is updated when a new commit is pushed to this PR.
- This comment was created by https://github.com/mlflow/mlflow/actions/runs/14597314264.

</details>

author_association: CONTRIBUTOR
user: login: artjen
created_at: 2025-04-18T17:26:29Z
body: /autoformat
author_association: COLLABORATOR