Repository: mlflow/mlflow
Pull request #15282: [ML-52305] mlflow.genai.evaluate() PoC commit
Author: login: dhruyads
Labels: name: rn/feature
description: Mention under Features in Changelogs.
name: area/tracking
description: Tracking service, tracking client APIs, autologging
Created: 2025-04-10T16:30:31Z | Merged: 2025-04-18T07:14:22Z

### Related Issues/PRs

<!-- Uncomment 'Resolve' if this PR can close the linked items. -->
<!-- Resolve --> #xxx

### What changes are proposed in this pull request?

<!-- Please fill in changes proposed in this PR. -->
Proof of concept for adding genai evaluate function.

### How is this PR tested?

- [x] Existing unit/integration tests
- [x] New unit/integration tests
- [ ] Manual tests

<!-- Attach code, screenshot, video used for manual testing here. -->

### Does this PR require documentation update?

- [ ] No. You can skip the rest of this section.
- [x] Yes. I've updated:
  - [ ] Examples
  - [ ] API references
  - [x] Instructions
Will update in follow up tasks.
### Release Notes

#### Is this a user-facing change?

- [ ] No. You can skip the rest of this section.
- [x] Yes. Give a description of this change to be included in the release notes for MLflow users.

Include new mlflow.genai.evaluate() api.
<!-- Details in 1-2 sentences. You can just refer to another PR with a description if this PR is part of a larger change. -->

#### What component(s), interfaces, languages, and integrations does this PR affect?

Components

- [ ] `area/artifacts`: Artifact stores and artifact logging
- [ ] `area/build`: Build and test infrastructure for MLflow
- [ ] `area/deployments`: MLflow Deployments client APIs, server, and third-party Deployments integrations
- [ ] `area/docs`: MLflow documentation pages
- [ ] `area/examples`: Example code
- [ ] `area/model-registry`: Model Registry service, APIs, and the fluent client calls for Model Registry
- [ ] `area/models`: MLmodel format, model serialization/deserialization, flavors
- [ ] `area/recipes`: Recipes, Recipe APIs, Recipe configs, Recipe Templates
- [ ] `area/projects`: MLproject format, project running backends
- [ ] `area/scoring`: MLflow Model server, model deployment tools, Spark UDFs
- [ ] `area/server-infra`: MLflow Tracking server backend
- [x] `area/tracking`: Tracking Service, tracking client APIs, autologging

Interface

- [ ] `area/uiux`: Front-end, user experience, plotting, JavaScript, JavaScript dev server
- [ ] `area/docker`: Docker use across MLflow's components, such as MLflow Projects and MLflow Models
- [ ] `area/sqlalchemy`: Use of SQLAlchemy in the Tracking Service or Model Registry
- [ ] `area/windows`: Windows support

Language

- [ ] `language/r`: R APIs and clients
- [ ] `language/java`: Java APIs and clients
- [ ] `language/new`: Proposals for new client languages

Integrations

- [ ] `integrations/azure`: Azure and Azure ML integrations
- [ ] `integrations/sagemaker`: SageMaker integrations
- [ ] `integrations/databricks`: Databricks integrations

<!--
Insert an empty named anchor here to allow jumping to this section with a fragment URL
(e.g. https://github.com/mlflow/mlflow/pull/123#user-content-release-note-category).
Note that GitHub prefixes anchor names in markdown with "user-content-".
-->

<a name="release-note-category"></a>

#### How should the PR be classified in the release notes? Choose one:

- [ ] `rn/none` - No description will be included. The PR will be mentioned only by the PR number in the "Small Bugfixes and Documentation Updates" section
- [ ] `rn/breaking-change` - The PR will be mentioned in the "Breaking Changes" section
- [x] `rn/feature` - A new user-facing feature worth mentioning in the release notes
- [ ] `rn/bug-fix` - A user-facing bug fix worth mentioning in the release notes
- [ ] `rn/documentation` - A user-facing documentation change worth mentioning in the release notes

#### Should this PR be included in the next patch release?

`Yes` should be selected for bug fixes, documentation updates, and other small changes. `No` should be selected for new features and larger changes. If you're unsure about the release classification of this PR, leave this unchecked to let the maintainers decide.

<details>
<summary>What is a minor/patch release?</summary>

- Minor release: a release that increments the second part of the version number (e.g., 1.2.0 -> 1.3.0).
  Bug fixes, doc updates and new features usually go into minor releases.
- Patch release: a release that increments the third part of the version number (e.g., 1.2.0 -> 1.2.1).
  Bug fixes and doc updates usually go into patch releases.

</details>

<!-- patch -->

- [ ] Yes (this PR will be cherry-picked and included in the next patch release)
- [x] No (this PR will be included in the next minor release)



Diff:
diff --git a/mlflow/genai/__init__.py b/mlflow/genai/__init__.py
new file mode 100644
index 0000000000000..9096b853480b9
--- /dev/null
+++ b/mlflow/genai/__init__.py
@@ -0,0 +1,3 @@
+from mlflow.genai.evaluation import evaluate
+
+__all__ = ["evaluate"]
diff --git a/mlflow/genai/evaluation/__init__.py b/mlflow/genai/evaluation/__init__.py
new file mode 100644
index 0000000000000..b3941a9214dff
--- /dev/null
+++ b/mlflow/genai/evaluation/__init__.py
@@ -0,0 +1,3 @@
+from mlflow.genai.evaluation.base import evaluate
+
+__all__ = ["evaluate"]
diff --git a/mlflow/genai/evaluation/base.py b/mlflow/genai/evaluation/base.py
new file mode 100644
index 0000000000000..8461f54b46b7c
--- /dev/null
+++ b/mlflow/genai/evaluation/base.py
@@ -0,0 +1,129 @@
+import logging
+from typing import Any, Callable, Optional
+
+from pyspark import sql as spark
+
+import mlflow
+from mlflow.data.evaluation_dataset import EvaluationDataset
+from mlflow.genai.evaluation.utils import (
+    _convert_scorer_to_legacy_metric,
+    _convert_to_legacy_eval_set,
+)
+from mlflow.genai.scorers import BuiltInScorer, Scorer
+from mlflow.tracing.utils import is_model_traced
+
+try:
+    # `pandas` is not required for `mlflow-skinny`.
+    import pandas as pd
+except ImportError:
+    pass
+
+logger = logging.getLogger(__name__)
+
+
+class EvaluationResult:
+    run_id: str
+    metrics: dict[str, float]
+
+
+def evaluate(
+    data: pd.DataFrame | spark.DataFrame | list[dict] | EvaluationDataset,
+    predict_fn: Optional[Callable[..., Any]] = None,
+    scorers: Optional[list[Scorer]] = None,
+    model_id: Optional[str] = None,
+) -> mlflow.genai.EvaluationResult:
+    """
+    TODO: updating docstring with real examples and API links
+    Args:
+        data: Dataset for the evaluation. It must be one of the following format:
+            * A EvaluationDataset entity
+            * Pandas DataFrame
+            * Spark DataFrame
+            * List of dictionary
+
+            If a dataframe is specified, it must contain the following schema:
+              - inputs (required): A column that contains a single input.
+              - outputs (optional): A column that contains a single output from the
+                   target model/app. If the predict_fn is provided, this is generated
+                   by MLflow so not required.
+              - expectations (optional): A column that contains a ground truth, or a
+                   dictionary of ground truths for individual output fields.
+              - traces (optional): A column that contains a single trace object
+                   corresponding to the prediction for the row. Only required when
+                   any of scorers requires a trace in order to compute
+                   assessments/metrics.
+
+            If a list of dictionary is passed, each dictionary should contain keys
+            following the above schema.
+
+        predict_fn: The target function to be evaluated. The specified function will be
+            executed for each row in the input dataset, and outputs will be used for
+            scoring.
+
+            The function must emit a single trace per call. If it doesn't, decorate
+            the function with @mlflow.trace decorator to ensure a trace to be emitted.
+
+        scorers: A list of Scorer objects that produces evaluation scores from
+            inputs, outputs, and other additional contexts. MLflow provides pre-defined
+            scorers, but you can also define custom ones.
+
+        model_id: Optional. Specify an ID of the model e.g. models:/my-model/1 to
+            associate the evaluation result with. There are several ways to associate
+            model with association.
+
+            1. Use the `model_id` parameters.
+            2. Use the mlflow.set_active_model() function to set model ID to global context.
+               ```python
+               mlflow.set_active_model(model_id="xyz")
+
+               mlflow.evaluate(data, ...)
+               ```
+    """
+    if mlflow.get_tracking_uri() != "databricks":
+        raise ValueError(
+            (
+                "The genai evaluation function is only supported on Databricks. ",
+                "Please set the tracking URI to Databricks.",
+            )
+        )
+
+    builtin_scorers = []
+    custom_scorers = []
+
+    for scorer in scorers:
+        if isinstance(scorer, BuiltInScorer):
+            builtin_scorers.append(scorer)
+        elif isinstance(scorer, Scorer):
+            custom_scorers.append(scorer)
+        else:
+            raise TypeError(
+                (
+                    f"Scorer {scorer} is not a valid scorer. Please use the @scorer decorator ",
+                    "to convert a function into a scorer or inherit from the Scorer class",
+                )
+            )
+
+    evaluation_config = {
+        "databricks-agents": {
+            "metrics": [],
+        }
+    }
+    for _scorer in builtin_scorers:
+        evaluation_config = _scorer.update_evaluation_config(evaluation_config)
+
+    extra_metrics = []
+    for _scorer in custom_scorers:
+        extra_metrics.append(_convert_scorer_to_legacy_metric(_scorer))
+
+    if not is_model_traced(predict_fn):
+        logger.info("Annotating predict_fn with tracing since it is not already traced.")
+        predict_fn = mlflow.trace(predict_fn)
+
+    mlflow.evaluate(
+        model=predict_fn,
+        # convert into a pandas dataframe with current evaluation set schema
+        data=_convert_to_legacy_eval_set(data),
+        evaluator_config=evaluation_config,
+        extra_metrics=extra_metrics,
+        model_type="databricks-agent",
+    )
diff --git a/mlflow/genai/evaluation/utils.py b/mlflow/genai/evaluation/utils.py
new file mode 100644
index 0000000000000..b57abbcdbc124
--- /dev/null
+++ b/mlflow/genai/evaluation/utils.py
@@ -0,0 +1,33 @@
+from databricks.agents.evals import metric
+from pyspark import sql as spark
+
+from mlflow.genai.scorers import Scorer
+
+try:
+    # `pandas` is not required for `mlflow-skinny`.
+    import pandas as pd
+except ImportError:
+    pass
+
+
+# TODO: ML-52299
+def _convert_to_legacy_eval_set(
+    data: pd.DataFrame | spark.DataFrame | list[dict], EvaluationDataset
+) -> dict:
+    """
+    Takes in different types of inputs and converts it into to the current eval-set schema that
+    Agent Evaluation takes in. The transformed schema should be accepted by mlflow.evaluate()
+    """
+    raise NotImplementedError("This function is not implemented yet.")
+
+
+# TODO: ML-52297
+def _convert_scorer_to_legacy_metric(scorer: Scorer):
+    """
+    Takes in a Scorer object and converts it into a legacy MLflow 2.x
+    Metric object.
+    """
+    # TODO: complete implementation
+    return metric(
+        eval_fn=scorer,
+    )
diff --git a/mlflow/genai/scorers/__init__.py b/mlflow/genai/scorers/__init__.py
new file mode 100644
index 0000000000000..80437bc19df5e
--- /dev/null
+++ b/mlflow/genai/scorers/__init__.py
@@ -0,0 +1,3 @@
+from mlflow.genai.scorers.base import BuiltInScorer, Scorer, scorer
+
+__all__ = ["BuiltInScorer", "Scorer", "scorer"]
diff --git a/mlflow/genai/scorers/base.py b/mlflow/genai/scorers/base.py
new file mode 100644
index 0000000000000..28c3a8fa7f8b1
--- /dev/null
+++ b/mlflow/genai/scorers/base.py
@@ -0,0 +1,103 @@
+import functools
+import inspect
+import types
+from typing import Callable, Literal, Optional, Union
+
+from pydantic import BaseModel
+
+from mlflow.evaluation import Assessment
+
+
+class Scorer(BaseModel):
+    name: str
+
+    def __call__(
+        self,
+        *,
+        inputs,
+        outputs=None,
+        expectations=None,
+        trace=None,
+    ) -> Union[float, bool, str, Assessment, list[Assessment]]:
+        """
+
+        Args:
+            inputs (required): A column that contains a single input.
+            outputs (optional): A column that contains a single output from the
+                target model/app. If the predict_fn is provided, this is generated
+                by MLflow so not required.
+            expectations (optional): A column that contains a ground truth, or a
+                dictionary of ground truths for individual output fields.
+            trace (optional): A column that contains a single trace object
+                corresponding to the prediction for the row. Only required when
+                any of scorers requires a trace in order to compute
+                assessments/metrics.
+        """
+        raise NotImplementedError("Please use an instance of BuiltInScorer")
+
+
+# TODO: ML-52304: Inherit the following class for every builtin scorer made available
+class BuiltInScorer(Scorer):
+    def update_evaluation_config(evaluation_config) -> dict:
+        """
+        The builtin scorer will take in an evaluation_config and return an updated version
+        of it as necessary to comply with the expected format for mlflow.evaluate().
+        More details about built-in judges can be found at
+        https://docs.databricks.com/aws/en/generative-ai/agent-evaluation/llm-judge-reference
+        """
+        raise NotImplementedError("Please use an instance of BuiltInScorer")
+
+
+def scorer(
+    func=None,
+    *,
+    name: Optional[str] = None,
+    aggregations: Optional[
+        list[Union[Literal["min", "max", "mean", "median", "variance", "p90", "p99"], Callable]]
+    ] = None,
+):
+    """
+    Syntactic sugar that ensures the decorated function has the correct parameters (with inputs
+    as required, outputs, expectations, trace as optional) and returns float | bool |
+    str | Assessment.
+    """
+
+    if func is None:
+        return functools.partial(scorer, name=name, aggregations=aggregations)
+
+    sig = inspect.signature(func)
+    params = sig.parameters
+
+    # Verify required param presence
+    if "inputs" not in params:
+        raise TypeError(
+            f"{func.__name__} must accept 'inputs' as a required keyword-only argument."
+        )
+
+    # Check param details
+    required_param = params["inputs"]
+    # Inputs must be required (no default)
+    if required_param.default is not inspect.Parameter.empty:
+        raise TypeError(f"'inputs' must not have a default value in {func.__name__}.")
+
+    for optional_arg in ("outputs", "expectations", "trace"):
+        if optional_arg not in params:
+            raise TypeError(
+                f"'{optional_arg}' must be present as an optional argument in {func.__name__}."
+            )
+
+    def wrapped_call(self, *, inputs, outputs=None, expectations=None, trace=None):
+        result = func(inputs=inputs, outputs=outputs, expectations=expectations, trace=trace)
+        if not isinstance(result, (float, bool, str, Assessment)):
+            raise ValueError(
+                (
+                    f"{func.__name__} must return one of float, bool, str, ",
+                    "or Assessment. Got {type(result).__name__}",
+                )
+            )
+        return result
+
+    instance = Scorer(name=name or func.__name__)
+    instance.__call__ = types.MethodType(wrapped_call, instance)
+
+    return instance
diff --git a/mlflow/genai/scorers/dummy_builtin_scorer.py b/mlflow/genai/scorers/dummy_builtin_scorer.py
new file mode 100644
index 0000000000000..58416b431ebd3
--- /dev/null
+++ b/mlflow/genai/scorers/dummy_builtin_scorer.py
@@ -0,0 +1,11 @@
+import copy
+
+from mlflow.genai.scorers import BuiltInScorer
+
+
+# TODO: ML-52304 example builtin scorer implementation
+class DummyBuiltInScorer(BuiltInScorer):
+    def update_evaluation_config(evaluation_config) -> dict:
+        updated_config = copy.deepcopy(evaluation_config)
+        updated_config["databricks-agents"]["metrics"].append("dummy_metric")
+        return updated_config
diff --git a/mlflow/tracing/utils/__init__.py b/mlflow/tracing/utils/__init__.py
index 1388f9600a591..a3a7e1add997c 100644
--- a/mlflow/tracing/utils/__init__.py
+++ b/mlflow/tracing/utils/__init__.py
@@ -9,6 +9,7 @@
 from dataclasses import asdict, is_dataclass
 from functools import lru_cache
 from typing import TYPE_CHECKING, Any, Optional, Union
+from unittest.mock import MagicMock, patch
 
 from opentelemetry import trace as trace_api
 from packaging.version import Version
@@ -457,3 +458,22 @@ def end_client_span_or_trace(
             status=status,
             end_time_ns=end_time_ns,
         )
+
+
+def is_model_traced(model):
+    """
+    Check if a PyFuncModel is being traced without logging to the database.
+
+    Args:
+        model (PyFuncModel): The model to check.
+
+    Returns:
+        True if the model is being traced, False otherwise.
+    """
+    with patch("mlflow.tracing.provider._get_tracer", return_value=None) as mock_get_tracer:
+        try:
+            model.predict(MagicMock())
+        except Exception:
+            pass
+
+        return 0 < mock_get_tracer.call_count
diff --git a/tests/tracing/utils/test_utils.py b/tests/tracing/utils/test_utils.py
index 06b2816254833..86f51a3d9117f 100644
--- a/tests/tracing/utils/test_utils.py
+++ b/tests/tracing/utils/test_utils.py
@@ -1,16 +1,26 @@
+import openai
 import pytest
 from pydantic import ValidationError
 
 import mlflow
-from mlflow.entities import LiveSpan
+from mlflow.entities import (
+    LiveSpan,
+    SpanType,
+)
 from mlflow.entities.span import SpanType
 from mlflow.exceptions import MlflowTracingException
+from mlflow.models import Model
+from mlflow.models.evaluation.base import _get_model_from_function
+from mlflow.pyfunc import PyFuncModel
 from mlflow.tracing import set_span_chat_messages, set_span_chat_tools
-from mlflow.tracing.constant import SpanAttributeKey
+from mlflow.tracing.constant import (
+    SpanAttributeKey,
+)
 from mlflow.tracing.utils import (
     construct_full_inputs,
     deduplicate_span_names_in_place,
     encode_span_id,
+    is_model_traced,
     maybe_get_request_id,
 )
 
@@ -193,3 +203,59 @@ def func(self, a, b, c=3, d=4, **kwargs):
 
     result = construct_full_inputs(TestClass().func, 1, 2)
     assert result == {"a": 1, "b": 2}
+
+
+def get_openai_predict_fn(with_tracing=False):
+    client = openai.OpenAI()
+
+    if with_tracing:
+        mlflow.openai.autolog()
+
+    def predict_fn(request):
+        response = client.chat.completions.create(
+            message=request["messages"],
+            model="gpt-4o-mini",
+        )
+        return response.choices[0].message.content
+
+    return predict_fn
+
+
+def get_dummy_predict_fn(with_tracing=False):
+    def predict_fn(inp):
+        return inp * 2
+
+    if with_tracing:
+        return mlflow.trace(predict_fn)
+
+    return predict_fn
+
+
+def get_pyfunc_model(predict_fn):
+    return PyFuncModel(
+        model_id=None,
+        model_meta=Model(),
+        model_impl=_get_model_from_function(predict_fn),
+    )
+
+
+@pytest.fixture
+def mock_openai_env(monkeypatch):
+    monkeypatch.setenv("OPENAI_API_KEY", "fake_api_key")
+
+
+@pytest.mark.usefixtures("mock_openai_env")
+@pytest.mark.parametrize(
+    ("test_case", "predict_fn_generator", "with_tracing", "expected_traced"),
+    [
+        ("dummy predict_fn without tracing", get_dummy_predict_fn, False, False),
+        ("dummy predict_fn with tracing", get_dummy_predict_fn, True, True),
+        ("openai predict_fn without tracing", get_openai_predict_fn, False, False),
+        ("openai predict_fn with tracing", get_openai_predict_fn, True, True),
+    ],
+)
+def test_is_traced(test_case, predict_fn_generator, with_tracing, expected_traced):
+    predict_fn = predict_fn_generator(with_tracing=with_tracing)
+    model = get_pyfunc_model(predict_fn)
+    is_actually_traced = is_model_traced(model)
+    assert is_actually_traced == expected_traced, f"Failed for {test_case}"


Reviews:
user: login: B-Step62
body: Overall structure LGTM! Added a few comments, but mostly minor.
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-04-15T14:38:27Z
user: login: B-Step62
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-04-16T13:34:12Z
user: login: B-Step62
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-04-18T03:26:07Z
user: login: B-Step62
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-04-18T03:27:07Z
user: login: B-Step62
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-04-18T03:27:49Z
user: login: B-Step62
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-04-18T03:30:04Z
user: login: B-Step62
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-04-18T06:24:50Z
user: login: B-Step62
state: APPROVED
author_association: COLLABORATOR
submitted_at: 2025-04-18T06:25:58Z

Review comments:
diff_hunk: @@ -0,0 +1,134 @@
+from typing import Any, Callable, Optional
+
+from pyspark import sql as spark
+
+import mlflow
+from mlflow.data.evaluation_dataset import EvaluationDataset
+from mlflow.genai.evaluation.utils import (
+    _convert_scorer_to_legacy_metric,
+    _convert_to_legacy_eval_set,
+)
+from mlflow.genai.scorers import BuiltInScorer, Scorer
+from mlflow.pyfunc import PyFuncModel
+from mlflow.tracing.fluent import is_traced
+
+try:
+    # `pandas` is not required for `mlflow-skinny`.
+    import pandas as pd
+except ImportError:
+    pass
+
+
+class EvaluationResult:
+    run_id: str
+    metrics: dict[str, float]
+
+
+def evaluate(
+    # See the "Input Data" section for the required format
path: mlflow/genai/evaluation/base.py
user: login: B-Step62
body: nit: we can drop these comments, they are just for design doc
created_at: 2025-04-15T14:21:45Z
author_association: COLLABORATOR
diff_hunk: @@ -0,0 +1,134 @@
+from typing import Any, Callable, Optional
+
+from pyspark import sql as spark
+
+import mlflow
+from mlflow.data.evaluation_dataset import EvaluationDataset
+from mlflow.genai.evaluation.utils import (
+    _convert_scorer_to_legacy_metric,
+    _convert_to_legacy_eval_set,
+)
+from mlflow.genai.scorers import BuiltInScorer, Scorer
+from mlflow.pyfunc import PyFuncModel
+from mlflow.tracing.fluent import is_traced
+
+try:
+    # `pandas` is not required for `mlflow-skinny`.
+    import pandas as pd
+except ImportError:
+    pass
+
+
+class EvaluationResult:
+    run_id: str
+    metrics: dict[str, float]
+
+
+def evaluate(
+    # See the "Input Data" section for the required format
+    data: pd.DataFrame | spark.DataFrame | list[dict] | EvaluationDataset,
+    predict_fn: Optional[Callable[..., Any]] = None,
+    # A list of scorers to compute metrics/assessments
+    scorers: Optional[list[Scorer]] = None,
+    # Optional argument to specify model ID to associate the Evaluation result with.
+    model_id: Optional[str] = None,
+) -> mlflow.genai.EvaluationResult:
+    """
+
+    Args:
+        data: Dataset for the evaluation. It must be one of the following format:
path: mlflow/genai/evaluation/base.py
user: login: B-Step62
body: nit: Let's add TODO for updating docstring with real examples and API links.
created_at: 2025-04-15T14:23:26Z
author_association: COLLABORATOR
diff_hunk: @@ -0,0 +1,134 @@
+from typing import Any, Callable, Optional
+
+from pyspark import sql as spark
+
+import mlflow
+from mlflow.data.evaluation_dataset import EvaluationDataset
+from mlflow.genai.evaluation.utils import (
+    _convert_scorer_to_legacy_metric,
+    _convert_to_legacy_eval_set,
+)
+from mlflow.genai.scorers import BuiltInScorer, Scorer
+from mlflow.pyfunc import PyFuncModel
+from mlflow.tracing.fluent import is_traced
+
+try:
+    # `pandas` is not required for `mlflow-skinny`.
+    import pandas as pd
+except ImportError:
+    pass
+
+
+class EvaluationResult:
+    run_id: str
+    metrics: dict[str, float]
+
+
+def evaluate(
+    # See the "Input Data" section for the required format
+    data: pd.DataFrame | spark.DataFrame | list[dict] | EvaluationDataset,
+    predict_fn: Optional[Callable[..., Any]] = None,
+    # A list of scorers to compute metrics/assessments
+    scorers: Optional[list[Scorer]] = None,
+    # Optional argument to specify model ID to associate the Evaluation result with.
+    model_id: Optional[str] = None,
+) -> mlflow.genai.EvaluationResult:
+    """
+
+    Args:
+        data: Dataset for the evaluation. It must be one of the following format:
+            * A EvaluationDataset entity
+            * Pandas DataFrame
+            * Spark DataFrame
+            * List of dictionary
+
+            If a dataframe is specified, it must contain the following schema:
+              - inputs (required): A colume that contains a single input.
+              - outputs (optional): A column that contains a single output from the
+                   target model/app. If the predict_fn is provided, this is generated
+                   by MLflow so not required.
+              - expectations (optional): A column that contains a ground truth, or a
+                   dictionary of ground truths for individual output fields.
+              - traces (optional): A column that contains a single trace object
+                   corresponding to the prediction for the row. Only required when
+                   any of scorers requires a trace in order to compute
+                   assessments/metrics.
+
+            If a list of dictionary is passed, each dictionary should contain keys
+            following the above schema.
+
+        predict_fn: The target function to be evaluated. The specified function will be
+            executed for each row in the input dataset, and outputs will be used for
+            scoring.
+
+            The function must emit a single trace per call. If it doesn't, decorate
+            the function with @mlflow.trace decorator to ensure a trace to be emitted.
+
+        scorers: A list of Scorer objects that produces evaluation scores from
+            inputs, outputs, and other additional contexts. MLflow provides pre-defined
+            scorers, but you can also define custom ones.
+
+        model_id: Optional. Specify an ID of the model e.g. models:/my-model/1 to
+            associate the evaluation result with. There are several ways to associate
+            model with association.
+
+            1. Use the `model_id` parameters.
+            2. Use the mlflow.set_active_model() function to set model ID to global context.
+               ```python
+               mlflow.set_active_model(model_id="xyz")
+
+               mlflow.evaluate(data, ...)
+               ```
+    """
+    if mlflow.get_tracking_uri() != "databricks":
+        raise ValueError(
+            (
+                "The genai evaluation function is only supported on Databricks. ",
+                "Please set the tracking URI to Databricks.",
+            )
+        )
+
+    builtin_scorers = []
+    custom_scorers = []
+
+    for scorer in scorers:
+        if isinstance(scorer, BuiltInScorer):
+            builtin_scorers.append(scorer)
+        elif isinstance(scorer, Scorer):
+            custom_scorers.append(scorer)
+        else:
+            raise TypeError(
+                (
+                    f"Scorer {scorer} is not a valid scorer. Please use the @scorer decorator ",
+                    "to convert a function into a scorer or inherit from the Scorer class",
+                )
+            )
+
+    if not is_traced(predict_fn):
+        predict_fn = mlflow.trace(predict_fn)
+
+    pyfunc_model = PyFuncModel(
path: mlflow/genai/evaluation/base.py
user: login: B-Step62
body: The `predict_fn`'s interface is not equal to the one `model_impl` expects. We can instead use [_get_model_from_function](https://github.com/mlflow/mlflow/blob/5f9cf84b1f6531d34f3581881ed757d0eca4924a/mlflow/models/evaluation/base.py#L1058-L1066).
created_at: 2025-04-15T14:27:47Z
author_association: COLLABORATOR
diff_hunk: @@ -0,0 +1,134 @@
+from typing import Any, Callable, Optional
+
+from pyspark import sql as spark
+
+import mlflow
+from mlflow.data.evaluation_dataset import EvaluationDataset
+from mlflow.genai.evaluation.utils import (
+    _convert_scorer_to_legacy_metric,
+    _convert_to_legacy_eval_set,
+)
+from mlflow.genai.scorers import BuiltInScorer, Scorer
+from mlflow.pyfunc import PyFuncModel
+from mlflow.tracing.fluent import is_traced
+
+try:
+    # `pandas` is not required for `mlflow-skinny`.
+    import pandas as pd
+except ImportError:
+    pass
+
+
+class EvaluationResult:
+    run_id: str
+    metrics: dict[str, float]
+
+
+def evaluate(
+    # See the "Input Data" section for the required format
+    data: pd.DataFrame | spark.DataFrame | list[dict] | EvaluationDataset,
+    predict_fn: Optional[Callable[..., Any]] = None,
+    # A list of scorers to compute metrics/assessments
+    scorers: Optional[list[Scorer]] = None,
+    # Optional argument to specify model ID to associate the Evaluation result with.
+    model_id: Optional[str] = None,
+) -> mlflow.genai.EvaluationResult:
+    """
+
+    Args:
+        data: Dataset for the evaluation. It must be one of the following format:
+            * A EvaluationDataset entity
+            * Pandas DataFrame
+            * Spark DataFrame
+            * List of dictionary
+
+            If a dataframe is specified, it must contain the following schema:
+              - inputs (required): A colume that contains a single input.
+              - outputs (optional): A column that contains a single output from the
+                   target model/app. If the predict_fn is provided, this is generated
+                   by MLflow so not required.
+              - expectations (optional): A column that contains a ground truth, or a
+                   dictionary of ground truths for individual output fields.
+              - traces (optional): A column that contains a single trace object
+                   corresponding to the prediction for the row. Only required when
+                   any of scorers requires a trace in order to compute
+                   assessments/metrics.
+
+            If a list of dictionary is passed, each dictionary should contain keys
+            following the above schema.
+
+        predict_fn: The target function to be evaluated. The specified function will be
+            executed for each row in the input dataset, and outputs will be used for
+            scoring.
+
+            The function must emit a single trace per call. If it doesn't, decorate
+            the function with @mlflow.trace decorator to ensure a trace to be emitted.
+
+        scorers: A list of Scorer objects that produces evaluation scores from
+            inputs, outputs, and other additional contexts. MLflow provides pre-defined
+            scorers, but you can also define custom ones.
+
+        model_id: Optional. Specify an ID of the model e.g. models:/my-model/1 to
+            associate the evaluation result with. There are several ways to associate
+            model with association.
+
+            1. Use the `model_id` parameters.
+            2. Use the mlflow.set_active_model() function to set model ID to global context.
+               ```python
+               mlflow.set_active_model(model_id="xyz")
+
+               mlflow.evaluate(data, ...)
+               ```
+    """
+    if mlflow.get_tracking_uri() != "databricks":
+        raise ValueError(
+            (
+                "The genai evaluation function is only supported on Databricks. ",
+                "Please set the tracking URI to Databricks.",
+            )
+        )
+
+    builtin_scorers = []
+    custom_scorers = []
+
+    for scorer in scorers:
+        if isinstance(scorer, BuiltInScorer):
+            builtin_scorers.append(scorer)
+        elif isinstance(scorer, Scorer):
+            custom_scorers.append(scorer)
+        else:
+            raise TypeError(
+                (
+                    f"Scorer {scorer} is not a valid scorer. Please use the @scorer decorator ",
+                    "to convert a function into a scorer or inherit from the Scorer class",
+                )
+            )
+
+    if not is_traced(predict_fn):
+        predict_fn = mlflow.trace(predict_fn)
+
+    pyfunc_model = PyFuncModel(
+        model_id=model_id,
+        model_impl=predict_fn,
+    )
+
+    evaluation_config = {
+        "databricks-agents": {
+            "metrics": [],
+        }
+    }
+    for _scorer in builtin_scorers:
+        _scorer(evaluation_config)
path: mlflow/genai/evaluation/base.py
user: login: B-Step62
body: Let's add an internal method like `update_evaluation_config` to keep the `__call__` interface consistent between `Scorer` and `BuiltInScorer`. While `BuildInScorer` will be a stub in MVP, it can be confusing that the `__call__` method has different signature.
created_at: 2025-04-15T14:29:59Z
author_association: COLLABORATOR
diff_hunk: @@ -0,0 +1,35 @@
+from pyspark import sql as spark
+
+from mlflow.genai.scorers import Scorer
+from mlflow.models import make_metric
+
+try:
+    # `pandas` is not required for `mlflow-skinny`.
+    import pandas as pd
+except ImportError:
+    pass
+
+
+# TODO: ML-52299
+def _convert_to_legacy_eval_set(
+    data: pd.DataFrame | spark.DataFrame | list[dict], EvaluationDataset
+) -> dict:
+    """
+    Takes in different types of inputs and converts it into to the current eval-set schema that
+    Agent Evaluation takes in. The transformed schema should be accepted by mlflow.evaluate()
+    """
+    raise NotImplementedError("This function is not implemented yet.")
+
+
+# TODO: ML-52297
+def _convert_scorer_to_legacy_metric(scorer: Scorer):
+    """
+    Takes in a Scorer object and converts it into a legacy MLflow 2.x
+    Metric object.
+    """
+    # TODO: pass in more arguments such as aggregations, etc
+    return make_metric(
path: mlflow/genai/evaluation/utils.py
user: login: B-Step62
body: As discussed offline, let's use the `@metric` decorator for defining the legacy metric.
https://docs.databricks.com/aws/en/generative-ai/agent-evaluation/custom-metrics
created_at: 2025-04-15T14:31:34Z
author_association: COLLABORATOR
diff_hunk: @@ -0,0 +1,93 @@
+import functools
+import inspect
+import types
+from typing import Any, Callable, Literal, Optional, Union
+
+from pydantic import BaseModel
+
+from mlflow.entities import Trace
+from mlflow.evaluation import Assessment
+
+
+class Scorer(BaseModel):
+    name: str
+
+    def __call__(
+        self,
+        *,
+        inputs: dict[str, Any],
+        outputs: Optional[dict[str, Any]] = None,
path: mlflow/genai/scorers/base.py
user: login: B-Step62
body: nit: inputs and outputs can be any type not only dictionary
created_at: 2025-04-15T14:32:07Z
author_association: COLLABORATOR
diff_hunk: @@ -174,6 +175,19 @@ def decorator(fn):
     return decorator(func) if func else decorator
 
 
+def is_traced(fn: Callable) -> bool:
path: mlflow/tracing/fluent.py
user: login: B-Step62
body: As discussed offline, we need to check if the `predict_fn` generates a trace or not, because the function itself might not be decorated with `@mlflow.trace`.

For example, in the following case, the `client.chat.completions(...)` call generates a trace as OpenAI autologging is enabled. We can use that trace for evaluation and no need to decorate the `predict_fn`.
```
import openai

client = openai.OpenAI()

mlflow.openai.autolog()

def predict_fn(request):
     response = chat.completions(
        message=request["messages"],
        model="gpt-4o-mini",
     )
     return response.choices[0].message.content
```
created_at: 2025-04-15T14:36:45Z
author_association: COLLABORATOR
diff_hunk: @@ -0,0 +1,103 @@
+import functools
+import inspect
+import types
+from typing import Callable, Literal, Optional, Union
+
+from pydantic import BaseModel
+
+from mlflow.evaluation import Assessment
+
+
+class Scorer(BaseModel):
+    name: str
+
+    def __call__(
+        self,
+        *,
+        inputs,
+        outputs=None,
+        expectations=None,
+        trace=None,
+    ) -> Union[float, bool, str, Assessment]:
path: mlflow/genai/scorers/base.py
user: login: B-Step62
body: Agent team told custom metric actually support `list[Assessment]` as well, let's include it in the new scorer.
created_at: 2025-04-16T13:34:11Z
author_association: COLLABORATOR
diff_hunk: @@ -174,6 +176,20 @@ def decorator(fn):
     return decorator(func) if func else decorator
 
 
+def is_traced(model: PyFuncModel):
path: mlflow/tracing/fluent.py
user: login: B-Step62
body: @dhruyads Let's move this function under `tracing/utils/__init__.py`. This should not be exposed as public API. Also let's give a more descriptive name like `is_model_traced`.
created_at: 2025-04-18T03:26:06Z
author_association: COLLABORATOR
diff_hunk: @@ -20,6 +21,7 @@
 from mlflow.entities.trace_status import TraceStatus
 from mlflow.exceptions import MlflowException
 from mlflow.protos.databricks_pb2 import BAD_REQUEST
+from mlflow.pyfunc import PyFuncModel
path: mlflow/tracing/fluent.py
user: login: B-Step62
body: This line causes `no module named 'numpy'` error in CI. Numpy is not installed in mlflow-skinny hence should not be imported at global level under `tracing` module. Can we move this line inside `is_traced` function?
created_at: 2025-04-18T03:27:07Z
author_association: COLLABORATOR
diff_hunk: @@ -0,0 +1,134 @@
+from typing import Any, Callable, Optional
+
+from pyspark import sql as spark
+
+import mlflow
+from mlflow.data.evaluation_dataset import EvaluationDataset
+from mlflow.genai.evaluation.utils import (
+    _convert_scorer_to_legacy_metric,
+    _convert_to_legacy_eval_set,
+)
+from mlflow.genai.scorers import BuiltInScorer, Scorer
+from mlflow.models import Model
+from mlflow.models.evaluation.base import _get_model_from_function
+from mlflow.pyfunc import PyFuncModel
+from mlflow.tracing.fluent import is_traced
+
+try:
+    # `pandas` is not required for `mlflow-skinny`.
+    import pandas as pd
+except ImportError:
+    pass
+
+
+class EvaluationResult:
+    run_id: str
+    metrics: dict[str, float]
+
+
+def evaluate(
+    data: pd.DataFrame | spark.DataFrame | list[dict] | EvaluationDataset,
+    predict_fn: Optional[Callable[..., Any]] = None,
+    scorers: Optional[list[Scorer]] = None,
+    model_id: Optional[str] = None,
+) -> mlflow.genai.EvaluationResult:
+    """
+    TODO: updating docstring with real examples and API links
+    Args:
+        data: Dataset for the evaluation. It must be one of the following format:
+            * A EvaluationDataset entity
+            * Pandas DataFrame
+            * Spark DataFrame
+            * List of dictionary
+
+            If a dataframe is specified, it must contain the following schema:
+              - inputs (required): A column that contains a single input.
+              - outputs (optional): A column that contains a single output from the
+                   target model/app. If the predict_fn is provided, this is generated
+                   by MLflow so not required.
+              - expectations (optional): A column that contains a ground truth, or a
+                   dictionary of ground truths for individual output fields.
+              - traces (optional): A column that contains a single trace object
+                   corresponding to the prediction for the row. Only required when
+                   any of scorers requires a trace in order to compute
+                   assessments/metrics.
+
+            If a list of dictionary is passed, each dictionary should contain keys
+            following the above schema.
+
+        predict_fn: The target function to be evaluated. The specified function will be
+            executed for each row in the input dataset, and outputs will be used for
+            scoring.
+
+            The function must emit a single trace per call. If it doesn't, decorate
+            the function with @mlflow.trace decorator to ensure a trace to be emitted.
+
+        scorers: A list of Scorer objects that produces evaluation scores from
+            inputs, outputs, and other additional contexts. MLflow provides pre-defined
+            scorers, but you can also define custom ones.
+
+        model_id: Optional. Specify an ID of the model e.g. models:/my-model/1 to
+            associate the evaluation result with. There are several ways to associate
+            model with association.
+
+            1. Use the `model_id` parameters.
+            2. Use the mlflow.set_active_model() function to set model ID to global context.
+               ```python
+               mlflow.set_active_model(model_id="xyz")
+
+               mlflow.evaluate(data, ...)
+               ```
+    """
+    if mlflow.get_tracking_uri() != "databricks":
+        raise ValueError(
+            (
+                "The genai evaluation function is only supported on Databricks. ",
+                "Please set the tracking URI to Databricks.",
+            )
+        )
+
+    builtin_scorers = []
+    custom_scorers = []
+
+    for scorer in scorers:
+        if isinstance(scorer, BuiltInScorer):
+            builtin_scorers.append(scorer)
+        elif isinstance(scorer, Scorer):
+            custom_scorers.append(scorer)
+        else:
+            raise TypeError(
+                (
+                    f"Scorer {scorer} is not a valid scorer. Please use the @scorer decorator ",
+                    "to convert a function into a scorer or inherit from the Scorer class",
+                )
+            )
+
+    evaluation_config = {
+        "databricks-agents": {
+            "metrics": [],
+        }
+    }
+    for _scorer in builtin_scorers:
+        evaluation_config = _scorer.update_evaluation_config(evaluation_config)
+
+    extra_metrics = []
+    for _scorer in custom_scorers:
+        extra_metrics.append(_convert_scorer_to_legacy_metric(_scorer))
+
+    if not is_traced(predict_fn):
path: mlflow/genai/evaluation/base.py
user: login: B-Step62
body: nit: Can we add a logging message indicates that MLflow annotate the function to produce a trace?
created_at: 2025-04-18T03:27:49Z
author_association: COLLABORATOR
diff_hunk: @@ -0,0 +1,134 @@
+from typing import Any, Callable, Optional
+
+from pyspark import sql as spark
+
+import mlflow
+from mlflow.data.evaluation_dataset import EvaluationDataset
+from mlflow.genai.evaluation.utils import (
+    _convert_scorer_to_legacy_metric,
+    _convert_to_legacy_eval_set,
+)
+from mlflow.genai.scorers import BuiltInScorer, Scorer
+from mlflow.models import Model
+from mlflow.models.evaluation.base import _get_model_from_function
+from mlflow.pyfunc import PyFuncModel
+from mlflow.tracing.fluent import is_traced
+
+try:
+    # `pandas` is not required for `mlflow-skinny`.
+    import pandas as pd
+except ImportError:
+    pass
+
+
+class EvaluationResult:
+    run_id: str
+    metrics: dict[str, float]
+
+
+def evaluate(
+    data: pd.DataFrame | spark.DataFrame | list[dict] | EvaluationDataset,
+    predict_fn: Optional[Callable[..., Any]] = None,
+    scorers: Optional[list[Scorer]] = None,
+    model_id: Optional[str] = None,
+) -> mlflow.genai.EvaluationResult:
+    """
+    TODO: updating docstring with real examples and API links
+    Args:
+        data: Dataset for the evaluation. It must be one of the following format:
+            * A EvaluationDataset entity
+            * Pandas DataFrame
+            * Spark DataFrame
+            * List of dictionary
+
+            If a dataframe is specified, it must contain the following schema:
+              - inputs (required): A column that contains a single input.
+              - outputs (optional): A column that contains a single output from the
+                   target model/app. If the predict_fn is provided, this is generated
+                   by MLflow so not required.
+              - expectations (optional): A column that contains a ground truth, or a
+                   dictionary of ground truths for individual output fields.
+              - traces (optional): A column that contains a single trace object
+                   corresponding to the prediction for the row. Only required when
+                   any of scorers requires a trace in order to compute
+                   assessments/metrics.
+
+            If a list of dictionary is passed, each dictionary should contain keys
+            following the above schema.
+
+        predict_fn: The target function to be evaluated. The specified function will be
+            executed for each row in the input dataset, and outputs will be used for
+            scoring.
+
+            The function must emit a single trace per call. If it doesn't, decorate
+            the function with @mlflow.trace decorator to ensure a trace to be emitted.
+
+        scorers: A list of Scorer objects that produces evaluation scores from
+            inputs, outputs, and other additional contexts. MLflow provides pre-defined
+            scorers, but you can also define custom ones.
+
+        model_id: Optional. Specify an ID of the model e.g. models:/my-model/1 to
+            associate the evaluation result with. There are several ways to associate
+            model with association.
+
+            1. Use the `model_id` parameters.
+            2. Use the mlflow.set_active_model() function to set model ID to global context.
+               ```python
+               mlflow.set_active_model(model_id="xyz")
+
+               mlflow.evaluate(data, ...)
+               ```
+    """
+    if mlflow.get_tracking_uri() != "databricks":
+        raise ValueError(
+            (
+                "The genai evaluation function is only supported on Databricks. ",
+                "Please set the tracking URI to Databricks.",
+            )
+        )
+
+    builtin_scorers = []
+    custom_scorers = []
+
+    for scorer in scorers:
+        if isinstance(scorer, BuiltInScorer):
+            builtin_scorers.append(scorer)
+        elif isinstance(scorer, Scorer):
+            custom_scorers.append(scorer)
+        else:
+            raise TypeError(
+                (
+                    f"Scorer {scorer} is not a valid scorer. Please use the @scorer decorator ",
+                    "to convert a function into a scorer or inherit from the Scorer class",
+                )
+            )
+
+    evaluation_config = {
+        "databricks-agents": {
+            "metrics": [],
+        }
+    }
+    for _scorer in builtin_scorers:
+        evaluation_config = _scorer.update_evaluation_config(evaluation_config)
+
+    extra_metrics = []
+    for _scorer in custom_scorers:
+        extra_metrics.append(_convert_scorer_to_legacy_metric(_scorer))
+
+    if not is_traced(predict_fn):
+        predict_fn = mlflow.trace(predict_fn)
+
+    pyfunc_model = PyFuncModel(
path: mlflow/genai/evaluation/base.py
user: login: B-Step62
body: Actually, if we directly pass the function to `mlflow.evaluate`, does it fail?
created_at: 2025-04-18T03:30:04Z
author_association: COLLABORATOR
diff_hunk: @@ -193,3 +203,60 @@ def func(self, a, b, c=3, d=4, **kwargs):
 
     result = construct_full_inputs(TestClass().func, 1, 2)
     assert result == {"a": 1, "b": 2}
+
+
+def get_openai_predict_fn(with_tracing=False):
+    client = openai.OpenAI()
+
+    if with_tracing:
+        mlflow.autolog()
path: tests/tracing/utils/test_utils.py
user: login: B-Step62
body: ```suggestion
```
created_at: 2025-04-18T06:24:49Z
author_association: COLLABORATOR

PR comments:
user: login: github-actions[bot]
created_at: 2025-04-10T16:31:13Z
body: <!-- documentation preview -->


Documentation preview for 7552d33b631f492d88c9484a5f0e09b2c74dae91 will be available when [this CircleCI job](https://circleci.com/gh/mlflow/mlflow/154088)
completes successfully. You may encounter a `{"message":"not found"}` error when reloading
a page. If so, add `/index.html` to the URL.

- [Top page](https://output.circle-artifacts.com/output/job/cfaae56a-42af-4cf2-b9a7-4c1ff89f8cc5/artifacts/0/docs/build/latest/index.html)
- [Changed pages](https://output.circle-artifacts.com/output/job/cfaae56a-42af-4cf2-b9a7-4c1ff89f8cc5/artifacts/0/docs/build/latest/diff.html) (⚠️ only MDX file changes are detected ⚠️)

<details>
<summary>More info</summary>

- Ignore this comment if this PR does not change the documentation.
- It takes a few minutes for the preview to be available.
- The preview is updated when a new commit is pushed to this PR.
- This comment was created by https://github.com/mlflow/mlflow/actions/runs/14530823005.

</details>

author_association: CONTRIBUTOR
user: login: artjen
created_at: 2025-04-18T01:18:48Z
body: Followup PR exposing builtin scorers: https://github.com/mlflow/mlflow/pull/15376
author_association: COLLABORATOR