Repository: mlflow/mlflow
Pull request: Support string guidelines for `meets_guidelines` judge
The `mlflow.genai.judges.meets_guidelines` function was documented to accept both single strings and lists of strings for the `guidelines` parameter, but the underlying `databricks.agents.evals.judges.guideline_adherence` function only accepts lists. This caused runtime errors when users passed a single string guideline.

## Changes Made

1. **Fixed `meets_guidelines` function**: Added logic to wrap string guidelines in a list before passing to the underlying databricks judge
2. **Enhanced test coverage**: Added comprehensive tests in `test_guideline_adherence` to verify both string and list input scenarios work correctly

## Example Usage

Both of these now work correctly:

```python
from mlflow.genai.judges import meets_guidelines

# Single string guideline (now works!)
feedback = meets_guidelines(
    guidelines="Be polite and respectful.",
    context={"response": "Hello, how are you?"}
)

# List of guidelines (already worked)
feedback = meets_guidelines(
    guidelines=["Be polite and respectful.", "Must be in English."],
    context={"response": "Hello, how are you?"}
)
```

The fix is minimal (4 lines) and maintains full backward compatibility while enabling the documented API to work as expected.

Fixes #16013.

---

💡 You can make Copilot smarter by setting up custom instructions, customizing its development environment and configuring Model Context Protocol (MCP) servers. Learn more [Copilot coding agent tips](https://gh.io/copilot-coding-agent-tips) in the docs.

Problem statement:
Support string guidelines for `meets_guidelines` judge
The `mlflow.genai.judges.meets_guidelines` should allow both single string and list of strings inputs. The underlying databricks judge only accept a list of judge, so our judge API should wrap it in a list. The change should be tested in `test_guideline_adherence` test case in `tests/genai/scorers/test_builtin_scorers.py`

Hints from issue discussion:

Commit message: Support string guidelines for meets_guidelines judge

Co-authored-by: B-Step62 <31463517+B-Step62@users.noreply.github.com>

Patch to review:
diff --git a/mlflow/genai/judges/databricks.py b/mlflow/genai/judges/databricks.py
index 5c863d644ea4d..5f89557cfd2bd 100644
--- a/mlflow/genai/judges/databricks.py
+++ b/mlflow/genai/judges/databricks.py
@@ -278,6 +278,10 @@ def meets_guidelines(
     """
     from databricks.agents.evals.judges import guideline_adherence
 
+    # Ensure guidelines is a list, as the underlying databricks judge only accepts lists
+    if isinstance(guidelines, str):
+        guidelines = [guidelines]
+
     return guideline_adherence(
         guidelines=guidelines,
         guidelines_context=context,
diff --git a/tests/genai/scorers/test_builtin_scorers.py b/tests/genai/scorers/test_builtin_scorers.py
index f3c1a5c793956..d2a4fcce642d0 100644
--- a/tests/genai/scorers/test_builtin_scorers.py
+++ b/tests/genai/scorers/test_builtin_scorers.py
@@ -3,6 +3,7 @@
 from mlflow.entities.assessment import Feedback
 from mlflow.entities.assessment_error import AssessmentError
 from mlflow.entities.span import SpanType
+from mlflow.genai.judges import meets_guidelines
 from mlflow.genai.scorers import (
     Correctness,
     GuidelineAdherence,
@@ -243,6 +244,33 @@ def test_guideline_adherence():
         assessment_name="is_english",
     )
 
+    # 3. Test meets_guidelines judge with string input (should wrap in list)
+    with patch("databricks.agents.evals.judges.guideline_adherence") as mock_guideline_adherence:
+        meets_guidelines(
+            guidelines="Be polite and respectful.",
+            context={"response": "Hello, how are you?"},
+        )
+
+    mock_guideline_adherence.assert_called_once_with(
+        guidelines=["Be polite and respectful."],
+        guidelines_context={"response": "Hello, how are you?"},
+        assessment_name=None,
+    )
+
+    # 4. Test meets_guidelines judge with list input (should pass through)
+    with patch("databricks.agents.evals.judges.guideline_adherence") as mock_guideline_adherence:
+        meets_guidelines(
+            guidelines=["Be polite and respectful.", "Must be in English."],
+            context={"response": "Hola, ¿cómo estás?"},
+            name="custom_name",
+        )
+
+    mock_guideline_adherence.assert_called_once_with(
+        guidelines=["Be polite and respectful.", "Must be in English."],
+        guidelines_context={"response": "Hola, ¿cómo estás?"},
+        assessment_name="custom_name",
+    )
+
 
 def test_relevance_to_query():
     with patch(


Code review comments:
diff_hunk: @@ -243,6 +244,33 @@ def test_guideline_adherence():
         assessment_name="is_english",
     )
 
+    # 3. Test meets_guidelines judge with string input (should wrap in list)
+    with patch("databricks.agents.evals.judges.guideline_adherence") as mock_guideline_adherence:
+        meets_guidelines(
+            guidelines="Be polite and respectful.",
+            context={"response": "Hello, how are you?"},
+        )
+
+    mock_guideline_adherence.assert_called_once_with(
+        guidelines=["Be polite and respectful."],
+        guidelines_context={"response": "Hello, how are you?"},
+        assessment_name=None,
+    )
+
+    # 4. Test meets_guidelines judge with list input (should pass through)
path: tests/genai/scorers/test_builtin_scorers.py
text: @user1:
Remove this test as it is already covered by #2 above.

@author:
Done. Removed the redundant test #4 as it's already covered by test #2 which exercises the same underlying functionality with list guidelines.