Repository: mlflow/mlflow

Issue:
[BUG] Metric 'faithfulness': Error: Values for grading_context_columns are malformed when sampling from a pandas dataframe
### Issues Policy acknowledgement

- [X] I have read and agree to submit bug reports in accordance with the [issues policy](https://www.github.com/mlflow/mlflow/blob/master/ISSUE_POLICY.md)

### Where did you encounter this bug?

Local machine

### Willingness to contribute

No. I cannot contribute a bug fix at this time.

### MLflow version

- Client: 2.10.0
- Tracking server: 2.10.0


### System information

- **OS Platform and Distribution (e.g., Linux Ubuntu 16.04)**: Windows 10
- **Python version**: 3.10.13
- **yarn version, if running the dev UI**:


### Describe the problem

The faithfulness metric doesn't work when I sample from a dataframe `dataset["train"].to_pandas().sample(n=10, random_state=0)`. I saw the index in the error message "Values: {'context': 50436    During Vatican", so I tried `eval_df.reset_index(drop=True)` and then it worked. Is this intended?

```
MlflowException: Metric 'faithfulness': Error:
Values for grading_context_columns are malformed and cannot be formatted into a prompt for metric 'faithfulness'.
Required columns: ['context']
Values: {'context': 50436    During Vatican II, the Council Fathers avoided...
```


### Tracking information

<!-- PLEASE KEEP BACKTICKS AND CHECK PREVIEW -->
```shell
System information: Windows 10.0.19045
Python version: 3.10.13
MLflow version: 2.10.0
MLflow module location: [c:\Users\_\miniconda3\envs\ml-3.10\lib\site-packages\mlflow\__init__.py](file:///C:/Users/_/miniconda3/envs/ml-3.10/lib/site-packages/mlflow/__init__.py)
Tracking URI: http://127.0.0.1:5000/
Registry URI: http://127.0.0.1:5000/
MLflow dependencies: 
  Flask: 3.0.1
  Jinja2: 3.1.3
  aiohttp: 3.9.2
  alembic: 1.13.1
  boto3: 1.34.37
  botocore: 1.34.37
  click: 8.1.7
  cloudpickle: 3.0.0
  databricks-cli: 0.18.0
  docker: 7.0.0
  entrypoints: 0.4
  fastapi: 0.109.0
  gitpython: 3.1.41
  importlib-metadata: 6.11.0
  kubernetes: 29.0.0
  markdown: 3.5.2
  matplotlib: 3.8.2
  numpy: 1.26.3
  packaging: 23.2
  pandas: 2.2.0
  protobuf: 4.23.4
  pyarrow: 15.0.0
  pydantic: 2.6.0
  pytz: 2023.4
  pyyaml: 6.0.1
  querystring-parser: 1.2.4
  requests: 2.31.0
  scikit-learn: 1.4.0
  scipy: 1.12.0
  slowapi: 0.1.8
  sqlalchemy: 2.0.25
  sqlparse: 0.4.4
  tiktoken: 0.5.2
  uvicorn: 0.27.0.post1
  waitress: 2.1.2
  watchfiles: 0.21.0
```


### Code to reproduce issue

<!-- PLEASE KEEP BACKTICKS AND CHECK PREVIEW -->
```python
from datasets import load_dataset

dataset = load_dataset("squad")
eval_df = dataset["train"].to_pandas().sample(n=10, random_state=0)

def get_answer(row):
    answers = row["answers"]["text"]
    return answers[0]

eval_df["answer"] = eval_df.apply(get_answer, axis=1)
eval_df = eval_df.drop(columns=["answers", "id", "title"])

import mlflow

mlflow.set_tracking_uri("http://127.0.0.1:5000")
mlflow.set_experiment("squad-qa-mistral")

from langchain_core.prompts import ChatPromptTemplate, PromptTemplate
from langchain_core.output_parsers import StrOutputParser
from langchain_core.runnables import RunnablePassthrough
from langchain_core.runnables import RunnableLambda, RunnablePassthrough
from langchain_openai import ChatOpenAI

template = """Answer the question based only on the following context:
{context}

Question: {question}
"""
prompt = ChatPromptTemplate.from_template(template)

llm = ChatOpenAI()
chain = (
    RunnablePassthrough()
    | prompt 
    | llm
    | StrOutputParser()
)
chain.invoke({"question": "What is 1 + 1?", "context": "1 + 1 is 3"})

from mlflow.metrics.genai import answer_similarity, faithfulness, relevance

# Construct the metric using OpenAI GPT-4 as the judge
judge_model = "openai:/gpt-4"

answer_similarity_metric = answer_similarity(model=judge_model)
faithfulness_metric = faithfulness(model=judge_model)
relevance_metric = relevance(model=judge_model)

def model(input_df):
    answers = []
    for _, row in input_df.iterrows():
        answer = chain.invoke({"question": row["question"], "context": row["context"]})
        answers.append(answer)
    return answers

with mlflow.start_run(run_name="mistral"):
    mlflow.evaluate(
        model,
        eval_df, # eval_df.reset_index(drop=True)
        model_type="question-answering",
        evaluators="default",
        targets="answer",
        # extra_metrics=[answer_similarity_metric, faithfulness_metric, relevance_metric],
        extra_metrics=[faithfulness_metric],
        evaluator_config={
            "col_mapping": {
                "inputs": "question",
                "context": "context"
            }
        },
    )
```


### Stack trace

<!-- PLEASE KEEP BACKTICKS AND CHECK PREVIEW -->
```
---------------------------------------------------------------------------
MlflowException                           Traceback (most recent call last)
Cell In[5], line 9
      6     return answers
      8 with mlflow.start_run(run_name="mistral"):
----> 9     mlflow.evaluate(
     10         model,
     11         eval_df,
     12         model_type="question-answering",
     13         evaluators="default",
     14         targets="answer",
     15         # extra_metrics=[answer_similarity_metric, faithfulness_metric, relevance_metric],
     16         extra_metrics=[faithfulness_metric],
     17         evaluator_config={
     18             "col_mapping": {
     19                 "inputs": "question",
     20                 "context": "context"
     21             }
     22         },
     23     )

File c:\Users\_\miniconda3\envs\ml-3.10\lib\site-packages\mlflow\models\evaluation\base.py:1880, in evaluate(model, data, model_type, targets, predictions, dataset_path, feature_names, evaluators, evaluator_config, custom_metrics, extra_metrics, custom_artifacts, validation_thresholds, baseline_model, env_manager, model_config, baseline_config)
   1877 predictions_expected_in_model_output = predictions if model is not None else None
   1879 try:
-> 1880     evaluate_result = _evaluate(
   1881         model=model,
   1882         model_type=model_type,
   1883         dataset=dataset,
   1884         run_id=run_id,
   1885         evaluator_name_list=evaluator_name_list,
   1886         evaluator_name_to_conf_map=evaluator_name_to_conf_map,
   1887         custom_metrics=custom_metrics,
   1888         extra_metrics=extra_metrics,
   1889         custom_artifacts=custom_artifacts,
   1890         baseline_model=baseline_model,
   1891         predictions=predictions_expected_in_model_output,
   1892     )
   1893 finally:
   1894     if isinstance(model, _ServedPyFuncModel):

File c:\Users\_\miniconda3\envs\ml-3.10\lib\site-packages\mlflow\models\evaluation\base.py:1120, in _evaluate(model, model_type, dataset, run_id, evaluator_name_list, evaluator_name_to_conf_map, custom_metrics, extra_metrics, custom_artifacts, baseline_model, predictions)
   1118     if evaluator.can_evaluate(model_type=model_type, evaluator_config=config):
   1119         _logger.info(f"Evaluating the model with the {evaluator_name} evaluator.")
-> 1120         eval_result = evaluator.evaluate(
   1121             model=model,
   1122             model_type=model_type,
   1123             dataset=dataset,
   1124             run_id=run_id,
   1125             evaluator_config=config,
   1126             custom_metrics=custom_metrics,
   1127             extra_metrics=extra_metrics,
   1128             custom_artifacts=custom_artifacts,
   1129             baseline_model=baseline_model,
   1130             predictions=predictions,
   1131         )
   1132         eval_results.append(eval_result)
   1134 _last_failed_evaluator = None

File c:\Users\_\miniconda3\envs\ml-3.10\lib\site-packages\mlflow\models\evaluation\default_evaluator.py:1883, in DefaultEvaluator.evaluate(self, model_type, dataset, run_id, evaluator_config, model, custom_metrics, extra_metrics, custom_artifacts, baseline_model, predictions, **kwargs)
   1881     if baseline_model:
   1882         _logger.info("Evaluating candidate model:")
-> 1883     evaluation_result = self._evaluate(model, is_baseline_model=False)
   1885 if not baseline_model:
   1886     return evaluation_result

File c:\Users\_\miniconda3\envs\ml-3.10\lib\site-packages\mlflow\models\evaluation\default_evaluator.py:1762, in DefaultEvaluator._evaluate(self, model, is_baseline_model, **kwargs)
   1759 if self.dataset.has_targets:
   1760     eval_df["target"] = self.y
-> 1762 self._evaluate_metrics(eval_df)
   1763 if not is_baseline_model:
   1764     self._log_custom_artifacts(eval_df)

File c:\Users\_\miniconda3\envs\ml-3.10\lib\site-packages\mlflow\models\evaluation\default_evaluator.py:1573, in DefaultEvaluator._evaluate_metrics(self, eval_df)
   1571 def _evaluate_metrics(self, eval_df):
   1572     self._check_args(self.builtin_metrics + self.extra_metrics, eval_df)
-> 1573     self._test_first_row(eval_df)
   1575     # calculate metrics for the full eval_df
   1576     self._evaluate_builtin_metrics(eval_df)

File c:\Users\_\miniconda3\envs\ml-3.10\lib\site-packages\mlflow\models\evaluation\default_evaluator.py:1569, in DefaultEvaluator._test_first_row(self, eval_df)
   1566             exceptions.append(f"Metric '{metric.name}': Error:\n{e!r}\n{stacktrace_str}")
   1568 if len(exceptions) > 0:
-> 1569     raise MlflowException("\n".join(exceptions))

MlflowException: Metric 'faithfulness': Error:
Values for grading_context_columns are malformed and cannot be formatted into a prompt for metric 'faithfulness'.
Required columns: ['context']
Values: {'context': 50436    During Vatican II, the Council Fathers avoided...
22762    The structure of hardwoods is more complex. Th...
45227    In 1982, the band released the album Hot Space...
54941    The first appearance of the term 'affirmative ...
33360    On 28 January 1871 the Government of National ...
78996    The original town of San Diego was located at ...
86439    The most-commonly accepted theory is that thes...
24103    Other than by appearance, "Coloureds" can usua...
50462    Pope Paul VI knew the Roman Curia well, having...
75702    In the 2005 National Family Health Survey, it ...
Name: context, dtype: object}
Error: KeyError(0)
Please check the following: 
- predictions and targets (if required) are provided correctly
- grading_context_columns are mapped correctly using the evaluator_config parameter
- input and output data are formatted correctly.
Traceback (most recent call last):
  File "c:\Users\_\miniconda3\envs\ml-3.10\lib\site-packages\pandas\core\indexes\base.py", line 3802, in get_loc
    return self._engine.get_loc(casted_key)
  File "index.pyx", line 153, in pandas._libs.index.IndexEngine.get_loc
  File "index.pyx", line 182, in pandas._libs.index.IndexEngine.get_loc
  File "pandas\\_libs\\hashtable_class_helper.pxi", line 2606, in pandas._libs.hashtable.Int64HashTable.get_item
  File "pandas\\_libs\\hashtable_class_helper.pxi", line 2630, in pandas._libs.hashtable.Int64HashTable.get_item
KeyError: 0

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "c:\Users\_\miniconda3\envs\ml-3.10\lib\site-packages\mlflow\metrics\genai\genai_metric.py", line 277, in eval_fn
    arg_string = _format_args_string(grading_context_columns, eval_values, indx)
  File "c:\Users\_\miniconda3\envs\ml-3.10\lib\site-packages\mlflow\metrics\genai\genai_metric.py", line 34, in _format_args_string
    args_dict[arg] = eval_values[arg][indx]
  File "c:\Users\_\miniconda3\envs\ml-3.10\lib\site-packages\pandas\core\series.py", line 1111, in __getitem__
    return self._get_value(key)
  File "c:\Users\_\miniconda3\envs\ml-3.10\lib\site-packages\pandas\core\series.py", line 1227, in _get_value
    loc = self.index.get_loc(label)
  File "c:\Users\_\miniconda3\envs\ml-3.10\lib\site-packages\pandas\core\indexes\base.py", line 3809, in get_loc
    raise KeyError(key) from err
KeyError: 0

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "c:\Users\_\miniconda3\envs\ml-3.10\lib\site-packages\mlflow\models\evaluation\default_evaluator.py", line 1558, in _test_first_row
    metric.eval_fn(*eval_fn_args)
  File "c:\Users\_\miniconda3\envs\ml-3.10\lib\site-packages\mlflow\metrics\genai\genai_metric.py", line 279, in eval_fn
    raise MlflowException(
mlflow.exceptions.MlflowException: Values for grading_context_columns are malformed and cannot be formatted into a prompt for metric 'faithfulness'.
Required columns: ['context']
Values: {'context': 50436    During Vatican II, the Council Fathers avoided...
22762    The structure of hardwoods is more complex. Th...
45227    In 1982, the band released the album Hot Space...
54941    The first appearance of the term 'affirmative ...
33360    On 28 January 1871 the Government of National ...
78996    The original town of San Diego was located at ...
86439    The most-commonly accepted theory is that thes...
24103    Other than by appearance, "Coloureds" can usua...
50462    Pope Paul VI knew the Roman Curia well, having...
75702    In the 2005 National Family Health Survey, it ...
Name: context, dtype: object}
Error: KeyError(0)
Please check the following: 
- predictions and targets (if required) are provided correctly
- grading_context_columns are mapped correctly using the evaluator_config parameter
- input and output data are formatted correctly.
```


### Other info / logs

<!-- PLEASE KEEP BACKTICKS AND CHECK PREVIEW -->
```
REPLACE_ME
```


### What component(s) does this bug affect?

- [ ] `area/artifacts`: Artifact stores and artifact logging
- [ ] `area/build`: Build and test infrastructure for MLflow
- [ ] `area/deployments`: MLflow Deployments client APIs, server, and third-party Deployments integrations
- [ ] `area/docs`: MLflow documentation pages
- [ ] `area/examples`: Example code
- [ ] `area/model-registry`: Model Registry service, APIs, and the fluent client calls for Model Registry
- [ ] `area/models`: MLmodel format, model serialization/deserialization, flavors
- [ ] `area/recipes`: Recipes, Recipe APIs, Recipe configs, Recipe Templates
- [ ] `area/projects`: MLproject format, project running backends
- [ ] `area/scoring`: MLflow Model server, model deployment tools, Spark UDFs
- [ ] `area/server-infra`: MLflow Tracking server backend
- [ ] `area/tracking`: Tracking Service, tracking client APIs, autologging

### What interface(s) does this bug affect?

- [ ] `area/uiux`: Front-end, user experience, plotting, JavaScript, JavaScript dev server
- [ ] `area/docker`: Docker use across MLflow's components, such as MLflow Projects and MLflow Models
- [ ] `area/sqlalchemy`: Use of SQLAlchemy in the Tracking Service or Model Registry
- [ ] `area/windows`: Windows support

### What language(s) does this bug affect?

- [ ] `language/r`: R APIs and clients
- [ ] `language/java`: Java APIs and clients
- [ ] `language/new`: Proposals for new client languages

### What integration(s) does this bug affect?

- [ ] `integrations/azure`: Azure and Azure ML integrations
- [ ] `integrations/sagemaker`: SageMaker integrations
- [ ] `integrations/databricks`: Databricks integrations


Discussion:

Resolving patch:
diff --git a/mlflow/metrics/genai/genai_metric.py b/mlflow/metrics/genai/genai_metric.py
--- a/mlflow/metrics/genai/genai_metric.py
+++ b/mlflow/metrics/genai/genai_metric.py
@@ -28,10 +28,16 @@
 
 
 def _format_args_string(grading_context_columns: Optional[List[str]], eval_values, indx) -> str:
+    import pandas as pd
+
     args_dict = {}
     for arg in grading_context_columns:
         if arg in eval_values:
-            args_dict[arg] = eval_values[arg][indx]
+            args_dict[arg] = (
+                eval_values[arg].iloc[indx]
+                if isinstance(eval_values[arg], pd.Series)
+                else eval_values[arg][indx]
+            )
         else:
             raise MlflowException(
                 f"{arg} does not exist in the eval function {list(eval_values.keys())}."


Test patch:
diff --git a/tests/metrics/genai/test_genai_metrics.py b/tests/metrics/genai/test_genai_metrics.py
--- a/tests/metrics/genai/test_genai_metrics.py
+++ b/tests/metrics/genai/test_genai_metrics.py
@@ -769,6 +769,14 @@ def test_faithfulness_metric():
             examples=[mlflow_example],
         )
 
+    faithfulness_metric.eval_fn(
+        # Inputs with different indices
+        pd.Series([mlflow_prediction], index=[0]),
+        {},
+        pd.Series([input], index=[1]),
+        pd.Series([mlflow_ground_truth], index=[2]),
+    )
+
 
 def test_answer_correctness_metric():
     answer_correctness_metric = answer_correctness()