<figure><img src="image_1.png" /></figure>

HAL
open science

# MILE-RefHumEval: A Reference-Free, Multi-Independent LLM Framework for Human-Aligned Evaluation

Nalin Srun, Parisa Rastin, Guénaël Cabanes, Lydia Boudjeloud Assala

► To cite this version:

Nalin Srun, Parisa Rastin, Guénaël Cabanes, Lydia Boudjeloud Assala. MILE-RefHumEval: A Reference-Free, Multi-Independent LLM Framework for Human-Aligned Evaluation. 2025. hal-05427760

HAL Id: hal-05427760

https://hal.science/hal-05427760v1

Preprint submitted on 22 Dec 2025

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_2.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# MILE-RefHumEval: A Reference-Free, Multi-Independent LLM Framework for Human-Aligned Evaluation

Nalin Srun¹, Parisa Rastin¹, Guénaël Cabanes¹, Lydia Boudjeloud-Assala¹

¹LORIA, Université de Lorraine
{firstname.lastname}@univ-lorraine.fr

## Abstract

We introduce MILE-RefHumEval, a reference-free framework for evaluating Large Language Models (LLMs) without ground-truth annotations or evaluator coordination. It leverages an ensemble of independently prompted evaluators guided by a human-aligned schema, supporting both discrete and continuous scoring judgement. With task-specific prompts from best candidate selection, summarization and image captioning to dialogue, MILE-RefHumEval provides flexible, interpretable, and scalable assessments. Experiments show it aligns closely with human judgments, outperforms prior methods, and reduces computational overhead, offering an efficient, robust, and human-aligned solution for real-world LLM evaluation.

## 1 Introduction

Large language models (LLMs) have transformed NLP, enabling fluent generation, complex reasoning, and domain adaptation. However, evaluating LLMs, especially for tasks involving structured synthesis, factual accuracy, or domain-specific reasoning, remains difficult. Common metrics like BLEU (Papineni et al., 2002), ROUGE (Lin, 2004), METEOR (Banerjee and Lavie, 2005), and BERTScore (Zhang et al., 2020) depend on reference outputs and struggle with semantic nuance, dynamic placeholders, and compositional structure.

Recent LLM-as-judge approaches cooperative (Liang et al., 2024; Xu et al., 2023), competitive (Chan et al., 2023; Zhao et al., 2024a), and aggregation-based (Ning et al., 2025; Shu et al., 2024) have advanced the field but remain sensitive to evaluator bias, inter-model noise, or lack interpretability (Liu et al., 2024).

We introduce MILE-RefHumEval, a reference-free, multi-evaluator framework where each LLM assesses candidate responses independently using a shared, human-aligned judgment criteria. This

structure eliminates interaction bias and enhances objectivity. We aim to investigate how multiple evaluators independently assess a single role, highlighting the distinct reasoning and judgments that emerge without interaction.

Importantly, we designed the framework to support flexible evaluation across a wide range of tasks without the need for excessive time to redesign model architectures to adapt to new tasks. The framework allows effortless adaptation. Users simply modify the evaluation prompt criteria based on the specific task and swap the evaluator type. For example, image-to-text evaluators can be used for image captioning, and text-to-text evaluators can be used for summarization. This enables seamless evaluation across diverse modalities and tasks.

Our main contributions are: (1) a reference-free, task-agnostic evaluation framework suited for complex, open-ended tasks; (2) an unbiased scoring design using isolated LLM evaluators to prevent cross-influence; (3) task-dependent evaluation dimensions that can be flexibly adapted to different benchmarks; (4) evaluator-type interchangeability that enables users to seamlessly switch modalities; and (5) an efficient structure that reduces query load while preserving evaluation quality.

## 2 Related Work

Collaborative frameworks like ABSEval (Liang et al., 2024) use role-specialized agents, e.g., commonsense reasoners and code executors, to produce multi-perspective assessments. Others, such as peer-review-style systems (Xu et al., 2023), involve iterative critique and revision among agents, emulating human feedback cycles. While these methods enhance interpretability and simulate diverse reasoning, they often suffer from consensus bias, where shared context inflates agreement even on flawed outputs. Debate-style setups assess model quality through structured ad-
versarial exchanges. Auto-Arena (Zhao et al., 2024a) and CHATEVAL (Chan et al., 2023) have models argue under LLM supervision or peer-ranking. JudgeLM (Wang et al., 2024) and DebateSum (Zhang et al., 2023) use third-party judges to rate rhetorical strength. MORE and SAMRE (Bandi et al., 2025) introduce advocate roles and multi-round scoring. These setups better surface qualitative differences but risk dominance bias, where verbosity skews judgment and lowers inter-rater reliability. Ensemble and optimization-driven methods aim for robustness and cost-efficiency. PiCO (Ning et al., 2025) uses learnable weights to merge rankings from multiple LLMs, while AIME (Patel et al., 2024) assigns diverse roles to evaluators for adversarial robustness. PoLL (Verga et al., 2024) reduces reliance on large models via lightweight mixtures. Other work incorporates voting (Badshah and Sajjad, 2024), confidence-based cascades (Jung et al., 2025), and hybrid metrics (Shu et al., 2024), though many require references or are compute-heavy. Domain-specific solutions include recommendation evaluation (Zhao et al., 2024b) and autonomous exam setups (Bai et al., 2023). In contrast, MILE-RefHumEval introduces a novel evaluation paradigm both reference-free and interaction-free, relying on multiple independently prompted LLMs to assess a single role without cross-agent influence. This design explicitly avoids consensus or dominance biases introduced by conversational or role-overlapping setups, while still producing human-aligned, interpretable judgments. To our knowledge, it is the first framework to combine evaluator independence with a structured, multi-criteria schema in a fully decentralized manner.

## 3 Proposed Framework Design

As illustrated in Figure 1, MILE-RefHumEval adopts a reference-free, multi-stage procedure for evaluating LLM-generated responses. Each candidate output is independently evaluated by a diverse ensemble of evaluators using a unified set of evaluation dimensions designed to comprehensively assess response quality. The specific dimensions are task-dependent and can be flexibly adapted based on the evaluation benchmark.

For example, Figure 2 shows that in the candidate comparison task, the evaluation focuses on identifying the best candidate.

In image captioning (Figure 3), custom prompts

<figure><img src="image_3.png" /><figcaption>Figure 1: Overall Framework of MILE-RefHumEval.</figcaption></figure>

assess caption quality relative to images. Similarly, for summarization (Figure 4), evaluation focuses on coherence, consistency, fluency, and relevance. In the topical chat task (Figure 5), the evaluation prompt can assess dimensions like contextual understanding, naturalness, coherence, engagement, and groundedness.

The key advantage of MILE-RefHumEval lies in its flexibility and adaptability. Evaluation prompts can be easily customized to suit different tasks by simply redefining the evaluation criteria, without requiring extensive domain expertise. Additionally, evaluators can be dynamically added, removed, or replaced depending on the task requirements. In image captioning, an image-to-text LLM like InternVL can boost evaluation quality.

This framework enables researchers to define any number of evaluation dimensions with detailed criteria tailored to their objectives. Finally, regardless of the task, the final evaluation scores are computed through majority voting across evaluators, which enhances robustness and mitigates individual model bias. However, for tasks where evaluators provide numerical ratings such as summarization (Figure 4) or other continuous scoring benchmarks, the final score is obtained by averaging the ratings from all evaluators instead of majority voting. By
You are Evaluator "A", assigned to assess the quality of two candidate answers for the given question.

— Question & Candidate Answers —
{str(instruction)}

[The Start of Question]
{str(input)}
[The End of Question]
[The Start of Assistant 1's Answer]
{response1}
[The End of Assistant 1's Answer]
[The Start of Assistant 2's Answer]
{response2}
[The End of Assistant 2's Answer]

Which Assistant's answer is the best?

Final Best Assistant: "Assistant 1" or "Assistant 2" or "Equal"

<caption>Figure 2: Evaluation prompt used in MILE-RefHumEval for pairwise comparison of two candidate answers (PandaLM Benchmark).</caption>

You are Evaluator "A", assigned to assess the quality of "Caption Generation" from the "Image".

[The Start of Caption]
{caption}
[The End of Caption]

Is this a good caption for this image?

Final Score: (1=good, 0=bad)

<caption>Figure 3: Evaluation prompt used in MILE-RefHumEval for the Image Captioning benchmark to assess LLM-generated captions (OID Rated Image Caption Benchmark).</caption>

You are Evaluator "A", assigned to assess the quality of "Summarization" from the "Document text (News Article)".

— Document Text & Summary —
{document}

[The Start of Summary]
{summary}
[The End of Summary]

**1. Coherence (1–5)**

**Definition:** Logical flow and structure of the summary. A coherent summary is well-organized, with sentences connected naturally and presenting a clear progression of ideas.

**Steps:** 1. Read the news article carefully and identify the main topic and key points. 2. Read the summary and check whether it presents information in a logical order without contradictions or abrupt jumps. 3. Assign a score based on overall logical flow. **Final Score:** (number between 1 and 5)

**2. Consistency (1–5)**

**Definition:** Faithfulness of the summary to the source text. A consistent summary should not contradict, misrepresent, or invent details.

**Steps:** 1. Compare the summary's statements with the source text. 2. Check if the summary accurately reflects the article's facts without adding false information. 3. Assign a score based on factual accuracy. **Final Score:** (number between 1 and 5)

**3. Fluency (1–5)**

**Definition:** Readability and grammatical quality of the summary. A fluent summary uses correct grammar, natural phrasing, and smooth sentence transitions.

**Steps:** 1. Read the summary carefully, focusing on grammar, spelling, punctuation, and sentence flow. 2. Judge whether the writing is natural and easy to read. 3. Assign a score based on overall fluency. **Final Score:** (number between 1 and 5)

**4. Relevance (1–5)**

**Definition:** The degree to which the summary focuses on the important and central aspects of the article while avoiding unnecessary or trivial details.

**Steps:** 1. Identify the most important points in the article. 2. Check if the summary captures these main points while avoiding irrelevant or minor details. 3. Assign a score based on overall relevance. **Final Score:** (number between 1 and 5)

leveraging this decentralized, interaction-free approach, MILE-RefHumEval supports structured, interpretable, and scalable evaluation, even in scenarios where ground-truth reference answers are unavailable.

# 4 Experiment

## 4.1 Experimental Setup

FairEval Benchmark (Wang et al., 2023): The experiments were conducted on 80 samples from the Open-ended Question Answers dataset, comprising nine categories to ensure semantic and domain diversity. We benchmark MILE-RefHumEval against two baselines: CHATE-VAL (Chan et al., 2023), which employs a single evaluator in a multi-role configuration; and MILE-RefHumEval-Conv, a variant of conversational ensemble variant where evaluators sequentially review outputs which involves MILE-RefHumEval with inter-evaluator communication as in CHATE-VAL. MILE-RefHumEval uses seven diverse LLMs: DeepSeek-R1-Distill-Llama-8B, Mistral-Small-3.2-24B-Instruct-2506, GPT-4.1-mini, Meta-

<caption>Figure 4: Evaluation prompt used in MILE-RefHumEval for the Summarization benchmark to assess LLM-generated summaries (SummEval Benchmark).</caption>

Llama-3.1-8B-Instruct, Gemma-3-12b-it, Phi-4 and Qwen_qwq-32b as independent evaluators. These models range from 8B to 32B parameters and were selected to provide complementary architectural and linguistic perspectives. Though MILE-RefHumEval uses two more agents than CHATE-VAL, it requires fewer queries.

Evaluation metrics include Accuracy (Acc.), F1-score (F1), Cohen's Kappa (Kap.), Matthews Correlation Coefficient (MCC), and Query Efficiency to assess reliability, alignment, and cost-effectiveness.

SummEval Benchmark (Fabbri et al., 2021): We benchmark MILE-RefHumEval against G-Eval-4 (Liu et al., 2023) on a summarization task with 1,600 samples evaluated on coherence, consistency, fluency, and relevance (scored 1–5). Unlike previous benchmarks, which chose the best candidate,
<figure><img src="image_4.png" /><figcaption>Figure 5: Evaluation prompt used in MILE-RefHumEval for the Topical Chat benchmark to assess LLM-generated conversational responses (Topical Chat Benchmark).</figcaption></figure>

MILE-RefHumEval extends this by evaluating fine-
grained quality scores that closely approximate hu-
man evaluations. For this dataset experiment, we
used all seven evaluators, consistent with the setup
in the FairEval benchmark.

To assess how accurately MILE-RefHumEval
predicts human-like scores, we computed Mean
Squared Error (MSE), Root Mean Squared Error
(RMSE), and Mean Absolute Error (MAE). These
metrics quantify the variance and absolute differ-
ences between our predicted scores and human
annotations, providing a more precise evaluation
beyond simple correlation measures. Since G-Eval-
4 already provides evaluation scores result, we di-
rectly use its outputs for benchmarking without
rerunning the G-Eval original evaluation process.

**OID Rated Image Caption Benchmark** (Levinboim et al., 2019): We benchmarked MILE-RefHumEval, a flexible framework for image-to-text evaluation, on the OID Image Caption Quality Estimation Test dataset, which contains 6,182

samples with human-rated captions. Each sample
includes an image, a system-generated caption, and
the number of positive ratings, from which we com-
puted the rounded average of good ratings.

MILE-RefHumEval adapts easily to diverse
tasks by swapping in specialized evaluators.
Here, we use five: Mistral-Small-3.2-24B-Instruct-
2506, Gemma-3-12B-IT, InternVL3-14B-Instruct,
Qwen2.5-VL-7B-Instruct, and GPT-4.1-mini.

To evaluate MILE-RefHumEval’s accuracy, we
framed its judgments as a binary classification task:
a caption is labeled as “good” if its rounded average
positive rating exceeds 0.50 (i.e., more than 2 out
of 5 evaluators rated it positively). For comparison,
we employed ClipScore (Hessel et al., 2021) (CLIP-
ViT-L-14), using a cosine similarity threshold of
0.30 to separate satisfactory captions from unsat-
isfactory ones. This threshold aligns with recent
studies (Schuhmann et al., 2021; Birhane et al.,
2023), where values around 0.3–0.35 are com-
monly adopted for zero-shot caption quality evalu-
ation, balancing sensitivity and precision. Evaluation metrics include Acc., F1, Kap., MCC.

**PandaLM Benchmark** (Zhu et al., 2023): We also compared MILE-RefHumEval to JudgeLM and a PandaLM method specifically trained on the evaluation task dataset. The dataset consists of 1,000 samples, each covering diverse categories (e.g., Grammarly, Wikipedia, Coursera,...). Each sample includes an instruction, an input (on various topics), and two responses generated by the same model based on that instruction and input. Three annotators choose the best response.

MILE-RefHumEval use all seven evaluators to select the best response for each sample and then compute the agreement between MILE-RefHumEval's choice and the human annotations. Evaluation metrics include Acc., F1, Kap., MCC.

**Topical Chat Benchmark** (Mehri and Eskenazi, 2020): To compare MILE-RefHumEval with G-Eval, we used the USR: Dialog Quality Annotations dataset. This dataset provides annotations across five dimensions: Understandable, Uses Knowledge (0 or 1), Natural, Maintains Context, Engaging (1, 2, or 3), and Overall Quality (1, 2, 3, 4, or 5). We evaluate on the Amazon Topical-Chat datasets (360 samples), with 3 human evaluators.

Evaluation metrics include Spearman and Kendall correlation coefficients between MILE-RefHumEval's judgment scores and those of all three human annotators on Naturalness (average of understandability and naturalness), Coherence (main context), Groundedness (use of knowledge), and AVG (overall quality). This procedure enables a direct comparison with G-Eval-4, as reported in the original paper.

## 4.2 Results

**FairEval Benchmark Result:** Table 1 presents the comparison of the three evaluation strategies. The proposed MILE-RefHumEval approach achieves the highest overall performance. The full ensemble (DeepSeek, Mistral, GPT, LLaMA, Gemma, Phi, and Qwen) reaches 63.75% accuracy, 49.87% macro F1, 39.04% MCC, and 37.42% Cohen's kappa. This represents an absolute gain of +5% F1, +1.38% MCC, and +1.58% Kap. over CHATEVAL's best result, while reducing the number of queries by 320. These gains demonstrate the effectiveness of model diversity and independence in reducing bias and improving evaluation reliability.

While CHATEVAL is lightweight in design, it consistently underperforms MILE-RefHumEval.

Its best configuration, using GPT as the sole evaluator, yields 63.75% acc., 45.87% macro F1, 37.66% MCC, and a Kap. of 35.84%. These results indicate moderate alignment with human labels but limited robustness, likely due to role overloading and intra-agent bias. For CHATEVAL, selecting the best evaluator requires running all evaluators, which increases the number of queries. In contrast, for MILE-RefHumEval, we can simply combine the evaluators without increasing the query count.

Under MILE-RefHumEval-Conv, conversational evaluation exhibits significant variability depending on the order in which evaluators are applied, leading to differing performance across configurations. The top-performing variant reaches 61.25% acc., 43.67% macro F1, 32.51% MCC, and 30.80% kap., with GPT in the leading position. While CHATEVAL achieves its best results with GPT as the evaluator, in MILE-RefHumEval-Conv the highest performance occurs when the conversation is initiated by GPT, indicating that GPT strongly influences the overall evaluation outcome. This suggests that cross-evaluator dialogues introduce additional noise and potential bias, leading to inconsistent judgments rather than resolution or consensus. In terms of computational cost, MILE-RefHumEval is also more efficient. Even with two more agents than CHATEVAL, it requires only 560 LLM queries (80 examples × 7 evaluators), compared to 880 for CHATEVAL (80 × 11 roles) and up to 1760 for MILE-RefHumEval-Conv (80 × 22 steps). Thus, our method not only improves reliability and agreement but also reduces evaluation overhead by 36.4% relative to CHATEVAL and 68.2% relative to conversational ensembles.

<figure><img src="image_5.png" /><figcaption>Figure 6: Average accuracy for all possible combinations of *k* evaluators (binomial coefficient $\binom{N}{k}$).</figcaption></figure>
<table><caption>Table 1: **Evaluation on FairEval benchmark.** This table compares CHATEVAL, a single evaluator performing multiple roles through cross-role agent conversations, and MILE-RefHumEval-Conv, representing multi-evaluator conversations, with our MILE-RefHumEval approach, employing multiple independent evaluators without interaction. Metrics **Accuracy** (Acc.%), **Macro F1-score** (F1-ma.%), **Matthews Correlation Coefficient** (MCC%), **Cohen's Kappa** (Kap.), and **LLM Queries**, illustrate differences in evaluation effectiveness and computational cost.</caption><thead><tr><th>Evaluator</th><th>Acc.(%)</th><th>F1-ma.(%)</th><th>MCC(%)</th><th>Kap.</th><th>LLM Queries</th></tr></thead><tbody><tr><td colspan="6"><strong>CHATEVAL: One Evaluator, Many Roles (Cross-Role Agent Conversations)</strong></td></tr><tr><td>Deepseek</td><td>52.50</td><td>39.16</td><td>24.96</td><td>22.78</td><td></td></tr><tr><td>Mistral</td><td>45.00</td><td>33.26</td><td>14.77</td><td>12.48</td><td></td></tr><tr><td>GPT</td><td><strong>63.75</strong></td><td>45.87</td><td>37.66</td><td>35.84</td><td>880 (80×11)</td></tr><tr><td>Llama</td><td>52.50</td><td>39.05</td><td>31.06</td><td>25.51</td><td></td></tr><tr><td>Gemma</td><td>57.50</td><td>41.63</td><td>27.78</td><td>26.63</td><td></td></tr><tr><td>Phi</td><td>55.00</td><td>43.88</td><td>23.90</td><td>23.32</td><td></td></tr><tr><td>Qwen</td><td>62.50</td><td>47.79</td><td>34.62</td><td>33.05</td><td></td></tr><tr><td colspan="6"><strong>MILE-RefHumEval: One Role, Many Evaluators (No Conversations)</strong></td></tr><tr><td>DeepSeek+Mistral+GPT+Llama+Gemma+Phi+Qwen</td><td><strong>63.75</strong></td><td><strong>49.87</strong></td><td><strong>39.04</strong></td><td><strong>37.42</strong></td><td>560 (80×7)</td></tr><tr><td colspan="6"><strong>MILE-RefHumEval-Conv: One Role, Many Evaluators (With Conversations)</strong></td></tr><tr><td>DeepSeek→Mistral→GPT→Llama→Gemma→Phi→Qwen</td><td>58.75</td><td>42.84</td><td>32.66</td><td>30.38</td><td></td></tr><tr><td>Qwen→Mistral→GPT→Llama→Gemma→Phi→DeepSeek</td><td>58.75</td><td>42.49</td><td>28.82</td><td>27.97</td><td></td></tr><tr><td>Phi→Mistral→GPT→Llama→Gemma→Qwen→DeepSeek</td><td>58.75</td><td>42.42</td><td>30.05</td><td>28.57</td><td>1760 (80×22)</td></tr><tr><td>Llama→Mistral→GPT→Phi→Gemma→Qwen→DeepSeek</td><td>53.75</td><td>38.45</td><td>19.79</td><td>18.86</td><td></td></tr><tr><td>GPT→Mistral→Qwen→Llama→Gemma→Phi→DeepSeek</td><td>61.25</td><td>43.67</td><td>32.51</td><td>30.80</td><td></td></tr><tr><td>Gemma→DeepSeek→GPT→Llama→Qwen→Phi→Mistral</td><td>57.50</td><td>41.00</td><td>25.01</td><td>23.66</td><td></td></tr></tbody></table>

Figure 6 illustrates how mean accuracy and its associated standard error (SE) evolve as the number of evaluators ($k$) increases. Mean accuracy rises steadily with each additional evaluator, showing that aggregating multiple judgments systematically enhances evaluation reliability. A single evaluator performs poorly (54.82%), reflecting both low accuracy and high variability. With two evaluators, performance improves only marginally (55.12%), but beyond this point, the benefits become clear: by $k=3$, accuracy reaches 58.64%, and at $k=4$, it climbs to 60.00%. Additional evaluators continue to improve performance, although with diminishing returns, reaching 63.75% at $k=7$. This trend suggests convergence toward a stable, high-quality consensus with more evaluators.

The behavior of the SE values further clarifies this pattern. SE drops sharply as $k$ increases from 1 to 4 declining from 2.24 at $k=1$ to 0.90 at $k=2$, and stabilizing around 0.50 from $k=3$ onward. This indicates that variability among evaluator subsets decreases rapidly when multiple evaluators are combined. For higher $k$ values, SE remains consistently low (0.38–0.53), reflecting stable estimates, while at $k=7$ it becomes undefined since only one subset exists. When $\binom{n}{k}$ is large such as 35 subsets for $k=3$ and $k=4$ the uncertainty is averaged over many combinations, yielding robust and minimal SE values. Conversely, small subset counts (e.g., $k=1$) lead to more volatile SE due to limited sampling.

Overall, these results demonstrate that increasing the number of evaluators not only improves

<table><caption>Table 2: Performance comparison of MILE-RefHumEval and G-Eval on the SummEval benchmark. Metrics include MSE, RMSE, and MAE across Coherence, Consistency, Fluency, and Relevance. Lower values indicate better agreement with human evaluations.</caption><thead><tr><th colspan="4">MILE-RefHumEval</th></tr><tr><th>Dimension</th><th>MSE</th><th>RMSE</th><th>MAE</th></tr></thead><tbody><tr><td>Coherence</td><td>0.927</td><td>0.963</td><td>0.761</td></tr><tr><td>Consistency</td><td>0.596</td><td>0.772</td><td>0.567</td></tr><tr><td>Fluency</td><td>0.641</td><td>0.801</td><td>0.685</td></tr><tr><td>Relevance</td><td>0.495</td><td>0.704</td><td>0.557</td></tr></tbody><thead><tr><th colspan="4">G-Eval-4</th></tr><tr><th>Dimension</th><th>MSE</th><th>RMSE</th><th>MAE</th></tr></thead><tbody><tr><td>Coherence</td><td>1.258</td><td>1.122</td><td>0.910</td></tr><tr><td>Consistency</td><td>1.849</td><td>1.360</td><td>1.083</td></tr><tr><td>Fluency</td><td>4.473</td><td>2.115</td><td>2.037</td></tr><tr><td>Relevance</td><td>0.978</td><td>0.989</td><td>0.811</td></tr></tbody></table>

expected accuracy but also makes poor outcomes less likely to occur by chance. This is especially important in settings where the individual quality or reliability of evaluators is uncertain.

**SummEval Benchmark Result:** Table 2 shows that MILE-RefHumEval consistently outperforms G-Eval-4 across all evaluation dimensions, achieving substantially lower MSE, RMSE, and MAE scores. For example, MILE-RefHumEval reduces MSE by 26.3% in Coherence (0.927 vs. 1.258), 67.7% in Consistency (0.596 vs. 1.849), and an impressive 85.7% in Fluency (0.641 vs. 4.473). Relevance also improves significantly, with an MSE of 0.495, nearly 50% lower than G-Eval-4's 0.978. Moreover, the variance of the proposed approach's
predictions remains consistently below 1, demonstrating more stable, accurate, and human-aligned evaluations.

<table><caption>Table 3: Evaluation on the OID Rated Image Caption Benchmark. We compare the ensemble MILE-RefHumEval with individual evaluators and the non-LLM baseline ClipScore.</caption><thead><tr><th>Method</th><th>Acc.</th><th>F1</th><th>Kap.</th><th>MCC</th></tr></thead><tbody><tr><td>MILE-RefHumEval</td><td>58.91</td><td><strong>57.97</strong></td><td><strong>22.44</strong></td><td><strong>26.67</strong></td></tr><tr><td>Mistral</td><td><strong>62.42</strong></td><td>55.35</td><td>17.41</td><td>21.28</td></tr><tr><td>Gemma</td><td>56.63</td><td>55.31</td><td>18.60</td><td>22.82</td></tr><tr><td>InternVL</td><td>57.31</td><td>56.85</td><td>13.80</td><td>13.84</td></tr><tr><td>QwenVL</td><td>50.92</td><td>46.09</td><td>10.80</td><td>17.69</td></tr><tr><td>GPT</td><td>56.96</td><td>54.99</td><td>19.97</td><td>26.05</td></tr><tr><td>ClipScore</td><td>43.27</td><td>30.37</td><td>0.14</td><td>2.23</td></tr></tbody></table>

**OID Rated Image Caption Benchmark Result:** Table 3 shows that MILE-RefHumEval, an ensemble of Mistral, Gemma, InternVL, QwenVL, and GPT, achieves 58.91% Accuracy, slightly below Mistral alone (62.42%), but surpasses all individual evaluators in Macro F1 (57.97%), kap. (22.44%), and MCC (26.67%), indicating more balanced and human-aligned predictions. Single evaluator like Mistral excel in raw Accuracy, while InternVL maintains strong class-wise performance (Macro F1 56.85%). QwenVL underperforms across metrics, and ClipScore serves as a low baseline (Accuracy 43.27%, Kappa 0.14%). Notably, even on image captioning tasks, the ensemble achieves superior overall performance under the same evaluation criteria, demonstrating that combining evaluators leads to more reliable and robust results than relying on any single evaluator alone. These results highlight that ensembling evaluators improves reliability and alignment with human judgments.

**PandaLM Benchmark Result:** Table 4 reports the performance of MILE-RefHumEval, PandaLM-7B, and JudgeLM-7B across three independent annotators and their majority vote on the PandaLM benchmark. MILE-RefHumEval consistently outperforms both baselines across all metrics and annotators, achieving the highest overall performance with an average Acc. of 78.98%, Macro F1 of 71.90%, Kap. of 63.71%, and MCC of 63.85% under majority voting. In comparison, PandaLM-7B performs moderately well (Accuracy 66.77%, F1 57.43%), while JudgeLM-7B lags significantly behind (Accuracy 57.76%, F1 40.24%).

Across individual annotators, MILE-RefHumEval maintains consistently high Accuracy (76.98–78.48%) and Macro F1 (69.76–71.58%),

<table><caption>Table 4: Evaluation on the PandaLM benchmark using three human annotators and their majority vote. We compare MILE-RefHumEval, PandaLM-7B, and JudgeLM-7B across Accuracy, Macro F1, kap., and MCC. PandaLM-7B results are computed from the official repository outputs, while JudgeLM-7B was downloaded and evaluated locally using the same settings as MILE-RefHumEval.</caption><thead><tr><th>Method</th><th>Acc. (%)</th><th>F1 (%)</th><th>Kap.</th><th>MCC (%)</th></tr></thead><tbody><tr><td colspan="5"><strong>Annotator 1</strong></td></tr><tr><td>MILE-RefHumEval</td><td><strong>77.78</strong></td><td><strong>70.76</strong></td><td><strong>61.43</strong></td><td><strong>61.52</strong></td></tr><tr><td>PandaLM-7B</td><td>65.97</td><td>56.03</td><td>41.91</td><td>41.92</td></tr><tr><td>JudgeLM-7B</td><td>57.46</td><td>39.79</td><td>23.69</td><td>26.40</td></tr><tr><td colspan="5"><strong>Annotator 2</strong></td></tr><tr><td>MILE-RefHumEval</td><td><strong>76.98</strong></td><td><strong>69.76</strong></td><td><strong>60.54</strong></td><td><strong>60.73</strong></td></tr><tr><td>PandaLM-7B</td><td>65.17</td><td>56.24</td><td>41.19</td><td>41.20</td></tr><tr><td>JudgeLM-7B</td><td>56.66</td><td>39.69</td><td>23.59</td><td>26.45</td></tr><tr><td colspan="5"><strong>Annotator 3</strong></td></tr><tr><td>MILE-RefHumEval</td><td><strong>78.48</strong></td><td><strong>71.58</strong></td><td><strong>63.04</strong></td><td><strong>63.26</strong></td></tr><tr><td>PandaLM-7B</td><td>66.97</td><td>58.55</td><td>44.12</td><td>44.15</td></tr><tr><td>JudgeLM-7B</td><td>56.16</td><td>39.31</td><td>22.88</td><td>25.78</td></tr><tr><td colspan="5"><strong>Human Annotator Majority Vote</strong></td></tr><tr><td>MILE-RefHumEval</td><td><strong>78.98</strong></td><td><strong>71.90</strong></td><td><strong>63.71</strong></td><td><strong>63.85</strong></td></tr><tr><td>PandaLM-7B</td><td>66.77</td><td>57.43</td><td>43.54</td><td>43.55</td></tr><tr><td>JudgeLM-7B</td><td>57.76</td><td>40.24</td><td>24.82</td><td>27.75</td></tr></tbody></table>

showing strong robustness to annotator disagreement. The improvements over PandaLM-7B are substantial, averaging +12.2 points in Accuracy, +14.5 in Macro F1, and +20.2 in kap., demonstrating significantly stronger agreement with human judgments. JudgeLM-7B's low Kap. scores (22.88–26.45%) confirm its weaker alignment with human preferences.

**Amazon Topical-Chat Benchmark Result:** Table 5 shows MILE-RefHumEval outperforms G-Eval across all dialogue evaluation dimensions, yielding the highest correlations with human judgments. Specifically, compared to G-Eval, MILE-RefHumEval improves Naturalness correlations by $r = 0.257$ and $\rho = 0.185$, Coherence by $r = 0.211$ and $\rho = 0.134$, Engagingness by $r = 0.119$ and $\rho = 0.052$, and Groundedness by $r = 0.332$ and $\rho = 0.262$, respectively, demonstrating clear gains in capturing naturalness, coherence, engagement, and factual consistency.

On average, MILE-RefHumEval achieves $r = 0.117$ and $\rho = 0.024$ improvements over G-Eval, confirming that our framework provides the most reliable and human-aligned evaluations across all dimensions. These results strongly validate MILE-RefHumEval as the state-of-the-art method for reference-free dialogue evaluation.

# 5 Discussion

Our results reveal a surprising yet consistent trend: independent, non-conversational evaluator
<table><caption>Table 5: Average Correlation Scores for MILE-RefHumEval Across Dimensions on the Amazon Topical-Chat benchmark. *G-Eval correlation score is directly from (Liu et al., 2023)</caption><thead><tr><th rowspan="2">Method</th><th colspan="2">Naturalness</th><th colspan="2">Coherence</th><th colspan="2">Engagingness</th><th colspan="2">Groundedness</th><th colspan="2">AVG</th></tr><tr><th>r</th><th>ρ</th><th>r</th><th>ρ</th><th>r</th><th>ρ</th><th>r</th><th>ρ</th><th>r</th><th>ρ</th></tr></thead><tbody><tr><td>MILE-RefHumEval</td><td><strong>0.806</strong></td><td><strong>0.750</strong></td><td><strong>0.805</strong></td><td><strong>0.739</strong></td><td><strong>0.746</strong></td><td><strong>0.683</strong></td><td><strong>0.863</strong></td><td><strong>0.813</strong></td><td><strong>0.692</strong></td><td><strong>0.612</strong></td></tr><tr><td>G-Eval*</td><td>0.549</td><td>0.565</td><td>0.594</td><td>0.605</td><td>0.627</td><td>0.631</td><td>0.531</td><td>0.551</td><td>0.575</td><td>0.588</td></tr></tbody></table>

ensembles outperform conversational configurations across all metrics (Table 1). While conversational paradigms are often assumed to emulate human collaborative reasoning, our analysis shows they introduce inter-model bias, where earlier model outputs unduly influence subsequent evaluators. This undermines objectivity and reduces alignment with human judgments. Crucially, the order and how much it strong influence in which models participate in conversations significantly affects outcomes.

**Diversity matters more than size.** The results suggest that diversity of evaluators contributes more to human alignment than simply using a larger or stronger single LLM. For example, while GPT alone achieved high accuracy in some tasks, the ensemble's balanced performance across dimensions like coherence, fluency, and engagement shows that a mix of reasoning styles outperforms raw scale. This implies that evaluation reliability stems not from model strength alone, but from the complementarity of evaluators.

**Order sensitivity exposes hidden fragility in conversational evaluators.** The poor stability of conversational ensembles reveals an overlooked issue: the order in which evaluators “speak” acts as a hidden variable that can sway results dramatically. This is analogous to psychological priming in human group discussions, where the first speaker heavily shapes consensus. By removing order effects, MILE-RefHumEval uncovers that evaluator independence is essential for trustworthy automation a point rarely considered in prior benchmarks.

**Fluency is the hardest dimension for evaluators to approximate.** In the SummEval results, fluency showed the largest error reduction (85.7%) compared to baselines, suggesting that models struggle most with accurately judging naturalness and readability. That MILE-RefHumEval narrowed this gap highlights that diverse evaluators collectively capture surface-level language features better than individual judges.

**Single models can “spike” in performance, but ensembles stabilize.** In the Image Caption benchmark, Mistral alone slightly outperformed the en-

semble in raw accuracy, but its gains came at the cost of much lower agreement metrics. This finding mirrors the idea of “overfitting” in supervised learning where one evaluator might excel on certain samples but lack generalizable reliability. Ensembles act as a variance-reduction mechanism, producing more human-like stability even if they sometimes underperform on headline accuracy.

**Where humans disagree, ensemble evaluators thrive.** In PandaLM, human annotators disagreed on many samples, yet MILE-RefHumEval maintained consistently high agreement across annotators. This suggests that ensembles may not just replicate human judgment, but actually mediate between conflicting human opinions, offering a form of consensus-building absent in individual LLM judges. That makes ensembles not just a human-aligned tool, but potentially a human-reflective evaluator capturing the “average” of human perspectives more reliably than any one person.

# 6 Conclusion

MILE-RefHumEval introduces a reference-free simple yet powerful shift in LLM evaluation: independence over interaction. By leveraging diverse, non-conversational ensembles, the framework reduces bias, stabilizes judgments, and achieves stronger alignment with human evaluations across tasks and modalities. Our findings highlight that evaluation quality stems not from mimicking human debate, but from diversity, independence, and structured criteria. As such, MILE-RefHumEval offers a scalable and reliable foundation for future human-aligned assessment of LLMs.

Looking ahead, the benchmark should be extended to include audio and speech-based tasks (e.g., MMAU benchmark (Sakshi et al., 2024) for sound, music, and other audio modalities). Incorporating specialized evaluators such as Audio Flamingo 3, Qwen 2.5 Omni, and Gemini 2.5 Pro will help determine whether these enhancements improve evaluation reliability. Identifying how many evaluators maximize accuracy without redundancy is an important future direction.
## Limitations

**Evaluator Generalization:** Our framework assumes uniform evaluator reliability across tasks, yet LLMs often exhibit domain-specific capabilities. Without mechanisms for domain-aware weighting or specialization, our current design may miss contextual subtleties in judgment, limiting its generalization across varied evaluation domains.

**Interaction Bias:** The fixed turn order in our conversational protocols introduces directional bias, where early-stage outputs disproportionately shape subsequent evaluations. This effect is especially pronounced when weaker models initiate the sequence. While we quantify this cascading influence, we do not explore adaptive, randomized, or model-quality-aware turn ordering as potential mitigation strategies.

**Metric Granularity:** We rely on standard agreement metrics (e.g., Pearson correlation, kap.), which may fail to capture nuanced shifts in evaluative reasoning introduced through interaction. Richer analytical tools such as causal attribution methods, latent trajectory comparisons, or fine-grained human annotations are needed to trace how evaluators influence one another.

**Prompt Sensitivity:** We observe significant variation in model behavior across prompt formulations, particularly among smaller models like Phi and Mistral. This sensitivity challenges reproducibility and points to the need for prompt-invariant evaluation strategies or dynamic prompt calibration mechanisms to ensure fairness and stability.

**Unexplored Optimal Evaluator Count:** We have not studied how many evaluators are needed to achieve maximal accuracy without introducing redundancy. Determining this optimal ensemble size remains an open question for future work.

**Domain Generalization:** MILE-RefHumEval has been tested primarily on general-purpose tasks such as summarization, captioning, and dialogue. Its reliability in highly specialized domains (e.g., legal, medical, or technical reasoning) remains untested, and model evaluators trained on general data may misjudge domain-specific quality.

## References

Sher Badshah and Hassan Sajjad. 2024. Reference-guided verdict: Llms-as-judges in automatic evaluation of free-form text. *arXiv preprint arXiv:2408.09235*.

Yushi Bai, Jiahao Ying, Yixin Cao, Xin Lv, Yuze He, Xiaozhi Wang, Jifan Yu, Kaisheng Zeng, Yijia Xiao, Haozhe Lyu, Jiayin Zhang, Juanzi Li, and Lei Hou. 2023. Benchmarking foundation models with language-model-as-an-examiner. In *Advances in Neural Information Processing Systems*, volume 36, pages 78142–78167. Curran Associates, Inc.

Chaithanya Bandi, Hari Bandi, and Abir HARRASSE. 2025. Adversarial multi-agent evaluation of large language models through iterative debate.

Satanjeev Banerjee and Alon Lavie. 2005. METEOR: An automatic metric for MT evaluation with improved correlation with human judgments. In *Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization*, pages 65–72, Ann Arbor, Michigan. Association for Computational Linguistics.

Abeba Birkhane, Sanghyun Han, Vishnu Boddeti, Sasha Luccioni, and 1 others. 2023. Into the laion's den: Investigating hate in multimodal datasets. *Advances in Neural Information Processing Systems*, 36:21268–21284.

Chi-Min Chan, Weize Chen, Yusheng Su, Jianxuan Yu, Wei Xue, Shanghang Zhang, Jie Fu, and Zhiyuan Liu. 2023. Chateval: Towards better llm-based evaluators through multi-agent debate. *The Twelfth International Conference on Learning Representations (ICLR 2024)*.

Alexander R Fabbri, Wojciech Kryściński, Bryan McCann, Caiming Xiong, Richard Socher, and Dragomir Radev. 2021. Summeval: Re-evaluating summarization evaluation. *Transactions of the Association for Computational Linguistics*, 9:391–409.

Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi. 2021. Clipscore: A reference-free evaluation metric for image captioning. *arXiv preprint arXiv:2104.08718*.

Jaehun Jung, Faeze Brahman, and Yejin Choi. 2025. Trust or escalate: LLM judges with provable guarantees for human agreement. In *The Thirteenth International Conference on Learning Representations*.

Tomer Levinboim, Ashish V Thapliyal, Piyush Sharma, and Radu Soricut. 2019. Quality estimation for image captions based on large-scale human evaluations. *arXiv preprint arXiv:1909.03396*.

Sirui Liang, Baoli Zhang, Jun Zhao, and Kang Liu. 2024. Abseval: An agent-based framework for script evaluation. In *Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing*, pages 12418–12434.
Chin-Yew Lin. 2004. Rouge: A package for automatic evaluation of summaries. In *Text summarization branches out*, pages 74–81.

Xinyi Liu, Pinxin Liu, and Hangfeng He. 2024. An empirical analysis on large language models in debate evaluation. In *Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)*, pages 470–487, Bangkok, Thailand. Association for Computational Linguistics.

Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu. 2023. G-eval: NLG evaluation using gpt-4 with better human alignment. In *Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing*, pages 2511–2522, Singapore. Association for Computational Linguistics.

Shikib Mehri and Maxine Eskenazi. 2020. USR: An unsupervised and reference free evaluation metric for dialog generation. In *Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics*, pages 681–707, Online. Association for Computational Linguistics.

Kun-Peng Ning, Shuo Yang, Yuyang Liu, Jia-Yu Yao, Zhenhui Liu, Yonghong Tian, Yibing Song, and Li Yuan. 2025. PiCO: Peer review in LLMs based on consistency optimization. In *The Thirteenth International Conference on Learning Representations*.

Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In *Proceedings of the 40th annual meeting of the Association for Computational Linguistics*, pages 311–318.

Bhrij Patel, Souradip Chakraborty, Wesley A. Suttle, Mengdi Wang, Amrit Singh Bedi, and Dinesh Manocha. 2024. AIME: AI system optimization via multiple LLM evaluators.

Sakshi, Utkarsh Tyagi, Sonal Kumar, Ashish Seth, Ramaneswaran Selvakumar, Oriol Nieto, Ramani Duraiswami, Sreyan Ghosh, and Dinesh Manocha. 2024. Mmau: A massive multi-task audio understanding and reasoning benchmark. *arXiv preprint arXiv:2410.19168*.

Christoph Schuhmann, Richard Vencu, Romain Beaumont, Robert Kaczmarczyk, Clayton Mullis, Aarush Katta, Theo Coombes, Jenia Jitsev, and Aran Komatsuzaki. 2021. Laion-400m: Open dataset of clip-filtered 400 million image-text pairs. *arXiv preprint arXiv:2111.02114*.

Lei Shu, Nevan Wichers, Liangchen Luo, Yun Zhu, Yinxiao Liu, Jindong Chen, and Lei Meng. 2024. Fusion-eval: Integrating assistant evaluators with LLMs. In *Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing: Industry Track*, pages 225–238, Miami, Florida, US. Association for Computational Linguistics.

Pat Verga, Sebastian Hofstatter, Sophia Althammer, Yixuan Su, Aleksandra Piktus, Arkady Arkhangorodsky, Minjie Xu, Naomi White, and Patrick Lewis. 2024. Replacing judges with juries: Evaluating llm generations with a panel of diverse models. *Preprint*, arXiv:2404.18796.

Linjun Wang, Bowen Zhang, Xurui Zhang, Xiang Ren, and Kai-Wei Chang. 2024. Judgelm: Pairwise judgment of question-answering pairs using large language models. In *Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics (NAACL)*.

Peiyi Wang, Lei Li, Liang Chen, Dawei Zhu, Binghuai Lin, Yunbo Cao, Qi Liu, Tianyu Liu, and Zhifang Sui. 2023. Large language models are not fair evaluators. *ArXiv*, abs/2305.17926.

Zhenran Xu, Senbao Shi, Baotian Hu, Jindi Yu, Dongfang Li, Min Zhang, and Yuxiang Wu. 2023. Towards reasoning in large language models via multi-agent peer review collaboration. *arXiv preprint arXiv:2311.08152*.

Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2020. Bertscore: Evaluating text generation with bert. In *International Conference on Learning Representations*.

Ziyou Zhang, Yuchen Duan, Yuanhe Tian, Luke Zettlemoyer, and Xi Victoria Liu. 2023. Debatesum: Evaluating summarization via structured debates. In *Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing (EMNLP)*.

Ruochen Zhao, Wenxuan Zhang, Yew Ken Chia, Weiwen Xu, Deli Zhao, and Lidong Bing. 2024a. Autoarena: Automating llm evaluations with agent peer battles and committee discussions. *arXiv preprint arXiv:2405.20267*.

Zihuai Zhao, Wenqi Fan, Jiatong Li, Yunqing Liu, Xi-aowei Mei, Yiqi Wang, Zhen Wen, Fei Wang, Xiangyu Zhao, Jiliang Tang, and Qing Li. 2024b. Recommender systems in the era of large language models (llms). *IEEE Trans. on Knowl. and Data Eng.*, 36(11):6889–6907.

Lianghui Zhu, Xinggang Wang, and Xinlong Wang. 2023. Judgelm: Fine-tuned large language models are scalable judges. *arXiv preprint arXiv:2310.17631*.
A Appendices

Please evaluate the accuracy, clarity, and relevance of the answer generated for the following question. Provide percent-
age scores along with detailed explanations for the following criteria:

1. Answer Relevance:

- Score: [Provide score between 0 and 100]

- Explanation: [Assess the clarity of the response. Is the answer easy to understand and logically structured? Are there any ambiguities or confusing phrases that could hinder comprehension?]

2. Depth and Completeness:

- Score: [Provide score between 0 and 100]

- Explanation: [Evaluate the depth of the response. Does it provide a thorough explanation, or does it lack detail? Are all key aspects of the question addressed adequately?]

3. Grammar and Linguistic Accuracy:

- Score: [Provide score between 0 and 100]

- Explanation: [Assess the grammatical correctness of the response. Are there any spelling or punctuation errors? Is the sentence structure correct and appropriate for the context?]

4. Contextual Appropriateness:

- Score: [Provide score between 0 and 100]

- Explanation: [Evaluate how well the answer adapts to the specific question category (e.g., "generic" or "knowl-edge"). Does it use relevant terminology and concepts appropriate to the subject matter?]

5. Conciseness and Precision:

- Score: [Provide score between 0 and 100]

- Explanation: [Evaluate whether the response is concise and to the point. Does the answer provide the necessary details without unnecessary elaboration or repetition?]

6. Creativity and Insight:

- Score: [Provide score between 0 and 100]

- Explanation: [Assess the creativity of the response. Does it offer a unique perspective or innovative solution, especially when dealing with complex or thought-provoking questions?]

7. Bias and Fairness:

- Score: [Provide score between 0 and 100]

- Explanation: [Evaluate the response for any signs of bias or unfairness. Does the answer exhibit neutrality, or does it reflect a particular perspective that may be considered biased or imbalanced?]

8. Knowledge Accuracy:

- Score: [Provide score between 0 and 100]

- Explanation: [Assess the accuracy of factual information presented in the answer. Are the facts correct? Are any misconceptions, errors, or outdated information included?]

9. Adaptability to Various Categories:

- Score: [Provide score between 0 and 100]

- Explanation: [Evaluate how well the answer adapts to different categories (e.g., "generic" vs. "knowledge"). Does the response adjust its tone, depth, and complexity based on the question category?]

10. Scalability of Responses:

- Score: [Provide score between 0 and 100]

- Explanation: [Evaluate how well the model handles a variety of question types and complexities. Does the model handle simple and complex questions effectively, or does its performance degrade with more challenging queries?]

11. Overall Answer Quality:

- Score: [Provide score between 0 and 100]

- Explanation: [Provide a final summary explanation for the overall score (Explain how it is calculated to show the calculation.) Consider how well all individual criteria performed. Offer insight into the general quality of the response based on the evaluation results.]

12. Error Detection and Correction:

Please analyze the provided response and identify any errors, inconsistencies, or areas where the response could be
improved. The model should help by suggesting a corrected version of the output, focusing on the following points:

- Identifying any missing or inaccurate information.

- Correcting any grammatical or linguistic errors.

- Ensuring the relevance and completeness of the answer.

- Suggesting any improvements in clarity or coherence.

- Making the answer more concise and precise where necessary.

13. Corrected Version:

- Updated Answer: [Provide a corrected, revised version of the answer based on the analysis above.]

Figure 7: Evaluation prompt on FairEval Benchmark used in MILE-RefHumEval to assess LLM-generated answers. The prompt guides each evaluator to score responses across 12 dimensions, including relevance, completeness, grammar, factual accuracy, bias, **Overall Answer Quality (Automatically computed by each evaluator as the average across the 10 preceding criteria)**, and finally provide justification and revisions. This structured criteria ensures consistent, multi-faceted evaluation without human intervention.
<figure><img src="image_6.png" /><figcaption>Figure 8: Pearson correlation between multiple LLM evaluators under MILE-RefHumEval-Conv. The figure highlights varying levels of agreement depending on model order and interaction style, suggesting that conversational sequences introduce inconsistency.</figcaption></figure>
<figure><img src="image_7.png" /><figcaption>Figure 9: Pearson correlation produced by a single evaluator (CHATEVAL) across different role-played personas. This figure demonstrates that repeating the same model across multiple roles can inflate agreement artificially, reducing diversity of judgment and interpretive value.</figcaption></figure>
<figure><img src="image_8.png" /><figcaption>Figure 10: Inter-evaluator agreement analysis in MILE-RefHumEval, showing variance and correlation among independently operating LLMs. The figure emphasizes that non-conversational, diverse evaluators lead to broader reasoning coverage and stronger alignment with human assessments compared to role-based or interactive setups.</figcaption></figure>

<table><caption>Table 6: Evaluation of six response-generation systems (S1–S6) on the Amazon Topical-Chat benchmark across six quality dimensions. Three human annotators rated the responses, and MILE-RefHumEval automatically predicted scores for the same responses. Accuracy and Weighted F1 show how closely MILE-RefHumEval aligns with human judgments, demonstrating its effectiveness in approximating human evaluation.</caption><thead><tr><th rowspan="2">Dimension</th><th colspan="2">S1</th><th colspan="2">S2</th><th colspan="2">S3</th><th colspan="2">S4</th><th colspan="2">S5</th><th colspan="2">S6</th></tr><tr><th>Acc</th><th>F1</th><th>Acc</th><th>F1</th><th>Acc</th><th>F1</th><th>Acc</th><th>F1</th><th>Acc</th><th>F1</th><th>Acc</th><th>F1</th></tr></thead><tbody><tr><th colspan="14" style="text-align: center;">MILE-RefHumEval vs. Human Quality Judgement 1 (sm)</th></tr><tr><td>Understandable</td><td>98.33</td><td>97.51</td><td>78.33</td><td>77.65</td><td>75.00</td><td>75.56</td><td>78.33</td><td>78.81</td><td>80.00</td><td>80.28</td><td><strong>100</strong></td><td><strong>100</strong></td></tr><tr><td>Natural</td><td>60.00</td><td>48.62</td><td>46.67</td><td>45.93</td><td>48.33</td><td>47.85</td><td>53.33</td><td>53.43</td><td>51.67</td><td>55.56</td><td><strong>93.33</strong></td><td><strong>90.11</strong></td></tr><tr><td>Maintains Context</td><td>68.33</td><td>60.29</td><td>50.00</td><td>48.68</td><td>55.00</td><td>56.31</td><td>38.33</td><td>37.60</td><td>41.67</td><td>42.58</td><td><strong>73.33</strong></td><td><strong>62.05</strong></td></tr><tr><td>Engaging</td><td>51.67</td><td>45.08</td><td>38.33</td><td>38.41</td><td>48.33</td><td>50.24</td><td>56.67</td><td><strong>61.16</strong></td><td>56.67</td><td>61.98</td><td><strong>65.00</strong></td><td>52.80</td></tr><tr><td>Uses Knowledge</td><td>60.00</td><td>56.94</td><td>68.33</td><td>72.84</td><td><strong>78.33</strong></td><td>80.70</td><td>75.00</td><td>79.93</td><td><strong>78.33</strong></td><td><strong>81.13</strong></td><td>63.33</td><td>50.73</td></tr><tr><td>Overall Quality</td><td>40.00</td><td>31.01</td><td>43.33</td><td>41.44</td><td>36.67</td><td>38.29</td><td>50.00</td><td><strong>51.91</strong></td><td>45.00</td><td>46.41</td><td><strong>55.00</strong></td><td>43.23</td></tr><tr><th colspan="14" style="text-align: center;">MILE-RefHumEval vs. Human Quality Judgement 2 (me)</th></tr><tr><td>Understandable</td><td>91.67</td><td>94.06</td><td>68.33</td><td>69.33</td><td>55.00</td><td>56.69</td><td>63.33</td><td>64.33</td><td>61.67</td><td>63.14</td><td><strong>98.33</strong></td><td><strong>99.16</strong></td></tr><tr><td>Natural</td><td>55.00</td><td>47.10</td><td>58.33</td><td>58.38</td><td>50.00</td><td>49.79</td><td>43.33</td><td>41.43</td><td>48.33</td><td>45.70</td><td><strong>95.00</strong></td><td><strong>95.48</strong></td></tr><tr><td>Maintains Context</td><td>68.33</td><td>64.22</td><td>63.33</td><td>64.16</td><td>50.00</td><td>50.56</td><td>56.67</td><td>56.09</td><td>51.67</td><td>52.47</td><td><strong>78.33</strong></td><td><strong>72.27</strong></td></tr><tr><td>Engaging</td><td>55.00</td><td>50.12</td><td>66.67</td><td><strong>68.02</strong></td><td>61.67</td><td>62.72</td><td>56.67</td><td>58.74</td><td>61.67</td><td>64.70</td><td><strong>73.33</strong></td><td>67.45</td></tr><tr><td>Uses Knowledge</td><td>63.33</td><td>61.84</td><td>65.00</td><td>69.98</td><td><strong>73.33</strong></td><td><strong>76.30</strong></td><td>70.00</td><td>75.84</td><td>70.00</td><td>74.18</td><td>65.00</td><td>54.17</td></tr><tr><td>Overall Quality</td><td>31.67</td><td>28.45</td><td>40.00</td><td>44.01</td><td>28.33</td><td>28.30</td><td>28.33</td><td>26.96</td><td>36.67</td><td>40.57</td><td><strong>61.67</strong></td><td><strong>54.17</strong></td></tr><tr><th colspan="14" style="text-align: center;">MILE-RefHumEval vs. Human Quality Judgement 3 (er)</th></tr><tr><td>Understandable</td><td>90.00</td><td>93.16</td><td>71.67</td><td>73.06</td><td>66.67</td><td>68.19</td><td>71.67</td><td>72.07</td><td>70.00</td><td>71.09</td><td><strong>98.33</strong></td><td><strong>99.16</strong></td></tr><tr><td>Natural</td><td>63.33</td><td>60.05</td><td>61.67</td><td>61.65</td><td>53.33</td><td>53.26</td><td>58.33</td><td>58.42</td><td>63.33</td><td>63.37</td><td><strong>81.67</strong></td><td><strong>84.76</strong></td></tr><tr><td>Maintains Context</td><td>68.33</td><td>67.58</td><td>60.00</td><td>62.07</td><td>56.67</td><td>60.05</td><td>56.67</td><td>57.40</td><td>58.33</td><td>58.02</td><td><strong>76.67</strong></td><td><strong>71.46</strong></td></tr><tr><td>Engaging</td><td>75.00</td><td>70.77</td><td>66.67</td><td>67.59</td><td>60.00</td><td>60.74</td><td>66.67</td><td>66.81</td><td>65.00</td><td>67.14</td><td><strong>76.67</strong></td><td><strong>73.58</strong></td></tr><tr><td>Uses Knowledge</td><td>65.00</td><td>64.27</td><td>61.67</td><td>67.12</td><td>75.00</td><td>77.81</td><td><strong>76.67</strong></td><td><strong>81.21</strong></td><td>68.33</td><td>72.80</td><td>70.00</td><td>63.29</td></tr><tr><td>Overall Quality</td><td>56.67</td><td>52.93</td><td>46.67</td><td>49.37</td><td>41.67</td><td>42.28</td><td>53.33</td><td>52.49</td><td>43.33</td><td>44.83</td><td><strong>61.67</strong></td><td><strong>56.98</strong></td></tr></tbody></table>
<figure><img src="image_9.png" /><figcaption>Figure 11: Illustration of CHATEVAL, a baseline evaluation method where a single LLM evaluator sequentially adopts multiple roles (e.g., critic, scientist) to assess responses. This approach introduces potential role contamination and biases due to repeated model use across evaluative stages.</figcaption></figure>
<figure><img src="image_10.png" /><figcaption>Figure 12: Diagram of MILE-RefHumEval-Conv, where multiple LLM evaluators engage in a conversation chain, each responding to the previous model's input. This method mimics human dialogue but risks accumulating bias from earlier turns, affecting independent judgment.</figcaption></figure>
Question: How can I improve my time management skills?

Assistant 1's answer (GPT):

Here are some tips to improve your time management skills:

1. Create a schedule: Make a to-do list for the day, week or month and prioritize tasks by importance and deadline.

2. Set realistic goals: Be realistic about what you can accomplish in a given amount of time. Set goals that can be achieved but are also challenging.

3. Minimize distractions: Minimize distractions such as emails, phone calls, and social media notifications. Turn off notifications or use apps that block these distractions.

4. Take breaks: Taking a break can help you recharge and increase productivity. Take a break every hour or so to stretch, go for a walk, or listen to some music.

5. Learn to say 'no': Learn to say 'no' to tasks and requests that are not a priority or are not within your capabilities.

6. Delegate tasks: If possible, delegate tasks to others who are capable of completing them. This will help to reduce your workload.

7. Prioritize self-care: Make sure to take care of yourself by prioritizing self-care activities such as exercise, meditation or spending time with friends and family.
Remember, time management is a skill that takes time and practice to develop.

Assistant 2's answer (VICUNA):

Improving your time management skills can help you to be more productive, focused, and less stressed. Here are some tips to help you improve your time management skills:

1. Set clear goals: Identify what you want to achieve and set specific, measurable, and realistic goals. Prioritize your tasks and focus on the most important ones first.

2. Use a calendar or planner: Use a tool to organize your time and schedule your tasks. This can help you to stay on track and make the most of your time.

3. Break down tasks: Break down larger tasks into smaller, more manageable steps. This can help you to focus on one step at a time and make steady progress.

4. Eliminate distractions: Identify and eliminate distractions that may be preventing you from being productive. This can include turning off your phone, closing unnecessary tabs on your computer, or finding a quiet place to work.

5. Practice self-discipline: Use self-discipline to stay focused and avoid procrastination. Set a timer for each task and challenge yourself to complete it before the timer goes off.

6. Take breaks: Take regular breaks to rest and recharge. This can help you to stay focused and avoid burnout.

7. Review your progress: Regularly review your progress and adjust your strategy as needed. This can help you to stay on track and make improvements over time.

<caption>Figure 13: Assistant responses to the question "How can I improve my time management skills?" Two LLMs (GPT and VICUNA) provide advice independently. The figure displays these responses side-by-side to support qualitative comparison of tone, clarity, structure, and depth.</caption>
## Candidates Evaluation using CHATEVAL: One Evaluator (DeepSeek), Many Roles (Cross-Role Agent Conversations)

**General Public Role:** <think> Okay, so I need to figure out which of these two AI assistants did a better job answering the question about improving time management skills. Let me start by reading both responses carefully. Assistant 1's answer has seven tips: creating a schedule with prioritization, setting realistic goals, minimizing distractions, taking breaks, learning to say no, delegating tasks, and prioritizing self-care. It seems pretty comprehensive, covering both personal productivity and some strategic aspects like delegation and goal-setting. On the other hand, Assistant 2 also lists seven tips but they're slightly different. They mention setting clear goals, using a calendar or planner, breaking down tasks into smaller steps, eliminating distractions, practicing self-discipline with timers, taking breaks, and reviewing progress. This approach is more about structure and discipline, focusing on tools like planners and timers. I think both assistants are helpful, but I notice that Assistant 1 includes tips related to self-care and delegating, which might be more beneficial for someone managing multiple responsibilities or needing support. Meanwhile, Assistant 2's emphasis on using tools and setting clear goals could be very practical for someone looking to organize their schedule effectively.

However, considering the content, both are quite similar in structure but maybe differ slightly in depth. I don't see any major inaccuracies in either response. They both cover essential time management strategies, so it's a bit tricky to choose one over the other.

But if I have to pick the better one, perhaps Assistant 1 is slightly more comprehensive because they include self-care and delegating, which are important aspects that can help in long-term productivity and reducing stress. Alternatively, maybe Assistant 2's use of specific tools like planners makes their advice more actionable.

Wait, but both are equally good in terms of structure and relevance. Maybe it's a tie? Or perhaps one is better than the other based on certain criteria. Since I have to choose one, I might lean towards Assistant 1 because they include self-care, which is crucial for maintaining motivation and reducing burnout. </think>

- Final Score for Assistant 1: 9

- Final Score for Assistant 2: 8.5

- Final Best Assistant: Assistant 1

**General Public Role to Critic Role:**

- Final Score for Assistant 1: 9

- Final Score for Assistant 2: 8.5

- Final Best Assistant: Assistant 1

**General Public Role to News Author Role:**

- Final Score for Assistant 1: 9

- Final Score for Assistant 2: 8.5

- Final Best Assistant: Assistant 1

**General Public Role to Psychologist Role:**

- Final Score for Assistant 1: 9

- Final Score for Assistant 2: 8.5

- Final Best Assistant: Assistant 1

**General Public Role to Scientist Role:**

- Final Score for Assistant 1: 9

- Final Score for Assistant 2: 8.5

- Final Best Assistant: Assistant 1

<caption>Figure 14: Evaluation of assistant responses using CHATEVAL (one LLM acting across multiple roles). Each role (e.g., Critic, Psychologist) scores the same responses using a shared criteria. The figure highlights score consistency across roles but potential redundancy due to a single model's repeated use.</caption>
<figure><img src="image_11.png" /><figcaption>Figure 15: Assistant response evaluations using MILE-RefHumEval-Conv, where multiple LLMs (e.g., DeepSeek, Mistral) sequentially critique the same responses in a conversational format. Each evaluator applies the 12-criteria and builds on prior outputs, revealing strengths and weaknesses of multi-step dialogic assessment.</figcaption></figure>
## Candidates Evaluation using MILE-RefHumEval: (One Role, Many Evaluators (No Conversations))

**DeepSeek:** <think> Okay, so I need to evaluate the answers from two assistants for the question How can I improve my time management skills? The evaluation criteria are quite detailed with 12 points each, but I'll try to break it down step by step. First, let's look at Assistant 1. Their answer lists seven tips: creating a schedule, setting realistic goals, minimizing distractions, taking breaks, learning to say no, delegating tasks, and prioritizing self-care. Each point is clear and directly addresses time management. The structure is logical, starting from planning to self-care. It's concise without unnecessary details. Now, Assistant 2 also provides seven tips: setting clear goals, using a calendar, breaking down tasks, eliminating distractions, practicing self-discipline, taking breaks, and reviewing progress. Their answer is similarly structured but includes slightly different points, like using a tool or breaking tasks into steps. Comparing both, they cover similar ground but with slight variations in their strategies. Both answers are relevant because they directly address the question about improving time management. The clarity and coherence are good; each point is easy to understand and logically ordered. Looking at depth, both assistants provide enough detail without being too verbose. They include practical tips that can be applied. Grammatical accuracy seems solid with no obvious errors. Contextually, both answers use appropriate terminology for the subject. In terms of conciseness, both are well-structured but could perhaps add a bit more on how to implement each tip. Creativity-wise, they stick to common strategies which is good for time management advice. There's no bias detected; both answers are neutral and helpful. Knowledge accuracy seems spot-on as these are standard tips in time management literature. Both answers adapt well to the question category by providing actionable advice. They handle simple questions effectively but might struggle with more complex ones, though that's not evident here. Overall, both assistants did a good job. Assistant 1's answer is slightly more comprehensive with additional points like delegating and learning to say no, which adds depth. Assistant 2 includes practical steps like using a tool and reviewing progress, which are also valuable. I think they're on par, so the best assistant would be equal.
</think> - Final Score for Assistant 1: 90- Final Score for Assistant 2: 90- Final Best Assistant: Equal

**Mistral:** 1. **Answer Relevance**:- **Score**:- 95- **Explanation**:- Both answers directly address the question by providing tips to improve time management skills. There are no off-topic details included in either response. 2. **Clarity and Coherence**:- **Score**:- 90- **Explanation**:- Both responses are easy to understand and logically structured. Assistant 1's answer is slightly more conversational, while Assistant 2's answer is more formal but equally clear. 3. **Depth and Completeness**:- **Score**:- 85 (Assistant 1) / 90 (Assistant 2)- **Explanation**:- Assistant 2 provides a slightly more detailed explanation by including the concept of SMART goals and breaking down tasks into smaller steps. Assistant 1's answer is also comprehensive but lacks this level of detail. 4. **Grammar and Linguistic Accuracy**:- **Score**:- 95 (both)- **Explanation**:- Both answers are grammatically correct with no spelling or punctuation errors. The sentence structure is appropriate for the context. 5. **Contextual Appropriateness**:- **Score**:- 90 (both)- **Explanation**:- Both responses use relevant terminology and concepts appropriate to the subject matter of time management. 6. **Conciseness and Precision**:- **Score**:- 85 (Assistant 1) / 90 (Assistant 2)- **Explanation**:- Assistant 2's answer is slightly more concise and to the point, while Assistant 1 includes a few extra details that are still relevant. 7. **Creativity and Insight**:- **Score**:- 80 (both)- **Explanation**:- Both answers offer standard but effective tips for time management. Neither provides a unique perspective or innovative solution. 8. **Bias and Fairness**:- **Score**:- 100 (both)- **Explanation**:- Both responses exhibit neutrality and do not reflect any particular bias or imbalanced perspective. 9. **Knowledge Accuracy**:- **Score**:- 95 (both)- **Explanation**:- Both answers provide accurate factual information about time management strategies. There are no misconceptions or errors included. 10. **Adaptability to Various Categories**:- **Score**:- 90 (both)- **Explanation**:- Both responses adapt well to the generic question category, providing practical advice suitable for a broad audience. 11. **Scalability of Responses**:- **Score**:- 85 (both)- **Explanation**:- Both models handle this simple and straightforward question effectively. Their performance would need to be tested with more complex queries to fully evaluate scalability. 12. **Overall Answer Quality**:- **Final Score for Assistant 1**:- 87- **Final Score for Assistant 2**:- 90- **Final Best Assistant**:- Assistant 2

<caption>Figure 16: Assistant response evaluations using MILE-RefHumEval. Multiple independent LLMs assess the same candidate responses without communicating with each other. This configuration demonstrates high agreement with human ratings and improved reliability due to evaluator diversity and isolation.</caption>
<figure><img src="image_12.png" /><figcaption>Figure 17: Illustration of MILE-RefHumEval's error detection and refinement process. Each LLM evaluator independently identifies weaknesses in the candidate answer and proposes corrections based on a 12-point criteria. The figure shows improved versions, highlighting the framework's ability to enhance response quality without human input.</figcaption></figure>