References Improve LLM Alignment in Non-Verifiable Domains
Authors: Kejian Shi
Yixin Liu
PeiFeng Wang
Alexander Fabbri
Shafiq Joty
Arman Cohan
Venue: ICLR 2026 Poster (2026)
Keywords: LLM Alignment; LLM-as-a-Judge; Alignment Evaluation; Preference Optimization
Primary area: alignment, fairness, safety, privacy, and societal considerations

Abstract: While Reinforcement Learning with Verifiable Rewards (RLVR) has shown strong effectiveness in reasoning tasks, it cannot be directly applied to non-verifiable domains lacking ground-truth verifiers, such as LLM alignment. In this work, we investigate whether high-quality reference outputs can be effectively leveraged to bridge this gap.  First, we design evaluation protocols that enhance LLM-based evaluators for LLM alignment using reference outputs. Through comprehensive experiments, we show that a reference-guided approach substantially improves the accuracy of less capable LLM-judges using references from frontier models; stronger LLM-judges can also be enhanced by human-written references. We then demonstrate the utility of high-quality references in alignment tuning, where LLMs guided with references are used as judges to self-improve. We show that reference-guided self-improvement yields clear gains over both SFT distillation and reference-free baselines, achieving performance comparable to training with finetuned reward models. Specifically, our method achieves scores of 73.1% and 58.7% on AlpacaEval and Arena-Hard with Llama-3-8B-Instruct, and 70.0% and 74.1% with Qwen2.5-7B. These results highlight the potential of using reference-guided LLM-evaluators to enable effective post-training in non-verifiable domains.

Published as a conference paper at ICLR 2026
REFERENCES IMPROVE LLM ALIGNMENT IN NON- VOO-FEEIFIABBE DOMAINS
Kejian Shi"l Yixin Liu Peifeng Wang
Alexander R. Fabbri*, Shafiq Joty4.5, Arman Cohan'
Yale University Meta SSaale "Salesforce Research 3 Nanyang Technological University kejian. shi@yale. edu, yixin. vixin.liu@yale. liu@yale..dd. edu, arman. wwwwneyale...comoooo edu
ABSTRACT
While Reinforcement Learning with Verifiable Rewards (RLVR) has shown strong effectiveness in reasoning tasks, it cannot be directly applied to non-verifiable domains lacking ground-truth verifiers, such as LLM alignment. In this work, we investigate whether high-quality reference outputs can be effectively leveraged to bridge this gap. First, we design evaluation protocols that enhance LLM-based evaluators for LLM alignment using reference outputs. Through comprehensive experiments, we show that a reference-guide guided approach substantially improves the accuracy of less capable L.MMjuugess using references from frontier models; stronger LLM-judges can also be enhanced by Human-written references. We then demonstrate the utility of high-quality references in alignment tuning, where LLMs guided with references are used as judges to self-improve. We show that reference- guided self- improvement yields clear gains over both SFT distillation and reference- free baselines, achieving performance comparable to training with finetuned reward models. Specifically, our method achieves scores of 73.1% and 58.7% % on AlpacaEval and Arena-Hard with 74.1% with Qwen2.5-7B These results highlight the potential of using reference- guided LLM-evaluators to enable effective post-training in non-verifiable domains.
and 70.0% and
0.9
Instruction
Contrast nuclear fissoo & fusion energy output
completions
0.8
Fissoo: splis nudei, energy. combines nuclel, greater fower hammful byproducts like hallum).
Fuslon affoms energy. more energy. Both are powerfu..
spiits Output
21 Create
Reference
0.6
30 DPO updatt
1.40b
Mistral -temo
Queen
2.778
Oween
CP
Fission (xplits heavy nuclel:: significant energy, radioactive byproducts. Fusion (jeins light vastell-vastty mass, primarily stable byproducts legg. helium).
vastly more energy
AlpacaEval
Reference-guinn ceerrenceeuiedd guided Evaluation
Reference Generation
II. Reference-guided Self- Self-lmpprvement
Figure 1: Overview of our study on reference-guidee guiieed LLM-as-a-Jude for LLM alignment. Con- ceptual plots illustrating (I) the improvement in average accuracy from reference guided evaluation
3.3 and (II) the reference-guided self-improvement ($4).
INTRODUCTION
Recently, Reinforcement Learning (RL) from Verifiable Reward (RLVR) (Liu al. 2024a Lambent et al. 2025) has shown strong effectiveness in improving L.LMs' reasoning capabilities. However, RLVR cannot be directly applied non-verifiable domains, such as alignment tuning (Ouyang et
2022 Bai et al.) 2022), because it is non-trivial to design verifiablecreliable rewards for these tasks.
Consequently, RL from Human Feedback (RLHF) (Stiennon et 2020 Ouyang al. 2022 or AI
Equua Contribution.

---

Published as a conference paper at ICLR 2026
Feedback(RLAIF)) (Bai et al. 2022). remains the predominant paradigm for LLM post-training in these domains.
In this work, propose to improve LLM alignment tuning by developing automatic evaluators that can effectively leverage high-quality reference outputs, which reduces the gap between RLVR and RLHF/RLATE Specifically, in RLHF/RLAIF, reward models or LLMs-as-Judges (Zheng et al. 2024
Li et al. 2023: 2024) are used as the automatic evaluators to provide the supervision/rewarrd signals. These evaluators usually assess the model outputs in a reference-free manner. In contrast, various RLVR methods use "verifiers" to evaluate outputs against gold-standard solutions for tasks such as math reasoning (Liu et al. 2024a). This difference motivates us to explore whether high- quality references can serve as soft verifiers. We study: can high-quality references support effective L.M alignment tuning in self-improvement setting, without external human or AT supervision? We argue that this question is important for two reasons. First, it mirrors the advantages of RLvvR-eaming from efference-based domains where rule-based verifiers are infeasible. Second, in practice, reference outputs may be available even when human feedback is costly or unavailable, making it critical to enable model improvement directly from references.
supervision-whil extending them to non-verifiable
To address this question, we develop LLM-judge+" that can effectively leverage reference outputs to provide supervision signals for preference optimization algorithms such DPO (Rafalov et al. 2023
Tunstall et al. 2024). Critically, these reference-guien guidedd M.LM-ju are used a self-improvement
manner, where an LLM serves as the judge to supervise its own training process (Yuan al. 2024
Wu et al. 2024), so no external human or Al feedback is required
In $3 we first develop effective reference-guided LLM-judges for alignment evaluation. Several recent studies have explored guiding LLM LLj-de using references (Zeng et al. 2024 Lyu et al.
2024] Zhang al., 2025 Krumdick al., 2025). However, their evaluation settings are limited in the types of tasks considered and the number of LLMs used as judges, and a more systematic and comprehensive investigation lacking (further discussed in $2). To this end, we first introduce targeted prompting protocols designed to leverage strong references for alignment evaluations. We then conduct comprehensive evaluations for the developed reference -uided evaluation method based on the prompting protocols. Specifically, our proposed method achieves a 6.8% absolute improvement over the reference-free baseline evaluated across 11 LLM-judges using reference outputs generated by
stronger LLM, GPT-4o (Hurst et al. 2024 (3.4). Furthermore, this improvement is generalizable when different frontier LLMs are used to provide references ($3.5). and frontier LLMs like GPT-4o can also be enhanced as judges when provided with high-quality human references ($3.6).
Having developed the L.L.-judges that can effectively leverage references, we apply them in a seff-imprrvemmnt setting for alignment tuning ($4). Specifically, use the instructions in the widely used UltraFeedaack dataset (Cui et al.) (023) to fine-tune Baama----B-Instueee (Meta AI, 2024) and
Qwen2.5-7B (Yang et al. 2024), with high-quality references generated by DeepSeek-Vi -3 (Liu et al. 2024a). We conduct a reference-focused training process involving two stages - (1) first performing distillation, i.e., supervised fine-tuning (SFT) on the reference outputs, (2) then applying the LLMs to be improved as reference guidedddggdddddded LLM-judges in further preference optimization using DPO.
The experimental results highlight the clear benefits of high-quality references: (1) At the first stage, the SFT distillation on reference outputs outperforms preference optimization (DPO) based on finetuned reward model; (2) At the second stage, reference- guided self eeffiimprvemmet using DPO further improves upon the SFT baseline, and shows greater gains compared to reference- free self-improvemenn, Furthermore, DPO with a reference-guides self-LLM-juuge achieves performance comparable to DPO using finetuned reward model of the same parameter size, without requiring the additional human or AI feedback typically needed to train such reward models. On AlpacaEval (Li et al. 2023 and Arena-Hard (Li et al. 2024), our reference-guide self- self-improved models show superior performance, achieving scores of 73.1 and 58.7 with Llama-3, and 70.0 and 74.1 with Qwen2.5, respectively.
serreece-guuied
Our contributions are primarily twofold:
We use "I...Mj to refer to an evaluation method that uses an I.M as the judge for brevity.
Reeent work proposes rubric-based rewards for RI. in non-verifiable domains (Gunjal et al. 2025 Huang
2025) to address RIVR'ss limitations. We note these methods complement ours and be used jointly.
2

---

Published as a conference paper at ICLR 2026
(I) We propose effective methods for enhancing LLM-judges with reference outputs in alignment evaluation, and demonstrate through comprehensive experiments that high-qualit references can substantially improve LLM-judges' accuracy in alignment evaluation.
(2) We show that veference-guides, LLM-judges are effective in semi-self -ippoooeemeeeeettt settings for LLM alignment, providing empirical evidence that high-quality reference outputs can be leveraged for effective model training in non-verifiable domains with reference- guriied-ggdeddddeee automatic evaluators
2 RELATED WORK
LLM-as-a-Judge. Using powerful LLMs as automated evaluators (LLM-as-a-Judge) is growing
practice for scalable evaluation, especially in instraction-following tasks (Zheng al. 2024 Li al.
2023 Dubois et al. 2024). Benchmarks like MT-Bench and grena-Hard (Zheng et al. 2024 Li
al. 2024) utilize strong LLMs (e.g.. GPT-4) as judges, and this paradigm also supports training
data annotation for preference optimization algorithms like DPO (Yuan et al. 2024 Rafailov et al.
2023). However, LLM judges are known to exhibit limitations such as positional and verbosity
biases (Zheng et al.) 2024]] Zhu et al. 2023 Ye et al. (2024). Mitigation efforts include Chain-of-
Thought (CoT) prompting (Wei et al. 2022). answer swapping (Shi et al.] 2024), and developing more robust evaluation protocols (Zeng et al. 2024 Liu et al. 2024c). Our work builds this by
investigating efference-guider prompting to enhance LLM judge accuracy and robustness, [rivedi
et al. (2024) explores improving judges via internal relfrrationallzation (Chain-of- Thought). Our work is orthogonal, focusing on external grounding via references to address knowledge gaps rather than reasoning gaps.
The Role of References in LLM Evaluation. Traditional NLG evaluation often relies on reference outputs (e.g., BLEU (Papineni et al. 2002). ROUGE (Lin 2004)), but their role in LLM-as-a- Judge for alignment evaluation, where single ground-truth references are often insufficient, has been less explored. Recent work has begun revisiting references: LLMBar (Zeng al. 2024) used prompts that guide LLM-judges to generate reference outputs before evaluation; HREF (Lyu et al. 2024) incorporated Human-written responses and reported improved performance over reference-free methods. However, these studies are limited in both the number of L.LMs evaluated and dataset scale. Krumdikku et al. (2025) focused on reference use for question answering, RevisEval (Zhang al. 2025) proposed generating response-adapted references to improve evaluation accuracy. Unlike RvvisEval, which focuses on dynamic reference modification for static evaluation, our work extends the setting to model training and demonstrates the benefits of reference- seereencegguiedd supervision in self-improvement scenarios. Furthermore, our work provides a more systematic and large-scale investigation into reference-guided LLM-judges, covering 5 datasets and 13 LLMs.
Self-Improving LMs and Generative RMs. LLM-judges have also been used in model training. particularly in ceff-improvement settings where L.MM supervises its training Yuaan al.) 2024; Wu et al.] 20244] Yasunaga et al.. 2024). A related line of work explores Generative Reward Models (Zhang et 2024 Mahan et al. 2024), where LLMs serve as reward models in preference optimization. Recent studies show that general-purpose frontier LLMS can perform competitively with finetuned discriminative reward models in this setting (Zhou et al., 2025; Frick al. 2025). Our work builds on this direction by studying reference- sierence--uiied guided LLM-judges for improimmroeeeent, making the setting more feasible by providing additional grounding through high-quality references.
DEVELOPING REFERENCE-GUINE LLM-JUDGES
To enable effective use references in improving LLM alignment evaluation, we first develop robust reference-guie guided evaluation methods for LLMs-as-Judges, and conduct comprehensive evaluations of them against strong baselines.
3.1 PRELIMINARY
LLM-judges for alignment evaluation typically perform pointwise scoring of a single output given an instruction /Zheng et al. 2024), or pairwise comparison of two outputs (Li al. 2024). In this
We release the codebase https: http://thuu..cmmallllllll//./ap RL from Reference-Guided Rewards).
3

---

Published as a conference paper at ICLR 2026
study, we focus on the pairwise comparison setting, as it matches the annotation format of various high-quality human-labeled alignment datasets such as LLMBar (Zeng et al. 2024). and is directly applicable to preference optimization algorithms like DPO. While our primary analysis focuses on this pairwise setting, we also conducted experiments on pointwise scoring to ensure the robustness of our findings. We present these results in Appendix [ which confirms that reference-guide evaluation also improves performance in a pointwise scoring setting. To evaluate an LLM-judge, human annotations are typically used as ground truth. Specifically, in the pairwise comparison task, the LLM-judge evaluation accuracy is measured by the proportion of instances where the LLM-judge selects the same preferred output as the human annotators.
3.2 REFERENCE-GUIDED PROMPTING FOR LLM-JUDGES
We introduce targeted prompting strategies designed to effectively leverage reference answers in the LLM-as-a-Judge paradigm. As a baseline, we first introduce strong reference-free prompting method, which we refer to Ref-Free (Ours). This prompt is designed for direct pairwise com- parison without relying on any external reference answer (its template is in Figure 2 Its general structure follows the base prompt proposed in Zeng al. (2024). However, specifically instruct the model to assess instruction instruction-following following quality along with other critical aspects
design the prompt to
such factuality and verbosity. As results show, this method outperforms many existing baselines.
approach, we extend it to a reference guided setting by instructing the LLM to assess which candidate output more closely aligns with the quality and content exemplified by the reference, while still addressing the original instruction. We refer to this method as RefEval. While prior work has proposed similar reference-guide prompting methods (Zeng al. [0224]]] Lyu et al. 2024), our approach offers more explicit guidance on how the reference output should be used (See prompt design in Appendix A.2
Building on the reference-
RefEval
User Message:
Decide which output is hetter following the instruction. An effective and factually correct Reference Output provided aid your evaluation. This Reference Output demonstrates successful instruction-followini Here are some aspects to consider:
1. Outputs should precisely follow the instruction. If output contains unrelated information or does not complete each and all requirements in the instruction, means that output docs not precisely follow the instruction.
2. You should check for factual correctness and accuracy of outputs. If an output contains factual errors (especially with numbers), it should be considered lower quality. Compare the output against the Reference Output verify if that output is factually correct.
Outputs should contain only a brief effective response without any verbose explanation, unless instruction explicitly asks for an explanation.
4. Understand how the Reference Output properly delivers a helpful, accurate, and natural response, and then compare how closely output matches this successful Reference Output. Extrancous content
output that goes beyond what is present in the Reference Output should be
discouraged.
6. The order in which the outputs are presented to you should NOT affect your judgment.
Select which output. "Output (a)" or "Output (b)", is better following the instruction. Your should ONLY contain: "Output (a)" or "Output (b)":
Figure 2: A snapshot of RefE'vl method.
We show a snapshot of our core prompt in Figure 2 while the full prompt template provided in Figure 10 in Appendix H As shown in 3 this emphasis reference utilization leads to clear improvements over previous methods.
To further emphasize the role of references, design additional prompting method, RefMatch (Figure [T], which instructs the Ludee to act primarily as a semantic and stylistic matcher, determining which candidate output more closely resembles the reference. Specifically, the L.M is
4

---

Published as a conference paper at ICLR 2026
Table Average evaluation accuracy (%) across five datasets using 11 open-source models as judges. We report the 95% bootstrap confidence interval for the average performance. indicates the method significantly worse than RefEval (p 0.05). Dataset acronyms are: Natural (Nat), Adversarial (Adv), MTBench (MT), and InstruSum (Ins).
Method
Nat
Adv
MT
Ins
HREF
Avg
L.MBBr-Base
83.1
61.7
74.6
70.2
72.0
72.3 (-1.4, +1.5)
HREF-Base
84.1
54.0
76.5
70.8
77.3
72.5 (-1.5,+,,1116)
CoT
82.0
60.1
75.4
69.1
69.6
71.2 (-1.5,
Prepair
81.7
71.5
72.6
68.8
75.2
74.0 (-133,++1)
Self-Ref
84.6
66.7
73.5
69.5
72.4
73.3 (-13,+1.33)
Self-Metric-Rer Ref
85.5
67.4
75.0
70.5
74.4
(114)
Ref-Free (Ours)
83.4
67.8
70,9
71.0
75.4
73.7 (-1.,++1.4)
LLMBar-Ref
85.5
66.3
74.5
70.7
72.8
74.0 (-1.3, +1.4)
HREF-Ref
85.3
62.3
76.5
70.8
79.2
74.8 (-1.3, +1.4)
RefMatch
84.6
74.1
76.3
72.9
80.4
77.7 (-1.6, +1.6)
RefEval
86.8
74.9
76.7
74.5
82.7
79.1 (-1.4,
explicitly instructed with: "Your goal is determine which output demonstrates closer similarity to the reference."
For comprehensiveness, we also explored several variants of these core methods. While these variants exhibited interesting characteristics in certain scenarios, our primary focus in the main paper will be on RefEval and RefMatch due to their consistent strong performance and clarity. Detailed descriptions and results for all explored variants are provided in Appendix A and Appendix
3.3 EVALUATION SETUP
Evaluation Setting and Metric. We use evaluation accuracy the main metric, computed based on human annotations as ground truth the pairwise comparison setting. To mitigate potential positional biases, where the LLMM-juge might favor output based on its presentation order (Park et al., 2024), all reported accuracies are averaged across two evaluation passes with the order of candidate outputs swapped. All LLM-judge evaluations use greedy decoding.
Datasets. We use five human-annotate datasets to ensure robust evaluation across varied instruc- tion types and complexities alignment evaluation. These datasets are: (1) LLMBar-Natural (Nat) and (2) L.MMBar-Avveraarti (Adv) (Zeng al. 2024). which contain carefully curated cntructiio-followiing examples; (3) MTBench (MT) (Zheng et al. 2024). a benchmark for multi-turn conversational abilities: (4) Instrusum (Ins) (Liu 2024b),, focusing on instruction-controllabbe summarization; (5) HREF (Lyu et al. 2024). a benchmark with human-written reference responses for instruction following across diverse scenarios. Further details on dataset characteristics and processing are provided in Appendix
Obtaining Reference Outputs Most the datasets we use do not provide Human--witten ref- erences. Therefore, study reference-guided LLM-judges, we focus on setting where strong frontier LLM, GPT-4o, is used as an oracle to generate reference outputs, while less capable LLMs are used as judges, guided by the generated references.
LLMs Evaluated as Judges As GPT-4o is used the oracle to provide references, primarily evaluate LLMs that are less capable than GPT-4o but represent diverse range of model families and sizes: qwen-2.5-72b,, .lama-3.1-700 gemma-2-27b, qwen-2.5-14b, mistral-nemo, gemma-2-9h, LJama-3.1-80 qwen-2.5-7b, Llama-3-8b, glm-4-9b, and mistral-7b-v0.3. We provide additional details in Appendix
Baseline Evaluation Methods. We compare proposed reference-guided evaluation methods
against the following established LLM-as-a- a-dgge prompting strategies. LLMBar-Bau The vanilla pairwise comparison approach from Zeng et al. (024), where the L.LM directly predicts the preferred
5

---

Published as a conference paper at ICLR 2026
Table 2: Comparison of average evaluation accuracy across 11 LLM judges (averaged over five datasets). References for RefEval were generated by GPT-4o. Best performance for each model is bolded. Base denotes -Base Full statistical significance analysis is provided in Appendix A.1
Method
qwen-2.5
Ilama-3.1
gemmaa22
-2b
qwen-2.5
2.5
-14b
mistral -nemo
gemmu-2
glm--d
Ilama-3.1
qwen-2_5
Ilama-3
mistral-7b
Base
79.4
85.2
82.3
81.5
65.6
80.8
71.8
65.0
73.5
60.1
47.0
RefFree
83.4
85.6
80.1
83.3
63.7
80.8
77.3
71.8
74.5
72.3
61.2
RefEval
85.9
73.2
79.5
79.4
77.4
77.5
output. CoT: Also from Zeng et al. 2024). this method prompts the LLM to provide a Chain-of- Thought explanation before making final judgment. Self-Ref: Adapted from Zeng al. (2024), the LLM first generates its own reference answer to the instruction, which is then used as context during the pairwise evaluation of candidate outputs. Seff-Metric-Ref: Combines Self-Ref with an initial step where the LLM generates key evaluation metrics (aspects to consider) for the given instruction, as proposed by Zeng al. (2024). LLMBar-Ref: uses the same prompting method as Self-Ref, but uses the references generated by the oracle, GPT-4o, instead of self-generated references. PrePAIR: Proposed by Jeong al. (2024), this protocol first elicits pointwise analysis for each candidate, identifying barks and then performs pairwise comparison informed by these analyses. HREF-Base: The reference-free prompting method proposed by Lyu et al. (2024).
HREF-Ref: The reference-based prompting method from Lyu et al. (2024), which incorporated reference output into the prompt. The prompt templates of these methods are in Appendix H
3.4 OVERALL RESULT ANALYSIS
We first examine the average performance across our suite of 11 open-source LLM judges.
As shown in Table T RefEval achieves the highest average evaluation accuracy (79.1%). This significantly outperforms reference-fiee free baselines such as LLMB BaMeBa--Baee (72.3%) and CoT (71.2%), as well as other reference-basee methods, HREF-Ref (74.8%) and LLMBar-Ref (74.0%). Our other core reference-based method, RefMatch, also demonstrates strong performance (77.7%), ranking second overall. These results show the effectiveness of directly grounding LLM judgments with strong reference via our proposed methods.
Inter-Judge Agreement. Beyond accuracy, we analyze whether references help different LLM- judges align with each other. shown in Appendix C.2 RefEvvl substantially improves the average pairwise agreement between judges compared to the reference-free baseline (81.4% vs 76.6%),
indicating that references provide a shared grounding that reduces variance in decision-making
Table 2 illustrates the individual performance for the 11 LLM-judges. It shows that RefEval overall achieves higher or comparable accuracy, and that smaller, less capable models benefit more from evaluations. For instance, with Llama-3-8b, RefEval achieves an absolute
the reference-gue improvement of approximately 17.4% over LLMBar-Basee While stronger models like qwen-2.5- 72b also benefit (84.6% for RefEval vs. 79.4% for LLMBar-Base), the relative gains more pronounced for models that initially struggle with reference-free evaluation. Providing a strong reference through RefEval effectively enables small L.LMs to achieve evaluation quality closer to that of much larger ones. In $3.6) we present a case study showing that stronger LLM-judges can also be further enhanced with human references. Appendix (A.5 provides providos more detailed analysis regarding the LLM-judge's 's performance grouped by their sizes.
3.5 REFERENCES FROM VARIOUS FRONTIER LLMS
Our primary experiments in 3.4 established the efficacy of RefEval and RefMatch using a single strong reference from GPT-4o. Therefore, we explore the impact of varying the source of this single reference and investigate strategies for leveraging multiple references. Specifically, we maintain the 11 open-source LLMs judges to be evaluated, but generate reference outputs using four additional frontier LLMs here: Claude-3.5-Someet (Anthropic 2024), Blaude-3.7-oomm (Anthropic) 2025).
Gemini-2.0-FFaast (Google Cloud 2025). DeepSeek- V3 (Liu et al. 2024a).

---

Published as a conference paper at ICLR 2026
76
79.1
78.2
77.5
78,1
77.8
78,5
78,7
Ang. Accuracy
74
70
GPT-4o
Claude-3.5
Gomini-2.0
Claude-3.7 3.7
DS-V3
Vote
Ref-Free (Ouns): 74.2%
LLMBar-Base: 72.3%
RefMatch
RefEval
Figure 3: Evaluation accuracy of 11 open- open-source LLM-judges using RefEval and RefMatch with single references from various frontier models, and their voted versions. Horizontal dashed lines indicate reference-free baselines. Results are averaged over five datasets.
Figure 3 illustrates the performance of our RefEval and RefMatch methods when guided by single reference from each of these frontier models. Both methods consistently outperform the reference-free baselines regardless of which frontier model generated the single reference, indicating their robustness.
Given the consistent benefit from different strong references, we then explored a Multi- Reference Voting strategy. Here, the LLM- judge performs independent pairwise evalua- tions for each candidate pair, each guided by a different reference. The final decision made by majority vote. As shown in Figure 3 (la- heled Vote), this voting approach yields the high- average accuracies for both protocols.
Table 3: Accuracy (%)
comparing standard vs. human-edited "Oracle" ref- erences. Qwen Qwen-2.5-72B and Llama is Llama-3.1-70B.
GPT-4o
GPT-4.1
Qwen
Llama
Ref-Free (Ours)
85.3
71.0
Robustness to Reference Quality. While our primary experiments use frontier models as ref- erence providers, we also investigated the im- pact of using references from less capable mod- els. In an ablation study using Llama-3.1-70B
RefMatch Re/Match-Oracle
84.2
86.1
81.0
79.9
85.9
88.2
82.6
83.9
RefEval
86.8
86.7
79.9
82.8
RefEval-OOracle
88.6
81.8
84.6
as the judge, we replaced GPT-4o references with those generated by Mistral-Nemo-122 and Tulu- 2-7B. As detailed in Appendix C.I with these weaker references, Reffival achieved scores of 85.0%
(Nemo) and 83.4% (Tulu), outperforming the reference-free baseline (80.3%).
3.6 HUMAN REFERENCES AS ORACLES
Our evaluation above uses GPT-4o generate references to supervise less-capable LLM-judges. Therefore, to investigate the impact of high-quality references to stronger LLM-judges we conduct focused experiment on the LLMMBarAAversariill dataset, which is adversarially constructed and challenging to LLMM-jdgee. To this end, we create "Oracle" references by having humans edit subset of the machine- genceree-eeeerated references (from GPT--4o ensure near-gold standard quality. Details of this editing process in Appendix D.2 We evaluated four models, GPT-4o, GPT-4.1, Qwen-2.5-72B, and 3. using our RefEval and RefMatch protocols with both standard, sometimes flawed, machine references and these human-edites oracle references.
Table shows that human-edited references enhance evaluation accuracy. For example, when GPT-4o serves the judge, its RefEval accuracy increased from 86.8% with its self- seff-aenerteed reference to 88.4% with the Oracle GPT-4o reference. Similar improvements are observed across the models and protocols tested. These findings highlight that even highly capable I.IM judges can benefit from
7

---

Published as a conference paper at ICLR 2026
references of high, numan-verified quality, particularly evident adversarial or complex instructions where standard machine- machine-generated references might contain flaws.
REFERENCE-GUIDER SELF-IMPROVEMENT
Having demonstrated the benefits of references in aiding LLM-judges' evaluations, we now explore their utility in model training. Specifically, we consider a self-immrovement setting where an LLM supervises its own training using preference optimization algorithms Yuaan et al. [024] Wu et al. 2024). Unlike prior work, however, the L.LM is provided with high-quality reference outputs to guide its evaluations, making this setup more practical. This setting resembles classic NLG training, where reference outputs are available for supervised finetuning (SFT), but explicit preference annotations absent. The reference-basee (self-)LLM-juude offers flexible alternative extending the use of references beyond SFT and into preference optimization, without requiring preference-annotated data to train separate reward model.
4.1 TRAINING PROCESS
Training Stage 1: Distillation. The first stage of the training process is direct distillation, where the base models are finetuned by SFT on the reference outputs. We found that this superior to directly applying the preference optimization algorithms, which will be further discussed in $4.3
Training Stage DPO. At the second stage, the models are further finetuned using DPO Rafailov et al. 2023):
CDPo(p6iPeo) = -- (3log log (o( Blog
po ()(())))) Pret ((rl))
(1)
(rel(wu)z))
where is input in the training dataset D, y and yu is pair of outputs where yhe is the better, and (.)) is the sigmoid function. Pe is the model under training, Prer the reference model that usually instantiated using the model checkpoint be finetuned, and / is hyperparameter controlling the strength of the KL-divergence regularization from the reference model.
Here. the preference annotations constructed "on-policy", where the output pair (Ve- y1 is sampled from the model to be finetuned, Pref, at the beginning of the training, and annotated by an LLM-judge or a reward model. More specifically, we follow the setting of Meng al. (2024) sampling 5 candidate outputs for each instruction with temperature of 0.8 and constructing the output pair by selecting the best and the worst candidates. When using L.MM-juges that perform pairwise comparisons, all output pairs candidate output. This process adopted since previous studies (Dong et al. 2024 Meng et al. al. (2024) have found that such "on-policy" data generation methods lead to better performance compared to using static preference annotations, of which the output pairs are generated by different models.
compared to derive the average quality score of each
4.2 EXPERIMENTAL. SETTINGS
Base Models. We use two base models in our training experiments. The first model is Meta- Llama-3-8B-Instrun which has already undergone post-training. To verify the effectiveness of guideed self-imprrvemmnt on models that have not being substantially posttrained, the second model we use Qwen2.5-7B-SPP 2024) on the Tulu3 SFT data mixture (Lambert et al. 2024a The SFT training setting follows the exact recipe used in Lambert et al. 2024a) (Appendix
which we finetuned from the pretrained Qwen2.5-7B (Yang et al.,
Evaluation Benchmarks. We use two widely-used benchmarks for LLM alignment and instruction- following evaluation: AlpacaEval (Li et al.) 2023) and Arena-Hard Hard (Li et al. 2024). On both benchmarks, models' outputs are compared against GPT-4's outputs using a strong LLM as the the default LLM-judge, gpt-4-1 106-preview. On Arena-Hard,
judge. On Alpa,aEval, gpt-40-2024-08. as the judge, which more cost-efficient and capable.
8

---

Published as a conference paper at ICLR 2026
Table Performance comparison of training methods. Scores are for length-controlled AlpacaEval (AE) and hena-Hard (AH).
-lama-3-8--Instrre
Qwen2.5-7--SF
Method
AE
AH
AE
AH
Base
25.0 +0.3)
27.1 (-1.8, 1.7)
14.4 +0.2)
23.4 (-2.1,,
ArmoRM-Base
49.2 (-0.5, +0.5)
40.4 (-2.4, 2.4)
32.6 (-0.3, +0.3)
58.6 (-2.3, 2.6)
V3-Distill
530. +0.6)
4-2. 2.3)
48.8 +0.5)
56.5 2-113,2.1)
ROUGE
(-0.6 +0.6)
52.1 (-2.2, 2.5)
5-.. +0.5)
(-2.0, 2.5)
BERTScore
58.8 (-0.6, +0.6)
53.0 (-2.8, 1.9)
5-0. +0.6)
64.5 (-2.3,
ArmoRM
58.6 (-0.7.
66.8 (-0.7, +0.7)
72.2 2-112,2.1)
RefFree
6-.7, +0.7)
53.8 (-2.2, 2.6)
65.1 (-0.6, +0.7)
71.8 (-2.1, 2.1)
RefEval
(-0.,,
58.7 (-2.8, 2.6)
70.0 (-0.7, +0.7)
74.1 (-2.4, 2.0)
Data Sources. The instruction set used for preference optimization is from UltraFeedhack (Cui et al. 2024), which consists of 60K instructions covering diverse scenarios. It has become a standard testbed for evaluating preference optimization algorithms (Tunstall et al. 2024 Meng et al. 2024). To obtain high-quality references, use DeepSeek-V3 (Liu et al. (0224)) to generate outputs over the entire instruction set. DeepSeek-VV V3 is a frontier LLM that demonstrates strong capabilities across various domains, including competitive performance on AlpacaEval and Arena-Hard. Its strong performance and moderate API cost make it a suitable choice for reference generation
Hyperparameters. For the first training stage, SFT the reference outputs, we the same hyperparameter settings as those applied when finetuning Qwen2.5-7B on Tulu3 SFT as described above, following the recipe in Lambert al. (2024a). For DPO training, we follow similar setting as used in Meng et al. (2024). Specifically, the number of training epochs is 1, the batch size is 64, the maximum learning rate is 5e-7 with cosine learning rate scheduler and 10% warmup steps. As for the hyperparameter / in the DPO objective (Eq. []] we perform a grid search within the range of 0.005 0.1, and compare each algorithm using its best hyperparameter configuration, following evaluation settings in previous work (Tunstall et al. 2024
Meng et al. 2024).
4.3 RESULT ANALYSIS
Table 4
compares the performance of several models: (1) Base is the base model to be finetuned; (2) ArmoRM-Base applies DPO on the base model using the AmmoRM-Laama3-8---/
reward
model (Wang et al. 2024), which achieves strong performance on RewardBench (Lambert et al.
2024b); and (3) Di-Disl the SFT model distilled from DeepSeek references. The following
models are finetuned from V3-Distill: (4) ROUGE, which uses ROUGE scores as the reward
(5) BERTScore, which uses the BERTScore metric (Zhang et al./ 2020 Zhao al., (025); (6) Ar- moRM, which uses the ArmoRM reward model; (7) RefFree, our reference-fres self-improvement method (FigureP) and (8) RefEval, our reference-guides self-imppovemmnt method (Figure I0). where the judges are the dist-distilled Llama-3-88 Instructtt and Dwwen..5-7B-FFT
Table 4 highlights the following findings:
(1) SFT training high-quality reference outputs is more effective than performing preference opti- mization with a finetuned reward model. Specifically, V3-Distill, the distillation model, outperforms the model trained with ArmoR.. This underscores the benefits of strong reference outputs.
(2) LLMs can effectively self-improve, which is demonstrated by the substantial improvement of RefFree over V3-Distill.
(3) References help LLMs better self-improve, as RefEval, the model trained with the reference- guided self-judge, consistently outperforms RefFree. It also performs much more strongly than the traditional reference-based metrics, ROUGE and BERTScore, and achieves comparable or better
"The total cost of generating 60K reference outputs through DeepSeek's 's API is around 40 US dollars. 'aah output's quality score is the average of ROUGE-1 and ROUGE-2 scores against the reference outputs.

---

Published as a conference paper at ICLR 2026
performance compared to the finetuned reward model ArmoRM. It shows that the LLM-judges' improvement from references observed in can result substantial improvement in training.
Table provides further comparison between the resulting model of our reference-basee self- improved training pipeline, and strong baselines. For Llama-3-8P 3---Instruc, we compare against SimPO (Meng al., 2024), which is compet- itive method that outperforms DPO, The com- pared checkpoint trained from the same base model, on the same instruction set, following similar training process, and uses ArmoRM as the reward model. For Qwen2.5-7B, we di- rectly compare against its post-trained check- point, Qwen2.5-7B-Ine Instruct (Yang et al. 2024). The results in Table demonstrate clear advantage of our trained models, validating the effectiveness of leveraging high-quality references.
Table 5: Performance of our self- self-improved models vs. strong baselines on AE and AH.
AlpacaEval Arena-Hare
DeepSeck-V V3
84.8
94.9
SimPO-Llamaa-33 BB--nn------sttttt
51.6
36.2
RefEval Llama3-8B Inst
73.1
58.7
Owen2.557B-Im
29.9
58.0
RefEval Sattval-Qwen..5755
70.0
74.1
Impact of Reference Quality on Training. To verify the training benefits are exclusive to frontier references, repeated the RefEval seff-improvement pipeline using references gen- erated by GPT-4o-mini (a weaker model than DeepSeek-V33 V3). Detailed results are provided in Appendix C.I We find that the resulting model still outperforms the reference-free self- improvement baseline using the same generator. This shows that while higher-quality references yield better results, the reference- guidedd super- vision mechanism itself provides structural ben- efits regardless of the specific generator.
0.75
0.67
6.75
2.71 0.25
Score
01:55
Catag/Mte
Coscbe Toda
Infknnatin Sakitg
Qwen2.5 5--55-PT
RaFaaee Retlaue
3.7
0.67
0:18
0.68
Understanding the Benefits of Reference- Based Seff-Improvement. To better under- stand the distinction between reference-basee based and reference- free supervision, we use GPT-4o categorize the instructions in AlpacaEval and Arena-Hard into four types: Coding&Matt Math, Cre- ative Tasks, Information Seeking, and Reason- Plannian
Scoce
Ceattee
Intemsatie Sakking (kcostaape/aastty
Figure 4: Comparison of reference- and reference-guided self-impprovement across task cat- egories on AlpacaEval and Arena-Hard.
We then compare the performance of the Ref- Free and RefEval models across each category Figure shows that for both Llama-3-81 -a-aa------Innntueee and Qwen2.5-7B-SH.
reference-basee supervision yields a substantial improvement in the Cod- inth category. However, its benefit on Creative Tasks is less significant for Qwen2.5-788 7---SFF.. while remaining considerable for Llama-3-8B-Ine Instrrcc... We posit that this is because leveraging references effectively in open-ended tasks is more challenging, and doing so requires more extensive post-training (as Llama-3-8B 3-8B-Innstucc) rather than the standard SFT (as in Qwen2.5-7B-SFT),
CONCLUSION
In this study, we investigate whether high-quality reference outputs can enable effective LLM alignment tuning, Across five datasets, we show that high-quality references can consistently improve LLM-judge performance. Using the developed referencc-guudeer LLM-judges in alignment tuning, we demonstrate that they can lead to effective semi-self-imme self-improovemeet by using high-quality references, even achieving performance comparable to that of trained reward models. Our findings highlight the potential of leveraging references to improve LLMSs in non-verifiable domains, while reducing the methodology gap between RIHF/RLATF and RLVR for LLM post-training.
"Tee prompt used for classification and the distribution of instruction types are Appendix
10

---

Published as a conference paper at ICLR 2026
REFERENCES
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, lige Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, al. GPT-4 technical report. arXiv preprint Stti::2303.087777/ 2023.
Anthropic. Claude 3,5 sonnet model card addendum, 2024. URL ht /wwp::////wwwwccc.c.c www.wwnw.nttoooi................ com/fed9cc1 193a 411414444111111
Card Claude Addendum, pdf]
Anthropic. Claude 3.7 sonnet system card, 2025. URL httpp:::/aaatsstatttrrrrostp
Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, Binyuan Hui, Luo Ji, Mei Li, Junyang Lin, Runji Lin, Dayiheng Liu, Gao Liu, Chengqiang Lu, Keming Lu, Jianxin Ma, Rui Men, Xingzhang Ren, Xuancheng Ren, Chuanqi Tan, Sinan Tan, Jianhong Tu, Peng Wang. Shijie Wang, Wei Wang, Shengguang Wu, Benfeng Xu, Jin Xu, An Yang. Hao Yang, Jian Yang. Shusheng Yang, Yang Yao, Bowen Yu, Hongyi Yuan, Zheng Yuan, Jianwei Zhang, Xingxuan Zhang, Yichang Zhang, Zhenru Zhang, Chang Zhou, Jingren Zhou, Xiaohuan Zhou, and Tianhang Zhu. Qwen technical report. arXiv preprint 2309v 16699 2023.
Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, Nicholas Joseph, Saurav Kadavath, Jackson Kernion, Tom Conerly, Sheer El-Showk. Nelson Elhage, Zac Hatfield-Dodde Danny Hernandez, Tristan Hume, Scott Johnston, Shauna Kravec, Liane Lovitt, Neel Nanda, Catherine Olsson, Dario Amodei, Tom Brown, Jack Clark, Sam McCandlish, Chris Olah, Ben Mann, and Jared Kaplan. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv: 2204.05864 2022.
Claude. Introducing claude, Mar 2023.
URL https:/www.anthrroicccoomm
Introducing-
Ganqu Cui, Lifan Yuan, Ning Ding, Guanming Yao. Wei Zhu, Yuan Ni, Guotong Xie, Zhiyuan Liu, and Maosong Sun. Ultrafeedback: Boosting language models with high-quality feedback. arXiv preprint 130.0:2310.01377, 2023.
Ganqu Cui, Lifan Yuan, Ning Ding, Guanming Yao, Bingxiang He, Wei Zhu, Yuan Ni, Guotong Xie, Ruobing Xie, Yankai Lin, Zhiyuan Liu, and Maosong Sun. ULTRAFFEDBACKK Boosting language models with scaled AI feedback. In Forty-first International Conference on Machine Learning, 2024. URL htttps://opennrviiew.eee.
DeepSeek-Al, Aixin Liu, Bei Feng. Bing Xue, Bingxuan Wang. Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai Dai, Daya Guo, Dejian Yang, Deli Chen, Dongjie Ji, Erhang Li, Fangyun Lin, Fucong Dai, Fuli Luo, Guangbo Hao, Guanting Chen, Guowei Li, H. Zhang. Han Bao, Hanwei Xu, Haocheng Wang, Haowei Zhang, Honghui Ding, Huajian Xin, Huazuo Gao, Hui Li, Hui Qu, J. L. Cai, Jian Liang. Jianzhong Guo, Jiaqi Ni, Jiashi Li, Jiawei Wang. Jin Chen, Jingchang Chen, Jingyang Yuan, Junjie Qiu, Junlong Li, Junxiao Song, Kai Dong, Kai Hu, Kaige Gao, Kang Guan, Kexin Huang, Kuai Yu, Lean Wang. Lecong Zhang, Lei Xu, Leyi Xia, Liang Zhao, Litong Wang. Liyue Zhang, Meng Li, Miaojun Wang, Mingchuan Zhang, Minghua Zhang, Minghui Tang, Mingming Li, Ning Tian, Panpan Huang, Peiyi Wang, Peng Zhang. Qiancheng Wang. Qihao Zhu, Qinyu Chen, Qiushi Du, R. J. Chen, L. Jin, Ruiqi Ge, Ruisong Zhang, Ruizhe Pan, Runji Wang, Runxin Xu, Ruoyu Zhang, Ruyi Chen, S. S. Li, Shanghao Lu, Shangyan Zhou, Shanhuang Chen, Shaoqing Wu, Shengfeng Ye, Shengfeng Ye. Shirong Ma, Shiyu Wang, Shuang Zhou, Shuiping Yu, Shunfeng Zhou, Shuting Pan, T. Wang, Tao Yun, Tian Pei, Tianyu Sun, W. L. Xiao, Wangding Zeng, Wanjia Zhao Wei An, Wen Liu, Wenfeng Liang, Wenjun Gao, Wenqin Yu, Wentao Zhang, X. Q. Li, Xiangyue Jin, Xianzu Wang, Xiao Bi, Xiaodong Liu, Xiaohan Wang. Xiaojin Shen, Xiaokang Chen, Xiaokang Zhang, Xiaosha Chen, Xiaotao Nie, Xiaowen Sun, Xiaoxiang Wang, Xin Cheng, Xin Liu, Xin Xie, Xingchao Liu, Xingkai Yu, Xinnan Song, Xinxia Shan, Xinyi Zhou, Xinyu Yang. Xinyuan Li, Xuecheng Su, Xuheng Lin, Y. K. Li, Y. Q. Wang, Y. X. Wei, Y. X. Zhu, Yang Zhang, Yanhong Xu, Yanhong Xu, Yanping Huang, Yao Li, Yao Zhao, Yaofeng Sun, Yaohui Li, Yaohui Wang. Yi Yu, Yi Zheng, Yichao Zhang, Yifan Shi, Yiliang Xiong, Ying He, Ying Tang, Yishi Piao, Yisong Wang. Yixuan
11

---

Published as a conference paper at ICLR 2026
Tan, Yiyang Ma. Yiyuan Liu, Yongqiang Guo, Yu Wu, Yuan Ou, Yuchen Zhu, Yuduan Wang, Yue Gong, Yuheng Zou, Yujia He, Yukun Zha, Yunfan Xiong, Yunxian Ma. Yuting Yan, Yuxiang Luo, Yuxiang You, Yuxuan Liu, Yuyang Zhou, Z. F. Wu, Z. Z. Ren, Zehui Ren, Zhangli Sha, Zhe Fu, Zhean Xu, Zhen Huang, Zhen Zhang, Zhenda Xie, Zhengyan Zhang, Zhewen Hao, Zhibin Gou, Zhicheng Ma, Zhigang Yan, Zhihong Shao, Zhipeng Xu, Zhiyu Wu, Zhongyu Zhang. Zhuoshu Li. Zihui Gu. Zijia Zhu, Zijun Liu, Zilin Li, Ziwei Xie, Ziyang Song, Ziyi Gao, and Zizheng Pan. 2024.
Deepseek-v3 technical report. arXiv preprint arXiv:
Hanze Dong, Wei Xiong, Bo Pang, Haoxiang Wang. Han Zhao. Yingbo Zhou, Nan Jiang, Doyen Sahoo, Caiming Xiong, and Tong Zhang. RLHF workflow: From reward modeling to online RI.HF. Transactions Machine Learning Research, 2024. ISSN 2835-8856. URI. https
//openreview.nee wwwpiiewwne/fooummiiii a Baa
Zhengxiao Du, Yujie Qian, Xiao Liu, Ming Ding, Jiezhong Qiu, Zhilin Yang, and Jie Tang. GLM: General language model pretraining with autoregressive blank infilling. In Proceedings of the 60th Annual Meeting of the Association for Compulational Linguistics (Volume 1: Long Papers), pp. 320-335, 2022.
Yann Dubois, Chen Xuechen Li, Rohan Taori, Tianyi Zhang, Ishaan Gulrajani, Jimmy Ba, Carlos Guestrin, Percy S Liang, and Tatsunori B Hashimoto. Alpacafarm: A simulation framework for methods that learn from human feedback. Advances in Neural Information Processing Systems, 36, 2024.
Evan Frick, Tianle Li, Connor Chen, Wei-Lin Chiang, Anastasios Nikolas Angelopoulos, Jiantao Jiao, Banghua Zhu, Joseph E. Gonzalez, and lon Stoica. How to evaluate reward models for RLHF. In The Thirteenth International Conference on Learning Representations, 2025. URL http://wwweeeeooooiioootttttoooo
Team Gemini, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al. Gemini: family of highly capable multimodal models. arXiv preprint 2rXiv:2312.1188000 2023.
Team Gemma, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rívière Mihir Sanjay Kale, Juliette Love, Pouya Tafti, Léonard Hussenot, Pier Giuseppe Sessa, Aakanksha Chowdhery, Adam Roberts, Aditya Barua, Alex Botev, Alex Castro-Ros, Ambrose Slone, Amélie Héliou, Andrea Tacchetti, Anna Bulanova, Antonia Paterson, Beth Tsai, Bobak Shahriari, Charline Le Lan, Christopher A. Choquette- Choo, Clément Crepy, Daniel Cer, Daphne Ippolito. David Reid, Elena Buchatskaya, Eric Ni, Eric Noland, Geng Yan, George Tucker, GGeorge-Christian Muraru, Grigory Rozhdestvenskiy.. Henryk Michalewski, lan Tenney, Ivan Grishchenko, Jacob Austin, James Keeling, Jane Labanowski, Jean-Baptiste Lespiau, Jeff Stanway, Jenny Bremnan, Jeremy Chen, Johan Ferret, Justin Chiu, Justin Mao-Jones, Katherine Lee, Kathy Yu, Katie Millican, Lars Lowe Sjoesund, Lisa Lee, Lucas Dixon, Machel Reid, Maciej Mikuła, Mateo Wirth, Michael Sharman, Nikolai Chinaev, Nithum Thain, Olivier Bachem, Oscar Chang. Oscar Wahltinez, Paige Bailey, Paul Michel, Petko Yotov, Rahma Chaabouni, Ramona Comanescu, Reena Jana, Rohan Anil, Ross Mcllroy, Ruibo Liu, Ryan Mullins, Samuel L. Smith, Sebastian Borgeaud, Sertan Girgin, Sholto Douglas, Shree Pandya, Siamak Shakeri, Soham De, Ted Klimenko, Tom Hennigan, Vlad Feinberg, Wojeiech Stokowiec, Yu hui Chen, Zafarali Ahmed, Zhitao Gong. Tris Warkentin, Ludovic Peran, Minh Giang, Clément Farabet, Oriol Vinyals, Jeff Dean, Koray Kavukcuoglu, Demis Hassabis, Zoubin Ghahramani, Douglas Eck, Joelle Barral, Fernando Pereira, Eli Collins, Armand Joulin, Noah Fiedel, Evan Senter, Alek Andreev, and Kathleen Kenealy. Gemma: Open models based gemini research and technology. arXiv preprint arXiv: 2403.08295, 2024.
Google Cloud. Gemini 2.0 flash, 2025. URL https:///lloud.goggle cttpoooee...m//eeeeeaaaiaii.. seterrativeeaa//occ////odo
gttpni/mmmnneeeem f-fsash
Anisha Gunjal, Anthony Wang, Elaine Lau, Vaskar Nath, Bing Liu, and Sean Hendryx. Rubrics as rewards: Reinforcement learning beyond verifiable domains. arXiv preprint 2.0.: 2025.
Zenan Huang, Yihong Zhuang, Guoshan Lu, Zeyu Qin, Haokai Xu. Tianyu Zhao, Ru Peng, Jiaqi Hu, Zhanming Shen, Xiaomeng Hu, et al. Reinforcement learning with rubric anchors. arXiv preprint ar0iv:2508. 12790, 2025.
12

---

Published as a conference paper at ICLR 2026
Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Os- trow, Akila Welihinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card. arXiv preprint arXiv:2410.21276, 2024.
Hawon Jeong, ChaeHun Park, Jimin Hong, and Jaegul Choo. PRePair: Pointwise reasoning enhance pairwise evaluating for robust iestruction-ooloowiin assessments, arXiv preprint arXiv: 2406. 12319, 2024.
Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7b. arXiv preprint arXiv:2310.06825, 2023.
Michael Krumdick, Charles Lovering, Varshini Reddy, Seth Ebner, and Chris Tanner. No free labels: Limitations of llm-as-a-judge without human grounding. arXiv preprint arXiv: 2503.05061, 2025.
Nathan Lambert, Jacob Morrison, Valentina Pyatkin, Shengyi Huang, Hamish Ivison, Faeze Brahman, Lester James V Miranda, Alisa Liu, Nouha Dziri, Shane Lyu, al. T\" ulu 3: Pushing frontiers in open language model post-training. arXiv preprint arXiv:241 1. 15124, 2024a.
Nathan Lambert, Valentina Pyatkin, Jacob Morrison, LJ Miranda, Bill Yuchen Lin, Khyathi Chandu, Nouha Dziri, Sachin Kumar, Tom Zick, Yejin Choi, Noah A. Smith, and Hannaneh Hajishirzi. Re- wardBench: Evaluating reward models for language modeling. arXiv preprint arXiv: 2403.13787 2024b.
Nathan Lambert, Jacob Morrison, Valentina Pyatkin, Shengyi Huang, Hamish Ivison, Faeze Brahman, Lester James Validad Miranda, Alisa Liu, Nouha Dziri, Xinxi Lyu, Yuling Gu, Saumya Malik, Victoria Graf, Jena D. Hwang, Jiangjiang Yang. Ronan Le Bras, Oyvind Tafjord, Christopher Wilhelm, Luca Soldaini, Noah A. Smith, Yizhong Wang. Pradeep Dasigi. and Hannaneh Hajishirzi. Tulu 3: Pushing frontiers in open language model post-training. In Second Conference on Language Modeling, 2025. URL. http:htts://o/t pttttp:://operreiieee www.pppereeiieeeeeeerrrriidddddo weepiidddiuuGffHHHH. HHGH
Tianle Li, Wei-Lin Chiang. Evan Frick, Lisa Dunlap, Tianhao Wu, Banghua Zhu, Joseph E Gonzalez, and Ion Stoica, From crowdsourced data to high-quality benchmarks: Arena-hard and benchbuilder pipeline. arXiv preprint arXiv:2406. 1939, 2024.
Xuechen Li, Tianyi Zhang, Yann Dubois, Rohan Taori, Ishaan Gulrajani, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Alpacaeval: An automatic evaluator of instruction-followingg models. httpp:://gt thub. cww/ ccmm/tatsu--ab/allacaa.e
2023.
Chin-Yew Lin. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out, pp. 74-81, Barcelona, Spain, July 2004. Association for Computational Linguistics.
URL htthhtts///cc.aanoollo//o/ org/We4-1013
Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. Deepseek-v3 technical report. arXiv preprint arXiv:2412.19437. 2024a.
Yixin Liu, Alexander R Fabbri, Jiawen Chen, Yilun Zhao, Simeng Han, Shafiq Joty, Pengfei Liu, Dragomir Radev, Chien-Sheng Wu, and Arman Cohan. Benchmarking generation and evaluation capabilities of large language models for instruction controllable summarization. In Findings of the Association for Computational Linguistics: NAACL 2024,2022, 2024b.
Yixin Liu, Kejjan Shi, Alexander R. Fabbri, Yilun Zhao, Peifeng Wang, Chien-Sheng Wu, Shafiq Joty, and Arman Cohan. Reife: Re-evaluating inslloction-following evaluation. arXiv preprint arXiv: 2410.07069, 2024c.
Team Llama3. The llama 3 herd of models. arXiv preprint arXiv: 2407.21783, 2024.
Xinxi Lyu, Yizhong Wang, Hannaneh Hajishirzi, and Pradeep Dasigi. Href: Human response-guided evaluation of instruction following in language models. arXiv preprint arXiv: 2412. 155244 2024.
Dakota Mahan, Duy Van Phung, Rafael Rafailov, Chase Blagden, Nathan Lile, Louis Castricato, Jan- PhillPh Fränken, Chelsea Finn, and Alon Albalak. Generative reward models. arXiv preprint arXiv:2410. 11.00.22833 /2832, 2024.
13

---

Published as a conference paper at ICLR 2026
Yu Meng, Mengzhou Xia, and Danqi Chen. SimPO: Simple preference optimization with reference- free reward. arXiv preprint WxXi::2455.14734 2024.
Meta AI. Introducing meta llama 3: The most capable openly available 11m to date. https: 2024. Accessed: 2024-05-30.
//ai. www.m...ccmmmmmmmmmm.. wwtp:/coo//oo/meaaa//iinooe/
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedback. In S. Koyejo, S, Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (eds.), Advances in Neural Information Processing Systems, volume 35, pp. 27730-27744. Curran Associates, Inc., 2022.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a method for automatic evaluation of machine translation. In Pierre Isabelle, Eugene Charniak, and Dekang Lin (eds.), Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pp- 311-318, Philadelphia, Pennsylvania, USA, July 2002. Association for Computational Linguistics. doi: 1131111/007303.100311311 URL
org/ 1040
Junsoo Park. Seungyeon Jwa, Meiying Ren, Daeyoung Kim, and Sanghyuk Choi. OffsetBias: 2024.
Leveraging debiased data for tuning evaluators. arXiv preprini
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly reward model. In shirty-seventh Conference on Neural Information Processing Systems, 2023.
Lin Shi, Chiyu Ma, Weicheng Ma, and Soroush Vosoughi. Judging the judges: A systematic investigation of position bias in pairwise comparative assessments by llms. arXiv preprint arXiv: 2406.07791, 2024.
Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Rad- ford, Dario Amodei, and Paul F Christiano. Learning to summarize with human feedback. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 3008-3021. Curran Associates, Inc., 2020.
Prapti Trivedi, Aditya Gulati, Oliver Molenschot, Meghana Arakkal Rajeev, Rajkumar Ramamurthy, Keith Stevens, Tanveesh Singh Chaudhery, Jahnavi Jambholkar, James Zou, and Nazneen Rajani. Self-rationalizztion improves IIm fine-grained judge, 2024. UtL/htts:///ari .rgg / 2410. 05495)
Lewis Tunstall, Edward Emanuel Beeching, Nathan Lambert, Nazneen Rajani, Kashif Rasul, Younes Belkada, Shengyi Huang, Leandro Von Werra, Clémentine Fourrier, Nathan Habib, Nathan Sarrazin, Omar Sanseviero, Alexander M Rush, and Thomas Wolf. Zephyr: Direct distillation of LM alignment. In First Conference on Language Modeling, 2024. http::////////er..iiee.... forum22d=akKAww2667
Haoxiang Wang. Wei Xiong, Tengyang Xie, Han Zhao, and Tong Zhang. Interpretable preferences via multi-objective reward modeling and mixture-of-exee offture-f--expers.. arXiv preprint arXiv:2406. 12845, 2024.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc Le, Denny Zhou, al. Chain-of-thought prompting elicits reasoning in large language models, Advances in neural information processing systems, 35:24824-24882 2022.
Tianhao Wu, Weizhe Yuan, Olga Golovneva, Jing Xu, Yuandong Tian, Jiantao Jiao, Jason Weston, and Sainbayar Sukhbaatar. Meta-rewarding language models: Self-improving alignment with llm-as-a- m-ea--uuee arXiv preprint arXiv: 2407. 19594, 2024.
An Yang. Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang. Haoran Wei, et al. Qwen2. 5 technical report. arXiv preprint 1511 2024.
14

---

Published as a conference paper at ICLR 2026
Michihiro Yasunaga. Leonid Shamis, Chunting Zhou, Andrew Cohen, Jason Weston, Luke Zettle- moyer, and Marjan Ghazvininejad. Alma: Alignment with minimal annotation. arXiv preprint 3.XYv:24122043055.. 2024.
Jiayi Ye, Yanbo Wang, Yue Huang, Dongping Chen, Qihui Zhang, Nuno Moniz, Tian Gao, Werner Geyer, Chao Huang, Pin-Yu Yu Chen, Nitesh V Chawla, and Xiangliang Zhang. Justice or prejudice? quantifying biases in Ilm-as-a-judge. arXiv preprint arXiv: 2410.02736, 2024.
Weizhe Yuan, Richard Yuanzhe Pang, Kyunghyun Cho, Xian Li, Sainbayar Sukhbaatar, Jing Xu, and Jason E Weston. Self-rewarding language models. In Forty-first International Conference on Machine Learning, 2024.
Zhiyuan Zeng. Jiatong Yu, Tianyu Gao. Yu Meng, Tanya Goyal, and Danqi Chen. Evaluating large language models at evaluating instruction following. In The Twelfth International Conference on Learning Representations, 2024.
Lunjun Zhang, Arian Hosseini, Hritik Bansal, Mehran Kazemi, Aviral Kumar, and Rishabh Agarwal. Generative verifiers: Reward modeling as next-token prediction. arXiv preprint arXiv:2408. 15240, 2024.
Qiyuan Zhang, Yufei Wang, Tiezheng YU, Yuxin Jiang, Chuhan Wu, Liangyou Li, Yasheng Wang, Xin Jiang, Lifeng Shang, Ruiming Tang, Fuyuan Lyu, and Chen Ma. Reviseval: Improving LLM-as---judge via response-adapted references. In The Thirteenth International Conference on Learning Representations, 2025. URL /ttpps://ppnreviiiii eeipeerreeie....m. http:iiwwwwee.fooo.eoee waee//ooummidd11111 tBvzOYTLF
Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. Bertscore: Evaluating text generation with bert. In International Conference Learning Representations, 2020.
Shuai Zhao, Linchao Zhu, and Yang. Learning from reference answers: Versatile language model alignment without binary human preference data. arXiv preprint arXiv:250.009895 2025.
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang. Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging Ilm-as judgee with mt-bench and chatbot arena. Advances in Neural Information Prucessing Systems, 36, 2024.
Enyu Zhou, Guodong Zheng, Binghai Wang, Zhiheng Xi, Shihan Dou, Rong Bao, Wei Shen, Limao Xiong, Jessica Fan, Yurong Mou, Rui Zheng, Tao Gui, Qi Zhang, and Xuanjing Huang. RMB: Comprehensively benchmarking reward models in LLM aligmment. In The Thirteenth International Conference on Learning Representations, 2025. URL
kGgr1G
Lianghui Zhu, Xinggang Wang, and Xinlong Wang. Judgelm: Fine-tuned large language models scalable judges. International Conference on Learning Representations, 2023. doi: 10.48550/
arXiv.2310. 17631.
A FULL EXPERIMENTAL RESULTS FOR EVALUATION PROTOCOLS
This appendix provides the complete evaluation accuracy results for all tested prompting protocols, including variants discussed the main text as well as those explored during our broader investigation. All averages are computed over the five datasets: LLMBar-Natura (Nat), (Adv), MTBench (MT), Instrusum (Ins), and HREF. References for reference- methods were generated by GPT-4o unless otherwise specified.
A.1 FULL STATISTICAL ANALYSIS
Table 6 presents the detailed statistical analysis for individual LLM-judges, corresponding to the summarized results in Table of the main text. We report the 95% bootstrap confidence intervals and paired significance tests. By transposing the view (Models rows), observe that RefEval provides a statistically significant improvement over the reference-fre free baselines for every single model tested.
15

---

Published as a conference paper at ICLR 2026
Table 6: Detailed evaluation accuracy (%) by model with 95% Bootstrap Confidence Intervals. *
indicates the method is significantly worse than RefEval (p 0.05). Comparison across 11 Open- Source Models. This table provides the full statistical context for Table 2
Judge Model
LLMBBr-Base
Ref-Free (Ours)
RefEval
gwen-2.5-72b
79.4 (-1.9, +1.8
83.4 +1.8)
(-1.8 +1.8)
llama-3.1-70b
(-1.8.11..
(-1..
85,9 (-1.7,+1.8
gemma-2-277
82.3 (-2.1,
8-.1
(-1.7
gwen-2.5-14b
81.5 (-19,,+199)
8-..,
8-2.,
mistral-nemo
(-2.,
(-22.422)2
(221,44))
gemma-2-9b
80.8 (-1.8,+1119))
80..
8-.7, +2.0)
glm-4-9h
71.8 (-18,+222)
77.3 (-2.0,++22200
(-19,
Ilama-3.1-80
65.0 (22.,+2..3333
71.8 (221,+2.1))
79.4 (222.4222
(wen-2.5-77
73.5 (-19.+2.11)
74.5 (-2.0, +2.1)
(+200,+.2.
llama-3-8h
60.1 (-2.4 +2.5)
72.3 (2211
77.5 (221,
mistral-7b-v0.3
47.0 (-2.6, +2.6)
61.2 (-2.3, 42,4)
69.6 (4.2, 42.4)
A.2 PROMPT PROTOCOL DESIGN AND VARIANTS
We argue that many existing reference-guided approaches, such as HREF-Re (Lyu et 2024
(Figure [3) or reference-augie augmented versions of LLMBar prompts (e.g., Figure 22) Zeng et al. 2024 often treat the reference as supplementary information rather than central anchor for judgment. These methods typically provide general evaluation criteria alongside the reference, without explicit instructions on how reference should be actively used to ground the decision.
In contrast, our core protocols, RefEval and RefMatch, are designed to make the static reference a primary component of the evaluation process. RefEval explicitly instructs the L.MM-udgee to use the provided "Reference Output" benchmark for successful enstrcctonnfollloinn guiding it to compare candidates against this exemplar for factual correctness and overall quality (Figure 10).
RefMatch directs the judge act semantic and stylistic matcher, determining which candidate shows closer similarity to the "ground-trutt Reference Output" based on specific matching rules and understanding of the reference's instruction-followee pattern. Both protocols aim provide more direct and robust guidance on leveraging the reference effectively (Figure T1.
For baseline comparisons, Ref-Free (Ours) prompt (Figure was developed strong reference-free method.
Table 7: Full average evaluation accuracy (%) across five datasets using GPT-4o as the judge, with references generated by GPTT-0 itself.
Method
Nat
Adv
MT
Instrusum
HREF
Avg
RefEval
0.975
0.889
0.800
0.807
0.907
0.876
0.970
0.865
0.820
0.803
0.899
0.871
LLMBar- Base
0.975
0.845
0.798
0.818
0.907
0.869
Metric-Reference
0.970
0.862
0.805
0.793
0.909
0.868
Ref-Free (Ours)
0.960
0.854
0.788
0.810
0.928
0.868
Self-Reference
0.960
0.853
0.805
0.793
0.906
0.863
RefEval
0.960
0.853
0.805
0.793
0,906
0.863
Prepair
0.965
0.865
0.788
0.799
0.876
0.859
CoT
0.975
0.832
0.800
0.788
0.854
0.850
HREF-Ref
0.945
0.801
0.812
0.794
0.888
0.848
RefMatch
0.925
0.842
0.760
0.813
0.888
0.846
0.960
0.851
0.800
0.755
0.835
0.840
HREF-Bas -aEF-Basc
0.950
0.762
0.795
0.800
0.853
0.832
To further explore the design space, also developed variants:
16

---

Published as a conference paper at ICLR 2026
Table Full average evaluation accuracy (%) across five datasets using 11 open-source models as judges, with references generated by GPT-40.
Method
Nat
Adv
MT
Instrusum
HREF
Avg
RefEval
0.868
0.749
0.767
0.745
0.827
0.791
RefMatch
0.846
0.741
0.763
0.729
0.804
0.777
HREF-Ref
0.853
0.623
0.765
0.708
0.792
0.748
ReiMatch-Rules-CC
0.866
0.733
0.757
0.704
0.749
0.762
RefEval-Rules
0.862
0.672
0.760
0.718
0.749
0.752
HREF-Base
0.841
0.540
0.765
0.708
0.773
0.725
Metric-Ref
0.855
0.674
0.750
0.705
0.744
0.746
Ref-Free (Ours)
0.834
0.678
0.709
0.710
0.754
0.737
Prepair
0.817
0.715
0.726
0.688
0.752
0.740
LLMBar-Ref
0,855
0.663
0.745
0.707
0.728
0.740
Self-Ref
0,846
0.667
0.735
0.695
0.724
0.733
LLMBar-Baa -Bss
0.831
0.617
0.746
0.702
0.720
0.723
CoT
0.820
0.601
0.754
0.691
0.696
0.712
Combines RefEyal with a structured list of LLMBBr-sttye evaluation rules, explicitly linking rule adherence the reference.
RefEval-Rules (Figure 17):
Augments RefMatch by requiring a Chain-of-Thought reasoning step before the final similarity judgment.
Ceantch-Rules-Caee (Figure 16
For multi-reference scenarios (Section
we explored:
Multi-Ref Avg (Figure three references.
14
A.6
Asks the judge to consider overall similarity to a set of
Multi-Ref MAX (Figure 15
A.6 Asks the judge to find the best match to any single
reference within set of three.
A.3
PERFORMANCE WITH FRONTIER MODEL. JUDGE
Table 7
presents the full evaluation accuracy across the five datasets for GPT-4o judge.
A.4
OVERALL PERFORMANCE WITH OPEN-SOURCE LLM JUDGES
Table 8 provides the complete average performance of all prompting protocols when using the 11 open- spen-source models as judges.
A.5 PERFORMANCE BREAKDOWN BY OPEN-SOURCE MODEL. SCALE
Tables and 10 present the detailed performance breakdown for larger (29B parameters) and smaller (s9B parameters) open-source model groups, respectively.
Figure presents aggregate performance of LLMBar-Base. Ref-Free (Ours), and RefEval on each of the five datasets, segmented by model capability groups (Larger Models: 29B, including GPT-4o variants; Smaller Models: <9B).
For the larger model group (top panel), RefEval shows clear advantages the challenging LLMBar- Adversarial and HREF datasets, while maintaining competitive performance elsewhere.
For the smaller model group, RefEval provides substantial gains, most notably LLMBar- Adversarial (72.7% for RefEval vs. 56.8% for LLMBar-Base) and HREF (80.8% vs. 64.5%). This consistent improvement across diverse datasets reinforces the robustness and general appli- cability the RefEval protocol, particularly its capacity to elevate the evaluation performance of resource-efficient smaller models.
17

---

Published as a conference paper at ICLR 2026
Table 9: Full average evaluation accuracy (%) for larger open-source models (Qwen-2.5-72B. Llama3.1-70B) as judges, across five datasets, References by GPT-4o.
Method
Nat
Adv
MT
Instrusum
HREF
Avg
RefEval
0.915
0.813
0.795
0.797
0.878
0.840
Metric-Ref
0.935
0.790
0.825
0.773
0.857
0.836
Self-Ref
0.938
0.775
0.816
0.768
0.846
0.829
RefMatch
0.893
0.813
0.793
0.773
0.863
0.827
Ref-Free (Ours)
0.910
0.756
0.811
0.772
0.864
0.823
LLMBar-Ref
0.915
0.786
0.821
0.777
0.832
0.826
HREF-Ref
0.920
0.680
0.820
0.779
0.838
0.807
LLMBar-Baad
0.905
0.735
0.824
0.768
0.835
0.813
RefEval-Rules
0,923
0.769
0.820
0.767
0.809
0.817
Prepair
0.933
0.827
0.785
0.769
0.807
0.824
CTMMatch-Rule-CCon
0.942
0.812
0.819
0.748
0.785
0.821
HREF-Base
0.885
0.601
0.829
0.769
0.820
0.781
CoT
0.897
0.771
0.812
0.743
0.773
0.799
Table 10: Full average evaluation accuracy (%) for smaller open-source models (s 9B parameters) as judges, across five datasets. References by GPT-4o.
Method
Nat
Adv
MT
Instrusum
HREF
Avg
RefEval
0.845
0.727
0.759
0.715
0.808
0.771
RefMatch
0.817
0.701
0.751
0.701
0.759
0.746
HREF-Ref
0.828
0.580
0.741
0,684
0.759
0.718
RefMatch-Rules-Co
0.837
0.696
0.735
0.684
0.722
0.735
HREF-Base
0.817
0.492
0.740
0.674
0.744
0.693
Prepair
0.784
0.678
0.698
0.646
0.725
0.706
RefEval-Rules
0,826
0.629
0.732
0.689
0.688
0.713
Ref-Free (Ours)
0,806
0.652
0.678
0.689
0.702
0.705
Metric-Ref
0.822
0.621
0.718
0.669
0.665
0.700
LLMBar-
0.824
0.611
0.712
0.674
0.648
0.694
CoT
0.785
0.535
0.730
0.666
0.653
0.674
Self-Ref
0.805
0.621
0.703
0.656
0.641
0.685
I.MMBar-Base
0.795
0.568
0.718
0.670
0.646
0.679
Table 11: Full average evaluation accuracy (%) with MultiReeference Strategies using 11 open-source models as judges. "Vote" indicates Multi-Voting Voting Aggregation. Methods without "Vote" (except Multi- (ulti-Prompt) use single GPT-4o reference unless they are inherently reference-free. Averages are over 4 datasets (Nat, Adv, MT, Ins).
Average Accuracy (%) on Datasets
Method
Nat.
Adv.
MT.
Ins.
Avg.
RefEval-Vort
0.885
0.748
0.768
0.756
0.789
RefEval (Single Ref)
0.868
0.749
0.767
0.745
0.782
RefMatch- Vote
0,858
0.748
0.775
0.736
0.779
0.888
0.745
0.755
0.728
0.779
RefMatch (Single Ref)
0.846
0.741
0.763
0.729
0.770
RefManch-Rules-CoTT (Single Ref)
0.866
0.733
0.757
0.765
Mulū-Promppf-Ave
0,857
0.674
0.78T
0.736
0.762
Multi-Prompt-Max
0.863
0.681
0.748
0.702
0.748
Metric-Ref (Single Ref)
0.855
0.674
0.750
0.705
0.746
HREF-Ref (Single Ref)
0.853
0.623
0.765
0.708
0.737
Prepair (OS Avg. Single Ref)
0.817
0.715
0.726
0.688
0.737
LLMBar- Base (OS Avg)
0.831
0.617
0.746
0.702
0.724
18

---

Published as a conference paper at ICLR 2026
Performance by Datastt for Large Models (~913)
100
90.9
$8.4
86.2
84.3
Avg. Accuracy
82.7
80
79.2
78.7............
77.8
7886
73.6
75.2
75.5
Nat
Adv
Ins
HREF
L.M.Bar Base (Ref-Free)
Ref-Free (Ours)
RefEEval (Ours, Ref-Based)
Performance by Darasct Small Models (<=9B)
100
X4.5
Avg. Accuracy
79.5
$0.6
80.8
75.9
75.2
72.7
71.8
71.5
67.8
68.9
65.2
64.5
$6.8
Nat
Adv
MT
Ins
HREF
Figure 5: Aggregate performance by dataset for Larger Models (2 9B parameters, including GPT-4o variants; top panel) and Smaller Models (s9B parameters; bottom panel). RefEval demonstrates consistent improvements across most datasets for both model groups.
A.6 FULL RESULTS FOR MULTI- STRATEGIES
Table II presents the complete results for our exploration of multi-reference strategies, including Multi- Voting and Multi-Prompt variants, averaged across all 11 open-source LLM judges. The single-reference results for RefEval and RefMatch (using GPT-4o reference) are included for direct comparison.
B ADDITIONAL RESULTS ON POINTWISE SCORING
Table 12: Evaluation accuracy (%) of pointwise scoring methods. Pointwise scores are used to infer pairwise preferences. which are then compared against human labels.
Method
Nat
Adv
MT
Instrusum
HREF
Avg.
Average of 11 Open-Source Models
Base-point
77.8
65.0
68.4
60.8 67.4
67.2
67.8
SeiEval--piint 82.7
71.7
73.1
73.1
73.6
Average of GPT-4o and GPT-4.1
Base-point
84.0
74.5
79.9
80.0
RefEvvl--pontt 89.5 87.1
75.2
72.6
83.9
81.7
We conducted additional experiments pointwise scoring 3.11) For this evaluation, adapted our reference-basee RefEval and the reference-free LLMBBr-Base protocols to pointwise scoring formats. namely valeEval-point) and (Base-point). In this setup, the LLM-judge is asked to rate a single model output on a Likert scale from compute evaluation accuracy, the output with the higher score is designated the winner. This
(see prompt at Figure 23 and Figure 24). To
19

---

Published as a conference paper at ICLR 2026
Table 13: Model registry and metadata described in $3.3 All models are post-trained.
Name
Size
License
Description
gemma-2-99
9b Gemma
Gemma is family of open models from Google
gemma-2-276
27b
Gemma
(Gemma et al. 2024). These instruct-tuned versions.
GLM-4-9B is an open-source version of the latest gener-
glm-4-9b
9b
GLM-4
ation of pre-trained models launched by Zhipu Al (Du)
al. 2022).
Ilama the latest open models from Meta Al (Meta AI 2024), pretrained on 1ST tokens.
1amma--8bb
8b
llama 3 Community
llama-3. 1-bb
8b
Ilama 3.1 Community
Ilama 3.1 collection offers a series of multilingual mod-
lamaa-31-700
70b
llama 3.1 Community
els that outperform many open and closed chat models on industry benchmarks (Llama3 2024).
qwen-2.5-7h
7b
Qianwen
Qwen is family of models built by Alibaba Cloud (Bai jtt 2023). Qwen2.5 are recent additions to this serees. featuring strong performance.
gwen-2.3-14b
14b
Qianwen
qwen-2.5-72b
72b
Qianwen
Instruction-tuned
version of Mistral-7B v0.3
mistral-7b-v0.33
7b
Apache 2.0
model (Jiang et al. 2023) from Mistral AL.
DeepSeek V3 represents the latest models from
deepseek-v3
DeepSeek License
DeepSeek AL building on their V2 architec- ture (DeepSeck-Al et al. (024).
Mistral Nemo a 12B parameter model developed by Mistral Al in partnership with NVIDIA Blog
mistral-nemo
12b
Mistral AI Non-Prod.
Gemini 2.0 Flash is part of Google's 's latest generation of capable multimodal models (Gemimi et al. 2023).
gemini-2.00 2.0-fllh
Proprietary
Baude-3.5-somme
Proprietary
Claude 3.5 and 3.7 Sonnet are advanced proprietary
Laade-3.... 3-ssoneet
Proprietary
models by Anthropic PBC (Claude 2023
gpt-4o
Proprietary
GPT-4o and GPT-4.1 are powerful proprictary models from OpenAl (Achiam et al. 2023
gpt-4.
Proprietary
inferred preference then compared the ground-truth human label, allowing us to use the same accuracy metric as in our main pairwise experiments.
As shown in Table 12 the efference-guider RefEval-point method consistently outperforms the freeeence-free Base-point baseline for both the average of 11 open-source models and for the stronger frontier model group (GPT40 and GPT-4.1).
C ADDITIONAL ANALYSES AND ABLATIONS
C.1 IMPACT OF REFERENCE QUALITY
Evaluation Robustness. We investigated whether our proposed RefEval method relies heavily on "Oracle" quality references (c.g., GPT-4o). Table shows the performance of Llama-3.11700 and Qwen-2.5-7B as judges when guided by references from significantly smaller/weaker models 12B and Tulu-2-7B). The results demonstrate that while stronger references yield the highest accuracy, even references from 7B models provide a clear signal that improves over the reference-free baseline.
Training Robustness. We further tested the robustness of our reference feference-guiiedd ceff-improvement pipeline (S??) by replacing the DeepSeek-V V3 references with those generated by GPT-4o-mini. Table [15] shows that even with weaker references, the RefEval pipeline enables effective self- improvement, outperforming the reference-free equivalent.
20

---

Published as a conference paper at ICLR 2026
Table 14: Ablation of Reference Source Quality on Evaluation Accuracy (Avg over datasets: Nat, Adv, MT, Ins).
Judge Model
Reference Source
Method
Avg. Acc.
GPT-4o (Oracle)
RefEval
0.856
Nemo-12B
RefEval
0.850
Llama-3.1-70B
Tulu-2-7B
RefEval
0.834
None
Ree-Free
0.832
GPT-4o (Oracle)
RefEval
0.735
Nemo-12B
RefEval
0.726
Qwen-2.5-7B
Tulu-2-7B
RefEval
0.731
None
Ref- Ref-Free 0.7I0
Table 15: Ablation of Reference Source Quality on Training (Llama-3-8B-Instrues
Method
AlpacaEval
Arena-Hard
Base
25.0
27.1
Reference Source:
DeepSeek-V3
V3-Distill
42.2
V3-RefFree
67.5
53.8
V3-RefEval
73.1
58.7
Reference Source: GPT-4o-mini
4o-mini-Distill
28.7
40.7
4o-mini-ReffFre DominiiReefEvee
42.6
41.7
58.3
C.2 INTER-JUDGE AGREEMENT
We analyzed the consistency of open-source judges by computing the average pairwise agreement between all 11 models. Table 16 shows that RefEval significantly increases agreement across all datasets compared to Ref-Free, suggesting that references reduce the variance in subjective judgment.
Table 16: Average Pairwise Inter-Judge Agreement (%) among 11 open-source models.
Dataset Group
Ref-Free
RefEval
Difference
LLMBar-Nature
80.98
85.31
+4.33
SlMBaarAdvvesariie
75.59
77.61
+2.02
MTBench
75.35
82.42
+7.07
Instrusum
76.95
79.69
+2.74
HREF
74.17
81.83
+7.66
Average
76.61
81.37
+4.76
C.3 MARGINAL. GAINS OF MULTI-REFERENCE VOTING
We analyze the performance gain from increasing the number of references used in voting ensemble (Figure 3.. Table 17 shows the average accuracy of RefEval using voting ensembles of increasing size, drawn from pool of diverse frontier models. While adding references consistently improves performance, the marginal gain diminishes, justifying our focus on single-reference efficiency in the main experiments.
D DATASET DETAILS
This section provides further details on the datasets used for evaluating the LLM-as-a-Judee protocols described in 33. All datasets consist of instances with input instruction and two candidate model
21

---

Published as a conference paper at ICLR 2026
Table 17: Marginal gains from increasing reference count (Average across 11 judges).
Configuration
# Refs
Avg. Accuracy
Gain
Single Best (Avg)
81.4%
Pairwise Ensemble (Simulated)
81.8%
+0.4%
Majority Vote
82.2%
+0.4%
Majority Vote
5
82.3%
+0.1%
outputs, along with a human preference label indicating which output superior or if they tied. For our primary evaluation metric (accuracy), ties are typically excluded or handled according to the original benchmark's protocol if specified for pairwise win-rate calculations. Our reported accuracies are averaged over two evaluation passes, swapping the order of candidate outputs.
D.1 CORE EVALUATION DATASETS
LLMBar-Natural (Nat) and LLMBar-Adversariall (Adv) Zeng et al. 2024): These datasets were designed for meta-evaluating LLM evaluators instruction following. LLMBar-Natural comprises 100 instances collected and filtered from existing human preference datasets, focusing on objective quality differences. L.MBBarAAverssriia contains 319 instances where the dispreferred output is adversarially crafted to possess superficially appealing qualities (e.g., engaging tone, better formatting) that might mislead LLM judge, despite deviating from the instruction. Both datasets feature high anter-annotator agreement (90% for Natural, 95% for Adversarial). We utilize the provided pairwise human preference labels.
MTBench (MT) (Zheng et al. 2024): This benchmark consists of 80 unique multi-turn conversation
prompts spanning eight categories (e.g., writing, roleplay, math, coding). For each prompt, responses from various models are collected. The evaluation involves pairwise comparisons of these responses, judged by strong LLMs (typically GPT-4) based on human-defined criteria, simulating expert human judgments. We use the 200 expert-annotated pairwise comparisons (excluding ties) from the publicly released data, which have an 81% anceotootttoo agreement rate.
Instrusum (Liu et al. 2024b): This dataset focuses instruction-contiie controllable summarization. Each instance includes a source document, a specific summarization instruction (c.g., varying length, style, or focus), and model- -enerrted summaries. Human annotators provide pairwise preference labels for summaries based on adherence to the instruction and overall quality. We use the 411 instances from this dataset that have perfect signal for our meta-evaluation. The instructions in Instrusum are notably longer and more complex on average compared to other datasets.
agreement (100%) ensure high-quality, low-noise
(Lyu et al. 2024): The Human Response- Guided Evaluation of Instruction Following (HREF) benchmark provides human-written instructions and corresponding human-written reference re- sponses. For our study, we selected subset of the HREF human agreement set, which contains pairwise comparisons of model podeel-genertted outputs against these instructions, annotated by humans for preference. Specifically, we utilized instances from the following five task categories: 1) Classification (cls), 2) Closed QA (cqa), 3) Extraction (ext), 4) Generation (gen), 5) Rewriting (rew).
HREF
We excluded task categories such as summarization (to avoid overlap with Instrusum), brainstorming (brn), and open QA (oqa) to maintain dataset diversity and focus, or due their more subjective nature which might introduce variance and noisiness.
We present the number of instances for each dataset in Table 18
D.2 CREATION OF HUMAN ORACLE REFERENCES
In Section 3.6
investigate the impact of exceptionally high-quality human references, particularly for strong LLM M-jdgge on challenging tasks, we created "Human Oracle" references for a subset of the ALMBar-AAvvessariaa dataset (Zeng al. 2024).
The creation process involved randomly selecting 23 instances where our standard GPT-4o judge (using its own generated reference) had previously made evaluation errors against the dataset's
22

---

Published as a conference paper at ICLR 2026
Table 18: Number of instances for each evaluation dataset detailed in Appendix D
Dataset
No. of Instances
L.M.Bar-Nauue (Nat)
100
LLMBBrr-dversan (Adv)
319
MTBench (MT)
200
Instrusum (Ins)
411
HREF CLS
56
HREF CQA
73
HREF EXT
64
HREF GEN
70
HREF- REW
92
Total
1385
original human annotations. For these selected instances, an NLP expert (a co-author of the paper) revised the initial GPT-4o-generated references. Critically, this revision was performed blindly: the expert was provided only with the original instruction for each instance and did not see the candidate model outputs (Output A and Output B) the original human preference label. The task to create an optimal reference answer for the given instruction, focusing on correctness, completeness, and precise adherence all instructional requirements. This methodology ensured the oracle references were developed independently of the specific outputs they would later be used to evaluate, providing rigorous test of reference quality impact.
E TRAINING DETAILS
Here, we provide additional training details ($ 4.2) We use number of epochs of 2, batch size of 128, maximum learning rate of 5e-6, with a linear learning rate scheduler and 3% warmup steps. The maximum sequence length is 2048 tokens and the training instances that exceed this length are filtered out, resulting in 883K training instances.
F MODEL REGISTRY
Table 13 lists details on the models used in this research.
G DETAILS OF INSTRUCTION CLASSIFICATION FOR ALPACAEVAL AND ARENA-HARD
In $4.3 we classify the instructions in AlpacaEval and Arena-Hard into different categories to further compare the reference-based I.L.Mjudges' performance against the baselines. Specifically, GPT-4o to classify the instructions into 4 categories: Coding&Matth, Creative Tasks, Information Seeking, and Reasoning&PPaaningg The prompt template used is shown in Figure 8
Figure 6 and Figure demonstrate the instruction type distributions of AlpacaEival and Arena-Hard, respectively. They show that AlpacaEval contains more open-ended instructions, while Arena-Hard has a larger emphasis on coding and math reasoning.
H PROMPT TEMPLATES
This section provides the prompt templates used for our proposed methods and selected baselines discussed in the main paper. Placeholders like (INSTRUCTION), (OUTPUT {OUTPUT 2). and [REFERENCE] are filled at runtime.
Below provide all the prompt templates used in this work.
23

---

Published as a conference paper at ICLR 2026
2128
Cratre
32.0
Coeking
Figure 6: Distribution of different instruction types in AlpacaEval.
CON
3005
1106
Figure 7: Distribution of different instruction types in Arena-Hard
1. Ref-Free (Ours): Figure
2. RefEval: Figure
10
3. RefMatch: Figure
4. HREF-Base (Lyu et al.
2024
Figure
5. HREF-Ref (Lyu al. 2024
Figure
6. Multi-Ref Avg: Figure
14
7. Multi-Ref Max: Figure
15
8. ReiMatch-Rules-CoTT Figure 16
9. RefEval-Rules: Figure
Metric-Ref: Figure
19
11. Prepair: Figure 20 and Figure
12. Relf-Ref (Zeng al. 2024 Figure 22
24

---

Published as a conference paper at ICLR 2026
Instruction Classification
User Message:
are helpful assistant that classifies prompts/instruuttonss one 4 categories:
1. Coding & Math
2. Information Seeking
Reasoning & Planning Creative Tasks
Respond with ONLY ONE of these category NUMBERS 2, 4), with additional explanation text.
Figure 8: Utility prompt classification of instruction type introduced in 4.3
Ref-Free (Ours)
System Message:
You are helpful assistant helps rate Al models' responses to instructions.
User Message:
Decide which output better following the instruction. The two outputs generated by two different chatbots respectively.
Here are some aspects to consider:
Outputs should precisely follow the instruction. an output contains unrelated information
or does not complete each and all requirements in the instruction, it means that output does not precisely follow the instruction.
You should check for factual correctness and accuracy of outputs. If output contains factual errors (especially with numbers), it should be considered lower quality.
3. Outputs should contain only brief effective response without any verbose explanation, unless the instruction explicitly asks for explanation.
4. The order in which the outputs are presented to you should NOT affect your judgment.
Select which output, "Output (a)" or "Output (b)", is better following the instruction. Your should ONLY contain: "Output (a)" or "Output (b)".
# Instruction: [INSTRUCTION:
# Output (a): (OUTPUT_1)
# Output (b): (OUTPUT 2}
# Which is the better. "Output (a)" or "Output (b)"? Your answer should ONLY contain either "Output (a)" or "Output (b)":
Figure 9: Prompt template for our Ref-Free (Ours) baseline prompting method introduced in
25

---

Published as a conference paper at ICLR 2026
RefEval
System Message:
You are helpful assistant that helps rate Al models' responses to instructions.
User Message:
Decide which output better following the instruction. The two outputs generated by two different chatbots respectively.
An effective and factually correct Reference Output is provided to aid your evaluation. This Reference Output demonstrates successful instruction-oolloing Here are some aspects consider:
1. Outputs should precisely follow the instruction. If an output contains unrelated information does complete each and all requirements in the instruction, it means that output does not precisely follow instruction.
2. You should check for factual correctness and accuracy of outputs. If an output contains factual (especially with numbers), it should be considered lower quality. Compare the output against the Reference Output verify if that output factually correct.
3. Outputs should contain only brief effective response without verbose explanation, unless the instruction explicitly asks for explanation.
4. Understand how the Reference Output properly delivers helpful, accurate, and natural response, and then compare how closely an output matches this successful Reference Output.
Extrancous content in an output that goes beyond what is present in the Reference Output should be discouraged.
6. The order in which the outputs are presented you should NOT affect your judgment.
Select which output, "Output (a)" or "Output (b)", is better at following the instruction. Your answer should ONLY contain: "Output (a)" or "Output (b)".
# Instruction:
[INSTRUCTION]
# Reference Output: (REFERENCE)
Output (a): (OUTPUT 1}
# Output (b): (OUTPUT_2)
# Which is the better, "Output (a)" or "Output (b)"? Your answer should ONLY contain either "Output (a)" or "Output (b)";
Figure 10: Prompt template for our RefEval prompting method introduced in
26

---

Published as a conference paper at ICLR 2026
RefMatch
System Message:
You are helpful assistant tasked with comparing how similar outputs ground-truth Reference Output. Your goal to determine which oulput demonstrates closer similarity the reference.
User Message:
You will be given Output (a) and Output (b) for the Instruction, and ground-truth Reference Output. Rules similarity comparison:
1. The Instruction determines what to match for extrancous information or incorrect number of elements means match, even if there are word overlaps
Surface-level similaritics (word matches, format) are not considered matches they don't satisfy the Instruction requirements
First understand how the ground-trui -ound-truth Reference Output properly follows the Instruction to see what successful answer looks like, then compare how closely Output (a) and Output (b) match this proper instruction-folle fottiion---ooiinngggg pattern
4. Extraneous content in an output that goes heyond what is present in the Reference Output should be discouraged
Compare how each output relates the ground-truth Reference Output. Before comparison, identify which aspects of the ground-truth Reference Output essential to match given the context of the Instruction.
Then determine which output demonstrates closer similarity to the ground-trun Reference Output. You should answer using ONLY "Output (a)" or "Output (h)". Do NOT output any other words.
Ground-truth Reference Output: (REFERENCE)
# Instruction: (INSTRUCTION]
# Output (a): (OUTPUT_I]
Output (b): (OUTPUT.2)
# Which similar the Reference Output, Output Output (b)? Your response should ONLY be either "Output (a)" or "Output (b)" verbatim:
Figure 11: Prompt template for the RefMatch prompting method introduced in
27

---

Published as a conference paper at ICLR 2026
HREF-Basse LLRE--Base(Lyu al., 2024)
System Message: You helptul assistant that helps rate an Al model's responses instructions.
User Message:
Decide which response from the Al system following the instruction is better, considering the following questions:
1. Does the response precisely follow the instruction? For example, response that includes unrelated information does not fulfill the task not precisely following the instruction.
2. Is the response helpful? For example, the instruction asks for recipe for healthy food, and the response is useful recipe, then you can consider it helpful.
Is the response language natural? For example, Al responses are often verbose or repetitive, which is not natural.
4. Is the response factual/accurate? Al responses often make up information. For example, if response claims Donald Trump the current U.S. president, then you should consider inaccurate.
5. Based your aestheties, which do you prefer? For example, you might prefer poem over another poem.
Select the response
that you prefer. Your answer should ONLY contain: A B.
Now is the real task, just select among: A or B.
# Task:
## Instruction:
(INSTRUCTION]
## Response A: (OUTPUT_1)
## Response B: (OUTPUT_2)
## Which the hest, "A" or "B"? Your should ONLY contain either "A" "B":
Figure 12: Prompt template for the HREF-Base prompting method from Lyu al. (2024).
28

---

Published as a conference paper at ICLR 2026
HREF-Ref (Lyu et al.) 2024)
System Message:
You are helpful assistant that helps us rate an Al model's responses to instructions.
User Message:
Decide which response from the Al system following the instruction is better, considering the following questions:
Does the response precisely follow the instruction? For example, response that includes unrelated information or does not fulfill the task is not precisely following the instruction. Compare each response with the provided human response to decide if response faithfully follows the instruction, especially when the instruction asks for expected word count or format.
2. Is the response helpful? For example, the instruction asks for recipe for healthy food, and the response is useful recipe, then you can consider it helpful.
3. Is the response language natural? For example, Al responses often verbose repetitive, which not natural. Compare with the provided human response to decide whether a response is natural.
4. Is the response factual/accurate? Al responses often make up information. For example, if response claims Donald Trump the current U.S. president, then you should consider it inaccurate. Compare with the provided human response to verify whether response is factual and accurate, especially with numbers.
Based on your aesthetics, which one you prefer? example, you might prefer one poem over another poem.
Select the response
B that you prefer. Your answer should ONLY contain: A or B.
Now is the real task, just select among: or B.
# Task:
## Instruction:
[INSTRUCTION]
## Response A: (OUTPUT_1)
## Response B: (OUTPUT 2}
## Human Response: (REFERENCE)
## Which is the best, "A" or "B"? Your answer should ONLY contain either "A" or "B":
Figure 13: Prompt template for the HREF-Ref prompting method from Lyu et al. (2024).
29

---

Published as a conference paper at ICLR 2026
Multi- Ref Avg
System Message:
You are an expert Al assistant tasked with identifying which two outputs more closely matches multiple Reference Outputs for a given Instruction.
User Message:
You will be given Instruction, Output (a), Output (b), and three Reference Outputs.
Determine which Output (a) Output (b) has higher degree of similarity to the set Reference Outputs, while considering the context of the Instruction.
# Instruction:
[INSTRUCTION]
# Reference Output 1:
[REFERENCE.] 1}
Reference Output 2: (REFERENCE.2]
# Reference Output 3: (REFERENCE.3)
# Output (a):
(OUTPUT }}
Output (b): (OUTPUT_2)
# Decision (You should carry concise reasoning. Conclude your reasoning with either "Therefore, Output (a) is overall more similar the Reference Outputs." "Therefore, Output (b) is overall more similar to the Reference Outputs." VERBATIM. Always state which is more similar at the end. In your explanation, always use "Output (a)" or "Output (b)" to refer to the two outputs.):
Figure 14: Prompt template for the Ref Avg prompting method. Details in A.2
30

---

Published as a conference paper at ICLR 2026
Multi-Ret Ref MAX
System Message:
You are an expert Al assistant tasked with selecting the output that best matches any the provided Reference Outputs.
User Message:
will be given candidate Output (a) and candidate Output (b) Instruction, and three Reference Outputs.
Compare candidate Output (a) and candidate Output (b) to each of the three Reference Outputs individually. Identify if there's standout match between any candidate output (a h) and any single Reference Output. Select the candidate output that matches best with ANY ONE of the Reference Outputs while considering the context the Instruction. # Instruction:
[NSTRUCTION]
# Output (a):
(OUTPUT_I)
# Output (b):
(OUTPUT_2) 2
# Reference Output 1:
[REFERENCE_]]
Reference Output 2:
[REFERENCE 2)
# Reference Output 3:
(REFERENCE.3)
Decision (You should carry concise reasoning. Conclude your reasoning with either "Therefore, Output (a) has a best match." or "Therefore, Output (b) has best match." " verbatim. Always state which has best match AT THE END. In your explanation, always use "Output (a)" or "Output (b)" to refor to the two outputs.):
Figure 15: Prompt template for the Multi-Ref MAX prompting method. Details in A.2
31

---

Published as a conference paper at ICLR 2026
ReMMatch-RulesCol
System Message:
You are an expert Al assistant tasked with comparing how similar two outputs are to Reference Output. Your goal is to determine which output demonstrates closer similarity to the reference.
User Message:
You will be given Output (a) and Output (h) for the Instruction, and a Reference Output.
Rules for similarity comparison:
1. The Instruction determines what to match for extraneous information or incorrect number of elements means match, even if there word overlaps
2. Surface-level similarities (word matches, format) are not considered matches if they don't satisfy the Instruction requirements
First understand how the Reference Output properly follows the Instruction, then compare similarity based on this proper instruction instraction-folloinng
Compare how each output relates to the Reference Output.
Before comparison, identify which aspects of the Reference Output are essential match given context of the Instruction. Then determine which output demonstrates closer similarity to the Reference Output.
Instruction:
[INSTRUCTIO]]
# Output (a): (OUTPUT_1)
# Output (b): (OUTPUT 2}
Reference Output: REFERENCE
# Decision (You should carry brief reasoning. Conclude your reasoning with either "Therefore, Output (a) shows closer similarity the Reference Output." "Therefore, Output (b) shows closer similarity to the Reference Output." VERBATIM. Always state which shows closer similarity at end. In your explanation, always use "Output (a)" or "Output (b)" to refer to the two outputs.):
Figure 16: Prompt template for the MefMatch-Rules-Con prompting method variant. $ A.2
32

---

Published as a conference paper at ICLR 2026
RefEval-Rules
System Message:
You are helpful assistant in evaluating the quality of outputs for given instruction. Your goal to select the best output for the given instruction.
User Message:
Select the Output or Output (b) that better for the given instruction. The outputs are generated by two different Al chatbots respectively.
A ground-truth Reference Output is provided to aid your evaluation. This Reference Output demonstrates successful instruction-followie and help inform your judgment.
Here are some rules of the evaluation:
You should prioritize evaluating whether the output honnestlyprrecieely/loosyyy executes the instruction, then consider helpfulness, accuracy, detail, harmlessness, etc.
2. Outputs should NOT contain more/less than what the instruction asks for, as such outputs do NOT precisely exccute the instruction.
3. You should avoid any potential bias and your judgment should be as objective as possible. For example, the order in which the outputs were presented should NOT affect your judgment, as Output (a) and Output (b) are **equally likely** to be the better.
Use the ground-truth Reference Output to understand what a successful answer looks like. Evaluate whether Output (a) or Output (b) achieves similar effectiveness as the Reference Output in addressing the instruction's 's requirements.
Do NOT provide any explanation for your choice. Do NOT say both neither are good.
You should answer using ONLY "Output (a)" or "Output (b)". Do NOT output any other words.
Instruction: [NSTRRCTIOO]
# Output (a): (OUTPUT_1)
# Output (b): (OUTPUT-2)
Ground truth Reference Output: (REFERENCE)
# Which is hetter, Output (a) Output (h)? Your response should be either "Output (a)" or "Output (b)":
Figure 17: Prompt template for the RefEval-Rule prompting method variant. $..
33

---

Published as a conference paper at ICLR 2026
CoT
[System Message]
You are helpful assistant in evaluating the quality of the outputs for given instruction. Your goal is to select the best output for the given instruction.
[User Message]
After giving brief explanation, select the Output (a) or Output (b) that is better for the given instruction. The two outputs are generated by two different Al chatbots respectively.
Here are some rules the evaluation:
(1) You should prioritize evaluating whether the output honestly/preciselylllossey executes the instruction, then consider its helpfulness, accuracy, level of detail, harmlessness, etc. (2) Outputs should NOT contain more/less than what the instruction asks for, such outputs do NOT precisely execute the instruction.
(3) You should avoid any potential bias your judgment should be objective as possible, example, the order in which the outputs were presented should NOT affect your judgment, as Output (a) and Output (b) are equally likely to be the better.
You should first provide brief explanation of your ovaluation, and then always end your re- sponse with either "Therefore, Output (a) is better." or "Therefore, Output (b) is better." verbatim. Do NOT say both / neither are good.
Do NOT output any other words.
NOT say "Output (a) better" or "Output (b) better" the beginning. You should do reasoning and thinking before claiming which better.
# Instruction: (INSTRUCTION]
# Output (a): (OUTPUT 1}
Output (b): (OUTPUT.2)
# Decision (Give brief explanation of your evaluation followed by either "Therefore, Out- put better." "Therefore, Output (b) is hetter." verbatim. Always claim which is better at the end. your explanation, you should always use "Output (a)" or "Output (b)" to refer to the two outputs respectively.):
Figure 18: Prompt for cot prompting method described in
34

---

Published as a conference paper at ICLR 2026
Metric Reference
[System Message]
You are helpful assistant in evaluating the quality of the outputs for given instruction. Your goal to select the best output for the given instruction.
[User Messagel
Seleet the Output (a) or Output (b) that better for the given instruction. The two outputs are generated by two different Al chatbots respectively.
Here
rules of the evaluation:
(1) You should prioritize evaluating whether the output lonessly/precisely/looeey executes instruc- tion, then consider its helpfulness, accuracy. level of detail, harmlessness, etc.
(2) Outputs should NOT contain more/less than what the instruction asks for, as such outputs do NOT precisely execute the instruction.
(3) You should avoid any potential bias and your judgment should be objective possible. For example, the order which outputs were presented should NOT affect your judgment, as Output (a) and Output (b) are equally likely to be the hetter.
Do NOT provide any explanation for your choice. NOT say both neither are good. You should answer using ONLY "Output (a)" or "Output (b)". Do NOT output any other words.
# Instruction: (INSTRUCTION]
# Output (a): (OUTPUT 1}
Output (OUTPUT_2)
# Questions about Outputs:
Here most three questions about the outputs, which presented from most important to least important. You can do the evaluation based on thinking about all the questions. (QUESTIONS)
# A reference output generated strong Al assistant:
(REFERENCE]
# Which is better, Output (a) or Output (b)? Your response should be either "Output (a)" or "Output (b)":
Figure 19: Prompt for metric + reference prompting method described in
35

---

Published as a conference paper at ICLR 2026
Prepair (pointwise analysis)
[System Message]
You are helpful assistant in evaluating the quality of the outputs for given instruction. Your goal to evaluate the quality of output for the given instruction.
[User Message]
Giving a brief explanation evaluate the quality the response to the given instruction. output generated by an AT chatbot.
Here
rules of the evaluation:
(1) should prioritize evaluating whether the output honessly/preciselylloooeyy executes instruction, then consider its helpfulness, accuracy, level of detail, harmlessness, etc. (2) The model outputs should NOT contain more/less than what the instruction asks for, as such outputs do NOT precisely execute the instruction.
(3) You should avoid any potential bias and your judgment should be as objective possible.
You should provide a brief explanation of your evaluation. Your explanation should identify critical drawbacks in model outputs that do not meet the above evaluation rules.
# Instruction:
[ISTTRUCTIO]]
Output: (OUTPUT)
# Provide your explanation:
Figure 20: Prompt for prepair prompting method described in 3 This is the prompt for pointwise analysis (the first stage) within the method.
36

---

Published as a conference paper at ICLR 2026
Prepair (pairwise evaluation)
[System Message]
You helpful assistant in evaluating the quality of the outputs for given instruction. Your goal is select best output for given instruction.
[User Message]
After giving brief explanation, select the Output (a) or Output (b) that is better for the given instruction. The outputs generated two different Al chatbots respectively.
Here are some rules of the evaluation:
(I) You should prioritize evaluating whether the output bonesttyprrecielydddsssy executes the instruc- tion, then consider its helpfulness, accuracy, level of detail, harmlessness, (2) Outputs should NOT contain more/less than what instruction for, such outputs NOT precisely execute the instruction.
(3) You should avoid any potential hias and your judgment should be as objective as possible. For example, the order in which the outputs were presented should NOT affect your judgment, as Output (a) and Output (b) are **quully likely*# to be the better.
You should first provide a brief explanation of your evaluation, and then always end your response with cither "Therefore, Output (a) is better." or "Therefore, Output (b) is better." verbatim. Do NOT say both / neither are good.
Do NOT output any other words.
Do NOT say "Output (a) is better" or "Output (b) better" at beginning.
You should do reasoning and thinking **bforr*** claiming which is better. Your explanation should identify critical drawbacks in model outputs that do not meet the above evaluation rules.
# Instruction: [ISTTRCCTIO]
Output (a): (OUTPUT 1}
# Output (b): (OUTPUT_2)
# Here's the analysis for each output you wrote carlier: (PER OUTPUT ANALYSES}
Your Response (Provide your evaluation and reasoning, followed by either "Therefore, Output (a) better." "Therefore, Output (b) hetter." verbatim):
Figure 21:
Prompt for prepair prompting method described in This is the pairwise evaluation stage (the second stage) within the method.
37

---

Published as a conference paper at ICLR 2026
Self-Ree Ref-Ref
[System Message]
You are helpful assistant in evaluating the quality of outputs for given instruction. Your goal to select the best output for the given instruction.
[User Message]
Select the Output or Output (b) that better for the given instruction. The two outputs are generated by two different Al chatbots respectively.
Here are some rules of the evaluation:
(I) You should prioritize evaluating whether the output honessly/precxselyllleeey executes the instruction, then consider its helpfulness, accuracy, level of detail, harmlessness, etc. (2) Outputs should NOT contain more/less than what the instruction asks for, as such outputs do NOT precisely execute the instruction.
(3) You should avoid any potential bias and your judgment should be objective possible. For example, the order which the outputs were presented should NOT affect your judgment, as Output (a) and Output (b) are equally likely to the better.
Do NOT provide any explanation for your choice. NOT say both neither are good. You should answer using ONLY "Output (a)" or "Output (b)". Do NOT output any other words.
Instruction:
[NSTRUCTION]
# Output (a): (OUTPUT_1)
# Output (b): (OUTPUT-2)
# A reference output generated strong Al assistant: (REFERENCE)
# Which is better, Output (a) or Output (b)? Your response should be either "Output (a)" or "Output (b)":
Figure 22: Prompt for Self-Ref prompting method described in
38

---

Published as a conference paper at ICLR 2026
Base-point
System Message:
You are helpful assistant in evaluating the quality of the outputs for given instruction. Your goal is to the output for the given instruction.
User Message:
Score the Output on Likert scale from to 5 for the given instruction, where score of one means "poor quality" and score five means "perfect quality". The output is generated Al chatbot.
Here are some rules the evaluation:
You should prioritize evaluating whether the output honessty/preciselyyloooyyy executes the instruction, then consider its helpfulness, accuracy, level of detail, harmlessness,
2. Outputs should NOT contain more/less than what the instruction asks for, as such outputs do NOT precisely execute the instruction.
3. You should avoid any potential bias and your judgment should be as objective as possible.
Do NOT provide any explanation for your choice. You should answer 1, 2, 3, 4, or 5 only. Do NOT output any other words.
# Instruction:
[INSTRUCTIO]]
Output: (OUTPUT)
# What is your rating for the Output?
Figure 23: Prompt for Base-point prompting method described in B
39

---

Published as a conference paper at ICLR 2026
RefEval-point
System Message:
You helpful assistant that helps rate an model's response an instruction.
User Message:
Score the given Output Likert scale from to 5, where score of one means "very poor quality" a score five means "perfect quality".
An effective and factually correct Reference Output is provided to aid your evaluation. This Reference Output demonstrates successful instruction-follooingg
Here are some aspects to consider when scoring:
1. The Output should precisely follow the instruction. If the Output contains unrelated informa- tion does not complete each and all requirements in the instruction, it should be scored lower.
2. You must check for factual correctness and accuracy. If the Output contains factual errors. it should be considered lower quality. Compare the Output against the Reference Output verify factual correctness.
3. The Output should contain only brief effective response without any verbose explanation, unless the instruction explicitly asks for one.
4. Understand how the Reference Output properly delivers helpful, accurate, and natural response, and then evaluate how closely the given Output matches this successful Reference Output.
Extrancous content the Output that goes beyond what present the Reference Output should be discouraged and result in lower score.
You should answer with single digit from to only. Do NOT provide any explanation for your choice.
# Instruction:
[INSTRUCTIO]]
Reference Output:
(REFERENCE]
# Output Score: (OUTPUT)
# What your rating for the Output?
Figure 24: Prompt for RefEval-point prompting method described in
40


Reviews:
[{"review_id": "Bzz5uNtI9f", "reviewer": "Reviewer_yDFd", "summary": "This paper investigates whether high-quality reference outputs can improve LLM alignment through enhanced evaluation and training methods. The authors develop reference-guided LLM-as-a-Judge protocols (RefEval and RefMatch) and demonstrate their effectiveness in both evaluation accuracy and self-improvement training scenarios.", "questions": "N/A", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "* Evaluation across diverse datasets (Natural, Adversarial, MTBench, Instrusum, HREF, AlpacaEval and ArenaHard) and models (Qwen, Llama, GPT, Gemma, GLM, etc)\n* Useful settings and approaches. RefEval provides explicit guidance on using reference outputs as benchmarks for instruction-following quality. RefMatch provides a good baseline to compare.", "weaknesses": "I mostly concern experiment design.\n* Missing comparison to recent reference-based methods like RevisEval beyond a brief mention\n* No systematic study of how reference quality affects downstream performance\n* Unclear how to obtain high-quality references in domains where frontier models struggle\n* No analysis of how reference diversity affects evaluation robustness, especially for aspects that reference responses miss.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "XwPcp92OCN", "reviewer": "Reviewer_2bbc", "summary": "The paper asks whether giving judges and students a good reference answer helps align smaller LLMs. It introduces three prompt styles for judging—Ref-Free (no reference), RefEval (compare to a reference), and RefMatch (pick the candidate closest to the reference)—and then uses references twice: (i) SFT to distill style/quality from strong references, and (ii) DPO where a reference-guided judge labels on-policy pairs for preference training. Across roughly 5 datasets and 11 judges, reference-guided prompts improve judge reliability, and the SFT→DPO pipeline lifts 7–8B models on common leaderboards (e.g., AlpacaEval, Arena-Hard). Overall, it’s a practical recipe: simple prompts, references as grounding, and a two-stage training loop that competes with a reward-model baseline without extra human feedback.", "questions": "Reference robustness: How do results change if you paraphrase, shorten, or inject minor errors into references, or use weaker LLMs as references? Any curve of judge accuracy vs. reference quality?\n\nBaseline coverage: Can you add at least one strong SimPO/ORPO/KTO line and one fine-grained baseline (e.g., token/segment-level DPO or span-supervised MT like TWA) under the same data and compute?\n\nTraining schedule: Beyond two-stage SFT→DPO, can you try mixed SFT+DPO (interleaved or joint loss) and report its effect vs. your current recipe?\n\nJudge calibration: How sensitive are outcomes to the particular judge and prompt? Any comments on the inter-judge agreement? Any human-correlation study for Ref-Free vs. RefEval/RefMatch?\n\nMulti-reference: (1) I don't see the multi-reference results in Figure 3. (2) Do multi-reference votes always help? What’s the marginal gain of 1→2→3 references? How did you create the multiple references?\n\nSignificance: Please add 95% CIs (bootstrap) and a simple permutation test for key deltas in main tables.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The paper asks a clear, practical question—can we ground LLM judges and training with strong references—and answers it with simple, reproducible tooling (Ref-Free, RefEval, RefMatch) and a clean SFT→DPO pipeline. The experimental setup is broad (many judges, several datasets) and the gains are consistent: reference-guided judging improves agreement/utility, and reference-distilled SFT followed by preference optimization moves mid-size models meaningfully on common leaderboards. I also like that prompts and templates are easy to reuse, and that multi-reference variants are explored rather than just a single “gold” answer.", "weaknesses": "Methodologically, the novelty feels incremental: reference-guided evaluation and reference-distilled training have both appeared before, and the paper mostly scales and systematizes them rather than introducing a new objective or learning signal. The comparisons also underplay strong preference-optimization baselines (e.g., SimPO, ORPO, KTO) and fine-grained supervision relevant to credit assignment (token/segment-level DPO; span-supervised MT like TWA). \n\nRobustness is not fully stress-tested: results rely on high-quality references, but we don’t see sensitivity to noisy, weaker, or stylistically different references, nor clear evidence against style coupling to the reference generator. Finally, statistical reporting is light (few CIs/permutation tests), and most end metrics are still LLM-judge based; more targeted human spot-checks would help address circularity.\n\nMissing citation and baseline:\nRevisEval: Improving LLM-as-a-Judge via Response-Adapted References — Qiyuan Zhang, ICLR 2025.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "GFYMdZgVy0", "reviewer": "Reviewer_r7ga", "summary": "The paper studies using references from frontier LLMs for LLM alignment. Specifically in domains where verifiable rewards are not available, the quality of less capable LLM evaluators can be much improved by guiding those with references from frontier LLMs.", "questions": "In section 4.3 you wrote\n\n (3) V3-Distill is the SFT model\ndistilled from DeepSeek v3 references. The following models are finetuned from V3-Distill: (4) ROUGE,\nwhich uses ROUGE scores as the reward5\n; (5) BERTScore, which uses\nthe BERTScore metric (Zhang et al.,\n2020; Zhao et al., 2025); (6) ArmoRM, which uses the ArmoRM reward model;\n\nWhat does it mean to finetune ROUGE, BERTScore etc to be finetuned from V3-Distill?", "limitations": "", "rating": 10, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "1. The proposed approach is a step towards something similar to RLVR for non-verifiable domains, which makes this work interesting to the broad community.\n2. Comparisons with strong baselines from literature are provided. In addition the authors design a strong reference-free baseline that is directly comparable to their referenced based approach in terms of prompt quality.\n3. Experiments with sources from different frontier LLMs are conducted so that the results are not specific to a particular frontier LLM. This validates the the generalization of the proposed approach.\n4. Finally, the paper shows results for actual preference alignment training using the reference guided evaluator. The results conclusively show that the proposed approach is superior.", "weaknesses": "1. The baselines in table 4 are not as strong. What I mean is that some of the baselines for table 4 should probably have been based on the baselines in table 1. If we are to choose a evaluator approach based on results from table 1, we would like to know whether the results in table 1 can serve as a robust tool for choosing a llm evaluator.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "9s2FsxazuF", "reviewer": "Reviewer_P3EF", "summary": "This paper studies how high-quality reference outputs can be used to improve LLM alignment. The authors first design prompting strategies that allow LLM-based evaluators to compare candidate answers with a reference, showing consistent accuracy gains across 11 open-source models on 5 datasets. They then extend the idea to training, where the model generates multiple outputs, ranks them with a reference-guided judge, and performs preference optimization (DPO). Experiments on AlpacaEval and Arena-Hard show that reference-guided self-improvement outperforms reference-free methods and reaches performance comparable to reward-model-based approaches, without requiring additional human feedback.", "questions": "Q1: What criteria are used to judge that a reference output is \"high quality,\" and how sensitive is the method to different levels of reference quality?\n\nQ2: Since GPT-4o is used to generate reference outputs and the evaluation benchmarks (e.g., AlpacaEval, Arena-Hard) are also judged by GPT-4 family models, how can we be sure that the improvements indicate stronger human preference alignment rather than closer imitation of GPT-4o-style responses? Incorporating human evaluation or cross-family judging would better support the alignment claims.\n\nQ3: Can the proposed method work when references are unavailable or noisy in new domains, and what are the expected failure conditions when references deviate from correct or complete content?", "limitations": "", "rating": 2, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "**Strength 1**: The study evaluates 11 open-source LLMs across five human-annotated datasets (LLMBar-Natural/Adversarial, MTBench, InstruSum, HREF), covering diverse architectures and capability tiers. This broad and systematic evaluation strengthens result credibility and supports reproducibility.\n\n**Strength 2**: Reference-guided approaches such as RefEval and RefMatch consistently outperform both reference-free methods and prior reference-based baselines. Evidence from Table 1, Table 2, and Table 4 shows notable gains in both judgment accuracy and downstream alignment tuning, highlighting the practical effectiveness of references.\n\n**Strength 3**: The analysis reveals stronger relative gains for smaller models and demonstrates that higher-quality references (e.g., human-edited) further improve judge performance. Results in Table 3 and Figure 4 provide helpful insights into when reference guidance is most beneficial.", "weaknesses": "**Weakness 1**: While the paper offers a more systematic and larger-scale study, the core concept of reference-guided judging remains closely related to prior work. In addition, recent studies like Self-Rationalization Improves LLM as a Fine-Grained Judge[1] similarly explore LLM-as-a-Judge and preference-based self-improvement. A clearer discussion of methodological differences and contribution boundaries would help clarify the incremental novelty introduced by leveraging external references.\n\n**Weakness 2**: The pipeline uses frontier LLMs to generate references and also as strong evaluators in benchmarks like AlpacaEval and Arena-Hard. This may cause circular style or preference bias, making it difficult to confirm whether improvements reflect better human alignment or closer imitation of the reference model family.\n\n**Weakness 3**: The paper does not provide clear criteria or measurements for selecting or validating high-quality references. This makes it difficult to assess generalization, especially when references may vary in coverage or factual reliability across datasets.\n\n[1]Trivedi, Prapti, et al. \"Self-rationalization improves llm as a fine-grained judge.\" arXiv preprint arXiv:2410.05495 (2024).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)