LFQA-E: Carefully Benchmarking Long-form QA Evaluation
Authors: Yuchen Fan
Chen Ling
Xin Zhong
Shuo Zhang
Heng Zhou
Yuchen Zhang
Mingyu Liang
Chengxing Xie
Ermo Hua
Zhizhou He
Cheng Huang
Gang Chen
Ning Ding
Bowen Zhou
Venue: ICLR 2026 Poster (2026)
Keywords: LFQA
Evaluaton
Benchmark
Primary area: datasets and benchmarks
TL;DR: We introduce a benchmark on Long-form QA evaluation and provide further analysis.

Abstract: Long-Form Question Answering (LFQA) involves generating comprehensive, paragraph-level responses to open-ended questions, which poses a significant challenge for evaluation due to the richness of information and flexible response format. Existing LFQA-evaluation benchmarks often lack reference answers and are limited in size and topic coverage, reducing their reliability. To address this gap, we introduce LFQA-E, a well-constructed, multilingual, and reference-based benchmark designed to rigorously evaluate automatic metrics for LFQA. LFQA-E comprises 1,625 questions and 7,649 pairwise comparisons across 15 topics, drawn from diverse sources such as online queries and examination questions, thereby enabling a comprehensive assessment of evaluation metrics. We examine five categories of metrics, encompassing 17 specific methods, using LFQA-E. The results demonstrate that none of the existing automatic metrics perform comparably to human judgments, highlighting their inability to capture the dense information in long-form responses. Furthermore, we present a detailed analysis of the failure cases and the generalization capacity of these metrics, offering insights to guide the future development of LFQA evaluation methods.

Accepted at ICLR 2026
LFQA-E: CAREFULLY BENCHMARKING LONG-FORM QA EVALUATION
Yuchen Fan 1.2.*, Chen Ling",, Xin Zhong",, Shuo Zhang', Heng Zhou', Yuchen Zhang? Mingyu Liang', Chengxing Xie?, Ermo Hua Gang Chen Zhizhou He², Cheng Huang Ning Ding Bowen 2..
Shanghai Jiao Tong University Shanghai AI Lab
Tsinghua University
4 Beijing University of Posts and Telecommunications
5 Zhejiang University
University of Tokyo
"qqual contributions 'Corresponding author
yuchenfan 48 @gmmmi...... com
thoubowen@tssigguueedtu
ABSTRACT
Long-Form Question Answering (LFQA) involves generating comprehensive, paragraph-Ievel responses to open-ended questions, which poses a significant challenge for evaluation due to the richness of information and flexible response format. Existing LFQA-evaluation benchmarks often lack reference answers and are limited in size and topic coverage, reducing their reliability. To ad- dress this gap, we introduce LFQA-E, well-constructed, multilingual, and reference-based benchmark designed to rigorously evaluate automatic metrics for LFQA. LFQA-E comprises 1618 questions and 7323 pairwise comparisons across 15 topics, drawn from diverse sources such as online queries and ex- amination questions, thereby enabling a comprehensive assessment of evalua- tion metrics. We examine five categories of metrics, encompassing 17 spe- eific methods, using LFQA-E. The results demonstrate that none of the exist- ing automatic metrics perform comparably to human judgments, highlighting their inability capture the dense information in long-form responses. Fur- thermore, we present detailed analysis of the failure cases and the general- ization capacity of these metrics, offering insights to guide the future develop- ment of LFQA evaluation methods. The benchmark and code are available at com https:/ / wtwpuu/.cooo/cccccaaaaaa///////
INTRODUCTION
Long-form Question Answering (LFQA) Fan et al. 2019) targets at generating in-depth, paragraph- level responses to open-ended questions. It requires models to have comprehensive domain-specific knowledge or use evidence from retrieved documents (Nakano et al., [022] Akash et al.] (023)) to provide relevant and accuracy answers. Despite efforts enhance the quality long-form answers, developing automatic and reliable evaluation metrics for LFQA is still underexplored.
Evaluating long-form answers presents significant challenges, evaluators must possess compre- hensive domain knowledge. Previous manual evaluations typically employed crowd-sourced work- ers for annotation. However, their limited domain expertise inevitably compromises reliability. In contrast, expert annotation would ensure higher quality, while the cost of employing experts to an- notate large-scale datasets is prohibitive. Consequently, automatic evaluation metrics are essential. In automatic evaluation of LFQA, ROUGE (Lin 20044 has been widely adopted. However, Krishna ct al. (202I) argue that ROUGE provides limited informativeness in long-form contexts, weakening its reliability. With the advancement of LLMs (OpenAl) 2023 2024) and Large Reasoning Models (LRMs) (DeepSeek-Al et al. 2025a Zhang et al. 2025), numerous studies have leveraged these models to develop evaluation metries through various approaches (Chang al.] 2023), including prompting (Wei et al., 2023 Zhou et al. 2025). fine-tuning (Li et [d.3] Fan et al.] 2024a Tiagg
al. 2024), and training LLMs Reward Models (RMs) (Liu et al. 2024al Chen et al. 2025

---

Accepted at ICLR 2026
Categories
LFQA-B Example
Paychology. 6.80%
Medicine,. 8. 0.20%
Engineering. 300%
Q: Why doesn't kidney pain get worse with hydration?
Biology 7.70%
Law, 6.10%
Ref: One of the major causes of kidney stone pain is ureteral colic... The idea behind drinking lot fluid is that you will produce more urine and propel...
Long-form Response
Expert Reference
Other, 8%
Hard Comparison
Politiss 5.20%
Geography, 8.20%
Technology
Resp1: From what understand, kidney stones cause arount
Identify Core
11 3.30%%
pain..cause the ureters themselves to swell to pass the greater volume of fluid, which then prevents kidney stone from getting stuck and causing the painful spasms.
Potential Solution: Produce more urines
Physics, 8.50%
History,
6. 10%
Chemistry.... 4%
Planetary Staence 5.50%
Mathemedics, 3.40%
Economics, 8,70%
Discard Useless
Resp2: does. When in the hospital for kidney stone and medical professional sympathizes with me about the pain and talks about how that_- scraping painful, know right away that they have no idea what they are talking about...
Sources
Settings
Around Topic: But no specific solution
# # MONRETIGON
DEYSENN
Resp1 is better
EL15
Which one better according to the reference?
Figure 1: The figure shows the overview of LFQA-E. The left side displays the categories, sources, and three settings, showcasing its diversity. The right side illustrates an example of LFQA-E.
either generative scalar-based. Despite the advances of evaluation metrics, determining which metrics are most effective and best aligned with human judgment for LFQA evaluation requires systematic verification and benchmarking.
Previous benchmark for LFQA evaluation (Xu et al. 2023) samples records from reddit/ELIS, hir- ing experts annotate the better one between two responses without references, and test alignment between automatic evaluation metrics and expert labels. However, their benchmark has several limitations: 1) Lack of authorized references reference answer provides baseline for assess- ing whether a response covers key details and maintains factual accuracy. Without ground-truth references, the comparison between metrics may be unfair, and evaluations without clear criteria or rubrics are inherently unreliable. 2) Limited diversity The benchmark consists of only 260 examples, all in English, constraining its linguistic and topical diversity. Moreover, it treats the comparison an A/B task, but in real scenarios, "tie" option always exists.
To fill the gap, we introduce LFQA-E, towards evaluating the ability of different metrics. 1) To evaluate whether current automatic evaluation metrics select better from two nuanced re- sponses, we gather references that are examined by the experts, and judge based on them. To ensure the diffieulty, we choose human responses based their upvotes their scores, and model sponses based on two models with comparable capabilities. 2) To analyze the systematic differences in validity among evaluation metries, rigorously assess the performance from several aspects. First we evaluate them based on three settings, i.e, human vs human (h v. h). human vs model (h m), and model model (m m). Moreover, we collect multilingual responses, i.e, English and Chinese, and multiple domain-specific responses, e.g., Engineering, Law, Medicine, to ensure thorough analysis. 3) To prevent data contamination, we collect data from offline examination, i.e. College Entrance Examination Simulation Questions (CEESQ) and Postgraduate Entrance Ex- amination Questions (PHEQ) and online platform questions (reddit/FL115) from the recent half-year. The overview of LFQA-E Benchmark is shown in Figure
Using LFQA-E, we critically assess the efficacy of 17 evaluation metrics. The experimental results show that current leading evaluation metrics fail to capture core information as human beings from verbose responses when differentiating the better one between two responses with similar quality. Furthermore, we provide analysis on why automatic evaluation metries fail in LFQA evaluation and find the misalignment between evaluation metrics. Lastly, we try TTRL (Zuo et al. (2025) to improve the evaluation performance of model-based metrics and provide some actionable insights.
2

---

Accepted at ICLR 2026
2 RELATED WORK
Development of LFQA LFQA Fan et al. 2019) requires models to generate paragraph-Ievel responses to open-ended questions which is more complex compared datasets like SQuAD (Ra- jpurkar et al., 2016), TriviaQA (Joshi et al.) 2017), and Narrative (Kočiský et al. 2017), where answers are primarily words or phrases extracted directly from documents. In LFQA, models must generate comprehensive yet correct responses based on their knowledge or existing evidence docu- ments. Several studies have analyzed the discourse structure of long-form answers (Xu et al. 2022) and have sought enhance the performance on LFQA (Chen al. 2023 Akash al. 2023
Evaluation of LFQA
The automatic evaluation of LFQA remains challenging and underexplored.
For human annotation, HURDLES (Krishna et al. 202T) and WEBGPT (Nakano et al. 2022) employ A/B testing, where sourced annotators swers. Since annotation of LFQA requires high expertise, the results of crowd-sourced workers may be unreliable. To address the gap, Xu al. (2023) employs experts for annotation, and tests several evaluation metrics, such as ROUGE (Lin) 2004), BERTScore (Zhang et al.] 2020), and BARTScore (Yuan et 2021). on an expert-annotated dataset. Their findings validated that existing metrics fully align with human judgment. However, the dataset they used lacks expert-written references, which are sourced from RedditEELI,, and is limited in scale, comprising only about 260 samples. More recently, since the development of LLMs and LRMs, many work uses them for evaluation of free-form answers, using prompt (Wei et 2023), fine-tuning using specific data (Liu al. 2023). and reinforcement-leariing (Chen et al. 2025)
instructed to choose the best of two candidate an-
3
METHODOLOGY
3.1 OVERVIEW
To reasonably test the evaluation ability of different metrics for LFQA when having reference, we introduce LFQA-E, a multilingual and comprehensive benchmark composed of different topies and questions. LFQA-E consists of the Chinese version LFQA-E-ZA and the English version LFQA- E-EN. Table [ shows its overview. includes 1618 questions and 7323 comparisons, consisting of 1193 comparisons in Chinese and 6130 comparisons in English. It spans 15 topics, ranging from history to engineering, ensuring its diversity. LFQA-E comprises expert-annotated references for fair comparison and nuanced responses. Therefore, it is naturally hard yet reasonable benchmark for LFQA evaluation.
Table Detailed statistics of LFQA-E. Avg Que. Lens, Avg Ref. Lens, Avg Res. Lens corre- sponds to question lengths, reference lengths, and response lengths, respectively.
Reference-Based Evaluation For LFQA-E, references are sourced from academic exami- nations or widely discussed questions Red- div/ELS.. After being reviewed by experts with relevant academic backgrounds, these ref- erences are ensured to cover all the key points needed to answer the question. This provides baseline for evaluation metrics to look up and provide a more precise comparison.
LFQA-E-EN LFQA-E-2H
# Topies # Questions
6
1026
592
# Comparisons
6130
1193
# Avg Que. Lens
13.4
24.6
# Avg Ref. Lens
299.1
187.2
Difficult Comparisons All the questions contained in LFQA-E have been carefully ex- amined by domain experts to ensure is an- swerable and clear to understand. We ensure that models have not seen the data by collecting data from recent examinations and forum questions. The responses are collected from human-wring re- sponses, with close scores or upvotes, and model responses generated by comparable LLMs. There- fore, it is hard to distinguish the better one at a glance.
# Avg Res. Lens Annotate
245.0
308.3
Expert
Expert
Diverse Benchmark We collect 1618 questions and 7323 comparisons in 15 distinct domains, from natural science to social science, guarantee diverse and representative benchmark. Also, LFQA-E is multilingual, consisting of examples in both Chinese and English. Moreover, LFQA-E includes three kinds of comparisons, guaranteeing the comprehensibility of the benchmark.
3

---

Accepted at ICLR 2026
3.2 DATA PROCESSING
Data Collection For LFQA-E-ZH, we source our data from CEESQ and PEEQ, where ques- tions, references, and scoring schemas are developed by domain experts, including teachers and professors from high schools and colleges. The records are sampled from 2024 and are based on local examinations restored from PDF files that have not been submitted to online platforms. These questions cover diverse subjects, including politics, history, medicine, psychology, law, and geog- raphy. We provide two examples of CEESQ and PEEQ in Appendix [.I] For LFQA-E-ENT data is sourced from Reddit/ELI5, where each question is explained without specialized terminology or complex concepts, and we use the top-ranked answer as our candidate reference. For LFQA-E-ZH, to prevent overlap with potential training data, we avoid using data from actual College Entrance Examinations. All problems are sourced from PDF files and have not been uploaded online. Addi- tionally, the questions captured from ELI5 are all from the past six months. We provide benchmark contamination study in See Sec To ensure that all questions are clear and answerable, we instruct GPT-4o with temperature=0.7 to filter out questions with unclear descriptions. The instruction used is listed in Appendix 1 We conducted an experiment on the effectiveness of leveraging GPT-4o as a filter, and the results reveal that GPT-4o demonstrates superior performance for this task, achiev- ing 97% accuracy. Subsequently, to ensure our references contain all information needed to answer the questions, we submit the remaining data for expert annotation. The annotation guidelines are presented in Table 29. Each reference was annotated by two annotators, and references were dis- carded if either annotator labeled them as invalid. The Cohen's ' kappa coefficient is 0.78, indicating substantial inter-annotator agreement. Through this process, obtained 1618 questions.
Human Response Collection For LIFAA-EZIH we gather examination papers primarily in im- age format and employ Optical Character Recognition (OCR) systems to extract student responses. Specifically, we choose student answers with close scores to ensure the comparison difficulty. The OCR is conducted using the Volcano Engine API. For LFQA-E-EN we collect responses from the forum section of the corresponding question. Also, we select answers within the many-voted yet close up-votes to make them hard to differentiate. However, the responses we collect for LFQA- E-ZH are mainly written during examination, it is concise and well structured, and the responses collect for LFQA-E-EI EN include some special characters like URLs, which deteriorate our data quality. To handle it, we use GPT-4o to paraphrase and clean our human responses. The instruction used is shown in Appendix [ We randomly sample 100 records and annotate the paraphrased responses to validate the performance of GPT-4o, with the results in Appendix [B.]] which indicates that leveraging LLMs doesn't 't introduce any error. We also provide a case study in Appendix [.2]
Model Response Generation When generating model responses, we focus on evaluating whether LLMs can understand the semantic meaning of texts well and properly select the better response. Therefore, we do not impose extremely strict requirements on answer quality. Instead, we ensure the difficulty of LFQA-E BENCH by selecting models with similar ranking in the LMSYS Arena (Chiang et al. 2024 Zheng et al. 2023) 2024). On account of responses generated by stronger mod- els like GPT-4o or Claude-4-sonnet will pose great challenge for our annotators to differentiate, increasing the cost of annotation, we leverage Llama-3-8B-Instte
(Dubey et al. 2024) and GPT- 3.5-turbo (OpenAT 2023) for response generation. For model-generated answers, use "Generate reasonable answers to the following questions. Use references or examples if needed" to prompt LLMs. The generation temperature is set to 1.0 encourage diverse and creative responses.
3.3 HUMAN ANNOTATION
Annotator Decision We hire 10 annotators from relevant aspects or who have taken relevant courses. Then we provide them with clear annotation recipes for better quality control. The an- notation recipe is in Appendix F Each annotator receives 2$ for annotating question, including 4-6 comparisons. To ensure the effectiveness of the annotation, we pre-annotate a subset consisting of 35 records. The annotator will start their work until they reach a 90% consistency on the subset.
Annotation Setting Guided by Xu et al. (2023), our evaluation criteria mainly focus on factuality and completeness according to the reference, since almost all responses we collect are already very fluent. Unlike typical A/B testing, our method employs triple-choice format, giving tie option,
4

---

Accepted at ICLR 2026
to better capture the subtle differences between answers, they often show comparable levels of information overlap with the reference while the other information is useless or verbose according to the central topic which can de dropped without hindering the comprehension of the response.
Table 2:
Performance of evaluation metrics on LFQA-E. The largest value is denoted in bold. We provide the results of other models in Appendix B.4
LFQA-E-EN
LFQA-E-ZH
Model
Avgr1
Avg.te
F1
Accuracy
F1
Accuracy
Static Evaluation Metric
Human Baseline
77.7
83.3
68.9
76.5
73.3
79.9
Length ROUGE
26.0
42.8
33.5
52.6
30.8
47.7
37.5
55.5
34.0
49.7
35.8
52.6
BERTScore
35.9
54.1
36.6
52.4
36.3
53.3
M.LMs-based Evaluation Metric
Breen.5-32B-Instuuee
45.8
56.7
43.8
60.1
Qwcn2.5-72B-Instee
43.1
61.2
39.0
53.0
41.1
57.1
1-ama..--70BIImst
42.5
59.6
29.4
30.7
36.0
45.2
GPT-4o
46.4
61.7
42.6
53.2
44.5
57.5
DeepSeck-V3
39.3
57.9
41.1
53.8
40.2
55.9
RM-based Evaluation Metric
Skywork- Skywork-Reward-1
37.3
54.4
38.2
53.6
37.8
54.0
Skywork-Rewad Gemma
Beeward-Gemma
37.5
56.0
33.0
48.3
35.3
52.2
RM-R TT1-RI-Qwe22.5-Inntte I-nsttuccttt
36.4
64.9
35.1
51.9
35.8
58.4
RM-R Dest-e-Depecc---isillll-----eee
43.9
65.7
36.7
53.2
40.3
59.5
LRM-baseed Evaluation Metric
ol-mini
45.9
45.2
58.9
45.6
60.9
Deepscek-RI
42.9
59.6
42.4
57.8
42.7
58.7
Trained Evaluation Metric
Auto-J--B-bilingual
46.0
66.8
35.4
51.9
40.7
59.4
Prometheus-7B-..
41.8
64.2
34.1
50.1
38.0
57.2
M-Prometheus-1444
41.6
60.8
33.9
49.4
37.8
55.1
The annotators assess two responses against given reference and select the more informative and complete answer or declare a "tie" both are of similar quality. During the process, treat piece of information the basic unit FActScore (Min al., 2023). Initially, annotators extract the key information needed to answer the question from the provided reference and check whether the responses under evaluation contain similar statements. Then, they will select a better one based on the overlapped information. To minimize bias and subjectivity, each record annotated by two independent reviewers. Each comparison takes around minutes annotate. comparisons, detailed justification is saved to help understand. After
Annotation Process
For some annotation, we find the Cohen's 's kappa correlation of inter-annotator agreement approximately 0.65, indicating substantial agreement. We show screenshot in Figure
4 EXPERIMENTS
4.1
MODELS
We evaluate various metrics on LFQA-E-EN and LFQA-E-ZE respectively, including: Static Met- rics: We use Length-orientation, ROUGE-1 (Li 2004) and BERTScore (F1) Zhang et al. (2020)
since they are widely used as the evaluation metric for LFQA. LLMs:: We select Owen2.5-320
Instruct (Qwen et al.) 2025). Deepseek-Ve V3 (DeepSeek-Al et al. 2025b), and GPT-4o (OpenAl et al. 2024). LRMs: Considering
Llama-3.1 1.mma-3.1-70B5 18-70B-IIntten (Dubey et al.) 2024),
the high time complexity and cost, we use ol-mini and Deepseek-RI (DeepSeck-Al et al. 2025a)
RMs: We test on Skywork-Rewse Rww.-eewadddrre Beeaard-Gemma-2277BBB
(Liu et al.) 2024a). Skywork Metoork-Rewaar Reward-
1.ama-3.1-BB-ve
considering their leading position Reward Bench (Lambert al.) 2024).
We also test RM-RI --wee2.55-nnstttee castruct-14B (Liu et al. 2024b) and RM- Re---eepseek--isiiie Distilled- Qwen- 14B since they represent another paradigm of reward models. We refer to Skywork-
5

---

Accepted at ICLR 2026
1aamma---27--000 and Rewook-Reward-Llmmaee 3B-v Skywork- Reward- Gemmma and Skywork-Rews Reward-Llama for simplification. Evaluation-Specifee Models: There are some mod- els trained be evaluation models. Among these models, select Auto-J-Bilingus (Li et al. 2023). Prometheus-7B-22.. and M-Prometheus-14B (Kim et al. 2024).
4.2
IMPLEMENTATION DETAILS
We evaluate all the metrics in both LFQA-E-EN and LFQA-E-ZII We Jieba cut for ROUGE- zh. For BertScore, we use roberta-large for LFQA-E evaluation and bert-base-chinese for LFQA-E-ZH. We set the temperature 1.0 for all LLM-based evaluation metrics to encourage diverse responses. We show the results of temperature shown in Appendix E.2 We further conduct prompt-sensitivit analysis in Appendix [B.2 For models with specific training templates, we adopt them. We include references for models to look up in all our settings. We accuracy and macro-F1 as indicators:
in Sec The prompts we used are
N
Acc
(pred; label,)
(1)
N
1=1
P.R. P& R.
F1 macro
2
C
where C = [A, B, tie). For LLM-based methods, include "tie" option in the instruction, while for other methods that return a scalar, we round the scalar to decimal places. For the human baseline, hire another 3 annotators with doctor's degrees ensure the quality. The final scores are obtained by averaging annotators' results. We use the annotation recipe as previously claimed.
4.3 MAIN RESULTS
Table[liste our results. The overall low accu- racies and Fi-scores of all evaluation metrics indicate the challenge LFQA-E poses cur- rent models and methods. We also provide a cost analysis in Appendix(B.6
Table 3: Performance of different models com- parisons that humans labeled as tie. The largest value in each column is in bold.
Model
LFQA-E-EN
LFQA-E-ZH
Comparison Between Metries Though none of the evaluation metrics achieves a high performance on LFQA-E, we observe that scaling model size doesn't definitely yield better result. For example, Qwen2.5- 32B-Instruct beats Qwen2.5-72BB-Insing by 3%. What's more, LRMs show a great performance compared with LLMs, thanks their long CoT and extended thinking. RM-based evaluation metrics don't show promising results when generalizing to LFQA evaluation, perhaps because they are trained give better between two responses, renouncing the "tie" option. We will analyze further and give a fairer comparison in Section B.5
Deepseek-V?
1.8
10.2
Qwen2.5-32BB-In
7.2
7.5
Qwen2.5-72B-Instruu
2.6
3.6
Llama-3.1-70B-Inntmmic
7.7
ol-mini
7.1
14.1
Deepseek-RI
7.4
7.2
GPT-4o
9.2
14.6
Comparison Between Indicators All evaluation metrics struggle to give tie as good as human beings. Table indicates that among the evaluation metrics we test, the best result just 9.2% for LFQA-E-EN and 14.6% for LFQA-E-ZHR Observing the responses, we find that they are too conservative to claim two responses are of equal quality. This explains why accuracy is always larger than Macro-F1. The low accuracy tie comparison reflects the difficulty of LFQA-E again.
Specialized Tuned models Help Boost Performance. Observing the table above, find that tuning a base model to be a robust generative reward model helps in LFQA evaluation. Moreover, when changing the base model to strong reasoning model, the performance gains continually. In addition, SFT models can also achieve performance comparable with models of larger sizes. These phenomenon indicates that tuning is essential for LFQA evaluation.

---

Accepted at ICLR 2026
human human
human va model
model model
human vs human
human model
model vs model
70
05
00
9.5
1.4
12.7
11.6
7.0
60
40
Despeeok-R1
01-miri
Deepppek-V3
RMM-lama
Deepsook-R1
o1-mini
Deepeeok-V3
0 RMM-lamm
(a) LFQA-E-EN
LFQA-E-ZH
Figure 2: Performance of different models on our three settings on LFQA-E.
ANALYSIS
5.1 TEMPERATURE MATTERS
To ablate the effect of temperature on our validation, we further experiment using temperature equals 0.0 for deterministic results. The result in Table&ssoows a noticeable shift in metrics. For instance, the performance of LLM-based Evaluation Metrics such as Qwen2.5-32B-Instee InI-ructt and GPT-4o are higher compared to when the temperature is set to 1.0. This suggests that deterministic behavior likely reduces the model's variability in responses, leading more consistent evaluation results. However, the LRM-based evaluation metrics show significant drop, indicating that models relying on more exploration perform worse or even collapse under deterministic conditions.
Table 4: Performance metrics on LFQA-E when temperature is set to The largest value is denoted in bold.
LFQA-E-EN
LFQA-E-ZH
Model
AvgF1
Avg
F1
Accuracy
FI
Accuracy
LLMs-based Evaluation Metric
Twen2.5-32BBIInse
46.0
64.5
40.0
54.0
43.0
59.3
Swwen.55-72B-Instre
41.8
58.6
38.5
52.7
40.2
55.7
Llama3,1-70B-Innss
36.8
53.5
40.5
50.4
38.6
52.0
GPT-4o
49.5
63.1
43.9
52.7
46.7
57.9
DeepSeek-V? -3
40.1
57.8
40.9
55.1
40.5
56.5
LRM-based Evaluation Metric
ol-mini Deepseek-RP R1
43.4
56.2
4.3
5.8
23.9
31.0
2.0
2.7
32.1
45.3
17.1
24.0
5.2 EVALUATION METRICS CAN'T EXCEL ALL SETTINGS.
To have a thorough understanding of whether the model evaluates human response or model sponse differently, we experiment on a different group of LFQA-E. We break it into three groups, i... h W. h, v. m, and m v. m, where indicates human response and represents model response, and see the accuracy changes. The results are listed in Figure 2a for LFQA-E-EN and Figure 2b for LFQA-E-ZIL. We can observe that for many evaluation metrics, there exists huge difference between different comparison settings. In LFQA-E-EN, the RMs show steady ability while others exhibit degradation when model responses are introduced. LFQA-E-ZH, all the metrics show a drastic accuracy decline under m, with maximum drop of 14.2% from Deepseck-V33 This further validates our assumption that current evaluation metrics can't differentiate between two nu- anced responses. also show the performance across subjects in Appendix B.3

---

Accepted at ICLR 2026
Owan 1.302
0.362
0.014
0.172
0.003
0.014
1.00
Qwen
6.75
e1-mini-
0,561
1.003
0.394
6.111
0.423
0 0.005
62-minl
9.229
0.50
DeepseeeV33-
0.214
1.002
+0.22
0.25
Deepsesk-V3
3,533
9.238
1.00
0.54
0.092
0.421
0..00
Auto-j-
0.25
Auto-j
2.503
0.242
0.:41
1.002
0.472
GPT-4o
6,077
Usma 0,014
4,,05
4.,05
6,093
4,021
1,000
Uama
9,303
9.923
9.421
0.
1.388
0.78
1,500
Qwenn
mimie
Deepatook 3g
Aute-)
0PP40
Ownn
al-mii
Demplesk
Autte
(P 4p
Liana
(a) LFQA E-EN
LFQA-E--- B--ZH
Figure 4: The Cohen's Kappa Correlation Matrix in LFQA-E.
5.3 REASONS EVALUATION METRICS FATL WHEN EVALUATING LONG-FORM RESPONSES.
For LM-based Evaluation Metric We observe the outputs of several LLMs and find that almost all errors arise from the following aspects.
The model fails correctly identify and separate bullet keypoints enumerated lists in responses, leading to poorly structured answers.
Keypoints Identification Error:
Irrelvvan/lncorrect Information Error: The model does not penalize or filter out irrelevant or factually incorrect details in its responses, reducing accuracy.
Contradiction Error: During reasoning, model generates inconsistent or contradictory statements due to factual hallucinations.
Formatting Error: The model produces responses with improper format.
We show the probability of each error occurring in Figure3 We choose Deepseek- V3 and o1-mini for representation. Point Identification Error and Irrelevant/nnorreet Information Error happen most time, indicating the relatively low inherent ability for LMs when evaluating long-form answers.
Static Evaluation Metrics These methods simply leverage word-level or embedding-level similarities, which scratch surface when evaluating. As de- scribed in Fan et al. (2024b), when considering eval- uating two long responses around topic, there may be many words overlapping. Also, overly long re- sponses dilute semantics, making originally impor- tant key information trivial, so metrics fail to con- sider informativeness, but only focus similarity.
Identiicaaiion
Contradiction
5.4 DIFFERENT EVALUATION METRICS DON'T AGREE WITH EACH OTHER.
0%
25%
50%
75%
10%
Figure 3: Percentage of error types for LMs on the LFQA-E dataset.
To find whether there is a correlation between dif- ferent evaluation metrics, we observe detailed eval- uation results. Specifically, we select ROUGE Qwen2.5-32B-Inntruu
(simplified as Qwen), GPT-4o, Skywork-Reward-LLane (simplified as Llama), ol ol-mini, and Auto-J-6B-biling ter performance on LFQA-E. Figure 4a and 4b show the results. We observe that neither of the two metrics achieves high correlation, indicating two metrics may contradict each other to large degree. There are even some negative correlations between the two metrics under LFQA-E -EEENN--N EN. This phenomenon further illustrates that there is stable evaluation result across different metrics.
(simplified as Auto-J), considering their relatively bet-
8

---

Accepted at ICLR 2026
5.5 BENCHMARK CONTAMINATION ANALYSIS
To examine whether our evaluation benchmark has potential contamination from pretraining cor- pora, conducted perplexity (PPL) and n-gram overlap, the same Xu et al.] (2024). us- ing two widely adopted open-source instruction-tuned models: 1,1 a1lama-3. ---8--nnstttnt ruct.. The underlying intuition is that evaluation data memorized during pretraining, the models would exhibit abnormally low perplexity and high n-gram overlap the evaluation benchmark.
and
Perplexity Analysis. Table 5 reports the PPL values on both English and Chinese sub- sets. For Qwen2. 5- 7B Instruct, the PPL is 11.60 (en) and 11.73 (zh). For 1aaaa-..1-8---- Inst Instrrttt, the PPL is 11.70 (en) and lower at 7.21 (zh), which all fall within tolerable limits and low data contamination potential.
Table Perplexity (PPL) on the benchmark.
Table 6: n-gram accuracy on the benchmark.
Model
en
zh
Model
en
zh
Qwen2.5-7B Bien2.5-7B-Instreen IBI-nuutcttt
11.60
11.73
Brren..5-B-IInttere
0.030
0.047
Llama-3. Baama-3.1-BB---- 3.1-8B-Instrrct
11.70
7.21
Llama-3.1 1-8B-Instruct
0.025
0.093
w-gram Overlap. We further computed n-gram exact match accuracy between the benchmark and model generations. shown in Table [ the overlap scores remain low across both models, e.g., 0.025-0.030 in English and 0.047-0.093 in Chinese. These values significantly below contamination thresholds observed in prior work, reinforcing the view that large-scale memorization unlikely to happen in benchmark.
Overall, both analyses suggest that the bench- mark is not heavily contaminated. The English subset appears relatively safe across both mod- els. For the Chinese subset, while Llama-3.1- 8B-Instruct shows somewhat lower perplex- ity and higher n-gram overlap, though n-gram higher than expected, remain within acceptable bounds and don't indicate severe memorization.
Table 7: Performance of TTRL.
Model
LFQA-E-EN
Qwen2.5-3B-Instee
CoT
49.6
Structured Prompt
59.7
TTRL
63.9
5.6 COMPARISON
TTRL + Clip Higher
66.5
WITH ESTABLISHED BENCHMARKS
Qwen2.5-7B-Instrucl
To demonstrate the value and distinctiveness of our LFQA-E benchmark, we conduct a com- parative analysis against several established baselines using frontier models. Our findings consistently show that LFQA-E presents a sig- nificantly greater challenge, thereby better dif- ferentiating the capabilities of state-of-the-art models.
CoT
53.3
Structured Prompt
60.6
TTRL TTRL Clip Higher
68.2
68.6
General LLM Evaluation. First, we compare LFQA-E with the expert-annotated benchmark from Xu et al. (2023), denoted "Expert" and Feedback-Bench Kim et al.) (2024). As shown in Table 8 all leading models achieved substantially lower scores LFQA-E compared to the baselines. This performance gap validates that LFQA-E probes deeper into the models' reasoning and evaluation capabilities rather than merely surfacing superficial knowledge.
Reward Model Evaluation. We further extend our comparison specialized reward model
benchmarks, including RM-Bench Liu et al. (2024c) and Reward-Bennn BenrchhBBBhhhhh Lambert et al. (2024). The results, presented in Table(9] reaffirm our conclusion. The consistent performance drop LFQA-E across various reward models highlights its robustness and difficulty, even for systems specifically designed for evaluation tasks.

---

Accepted at ICLR 2026
Table 8: Performance comparison between LFQA-E and general evaluation benchmarks.
Model
Feedback-Bench
Expert
LFQA-E
Swen2.5-32B-Instrun
86.8%
74.2%
60.1% %
Qwen2.5-72B-Insten
94.4%
72.7%
57.1%
GPT-4o
89.2%
70.0%
57.5%
GPT-5
75.6%
68.9%
62.5%
DeepSeek- V3
83.6%
65.8%
55.9%
DeepSeek-R1
83.6%
68.9%
58.7%
Table 9: Comparison with specialized reward model benchmarks. results for that benchmark.
N/A denotes missing standard
Model
Reward-Bench
RM-Bench
LFQA-E
75.3%
69.5%
52.2%
71.8%
72.6%
54.0%
Gemini-2.5-fllash
77.7%
N/A
60.6%
RM-RRI-DeepSeeck-issiii wttpisstiiild--w-----llle -wen---eennnn--
N/A
71.8%
59.5%
RM-RI RMM-RI--Owea--nstruc-11111
N/A
75.6%
58.4%
5.7 TTRL BooST PERFORMANCE
Here, we try improve the performance LFQA-E through prompting and reinforcementt learning (RL). Firstly, we use a structured prompt to instruct models to embrace their response within kanswer>...// answer> tag. This leads to a non-trivial performance increase, considering the relatively lower instruction following ability of small language models. Building on this, since RL widely used to improve the reasoning ability of LLMs (Cui et al. 2025 Fan al. 2025). we use RL to leverage the performance of LLMs on LFQA evaluation. Considering the lack of high-quality data, we implement TTRL. (Zuo al., 2025) on our LFQA-E-BR as an example. We configure our model with batch size of 8, rollout temperature of 1.0, and generate 32 rollouts per prompt. The learning rate is 5e-7. During validation, the temperature 0.0 for consistent results. The reward signal is based Al et al. 2025a). As shown in Table 7 TTRL yield a substantial performance boost, demonstrating the effectiveness of using RL for this task. Also, the response length grows steadily during the training until the training rewards converge. However, we observe a rapid convergence where all rollouts produce identical preferences, which limits further improvements. We attribute this the underlying three-category classification which is easily overfitted when the model is over-confident. For remedy, we implement clip-higher mechanism used in DAPO (Yu al., (025). Also, we observe a performance boost, indicating that when we increase the diversity during rollouts, RL can take effect more stably. However, more sophisticated methods and high-quality data are needed for a better evaluation metric that mimics human preferences.
outcome-based rule, similar to the approach used in Deepseek-RI (DeepSeek-
CONCLUSION
We introduce LFQA-E, multilingual benchmark for LFQA evaluation. It consists of 1625 ques- tions and 7649 comparisons, spanning 15 topies, from natural science social science, consisting of 3 settings, i.e., h h, h v. and m m. Each records include clear question, an authorized ref- erence, and two hard-to-diiferreniaate responses, ensuring its difficulty. We conduct experiments on 15 automatic evaluation metrics. The results show that none of the metrics can evaluate long-form responses as well as human beings. We further analyze the generalization different metrics across languages and settings. The results further indicate that all models struggle generalize well to all comparisons. We find that LRMs and specifically trained evaluation models lead on LFQA-E. The test-time-scaled evaluation model may be used enhance the performance of I. L.FAA evaluation.
10

---

Accepted at ICLR 2026
ETHICS STATEMENT
This paper proposes a benchmark consisting of questions and references from examination papers and online forums. We perform data filtering ensure that there offense in the benchmark data.
REPRODUCIBILITY STATEMENT
We have provide our settings comprehensively for data construction and evaluation so that the result be reproduced using the same settings and prompt.
ACKNOWLEDGEMENT
We thank the China Construction Third Engineering Bureau Group Co.,Ltd. for their support.
REFERENCES
Pritom Saha Akash, Kashob Kumar Roy, Lucian Popa, and Kevin Chen-Chuan Chang. Long-form question answering: An iterative wlanning-rerrieval-geener approach, 2023. URL https:
T/axxiv.org/abs///
09383
Yupeng Chang, Xu Wang, Jindong Wang, Yuan Wu, Linyi Yang, Kaijie Zhu, Hao Chen, Xiaoyuan Yi, Cunxiang Wang. Yidong Wang, Wei Ye, Yue Zhang, Yi Chang, Philip S. Yu, Qiang Yang, and Xing Xie. survey on evaluation of large language models, 2023. URL https:
org/abs/ 2307. 03109
Hung-Ting Chen, Fangyuan Xu, Shane Arora, and Eunsol Choi. Understanding retrieval augmen- tation for long-form question answering, 2023. URL httppe://arrivv.rrrrbbb/////
12150
Xiusi Chen, Gaotang Li, Ziqi Wang, Bowen Jin, Cheng Qian, Yu Wang, Hongru Wang, Yu Zhang, Denghui Zhang. Tong Zhang, Hanghang Tong, and Heng Ji. Rm-rl: Reward modeling as reason- ing, 2025. URL L petttps://axiinn org. abs/ 25055/255..o0 02387
Wei-Lin Chiang, Lianmin Zheng, Ying Sheng, Anastasios Nikolas Angelopoulos, Tianle Li, Dacheng Li, Hao Zhang, Banghua Zhu, Michael Jordan, Joseph E. Gonzalez, and Ion Stoica. Chatbot arena: An open platform for evaluating lims by human preference, 2024.
Ganqu Cui, Lifan Yuan, Zefan Wang, Hanbin Wang, Wendi Li, Bingxiang He, Yuchen Fan, Tianyu Yu, Qixin Xu, Weize Chen, et al. Process reinforcement through implicit rewards. arXiv preprint arXiv:2502.01456, 2025.
DeepSeek-AI, Daya Guo, Dejian Yang. Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang. Xiao Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, Z. F. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, Aixin Liu, Bing Xue, Bingxuan Wang, Bochao Wu, Bei Feng, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai Dai, Deli Chen, Dongjie Ji, Erhang Li, Fangyun Lin. Fucong Dai, Fuli Luo, Guangbo Hao, Guanting Chen, Guowei Li, H. Zhang, Han Bao, Hanwei Xu, Haocheng Wang, Honghui Ding. Huajian Xin, Huazuo Gao, Hui Qu, Hui Li, Jianzhong Guo, Jiashi Li, Jiawei Wang, Jingchang Chen, Jingyang Yuan, Junjie Qiu, Junlong Li, J. Cai, Jiaqi Ni, Jian Liang, Jin Chen, Kai Dong, Kai Hu, Kaige Gao, Kang Guan, Kexin Huang, Kuai Yu. Lean Wang, Lecong Zhang, Liang Zhao, Litong Wang. Liyue Zhang, Lei Xu, Leyi Xia, Mingehuan Zhang, Minghua Zhang, Minghui Tang, Meng Li, Miaojun Wang, Mingming Li, Ning Tian, Panpan Huang, Peng Zhang, Qiancheng Wang, Qinyu Chen, Qiushi Du, Ruiqi Ge, Ruisong Zhang, Ruizhe Pan, Runji Wang, R. J. Chen, R. L. Jin, Ruyi Chen, Shanghao Lu, Shangyan Zhou, Shanhuang Chen, Shengfeng Ye, Shiyu Wang. Shuiping Yu, Shunfeng Zhou, Shuting Pan, S. S. Li, Shuang Zhou, Shaoqing Wu, Shengfeng Ye, Tao Yun, Tian Pei, Tianyu Sun, T. Wang, Wangding Zeng, Wanjia Zhao, Wen Liu, Wenfeng Liang, Wenjun Gao, Wenqin Yu, Wentao Zhang, W. L. Xiao, Wei An, Xiaodong
Liu, Xiaohan Wang, Xiaokang Chen, Xiaotao Nie, Xin Cheng, Xin I Liu, Xin Xie, Xingchao Liu,

---

Accepted at ICLR 2026
Xinyu Yang, Xinyuan Li, Xuecheng Su, Xuheng Lin, X. Q. Li, Xiangyue Jin, Xiaojin Shen, Xi- aosha Chen, Xiaowen Sun, Xiaoxiang Wang, Xinnan Song, Xinyi Zhou, Xianzu Wang, Xinxia Shan, Y. K. Li, Y. Q. Wang. Y. X. Wei, Yang Zhang, Yanhong Xu, Yao Li, Yao Zhao, Yaofeng Sun, Yaohui Wang, Yi Yu, Yichao Zhang, Yifan Shi, Yiliang Xiong, Ying He, Yishi Piao, Yisong Wang, Yixuan Tan, Yiyang Ma, Yiyuan Liu, Yongqiang Guo, Yuan Ou, Yuduan Wang, Yue Gong, Yuheng Zou, Yujia He, Yunfan Xiong, Yuxiang Luo, Yuxiang You, Yuxuan Liu, Yuyang Zhou, Y. X. Zhu, Yanhong Xu, Yanping Huang, Yaohui Li, Yi Zheng, Yuchen Zhu, Yunxian Ma, Ying Tang, Yukun Zha, Yuting Yan, Z. Z. Ren, Zehui Ren, Zhangli Sha. Zhe Fu, Zhean Xu, Zhenda Xie, Zhengyan Zhang, Zhewen Hao, Zhicheng Ma, Zhigang Yan, Zhiyu Wu, Zihui Gu, Zijia Zhu, Zijun Liu, Zilin Li, Ziwei Xie, Ziyang Song. Zizheng Pan, Zhen Huang. Zhipeng Xu, Zhongyu Zhang, and Zhen Zhang. Deepseek-rl: Incentivizing reasoning capability in Ilms via reinforce- ment learning, 2025a. URL.
org / 12948
DeepSeek-AL, Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Cheng- gang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai Dai, Daya Guo, Dejian Yang. Deli Chen, Dongjie/ Ji, Erhang Li, Fangyun Lin, Fucong Dai, Fuli Luo, Guangbo Hao, Guanting Chen. Guowei Li, H. Zhang, Han Bao. Hanwei Xu, Haocheng Wang. Haowei Zhang. Honghui Ding, Huajian Xin. Huazuo Gao, Hui Li. Hui Qu, J. L. Cai, Jian Liang, Jianzhong Guo, Jiaqi Ni, Jiashi Li, Jiawei Wang, Jin Chen, Jingchang Chen, Jingyang Yuan, Junjie Qiu, Junlong Li, Junxiao Song, Kai Dong Kai Hu, Kaige Gao, Kang Guan, Kexin Huang, Kuai Yu, Lean Wang, Lecong Zhang, Lei Xu, Leyi Xia, Liang Zhao, Litong Wang, Liyue Zhang, Meng Li, Miaojun Wang, Mingchuan Zhang, Minghua Zhang, Minghui Tang. Mingming Li, Ning Tian, Panpan Huang, Peiyi Wang. Peng Zhang, Qiancheng Wang, Qihao Zhu, Qinyu Chen, Qiushi Du R. J. Chen, R. L. Jin, Ruiqi Ge. Ruisong Zhang, Ruizhe Pan, Runji Wang, Runxin Xu, Ruoyu Zhang, Ruyi Chen, S. S. Li, Shanghao Lu, Shangyan Zhou, Shanhuang Chen, Shaoqing Wu, Shengfeng Ye, Shengfeng Ye. Shirong Ma. Shiyu Wang. Shuang Zhou, Shuiping Yu, Shunfeng Zhou, Shut- ing Pan, T. Wang, Tao Yun, Tian Pei, Tianyu Sun, W. L. Xiao, Wangding Zeng, Wanjia Zhao, Wei An. Wen Liu, Wenfeng Liang, Wenjun Gao. Wenqin Yu, Wentao Zhang, X. Q. Li, Xiangyue Jin, Xianzu Wang, Xiao Bi, Xiaodong Liu, Xiaohan Wang, Xiaojin Shen, Xiaokang Chen, Xi- aokang Zhang, Xiaosha Chen, Xiaotao Nie, Xiaowen Sun, Xiaoxiang Wang. Xin Cheng, Xin Liu, Xin Xie, Xingchao Liu, Xingkai Yu, Ximnan Song, Xinxia Shan, Xinyi Zhou, Xinyu Yang, Xinyuan Li, Xuecheng Su, Xuheng Lin, Y. K. Li, Y. Q. Wang. Y. X. Wei, Y. X. Zhu, Yang Zhang. Yanhong Xu, Yanhong Xu, Yanping Huang, Yao Li, Yao Zhao, Yaofeng Sun, Yaohui Li, Yaohui Wang. Yi Yu, Yi Zheng, Yichao Zhang, Yifan Shi, Yiliang Xiong, Ying He, Ying Tang, Yishi Piao, Yisong Wang, Yixuan Tan, Yiyang Ma, Yiyuan Liu, Yonggiang Guo, Yu Wu, Yuan Ou, Yuchen Zhu, Yuduan Wang, Yue Gong, Yuheng Zou. Yujia He, Yukun Zha, Yunfan Xiong, Yunxian Ma. Yuting Yan, Yuxiang Luo, Yuxiang You, Yuxuan Liu, Yuyang Zhou, Z. F. Wu, Z. Z. Ren, Zehui Ren, Zhangli Sha, Zhe Fu, Zhean Xu, Zhen Huang, Zhen Zhang, Zhenda Xie, Zhengyan Zhang, Zhewen Hao, Zhibin Gou, Zhicheng Ma, Zhigang Yan, Zhihong Shao, Zhipeng Xu, Zhiyu Wu, Zhongyu Zhang, Zhuoshu Li, Zihui Gu, Zijia Zhu, Zijun Liu, Zilin Li, Ziwei Xie, Ziyang Song. Ziyi Gao, and Zizheng Pan. Deepseek-v3 technical report, 2025b. URL https: //arxiv org. wwwp://g.//.ooooooeooooooseeeess 2412.122211 19437
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshom, Aobo Yang. Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Zhang, Aurelien Rodriguez, Austen Gregerson, Ava Spataru, Baptiste Roziere, Bethany Biron, Binh Tang, Bobbie Chern, Charlotte Caucheteux, Chaya Nayak, Chloe Bi, Chris Marra, Chris McConnell, Christian Keller, Christophe Touret, Chunyang Wu, Corinne Wong. Cristian Canton Ferrer, Cyrus Nikolaidis, Damien Allonsius, Daniel Song, Danielle Pintz, Danny Livshits. David Esiobu, Dhruv Choudhary, Dhruv Mahajan, Diego Garcia-Olano, Diego Perino, Dicuwke Hupkes, Egor Lakomkin, Ehab AlBadawy, Elina Lobanova, Emily Dinan, Eric Michael Smith, Filip Radenovic, Frank Zhang, Gabriel Synnaeve, Gabrielle Lee, Georgia Lewis Ander- son, Graeme Nail, Gregoire Mialon, Guan Pang, Guillem Cucurell, Hailey Nguyen, Hannah Korevaar, Hu Xu, Hugo Touvron, Iliyan Zarov, Imanol Arrieta Ibarra, Isabel Kloumann, Ishan Misra, Ivan Evtimov, Jade Copet, Jaewon Lee, Jan Geffert, Jana Vranes, Jason Park, Jay Ma- hadeokar, Jeet Shah, Jelmer van der Linde, Jennifer Billock, Jenny Hong, Jenya Lee, Jeremy Fu, Jianfeng Chi, Jianyu Huang, Jiawen Liu, Jie Wang, Jiecao Yu, Joanna Bitton, Joe Spisak, Jongsoo Park, Joseph Rocca, Joshua Johnstun, Joshua Saxe, Junteng Jia, Kalyan Vasuden Al- wala, Kartikeya Upasani, Kate Plawiak, Ke Li, Kenneth Heafield, Kevin Stone, Khalid El-Arini,
12

---

Accepted at ICLR 2026
Krithika Iyer, Kshitiz Malik, Kuenley Chiu, Kunal Bhalla, Lauren Rantala- Yeary, Laurens der Maaten, Lawrence Chen, Liang Tan, Liz Jenkins, Louis Martin, Lovish Madaan, Lubo Malo, Lukas Blecher, Lukas Landzaat, Luke de Oliveira, Madeline Muzzi, Mahesh Pasupuleti. Man- nat Singh, Manohar Paluri, Marcin Kardas, Mathew Oldham, Mathieu Rita, Maya Pavlova, Melanie Kambadur, Mike Lewis, Min Si, Mitesh Kumar Singh, Mona Hassan, Naman Goyal, Narjes Torabi, Nikolay Bashlykov, Nikolay Bogoychev, Niladri Chatterji, Olivier Duchenne, Onur Çelebi, Patrick Alrassy, Pengchuan Zhang, Pengwei Li, Petar Vasic, Peter Weng, Prajjwal Bhar- gava, Pratik Dubal, Praveen Krishnan, Punit Singh Koura, Puxin Xu, Qing He, Qingxiao Dong, Ragavan Srinivasan, Raj Ganapathy. Ramon Calderer, Ricardo Silveira Cabral, Robert Stojnic, Roberta Raileanu, Rohit Girdhar, Rohit Patel, Romain Sauvestre, Ronnie Polidoro, Roshan Sum- baly, Ross Taylor, Ruan Silva, Rui Hou, Rui Wang, Saghar Hosseini, Sahana Chennabasappa, Sanjay Singh, Sean Bell, Seohyun Sonia Kim, Sergey Edunov, Shaoliang Nie, Sharan Narang, Sharath Raparthy, Sheng Shen, Shengye Wan, Shruti Bhosale, Shun Zhang, Simon Vandenhende, Soumya Batra, Spencer Whitman, Sten Sootla, Stephane Collot, Suchin Gururangan, Sydney Borodinsky, Tamar Herman, Tara Fowler, Tarek Sheasha, Thomas Georgiou, Thomas Scialom, Tobias Speckbacher, Todor Mihaylov, Tong Xiao, Ujjwal Kam, Vedanuj Goswami, Vibhor Gupta, Vignesh Ramanathan, Viktor Kerkez, Vincent Gonguet, Virginie Do, Vish Vogeti. Vladan Petro- Vic, Weiwei Chu, Wenhan Xiong, Wenyin Fu, Whitney Meers, Xavier Martinet, Xiaodong Wang. Xiaoqing Ellen Tan, Xinfeng Xie, Xuchao Jia, Xuewei Wang, Yaelle Goldschlag, Yashesh Gaur, Yasmine Babaei, Yi Wen, Yiwen Song. Yuchen Zhang. Yue Li, Yuning Mao, Zacharie Delpierre Coudert, Zheng Yan, Zhengxing Chen. Zoe Papakipos, Aaditya Singh, Aaron Grattafiori, Abha Jain, Adam Kelsey, Adam Shajnfeld, Adithya Gangidi, Adolfo Victoria, Ahuva Goldstand, Ajay Menon, Ajay Sharma, Alex Boesenberg, Alex Vaughan, Alexei Baevski, Allie Feinstein, Amanda Kallet, Amit Sangani, Anam Yunus, Andrei Lupu, Andres Alvarado, Andrew Caples, Andrew Gu, Andrew Ho. Andrew Poulton, Andrew Ryan, Ankit Ramchandani, Annie Franco, Aparajita Saraf, Arkabandhu Chowdhury, Ashley Gabriel, Ashwin Bharambe, Assaf Eisenman, Azadeh Yazdan, Beau James, Ben Maurer, Benjamin Leonhardi, Bernie Huang, Beth Loyd, Beto De Paola, Bhargavi Paranjape, Bing Liu, Bo Wu, Boyu Ni, Braden Hancock, Bram Wasti, Brandon Spence, Brani Stojkovic, Brian Gamido, Britt Montalvo, Carl Parker, Carly Burton, Catalina Mejia, Changhan Wang. Changkyu Kim, Chao Zhou, Chester Hu, Ching-Hsiang Chu, Chris Cai, Chris Tindal, Christoph Feichtenhofer, Damon Civin, Dana Beaty, Daniel Kreymer. Daniel Li, Danny Wyatt, David Adkins, David Xu, Davide Testuggine, Delia David, Devi Parikh, Di- ana Liskovich, Didem Foss, Dingkang Wang, Due Le. Dustin Holland, Edward Dowling, Eissa Jamil, Elaine Montgomery, Eleonora Presani, Emily Hahn, Emily Wood, Erik Brinkman, Es- teban Arcaute, Evan Dunbar, Evan Smothers, Fei Sun, Felix Kreuk, Feng Tian, Firat Ozgenel, Francesco Caggioni, Francisco Guzmán, Frank Kanayet, Frank Seide, Gabriela Medina Flo- rez, Gabriella Schwarz, Gada Badeer, Georgia Swee, Gil Halpern, Govind Thattai, Grant Her- man, Grigory Sizov, Guangyi. Zhang, Guna Lakshminaray Hannah Wang. Hanwen Zha, Haroun Habeeb, Harrison Rudolph, Helen Suk, Henry Aspegren, Hunter Goldman, Igor Molybog, Igor Tufanov, Irina-Elena Veliche Itai Gat, Jake Weissman, James Geboski, James Kohli, Japhet Asher, Jean-Baptiste Gaya, Jeff Marcus, Jeff Tang, Jennifer Chan, Jenny Zhen, Jeremy Reizenstein, Jeremy Teboul, Jessica Zhong, Jian Jin, Jingyi Yang. Joe Cummings, Jon Carvill, Jon Shepard, Jonathan McPhie, Jonathan Torres, Josh Ginsburg, Junjie Wang, Kai Wu, Kam Hou U. Karan Saxena, Karthik Prasad, Kartikay Khandelwal, Katayoun Zand, Kathy Matosich, Kaushik Veeraraghavan, Kelly Michelena, Keqian Li, Kun Huang, Kunal Chawla, Kushal Lakhotia, Kyle Huang, Lailin Chen, Lakshya Garg, Lavender A. Leandro Silva, Lee Bell, Lei Zhang, Liangpeng Guo, Licheng Yu, Liron Moshkovich, Luca Wehrstedt, Madian Khabsa, Manav Avalani, Manish Bhatt, Maria Tsimpoukelli, Martynas Mankus, Matan Hasson, Matthew Lennie, Matthias Reso, Maxim Groshev, Maxim Naumov, Maya Lathi, Meghan Ke- neally, Michael L. Seltzer, Michal Valko, Michelle Restrepo, Mihir Patel, Mik Vyatskov, Mikayel Samvelyan, Mike Clark, Mike Macey, Mike Wang, Miquel Jubert Hermoso, Mo Metanat, Mo- hammad Rastegari, Munish Bansal, Nandhini Santhanam, Natascha Parks, Natasha White, Navy- Bawa, Nayan Singhal, Nick Egebo, Nicolas Usunier, Nikolay Pavlovich Laptev, Ning Dong. Ning Zhang, Norman Cheng, Oleg Chernoguz, Olivia Hart, Omkar Salpekar, Ozlem Kalinli, Parkin Kent, Parth Parekh, Paul Saab, Pavan Balaji, Pedro Rittner, Philip Bontrager, Pierre Roux, Piotr Dollar, Polina Zvyagina, Prashant Ratanchandani, Pritish Yuvraj, Qian Liang, Rachad Alao, Rachel Rodriguez, Rafi Ayub, Raghotham Murthy, Raghu Nayani, Rahul Mitra, Raymond Li, Rebekkah Hogan, Robin Battey, Rocky Wang, Rohan Maheswari, Russ Howes, Ruty Rinott, Sai Jayesh Bondu, Samyak Datta, Sara Chugh, Sara Hunt, Sargun Dhillon, Sasha Sidorov, Sa-
Hamid Shojanazeri, Han Zou,
13

---

Accepted at ICLR 2026
tadru Pan, Saurabh Verma, Seiji Yamamoto, Sharadh Ramaswamy, Shaun Lindsay, Shaun Lind- say, Sheng Feng, Shenghao Lin, Shengxin Cindy Zha, Shiva Shankar, Shuqiang Zhang, Shuqiang Zhang, Sinong Wang, Sneha Agarwal, Soji Sajuyigbe, Soumith Chintala, Stephanie Max. Stephen Chen, Steve Kehoe, Steve Satterfield, Sudarshan Govindaprasad, Sumit Gupta, Sungmin Cho, Sunny Virk, Suraj Subramanian, Sy Choudhury, Sydney Goldman, Tal Remez, Tamar Glaser, Tamara Best, Thilo Kohler, Thomas Robinson, Tianhe Li, Tianjun Zhang, Tim Matthews, Tim- othy Chou, Tzook Shaked, Varun Vontimitta, Victoria Ajayi, Victoria Montanez, Vijai Mohan, Vinay Satish Kumar. Vishal Mangla, Vlad Ionescu, Vlad Poenaru, Vlad Tiberiu Mihailescu, Vladimir Ivanov, Wei Li, Wenchen Wang, Wenwen Jiang, Wes Bouaziz, Will Constable, Xi- aocheng Tang, Xiaofang Wang, Xiaojian Wu, Xiaolan Wang, Xide Xia, Xilun Wu, Xinbo Gao, Yanjun Chen, Ye Hu, Ye Jia, Ye Qi, Yenda Li, Yilin Zhang, Ying Zhang, Yossi Adi, Youngjin Nam, Yu, Wang. Yuchen Hao, Yundi Qian, Yuzi He, Zach Rait, Zachary DeVito, 7ef Rosnbrick, Zhaoduo Wen, Zhenyu Yang, and Zhiwei Zhao. The llama 3 herd of models, 2024. URL https://arxe org/ abs 2407 21783
Angela Fan, Yacine Jernite, Ethan Perez, David Grangier, Jason Weston, and Michael Auli Eli5: www.b///11....00 09190
Long form question answering. 2019.
Yuchen Fan, Yantao Liu, Zijun Yao, Jifan Yu, Lei Hou, and Juanzi Li. Evaluating generative language models in information extraction as subjective question correction. arXiv preprint WrXiv:2404.03332 2024a.
Yuchen Fan, Xin Zhong, Yazhe Wan, Chengsi Wang. Haonan Cheng, Gaoche Wu, Ning Ding, and Bowen Zhou Eva-score: Evaluating abstractive long-form summarization on informativeness through extraction and validation. arXiv preprint arXiv:2407.04905 2024b.
Yuchen Fan, Kaiyan Zhang, Heng Zhou, Yuxin Zuo, Yanxu Chen, Yu Fu, Xinwei Long, Xuekai Zhu, Che Jiang, Yuchen Zhang, al. Ssrl: Self-search reinforcement learning. arXiv preprint arXiv:2508. 10874, 2025.
Dongfu Jiang, Yishan Li, Ge Zhang, Wenhao Huang, Bill Yuchen Lin, and Wenhu Chen. Tiger- score: Towards building explainable metric for all text generation tasks, 2024. URL https:
//arxiv. abs 20t../211...0 00752
Mandar Joshi, Eunsol Choi, Daniel S. Weld, and Luke Zettlemoyer. Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension, 2017. utt.httss:/arxxiv.oo
0as/17005035511
Seungone Kim, Juyoung Suk, Shayne Longpre, Bill Yuchen Lin, Jamin Shin, Sean Welleck, Graham Neubig, Moontae Lee, Kyungjae Lee, and Minjoon Seo. Prometheus 2: An open source language model specialized in evaluating other language models, 2024.
Tomáš Kočiský, Jonathan Schwarz, Phil Blunsom, Chris Dyer, Karl Moritz Hermann, Gábor Melis, and Edward Grefenstette. The narrativeqa reading comprehension challenge, 2017. htRptt //arxivv.rrg/ab///112....
Kalpesh Krishna, Aurko Roy, and Mohit lyyer. Hurdles progress in long-form question answering, 2021.
Nathan Lambert, Valentina Pyatkin, Jacob Morrison, LJ Miranda, Bill Yuchen Lin, Khyathi Chandu, Nouha Dziri, Sachin Kumar. Tom Zick, Yejin Choi, Noah A. Smith, and Hannanch Hajishirzi. Rewardbench: Evaluating reward models for language modeling. https: htta:////ggggigggeeeee
co/spaces/al lena / wwwwardd-eenee 2024.
Junlong Li, Shichao Sun, Weizhe Yuan, Run-Ze Fan, Hai Zhao, and Pengfei Liu. Generative judge for evaluating alignment, 2023. http::::::/ //tp:v/.////vva//a orgo abs////////////// 2310.00000 05470
Chin-Yew Lin. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out pp. 74-81, Barcelona, Spain. July 2004. Association for Computational Linguis-
tics. URL https: /etts::/aclantholoogr crg. / MO4
1013
Chris Yuhao Liu, Liang Zeng, Jiacai Liu, Rui Yan, Jujie He, Chaojie Wang, Shuicheng Yan, Yang Liu, and Yahui Zhou. Skywork-rewee reward:: Bag of tricks for reward modeling in 1lms. arXiv preprint 241iv:2410. 18451, 2024a.
14

---

Accepted at ICLR 2026
Xiusi Liu, Gaotang Li, Ziqi Wang, Bowen Jin, Cheng Qian, Yu Wang, Hongru Wang, Yu Zhang, Denghui Zhang. Tong Zhang, Hanghang Tong, and Heng Ji. Rm-rl: Reward modeling as reason- ing, 2024b.
http:///axvvvvvooo org/ wwwp/r/////////oo//////.//ea.aee 2505. 02387
Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu. G-eval: Nlg evaluation using gpt-4 with better human alignment, 2023, URL/ttps://axxiv.orr abs/
2663416634
Yantao Liu, Zijun Yao, Rui Min, Yixin Cao, Lei Hou, and Juanzi Li. Rm-bench: Benchmarking reward models of language models with subtlety and style, 2024c. URL https://arxiv.
org/abs/2410, 16184
Sewon Min, Kalpesh Krishna, Xinxi Lyu, Mike Lewis, Wen tau Yih, Pang Wei Koh, Mohit lyyer, Luke Zettlemoyer, and Hannaneh Hajishirzi. Factscore: Fine-grained atomic evaluation of fac- tual precision in long form text generation, 2023. URL
14251
Reiichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christopher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, Xu Jiang, Karl Cobbe, Tyna Eloundou, Gretchen Krueger, Kevin Button, Matthew Knight, Benjamin Chess, and John Schulman. Webgpt: Browser-assisted question-answering with human feedback, 2022. URL https://arxiv:
org/abs/2112. 09332
OpenAl. Chatgpt: Chat generative pre-trained transformer.
itttp::/chat.opeeed
2023. Accessed: 2024-08-05.
OpenAl. Hello gpt-4o. 2024-08-05.
-otpg//eeeeellllgt--// 2024. Accessed:
OpenAI, :, Aaron Hurst, Adam Lerer, Adam P. Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, Aleksander Mądry, Alex Baker- Whitcomb, Alex Beutel, Alex Borzunov, Alex Carney, Alex Chow, Alex Kirillov, Alex Nichol, Alex Paino, Alex Renzin, Alex Tachard Passos, Alexander Kirillov, Alexi Christakis, Alexis Con- neau, Ali Kamali, Allan Jabri, Allison Moyer, Allison Tam, Amadou Crookes, Amin Tootoochian, Amin Tootoonchian, Ananya Kumar, Andrea Vallone, Andrej Karpathy, Andrew Braunstein, Andrew Cann, Andrew Codispoti, Andrew Galu, Andrew Kondrich, Andrew Tulloch, Andrey Mishchenko, Angela Baek, Angela Jiang, Antoine Pelisse, Antonia Woodford, Anuj Gosalia, Arka Dhar, Ashley Pantuliano, Avi Nayak. Avital Oliver, Barret Zoph, Behrooz Ghorbani, Ben Leimberger, Ben Rossen, Ben Sokolowsky, Ben Wang, Benjamin Zweig, Beth Hoover, Blake Samic. Bob McGrew, Bobby Spero, Bogo Giertler, Bowen Cheng, Brad Lightcap, Brandon Walkin, Brendan Quinn, Brian Guarraci, Brian Hsu, Bright Kellogg, Brydon Eastman, Camillo Lugaresi, Carroll Wainwright, Cary Bassin, Cary Hudson, Casey Chu, Chad Nelson, Chak Li, Chan Jun Shern, Channing Conger, Charlotte Barette, Chelsea Voss, Chen Ding, Cheng Lu, Chong Zhang, Chris Beaumont, Chris Hallacy, Chris Koch, Christian Gibson, Christina Kim, Christine Choi, Christine McLeavey, Christopher Hesse, Claudia Fischer, Clemens Winter, Colcy Czarnecki, Colin Jarvis, Colin Wei, Constantin Koumouzelis, Dane Sherburn, Daniel Kappler, Daniel Levin, Daniel Levy, David Carr, David Farhi, David Mely, David Robinson, David Sasaki, Denny Jin, Dev Valladares, Dimitris Tsipras, Doug Li, Due Phong Nguyen, Duncan Findlay, Edede Oiwoh, Edmund Wong, Ehsan Asdar, Elizabeth Prochl, Elizabeth Yang, Eric Antonow, Eric Kramer, Eric Peterson, Eric Sigler, Eric Wallace, Eugene Brevdo, Evan Mays, Farzad Kho- rasani, Felipe Petroski Such, Filippo Raso, Francis Zhang, Fred von Lohmann, Freddie Sulit, Gabriel Goh, Gene Oden, Geoff Salmon. Giulio Starace. Greg Brockman, Hadi Salman, Haiming Bao, Haitang Hu, Hannah Wong. Haoyu Wang, Heather Schmidt, Heather Whitney, Heewoo Jun, Hendrik Kirchner, Henrique Ponde de Oliveira Pinto, Hongyu Ren, Huiwen Chang, Hyung Won Chung, Ian Kivlichan, Ian O' O'Connell, Ian O' O'Connell, Ian Osband, Ian Silber, Ian Sohl, Ibrahim Okuyucu, Ikai Lan, Ilya Kostrikov, Ilya Sutskever, Ingmar Kanitscheider, Ishaan Gulrajani, Ja- cob Coxon, Jacob Menick, Jakub Pachocki, James Aung, James Betker, James Crooks, James Lennon, Jamie Kiros, Jan Leike, Jane Park, Jason Kwon, Jason Phang. Jason Teplitz, Jason Wei, Jason Wolfe, Jay Chen, Jeff Harris, Jenia Varavva, Jessica Gan Lee, Jessica Shich, Ji Lin, Jiahui Yu, Jiayi Weng. Jie Tang, Jieqi Yu, Joanne Jang, Joaquin Quinonero Candela, Joe Beutler, Joe Landers, Joel Parish, Johannes Heidecke, John Schulman, Jonathan Lachman, Jonathan McKay,
15

---

Accepted at ICLR 2026
Jonathan Uesato, Jonathan Ward, Jong Wook Kim, Joost Huizinga, Jordan Sitkin, Jos Kraaijeveld, Josh Gross, Josh Kaplan, Josh Snyder, Joshua Achiam, Joy Jiao, Joyce Lee, Juntang Zhuang, Justyn Harriman, Kai Fricke, Kai Hayashi. Karan Singhal, Katy Shi, Kavin Karthik, Kayla Wood, Kendra Rimbach, Kenny Hsu, Kenny Nguyen, Keren Gu-Lemberg, Kevin Button, Kevin Liu, Kiel Howe, Krithika Muthukumar, Kyle Luther, Lama Ahmad, Larry Kai, Lauren Itow, Lauren Work- man, Leher Pathak, Leo Chen, Li Jing, Lia Guy, Liam Fedus, Liang Zhou, Lien Mamitsuka, Lilian Weng, Lindsay McCallum, Lindsey Held, Long Ouyang, Louis Feuvrier, Lu Zhang, Lukas Kondraciuk, Lukasz Kaiser, Luke Hewitt, Luke Metz, Lyrie Doshi, Mada Aflak, Maddie Simens, Madelaine Boyd, Madeleine Thompson, Marat Dukhan, Mark Chen, Mark Gray, Mark Hudnall, Marvin Zhang, Marwan Aljubeh, Mateusz Litwin, Matthew Zeng. Max Johnson, Maya Shetty, Mayank Gupta, Meghan Shah, Mehmet Yatbaz, Meng Jia Yang. Mengchao Zhong, Mia Glaese, Mianna Chen, Michael Janner, Michael Lampe, Michael Petrov, Michael Wu, Michele Wang, Michelle Fradin, Michelle Pokrass, Miguel Castro. Miguel Oom Temudo de Castro. Mikhail Pavlov, Miles Brundage. Miles Wang. Minal Khan, Mira Murati, Mo Bavarian, Molly Lin, Murat Yesildal, Nacho Soto, Natalia Gimelshein, Natalie Cone, Natalie Staudacher, Natalie Summers, Natan LaFontaine, Neil Chowdhury, Nick Ryder, Nick Stathas, Nick Turley, Nik Tezak, Niko Fe- lix, Nithanth Kudige, Nitish Keskar, Noah Deutsch, Noel Bundick, Nora Puckett, Ofir Nachum, Ola Okelola, Oleg Boiko, Oleg Murk, Oliver Jaffe, Olivia Watkins, Olivier Godement, Owen Campbell-Moore, Patrick Chao. Paul McMillan, Pavel Belov, Peng Su, Peter Bak, Peter Bakkum, Peter Deng. Peter Dolan, Peter Hoeschele, Peter Welinder, Phil Tillet, Philip Pronin, Philippe Tillet, Prafulla Dhariwal, Qiming Yuan, Rachel Dias, Rachel Lim, Rahul Arora, Rajan Troll, Ran- dall Lin, Rapha Gontijo Lopes, Raul Puri, Reah Miyara, Reimar Leike Renaud Gaubert, Reza Zamani, Ricky Wang. Rob Donnelly, Rob Honsby, Rocky Smith, Rohan Sahai, Rohit Ramchan- dani, Romain Huet, Rory Carmichael, Rowan Zellers, Roy Chen, Ruby Chen, Ruslan Nigmat- ullin, Ryan Cheu, Saachi Jain, Sam Altman, Sam Schoenholz, Sam Toizer, Samuel Miserendino, Sandhini Agarwal, Sara Culver, Scott Ethersmith, Scott Gray, Sean Grove, Sean Metzger, Shamez Hermani, Shantanu Jain, Shengjia Zhao, Sherwin Wu, Shino Jomoto. Shirong Wu, Shuaiqi. Xia, Sonia Phene, Spencer Papay, Srinivas Narayanan, Steve Coffey, Steve Lec, Stewart Hall, Suchir Balaji, Tal Broda, Tal Stramer, Tao Xu, Tarun Gogineni, Taya Christianson, Ted Sanders, Tejal Patwardhan, Thomas Cunninghman, Thomas Degry, Thomas Dimson, Thomas Raoux, Thomas Shadwell, Tianhao Zheng, Todd Underwood, Todor Markov, Toki Sherbakov, Tom Rubin, Tom Stasi, Tomer Kaftan, Tristan Heywood, Troy Peterson, Tyce Walters, Tyna Eloundou, Valerie Qi, Veit Moeller, Vinnie Monaco, Vishal Kuo, Vlad Fomenko, Wayne Chang. Weiyi Zheng, Wenda Zhou, Wesam Manassra, Will Sheu, Wojeiech Zaremba, Yash Patil, Yilei Qian, Yongjik Kim, Youlong Cheng, Yu Zhang, Yuchen He, Yuchen Zhang, Yujia Jin, Yunxing Dai, and Yury Malkov. Gpt-4o system card 2024. URL https: /aapivv///rvvvv.v/a orgg abs/ 2410 0. 21276
Qwen, An Yang. Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mei Li, Mingfeng Xue, Pei Zhang, Qin Zhu. Rui Men, Runji Lin, Tianhao Li, Tianyi Tang, Tingyu Xia, Xingzhang Ren, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang. Yu Wan, Yugiong Liu, Zeyu Cui, Zhenru Zhang, and Zihan Qiu. Qwen2.5 technical report, 2025. URL /tttps://aaat wawiiv.ivvvv..////a//...//aaaaz org/ abs 22oo/24125 221224122..2 15115
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. Squad: 100,000+ questions for machine comprehension of text, 2016. URL https:/ http///axvvv... org/ aaa/a/bssii/ 1606.16666.00000 05250
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, and Denny Zhou. Ohain-of-thouglle prompting elicits reasoning in large language models, 2023. (RLttttt::
.rgg abs 0200. 11903
Fangyuan Xu, Junyi Jessy Li, and Eunsol Choi. How do we answer complex questions: Discourse structure of long-form answers, 2022. URL
Fangyuan Xu, Yixiao Song, Mohit Iyyer, and Eunsol Choi. A critical evaluation of evaluations for forg-form question answering. 2023. URL https: //t::///aiiv.eeee wttp://.gg/......../ 18201
Ruijie Xu, Zengzhi Wang, Run-Ze Fan, and Pengfei Liu. Benchmarking benchmark leakage in large language models. arXiv preprint arXiv:2404.1 18824, 2024. URL htthhtts:t.. htt/:tttt////xi... orq/ abs/ 2244/4444.. 8824
16

---

Accepted at ICLR 2026
Qiying Yu, Zheng Zhang. Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Weinan Dai, Tiantian Fan, Gaohong Liu, Lingjun Liu, Xin Liu, Haibin Lin, Zhiqi Lin, Bole Ma, Guang- ming Sheng, Yuxuan Tong, Chi Zhang. Mofan Zhang, Wang Zhang, Hang Zhu, Jinhua Zhu, Jiaze Chen, Jiangjie Chen, Chengyi Wang, Hongli Yu, Yuxuan Song, Xiangpeng Wei, Hao Zhou, Jingjing Liu, Wei- Ying Ma, Ya-Qin Zhang. Lin Yan, Mu Qiao, Yonghui Wu, and Mingx- uan Wang. Dapo: An open-source Ilm reinforcement learning system at scale, 2025. URL https:
htrg/abs/25550 14476
Weizhe Yuan, Graham Neubig, and Pengfei Liu. Bartscore: Evaluating generated text as text gener- ation, 2021. URL
wtt../////................. 11520
Kaiyan Zhang, Yuxin Zuo, Bingxiang He, Youbang Sun, Runze Liu, Che Jiang, Yuchen Fan, Kai Tian, Guoli Jia, Pengfei Li, et al. A survey of reinforcement learning for large reasoning models. arXiv preprint arXiv:2509.08827 2025.
Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. Bertscore: Evalu- wwtp://.rggabbbb/bbgbbae 09675
ating text generation with bert, 2020. URL
Lianmin Zheng, Wei Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, Hao Zhang. Joseph E. Gonzalez, and lon Sto- ica. Judging llm-as-ajjudge with mt-bench and chatbot arena. In Thirly-seventh Confer- ence on Neural Information Processing Systems Datasets and Benchmarks Track, 2023. URL net horrm/ormmmmmm wwt.:ntooooommmdduceieeeeeeeee dd-ummiiicaccccccaaaoo.oo
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Tianle Li, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zhuohan Li, Zi Lin, Eric Xing. Joseph E. Gonzalez, Ion Stoica, and Hao Zhang. Lmsys- chat-Imp A large-scale real-world I1m conversation dataset. The Twelfth International Confer- ence on Learning Representations, 2024. URL BOfDKxfwtO
net htw.://wweee/touuueemmmm forum?id
Heng Zhou, Hejia Geng, Xiangyuan Xue, Li Kang, Yiran Qin, Zhiyong Wang, Zhenfei Yin, and Lei Bai. Reso: A reward-driven self-organizing Ilm-based multi-agent system for reasoning tasks. arXiv preprint wwXiv:2503.02399, 2025.
Yuxin Zuo, Kaiyan Zhang, Li Sheng, Shang Qu, Ganqu Cui, Xuekai Zhu, Haozhan Li, Yuchen Zhang, Xinwei Long, Ermo Hua, al. Ttrl: Test-time reinforcement learning. arXiv preprint a50iv:2504. 16084, 2025.
17

---

Accepted at ICLR 2026
A USE OF LLMS
We use LLMs to refine our writing using Gemini2.5-Pro, GPT-5, and Claude-4.1. We check the refined phrases after generation.
B ADDITIONAL RESULTS
B.1 ANNOTATION OF L.M PARAPHRASING
We randomly sample 100 records to compare the original responses and the paraphrased responses. We define three types of errors: The paraphrased response contains factual errors; The para- phrased response adds new core points to answer the problem, which may reverse the order; 3) The paraphrased response drops original points introduces influency or inconsistency. After annota- tion, we find error in only out of 100 paraphrased response, indicating the effectiveness of using GPT-4o to paraphrase responses.
B.2 PROMPT SENSITIVITY ANALYSIS
To further investigate the sensitivity to prompt design, we conduct an additional ablation by instruct- ing models solely provide final answer without CoT. The results presented in the table below. This experiment further highlights the importance of CoT for enhancing the ability when evaluating: for most settings under LFQA-E-EN, the performance degrades greatly. While for LFQA-E-ZH. the performance is hard to predict, indicating the language vulnerability.
Table 10: Ablation study prompt design.
LFQA-E-EN
LFQA-E-ZH
Model
Acc
Fl
Acc
Fl
Qwenn-5-33B-Inntruut
57.3
45.5.0.3
52.813.9
38.613.2
Qwen2.5-72B-Instuue
60.4jo.8
45.142.0
53.940.9
37.0
GPT-4o
61.41o.3
44.9
51.511.7
37.7
DeepSeck-V3
51.9
42.2
54.470.6
37.0
DeepSeck-RI
54.8
41.4J1.5
60.0ç2.2
44.1ç1.7
B.3 THE ABILITY OF EVALUATION METRICS VARIES ACROSS DOMAINS.
The performance evaluation metrics in LFQA-E varies significantly across different domains. As shown in Tables TT and 12 models and metrics exhibit distinct strengths and weaknesses depending on the subject area. For instance, on the LFQA-E-ZI dataset, models like Owen2.5-32B-Inseee IIstructt and GPT-4o consistently excel in subjects such Geography, Law, and Medicine but perform less effectively in more complex domains like Psychology and History. Conversely, models like DeepSeek-V3 and RM-hased Evaluation Metrics show particular strengths in fields like Politics and Law, where the emphasis is on factual accuracy and legal context. Similarly, in the LFQA-E-EN dataset, LLM-based models such as Qwwen2.5-32B-Instruu perform exceptionally well in Engineer- ing and Technology but show noticeable decline in subjects like Psychology and Mathematics, possibly due to the more abstract nature of these domains. In contrast, RM-based Evaluation Met- rics, such as AM-RI-DeepSeek. Distilled-Qweet 14B, demonstrate impressive performance in fields like Planetary Science and Chemistry, where the data often more structured and less ambiguous.
B.4 RESULTS OF ADDITIONAL METRICS
We incorporate several additional baselines, including cectuality-orientte and evaluation eenluation-ensemlle measures, shown below in Table 13. Through the experimental results, find that the agentic
18

---

Accepted at ICLR 2026
Table 11: Results of different topics from LFQA-E-ZHH The largest value is denoted using bold.
METRICS
GEOGRAPHY
HISTORY
POLITICS
PSYCHOLOGY
MEDICINE
LAW
LLM--based Evaluation Metric
Qwen2.5-32B 5332--tstttl
64.3
58.2
50.4
44.8
65.1
49.0
Owen2.5-728-Insttne
65.5
54.3
42.7
58.3
Llama-3, 1--00B 1-70B-Instruct
41.5
34.4
37.0
46.3
46.6
62.6
GPT-4o
62.0
51.8
49.1
44.8
58.3
46.1
Deepscck-V3
60.0
50.5
54.4
40.6
57.3
49.0
RM-bused Evaluation Metric
Skywork-Reward- Reward-Llama
58.3
54.1
51.4
34.4
59.2
54.9
Skywork- Reward-Geenet Gemma
54.3
50.0
41.0
36.5
48.5
56.9
RM-RI RM-RI-Qwen2.55
51.0
45.5
44.8
58.4
RM-RI RM-RI-
61.4
51.8
62.5
37.5
54.5
57.8
LRM--hased Evaluation Metric
ol-mini
53.5
46.9
42.5
49.0
66.0
56.9
Deepseck-RI
64.0
58.4
53.2
42.7
67.0
54.9
Trained Evaluntion Metric
Auto-I-BB-biliggee
58.5
55.4
41.7
38.5
68.4
59.8
Promethus77B-v2..
54.0
52.6
43.2
40.6
51.5
59.8
M-Promethus-11
51.3
43.7
41.7
44.7
50.0
Table 12: Results of different topics from LFQA-E-EN. The largest value is denoted using bold.
METRICS
ENGINBERING BEOLOGY
TECHNOLOGY
PaySicS MATHEEMATIES
ECONONICS
PLANCTARY SCIENCE
CHEMISTRY
OTHER
Eviitation
5-328 Qnee2 5-728 Intruct
$5.6
59.8
65.3
69.2
12.2
57.1
61.9
01.5
67.5
63.4
Liano 706 Iastree
59.7
61.4
53.2
$1.0
35011
61.2
00.5
62.6
$5.0
80-4
830
RM-lvwrd Th sluntin Meric
Skywoukk Lama
Itl ---enm S-benna--
90.2
56.8
500
58.5
556
60.0
RM Deepiterk Maaattdd-eer---aa
65.7
71.1
61.5
63.9
613
677
73.0
71.4
69.0
ol-mus
56.2
64.3
$3.9
$4.0
58.2
643
63.3
616
$1.0
Despoccker
$4.0
58.6
$4.3
48.3
45.1
588
5633
62.6
56.1
Evaleation Menic
4-08--BBa bilingul
12.3
65.3
73.1
65.2
643
62.1
67.4
6966
72.9
Pronchus- 7B--2.0
66.8
65.4
690
65.1
68.1
60.0
64.0
72.0
33.0
600
611
framework is the most effective method for LFQA evaluation, and majority voting also contributes to improvement.
Table 13: More baselines on LFQA-E.
Model
LFQA-E-EN
LFQA-E-ZH
Average Acc
Acc
Fl
Acc
F1
F1
SOTA Models
Gamini2.5-flash
63.3
45.6
57.9
44.7
60.6
45.2
GPT-5
64.8
47.7
60.1
46.7
62.5
47.2
FactScore variants
FineSurE (GPT-4o)
47.8
35.5
42.9
32.5
45.4
34.0
Evaluation Ensembles
ChatEval (GPT-4o)
70.7
48.0
49.9
40.3
60.3
44.2
G-Eval (GPT-4o)
68.0
50.0
55.2
42.2
61.6
46.1
19

---

Accepted at ICLR 2026
Performance of different evaluation metries on LFQA-E. The examples whose labels are Table 14:
tie are discarded for fairer comparison. The largest value is denoted using bold.
MODEL
LFQA-E-EN
LFQA-E-ZH
Avg
Static Evaluation Metric
Length ROUGE
42.7
56.9
49.8
2.1%
57.5
53.8
55.7
3.1%
BERTScore
56.0
56.6
56.3
3.0%
LLMM-ased Evaluation Metric
Swen2.5-33B-Insttuue
66.8
62.8
64.8 1 4.7%
Brren25--7BB-Insee
63.4
57.6
60.5 3.4%
Llama-3.1-77 3.1-00B...--0----- 1-700-Innstrctt
66.1
34.0
50.1 1 4.9%
GPT-4o
66.3
61.4
63.9 6.4%
Deepseek-V.
60.0
60.0
60.0 4.1%
R.M-based Evaluation Metric
Skeyorrk-Reward-Llamd
56.7
58.4
57.6 1 3.6%
Skywork-Reward-Gemmt
58.2
52.5
55.4
3.2%
67.5
56.2
61.9 1 3.5%
RM RM-RRI-DeepSec--Dstii Distilled-Qweert 14B
68.0
57.5
62.8
3.3%
LRM.-bsee Evaluation Metric
ol-mini
67.3
64.2
65.8
4.9%
Deepseek-R
61.6
63.1
62.4
3.7%
Trained Evaluation Metric
Auto-J-6B--illinguu
70.0
57.3
63.7
4.3%
Prometheus-7B-ie
66.5
54.1
60.3
3.1%
M-Prometheus-l44
63.7
53.3
58.5
3.4%
B.5 LLMS ARE BETTER FINDING SOMETHING BETTER.
Considering that giving a tie option difficult for both humans and models, we drop out the records that are labeled tie and conduct the experiments again. We show the results in Table [4 After discarding the tied comparison, all the evaluation metrics show nontrivial performance boosts. GPT- 4o even gets 6.4% bonus. This increase matches what we find when comparing indicators. Similar to what we observe above. LRMs remain leading on the fairer comparison, and RMs still struggle to generalize to long-form response evaluation. Static evaluation metrics, however, show the least im- provement. The experimental results demonstrate the potential of test-time scaling, while reflects the generalization problem of RMs. What's 's more, specific evaluation models show their great potential once again, ranking first on LFQA-E-EN, displaying its future for LFQA evaluation.
B.6 HOW TO BALANCE THE COST AND TIME FFFICIENCY BETWEEN MODELS?
To further discuss the tradeoff between efficiency and effectiveness among language models, we cal- culate the average inference time per question of several L.LMs, LRMs and RMs. Table 17 lists the results. Specifically, for the closed-source models that need to invoke api for inference, we calculate their costs on both LFQA-E-EI and LFQA-E-ZII in Table I8, We observe that ol-mini incurs the highest cost, which is aligned with its relatively higher accuracy and Fl scores. DeepSeek- V3, on the other hand, offers a more cost-effective alternative, though its performance somewhat practical and cost-efficient evaluation, models like Skywork-Reward- Reward-LLame and lower. For -eemaa have much smaller parameter scales and can be deployed on local re- Skywork-Reward Reward- sources, showing strong promise for reducing costs while still providing reasonable evaluation.
20

---

Accepted at ICLR 2026
C DISCUSSION
We suggest training open-domain RMs or evaluation models, which may help for the evaluation of LFQA, considering their relatively low cost and GPU requirements, but with decent score. Also, we recommend future work to focus on evaluation workflows that combine the strengths of both LLM-based models and more efficient reward models. Particularly, for open-sourced models, we don't observe steady performance gains as model size scales. Therefore, smaller models with more training data may help more than larger models with some well-designed prompts.
D CASE STUDY
D.1 CASE FROM CEESQ AND PEEQ
We provide two cases from CEESQ and PEEQ in Table [3 We translate them into English for easier
comprehension.
Table 15: Case Study from CEESQ and PEEQ.
QUESTION FROM PEEQ:
What are the classifications and percentages of white blood cells?
REFERENCE:
White blood cells are divided into granular cells and non-granular cells. Granular cells include: neutrophils (50%-70% basophils (0-1%); eosinophils (0.5%-5%). Non- Non-granular cells include: monocytes (5%-8%) lymphocytes (20%-40%)
QUESTION FROM CEESQ:
Analyze the natural reasons for the numerous sandbars in the Yangtze River estuary area.
REFERENCE
The river has large discharge volume and carries large amount of sediment; located at the river estuary, the terrain low and flat, the flow velocity is slow, with deposition as the main pro- leading to massive sediment accumulation; situated at the river-sea interface, tidal backing enhances the deposition process, forming numerous sandbars in the estuary area.
D.2 CASE FOR PARAPHRASING
We provide case study from LFQA-E-ZH for paraphrasing in Table [6
Table 16: Case Study for parahrasing.
QUESTION FROM LFQAAE-ZH:
Reasons for the extremely fragile ecological environment Guizhou Province.
ORIGINAL RESPONSE:
1. Guizhou features karst topography with undulating terrain
2. Serious soil erosion
Guizhou Province has few plains and is susceptible to natural disasters
4. Rugged terrain with inconvenient transportation
PARAPHRASED RESPONSE:
The ecological environment of Guizhou Province extremely fragile for four main reasons: First. Guizhou features karst topography with undulating terrain and complex landforms; Sec- ond, serious soil erosion has caused severe depletion of land resources: Third, Guizhou Province has few plains, with most areas being mountainous, making susceptible to natural disasters; Fourth, the rugged terrain with inconvenient transportation creates certain difficulties for ecolog- ical environment protection and restoration.
21

---

LFQA-E-ZH Accepted at ICLR 2026
Table 17: The average inference time of different language models per question, arranged in de- scending order.
Model
TIME (SECONDS)
Skywork-Rese -eee-R--w-aadaaa
0.21
Qwen2.5-328) -----t-ntttttt
0.27
Barook--eewrr-GGer
0.64
Qwen2.5-722 -25-sstteet
1.03
ol-mini
1.80
GPT-4o
3.56
DeepSeek-V V3
12.00
DeepSeek-RI
14.13
Table 18: Overall closed-source models using API. The largest value is denoted in bold.
Model
LFQA-E-EN
3.6S DeepSeek-RI
DeepSeek- V3
0.75
18.05
6.95
ol-mini
67.0S
14.2S
GPT-4o
35.65
5.7S
D.3 FAILED CASE
We show eтrоr cases in Table 19, Table 20 and Table 21 Table 19 shows the Incorrect Information Error.
Table 20 shows the Point Identification Error. Table shows the Format Error.
D.4 SUCCESS CASE
We show successful cases in Table 22 and Table
E INSTRUCTIONS
The following instructions used.
E.1
DATA COLLECTION INSTRUCTIONS
The instruction for the question filter is in Table 24 The instruction for the paraphrase is in Table
25
E.2
ENGLISH LLM EVALUATION INSTRUCTION
The instruction for all LLMs and LRMs is in Table 26 The instruction for Prometheus series is in
Table 27
The instruction for Auto-J is in Table 28
F
ANNOTATION
We show the annotation recipe in Table 29 and Table 30
We show a screenshot of annotation
pipeline in Figure
22

---

Accepted at ICLR 2026
Table 19: Case Study from LFQA-E from GPT-4o. This results from an incorrect information error.
QUESTION:
How does law enforcement track down criminal during manhunt?
REFERENCE:
1) Do we know where he/she is likely to "go to ground?" We try find people he/she relies from phone taps, social media, prior crimes etc. You'd be amazed how many guys are found at their aunt's/es s/former cellmate's 's place.
2) Put the word out to local TV news. In my experience they plaster the manhunt news and mugshot on their website landing page right away.
3) License plate scanner hits on patrol cars and traffic cameras, if available. This is surprisingly effective.
4) Social media phishing. Is he/she into certain drugs? Certain types of prostitutes? Is he trying to find his wife or kids who are working with victim services and in hiding (very common bad domestic abuse cases)?
RESPONSE 1:
There are a lot of ways that law enforcement can track someone during a manhunt. First and foremost, live in the age of technology. and technology is a really powerful tool for tracking someone. cell phone is pretty much always transmitting your approximate location, and law enforcement can gain access the satellite that keeps track of that location.
There are also a few tried and true methods: the public. Law enforcement will almost always provide a physical description of a suspect, complete with a photograph urge people to report anyone who fits the bill close enough to be the person. This is a powerful tool when to get anything modern, you need interact with other humans or come into close contact with human society.
artist rendition, and
There are also recordings, photographs, and footage captured by CCTV cameras everywhere. Police may also deploy tracking hounds if they know who you are and have anything that has your smell on it and escaping on foot from persistent, trained hunting dogs is unbelievably difficult."
RESPONSE: 2:
Ohhhhh so many ways. Really depends on what resources are on hand. 1. Suspect probably has friends and family. Check their houses, interview them, see what they know about the suspects hobbies and motives. Do they have a goal they're 're trying achieve? Is that goal local or would it send them running elsewhere?
2. Flowing from above, roadblocks. If you generally know where your suspect is, where they might try to go, and what means of transportation are available to them, you can monitor those points of access until they show up.
3. Technology. Where did the suspect use their credit card. Oh, they pulled out cash to avoid being traced? We can see where they made the withdrawal and go from there. Is their phone still online? Can we call and trace them? Were they recently active social media? How about their friends? A lot of those posts have geotags in the metadata
MODEL JUDGE:
Response aligns well with the standard answer by emphasizing technology (cell phone track- ing), public involvement (providing descriptions and photos), and surveillance methods (CCTV cameras and tracking dogs). touches upon finding the suspect's connections, similar to the standard answer's first point about identifying likely places to "go to ground."
Response also covers relevant points such as investigating friends and family, setting up roadblocks, and using technology like phone tracing and social media analysis. However, it delves into areas like credit card tracking, which the standard answer specifically mentions as ineffective.
Overall, Response more closely mirrors the strategies outlined in the standard answer. (The Judgement incorrect by incorrectly estimating the weights of irrelevant unimportant key- points.)
23

---

Accepted at ICLR 2026
Table 20: Case Study from LFQA-E from GPT-4o. This results from an identification error.
QUESTION:
Why do swear words exist? What is it about swear words that make us feel better? Why don't t other normal words make us feel the same way swear words do?
REFERENCE:
Swear words fall broadly into two categories: profanity and obscenity. Doscenity.Profanity Profanity originates with the Ten Commandments, specifically "Do not take the Lord's name in vain" Abrahamic reli- gions. Generally the idea though is "don't disparage term "curse words" comes from, as you would curse someone by saying "God damn you". It's also where the term "swear words" comes from, i.e. "I swear to God". Obscenity, by contrast, is primarily words that relate to sex or using the toilet. These are activities during which you would be vulnerable to being attacked, and also that produce excrement that could be harmful for oth- ers. There is sometimes a need to discuss these things in polite company, so terms were divided into "polite" and "rude". In English, this distinction largely comes from the language of origin. English has linguistic roots in both Romance languages (ones based on Latin like French) and Germanic languages. At the time when the language was developing, higher society had more background in Romance languages while the lower classes were more Germanic. So the former words were considered refined and the latter crude and crass. Most English obscenities have close analogs in German. In modern usage, there's 's actually official definitions of profanity and obscenity by the FCC. Profanity is using these words in an exclamatory sense, while obscenity is using these words to describe the actions. The former is more allowable than the latter. As for why it feels good to swear, it's mostly because breaking a taboo gives a feeling of exhilaration, and if nobody actually suffers result, you also get feel like you're 're "getting away with it". Basically, if we didn't put so much emphasis on NOT saying the words, it wouldn' 't feel so good when we do say them.
devalue the sacred". This is where the
RESPONSE 1:
Language is tool of communication. Expletives communicate something that you basically cannot express using polite language. There's certainly an element to them where the social transgression is part of what makes them function as they do. What is isn't considered swear word largely depends on cultural values. In the Middle Ages it's pretty apparent that words related to bodily functions sexuality were not considered particularly offensive, while blas- phemies were generally considered a lot more offensive than they are in most modern societies. Medieval folks appear to have been simultaneously less prudish but more fearful of the almighty.
RESPONSE 2:
Language has been used to separate classes for a very long time. Do you know why a pig is pig in the pen but pork on your plate? Or why a cow becomes beef when you serve it up? Because one word was used by the lower class who raised the animals and the other word was used by the upper class who could afford to consume it (in this case, Normans). Words used by the lower classes frequently, but not by upper classes, will be found to be taboo in the upper classes, leading them to be essentially banned in polite society. They have power because they are taboo, and for no other reason. The word 'merde' has power to you unless you speak French, and the word 'crap' has no power to you unless you're 6. You've been conditioned to believe these words are strong, so you feel strong when saying them. It really is that simple. It's nothing more than the placebo effect.
MODEL. JUDGE:
Upon evaluating both responses against the reference answer:
Response aligns closely with the standard answer by discussing the role of swear words in communication, the significance of social transgression, and cultural variations in what is considered offensive. It also touches on historical perspectives, similar to how the reference answer differentiates profanity and obscenity based on origins and societal values.
Response 2 introduces the concept of language separating classes and the power of taboo stem- ming from class distinctions. While it addresses the conditioning and perceived strength of swear words, it deviates more from the primary focus of the standard answer, which centers on cultural and linguistic origins rather than class-bsee language differentiation.
Therefore, Response better. (The Judgement is incorrect by observing the number of infor- mation coexisting.)
24

---

Accepted at ICLR 2026
Table 21: Case Study For LFQA-E from GPT-4o. This results from a format error.
QUESTION:
Can someone explain Benford's Law. I get that certain numbers show up more often in large data sets, but why?
REFERENCE:
Benford's Law is typically stated about data that span multiple orders magnitude; so we have values that are many times larger than others instead of them being close together. Next is that we have at least a bit what is called scale-im inleeinvaria unlike many simpler probability problems, don't assume that all numbers are equally likely, but instead that larger ones rarer. More precisely, we want roughly the same chance to land between x and 2x as for landing between y and 2y. If those two satisfied, Benford's Law states that the leading digit is more likely than an other, and the larger digits are the less common ones. There can he no perfect examples of ccale-invariance in probability. But between those lie dozens of orders of magnitude, easily more than enough within reason...
RESPONSE 1:
Benford's law says you gather up a bunch of random numbers that appear "in the wild" and you tally up the first digits of each of these numbers then you will tend to find lots of I's and not many 9's. This tends to happen when you have a range of numbers which spread out over spread out over many orders of
multiple orders of magnitude. The reason is when numbers magnitude, it tends to be roughly uniform when measured on a "log scale". This means that the number of data points between and 2x will be about the same for different values of x. In other words, there should be as many values between 100 and 200 as there are between 200 and 400.
RESPONSE 2:
Benford's law says the first digit of a number picked from large range of numbers tends to be I. Think of it like this between I and 20, 11 of the 20 numbers start with 1. Between I and 99, 11 of the 99 numbers start with a 1. This repeats for 0-200 and 0-999 etc always holding the if you average that for all ranges, you get that about 30 percent of numbers in an unknown range start with 1. Between I and 89, number starts with 9, basically 1/89 percent. Going up to 99 brings back to 1/99, but 11 is the max and "almost 0" is the min, so again average over all ranges, and you get more like 4 percent of lead numbers. You can then apply this some fraud cases. If the numbers span multiple orders of magnitude AND should be roughly random, AND there are a lot of them, you should expect them match Benford's law pretty well. If they don't tt one of the 3 requirements is probably false. If you know the first and last are true, you can say "these probably arent actually random".
max of "just over half" and the min of "about 11 percent"
MODEL JUDGE:
Strengths of Response 1:
Correctly identifies that Benford's 's Law applies to numbers spanning large ranges. Explains the higher probability of 1 as leading digit by comparing intervals which aligns with the scale-invariance idea.
Mentions the decreasing probability for higher digits. Correctly notes the conditions for applying Benford's Law and its use in fraud detection.
Weaknesses of Response 1:
Doesn't 't explicitly mention scale-invariance though the intuition is somewhat captured. The explanation for why 1 more likely is bit simplistic.
Strengths of Response 2: Introduces Benford's 's Law in a simple, relatable context. Correctly notes that and are more common as leading digits.
Weaknesses of Response 2:
Misrepresents the core reasons for Benford's Law. The explanations given are incorrect or mis- leading. Benford's Law is not about human behavior but about inherent properties of scale- invariant data.
Doesn't mention orders of magnitude scale-invariance at all, which are central the reference answer.
The references the end are irrelevant to the actual content of the response. No preference provided!
25

---

Accepted at ICLR 2026
Table 22: Case Study For LFQA-E from Deepseck-Ve V3.
QUESTION:
how do metals put together with electricity cause technology and chips and computers and data ctc?
REFERENCE:
Silicon along with a little bit of some other elements combined form what is called a semicon- ductor, of which there are 2 types, the n type and the p type. When you then sandwich these semiconductors in a specific way, has the property that applying a current to these sandwiches causes it to become conductive, or if you invert the sandwich it has the property that it's con- ductive when you don't It apply a current and then not conductive when you do. So essentially, what you have created is an electrical switch, which is controlled by electricity, and call these transistors.
If you'te re looking at how these transistors then form a computer, you can arrange transistors to perform logical functions. A simple one would be the NOT gate, which just uses the second switch mentioned that turns off when there is an input. An AND gate would be if you took two switches, placed them one after the other, and each switch is then controlled by an input. Both inputs need to be on for there to be an output.
RESPONSE 1:
For that, you need to look at boolian values and at turing machines. A boolian value is just yes no answer. Yes vs no, on vs off, one vs zero. Technically you use boolian values all the time in speech. "Is Tommy in the bathroom?" You could answer "Yes", but you could also say "one", and anyone who knows Boolian will understand.
A Turing machine is just a name for any machine that uses Boolian values. has rules, for example "if you see seven zeros in row followed by a one, then do".. It seems pretty abstract. But math people have actually proved that you can do any calculation, think any thought, perform any task, if you have the right rules designed for your turing machine.
So once we figured that out, it just became matter of inventing the right rules for the machine, and inventing the best form for a machine to store boolian values. A transistor is always either One or a Zero, and they can be made very small, so makes sense to use them for your Turing Machine.
RESPONSE 2:
When metals are put together with electricity in the context of technology, it typically refers to the process of creating integrated circuits or microchips. These chips are made up of layers of different metals, such as silicon, aluminum, and copper, that are carefully arranged on a silicon wafer.
Electricity is used to power the various components on the chip, allowing them to perform their specific functions. The flow of electricity through these metal layers enables the chip to process and store information, ultimately allowing computers and other electronic devices to function. Additionally, metals such as gold and copper are used in the construction of circuit boards and connectors within electronic devices. These metals have excellent conductivity properties, which allow for the efficient flow of electricity throughout the device.
MODEL JUDGE:
Response delves into the concept of Boolean values and Turing machines, which extends the topic to how computers "think" and process data. It links the use of transistors to Turing ma- chines and emphasizes the binary nature of data processing, which aligns well with the reference answer's explanation of logical functions and binary systems accessible through transistors.
Response 2 focuses on the physical aspects of creating chips, such as the use of various metals and their conductivity. While it provides a general overview of chip construction and functional- ity, it lacks depth on the conceptual workings, like the function of transistors and binary systems, which is a significant aspect of the reference answer.
The reference answer provides an explanation centered around the role of transistors as switches and their organization perform logical operations. It links this the representation of numbers and data processing in computers through binary systems, which Response I addresses more comprehensively compared to Response 2.
Therefore, Response 1 is better. (The evaluation is correct!)
26

---

Accepted at ICLR 2026
Table 23: Case Study For LFQA-E from Deepseck-V2
QUESTION:
How did human calculate time?
REFERENCE:
For long time we didn't and no one cared. However, once you start farming, you need to know what season it is, what hour of the day it is more or less, you know what you can and should do on the fields for a bountiful yield. The easiest way to do is to watch the celestial bodies, namely the two major ones, the Sun and the Moon (the Stars also helped, but eventually it was more of a tool for navigation). Over time people figured out that the Sun shifts, days are not evenly distributed over the year, but luckily goes in cycles, so it is predictable. Once you make chart of these shifts throughout the year, you got yourself more or less a calendar. Civilizations with accurate-enough calendars survived and thrived. Calendars got more and more accurate over time as our math and science got better and here we are, keeping time down to split seconds. By the way, it does not fit perfectly. We accumulate quite the error over time, but the Gregorian Calendar is designed as way to correct for these accumulative errors. The Persian Solar Calendar is the most accurate one, because the new year starts after a celestial event. Gregorian calendar has bigger focus on religious events rather than any celestial event, so it needs corrections. We have a very accurate time measurement, but that NOT what is used on the daily, because we don't need it.
RESPONSE: 1:
Absolutely nothing about the way humans decided to calculate time "fits neatly" into Earth's behaviours. Years are actually 365 days plus hours and minutes, days are actually 23 hours and 56 minutes long.On Ot her time systems have been attempted, with the Romans dividing day and night into 12 hours, which varied in actual length as the days and nights shifted through the year. They also had the "Civil Day" system: formal names given to time according to certain markers (such as separate named times for "rooster is crowing" and "rooster stops crowing", or "time we light candles" and "bed time"). Humans have been struggling with measuring it forever.
RESPONSE 2:
Time has been calculated and measured in various ways throughout history. One of the earliest methods used by ancient civilizations was observing the movement of celestial bodies such as the sun, moon, and stars, By tracking the positions of these objects in the sky, early astronomers were able to develop calendars and timekeeping systems. The modern concept of time mea- surement is based on the International System of Units (SI). In terms of dividing the day into hours, minutes, and seconds, this system was developed by the ancient Babylonians and Egyp- tians. The Babylonians used base-60 system for counting, which led to the division of the hour into 60 minutes and the minute into 60 seconds. The Egyptians also used a similar system of timekeeping. Overall, the calculation of time has evolved over centuries through combination of astronomical observations, scientific advancements, and cultural practices. The precise mea- surement of time today is achieved through the use of atomic clocks and coordinated universal time (UTC) to ensure accurate timekeeping worldwide.
MODEL JUDGE:
The reference answer emphasizes the historical development of time measurement, focusing on the transition from no precise timekeeping to the necessity of using celestial bodies for farming, Response discusses the inconsistency and challenges of time measurements historically, noting the Roman system and difficulties faced by humans in tracking time over time. While it touches on early attempts to measure time, it doesn't delve specifically into celestial observations or the evolution of calendars as much.
Response 2 covers the historical aspect of using celestial bodies and tracks the progression to modern timekeeping with atomic clocks. It also discusses the Babylonian and Egyptian contri- butions to dividing time. This response aligns more closely with the standard answer by tracing the transition from ancient methods to modern precision, even mentioning cultural influences timekeeping.
Overall, while both responses provide historical context, Response 2 is closer to the reference answer as it better follows the progression of timekeeping from ancient observations leading to the precise systems we have today.
Therefore, Response 2 is better. (The evaluation is correct!)
27

---

Accepted at ICLR 2026
Table 24: Prompt used for LLM-based question filtering.
PROMPT FOR LLM FILTER
Question Filtering Instructions
Objective
Filter out questions that are either unclear their description or too broad to provide a mean-
ingful reference.
Filtering Criteria
1. Unclear Questions
Reject questions that exhibit:
Ambiguous wording phrasing
Multiple possible interpretations
Missing critical context parameters
Vague or undefined terms
Grammatical issues that obscure meaning
Incomplete or fragmented thoughts
2. Overly Broad Questions
Reject questions that:
Request information topics with reasonable boundaries
Would require encyclopedic or book-length answers
Ask for opinions on vast, nulti-faceted subjects
Lack specific focus or scope constraints
Would yield references too general to useful
Cover multiple unrelated topics simultaneously
Process
1. Read the question carefully and completely
Evaluate against both clarity and breadth criteria
3. Make filtering decision:
PASS:
Question is clear and appropriately scoped
REJECT UNCLEAR: Question lacks clarity (provide specific reason)
REJECT- TOO BROAD: Question is overly broad (provide specific reason)
Examples of Questions to Reject
"What about technology?" (unclear)
"Explain everything about human history" (too broad)
"How does stuff work in general?" (both unclear and too broad)
"What are all the factors affecting everything in the world?" (too broad)
Examples of Questions to Pass
"What is the boiling point of water at sea level?"
"How does photosynthesis work in green plants?"
"What were the main causes of World War I?"
28

---

Accepted at ICLR 2026
Table 25: Prompt for LLM Paraphrase.
PROMPT FOR LLM PARAPHRASE
Objective:
Transform the provided response into a more verbose version while strictly preserving the original meaning and information.
Requirements:
Expand the original text by adding descriptive language, elaborations, and explanatory phrases
Maintain complete fidelity the original information-do do not introduce any new facts,
claims, or insights
Preserve the tone and intent of the original message
Use stylistic techniques such as:
Adding clarifying phrases and parenthetical explanations
Employing more elaborate sentence structures
Incorporating synonyms and varied vocabulary
Adding transitional phrases between ideas
Expanding brief points into full explanations
Ensure the final text feels natural and not artificially inflated
Process:
1. Thoroughly analyze the original response understand its complete meaning
Identify core points and supporting details
3. Expand each point methodically while maintaining the original structure
Review to confirm no new information has been introduced
5. Polish the text for readability and flow
Table 26: Prompt for English LLM Evaluation.
PROMPT FOR ENGLISH LLM EVALUATION
We have the following question:
Question: question
The reference (standard) answer to this question is as follows: reference We now have two student responses this question.
Response is as follows: resp1
Response is as follows: resp2
Now, you should evaluate the two responses based on the content of the question, using the standard answer as the sole basis for judgment.
Determine which of the two "ro-Response or Response 2 is closer to the reference an- swer terms of content.
Please begin with brief analysis, and then provide your final judgment in one of the following forms:
If one response is better:
"Therefore, [Response is better." or "Therefore, [Response 2] is better." " If the two responses are roughly equal in quality: "Therefore, [Both responses are equal]."
29

---

Accepted at ICLR 2026
Table 27: Prompt for Prometheus Evaluation.
PROMPT FOR PROMETHEUS EVALUATION
Task Description:
An instruction (might include Input inside it), response to evaluate, and score rubric representing a evaluation criteria are given.
1. Write detailed feedback that assess the quality of two responses strictly based the given score rubric, not evaluating in general.
After writing feedback, choose better response between Response A and Response
B. You should refer to the score rubric.
The output format should look as follows:
"Feedback: (write feedback for criteria)
[RESULT] (A or B)"
Please not generate any other opening, closing, and explanations.
Instruction: [orig_instrun instruction]
Response A: [response |
Response B: (response_Bi
Score Rubric: (score_rubric)
Feedback:
Table 28: Prompt for Auto-J Evaluation.
PROMPT FOR AUTO-J EVALUATION
You are a helpful and precise assistant for checking the quality of the feedback.
Two pieces of feedback have been provided for the same response to particular query. Which one is better with regard their correctness, comprehensiveness, and specificity the query?
[BEGIN DATA]
[Query]: [prompt)
[Response]: (response)
FFeddacck 1]: (feedback1)
Feedback 2]: [feedback2)
[NDD DATA]
Please choose from the following options, and give out your reason in the next line. A: Feedback is significantly better.
B: Feedback 2 is significantly better.
C: Neither is significantly better.
30

---

Accepted at ICLR 2026
Table 29: Annotation recipe for deciding valid references of LFQA-E.
ANNOTATION RECIPE FOR DROPPING INVALID REFERENCES
Goal
Keep only real, helpful explanation. Discard those that are uninformative, incorrect, or not actual expla-
nations.
Keep if the answer:
-Directly answers the question.
Gives simple but meaningful explanation (even if simplified). Is factually reasonable- - not misleading or false.
Stands alone (no "see link" or "I don't 't know"). )).
Good example: "We yawn to help cool the brain and stay alert. It brings in oxygen and improves blood flow."
Discard if the answer is:
Not an explanation - e.g.. "Google it," "It's magic." jokes, memes, one- me-linees ("Because science").
Irrelevant Doesn't I address the question or misunderstands it.
Factually wrong - Clear misinformation (c.g. "Rain comes from clouds crying").
Too vague No real content: "It's complicated," " "There are many reasons."
Avoids answering- "Great question!", "Not sure, but here's thought..." with nothing useful.
Circular - Repeats the question: "We sleep because we're tired."
Inappropriite- - Offensive, harmful, or unprofessional content.
Question:
What is an instigator in Chinase oriminal law?
1. Identify Key Information reference
Belerence:
Overall, an insligator in Chinese criminal law refers to an indiviuull who incites others to commit a crime Specifically, such individuals incentionally lead or provoke others to decide carry criminal activitiessppp// intil the fundamental characteritic of inttigator that inctte others commit instance, person may
without directly partic pating encourage, persuade, or offer some benefit successfully prompt another to decide commit acts such theft froud while not partiking in the actual cemmnna process inwnnnennddyyy...o. certair condibons must be met to establish instigation. Subjectuely, the instgator must have the intention induce others This intent
criminal behavior themselves. They a inducing
intentions.
form the criminal intent and decide
the crime, which known instiggtigg Without explictt intent, instigation be established;
others
cxplittt the
inst gater must dearlly know whom they are instigtinn behavior that inaveretyyyy causes others to forn cominal bons cannot constitute instaatioon Objectively, the insigattio must have carried acts of Inst gation, commonty manifosted as encuragement, deception. persuasion, requests bribery, coorcion, prompting specific indiv duals to commit specific Whether
actualizes ciminal intert determination and whother instigatue actually carrils carries the criminal sstablismant instigatar/niiiiieueee ng the criminal libility insiigators several kay points noteworthy. Firs, instigators are punished according, to thwirools the joint crime. the insigator plays major role the criee process, they will be purished as the main pepetrator, thy only play minor ausilarry role, they will punished according the standard for accomplion. Although instgattors typically a major role and are generally punished accomplice.
affect
main perpetrators, inst gated crime, the instggator may sill be deemed puilyy but subect lighter reduced punishment This scemario, usually referred as l'attempted inst gabion.' indicates that attempted inst gation is also punshablee undersooing the independent and punshable nature of inst instugative behawier under Chinese criminal law. Third, instigating minors commit crimes should resultin harsher punishment, especially commit cnminol acts houghoughh instigators possess indepondent criminality punishobilly, they are an independent should be charged according to the spec crime instgated For esample, the instigation pertains to robbery, the instigator should be prosecuted under rebbery charges. summary, through these three points, we can gain deeper understanding of the provisions conceming instigators in Chinese criminal law cianty
possible
gator
instigatte does
where those under 15 inclted to category instigotor
specific determin practice require dotalled analysis based on individual cases, However, thess basic principles provide relativaly comprchensive understanding in onstttiits These principles stam from Chinese criminal lawe and reflect our serious attiude towards and strict control over instgatveee behavior.
responsibilttiss
summary
possible
answer
An inst gator in Chinese criminal law refers to person who directly on onindrrectl inst or incites others commit a crime. According to the relevant provisions the Criminal Law of the People's Republic China, instigator not only bears the corresponding comminal respo ns billty for the criminal act but may punished
For ssample, anline platforn, leading to commisiion of haud crime by incited person, instructing behavlor, although not directly involved recognized as an instigator, bearing cor respenddingg criminal responsibility In lugal practice, the duterinaation of an instigator generally requires comproharsive judgment based on the spacitic dircumstanoss the case, including the nature, content, and result of the instruction, wel. as the identity of instructing
soverely based their
influence criminal
and instructs others through fraud, may be
answer
Check for Key Information in Answers
According Chinese criminal law, instigator is person who deliberately entices incites others commit criminal behawior through orai, written, otherr means. The instigator usually does not directly commit the crima but their actions lead directly to the commission the crime by others. In criminal law, an instigator, like a principall or accomplics, must bear copespanding logal responsibiiity, For example, someone inchts another person online to rob store, and incited person commits the robbary, the instigator constitutes Instigator.
3. Handling Responses
Compare Overlapping Information
The handling responses criteria classify answers totally correct, partially correct, totally wrong, affecting the overlapping information score accordingly, Use purple to display common points.
Which will considered the better response?
AnowerA
Anower
Figure 5: The annotation pipeline.
31

---

Accepted at ICLR 2026
Table 30: Annotation recipe of LFQA-E.
Overview
This guide helps annotators evaluate and compare long-form responses against reference determine which response is more informative and complete. The process uses triple-choice format (Response Better, Response Better, or Tie).
Key Principles
Focus on factuality and completeness according to the reference
Fluency is not a primary evaluation criterion (all responses are expected to be fluent)
Use information units as the basic evaluation unit
Minimize bias through systematic comparison
Prerequisites
Domain knowledge relevant the question topic
Understanding of the subject matter through academic coursework or professional experience
Ability to maintain focus during paragraph-Ieve analysis
Evaluation Process
Step 1: Extract Key Information from Reference
1. Read the question carefully to understand what information is being requested
2. Read the reference thoroughly
Identify and list all key information units that:
Directly answer the question
Provide necessary context or background
Support the main answer with evidence examples
4. Organize key information into logical categories themes
Step 2: Check for Key Information in Responses
For each response (A and B):
1.
Read the response completely
2. Map each key information unit from the reference to the response
3. Mark which key information units are:
Present and accurate
Present but inaccurate
Missing entirely
4. Note any additional information not in the reference
Step 3: Handle Response Content
1. Evaluate additional information:
Is relevant to the central topic?
Does it enhance understanding or is verhose/umneceessaryy
2. Identify intertwined information:
For sentences containing both correct and incorrect information. separate the components
Assess the impact any inaccuracies on the overall response quality
Step Compare Overlapping Information
1. Compare how well each response covers the key information units
2. Consider:
Completeness: Which response includes more key information?
Accuracy: Which response presents information more correctly?
Relevance: Which response stays more focused on the question?
3. Compare the quality of overlapping information presentation
Step 5: Make Final Decision
Select one of three options:
Ressonse is Better: A contains key information and/or presents accurately
Response is Better: B contains more key information and/or presents it more accurately
Tie: Both responses are comparable information coverage and accuracy
Common Pitfalls to Avoid
1. Losing focus due to long paragraphs use the systematic approach
2. Allowing domain bias to influence decisions stick the reference
Confusing eloquence with accuracy
Missing subtle differences between comparable responses
32


Reviews:
[{"review_id": "YWY10BmB1m", "reviewer": "Reviewer_awMT", "summary": "This paper introduces LFQA-E, a reference-based multilingual benchmark for evaluating automatic metrics on long-form question answering (LFQA). The motivation is that existing automatic metrics fall short on LFQA, and also there are gaps in metric generalization. The benchmark contains questions and pairwise comparisons across 15 topics in English and Chinese, with expert-written references and a triple-choice annotation protocol (A / B / tie). The authors evaluate 17 evaluation methods spanning static, LLM-based, reward/model-based, LRM, and trained evaluation models.", "questions": "Please address the weaknesses O1-O4.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "S1) The paper addresses an important, underexplored issue: reliable automatic evaluation for paragraph-length, information-dense LFQA answers. The motivation and relation to prior small/weak benchmarks are convincing. \n\nS2) The paper offers a new substantial and diverse dataset for LFQA evaluation, offering 1,625 Questions and 7.6k comparisons, multilinguality (EN/ZH), and topic breadth. Statistics and careful sourcing are provided. \n\nS3) The annotation recipe is thorough. Using experts, double annotation, a triple-choice rubric, and inter-annotator agreement reporting improves trust in labels. \n\nS4) Authors test a wide range of metrics, give per-domain breakdowns, error taxonomy, contamination checks (PPL, n-gram overlap), and also try improvement (TTRL). This breadth of evaluation is useful for diagnosing metric weaknesses.", "weaknesses": "O1) The English references use the “top-ranked ELI5 answer” as candidate references. Top reddit answers are not guaranteed to be expert or correct; relying on them (even after expert review) risks reference quality variance. Although the authors state expert review, more detail is needed: how often did experts modify reddit content, and can we get quantitative measures of reference quality across sources? \n\nO2) Annotators are paid $2 per question (4–6 comparisons). This low pay risks rushed judgments despite training. The paper reports κ, but does not report annotator qualification statistics, and time per item distribution (they say ~7 minutes per comparison — is that per comparison or per question?). These details matter for external confidence. \n\nO3) Several choices need clarification or stronger controls: prompts, temperature settings, and inclusion of references differ across evaluated models. The procedural differences can advantage/disadvantage classes of metrics. The authors report a temperature ablation, but more systematic control (same prompt templates, prompt-sensitivity analysis, ensemble-vs-scalar handling) is needed to ensure fair comparisons. \n\nO4) Some leading recent evaluation approaches (explainable/extractive hybrid metrics, factuality-oriented measures such as FactScore variants, or evaluation ensembles) are missing or treated superficially. It’s unclear whether the 17 chosen methods fairly represent the state of the art; the paper would benefit from explicitly listing and justifying excluded but relevant baselines.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5MUNhjGRyl", "reviewer": "Reviewer_Yu2f", "summary": "This paper introduces a new comprehensive evaluation benchmark for evaluating long form QA systems and models. The main contributions of this paper are as follows:\n1. LFQA-E a benchmark dataset for evaluation of automatic metrics for LFQA.\n2. Detailed analysis of the proposed benchmark\n3. Experimental evaluation on 15 evaluation metrics to show that current LFQA evaluation metrics fail to capture core information.\n4. Analysis of generalisation of different metrics across languages and settings.", "questions": "1. Is there a metrics used to assess difficulty of the questions in the benchmark?\n2. Is there a plan to release a multilingual version of this dataset?\n3. Reddit license has some restrictions, do the authors plan to release the benchmark and setup a leaderboard?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The main strengths of the paper are as follows:\n1.  The paper identifies important limitations of the existing LFQA benchmarks like lack of authorized references and limited diversity and proposes a new large scale benchmark for better LFQA evaluation.\n2. The benchmark is very diverse and spans 15 topics and domains.\n3. Human experts based validation to makes the benchmark trustworthy.\n4. Detailed comparison of standard evaluation metrics on a variety of reasoning models using frontier models like deepseek and qwen family of models.\n5. Detailed benchmark contamination analysis to ensure that the evaluation data is not memorised during pre-training.", "weaknesses": "The main weaknesses of the paper are as follows:\n1. The comparisons of evaluation results using previous LFQA benchmarks using frontier models are missing.\n2. The paper writing and presentation can be improved.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "QW7jq7UPg3", "reviewer": "Reviewer_H1wd", "summary": "- Focuses on LFQA (Long-Form Question Answering) \n - Existing LFQA benchmarks are small, narrow and often lack reference answers.\n - LFQA-E is a multilingual, reference-based benchmark with 1,625 questions and 7,649 pairwise comparisons across 15 diverse topics.\n - The dataset covers sources like online queries and exam questions, supporting broad metric evaluation.\n - The study assesses 17 automatic metrics across five categories using LFQA-E.\n - Findings: no current automatic metric aligns well with human judgment; all struggle to capture dense, nuanced information in long-form responses.", "questions": "See the previous box.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The goal of effective evaluation of long responses is clearly an important for AI/NLP.", "weaknesses": "Assuming that table 1 refers to tokens (Table 1 are tokens, not chars?), 180-200 tokens is not much really, compared to what modern models can generate. \n\nFor your LLM-based evaluation metrics, have you tries using prompt that involves generating intermediate thoughts before conclusing a final answer? \n\nFor reasoning models, can you please see the quality metrics as a function of the thinking budget? \n\nThere have been various benchmarks for reward modeling (input Q, and two responses with preference label: A1 > A2). These benchmarks are essentially equivalent to what you're doing here. Can you distinguish yourself against these works and what gap your work is filling? \n\nMake sure your paragraph heading have a consistent style (e.g., they all end with \".\" at the end). \n\nRelatedly, this an off way to format your paragraph heading. Like it's part of the subsequent sentence?? \n> **For LM-based Evaluation Metric** We observe ...", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "EnanURPcMW", "reviewer": "Reviewer_aZgf", "summary": "This paper presents an evaluation dataset consisting of -- 1k data with question, pairwise answer, a reference answer and a human annotated preference covering English and Chinese. The paper then conducted evaluation of different evaluation metrics' correlation with the human annotation, concluding that none of them reliably model the human preference.\n\nOverall the paper is written in a clear manner. However I do not find it to bring substantial contribution compared to prior work in the space of evaluating evaluation metrics. Please see discussion below.", "questions": "* Given that there are only two annotators, how is the gold label determined when the two annotators disagree?\n* The prompt for LLM evaluation (table 22) appears to be incorrect.\n* Figure 2 reports performance of different comparison setting (human v.s. human, model v.s. model) and shows that evaluation metrics perform worse on model v.s. model comparison -- what's the human performance for these settings?", "limitations": "", "rating": 2, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "* The paper is written in a clear manner.\n* The paper conducted comprehensive experiments with different types of metrics (reference-based, reward models, etc.)\n* The annotation process is conducted clearly.", "weaknesses": "* Limited contribution: the paper aims to contribute a dataset with reference-based evaluation for long-form answers, yet there are existing resources such as Feedback-Bench from [Prometheus-2(EMNLP 2024)](https://arxiv.org/abs/2405.01535). It would be good to further clarify the new contribution this new dataset introduces. Also while the benefit of having a reference answer is understandable, it is also a somewhat unrealistic setting -- many user queries would not contain a reference answer. Also it is possible that a single reference answer does not comprehensively cover all the possible answers, especially for more open-ended questions studied in this paper.\n* Annotation noise: while the paper reported a cohen's kappa of 0.77 between annotators, each question is only annotated by two annotators -- this raises the question of whether the dataset capture annotators' preference, or other aspects. On a related note, it would be good to identify what are the **aspects** that these preference capture.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)