DiscoX: Benchmarking Discourse-Level Translation in Expert Domains
Authors: Xiying ZHAO
Zhoufutu Wen
Zhixuan Chen
Jingzhe Ding
Jianpeng Jiao
Shuai Li
Xi Li
Danni Liang
Shengda Long
Qianqian Liu
Xianbo Wu
Hongwan Gao
Xiang Gao
LIANG HU
Jiashuo Liu
Liumengyun
Weiran Shi
Chenghao Yang
Qianyu Yang
Xuanliang Zhang
Ge Zhang
Wenhao Huang
Venue: ICLR 2026 Poster (2026)
Keywords: translation
discourse-level
expert-level
benchmark
LLM
automatic evaluation system
Primary area: datasets and benchmarks
TL;DR: We introduce DiscoX, a benchmark for the evaluation of  LLMs on discourse- and expert-level translation tasks. We also propose Metric-S, an automatic evaluation system for translation tasks.

Abstract: The evaluation of discourse-level translation in expert domains remains inadequate, despite its centrality to knowledge dissemination and cross-lingual scholarly communication. While these translations demand discourse-level coherence and strict terminological precision, current evaluation methods predominantly focus on segment-level accuracy and fluency. To address this limitation, we introduce DiscoX, a new benchmark for discourse-level and expert-level Chinese-English translation. It comprises 200 professionally-curated texts from 7 domains, with an average length exceeding 1700 tokens. To evaluate performance on DiscoX, we also develop Metric-S, a reference-free system that provides fine-grained automatic assessments across accuracy, fluency, and appropriateness. Metric-S demonstrates strong consistency with human judgments, significantly outperforming existing metrics. Our experiments reveal a remarkable performance gap: even the most advanced LLMs still trail human experts on these tasks. This finding validates the difficulty of DiscoX and underscores the challenges that remain in achieving professional-grade machine translation. The proposed benchmark and evaluation system provide a robust framework for more rigorous evaluation, facilitating future advancements in LLM-based translation. Our data and code are available at https://github.com/ByteDance-Seed/DiscoX.

Published as a conference paper at ICLR 2026
DISCOX: BENCHMARKING DISCOURSE-LEVEL TRANSLATION TASK IN EXPERT DOMAINS
Xiying Zhao'; ZZoofutuu Zhixuan Chen', ,ingz,e " Jiao', Shuai Li', Xi Li', Danni Liang', Shengda Long'.? QQanqqia Liu', Xianbo Wu!, Hongwan Gao', Xiang Gao', Liang Hu', Jiashuo Liu', Mengyun Liu', Weiran Shi', Chenghao Yang', Qianyu Yang', Xuanliang Zhang', Ge Zhang', Wenhao Huang
BByteDance *Peking University (zhaoxiying, Limiuniu/@byreedncee.
ABSTRACT
The evaluation of discourse-level translation in expert domains remains inade- quate, despite centrality knowledge dissemination and cross-lingual schol- arly communication. While these translations demand discourse-level coherence and strict terminological precision, current evaluation methods predominantly focus on segment-level accuracy and fluency. To address this limitation, we introduce DiscoX, a new benchmark for discourse-level and expert-level Chinese- English translation. It comprises 200 professionalll-curattee texts from domains, with an average length exceeding 1700 tokens. To evaluate performance on Dis- coX, we also develop Metric-S, reference-free system that provides fine-grained automatic assessments across accuracy, fluency, and appropriateness. Metric-S demonstrates strong consistency with human judgments, significantly outper- forming existing metrics, Our experiments reveal a remarkable performance gap: even the most advanced LLMs still trail human experts on these tasks. This find- ing validates the difficulty of DiscoX and underscores the challenges that remain in achieving professional-gres grade machine translation. The proposed benchmark and evaluation system provide a robust framework for more rigorous evalua- tion, facilitating future advancements in LL-M-based translation. Our data and Sww.eeeeeaaaeeeeeiiieieeeeee
code are available at
Pickcy Aperasioseeee
43
51.6
40.05
40
20
Aigh
Cheate
cater
Deselied
Dosther
1017
Figure 1: Leaderboard of DiscoX. This chart compares the overall translation performance of L.LMs against human expert baseline, which refers to the score of translation produced by an expert human translator. Each bar is segmented show the score composition from three dimensions: Accuracy (blue), Fluency (light blue), and Appropriateness (green).
"Equal contribution and Corresponding authors.

---

Published as a conference paper at ICLR 2026
INTRODUCTION
Translation, a critical application of intelligent systems, is central to enabling cross-lingual commu- nication and knowledge access. Recent advances in large language models (LLMs) have yielded substantial progress in segment-level translation, with state-of-the-art (SOTA) systems approach- ing human performance (Koemi et al.) 2024). However, notable shortcomings persist in the con- text of longer and more specialized texts (Wang al. 2024b). Yet, existing benchmarks, such as WMT et al.] 2023b), FI FLORES Cooyal et al., 2022) and Redtrans Bench(Guo et al., (025b) predominantly focus segment-level which means evaluating one or several sentences time. Consequently, they fail to assess whether models can sustain discourse-level coherence, han- dle domain-intente terminology, meet expert stylistic standards. This gap underscores the need for benchmarks designed to evaluate these advanced capabilities.
Such capabilities are critical in expert domains such as scientific articles, legal contracts, and tech- nical manuals. For instance, as shown in Figure in scientific articles, failing to sustain coherence across sections can distort the logical flow of arguments: in legal contracts, inconsistent translation of specialized terminology may weaken their binding force (Briva-Iglesias et al.) 2024) and in tech- nical manuals, imprecise or stylistically inappropriate renderings may lead to misunderstandings that jeopardize operational safety.
To address this gap, we introduce DiscoX, the first benchmark to evaluate the translation at the discourse-level and expert-level translation between Chinese and English (shown in Table [ The benchmark, which cost 1,330 person-hours to create, is designed to simulate professional, real-world scenarios, comprising 200 cases across 7 domains, spanning both academic and non-academic texts, with an average length of 1712 tokens. The construction of the dataset followed a multi- stage expert curation process. This process ensures that the test cases reflect authentic professional demands and incorporate domain-critical aspects such as terminology and cultural expressions. For each case, the key challenges are systematically collected and organized into expert-authored rubrics, such as the handling of ambiguous terminology. For example, in academic contexts, the ab- breviation LLM may refer Large Language Model in natural language processing, but to Master of Laws in the legal domain, highlighting the importance of precise terminology handling in context.
Long Coherant and Cortentattet
Discaurse- level translaian task
Discot
Spaciie Duman Sneweledge
intudee
Laway Worts
Mull Comosiiu Prodeen
Solket miiiin Aadenic taper
Lil of Toot Temo
Socal Vedio
Indaary Repon
Closaes Chnnna Horty
Expert level translation task
Comparative Raview Febrile
When Translation
Single word:
In context:
Terminology should remain come stant
0238 worh)
Figure 2: Definition of DiscoX. It illustrates the definition of two core concepts in DiscoX bench- mark: "discourse-level translation" and "expert-level translation".
However, evaluating performance on a benchmark like DiscoX presents its own set of challenges, as conventional reference-basee metrics are inadequate for long-form text and single-judge LLM evaluations can be unreliable (shown in Table D (Szymanski et al. 2025). To overcome this, we developed Metric-S, a novel, automated evaluation system based on the LLM-as-a-judge paradigm. Metric-S orchestrates multiple LLM agents in a structured workflow that includes pre-checking, quality estimation across accuracy, fluency, and appropriateness, error deduplication and attribution, and severity weighting. This modular framework exhibits strong alignment with human judgments (70.3% consistency on DiscoX) and significantly outperforms existing excellent reference-fee met-
2

---

Published as a conference paper at ICLR 2026
rics like XCOMET-QE (34.7%) (Guerreiro al. 2024) DiscoX. Together, DiscoX and Metric-S provide a principled and robust foundation for assessing professional grade translation quality.
Table 1: Comparison of DiscoX with existing translation benchmarks. DiscoX distinguishes itself by (a) targeting discourse-level texts with a larger average length and focusing on expert domains. And, (b) its companion metric, Metric-S, offers reference-free and explainable evaluation, unique feature among the compared methods.
(a) Benchmark Comparison
(b) Metric Comparison
Benchmark
Scenarios
Avg. Tokens
Metrie
Ref-bused
Explainability
FLORES
General
48 88
N-grams
Yes
No
WMT (2024)
News, Speech, Social, Lit.
45.84
Neural Metric
Optional
No
Redtrans Bench
Social Conversation
59.46
LLM-as-a-judge
Optional
Optional
DiscoX (Ours)
Academic & Non-academic
1712.17
Metric-S (Ours)
No
Yes
We apply DiscoX broad set of LLMs test discourse-level and expert-level translation. As shown in Figure even the strongest LLM (GPT-5-high) still lags behind professional human trans- lators. This gap demonstrates that DiscoX serves as realistic and challenging stress test for pro- fessional translation. Performance also diverges across dimensions: some models excel in accuracy, others fluency, but none achieve a balanced, human-level competence. Overall, current models fall short of expert-level and discourse-level translation, underscoring both the difficulty of DiscoX and the need for future progress. Our contributions are:
We present DiscoX, a benchmark that rigorously evaluates LLMs on discourse-level and expert-level translation tasks.
We design Metric-S, workflow-based automatic evaluation system for DiscoX, which enables more accurate and comprehensive assessments tailored to the requirements of pro- fessional translation domains.
We conduct an extensive empirical study across multiple models, leveraging the explain- ability of Metric-S to reveal their respective strengths and limitations challenging trans- lation tasks.
We discuss implications for system development and propose concrete recommendations to advance translation evaluation practices.
2
DISCOX
This section details the construction and composition DiscoX. The benchmark designed around two core concepts: (1) Discourse-level translation, which requires rendering a complete text as single coherent unit, ensuring consistency in logic and style; and (2) Expert-level translation, which addresses highly specialized fields where the primary challenge is accurately conveying complex concepts and terminology, demanding subject-matter expertise. Based on these principles, we design three-stage construction pipeline (Figure 3) to source texts from two primary and seven secondary domains. We next detail the expert team, construction process, and dataset composition.
2.1 DATA CONSTRUCTION
The construction of DiscoX is large-scale collaborative effort involving 133 professionals (115 vertical domain experts and 18 linguistic specialists; see Appendix for detailed profiles). The process is structured into three stages (illustrated in Figure 3).
Data Annotation. In the first stage, Vertical Domain Experts collect texts from their respective fields. Each source text has to meet three core requirements: (1) reflect authentic professional sce- narios, (2) exceed a minimum length of 1 1,500 characters (Chinese) or words (English), and (3) be specific, self-contained, and amenable to the creation of unambiguous rubrics. Each text is then paired with a comprehensive set of expert-authored rubrics that delineate specific, verifiable eval- uation criteria, including Grammar, Topic Terms, Terminology, and Culdur-lloade Words. For
3

---

Published as a conference paper at ICLR 2026
( bete Ansetinian
90 Quality Centrll & Fiterieg
30 Revinaing Seleciin
Vervica Expert
Redl Seurcno
Cetrets Gongresenin
orguetici Export
Demaie Butr bur we bslonce Qui
Branaer
zoe lect
Teurpera Resed On korfarmenes
Tere nc
DOOK
ACADENCZ
diffimlly Vertfrstive
modeld made (TOC EASY R)
DiscoX
eccept
Figure 3: Overview of the rigorous curation pipeline for the DiscoX benchmark. This process transforms real-world texts into validated evaluation set by employing synergistic system of expert judgment and automated difficulty filtering.
instance, rubric for literary text: "Checkpoint 1: The term ('yuanzi") ) in context must be trans- lated as 'Ditan Park' or 'the park', not 'garden'." This initial phase yields 665 potential tasks, with an average of 9.38 rubrics each. Detailed cases of rubrics can be found in Appendix C.3
Quality Controlling and Filtering. The initial pool of 665 tasks then undergoes rigorous quality control and filtering stage. This stage begins with peer review by linguistic specialists to ensure textual professionalism. To establish high and standardized difficulty, each task subsequently tested against two SOTA LLMs (see Appendix C.I for prompt structure). A task advances to the next stage only if both models fail on a minimum of eight predefined rubrics, signifying a stringent difficulty threshold. Specifically, a task is deemed sufficiently challenging only if both tested SOTA models fail to correctly translate content related to a minimum of eight predefined rubrics.
Reviewing and Selection. In the final stage, domain experts review the tasks that pass the difficulty filter. From this pool, they select the final 200 tasks, representing a selection rate of approximately 30%, ensuring balanced and diverse distribution across domains. The experts then perform final refinement, correcting any remaining flaws in the source texts and honing the rubrics based on the error patterns that are observed in the LLM outputs from the filtering stage. This step serves to maximize the benchmark's 's precision and evaluative utility.
2.2 DISCOX COMPOSITION
The resulting DiscoX benchmark comprises 200 high-quality translation tasks. As detailed in Ta- ble[] the dataset is balanced across two primary domains: Academic (121 tasks) and Non-Academic (79 tasks), which are further subdivided into seven secondary domains. It spans the language pairs of English-Chine Chinese (en-zhh and Chinese-Englhn (zh-ten). Notably, with an overall average input length of 1712.17 17 tokens, DiscoX texts are substantially longer than those in typical segment-level translation benchmarks.
METRIC-S: AUTOMATIC EVALUATION SYSTEM FOR DISCOX
Evaluating the discourse-level translations in DiscoX demands metrics that capture nuances beyond segment-level accuracy (Jiang et al. 2023 Zerva et al. 2024). such as fluency and appropriateness, which conventional automated metrics often fail to address (Wang et al., 2023b). We developed Metric-S, an automated evaluation system that leverages a multi-agent LLM system. As illustrated in Figure it employs multiple LLMs to assess content in three dimensions: accuracy, fluency, and appropriateness. It then applies hierarchical de-duplication process to attribute errors to their root causes before calculating a final, severity score. The prompt of different LLM judges is shown in Appendix C.2
4

---

Published as a conference paper at ICLR 2026
Table 2: Composition of the DiscoX benchmark. The table shows the breakdown of the 200 tasks into Academic and Non- Acad-mic domains, further subdivided into seven specific fields. It presents the task count for each category and the average token length.
Primary Domain
Secondary Domain
Count
Average tokens
Social Sciences
38
Natural Sciences
35
Academic Papers
1875.58
Humanities
28
Applied Disciplines
20
News and Information
37
Non-Academic Tasks
Nomain-Speciin Scenarios
28
1450.49
Literature and Arts
14
Overall
200
1712.17
Plattep Quality Extination
gdd stee: Dvelsate Verificttinn
yrd Stsp Calcakthip Stare
Trunskaiin Instraction
Untrendicted
Level major)
(max)60))
Watroosien on Over
Jass (Ervor Level crriczl)
Eoch Subris -5 points
Rubrica (expert lobel)
(Erer Leiel: evesr)
eman -3/10/00
Ascinsa Judee
FLAI
-Srummatioll Errer Logiet Incsharance
biifkan Phinsing
Erar
(mox:20)
Ench Flanary enner: paints
Incorlitter Tereindayy
Laws
(Erer Lend majer)
Surge
Zonee
(Errr Leeel majer]
Shyle Misnstch
Find Score
Guffural scappattinn
Tand Micestchh
Liierery
Suss
4bbbb
Aprirtinano time majer)
(Brnor Lend miter]
Each Approprittersss srreet paints
(mar 20)
Find Score Soarc Seanc_fuu Scans
Figure 4: Overview of the Metric-S automated evaluation workflow. The system first employs cnnrruction-oolowwiii judge to filter out invalid outputs. It then evaluate the translation's 's Accuracy, Fluency, and Appropriateness. Identified errors undergo hierarchical de-duplication process to isolate root causes before a final score is computed based on the number and severity of the unique errors.
3.1 INSTRUCTION FOLLOWING CHECK
LLMs exhibit a tendency to deviate from translation instructions in discourse-level tasks, often de- faulting to text continuation or summarization. To mitigate this issue, we employ an instruction- following check. Any output that fails to constitute valid translation is immediately assigned zero score and excluded from further evaluation.
3.2 QUALITY ESTIMATION
For outputs that pass the initial check, the system proceeds to evaluate quality across three dimen- sions: accuracy, fluency, appropriateness.
Accuracy The accuracy dimension evaluates how faithfully the translation preserves the source text's meaning, factual information, and emotional tone, while identifying issues such as mistrans- lation, omission, over-translation, or code-switching. Given that discourse includes professional articles across diverse domains, this metric also introduces the rubrics completed in the data annota- tion stage, enabling predefined specification of key terms, such as proper nouns and domain-speciin terminology, that the model must handle correctly to receive credit.
Fluency The fluency dimension focuses on the quality of the translation from the perspective of the target language. The translated text is expected to be evaluated as if read by native speaker,
5

---

Published as a conference paper at ICLR 2026
with attention to linguistic smoothness, lexical consistency, and overall logical coherence. This requirement highlights a critical difference between discourse-leve evaluation and segment-level evaluation, fluency in extended discourse cannot be fully captured by traditional automatic met- rics. It relies on L.Ms'' extensive linguistic resources and their capacity for self-assessment through learned analytic abilities.
Appropriateness The appropriateness dimension reflects a higher-level expectation translation quality. Beyond basic usability, this metric seeks to discover the upper boundary of LLMs' trans- lation capabilities. In addition to accuracy and fluency, appropriateness assesses whether culturally loaded expressions are properly rendered, whether the stylistic features of the source text are pre- served, and whether the emotional tone and literary flavor are faithfully maintained in the translation.
3.3 ERROR DEDUPLICATION AND ATTRIBUTION
In our multi dittesiiimmnii evaluation system, a single root error can propagate into multiple derivative issues. To prevent double-counting, Metric-S employs hierarchical de-duplication and attribution process. This process isolates the fundamental cause each error, ensuring that a single mistake is penalized only once. Specifically, errors marked as "Extremely Critical" in Accuracy take overriding priority, rubric-defined violations are systematically attributed to Accuracy, and for other overlaps, causal analysis determines which error is primary. For example, if lexical choice error leads to dissluent phrasing, only the Accuracy error is retained while the Fluency symplom is discarded. Detailed example of de-duplication can be found Appendix C.4
3.4 DISCOX SCORE CALCULATION
The final score is defined Score = SACe + SPT + SApp- where for each dimension x (Ace, Flu, App). S, MAX,, EN= ef Accuracy, Fluency, and Appropriateness are weighted at 60, 20, and 20 points, respectively, which correspond to the maximum scores MAX, For each error, deductions are applied based on severity: points for minor, points for major, 10 points for critical, and 50 points for extremely critical. The details of severity levels in different domains can be found at
EXPERIMENTS
4.1 MAIN RESULTS
Evaluation Setting. In this section, we present the main results of DiscoX, covering 20 systems: open-source LLMs, 11 closed-source LLMs, 1 domain-specific LLMs, and I neural machine trans- lation (NMT) system. Gemini-2.5-PPr is further employed as the judge model within the Metric-S workflow [ The list of all models can be found Appendix
Results Analysis. From the results in Table we have the following observations. Discourse-level and expert-level translation remain a formidable challenge, and while general-purpose LLMs sig- nificantly outperform traditional MT systems, their performance still falls short of human standards. The results reveal a clear hierarchy, with the top model, GPT-5-hight achieving an overall score of 76.66 on the strength of its accuracy, yet still trailing human experts (80.16). Furthermore, perfor- mance is imbalanced across different evaluation dimensions. indicating distinct and complementary strengths. For instance, while GPT-5 excels in accuracy, other models like Kimi-K2 lead in fluency and appropriateness, and Claude-4 variants are highly accurate but struggle with fluency, indicating distinct and complementary strengths among the models. Specified cases of model performance on different dimensions can be found in Appendix D.2.1
DDe 2025 update: The result of using detailed instruction prompt and employing Gemini-3-PMe as the judge
shown in Appendix T

---

Published as a conference paper at ICLR 2026
Table 3: A ranked comparison of model performance on the DiscoX benchmark. The results high- light that even the most advanced models still trail the human expert. The data reveals imbalanced performance profiles, with different models excelling in distinct dimensions.
Models
Overall
Accuracy
Fluency
Appropriateness
Open-source
Human Expert
80.16
49.80
15.96
14.40
GPT-5-high
76.66
48.65
15.21
12.80
X
Gemini-2.5-Pro
71.25
46.68
13.14
11.43
X
Qwen-3-235B
59.66
33.15
14.96
11.55
Kimi-K2
55.80
27.63
16.44
11.73
03-high
55.57
28.78
15.79
11.00
X
04-mini-high
55.09
29.55
14.29
11.25
Claude-4
54.04
39.38
5,98
8.68
X
Claude-4-T
53.53
38.98
5.47
9.08
Qwen-3-235B-T
49.97
23.20
15.54
11.23
GPT-4.1
49.65
29.25
11.05
9.35
X
DeepSeek-V3
49.60
22.80
16.20
10.60
Doubao-1.66 1.6-T
49.51
29.30
10.11
10.10
DeepSeek-R1
46.06
19.75
16.11
10.20
Gemini-22.5-Flahh-Liite
44.01
26.70
7.91
9.40
Grok-4
43.82
31.38
4.71
7.73
GPT-4o
39,93
20.35
11.28
8.30
Qwen-3-14B
39.36
22.40
7.73
9.23
Qwen-3-8B
28.37
15.13
5.84
7.40
Youdao- 14ndao-14B
46.37
28.50
9.82
8.05
Google-NMT
37.10
18.96
10.12
8.02
4.2 EFFECTIVENESS OF METRIC-S EVALUATION SYSTEM
To validate the effectiveness and reliability of Metric-S, we measured its pairwise consistency Free- itag et al. 2024) with professional human judgments. The experiment designed demonstrate Metric-S's superiority, particularly our proposed benchmark, DiscoX.
Evaluation Setting. The evaluation is conducted on two test sets: our DiscoX benchmark and the WMT 2024 general translation task (Koemi et al.) 2024). We compared Metric-S against two high-performing baselines from the WMT 2024 metric shared task (Freitag et al. 2024): XCOMET- QE (Guerreiro al.) 2024) and ChrF (Popović 2015). For each dataset, randomly sampled 50 cases. In our evaluation, translations are scored by linguistic experts, and both human and metric scores normalized to [0, 1] range for fair comparison. Since ChrF requires reference, we only evaluate its performance on WMT 2024 tasks.
To measure alignment with human judgments, we adopted the unified pairwise ranking consistency framework (Freitag et al., 2024). This method evaluates how often our metric agrees with human experts on which two system outputs is superior. To ensure robust evaluation, we made specific adjustments for different levels: at the system level, we used Soft Pairwise Accuracy (SPA) to provide more nuanced comparison by accounting for statistical uncertainty human rankings. To handle ties in the segment level, we treat metric scores as consistent if their difference is less than 0.05. Details of alignment framework can found Appendix G
Results Analysis. The experimental results, summarized in Table clearly demonstrate the superi- ority of Metric-S, particularly on our proposed DiscoX benchmark. The most striking finding is in the overall average consistency: Metric-S achieves 70.3%, more than doubling the 34.7% score of XCOMET-QE, SOTA baseline. This significant gap highlights the failure traditional metrics to handle complex, Aiscourse-level phenomena. The divergence especially pronounced at the system level, where XCOMET-QE'S consistency plummets to just 10.0% on the zh-ten task, while Metric- S maintains robust 80.0%. These findings confirm that Metric-S provides much reliable

---

Published as a conference paper at ICLR 2026
Table 4:
Pairwise consistency of Metric-S and XCOMET-QE with human judgments. The table
presents a comparison of evaluation metrics at both the system and segment levels for the DiscoX benchmark. ChrF is excluded because it requires reference.
System
Segment
Metric
Overall Avg.
zh ten
en en-
zh-ten
en-
Metric-S 70.3% 34.7%
80.0% 10.0%
90.0%
54.8%
56.4%
XCOMET-QE ChrF
70.0%
26.4%
32.4%
and faithful alignment with human judgments for discourse-level translation evaluation. Case of the output of different evaluation methods can be found at Appendix D.I
Furthermore, we investigated the internal validity of Metric-S to confirm that its advantage stems from the synergistic contribution of core design components. Our ablation studies (detailed in Appendix[A) AA reveal that while the full framework achieves 90% system-level consistency, removing the error de-duplication mechanism reduces this to 80%. More drastically, a simplified approach using a single LLM judge yields mere 20% consistency. This evidence underscores that the effec- tiveness and stability Metric-S are rooted in its integral, multi-component design.
5 ANALYSIS AND DISCUSSION
In this section, we move beyond raw scores to analyze what drives the observed performance dif- ferences. By investigating different perspectives, we seek better understand both the current capabilities and the remaining limitations of LLMs in discourse-level translation.
5.1 LLM BETTER AT CHINESE- TO- TO-ENGLISH THAN ENGLISH-TO-CHINE
Our evaluation in Figure detailed in Appendix reveals significant performance asymmetry between translation directions, with LLMs consistently achieving higher scores in zh-ten than in en en- tasks. This gap particularly pronounced for models like DeepSeek-V3, which shows a 34.8-point performance difference. In contrast, Doubao-1.6- is the most balanced model, with gap of only 7.2 points. This disparity is primarily driven by lower accuracy en-tzh translation, which we attribute to three main factors: (1) data imbalance, where high-quality English corpora are more abundant than Chinese ones; (2) the prevalence of English-centric model architectures that require greater adaptation for generating Chinese; and (3) the inherent linguistic complexities of Chinese, such as its rich morphemes, implicit logic, and strong reliance on context and word order. Detailed cases showing performance asymmetry provided in Appendix D.2.2
Acstemic Non-Academiz Tode
rig
Gemeeo A-Pre
Degr (2
Dostany br
linsaa
deute-d
Upend 200p
Demooi 2.5m Suee
Deep:ieat 3g
Gosten-1 A1
Glsst 4+r
Googlettol todl
(a) Performance by Translation Direction
(b) Performance by Domain
Figure 5: Subplot (a) compares performance across translation directions, showing that models are stronger when translating into English. Subplot (b) compares performance across text domains, showing a clear advantage in translating academic papers.
8

---

Published as a conference paper at ICLR 2026
5.2 STRONGER IN ACADEMIC PAPERS, WEAKER IN LITERATURE
Model performance exhibits clear domain-based disparity, with significantly higher scores in aca- demic translation compared literary translation (illustrated in Figure [ with detailed results in Appendix mp. This gap is attributable to the inherent differences between the domains. Academic papers, with their structured and logical format, are more amenable accurate machine transla- tion. In contrast, literary works require models to interpret complex syntax, rhetorical devices, and cultural nuances- abilities that are difficult to evaluate and represent key challenge for current LLMs. At the individual model level, GPT---highh demonstrates strong generalizability by leading both domains, whereas Kimi-K2 displays particular strength in literary translation, reflecting its fluncc-focuued design.
5.3 PERFORMANCE DEFICIT OF THINKING MODELS
As shown in Table 5 Contrary to expectations, our results indicate that thinking-enhanced mod- els consistently underperform their non-thinking counterparts in translation tasks. This performance deficit is particularly stark for Qwen-3-235B, where the non-thinking version scores nearly 10 points higher (59.66 vs. 49.97), a gap driven primarily by significant drop in accuracy. A similar, though smaller, trend is observed with Claude-4. Error analysis suggests this underperformance is due the thinking models' propensity to either over-summarize the source text, leading to omissions (under-translation), or introduce extraneous structural content (over-translation) thereby compro- mising translation faithfulness. Specified cases can be found in Appendix D.2.3
D.22.3
Table 5:
Comparison of Thinking vs. Non-thinking Models. The data shows that Non-thinking
versions generally outperform their Thinking counterparts in translation tasks.
Models
Type
Score
Accuracy
Fluency
Appropriateness
Thinking 49.97
23.20
15.54
11.23
Qwen-3-235B
Non- Non-thinking
59.66
33.15
14.96
11.55
Thinking 53.53
38.98
5.47
9.08
Claude-4
Non-thinkn non-thinking
54.03
39.38
5.98
8.68
5.4 LIMITATIONS OF TRADITIONAL MT AND DOMAIN-SPECIIA LLMS
Our results indicate that general-purpose LLMs significantly outperform traditional MT models and domain-specifie LLMs in discourse-level translation tasks, whose performance is hindered by two primary factors.
Input Length Constraints. These models struggle with the discourse-level texts in DiscoX due to strict character limits. Even when document translation features are available, their output quality is often inferior, and some models produce disordered content like summarization or omissions on long inputs.
Inferior Accuracy Compared to LLMs. Compared to LLMs, these models exhibit lower accu- racy, primarily due to omissions and mistranslations. They particularly struggle to correctly pro- cess domain-specific terminology and modern internet neologisms, weakness that persists even in domain-specialized systems.
RELATED WORK
6.1 EVOLUTION OF MACHINE TRANSLATION BENCHMARKS
2023b) and IWSLT Agostinelli et al. (2025) have long served as the standard for evaluating translation qual- ity. WMT. in particular, provides annually released test sets covering multiple language pairs and domains, but its focus has primarily been on sentence- or paragraph-level translation tasks. Simi- larly, IWSLT is specialized in speech and TED talk translations, offering relatively short and well-
Traditional machine translation benchmarks such WMT Wang al. 2002a

---

Published as a conference paper at ICLR 2026
structured content. While these benchmarks have played crucial role in advancing MT research, they remain limited in both domain diversity and task complexity, leaving a gap in assessing trans- lation performance under more challenging, real-world conditions.
In recent years, with the rapid improvement of models, translation benchmarks have increasingly emphasized broader coverage, higher task diffieulty, and more diverse application scenarios. The FLORES-101 benchmarkGoyal et al.] (2022) covers 101 languages with professionally translated sentences from diverse domains, enabling rigorous many-to-many multilingual translation evalua- tion. TransBenchLii et al. (2025) introduces a large-scale benchmark of 17k expert-curated tasks across 33 language pairs and multiple e-commerce scenarios, designed evaluate machine transla- tion systems at an industrial scale with greater domain coverage and linguistic difficulty. Redtrans BenchGuu et al. (025b) introduces benchmark of 2,858 zh->en and en->zh test cases covering in- formal, culturalyy loaded, and humor-rich SNS content that demands nuanced, context-aware trans- lation beyond conventional MT settings. However, despite their contributions, these benchmarks mainly remain confined to sentence- or paragraph-Ievel tasks, without extending to discourse- translation or highly specialized domain-specific scenarios that demand expert-level knowledge and in-depth contextual reasoning. For instance, the WMT 2023 shared task discourse-leve literary translation Wang et al. (2024a) highlighted this issue, as the low accuracy of traditional metrics led to results with questionable credibility.
6.2 EVALUATION METRICS ON TRANSLATION TASKS
Traditional machine translation metrics, including rule-based ones (e.g., n-gram overlap) and deep learning-based ones (e.g. COMET Rei et al. 2020), Metric-XJuraska et al. (024)), ), achieve strong performance on sentence or paragraph level tasks because translation quality in this setting largely depends on local adequacy and fluency, which can be reliably captured by these methods. How- ever, their evaluation accuracy declines when dealing with discourse- -eeeel texts, where evaluation requires modeling discourse-Iewe coherence, consistency, and stylistic continuity- dimension that sentence-level overlap embedding similarity fail to represen Wang al. (2023a). This mismatch translation leads to weaker cor- al. (2024). Their reliance on reference translations also
between metric design and the global properties of discourse- relation with human judgmentsKocmi makes them poorly suited for discourse-level evaluation, where no single reference can capture the full range of valid output:(Kim et al. (2024). In such cases, scores become unreliable, and in the absence of references, these methods often degrade sharply or cannot be applied at all. Compared with traditional metries, LLMMassJudge methods offer greater flexibility and interpretabilitDeettse et al. (025). as they can directly evaluate translations without strictly depending on reference out- puts. However, in discourse-level scenarios, the capabilities of a single LLM judge may be con- strained, leading to potential shortcomings in terms of comprehensive coverage across evaluation dimensions as well as accuracy. Evaluation using a single, general-purpose LLM is fraught with issues, including hallucinationsSi et al. (2023) and various biase: (ang et al. (202c)). To better adjust general-purpos LLMs for single-model evaluation tasks, some researchers have begun to train smaller and more specialized evaluation models. These models are fine-tuned with extensive datasets featuring domain-specific evaluation criteria and illustrative examplesZheng et al. (023).
7 CONCLUSION
We present DiscoX the first benchmark for assessing discourse-level, expert-level LLM translation, alongside Metric-S, new reference-free evaluation system that aligns with human judgment. Our experiments show that while current LLMs are promising, they still underperform expert translators, struggling with global coherence, domain-specific terminology, and professional style. We are re- leasing both DiscoX and Metric-S to the community to foster research and advance the development of professional grade translation models.
10

---

Published as a conference paper at ICLR 2026
ETHICS STATEMENT
All data used in this study are collected from publicly available sources and do not contain any private personally identifiable information. For data annotation, we engage qualified experts and provided fair compensation in accordance with market rates. We believe these practices ensure ethi- cal compliance in terms of data usage, privacy protection, and fair treatment human contributors.
REPRODUCIBILITY STATEMENT
We have made significant efforts to ensure the reproducibility of our work. GitHub repository is provided in the abstract, which contains the full dataset, and implementations of the proposed evaluation metries. Section details the evaluation methodology, and Section 4 describes the exper- imental details.
REFERENCES
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Ale- man, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. Xiv preprint arXiv:2303.08774 2023.
Victor Agostinelli, Tanel Alumäe, Antonios Anastasopoulos, Luisa Bentivogli, Ondrej Bojar, Clau- dia Borg, Fethi Bougares, Roldano Cattoni, Mauro Cettolo, Lizhong Chen, al. Findings of the iwslt 2025 evaluation campaign. In Proceedings of the 22nd International Conference on Spoken Language Translation (IWSLT 2025), pp. 412-481, 2025.
Anthropic. System card: Claude opus 4 & claude sonnet https: www cdn .pd 2025, Accessed: 2025-
com (22553640aab55500033333333333333
09-13.
Vicent Briva-lglesias, Joao Lucas Cavalheiro Camargo, and Gokhan Dogru. Large language models" ad referendum": How good are they at machine translation in the legal domain? arXiv preprint arXiv:2402.0768. 2024.
Daniel Deutsch, Eleftheria Briakou, Isaac Caswell, Mara Finkelstein, Rebecca Galor, Juraj Juraska, Geza Kovacs, Alison Lui, Ricardo Rei, Jason Riesa, Shruti Rijhwani, Parker Riley, Elizabeth Salesky, Firas Trabelsi, Stephanie Winkler, Biao Zhang, and Markus Freitag. Wmt24++: Ex- panding the language coverage of wmn24 to 55 languages & dialects, 2025.
URL https:
wwwxiv..rr/abs//255502 12404
Markus Freitag, Nitika Mathur, Daniel Deutsch, Chi kiu Lo, Eleftherios Avramidis, Ricardo Rei, Brian Thompson, Frédéric Blain, Tom Koemi, Jiayi Wang, David Ifeoluwa Adelani, Marianna Buchicchio, Chrysoula Zerva, and Alon Lavie. Are llms breaking mt metrics? results of the wmn24 metrics shared task. In WMT, pp. 47-81, 2024. URL https:://ccaatte
ogy.
org/2024. wmn 1.2
Google Cloud. Gemini 2.5 Flash Lite Model Card, May 2025a. 7/cloud. google. cow/vertex 2-5-flash- lite
URL https:
gemini/
Google Cloud. Gemini 2.5 Pro Model Card, May 2025b. URL httph:httthllccc google com/vertexe ieaerteexea//geeeraaiie----e
mode s/ gew.geimiimmpppeeeeee
Naman Goyal, Cynthia Gao, Vishrav Chaudhary, Peng-Jen Chen, Guillaume Wenzek, Da Ju, San- jana Krishnan, Marc' Aurelio Ranzato, Francisco Guzmán, and Angela Fan. The flores- 101 eval- uation benchmark for low-resource and multilingual machine translation. Transactions of the Association for Computational Linguistics, 10:522-538, 2022.
Nuno M Guerreiro, Ricardo Rei, Daan Van Stigt, Luisa Coheur, Pierre Colombo, and André FT Martins, xcomet: Transparent machine translation evaluation through fine-grained error detection. Transactions of the Association for Computational Linguistics, 12:979-999 2024.
11

---

Published as a conference paper at ICLR 2026
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song. Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501. 2025a.
Hongcheng Guo, Fei Zhao, Shaosheng Cao, Xinze Lyu, Ziyan Liu, Yue Wang, Boyang Wang, Zhou- jun Li, Chonggang Lu, Zhe Xu, and Yao Hu. Redefining machine translation on social net- work services with large language models, 2025b. URL 07901
Ziwei Ji, Naycon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Ye Jin Bang, Andrea Madotto, and Pascale Fung. Survey of hallucination in natural language generation. ACM Computing Surveys, 55(12): 38, March 2023. ISSN 1557-7341. doi: 10.1 10.1145/3571730. URL
httt:://xx.oi..coo www.
Yuchen Eleanor Jiang, Tianyu Liu, Shuming Ma, Dongdong Zhang, Mrinmaya Sachan, and Ryan Cotterell. Discourse centric evaluation machine translation with a densely annotated parallel corpus, 2023. URL/ttps:: /aaxivv.. org/ abs 2305. 11142
Juraj Juraska, Daniel Deutsch, Mara Finkelstein, and Markus Freitag. MetricX-24: The Google submission to the WMT 2024 metries shared task. In Barry Haddow, Tom Koemi, Philipp Koehn, and Christof Monz (eds.), Proceedings of the Ninth Conference on Machine Translation, pp. 492- 504, Miami, Florida, USA, November 2024, Association for Computational Linguistics. doi: 10.18653/v wmt- 1.35. httpst httpss///caaaatoooooooooo org/ 2024 wmt 11. 35/
Seungone Kim, Jamin Shin, Yejin Cho, Joel Jang, Shayne Longpre, Hwaran Lee, Sangdoo Yun, Scongjin Shin, Sungdong Kim, James Thorne, and Minjoon Sco. Prometheus: Inducing fine- grained evaluation capability in language models, 2024. URL
08491
Tom Koemi, Eleftherios Avramidis, Rachel Bawden, Ondrej Bojar, Anton Dvorkovich, Christian Fe- dermann, Mark Fishel, Markus Freitag, Thamme Gowda, Roman Grundkiewicz, Barry Haddow, Marzena Karpinska, Philipp Koehn, Benjamin Marie, Christof Monz, Kenton Murray, Masaaki Nagata, Martin Popel, Maja Popovic, Mariya Shmatova, Steinthór Steingrímsson, and Vilém Zouhar. Findings of the wmt24 general machine translation shared task: The Ilm era here but mt is not solved yet. In WMT, pp. 1-46, 2024. URL https::///aaeecc http:::///laaaahhooooooooooooe org/ 2024. wmi 1.1
Haijun Li, Tianqi Shi, Zifu Shang. Yuxuan Han, Xueyu Zhao, Hao Wang. Yu Qian, Zhiqiang Qian, Linlong Xu, Minghao Wu, Chenyang Lyu, Longyue Wang, Gongho Tang, Weihua Luo, Zhao Xu, and Kaifu Zhang. Transbench: Benchmarking machine translation for industrial-scale applica- tions, 2025. https:/t:
org/ abs 25002255.200 14244
Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. Deepseck-v3 technical report. arXiv preprint arXiv:2412.19437,, 2024.
OpenAl. Gpt-4.1: Product research publication. hutp:://opeenaa.com//neee/ 2024. Accessed: 2025-09) -3.
4-1/
OpenAl. Gpt-5 system card. Technical report, OpenAl, August 2025a. URL https://cdn openai areenai.cmm/ptt. 5 system card. pdf Accessed: 2025-09-13.
OpenAl. Openai o3 and mi-mini system card, 2025b. URL o3 23--od-mini-ssstem--ee
Maja Popović. chrf: character n-gram f-score for automatic mt evaluation. In Proceedings of the tenth workshop on statistical machine translation, pp. 392-395, 2015.
Qwen Team. Qwen3: Think deeper, act faster, 2025. URL
io
#log/gwen3/
Ricardo Rei, Craig Stewart, Ana C Farinha, and Alon Lavie. Comet: A neural framework for mt evaluation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 2685-2702, 2020.
12

---

Published as a conference paper at ICLR 2026
Annalisa Szymanski, Noah Ziems, Heather A Eicher-Miller, Toby Jia-Jun Li, Meng Jiang, and Ronald Metoyer. Limitations of the llm-as-a-judge approach for evaluating Ilm outputs in expert knowledge tasks. In Proceedings of the 30th International Conference Intelligent User Interfaces, pp. 952-966, 2025.
Longyue Wang. Zefeng Du, Donghuai Liu, Deng Cai, Dian Yu, Haiyun Jiang, Yan Wang, Leyang Cui, Shuming Shi, and Zhaopeng Tu. Disco-bench: A discourse-aware evaluation benchmark for language modelling, 2023a. URL https: http://axxiv...ioo.. wbt..//////./......///// abb02237000774
Longyue Wang. Zhaopeng Tu, Yan Gu, Siyou Liu, Dian Yu, Qingsong Ma. Chenyang Lyu, Liting Zhou, Chao-Hong Liu, Yufeng Ma, Weiyu Chen, Yvette Graham, Bonnie Webber, Philipp Koehn, Andy Way, Yulin Yuan, and Shuming Shi. Findings of the wmt 2023 shared task discourse- level literary translation: A fresh orb in the cosmos llms, 2023b. URL https: arxiv.
org/ hrg//bb/23111 03127
Longyue Wang, Siyou Liu, Chenyang Lyu, Wenxiang Jiao, Xing Wang, Jiahao Xu, Zhaopeng Tu, Yan Gu. Weiyu Chen, Minghao Wu, Liting Zhou, Philipp Koehn, Andy Way. and Yulin Yuan. Findings of the wmt 2024 shared task on discourse-level literary translation, 2024a. URL https:
//arxiv.org abs / 2412.11 11732.1122 1322
Longyue Wang. Siyou Liu, Chenyang Lyu, Wenxiang Jiao, Xing Wang, Jiahao Xu, Zhaopeng Tu, Yan Gu, Weiyu Chen, Minghao Wu, et al. Findings of the wmt 2024 shared task on discourse- level literary translation. In Proceedings of the Ninth Conference on Machine Translation, pp- 699-700, 2024b.
Peiyi Wang, Lei Li, Liang Chen, Zefan Cai, Dawei Zhu, Binghuai Lin, Yunbo Cao, Qi Liu, Tianyu Liu, and Zhifang Sui. Large language models are not fair evaluators, 2023c. URL https:
7/arxiv. org abs /00bs/2305..0 2305 17926
Chrysoula Zerva, Frederic Blain, José G. C. De Souza, Diptesh Kanojia, Sourabh Deoghare, Nuno Guerreiro, Giuseppe Attanasio, Ricardo Rei, Constantin Orasan, Matteo Negri, Marco Turchi, Rajen Chatterjee, Pushpak Bhattacharyya, Markus Freitag, and André Martins. Find- ings of the quality estimation shared task at WMT 2024: Are LLMs closing the gap in QE? In Barry Haddow, Tom Kocmi, Philipp Koehn, and Christof Monz (eds.), Proceedings of the Ninth Conference on Machine Translation, pp. 82-109, Miami, Florida, USA, November 2024. Association for Computational Linguistics. doi: 10. 1111165//1/2224.mmml wmm--11..... URL https:
//aclanthology wtw.ooooooy..ggg///222 org/2024 wnn 1.3/
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph Gonzalez, and Ion Stoica. Judging lld-as-a-judge with mt-bench and chatbot arena, 2023. Utt ptttttss//axxiv..om org/
abs/ 22066 05685
13

---

Published as a conference paper at ICLR 2026
A ABLATION AND AUXILIARY EXPERIMENTS ON DISCOX AND METRIC-S
A.1 MULTIPLE RUNS DISCOX
To probe the stability of model outputs and the quality of Data, We further perform multiple inde- pendent runs (three trials) for each model. As shown in Table 6 the negligible performance differ- ence across trials indicates that the model produces consistent high-quality outputs across multiple samples, suggesting both strong model robustness and reliable evaluation.
Table 6: Scores of different models across three runs with mean and standard deviation.
Model
First Score
Second Score
Third Score
Mean
Std. Dev.
Claude-4
52.73
53,21
53.52
53.15
0,40
Claude-4-T
53.28
54.03
54.33
53.88
0.54
-e5ini-2.5-pro
71.53
72.51
73.47
72.50
0.97
GPT-5
76.66
77.02
77.03
76.90
0.21
Qwen3-235h
57.10
59.70
59.00
58.60
1.36
Qwen3-235B-T
50.00
48.00
47.90
48.63
1.16
A.2 ABLATION EXPERIMENTS ON METRIC-S
To validate the robustness and necessities every elements Metric-S, we carry out series of ablation experiments.
A.2.1 EFFECTIVENESS OF THE WORKFLOW OF METRIC-S SYSTEM
We carry out the following experiments illustrate Metric-S's effectiveness and necessities of ele- ments in Metric-S system:
Effectiveness of етrог deduplication mechanism: We evaluate the performance when the result of different judges is direcity applied claculate final score without error dedupli- cation.
Metric-S vs single LLM judge: We try to merge all the prompts dimensions into one and employ single L.M judge model performances DiscoX.
Effectiveness of different dimensions and their weights: We also analyse the results when only the accuracy dimension taken into consideration and when all dimension exists but they are in the same weights.
We compare alignment with human judgments across Metric-S and all experimental settings. As shown in Table 7 Metric-S system achieves the best performance among all settings, demonstrating the integrity of the Metric-S framework and the necessity of its individual components.
Table 7: Pairwise consistency with human judgments under different experimental evaluation set-
tings. The "Average" column reports the mean system-level and segment-level consistency scores. All the experiments employ Gemini-2.5-pro as the baseline judge model.
Metric
Average System-level
Segment-level
Metric-S (Original)
70.30%
85,00%
55.60%
Metric-S (No-duplicate)
66.00%
80.00%
52.00%
Single Accuracy judge
62.50%
70.00%
55.00%
Single L.LM (Detailed prompt)
55.90%
60.00%
51.80%
Metric-S (Same weight for all dimensions)
48.20%
50.00%
46.40%
Single LLM (Simple prompt)
24.90%
20.00%
29.80%
14

---

Published as a conference paper at ICLR 2026
A.2.2 EFFECTIVE OF METRIC-S ON DIFFERENT LLM JUDGES
To examine which model suits best the judge model and whether the strong performance of Metric-S is highly dependent on the choice of the judge model, we replace Gemini-2.55 Semini-2.5-PPro with several alternative baseline models as judges and compare their consistency with human judgments. As shown in Table Gemini-2.5-PPr achieves the best performance among all models, illustrating the rationality of selecting it as the judge model. Moreover, Although the absolute scores of Metric- S vary moderately across different judge models, in all settings, Metric-S consistently outperforms the baseline evaluation metric XCOMET-QE by a substantial margin.
These results indicate that although the absolute accuracy of Metric-S is influenced by the underlying capability of the judge model, the observed robustness is not specific to Gemini. Instead, the stability of Metric-S reflects properties that inherent to its evaluation design, rather than being an artifact of a particular judge model.
Table 8: Effect of different LLM judges on the consistency between Metric-S and human judgments. The table reports system-level, segment-level, and overall (average) consistency scores.
Judge Type
System-level Segment-level
Average
Metric-S Deemini-2.5-pro
85.0%
56.0%
70.3%
Metric-S (DeepSeek-RI)
70.0%
45.6%
57.8%
Metric-S (o3-high)
70.0%
46.4%
58.2%
XCOMET-QE.
40.0%
29.4%
34.7%
A.3 EFFECTIVENESS OF
BIAS FOR LLM JUDGE
To validate whether self- self- ceeferrffeeece bias of models affects the reliability of Metric-S, we compare the results of judgments of the three models from the previous subsection against human judgments. As shown Table g Gemini-2.5-pro does not exhibit a self-preference bias when compared with human judgments. In contrast, o3 demonstrates pronounced bias toward its own outputs, ranking itself first as a judge even though human experts placed it only third. These findings further justify the choice of Gemini-2.5-pp as the evaluation judge.
Table 9: Evaluation results of different judges on DiscoX samples.
Model
Human
Metric-S+Gemini
Metric-SS+DeepseekkH
Metrie-S+o3-high
Gemini-2.5-PP
61.35
69.66
64.70
74.96
Claude-4
57.45
67.90
64.86
71.10
03-high
51.70
62.96
62.50
77.54
DeepSeek-RI
51.25
67.82
63.04
72.78
A.4 PERFORMANCE OF METRIC-S ON MWT 2024 TASKS
To further validate the robustness of Metric-S, we conducted additional experiment comparing its performance with other traditional evaluation metrics on the WMT 2024 task. Considering these tasks include reference answers, we selected XCOMET Guerreiro al. (2024)and ChrF Popović 2015), both reference-based metrics, for comparison.
As shown in Table TO our experimental results indicate that, even without a reference, Metric-S can achieve human evaluation agreement rates close to those of traditional reference-based metrics on translation tasks beyond DiscoX. This reveals that Metric-S, as a robust, reference-free evaluation metric, is capable of effectively assessing model performance across various tasks without reliance on reference translations. More importantly, our evaluation method not only provides scores but also identifies error types and their severity, offering clear, actionable feedback. This transparency helps the evaluated parties understand their strengths and weaknesses, providing a clear path for targeted improvements.
15

---

Published as a conference paper at ICLR 2026
Table 10: Pairwise consistency with human judgments the WMT e0 en- general translation task. This table compares the performance of the reference-free Metric-S against reference-basee metrics. The "Average" column shows the mean of the system-level and segment-level scores.
Metric
Average
System-level Segment-level
Metric-S
72.30%
90.00%
54.60%
XCOMET
68.80%
80.00%
57.60%
ChrF 55.70%
60.00%
51.40%
A.5 ACCUARCY OF LLM JUDGE OUTPUT IN DIFFERENT DIMENSIONS
Beyond consistency with human judgments the score level, further examine whether LLM- based judges produce accurate assessments at finer-grained, dimension-speciiie levels. To this end, select five representative judge models and analyze their evaluations on the three assessment dimensions for the 50 cases introduced in Section 4.3. For each case, linguistic experts manually verify whether the model-generated judgments covering both error types and error severity are correct for each dimension.
As shown in Table [T] the accuracy for all 3 dimensions exceeds 95%, illustrating that Metric-S not only achieves high agreement with human judgments overall, but also demonstrates strong reliability and precision in detailed, dimension-level evaluations.
Table 11: Accuracy of judgment on different dimensions of Metric-S. The number of total cases is 50 5 250.
Evaluation Type
Number of Correct Cases
Correct Rate
Accurate Judge
246
98.40%
Fluency Judge
238
95.20%
Appropriateness Judge
241
96.40%
B ANNOTATOR BACKGROUND
The integrity of the DiscoX dataset is ensured by 133-member curation team operating under dual-expertise model (summarized in Table[12). The team integrated 115 Vertical Domain Experts responsible for sourcing texts and authoring initial rubrics. They were supported by 18 Linguis- tic Experts, a group of certified MTI graduates with 5-16 years of experience. This latter group tasked with ensuring linguistic accuracy, refining the evaluation criteria, and exercising overall project oversight.
Table 12: Table overview of the annotator team provides a breakdown of 133 experts responsible for the dataset curation, detailing the composition, number, professional experience, and background of the Vertical Domain Experts and Linguistic Experts.
Composition
Number of Experts
Avg. Years
Background
Experts practical fields are primarily professionals years of experience, while those in academic disciplines are mainly Master's holders from top-tier Chinese universities.
Vertical Domain Experts
115
4-10
Composed professional linguists and translation specialists, primarily composed of Master's graduates in Translation and Interpreting (MTI) with high-level language certifications (c.g.. TEM-8, 8.0) 5- years professional experience.
Linguistic Experts
5-16
Overall
133
4-16
Expert in specific domain linguistic.
16

---

Published as a conference paper at ICLR 2026
C
DETAILED PROMPT AND RUBRICS
C.I SYSTEM PROMPT
System Prompt
Translate from Source Language to Target Language. Do not output content beyond the
translation.
C.2 JUDGE PROMPT
Below is the complete prompt provided to the judge model, including the main stages of Instruc- tion Following Judge, Aceuracy Judge, Flueney Judge, Appropriateness Judge, and De-duplication Judge.
C.2.1
INSTRUCTION FOLLOWING JUDGE
Instruction Following Judge
fi
# (
1. -
. . M
#  50
# _ 7
$. T 50% LLE,, 1:
#
A
. /
$
*
J 1: Les moyennes quadratiques
F A:
LLes moyennes quadratiques sont un type particulier de moyenne utilisée en mathématiques pour des ensembles de valeurs, notamment lorsque I'on s' s'intéresse des quantités qui dépendent de carrés, comme l'énergie (en physique) ou dispersion statistique."
#
17

---

Published as a conference paper at ICLR 2026
C.2.2
ACCURACY JUDGE, FLUENCY JUDGE AND APPROPRIATENESS JUDGE
PE of Checkpoints
- Ж
[checkpoints] + K #i
[prompt] x [response] A
[[ :
: 1エ
; /
:
+ Мリ H
Judge PE of Accuracy
MR. 14
MRME S
A HENEEEEEEEEEEENNNNEERRRRSRRTRRER
X
1. 1
7
2. + 2.. F $
3. A :
4. BhE RR -
5. - X1A.::0-88..EEE....X22222E....ES
M11
M /
: The H is pretty good.
18

---

Published as a conference paper at ICLR 2026
2
: M
x The Ā5 (weather) is pretty good.
3 [
3:
: Victor Hugo is writer.
WR (Victor Hugo) -
B13 REESHASTEX, FEXCH X.
1. # (Maorr 1 M.. 7
.
2. نm (Critical) : -
M E
* X
3. # Extremely Critical) / 42 3 3 3
- ""
TT [ (
(Critical) ]
MT #7
, , 3 W [ Critical ]
(#
Extremely Critical)
T HR MA. #
* 1
i 7hs HH
[[ 7 E M1
1 6
- /
I"
)1
A
[[ : 1
Judge PE of Fluency
Miprompt AIresponse X HOTE # M.
1. 19 TAA
2. h #2 -
19

---

Published as a conference paper at ICLR 2026
3. E WA HH N
. HA 888T
4.
-
F.
# 7
1. 11
1--1-
2. ΣM:. 10 M
I(
A1
: न म
3 2,
nj : : Xr
: ff
}
:
**
Judge PE of Appropriateness
#2 TT H [response - # XX
& Alresponsee X. 4 (prompt((x) T
20

---

Published as a conference paper at ICLR 2026
1. R ( As X (
- - & SEA Н
2. GA -
1. K 2 R
X MA
2. : :
3. X EICEIAAAAAEENNNNGHHHCIHIIEEEEEEE TEITH8E  TTTTTTTNE33    33E3E3EE
4. X TI # # T
E i * " 1
F -
R. i
+
E : ,1
40:
[(
M #A
[
iprompt;
The prevailing neuroscientific paradigm posits that cognitive flexibility is fundamentally con- tingent upon neuroplasticity-thhe brain's 's capacity to reorganize its neural pathways. This process is largely driven by synaptic pruning and long-term potentiation, mechanisms that dynamically adapt in response to novel environmental stimuli and learning experiences.
presponsez
;
:: 1,
10
; R
J # |π 'prevailing neuroscientific paradigm posits' - $ E # I / # #
T 1
21

---

Published as a conference paper at ICLR 2026
C.2.3
DE-DUPLICATION JUDGE
Judge PE of De-duplication
$ & flu-
ency, appropriateness. checkpoints G #
HE #*
プ A|
$ accuracyfi (#:
(Extremly Critical)
.,
11211
T H. # !
(Exurmlly Critical)
ル E 1 BE #lcchckkpoints
サ cccheckposnss
-
7 H..
1.
2. checkpoints [i
checkpoints
1 3m #4
* T A CA 35 A
,
[checkpoints 5 [#luency[:][[]] サ
m + tī: :
Fchhe@ (fluency|1] [1 T
: 2,
H 3
c प A tT: サN
E :
22

---

Published as a conference paper at ICLR 2026
C.3 CASE OF RUBRICS
The following are sample rubrics for evaluating [a specific task, e.g., academic papers]. In our evaluation, we use rubrics with similar structure but with adjustments the details for other categories of text to better accommodate the unique characteristics of each genre.
Case 1
Primary Domain: Academic Papers
Secondary Domain: Natural Sciences
Prompt (excerpt)
Rubrics (excerpt)
1 :
" T
AÚ. # H
Histologic type म Histo- logic subtype, # x
* 1E N
-
x
-# # #H fa the ## े Biologically in-
メ) = 1O AL (ripllennegative breast can-
cer, TNBC) +
X M
[
dolent, T
A SEAA vague Je
D M B
t1 5) AV 1
XX
101 .
X H
HN
1 की iे 3タ Fibromatosis- like
#7
spindle cell carci- noma
M A B
"
2 Sys- 1 JI JE
K mi (4
temic therapy, #
*
# + ME M10
"
A
म #
AT 7 " D*
Gene rearrangement. $
# मे HMYB-NFIB MOWEE
# CRICI-MAMI2HEEE 10
A # } ý Histologic grade.
#
Tht
2 1L 33
5 Histological grading X
# 5 TH
Nottingham
* #
......
il
/ / 6 AN 18 A2
i 3j Pyknosis and kary-
$
orrhexis E Apoptotic bod- ies
3
> A NY 1
........
#
#
- HA
# Gross pathologic measurement, A E
pathologic S pathological 5
23

---

Published as a conference paper at ICLR 2026
Case 2
Primary Domain: Non-academic Tasks narios
Secondary Domain: Domain
Domain-Specifie Specific Sce-
Prompt (excerpt)
Rubrics (excerpt)
Liability, Indemnification and Release
"sevvants
"
charges and expenses suffered or incurred by it in connection with the termination due to the negligence, breach of duty other default or wrongdoing of the defaulting party, its servants, anployees,agents contractors.
ft
E i0
M
5"'mployces, agents, con-
tractors" #5
"any loss or damage" デ
...The CJV shall indemnify Party B against any loss damage directly indirectly suf- fered by Party B as a result the failure of the Products manufactured hereunder com- ply with the Technical Data("Defecting ucts")or to comply with such laws or regula- tions; provided, however, that such indemnifica- tion shall not exceed the total of ex-factory sales price, costs of delivery and transportation and other costs associated with the recall of these De- fective Products.
#J क i
fE"
प Hil # oZ
fill $ T *"
"provided.
however. 3e
Prod-
that "
"shall not exceed... ix i " 50
1
hereby the other Party and undertakes to hold harmless and defend the other Party against any and all claims, suits, losses, dam- ages, disbursements (including legal and management costs) arising out of any alleged or actual breach or failure to comply with the terms and conditions hereof including but not limited to any infringement of the other Party's intellectual property or other rights occurring as result of the offending Party's 's fault, omission or activities in connection with the Project...
Each Party
indemnifies
ik.
"hold harmless and de- " 50
$ F
"any and all claims, suits, losses" ix t 3 f X f i/Fi2 #
"offending Patty t iे メ # 3//2 1 J7" T
""
(i) promptly notifies Consultant of any third party claim subject to indemnification hereun- der...
"promptly notifies + "
24

---

Published as a conference paper at ICLR 2026
Case 3
Primary Domain: Academic Papers
Secondary Domain: Humanities
Prompt (excerpt)
Rubrics (excerpt)
*t, - # 1 sharp verbal repartee".
:
 1
" MI $ "no reliance on words and let- ters" ters" nonddddeeeeeeen on words and letters".
" "イΣX?. $
AI
+X
- XX
"M E _2r THE #
$
"y"uning words elim- inate words" "using words to dispel words"
-
"
,
X "MXX
# 111
MA #X, XX "F
"I Я Z f # i "/e finger pointing the moon et "a finger pointing at the moon".
"1
"S ilf # M E Word Contemplation Chan (Kan- hua Chan)"
7 1 "AX"" AL
"/
,
"g""oan(s)"; "gongan (koans)"
30 1A 11
24
"x Great doubt"
A P "he Doubt" "reat
#B
# M
3
IX:
"
" "te criti- cal phrase (Huatou)".
- #
*
despprrte state"
F. XX
"IT Øh X 1 MT ? "smmmahiin the lacquer bucket" "brekkngg the lacquer bucket".
EE 2 #13 ("
X " X,
+
CARE
#
-タ -
25

---

Published as a conference paper at ICLR 2026
C.4 CASE OF DE-DUPLICATION
A single root error may lead to multiple, unfairly penalized issues across different dimensions. To address this, we introduce the Hierarchical De-duplications. This system ensures each error is traced back to its single fundamental dimension and penalized only once, eliminating redundant scoring. case is shown blow.
First Judge
acc first judge
flu first judge
app first judge
: 1 2 1 Fi
: :
A : : H1 K [ | 4 Øi: "Does Machiavelli hold up Borgia the model
3
+ F *
μ A 5 t7: "Does Machiavelli hold up Borgia the model
19] - 5 TP: "Does Machiavelli hold up Borgia as the model
? A # - Tor iC (#ER)
6 0 1
X
b... 3 t T
P सा
1 NM 1 मां
: 2
A F : 6
A 3 2
#
3 7
1/j Eo f5:
X #9
H M
n E 5 Mr: 1 x +
fij 'Machiavelli re- inforces the prince's need to be feared by stating: a 1 म 9 1X *
-
# i
X B 2
A B2
3 #L X
X 12 iE 30
#
T
सा
A - मम
De-duplication
: 2
: [
[1luecy([]]011 5 (syle
A 5 =
7 T
Dooss Machiavelli hold up Borgia
as the model "rince?"Af H Xf i of
* stylelllii tf T f X It" :
# H T'acuurayy.
(fluency(i] S [style M
26

---

Published as a conference paper at ICLR 2026
Final Judge
acc sec..final-jue judge
flu fiu.finaljjue judge
app final judge
:
1 2
:
1 : 6
3 7
:E:: WDi
M 10
#
A و tी: "Does
A tं: x
Machiavelli hold up Borgia as the model
X 09 M # # Н
fij MMachiavelli re-
inforces the prince's s need be feared by stating:' E - T #
x
# -
1 7
# R
*
# 33 # 1
6).
B A # 77
H iE. 1
T
: मा
# A1 N M/ ~ #
18 AM
D CASE ANALYSIS
D.1 CASE OF METRIC FIDELITY VALIDATION
The assessment indicates that for the evaluation of model performance, the Metric-S system demon- strates a significantly higher correlation with expert human judgments compared to other metrics. Illustrative examples from the evaluation provided below.
Case is shown below:
Prompt
This phase is called the preliminary design, since this design is still subject to change. Usu- ally, crude, fast analysis method is adequate. At this stage, the cost of the structure estimated, loads and member properties are revised, and the design is checked for possible improvements. The changes are now incorporated in the structure, more refined analysis performed, and the member design is revised. This project is carried out to convergence, the rapidity of which will depend on the capability of the designer. analysis methods, ranging from "quick and dirty to exact", are needed for design purposes. efficient analyst must thus be in command of the rigorous methods of analysis, must be able to reduce these to shortcut methods by appropriate assumptions, and must be aware of available design and analysis aids, as well as simplification permitted by applicable building codes. An up-to-date analyst must likewise be versed in the basics of matrix structural anal- ysis and its use in digital computers, as well as in the use of available analysis programs or software.
clear that variety of
27

---

Published as a conference paper at ICLR 2026
Response
it H S N
Human Judge
Score: 0.675
Output:
i " this design is still subject change":
i "4 up-to-date ana- lyst must likewise..."
i " MU
Metric-S
Score: 0.580
Output:
A M Te!
"M
h Hi
fr: *sice this design is still subject to change iF
+ M 1
m M
THT :
"I"
b m Ey.
"#
M J "epficiann"
#
I B.......
A M
"M"""
n ø
デラ tT:
SuT F "M"";;
X-COMET
Score: 0.177
Output: None
Case is shown below:
Prompt
#R SESAHEAAEESTTTTTTAAAAAAAAAAAAAAE THA SAE 3 +
M XENEIEAEEEEESSSSSSSSSSEEEAAAAAEE SAA K
28

---

Published as a conference paper at ICLR 2026
Response
One in Every Four Ketchup Bottles Worldwide Comes from Xinjiang-A A Pride and Contri- bution of the Region Why Xinjiang Tomato "Red"? Unrivaled Geographic Conditions: The "Golden Land" for Tomatoes Tomatoes, originally introduced to China from abroad, thrive under abundant sunshine and dramatic temperature swings-condition Xinjiang per- fectly provides. With annual sunshine hours reaching 2,550 3,300, Xinjiang ranks first among all Chinese provinces and regions. The region's 's tomatoes are further enhanced by irrigation from Tianshan mountain snowmelt, rich in minerals and trace elements.
Human Judge
Score: 0.525
Output:
X MV Why Xinjiang Tomato So "Red">
i "With annual sunshine hours reaching 2,550 to 3,300, Xinjiang ranks first
among all Chinese provinces and regions. Mf
A i "The region's 's tomatoes are further enhanced by irrigation from Tianshan moun- tain snowmelt, rich in minerals and trace " H i
Metric-S
Score:
0.400
Output:
i .y f
T 7
; : "F"
| "X/D"""" B # X
- AI "#""""
X-COMET
Score: 0.211
Output: None
D.2
CASE ANALYSIS OF DIFFERENT LLM MODELS
D.2.1 CASE ANALYSIS OF LLM PERFORMANCE VARIATIONS ACROSS DIFFRRENT DIMENSION
The models' performance is not balanced across the three dimensions (Accuracy, Fluency and Ap- propriateness). For example, Claude-4 performs well on Accuracy but poorly on Fluency-its trans- lations are semantically correct but not smooth or natural. Conversely, DeepSeek- V3 performs well on Fluency but poorly Accuracy.
29

---

Published as a conference paper at ICLR 2026
Prompt
" TUI THA.EEE....ITTTTTT EEEEEEEEEEEE 
SHATAHAAFAAFPPPP  RRREEEEEEEEEEE NU SHAEAHA HHHTNNNEEEEEEEEE EEEEEEE E SAEAA T
Classification: News and Information; zh-en
GPT-5-high Response
"Globally, the genomes of 13 tomato species have been characterized; now have 11 of them," said Yu Qinghui, [...] a world-leading reservoir of tomato genetic resources.
Score:
accuracy: 60
fluency: 20
appropriateness: 10
Analysis: Accurate terminology, fluent prose, the translation exemplifies the professional style of science news.
Claude-4 Response
"Worldwide, 13 tomato species genomes have been researched in total, and we now possess 11 of them," said Yu Qinghui, [...] opening up treasure trove research in the field of tomato genetic resources, transforming Xinjiang from a major planting region to world-rich area for tomato genetic resources.
Score: 63
accuracy: 45
fluency: 8
appropriateness: 10
Analysis: The meanings "research" and "possess" are correct, but the contextual flow is not smooth; "opening up" is a literal translation and fails to capture the metaphorical meaning.
DeepSeek V3 Response
"Out of the 13 tomato species genomes researched worldwide, we now have mastered 11," said Yu Qinghui, [...] elevating Xinjiang from major planting region to a global hub for tomato genetic resources.
Score: 46
accuracy: 25
fluency: 16
appropriateness:
Analysis: "mastered" is mistranslation and does not fit the original meaning.
30

---

Published as a conference paper at ICLR 2026
D.2.2 ASYMMETRY IN TRANSLATION DIRECTIONALITY: AN ENGLISH- TO-CHINESE CASE STUDY
The case reveals a significant performance disparity, with models performing considerably worse when translating from English to Chinese compared to the reverse direction. For instance, even the top-ranked GPT-5-high shows noticeable quality gap between its Paglish-to-Chinee output and the translations produced by professional human experts.
Prompt
Machiavelli further went on to question the loyalty of the citizens and advised the Prince that "...because men wretched creatures who would not keep their word to you, you need not keep your word them."
The same to Hanfeizi.
He believed that human beings were driven by the greed for profit. We can see some trace in his famous saying: strict mother has kind children, while kind mother has brute children.
Both of them believed that the relationship among human beings was a kind of naked inter- est_oriented relationship.
Classification: Humanities; en-zh
Reference Response (Human)
SAAE::/AEEEEEEAATTTP............ f 1 -
BU EETT- & ビ A
GPT-5-high Response
,.. H SHAN $
T AE x
Score: 81
accuracy: 45
fluency: 16
appropriateness: 20
GPT-5-high Response Judge Analysis
2.
mother has kind children, while kind mother has 34 F H
brute dhindter 30 b
F.
A ) #
10 17 >4 same to Hanfeizi."
# B2 5
31

---

Published as a conference paper at ICLR 2026
D.2.3 CASE ANALYSIS BETWEEN THINKING LLM AND NON-THINKING LLM
The study finds that thinking models generally underperform in non-thinking translation tasks. Thinking models are more prone to omitting information providing summarized translations, which may lead to information loss.
Prompt
Incidents may be diagnosed and resolved by people in many different groups, depending the complexity of the issue or the incident type. All of these groups need to understand the incident management process, and how their contribution this helps to manage the value, outcomes, costs, and risks of the services provided: Some incidents will be resolved by the users themselves, using self-help. Use of specific self-help records should be captured for use in measurement and improvement activities.
Classification: Academic Papers; en-zh
Non-thinking Response
30 F2 H4 TAAAATAAAAAAATTTTTTTAATAEEEEEEEE SIRTE ME H
Non-thinking Score: 71
Accuracy: 45 Fluency: 6
Appropriateness: 20
Thinking Response
Thinking Score: 14
Accuracy: 0 - Fluency: 4
Appropriateness: 10
E SEVERE LEVELS IN DIFFERENT EVALUATION DIMENSIONS OF METRIC-S
Accuracy. Representing the foundation of a translation, this dimension assesses factual correct- ness and usability. Errors are heavily penalized and stratified by their impact: major for word- or phrase-level mistakes, critical for paragraph-Ieva deviations, and extremely critical for discourse- level failures like large-scale omissions.
Fluency. Evaluating the "goodness" of translation, this dimension focuses on whether the text well-formed and reads naturally. As fluency issues typically affect the reading experience rather than the core meaning, all such errors are uniformly classified as minor.
Appropriateness. Measuring the pursuit of "excellence". this dimension assesses stylistic and cultural resonance. Errors tone, style, or cultural adaptation can significantly alter text's intended impact; thus, any such emor is classified as major.
32

---

Published as a conference paper at ICLR 2026
F MODELS EVALUATED ON DISCOX
Opencs LLMs (7): Kimi-K2, DeepSeek-V3 (Liu al. 2024), DeepSeek-RI (Guo
et al. 2025a Qwen-3-235B 1-1555---2BBB-1 A22B-Instruc Qwen-3-235B A22B-222B--100 -2inniig Qwen-3-
14B, Qwen-3-8B (Qwen Team 2025).
Closed-source LLMs (11):
GPT-4o.
GPT-4.1.
o3-high, 04-mini-high, GPT-5- 2025a).
high (Achiam et al. 2023 OpenAl 2025b
-laude-4Soonne--hinnke Doubao- 1.6-- Thhnkiin
2024
Claude-4-Sonner (Anthropic 2025
Gemini-2.5-Pro. Gemini-
2.5-Flash-Lite Google Cloud
2025b
WebCrawi-gook--oommof
Domain-specific LLMs (1): Youdao- 14B
NMT system (1): Google Translate 9
G CORRELATION FRAMEWORK OF EVALUATING ALIGNMENT OF DIFFERENT METRICS WITH HUMAN JUDGMENTS.
To evaluate alignment with human judgments, we adopt unified pairwise ranking consistency framework (Freitag al. 2024). For any two model outputs a and under the same evaluation unit (system segment), let s (a), (b) denote human scores and g (a), s (b) denote metric scores.
pair counted as consistent
sign (a) ss (b)b) sign(s (a) ((b))
(b)).
At the system-level, we employ Soft Pairwise Accuracy (SPA). Unlike standard pairwise accuracy, SPA accounts for the statistical uncertainty in rankings, providing a more nuanced comparison of system performance without arbitrarily penalizing metrics for statistical ties.
At the segment-level, we use the metric, segment-level accuracy measure with tie cal- ibration. If s ("(a) s (b). we additionally treat the pair as consistent when |s (a) dM (b) < 0.05 during the segment-level calculation.
The final correlation score for each metric is the average of the results across all distinct tasks: system-level and segment-level for both en en-zhh and zh-ten translation directions.
H DETAILED RESULTS OF MODEL PERFORMANCE ON CHINESE- AND ENGLISH-TO-CHINER TASKS
The detailed results of model performance on Chinese-to-Engliee and English-to-Chines tasks are shown in Table 13
I DETAILED RESULTS OF MODEL PERFORMANCE ON DIFFERENT DOMAINS
The results of model performance on primary and secondary domains is shown in Table 14
MODEL PERFORMANCE WHEN USING DETAILED INSTRUCTION PROMPTS AND EMPLOYING CEMINI-3-PRO AS THE JUDGE
During our experiment, we detect that with the simple translation instruction as the input, some models will produce somewhat unprofessional translations to align with preferences, hindering to
abbr. Qwen-3-235B
abbr..
Owen-3-235B-T
"abbr. Claude-4-T
abbr.. Claude-4
"abbr. Doubao-1.6-T
abbr.. Grok-4
IIpptt limits: 10,000 characters
abbr. Google-NMT; 5,000 character input limit, with document upload support for longer
33

---

Published as a conference paper at ICLR 2026
Table 13: Comparison of model performance zh-sen and en-*zh translation tasks. The table presents a detailed breakdown of scores for each model across the two translation directions. The 'Diff' column quantifies the performance gap between zh-ten and en-tzh translations.
Models
Diff
Score
Accuracy
Fluency
Approrrittenes
Score
Accuracy
Finency
Appropriateness
GPT-5-high
84.49
$2.35
17.24
14.90
68.83
44.95
13.18
10.70
15.66
Gemini-2.5-Pro
14.15
43.10
10.46
8.70
17.96
Qwen-3-235B
66.15
36,35
16.80
13.00
53.17
29.95
13.12
10.10
12.98
Kimi-K2
64.12
32.90
18.32
12.90
47,46
22.35
14.56
10.55
16.66
a3-high
67.18
36.10
13.10
43.95
21.45
13.60
8.90
23.23
o4-mini-high
70.34
40.10
15.94
14.30
39.84
19.00
12.64
8.20
30.50
Claude-4
62.44
43.45
7.84
11.15
52.62
35.30
4.12
6,20
9.82
Claude-4-T
62.34
44.15
6.94
11.25
44.70
33.80
4.00
6,90
17. 74
Qwen-3-235b-T
58.12
28.45
13.75
17.95
15.16
16.31
GPT-4.1
65.82
39.65
13.62
12.55
33.48
18.85
8.48
6.15
32.34
DeepSeck-V3 V3
66.97
36.55
17.62
12.80
32.23
9.05
14.78
8.40
34,74
Doubuo-L-6T
53.13
33.65
9.18
10.30
45.89
24.95
11.04
9,90
7.24
DeepSeck-R1
58.12
28.60
16.72
12.80
34.00
10.90
15.50
7.60
24.12
Semmnni22.5-Fladtt
62,51
38.75
11,26
12.50
25.51
14.65
4.56
6.30
37.00
Grok-4
59.29
40.70
7.04
11,55
28.33
22.05
2.38
3.90
30.96
GPT-4o
58.13
30,95
9.75
5.30
36.40
Qwen-3-14R
47.20
26.80
9.00
11.40
31.51
18.00
6.46
7.05
15.69
Goo
46.49
25.51
11.90
9.08
27.80
12.47
8.36
6,97
18.69
Qwen-3-8B
32.95
18.70
6.20
8.05
23.78
11.55
5.48
6.75
9.17
Average
61.37
36.00
13.22
12.15
39.94
22.11
9.71
7.75
21.43
evaluate their translations performances accurately. So we update version of dataset with detailed translation instruction prompts to reduce the impact of preference alignment(data released in our Github codebase). Moreover, we also find that Gemini-3-Pro behaves well on multiple benchmarks, revealing its potential to be good judge.
Taking all things into consideration, we add an experiment using detailed instruction prompt and Gemini-3-Pro the judge. Notably, this experimental setting is consistent with that adopted in the Seed-1.8 1.8 technical report. The results are shown Table 15
K USE OF LLMs
In this paper, we use LLM for polishing writing only. Specifically, the models are used to refine grammar, improve sentence fluency, and enhance the clarity and readability of the text, without altering the underlying meaning or introducing new content.
34

---

Published as a conference paper at ICLR 2026
Table 14: Detailed model performance on academic versus non non-academic tasks in DiscoX. The table provides a fine- fine-grainnn -rainednedd breakdown of model rankings and scores across seven sub-domains, categorized under Academic and Non- Non-Academin Academic Texts. In the header, 'R' and stand for Rank and Score, respectively.
News and Information
78.97
71 1..66
55.05
60.76
65.41
62.73
59.19
52.62
56.78
55.86
50.95
44.79
47.92
54.59
52.38
49.51
42.76
44.83
28.00
59
2
6
10
3
4
7
12
8
9
14
18
11
13
15
19
17
20
Literature and Arts
64.07
43.00
55.57
49.00
40.86
35.00
33.36
41 19
33.50
39.14
32.21
28.50
21.93
34.29
33.07
15.50 550
10.57
68
28.
22
Non araddmid Tasks
R
1
2
5
3
4
7
9
12
11
8
14
15
18
10
13
17
19
20
Domain Specifiec Sccnarios
76.00
69.32
51.00
47.82
50.00
47.43
49.71
48.43
44.86
37.82
42.32
46.93
41.58
42.71
31.33
35.89
32.11
28.86
30.50
18.64
S
R
1
2
3
7
8
5
6
10
14
12
9
13
11
17
15
16
19
18
20
Score
76.03
69.58
53.66
52.58
54,86
54.68
53.20
50.80
47.86
45.94
48.10
46.20
40,72
42.62
40.59
43.33
40.43
34,27
34,42
21.59
Overall
Rank
2
5
7
3
4
6
8
12
9
11
15
16
13
17
19
18
20
Natural Sciences
75.23
70.37
61.03
53.80
45.80
48.40
52.51
54.57
56.09
48.11
49.71
49.86
46.79
39.97
41.26
42.66
32.51
42.40
36.03
33.94
2
3
10
7
5
11
9
8
12
16
14
20
15
19
Applied Disciplines
1 84.25
76.35
69.70
63.15
63.70
64.25
63.80
62.95
57.40
58.80
51.40
62.50
55.95
58.50
54.20
52.60
54.70
47.00
46.65
49.
R
2
3
7
6
4
5
8
18
12
10
17
9
11
15
16
14
19
20
Academic Papers
Social Sciences
1 78.
78.79
75 65.63
66 66.18
18
61.11
64 29
58.45
55.61
57.68
49.74 74
52.84
51.97
54.39
47,00
50.68
49.58
47.21
43.99
43.97
97
41.35
31 31.66
R
2
3
5
4
6
8
7
13
10
11
9
16
12
14
15
17
17
19
20
Humanities
S
71.93
67.39
58 58.86
54 86
52.
53.50
49.14
14
47.50 57.50
49. 18
52.18
43.89
50.43
45. 12
50.04
39.18
34.57
33.25
32.68
68
33.18
22. 96
R
1
2
3
11
12
10
6
14
8
13
15
16
17
19
18
20
Score
77.07
72.32
63.58
57.88
5 56.02
55.36
54.57
55.30
$1.34
52.07
50.58
51.67
49.98
48.31
46.24
44.12
39.60
42. 68
38.86 86
32.7 79
Overall
Rank
2
3
4
6
8
7
11
12
10
13
15
16
18
17
19
20
Gemini-2.3 2.5ni-2.55-Faa Flash-hhLii Lite
Gemini-2.5-PPt 2,5-Pro
Qwen-3-235h 3-235B-1T T
gwen-3-235B
Doubao-1.66 16-TT
GPT-5-high h
04-mini-high
Claude-4-1 4-T
DeepSeek-VV
Youdao- 14B
DeepSeck-RR
owen-3-14B
Google-NMT
Qwen-3-8B 8BB
Models
Kimi-K2 K2
03-high
Claude-4
GPT-4.1
Grok-4
GPT-40
35

---

Published as a conference paper at ICLR 2026
Table 15: A ranked comparison of model performance on the DiscoX benchmark using detailed instruction prompt and Gemini-3-Pro the judge model.
Models Gemini-3-Pro
Overall
Accuracy
Fluency
Appropriateness
75.83
48.75
17.82
8.88
1-highh
75.06
45.85
18.71
10.50
DeepSeek-RI
73.90
46.33
18.76
8.85
Gemini-2.5-Pro
73.48
46.15
18.08
9.25
Claude-Sonnet445
71.65
46.76
16.55
8.69
Instrcutt
65.70
37.10
18.47
10.18
DeepSeek-V3.
64.55
38.80
16.60
9.15
GLM-4.6
63.90
39.40
16.59
7.88
o3-high
60.60
32.40
18.83
9.40
04-mini
59.10
31.33
17.77
9.98
Qwen3-T
59.10
30.78
18.20
10.10
2i5ini-2.5-Flash-Laa
56.00
32.18
15.75
8.06
DeepSeek-V3
55.90
28.33
18.50
9.05
Kimi-K2
52.07
24.53
18.55
9.18
GPT-4o
50.40
25.70
16.81
7.90
36


Reviews:
[{"review_id": "sVmNWDfTFR", "reviewer": "Reviewer_Lndd", "summary": "The authors present DiscoX, a new Chinese-English benchmark composed of 200 long-form texts (average 1700+ tokens) from seven professional and academic domains. The construction involved a multi-stage process with 133 domain and linguistic experts. To evaluate translations on this challenging benchmark, the paper also introduces Metric-S, a reference-free, multi-agent LLM-based evaluation system. Metric-S assesses translations along three dimensions: Accuracy, Fluency, and Appropriateness, and shows a high correlation with human judgments (70.3% consistency). The experimental results on 20 different systems reveal a significant performance gap between the most advanced LLMs and human expert translators, demonstrating the difficulty of the benchmark and highlighting key areas for future MT research.", "questions": "Some crucial methodological details are located in the appendix, such as the definition of severity levels (Appendix D) and the correlation framework (Appendix F). Why not place them in the main body of the paper?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- The shift of focus from sentence-level to discourse-level translation, especially in expert domains, is a critical research direction. Current benchmarks are inadequate for this, and this paper makes a convincing case for the need of a new evaluation paradigm.\n\n- The construction of the DiscoX benchmark is commendable. The multi-stage process involving a large number of domain experts and linguistic specialists ensures high quality and relevance of the test data. The inclusion of expert-authored rubrics is a particularly strong feature that allows for fine-grained, domain-specific evaluation beyond generic quality aspects.\n\n- The development of Metric-S is a good contribution. A reference-free, multi-dimensional evaluation system is suitable for long-form texts where a single reference is insufficient. The three dimensions of Accuracy, Fluency, and Appropriateness are well-defined, and the error de-duplication mechanism is a thoughtful detail that improves the metric's fairness and interpretability. The empirical evaluation is also very extensive, covering many recent powerful LLMs. The analysis provides interesting findings that go beyond a simple leaderboard, such as the performance asymmetry between translation directions (zh→en vs. en→zh), between domains (academic vs. non-academic), and between 'thinking' and 'non-thinking' model versions. These insights are valuable for the community.", "weaknesses": "- The reliance on a single LLM (Gemini-2.5-Pro) as the judge is a concern. While Appendix G.3 tests for self-preference bias, the general stability of the \"LLM-as-a-judge\" paradigm is a known challenge. The paper's own results in Table 9 show that the choice of judge model can significantly alter the rankings (e.g., o3-high ranking itself first). This point is critical and deserves more discussion in the main paper. How can we be confident that the high consistency of Metric-S (70.3%) is not specific to this particular judge model? The paper would be much stronger if it discusses the sensitivity to the judge model more deeply.\n\n- The scale of the benchmark, at 200 texts, is relatively modest, even if the texts are long. This could limit the statistical power of the conclusions, especially when results are broken down into sub-domains with very few texts (e.g., only 14 for \"Literature and Arts\"). I suggest the authors acknowledge this limitation and perhaps discuss plans for future expansion. Furthermore, the benchmark is limited to Chinese-English. The claims about discourse-level translation are broad, and it would be beneficial to discuss the potential challenges in extending this framework to other language pairs.\n\n- The analysis in Section 5.3 is interesting, but the conclusion that \"thinking-enhanced models consistently underperform\" might be too strong. The underperformance is attributed to over-summarization or adding extraneous content. However, this could be an artifact of prompt-engineering. The system prompt used (Appendix B.1) is very generic. It is possible that 'thinking' models require more specific instructions to constrain their behavior to pure translation. Did the authors experiment with different prompts for these models? This would be a valuable ablation study to understand if the issue is a fundamental flaw or a behavioral artifact that can be controlled.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "IrQxApkhos", "reviewer": "Reviewer_A2t8", "summary": "- This paper introduces DiscoX, the first benchmark designed to evaluate discourse-level and expert-level Chinese–English translation. The benchmark comprises 200 cases drawn from seven domains, covering both academic and non-academic contexts.\n- In addition, the authors propose Metric-S, a reference-free, multi-agent LLM-based evaluation framework that judges translations along accuracy, fluency, and appropriateness dimensions. With this score, extensive experiments show that even leading LLMs such as GPT-5-high still trail human experts.", "questions": "1. What kinds of linguistic or contextual features are meant to define this level of translation? How are these aspects represented or evaluated in the DiscoX benchmark? Could you provide an example?\n\n2. What mechanisms in Metric-S (e.g., multi-agent judging, rubric-based scoring, error deduplication) directly enable it to capture document-level phenomena more effectively than previous metrics?\n\n3. Could the authors provide empirical examples or case analyses demonstrating when Metric-S identifies discourse-level errors that traditional metrics overlook?", "limitations": "", "rating": 2, "confidence": 4, "soundness": 1, "presentation": 2, "contribution": 2, "strengths": "1. Expert-Curated Dataset: DiscoX was constructed by 133 qualified professionals, including 115 vertical domain experts (4–10 years of field experience, many Master’s or PhD holders from top-tier Chinese universities) and 18 linguistic specialists (Master’s level in Translation and Interpreting (MTI) graduates with professional translation experience). \n2. Extensive Domain Coverage: The benchmark spans seven domains covering both academic and non-academic.\n3. DiscoX was evaluated on a wide spectrum of models, including 7 open-source and 11 closed-source LLMs.", "weaknesses": "Although the authors emphasize the benchmark's careful construction (claiming over 1,300 person-hours of expert effort), the presentation of DiscoX in the paper remains vague, incomplete, and insufficiently grounded in established machine translation literature. Several weaknesses are evident:\n\n1. Unclear Problem Definition and Weak Motivation:\nThe Introduction fails to clearly articulate why discourse-level and expert-domain benchmarks are needed. While the authors mention that existing benchmarks \"fail to assess whether models can sustain discourse-level coherence, handle domain-intensive terminology, or meet expert stylistic standards,\" these claims are not substantiated with concrete examples or empirical evidence. Only a few representative MT benchmarks (e.g., WMT, FLORES, RedTrans) are briefly cited, without a systematic survey of prior discourse-level or document-level evaluation studies. \n\n\n2. Lack of Theoretical Definition of \"Discourse-Level Translation\":\nThe paper repeatedly uses the term \"discourse-level translation\" but never provides a precise or operational definition. There is little to no discussion linking this notion to existing research in discourse-level NLP or translation studies, such as works on coreference, discourse connectives, or coherence modeling. This weakens the conceptual grounding of the benchmark. Furthermore, Figure 1, supposed to explain the concept, appears incomplete and uninformative, offering no clear illustration of the discourse-level concept or the nature of evaluation differences from \"problematic\" segment-level translation.\n\n3. Superficial Dataset Construction Description:\nAlthough the data collection process involves \"expert-authored rubrics,\" the paper does not adequately describe how these rubrics were designed, validated, or applied. There is no verification of rubric quality or consistency across experts. Likewise, no inter-annotator agreement or reliability measure is reported, despite the benchmark's heavy reliance on human expertise for quality control.\n\n4. Evaluation Metric Criteria Lack Scholarly Basis:\nThe Metric-S framework assesses accuracy, fluency, and appropriateness, but these categories are introduced without theoretical justification or reference to prior translation evaluation literature (e.g., DA, MQM, COMET, BLEURT, or human adequacy–fluency frameworks). As a result, the metric design appears ad hoc, lacking grounding in established MT evaluation standards.\n\n(minor) At table 4, what is the point of including chrF row?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "4uH2wYvIUl", "reviewer": "Reviewer_DyVE", "summary": "The paper proposes a novel data set, DiscoX, designed for evaluating accuracy, fluency and appropriateness in a larger context, and also in specific domains.\nA novel metric is proposed, too, which is better suitable to assess context-related aspects.\n\nSeveral translations were evaluated on this test set, including open and closed LLMs, NMT systems and a human professional translation. It is shown that LLMs still do not reach the human performance on the given data set.", "questions": "Related work should be placed after introduction, not at the end.\n\n\n043: the term \"expert-level\" is not fully clear -- it seems that it refers to very specific domains where expert knowledge is needed in addition to language knowledge?\n\n051: translation between Chinese and English: which direction? \n\n052: what are \"tokens\" in the given context? Chinese characters, or English words, or English sub-word units, or Chinese units after some segmentation? \n\n102: what does \"domain-intensive\" mean? a difficult domain? \n\n\n135: are discourse-level and expert-level parts separated, or there is some overlap? \n\n145: what are exactly \"vertical domain experts\"? \n\n157: 665 tasks -- what does \"task\" mean in the given context? a coherent text to be translated? \n\n192, Table 2: again, what is a \"token\" exactly?\nAlso, does Table 2 refer to English or to Chinese?\nFurthremore, what are \"domain specific scenarios\"?\n\nAlso, why the main distinction between domains is \"academic\" vs \"non-academic\"?\n\nSection 3: Why is an LLM used for calculating the score? Once the errors are identified, the score can be easily calculated using a simple script/code.\n\nTable 4: the results on WMT24 test set is missing\n\nSection 5.3: what are the thinking and non-thinking models? \n\n\n\noverall: spelling should be checked/revised, some spaces are missing between words and punctuations (commas, brackets, etc)", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "the data set is carefully constructed to be difficult for translation, which is a valuable contribution to the community\n\nthree different quality criteria are covered (adequacy, fluency, appropriateness) \n\ndifferent domains are included", "weaknesses": "some details are not clearly explained (see \"Questions\" for details\n\nfor example: what are \"tokens\", how many Chinese and how many English source texts are collected, why is the main domain distinction between academic and non-academic texts, what are \"thinking\" and \"non-thinking\" models", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Eg6uYo2FVI", "reviewer": "Reviewer_y79T", "summary": "This paper presents a benchmark and evaluation system, DiscoX, for discourse-level and expert-level translation tasks, particularly for Chinese-English translation. The system introduces Metric-S, a reference-free evaluation metric that uses large language models (LLMs) to score translations based on three key dimensions: accuracy, fluency, and appropriateness. The authors demonstrate that Metric-S aligns closely with human judgment, achieving significant improvements over traditional reference-based metrics. The paper also introduces a novel multi-stage task construction pipeline to create high-quality translation tasks for expert domains, addressing a significant gap in discourse-level machine translation evaluation.", "questions": "1. How robust is Metric-S across different domains, such as technical writing or legal texts, where semantic trade-offs may differ from general text translation?\n2. Has Metric-S been tested on out-of-domain or noisy data (e.g., user-generated content) to check its stability and potential biases?\n3. Would combining Metric-S with embedding-based metrics (e.g., BERTScore) improve the overall evaluation reliability, especially for more diverse domains?\n4. How computationally intensive is the multi-round feedback pipeline, and is it scalable for use on large translation datasets?", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper tackles the important issue of evaluating discourse-level translation, a topic often neglected in standard translation benchmarks that typically focus on segment-level accuracy. This approach is timely and relevant, especially given the increasing use of LLMs for complex, domain-specific translation tasks.\n2. The authors propose a robust evaluation metric (Metric-S) that combines accuracy, fluency, and appropriateness in a reference-free manner. The empirical results show that the metric aligns closely with human evaluation, demonstrating the potential of LLMs for fine-grained translation assessments.\n3. The multi-stage task creation pipeline for DiscoX ensures that the translation tasks reflect real-world professional demands. The quality control process, including peer reviews and filtering based on SOTA models, adds credibility to the dataset construction.\n4. The proposed score-guided multi-round translation pipeline shows clear improvement in translation quality across various models, such as QWQ, GPT-5, and DeepSeek. This is a practical and useful contribution for real-world applications of machine translation.", "weaknesses": "1. While the Metric-S system is empirically sound, the paper does not provide a theoretical framework explaining why certain dimensions (accuracy, fluency, appropriateness) are weighted in the manner they are, or how these dimensions interact. The authors should consider adding an analysis of the sensitivity of these weights, or provide a more detailed justification for their choices.\n2. Metric-S relies on a single evaluation model, and there is limited analysis of how different LLMs might perform as evaluators. To validate the robustness and independence of Metric-S, the authors should consider conducting experiments comparing different LLM models as judges and analyzing the consistency of their evaluations across models.\n3. The current experiments focus on Chinese-English translation. To make the framework more generalizable, the authors should consider testing Metric-S on other language pairs (e.g., English-Spanish, English-German) or multilingual tasks to assess its robustness across languages and domains.\n4. The paper mentions that a task is included in the DiscoX benchmark only if it passes a stringent threshold (at least 8 rubric failures from two SOTA models). However, the rationale behind this threshold is not explained. It would be helpful if the authors clarified the reasoning for this cutoff or conducted experiments showing how the difficulty threshold affects the overall task difficulty distribution.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)