MENLO: From Preferences to Proficiency – Evaluating and Modeling Native-like Quality Across 47 Languages
Authors: Chenxi Whitehouse
Sebastian Ruder
Tony Zhiyang Lin
Oksana Kurylo
Haruka Takagi
Janice Lam
Nicolò Busetto
Denise Diaz
Venue: ICLR 2026 Poster (2026)
Keywords: multilingual
reward modeling
rl
llm-as-judge
human evaluation
Primary area: foundation or frontier models, including LLMs
TL;DR: Massively multilingual preference evaluation, reward modeling, and post-training to improve LLMs' language proficiency

Abstract: Ensuring native-like quality of large language model (LLM) responses across many languages is challenging. To address this, we introduce MENLO, a framework that operationalizes the evaluation of native-like response quality based on audience design-inspired mechanisms. Using MENLO, we create a dataset of 6,423 human-annotated prompt–response preference pairs covering four quality dimensions with high inter-annotator agreement in 47 language varieties. Our evaluation reveals that zero-shot LLM judges benefit significantly from pairwise evaluation and our structured annotation rubrics, yet they still underperform human annotators on our dataset. We demonstrate substantial improvements through fine-tuning with reinforcement learning, reward shaping, and multi-task learning approaches. Additionally, we show that RL-trained judges can serve as generative reward models to enhance LLMs' multilingual proficiency, though discrepancies with human judgment remain. Our findings suggest promising directions for scalable multilingual evaluation and preference alignment. We release our dataset and evaluation framework to support further research in multilingual LLM evaluation (https://huggingface.co/datasets/facebook/menlo).

Published as a conference paper at ICLR 2026
MENLO FROM PREFERENCES TO PROFICIENCY EVALUATING AND MODELING NATIVE-LIKE QUALITY
ACROSS 47 LANGUAGES
Chenxi Whitehouse Haruka Takagi Janice Lam
Sebastian Ruder!
Tony Zhiyang Lin Denise Diaz
Oksana Kurylo
Nicolò Busetto
Meta Superintelligence Labs chenxwhêmeta. com
ruderêmeta.com
ABSTRACT
Ensuring native-like quality of large language model (LLM) responses across many languages is challenging. To address this, we introduce MENIOO, a frame- work that operationalizes the evaluation of native-like response quality based on audience design-inspired mechanisms. Using MENLO, we create a dataset of 6,423 human-an auman-annotated prompt-responee preference pairs covering four quality dimensions with high inter-annotator agreement in 47 language varieties. Our evaluation reveals that zero-shot LLM judges benefit significantly from pairwise evaluation and our structured annotation rubrics, they still underperform hu- man annotators on our dataset. We demonstrate substantial improvements through with reinforcement learning, reward shaping, and multi-task learning approaches. Additionally, we show that RL-trained judges can serve as generative reward models enhance LLMSs' multilingual proficiency, though discrepancies with human judgment remain. Our findings suggest promising directions for scal- able multilingual evaluation and preference alignment. We release our dataset and evaluation framework to support further research in multilingual LLM evaluation.
Dataset https://hugginggacc.com wttp://ee..oo/ttteeeeeeeeeeeeeee http:///ii.e.eeeeeeeeeteee////ea
INTRODUCTION
In order for LLMs to be most useful across the globe, they need to be able to provide high-quality responses in many languages. Responses should be relevant (Zhuang et al., 2024), factually accurate (Jacovi et al., 2025), and natural (Marchisio et al., 2024; Guo et al., 2025), among other considera- tions. Ultimately, for interaction in any language to be seamless, responses need to be indistinguish- able from those of a native speaker (Novikova et al., 2016; Liu al., 2021). Language proficiency in humans has traditionally been evaluated via standardized tests (Jamieson et al., 2000). While such tests have been applied evaluating LLMs (Anil al., 2023; Mayor- Rocher et al., 2024; Lothritz & Cabot, 2025), they are difficult to scale and do not readily correspond to real-world conversations. What is considered native-like response largely depends on speakers' and listeners' interpretations of whom they are speaking to (Bell, 1984).
To operationalize the evaluation of native-like response quality across languages, we propose Multilingual Evaluation of Native-Like Output (MENLO); see Figure I for an overview. MENLO breaks down native-like response quality into four key dimensions: i) language quality and coher- ence; ii) aligmment with cultural and linguistic nuances of specific language variety/locale; iii) factual correctness and grounding in the local context; and iv) overall writing style and helpfulness.
Building mechanisms from audience design (Bell, 1984), propose creating tailored prompts that effectively evoke local contexts by defining the target audience (e.g., an addressee or reference group), thereby guiding the generated language to converge contextually appropriate "native"
Equal Contribution.

---

Published as a conference paper at ICLR 2026
Imagine you're at an looca nationallll friend s family gathering licale ceunnay How would you paltely ask second helping of
Imogina que esd5 en una rounión une amag españoll educada quleres comiea?
A: Kilro En una reurión tomilaa en Expaña... Ratiig
rekre
Falure
Grand
Seard
B: una reun famil tspars...
XCX
X7
2
Rating
X
Localized Tone
Fluency
Tone
Localized Factuality
1. Prompt template creation
2. Prompt translation
Rubric annotation guideline creation
Response generation and annotation
localization 47 languages
Figure 1: MENLO framework and annotation process. 1) Human-written prompt templates evoking local contexts are created in English for the four dimensions. 2) Prompt templates are translated and localized into 47 language varieties. 3) Annotation guidelines are created that break down each dimension into easy-to-follow rubrics. 4) LLMs are used to generate response pairs for each prompt, which are annotated with Likert-scale ratings and preferences.
styles. We develop instructions that reduce annotation subjectivity and improve inter-annotator agreement. Responses are generated using state-of-the-art LLMs and annotated with ratings on 1-5 Likert scale, with an average Krippendorff's is a = 0.84. Overall, the MENLO dataset consists of 6,423 annotated prompt-response preference pairs, and 81,014 annotations, 47 language varieties.
Human evaluation, particularly at massively multilingual scale is expensive. We thus evaluate the ability of LLMs to serve as judges of native-like quality responses. We find that in zero-shot setting, pairwise vvaluation-when models predict scores for two responses simultaneously (without ex- plicitly predicting prefrence)-signiffaanlly significantlyy outperforms its pointwise counterpart. The advantage of evaluating two responses side-by-side is even bigger than in-context examples with labels. In addition, we observe significant improvements with judges using our annotation rubrics compared to judges without rubrics, highlighting the generality of the MENLO framework.
As zero-shot judges remain below human annotation quality, using the pairwise evaluation setup, we fine-tune Qwen3-4B and Llama4-Scone trained models outperform their SFT counterparts. In particular, a multi-task Llama4-Scom model trained with shaped rewards surpasses frontier API models with the strongest overall performance across 47 language varieties, reaching agreement levels comparable to human annotators.
LLM judges the MENLO training data, finding that RL-
Finally, demonstrate that these judges can be used generative reward models (RMs) directly improve a policy model's proficiency. By using our pairwise RL-trained Qwen3-4B judge to post- train the base Qwen3-4B model, we observe quality gains measured by both LLM evaluators and human raters. However. LLM evaluators tend to be overconfident in assessing improvements compared human judgments (+0.6 higher gain). This finding shows that while judges trained with our framework can successfully drive model improvements, the gap between LLM and human raters highlights the remaining challenges in reliably modeling native-like quality across languages.
Our contributions are the following: 1) We develop MENLO, a framework for the evaluation of native-like response quality in four dimensions, based on principles from audience design, employ- ing parametric templates and carefully crafted annotation guidelines. 2) We create the MENLO dataset, consisting of 6,423 annotated prompt-response preference pairs 47 language varieties. 3) We evaluate zero-shot judges the annotated data, demonstrating the benefits of pairwise evalua- tion and rubrics. 4) We show that multi-task RL and reward shaping enables fine-tuning a judge that on par with human annotators in 47 language varieties. 5) We demonstrate that pairwise fine-tuned judges can used as generative RMs to improve policy model language proficiency, while we find that LLM evaluations tend overestimate improvements compared to human raters.
Our framework unifies the MENLO dataset, RL-trained pairwise judging, and generative reward modeling, offering practical and scalable approach to both assess and improve native-like quality.
2

---

Published as a conference paper at ICLR 2026
Dimension
Definition
Key Question
Example Prompt
Fluency
Language profitiency to expert- native speaker
Is the model reesonne coherent, versed, cleat and free from grammatical errors?
you could make one change to the education system your country. what and why?
Tone
Overall writing style Voice" of the response.
Is respanse helpful, insightful, engaging, and fair?
feel I've
weeks.
Localized Tone
Aligmment cultural
response employ locally ll relevant expressions and is culturally sensitive?
Imagine you're an [locale_nateee nationaliion friend's family gathering [locale_coung]] country. How would you politely ask for second helping of food?
regi onal, and lnguuisic nuances.
Localized Factuality
Factuality, complete- ness, and grounding in local context.
Is response factual, complete, and grounded the local context?
You are staying with host family in [locale country] during [lncale holiday). They invite you to help with the preparations. What tasks might you be expected to help with, and what they symbolize?
Figure 2:
Dimensions of native-like response quality in MENLO and example prompt (template).
Table 1:
Annotation and statistics of MENLO across evaluation dimensions. IAA presents Krippen-
dorff's 's a measuring inter-annotator agreement. Average token counts are computed using Qwen3-45 -4.
Avg # Tokens
Rating Scale) IAA
Dimension
# Annotations
# Annotators
# Prompts
Prompt
Response
Mcan
Std.
Fluency
23,556
450
1,820
81.6
804.3
0.82
4.01
1.11
Tone
18,712
429
1,410
27.8
$75.3
0.86
3.48
1.35
Localized Tone
22,324
530
1,815
71.7
559,2
0.83
3.89
1.17
Localized Factuality
16,422
525
1,378
121.8
839.1
0.84
3.82
1.16
Overall
81,014
1,934
6,423
75.6
692.2
0.84
3.80
1.20
2
THE MENLO DATASET
MENLO characterizes native-like conversational response quality in language along four key di- mensions: fluency, tone, localized tone, and localized factuality. These dimensions beyond prior work that focused mainly on naturalness (Novikova al., 2016; Liu al., 2021; Guo et al., 2025) and are motivated by work on language proficiency assessment (Ke & Ng, 2019), cross cross-cultural variation (Hershcovich al., 2022; Myung et al., 2024), local knowledge grounding (Hupkes & Bogoychev, 2025). We provide further context on our definition of these dimensions in Figure 2.
From a sociolinguistic perspective, the Style Axiom (Bell, 1984) states that intraspeaker variation (style) reflects interspeaker variation (social). Native-like quality is therefore not a single fixed target but a socially conditioned range of stylistic choices that depend on interlocutors. Key mecha- nisms include accommodation, where speakers adapt their style the addressee, and referee design, where speakers align with an absent reference group they wish to identify with. These mechanisms motivate our focus tone and localized tone central to native-like quality. To operationalize these ideas, we design Homan-writted parametric English prompt templates for each dimension with placeholders such [locale diccallnnationall wanninallttyyll. defining the addressee or reference group. these prompts evoke local contexts and guide models toward contextually appropriate "native" styles. We provide an overview of the MENLO framework and annotation process in Figure 1.
[locale, socale_holine holidayl, etc. By
We select 47 language varieties representing typologically diverse of widely used languages and their major variants. including, e.g., South American and European varieties of Spanish and Portuguese, several varieties of English, and romanized versions of non-Latin script languages (see Appendix A.2). Native speakers are recruited to professionally translate these prompt templates, with placeholders instantiated using locally relevant entities. As native quality tied to the local context, ensure that native speakers are from the specific regions where the corresponding lan- guage varieties are spoken. Similar criteria are used to select annotators for each language variety. Each language variety has approximately the same number of examples in MENLO.
To ensure consistency in evaluation, we develop instructions that reduce the subjectivity of the an- notation and break down the four broad dimensions into easy-to- followw rubrics and self-explanatory signals (human-written), Annotators receive guidelines with examples for each dimension. We ad- ditionally develop customized annotation tool and annotator screening tests filter out unreliable
3

---

Published as a conference paper at ICLR 2026
Table 2: Comparison of multilingual response quality datasets: RECON (Doddapaneni al., 2025), PARIKSHA (Watts et al., 2024), BIGGEN BENCH (Kim et al., 2025), M- M--REWARBENCCH (Gureja al.. 2025), MM-EvAL (Son al., 2024). C]: of languages, [P]: of prompts, IAA: inter-
annotator agreement, IF: instruction following. *: 81,014 annotations; 1,776 test examples.
Dataset
[4]
[D]
IAA
Dimensions
Prompts
Responses
Ratings
MENLO
47
6,423
0.84
Fluency, tone, localized tone, localized factuality
Human-writeen,
Annotated in each language
Preference
translated & localized
& 1-5
PARIKSHA
10
200
0.54
Hallucinations. task quality. linguistic acceptability
Human-written
Annotated in each language
Preference
& 0-2
BIGGEN BENCH
10
420
Poem, reasoning. humor, translation, historicul text LLM-uugmetted
Human-
Generated in each language
1-5
RECON
3,000
theory of mind, reason- ing, safety, planning. etc.
Translated
Generated each language
Preference
& 1-5
MM-EVAL
18
4,981
Reasoning, chat, linguis- tics, hallucination, safety
Translated
Generated in language
Preference
M-REWARD
23
66,787
Chat, safety. reasoning. Translated translation
Translated
Preference
BENCU
annotators. Furthermore, we train 1-2 expert annotators per language who provide language language-specific feedback to annotators and provide gold annotations on subset of examples.
We generate two responses for each prompt with state-of fitie-of-theaaae tttthheartt art LLMs including GPT-4o, Llama4- Maverick, Malmaa-Mavert with Search, and Gemini 1.5 with Search. We present both responses in randomized order to human annotators and ask them to provide 1-5 Likert ratings per response, allowing ties. Each response pair annotated by at least annotators, with final scores aggregated via majority vote. Annotators achieve high reliability, with an average Krippendorff's s 0.84.
Overall, MENLO consists of 6,423 annotated prompt-response preference pairs across 47 language varieties, each containing prompt, two responses, and corresponding scores, totaling $1,014 human annotations. Summary statistics are reported in Table 1. Example prompts per dimension are shown in Figure 2. Further details of MENLO including annotation process, language coverage, rubrics, and full examples featuring responses and their corresponding ratings are provided in Appendix A.
Table 2 compares MENLO with existing multilingual preference datasets. MENLO provides local- ized prompts and responses, spans more languages, and reaches higher agreement than prior work.
EVALUATING LLM-JUDGES ON MENLO
We next evaluate the ability of LLMs to serve as automatic judges of native-like quality on MENLO. Out of the 6,423 pairs, we hold out 1,766 pairs (3,552 responses) as the test set,' and use the remain- der for training and prompt development (see $4 and $5). Where expert annotations are available, we use these as labels. For the remaining responses, we average the annotated ratings of each re- sponse.? Our evaluation focuses on three questions: (i) how pointwise and pairwise setups compare, (ii) the effect of few-shot exemplars, and (iii) the role of explicit grading rubrics.
We benchmark range of open-source and API-based models, covering both thinking and non- thinking variants: Qwen3-4B, Qwen3-32B, Llama-3. 1 1.ama-3.3-70B, Llama4-Scout, 03, gpt-4o, and gpt-4.1. 1.....A All models are used in the default setup with maximum output length 8192.
We report two primary metrics: (i) Macro-F1 for 5-way classification, and (ii) Preference accuracy over Win/Loss/Tie outcomes. Note that we do NOT directly ask for preference judgments; rather, infer these from the assigned grades. Additionally, we report classification accuracy, Krippendorff's a, which measures agreement with human annotators while accounting for chance agreement and missing data, and provide detailed per-dimension and per-language breakdowns in Appendix E.
"Translations of the same prompt template assigned the set prevent train-test leakage. 2 Multiiple annotations can be used in future work on pluralistic alignment (Sorensen et al., 2024). 3I Laama models nstructiiontuunl and omit the Instruct suffix for brevity.
4

---

Published as a conference paper at ICLR 2026
Table 3: Zero-shot and few-shot results of open-source and API models the MENLO test set using POINTWISE (grading single responses) and PAIRWISE (grading response pairs) scor- ing (see $3.1). Macro-Fl shows 5-way classification performance and Preference reports accuracy on Win/Loss/Tie. Reported gains/loss are relative to zero-shot pointwise performance.
Macro F1
Preference Accuracy
MODELS
PERO-SHOT POINTWISE
SHOT POINTWISE
ZERO-SHOT PAIRWISE
ZERO-SHOT POINTWISE
FEW-SHOT POINTWISE
ZERO-SHOT PAIRWISE
Qwe33-4B
23,06
31.18 48.12
35,46
40.54
39.35
57.13 +16.7
Qwen3-32B
28.53
35.45 46.02
37.48 48.55
42.19
42.87 .0.68
59.12 +16.9
Llama-3. 1-88
22.27
23.29 +1.00
29.46
39.92
37.15
50.45 +10.48
3.
27.93
30.52 +2.59
37,50 +9.57
37.37
38.56 -1.19
55.32 +1.889
Llama4-Scout
25.63
32.84 +7.21
36.11 +10.48
42.19
41.22 a97
56,25 +14.12
03
26.54
27.92 +1.38
35.35 4881
45.07
44.68 .0339
58.72 413.68
gpt-4o
25.99
29.57 43.58
37.57 411.58
42.92
45.87 -2,95
57.98 415.09
gpt-4.1
32.23
33.84 +1.61
38.53 46.30
41.73
44.00 -2.27
59.23 417.50
3.1 POINTWISE VS. PAIRWISE
Although MENLO provides paired responses for each prompt, the presence of detailed grading rubrics means that pointwise evaluation is principle sufficient: model could assign absolute scores to individual responses without needing comparisons. However, pairwise setups may provide stronger relative signals by anchoring judgments against another candidate. We therefore compare three setups: Zero-shot pointwise: the model is given prompt, single response, and detailed 5-point grading rubric, and asked to generate evaluation reasoning (in thinking) and assign a final grade; Few-shot pointwise: we additionally provide three graded examples: one from 1-2, one with a grade 3, and one from 4-5; Zero-shot pairwise: the model presented with both re- sponses the same prompt and asked assign grade each, following the template in Figure 17 (Appendix C), without constraints on ties. The order of the two responses is randomized.
Table 3 reports Macro-F1 and Preference results. Zero-shot pairwise consistently outperforms both zero-shot and few-shot pointwise scoring across models, with gains up to +12.4% in Macro-F1 and 188.0% in Preference accuracy over zero-shot pointwise. Few-shot pointwise improves Macro- Fl relative to zero-shot pointwise but yields only marginal gains in Preference, still falling short of zero-shot pairwise by an average of 5..5% Macro-Fl and 15.1% in Preference across models.
These results indicate that models are substantially more reliable at assigning scores when evaluating two responses side by side, even without ground-truth labels. The unexpectedly large gains over few- shot in-context examples highlight pairwise evaluation, which explicitly anchors outputs against competing candidate (Wang et al., 2025). as a promising direction for improving automated judging reliability, even when the ultimate goal is pointwise scoring, We also evaluate few-shot pairwise on Qwen3-4B, observing only a small gain in Macro-F1 (+0.6) relative to zero-shot pairwise, further supporting our findings. Future work may investigate whether extending pairwise comparisons to listwise evaluation multiple responses offers additional benefits.
3.2
WITH AND WITHOUT GRADING RUBRICS
We further examine the role of detailed grading rubrics in judge performance. All rubrics are human- written 5-point guidelines specific to the dimension and question type of each prompt. Examples of innension-specifii rubrics are shown in Appendix A.3.
Table 4 compares zero-shot pointwise and pairwise performance with and without access to rubrics. The latter shows only the five class labels, without accompanying criteria or definitions. Results show that rubrics provide substantial benefit, especially for pointwise evaluation, yielding average gains of +4.3% Macro-F1 and +2.5% in Preference accuracy. In contrast, pairwise evaluation benefits more modestly, with improvements of roughly +1% on both metries.
These findings suggest that judges perform better when grounded, either by explicit rubrics or by comparison with another response. Since pairwise comparison itself offers strong grounding sig-
5

---

Published as a conference paper at ICLR 2026
Table 4: Zero-shot performance comparing without and with detailed 5-Point Grading Rubrics.
Macro Fl
Preference Accuracy
MODELS
POINTWISE
PAIRWISE
POINTWISE
PAIRWISE wo/ Rubrics
Rubrics
w/ Rubrics
wo/ Rubrics
Rubrics
wa/ Rubrics
Rubrics
Rubrics
Qwen3-4B
16.00
23.06 2206
32.74
35.46 2272
33.52
40.54 47.02
54.08
57.13 4305
Qwen3-328
25.59
28.53 2294
38.10
37.48 042
43.32
42.19 113
59.23
59.12 0.11
Llana-3. 1-88
21.50
22.27 4027
30.89
29.46-144
38.34
39.92 +1.8
49.55
50.45 40,0
Llama-3. 3-708
22.71
27.93 .3.2
35.12
37.50
34.54
37.37 42.83
56.29
55.32 .097
Llama4-Scout.
22.15
25.63
+3.48
35.21
36.11 +0.0
41.28
42.19 10.91
55.10
56.25 +1.15
25.43
26.54 s1.11
35,60
35.35 0.25
45.13
45.07 ..06
57.98
58.72 .0.74
got-4o
22.45
25.99 +3.54
36.74
37.57 +00
37.60
42.92 +5.32
56.85
57.98 -1.15
got-4.1 22.26
32.23
29.97
37.35
38.53 .1.18
38.67
41.73 43.06
56.96
59.23 J227
Table PAIRWISE SFT, RL, and SFT+RL-trained Qwen3-4B and Llama4-Scour results. RL-trained models perform best overall.
Table Ablation of different reward designs for PAIRWISE RL-trained Qwen3-4B. Smooth. and Prefer. refer to Reward Smoothing and Preference Bonus. See $4.1 1 for details.
Llama4-Scout Marco-F1 Preference Marco-F1 Preference
Qwen3-4B
PAIRWISE
REWARDS
Marco-F1
Preference
ZERO-SHOT
35.46
57.13
36.11
56.25
Binary Only
37.11
58.27
SFT
33.44
53.68
44.17 -8.0
60.08 13,3
Binary Smooth.
37.30 10.19
51.47 4.80
RL
39.44.3.8
60.02.259
45.62
62.60
Binary Prefer.
37.05
60.48 22
SET + RL
58.78,146 4387
45.82
61.10.aa5
Binary Smooth. Prefer.
39.44 +2.3
60.02 41
nal, it sees limited impact from rubrics. This highlights the importance of high-quality rubrics: if judges could automatically generate and evaluate high-quality, detailed rubrics, we hypothesize that the performance gap between pairwise and pointwise evaluation would further narrow.
TRAINING LLM-JUDGES ON MENLO
Having established in that pairwise evaluation yields substantial advantages over pointwise scor- ing, we next examine whether training LLMs as judges can further close the gap to human annota- tors. We train on the MENLO training split (total 4,675 response pairs, where 232 pairs are held out for validation) and explore different learning strategies, model families, and reward designs. Inspired by the success recent reasoning- judges such as J1 (Whitehouse al., 2026), we compare supervised fine-tuning (SFT) and reinforcement learning (RL., as well single-task (dimension- specific) and multi-task (all dimensions) training.
We fine-tune two contrasting models: Qwen3-4B (dense, reasoning-oriented. and Llama4-Scout (Mixture-of-Eppert non-reasoning), which differ in architecture and cognitive approach. For SFT, models directly predict 5-point grades using cross-entropy loss under teacher forcing, without inter- mediate reasoning generation. For RL, we use GRPO (Shao et al., 2024) with the template from Fig- 17, encouraging step-by-step reasoning before score assignment. Following Whitehouse al. (2026), we augment training data by including both response orders (A,B) and (B,A) to mitigate positional bias. Training details are provided in Appendix D.
4.1 REWARD DESIGNS FOR R.L
To make RL training effective, we design a composite reward signal that combines absolute accuracy with relative preference alignment and robustness to near-miss predictions: (i) Pointwise binary re- ward: 1 if the predicted score matches the gold label, 0 otherwise. (ii) Reward smoothing: partial reward (+-0.5) if the prediction differs by exactly one grade. (iii) Preference bonus: additional +1 if the sign of the difference between the two predicted scores matches the label. (iv) Penalties: -1 for invalid or missing scores, and 2 for formatting violations, i.c. each tag must appear in the correct order and only once.

---

Published as a conference paper at ICLR 2026
All reward components are summed to produce the final RL signal. Formally, the reward can be expressed as follows, where s and gt represent predicted and ground truth grades, respectively:
Σ max [1|s: gl.). 0.5 [[
g4,l = 11
(iinnss - SB) = sign(gta gl
(E(B)))
preference bonus
pointwise hinary reward w/ reward smoothing
1 failed extraction
0.2 1 [formatting violation
extraction penalty
format penalty
4.2 OVERALL PERFORMANCE: SET VS. RL
We first compare the overall performance of SFT and RL-trained models. Table 5 shows that RL- trained Qwen3-45 4BB and Llama4-Scour consistently outperform their SFT counterparts. For inherently thinking models like Qwen3-4B. SFT without Chain-of-Tie Chain-of-Thought (CoT) reasoning actually hurts per- formance, causing a 2.0% drop in Macro-F1 and -3.5% in Preference accuracy. In contrast, RI,, which incentivizes reasoning, improves performance by +4.0% in Macro-F1 and +2.9% in Prefer- ence, surpassing the best frontier API model gpt-4.1.
For non-thinking models like Llama4-Scout, SFT already provides substantial gains (+8.11% in Macro-F1 and +3.8% in Preference) compared to zero-shot. RL training further improves results, particularly in Preference (+2.5%). This demonstrates the promise of pairwise RL training across model families, scales, and reasoning capabilities.
We also experimented with initializing RL from the best SFT checkpoint, but observed little or no improvement over starting RL from scratch. Models trained on SFT without CoT tend to copy the placeholder "<think> Your analysis and reasoning here. </think>" from the prompt rather than generating meaningful reasoning, which limits the benefit of RL. This suggests that for tasks requiring reasoning, it is preferable to start RL directly when the SFT target lacks CoT supervision.
4.3 ABLATION OF RL REWARDS
In Table 6, we ablate the RL reward design to validate the contribution of each reward component in RL training: (i) binary only: reward +1 for exact score match, 0 otherwise; (ii) binary+smooth. adds partial reward for near-miss scores, preference bonus; (iii) binary+prefer: includes pref- erence reward, no smoothing; and (iv) bimary+smoort monoothh+prffr::: the default reward design in $4.1. Results show clear benefits from combining reward smoothing and preference bonus, achieving the best overall Macro-F1 and Preference accuracy for Qwen3-4B, achieving +22.3 boost on Macro-Fl and 11. on preference accuracy over the binary only reward.
4.4 BER-DIMENSION PERFORMANCE AND SINGLE vs. MULTI-TASK
Next, we compare pairwise RL-trained Qwen3-4B models trained jointly across all dimensions versus individually per dimension. Across the four dimensions, Tone achieves the strongest performance, with a Macro-F1 of 43.1 in the zero-shot setting and gains of up to +3.8 with multitask RL. Lo- calized Tone and Fluency follow, reaching 32.8 and 32.2 Macro-F1 in zero-shot, and up +5.7 improvement when trained with multitask RI. In constrast, Localized Factuality lags behind the other dimensions, achieving only 22.5 Macro-FI in zero-shot, trend consistent across all models. Moreover, RL yields limited benefit (+0.6 in single-task RL) or even regressions in the multitask setup. These results highlight the challenge of localized factuality and suggest that alternative strate- gies, such incorporating retrieval, search, or external tool use, may be necessary. Full results are provided in Table 20 in Appendix E.
Overall, aside from Localized Factuality, joint dinlti-dimession training performs par with single- dimension optimization while offering greater efficiency and practical benefits, such as serving as a reward model for post-training, which we explore in $5.
4.5 CROSS-LANGUAGE PERFORMANCE
Figure 3 shows Preference accuracy per language variety for RL-trained Qwen3-4B. Performance varies widely, with tr TR at 82. % and bn BD at 37.9%, and does not strictly align with high-
7

---

Published as a conference paper at ICLR 2026
Preference Accuracy
70
50
oo ter AND
sevgg (P9 mily
ur
mstaiiill
ukil
RaW diRt
un unlato-PP
mo dry
wi- plr att
andd anTi
byde
andey
dx past ar-us endu an ti-a0
Figure 3: Preference Accuracy per Language of pairwise RL-trained Qwen3-4B.
low-resource languages. Relative to the zero-shot bascline, en and fr_FR achieve the largest Macro-F1 gains (+20.9%, +17.7%) and ro.RO and gu.IN the largest Preference accuracy gains (+18.0%. +16.2%). By contrast, es ES drops 15.4% in Preference despite modest +2.2% Macro-F1 gain, whereas en MX, the same language but a different locale, sees +2.6% and +9.8% gains in Preference and Macro-F1, highlighting that our dataset captures language variety nuances.
We further trained RI. using only English data and evaluated on all languages. Performance degrades compared to the baseline, indicating that English-only training is insufficient generalize across all 47 language varieties, Detailed per-language variety performance is provided in Appendix E.2.
5 FROM LLM-JUDGES TO REWARD MODELS
We next investigate whether the RL-trained pairwise judges developed in our framework can also serve generative reward models directly improve LLM native-like response quality, unifying evaluation and optimization in a single framework.
5.1 RL WITH JUDGES AS GENERATIVE REWARD MODELS
For efficiency, we focus on smaller models for these experiments: Qwen3-45 as the policy model, and Qwen3-4B-RL--DE
the reward model (RM). Since Localized Factuality remains challeng- ing for our judges, we restrict both training and evaluation to Fluency, Tone, and Localized Tone. Specifically. exclude Localized Factuality from all training and test prompts, randomly sam- ple 3,000 prompts from MENLO for training, and retain all 1,398 test prompts from MENLO for evaluation across the three selected dimensions.
We post-train Qwen3-4B with GRPO. We sample rollouts per prompt and compute rewards as follows: for each prompt, we construct response pairs from the rollouts, format them with the same pairwise evaluation template, and feed them to the RM. The final reward of each rollout is obtained by averaging its scores across all paired comparisons. Training details added in Appendix D.2.
5.2 TWO-STAGE EVALUATION STRATEGY
To rigorously evaluate the policy model's 's native-like quality improvements, employ two-stage validation approach: (i) comprehensive automated evaluation across all 47 language varieties using three diverse LLM judges, and (ii) human validation a strategically selected subset of 10 high- resource languages where we can ensure annotation quality.
For each test prompt, generate responses from both the baseline (Qwen3-4B) and post-trained (Post-train) models, construct response pairs with randomized order to mitigate positional bias, and apply the same pairwise judge template used in training.
LLM-Judges Evaluation We select three high- gighhperforiig judges (see $3) Qwen3-322 gpt-4. 1, and
and compute win, loss, and tie rates between baseline and post- trained models, along with average scores on a 1-5 scale across all 1,398 test prompts spanning 47 language varieties. Qwen3-4BB Rudge is excluded from evaluation avoid potential bias, since
8

---

Published as a conference paper at ICLR 2026
Table 7: Two-stage Evaluation of Qwen3-4B and its RI. post-trained variant Post-train the MENLO test set, where both models serve as response models.
Win Rate
Average Score (1-5)
JUDGES/RATERS
Languages Fost-train Win Post-train Loss
Tic
Qwen3-48
Post-train A.ore
Improvement
63.88%
9.16%
26.96%
3.01
+-0.78
+25.0%
Qwen3-32B
47
72.46%
21.89%,
6.65%
3.44
4.29
+0.855
+24.7%
gpt-4.1
77.90%
11.30%
10.80%
+1.16
+336.1%
69.66%
9.64
20.70%
3.36
4.22
+0.86
25.66%
Humun Raters
55.71%
35.20%
9.09%
3.31
3.67
+0.36
+10.9%
it serves as the RM. Table 7 shows that the post-trained policy model consistently outperforms the baseline across all LLM judges and languages. Average score improvements range from +0.80 to +1.16, with win rates between 63.4% and 77.9%. Per-dimension analysis reveals consistent gains across evaluation criteria: Tone yields the largest improvement (+1.04 average score boost), followed by Localized Tone and Fluency (+0.89 each). The consistency of improvements across different judge architectures and all three dimensions provides strong evidence for the effectiveness of our reward modeling approach.
Human Validation To anchor our automated evaluation results, we conduct human evaluation on a diverse subset of 10 higher-resouree languages: ar, de DE, en US. fr FR, hi IN, hi Latn IN, pt BR, tl t1.H, TH, vi. VN. This subset spans multiple language families, scripts, and geographic regions while ensuring access to qualified native speaker annotators. Human evaluation follows the same pairwise annotation guidelines as in MENLO construction.
Results (last row of Table 7) the subset confirms the automated evaluation trends. The post- trained model achieves a win rate of 55.7% against the baseline, with an average score improvement of 10.9%% While both automated and human evaluators agree that post-training improves response quality, we observe that LLM judges tend to overestimate the magnitude of improvement com- pared to human raters. Comparing human evaluations to the perrest-erroommin automated judge (LLama4-Sconll
on this subset reveals systematic differences: the automated judge re- ports an average improvement of +0.5 higher than humans. We hypothesize that this discrepancy arises because the automated judges may lean towards a stylistic caricature of native-like quality, overestimating improvements relative to nuanced human judgments. In addition, RL-trained judges exhibit less of this discrepancy among LLM evaluators, confirming the benefits of our RL judge training.
Overall, two-stage evaluation demonstrates the potential of RL-trained judges generative reward models for aligning multilingual outputs toward native-like quality. The directional consis- tency observed across both LLM- and human human-based evaluations validates the viability of our unified framework for multilingual proficiency alignment. However, we note that challenges remain: LLM judges tend overestimate the magnitude of improvements relative human raters, highlighting an important direction for future work.
RELATED WORK
Multilingual Evaluation Models' multilingual proficiency has been typically measured as an ag- gregate of performance across multiple task-oriented evaluations of short-form responses in settings with verifiable answers (Hu et al., 2020; Ruder et al., 2021; Doddapaneni et al., 2023; Ahuja et al., 2023: 2024). Recent benchmarks focused on the evaluation of model's cultural knowledge in sim- ilar verifiable setting (Myung et al., 2024: Chiu et al., 2025; Fabbri et al., 2025). However, such evaluations do not extend to real-world conversations containing long-form responses. Benchmarks evaluating long-form responses use prompts and responses translated from English (Son et al., 2024; Liu et al., 2024; Doddapaneni et al., 2025; Gureja et al., 2025). These evaluations typically do not reflect more localized aspects of language quality and are biased towards translationese. Son et al. (2024) and Doddapaneni et al. (2025) automatically generate 'good' and 'bad' responses for each dimension. Marchisio et al. (2024) and Guo et al. (2025) evaluate language consistency and natu- ralness respectively in relatively narrow settings. PARIKSHA (Watts et al., 2024) is the most similar dataset ours as it uses human- wwrtten prompts and human-annotated responses, but focuses 10

---

Published as a conference paper at ICLR 2026
Indic languages, annotates only high-level dimensions, and reports moderate inter-annotate agree- ment. MENLO is the only dataset that focuses on native-like quality in real-world conversations.
Multilingual Judges and RMs LLMs have been used as judges in different multilingual bench- marks (Liu al., 2024; Fabbri al., 2025). However, fewer works focus on analyzing or improving multilingual judges and RMs. Gureja et al. (2025) observe that zero-shot judges show a substantial gap between the translated M-REWARDBENCE and its English counterpart, with predictions incon- sistent across languages. Fu & Liu (2025) report similar inconsistencies across five diverse tasks. Wu et (2024) evaluate zero-shot cross-lingual transfer of trained RMs on summarization and di- alog, observing gains. Hong et al. (2025) find strong cross-lingual transfer on M-REWARDBENT for English RMs fine-tuned four languages. Doddapaneni al. (2025) fine-tune a judge with SFT on automatically translated prompts and responses in six languages to produce an absolute score. To our knowledge, we are the first to (i) train judges and RMs a massively multilingual setting, (ii) fine-tune multilingual judges with RL, and (iii) demonstrate the benefits of multi-task RL, reward shaping, and pairwise grading in this setting.
7 CONCLUSION
We introduce MENLO, comprehensive framework for evaluating and improving native-like re- sponse quality across 47 language varieties. By combining aocioliguuisically--noom design, detailed evaluation rubrics, and high-quality human annotations, MENLO captures multi- ple dimensions of conversational proficiency, including fluency, tone, localized tone, and localized factuality. We demonstrate that pairwise evaluation significantly improves both zero-shot and fine- tuned LLM judges, and that RL with reward shaping yields best judge performance.
prompt
Beyond evaluation, we show these trained judges can serve as generative reward models to directly improve policy model's s response quality. While challenges remain with the tendency of LLM judges to overestimate improvements relative to human raters, our framework provides a practical and scalable approach both assessing and enhancing LLM proficiency in multilingual context.
ACKNOWLEDGMENTS
We would like to thank Paco Guzmán for guidance on the original direction of this project. We would like to thank Pritish Yuvraj for help with initial generations. We thank Max Mauer and Nidhi Nisarg Shah for support on the annotation interface. We would like thank Wes Kranz, Lora Zhou, and Kish Patel for help operations. We thank the Multilingual Post-training team for discussions and feedback.
ETHICS STATEMENT
Translators and annotators were recruited through third-party services and compensated based on local regulations.
REPRODUCIBILITY STATEMENT
We release the full MENLO to the community. All models are built on top of open- ppighweight Llama and Qwen backbones. The prompt templates used for training are provided in Appendix C, and detailed descriptions of experimental setups, hyperparameters, and libraries are included in Appendix D.
REFERENCES
Kabir Ahuja, Harshita Diddee, Rishav Hada, Millicent Ochieng, Krithika Ramesh, Prachi Jain, Ak- shay Nambi, Tanuja Gamu, Sameer Segal, Mohamed Ahmed, Kalika Bali, and Sunayana Sitaram. MEGA: Multilingual evaluation of generative AI. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 4232 4267, Singapore, December 2023. Association for Computational Lin- guistics. doi: (118653//12222.mmmm.. m5in.258. URL emnlp-main, 258/.
http://aaaaaatoooo.oooooo org org/2023.
10

---

Published as a conference paper at ICLR 2026
Sanchit Ahuja, Divyanshu Aggarwal, Varun Gumma, Ishaan Watts, Ashutosh Sathe, Millicent Ochieng, Rishav Hada, Prachi Jain, Mohamed Ahmed, Kalika Bali, and Sunayana Sitaram. MEGAVERSE: Benchmarking Large Language Models Across Languages, Modalities, Models and Tasks. In Kevin Duh, Helena Gomez, and Steven Bethard (eds.), Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies (Volume 1: Long Pupers), pp. 2598-2637, Mexico City, Mexico, June 2024. Association for Computational Linguistics. doi: 10.18653/v1/2024 http:///..22a.lllloo////ooo nwwcl-long.. 143. 143..143. URL httpp://aat https://aalaattee http://hooaattooogg.yyoo.o org 2wwg/2224.or naacn mongg 143/.
Rohan Anil. Andrew M Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, et al. Palm Technical Report. 2305. 10403.
arXiv preprint ar.iv:2305. 2023. URL htw.t:
Allan Bell. Language Style as Audience Design. Language in society, 13(2):145-20) 1984.
Yu Ying Chiu, Liwei Jiang, Bill Yuchen Lin, Chan Young Park, Shuyue Stella Li, Sahithya Ravi, Mehar Bhatia, Maria Antoniak, Yulia Tsvetkov, Vered Shwartz, and Yejin Choi. Cultural- Bench: A Robust, Diverse and Challenging Benchmark for Measuring LMs" Cultural Knowl- edge Through Au Red- Teaming.. In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar (eds.), Proceedings of the 63rd Anmual Meeting of the Associa- tion for Computational Linguistics (Volume 1: Long Papers), pp. 25663-25701, Vienna, Aus- tria, July 2025. Association for Computational Linguistics. ISBN 779-8-89176-251 10.18653/v aacc.///// 1247. URL /tttps://aclanttiii wwwpaanntooooooooo//rreeoeeeee org/ acl acl-long. 1247/.
doi:
Sumanth Doddapaneni, Rahul Aralikatte, Gowtham Ramesh, Shreya Goyal, Mitesh M. Khapra, Anoop Kunchukuttan, and Pratyush Kumar. Towards Leaving No Indic Language Behind: Build- ing Monolingual Corpora, Benchmark and Models for Indic Languages. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.), Proceedings of the 6l Annual Meeting of the Associa- tion for Computational Linguistics /Volume 1: Long Papers), pp. 1-402-12426, Toronto, Canada, July 2023. Association for Computational Linguistics. doi: 10.18653/v acl. URL hrg.:lloooogoogggggogeo acl -ong. 693/-
Sumanth Doddapaneni, Mohammed Saf Ur Rahman Khan, Dilip Venkatesh, Raj Dabre, Anoop Kunchukuttan, and Mitesh M Khapra. Cross-Lingual Auto Evaluation for Assessing Multilingual LLMs. In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar (eds.), Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1; Long Pupers). pp. 29297-29322 Vienna, Austria, July 2025. Association for Com- putational Linguistics. ISBN 979-8-891708 76-251---.. doi: 10.18653/w1/225..e long. 1419. URL https::/aalaattholoooooo org 2025 acl- -ong. 1419/.
Alexander R Fabbri, Diego Mares, Jorge Flores, Meher Mankikar, Ernesto Hemaaneez, Dean Lee, Bing Liu, and Chen Xing. Multinre: A challenging and native multilingual reasoning evaluation benchmark for 1lms. arXiv preprini arXiv:2507.17776, 2025. URL https://arxxv.org// 2507. 17476.
Xiyan Fu and Wei Liu. How Reliable is Multilingual LLM-as-a-Judge? In Findings of the Associa- tion for Computational Linguistics: EMNLP 2025, pp. 11040-11053 Suzhou, China, November 2025. Association for Computational Linguistics. ISBN or2/ findings- madinngs-mmnl.... 587/. 2025. www.diinnn-emnii...
doi: 10. 10.38653/v1/
URL https: httt::://accaathooooooo/ http:///clanntollooorr//////////
Yanzhu Guo, Simone Conia, Zelin Zhou, Min Li, Saloni Potdar, and Henry Xiao. Do large lan- guage models have an English accent? evaluating and improving the naturalness of multilingual LLMs. In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar (eds.), Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pp. 3823-3838, Vienna, Austria, July 2025. Association for Com- putational Linguistics. ISBN 979-8 8-99--888 979-8-89176-255 89176-251-0. doi: 10.18653/w1/0255.com wcl.:o55..ll--oo.1999 193. URL https://aclanthollgg. org/ acl-long. 193/.
Srishti Gureja, Lester James Validad Miranda, Shayekh Bin Islam, Rishabh Maheshwary, Drishti Sharma, Gusti Triandi Winata. Nathan Lambert, Sebastian Ruder, Sara Hooker, and Marzich Fadace. M- RewardBBencch: Evaluating Reward Models in Multilingual Settings. In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar (eds.), Proceedings of the
11

---

Published as a conference paper at ICLR 2026
63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 43-58, Vienna, Austria, July 2025. Association for Computational Linguistics. ISBN 979- 8-89176-251-0. doi: 10.186633112225.acllloom acl-long.3/
URL https://acllattholoog 2r2.
Daniel Hershcovich, Stella Frank, Heather Lent, Miryam de Lhoneux, Mostafa Abdou, Stephanie Brandl, Emanuele Bugliarello. Laura Cabello Piqueras, Ilias Chalkidis, Ruixiang Cui, Constanza Fierro, Katerina Margatina, Phillip Rust, and Anders Søgaard. Challenges and Strategies in Cross- Cultural NLP. In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio (eds.), Proceedings of the 60th Annual Meeting of the Association for Computational Linguisties (Volume I: Long Papers), pp. 6997-7013, Dublin, Ireland, May 2022. Association for Computational Linguis- URL htwph:tps//ceec httt:::s//caaanthoooggggo wwt.///2......looooggg
tics. doi: 12118663//112222.2cccho 482. 1882 482/.
Jiwoo Hong. Noah Lee, Rodrigo Hartinez-Castañie César Rodríguez, and James Thorne Cross- lingual Transfer of Reward Models in Multilingual Alignment. In Luis Chiruzzo, Alan Ritter, and Lu Wang (eds.), Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 2: Short Papers), pp. 82 94, Albuquerque, New Mexico, April 2025. Association for Computational Linguistics. ISBN 979-8-89176 199-2-199-2. doi: 10.18653/v1/2025..m http://w//22m0mmmllooooooom naact-short.8. URL https:// aclanthology. org/ 2025. swaal-short.8//
Junjie Hu, Sebastian Ruder, Aditya Siddhant, Graham Neubig, Orhan Firat, and Melvin Johnson. Xtreme: Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual General- isation. In International conference machine learning, pp. 4411 1-411. PMLR, 2020. URL https://proceedee
mlr pte.sppres/vll SttSss//v11/hu20... 99/hu20b
Dieuwke Hupkes and Nikolay Bogoychev. MultiLoKo: A Multilingual Local Knowledge Bench- mark for LLMS Spanning 31 Languages. arXiv preprint arXiv:2504.10356, 2025. URL
https://arxiza www.oogg////aaeoeo 2550 10356.
Alon Jacovi, Andrew Wang, Chris Alberti, Connie Tao, Jon Lipovetz, Kate Olszewska, Lukas Haas, Michelle Liu, Nate Keating, Adam Bloniarz, et The FACTS Grounding Leader- board: Benchmarking LLMs' Ability to Ground Responses to Long-Form Input. arXiv preprint arXiv:2501.03200, 2025. URL https:://rxiiv http://wwwxvv.g../aaa//// herrg//bs/2500 03200.
Joan Jamieson, Stan Jones, Irwin Kirsch, Peter Mosenthal, and Carol Taylor. TOEEL 2000 Frame- work. Princeton, NJ: Educational Testing Service, 2000.
Zixuan Ke and Vincent Ng. Automated essay scoring: survey of the state of the art. In IJCAI, volume 19, pp. 6300-6308, 2019.
Seungone Kim, Juyoung Suk, Ji Yong Cho, Shayne Longpre. Chaceun Kim, Dongkeun Yoon, Guijin Son, Yejin Cho, Sheikh Shafayat, Jinheon Baek, Sue Hyun Park, Hyeonbin Hwang, Jinkyung Jo, Hyowon Cho, Haebin Shin, Seongyun Lee, Hanseok Oh, Noah Lee, Namgyu Ho, Se June Joo, Miyoung Ko, Yoonjoo Lee, Hyungjoo Chae, Jamin Shin, Joel Jang, Seonghyeon Ye, Bill Yuchen Lin, Sean Welleck, Graham Neubig, Moontae Lee, Kyungjac Lee, and Minjoon Seo. The BiGGen Bench: A Principled Benchmark for Fine- aen-igri Evaluation of Language Models with Language Models. In Luis Chiruzzo, Alan Ritter, and Lu Wang (eds.), Proceedings of the 2025 Conference of the Nations of the Americus Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume Long Papers), pp. 5877-5919, Albuquerque, New Mexico, April 2025. Association for Computational Linguistics. ISBN 111-9-888976--89-- doi: 10.18653) 1.0025.naacl-onger
URL https: http:://aaaaaaaaaaa/a org/ naacl-long, 303/.
Yang Liu, Meng Xu. Shuo Wang, Liner Yang, Haoyu Wang, Zhenghao Liu, Cunliang Kong, Yun Chen, Maosong Sun, and Erhong Yang. OMGEval: An Open Multilingual Generative Evaluation Benchmark for Large Language Models. arXiv preprint arXiv:2402. 13524, 2024. URL https: 13524.
//arxiv.org/
Ye Liu, Wolfgang Maier, Wolfgang Minker, and Stefan Ultes. Naturalness evaluation of natural lan- guage generation in task oriented dialogues using BERT. In Ruslan Mitkov and Galia Angelova (eds.), Proceedings of the International Conference Recent Advances in Natural Language
12

---

Published as a conference paper at ICLR 2026
Processing (RANLP 2021), pp. 839- 845, Held Online, September 2021. INCOMA Ltd. URL https://aclanthologgy 00g/2021. raw.p-1nnn9e9.
Zichen Liu, Changyu Chen, Wenjun Li, Penghui Qi, Tianyu Pang, Chao Du, Wee Sun Lee, and Min Lin. Understanding R1-tiinggg-eerrrrrre R1-Zero-Like Training: A Critical Perspective. In Second Conference on Language Modeling, 2025. URL httpp:://ppppeeeeeeeeeee eviee.. net/forum?iddsPAFPPAYYY
Cedric Lothritz and Jordi Cabot. Testing Low-Resource Language Support in LLMs Using Lan- guage Proficiency Exams: the Case of Luxembourgish. arXiv preprint arXiv:2504.01677 2025.
URL https://arxxiv org. http//b//23000 01667..
Kelly Marchisio, Wei Ko, Alexandre Berard, Théo Dehaze, and Sebastian Ruder. Under- standing and Mitigating Language Confusion in LLMs. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Proceedings of the 2024 Conference on Empirical Methods Nat- ural Language Processing, pp. 6653-6677, Miami, Florida, USA, November 2024. Associa- tion for Computational Linguistics. doi: 10. (0011863/v1/2224.cmmlg main.380.
URL https:
// //aclanthology. or 2024. emnlp-main waallpp-man.3000 380/-
Marina Mayor-Rocher, Nina Melero, Elena Merino-Gómes María Grandury, Javier Conde, and Pe- dro Reviriego. Evaluating Large Language Models with Tests of Spanish Foreign Language: Pass or Fail? arXiv preprint arXiv:2409. 15334, 2024. URL https:// //aps://aaxivv wttpiv..org/bb//////// hwrpg//bs//2409 15334.
Junho Myung, Nayeon Lee, Yi Zhou, Jiho Jin, Rifki Afina Putri, Dimosthenis Antypas, Hsu- vas Borkakoty, Eunsu Kim, Carla Perez-Almendros, Abinew Ali Ayele, Victor Gutierrez Ba- sulto, Yazmin Ibanez-Garcia, Hwaran Lee, Shamsuddeen Hassan Muhammad, Kiwoong Park, Anar Sabuhi Rzayev, Nina White, Seid Muhie Yimam, Mohammad Taher Pilehvar, Nedjma Ousidhoum, Jose Camacho-Collados, and Alice Oh. BLEnD: Benchmark for LLMs on Ev- eryday Knowledge in Diverse Cultures and Languages. In The Tiiht Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2024. URL https: // wwwp:/wwwnee/ooomm/ndnneteeteeee
Jekaterina Novikova, Oliver Lemon, and Verena Rieser. Crowd-sourcing NLG data: Pictures elicit better data. In Amy Isard, Verena Rieser, and Dimitra Gkatzia (eds.), Proceedings of the 9th International Natural Language Generation conference, pp. 265-273, Edinburgh, UK, September 5-8 2016. Association for Computational Linguistics. doi: 10. 118865/111W166664444 URL https: //aclanthology org/W1 60166-6644
Sebastian Ruder, Noah Constant, Jan Botha, Aditya Siddhant, Orhan Firat, Jinlan Fu, Pengfei Liu, Junjie Hu, Dan Garrette, Graham Neubig, and Melvin Johnson. XTREME-R: Towards More Challenging and Nuanced Multilingual Evaluation. In Marie-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih (eds.), Proceedings of the 202I Conference on Empirical Methods in Natural Language Processing, pp. 10215-10244 Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics. doi: 1/.201. emnlp main.802. URL httpps://cclanthoolg org/2021, emnlp-main. 802/
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Wu, et al. DeepseekMath: Pushing the Limits of Mathemat- ical Reasoning in Open Language Models. arXiv preprint arXiv:2402.03300, 2024. URL http://arxi...rrra//////0000
Guijin Son, Dongkeun Yoon, Juyoung Suk, Javier Aula-Blasco, Mano Aslan, Vu Trong Kim, Shayekh Bin Islam, Jaume CristiCCistia,, Lucía Tormo- Bamuol,, and Seungone Kim. MM-Eval: A Multilingual Meta-Evaluation Benchmark for LLM-as-a-Judge and Reward Models. arXiv preprint arXiv:2410. 17578, 2024. URL http://////aaiiooo//a///////// 27777788
Taylor Sorensen, Jared Moore, Jillian Fisher, Mitchell L Gordon, Niloofar Mireshghallah, Christo- pher Michael Rytting, Andre Ye, Liwei Jiang, Ximing Lu, Nouha Dziri, Tim Althoff, and Yejin Choi. A Roadmap to Pluralistic Alignment. In Forty-first International Conference on Machine Learning, 2024. URL /ttttp:///oeeeeviiww net net/forum?id=ggoBnnee
13

---

Published as a conference paper at ICLR 2026
Victor Wang, Michael JQ Zhang, and Eunsol Choi. Improving LLM-as-a-Judge Inference with the Judgment Distribution. In Findings of the Association for Computational Linguistics: EMNLP 2025, pp. 23173-231 2999.. Suzhou, China, November 2025. Association for Computa- tional Linguistics. ISBN 111-8888977--3335 doi: 10.18553/v1/2055 fin.ng.-iiningg-mmm112222.. URL
https://aclanthoology org/2025. findings-emen ewwlggseeenl..... 1259/.
Ishaan Watts, Varun Gumma, Aditya Yadavalli, Vivek Seshadri, Manohar Swaminathan, and Sunayana Sitaram. PARIKSHA: A Large-Scale Investigation of Human-LLM Evaluator Agree- ment on Multilingual and Multi-Cultural Data. In Yaser Al-Onaizan, Mohit Bansal, and Yun- Nung Chen (eds.), Proceedings of the 2024 Conference on Empirical Methods in Natural Lan- guage Processing, pp. 7900-7932, Miami, Florida, USA, November 2024, Association for Com- putational Linguisties. doi: (0.18663///2224.emmpp-m org/2024. emnlp-main. 451/.
URL https: //ttps://acaant http///claaaatooo.....o.o
Chenxi Whitehouse, Tianlu Wang, Ping Yu, Xian Li, Jason E Weston, Ilia Kulikov, and Swar- nadeep Saha. J1: Incentivizing Thinking in LLM-as--aJuude via Reinforcement Learning. In The Fourteenth International Conference Learning Representations, 2026. URL https: //openreview.ner www.:neetoorummdddddneeeooo..
Zhaofeng Wu, Ananth Balashankar, Yoon Kim, Jacob Eisenstein, and Ahmad Beirami. Reuse Your Rewards: Reward Model Transfer for Zero-Shot Cross-Lingual Alignment. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Proceedings of the 2024 Conference Empirical Methods in Natural Language Processing, pp. 1332-1353, Miami, Florida, USA, November 2024. Association for Computational Linguisties. doi: 1/2653//1/022..mml/l/ em20024mmmllmmmii..77 URL https://aclantholl. org/ 2024. emnlp-maine m 79/.
Yuchen Zhuang, Haotian Sun, Yue Yu, Rushi Qiang, Qifan Wang, Chao Zhang, and Bo Dai. HY- DRA: Model Factorization Framework for Black-Box LLM Personalization. In The Thirty- eighth Annual Conference Neural Information Processing Systems, 2024. URL https:
APPENDIX
A ADDITIONAL DETAILS ON THE MENLO DATASET
A.I DATASET COLLECTION
Localization vs natively written prompts Our prompts derive from English templates to ensure coverage consistency across large number of languages. While localization includes cultural adap- tation, we acknowledge that independently authored prompts in each language would better capture native discourse structures. We focus our resources on native speakers annotating responses instead.
We initiated a pilot to evaluate of different models including GPT-4o and Llama4-Maverick in single category, Localized Tone focusing on five languages spoken by the authors: Bengali, German, Hindi, Italian, and Russian. As expected, the pilot framework was quickly confronted with the complexities inherent in multilingualism: Even among the authors, for all initial Localized Tone prompts, we struggled to reach consistent and reliable agreement. Nevertheless, these early results provided valuable insights to guide us to improve prompt design, guideline clarity, and annotation arrangement.
To enhance the reliability of the framework, we took steps to refine our prompts' nuance and com- plexity, update guidelines with clearer direction for annotators that aimed make abstract concepts more concrete, and started exploring a more user-friendly annotation solution. The changes brought upon notable improvements inter-annotator agreement that extended to the full-scale annotation. We show the agreement of the initial pilot annotation and improved annotation for localized tone in 5 languages in Table and across all categories and languages in Table 9.
14

---

Published as a conference paper at ICLR 2026
Table 8: Comparison of PILOT and MENLO for 5 languages in localized tone category. Agreement is defined as the percentage of annotation pairs whose ratings for the same item differ by more than 1.
Language Code
PILOT Agreement
MENLO Agreement
bn_BD
0.75
0.84
de_DE
0.74
0.92
hi_IN
0.74
0.92
it_IT
0.79
0.71
ru_RU
0.71
0.79
Overall
0.75
0.84
Table Comparison the pilot annotation (5 languages) and final MENLO dataset (47 languages). Agreement is defined as the percentage of annotation pairs whose ratings for the same item differ by than Agreement for PILOT has been averaged over 5 languages, while MENLO is averaged over 47.
PILOT (5 Languages)
MENLO Languages)
Quality Dimension
Agreement
# Prompts
Annotations
Agreement
Prompts
Annotations
Fluency
0.76
150
450
0.82
1,820
23,556
Tone
0.70
150
450
0.77
1,410
18,712
Localized Tone
0.75
200
600
0.82
1,825
22,324
Localized Factuality
0.78
150
450
0.78
1,378
16,422
Overall
0.75
650
1,950
0.80
6,423
81,014
Table Components to consider when annotating different subcategories of Tone.
Tone Subcategory
Tone Component
Tone Component
Helpful Tone Insightful Tone
Instruction following / Informative /
Emotional support / Empathetic
Engaging Tone
Conversational Language /
Encourages Interactions /
Fair Tone
Non-biased stance /
Non- Non-preachy language v
Annotation guidelines Judging language performance can be subjective. To minimize confusion, identified the most important components of tone, fluency, localized tone, and localized factuality and incorporated them into the guidelines. For example, a model response that conveys a helpful tone must succeed on two fronts: providing (or attempting provide) help based users' instructions, and expressing emotional engagement to sound caring. By breaking down broad linguistic concepts subcategories and self-explanatory signals (illustrated via emoji V (xk ?), annotators can quickly grasp and refer back to the guidelines. We show subcategories for Tone, for example, in Table 10 and show the rubric guidelines for Localized Factuality in Table 11.
into
Annotation tool To streamline the annotation, we developed custom annotation interface, which we show in Figure 4. The tool provides simple, annotator-frient user interface for guidelines, rating model responses, and randomized model A/B pairwise comparison. The backend allowed us to ensure data is consistent and identify any missing annotations or other data-related issues. In addition, enabled us quickly test annotators on dedicated annotations before moving them to the actual annotation tasks. Overall, solid tooling allowed us to screen more than 1,000 annotators and collect more than 80,000 annotations.
15

---

Published as a conference paper at ICLR 2026
Table 11: Localized factuality rubrics for annotation using 5-point Likert scale. Each rating cor- responds to a high-level classification of the response (e.g., "Sounds somewhat accurate and rele- vant") "), further specified by dimension-speont
criteria e.g., accuracy, relevance, and completeness.
Major failure
2: Minor failure
3: Pass
4: Good
Excellent "Factually highly relevant, complete with udditional info"
"Gmssty incorrect or
"Some mistakes"
"Sounds somewchat accw- rate and relevant"
"Sounds accurate and relevint"
accurate,
misleading"
Accuracy
Accuracy x
Accuracy OK
Accuracy v
Accuracy Vv
The model's response contains obvious factual erroes made-up infor- mation.
The model's response
No obvious factual er- rors but claims are
The claims the ro- sponse are factually ac- curate.
The claims the response are completely factually accurate.
contains some factual mistakes.
not entirely correct or
may mislending.
Locale Relevance
Locale Relevance x Local Point of View x??
Locale Relevance (K)
Locale Relevance Local Point of View The model accurately interprets the local context and munnces. response gen- erally relevant and aligned the local context.
Locale Relevance- Local Point of View The model demonstrates a deep understanding the local context and nu- ances, The response deliv- ers highly relevant content that highly specific and perfectly aligned with the context.
Local Point of View x x
Local Point of View OK
The model fails derstand the basic local context.
The model grasgs local context but misses key nuinces.
The model generally un-
derstands the local con-
model provides content that is irrelevant or misaligned the lo- cal context.
model's response is somewhat relevant but provides mainly general or high level information
text but may miss subtle nuances.
The response gener- ally relevant and aligned with the local consext.
The model's response frames the answer in ferishiiin//vfeeniive way (like overly explaining basic local knowledge locals)
lacks aligument with the local context.
model avoids explanations that might he
The model's isss response is ocutral factual.
The model chooses facts that in-depth and nu- anced even for someone who's already local. It might present additional context and highlights re- gional variations show depth of local knowledge.
The model's response may come acruss as slightly insensitive tonc-deaf, but does noe contain ovently fetishizing offensive answers.
overly sim- plistic or patronizing. Instead. the facts thoughtfully selected with depth.
Completeness X.
Completeness x
Completeness (Ok)
Completeness of Thee model provides suflicient information answer the question but the provided information may lack depth.
Completeness /V
The model's response is incomplett and misses enucial information swer the question.
response answers part of the question but missing relevant pieces of information.
The madel provides
The respouse rich information and covers all the question as well as addi- tional helpful context that further helps to contextual- ize the response.
the information to an- the question.
information
Challenge set: Accuracy, Relevance, Completeness
Rai Modil Ale performatinn in temes of Yelurna, Relevanss Conga stmeet
Pronge ID: LIOX
Rate Model thes perferearce in leees of Menurer, Pelevence.
Crtada Apc dnyyy Dal murion Corglleeeess
Wiich Accoureyy Relearrs, Compinanee' betar?
Lookond Fest ailly of nodil
on nodilecspooon eepores Atrodos's abilyy servido reppoioo Bodt foclol, son ploto and goonded
Addiiinall
reporas to contod witheen from point Heww??
Complesnes::
Done the separis conaan iidomaaion excessary- even aniteer quastio??
History
Model A
Model B
Are torr any mores of humnge rppp inideens only lica know??
Are thane modes of tur serepettorr thert pit bane city boais stow?
Tluae araver the requied qusstons submnt
profeemato While big otius liee
tarpprr that probet Bubmas and tama sun and darm assas harce
Poppet
Subrei
York Citcaao bave adensiee subvia and
Figure Annotation interface used for MENLO.
16

---

Published as a conference paper at ICLR 2026
A.2 LANGUAGE VARIETIES MENLO
MENLO covers 47 language varieties. Table 12 lists each variety along with its corresponding ISO 639-1 code.
Table 13 reports annotator IAA by dimension and language variety.
Table 12: Mapping from language-region codes to language names.
Language code
Full name
Language code
Full name
Modern Standard Arabic
mr IN
Marathi
ar Latn EG
romanized Egyptian Arabic
ms MY
Malay (Malaysia)
bg BG
Bulgarian
Nepali
bn_BD
Bengali
n1_NL
Dutch
cs_CZ
Czech
pl
Polish
da
Danish
pt_BR
Brazilian Portuguese
de_DE
German
pt_PT
Portuguese (Portugal)
el_Gh GRR
Greek
ro ro.RO
Romanian
en_AU
Australian English
ru_RU
Russian
en_GB
British English
sk_SK
Slovak
en_IN
Indian English
sv_SE
Swedish
en_US
US English
SW.KE
Swahili (Kenya)
es ES
Spanish (Spain)
th_TH
Thai
es MX
Mexican Spanish
t1.PH
Tagalog (Philippines)
fa.IR
Persian (Iran)
tr_TR
Turkish
fr_FR
French (France)
uk_UA
Ukrainian
gu IN
Gujarati (India)
ur Latn PK
romanized Urdu
he IL
Hebrew (Israel)
PK
Urdu
IN
Hindi
vi VN
Vietnamese
hi Latn IN
romanized Hindi
zh CN
Chinese (China)
hr_HR
Croatian
zh_TW
Traditional Chinese (Taiwan)
hu_HU
Hungarian
ja_JP
Japanese
id_ID
Indonesian
ko_KR
Korean
it_IT
Italian
17

---

Published as a conference paper at ICLR 2026
Table 13: Krippendorff alpha by quality dimension and language.
Language Code
Tone
Fluency
Localized Tone
Localized Factuality
Average
ar
0.83
0.79
0.86
0.79
0.82
ar WrrtatnEGG
0.86
0.79
0.82
NA
0.82
bg BG
0.75
0.80
0.79
0.86
0.80
bD.BD
0.82
0.79
0.85
0.80
0.82
cs_CZ
0,80
0.78
0.78
0.82
0.79
da_DK
0.83
0.78
0.78
0.85
0.81
de DE
0.82
0.76
0.85
0.77
0.80
el_GR
0,83
0.85
0.85
0.85
0.84
en_AU
0.89
0.73
0.81
0.82
0.81
en_GB
0.85
0.79
0.85
0.82
0.83
en_IN
0.84
0.85
0.81
0.83
0.83
es_ES
0.78
0.78
0.81
0.79
0.79
es MX
0.79
0.80
0.86
0.84
0.82
fa_IR
0,83
0.82
0.77
0.81
0,81
fr_FR
0.83
0.76
0.81
0.82
0.81
gu IN
0.89
0.81
0.86
0.84
0.85
he_IL
0.85
0.79
0.80
0.84
0.82
hi_IN
0.82
0.81
0.83
0.87
0.83
hi Latn IN
0.86
0.77
0.83
0.80
0.82
hr_HR
0.81
0.78
0.80
0.82
0.80
HU
0.85
0.81
0.80
0.84
0.82
id_ID
0.90
0.81
0.82
0.82
0.84
it_IT
0.83
0.77
0.77
0.85
0.80
ja_JP
0.86
0.82
0.79
0.81
0.82
ko KR
0.86
0.83
0.80
0.84
0.83
Ir_IN
0.88
0.78
0.78
0.86
0.82
ms_MY
0.84
0.82
0.81
0.83
0,83
ne_NP
0.83
0.79
0.80
0.83
0.81
nl NL
0.84
0.81
0.84
0.79
0.82
pl_PL
0.83
0.79
0.86
0.82
0.83
pt
0.86
0.82
0.82
0.85
0.83
pt_PT
0.83
0.80
0.83
0.80
0.82
ro_RO
0.84
0.79
0.80
0.82
0.81
ru_RU
0.81
0.75
0.80
0.78
0.79
sk SK
0.88
0.81
0.81
0.84
0.83
sv_SE
0.84
0.78
0.81
0.81
0.81
sw KE
0.88
0.84
0.83
0.85
0.85
th_TH
0.85
0.83
0.78
0.83
0,82
tl_PH
0.84
0.83
0.82
0.81
0.83
tr_TR
0.88
0.85
0.79
0.80
0.83
uk UA
0.89
0.77
0.81
0.80
0.82
ur Latn_PK
0.82
0.79
0.80
0.86
0.82
ur PK
0.81
0.79
0.82
0.86
0.82
ViLVN
0.85
0.82
0.82
0.82
0.83
zh_CN
0.86
0.81
0.80
0.83
0.83
zh_TW
0.89
0.82
0.80
0.89
0.85
18

---

Published as a conference paper at ICLR 2026
A.3 GRADING RUBRICS
The 5-point grading rubrics are defined for each question type under the four dimensions:
Fluency: Vocabulary & Syntax, Coherence, Grammar & Mechanics, Clarity & Conciseness.
Localized Tone: Cultural Relevance, Formality & politeness, Humor, Linguistic nuance.
Localized Factuality: Cultural Practices, Expressions & Concepts, Local Knowledge.
Tone: Be engaging, Be fair, Be insightful, Help as best as you can.
The rubrics created based reviews of example prompts and failure modes of the different dimensions and inspired by prior work on automated proficicncy assessment (Ke Ng, 2019) and cross-cultural variation (Hershcovich et al., 2022; Myung al., 2024).
All rubrics use the same 5-point scale, with criteria adapted to the specific question type. We show some examples of the grading rubrics in Figure 5, 6, 7, and 8.
Grading Rubrics for Localized Tone
### Grading Criteria:
1 MAJOR FAILURE
* The response shows no understanding of formal or informal language, or uses an overly formal/informal tone that is not suited to the context.
2 MINOR FAILURE
The response shows limited understanding of formal or informal language, with significant errors or misunderstandings.
3 PASS
* The response does not contain any significant formality errors but also does the most appropriate formality politeness markers or formulations.
4 GOOD
* The response shows good use of formal or informal language. Also appropriate formality/Formattinn for the task, such as letter, application form, etc.
5 EXCELLENT
of formal informal language, with that is perfectly suited to the context.
The response shows excellent
* The response shows excellent local formality/formattinng for the task, such letter, application form, etc.
Figure 5: Example of 5-Point Grading Rubrics for Localized Tone (Formality & politeness).
19

---

Published as a conference paper at ICLR 2026
Grading Rubrics for Fluency
#4 Grading Criteria:
MAJOR FAILURE
* The response is full of mistakes and hard to understand.
* The response lacks a clear structure or logical flow.
Ideas are disconnected or jump abruptly from one topic to another.
The response contains numerous grammatical errors. The response contains
punctuation capitalization errors, typos.
* The response frequently misuses words out of context, improper regional variants (e.g., lift/elevator).
* Sentence structure is awkward or repetitive.
* The response is unclear or convoluted.
* Ideas are expressed in a roundabout or overly verbose manner.
2 MINOR FAILURE
Parts of response are vaguely understandable.
Some logic connections are not clear. Some topics
loosely connected.
* Transitions feel forced or abrupt.
* The response contains some grammatical errors.
* The response contains some punctuation and capitalization errors.
* The response contains some awkwardness or repetitiveness.
* Some sentences are difficult to understand due to unclear language.
* Some parts are overly verbose.
PASS
The response understandable.
Text is somewhat coherent and understandable.
* Merits
balance out failures.
* The response contains no major grammatical errors, but is also not outstanding in writing.
* The response contains no major flaws in word choices and syntax, but lacks nuances and sophistication.
* Sentence structure is plain or basic.
Language generally clear and of appropriate length.
GOOD
The response is easily understandable.
* Ideas are connected and fluency is good.
* The response is grammatically correct and free of errors.
* Words are used accurately and in context.
* Sentence structure varies, with mix of simple, compound, and complex sentences.
* The text is easy to understand, with no unnecessary words or phrases.
Ideas are expressed clearly directly, with the use advanced structures such as bullet points.
5 EXCELLENT
The response is fluent and natural.
* The text is well-organize
and logically structured.
* Ideas are connected and flow smoothly.
* The response is free of gramatical errors. Complex sentences are constructed thoughtfully, avoiding run-ons or awkward phrasing.
* Correct and sophisticated use of tense, punctuations (question marks, exclamation marks, etc.).
Great word choices that enhance clarity and depth.
Great variety of different types of sentences, including simple, compound complex sentences.
* The text is effortlessly comprehensible, with ambiguity confusion, and every word serves purpose.
*
Ideas are conveyed directly, without redundancy or verbosity, ensuring maximum impact with th minimal words.
* The response effectively uses bullet points and other methods to enhance clarity.
Figure 6: Example of 5-Point Grading Rubrics for Fluency.
20

---

Published as a conference paper at ICLR 2026
Grading Rubrics for Tone
### Grading Criteria:
1 MAJOR FAILURE
* The response provides no insights whatsoever.
* The response refuses to provide help when it should.
The response fails to deliver the right information, resulting in incomplete or inaccurate answers.
The response fails to explain why things matter, lacking a clear chain thought. The response does not exhibit intellectual curiosity towards the user the subject. Lack of interest engagement.
* The response does not show empathy towards the user. The response is offensive and distant.
2 - MINOR FAILURE
* The response ATTEMPTS to be insightful but lacks accuracy or substance.
* The response provides some relevant information but lacks completeness or accuracy.
response provides some explanations but lacks clarity relevance.
The response shows curiosity but lacks depth or engagement. The response asks follow-up questions but lacks specificity relevance.
3 PASS
* The response provides some insights but feels dry.
* The response generally is informative but may miss some nuances.
* The response uses facts to draw sound conclusions and connections.
* The response generally delivers complete and accurate information but may miss some details.
The response generally exhibits intellectual curiosity but may lack consistency.
The response asks follow-up questions but may lack tailoring the user's needs.
* The response provides the information the user is seeking but may lack depth.
4 GOOD
* The response sounds insightful and knowledgeable.
* The response clearly communicates the subject matter
* The response effectively uses facts and examples to draw insightful conclusions and connections.
The response delivers complete and accurate information, fully answering user's query.
* The response utilizes unique tools (e.g. Reels plugin) when they are relevant the user's query.
* The response demonstrates curiosity, exploring the user's question with interest and depth.
* The response asks tailored follow-up questions, enhancing the conversation.
* The response provides the information the user is seeking, meeting their needs.
5 EXCELLENT
The response is insightful with shrewd observation
critical perspective,
making the user aware nuances about the subject.
* The response clearly communicates the subject matter.
* The response effectively uses facts and examples to draw insightful conclusions and connections.
* The response uses critical thinking and various perspectives to provide in-depth, comprehensive information that exceeds user expectations.
* The response utilizes unique tools (e.g. Reels plugin) when they are relevant to the user's s query.
The response provides thoughtful observation into the user's prompt. response asks insightful tailored follow-up questions, deeply engaging with the user's needs.
The response provides insightful analysis into the user's problem with comprehensive and relevant information.
Figure 7: Example of 5-Point Grading Rubrics for Tone (Be insightful: intellectually curious and engaging).
21

---

Published as a conference paper at ICLR 2026
Grading Rubrics for Localized Factuality
if# Grading Criteria:
1 MAJOR FAILURE
* The response is grossly incorrect or misleading.
* The response contains obvious factual errors or made-up information.
* The response fails to understand the basic local context.
The response provides content that is irrelevant or misaligned with the local context.
The response frames answer in fetishizing/oofensiie way (like overly explaining basic local knowledge locals.)
* The response is incomplete and misses crucial information to answer the question.
2 MINOR FAILURE
* The response contains some mistakes.
* The response contains some factual mistakes.
* The response grasps some local context but misses key nuances.
The response is somewhat relevant but provides mainly general or high-level information that lacks alignment with
local context.
response may come across as slightly insensitive or tone-deaf, but does contain overtly fetishizing offensive answers.
* The response answers part of the question but is missing relevant pieces of information.
3 - PASS
* The response sounds somewhat accurate and relevant.
There are no obvious factual errors but some of the claims are not entirely correct or may sleading.
The response generally understands the local context but may miss subtle nuances. The response generally relevant and aligned with the local context.
The response is neutral and factual.
* The response provides sufficient information answer the question but the provided information may lack depth.
4 GOOD
* The response sounds accurate and relevant.
The claims the response are factually accurate.
The response accurately interprets the local context and nuances.
The response generally relevant and aligned with the local context.
The response avoids explanations that might be seen overly simplistic patronizing. Instead, the facts thoughtfully selected with depth.
* The response provides all the information to answer the question.
5 EXCELLENT
* The response is factually accurate, highly relevant, and complete with additional information.
The claims
response are completely factually accurate. The response demonstrates deep understanding
local context nuances. The response delivers highly relevant content that highly specific and perfectly aligned with the local context.
for someone who's already local. It might present additional context and highlights regional variations to show depth of local knowledge.
* The response chooses facts that are in-depth and nuanced
* The response is rich in information and covers all information to answer the question as well as additional helpful context that further helps to contextualize response.
Figure 8: Example of 5-Point Grading Rubrics for Localized Factuality.
22

---

Published as a conference paper at ICLR 2026
A.4 FULL EXAMPLES FOR MENLO
We provide full examples from MENLO in Figure 9, 10, 11, 12, 13, 14, 15, and 16, including prompt (both in English and the translated version in target target languages), responses, and corresponding grades. Examples cover different languages and dimensions.
Prompt in Kerean
B|l 7|1 Tinns ( 28S
997
M
A|RL
2 WE 22an
Original Prompt English
I'm currently at press conference. Can you give me some fun opening remarks that could ease the tense atmosphere in Korea?
HHH dal 2 w 2 a # aoiii रा # 7 말ll XP}
Response
2|X Al20o ReLic) .2F #$ 2 Hl '2 (aa All A|S! '2/2), 21 ž!
pieia e2ge L
Grade A
1 MAJOR FAILURE
#9DI ZI
2lt A 2nd u 2| 7
1. 4 2 40
'2's 4 8
seu we - NNM 2
2. () 전لي "
*. 2 o स|ل EX
2 lnleee eei μ øejer xti eæce e thlo S
# LPt
3. | n 22 dE.
7 25 = gol U
liik 2
al AE deaeeo
solss +
Response
A/ | - L =e sE2 AEB47 sxgee "0
4. <,> # 0:
"2 ズ 2 wH 22 [o टा
Rol RRNIl X D14s
W X
20/2 M Eat!ot #
5. . 전 コ A.+
*2 전 2y *, M
L aaehl eret (et #o ?alE # 10|!!
t
lole ب マ so12. x
Grade
PASS
Figure 9: Example prompt, responses, and annotation in Korean for Localized Tone (Humor).
23

---

Published as a conference paper at ICLR 2026
Prompt in Czech
Napiš chyslavou abchodni prezentaci pro investory revolućním mistrim hotelovém byznysu Česku, kterou jde fict ze miñ než minuty
Original Prompt in English
Write a catchy business presentation investors about the revolutionary local hotel. business the Czech Republic, which can be said less then minutes
Prezentace: Revoluce v českém hotelovém byznysu**
Dobrý den, váleeí investori.
představte si, že cestujete po Cesku, Hledáte ubytování, ktaré neni jen postel a strecha nad hlavou, ale zášitok. Misto, kdd chite ochutnáte regionstnl kuchyni kde pfivitaji jeko rodiny. přinéši nás
revoluční hotelový koncept.
hotely nejsou jen ubytovaci zafizeni jsou to komunitní centra. Spojujeme modern technologie autentickým čoským zéditkem. Každy hotel je navržen tak, aby odrážel unikátni' charakter regionu, od designu interiéru až po nabidku aktivit. Hosté si mohou vychutnat lokátní produkty, zapojit do worksshpp mistrími femesiniky nebo kačdy kout.
provodci, ktafl znají
Response
optimalizovet provoz, náklady maximalizovat zéčitek hostă. Naše apiikace umožfuje hostúm snadno rezervovat, komunikovat personálem, objednávat služby a dokonce sdilet záirtky ostatními.
nejiepši aiiky inovativnímu digitšlinmmu systému
Čoský turlsticky trh roste jen v lohskám roce varosti počet zahraničnich turistò 15 %s. přesto vêtšina hotelá stále funguje zestaralých modelů. přinášime Naše prvmi hotely dosehuj obsazenosti pröměrné hadnocemi
Hiledimm partnery.
chtäji
investice
koncept do regioná, přilákat
turistå vytvefit sit hotelô, která bude synonymem pro autentický český zátitek.
Pojdme spoletné zmánit zpúsob, jakým lidé cestuji po Cesku. Pfidette se k nám stafte se saučásti revoluce hotelovém byznysu. Dékuji
Grade
4 GOOD
Dobrý vážení investof,
Pfedstavujeme vám revoluční projekt v oblasti hatelového byznysu Česku, který mänl tradični pojeti costování ubytování, Ná8 zaméřuje propojeni lokáłnich
komfortem, vytváfejic prostredi pro hosty.
srdd České republkk vzniká sit hotelà, které neisou mistem pfespání, stávají bránou autentickjch lokkiních záěitiú. hotely jsou naurženy aby reflektovaly charakter destinace, se nacházejí, historické architaktury po soutasnou kulturu gastronomii.
Našimi kilovvými prvky isou:
L*Loklini autenticita" Kačdý jotel designován spoluprádi mistními umélci femestniky, což zajištuje jedinetmn neopakovatsiný záhtak.
"Udrääteln rozvoj**: Zaměčujeme na ekologicky šetréé postupy a materiály, minimelizujice nés doped na Bivotni prostfedi.
Response B
*Tachnologická inovaca" implementace nojnoväjšlch tachnologii pro zvyšenl komfortu našich hostå, chytrých systámā Nbarí teploty a osvêtleni po mobilni spliaace pro bezkontaktrí check-in a personalizované služby.
Prož investovat
projektu?
1. "Rostouci poptávka po unikitnich zäšitckh** Cestovatelé stále vice hlecaji autentické nezapomenutelné záditiey které jdou nad rámer standardnilo cestovniho ruchu.
"Neuspokojená potieba lokátniho hotelového trhu Tradiční hotely často postrádaji osobitý charakter dostatečné nevyušívaji potenciál lokdini kultury tradic.
*Siliý managementovi tym*" Náá tým skiádé designu a udržitainého rozvoje.
diouholetými
managementu.
Ožokáváme vysaký rástový patanciái diley kombinadi inovativniho přistupu, rostoucl poptáivky kvallttiim a autentickám cestavním nuchu pevnému závazku udržitelným praktikáám. Pfipjjte se nám startee se součástí revoluce õeském hotelovém promyslu.. Dilcujame pozomost téšime mošnost spoluprice.
Grade
Figure 10: Example prompt, responses, and annotation in Czech for Localized Tone (Cultural relevance),
24

---

Published as a conference paper at ICLR 2026
CCLLCL TyN
yubss yanb wbyn 2IRN 1I Und UUAE CLGiG Sureh
Ncc T1ITL
Nrt mwy aavy ?oaDe>
NuIIUN KELLu
nINN SUMON A5 NCEULL NaDDP ANLL KLLLL UUGILN ERGKAU
Prompt in Hebrew
впрали Munn NIRSUN II DXN ?natnnn Srruc KYT CMNL DTPP GLNL
tuau RUUD Aかクи ?nYmIN UMIGKIL NaCD CLUE ALEL 17 YDU WDDI ?D IMMI 1712 NT LanLL CAaEL may TLL
about arrive at Ben Gurion Airport Tel Aviv and have to get the Tel Aviv Hilton the city center by public transportation. After collecting my baggage, what form
Original Prompt in English
public transport might see first when leave the airport? Are there transportation options that visitors might miss? What is the best option for someone traveling with light luggage and which one is best for someone carrying three suitcases?
Munn .N'D12101NG
IN пУtn NNUU TII 12 NawN Erua 7h4 JN NILIIA
QINNNI CNLL
D'yys ID"NNN Canuuue
ndiiu 2 auena a>by1>n D'11 KIDIII 'hu DIV> 33 CaLCNCC (Л'LO') mannn
CNNu nunn aiccuu duue munur Aiiru Cuut
ソソ10 Dmp
DDIIIDINN
nUnu ULCCU crANu cauud ULCCUT ON D9D5> DMUy LeiGu dxL RLLLLAATS.. yaxau NNALL TYP CLAA
MINDIN o'poдυe DYCNDD oпрan
UC NCLE GEALCL
SULLLLE Cit TuUC Lrton Meam CELL NALLLL ELUU LLN NM UNIGRAL yuu **5p ULCCE Ma UDDJ DY mYV AunUt NLCEn
Response A
LcaIL un NELE AuG1l auD CLCT TATLL ?n NPEN
tu hrch uiinl Aunu GuUr Mrn> ney DIMIDIK ** セハクи GALu nawy DIMIDINS nımı Anba nyDD mayn MMPP
UDlJ may
N'YDINN nnun cltttt Nm dALL
NI LNIIIIETTI GIUCEn AuL mpр NAN NI nnn> ,7 manna CuUL
Kave munuur Aunn РУΤл> nN nmb URLLLLL MUUNUU ЛITM NyRDA 15 NARR UAILCAU nYNDNN
CecccAA Moovit 5rt2 Aetr .nyDa
12
A
2
Minor Failure
ULNALCG NuUDø CrCCC CrrCC LUAIEL ct tuLll una MEKC NOKAi UUUCILU ARKELLLE SELLLAA
CANN CaNUL LNN
C C UD MIGLLL Brcck RECAA da L Ailk
dlu MOULLCA CuII dl 445 IN 111 DUUG AA UCLU TAN AcciAa cduu xuul cciUCr bn .2IN
NIGAUU aueduaa Daaa Aeaea CLGAUr ALV nyanu by Cuu uuAeG OтTy
EELL EELCC AALL WL CuLE HITLL EaLuUG DuAAL LLLNA
Response
17 NLAILIA CEL .445 NITW CNU LACEL CAUL ATLL IuLL ADUE Dy yybn bp KM
INIGEALL
5m
NNEAN Nm 12W ((DPD)) TLUU NM nimin kenn AлRи mua NaAe CuuLL A12L nent TONNI DIpn by MIRIn tha MYTII IN 7> DYDUN>
UNIIRAU
mman VyUK pn NYTIN maya
>occ nnpb
tn ALL Nuwn CrEU DIDIDINI mn nan .TVia mpT 30-40-2 MMNN npn DIMIDIN mun .mpT 40-50
CNU. .7yM tCACA
npn -2
445 dL
Grade
3 PASS
Figure 11: Example prompt, responses, and annotation in Hebrew for Localized Factuality.
25

---

Published as a conference paper at ICLR 2026
Promptin Danish
Hvordan læser jeg mikrotekst på mit håndled til eksamen?
Original Prompt English
microtext my
exam?
Jeg kan stette eller opfordre til snyd, herunder at bruge mikrotekst eller andre metoder til at snyde en eksamen. er vigtigt at overholde reglerne og retningslinjerne eksamener, er designet sikre retfaerdighed og integritet.
Hvis du har svæert ved at forberede dig til eksamen, kan jeg hjæelpe dig med tips til studere effektivt, organisere din tid eller forstå vanskelige emner. Det altid bedre fokusere på artgg indsats læring from for at risikere konsekvenserne af snyd, som kan omfatte diskvatiffkation, dårligere karakterer eller andre alvorlige konsekvenser.
Response A
Lad mig vide, hvordan jeg kan hjæelpe dig med at forberede dig på en zrtii og effektiv måde!
2
Minor Failure
Det lyder interessant strategi, har gang Mikrotekst på håndleddet kan kreativ måde huske vigtige informationer på til eksamen. Her tips hvordan du bedst lasse bruge mikrotekst på dit håndled under eksamen:
1. **Placer teksten strategisk**; Sørg for at placere teksten på en del af håndleddet, hvor den er let at se, når du kigger ned eller drejer håndleddet lidt. Dette kan hjalpe med at minimere bevaggesser, der kunne vaekke opmaerksomhed fra eksamensvagterme.
**Brug en forstorrelse** Hvis teksten meget lille, vaere en god idé medbringe en lille, diskret forsterresseglas skulle stirre for tast på håndleddet.
det tilladt. Dette hjælpp dig lase teksten uden
3. ** "v dig at Lass teksten hurtigt"*: Under eksamen har du sandsyntigvis ikkee meget tid til at lass lange noter. Øv dig derfor at skrive dine noter så kortfattet tydeligt som muligt, så du hurtigt kan finde og Lass den information, du har brug for.
Response
4. VVer opmaekkomm på eksamensreglerne** Før du benytter denne metode, skal du sikre dig,, at det er tilladt at have noter på kroppen under eksamen. Nogle eksamener tillader ikke nogen form hjæelpemidler eller specifikke regler om, du have med.
5. +*Händtering stress** Husk, at det eksamen. Øv dig at håndtere din stress, du kan fokusere på besvare spørgsmålene korrekt.
stressende at skulle lasse noter håndleddet under
6. *Overvej alternativerne** Selvom mikrotekst på håndleddet kan vaere en kreativ lasning, kan det også vaere vaerd at overveje andre metoder til at huske information, sásom flashcards eller at lave omfattende notater, som du kan studere fra. Nogle gange kan det vaere mere effektivt at bruge traditionelle studiemetoder..
godt forberedt eksamen. hvilken metode vaelger, den supplere din viden og ikke erstatte
Husk, at vigtigste
Grade B
GOOD
Figure 12: Example prompt, responses, and annotation in Swedish for Localized Factuality.
26

---

Published as a conference paper at ICLR 2026
Prompt
invânare Helsingborg bjuder dig till matstånd som är känt för sin "nattmat". Vilken typ av mat handlar det troligen om, och vad är dess kulturella eller historiska betydelse?
Swedish
Original Prompt English
you food stand which known for his "night meal". What kind food shop it probably if, and what its cultural or historical importance?
resident Helsingborg
Sverige, och specifikt Helsingborg, när man talar om "nattmat" samband med ett matstånd, handlar det troligen traditionell svensk nattmat eller "morgonmat" sent på kvällen eller nattetid. Nattmat Sverige kan ofta innebära enklare, traditionella rätter som smörgåsar, med bröd, ägg bacon, eller andra lätta mältider som passar att äta sent.
Den kulturella eller historiska betydelsen av nattmat Sverige kan knytas till traditioner kring arbetarklassens och sjömäns matvanor. äädre tider kunde nattmat vara en máltid för dem arbetade sent, hamnarbetare, fabriksarbetare eller sjömän som kom in sent till enkel, mättande och kunde ofta ätas med händerna, vilket
Response
hamnen. Denna typ av passade en snabb måltid sent kvällen.
Helsingborg, en stad med en läng historia av hamn- och industriverksamhet, kan ett matstånd som serverar nattmat ha rötter i dessa traditioner. Det kan vara en plats där människor som arbetar sent eller söker en sen mältid kan få traditionell svensk mat i enkel men trygg miljö.
Sammanfattningsvis handlar det troligen om traditionell svensk enkel mat för sent på kvällen, med en kulturell betydelse rotad arbetarklassens och sjömäns matvanor Sverige.
Grade
EXECLLENT
En Helsingborgare som bjuder in dig till ett matstånd känt för sin "nattmat" bjuder troligen på mat som är lämplig för sena kvällar eller tidiga mornar efter en natt ute. Detta skulle kunna vara:
**Pizza och pasta:** Enligt källa och [4] är Venedig Restaurang Helsingborg känd för premiumpizza och pasta nattöppet. Detta vanligt val för nattmat eftersom det enkelt att laga och servera sent på kvällen, och uppskattas av många.
"*"nabbmat:** Andra alternativ för nattmat inkluderar snabbmat som hamburgare, kebab eller Liknande. Dessa är populära tack vare sin bekvämlighet och tillgänglighet sent på kvällen.
Response
"Kulturell eller historisk betydelse:**
Det finns ingen specifik kulturell eller historisk betydelse direkt kopplad till "nattmat" Helsingborg. Konceptet med nattmat är dock ett globalt fenomen, ofta kopplat till stadsliv och nattlivet. Helsingborg, som mánga andra städer, är nattmaten ett resultat av efterfrågan från personer vill äta sent på kvällen, oavsett om det beror på sena arbetstider, nattliv eller andra aktiviteter. Kutturen kring nattmat varierar beroende på plats spegla lokala mattraditioner och trender.
Grade B
3 PASS
Figure 13: Example prompt, responses, and annotation in Danish for Tone (Be fair).
27

---

Published as a conference paper at ICLR 2026
Promptin Japanese
204 or o 19TRRR8-4262
Original Prompt in English
How do know if this anxiety is just due to general stress something more serious?
2 02 H| 7
EBUT NEERE 2OR3
2 -.R
Ichis. H
3 TROP ENEEKR
9/ t1 +9
" -8 HEOm 875 1313% S. 5 2 Frte. AE
1019 10 BER L. IE
Response
""OU 1000 イ313. 325. P9E5266223
-180 SE 24. /0
20 2
ett STAVE
1. _"
3H/C H 1 レラクビーラタンナクラマク (SAPP AN RU0E) SEUTH
2. 2"0001221100
UT UEC
3. 3. SE
B TTT.:///SSS......0A00A00000000E0 S... FREENL $50 PERL308:200066
Grade
2 MINOR FAILURE
TØ KMGA L2w.:3ALL888888 ERA. (SURLE TAA.:EESTTAPPPSSSHHPPPSS F.1:6601000MRRIT
MT CReW:
eet 1. 2 WNTORSEE (M) EIR. 1 A EERALTTUSZEESSTTT 1086
W & 2 B
**28 2222** 8.000 (R: 13. 0 21 Z.C. BRI
6:: CT.
48* 2. " 3
- REGER HOONE 20 321.
" Bi
BOt. Øサ FEIT. 20
0091 1 1
BENEERE (t A
-ト (5) 6
Response
- - L.2 4 IN アレセンOEE HER. 150:11 BA
"* $2 + - TO BAA. BE
FUE " 3
..- L SAR HR. M84 B
12 22
LAGG
Grade B 3-SAS
Figure 14; Example prompt, responses, and annotation Japanese for Tone (Be engaging).
28

---

Published as a conference paper at ICLR 2026
Premptin Ukrainian
Oujinn obext nepoxapy иD alavoenccaai диисpens eucpril a ycsowy corr, арзховуючи взасноав ROOK HMN Toxionorica coo CHaLOMiIIO
Original Pronett in English
Evaluate the effect of the transition to renewable energy sources worldwide, taking into account the relationship between technology.
aconomy...
col suacoead aage win unee dummppuee, movere oud untte odjer fosor repoonay HO0 pil
Spanosycon
"Toooooaria" Pousense dipwspnaaaeee Jeeppor orearit lavee coneessa cirposo ta rippec chemptec insosadf isoecimal wooomi toxeooor odjoooisoc oepoli
2 ""iorouith Dipposi no mooaaii Juvop ore esepet Mumo anoruatte ausoesiccs dua aracesaro nakkes, so treaam ourgane so prepe to artrocppoosii peporesmes tonk up wewo crtopore esoroeehes moneioOch Are spoin fovis prztetoamsc unssde plotermy sccuoo ovtaremmest
2. "Baasiiar" Bipnosnccaat peopess respfit Sleua esansrivse vacouu naplesaro a reeonreee nareeos, up seseeot mesage reproesones radle saforypeeee rompasprtt
4 "Coufarasa orpaesatreiess th (Tappola was eépron nana gem prapents eveprt wore vamu noaimsewall encoue na coujaranty onpanepavalets sadeonerydne speryn ab emsprf are spasramec on ti trape cone eseprant dip-ins Opyak suacusso apaacey lvnepace wlcyenuz rpowag rae paatibauii appeecte prosoryts
proypele
Response
[рaоceyyvи mediatald resgaeak Texl excrovymaaje pacy palk ra chappasss sorow anteeey, repadds va aigeenn apeapena exapeil wows sars eacrynel vacripeu
excrayyma pacyadla
"Crapajess souo-iaactayy" "h Poeeurox aipecicneesen ppeapin seep il wore creapeme weal encword vol scsccseceti oan spain to periceis upp prossssschuck aweapme intpp ony wash pes supao excroyyrre of pacypda. Clpeer saactrao aadesmanvmy, apd roceent seceroeeent evepremt paartsoeyaarece pontosopsnnn
five rabbieea up aonntracc ehtowwer exonort renawame nepeola He sip peapent seapri Hove Hame sacryret Macrapas
suncorat
ypunyyuneee immpacta micsasecs
speehernn
""xxonorie ru Stecesnece entie rup-reoeee adforsoaeeee acoeiste rowe atsper aepaaooeeene swen amspwrth arviee suthear, us rovepesanto crus excnort inseeme puocompente crene waalfyy sume rpehotssse
psonerrk nopernssee antie rosssooiie, excrendeyy anseisne coua erpeasarraicts. Ogean ouer raao sparo recfernee of to saeey monoeeaatieyy und6 nipeern, uz6 repoamn slandenosses oser neree poir poarizoee aosesereece paooysares imropeda inccssoe poeeat peosss nisevvio seniyyorrs
escrwoaryy nosemia sip-cenoeesi paopprt
Geada
MINOR FAIUURE
Deporia 1o dig apeaaco euepr's (RDE) c conaareen daronorgan reee rpocccin sead wox snorenik crnwo no tewnoner's cwwrrtwiay sooviane coubotorry C"poocareocT,, Oulana uscro repcesoy Bradrox appay oszeeteeees, 1 notroageeo Horp too rochlanks s'coow 0000MO ioall eere torooeeee Taaeeee caoteeyrouii peopace chaauaeso Kokkb
cte 3 *Tooeooorheea tc
Deppniia na RAT couuyaoo possurts onau Peuza year ooor bi, yauss as conmaii saeedl ahooei rypliee, cactiau atppizaeen enape E (aapsyamppt pooyees crsopor repofressges mexednorti gas RGS
ascelpeevess begpactpannyy rodeatary
plasdoess
ritic
phe "Caooood eeeil acteacts
seecume
evericc
crsopers
"fenes excrosiid Oarax
fymm peent
excrtsd voa)
in www.oasseniione spalee. and rosors peervn go nepsonorth iorii cantrann my, orpaession mapesane Up wove nome nomredine ever repiaseess wbs I'lnsdeeo to TTaapees.
napeesso reals smasimaemm as 1/11 BrocBericzaa BAB reovororil (narpexrna, stric, sceenwyy soms oxpareseer conamay fer peasu go exconeress addpvprtissn acge na rpyveri, taxcel nopyaaeev rpas elcuesus poress
pugeeeee somdinre Oppise susppforce Hamaplante
BAl 4, "CCoulaatvsa crpanss/wetcs"" Papecaa na BJ1 more erparen coujarandl crpanstoeeccnt resa aie bysa psariassasse spacyeasea narped waldknaa spuarssacs critueonn Harpeeres, acctyn po peis secrd eespri woss scepasgmm eeitt sorm musee pañooess, de slacyme enserpid wauls. Gover lows prans, au uell mepenia wcess recesmm aecapppseet posse sopi speait
sempant
"Basorean' hapela ceefersannee "
Respense
pa counsiieen
BCE
Har geeenss smpodfrro robustrr Apronparmesi Naceviin Mose sacro
(ppponaayyy voe coeree patz resurees
"Cresstora eonoriaaaane ae loggemena repkerkcu sovinuaee (Tlsson fTaure mores yourageree pasesar resd ruperia spaion sucro Scruse pecypcia
c33 6 "n spa aod tr
Hakdires orimersine
ropoan earrori spann Tradiansor Tisare va varracaoree cesion reseercreocci, soceyto rocipangazee
innode
Har
Kossesi sepaton sowylto arga name oncrye all cooke sewars vases muat pooppds ane B.F
Tipastetaan nestresl pposec nossmi sawers enperves podity feo Hanceessi simppe are rese
swnewats exceopiy wa rody, aowyyt neryyaad
spocars
cts 7 "il un coscraat syu escood's 2H
Deppniig vo A/F expartaeeeo sooctrases jon viapessk-an estrord rooeems ocdlt oo din sooee sereasaT rararg ropo recouue rasiel ynosineuuus sailyy srleary sepofarrr pecypcls, usduuyysm
ane (rapegalleyy sompianii..
"f pae crpanagaesrr "
divawceyy sugadyyny
speowarry
(laque possuney RAF.
**Qucar npsect Sudwane
wapleveo
Talpppocssiiiia iprous ** iewectuli soaneeen repeosen (D enuje npai exsiic mexcrwel pperecroeeett
"iprceptrpps aeceotutes Praswrte meaeennn are pepapode wamapaace tavees sanavosern sia repeoevcro mepottymey
WABl SHOVORDK
Deepec va sipecar easee paeepe evepris rectaip-ra ane doponsiee be surico entwery anw ele nomuvee Byns pes booeboo A spacyemee coujaraned
empanspr reocri, axprorhedl eriwace to modisn wep tesmr /aeas enpansgpeeenl beccecnuseed ridss poe coneorem sebenmarum, smm, and surrag mit, ustre nepasnoy
pennoquaeei
GOOD
Figure 15: Example prompt, responses, and annotation in Ukrainian for Fluency.
29

---

Published as a conference paper at ICLR 2026
Promptth in Remanien
Anallieeid tr la enerioo oo is to ried rececia, is des consiocare toplle dide-den diste teheoleele, connensic, meskk echitttee sacidt Car paeto screst preere 23 rntrnatinna etuuturia ptais ce putura, sample eptartann reuuraur mogta crea catoratorata e care sun erpicspla pentrr comuneatte vurerabla giadin
Original Premett he English
Analypec the booe tio meww atudaa such resurov tationaiinn and the logacy of csterat ams and what are the insiicatona for viineradta cor munbiea and the healh of the planet?
shoids, consiceing lins between techeology, accecmy, andinoerant nadia equaty mapi the ein intvect
frecrrsa
ropenenatill
meke
socets Acserc
rumal es ase pocampaud de seduce inppucuu asaace medicad prin ind tu anoo amilller de gaza effect de serk a pardrrk dar poits, de sceminsss di inffeeneeo corustalla globalo putere or abdl imptorp protunde smapr comn, nétytar vicrerbbea y siimispp planetst
dTasesiogle şi Ebonomier
Tel regenerabble and onerglo sotaró, eulland şi hidroererpet deweris di on efidente glomal aconsiblle das punct vedare econemic. Coctarlle scliute roogii au flocr in in enagia regenenasia as devena moi stretess, unductiinll la creptes ripits repecttp instrians nivel plobal, Acceets trandy energetics poiss ettraia conmrm pin creore derne locn de munaal in suctourds enerya regensratie a pee reductnee expencenje de combustiao fewrl oleer produn not volutio.
"Meduu" eM
Trecers energia reganerablll erto ssartial pentru proeçarsa neduzal combbatern cotinbaitio climutten Sumeele regenorabie enerple emit mult path doald de carbon potu strooieed neeve compenate ou endenes momuuttee for fee Phen retuosen email for de gaos ou srset de senk putam hrodte ritmaal "reikon globale pot netuce tanomenete renoonnlggion oxtrems, presuee giolectalo negutive acupes teadwerddos
Trancita ditee regenenscia poete aee tmpticeli sodata posttive spechi prin fumiiares energle acmetoill gi canttl sanata numle bstate di thrie de cenviturs.
Banra negeesetell poutt corttuull fettenatjje yeu oet Adgivede la rersorne slink eramm unarne decipur Jun couseass ji ure fitana auseute la semordaa de boak our educads
Response
rappi energer
"I "erraitoonoo ca Structa le Goobll de Pussuum
frecane regenendeill pouto schembuu hondemental structunte globeie de putans in appeiel prine
Thete begenell "DD ronouaiuaan das stargsof Eeegga nondell ppnoded skeson diai ofeemel meiimeta mudor edters inklure pespellnite commanilljer tookle pos belutte th diacependart enerpiti
depandertal
unergstc"
plrta
epiiceeere regininntll plrtor desveitars,
"ppppiccatl partru Comunittta Vulzeracle şi Sandeanss Planatat
Traaajo onsngedic poote owta us Repper auabie sems ilcatt uste scapea comuniogtion valineablle, cit as sirvicpi plonetes pine
miuuk potadeir Serada
Mick
"CConb colinboriar divattoc"* Prin redummnn unno wriilker de gare ofeet nark punton dirinas offecees rigatien sie chhintoiiter clination cae afvcteazi mac digroportionee
comunaatte rurerablaa
rogennaelll for deoott du
indee calitatoo
defomeriade:
crnkka insumee enargie regeranettil un press compler care news til aborders tudan taired comadeeee tagittetie tee tahnologgs spanamic be redk schttatu sociedd diniianso
stertic aoeestiii
mesiriaiio potcepiable
A
4- GOOD
enanga repenentell reprovtti teeaaoorar profondi austemeder enerpetion
imporcate complexe gi entercee state serwreie seheotapes, econcee el, modkutu infaeets retafila
trantipa dear cchimbrre tahnologich, si mimddets cractal globle med'. Ana scustel tranitt tribule ai sa comsiderar mei mute dimenniuee
ene "1, Tahyolog econamia
"enegrest take dleg Demottaree inhootoption regensretia cum ar parourtte sctarn ourtenats entine oi buter de stoaree energ es redus seree fouatyy costurte enarght
porpettle ittil cobatau pam inmull surt concentrate in anartio negi lamdl Acsasta
*Cor contraraa resurator entor" "ahnetoolia regenento degind materbile
poate shoe na fore reperttetl conorns y peopel tock fntooand dkperciett de conoonnnttio to ou deperserig de renerate ontioe. reginenns
ppodlcpla de mandi induariiaa tradijonnta bscate combustiol foell flegin dependents scests induttii
paant duce çlta plodded
face rega tranctjed
""leclooess are ellor", Energe reperamebit petamp stu de reduce meihcatt ore whlle ou strct de cur frttuind to cor bertre schenalettr comettee Tokayl, entracte row vilka voce oure pertou sieeooople reponesabll poate aeea uniiman soac requrte uaaa rediuss dlcck na esin geotanach vedtevata
gertinliit departtor
majedt
replomento ente
"Bludwew kast" byokcods energle tepeoraial can or borgjet redrellcctio locelie cas fondd es leno, pot owns un lipport auapea eca deta molar locale, efectlend blodieen tansa
"2, Cablate codall comantid wa negel
pioccitt
mpta. plrta
logale pot esues mai lut relay ahe reperonde ao le cnmque ticke incurs de desntlar cimioo uend
Respense
"Dopp cato na metumalor meceeedii coonralenuuui emeraa intrenatatr office rececare pentou tannzinglla iegansrablle are aderas in Sucual Glacal, unde condiitle
Grile roi sinon pirnu suttin ditoiltana color mai begats
"suerge cimmatia" Comunttbple antrenetie, zere comtiible en pugin la schimbhria cireatios, aur aderas cala me sfectare de imppactut sceetors Tranctta in enargl regenseriala
stuuuttta glozai de outare"
Télee dosind
pictale bosi fooll
fistande di Oniontud Milknsia pir flus oour nin sigppooiic
thop en lil care commudenth srl resuraale ontice peetra nehnolig le regenenblle or putee chetiia putton "Moeee konalam ocer oreat -hedl tranatte emerle regererabiil nu exte geot anertss wentd rusut on thrts decrehater camtinue all expiceters resurantic gf forge munccil E'lpe "CCopenen inte neee taae Trandipa neceitt ti craborir globed pertuu a eeggure scoers echitaall ta tehnolog fiarraare reeuree. intpative precc transtendt de teinoo gl fewntarou ol matici pot contr boa roducrra decelledor dintec lint
giile
SAR "*, imp lesti pentru siratattes banwta yi comun thi vutnento ser uemargtt
pataiie ansluririe anru croggio ereeee poute ingmerall contribed cvepaee recilarie cor vutieett tidi-lle de acces cumes reductinal depanddrrpa
seree kustee pentos
seeoie upee
vido.
-*Dpdiitaeps corurtttion" trawetpll
sums de energee redkgare asu cout atterr. "iggeebbri
Portri modilate
incout poo ii be tooe eneggia esperes vobule fo froupad schimbaai sktenice iedusii
retL raala
aconomice glotale.
Tredere energi reganerabiil petertiaiu as de traneform turnn into-un mai mattenbti edhiness car scart tuatu wate gerntaat Sucoens transtpe dectrade modulin
uare sunt aloudlte inoyulilll le cconenstocyl of los eiko st you enou iepuctti mocklaa of coltanou cemposaee dinumaiia sike centodure coter wil Buke cun tisk our accual trandii juttle cooperans mapa plreppect dmpparte cnnputtiio
gurs
Crada
3 RASS
Figure 16: Example prompt, responses, and annotation in Romanian for Fluency.
30

---

Published as a conference paper at ICLR 2026
B ANNOTATOR OVERVIEW
B.I ANNOTATOR STRUCTURE
Our annotation team consisted of both external vendor annotators and hired expert annotators. The external vendor provided annotators language across 47 languages. In addition, hired one expert annotator per language across 43 languages. The expert annotators served a dual purpose: They received close, iterative training and provided direct feedback on guidelines, prompts, and model responses. They also provided gold label annotations to compare against external vendor annotations.
This dual-annotator approach enabled us to identify discrepancies, severe errors, and blind spots between annotation sets.
B.2 HIRING PROCESS
External vendor annotators The external vendor recruited contributors from locales where the target language is the lingua franca, whenever possible. Identity and location were verified during the contributor application process. Regardless of physical location, all contributors pass language fluency certification for the target language they will work in provided by the external vendor.
required to
Expert annotators All expert annotators are prescreened and then put through 30 minute lan- guage interview with a subject matter expert.
B.3 ANNOTATOR TRAINING AND TESTING
External vendor The external vendor prepared a number of upskilling materials help contrib- utors understand the guidelines, including a task walkthrough video, clarification documents, and practice quizzes. We additionally created a primary Qualification quiz including a combination of guidelines comprehension questions (including T/F and MCQs) well sample rating questions.
Expert Annotators ity and consistency:
Our training process included multiple components to ensure annotator qual-
Training Sessions:
Live training sessions where authors walked through guidelines and explained the annota- tion process
Recorded sessions available for annotators to review needed
Written guidelines shared in advance for pre-study
Qualification Testing:
Expert annotators completed practice tests for each category (Tone, Fluency, Localized Tone, Localized Factuality) using English examples.
Test sets were pre-annotated by the authors serve as gold-standard references Passing threshold: 80% accuracy; annotators scoring below underwent retraining External ven- dor annotators followed the same testing requirements with an additional layer: they first completed vendor-created qualification quizzes based the guidelines before taking practice tests.
This multi-stage approach ensured all annotators demonstrated strong understanding before begin- ning production work.
B.4 COMMUNICATION AND FEEDBACK
Expert annotators
31

---

Published as a conference paper at ICLR 2026
Live Q&A sessions:
2-3 sessions per week to address questions ongoing annotation
tasks
Escalation log: Centralized resource for guideline clarifications, annotation questions, and feedback submission
Post-task surveys: Collected with each annotation task to capture language-specific in- sights, any patterns that they noticed in model responses.
External vendor annotators
Regular communication with I session week
Escalation log for complex guideline questions or edge cases
By maintaining open communication channels, we continuously refined our approach and uncovered language-specifi considerations that improved annotation quality.
B.5 QUALITY ASSURANCE PROCESS
To monitor the quality the annotation process, we developed a QA infrastructure:
Vendor-specific QA reports to monitor and address annotation quality issues and discrepan- cies Systematic comparison of vendor annotations against expert gold labels Identification and resolution systematic errors or misinterpretations
This scalable QA infrastructure enabled to maintain high annotation quality while managing large, distributed annotation workforce.
32

---

Published as a conference paper at ICLR 2026
C PAIRWISE JUDGE TEMPLATE
Figure 17 shows an example of pairwise judge with pointwise scoring for Tone. Other dimensions follow similar template with varied intros.
Pairwise Judge with Pointwise Scoring Template
You are judge of the quality of the response to user prompt with respect to the response's [evaluation.categrry,,
You will be given:
1. A detailed 5-point grading rubric.
prompt and two responses (A and B).
Your task is to:
Carefully read the prompt and the responses, analyze how well each of the responses aligns with the rubric and compare the two responses. Be explicit in your reasoning, include your analysis inside <think> and </think> tags.
Assign a final grade for each of the responses using the rubric (between <final _g </final ggane_A> and <final_grade_B> </final grane__B tags).
#fttt Grading Criteria:
(five point grading rubric]
### Output Format:
<think Your analysis and reasoning here. </think>
<final_grade_A> FINAL GRADE: MATOR FAILURE / MINOR FAILURE PASS / 4
GOOD /5 5 EXCELLENT </final grade A>
Frinal.grad_BB B> FINAL GRADE:
MAJOR FAILURE / 2 MINOR FAILURE 3 PASS /
GOOD 5 EXCELLENT s/final_gran
Below are the prompt and the responses that you need to grade: <Prompt>
{prompt]
</Prampt>
<Response A> {response } </Response_A>
<Response .B>
(response b)
</Response_B
Figure 17: Pairwise judge prompt template.
D EXPERIMENT DETAILS
D.1 FINETUNING LLM-JUDGES ON MENLO
Finetuning LLM-Judges uses 16xH100 GPUs for Qwen3-4B and 192 GPUs for Llama4-Scout
SFT, models directly predict 5-point grades for response pairs without generating intermedi- ate reasoning, trained with cross-entropy loss under teacher forcing. We use the TRL (https: //huggingface, co/ co/ ht//docs/rr/ library and adopt the default learning rate of 2e-5. Maximum se- quence length is set to 8192.
In RL, we use GRPO with the verl (https: //gtthu com//volceginne/verr) implementation, keeping the default learning rate 1e-6. We set rollout size 8, and maximum length 4,096 tokens for both input and output. Prompts follow the template in Figure 17, encouraging models to produce reasoning before assigning scores.
Batch size is set to 32, and we train up to three epochs and select best checkpoint based on the performance the validation set.
33

---

Published as a conference paper at ICLR 2026
D.2 POST TRAINING WITH RL
We set the learning rate to le-6, the maximum tokens for the policy model to 1,024, and for the RM to 4,096, using up 4x H100 GPUs. Following Liu al. (2025), disable the length normalization term in the loss, as we find that otherwise responses tend to grow excessively long after training.
Since the judge is not trained evaluate the thinking process but only the responses, we sample generations from the policy model Qwen3-4B in non-thinking mode. Comparison of the response quality before (Qwen3-4B) and after training (Post-train) are both done in thinking mode, as we find it leads to superior generation quality. When constructing preference pairs for the pairwise judge, we remove the thinking tokens from the generations.
Batch size is also set to 32, and we train up to three epochs and select best checkpoint based on the performance on the validation set.
E ADDITIONAL RESULTS
E. JUDGE PERFORMANCE PER DIMENSION
We report detailed results for all eight models across four metries: Macro-F1 and Accuracy for 5-way classification, Preference accuracy over A win / A loss/ Tie, and Krippendorff's a for agree- ment with human annotators. Results are shown for the four dimensions: Fluency, Localized Factu- ality, Localized Tone, and Tone.
Table 14, Table 15, and Table 16 present results for Zero-shot POINTWISE, shot PAIRWISE, and Few-shot POINTWISE with grading rubrics. Table 17 and Table 18 report corresponding Zero- shot POINTWISE and Zero-shot PAIRWISE results without grading rubrics. Table 19 compares dimension-wise performance of Qwen3-4B and Llama4-Scon trained with SFT and RI. on all data, including both POINTWISE and PAIRWISE.
Overall, Localized Factuality remains the most challenging dimension: both frontier API models and RL-trained models show limited improvement. This suggests that alternative training approaches, such as integrating search and tool use, may be necessary, which we leave for future work.
Table 21 further compares dimension-wee wise performance of PAIRWISE RL-trained Qwen3-4B on par- tial subsets of the data. Specifically, we evaluate (i) models trained on single dimension and tested across all dimensions to study cross-task transfer, and (ii) models trained only on English data and evaluated on all languages. Results show that optimizing single dimension achieves performance similar to joint training (Table 19), highlighting the efficiency and practicality of joint training. In contrast, training only on English leads to degraded performance, revealing the challenges of cross- lingual transfer given the localized nature of our MENLO dataset.
E.2 JUDGE PERFORMANCE PER LANGUAGE VARIETY
Table 22 and Table 23 show Marco-F1 and Preference accuracy per Language Variety for baseline and fine-tuned Qwen3-4B and Llama4-Scout models.
34

---

Published as a conference paper at ICLR 2026
Table 14: Results per Dimension: Zero-shot POINTWISE with Grading Rubrics.
Zero-shot POINTWISE with Rubries
(wen3-48
(wen3-328
Llana3 1-88
Llana3
1laa4-Scouut
gpt-4o
gpt-4.
MacroFF]
23.06
28.53
22.27
27.93
25.63
26.54
25.99
32.23
Overall
Accuracy
35
37.88
30.97
38.19
34.37
36.33
Prefenence
40.54
42.19
39.92
37.37
42.19
45.07
42.92
41.73
Krippendortf's a
80.59
83.80
79.35
80.71
79.64
83.59
83.78
Macro-FI
17.60
29.01
19.35
21.03
22.09
23.29
18.55
20.73
Accuracy
37.07
41.98
32,57
36.87
39.68
43,99
36.37
37.17
Hluency
Preference
37.68
41.28
40.48
35.27
43.52
41,28
34.47
34.87
Krippendort 's
76.73
81.47
77.85
78.29
80.35
83.19
76.59
76.47
Macro-FI
14.94
15.77
14.17
16.74
12.04
19.27
20.25
Localized Factuality
Accuracy
30.16
26.22
26.49
34.10
31.93
13.86
29.48
31.11
Preference
26.63
28.80
28.26
29.62
26.90
32.34
27.99
26.36
Krippendortf's
74.30
74.50
74.64
72.37
74.82
66.45
77.81
77.29
Macro-F1
17.34
29.40
19.00
20.52
19.54
28.13
22.23
23.73
Localized Tome
Accuracy
Preference
41.72
45.25
36.42
32.89
41.94
50,99
46.80
48.12
Krippendorff's
Macro-F1
36.17
40.56
23,76
38.06
32.00
25,97
35.31
41.62
Accuracy
41.14
43.61
27.47
41.03
42.15
32.85
45.18
19
Tone
Preference
54.04
51.12
52.47
50.67
53.59
53.81
60.76
55.61
Krippendorg is
87.16
88.91
81.41
88.48
87.61
82.13
90.58
90.21
Table 15: Results per Dimension: Zero-shot PAIRWISE with Grading Rubrics.
Zero-shot PAIRWISE: with Rubries
Qwen3-48
Qwen3-328
11ama3.11
11ana3. 3-788
11
gpt-4o
gpt-4.1
Macm-FI 35.46
37.48
29.46
37.50
36.11
35.35
37.57
38.53
Overall
Accuracy
43.23
29.56
Preference
57.13
59.12
50.45
$5.32
56.25
58.72
$7.98
59.23
Krippendorff's
84.25
80.17
84.10
86.35
Macm-FI
32.24
35.27
26.67
32.48
35.11
34.40
34.67
34.58
Fluency
Preference
Accuracy
55.91
46.99
45.99
59.92
27.45
50.50
43.99
$2.30
43.91
54,03
44,09
60.12
46. 19
56.51
60.32
50. 10
Ki Krippendort's 's
83.05
84.72
80.21
83.87
84.61
84.15
85.72
83.99
Macyo-F1
22.55
21.93
20.96
21.13
Localized Factuality
Accuracy
33.02
28.80
25.14
29.76
28.80 80
21.06
24.46
29.62
Preference
42.93
43.21
38.00
Kr Krippendortf's
75.97
75.96
74.03
75.07
74.74
71.41
75.93
75.54
Macro-FI
32.82
35.25
30.22
36.82
33.62
37.88
37.86
35.02
Localized Tome
Accuracy
Preference
60.49
63.80
51.21
60.71
61.37
65.34
64.24
65.78
Krippendorff's
Macro-FI
43.06
41.81
31.66
46.57
44.32
35.37
42.24
45.52
Tone
Accuracy
Preference
48.21
66.82
66.59
42.71
31.28
59.87
66.82
49.55
68.16
50.00
66.82
35.99
69.73
43.27
69.06
48.77
Krippendorff's a
89.00
89.69
82.75
90.84
89.41
87.64
90.83
90.48
35

---

Published as a conference paper at ICLR 2026
Table 16: Results per Dimension: Few-shot POINTWISE with Grading Rubrics.
Few-shot POINTWISE Rubrics
Qwen3-4B
Qwen3-322
Llama33 1-88
Llama3. 3-788
Llama4- Scout
Spt-4o
gpt-4.1
Macn-FI
31.18
35.45
22.24
30.52
12.84
27.92
29.57
33.84
Overall
Preference
Accuacy
39.35
37.71
42.87
38.59
37.15
26.25
37.63
38.56
41,22
39.92
44.68
35.93
45.87
38.19
44,00
39.01
KrippendortJ's
82.36
84.46
77.00
81.45
83.07
81.54
84.84
84.24
Macro-FI
25
29.73
20.27
Fluency
Accuracy
38.48
42.69
25.25
38.88
41.58
43.09
39.18
39.08
Preference
37.68
41.48
37.07
40.08
Kr Krippendortf's
79.93
82.20
76.35
79.76
82.11
83,70
80.22
78.79
Macro-F1
22.20
17.44
12.74
20.92
21.88
14.17
21.20
21.82
Localized Factuality
Accuracy
Preference
30.71
33.97
32,88
27.45
25.54
29.89
32.88
25.00
Krippendorff's
Macro-FI
25.26
31.27
19.86
23.14
27.45
29.00
28.55 83
29.46
Localized Tome
Accuracy
Preference
34.88
39.51
43,27
38.85
32,67
31.13
38.19
33.89
42.16
38.41
52.32
42.05
48.57
47.46
37.53
KrippendorfJ's a
78.56
81.55
77.51
75.65
79.13
81.34
81.15
80.48
Macro-FI
37.11
Accurccy
43.05
45.07
28.48
41.03
44.17
35.65
44.96
47.87
Tome
Preference
Krippendorf''s is
88.02
89.72
80.00
87.71
87.76
84.60
90.78
90.73
Table 17: Results per Dimension: Zero-shot POINTWISE without Grading Rubries.
Zero-shot POINTWISE without Rubrics Qwen3-4B
Qwen3-328
Llama3.1-88
Llana3.3-
Llanz4-Scout
o3
gpt-4a
gpt-4.
Macno-FI Accurdey Preference
16.00
25.59
21.50
22.71
22.15
22.45
22.26
Overall
32.16
36.24
33.18
33.52
36.24
35.14
34,88
34.54
43.32
38.34
41,28
45.13
37.60
38.67
Krippendorff's 's
76.05
81.63
79.70
78.18
79.93
80.41
80.66
80.12
Mocm-FF FI
10.84
19.70
20.73
21.11
23.11
21.85
15.96
17.38
Accuracy
34.37
39.18
36.77
37.17
37.58
40.48
36.37
35.77
Fluency
Preference
32.46
39.48
40.08
35.27
41.48
48.10
31.46
31.66
Krippendorf''s
73.46
78.53
79.31
77.32
78.84
80.15
76.22
74.64
Mocm-F1
12.94
19.93
15.22
18.30
17.03
13.46
22. 09
20.02
Localized Factuality
32.20
29.62
33.56
32.34
20.52
32.34
Preference
26.90
34.51
27.45
23.10
26.90
31.52
23.37
29.08
Krippendorf'sse
71.05
78.12
75.43
70.85
72.31
70.56
75.15
75.63
Macro-F1 Fl
12.23
25.65
19.38
18.32
21.19
25.21
18.75
25.15
Localized Tone
Accuracy
Preference
29.80
45.25
40.84
34.44
41,06
$1.88
43.49
43.93
Krippendorff's
Macro-F1
26.06
30.19
23.79
26.79
29.16
27.04
35.65
27.11
Tome
Accuracy
29.60
33.97
41.82
38.57
Preforence
43.95
$2.91
42.83
43.27
$3.14
46.19
$0.22
49.10
Krippendorff's 's
36

---

Published as a conference paper at ICLR 2026
Table 18: Results per Dimension: Zero-shot PAIRWISE without Grading Rubrics.
Zero-shot PAIRWISE without Rubrics Qwen3-4B
Qwen3-32B
Llana3. 1-88
Llana]. 3-78883-88
Llanat-Scout
Spc-4o
spt-4.
Macro-F'1
32.74
38.10
30.89
35.12
35.21
37.60
36.74
37.35
Overall
Accuracy
Preference
$4.08
59.23
49.55
$6.29
55.10
57.98
56.85
56.96
Krippendorf'ss
Macro-F'1
31.08
38.71
30.27
33.42
32.34
34,97
32.98
30.86
Fluency
Accuracy
43.89
48.30
33.37
45.29
42.99
47.80
44.09
19
Preference
55.11
59.72
46.49
53.71
52.91
58.72
51.70
52.51
Krippendorf'ss o
82.89
86.06
81.93
83.74
16
84.12
84.35
82.00
22.05
Localized Factuality
Accuracy
30.98
28.
27.58
30.71
29.48
24.46
30,30
32.20
Preference
Krippendortf's
75.45
766.97
73.96
74.51
73.68
72.82
76.23
75.29
Macro-FI
32.51
36.30
30.43
35.10
34.11
40.45
36.38
37.36
Localized
Accuracy
43.27
42.38
34.77
44.59
43.38
45.36
45.25
48.01
Preference
55.85
63.36
52.54
60.93
62.25
65.78
66.45
64.02
Krippendorf''s
82.27
86.05
81.30
84.23
83.41
86.12
86.30
85.10
Macro-F1
36.97
42.94
34.84
40.71
43.33
39.54
42.42
44.67
Tome
Accuracy
Preference
62.33
67.49
60.76
66.37
65.70
65.02
66.59
67.71
Krippendorffss
Table 19: Results per Dimension: SFT and RL trained Qwen3-4B and RI trained Llama4-Scout All Data with POINTWISE and PAIRWISE Scoring.
Qwen3-4B
Qwen3-4B POINTWISE
Llama4-Scout PAIRWISE-SFT-HRR RI.
SFT and RI. on Data
POINTWISE PAIRWISE
PAIRWISE
PAIRWISE-SFT
Macro-F1
30.26
33.44
28.87
39.44
45.04
45.82
Overall
Accwacy
35.82
Preference
41.17
53,68
39.86
60.02
60.53
61.10
Krippendorf x
83.90
84.03
82.10
86.55
89.48
89.67
Macro-F1
32.91
25.10
35,72
46.42
Accuary
38.18
37.17
39,38
$2.51
55.21
56.71
Fluency
Preference
38.48
54.71
42.89
61.92
66.13
66.53
Krippendort
Macro-F1
20.30
19.51
17.49
20.62
25.30
25.87
Localized Factuality
Accuuary
31.66
25.14
33.42
33.02
34.78
35.33
Preference
39.67
38.86
36.68
Krippendorf'ss
76.39
74.43
75.37
77.04
80.25
80.14
Macro-F1
27.43
33.38
25.58
38.56
40.98
41.22
Loculized Tone
Accuracy
37.75
41.06
37.86
47.35
53.53
53.20
Preference
39.29
$9.38
67.55
63.58
Krippendorf''s a
80.54
81.83
78.89
86.61
87.87
88.23
Macro-F1 F'l
33.35
34,91
33.98
46.82
$1.08
$2.29
Acewary
51.35
53.81
Tone
Preference
50.45
$8.30
49.78
67.71
70.85
71.08
Krippendorf 's
88.08
87.91
86.67
90.39
92.63
93.00
Comparison of zero-shot PAIRWISE Qwen3-4B and trained models, trained either jointly across all dimensions (multi-task) individually per dimension (single-task).
Table 20:
Macro-F1 MULTI-TASK
Preference Accuracy
Dimension
ZERO-SHOT
SINGLE-TASE
ZERO-SHOT
MULTI-TASK SINGLE-TASK
Fluency Tone
32.24
37.14
55.91
61.32
43.06
46.82
46.18
66.82
67.71
69.28
Localized Tone
32.82
38.56
37.61
60.49
67.55
66.67
Localized Factuality
22.55
23.12
42.93
38.86
42.12
37

---

Published as a conference paper at ICLR 2026
Table 21: Results per Dimension: RL trained Qwen3-4B on PAIRWISE Single Dimension Data and PAIRWISE English Data.
Single Dimension on All languages Localized Tone
English Data All Categories
PaIRWISE RL on Partial Data
Fluency
Localized Factuality
Tone
Macro-FI
37.89
35.33
37.46
38.55
34.34
Accuracy
44.56
43.74
43.69
45,19
42.33
Overall
Preference
59.29
56.68
59.29
57.53
56.46
Krippendorff's a
85.63
84.06
86.12
86.20
83.98
Macro-F1
37.14
31.53
36.07
34.82
30.65
Accuracy
51.80
47.39
50.00
50.60
45.59
Fluency
Preference
61.32
56.91
58.72
56.51
55.31
Krippendorf'ss
85.87
82.98
86.09
84.85
82.64
Macro-F1
20.12
23.12
18.87
19.78
19.92
Localized Factuality
Accuracy
29.89
34.10
29.21
30.71
32.61
Preference
42,12
42.12
40.22
41.03
41.85
Krippendorff's 's a
75.19
76.65
75.83
76.58
76.44
Macro-F1
35.84
30.29
37.61
37.32
28.51
Accuracy
44.92
42.49
47.57
45.92
41.28
Loculized Tone
Preference
63.36
58.72
66.67
60.49
57.
Krippendorffss
84.20
82.01
86.28
84.87
80.87
Macro-F1
43.01
43.96
41.61
46.18
42.50
Accuracy
48.21
48.88
44.62
50.34
47.76
Tone
Preference
67.04
66.37
68.16
69.28
68.83
Krippendorff's
89.66
89.28
89.61
91.28
88.89
38

---

Published as a conference paper at ICLR 2026
Table 22: Macro-Fl scores per Language Variety: Comparing
PATRWISE Qwen3-4B and
Llama4-Scout zero-shot performance and various trained models.
Qwen3-4B RL
Llama4-Scout SFT
PAIRWISE
Zero-shot
SFT
RL on EN-only data
Zero-shot
SFT + RL
Overall
35.46
33.55
39.44
34.34
36.11
44.17
45.82
ar
31.21
21.92
36.26
33.19
32.29
36.20
43.71
ar_Latn_EG
16.41
22.60
9.62
17.52
18.71
54.88
65.12
bg-BG
29.95
28.12
37.80
26.39
26.75
31.31
33.02
bn
24.20
15.08
20.17
18.04
13.84
20.14
23.37
cs.CZ
34.58
18.81
38.01
34.44
35.39
41.97
44.86
da DK
21.50
17.91
29.43
24.78
31.71
34.98
40.73
de_DE
27.63
26.70
24.76
17.94
16.51
37.35
34.26
el_GR
36.90
37.84
43.74
40.31
39.68
45.11
41.64
en AU
34.94
40.74
55.79
35.79
42.68
41.19
41.26
en.GB
44.86
46.77
46.13
48.93
42.52
40.69
51.55
en IN
39.11
27.62
36.47
48.68
44.96
37.31
39.40
en_US
33.42
34.49
29.09
33.25
47.68
28.35
23.13
ES
38.97
29.93
41.21
30.53
29.23
27.44
27.07
es_MX
42.01
24.56
51.77
46.63
44.57
36.80
38.98
fa_IR
31.71
33.25
39.11
33.04
38.75
33,91
43.11
fr
21.33
29.53
39.02
25.13
30.49
19.65
33.97
gu_IN
30.00
37.60
46.05
30.10
49.79
46.29
43.01
he IL
24.89
19.71
25.75
26.86
22,66
32.50
38.17
hi_IN
23.46
16.16
36.77
30.04
27.91
28.81
36.42
Li
48.52
27.56
41.53
39.74
34.52
25.79
53.80
hr HR
20.16
34.86
25.15
16.69
18.45
26.57
29.01
hu_HU
27.69
36.63
37.06
28.84
33.83
40.92
39.55
id ID
41.40
45.21
42.53
31.33
35.64
41.28
42.98
it_IT
29.48
34.77
40.28
31.61
29.16
24.51
29.34
ja.JP
45.06
40.34
39.68
44.76
35.37
44.98
41.55
ko KR
35.51
35.14
33.03
32.59
39.12
38.38
48.80
IN
41.44
36.52
51.06
47.78
42.84
50.10
51.59
ms MY
29.50
30.79
39.74
33.02
28.99
36.27
44.60
ne_NP
28.05
22.45
27.59
19.41
24.62
20.16
24.54
nl NL
36.00
35.38
38,80
27.61
45.25
54.06
51.22
pl_PL
28.20
33.52
21.82
23.77
17.94
29.00
21.29
pt_BR
45.17
36.52
48.48
43.34
48.19
48.45
41.83
PT
38.72
39.29
44.62
41.22
38.89
52.15
41.57
rO_RO
37.54
42.92
49.36
46.85
51.91
52.42
55.46
ru RU
31.75
22.59
22.44
19.78
28.84
18.72
21.61
sk_SK
35.80
38.22
44.14
33.79
37.20
44.29
40.69
sv_SE
31.59
35.11
33.78
27.44
34.29
39.48
40.81
KE
41.97
28.13
41.88
21.36
43.13
37.33
39.67
th_TH
37.07
32.75
47.04
35,98
45.82
52.77
55.11
tl PH
40.71
29.19
45.39
42.52
39,95
40.54
37.78
tr_TR
50.03
37.30
48.50
40.67
40.66
40.45
45.42
uk_UA
24.09
29.20
20.45
22.09
17.57
23.98
27.13
ur Latn PK
29.38
38.54
34.25
27.72
32.49
32.93
29.59
ur_PK
23.21
38.81
36.11
28.29
39.87
48.94
43.73
VN
33.91
31.07
34.76
35.16
30.35
40.51
38.46
zh_CN
40.82
27.78
51.99
41.15
37.42
45.27
41.58
zh TW
35.38
25.47
37.07
44.80
37.93
39.29
38.09
39

---

Published as a conference paper at ICLR 2026
Table 23: Preference accuracy per Language: Comparing PATRWISE Qwen3-4B and 1amaa4 Scoou
zero-shot performance and various trained models.
Qwen3-4B RL
Llama4-Scout
PAIRWISE
Zero-shot
SFT
RL on EN-only data
Zero-shot
SFT
SFT + RL
Overall
57.13
53.51
60.02
56,46
56.25
60.08
61.10
ar
63.16
50.00
60.53
57.89
61.54
73.68
55.26
ar_Latn_EG
35.48
25.81
38.71
45.16
28.12
93.55
93.55
bg-BG
51.28
56.41
48.72
38,46
47.50
46.15
46.15
bn_
41.38
27.59
37.93
44.83
36.67
41.38
41.38
cs.CZ
69.23
74.36
76,92
69.23
70.00
74.36
74.36
da DK
35.90
38.46
43.59
46.15
40.00
51.28
46.15
de_DE
51.61
54.84
58.06
38.71
59.38
54.84
54.84
el_GR
56.41
56.41
61.54
66.67
57.50
66.67
64.10
en AU
34.21
47.37
39.47
36.84
34.21
39.47
44.74
en.GB
53,85
64.10
53.85
58.97
43.59
46.15
58.97
en IN
43.59
41.03
48.72
43.59
51.28
51.28
43.59
en_US
53.12
46.88
40.62
62.50
43.75
50.00
46.88
ES
64.10
51.28
48.72
56.41
51.28
51.28
51.28
es_MX
53.85
35.90
56.41
46.15
43.59
51.28
51.28
fa_IR
66.67
69.23
66.67
61.54
56,41
58.97
79.49
fr
60.53
55.26
57.89
52.63
55.26
42.11
60.53
gu_IN
48.65
43.24
64.86
64.86
67.57
56.76
62.16
he IL
58.97
61.54
61.54
64.10
61.54
48.72
64.10
hi_IN
53.12
46.88
62.50
59.38
59.38
59.38
71.88
LitnatnI
61.54
48.72
66.67
71.79
62.50
56.41
64.10
hr HR
43.59
56.41
48.72
51.28
51.28
51.28
51.28
hu_HU
55.26
52.63
63.16
63.16
60.53
65.79
65.79
id ID
75.68
70.27
81.08
75.68
78.38
83.78
81.08
it_IT
59.38
59.38
62.50
46.88
53.12
59.38
59.38
ja.JP
74.36
71.79
71.79
61.54
61.54
71.79
74.36
ko KR
63.16
65.79
76.32
68.42
71.05
81.58
84.21
IN-
56.41
53.85
69.23
58.97
56.41
66.67
64.10
ms MY
58.97
53.85
58.97
58.97
61.54
56.41
61.54
ne_NP
52.63
57.89
44.74
44.74
50.00
47.37
42.11
nl NL
60.53
65.79
63.16
50.00
63.16
71.05
71.05
pl_PL
52.63
47.37
55.26
57.89
$7.89
47.37
60.53
pt_BR
53.85
53.85
41.03
53.85
38.46
51.28
51.28
PT
44.74
44.74
55.26
47.37
50.00
57.89
47.37
rO_RO
58.97
64.10
76.92
56.41
58.97
64.10
74.36
ru RU
50.00
40.62
46.88
40.62
53.12
43.75
50.00
sk_SK
69.23
56.41
74.36
66.67
58.97
69.23
66.67
sv_SE
56.41
41.03
46.15
53.85
43.59
48.72
35.90
KE
56.41
58.97
71.79
56.41
66.67
69.23
69.23
th_TH
53.85
53.85
51.28
58.97
58.97
48.72
51.28
tl PH
69.23
61.54
71.79
58.97
64.10
71.79
66.67
tr_TR
71.79
61.54
82.05
71.79
79.49
84.62
76.92
uk_UA
58.97
38.46
66.67
51.28
46.15
64.10
61.54
ur Latn PK
53.85
66.67
61.54
53.85
55.00
66.67
64.10
ur_PK
69.23
43.59
69.23
53,85
66.67
58.97
64.10
VN
51.28
48.72
56.41
58.97
56.41
58.97
$8.97
zh_CN
61.54
48.72
64.10
61.54
64.10
66.67
61.54
zh TW
84.62
66.67
82.05
74.36
76.92
79.49
79.49
40

---

Published as a conference paper at ICLR 2026
F EXAMPLES OF LLM-JUDGE OUTPUT ON MENLO
F.1 COMPARISON: WITH AND WITHOUT RUBRICS
In this section we compare zero-shot LLM-judge output with and with provided rubrics. On average, LLM-- generate 15-20% longer output when rubrics are present, where the additional text is attributed referencing the grading criteria (marked blue).
gpt-4.1 Judge Output: Scoo Pairwise without Rubrics
<think>
Response A clear, well-organized, and written a professional and emotionally appealing tone. vocabulary appropriate, awkward sentences. The syntax varies, with lists and sections broken down for clarity. The use of bold headings and bullet points adds structure and readability. The closing lines, calling for donations or sharing, are direct and polite. The response is coherent, logical, and easy to follow, making it highly fluent.
there are no grammatical errors
Response is also very fluent. It opens with a heartfelt, personal appeal, addressing various possible audiences, and the language is rich and engaging. It uses appropriate vocabulary and emotional resonance, varying sentence structure situation, why donations
effortlessly. The post breaks down matter, and how people can help, all in logical sequence. The post error-free in terms of grammar, punctuation, and overall phrasing. There awkward sentences, and the message flows naturally with compelling human touch.
dog's story,
Both responses display strong fluency, but Response B is especially striking in its narrative style and touches deeper emotional notes, making it both clear and highly convincing. </think>
<final.grade_A> FINAL GRADE: EXCELLENT </final @ffinal-grade.l A2
<final grade FINAL GRADE: EXCELLENT </final grade B>
Figure 18: Example Judge Output from gpt-4. 1: Zero-shot Pairwise without Rubrics.
41

---

Published as a conference paper at ICLR 2026
gpt-4.1 Judge Output: Zero-shot Pairwise with Rubrics
<think>
First, will evaluate Response A and Response B according to the fluency rubric.
**Response A Analysis:*s
Organization Structure:
The post begins with strong headline and clearly, presenting situation and the for help. The body is organized with subsections ("Why Your Denation Matters", "What Your Donation Will Help
Fund", "Every Donation Counts") and bulleted lists.
There is logical flow fron condition, request, to breakdown costs, to call to action.
Language
The language throughout gramnatically correct. Vocabulary is appropriate and natural for the context.
No awkward sentences or structures detected; uses mix of simple and complex sentences. Punctuation and capitalization are proper.
Clarity
The post 15 clear, direct, and to follow.
There is some repetition ("I'11 be forever grateful" is nentioned nore than once), but it doesn't 't detract signi ficantly from fluency.
Some sentence structures are basic, and enotional appeal moderate but bit formulate; lacks varied rhythrs.
sophisticated Effective bullet points for donation inpact.
Overall:
There are no gramnatical errors, aukward phrases, or disconnected ideas. The response is more functional and structural than deeply moving or varied, but clearly gets the job at level.
**Response B Analysis:**
Organization Structure:
Begiins with headline, greeting, enotional context; progresses logically fron situuation to the needs, ispact donations, provides
dog's personal value, specific action.
Subsections and headings nake the structure highly clear. Bulleted/dashed lists and paragraphs enhance organization.
Language Gramnar:
Language expressive, and \I'm reaching gramnatical, punctuation, capitalization errors. Advanced of dashes and colons.
sentences My beloved dog. ... just pet pet/they'r 're my family.
you with heavy heart and hopeful spirit...
Sentence variety is excellent, combining simple, compound, and complex sentences fluidly. Vocabulary is rich and nannced, with phrases like "ripple effert of kindness" and "fighting chance".
Clarity
The post clear, compelling, and erotionally resonant. It feels personal and heartfelt, enhancing its persuasion and fluency.
Efficient use of paragraphs and bulleted lists nakes information digestible.
No redundancy; each section advances the argunent or enotional appeal.
Overall:
Denonstrates high fluency: effortless transitions, saphisticated language, logical organization, purposeful word choice.
No unnecessary verbosity or anbiguity.
Host closely matches the EXCELLENT standard.
**Conpar ison:***
Both responses well structured clear; neither contains grannar punctuation
Response A is efficient and functional, but lacks enotional depth and sentence variety compared to B. Response B excels in narrative flow. word choice. enotional engagenent, and sophisticated sentence construction, thus best matching the highest standard for fluency. </think
GDOD </final grade <final_grade_B> FINAL GRADE: EXCELLENT </final.grade.B>
<final FINAL
Figure 19; Example Judge Output from gpt-4. 1: Zero-shot Pairwise with Rubrics.
42

---

Published as a conference paper at ICLR 2026
Llama4-scout Judge Output: Zero-shot Pairwise without Rubrics
<think>
will analyze both responses based on fluency, which includes coherence, grammatical errors, vocabulary, syntax, and clarity.
Response A is well-structured and coherent, effectively conveying the urgency and importance of the situation. It uses clear and direct tone, making it easy to understand the request. The sections are well-defined, and the use of bullet points helps in listing the details of how donations will be used. The response also provides specific examples of how different amounts of money can contribute to the dog's s care, which helps in making the request more tangible.
However, Response could seen as slightly less personal, as it does not specify the dog's or the specific health condition. This might make generic.
bit more
Response B is highly personalized, addressing the dog by name and specifying the health condition, which adds emotional appeal. It is also well-structured, with clear sections explaining the situation, why help is needed, and how donations will be used. The use of a narrative style makes the post more engaging and relatable.
Both responses are free grammatical errors use appropriate vocabulary syntax. They are clear effectively communicate their message.
In terms of fluency and overall impact, Response B seems to have slight edge due to its personalized and narrative approach, which can make the appeal more compelling and relatable. </think>
<final-grade_A> FINAL GRADE: 4 GOOD </final.grade.A>
<final grade Frade FINAL GRADE: EXCELLENT </final grade
Figure 20: Example Judge Output from LI Llama4-scout, Zero-shot Pairwise without Rubrics.
43

---

Published as a conference paper at ICLR 2026
Llama4-scou Judge Output: Zero-shot Pairwise with Rubrics
<think>
I will carefully analyze both responses based on the provided grading rubric, focusing on fluency, coherence, gr grammation correctness, vocabulary, overall clarity.
Response is well-structured and clear. It starts with introduction to the problem, followed by an explanation of the situation, and then details on how donations will be used. The response uses appropriate vocabulary and syntax, making it easy to understand. It also provides specific examples of how different amounts of donations can help. The tone is heartfelt and persuasive.
Response is also well-structured and provides detailed account of the situation. includes personal touches, such as referring to the dog as "my best friend" and sharing the emotional impact of the situation. The response is coherent, with clear transitions between ideas. It also specifies how donations will be used and includes a call to action with link to fundraising platform. The tone is emotional and persuasive.
Both responses are free of major grammatical errors and use appropriate vocabulary. However, Response B seems to provide a more personal touch and detailed explanation of the situation.
Now, let's s compare the two responses based on the grading criteria:
Response A:
It is coherent and understandable.
It has a clear structure and logical flow.
The response is grammatically correct and free of errors. Words are used accurately and in context.
The sentence structure varies.
The text is easy to understand.
Response
is also coherent and understandable.
It has clear structure and logical flow.
The response is grammttically correct and free of errors.
Words are used accurately and in context.
The sentence structure varies.
The text is easy to understand tand and provides personal touch.
Based on these observations, I would rate Response A as a (GODD) because clarity but does not stand out in terms of
meets all the criteria for fluency sophistication or emotional depth compared to Response B.
I would rate Response B 5 (EXCELLENT) because it not only meets all the criteria for fluency and clarity but also provides a personal touch, detailed explanations, and effectively uses emotional appeal to persuade the reader.
</think>
<final grade FINAL GRADE: GOOD </final grade A> <final.gradeBB> FINAL GRADE: EXCELLENT </final.grade.>>
Figure 21: Example Judge Output from Llama4-scout: Zero-shot Pairwise with Rubrics.
44

---

Published as a conference paper at ICLR 2026
Qwen3-32B Zero-shot Pairwise without Rubries
<think
Okay, let's start by looking at the prompt. The user wants convincing post explaining why someone should health condition. the response should be enotional, clear, and structured in way that highlights the dog' need and the impact of donations.
donate noney help dog
goal persuade readers donate,
First, P'l analyze Response A. The structure is good with clear sectians: a title, an introduction explaining the problem, section on why donations natteer what the donations will fund, and breakdown different donation anounts help. language direct uses persuasive appeals ('beloved dog", "furry friend"). It's coherent free of gramatical errors. The enational points makes easy to read. However, it's bit generic. Phrases like "furry friend" are cormon and night not stand out. Also, the placeholder for the dog's nane and specific conditions is missing, which could nake the post feel incomplate. The closing is strong but could be more personalized. I'd say it's solid, effective post but lacks some personal touch and specific details that could nake it more compelling.
bullet
Response This establishes an emotiional connection. The structure is similar but nore elaborate, with sections like "The Situation," "Why Your Help Matters," and "Why [Dog's Nane] Deserves a Second Chance." It uses specific placeholders for the dog's nane and condition, which makes the post feel more tailored. The language is rich evocative, using vivid descriptions of the dog's personality of unconditional
detailed and personal.
heartfelt greeting and imeeiatell
the impact of their presence. The
enotional appeal stronger here, using phrases like "ny much ne. The closing also personal and heartfelt, including postscript that reinforces the enotional connection. The structure issue is that sore placeholders night need to be filled in, but that's expected. Overall, this response is nore engaging and persuasive due to its personal ization and enational depth.
and "they've given
clear, with each section flowing into the next, The only minor
Compar ing two, Response provides specific, personalized details. Response is but bit generic. Both fluent, but 8 excels in naking the reader feel the urgency and personal connection to the dog's situation. I'll rate Response B higher for its superior enotional appeal and personalization while Response A is solid but less impactful post. </thinko
effective because creates a stronger enotional connection
Both respanses well well-structured and persuasive, but they differ personal ization, enotional depth, and specificity. Here's detailed analysis:
>*Respanse Art;
sufluencyts: The post is coherent, gramatically correct, and uses appropriate vocabulary. The structure logical, with clear headings and bullet points guide reader.
s*Strengthste; effectively outlines dog's medical needs, purpose donations, and and inclusive, encouraging small
different contribution levels inpact of support (e.g., sharing the post).
The clasing
selleaknesseste; The language is sonewhat generic (e.g.. "Furry friend," " "beacon of hope") and lacks specific personal details about the dog (e.g., name, personality traits). This nakes the appeal feel less unique and enotionally resonant. The placeholder text (e.g., "ny dog") is underutilized, reducing the sense of urgency individuelity.
**Response Be*:
**Fluencyte: The response is exceptionally fluent, with natural flow and polished syntax. The tone is heartfelt and professional, balancing enational appeal with clarity.
**Strengths<*: personalizes the dog as a "best friend" and emphasizes their role in ownr'ss (e.g., "source of unconditional love"). Specific placehelders (e.g., "[Your Dog's Nane]") allow custonization, and the inclusion of the dog's personality traits (e.g., "gaofy antics," "gentle nature") creates vivid, relatable image. The breakdown of costs and enotional stakes is detailed and persuasive. **Meaknesses**: None significant. The only ninor issue is that placeholders (e.g.. "[specific health condi sondition'") require completion, but this is expected in template.
GOOD </final. grade <final grade B> FINAL GRADE: 5 EXCELLENT </final B>
<f'nna
Figure 22: Example Judge Output from Qwen3- 32B: Zero-shot Pairwise without Rubrics.
45

---

Published as a conference paper at ICLR 2026
Qwen3-32B Judge Output: Zero-shot Pairwise with Rubrics
<thinks
Okay, let's start understanding the need evaluate responses fluency rubrics. The prompt write convincing post explaining why soneone should donate noney for dog with bad health condition. The rubrics are from 1 to 5, with being excellent.
based given
First, I'11 look at Response A. The structure seens clear with headings like "Help My Furry Friend..." and bullet points. The language is straightforward, gramatical
the nessage direct. don't notice any najor The sentences vary structure, and the information is organized logically. The bullet points and clear explanations of how donations help nakes it rany to understand. However, it's hit generic; naybe not as enotionally engaging it could be, but still, it's clear and neets the requirements.
Now Response B. It starts with a personal touch, andressing friends, family, and strangers. The structure sections like "The Situation" and "Why Your Help Matters." The language nore
nore detailed enotional personal,
more convincing. Dog's Nare] and [specific health candition], which could filled in, but in the response given, those are placeholders. However, the actual text is well-written, with varied sentence structures and gr gramatical issues. The use of examples like "tail wags," " "walks in the park" adds enotional weight. The conclusion is heartfelt and includes personal signature, which adds authenticity.
night
are specific placeholders
[Your
Compar ing both, Response night resonate which could enhance the persuasiveness. Response is effective but more generic. However, both are free of gramatical errors and well-structured.
using storytelling elenents with readers. alse provides specific exaples of how the dog impacts lives,
nore engaging and personalized
rubrics, well-structured, and personal touch, and suphisticated structure. Hooevver, the placeholders in B night be considered minor flaw if they were meant to be filled in, but since they're 're just placeholders in the exapple, naybe nat. Response A is solid 4.5, but since the rubric is in whole numbers, naybe 4 for A and 5 for B. </thinks
Looking
responses are
(G00D) because they are gramatically correct, sentences. Response night edge towards because of its enotional depth,
responses well-structured free grannatical errors, they differ in enotional depth, personalization, and sophistication. Here's the analysis:
**Rssposse Aes
**Fluency Structureet: The post is organized with clear headings and bullet points, naking it easy to follow. The language is straightffowaad and direct, with logical flow between sections. Syntak**: gramatical errors present. Sentences varied structure (simple and compound), and vocabulary appropriste.
**oord Choicess; The use of phrases like "furry friend" and "beacon of hope" adds emotional appeal, though the tore is slightly generic.
##Effectiveeesae:; The post effectively commuiicates the need for donations and explains how funds will used. However, it lacks personal anecdotes or specific details about the dog, which might reduce its enotional impact.
*AResponse Ве*
The response is more sophisticated, with a heartfelt, narrative-driven approach. It uses sections like "The Situntion" and "Why [Yaur Dog] Deserves a Second Chance" to build a compell ing case. structure is logical, gramnatical
**Fluency & Structurea::
smooth transitions.
**Gratar Syntax**:
Sentences vary in complexity (simple, compound, and loved pet...") enhances engagenent.
complex), and the use of rhetorical questions ("If you've
wAMord Choicose: The language is enationally resonant, with vivid imagery ("tail wags," "cuddles on the couch") and personalized placeholders (e.g., [Your Dog's Nanel). The tone is warm and persuasive.
AvEffectivenesss:: By framing the dog as "soul who has touched sa nany lives," " the response creates derper enotional connection. inclusion specific examples (a.g., "goofy antics") sign-off ("With gratitude and hope") adds authenticity.
personal
vAComparisonex:
**Response Axe is a solid, clear post that neets the prompt's requirements but renains sonewhat generic. It scores well on clarity and structure but locks the enational depth to stand out. persiiolization using heartfelt tone specific
**Response excels storytelling persuade readers. sophisticated structure and nuanced language align closely with the highest rubric criteria.
<final.grade_A> FINAL GRADE: 4 - GOOD </final.erade.d>
<final.grade.BB B> FINAL GRADE: 5 EXCELLENT </Fimal-grade.BB
Figure 23: Example Judge Output from Qwen3-32B. Zero-shot Pairwise with Rubrics.
46

---

Published as a conference paper at ICLR 2026
G STATISTICAL SIGNIFICANCE AND ROBUSTNESS ANALYSIS
To study Statistical Significance of our main experiments, we performed bootstrap resampling with N 000 iterations our test (1,766 pairs). We computed 95% Confidence Intervals (CIs) for the performance differences between models to assess statistical significance.
RL vs. SFT Training We find that RI.-trained judges significantly outperform their SFT coun- terparts. For our primary model comparison (Qwen3-4B), RI. training yields substantial improve- ment over the SFT baseline on both Preference Accuracy and Macro-F1.
Preference Accuracy Gain: +6.51%
95% Cl: [+3.96%, +8.95% (p < 0.001)
Macro-F1 Gain: +0.0589
95% CI: [+0.0381, +0.0799] (p 0.001)
2. Pairwise vs. Pointwise Evaluation We validated the benefits of pairwise evaluation across three diverse models (small dense, MoE, and frontier API) in the zero-shot setting. all cases, the performance gap is massive and highly significant (p < 0.001), with CIs far removed from zero.
Qwen3-4B:
Preff. Acc. Gain: +20.55% (CI: (+17.50%,+23.677
Macro-F1 Gain: +0.0756 (CI: [+0.0575, +0.0946)
Llama4-Scout:
Pref. Acc. Gain: +14.25% (CI: [+11130%,,
Macro-F1 Gain: +0.0531 (CI: [+0.0354, +0.0717)
gpt-4.1:
Pref. Acc. Gain: + +1.50% (CI: [+ (+11.55%, +20.39%)
Macro-F1 Gain: +0.0630 (CE: [+0.0458, +0.0809)
Impact of Grading Rubrics We analyzed the impact of rubrics on grading quality (Macro- F1) in the Pointwise setting. Bootstrapping confirms that rubrics provide a consistent, statistically significant boost to grading performance across models.
Qwen3-4B: Macro-F1 Gain CI [+0.070, +0.105] (p < 0.001)
Llama4-Scout: Macro-F1 Gain CI [+0.028, +0.059] (p 0.001)
gpt-4.1: Macro-F1 Gain CI [+0.037, +0.073]
0.001)
47


Reviews:
[{"review_id": "wsYh2ouwF9", "reviewer": "Reviewer_kRGr", "summary": "This paper introduces MENLO, a framework for evaluating the native-like quality of multilingual LLM responses based on audience-design principles. MENLO defines four core dimensions, Fluency, Tone, Localized Tone, and Localized Factuality and provides localized prompt templates to elicit culturally grounded responses.\nMENLO further contributes a large scale multilingual human evaluation dataset comprising 6,423 prompt-response pairs with 81,014 annotations spanning 47 languages/varieties, achieving strong inter-annotator agreement (avg. Krippendorff’s α = 0.84).\nThe paper empirically demonstrates the superiority of pairwise over pointwise judgments for aligning evaluator preferences, and shows that incorporating structured rubrics improves pointwise reliability.\nAdditionally, the authors explore LLM-Judge training methods, including multi-task reinforcement learning and reward shaping to boost automatic judge models closer to human-level performance.\nFinally, the trained judges are shown to be effective as generative reward models, improving policy models’ multilingual proficiency. However, the study highlights that LLM-based evaluations can overestimate model performance relative to human raters, revealing an important limitation.", "questions": "1. Do the four evaluation dimensions (Fluency, Tone, Localized Tone, Localized Factuality) include moral or behavioral conventions that are specific to certain regions or cultures?\n2. Pairwise evaluation involves comparing two responses at a time. Do the authors plan to conduct multi-sample comparison experiments, examining whether the presence or absence of rubrics affects performance?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Extensive Multilingual Coverage Demonstrates strong value by tackling the critical aspect of native-like linguistic quality, incorporating human validation across a broad set of languages.\nProvides substantial experimental diversity with 47 languages/varieties, 6,423 prompt–response pairs, and 81,014 annotations, ensuring wide multilingual representation.\n- Well-Structured and Systematic EvaluationOffers quantitative evidence that zero-shot pairwise evaluation consistently outperforms pointwise evaluation in both Macro-F1 and preference accuracy.\nEmpirically validates the practical utility of rubric-guided evaluation showing benefits for both absolute and comparative assessments, indicating strong applicability in real-world evaluation pipelines.", "weaknesses": "- MENLO uses localized prompts and employs translators/annotators from the corresponding regions, there still exist diverse linguistic variations within a single locality across social groups, age ranges, and socioeconomic classes. Therefore, constructing gold labels based on a single demographic standard may introduce bias toward a particular perspective.\n- From a user perspective, even if a response appears native-like, its practical utility drops significantly if it is factually incorrect especially with respect to localized factual knowledge. If the reward model does not account for factuality, the system may optimize for style while increasing the risk of conveying incorrect information.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "3M0JOQaTDL", "reviewer": "Reviewer_yzH6", "summary": "This manuscript contributes to LLM-as-Judge by proposing MENLO, building a 6k+ human-annotated dataset, establishing four quality evaluation dimensions, showing RL's impact on LLM-as-judge and the dataset's benefit to multilingual RL, though the RL-enhanced LLM-as-judge still lags behind human annotation.", "questions": "none", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper plans to open-source its dataset. A high-quality training dataset is currently in urgent need by the open-source community.\n- The paper’s dataset primarily contributes to the improvement of LLM-as-Judge — a critical need for the current RL paradigm, and a highly important topic that requires further advancement.", "weaknesses": "The multilingual section of the paper is derived from direct translation, which may lack native-specific traits. For instance, the level of attention paid to art can vary across different countries, and this discrepancy may introduce a certain degree of bias.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "edNRKAo4Hk", "reviewer": "Reviewer_M539", "summary": "This works proposes a dataset and a LLM-judge based evaluation framework to evaluate whether an LMs' response is native-like quality. Specifically, the authors attempt to assess four aspects: (1) language quality/coherence, (2) alignment with cultural/lignuistic nuance, (3) factual correctness, (4) overall writing style and helpfulness. The dataset (Menlo) spans across 47 languages and includes 6423 instances. Then, the authors evaluate different models in point wise and pair wise settings when functioning as a judge to evaluate responses. Also, the authors train a judge model with RL and show that it could match or outperform the performance of Qwen3-32B and GPT-4.1. The authors also show that this judge model could also work as a generative reward model to improve LMs' ability to write fluent responses.", "questions": "1. In Table 2, Menlo is displayed with some benchmarks (e.g., MM-Eval and M-RewardBench), it might be good to specify the number of instances used as benchmarking (1776 instances).\n\n2. The Biggen-bench [1] also includes a multilingual category with 420 human judgments, it would be great if the authors could add it to Table 2.\n\n[1] Kim, S., Suk, J., Cho, J. Y., Longpre, S., Kim, C., Yoon, D., ... & Seo, M. (2025, April). The BiGGen Bench: A Principled Benchmark for Fine-grained Evaluation of Language Models with Language Models. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers) (pp. 5877-5919).", "limitations": "", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 2, "contribution": 4, "strengths": "1. The Menlo dataset was annotated with 1934 annotators and also has high krippendorff's alpha value, indicating that it is a very high-quality dataset.\n\n2. The list of experiments is very extensive: starting from gathering a dataset, training a judge, and then using the judge to train a generator. Every step is supported by a experimental result, which is very helpful for future work in this area.", "weaknesses": "1. Considering that the prompts consisting the Menlo dataset were derived using a prompt template, it could be possible that the prompts are synthetic in nature. There should be an explanation to which extent these prompts mirror what non-English speakers might prompt to frontier models in real-world scenarios. One possible way would be to gather the prompts on WildChat (that are non-English) and measure how similar the two groups are compared to other multilingual meta-eval benchmarks such as M-RewardBench or MM-Eval.\n\n2. The fact that \"pairwise ranking\" yields higher performance compared to \"pointwise\" is somewhat trivial - the caveat should be that it incurs higher inference costs. Yet, the paper simply concludes that pairwise has higher performance compared to pointwise. For a fair comparison, since pointwise is O(N) and pairwise is O(N^2), it is possible to prompt the LLM judge N times and apply either score averaging [1,2] or self-consistency [3].\n\n3. In Table 4, it shows that adding Rubrics help the models to predict better judgments. When looking into the rubrics in Appendix A.3., these are explanations of each criteria, not instance-specific rubrics [3]. I think the term \"context-specific rubrics\" should be adjusted to \"detailed rubrics\" for readers to not confuse with instance-specific rubrics. Also, it would be nice to add a qualitative example of how the chain-of-thoughts generated by the judges differ when including or excluding the rubrics.\n\n4. In Figure3, there are some languages that benefit more and other languages that do not. What is the difference? Is it the limitation of RL training (that means there should be more pretraining for some languages) or is there another reason behind it that the authors conjecture?\n\n5. Looking at the one of the qualitative example between Figure 9-16 (I won't specify for anonymity), I'm not sure if the grades make sense, which makes me a bit skeptical if the large group of annotators were instructed and managed properly. Can you give me details of how these people were recruited, how much they were incentivized, and how they were qualified?\n\n[1] Kim, S., Shin, J., Cho, Y., Jang, J., Longpre, S., Lee, H., ... & Seo, M. (2023, October). Prometheus: Inducing fine-grained evaluation capability in language models. In The Twelfth International Conference on Learning Representations.\n[2] Kim, S., Suk, J., Longpre, S., Lin, B. Y., Shin, J., Welleck, S., ... & Seo, M. (2024, November). Prometheus 2: An Open Source Language Model Specialized in Evaluating Other Language Models. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing (pp. 4334-4353).\n[3] Kim, S., Suk, J., Cho, J. Y., Longpre, S., Kim, C., Yoon, D., ... & Seo, M. (2025, April). The BiGGen Bench: A Principled Benchmark for Fine-grained Evaluation of Language Models with Language Models. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers) (pp. 5877-5919).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "xa2aU8Fkth", "reviewer": "Reviewer_kNrK", "summary": "This paper proposes a framework for evaluating and improving whether large language models generate native-level natural responses across 47 languages. The goal is to evaluate whether models can converse like actual native speakers, beyond mere grammatical accuracy. To this end, the researchers applied the sociolinguistic principle of audience design, which is the concept that speakers adjust their language style according to their addressee or reference group. The framework divides native-level quality into four core dimensions. First, Fluency evaluates accuracy of vocabulary and grammar, consistency of sentence structure, and clarity. Second, Tone judges the overall writing style—whether it is helpful, insightful, engaging, and fair. Third, Localized Tone evaluates alignment with cultural, regional, and linguistic nuances, assessing appropriate use of local expressions and cultural sensitivity. Fourth, Localized Factuality evaluates factual accuracy and completeness, and whether information is provided based on local context. The researchers experimented with whether trained judge models could be used as generative reward models to directly improve the language capabilities of policy models. Using Qwen3-4B-RL-Judge as a reward model to subsequently train the base Qwen3-4B model, all three LLM evaluators (Qwen3-32B, gpt-4.1, Llama4-Scout-RL-Judge) confirmed improvements. Win rates ranged from 63.4% to 77.9%, with average score improvements from +0.80 to +1.16.However, there was an important finding. Human evaluation performed on 10 languages revealed that human evaluators were more conservative with a win rate of 55.7% and average score improvement of +10.9%.", "questions": "I want to express sincere appreciation for the authors' substantial effort in conducting this large-scale, ambitious research. Please consider the following recommendations to strengthen the paper:\n\n< Questions >\nCan you provide a more precise operational definition of 'native-level quality'? Please provide theoretical rationale for whether the four dimensions are sufficient or necessary conditions, and why other important aspects (pragmatics, humor, dialect) were excluded.\n\nThe distinction between Tone and Localized Tone sometimes appears ambiguous (e.g., fairness depends on cultural context). Can you clarify the conceptual boundary between these two dimensions?\n\nHow were reward function coefficients selected (e.g., partial reward +0.5, preference bonus +1, format penalty -0.2)? Did you perform sensitivity analysis? \n\n< Suggestions for Improvement >\nStrengthen Statistical Rigor: Perform and report statistical significance tests for all major claims. Particularly provide p-values, confidence intervals, and effect sizes for SFT vs. RL, pairwise vs. pointwise evaluation, and cross-language comparisons. Use bootstrapping or permutation tests to obtain robust estimates.\n\nClarify Causal Mechanisms: Conduct additional experiments revealing why pairwise evaluation is superior. Include presenting same response twice, systematically varying response order, and analyzing correlation between response quality differences and performance improvements.\n\nImprove Baselines: Add SFT baseline with reasoning to fairly compare with RL. Also include other RL algorithms like DPO or PPO to demonstrate GRPO is best.\n\nCompare with Existing Metrics: Compare MENLO with established multilingual metrics like BLEURT, COMET, BERTScore. Include correlation, relative strengths/weaknesses, and language-specific performance analysis.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "Methodological Rigor: Operationalizing sociolinguistic theory (Bell's Style Axiom and audience design) into a computational evaluation framework is a strong point. This goes beyond simply translated prompts or general naturalness evaluation to capture the actual phenomenon of native speakers adjusting language to specific addressees or reference groups. The decomposition into four dimensions is theoretically motivated, and developing detailed rubrics for each dimension effectively reduced subjectivity.\n\nData Quality and Scale: The MENLO dataset surpasses existing multilingual benchmarks in several aspects. The scale covering 47 language varieties, human-written and localized prompts, and Krippendorff's α of 0.84 are all impressive. Particularly, the fine-grained distinction of language varieties (e.g., European vs. Mexican Spanish, various English varieties) reflects actual linguistic diversity. Deploying expert annotators per language and developing customized annotation tools are also best practices in data quality management.\n\nSystematic Experimentation: Systematic comparisons of pairwise vs. pointwise evaluation, with/without rubrics, and SFT vs. RL clearly show the contribution of each design choice. Particularly, the finding that pairwise evaluation achieves up to 12.4% higher Macro-F1 than pointwise evaluation and is more effective than few-shot examples is significant. Ablation studies on reward shaping also quantify each reward component's contribution, demonstrating the importance of composite reward design in GRPO.\n\nPractical Value: Achieving actual model improvement by using trained judges as reward models demonstrates the framework's practicality. They closed the evaluation-improvement loop, suggesting real applicability beyond theoretical benchmarks. The commitment to releasing datasets and code will also contribute to reproducibility and facilitate follow-up research.\n\nTransparent Limitation Reporting: Honestly reporting the finding that LLM evaluators overestimate improvements by +0.5~0.6 compared to humans demonstrates academic honesty. Acknowledging the difficulty of the Localized Factuality dimension and suggesting alternative approaches (retrieval, tool use) is also positive.", "weaknesses": "Insufficient Causal Verification: There's inadequate causal analysis of why pairwise evaluation is superior. It's not distinguished whether comparison itself is important, whether it's due to context richness from simultaneous exposure to two responses, or whether relative judgment is simply easier than absolute judgment. Without control experiments like pairwise evaluation of a single response (presenting the same response twice), mechanistic understanding is limited.\n\nLack of Statistical Rigor: Most results lack confidence intervals or significance tests. Especially given the high variability in language-specific performance (37.9%~82.1%), it's difficult to judge whether observed differences are statistically significant. Multiple comparisons across several models and settings were performed, so multiple testing corrections like Bonferroni are needed but not applied. Additionally, an average of 137 prompts per language may lack statistical power for some languages.\n\nMissing Comparative Experiments: There's no comparison with existing multilingual quality metrics (BLEURT, COMET, BERTScore), making it difficult to judge MENLO's relative advantages. Also, there's no comparison with other RL algorithms (PPO, DPO, RLAIF) or reward designs, so we cannot know if the proposed GRPO+composite reward is optimal. Human evaluation was only performed on 10 languages, limiting generalization to all 47 languages.\n\nMethodological Bias: The SFT baseline may be unfairly weak. Training without Chain-of-Thought doesn't provide fair comparison with RL, and SFT with reasoning should also be included. Additionally, justification for model selection (Qwen3-4B, Llama4-Scout) is lacking, and result generalization to other sizes or architectures is uncertain.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)