MENLO: From Preferences to Proficiency – Evaluating and Modeling Native-like Quality Across 47 Languages

## Abstract

Abstract Ensuring native-like quality of large language model (LLM) responses across many languages is challenging. To address this, we introduce Menlo , a framework that operationalizes the evaluation of native-like response quality based on audience design-inspired mechanisms. Using Menlo , we create a dataset of 6,423 human-annotated prompt–response preference pairs covering four quality dimensions with high inter-annotator agreement in 47 language varieties.
Our evaluation reveals that zero-shot LLM judges benefit significantly from pairwise evaluation and our structured annotation rubrics, yet they still underperform human annotators on our dataset. We demonstrate substantial improvements through fine-tuning with reinforcement learning, reward shaping, and multi-task learning approaches. Additionally, we show that RL-trained judges can serve as generative reward models to enhance LLMs’ multilingual proficiency, though discrepancies with human judgment remain. Our findings suggest promising directions for scalable multilingual evaluation and preference alignment. We release our dataset and evaluation framework to support further research in multilingual LLM evaluation. Dataset https://huggingface.co/datasets/facebook/menlo

## 1 Introduction

In order for LLMs to be most useful across the globe, they need to be able to provide high-quality responses in many languages. Responses should be relevant (Zhuang et al., 2024), factually accurate (Jacovi et al., 2025), and natural (Marchisio et al., 2024; Guo et al., 2025), among other considerations. Ultimately, for interaction in any language to be seamless, responses need to be indistinguishable from those of a native speaker (Novikova et al., 2016; Liu et al., 2021).
Language proficiency in humans has traditionally been evaluated via standardized tests (Jamieson et al., 2000). While such tests have been applied to evaluating LLMs (Anil et al., 2023; Mayor-Rocher et al., 2024; Lothritz & Cabot, 2025), they are difficult to scale and do not readily correspond to real-world conversations.
What is considered a native-like response
largely depends on speakers’ and listeners’ interpretations of whom they are speaking to (Bell, 1984).

To operationalize the evaluation of native-like response quality across languages, we propose Multilingual Evaluation of Native-Like Output, the Menlo framework; see [Figure 1](https://arxiv.org/html/2509.26601v1#S1.F1) for an overview. Menlo breaks down native-like response quality into four key dimensions: i) language quality and coherence; ii) alignment with cultural and linguistic nuances of a specific language variety or locale; iii) factual correctness and grounding in the local context; and iv) overall writing style and helpfulness.

Building on mechanisms from audience design (Bell, 1984), we propose creating tailored prompts that effectively evoke local contexts by defining the target audience (e.g., an addressee or reference group), thereby guiding the generated language to converge to contextually appropriate “native” styles.
We develop instructions that reduce annotation subjectivity and improve inter-annotator agreement. Responses are generated using state-of-the-art LLMs and annotated with ratings on a 1–5 Likert scale, with an average Krippendorff’s $\alpha=0.84$.
Overall, the Menlo dataset consists of 6,423 annotated prompt-response preference pairs, and 81,014 annotations, covering 47 language varieties.

Human evaluation, particularly at a massively multilingual scale is expensive. We thus evaluate the ability of LLMs to serve as judges of native-like quality responses. We find that in zero-shot setting, pairwise evaluation—where models predict scores for two responses simultaneously (without explicitly predicting preference)—significantly outperforms its pointwise counterpart.
The advantage of evaluating two responses side-by-side is even bigger than in-context examples with labels. In addition, we observe significant improvements with judges using our annotation rubrics compared to judges without rubrics, highlighting the generality of the Menlo framework.

Figure: Figure 1: Menlo framework and annotation process. 1) Human-written prompt templates evoking local contexts are created in English for the four dimensions. 2) Prompt templates are translated and localized into 47 language varieties. 3) Annotation guidelines are created that break down each dimension into easy-to-follow rubrics. 4) LLMs are used to generate response pairs for each prompt, which are annotated with Likert-scale ratings and preferences.
Refer to caption: figures/MENLO_graph.png

As zero-shot judges remain below human annotation quality, using the pairwise evaluation setup, we fine-tune Qwen3-4B and Llama4-Scout as LLM judges on the Menlo training data, finding that RL-trained models outperform their SFT counterparts. In particular, a multi-task Llama4-Scout model trained with shaped rewards surpasses frontier API models with the strongest overall performance across 47 language varieties, reaching agreement levels comparable to human annotators.

Finally, we demonstrate that these judges can be used as generative reward models (RMs) to directly improve a policy model’s proficiency. By using our pairwise RL-trained Qwen3-4B judge to post-train the base Qwen3-4B model, we observe quality gains as measured by both LLM evaluators and human raters. However, LLM evaluators tend to be overconfident in assessing improvements compared to human judgments ($+0.6$ higher gain). This finding shows that while judges trained with our framework can successfully drive model improvements, the gap between LLM and human raters highlights the remaining challenges in reliably modeling native-like quality across languages.

Our contributions are the following: 1) We develop Menlo, a framework for the evaluation of native-like response quality in four dimensions, based on principles from audience design, employing parametric templates and carefully crafted annotation guidelines. 2) We create the Menlo dataset, consisting of 6,423 annotated prompt-response preference pairs in 47 language varieties. 3) We evaluate zero-shot judges on the annotated data, demonstrating the benefits of pairwise evaluation and rubrics. 4) We show that multi-task RL and reward shaping enables fine-tuning a judge that is on par with human annotators in 47 language varieties. 5) We demonstrate that pairwise fine-tuned judges can be used as generative RMs to improve policy model language proficiency, while we find that LLM evaluations tend to overestimate improvements compared to human raters.

Our framework unifies the Menlo dataset, RL-trained pairwise judging, and generative reward modeling, offering a practical and scalable approach to both assess and improve native-like quality.

## 2 The Menlo Dataset

Menlo characterizes native-like conversational response quality in a language along four key dimensions: fluency, tone, localized tone, and localized factuality. These dimensions go beyond prior work that focused mainly on naturalness (Novikova et al., 2016; Liu et al., 2021; Guo et al., 2025) and are motivated by work on language proficiency assessment (Ke & Ng, 2019), cross-cultural variation (Hershcovich et al., 2022; Myung et al., 2024), local knowledge grounding (Hupkes & Bogoychev, 2025). We provide further context on our definition of these dimensions in [Figure 2](https://arxiv.org/html/2509.26601v1#S2.F2).

From a sociolinguistic perspective, the Style Axiom (Bell, 1984) states that intraspeaker variation (style) reflects interspeaker variation (social). Native-like quality is therefore not a single fixed target but a socially conditioned range of stylistic choices that depend on interlocutors. Key mechanisms include accommodation, where speakers adapt their style to the addressee, and referee design, where speakers align with an absent reference group they wish to identify with. These mechanisms motivate our focus on tone and localized tone as central to native-like quality.

Figure: Figure 2: Dimensions of native-like response quality in Menlo and example prompt (template).
Refer to caption: x2.png

To operationalize these ideas, we design human-written parametric English prompt templates for each dimension with placeholders such as [locale_nationality], [locale_country], [locale_holiday], etc. By defining the addressee or reference group, these prompts evoke local contexts and guide models toward contextually appropriate “native” styles. We provide an overview of the Menlo framework and annotation process in [Figure 1](https://arxiv.org/html/2509.26601v1#S1.F1).

We select 47 language varieties representing a typologically diverse set of widely used languages and their major variants, including, e.g., South American and European varieties of Spanish and Portuguese, several varieties of English, and romanized versions of non-Latin script languages (see Appendix [A.2](https://arxiv.org/html/2509.26601v1#A1.SS2)). Native speakers are recruited to professionally translate these prompt templates, with placeholders instantiated using locally relevant entities. As native quality is tied to the local context, we ensure that native speakers are from the specific regions where the corresponding language varieties are spoken. Similar criteria are used to select annotators for each language variety. Each language variety has approximately the same number of examples in Menlo.

**Table 1: Annotation and statistics of Menlo across evaluation dimensions. IAA presents Krippendorff’s $\alpha$ measuring inter-annotator agreement. Average token counts are computed using Qwen3-4B.**
| Dimension | # Annotations | # Annotators | # Prompts | Avg # Tokens | Rating (1–5 Scale) |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Prompt | Response | IAA | Mean | Std. |  |  |  |  |
| Fluency | 23,556 | 450 | 1,820 | 81.6 | 804.3 | 0.82 | 4.01 | 1.11 |
| Tone | 18,712 | 429 | 1,410 | 27.8 | 575.3 | 0.86 | 3.48 | 1.35 |
| Localized Tone | 22,324 | 530 | 1,815 | 71.7 | 559.2 | 0.83 | 3.89 | 1.17 |
| Localized Factuality | 16,422 | 525 | 1,378 | 121.8 | 839.1 | 0.84 | 3.82 | 1.16 |
| Overall | 81,014 | 1,934 | 6,423 | 75.6 | 692.2 | 0.84 | 3.80 | 1.20 |

To ensure consistency in evaluation, we develop instructions that reduce the subjectivity of the annotation and break down the four broad dimensions into easy-to-follow rubrics and self-explanatory signals (human-written). Annotators receive guidelines with examples for each dimension. We additionally develop a customized annotation tool and annotator screening tests to filter out unreliable annotators. Furthermore, we train 1–2 expert annotators per language who provide language-specific feedback to annotators and provide gold annotations on a subset of examples.

We generate two responses for each prompt with state-of-the-art LLMs including GPT-4o, Llama4-Maverick, Llama4-Maverick with Search, and Gemini 1.5 with Search. We present both responses in randomized order to human annotators and ask them to provide 1–5 Likert ratings per response, allowing ties. Each response pair is annotated by at least 3 annotators, with final scores aggregated via majority vote. Annotators achieve high reliability, with an average Krippendorff’s $\alpha=0.84$.

**Table 2: Comparison of Menlo with other multilingual response quality datasets: Recon (Doddapaneni et al., 2025), Pariksha (Watts et al., 2024), M-RewardBench (Gureja et al., 2025), MM-Eval (Son et al., 2024). $|\mathcal{L}|$: # of languages, $|\mathcal{D}|$: # of prompts, IAA: inter-annotator agreement, IF: instruction following.**
| Dataset | $|\mathcal{L}|$ | 00$|\mathcal{D}|$ | IAA | Dimensions | Prompts | Responses | Ratings |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Menlo | 47 | 06,423 | 0.84 | Fluency, tone, localized tone, localized factuality | Human-written, translated & localized | Annotated in<br>each language | Preference & 1–5 |
| Pariksha | 10 | 00,200 | 0.54 | Hallucinations, task quality, linguistic acceptability | Human-written | Annotated in<br>each language | Preference & 0–2 |
| Recon | .6 | 03,000 | 0.– | IF, theory of mind, reasoning, safety, planning, etc. | Translated | Generated in each language | Preference & 1–5 |
| MM-Eval | 18 | 04,981 | 0.– | Reasoning, chat, linguistics, hallucination, safety | Translated | Generated in each language | Preference |
| M-Reward<br>Bench | 23 | 66,787 | 0.– | Chat, safety, reasoning, translation | Translated | Translated | Preference |

Overall, Menlo consists of 6,423 annotated prompt–response preference pairs across 47 language varieties, each containing a prompt, two responses, and corresponding scores, totaling 81,014 human annotations. Summary statistics are reported in [Table 1](https://arxiv.org/html/2509.26601v1#S2.T1).
Example prompts for each dimension are shown in [Figure 2](https://arxiv.org/html/2509.26601v1#S2.F2). Further details of Menlo including annotation process, language coverage, rubrics, and full examples featuring responses and their corresponding ratings are provided in [Appendix A](https://arxiv.org/html/2509.26601v1#A1).

[Table 2](https://arxiv.org/html/2509.26601v1#S2.T2) compares Menlo with existing multilingual preference evaluation datasets. Notably, the Menlo dataset is the dataset with the largest language coverage and the first that focuses on native-like quality of LLM responses beyond linguistic acceptability. Compared to prior work
that relies on English-centric prompts and translated responses or achieves moderate inter-annotator agreement,
Menlo provides localized prompts and responses, spans more languages, and reaches higher agreement.

## 3 Evaluating LLM-Judges on Menlo

We next evaluate the ability of LLMs to serve as automatic judges of native-like quality on Menlo.
Out of the 6,423 pairs, we hold out 1,766 pairs (3,552 responses) as the test set,(^2^22Translations of the same prompt template are assigned the same set to prevent train-test leakage.) and use the remainder for training and prompt development (see §[4](https://arxiv.org/html/2509.26601v1#S4) and §[5](https://arxiv.org/html/2509.26601v1#S5)). Where expert annotations are available, we use these as labels. For the remaining responses, we average the annotated ratings of each response.(^3^33Multiple annotations can be used in future work on pluralistic alignment (Sorensen et al., 2024).)
Our evaluation focuses on three questions:
(i) how pointwise and pairwise setups compare,
(ii) the effect of few-shot exemplars, and
(iii) the role of explicit grading rubrics.

We benchmark a range of open-source and API-based models, covering both *thinking* and *non-thinking* variants:
Qwen3-4B, Qwen3-32B, Llama-3.1-8B,(^4^44Llama models are instruction-tuned and we omit the Instruct suffix for brevity.)
Llama-3.3-70B, Llama4-Scout, o3, gpt-4o, and gpt-4.1.
All models are used in the default setup with maximum output length $8192$.

We report two primary metrics:
(i) Macro-F1 for 5-way classification, and
(ii) Preference accuracy over Win/Loss/Tie outcomes. Note that we do NOT directly ask for preference judgments; rather, we infer these from the assigned grades. Additionally, we report classification accuracy, Krippendorff’s $\alpha$, which measures agreement with human annotators while accounting for chance agreement and missing data, and provide detailed per-dimension and per-language breakdowns in [Appendix D](https://arxiv.org/html/2509.26601v1#A4).

### 3.1 Pointwise vs. Pairwise

**Table 3: Zero-shot and few-shot results of open-source and API models on the Menlo test set using Pointwise (grading single responses) and Pairwise (grading response pairs) scoring (see §[3.1](https://arxiv.org/html/2509.26601v1#S3.SS1)). Macro-F1 shows 5-way classification performance and Preference reports accuracy on Win/Loss/Tie. Reported gains/loss are relative to zero-shot pointwise performance.**
| Models | Macro F1 | Preference Accuracy |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
| zero-shot | few-shot | zero-shot | zero-shot | few-shot | zero-shot |  |
| Pointwise | Pointwise | Pairwise | Pointwise | Pointwise | Pairwise |  |
| Qwen3-4B | 23.06 | 31.18 +8.12 | 35.46 +12.40 | 40.54 | 39.35 -1.19 | 57.13 +16.57 |
| Qwen3-32B | 28.53 | 35.45 +6.92 | 37.48 +8.95 | 42.19 | 42.87 +0.68 | 59.12 +16.59 |
| Llama-3.1-8B | 22.27 | 23.29 +1.02 | 29.46 +7.19 | 39.92 | 37.15 -2.77 | 50.45 +10.48 |
| Llama-3.3-70B | 27.93 | 30.52 +2.59 | 37.50 +9.57 | 37.37 | 38.56 +1.19 | 55.32 +17.89 |
| Llama4-Scout | 25.63 | 32.84 +7.21 | 36.11 +10.48 | 42.19 | 41.22 -0.97 | 56.25 +14.12 |
| o3 | 26.54 | 27.92 +1.38 | 35.35 +8.81 | 45.07 | 44.68 -0.39 | 58.72 +13.68 |
| gpt-4o | 25.99 | 29.57 +3.58 | 37.57 +11.58 | 42.92 | 45.87 +2.95 | 57.98 +15.09 |
| gpt-4.1 | 32.23 | 33.84 +1.61 | 38.53 +6.30 | 41.73 | 44.00 +2.27 | 59.23 +17.50 |

Although Menlo provides paired responses for each prompt, the presence of detailed grading rubrics means that *pointwise* evaluation is in principle sufficient: a model could assign absolute scores to individual responses without needing comparisons.
However, pairwise setups may provide stronger relative signals by anchoring judgments against another candidate.
We therefore compare three setups:
Zero-shot pointwise: the model is given a prompt, a single response, and a detailed 5-point grading rubric, and asked to generate evaluation reasoning (in thinking) and assign a final grade;
Few-shot pointwise: we additionally provide three graded examples: one from 1–2, one with a grade of 3, and one from 4–5;
Zero-shot pairwise: the model is presented with both responses to the same prompt and asked to assign a grade to each, following the template in [Figure 17](https://arxiv.org/html/2509.26601v1#A2.F17) ([Appendix B](https://arxiv.org/html/2509.26601v1#A2)), without constraints on ties.
The order of the two responses is randomized.

[Table 3](https://arxiv.org/html/2509.26601v1#S3.T3) reports Macro-F1 and Preference results. Zero-shot pairwise consistently outperforms both zero-shot and few-shot pointwise scoring across models, with gains of up to $+12.4\%$ in Macro-F1 and $+18.0\%$ in Preference accuracy over zero-shot pointwise. Few-shot pointwise improves Macro-F1 relative to zero-shot pointwise but yields only marginal gains in Preference, still falling short of zero-shot pairwise by an average of $-5.5\%$ in Macro-F1 and $-15.1\%$ in Preference across models.

These results indicate that models are substantially more reliable at assigning scores when evaluating two responses side by side, even without ground-truth labels. The unexpectedly large gains over few-shot in-context examples highlight pairwise evaluation, which explicitly anchors outputs against a competing candidate (Wang et al., 2025), as a promising direction for improving automated judging reliability, even when the ultimate goal is pointwise scoring. We also evaluate few-shot pairwise on Qwen3-4B, observing only a small gain in Macro-F1 ($+0.6$) relative to zero-shot pairwise, further supporting our findings. Future work may investigate whether extending pairwise comparisons to a listwise evaluation of multiple responses offers additional benefits.

### 3.2 With and without Grading Rubrics

**Table 4: Zero-shot performance comparing without and with detailed 5-Point Grading Rubrics.**
| Models | Macro F1 | Preference Accuracy |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Pointwise | Pairwise | Pointwise | Pairwise |  |  |  |  |  |
| wo/ Rubrics | w/ Rubrics | wo/ Rubrics | w/ Rubrics | wo/ Rubrics | w/ Rubrics | wo/ Rubrics | w/ Rubrics |  |
| Qwen3-4B | 16.00 | 23.06 +7.06 | 32.74 | 35.46 +2.72 | 33.52 | 40.54 +7.02 | 54.08 | 57.13 +3.05 |
| Qwen3-32B | 25.59 | 28.53 +2.94 | 38.10 | 37.48 -0.62 | 43.32 | 42.19 -1.13 | 59.23 | 59.12 -0.11 |
| Llama-3.1-8B | 21.50 | 22.27 +0.77 | 30.89 | 29.46 -1.43 | 38.34 | 39.92 +1.58 | 49.55 | 50.45 +0.90 |
| Llama-3.3-70B | 22.71 | 27.93 +5.22 | 35.12 | 37.50 +2.38 | 34.54 | 37.37 +2.83 | 56.29 | 55.32 -0.97 |
| Llama4-Scout | 22.15 | 25.63 +3.48 | 35.21 | 36.11 +0.90 | 41.28 | 42.19 +0.91 | 55.10 | 56.25 +1.15 |
| o3 | 25.43 | 26.54 +1.11 | 35.60 | 35.35 -0.25 | 45.13 | 45.07 -0.06 | 57.98 | 58.72 +0.74 |
| gpt-4o | 22.45 | 25.99 +3.54 | 36.74 | 37.57 +0.83 | 37.60 | 42.92 +5.32 | 56.85 | 57.98 +1.13 |
| gpt-4.1 | 22.26 | 32.23 +9.97 | 37.35 | 38.53 +1.18 | 38.67 | 41.73 +3.06 | 56.96 | 59.23 +2.27 |

We further examine the role of detailed grading rubrics in judge performance. All rubrics are human-written 5-point guidelines specific to the dimension and question type of each prompt. Examples of dimension-specific rubrics are shown in Appendix [A.3](https://arxiv.org/html/2509.26601v1#A1.SS3).

[Table 4](https://arxiv.org/html/2509.26601v1#S3.T4) compares zero-shot pointwise and pairwise performance with and without access to rubrics. The latter shows only the five class labels, without accompanying criteria or definitions.
Results show that rubrics provide a substantial benefit, especially for pointwise evaluation, yielding average gains of $+4.3\%$ in Macro-F1 and $+2.5\%$ in Preference accuracy. In contrast, pairwise evaluation benefits more modestly, with improvements of roughly $+1\%$ on both metrics.

These findings suggest that judges perform better when grounded, either by explicit rubrics or by comparison with another response. Since pairwise comparison itself offers a strong grounding signal, it sees limited impact from rubrics. This highlights the importance of high-quality rubrics: if judges could automatically generate and evaluate high-quality, context-specific rubrics, we hypothesize that the performance gap between pairwise and pointwise evaluation would further narrow.

## 4 Training LLM-Judges on Menlo

Having established in §[3](https://arxiv.org/html/2509.26601v1#S3) that pairwise evaluation yields substantial advantages over pointwise scoring, we next examine whether training LLMs as judges can further close the gap to human annotators. We train on the Menlo training split (total 4,675 response pairs, where 232 pairs are held out for validation) and explore different learning strategies, model families, and reward designs. Inspired by the success of recent reasoning-based judges such as J1 (Whitehouse et al., 2025), we compare supervised fine-tuning (SFT) and reinforcement learning (RL), as well as single-task (dimension-specific) and multi-task (all dimensions) training.

We fine-tune two contrasting models: Qwen3-4B (dense, reasoning-oriented) and Llama4-Scout (Mixture-of-Experts, non-reasoning), which differ in architecture and cognitive approach.
For SFT, models directly predict 5-point grades using cross-entropy loss under teacher forcing, without intermediate reasoning generation. For RL, we use GRPO (Shao et al., 2024) with the template from [Figure 17](https://arxiv.org/html/2509.26601v1#A2.F17), encouraging step-by-step reasoning before score assignment. Following Whitehouse et al. (2025), we augment training data by including both response orders (A,B) and (B,A) to mitigate positional bias. Training details are provided in Appendix [C.1](https://arxiv.org/html/2509.26601v1#A3.SS1).

### 4.1 Reward Designs for RL

To make RL training effective, we design a composite reward signal that combines absolute accuracy with relative preference alignment and robustness to near-miss predictions:
(i) Pointwise binary reward: $+1$ if the predicted score matches the gold label, 0 otherwise.
(ii) Reward smoothing: partial reward ($+0.5$) if the prediction differs by exactly one grade.
(iii) Preference bonus: additional $+1$ if the *sign* of the difference between the two predicted scores matches the label.
(iv) Penalties: $-1$ for invalid or missing scores, and $-0.2$ for formatting violations, i.e. each tag must appear in the correct order and only once.

All reward components are summed to produce the final RL signal. Formally, the reward can be expressed as follows, where $s$ and $gt$ represent predicted and ground truth grades, respectively:

$$ $R=\sum_{i\in\{A,B\}}\underbrace{\max\Big(\mathbf{1}[s_{i}=gt_{i}],\ 0.5\cdot\mathbf{1}[|s_{i}-gt_{i}|=1]\Big)}_{\text{pointwise binary reward w/ reward smoothing}}+\underbrace{\mathbf{1}[\text{sign}(s_{A}-s_{B})=\text{sign}(gt_{A}-gt_{B})]}_{\text{preference bonus}}$ $$

$$ $-\underbrace{\mathbf{1}[\text{failed extraction}]}_{\text{extraction penalty}}-\underbrace{0.2\cdot\mathbf{1}[\text{formatting violation}]}_{\text{format penalty}}.$ $$

**Table 5: Pairwise SFT, RL, and SFT+RL-trained Qwen3-4B and Llama4-Scout results. RL-trained models perform best overall.**
| Pairwise | Qwen3-4B | Llama4-Scout |  |  |
| --- | --- | --- | --- | --- |
| Marco-F1 | Preference | Marco-F1 | Preference |  |
| Zero-shot | 35.46 | 57.13 | 36.11 | 56.25 |
| SFT | 33.44 -2.02 | 53.68 -3.45 | 44.17 +8.06 | 60.08 +3.83 |
| RL | 39.44 +3.98 | 60.02 +2.89 | 45.62 +9.51 | 62.60 +6.35 |
| SFT + RL | 39.33 +3.87 | 58.78 +1.65 | 45.82 +9.71 | 61.10 +4.85 |

### 4.2 Overall Performance: SFT vs. RL

We first compare the overall performance of SFT and RL-trained models. [Table 6](https://arxiv.org/html/2509.26601v1#S4.T6) shows that RL-trained Qwen3-4B and Llama4-Scout consistently outperform their SFT counterparts.
For inherently thinking models like Qwen3-4B, SFT without Chain-of-Thought (CoT) reasoning actually hurts performance, causing a $-2.0\%$ drop in Macro-F1 and $-3.5\%$ in Preference accuracy. In contrast, RL, which incentivizes reasoning, improves performance by $+4.0\%$ in Macro-F1 and $+2.9\%$ in Preference, surpassing the best frontier API model gpt-4.1.

For non-thinking models like Llama4-Scout, SFT already provides substantial gains ($+8.1\%$ in Macro-F1 and $+3.8\%$ in Preference) compared to zero-shot. RL training further improves results, particularly in Preference ($+2.5\%$). This demonstrates the promise of pairwise RL training across model families, scales, and reasoning capabilities.

We also experimented with initializing RL from the best SFT checkpoint, but observed little or no improvement over starting RL from scratch. Models trained on SFT without CoT tend to copy the placeholder “<think> Your analysis and reasoning here. </think>”
from the prompt rather than generating meaningful reasoning, which limits the benefit of RL. This suggests that for tasks requiring reasoning, it is preferable to start RL directly when the SFT target lacks CoT supervision.

### 4.3 Ablation of RL Rewards

In [Table 6](https://arxiv.org/html/2509.26601v1#S4.T6), we ablate the RL reward design to validate the contribution of each reward component in RL training:
(i) binary only: reward $+1$ for exact score match, 0 otherwise;
(ii) binary+smooth.: adds partial reward for near-miss scores, no preference bonus;
(iii) binary+prefer.: includes preference reward, no smoothing; and
(iv) binary+smooth.+prefer.: the default reward design in §[4.1](https://arxiv.org/html/2509.26601v1#S4.SS1).
Results show clear benefits from combining reward smoothing and preference bonus, achieving the best overall Macro-F1 and Preference accuracy for Qwen3-4B, achieving $+2.3$ boost on Macro-F1 and $+1.7$ on preference accuracy over the binary only reward.

### 4.4 Per-Dimension Performance and Single vs. Multi-Task

Next, we compare pairwise RL-trained Qwen3-4B models trained jointly across all dimensions versus individually per dimension.
Across the four dimensions, Tone achieves the strongest performance, with a Macro-F1 of 43.1 in the zero-shot setting and gains of up to $+3.8$ with multitask RL. Localized Tone and Fluency follow, reaching $32.8$ and $32.2$ Macro-F1 in zero-shot, and up to $+5.7$ improvement when trained with multitask RL. In constrast, Localized Factuality lags behind the other dimensions, achieving only $22.5$ Macro-F1 in zero-shot, a trend consistent across all models. Moreover, RL yields limited benefit ($+0.6$ in single-task RL) or even regressions in the multitask setup. These results highlight the challenge of localized factuality and suggest that alternative strategies, such as incorporating retrieval, search, or external tool use, may be necessary. Full results are provided in [Table 20](https://arxiv.org/html/2509.26601v1#A4.T20) in Appendix [D](https://arxiv.org/html/2509.26601v1#A4).

Overall, aside from Localized Factuality, joint multi-dimension training performs on par with single-dimension optimization while offering greater efficiency and practical benefits, such as serving as a reward model for post-training, which we explore in §[5](https://arxiv.org/html/2509.26601v1#S5).

### 4.5 Cross-Language Performance

[Figure 3](https://arxiv.org/html/2509.26601v1#S4.F3) shows Preference accuracy per language variety for RL-trained Qwen3-4B. Performance varies widely, with tr_TR at $82.1\%$ and bn_BD at $37.9\%$, and does not strictly align with high- vs. low-resource languages. Relative to the zero-shot baseline, en_AU and fr_FR achieve the largest Macro-F1 gains ($+20.9\%$, $+17.7\%$) and ro_RO and gu_IN the largest Preference accuracy gains ($+18.0\%$, $+16.2\%$). By contrast, es_ES drops $-15.4\%$ in Preference despite a modest $+2.2\%$ Macro-F1 gain, whereas en_MX, the same language but a different locale, sees $+2.6\%$ and $+9.8\%$ gains in Preference and Macro-F1 , highlighting that our dataset captures language variety nuances.

We further trained RL using only English data and evaluated on all languages. Performance degrades compared to the baseline, indicating that English-only training is insufficient to generalize across all 47 language varieties. Detailed per-language variety performance is provided in Appendix [D.2](https://arxiv.org/html/2509.26601v1#A4.SS2).

Figure: Figure 3: Preference Accuracy per Language of pairwise RL-trained Qwen3-4B.
Refer to caption: x3.png

## 5 From LLM-Judges to Reward Models

We next investigate whether the RL-trained pairwise judges developed in our framework can also serve as generative reward models to directly improve LLM native-like response quality, unifying evaluation and optimization in a single framework.

### 5.1 RL with Judges as Generative Reward Models

For efficiency, we focus on smaller models for these experiments: Qwen3-4B as the policy model, and Qwen3-4B-RL-Judge as the reward model (RM). Since Localized Factuality remains challenging for our judges, we restrict both training and evaluation to Fluency, Tone, and Localized Tone. Specifically, we exclude Localized Factuality from all training and test prompts, randomly sample 3,000 prompts from Menlo for training, and retain all 1,398 test prompts from Menlo for evaluation across the three selected dimensions.

We post-train Qwen3-4B with GRPO.
We sample 8 rollouts per prompt and compute rewards as follows:
for each prompt, we construct response pairs from the rollouts,
format them with the same pairwise evaluation template, and feed them to the RM.
The final reward of each rollout is obtained by averaging its scores across all paired comparisons. Training details are added in Appendix [C.2](https://arxiv.org/html/2509.26601v1#A3.SS2).

### 5.2 Two-Stage Evaluation Strategy

To rigorously evaluate the policy model’s native-like quality improvements, we employ a two-stage validation approach: (i) comprehensive automated evaluation across all 47 language varieties using three diverse LLM judges, and (ii) human validation on a strategically selected subset of 10 high-resource languages where we can ensure annotation quality.

For each test prompt, we generate responses from both the baseline (Qwen3-4B) and post-trained (Post-train) models, construct response pairs with randomized order to mitigate positional bias, and apply the same pairwise judge template used in training.

#### LLM-Judges Evaluation

We select three high-performing judges Qwen3-32B, gpt-4.1, and Llama4-Scout-RL-Judge (see §[3](https://arxiv.org/html/2509.26601v1#S3)), and compute win, loss, and tie rates between baseline and post-trained models, along with average scores on a 1–5 scale across all 1,398 test prompts spanning 47 language varieties. Qwen3-4B-RL-Judge is excluded from evaluation to avoid potential bias, since it serves as the RM.
[Table 7](https://arxiv.org/html/2509.26601v1#S5.T7) shows that the post-trained policy model consistently outperforms the baseline across all LLM judges and languages. Average score improvements range from $+0.80$ to $+1.16$, with win rates between 63.4% and 77.9%.

Per-dimension analysis reveals consistent gains across evaluation criteria: Tone yields the largest improvement ($+1.04$
average score boost), followed by Localized Tone and Fluency ($+0.89$ each). The consistency of improvements across different judge architectures and all three dimensions provides strong evidence for the effectiveness of our reward modeling approach.

**Table 7: Two-stage Evaluation of Qwen3-4B and its RL post-trained variant Post-train on the Menlo test set, where both models serve as response models.**
| Judges/Raters | # Languages | Win Rate | Average Score (1–5) |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Post-train Win | Post-train Loss | Tie | Qwen3-4B | Post-train | $\Delta Sore$ | Improvement% |  |  |
| Llama4-Scout-RL-Judge | 47 | 63.88% | 09.16% | 26.96% | 3.01 | 3.79 | $+0.78$ | $+25.9\%$ |
| Qwen3-32B | 47 | 72.46% | 21.89% | 06.65% | 3.44 | 4.29 | $+0.85$ | $+24.7\%$ |
| gpt-4.1 | 47 | 77.90% | 11.30% | 10.80% | 3.21 | 4.37 | $+1.16$ | $+36.1\%$ |
| Llama4-Scout-RL-Judge | 10 | 69.66% | 09.64 % | 20.70% | 3.36 | 4.22 | $+0.86$ | $+25.6\%$ |
| Human Raters | 10 | 55.71% | 35.20% | 09.09% | 3.31 | 3.67 | $+0.36$ | $+10.9\%$ |

#### Human Validation

To anchor our automated evaluation results, we conduct human evaluation on a diverse subset of 10 higher-resource languages: ar, de_DE, en_US, fr_FR, hi_IN, hi_Latn_IN, pt_BR, tl_PH, th_TH, vi_VN. This subset spans multiple language families, scripts, and geographic regions while ensuring access to qualified native speaker annotators. Human evaluation follows the same pairwise annotation guidelines as in Menlo construction.

Results (last row of [Table 7](https://arxiv.org/html/2509.26601v1#S5.T7)) on the subset confirms the automated evaluation trends. The post-trained model achieves a win rate of $55.7\%$ against the baseline, with an average score improvement of $+10.9\%$. While both automated and human evaluators agree that post-training improves response quality, we observe that LLM judges tend to overestimate the magnitude of improvement compared to human raters.
Comparing human evaluations to the closest-performing automated judge (Llama4-Scout-RL-Judge) on this subset reveals systematic differences: the automated judge reports an average improvement of $+0.5$ higher than humans. We hypothesize that this discrepancy arises because the automated judges may lean towards a stylistic caricature of native-like quality, overestimating improvements relative to nuanced human judgments. In addition, RL-trained judges exhibit less of this discrepancy among LLM evaluators, confirming the benefits of our judge training.

Overall, our two-stage evaluation demonstrates the potential of RL-trained judges as generative reward models for aligning multilingual outputs toward native-like quality. The directional consistency observed across both LLM- and human-based evaluations validates the viability of our unified framework for multilingual proficiency alignment. However, we note that challenges remain: LLM judges tend to overestimate the magnitude of improvements relative to human raters, highlighting an important direction for future work.

## 6 Related Work

##### Multilingual Evaluation

Models’ multilingual proficiency has been typically measured as an aggregate of performance across multiple task-oriented evaluations of short-form responses in settings with verifiable answers (Hu et al., 2020; Ruder et al., 2021; Doddapaneni et al., 2023; Ahuja et al., 2023; 2024). Recent benchmarks focused on the evaluation of model’s cultural knowledge in a similar verifiable setting (Myung et al., 2024; Chiu et al., 2025; Fabbri et al., 2025). However, such evaluations do not extend to real-world conversations containing long-form responses. Benchmarks evaluating long-form responses use prompts and responses translated from English (Son et al., 2024; Liu et al., 2024; Doddapaneni et al., 2025; Gureja et al., 2025). These evaluations typically do not reflect more localized aspects of language quality and are biased towards translationese. Son et al. (2024) and Doddapaneni et al. (2025) automatically generate ‘good’ and ‘bad’ responses for each dimension. Marchisio et al. (2024) and Guo et al. (2025) evaluate language consistency and naturalness respectively in relatively narrow settings. Pariksha (Watts et al., 2024) is the most similar dataset to ours as it uses human-written prompts and human-annotated responses, but focuses on 10 Indic languages, annotates only high-level dimensions, and reports moderate inter-annotator agreement. Menlo is the only dataset that focuses on native-like quality in real-world conversations.

##### Multilingual Judges and RMs

LLMs have been used as judges in different multilingual benchmarks (Liu et al., 2024; Fabbri et al., 2025). However, fewer works focus on analyzing or improving multilingual judges and RMs. Gureja et al. (2025) observe that zero-shot judges show a substantial gap between the translated M-RewardBench and its English counterpart, with predictions inconsistent across languages. Fu & Liu (2025) report similar inconsistencies across five diverse tasks. Wu et al. (2024) evaluate zero-shot cross-lingual transfer of trained RMs on summarization and dialog, observing gains. Hong et al. (2025) find strong cross-lingual transfer on M-RewardBench for English RMs fine-tuned in four languages. Doddapaneni et al. (2025) fine-tune a judge with SFT on automatically translated prompts and responses in six languages to produce an absolute score. To our knowledge, we are the first to (i) train judges and RMs in a massively multilingual setting, (ii) fine-tune multilingual judges with RL, and (iii) demonstrate the benefits of multi-task RL, reward shaping, and pairwise grading in this setting.

## 7 Conclusion

We introduce Menlo, a comprehensive framework for evaluating and improving native-like response quality across 47 language varieties. By combining sociolinguistically-informed prompt design, detailed evaluation rubrics, and high-quality human annotations, Menlo captures multiple dimensions of conversational proficiency, including fluency, tone, localized tone, and localized factuality. We demonstrate that pairwise evaluation significantly improves both zero-shot and fine-tuned LLM judges, and that RL with reward shaping yields best judge performance.

Beyond evaluation, we show these trained judges can serve as generative reward models to directly improve policy model’s response quality. While challenges remain with the tendency of LLM judges to overestimate improvements relative to human raters, our framework provides a practical and scalable approach to both assessing and enhancing LLM proficiency in multilingual context.

#### Acknowledgments

We would like to thank Paco Guzmán for guidance on the original direction of this project. We thank Pritish Yuvraj for help with initial generations. We thank Max Mauer and Nidhi Nisarg Shah for support on the annotation interface. We would like to thank Wes Kranz, Lora Zhou, and Kish Patel for help on operations. We thank the Multilingual Post-training team for discussions and feedback.

## References

- Ahuja et al. (2023)
Kabir Ahuja, Harshita Diddee, Rishav Hada, Millicent Ochieng, Krithika Ramesh, Prachi Jain, Akshay Nambi, Tanuja Ganu, Sameer Segal, Mohamed Ahmed, Kalika Bali, and Sunayana Sitaram.
MEGA: Multilingual evaluation of generative AI.
In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), *Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing*, pp. 4232–4267, Singapore, December 2023. Association for Computational Linguistics.
doi: 10.18653/v1/2023.emnlp-main.258.
URL [https://aclanthology.org/2023.emnlp-main.258/](https://aclanthology.org/2023.emnlp-main.258/).
- Ahuja et al. (2024)
Sanchit Ahuja, Divyanshu Aggarwal, Varun Gumma, Ishaan Watts, Ashutosh Sathe, Millicent Ochieng, Rishav Hada, Prachi Jain, Mohamed Ahmed, Kalika Bali, and Sunayana Sitaram.
MEGAVERSE: Benchmarking Large Language Models Across Languages, Modalities, Models and Tasks.
In Kevin Duh, Helena Gomez, and Steven Bethard (eds.), *Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers)*, pp. 2598–2637, Mexico City, Mexico, June 2024. Association for Computational Linguistics.
doi: 10.18653/v1/2024.naacl-long.143.
URL [https://aclanthology.org/2024.naacl-long.143/](https://aclanthology.org/2024.naacl-long.143/).
- Anil et al. (2023)
Rohan Anil, Andrew M Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, et al.
Palm 2 Technical Report.
*arXiv preprint arXiv:2305.10403*, 2023.
URL [https://arxiv.org/abs/2305.10403](https://arxiv.org/abs/2305.10403).
- Bell (1984)
Allan Bell.
Language Style as Audience Design.
*Language in society*, 13(2):145–204, 1984.
- Chiu et al. (2025)
Yu Ying Chiu, Liwei Jiang, Bill Yuchen Lin, Chan Young Park, Shuyue Stella Li, Sahithya Ravi, Mehar Bhatia, Maria Antoniak, Yulia Tsvetkov, Vered Shwartz, and Yejin Choi.
CulturalBench: A Robust, Diverse and Challenging Benchmark for Measuring LMs’ Cultural Knowledge Through Human-AI Red-Teaming.
In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar (eds.), *Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)*, pp. 25663–25701, Vienna, Austria, July 2025. Association for Computational Linguistics.
ISBN 979-8-89176-251-0.
doi: 10.18653/v1/2025.acl-long.1247.
URL [https://aclanthology.org/2025.acl-long.1247/](https://aclanthology.org/2025.acl-long.1247/).
- Doddapaneni et al. (2023)
Sumanth Doddapaneni, Rahul Aralikatte, Gowtham Ramesh, Shreya Goyal, Mitesh M. Khapra, Anoop Kunchukuttan, and Pratyush Kumar.
Towards Leaving No Indic Language Behind: Building Monolingual Corpora, Benchmark and Models for Indic Languages.
In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.), *Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)*, pp. 12402–12426, Toronto, Canada, July 2023. Association for Computational Linguistics.
doi: 10.18653/v1/2023.acl-long.693.
URL [https://aclanthology.org/2023.acl-long.693/](https://aclanthology.org/2023.acl-long.693/).
- Doddapaneni et al. (2025)
Sumanth Doddapaneni, Mohammed Safi Ur Rahman Khan, Dilip Venkatesh, Raj Dabre, Anoop Kunchukuttan, and Mitesh M Khapra.
Cross-Lingual Auto Evaluation for Assessing Multilingual LLMs.
In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar (eds.), *Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)*, pp. 29297–29329, Vienna, Austria, July 2025. Association for Computational Linguistics.
ISBN 979-8-89176-251-0.
doi: 10.18653/v1/2025.acl-long.1419.
URL [https://aclanthology.org/2025.acl-long.1419/](https://aclanthology.org/2025.acl-long.1419/).
- Fabbri et al. (2025)
Alexander R Fabbri, Diego Mares, Jorge Flores, Meher Mankikar, Ernesto Hernandez, Dean Lee, Bing Liu, and Chen Xing.
Multinrc: A challenging and native multilingual reasoning evaluation benchmark for llms.
*arXiv preprint arXiv:2507.17476*, 2025.
URL [https://arxiv.org/abs/2507.17476](https://arxiv.org/abs/2507.17476).
- Fu & Liu (2025)
Xiyan Fu and Wei Liu.
How Reliable is Multilingual LLM-as-a-Judge?
*arXiv preprint arXiv:2505.12201*, 2025.
URL [https://arxiv.org/abs/2505.12201](https://arxiv.org/abs/2505.12201).
- Guo et al. (2025)
Yanzhu Guo, Simone Conia, Zelin Zhou, Min Li, Saloni Potdar, and Henry Xiao.
Do large language models have an English accent? evaluating and improving the naturalness of multilingual LLMs.
In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar (eds.), *Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)*, pp. 3823–3838, Vienna, Austria, July 2025. Association for Computational Linguistics.
ISBN 979-8-89176-251-0.
doi: 10.18653/v1/2025.acl-long.193.
URL [https://aclanthology.org/2025.acl-long.193/](https://aclanthology.org/2025.acl-long.193/).
- Gureja et al. (2025)
Srishti Gureja, Lester James Validad Miranda, Shayekh Bin Islam, Rishabh Maheshwary, Drishti Sharma, Gusti Triandi Winata, Nathan Lambert, Sebastian Ruder, Sara Hooker, and Marzieh Fadaee.
M-RewardBench: Evaluating Reward Models in Multilingual Settings.
In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar (eds.), *Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)*, pp. 43–58, Vienna, Austria, July 2025. Association for Computational Linguistics.
ISBN 979-8-89176-251-0.
doi: 10.18653/v1/2025.acl-long.3.
URL [https://aclanthology.org/2025.acl-long.3/](https://aclanthology.org/2025.acl-long.3/).
- Hershcovich et al. (2022)
Daniel Hershcovich, Stella Frank, Heather Lent, Miryam de Lhoneux, Mostafa Abdou, Stephanie Brandl, Emanuele Bugliarello, Laura Cabello Piqueras, Ilias Chalkidis, Ruixiang Cui, Constanza Fierro, Katerina Margatina, Phillip Rust, and Anders Søgaard.
Challenges and Strategies in Cross-Cultural NLP.
In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio (eds.), *Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)*, pp. 6997–7013, Dublin, Ireland, May 2022. Association for Computational Linguistics.
doi: 10.18653/v1/2022.acl-long.482.
URL [https://aclanthology.org/2022.acl-long.482/](https://aclanthology.org/2022.acl-long.482/).
- Hong et al. (2025)
Jiwoo Hong, Noah Lee, Rodrigo Martínez-Castaño, César Rodríguez, and James Thorne.
Cross-lingual Transfer of Reward Models in Multilingual Alignment.
In Luis Chiruzzo, Alan Ritter, and Lu Wang (eds.), *Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 2: Short Papers)*, pp. 82–94, Albuquerque, New Mexico, April 2025. Association for Computational Linguistics.
ISBN 979-8-89176-190-2.
doi: 10.18653/v1/2025.naacl-short.8.
URL [https://aclanthology.org/2025.naacl-short.8/](https://aclanthology.org/2025.naacl-short.8/).
- Hu et al. (2020)
Junjie Hu, Sebastian Ruder, Aditya Siddhant, Graham Neubig, Orhan Firat, and Melvin Johnson.
Xtreme: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalisation.
In *International conference on machine learning*, pp. 4411–4421. PMLR, 2020.
URL [https://proceedings.mlr.press/v119/hu20b](https://proceedings.mlr.press/v119/hu20b).
- Hupkes & Bogoychev (2025)
Dieuwke Hupkes and Nikolay Bogoychev.
MultiLoKo: A Multilingual Local Knowledge Benchmark for LLMs Spanning 31 Languages.
*arXiv preprint arXiv:2504.10356*, 2025.
URL [https://arxiv.org/abs/2504.10356](https://arxiv.org/abs/2504.10356).
- Jacovi et al. (2025)
Alon Jacovi, Andrew Wang, Chris Alberti, Connie Tao, Jon Lipovetz, Kate Olszewska, Lukas Haas, Michelle Liu, Nate Keating, Adam Bloniarz, et al.
The FACTS Grounding Leaderboard: Benchmarking LLMs’ Ability to Ground Responses to Long-Form Input.
*arXiv preprint arXiv:2501.03200*, 2025.
URL [https://arxiv.org/abs/2501.03200](https://arxiv.org/abs/2501.03200).
- Jamieson et al. (2000)
Joan Jamieson, Stan Jones, Irwin Kirsch, Peter Mosenthal, and Carol Taylor.
TOEFL 2000 Framework.
*Princeton, NJ: Educational Testing Service*, 2000.
- Ke & Ng (2019)
Zixuan Ke and Vincent Ng.
Automated essay scoring: A survey of the state of the art.
In *IJCAI*, volume 19, pp. 6300–6308, 2019.
- Liu et al. (2024)
Yang Liu, Meng Xu, Shuo Wang, Liner Yang, Haoyu Wang, Zhenghao Liu, Cunliang Kong, Yun Chen, Maosong Sun, and Erhong Yang.
OMGEval: An Open Multilingual Generative Evaluation Benchmark for Large Language Models.
*arXiv preprint arXiv:2402.13524*, 2024.
URL [https://arxiv.org/abs/2402.13524](https://arxiv.org/abs/2402.13524).
- Liu et al. (2021)
Ye Liu, Wolfgang Maier, Wolfgang Minker, and Stefan Ultes.
Naturalness evaluation of natural language generation in task-oriented dialogues using BERT.
In Ruslan Mitkov and Galia Angelova (eds.), *Proceedings of the International Conference on Recent Advances in Natural Language Processing (RANLP 2021)*, pp. 839–845, Held Online, September 2021. INCOMA Ltd.
URL [https://aclanthology.org/2021.ranlp-1.96/](https://aclanthology.org/2021.ranlp-1.96/).
- Liu et al. (2025)
Zichen Liu, Changyu Chen, Wenjun Li, Penghui Qi, Tianyu Pang, Chao Du, Wee Sun Lee, and Min Lin.
Understanding R1-Zero-Like Training: A Critical Perspective.
*arXiv preprint arXiv:2503.20783*, 2025.
URL [https://arxiv.org/abs/2503.20783](https://arxiv.org/abs/2503.20783).
- Lothritz & Cabot (2025)
Cedric Lothritz and Jordi Cabot.
Testing Low-Resource Language Support in LLMs Using Language Proficiency Exams: the Case of Luxembourgish.
*arXiv preprint arXiv:2504.01667*, 2025.
URL [https://arxiv.org/abs/2304.01667](https://arxiv.org/abs/2304.01667).
- Marchisio et al. (2024)
Kelly Marchisio, Wei-Yin Ko, Alexandre Berard, Théo Dehaze, and Sebastian Ruder.
Understanding and Mitigating Language Confusion in LLMs.
In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), *Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing*, pp. 6653–6677, Miami, Florida, USA, November 2024. Association for Computational Linguistics.
doi: 10.18653/v1/2024.emnlp-main.380.
URL [https://aclanthology.org/2024.emnlp-main.380/](https://aclanthology.org/2024.emnlp-main.380/).
- Mayor-Rocher et al. (2024)
Marina Mayor-Rocher, Nina Melero, Elena Merino-Gómez, María Grandury, Javier Conde, and Pedro Reviriego.
Evaluating Large Language Models with Tests of Spanish as a Foreign Language: Pass or Fail?
*arXiv preprint arXiv:2409.15334*, 2024.
URL [https://arxiv.org/abs/2409.15334](https://arxiv.org/abs/2409.15334).
- Myung et al. (2024)
Junho Myung, Nayeon Lee, Yi Zhou, Jiho Jin, Rifki Afina Putri, Dimosthenis Antypas, Hsuvas Borkakoty, Eunsu Kim, Carla Perez-Almendros, Abinew Ali Ayele, Victor Gutierrez Basulto, Yazmin Ibanez-Garcia, Hwaran Lee, Shamsuddeen Hassan Muhammad, Kiwoong Park, Anar Sabuhi Rzayev, Nina White, Seid Muhie Yimam, Mohammad Taher Pilehvar, Nedjma Ousidhoum, Jose Camacho-Collados, and Alice Oh.
BLEnD: A Benchmark for LLMs on Everyday Knowledge in Diverse Cultures and Languages.
In *The Thirty-eight Conference on Neural Information Processing Systems Datasets and Benchmarks Track*, 2024.
URL [https://openreview.net/forum?id=nrEqH502eC](https://openreview.net/forum?id=nrEqH502eC).
- Novikova et al. (2016)
Jekaterina Novikova, Oliver Lemon, and Verena Rieser.
Crowd-sourcing NLG data: Pictures elicit better data.
In Amy Isard, Verena Rieser, and Dimitra Gkatzia (eds.), *Proceedings of the 9th International Natural Language Generation conference*, pp. 265–273, Edinburgh, UK, September 5-8 2016. Association for Computational Linguistics.
doi: 10.18653/v1/W16-6644.
URL [https://aclanthology.org/W16-6644/](https://aclanthology.org/W16-6644/).
- Ruder et al. (2021)
Sebastian Ruder, Noah Constant, Jan Botha, Aditya Siddhant, Orhan Firat, Jinlan Fu, Pengfei Liu, Junjie Hu, Dan Garrette, Graham Neubig, and Melvin Johnson.
XTREME-R: Towards More Challenging and Nuanced Multilingual Evaluation.
In Marie-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih (eds.), *Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing*, pp. 10215–10245, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics.
doi: 10.18653/v1/2021.emnlp-main.802.
URL [https://aclanthology.org/2021.emnlp-main.802/](https://aclanthology.org/2021.emnlp-main.802/).
- Shao et al. (2024)
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, et al.
DeepseekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models.
*arXiv preprint arXiv:2402.03300*, 2024.
URL [https://arxiv.org/abs/2402.03300](https://arxiv.org/abs/2402.03300).
- Son et al. (2024)
Guijin Son, Dongkeun Yoon, Juyoung Suk, Javier Aula-Blasco, Mano Aslan, Vu Trong Kim, Shayekh Bin Islam, Jaume Prats-Cristià, Lucía Tormo-Bañuelos, and Seungone Kim.
MM-Eval: A Multilingual Meta-Evaluation Benchmark for LLM-as-a-Judge and Reward Models.
*arXiv preprint arXiv:2410.17578*, 2024.
URL [https://arxiv.org/abs/2410.17578](https://arxiv.org/abs/2410.17578).
- Sorensen et al. (2024)
Taylor Sorensen, Jared Moore, Jillian Fisher, Mitchell L Gordon, Niloofar Mireshghallah, Christopher Michael Rytting, Andre Ye, Liwei Jiang, Ximing Lu, Nouha Dziri, Tim Althoff, and Yejin Choi.
A Roadmap to Pluralistic Alignment.
In *Forty-first International Conference on Machine Learning*, 2024.
URL [https://openreview.net/forum?id=gQpBnRHwxM](https://openreview.net/forum?id=gQpBnRHwxM).
- Wang et al. (2025)
Victor Wang, Michael JQ Zhang, and Eunsol Choi.
Improving LLM-as-a-Judge Inference with the Judgment Distribution.
*arXiv preprint arXiv:2503.03064*, 2025.
URL [https://arxiv.org/abs/2503.03064](https://arxiv.org/abs/2503.03064).
- Watts et al. (2024)
Ishaan Watts, Varun Gumma, Aditya Yadavalli, Vivek Seshadri, Manohar Swaminathan, and Sunayana Sitaram.
PARIKSHA: A Large-Scale Investigation of Human-LLM Evaluator Agreement on Multilingual and Multi-Cultural Data.
In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), *Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing*, pp. 7900–7932, Miami, Florida, USA, November 2024. Association for Computational Linguistics.
doi: 10.18653/v1/2024.emnlp-main.451.
URL [https://aclanthology.org/2024.emnlp-main.451/](https://aclanthology.org/2024.emnlp-main.451/).
- Whitehouse et al. (2025)
Chenxi Whitehouse, Tianlu Wang, Ping Yu, Xian Li, Jason Weston, Ilia Kulikov, and Swarnadeep Saha.
J1: Incentivizing Thinking in LLM-as-a-Judge via Reinforcement Learning.
*arXiv preprint arXiv:2505.10320*, 2025.
URL [https://arxiv.org/abs/2505.10320](https://arxiv.org/abs/2505.10320).
- Wu et al. (2024)
Zhaofeng Wu, Ananth Balashankar, Yoon Kim, Jacob Eisenstein, and Ahmad Beirami.
Reuse Your Rewards: Reward Model Transfer for Zero-Shot Cross-Lingual Alignment.
In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), *Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing*, pp. 1332–1353, Miami, Florida, USA, November 2024. Association for Computational Linguistics.
doi: 10.18653/v1/2024.emnlp-main.79.
URL [https://aclanthology.org/2024.emnlp-main.79/](https://aclanthology.org/2024.emnlp-main.79/).
- Zhuang et al. (2024)
Yuchen Zhuang, Haotian Sun, Yue Yu, Rushi Qiang, Qifan Wang, Chao Zhang, and Bo Dai.
HYDRA: Model Factorization Framework for Black-Box LLM Personalization.
In *The Thirty-eighth Annual Conference on Neural Information Processing Systems*, 2024.
URL [https://openreview.net/forum?id=CKgNgKmHYp](https://openreview.net/forum?id=CKgNgKmHYp).

## Appendix A Additional Details on the Menlo Dataset

For Menlo, translators and annotators were recruited through third-party services and compensated based on local regulations.

### A.1 Dataset Collection

We initiated a pilot to evaluate of different models including GPT-4o and Llama4-Maverick in a single category, Localized Tone focusing on five languages spoken by the authors: Bengali, German, Hindi, Italian, and Russian. As expected, the pilot framework was quickly confronted with the complexities inherent in multilingualism: Even among the authors, for all initial Localized Tone prompts, we struggled to reach consistent and reliable agreement. Nevertheless, these early results provided valuable insights to guide us to improve prompt design, guideline clarity, and annotation arrangement.

To enhance the reliability of the framework, we took steps to refine our prompts’ nuance and complexity, update guidelines with clearer direction for annotators that aimed to make abstract concepts more concrete, and started exploring a more user-friendly annotation solution. The changes brought upon notable improvements in inter-annotator agreement that extended to the full-scale annotation. We show the agreement of the initial pilot annotation and improved annotation for localized tone in 5 languages in [Table 8](https://arxiv.org/html/2509.26601v1#A1.T8) and across all categories and languages in [Table 9](https://arxiv.org/html/2509.26601v1#A1.T9).

**Table 8: Comparison of Pilot and Menlo for 5 languages in localized tone category. Agreement is defined as the percentage of annotation pairs whose ratings for the same item differ by no more than 1.**
| Language Code | Pilot Agreement | Menlo Agreement |
| --- | --- | --- |
| bn_BD | 0.75 | 0.84 |
| de_DE | 0.74 | 0.92 |
| hi_IN | 0.74 | 0.92 |
| it_IT | 0.79 | 0.71 |
| ru_RU | 0.71 | 0.79 |
| Overall | 0.75 | 0.84 |

**Table 9: Comparison of the pilot annotation (5 languages) and final Menlo dataset (47 languages). Agreement is defined as the percentage of annotation pairs whose ratings for the same item differ by no more than 1. Agreement for Pilot has been averaged over 5 languages, while Menlo is averaged over 47.**
|  | Pilot (5 Languages) | Menlo (47 Languages) |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
| Quality Dimension | Agreement | # Prompts | # Annotations | Agreement | # Prompts | # Annotations |
| Fluency | 0.76 | 150 | 450 | 0.82 | 1,820 | 23,556 |
| Tone | 0.70 | 150 | 450 | 0.77 | 1,410 | 18,712 |
| Localized Tone | 0.75 | 200 | 600 | 0.82 | 1,825 | 22,324 |
| Localized Factuality | 0.78 | 150 | 450 | 0.78 | 1,378 | 16,422 |
| Overall | 0.75 | 650 | 1,950 | 0.80 | 6,423 | 81,014 |

**Table 10: Components to consider when annotating different subcategories of Tone.**
| Tone Subcategory | Tone Component 1 | Tone Component 2 |
| --- | --- | --- |
| Helpful Tone | Instruction following $\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}{\checkmark}$ | Emotional support $\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}{\checkmark}$ |
| Insightful Tone | Informative $\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}{\checkmark}$ | Empathetic $\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}{\checkmark}$ |
| Engaging Tone | Conversational Language $\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}{\checkmark}$ | Encourages Interactions $\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}{\checkmark}$ |
| Fair Tone | Non-biased stance $\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}{\checkmark}$ | Non-preachy language $\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}{\checkmark}$ |

##### Annotation guidelines

Judging language performance can be subjective. To minimize confusion, we identified the most important components of tone, fluency, localized tone, and localized factuality and incorporated them into the guidelines. For example, a model response that conveys a helpful tone must succeed on two fronts: providing (or attempting to provide) help based on users’ instructions, and expressing emotional engagement to sound caring. By breaking down broad linguistic concepts into easy-to-follow subcategories and self-explanatory signals (illustrated via emoji $\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}{\checkmark}$  OK $\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}{\boldsymbol{\times}}$ $\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}{?}$), annotators can quickly grasp and refer back to the guidelines. We show subcategories for Tone, for example, in [Table 10](https://arxiv.org/html/2509.26601v1#A1.T10) and show the rubric guidelines for Localized Factuality in [Table 11](https://arxiv.org/html/2509.26601v1#A1.T11).

**Table 11: Localized factuality rubrics for annotation using a 5-point Likert scale. Each rating corresponds to a high-level classification of the response (e.g., *“Sounds somewhat accurate and relevant”*), further specified by dimension-specific criteria e.g., accuracy, relevance, and completeness.**
| 1: Major failure<br>*“Grossly incorrect or misleading”* | 2: Minor failure<br>*“Some mistakes”* | 3: Pass<br>*“Sounds somewhat accurate and relevant”* | 4: Good<br>*“Sounds accurate and relevant”* | 5: Excellent<br>*“Factually accurate, highly relevant, complete with additional info”* |
| --- | --- | --- | --- | --- |
| Accuracy $\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}{\boldsymbol{\times}}$$\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}{\boldsymbol{\times}}$<br>- The model’s response contains obvious factual errors or made-up information. | Accuracy $\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}{\boldsymbol{\times}}$$\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}{?}$<br>- The model’s response contains some factual mistakes. | Accuracy OK<br>- No obvious factual errors but some claims are not entirely correct or may be misleading. | Accuracy $\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}{\checkmark}$<br>The claims in the response are factually accurate. | Accuracy $\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}{\checkmark}$$\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}{\checkmark}$<br>The claims in the response are completely factually accurate. |
|  |  |  |  |  |
| Locale Relevance $\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}{\boldsymbol{\times}}$$\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}{\boldsymbol{\times}}$<br>Local Point of View $\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}{\boldsymbol{\times}}$$\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}{\boldsymbol{\times}}$<br>- The model fails to understand the basic local context.<br>- The model provides content that is irrelevant or misaligned with the local context.<br>- The model’s response frames the answer in a fetishizing/offensive way (like overly explaining basic local knowledge to locals) | Locale Relevance $\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}{\boldsymbol{\times}}$$\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}{?}$<br>Local Point of View $\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}{\boldsymbol{\times}}$$\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}{?}$<br>- The model grasps some local context but misses key nuances.<br>- The model’s response is somewhat relevant but provides mainly general or high-level information that lacks alignment with the local context.<br>- The model’s response may come across as slightly insensitive or tone-deaf, but it does not contain overtly fetishizing or offensive answers. | Locale Relevance OK<br>Local Point of View OK<br>- The model generally understands the local context but may miss subtle nuances.<br>- The response is generally relevant and aligned with the local context.<br>- The model’s response is neutral and factual. | Locale Relevance $\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}{\checkmark}$<br>Local Point of View $\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}{\checkmark}$<br>- The model accurately interprets the local context and nuances.<br>- The response is generally relevant and aligned with the local context.<br>- The model avoids explanations that might be seen as overly simplistic or patronizing. Instead, the facts are thoughtfully selected with depth. | Locale Relevance $\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}{\checkmark}$$\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}{\checkmark}$<br>Local Point of View $\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}{\checkmark}$$\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}{\checkmark}$<br>- The model demonstrates a deep understanding of the local context and nuances. The response delivers highly relevant content that is highly specific and perfectly aligned with the local context.<br>- The model chooses facts that are in-depth and nuanced even for someone who’s already a local. It might present additional context and highlights regional variations to show depth of local knowledge. |
|  |  |  |  |  |
| Completeness $\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}{\boldsymbol{\times}}$$\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}{\boldsymbol{\times}}$<br>- The model’s response is incomplete and misses crucial information to answer the question. | Completeness $\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}{\boldsymbol{\times}}$$\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}{?}$<br>- The response answers part of the question but is missing some relevant pieces of information. | Completeness OK<br>- The model provides sufficient information to answer the question but the provided information may lack depth. | Completeness $\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}{\checkmark}$<br>- The model provides all the information to answer the question. | Completeness $\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}{\checkmark}$$\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}{\checkmark}$<br>- The response is rich in information and covers all information to answer the question as well as additional helpful context that further helps to contextualize the response. |

##### Annotation tool

To streamline the annotation, we developed a custom annotation interface, which we show in [Figure 4](https://arxiv.org/html/2509.26601v1#A1.F4). The tool provides a simple, annotator-friendly user interface for guidelines, rating model responses, and randomized model A/B pairwise comparison. The backend allowed us to ensure data is consistent and identify any missing annotations or other data-related issues. In addition, it enabled us to quickly test annotators on dedicated test annotations before moving them to the actual annotation tasks. Overall, solid tooling allowed us to screen more than 1,000 annotators and collect more than 80,000 annotations.

Figure: Figure 4: Annotation interface used for Menlo.
Refer to caption: figures/menlo_annotation_tool.png

### A.2 Language Varieties in Menlo

Menlo covers 47 language varieties.
[Table 12](https://arxiv.org/html/2509.26601v1#A1.T12) lists each variety along with its corresponding ISO 639-1 code.

[Table 13](https://arxiv.org/html/2509.26601v1#A1.T13) reports annotator IAA by dimension and language variety.

**Table 12: Mapping from language-region codes to language names.**
| Language code | Full name | Language code | Full name |
| --- | --- | --- | --- |
| ar | Modern Standard Arabic | mr_IN | Marathi |
| ar_Latn_EG | romanized Egyptian Arabic | ms_MY | Malay (Malaysia) |
| bg_BG | Bulgarian | ne_NP | Nepali |
| bn_BD | Bengali | nl_NL | Dutch |
| cs_CZ | Czech | pl_PL | Polish |
| da_DK | Danish | pt_BR | Brazilian Portuguese |
| de_DE | German | pt_PT | Portuguese (Portugal) |
| el_GR | Greek | ro_RO | Romanian |
| en_AU | Australian English | ru_RU | Russian |
| en_GB | British English | sk_SK | Slovak |
| en_IN | Indian English | sv_SE | Swedish |
| en_US | US English | sw_KE | Swahili (Kenya) |
| es_ES | Spanish (Spain) | th_TH | Thai |
| es_MX | Mexican Spanish | tl_PH | Tagalog (Philippines) |
| fa_IR | Persian (Iran) | tr_TR | Turkish |
| fr_FR | French (France) | uk_UA | Ukrainian |
| gu_IN | Gujarati (India) | ur_Latn_PK | romanized Urdu |
| he_IL | Hebrew (Israel) | ur_PK | Urdu |
| hi_IN | Hindi | vi_VN | Vietnamese |
| hi_Latn_IN | romanized Hindi | zh_CN | Chinese (China) |
| hr_HR | Croatian | zh_TW | Traditional Chinese (Taiwan) |
| hu_HU | Hungarian | ja_JP | Japanese |
| id_ID | Indonesian | ko_KR | Korean |
| it_IT | Italian |  |  |

**Table 13: Krippendorff alpha by quality dimension and language.**
| Language Code | Tone | Fluency | Localized Tone | Localized Factuality | Average |
| --- | --- | --- | --- | --- | --- |
| ar | 0.83 | 0.79 | 0.86 | 0.79 | 0.82 |
| ar_Latn_EG | 0.86 | 0.79 | 0.82 | NA | 0.82 |
| bg_BG | 0.75 | 0.80 | 0.79 | 0.86 | 0.80 |
| bn_BD | 0.82 | 0.79 | 0.85 | 0.80 | 0.82 |
| cs_CZ | 0.80 | 0.78 | 0.78 | 0.82 | 0.79 |
| da_DK | 0.83 | 0.78 | 0.78 | 0.85 | 0.81 |
| de_DE | 0.82 | 0.76 | 0.85 | 0.77 | 0.80 |
| el_GR | 0.83 | 0.85 | 0.85 | 0.85 | 0.84 |
| en_AU | 0.89 | 0.73 | 0.81 | 0.82 | 0.81 |
| en_GB | 0.85 | 0.79 | 0.85 | 0.82 | 0.83 |
| en_IN | 0.84 | 0.85 | 0.81 | 0.83 | 0.83 |
| es_ES | 0.78 | 0.78 | 0.81 | 0.79 | 0.79 |
| es_MX | 0.79 | 0.80 | 0.86 | 0.84 | 0.82 |
| fa_IR | 0.83 | 0.82 | 0.77 | 0.81 | 0.81 |
| fr_FR | 0.83 | 0.76 | 0.81 | 0.82 | 0.81 |
| gu_IN | 0.89 | 0.81 | 0.86 | 0.84 | 0.85 |
| he_IL | 0.85 | 0.79 | 0.80 | 0.84 | 0.82 |
| hi_IN | 0.82 | 0.81 | 0.83 | 0.87 | 0.83 |
| hi_Latn_IN | 0.86 | 0.77 | 0.83 | 0.80 | 0.82 |
| hr_HR | 0.81 | 0.78 | 0.80 | 0.82 | 0.80 |
| hu_HU | 0.85 | 0.81 | 0.80 | 0.84 | 0.82 |
| id_ID | 0.90 | 0.81 | 0.82 | 0.82 | 0.84 |
| it_IT | 0.83 | 0.77 | 0.77 | 0.85 | 0.80 |
| ja_JP | 0.86 | 0.82 | 0.79 | 0.81 | 0.82 |
| ko_KR | 0.86 | 0.83 | 0.80 | 0.84 | 0.83 |
| mr_IN | 0.88 | 0.78 | 0.78 | 0.86 | 0.82 |
| ms_MY | 0.84 | 0.82 | 0.81 | 0.83 | 0.83 |
| ne_NP | 0.83 | 0.79 | 0.80 | 0.83 | 0.81 |
| nl_NL | 0.84 | 0.81 | 0.84 | 0.79 | 0.82 |
| pl_PL | 0.83 | 0.79 | 0.86 | 0.82 | 0.83 |
| pt_BR | 0.86 | 0.82 | 0.82 | 0.85 | 0.83 |
| pt_PT | 0.83 | 0.80 | 0.83 | 0.80 | 0.82 |
| ro_RO | 0.84 | 0.79 | 0.80 | 0.82 | 0.81 |
| ru_RU | 0.81 | 0.75 | 0.80 | 0.78 | 0.79 |
| sk_SK | 0.88 | 0.81 | 0.81 | 0.84 | 0.83 |
| sv_SE | 0.84 | 0.78 | 0.81 | 0.81 | 0.81 |
| sw_KE | 0.88 | 0.84 | 0.83 | 0.85 | 0.85 |
| th_TH | 0.85 | 0.83 | 0.78 | 0.83 | 0.82 |
| tl_PH | 0.84 | 0.83 | 0.82 | 0.81 | 0.83 |
| tr_TR | 0.88 | 0.85 | 0.79 | 0.80 | 0.83 |
| uk_UA | 0.89 | 0.77 | 0.81 | 0.80 | 0.82 |
| ur_Latn_PK | 0.82 | 0.79 | 0.80 | 0.86 | 0.82 |
| ur_PK | 0.81 | 0.79 | 0.82 | 0.86 | 0.82 |
| vi_VN | 0.85 | 0.82 | 0.82 | 0.82 | 0.83 |
| zh_CN | 0.86 | 0.81 | 0.80 | 0.83 | 0.83 |
| zh_TW | 0.89 | 0.82 | 0.80 | 0.89 | 0.85 |

### A.3 Grading Rubrics

The 5-point grading rubrics are defined for each question type under the four dimensions:

Fluency:
Vocabulary & Syntax,
Coherence,
Grammar & Mechanics,
Clarity & Conciseness.

Localized Tone:
Cultural Relevance,
Formality & politeness,
Humor,
Linguistic nuance.

Localized Factuality:
Cultural Practices,
Expressions & Concepts,
Local Knowledge.

Tone:
Be engaging, Be fair, Be insightful, Help as best as you can.

The rubrics were created based on reviews of example prompts and failure modes of the different dimensions and inspired by prior work on automated proficiency assessment (Ke & Ng, 2019) and cross-cultural variation (Hershcovich et al., 2022; Myung et al., 2024).

All rubrics use the same 5-point scale, with criteria adapted to the specific question type.
We show some examples of the grading rubrics in Figure [5](https://arxiv.org/html/2509.26601v1#A1.F5), [6](https://arxiv.org/html/2509.26601v1#A1.F6), [7](https://arxiv.org/html/2509.26601v1#A1.F7), and [8](https://arxiv.org/html/2509.26601v1#A1.F8).

Figure: Figure 5: Example of 5-Point Grading Rubrics for Localized Tone (Formality & politeness).

Figure: Figure 6: Example of 5-Point Grading Rubrics for Fluency.

Figure: Figure 7: Example of 5-Point Grading Rubrics for Tone (Be insightful:Be intellectually curious and engaging).

Figure: Figure 8: Example of 5-Point Grading Rubrics for Localized Factuality.

### A.4 Full Examples for Menlo

We provide full examples from Menlo in Figure [9](https://arxiv.org/html/2509.26601v1#A1.F9), [10](https://arxiv.org/html/2509.26601v1#A1.F10), [11](https://arxiv.org/html/2509.26601v1#A1.F11), [12](https://arxiv.org/html/2509.26601v1#A1.F12), [13](https://arxiv.org/html/2509.26601v1#A1.F13), [14](https://arxiv.org/html/2509.26601v1#A1.F14), [15](https://arxiv.org/html/2509.26601v1#A1.F15), and [16](https://arxiv.org/html/2509.26601v1#A1.F16), including prompt (both in English and the translated version in target languages), responses, and corresponding grades.
Examples cover different languages and dimensions.

Figure: Figure 9: Example prompt, responses, and annotation in Korean for Localized Tone (Humor).
Refer to caption: x4.png

Figure: Figure 10: Example prompt, responses, and annotation in Czech for Localized Tone (Cultural relevance).
Refer to caption: x5.png

Figure: Figure 11: Example prompt, responses, and annotation in Hebrew for Localized Factuality.
Refer to caption: x6.png

Figure: Figure 12: Example prompt, responses, and annotation in Swedish for Localized Factuality.
Refer to caption: x7.png

Figure: Figure 13: Example prompt, responses, and annotation in Danish for Tone (Be fair).
Refer to caption: x8.png

Figure: Figure 14: Example prompt, responses, and annotation in Japanese for Tone (Be engaging).
Refer to caption: x9.png

Figure: Figure 15: Example prompt, responses, and annotation in Ukrainian for Fluency.
Refer to caption: x10.png

Figure: Figure 16: Example prompt, responses, and annotation in Romanian for Fluency.
Refer to caption: x11.png

## Appendix B Pairwise Judge Template

[Figure 17](https://arxiv.org/html/2509.26601v1#A2.F17) shows an example of pairwise judge with pointwise scoring for Tone. Other dimensions follow similar template with varied intros.

Figure: Figure 17: Pairwise judge prompt template.

## Appendix C Experiment Details

### C.1 Finetuning LLM-Judges on Menlo

Finetuning LLM-Judges uses 16$\times$H100 GPUs for Qwen3-4B and 192 GPUs for Llama4-Scout.

In SFT, models directly predict 5-point grades for response pairs without generating intermediate reasoning, trained with cross-entropy loss under teacher forcing. We use the TRL ([https://huggingface.co/docs/trl](https://huggingface.co/docs/trl)) library and adopt the default learning rate of $2\text{e-}5$. Maximum sequence length is set to 8192.

In RL, we use GRPO with the verl ([https://github.com/volcengine/verl](https://github.com/volcengine/verl)) implementation, keeping the default learning rate $1\text{e-}6$. We set rollout size to 8, and maximum length 4,096 tokens for both input and output. Prompts follow the template in [Figure 17](https://arxiv.org/html/2509.26601v1#A2.F17), encouraging models to produce reasoning before assigning scores.

Batch size is set to 32, and we train up to three epochs and select best checkpoint based on the performance on the validation set.

### C.2 Post Training with RL

We set the learning rate to $1\text{e-}6$, the maximum tokens for the policy model to 1,024, and for the RM to 4,096, using up to 4$\times$H100 GPUs.
Following Liu et al. (2025), we disable the length normalization term in the loss, as we find that otherwise responses tend to grow excessively long after training.

Since the judge is not trained to evaluate the thinking process but only the responses, we sample generations from the policy model Qwen3-4B in non-thinking mode. Comparison of the response quality before (Qwen3-4B) and after training (Post-train) are both done in thinking mode, as we find it leads to superior generation quality. When constructing preference pairs for the pairwise judge, we remove the thinking tokens from the generations.

Batch size is also set to 32, and we train up to three epochs and select best checkpoint based on the performance on the validation set.

## Appendix D Additional Results

### D.1 Judge Performance Per Dimension

We report detailed results for all eight models across four metrics: Macro-F1 and Accuracy for 5-way classification, Preference accuracy over A win / A loss / Tie, and Krippendorff’s $\alpha$ for agreement with human annotators. Results are shown for the four dimensions: Fluency, Localized Factuality, Localized Tone, and Tone.

[Table 14](https://arxiv.org/html/2509.26601v1#A4.T14), [Table 15](https://arxiv.org/html/2509.26601v1#A4.T15), and [Table 16](https://arxiv.org/html/2509.26601v1#A4.T16) present results for Zero-shot Pointwise, Zero-shot Pairwise, and Few-shot Pointwise with grading rubrics.
[Table 17](https://arxiv.org/html/2509.26601v1#A4.T17) and [Table 18](https://arxiv.org/html/2509.26601v1#A4.T18) report corresponding Zero-shot Pointwise and Zero-shot Pairwise results without grading rubrics.
[Table 19](https://arxiv.org/html/2509.26601v1#A4.T19) compares dimension-wise performance of Qwen3-4B and Llama4-Scout trained with SFT and RL on all data, including both Pointwise and Pairwise.

Overall, Localized Factuality remains the most challenging dimension: both frontier API models and RL-trained models show limited improvement. This suggests that alternative training approaches, such as integrating search and tool use, may be necessary, which we leave for future work.

[Table 21](https://arxiv.org/html/2509.26601v1#A4.T21) further compares dimension-wise performance of Pairwise RL-trained Qwen3-4B on partial subsets of the data. Specifically, we evaluate (i) models trained on a single dimension and tested across all dimensions to study cross-task transfer, and (ii) models trained only on English data and evaluated on all languages. Results show that optimizing on single dimension achieves performance similar to joint training ([Table 19](https://arxiv.org/html/2509.26601v1#A4.T19)), highlighting the efficiency and practicality of joint training. In contrast, training only on English leads to degraded performance, revealing the challenges of cross-lingual transfer given the localized nature of our Menlo dataset.

### D.2 Judge Performance Per Language Variety

[Table 22](https://arxiv.org/html/2509.26601v1#A4.T22) and [Table 23](https://arxiv.org/html/2509.26601v1#A4.T23) show Marco-F1 and Preference accuracy per Language Variety for baseline and fine-tuned Qwen3-4B and Llama4-Scout models.

**Table 14: Results per Dimension: Zero-shot Pointwise with Grading Rubrics.**
| Zero-shot Pointwise with Rubrics | Qwen3-4B | Qwen3-32B | Llama3.1-8B | Llama3.3-70B | Llama4-Scout | o3 | gpt-4o | gpt-4.1 |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Overall | Macro-F1 | 23.06 | 28.53 | 22.27 | 27.93 | 25.63 | 26.54 | 25.99 | 32.23 |
| Accuracy | 35.48 | 37.88 | 30.97 | 35.99 | 38.19 | 34.37 | 36.33 | 38.05 |  |
| Preference | 40.54 | 42.19 | 39.92 | 37.37 | 42.19 | 45.07 | 42.92 | 41.73 |  |
| Krippendorff’s $\alpha$ | 80.59 | 83.80 | 79.35 | 80.71 | 82.09 | 79.64 | 83.59 | 83.78 |  |
| Fluency | Macro-F1 | 17.60 | 29.01 | 19.35 | 21.03 | 22.09 | 23.29 | 18.55 | 20.73 |
| Accuracy | 37.07 | 41.98 | 32.57 | 36.87 | 39.68 | 43.99 | 36.37 | 37.17 |  |
| Preference | 37.68 | 41.28 | 40.48 | 35.27 | 43.52 | 41.28 | 34.47 | 34.87 |  |
| Krippendorff’s $\alpha$ | 76.73 | 81.47 | 77.85 | 78.29 | 80.35 | 83.19 | 76.59 | 76.47 |  |
| Localized Factuality | Macro-F1 | 14.94 | 15.77 | 14.17 | 18.57 | 16.74 | 12.04 | 19.27 | 20.25 |
| Accuracy | 30.16 | 26.22 | 26.49 | 34.10 | 31.93 | 13.86 | 29.48 | 31.11 |  |
| Preference | 26.63 | 28.80 | 28.26 | 29.62 | 26.90 | 32.34 | 27.99 | 26.36 |  |
| Krippendorff’s $\alpha$ | 74.30 | 74.50 | 74.64 | 72.37 | 74.82 | 66.45 | 77.81 | 77.29 |  |
| Localized Tone | Macro-F1 | 17.34 | 29.40 | 19.00 | 20.52 | 19.54 | 28.13 | 22.23 | 25.73 |
| Accuracy | 32.45 | 37.20 | 36.31 | 31.57 | 37.75 | 41.94 | 33.11 | 36.64 |  |
| Preference | 41.72 | 45.25 | 36.42 | 32.89 | 41.94 | 50.99 | 46.80 | 48.12 |  |
| Krippendorff’s $\alpha$ | 76.32 | 80.41 | 78.43 | 73.92 | 78.84 | 80.89 | 78.80 | 80.27 |  |
| Tone | Macro-F1 | 36.17 | 40.56 | 23.76 | 38.06 | 32.00 | 25.97 | 35.31 | 41.62 |
| Accuracy | 41.14 | 43.61 | 27.47 | 41.03 | 42.15 | 32.85 | 45.18 | 46.19 |  |
| Preference | 54.04 | 51.12 | 52.47 | 50.67 | 53.59 | 53.81 | 60.76 | 55.61 |  |
| Krippendorff’s $\alpha$ | 87.16 | 88.91 | 81.41 | 88.48 | 87.61 | 82.13 | 90.58 | 90.21 |  |

**Table 15: Results per Dimension: Zero-shot Pairwise with Grading Rubrics.**
| Zero-shot Pairwise with Rubrics | Qwen3-4B | Qwen3-32B | Llama3.1-8B | Llama3.3-70B | Llama4-Scout | o3 | gpt-4o | gpt-4.1 |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Overall | Macro-F1 | 35.46 | 37.48 | 29.46 | 37.50 | 36.11 | 35.35 | 37.57 | 38.53 |
| Accuracy | 43.23 | 40.88 | 29.56 | 43.12 | 42.29 | 37.26 | 40.86 | 44.48 |  |
| Preference | 57.13 | 59.12 | 50.45 | 55.32 | 56.25 | 58.72 | 57.98 | 59.23 |  |
| Krippendorff’s $\alpha$ | 84.25 | 85.60 | 80.17 | 85.29 | 84.10 | 83.97 | 86.35 | 85.65 |  |
| Fluency | Macro-F1 | 32.24 | 35.27 | 26.67 | 32.48 | 35.11 | 34.40 | 34.67 | 34.55 |
| Accuracy | 46.99 | 45.99 | 27.45 | 43.99 | 43.91 | 44.09 | 46.19 | 50.10 |  |
| Preference | 55.91 | 59.92 | 50.50 | 52.30 | 54.03 | 60.12 | 56.51 | 60.32 |  |
| Krippendorff’s $\alpha$ | 83.05 | 84.72 | 80.21 | 83.87 | 84.61 | 84.15 | 85.72 | 83.99 |  |
| Localized Factuality | Macro-F1 | 22.55 | 21.93 | 20.96 | 20.96 | 21.13 | 17.86 | 21.24 | 24.27 |
| Accuracy | 33.02 | 28.80 | 25.14 | 29.76 | 28.80 | 21.06 | 24.46 | 29.62 |  |
| Preference | 42.93 | 43.21 | 38.04 | 38.86 | 38.59 | 38.86 | 38.04 | 37.77 |  |
| Krippendorff’s $\alpha$ | 75.97 | 75.96 | 74.03 | 75.07 | 74.74 | 71.41 | 75.93 | 75.54 |  |
| Localized Tone | Macro-F1 | 32.82 | 35.25 | 30.22 | 36.82 | 33.62 | 37.88 | 37.86 | 35.02 |
| Accuracy | 42.49 | 43.27 | 33.77 | 46.69 | 43.82 | 44.15 | 45.92 | 46.14 |  |
| Preference | 60.49 | 63.80 | 51.21 | 60.71 | 61.37 | 65.34 | 64.24 | 65.78 |  |
| Krippendorff’s $\alpha$ | 82.95 | 85.18 | 79.38 | 84.28 | 83.10 | 86.93 | 87.15 | 85.44 |  |
| Tone | Macro-F1 | 43.06 | 41.81 | 31.66 | 46.57 | 44.32 | 35.37 | 42.24 | 45.52 |
| Accuracy | 48.21 | 42.71 | 31.28 | 49.55 | 50.00 | 35.99 | 43.27 | 48.77 |  |
| Preference | 66.82 | 66.59 | 59.87 | 66.82 | 68.16 | 66.82 | 69.73 | 69.06 |  |
| Krippendorff’s $\alpha$ | 89.00 | 89.69 | 82.75 | 90.84 | 89.41 | 87.64 | 90.83 | 90.48 |  |

**Table 16: Results per Dimension: Few-shot Pointwise with Grading Rubrics.**
| Few-shot Pointwise with Rubrics | Qwen3-4B | Qwen3-32B | Llama3.1-8B | Llama3.3-70B | Llama4-Scout | o3 | gpt-4o | gpt-4.1 |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Overall | Macro-F1 | 31.18 | 35.45 | 22.24 | 30.52 | 32.84 | 27.92 | 29.57 | 33.84 |
| Accuracy | 37.71 | 38.59 | 26.25 | 37.63 | 39.92 | 35.93 | 38.19 | 39.01 |  |
| Preference | 39.35 | 42.87 | 37.15 | 38.56 | 41.22 | 44.68 | 45.87 | 44.00 |  |
| Krippendorff’s $\alpha$ | 82.36 | 84.46 | 77.00 | 81.45 | 83.07 | 81.54 | 84.84 | 84.24 |  |
| Fluency | Macro-F1 | 25.64 | 29.73 | 20.27 | 25.39 | 29.71 | 23.64 | 26.94 | 23.78 |
| Accuracy | 38.48 | 42.69 | 25.25 | 38.88 | 41.58 | 43.09 | 39.18 | 39.08 |  |
| Preference | 37.68 | 41.48 | 37.07 | 39.08 | 44.29 | 40.08 | 38.88 | 40.08 |  |
| Krippendorff’s $\alpha$ | 79.93 | 82.20 | 76.35 | 79.76 | 82.11 | 83.70 | 80.22 | 78.79 |  |
| Localized Factuality | Macro-F1 | 22.20 | 17.44 | 12.74 | 20.92 | 21.88 | 14.17 | 21.20 | 21.82 |
| Accuracy | 33.70 | 24.86 | 18.89 | 36.41 | 34.38 | 19.02 | 30.57 | 30.03 |  |
| Preference | 30.71 | 33.97 | 32.88 | 27.45 | 25.54 | 29.89 | 32.88 | 25.00 |  |
| Krippendorff’s $\alpha$ | 75.81 | 74.84 | 69.38 | 74.54 | 76.44 | 69.40 | 77.81 | 76.25 |  |
| Localized Tone | Macro-F1 | 25.26 | 31.27 | 19.86 | 23.14 | 27.45 | 29.00 | 28.83 | 29.46 |
| Accuracy | 34.88 | 38.85 | 31.13 | 33.89 | 38.41 | 42.05 | 36.64 | 37.53 |  |
| Preference | 39.51 | 43.27 | 32.67 | 38.19 | 42.16 | 52.32 | 48.57 | 47.46 |  |
| Krippendorff’s $\alpha$ | 78.56 | 81.55 | 77.51 | 75.65 | 79.13 | 81.34 | 81.15 | 80.48 |  |
| Tone | Macro-F1 | 37.11 | 41.90 | 25.14 | 38.16 | 39.60 | 28.43 | 36.46 | 43.29 |
| Accuracy | 43.05 | 45.07 | 28.48 | 41.03 | 44.17 | 35.65 | 44.96 | 47.87 |  |
| Preference | 48.21 | 51.35 | 45.29 | 47.53 | 49.78 | 54.26 | 61.66 | 60.54 |  |
| Krippendorff’s $\alpha$ | 88.02 | 89.72 | 80.00 | 87.71 | 87.76 | 84.60 | 90.78 | 90.73 |  |

**Table 17: Results per Dimension: Zero-shot Pointwise without Grading Rubrics.**
| Zero-shot Pointwise without Rubrics | Qwen3-4B | Qwen3-32B | Llama3.1-8B | Llama3.3-70B | Llama4-Scout | o3 | gpt-4o | gpt-4.1 |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Overall | Macro-F1 | 16.00 | 25.59 | 21.50 | 22.71 | 22.15 | 25.43 | 22.45 | 22.26 |
| Accuracy | 32.16 | 36.24 | 33.18 | 33.52 | 36.24 | 35.14 | 34.88 | 34.54 |  |
| Preference | 33.52 | 43.32 | 38.34 | 34.54 | 41.28 | 45.13 | 37.60 | 38.67 |  |
| Krippendorff’s $\alpha$ | 76.05 | 81.63 | 79.70 | 78.18 | 79.93 | 80.41 | 80.66 | 80.12 |  |
| Fluency | Macro-F1 | 10.84 | 19.70 | 20.73 | 21.11 | 23.11 | 21.85 | 15.96 | 17.38 |
| Accuracy | 34.37 | 39.18 | 36.77 | 37.17 | 37.58 | 40.48 | 36.37 | 35.77 |  |
| Preference | 32.46 | 39.48 | 40.08 | 35.27 | 41.48 | 48.10 | 31.46 | 31.66 |  |
| Krippendorff’s $\alpha$ | 73.46 | 78.53 | 79.31 | 77.32 | 78.84 | 80.15 | 76.22 | 74.64 |  |
| Localized Factuality | Macro-F1 | 12.94 | 19.93 | 15.22 | 18.30 | 17.03 | 13.46 | 22.09 | 20.02 |
| Accuracy | 31.39 | 32.20 | 29.62 | 33.56 | 32.34 | 20.52 | 32.34 | 31.79 |  |
| Preference | 26.90 | 34.51 | 27.45 | 23.10 | 26.90 | 31.52 | 23.37 | 29.08 |  |
| Krippendorff’s $\alpha$ | 71.05 | 78.12 | 75.43 | 70.85 | 72.31 | 70.56 | 75.15 | 75.63 |  |
| Localized Tone | Macro-F1 | 12.23 | 25.65 | 19.38 | 18.32 | 21.19 | 25.21 | 18.75 | 25.15 |
| Accuracy | 28.70 | 33.33 | 35.65 | 29.03 | 32.45 | 40.84 | 31.68 | 33.44 |  |
| Preference | 29.80 | 45.25 | 40.84 | 34.44 | 41.06 | 51.88 | 43.49 | 43.93 |  |
| Krippendorff’s $\alpha$ | 72.07 | 78.80 | 79.17 | 73.46 | 75.85 | 81.75 | 78.59 | 78.90 |  |
| Tone | Macro-F1 | 26.06 | 30.19 | 23.79 | 26.79 | 29.16 | 27.04 | 35.65 | 27.11 |
| Accuracy | 33.86 | 39.24 | 29.60 | 33.97 | 41.82 | 35.43 | 38.57 | 36.55 |  |
| Preference | 43.95 | 52.91 | 42.83 | 43.27 | 53.14 | 46.19 | 50.22 | 49.10 |  |
| Krippendorff’s $\alpha$ | 81.59 | 85.54 | 80.92 | 83.85 | 85.49 | 82.81 | 86.68 | 85.60 |  |

**Table 18: Results per Dimension: Zero-shot Pairwise without Grading Rubrics.**
| Zero-shot Pairwise without Rubrics | Qwen3-4B | Qwen3-32B | Llama3.1-8B | Llama3.3-70B | Llama4-Scout | o3 | gpt-4o | gpt-4.1 |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Overall | Macro-F1 | 32.74 | 38.10 | 30.89 | 35.12 | 35.21 | 37.60 | 36.74 | 37.35 |
| Accuracy | 40.74 | 41.90 | 33.10 | 42.44 | 41.53 | 40.12 | 41.79 | 44.45 |  |
| Preference | 54.08 | 59.23 | 49.55 | 56.29 | 55.10 | 57.98 | 56.85 | 56.96 |  |
| Krippendorff’s $\alpha$ | 82.44 | 85.66 | 81.73 | 83.97 | 83.99 | 84.46 | 85.38 | 84.23 |  |
| Fluency | Macro-F1 | 31.08 | 38.71 | 30.27 | 33.42 | 32.34 | 34.97 | 32.98 | 30.86 |
| Accuracy | 43.89 | 48.30 | 33.37 | 45.29 | 42.99 | 47.80 | 44.09 | 46.19 |  |
| Preference | 55.11 | 59.72 | 46.49 | 53.71 | 52.91 | 58.72 | 51.70 | 52.51 |  |
| Krippendorff’s $\alpha$ | 82.89 | 86.06 | 81.93 | 83.74 | 84.16 | 84.12 | 84.35 | 82.00 |  |
| Localized Factuality | Macro-F1 | 22.05 | 22.01 | 19.59 | 21.35 | 18.58 | 20.04 | 24.20 | 22.67 |
| Accuracy | 30.98 | 28.53 | 27.58 | 30.71 | 29.48 | 24.46 | 30.30 | 32.20 |  |
| Preference | 40.49 | 43.48 | 36.41 | 41.85 | 36.41 | 38.86 | 40.22 | 41.30 |  |
| Krippendorff’s $\alpha$ | 75.45 | 76.97 | 73.96 | 74.51 | 73.68 | 72.82 | 76.23 | 75.29 |  |
| Localized Tone | Macro-F1 | 32.51 | 36.30 | 30.43 | 35.10 | 34.11 | 40.45 | 36.38 | 37.36 |
| Accuracy | 43.27 | 42.38 | 34.77 | 44.59 | 43.38 | 45.36 | 45.25 | 48.01 |  |
| Preference | 55.85 | 63.36 | 52.54 | 60.93 | 62.25 | 65.78 | 66.45 | 64.02 |  |
| Krippendorff’s $\alpha$ | 82.27 | 86.05 | 81.30 | 84.23 | 83.41 | 86.12 | 86.30 | 85.10 |  |
| Tone | Macro-F1 | 36.97 | 42.94 | 34.84 | 40.71 | 43.33 | 39.54 | 42.42 | 44.67 |
| Accuracy | 42.71 | 45.29 | 35.65 | 46.75 | 47.98 | 39.13 | 45.18 | 48.99 |  |
| Preference | 62.33 | 67.49 | 60.76 | 66.37 | 65.70 | 65.02 | 66.59 | 67.71 |  |
| Krippendorff’s $\alpha$ | 84.96 | 89.18 | 86.05 | 87.84 | 88.36 | 88.49 | 89.52 | 88.98 |  |

**Table 19: Results per Dimension: SFT and RL trained Qwen3-4B and RL trained Llama4-Scout on All Data with Pointwise and Pairwise Scoring.**
| SFT and RL on All Data | Qwen3-4B SFT | Qwen3-4B RL | Llama4-Scout |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Pointwise | Pairwise | Pointwise | Pairwise | Pairwise-SFT | Pairwise-SFT+RL |  |  |
| Overall | Macro-F1 | 30.26 | 33.44 | 28.87 | 39.44 | 45.04 | 45.82 |
| Accuracy | 36.64 | 35.82 | 38.22 | 46.83 | 50.17 | 50.99 |  |
| Preference | 41.17 | 53.68 | 39.86 | 60.02 | 60.53 | 61.10 |  |
| Krippendorff’s $\alpha$ | 83.90 | 84.03 | 82.10 | 86.55 | 89.48 | 89.67 |  |
| Fluency | Macro-F1 | 28.41 | 32.91 | 25.10 | 35.72 | 46.42 | 47.52 |
| Accuracy | 38.18 | 37.17 | 39.38 | 52.51 | 55.21 | 56.71 |  |
| Preference | 38.48 | 54.71 | 42.89 | 61.92 | 66.13 | 66.53 |  |
| Krippendorff’s $\alpha$ | 82.36 | 85.16 | 80.36 | 85.69 | 90.77 | 90.86 |  |
| Localized Factuality | Macro-F1 | 20.30 | 19.51 | 17.49 | 20.62 | 25.30 | 25.87 |
| Accuracy | 31.66 | 25.14 | 33.42 | 33.02 | 34.78 | 35.33 |  |
| Preference | 35.87 | 39.67 | 26.63 | 38.86 | 36.68 | 36.68 |  |
| Krippendorff’s $\alpha$ | 76.39 | 74.43 | 75.37 | 77.04 | 80.25 | 80.14 |  |
| Localized Tone | Macro-F1 | 27.43 | 33.38 | 25.58 | 38.56 | 40.98 | 41.22 |
| Accuracy | 37.75 | 41.06 | 37.86 | 47.35 | 53.53 | 53.20 |  |
| Preference | 39.29 | 59.38 | 37.53 | 67.55 | 63.58 | 65.12 |  |
| Krippendorff’s $\alpha$ | 80.54 | 81.83 | 78.89 | 86.61 | 87.87 | 88.23 |  |
| Tone | Macro-F1 | 33.35 | 34.91 | 33.98 | 46.82 | 51.08 | 52.29 |
| Accuracy | 37.89 | 37.78 | 41.26 | 51.35 | 53.81 | 55.27 |  |
| Preference | 50.45 | 58.30 | 49.78 | 67.71 | 70.85 | 71.08 |  |
| Krippendorff’s $\alpha$ | 88.08 | 87.91 | 86.67 | 90.39 | 92.63 | 93.00 |  |

**Table 20: Comparison of zero-shot Pairwise Qwen3-4B and RL trained models, trained either jointly across all dimensions (multi-task) or individually per dimension (single-task).**
| Dimension | Macro-F1 | Preference Accuracy |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
| Zero-shot | Multi-task | Single-task | Zero-shot | Multi-task | Single-task |  |
| Fluency | 32.24 | 35.72 | 37.14 | 55.91 | 61.92 | 61.32 |
| Tone | 43.06 | 46.82 | 46.18 | 66.82 | 67.71 | 69.28 |
| Localized Tone | 32.82 | 38.56 | 37.61 | 60.49 | 67.55 | 66.67 |
| Localized Factuality | 22.55 | 20.62 | 23.12 | 42.93 | 38.86 | 42.12 |

**Table 21: Results per Dimension: RL trained Qwen3-4B on Pairwise Single Dimension Data and Pairwise English Data.**
|  | Single Dimension Data on All languages | English Data on |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
| Pairwise RL on Partial Data | Fluency | Localized Factuality | Localized Tone | Tone | All Categories |  |
| Overall | Macro-F1 | 37.89 | 35.33 | 37.46 | 38.55 | 34.34 |
| Accuracy | 44.56 | 43.74 | 43.69 | 45.19 | 42.33 |  |
| Preference | 59.29 | 56.68 | 59.29 | 57.53 | 56.46 |  |
| Krippendorff’s $\alpha$ | 85.63 | 84.06 | 86.12 | 86.20 | 83.98 |  |
| Fluency | Macro-F1 | \cellcolor[HTML]F3F3F337.14 | 31.53 | 36.07 | 34.82 | 30.65 |
| Accuracy | \cellcolor[HTML]F3F3F351.80 | 47.39 | 50.00 | 50.60 | 45.59 |  |
| Preference | \cellcolor[HTML]F3F3F361.32 | 56.91 | 58.72 | 56.51 | 55.31 |  |
| Krippendorff’s $\alpha$ | \cellcolor[HTML]F3F3F385.87 | 82.98 | 86.09 | 84.85 | 82.64 |  |
| Localized Factuality | Macro-F1 | 20.12 | \cellcolor[HTML]F3F3F323.12 | 18.87 | 19.78 | 19.92 |
| Accuracy | 29.89 | \cellcolor[HTML]F3F3F334.10 | 29.21 | 30.71 | 32.61 |  |
| Preference | 42.12 | \cellcolor[HTML]F3F3F342.12 | 40.22 | 41.03 | 41.85 |  |
| Krippendorff’s $\alpha$ | 75.19 | \cellcolor[HTML]F3F3F376.65 | 75.83 | 76.58 | 76.44 |  |
| Localized Tone | Macro-F1 | 35.84 | 30.29 | \cellcolor[HTML]F3F3F337.61 | 37.32 | 28.51 |
| Accuracy | 44.92 | 42.49 | \cellcolor[HTML]F3F3F347.57 | 45.92 | 41.28 |  |
| Preference | 63.36 | 58.72 | \cellcolor[HTML]F3F3F366.67 | 60.49 | 57.40 |  |
| Krippendorff’s $\alpha$ | 84.20 | 82.01 | \cellcolor[HTML]F3F3F386.28 | 84.87 | 80.87 |  |
| Tone | Macro-F1 | 43.01 | 43.96 | 41.61 | \cellcolor[HTML]F3F3F346.18 | 42.50 |
| Accuracy | 48.21 | 48.88 | 44.62 | \cellcolor[HTML]F3F3F350.34 | 47.76 |  |
| Preference | 67.04 | 66.37 | 68.16 | \cellcolor[HTML]F3F3F369.28 | 68.83 |  |
| Krippendorff’s $\alpha$ | 89.66 | 89.28 | 89.61 | \cellcolor[HTML]F3F3F391.28 | 88.89 |  |

**Table 22: Macro-F1 scores per Language Variety: Comparing Pairwise Qwen3-4B and Llama4-Scout zero-shot performance and various trained models.**
| PAIRWISE | Qwen3-4B | Llama4-Scout |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Zero-shot | SFT | RL | RL on EN-only data | Zero-shot | SFT | SFT + RL |  |
| Overall | 35.46 | 33.55 | 39.44 | 34.34 | 36.11 | 44.17 | 45.82 |
| ar | 31.21 | 21.92 | 36.26 | 33.19 | 32.29 | 36.20 | 43.71 |
| ar_Latn_EG | 16.41 | 22.60 | 9.62 | 17.52 | 18.71 | 54.88 | 65.12 |
| bg_BG | 29.95 | 28.12 | 37.80 | 26.39 | 26.75 | 31.31 | 33.02 |
| bn_BD | 24.20 | 15.08 | 20.17 | 18.04 | 13.84 | 20.14 | 23.37 |
| cs_CZ | 34.58 | 18.81 | 38.01 | 34.44 | 35.39 | 41.97 | 44.86 |
| da_DK | 21.50 | 17.91 | 29.43 | 24.78 | 31.71 | 34.98 | 40.73 |
| de_DE | 27.63 | 26.70 | 24.76 | 17.94 | 16.51 | 37.35 | 34.26 |
| el_GR | 36.90 | 37.84 | 43.74 | 40.31 | 39.68 | 45.11 | 41.64 |
| en_AU | 34.94 | 40.74 | 55.79 | 35.79 | 42.68 | 41.19 | 41.26 |
| en_GB | 44.86 | 46.77 | 46.13 | 48.93 | 42.52 | 40.69 | 51.55 |
| en_IN | 39.11 | 27.62 | 36.47 | 48.68 | 44.96 | 37.31 | 39.40 |
| en_US | 33.42 | 34.49 | 29.09 | 33.25 | 47.68 | 28.35 | 23.13 |
| es_ES | 38.97 | 29.93 | 41.21 | 30.53 | 29.23 | 27.44 | 27.07 |
| es_MX | 42.01 | 24.56 | 51.77 | 46.63 | 44.57 | 36.80 | 38.98 |
| fa_IR | 31.71 | 33.25 | 39.11 | 33.04 | 38.75 | 33.91 | 43.11 |
| fr_FR | 21.33 | 29.53 | 39.02 | 25.13 | 30.49 | 19.65 | 33.97 |
| gu_IN | 30.00 | 37.60 | 46.05 | 30.10 | 49.79 | 46.29 | 43.01 |
| he_IL | 24.89 | 19.71 | 25.75 | 26.86 | 22.66 | 32.50 | 38.17 |
| hi_IN | 23.46 | 16.16 | 36.77 | 30.04 | 27.91 | 28.81 | 36.42 |
| hi_Latn_IN | 48.52 | 27.56 | 41.53 | 39.74 | 34.52 | 25.79 | 53.80 |
| hr_HR | 20.16 | 34.86 | 25.15 | 16.69 | 18.45 | 26.57 | 29.01 |
| hu_HU | 27.69 | 36.63 | 37.06 | 28.84 | 33.83 | 40.92 | 39.55 |
| id_ID | 41.40 | 45.21 | 42.53 | 31.33 | 35.64 | 41.28 | 42.98 |
| it_IT | 29.48 | 34.77 | 40.28 | 31.61 | 29.16 | 24.51 | 29.34 |
| ja_JP | 45.06 | 40.34 | 39.68 | 44.76 | 35.37 | 44.98 | 41.55 |
| ko_KR | 35.51 | 35.14 | 33.03 | 32.59 | 39.12 | 38.38 | 48.80 |
| mr_IN | 41.44 | 36.52 | 51.06 | 47.78 | 42.84 | 50.10 | 51.59 |
| ms_MY | 29.50 | 30.79 | 39.74 | 33.02 | 28.99 | 36.27 | 44.60 |
| ne_NP | 28.05 | 22.45 | 27.59 | 19.41 | 24.62 | 20.16 | 24.54 |
| nl_NL | 36.00 | 35.38 | 38.80 | 27.61 | 45.25 | 54.06 | 51.22 |
| pl_PL | 28.20 | 33.52 | 21.82 | 23.77 | 17.94 | 29.00 | 21.29 |
| pt_BR | 45.17 | 36.52 | 48.48 | 43.34 | 48.19 | 48.45 | 41.83 |
| pt_PT | 38.72 | 39.29 | 44.62 | 41.22 | 38.89 | 52.15 | 41.57 |
| ro_RO | 37.54 | 42.92 | 49.36 | 46.85 | 51.91 | 52.42 | 55.46 |
| ru_RU | 31.75 | 22.59 | 22.44 | 19.78 | 28.84 | 18.72 | 21.61 |
| sk_SK | 35.80 | 38.22 | 44.14 | 33.79 | 37.20 | 44.29 | 40.69 |
| sv_SE | 31.59 | 35.11 | 33.78 | 27.44 | 34.29 | 39.48 | 40.81 |
| sw_KE | 41.97 | 28.13 | 41.88 | 21.36 | 43.13 | 37.33 | 39.67 |
| th_TH | 37.07 | 32.75 | 47.04 | 35.98 | 45.82 | 52.77 | 55.11 |
| tl_PH | 40.71 | 29.19 | 45.39 | 42.52 | 39.95 | 40.54 | 37.78 |
| tr_TR | 50.03 | 37.30 | 48.50 | 40.67 | 40.66 | 40.45 | 45.42 |
| uk_UA | 24.09 | 29.20 | 20.45 | 22.09 | 17.57 | 23.98 | 27.13 |
| ur_Latn_PK | 29.38 | 38.54 | 34.25 | 27.72 | 32.49 | 32.93 | 29.59 |
| ur_PK | 23.21 | 38.81 | 36.11 | 28.29 | 39.87 | 48.94 | 43.73 |
| vi_VN | 33.91 | 31.07 | 34.76 | 35.16 | 30.35 | 40.51 | 38.46 |
| zh_CN | 40.82 | 27.78 | 51.99 | 41.15 | 37.42 | 45.27 | 41.58 |
| zh_TW | 35.38 | 25.47 | 37.07 | 44.80 | 37.93 | 39.29 | 38.09 |

**Table 23: Preference accuracy per Language: Comparing Pairwise Qwen3-4B and Llama4-Scout zero-shot performance and various trained models.**
| PAIRWISE | Qwen3-4B | Llama4-Scout |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Zero-shot | SFT | RL | RL on EN-only data | Zero-shot | SFT | SFT + RL |  |
| Overall | 57.13 | 53.51 | 60.02 | 56.46 | 56.25 | 60.08 | 61.10 |
| ar | 63.16 | 50.00 | 60.53 | 57.89 | 61.54 | 73.68 | 55.26 |
| ar_Latn_EG | 35.48 | 25.81 | 38.71 | 45.16 | 28.12 | 93.55 | 93.55 |
| bg_BG | 51.28 | 56.41 | 48.72 | 38.46 | 47.50 | 46.15 | 46.15 |
| bn_BD | 41.38 | 27.59 | 37.93 | 44.83 | 36.67 | 41.38 | 41.38 |
| cs_CZ | 69.23 | 74.36 | 76.92 | 69.23 | 70.00 | 74.36 | 74.36 |
| da_DK | 35.90 | 38.46 | 43.59 | 46.15 | 40.00 | 51.28 | 46.15 |
| de_DE | 51.61 | 54.84 | 58.06 | 38.71 | 59.38 | 54.84 | 54.84 |
| el_GR | 56.41 | 56.41 | 61.54 | 66.67 | 57.50 | 66.67 | 64.10 |
| en_AU | 34.21 | 47.37 | 39.47 | 36.84 | 34.21 | 39.47 | 44.74 |
| en_GB | 53.85 | 64.10 | 53.85 | 58.97 | 43.59 | 46.15 | 58.97 |
| en_IN | 43.59 | 41.03 | 48.72 | 43.59 | 51.28 | 51.28 | 43.59 |
| en_US | 53.12 | 46.88 | 40.62 | 62.50 | 43.75 | 50.00 | 46.88 |
| es_ES | 64.10 | 51.28 | 48.72 | 56.41 | 51.28 | 51.28 | 51.28 |
| es_MX | 53.85 | 35.90 | 56.41 | 46.15 | 43.59 | 51.28 | 51.28 |
| fa_IR | 66.67 | 69.23 | 66.67 | 61.54 | 56.41 | 58.97 | 79.49 |
| fr_FR | 60.53 | 55.26 | 57.89 | 52.63 | 55.26 | 42.11 | 60.53 |
| gu_IN | 48.65 | 43.24 | 64.86 | 64.86 | 67.57 | 56.76 | 62.16 |
| he_IL | 58.97 | 61.54 | 61.54 | 64.10 | 61.54 | 48.72 | 64.10 |
| hi_IN | 53.12 | 46.88 | 62.50 | 59.38 | 59.38 | 59.38 | 71.88 |
| hi_Latn_IN | 61.54 | 48.72 | 66.67 | 71.79 | 62.50 | 56.41 | 64.10 |
| hr_HR | 43.59 | 56.41 | 48.72 | 51.28 | 51.28 | 51.28 | 51.28 |
| hu_HU | 55.26 | 52.63 | 63.16 | 63.16 | 60.53 | 65.79 | 65.79 |
| id_ID | 75.68 | 70.27 | 81.08 | 75.68 | 78.38 | 83.78 | 81.08 |
| it_IT | 59.38 | 59.38 | 62.50 | 46.88 | 53.12 | 59.38 | 59.38 |
| ja_JP | 74.36 | 71.79 | 71.79 | 61.54 | 61.54 | 71.79 | 74.36 |
| ko_KR | 63.16 | 65.79 | 76.32 | 68.42 | 71.05 | 81.58 | 84.21 |
| mr_IN | 56.41 | 53.85 | 69.23 | 58.97 | 56.41 | 66.67 | 64.10 |
| ms_MY | 58.97 | 53.85 | 58.97 | 58.97 | 61.54 | 56.41 | 61.54 |
| ne_NP | 52.63 | 57.89 | 44.74 | 44.74 | 50.00 | 47.37 | 42.11 |
| nl_NL | 60.53 | 65.79 | 63.16 | 50.00 | 63.16 | 71.05 | 71.05 |
| pl_PL | 52.63 | 47.37 | 55.26 | 57.89 | 57.89 | 47.37 | 60.53 |
| pt_BR | 53.85 | 53.85 | 41.03 | 53.85 | 38.46 | 51.28 | 51.28 |
| pt_PT | 44.74 | 44.74 | 55.26 | 47.37 | 50.00 | 57.89 | 47.37 |
| ro_RO | 58.97 | 64.10 | 76.92 | 56.41 | 58.97 | 64.10 | 74.36 |
| ru_RU | 50.00 | 40.62 | 46.88 | 40.62 | 53.12 | 43.75 | 50.00 |
| sk_SK | 69.23 | 56.41 | 74.36 | 66.67 | 58.97 | 69.23 | 66.67 |
| sv_SE | 56.41 | 41.03 | 46.15 | 53.85 | 43.59 | 48.72 | 35.90 |
| sw_KE | 56.41 | 58.97 | 71.79 | 56.41 | 66.67 | 69.23 | 69.23 |
| th_TH | 53.85 | 53.85 | 51.28 | 58.97 | 58.97 | 48.72 | 51.28 |
| tl_PH | 69.23 | 61.54 | 71.79 | 58.97 | 64.10 | 71.79 | 66.67 |
| tr_TR | 71.79 | 61.54 | 82.05 | 71.79 | 79.49 | 84.62 | 76.92 |
| uk_UA | 58.97 | 38.46 | 66.67 | 51.28 | 46.15 | 64.10 | 61.54 |
| ur_Latn_PK | 53.85 | 66.67 | 61.54 | 53.85 | 55.00 | 66.67 | 64.10 |
| ur_PK | 69.23 | 43.59 | 69.23 | 53.85 | 66.67 | 58.97 | 64.10 |
| vi_VN | 51.28 | 48.72 | 56.41 | 58.97 | 56.41 | 58.97 | 58.97 |
| zh_CN | 61.54 | 48.72 | 64.10 | 61.54 | 64.10 | 66.67 | 61.54 |
| zh_TW | 84.62 | 66.67 | 82.05 | 74.36 | 76.92 | 79.49 | 79.49 |

## Peer Reviews

confidence: 3
contribution: 3
initial_rating: 6
presentation: 3
questions: 1. Do the four evaluation dimensions (Fluency, Tone, Localized Tone, Localized Factuality) include moral or behavioral conventions that are specific to certain regions or cultures?
2. Pairwise evaluation involves comparing two responses at a time. Do the authors plan to conduct multi-sample comparison experiments, examining whether the presence or absence of rubrics affects performance?
soundness: 3
strengths: - Extensive Multilingual Coverage Demonstrates strong value by tackling the critical aspect of native-like linguistic quality, incorporating human validation across a broad set of languages.
Provides substantial experimental diversity with 47 languages/varieties, 6,423 prompt–response pairs, and 81,014 annotations, ensuring wide multilingual representation.
- Well-Structured and Systematic EvaluationOffers quantitative evidence that zero-shot pairwise evaluation consistently outperforms pointwise evaluation in both Macro-F1 and preference accuracy.
Empirically validates the practical utility of rubric-guided evaluation showing benefits for both absolute and comparative assessments, indicating strong applicability in real-world evaluation pipelines.
summary: This paper introduces MENLO, a framework for evaluating the native-like quality of multilingual LLM responses based on audience-design principles. MENLO defines four core dimensions, Fluency, Tone, Localized Tone, and Localized Factuality and provides localized prompt templates to elicit culturally grounded responses.
MENLO further contributes a large scale multilingual human evaluation dataset comprising 6,423 prompt-response pairs with 81,014 annotations spanning 47 languages/varieties, achieving strong inter-annotator agreement (avg. Krippendorff’s α = 0.84).
The paper empirically demonstrates the superiority of pairwise over pointwise judgments for aligning evaluator preferences, and shows that incorporating structured rubrics improves pointwise reliability.
Additionally, the authors explore LLM-Judge training methods, including multi-task reinforcement learning and reward shaping to boost automatic judge models closer to human-level performance.
Finally, the trained judges are shown to be effective as generative reward models, improving policy models’ multilingual proficiency. However, the study highlights that LLM-based evaluations can overestimate model performance relative to human raters, revealing an important limitation.
weaknesses: - MENLO uses localized prompts and employs translators/annotators from the corresponding regions, there still exist diverse linguistic variations within a single locality across social groups, age ranges, and socioeconomic classes. Therefore, constructing gold labels based on a single demographic standard may introduce bias toward a particular perspective.
- From a user perspective, even if a response appears native-like, its practical utility drops significantly if it is factually incorrect especially with respect to localized factual knowledge. If the reward model does not account for factuality, the system may optimize for style while increasing the risk of conveying incorrect information.

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 3
questions: none
soundness: 3
strengths: - The paper plans to open-source its dataset. A high-quality training dataset is currently in urgent need by the open-source community.
- The paper’s dataset primarily contributes to the improvement of LLM-as-Judge — a critical need for the current RL paradigm, and a highly important topic that requires further advancement.
summary: This manuscript contributes to LLM-as-Judge by proposing MENLO, building a 6k+ human-annotated dataset, establishing four quality evaluation dimensions, showing RL's impact on LLM-as-judge and the dataset's benefit to multilingual RL, though the RL-enhanced LLM-as-judge still lags behind human annotation.
weaknesses: The multilingual section of the paper is derived from direct translation, which may lack native-specific traits. For instance, the level of attention paid to art can vary across different countries, and this discrepancy may introduce a certain degree of bias.

---

confidence: 5
contribution: 4
initial_rating: 8
presentation: 2
questions: 1. In Table 2, Menlo is displayed with some benchmarks (e.g., MM-Eval and M-RewardBench), it might be good to specify the number of instances used as benchmarking (1776 instances).

2. The Biggen-bench [1] also includes a multilingual category with 420 human judgments, it would be great if the authors could add it to Table 2.

[1] Kim, S., Suk, J., Cho, J. Y., Longpre, S., Kim, C., Yoon, D., ... & Seo, M. (2025, April). The BiGGen Bench: A Principled Benchmark for Fine-grained Evaluation of Language Models with Language Models. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers) (pp. 5877-5919).
soundness: 4
strengths: 1. The Menlo dataset was annotated with 1934 annotators and also has high krippendorff's alpha value, indicating that it is a very high-quality dataset.

2. The list of experiments is very extensive: starting from gathering a dataset, training a judge, and then using the judge to train a generator. Every step is supported by a experimental result, which is very helpful for future work in this area.
summary: This works proposes a dataset and a LLM-judge based evaluation framework to evaluate whether an LMs' response is native-like quality. Specifically, the authors attempt to assess four aspects: (1) language quality/coherence, (2) alignment with cultural/lignuistic nuance, (3) factual correctness, (4) overall writing style and helpfulness. The dataset (Menlo) spans across 47 languages and includes 6423 instances. Then, the authors evaluate different models in point wise and pair wise settings when functioning as a judge to evaluate responses. Also, the authors train a judge model with RL and show that it could match or outperform the performance of Qwen3-32B and GPT-4.1. The authors also show that this judge model could also work as a generative reward model to improve LMs' ability to write fluent responses.
weaknesses: 1. Considering that the prompts consisting the Menlo dataset were derived using a prompt template, it could be possible that the prompts are synthetic in nature. There should be an explanation to which extent these prompts mirror what non-English speakers might prompt to frontier models in real-world scenarios. One possible way would be to gather the prompts on WildChat (that are non-English) and measure how similar the two groups are compared to other multilingual meta-eval benchmarks such as M-RewardBench or MM-Eval.

2. The fact that "pairwise ranking" yields higher performance compared to "pointwise" is somewhat trivial - the caveat should be that it incurs higher inference costs. Yet, the paper simply concludes that pairwise has higher performance compared to pointwise. For a fair comparison, since pointwise is O(N) and pairwise is O(N^2), it is possible to prompt the LLM judge N times and apply either score averaging [1,2] or self-consistency [3].

3. In Table 4, it shows that adding Rubrics help the models to predict better judgments. When looking into the rubrics in Appendix A.3., these are explanations of each criteria, not instance-specific rubrics [3]. I think the term "context-specific rubrics" should be adjusted to "detailed rubrics" for readers to not confuse with instance-specific rubrics. Also, it would be nice to add a qualitative example of how the chain-of-thoughts generated by the judges differ when including or excluding the rubrics.

4. In Figure3, there are some languages that benefit more and other languages that do not. What is the difference? Is it the limitation of RL training (that means there should be more pretraining for some languages) or is there another reason behind it that the authors conjecture?

5. Looking at the one of the qualitative example between Figure 9-16 (I won't specify for anonymity), I'm not sure if the grades make sense, which makes me a bit skeptical if the large group of annotators were instructed and managed properly. Can you give me details of how these people were recruited, how much they were incentivized, and how they were qualified?

[1] Kim, S., Shin, J., Cho, Y., Jang, J., Longpre, S., Lee, H., ... & Seo, M. (2023, October). Prometheus: Inducing fine-grained evaluation capability in language models. In The Twelfth International Conference on Learning Representations.
[2] Kim, S., Suk, J., Longpre, S., Lin, B. Y., Shin, J., Welleck, S., ... & Seo, M. (2024, November). Prometheus 2: An Open Source Language Model Specialized in Evaluating Other Language Models. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing (pp. 4334-4353).
[3] Kim, S., Suk, J., Cho, J. Y., Longpre, S., Kim, C., Yoon, D., ... & Seo, M. (2025, April). The BiGGen Bench: A Principled Benchmark for Fine-grained Evaluation of Language Models with Language Models. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers) (pp. 5877-5919).

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: I want to express sincere appreciation for the authors' substantial effort in conducting this large-scale, ambitious research. Please consider the following recommendations to strengthen the paper:

< Questions >
Can you provide a more precise operational definition of 'native-level quality'? Please provide theoretical rationale for whether the four dimensions are sufficient or necessary conditions, and why other important aspects (pragmatics, humor, dialect) were excluded.

The distinction between Tone and Localized Tone sometimes appears ambiguous (e.g., fairness depends on cultural context). Can you clarify the conceptual boundary between these two dimensions?

How were reward function coefficients selected (e.g., partial reward +0.5, preference bonus +1, format penalty -0.2)? Did you perform sensitivity analysis? 

< Suggestions for Improvement >
Strengthen Statistical Rigor: Perform and report statistical significance tests for all major claims. Particularly provide p-values, confidence intervals, and effect sizes for SFT vs. RL, pairwise vs. pointwise evaluation, and cross-language comparisons. Use bootstrapping or permutation tests to obtain robust estimates.

Clarify Causal Mechanisms: Conduct additional experiments revealing why pairwise evaluation is superior. Include presenting same response twice, systematically varying response order, and analyzing correlation between response quality differences and performance improvements.

Improve Baselines: Add SFT baseline with reasoning to fairly compare with RL. Also include other RL algorithms like DPO or PPO to demonstrate GRPO is best.

Compare with Existing Metrics: Compare MENLO with established multilingual metrics like BLEURT, COMET, BERTScore. Include correlation, relative strengths/weaknesses, and language-specific performance analysis.
soundness: 2
strengths: Methodological Rigor: Operationalizing sociolinguistic theory (Bell's Style Axiom and audience design) into a computational evaluation framework is a strong point. This goes beyond simply translated prompts or general naturalness evaluation to capture the actual phenomenon of native speakers adjusting language to specific addressees or reference groups. The decomposition into four dimensions is theoretically motivated, and developing detailed rubrics for each dimension effectively reduced subjectivity.

Data Quality and Scale: The MENLO dataset surpasses existing multilingual benchmarks in several aspects. The scale covering 47 language varieties, human-written and localized prompts, and Krippendorff's α of 0.84 are all impressive. Particularly, the fine-grained distinction of language varieties (e.g., European vs. Mexican Spanish, various English varieties) reflects actual linguistic diversity. Deploying expert annotators per language and developing customized annotation tools are also best practices in data quality management.

Systematic Experimentation: Systematic comparisons of pairwise vs. pointwise evaluation, with/without rubrics, and SFT vs. RL clearly show the contribution of each design choice. Particularly, the finding that pairwise evaluation achieves up to 12.4% higher Macro-F1 than pointwise evaluation and is more effective than few-shot examples is significant. Ablation studies on reward shaping also quantify each reward component's contribution, demonstrating the importance of composite reward design in GRPO.

Practical Value: Achieving actual model improvement by using trained judges as reward models demonstrates the framework's practicality. They closed the evaluation-improvement loop, suggesting real applicability beyond theoretical benchmarks. The commitment to releasing datasets and code will also contribute to reproducibility and facilitate follow-up research.

Transparent Limitation Reporting: Honestly reporting the finding that LLM evaluators overestimate improvements by +0.5~0.6 compared to humans demonstrates academic honesty. Acknowledging the difficulty of the Localized Factuality dimension and suggesting alternative approaches (retrieval, tool use) is also positive.
summary: This paper proposes a framework for evaluating and improving whether large language models generate native-level natural responses across 47 languages. The goal is to evaluate whether models can converse like actual native speakers, beyond mere grammatical accuracy. To this end, the researchers applied the sociolinguistic principle of audience design, which is the concept that speakers adjust their language style according to their addressee or reference group. The framework divides native-level quality into four core dimensions. First, Fluency evaluates accuracy of vocabulary and grammar, consistency of sentence structure, and clarity. Second, Tone judges the overall writing style—whether it is helpful, insightful, engaging, and fair. Third, Localized Tone evaluates alignment with cultural, regional, and linguistic nuances, assessing appropriate use of local expressions and cultural sensitivity. Fourth, Localized Factuality evaluates factual accuracy and completeness, and whether information is provided based on local context. The researchers experimented with whether trained judge models could be used as generative reward models to directly improve the language capabilities of policy models. Using Qwen3-4B-RL-Judge as a reward model to subsequently train the base Qwen3-4B model, all three LLM evaluators (Qwen3-32B, gpt-4.1, Llama4-Scout-RL-Judge) confirmed improvements. Win rates ranged from 63.4% to 77.9%, with average score improvements from +0.80 to +1.16.However, there was an important finding. Human evaluation performed on 10 languages revealed that human evaluators were more conservative with a win rate of 55.7% and average score improvement of +10.9%.
weaknesses: Insufficient Causal Verification: There's inadequate causal analysis of why pairwise evaluation is superior. It's not distinguished whether comparison itself is important, whether it's due to context richness from simultaneous exposure to two responses, or whether relative judgment is simply easier than absolute judgment. Without control experiments like pairwise evaluation of a single response (presenting the same response twice), mechanistic understanding is limited.

Lack of Statistical Rigor: Most results lack confidence intervals or significance tests. Especially given the high variability in language-specific performance (37.9%~82.1%), it's difficult to judge whether observed differences are statistically significant. Multiple comparisons across several models and settings were performed, so multiple testing corrections like Bonferroni are needed but not applied. Additionally, an average of 137 prompts per language may lack statistical power for some languages.

Missing Comparative Experiments: There's no comparison with existing multilingual quality metrics (BLEURT, COMET, BERTScore), making it difficult to judge MENLO's relative advantages. Also, there's no comparison with other RL algorithms (PPO, DPO, RLAIF) or reward designs, so we cannot know if the proposed GRPO+composite reward is optimal. Human evaluation was only performed on 10 languages, limiting generalization to all 47 languages.

Methodological Bias: The SFT baseline may be unfairly weak. Training without Chain-of-Thought doesn't provide fair comparison with RL, and SFT with reasoning should also be included. Additionally, justification for model selection (Qwen3-4B, Llama4-Scout) is lacking, and result generalization to other sizes or architectures is uncertain.


Decision: Accept (Poster)