Evaluating LLMs on Chinese Idiom Translation
Authors: Cai Yang
Yao Dou
David Heineman
Xiaofeng Wu
Wei Xu
Venue: COLM 2025 (2025)
Keywords: Large Language Model Evaluation
Chinese Idioms
Meta Analysis
Multilingual Evaluation
TL;DR: We evaluate large language models on Chinese idiom translation across multiple domains and introduce new metrics that reliably measures idiom translation quality, significantly outperforming existing translation metrics.

Abstract: Idioms, whose figurative meanings usually differ from their literal interpretations, are common in everyday language, especially in Chinese, where they often contain historical references and follow specific structural patterns. Despite recent progress in machine translation with large language models, little is known about Chinese idiom translation. In this work, we introduce IdiomEval, a framework with a comprehensive error taxonomy for Chinese idiom translation. We annotate 900 translation pairs from nine modern systems, including GPT-4o and Google Translate, across four domains: web, news, Wikipedia, and social media. We find these systems fail at idiom translation, producing incorrect, literal, partial, or even missing translations. The best-performing system, GPT-4, makes errors in 28\% of cases. 
We also find that existing evaluation metrics measure idiom quality poorly with Pearson correlation below 0.48 with human ratings. We thus develop improved models that achieve F$_1$ scores of 0.68 for detecting idiom translation errors.

Published as a conference paper at COLM 2025
Evaluating LLMs on Chinese Idiom Translation
Cai Yang', Yao Dou², David Heineman', Xiaofeng Wu?, Wei Xu² "Independent Contributor "Georgia Institute of Technology caiyang. cywoutlook. com, {douy, (douy,david, heineman, www.annn..wut14eeaatcc.cc.c wei.xu@ce wei.xuecc.aatech. edu
gatech.
Abstract
Idioms, whose figurative meanings usually differ from their literal interpre- tations, are common in everyday language, especially in Chinese, where they often contain historical references and follow specific structural pat- terns. Despite recent progress in machine translation with large language models, little is known about Chinese idiom translation. In this work, introduce IDIOMEVAL, a framework with a comprehensive error taxonomy for Chinese idiom translation. We annotate 900 translation pairs from nine modern systems, including GPT-4o and Google Translate, across four do- mains: web, news, Wikipedia, and social media. We find these systems fail at idiom translation, producing incorrect, literal, partial, or even missing translations. The best-performing system, GPT-4, makes errors in 28% of cases. We also find that existing evaluation metrics measure idiom quality poorly with Pearson correlation below 0.48 with human ratings. We thus develop improved models that achieve F1 scores of 0.68 for detecting idiom translation errors.
Introduction
Input: F X
* F
0.00
BERTScore
76.12
Reforence: Before passing away, my dad said he had lived a life of poverty and hadn't left me anything.....
77.29
75.93
54.57
COMETKiNi
51.27
GPT-4:
Owan:
Google:
mIO
MetricX
9.82
10.23
heteer
fotteer
My befter
(onillae in
MetricX-QE
15.53
was
the
sald before
at the end his lle
transiation)
Sytoms
Metric scores not correlated with human retingg
cied thae
NEror Transistion
translate
Pestsonn Conreatiion
no special akill
86
strengits, iddn1t
and hodnl loft
and had
lcavo me
50
anything you for
anything betted
nothing
0.27
0.5
Mistranslation
Mistranalaation
Literal Translation
No Translation
No
BLSU
COMETIION BEmSSoore
COMET
Matrfxx
O6
Translation
GPT4 Ownn Alma mTO GTrans
Figure 1: Top left: example of Chinese input and translations from five systems. Top right: percentage of good translations in each system. Bottom left: evaluation metric scores of system translations. Bottom right: Pearson's 's between human rating and metric scores.
Idioms are phrases whose meanings typically differ from the literal meanings of their individual components. They are commonly used in everyday language, making their understanding crucial for tasks such as neural machine translation (Baziotis al., 2023) and writing assistance (Tan & Jiang, 2021; Dankers et al., 2022a). In particular, idioms have posed challenge translation as their meaning is usually non-compositional and cannot be inferred from their literal parts. For instance, my two cents means one's viewpoint, rather than actual money. Increasing attention has been paid to idiom translation recent years. Dankers et al. (2022b) has found that transformer encoders tend to process figurative idioms as single units more strongly, while Baziotis al. (2023); Liu et al. (2023) have shown different training methods can be helpful for idiom translation.
However, much of this research has focused on idioms in English, Japanese, and European languages, with little attention given to Chinese idioms. From linguistics perspective, the

---

Published as a conference paper at COLM 2025
Eroor Type Description
Example
Example (en)
50 R2
..Scimitals dlwene and rejicedd, but they discovered that..
Translation is of high qualitt.
....B 24
...oll tycoons stab the US dolaar again, King Zhuang's S swoodd Bance nimed at Pei Gong...
Mistranslationt Translation incomect disrupts idsoo understanding
Asking too many questions only bring them problems. Urring yoursolf prohiens)
Unnatural Translation understand- able yet sabuptimal,
warior not achieve a aiglaring, victory..... rosorlaatie victury)
im-
proved.
Litend Translation is literal and cohur- ent with the surrounding contied.
..I have merits in this life.. (fite ie pointly)
Addition: Translation contains irrel- evant content beyond the accurate ewaning.
...ttt only venle- in drama stadies, knowledgeable and sionate aspests. (Nuoollfgaatic
aspecis)
Partal Given idsoon translated par- tally.
there is no plane to stay, and againes these bad inns... (if you don't stoy here, torre is piace to otay chrmffen)
Repetition tranelation contains
the best relationship between fother and son should be like that of broliser and brolist... (lile list arothees)
repeated correct cuntent synonyms.
No Iransationt Transaatonn lacks the aliom's monning in the oulpul.
...was Byynng from Bloemfontein to Pretoria and had four other passengers on board, including the stake. (Ih unaviroone stale))
Table 1: The IDIOMEVAL error taxonomy with example translations with each error type taken from our collected dataset in Section 3.1. The corrected span included in italics. Participants can select More than One if there is more than one error present.
former falls into the Indo-Furopean language family, while Chinese belongs to the Sino- Tibetan language family (Katzner & Miller, 2002). Besides, idioms across languages differ in their definitions and classifications. For example, while English idioms often emphasize syntactic structures, Chinese idioms place more importance on semantic and pragmatic aspects (Wang, 2021). Meanwhile, Chinese idioms generally contain proverbs, allegorical sayings, and aphorisms which mostly originate from ancient literature (Wang, 2021; 2022). One example is the idiom B TE which first appeared in the Song dynasty (ruled China from 960 to 1279 CF) and literally translated into "tomorrow's yellow flower", but its figurative meaning is "things that have become outdated". Due to cultural and linguistic differences (Wang, 2022), Chinese idioms present unique challenges for translation systems.
In this work, we aim to shed light on how effectively modern translation systems handle Chinese idioms and whether current automatic metrics accurately capture the quality of idiom translations. We first introduce IDIOMEVAL, an error taxonomy capturing 7 different failure modes for Chinese idiom translation such as Mistranslation and Partial Translation (see Table 1. To explore the impact of domain context, contain idioms from four distinct domains: Web, News, Wikipedia, and Social Media. Using the IDIOMEVAL taxonomy, translation systems including GPT--0 (Achiam et al., 2023). With these annotations, we conduct an error analysis on system translations and investigate how well current evaluation metrics assess the quality of Chinese idiom translations. Our main findings are as follows:
collect the Chinese texts that
collect annotations for 900 translation pairs from 9 modern
The selected systems struggle to produce high-quality idiom translations. They perform better on News but worse on Web and Social Media. The systems commonly produce Mistranslation, Literal Translation and Partial Translation.
Current machine translation evaluation metrics do not correlate well with human annota- tions. They show moderate sensitivity to perturbed idiom translations and fail to detect idiom error spans reliably.
To improve the detection of good and bad idiom translations, we instruction-tune Qwen2.5 models and achieve Macro F1 score of 0.68, outperforming all existing metrics and prompt- ing GPT-4o. We also find performance increases as the model scales up, except at 72B. We hope our work highlights the impact of idioms on translation and paves the way for developing better models and evaluation metrics. We will release all annotations, code, and models. In summary, our contributions are:
1. We introduce IDIOMEVAL, new taxonomy of Chinese idiom translation errors.
2. We collect a high-quality dataset of 900 human-annotated translations spanning four domains and nine modern MT systems.
2

---

Published as a conference paper at COLM 2025
(8) Canoer
(c) ADDING EDIT
Tandiation
(BS56 juit) selisammes
loddg sere
A2300G
CANCEL
VANE
Figure 2: Idiom annotation framework.
3. We provide the first comprehensive analysis of how modern systems perform idiom translation and how existing metrics correlate with human judgments.
4. We show that instruction-tuned LLMs significantly outperform existing metrics and prompting approaches in detecting idiom translation errors.
IDIOMEVAL: An Evaluation Framework
In this section, we discuss the details of our framework for idiom translation evaluation. IDIOMEVAL involves two steps: annotators select the translation span for the idiom within the context and classify into 9 high-level linguistically grounded categories that are further broken down into 13 subcategories; annotators then rate the severity of the translation errors and provide their confidence in the annotations. Our annotation interface built with Thresh (Heineman et al., 2023), and we will release our configuration for public use.
Select Translation Span and Category. The framework begins with selecting the corre- sponding translation span of the idiom. Annotators are then asked to choose one category that best describes the idiom translation. For certain categories, annotators are also required to select subcategory. The IDIOMEVAL. categories defined in Table 1. We provide detailed description of each category, subcategory and examples in Appendix A.
Rate Severity and Confidence. As each translation category affect the overall sentence quality to varying degrees, we ask annotators to rate the severity of the errors. Following Karpinska et al. (2022); Dou al. (2022), somewhat (2), and a lot (3). Finally, we ask annotators their confidence that their selection is correct on the same scale. An overview of the annotation interface is shown in Figure 2.
define three levels of severity: minor (1),
3 Data Collection and Annotation
We describe our methodology for automatically constructing dataset of 623K Chinese sentences containing idioms across four domains. Using subset this data, we then collect and annotate 900 translations from 9 modern systems, analyzed in $4 and $5.
Domain
Source
Period Instances
Idioms
Avg. occurrence freq. range
VH
H
M
L.
N
News
Common Crawl News
Mlat-Apr,Oct
50,845
5,333
16.7
3.9
2.5
1.5
15
Web
Common Crawl
Dec.
463,642
15,319
1.4
1.4
9.3
Wikipedia
Wikipedia Meta History
Dec.
39,699
5,947
12.1
2.1
1.4
1.2
1.1
Social Media
Weibo Search Results
Jan.jull.ann-Dee
35,315
519.0
Table Statistics of collected Chinese data by domain. Source lists the data collection sources. Period lists the month of collected data in 2023 and 2024. Instances shows the number of data samples in each domain. Idiom column shows the total number of idioms in the collected data. The rightmost column shows the average occurrence of idioms within each frequency range. The number of idioms within each frequency range can found in Appendix B.2.
3

---

Published as a conference paper at COLM 2025
3.1 Collect Text with Idioms
We use the idiom vocabulary provided by Tan & Jiang (2021) for its high coverage, which contains 30,999 idioms with definitions. To evaluate idiom translations under different contexts, we first collect Chinese texts containing idioms from four domains: News, Web, Wikipedia, and Social Media. Our subsequent analysis involves LLMs that were released in 2023 and 2024. To minimize the chance that these texts appear in the pre-training corpora of LLMs used for translation ($3.2), we collect texts that were available after the respective release year of these LLMs (referred to as new corpus).
For each domain, we divide idioms into five groups according to the 1st 2nd and 3rd quartiles of the overall frequency: VH (very high), H (high), M (medium), L (low), and N (never appear in the data). This allows us measure the impact of idioms frequency in pre- training corpora. We use the old corpus (data before 2023 in the same domains, details in Appendix B.1) to calculate idiom frequencies, as idioms from this period are more likely to be included in the LLMs' training data. Given that idioms tend to have a long-standing presence in a language, their frequency distribution within the same domain is unlikely to change significantly over time. Table 2 displays statistics of the collected data. The details of data collection for each domain are described in Appendix C.I.
3.2 Collect Translations from Systems
We carefully select diverse set of translation systems to allow us to assess idiom translation performance across range of system types: GPT-4 (a SOTA LLM) (Achiam et al., 2023), Alma-13B (a fine-tuned translation model) (Xu et al., 2024), Qwen-14B-Chi 2023) (a bilingual LLM for Chinese and English), mTT0133B (a multilingual LLM) (Muen- nighoff et al., 2022), Google Translate (a widely used commercial service), GPT-4o (a SOTA LLM) (Hurst et al., 2024), and QWen2.5 instruction-tuned models (multilingual LLMs with 7B, 14B and 72B parameters) (Yang et al., 2024a;b).
(Bai et al.,
We sample instances from each frequency range, resulting in 25 Chinese texts be translated for each domain. In total, this produces 900 translation pairs from the nine systems. The authors, native speakers of both Chinese and English, manually translated each Chinese text into English. These translations are used as references in our analysis.
3.3 Human Annotation
We recruit native Chinese speakers fluent in English through Prolific. The annotation process has two phases: the pilot phase and the main phase. In the pilot phase, 20 participants are provided with annotation guidelines and quiz. The top 5 scorers are selected for the main phase. In the main phase, each translation pair is assigned to three annotators, ensuring equal workload distribution. The final annotations are determined by a majority vote, with manual resolution by the first author for any ties. More details on inter-annotator agreement and quality control are provided in Appendix C.2.
4 Translation System Evaluation
We conduct detailed analysis of the idiom translation quality of the modern systems. We discuss how each system performs in each domain and their relative comparison. We then examine how domain and idiom frequency impact translation quality, followed by discussion on the severity of translation errors. We present our results in Figure 3, which displays the ratio of different translation categories from each system in the four domains. Further analysis, such as commonly seen errors, subcategories selected, error distribution by frequency range, and average severity scores for each category, can be found in Appendix D. The following are the key findings:
1. GPT-4 and Qwen family have overall higher translation quality, while Alma and m TO have performed consistently badly. In 2023 data, GPT-4 has the highest number of Good Translations across all the domains, while Qwen achieves comparable second performance.
4

---

Published as a conference paper at COLM 2025
Web
News
Wikipedia
Social Media
cor osee
sona
ofreas
0.5
0
0.5
1
0.5
0.5
0.5
0.5
1
0.5
2023
2024
Orteed Ao
0.5
0.5
Mis.
Unnat.
Lit.
Add.
Par.
Rep
No.
MTO
Figure 3: System-centric view of translation results in each domain. Each dataset is ac- companied by two subfigures. Left subfigure: ratio of Good Translations. Right subfigure: composition of error categories for each system.
On News and Wikipedia, GPTT-4 surpasses the second-best system (Qwen) by margin of 0.24 and 0.2. The difference is minimized in the 2024 data, where both models have close quality. Despite the best performance among the models, the GPT-4 and Qwen family still fail to achieve satisfying results, with more than 20% all translations being problematic. For Alma and m00 fewer than 25% of their translations are rated as good (except for Alma on Web). This disparity is likely due to a lack of idiom-related Chinese data in the pre-training and fine-tuning phases. For example, Alma's backbone, LLaMA-2,, is primarily pre-trained in English, limiting its performance. In contrast, Qwen, which is pre-trained on both Chinese and English bilingual data, achieves better translation quality.
2. Systems' translation quality varies by domain, where they perform better on News but worse on Web and Social Media. Although GPT-4 and Qwen provide the highest translation quality overall, their number of Good Translations varies significantly across domains. For GPT-4 and GPT-4o, more than 80% of its translations are rated as good on News, but this drops to 50% on Web and 60% on Social Media. The Qwen models have shown similar fluctuations. Overall, translation quality is worse on Web and Social Media. This may be because the two domains often feature newly emerging contexts, where idioms are used in novel ways compared to other domains. These variations highlight intrinsic differences in idiom usage across domains and emphasize the need for diverse training data to ensure consistently high-quality translations.
3. Translation error types differ across systems, but Mistranslation and Partial Transla- tion are commonly seen. Errors made by GPT-4 and GPT-4o are mostly Mistranslation and Partial Translations. On Web, GPT-4o also produces number of Literal Translations. Qwen (including Qwen2.5) and Alma produce more diverse errors. They are the only two models producing Addition errors (Web and Social Media). mTO frequently produces No Translations, particularly on Wikipedia, where it fails to translate at least 40% of idioms across frequency ranges. Google Translate frequently produces Literal Translations (36% on Social Media) and Unnatural Translations across three domains (4%).
Automatic Metric Evaluation
We evaluate how well current automatic translation metrics assess idiom translations, focusing on their ability to evaluate translations containing idioms, their sensitivity to idiom translation errors, and whether they can detect specific idiom translation error spans.
5.1 Metric Performance
We consider six of the most widely-used and strongest automatic metrics (Freitag et al., 2023): BLEU (Papineni et al., 2002), BERTScore Zhang et al. (2019), COMET (Rei et al., 2022a), COMETKIWT (Rei et al., 2022b), MetrieX-XXT. (Juraska et al., 2023; 2024) and MetricX-QF.
5

---

Published as a conference paper at COLM 2025
Scope
Category
Size
Reference-Based Metrics
Reference-Free Metrics
MetricX
MetricX -24-XXL
MetricX-QE -23-XXL
MetricX-QE
BLEU
BERTScore
COMET
-23-XXL
COMETKIWI
-24-XXL
Mistranslation
128
0.071
0.119
0.170
0.118
0:170
0.215
0.080
0.140
Literal
0.250
0.325
0.376
0.305
0.270
0.100
0.144
0.083
Partial
105
0.208
0.284
0.244
0.136
0.322
0.341
No Translation
74
0.186
0.271
0,346
0.198
0.226
0.381
0.138
0.268
Mistranslation
128
-0,074
0.066
0.239
0,251
0.166
0.174
-0.021
0.055
Literal
0.113
0.140
0.217
0.219
0.119
0,081
+0.095
-0112
Fartial
105
0.131
0.109
0.107
0.332
0.326
No Translation
-0.052
40,025
-0.512
+0.08
40.502
-0.536
Full
All errors
452
0.277
0.522
0.376
0.264
0.241
0.208
0.186
0.175
All categories
900
0.274
0.352
0.786
0.341
0.292
0.151
0.131
0.050
Idiom
All categories
errors
900
0.343
0.399
0.229
0.473
0.483
0.036
0.026
-0.161
-0.277
+0.205
0.274
Table 3: Pearson's between metrics and human annotations. Evaluation measured on both Full translation and Idiom translation. More Than One is omitted here as all its instances are rated as the highest severity. Unnatural, Addition and Repetition are omitted due to small sample size. ": for No Translation on Idiom, BLEU and BERTScore outputs 0 and thus omitted. Kendall's results (Table 9 in Appendix E.1) show similar patterns.
ROC-AUC
Good Rad
MetrieX-QF: -23-XXL
MetricX-QE -24-XXL
Metric
COMETKIWI
Full
Idiom
Full
Idiom
Perturbation
BLEU
0.63
0.69
68%
54%
Full
Idiom
Full
Idiom
BEKTScore
0.67
0.71
72%
77%
Mistranslation
0.73
0.66
0.80
0.62
0.79
0.67
COMET
0.67
0.75
64%.
56%
Unnatural
0.59
0,57
0.56
0.52
0.66
0.52
MetrieX-23-XXXL
0.67
0.78
67%
78%
Literal
0.65
0.72
0.73
MetricX-24 24--24-XXXX XXI.
0.66
0.57
71%
76%
Addition
0.66
0.57
0.54
0.52
0.72
0.48
COMETKIWI
0.58
0.51
58%
51%
Partial
0.59
0.58
0.61
0.57
0.72
0.65
MetricX-QF. QF-2XXXF--
0.58
0.47
65%
60%
Repetition
0.67
0.54
0.74
0.71
0.72
0.69
MetricX- Petricx-QE-24-XXLL
0.59
0.45
64%
62%
No Translation
0.68
0,28
0.75
0.75
0.72
0.77
Table 4:
Performance of existing metrics on idiom translation. Left table: distinguishing good and bad translations based on ROC-AUC scores and pairwise comparison accuracy on Full text and Idiom translations. Right table: sensitivity of reference-free metrics on Full text and Idiom translations, measured by the ratio of instances where unperturbed translations score better than the perturbed translations.
XXL. (Juraska et al., 2023; 2024). Among these, COMETKIWT and MetricX-QE. are reference- free. To allow for comparison with human annotations, we collapse each annotation into single numeric score, which is the severity score (from 1 to 3) directly as the overall score. Good translations are marked as O.
To understand how well these metrics evaluate idioms both in context and individually, calculate their scores based on the full context and only the idiom part. Table 3 reports the Pearson correlation coefficient (r) between the metrics and human evaluation for different error types. We found that, at both the full context and idiom levels, the metrics struggle to provide effective fine-grained measurements, with the highest correlation being 0.386 for full context and 0.483 for idioms. In most cases, correlations are stronger when measured on idiom translations than in full text, the surrounding context of idioms also influences the metric scores. Nevertheless, existing metrics show weak correlations with human ratings, thus failing to measure the quality of idiom translation from human-like perspective. Similar trends are observed when examining the ranking correlation with Kendall's (details in Appendix F.1).
Given that the metrics cannot effectively measure the severity of errors, we next investigate a simpler question: Can they distinguish good idiom translations from bad ones? We report the ROC-AUC score of each metric and the percentage of pairs where good translations score better than bad ones (for the same Chinese sentence) in Table 4 (left). BERTScore and MetricX perform the best on both metrics. However, in more than 20% of cases, Good Translations are still rated lower than those with errors, indicating that these metrics are not consistently reliable for distinguishing between good and bad idiom translations.

---

Published as a conference paper at COLM 2025
5.2 Metric Sensitivity
We measure metric sensitivity based on a similar approach to Karpinska et al. (2022) by comparing how each metric scores gold translations versus perturbed translations. Metrics that are sensitive to perturbations can accurately identify the perturbations. We focus on reference-free metrics, COMETKIWI and MetricX-QE as reference-based metrics use the gold translation as reference. To obtain perturbed translations, we use GPT-4o (OpenAT, 2024) to edit the existing reference translations into different errors. The task is decomposed into two steps: 1) Extraction: extracting the idiom translation from the English sentence, and 2) Perturbation: perturbing idiom translations with errors (see Appendix E.2 for the prompts). For each error type, randomly sampled 10 instances and checked if the edited translations matched the intended error category. More than 70% of the instances were edited correctly for each category. Since the model was specifically instructed to introduce errors, the gold translation should always be better than the perturbed ones.
We calculate the accuracy on perturbation the ratio of instances where gold translations score better than perturbed translations, as shown in Table 4 (right). For full-context translations, the top-performing metric, MetricX-QE-24, has an accuracy of around 0.7 under most perturbations. In contrast, COMETKIWI and MetricX-QE-23 have an accuracy below 0.6 for certain perturbations. However, when evaluating translations on the idiom part, the metrics demonstrate lower performance in almost all perturbations, showing they are less effective at detecting errors when only the idiom is presented.
5.3 Can Existing Metrics Identify Idiom Error Span?
We use XCOMET Guerreiro et al. (2023), the state-of-the-art error span detection model, to examine if it can accurately detect the span of idiom translation errors. We evalu- ate the performance using character level F1 score and only consider target spans. Sim- ilar to Blain al. (2023), we also compute weighted F scores by converting severity scores into binary weights. Table 5 displays the corresponding performance. We find that xCOMET performs poorly in detecting the span of idiom translation error, with the best model achieving only 0.3 F1.
P
R
F
WP
WR
Wh.
XCOMET-XI.
12.5
44.7
16.6
0.4
2.2
0.7
XCOMET-XXI XCOMET-XL (src)
10.2
30.3
12.4
0.5
1.6
0.6
13.3
49.7
18.1
0.5
2.4
0.7
XCOMET-XXL (src)
9.3
11.4
30.0
0.5
1.6
0.5
Table 5: Precision and F1 scores for error span detection using XCOMET. src means reference- free version. W represents weighted metric output. All values are multiplied by 100 for readability.
6 Improving Detection of Good and Bad Idiom Translation
With our corpus, we aim to develop a model that determines whether Chinese idiom is correctly translated. The task formulated as a binary classification problem: let be the idiom, s be the source Chinese text containing i, and I be the English translation of We want to predict whether is correctly translated in In our experiments, we only use references for reference-based metrics, such as BLEU. All other methods remain reference- since references may be unavailable real-world scenarios.
6.1 Data and Baselines
In practice, metrics are used to evaluate newly released models, so we split our data by model: the training and validation sets include translations from older models (GPT-4, Qwen, Alma, mT0, and Google Translate), while the test set includes translations from newer models (GPT-4o, Qwen2.5-7B, Qwen2.5-14B, and Qwen2.5-72B) leading to different sampled idioms with minimum overlap. The train, validation, and test sets contain 450, 50, and 400 instances, respectively.
We consider two baselines: regression-based metrics and LLM-based prompting. The regression metrics produce continuous score, we use the training set to find an optimal
7

---

Published as a conference paper at COLM 2025
375
Outpe
200-
Ling detees
diee (Hibe) soure Jountyy junddey)
3.44
R-11
3.50
0.50
3.63
5.50
g.50
1.4G
0.45
5.45
1.50
146
72B
333
135 1001
Figure 4: Performance comparison across model sizes and strategies for training Qwen2.5 (left) and MetricX (right). For Qwen2.5 (1.5B-32B), a short thought plus final answer yields the best results, while at 72B, shorter answer format performs similarly. For MetricX, omitting the idiom gives better results at 1.2B and 13B. Overall, scaling model sizes improves performance, except at Qwen2.5-72B.
threshold from the ROC curve, then apply it to the test set. For prompting, we evaluate GPT-4o and Qwen2.5 with zero-shot direct-answer and CoT prompts (Wei et al., 2022).
6.2 Our Methods
We explore two fine-tuning approaches. First, we fine-tune MetricX-24, the top- performing machine translation metric in WMT24 (Freitag et al., 2024). Second, instruction-tune the Qwen2.5 family of models, the best open-source LLMs in Chi- nese and English.
Model
#Params. Fj GooGood Fy-Bad Fi-Macro
Dummy
0.76
0.00
0.38
Thresholded Metrics Based on Roc Curve
Reference- Based
BLEU
0.68
0.43
0.56
BERTScore
355M
0.72
0.43
0.57
COMET
580M
0.65
0.50
0.58'
Fine-tuning MetricX. MetricX is based on mI5 (Xue et al., 2020) and trained to predict regression score from 0 to 25 via the logits for special token from the LM head. To adapt it for binary classification, we replace the LM head with classification head that inputs the decoder's 's final-layer represen- tation the starting token. We compare two input formats for training: one that in- cludes the idiom ('idiom: (idiom) source: (source) candidate: (candidate)")) and other that omits it ("source: (source) candi- date: (candidate)"",, which follows the origi- nal MetricX-24 training format. This allows us to study how adding extra information interacts with the existing format in a lim- ited training data setting.
MetricX-23
13B
0.68
0.54
0.61*
MetrieX-24
13B
0.66
0.54
0.60
Reference-Free
COMETKIWI
3.5B
0.49
0.52
0.50
MetricX-QE-23
13B
0.60
0.37
0.49
MetricX-QE-24
13B
0.46
Prompting LLMs
Zero-shot
Qwen2.5
14B
0.65
0.49
0.57
Qwen2.5
72B
0.75
0.37
0.56
GPT-4o
0.74
0.51
0.63
Zeru-shut- CoT
Qwen2.5
14B
0.73
0.60*
Qwen2.5
72B
0.78
0.30
0.54
CPTT4o
0.78
0.42
0.60
Fine-tuned Models
Fine-tuning Encoder-Decoder Decoder with Classification Head
MetrieX-QU Eetricx-QQE-2 Instruction Tuming Dender--Oll LLMs
13B
0.66
0.50
0.58
Qwen2.5 Qwen2.5
14B
0.77
0.54
0.65
72B
0.70
0.63
0.67
Instruction tuning Qwen2.5. We format each instance as an instruction, where the input includes the idiom, source Chinese text, and English translation. For output, on which the model's loss is computed, we test four strategies: (1) a simple yes/ yes/no no answer, (2) slightly longer answer, (3) short thought process followed by the final answer, and (4) a longer thought process followed by the final answer. The latter three strategies use templates described in Appendix F, where also provide additional details for all methods.
Table 6: Comparison of our fine-tuning meth- ods with threshold-based metrics and LLM prompting. Fine-tuned models achieve higher performance than the baselines.
8

---

Published as a conference paper at COLM 2025
6.3 Results
Table 6 shows the Fj scores for both good and bad translation classes, as well as their Macro average. We conduct one-sided paired bootstrap significance tests (10,000 resamples) to compare each metric's macro F1 score against that of the best-performing model, fine-tuned Qwen2.5-72B. An asterisk (*) indicates statistical significance (p 0.05). We show only the top-performing metrics from each method category. Figure 4 provides more detailed comparison across different model sizes and training strategies for fine-tuning MetricX and instruction-tuning Qwen2.5.
Instruction tuning Qwen2.5 outperforms all baselines, followed by prompting GPT-4o and then thresholded reference-based metrics. Fine-tuning Qwen2.5-32B with a short thought process and final answer achieves the highest Macro Fj of 0.68. Fine-tuning MetricX- QF-24 raises its Macro F from 0.46 to 0.58. For prompting, GPT-40 reaches a macro F of 0.63, while the Qwen2.5-72B gets 0.56. Interestingly, chain-of-thought prompting lowers performance on both of these two models, showing these models are more inclined judge translations as correct when CoT is used. For thresholded metrics, reference-basee metrics outperform reference-free ones by significant margin, with simple BLEU outperforming COMETKIWI by 0.06 F,. Model size scaling generally helps Qwen2.5 and MetricX, with an exception at 72B. Across the 1.5B to 32B versions of Qwen2.5, using short thought process plus the final answer yields the best results. However, at 72B, training on short answer performs the best, on par with the best 32B variant. For MetrieX, using its original training format (omitting the idiom) works better at the 1.2B and 13B scales, indicating that data format consistency might be crucial when training data is limited.
We further investigate why the performance does not scale up at Qwen2.5 72B. In terms of training loss (cross-entropy) we found that when fine-tuned on thoughts, the 72B model plateaus around 0.2 to 0.3, while smaller models (32B, 14B, 7B) converge more smoothly to lower loss (0.02). In contrast, training the 72B model on answers only, especially short answers, results in lower loss and better performance.
In terms of model behavior, we find that the 72B model always outputs the short answer format (yes/ / no) even when fine-tuned on longer formats such as long answers or thought- based responses. This differs from all the smaller size models (1.5B to 32B), which follow the training output format. While we do not fully understand the root cause of this be- havior, it may explain why training the 72B model on the short answer produces the best performance.
Related Work
Machine Translation with Idioms. We consider multi-word expression (MWE) a close line of work to idioms, and there have been plenty of studies on MWEs (Sag et al., 2002; Calzolari et al., 2002). Zaninello & Birch (2020) explored annotation and data augmentation techniques for MWE translations. Fadaee et al. (2018) add additional input features by pretending special tokens to indicate the presence of idioms. Gamallo & Garcia (2019) proposed compositional translation using cross-lingual embeddings, More recent works focus on idiom identification (Tedeschi et al., 2022), characterization (Socolof et al., 2021) and representation (Zeng & Bhat, 2022). Dankers et al. (2022b); Baziotis et al. (2023) provide an analysis of idiom processing with Transformers in the context of neural machine translation. Liu et al. (2023) provide characterization of idiomatic translation and its issues and introduce upweighting and a retrieval module to improve the translation. In contrast to these works, our research introduces a fine-grained evaluation framework for idiom translation and highlights the limitations of modern LLMs, such as GPT-4, and automatic metrics in generating and evaluating idiom translations.
Studies with Chinese Idioms. There has been limited research involving Chinese idioms. Zheng et al. (2019) proposed a large-scale Chinese idiom dataset ChID for cloze tests. It is the first work to study Chinese idiom understanding in the form of machine reading comprehension. Tan & Jiang (2021) constructs a dataset to facilitate the evaluation of Chinese idiom embeddings. Tan et al. (2021) collected large pretaining corpus of Chinese text by

---

Published as a conference paper at COLM 2025
crawling e-books online. Liao & Cheng (2023) studied text polishing with Chinese idioms, where sentences are rewritten to be more elegant with idioms. Other works have focused on creating parallel corpus with Chinese idioms. Shao et al. (2018) constructed the CIBB dataset that contains the English translations of 50 distinct Chinese idioms, together with a blacklist of literal translations. Wang & Yu (2010) created CTKB dataset that contains about 38k idioms and different English translations. However, only 28% of all samples have complete translations, and the dataset is publicly available. Tang (2022) constructed PETCI dataset that contains idioms collected from dictionary, together with translation results from Google and DeepL. More recently, Tang et al. (2024) used GPT-4 to generate context-aware idiom translations, yet there is a lack of external datasets and expert evaluations.
Conclusion
We introduce IDIOMEVAL, a framework with a carefully designed taxonomy that facilitates evaluating Chinese idiom translation. We posit that our framework can be extended to different languages. Our evaluation of various systems across multiple domains reveals that they struggle to produce high-quality idiom translations. We also find that existing metrics fail to correlate with human ratings or reliably distinguish good translations from poor ones. Furthermore, we demonstrate that the fine-tuned Qwen2.5 model is able to identify good and bad idiom translations with Macro F1 of 0.68, surpassing existing metrics and prompting-bare LLMs. Our work lays the groundwork for future research to improve idiom translation performance and to develop metrics focused on idioms.
Acknowledgement
This research is supported in part by the NSF CAREER award IIS-2144493. The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the official policies, either expressed or implied, of NSF or the U.S. Government. The U.S. Government authorized to reproduce and distribute reprints for governmental purposes notwithstanding any copyright annotation therein.
References
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arviv preprint ar arXiv:2303.08774 2023.
Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. Qwen technical report. arXiv preprint ar Xii:22309.16009 2023.
Christos Baziotis, Prashant Mathur, and Eva Hasler. Automatic evaluation and analysis of idioms neural machine translation. In Andreas Vlachos and Isabelle Augenstein (eds.), Proceedings of the 17th Conference of the European Chapler of the Association for Computational Linguistics, pp. 3682-3700, Dubrovnik, Croatia, May 2023. Association for Computational URL
Linguistics. doi: 10.18653 http://11122.2lllmmmmm eacl-man m
/0233.
Frederic Blain, Chrysoula Zerva, Ricardo Rei, Nuno M Guerreiro, Diptesh Kanojia, José GC de Souza, Beatriz Silva, Tânia Vaz, Yan Jingxuan, Fatemeh Azadi, et al. Findings of the wmt 2023 shared task on quality estimation. In Proceedings of the Eighth Conference Machine Translation, pp. 629-653, 2023.
Nicoletta Calzolari, Charles J. Fillmore, Ralph Grishman, Nancy Ide, Alessandro Lenci, Catherine Macleod, and Antonio Zampolli. Towards best practice for multiword expres- sions in computational lexicons. In International Conference on Language Resources and Evaluation, 2002.
10

---

Published as a conference paper at COLM 2025
Verna Dankers, Christopher Lucas, and Ivan Titov. Can transformer be too compositional? analysing idiom processing in neural machine translation. ArXio, abs/ 0tbs/2205.153000 2022a.
Verna Dankers, Christopher Lucas, and Ivan Titov. Can transformer be too composi- tional? analysing idiom processing in neural machine translation. In Smaranda Mure- san, Preslav Nakov, and Aline Villavicencio (eds.), Proceedings of the 60th Annual Meet- ing of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 3608- 3626, Dublin, Ireland, May 2022b. Association for Computational Linguistics. doi: 10.18653/ 11118633/v1/2000 /122//12.22222//oog 252. URL
hwtphhhllloo..gg/2222222/ wrg. acl- -ong 252.
Yao Dou, Maxwell Forbes, Rik Koncel-Kedziorsii, Noah A. Smith, and Yejin Choi. GPT-3 text indistinguishable from human text? scarecrow: A framework for scrutinizing machine text. In Smaranda Muresan, Prreslav Nakov, and Aline Villavicencio (eds.), Proceedings of the 60th Annual Meeting of the Association for Compulational Linguistics (Volume 1: Long Papers), pp. 7250-7274, Dublin, Ireland, May 2022. Association for Computational Linguistics. doi: 10.18653 /v1/ w122022.aclloong.555 URL https::/acllattee
wrg/2022.co2. acl acl-long 501.
Marzieh Fadaee, Arianna Bisazza, and Christof Monz. Examining the tip of the iceberg: A data set for idiom translation. ArXiv, abs/ abs/1802.0445 1802.04681, 2018.
Markus Freitag, Nitika Mathur, Chi-kiu Lo, Eleftherios Avramidis, Ricardo Rei, Brian Thompson, Tom Kocmi, Frédéric Blain, Daniel Deutsch, Craig Stewart, al. Results of wmt23 metrics shared task: Metrics might be guilty but references are not innocent. In Proceedings of the Eighth Conference on Machine Translation, pp. 578-625- 628, 2023.
Markus Freitag, Nitika Mathur, Daniel Deutsch, Chi-Kiu Lo, Eleftherios Avramidis, Ricardo Rei, Brian Thompson, Frédéric Blain, Tom Koemi, Jiayi Wang, et al. Are Ilms breaking metrics? results of the wmt24 metrics shared task. In Proceedings of the Ninth Conference on Machine Translation, pp. 47-81, 2024.
Pablo Gamallo and Marcos Garcia. Unsupervised compositional translation of multiword expressions. In MWE-WN@ACH 2019.
Nuno M Guerreiro, Ricardo Rei, Daan van Stigt, Luisa Coheur, Pierre Colombo, and An- dré FT Martins. xcomet: Transparent machine translation evaluation through fine-grained error detection. Xrviv preprint ar Xfto:2231010042,, 2023.
David Heineman, Yao Dou, and Wei Xu. Thresh:
A unified, customizable and deployable platform for fine-grained text evaluation. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pp. 336 345, 2023.
Edward Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. ICLR, 1(2):3, 2022.
Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card. ar Xiv preprint ar Xrii:24400.21776 2024.
Juraj Juraska, Mara Finkelstein, Daniel Deutsch, Aditya Siddhant, Mehdi Mirzazadeh, and Markus Freitag. MetricX-23: The Google submission to the WMT 2023 metrics shared task. In Philipp Koehn, Barry Haddow, Tom Kocmi, and Christof Monz (eds.), Proceedings of the Eighth Conference on Machine Translation, pp. 756-767, Singapore, December 2023. Association for Computational Linguistics. doi: 10.18653 / ht/p//112222//////// wwt-1.63..... https://aclanthologg, www.o/gg.........oo wmt-1.63. 63.
URL
Juraj Juraska, Daniel Deutsch, Mara Finkelstein, and Markus Freitag. MetricX-24: The Google submission to the WMT 2024 metrics shared task. In Barry Haddow, Tom Kocmi, Philipp Koehn, and Christof Monz (eds.), Proceedings of the Ninth Conference Machine Translation, pp. 492-504, Miami, Florida, USA, November 2024. Association for Computa- tional Linguistics. URL https: http::///aclaahhllogggoo .rgyy.org/// org/2024. wmt-1. 35.
11

---

htt////112222m//////// Published as a conference paper at COLM 2025
Marzena Karpinska, Nishant Raj, Katherine Thai, Yixiao Song, Ankita Gupta, and Mohit Diagmosing evaluation metrics for translation. In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang (eds.), Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp. 9540-9561. Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics. doi: 10.1 10.18653// 1118885/1//// /222222 emmlp-main.649. URL hps: //sss//thllaaoogy org/2022 1ttg/222.eemmppmaii.. emnlp-main. 649.
lyyer. DEMETR:
Kenneth Katzner and Kirk Miller. The languages of the world. Routledge, 2002.
Junwei Liao and Shuai Cheng. Text polishing with chinese idiom: Task, datasets and
pre-trained baselines. ACM Transactions on Asian and Low-Reson Resouree Language Information Processing, 2023.
Emmy Liu, Aditi Chaudhary, and Graham Neubig. Crossing the threshold: Idiomatic machine translation through retrieval augmentation and loss weighting. Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 15095-15111, Singapore, December 2023. URL
Association for Computational Linguistics. doi: https://aclanthhology org/2023 emnlp-main mwi.maai.n.....
Niklas Muennighoff, Thomas Wang, Lintang Sutawika, Adam Roberts, Stella Biderman, Teven Le Scao, M Saiful Bari, Sheng Shen, Zheng-Xin Yong, Hailey Schoelkopf, et al. Crosslingual generalization through multitask finetuning, arXiv preprint or arXiv:2211.01788 2022.
OpenAL. Hello gpt-4o, May 2024. URL htttp:://openaa.com/iiii http://ddeweelllo-g--ooo..
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pp. 311-318, 2002.
Ricardo Rei, José G. C. de Souza, Duarte Alves, Chrysoula Zerva, Ana C. Farinha, Taisiya Glushkova, Alon Lavie, Luisa Coheur, and André F. T. Martins. COMET-22: Unbabel-IST
2022 submission for the metrics shared task. In Philipp Koehn, Loic Barrault, Ondřej Bojar, Fethi Bougares, Rajen Chatterjee, Marta R. Costa-jussä, Christian Federmann, Mark Fishel, Alexander Fraser, Markus Freitag, Yvette Graham, Roman Grundkiewicz, Paco Guzman, Barry Haddow, Matthias Huck, Antonio Jimeno Yepes, Tom Kocmi, André Martins, Makoto Morishita, Christof Monz, Masaaki Nagata, Toshiaki Nakazawa, Matteo Negri, Aurélie Névéol, Mariana Neves, Martin Popel, Marco Turchi, and Marcos Zampieri (eds.), Proceedings of the Seventh Conference Machine Translation (WMT), pp. 578-585, Abu Dhabi, United Arab Emirates (Hybrid), December 2022a. Association for Computational Linguistics. URL https: / http://aaaaaaoooogoooyoo wrw.:o2g//2222 1mt-1.52
Ricardo Rei, Marcos Treviso, Nuno M. Guerreiro, Chrysoula Zerva, Ana Farinha, Christine Maroti, José G. C. de Souza, Taisiya Glushkova, Duarte Alves, Luisa Coheur, Alon Lavie, and André F. Martins. CometKiwi: IST-unbabel 2022 submission for the quality esti- mation shared task. In Philipp Koehn, Loïc Barrault, Ondfej Bojar, Fethi Bougares, Rajen Chatterjee, Marta R. Costa-jusså, Christian Federmann, Mark Fishel, Alexander Fraser, Markus Freitag, Yvette Graham, Roman Grundkiewicz, Paco Guzman, Barry Haddow, Matthias Huck, Antonio Jimeno Yepes, Tom Kocmi, André Martins, Makoto Morishita, Christof Monz, Masaaki Nagata, Toshiaki Nakazawa, Matteo Negri, Aurélie Névéol, Mariana Neves, Martin Popel, Marco Turchi, and Marcos Zampieri (eds.), Proceedings of the Seventh Conference on Machine Translation (WMT), pp. 634-645, Abu Dhabi, United Arab Emirates (Hybrid), December 2022b. Association for Computational Linguistics. URL https::/aclaantholggy/ hwwpoollogyooggog//gooooo hetg/2022.cc0 w 10.
Ivan A. Sag, Timothy Baldwin, Francis Bond, Ann A. Copestake, and Dan Flickinger. Multiword expressions: A pain in the neck for nlp. In Conference on Intelligent Text Processing and Computational Linguistics, 2002.
Yutong Shao, Rico Sennrich, Bonnie Webber, and Federico Fancellu. Evaluating machine translation performance Chinese idioms with blacklist method. In Proceedings of
12

---

Published as a conference paper at COLM 2025
the Eleventh International Conference Language Resources and Evaluation (LREC 2018), Miyazaki, Japan, May 2018. European Language Resources Association (ELRA). URL
nttps://aclanthol..gg http:/ooooooooo/1--1// org/L18-1005
Michaela Socolof, Jackie Chi Kit Cheung, Michael Wagner, and Timothy J. O'Donnell. Characterizing idioms: Conventionality and contingency. ArXio, abs/2104.08666 2021.
Minghuan Tan and Jing Jiang. Learning and evaluating chinese idiom embeddings. In Recent Advances in Natural Language Processing, 2021.
Minghuan Tan, Jing liang, and Bingtian Dai. bert-based two-stage model for chinese chengyu recommendation. ACM Trans. Asian Low Resour. Lang- Inf. Process., 20:92:1-92:18, 2021.
Kenan Tang. Petci: A parallel english translation dataset of chinese idioms. ArXio, /bs//220.09509, 2022.
Kenan Tang, Peiyang Song, Yao Qin, and Xifeng Yan. Creative and context-aware translation of east asian idioms with gpt-4. ar arXiv preprint ar arXio:2410.0090 2024.
Simone Tedeschi, Federico Martelli, and Roberto Navigli. Id10m: Idiom identification in 10 languages. In NAACL-HLT, 2022.
Lei Wang and Shiwen Yu. Construction of Chinese idiom knowledge-base and its ap- plications. In Proceedings of the 2010 Workshop on Multiword Expressions: from Theory to Applications, pp. 11-18, Beijing, China, August 2010. Coling 2010 Organizing Committee. URL https: andll antoooogy hwwphoolg...ooo/o/ooooeeoeoooeee org/w//g/// 00rg//10-3770 07033.
Ran Wang. The cultural differences between english and chinese idioms and translation. In 2022 International Conference Creative Industry and Knowledge Economy (CIKE 2022), pP- 610-614. Atlantis Press, 2022.
You Wang. A contrastive research on the definitions and categories of chinese and english idioms. In 7th International Conference Humanities and Social Science Research (ICHSSR 2021), pp. 423-426. Atlantis Press, 2021.
Jason Wei, Xuezhi Wang. Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824-24837 2022.
Haoran Xu, Young Jin Kim, Amr Sharaf, and Hany Hassan Awadalla. A paradigm shift in machine translation: Boosting translation performance of large language models. In ICLR, 2024.
Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Adilya Barua, and Colin Raffel. mt5: A massively multilingual pre-trained text-to-text transformer. Xiv preprint ar 201::200001199.... 2020.
An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, Guanting Dong, Haoran Wei, Huan Lin, Jialong Tang, Jialin Wang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Ma, Jin Xu, Jingren Zhou, Jinze Bai, Jinzheng He, Junyang Lin, Kai Dang, Keming Lu, Keqin Chen, Kexin Yang, Mei Li, Mingfeng Xue, Na Ñi, Pei Zhang, Peng Wang, Ru Peng, Rui Men, Ruize Gao, Runji Lin, Shijie Wang, Shuai Bai, Sinan Tan, Tianhang Zhu, Tianhao Li, Tianyu Liu, Wenbin Ge, Xiaodong Deng, Xiaohuan Zhou, Xingzhang Ren, Xinyu Zhang, Xipin Wei, Xuancheng Ren, Yang Fan, Yang Yao, Yichang Zhang, Yu Wan, Yunfei Chu, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, and Zhihao Fan. Qwen2 technical report. arXie preprint ar 4rX1122007.16671 2024a.
An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mei Li, Mingfeng Xue, Pei Zhang, Qin Zhu, Rui Men, Runji Lin,
13

---

Published as a conference paper at COLM 2025
Tianhao Li, Tingyu Xia, Xingzhang Ren, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yu Wan, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, and Zihan Qiu. Qwen2.5 technical report. Xiv preprint ar X1X:::2412.1511,, 2024b.
Andrea Zaninello and Alexandra Birch. Multiword expression aware neural machine translation. In International Conference on Language Resources and Evaluation, 2020.
Ziheng Zeng and S. Bhat. Getting bart to ride the idiomatic train: Learning to represent idiomatic expressions. Transactions of the Association for Computational Linguistics, 10: 1120-1137, 2022.
Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. Bertscore: Evaluating text generation with bert. arviv preprint ar Xiio:1904.09665, 2019.
Chujie Zheng, Minlie Huang, and Aixin Sun. Chid: A large-scale chinese idiom dataset for cloze test. ArXio, (bbs/190..0126 2019.
14

---

Published as a conference paper at COLM 2025
A Idiom Translation Examples
Below are the input and output of different translation categories and subcategories, along with corresponding explanation. We skip trivial cases.
No Error:
Correct Underlying Meaning
Chinese: Ti (
TTF HEUT F
English:
The main fault bypassed the nuclear power plant and came to a stop. Scientists from the United States Geological Survey cheered and rejoiced, but they discovered that the fault had not been completely halted.
Explanation:
X MB
is translated correctly into "cheered and rejoiced".
No Error:
Literal and Coherent
Chinese: 1
L M 0F
t 14 2F 3 2 - A HEL HE
English: This move has clear direction provoking the US dollar and putting the Federal Reserve in a difficult position. (For more content, see: "Saudi-led oil tycoons stab the US
dollar again, Xiang Zhuang's sword dance is aimed at Pei Gong") In March, Saudi Arabia's series of 'ooess' is important step in its strategic transformation.
The original meaning of 3 means the actual intention is not what it appears to be. However, in this case, the idiom is used to refer to the historical story and literal translation is more appropriate.
Explanation:
Mistranslation
Chinese: M
-
The other half sometimes needs someone to think quietly by themselves, at which point you don't need to disturb them too much. Asking too many questions will only bring them problems.
English:
tu means asking for trouble trouble someone him/herself, not "bringing other people trouble".
Explanation:
Unnatural
Chinese:
English: A good warrior does not achieve glaring victory.
L means "a great achievement" or "a great victory". "A glaring victory" conveys a rough meaning but the usage of "glaring" is not perfect enough.
Explanation:
Literal Translation
Chinese: HH X
T F1H
My father said when he was dying, have no merits in this life, I have not left anything for you.'
English:
Explanation:
& is translated literally here. 4 in this life. F: K&L- merits.
15

---

Published as a conference paper at COLM 2025
Addition
Chinese:
Cheng was not only "well-versed in drama studies, knowledgeable and passionate in all aspects", but also harbored lofty theatrical ideals and ambitions.
English:
Explanation: T passion.
means knowledgeable in all aspects, and not related one's
Partial Translation: Missing Modifier
Chinese:
"H (E A. HI TE r
TE ( "K
English:
The "I Ching" says, "The expedition successful, the chief evil is beheaded, and his accomplices are captured." " It praises the execution of those who commit evil crimes, and those who do not submit have come to surrender.
Explanation: AB refers to extremely evil crimes. Here, the translation has conveyed the core idea of the idea, but the degree of "evil" is missing.
Partial Translation: Missing Core Info
म FUUM #
Chinese:
T NMT
Also, there are many agencies, there no place to stay, and against these bad inns, only the agencies with the current rules cannot adequately curb the abuse.
English:
Only the first half of ! A is translated. The second half of the idiom contains important information for us to understand the idiom, too.
Explanation:
Partial Translation: Inaccurate Modifier
Chinese: #
English:
The captured Russian soldiers are a bit terrified, but in reality, they are much luckier than their comrades on the ground.
means "very terrified". The translation inaccurately used "a bit" as Explanation: 1 modifier.
Repetition
Chinese:
#
English: I think the best relationship between father and should be like that of brother and brother.
Explanation:
Ε means the relationship is like brothers. In this case, the translation is repeated.
No Translation: Missing from Output
Chinese: AE J .
T #H #LETT T T4%
English:
Irascible was flying from Bloemfontein to Pretoria and had four other passengers on board, including the snake.
Explanation:
2 is not translated and is missing from the English sentences.
16

---

Published as a conference paper at COLM 2025
No Translation: Chinese Idioms Copied or Paraphrased
Chinese: ス
English: Σ<
Explanation:
not translated and remains in Chinese text.
B Statistics of Corpus
B.1 Statistics of Old Corpus
Table shows the statistics of the old corpus.
Avg. occurrence by freq. range
Domain
Source
Period
Instances
Idioms
VH
M
News Web
News Crawl
2019
17,931,873
10,830
425.6
38.6
8.0
1.5
me4
2008-22
54,542,308
30,652
8135.5
220.0
53.0
26.0
Wikipedia
wiki2019zh
2018-19
1,043,224
11,754
95.1
13.0
3.4
1.0
Social Media
PchatbotW
2008-22
139,448,339
15,089
2313.1
189.9
16.2
1.8
Table 7: Statistics of old corpus. Source lists the data collection sources. Period lists the year of collected data. Instances shows the number of data samples in each domain. Idiom column shows the total number of idioms. The rightmost column shows the average occurrence of idioms within each frequency range.
B.2 Number of Idioms within Each Frequency Range
Table displays the number of idioms each frequency range in our newly collected data.
Domain
VH
H
M
L
N
News
2,486
1,768
692
225
162
Web
7,310
4,344
2,117
1,442
106
Wikipedia Social Media
2,814
1,702
772
269
390
60
60
60
104
67
Table 8: Number idioms within each frequency range in our collected data.
Data Collection and Annotation Details
C.1 Data Collection for Each Domain
News. We collect data from Common Crawl News (2023-04 and 2024-10 snapshots), con- sisting of 41,354 Chinese news articles, which we converted to simplified Chinese for consistency. For each article, we extracted sentences containing idioms, along with the preceding and following sentences for context, resulting in total of 50,845 instances.
Web. We use Common Crawl (2023-06 and 2024-12 snapshots) to collect web page data. After retrieving the HTML content from each URL, we extracted sentences containing idioms along with their surrounding context, similar to the news data. This resulted in total of 463,642 instances.
Wikipedia. We select the Chinese Wikipedia meta history (2023-07 and 2024-12 snapshots) for data collection. We save the revision IDs of articles whose first revision is after the cutoff
17

---

Published as a conference paper at COLM 2025
date. We obtain HTML pages for each revision through Wikimedia REST API'. We then extract sentences with idioms and their contexts from the revised text. There are 39,699 instances in total.
Social Media. We use Weibo, a popular Chinese social media platform similar to Twitter, to collect posts containing idioms. We rely on Weibo's search function to collect posts containing idioms. We check the publishing date each post to ensure was published after 2023. Unlike the previous domains, where we collected new data with all present idioms from the old corpus, querying each idiom will incur a high overhead in API usage. We instead sample 400 idioms from old corpus based on frequency range (60 for VH, H, and 110 for L and N). We double the size for L and N to alleviate the scarcity of data. We managed to collect posts for 351 out of 400 idioms, and there are 55,315 instances in total.
C.2 Quality Control
Hit 300
Instructions
e A
Contect:
Idiom: 35
Chinese Ingul
2 HALELL
Translation
Germany's 's main opposition party leader calls for a European front on China. In an interview with Reuttrs, Friedrich Meltz, the conservative leader of the canter-right Knesset party, said Germany longer rely buying cheap natural gas from Russia and buying East and West from around the world while relying on the US for security.
EDIT ANNOTATIONS (0/0)
+ Add Edit
Figure 5: Example quiz question.
We use Prolific to recruit annotators who are native Chinese speakers fluent in English. We only collect the annotations the data we provide, and no information about the annotators themselves is collected. We have received full consent from the annotators on this task. The annotation process is divided into two phases: pilot phase and main phase. In the pilot phase, 20 participants are selected for pre-screening. They are provided with detailed guidelines and must complete a quiz to demonstrate their understanding, The quiz consists of 30 questions, covering each category and subcategory with 2-3 questions. An example quiz question is shown in Figure 5. The top performers from the pilot phase are selected for the main phase. In the main phase, we randomly assign each translation pair to three annotators while we ensure each annotator sees an equal number of pairs. We sample 100 annotations and evaluate them against the authors' own annotations, achieving category accuracy 66% and subcategory accuracy of 88%. For the final annotations, we take the majority vote and the first author manually resolves any tie cases. This resolution improves the average Cohen's K among annotators on category (subcategory) agreement from 0.47 (0.38) to 0.73 (0.69), indicating strong annotator agreement given some translations may have more than answer. For instance, translating #n f # (false feelings and intentions, indicating insincerity in one's intention) into "false" can be considered as either Mistranslation Partial Translation. Annotators compensated at $22 per hour.
18

---

Published as a conference paper at COLM 2025
Mis.
Lit.
Par.
No.
Unnat
Minor
Somewhal
A lo!
GPT-4
0.28
0.24
0.38
0.06
0.59
0.35
Owen
0.21
0.23
0.21
0.15
0.23
0.62
Alma
0:33
0.29
0:19
0.38
0.62
mTQ
0:35
0.18
0.38
0/04
0.21
0.75
GTrans
0.25
0.5
0.75
0
2023
2004
GPT-40
0.09
0.14
0.07
0.24
0.24
0.52
Owen2 5-7b
0.12
0.11
0.11
0.19
0:31
0.50
Owen2.5 5-14b
0.09
0.11
0.08
0.36
0.40
0.27
0.3
0.4
Figure 6: Left: Top three most commonly seen errors for each system. Mis.: Mistranslation;
Lit.: Literal Translation; Par.: Partial Translation; No.: No Translation. Right: Ratio of each severity score for each system, calculated overall all errors and domains.
D Further Analysis on Translation Systems
D.1 Category Breakdown
Figure shows the subcategory composition for Good Translation, Mistranslation, and Partial Translation. It can be seen that subcategories vary across domains and systems. For instance, for Partial Translations, On News and Wikipedia, Alma is likely to omit modifiers (100% and 75%), while Web and Social Media, it misses the core meaning in most cases (100% and 67%). mTO accomplished by significant number of No Translations each domain, where its translations mostly missing from the output.
D.2 Frequency Breakdown
Figure 9 illustrates each system's performance in each frequency range across domains. can be seen that the frequency ranges of idioms are not correlated with translation quality. For idioms that appear in old corpus, their translation quality does not decrease with their frequency ranges. For instance, idioms in Web from the M range (100% Good Translations) get the best translation by GPT-4, while those from the H and L range (60% Good Translations) achieve comparable translation quality. Similar patterns can be found in other models. Interestingly, idioms that never appeared in the old corpus (N range) do not always result in poor translations. For instance, on Wikipedia, Alma produces the highest-quality translations for N-range idioms over other frequencies. This observation is particularly remarkable on Social Media data, where idioms that never appear get the best translation quality by almost all models in 2023 data. Such idioms get relatively good translation quality in 2024 data, too. Since the N range is defined regarding the appearance of idioms old corpus for that domain. Idioms not showing up in one domain can still appear in another domain and be included in the pre-training corpus. This shows that language models transfer the knowledge from different domain and translate the idiom correctly.
D.3 Common Errors and Severity Scores
Figure displays the top three most commonly seen errors for each system and the ratio of each severity score for each system. Figure shows the breakdown of the severity score of each system on each error category across domains. We note that errors that interrupt
htttps: wwt.://eaiia....gapprreeees...
19

---

Published as a conference paper at COLM 2025
News
Unieera
Undeling
Daves
Wikipedia
Social Media
13
10
Morr
Cape
6PP4
Cewn
GPF4
Ccreet
Hadie
Wiklpodia
Social Media
Moarg
1
Dape
Crppe
Oventi-Tt
Geeed S-14
Ovendd-1t
Geeeds 5-40
Figure 7: Subcategory composition for Good Translation, Partial Translation and No Trans- lation.
meaning are more severe than errors that preserve the overall meaning. When models make errors, over half of them significantly impact the understanding of the text. For 2023 data, despite GPT-4's top performance, only 6% of its incorrect translations have a minor impact, placing it third behind Qwen and Google Translate. Alma makes the most severe errors, with none of its errors classified as minor. All the models have more than 40% of their errors marked as the highest severity in 2024 data.
Mistranslation and No Translation are the most severe error types, as they disrupt meaning significantly. Except for wikipedia in 2024 data, Mistranslation has minimum average severity score of 2.43. Meanwhile, No Translation has a severity score of 3 in many domains. In contrast, Unnatural Translation (max severity of 2), Addition (max severity of 2.5), and Repetition (max severity of 2) are less severe, as they typically preserve some correct or near-correct idiom parts.
20

---

Published as a conference paper at COLM 2025
Web
News
2.38
3.00
2.00
3.00
QVAin
2.00
3,00
1.00
1.00
3.00
3.00
2.25
2.5
QWen
2.67
1.00
3,00
2.00
3.00
2.43
2.5
Ama
3,00
2,00
2.50
3.00
3.00
2.6 67
2.0
Alma
3.00
3.00
2.00
2.00
2.67
2.0
2.00
1.5
3.00
1.5
1.00
3.00
3.00
2.30
AV3
Aug
Avg
Uowt
U2
Par
Rop
Nn.
ATO
Mi
Uniot
Add
Par
ANo
Wikipedia
Social Media
GPT-4
2.00
2.00
GPT-4
2:50
1.00
QWen
3.00
2.00
1.00
3.00
25
2.5
QWen
3.00
2.00
3.00
3.00
3.00
3.00
3.00
2.86
2.5
Ama
3.00
2.00
3.00
3.00
3.00
83
2.0
Alma
2.00
3.00
2.00
2.00
2.00
3.00
3.00
2.39
2.0
Girans
3.00
2.50
3.00
Girans
2:00
00
2.50
3.00
00
2.43
Arg
Avg.
2.00
Avg.
2.43
2.50
2.20
3.00
3.00
Umnot.
PBc
Rep.
MTO
Mis.
Umaor
Ada.
Par
Rep.
M70
GPT446
3.0
GPT4c
3.0
0Wer2.5-76
2.50
3.00
2.00
2.60
0WAr2.5-77 5-7b
2.00
2.00
3.00
3.00
2.50
2.0
2.0
-15
Owen2.5 72b
3,00
3.00
1.00
3.00
2.20
-10
OWen2.5
726
2.00
1.00
3.00
3.00
2.00
1.80
Arg.
2.00
2.75
MDx
Unove
A68
Pbc
Rep.
NTO
Degh
M20
Wikipedia
Social Media
1.50
3.0
1.00
3.00
OWen2.577
2.00
OWen25-77
3.00
3.00
2.80
2.0
oWen2.5 5 14b
3.00
1.00
1.00
2.00
3.00
1.20
@Wen2.5 2. 14b
3.00
2.00
2.50
2.00
3.00
2.57
3.0
@Wen2
2.00
3.00
1.0
Arg.
Aug
1.67
3.50
1.00
1.50
Auga
2.00
2.00
1.00
No.
ATO
Unuue
6K
Rep.
No.
KTO
Figure 8: Average severity scores for each translation category in each domain. The right- most column represents the average severity score across all errors for each system. The bottom row represents the average severity score across all systems for each error.
21

---

Published as a conference paper at COLM 2025
GPT-4
Web
News
Wikepedls
Social Media
Mis
Urnatt
Add
Par
Rep
No
MTO
Wikipedaa
0.6
0.5
0.5
0.5
96
Mis.
Unnat:
Add
Par
Alma
Web
News
Wikipedia
Social Media
Dual
0.5
0.E
0
0.5
0.5
066
065
0.5
Rep
mTo
Web
News
Wikipedia
Social Media
Good
VHH
Mis.
Unroal
Add
Pan.
Rep
No.
MTO
Wikipedie
Social Media
Mis.
Unnat
Lt
Aad
Par
Hop
No.
MTO
GPT-4o
Wikipedia
Social Media
Meal
0.5
0.2
0
0.5
0.5
05
355
1
0.5
Unnal.
Qwen2.5-1444
Web
News
Wikipedia
Social Media
VHI
0.5
Unnat.
Rep
Qwen2.5 5-
Web
News
Whkipedis
Secial Media
Mis
Unnat
Add
Paat
Rep
No.
MTO
Qwen2.5-72h
Whlipedta
0.5
0.5
1.6
0.5
0.
1.0
96
C
Unnal.
L1
Add.
Pgr.
Figure 9: Frequency-centric view of translation results in each domain. There are two subfigures in each dataset. Left subfigure: mean ratio good translations across all systems. Right subfigure: mean ratio of error categories across all systems.
22

---

Published as a conference paper at COLM 2025
E Further Analysis on Evaluation Metrics
E.1
Kendall's T between Evaluation Metrics and Human Annotations
Table shows Kendall's between evaluation metrics and human annotations. Similar to Table 3, Kendall's T remains weak across categories, indicating low agreement between ranking based on metric scores and human ratings.
Reference- Metrics
Reference-Free Metrics
Scope
Category
Size
MetricX
MetricX -24-XXL
MetricX-QE
MetricX-QE
-24-XXL
Mistranslation
128
0.064
0.099
0.141
0,081
0.129
0.192
0,049
0.097
Full
Lateral
90
0.183
0212
0.314
0.248
0.210
0.991
0.158
0.000
No Translation
74
0.182
0.271
0.345
0.204
0.232
0.072
0.156
0.273
Mistranslation
128
-0.011
0.080
0.196
0.210
0.181
0.184
0.120
0.123
Idiom
+0.052
Partial
105
0.122
0.104
0.072
0.034
40.107
+0.043
-0.171
+0.195
No Translation
74
-0.016
40.099
40.269
0.031
40.73
-0.252
All errors
452
0.199
0.269
0.323
0.133
0.162
calegories
0.208
0.274
0.285
0.264
0.246
0.140
0.137
0.147
Idiom
All erroes
452
0.128
0.218
0.259
0.269
0.006
0.012
-0.08
00.140
categories
-0005
-0.090
Table 9: Kendall's between evaluation metrics and human annotations. Evaluation is measured on both Full translation and Idion translation. Good is omitted since does not have severity scores. More Than One is omitted here as all its instances are rated as the highest severity. Unnatural, Addition and Repetition are omitted due to small sample size. +. for No Translation on Idiom, BLEU and BERTScore outputs 0 and thus omitted. Existing metrics fail to produce measures that are correlated strongly with human annotations.
E.2 Prompts
Figure 10 shows prompt for extracting idiom translations. Figure 11 shows prompt for editing idiom translations. Figure 12 shows prompt for identifying idiom translations.
F Implementation Details for Good and Bad Idiom Translation Detection
E1
Prompting Methods
We use the latest GPT-40-20241120 (referred to as GPT-4o) and Qwen2.5 Instruct models for prompting. Generation is done via greedy decoding. Figure 13 displays the zero-shot prompt, and Figure 14 displays the zero-shot prompt with thain-of-thought.
F2 Fine-tuning Methods
We fine-tune Qwen2.5 Base models using LoRA (Hu et al., 2022), and use greedy decoding during inference. For MetricX-24, we add classification head to the decoder and update both the LoRA parameters and the classification head during training. We sweep learning rates (5e-5, 1e-4, 3, 5e-4) for epochs and select the best checkpoint on the validation set for final testing.
Input Format for Fine-tuning Qwen2.5:
Evaluate whether the Chinese idiom is correctly translated in the following text:
Chinese Idiom: (idiom}
Chinese Text: (zh_sentence)
English Translation: (en_sentence)
23

---

Published as a conference paper at COLM 2025
Output Formats of Different Strategies for Fine-tuning Qwen2.5:
1. Short Answer Template: no
yes
Description: The model outputs single token indicating correctness ("yes") or incorrect- ness ("no").
2. Long Answer Template:
the idiom [idiom} got translated (correctly
incorrectly).
Description: The model outputs one short sentence describing whether the idiom transla- tion is correct or incorrect, but does not include any reasoning.
Short Thought Process Final Answer Template:
The translation of the idiom is "(idiom_trea translation)" Given the context, I think this is [category), my final answer is: the idiom got translated {decision).
The model provides a brief rationale (e.g,, mentioning the idiom translation) and then states whether is correct or incorrect.
Description:
4. Long Thought Process + Final Answer Template:
Analyzing the translation of "(idiom)" [[idiom_transsatte When examining this translation: (definition based on category) Therefore, conclude this is (not) good translation. The idiom "(idiom)" has been translated (decision).
Description: The model generates a more elaborate thought, referencing predefined definitions for the relevant error category (or "Good Translation"), It concludes by stating whether the translation is correct or incorrect.
24

---

Published as a conference paper at COLM 2025
Prompt for Extracting Idiom Translations
## Task
Analyze the provided Chinese idiom, Chinese sentence containing the idiom
and its English translation to extract corresponding idiom translation.
## Input to Analyze
Chinese idiom: [PLACEHOLDER]
Chinese sentence: [PLACEHOLDER]
English translation: [PLACEHOLDER]
Please note:
1. Ensure the extracted idiom translation is short and concise, and do not include irrelevant translation.
2. If the given idiom appears multiple times in the Chinese sentence, only analyze the first occurrence.
3. If corresponding translation is found, output empty string.
Read the input carefully. Write down brief thought process first, with the 3 notes in mind. Then extract the translation of the idiom. Respond in JSON format ["Idiom translation": The key must be "Idiom translation", and the value must be the corresponding idiom translation in the English sentence.
Figure 10: Prompt for Extracting Idiom Translations
25

---

Published as a conference paper at COLM 2025
Prompt for Perturbing Idiom Translations
## Task
Analyze the provided Chinese idiom, idiom meaning, Chinese sentence containing the idiom and its English translation. The goal is to edit the given idiom translation to comply with the given category.
## Categories
There categories consider:
Mistranslation: idiom translation is incorrect due to wrong choices of words or phrases, and it affects our understanding the translated sentence.
Partial Translation: idiom is translated partially. Part of the idiom meaning is missing or the extremity is inaccurate.
Repetition: translation of the idiom contains repeated words or phrases, their synonyms.
Unnatural: the translation is ok but not perfect enough due to improper choices of words or grammar errors. In other words, the translation can be improved by more appropriate choices of words.
Literal Translation: idiom is translated literally and translation is not coherent with context.
Addition: addition good translation of idiom, the translation also contains non-present content in the source.
No Translation: there is translation of the idiom in the output.
## Input Edit
* Chinese idiom: [PLACEHOLDER]
Idiom meaning: [PLACEHOLDER]
Chinese sentence: [PLACEHOLDER]
English translation: [PLACEHOLDER]
Idiom translation: [PLACEHOLDER]
Please note:
1. Given category only applies to the idiom translation itself.
2. An idiom can have multiple meanings. Find the appropriate meaning under the given context.
Read the input carefully. YOU NEED MAKE THE IDIOM TRANSLATION [PLACEHOLDER].
First, modify the idiom translation to comply with the given category. Then, rewrite the given translation by replacing the original idiom translation with the new translation.
Do not include your thought process.
Output the corresponding edited translation only. Do not include anything else in your output.
Figure 11: Prompt for Perturbing Idiom Translations
26

---

Published as a conference paper at COLM 2025
Prompt for Editting Idiom Translations
## Task
Analyze the provided Chinese idiom, idiom meaning, Chinese sentence containing the idiom and its English translation to identify idiom translation category.
## Categories
There are categories to consider:
Good Translation: the idiom is translated perfectly.
Mistranslation: idiom translation is incorrect due to wrong choices of words or phrases, and it affects our understanding the translated sentence.
Partial Translation: idiom is translated partially. Part of the idiom meaning is missing or the extremity is inaccurate.
Repetition: translation of the idiom contains repeated words or phrases, their synonyms.
Unnatural: the translation is ok but not perfect enough due to improper choices of words or grammar errors. In other words, the translation can be improved by more appropriate choices of words.
Literal Translation: idiom is translated literally and translation is not coherent with context.
Addition: addition good translation of idiom, the translation also contains non-present content in the source.
No Translation: there is translation of the idiom in the output.
Input Analyze
* Chinese idiom: [PLACEHOLDER]
Idiom meaning: [PLACEHOLDER]
Chinese sentence: [PLACEHOLDER]
English translation: [PLACEHOLDER]
Idiom translation: [PLACEHOLDER]
Please note:
1. Focus on analyzing the translation of the idiom under the context of the sentence.
2. Compare the idiom meaning and provided idiom translation to make the judgement. Note an idiom can have multiple meanings.
3. If the given idiom appears multiple times in the Chinese sentence, only analyze the first occurrence.
4. If the idiom translation only captures the essence of the meaning, it is "Partial Translation".
Read the input carefully. Write down a brief thought process first, with the 4 notes in mind. Then identify the translation category of the idiom.
Respond in a JSON format: the keys must be "Category", and the values must be the corresponding category of the idiom translation.
Figure 12: Prompt for editting idiom translations
27

---

Published as a conference paper at COLM 2025
Zero-shot Prompt for Idiom Translation Correctness Detection
Evaluate whether the Chinese idiom is correctly translated in the following text:
Chinese Idiom: (idiom}
Chinese Text: (zh_sentence)
English Translation: (en_sentence)
Note: only output Yes or No in your response. Do not include anything else.
Figure 13: Zero-shot Prompt for Idiom Translation Correctness Detection
Zero-shot with CoT Prompt for Idiom Translation Correctness Detection
Evaluate whether the Chinese idiom is correctly translated in the following text:
Chinese Idiom: (idiom}
Chinese Text: (zh_sentence)
English Translation: (en_sentence)
Note: Analyze this step by step with the following output format:
Thought Process: [[your analysis of the idiom's meaning and translation accuracy)]
Final Answer: ((correct translation wrong translation))
Figure 14: Zero-shot with CoT Prompt for Idiom Translation Correctness Detection
28


Reviews:
[{"review_id": "lb8A82C1nF", "reviewer": "Reviewer_vzbR", "summary": "The paper focuses on translation of Chinese idioms. The authors propose an error taxonomy, create an annotated dataset and evaluate how well existing MT metrics capture errors in the translation of idioms. Finally, experiments are carried out with the goal of improving the idiom translation evaluation.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "* The annotation interface, the dataset and evaluation methodology can potentially be useful to other researchers.\n* The paper is relatively well written and easy to follow.\n* The scope of the evaluation is relatively large, including various SotA translation systems/LLMs and evaluation metrics.", "reasons_to_reject": "* There is not much novelty in the work; the paper mostly just introduces a dataset and the results of an evaluation.\n* Limited language coverage, focusing only on Chinese idioms.\n* Various inaccuracies and confusing parts in the text (see questions to authors).\n* I have some doubts about the improved methods for evaluation; based on the training data description it looks like the same idioms were present in both training and evaluation, which casts doubt on the positive result.", "questions_to_authors": "The taxonomy is documented in different places throughout the paper and the number of categories varies. At first you mention _7 different failure modes_ and Table 1 indeed contains 7 error categories. However, then you mention 9 categories and 13 subcategories. Which version is correct?\n\nI would suggest using \"minor\", \"major\" and \"critical\" for severity. For annotator confidence, something like \"low\", \"medium\", \"high\" might work better as well.", "extra_scores": {}}, {"review_id": "q8oUkQhcxL", "reviewer": "Reviewer_9war", "summary": "This paper addresses the evaluation of machine translation systems with a focus on Chinese idioms, which are rich in figurative meaning and deeply embedded cultural context. The authors introduce IDIOMEVAL, a comprehensive error taxonomy designed to assess the accuracy of idiomatic translation. \nSince existing metrics have a low correlation with human judgment, they developed an improved model that detects idiom translation errors.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "+ This paper presents the potential of LLMs in the nuanced evaluation of idiomatic translations, while also revealing that LLM systems make critical errors.\n\n+ By focusing on Chinese idiom, which is a relatively underexplored are, the authors contribute a framework that systematically categorizes translation errors in a way that reflects the linguistic and cultural complexity of Chinese.\n\n+ This study is comprehensive in scope, encompassing data collection, error taxonomy design, and evaluation methodology, and it appears to promise for adaptation to other languages.", "reasons_to_reject": "+ While the analysis involving major LLMs and CoT prompting is valuable, the experimental section feels limited in depth due to space constraints. There is a lack of variation and exploration across different conditions.\n\n+ For instance, in Figure 4, the performance drop at the 72B scale for both “short thought with answer” and “long thought with answer” prompts is noted, yet the paper does not sufficiently explore or explain the underlying causes of this decline.", "questions_to_authors": "+ As mentioned above, the “Results” section would benefit from a more focused analysis. Consider narrowing the scope of the key arguments to allow for deeper discussion.\n\n+ One suggestion is to move the CoT experiment details to the appendix, thereby freeing up space to expand on the qualitative evaluation—perhaps by including more illustrative examples of idiom translation errors and model behavior.", "extra_scores": {}}, {"review_id": "w1H2EquG4G", "reviewer": "Reviewer_j3Fh", "summary": "In this paper, an evaluation framework (i.e., IDIOMEVAL) is introduced as a new taxonomy of Chinese idiom translation errors. Using this framework, a high-quality dataset of 900 human-annotated translations based on four domains and nine LLMs is collected. The result of experiments using this dataset indicated that the translation qualities of nine modern systems and the evaluation qualities of existing metrics and prompting approaches are insufficient. Moreover, this paper proposed new evaluation models by instruction-tuning LLMs for improvement of evaluation of Chinese Idiom translation, and showed that the proposed models which are instruction-tuned LLMs outperform existing metrics and prompting approaches.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The advantages of the paper is as follows:\n\n1. This paper indicated that the modern translation systems are insufficient in Chinese idiom translation of some domains using high-quality dataset based on human-annotated translations.\n\n2. This paper indicated that the evaluation quality of existing metrics and prompting approaches is low based on the coefficient correlation with human judgements (i.e., under 0.48) in Chinese idiom translation.\n\n3. This paper proposed the instruction-tuned LLMs using dataset by human-annotated translations to detect good and bad idiom translation as a binary classification problem. Moreover, though the experimental results, it is indicated that Macro F1 of the proposed models (i.e., 0.68) are higher than metrics and prompting approaches.", "reasons_to_reject": "The effectiveness of the proposed approach is not clear for the following points;\n\n1. In the proposed models, native Chinese speakers fluent in English as annotators are required to obtain the dataset for instruction-tuning. Therefore, the annotators and cost are serious problems.\n\n2. In detection of idiom translation errors of Table 6, it is not clear whether there are statistical significances or not in F1-Macro between models.\n\n3. The reference-based or reference-free metrics are formulated as a binary classification task using ROC curve. However, it may be a little unfair because the metrics except BLEU based on n-gram matching is not learned to perform a binary classification task.\n\n4. This paper needs to evaluate idiom translation of other languages, not only Chinese.", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Orpr3RDGat", "reviewer": "Reviewer_u3jM", "summary": "This paper introduces an evaluation framework for identifying errors in Chinese idiom translation. They apply it to compare the performance of nine contemporary LLM-based systems, to translate idioms in multiple domains. The results show that these systems frequently mistranslate idioms, while existing evaluation metrics poorly assess idiom quality. The authors describe and extensive evaluation and propose alternative metrics to alleviate such an issue. \n\nThe paper is very clear and well written. The presented analysis is quite complete and well motivated. They tackle the problem (idioms translation) from the evaluation perspective, performing a comprehensive comparison between LLM-based SoA systems, proposing a new taxonomy and evaluation framework. Interesting conclusions are drawn, most of them are as expected but now they are supported by evidence. One of them is surprising, though, which is that the performance increases as the model scales up, except at 72B. It would have been interesting to hear a hypothesis about this behavior in the paper.\n\nThe proposed evaluation metric correlates better with human opinion than current ones. My impression, not being an expert in MT myself, is that the proposed framework is general enough to be applied to other languages.\n\nDespite the paper is very clear, there are a few minor issues to be fixed (see below)", "questions": "", "limitations": "", "rating": 9, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "* The paper is very clear and well written. \n* The presented analysis is quite complete and well motivated. \n* Interesting conclusions, which highlight the difficulties faced by LLMs in tasks as difficult as capturing idioms. \n* The proposed evaluation metric correlates better with human opinion than current ones.", "reasons_to_reject": "I see NO reasons to reject. I write here some minor issues to be fixed:\n\n- In the introduction, there is a missing closing parenthesis in \"... (see Table 1\"\n- Fonts in some figures (e.g., figure 2, and many other in the annex) are too small and not easily readable in printed version\n- IDIOMEVAL is sometimes introduced as a taxonomy (e.g., the abstract) and other times as a framework (e.g., the introduction). Authors need to be consistent with this. My understanding it that it is a framework that, among other things, defines a taxonomy.", "questions_to_authors": "Only one: \n* why do you think that the performance increases as the model scales up except at 72B? (no need to prove the answer, but just to formulate an hypothesis from your expert view)", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper introduces an evaluation framework for Chinese idiom translation that could potentially be extended to other languages (9war, u3jM). It includes a supporting dataset (with 900 translation pairs annotated for evaluation) and taxonomy, evaluates a range of modern MT/language models, and shows that existing metrics correlate poorly with human judgments. Finally, it develops models that achieve better Chinese idiom translation results. 

As for reasons to accept: the paper is clear and well written (j3FH, u3JM), and the dataset and evaluation methodology for Chinese idiom translation (which is relatively underexplored, 9war), including taxonomy, are quite comprehensive could be useful to other researchers (vzbR) and could be extended to other languages (9war, u3JM). The paper highlights the limitations of existing translation models and metrics with interesting results (j3FH, u3JM) and its (revealing) analysis is seen as a strong point (9war, u3JM).

As for reasons to reject, two reviewers would have liked the approach to be applied to languages beyond Chinese (vzbR, j3FH) (even though the authors reasonably make it clear that a lot of expertise is needed for idioms and annotation costs are high, so it seems reasonable to focus on just one language pair here, if compensated by e.g. a strong analysis). Only one reviewer finds the paper limited in novelty (vzbR), and one finds the experimental section somewhat shallow (9war). One reviewer (out of four) does not find any reasons to reject the paper. In author discussion it became clear that one idiom was both in the train and test set, and the authors showed that this had minimal impact on the results.

Even though idiom translation, especially using automatic systems, is relatively underexplored, I find that the paper does not situate itself very well in existing literature (its related work section is shallow and it hardly interacts with that work), and its main novelties are its mined dataset and (detailed) focus on Chinese and reporting idiom translation results and analyses of modern systems like GPT-4o, including with the metrics analysis. There is still a lot of value in that, and the reviewers found ample reasons to (weakly) accept it despite this limitation, but the paper would have been stronger had it built more on previous work. Overall, three reviewers suggest this is marginally above the acceptance threshold, and based on the above I believe it is just that.
Decision: Accept