Proverbs Run in Pairs: Evaluating Proverb Translation Capability of Large Language Model
Authors: Minghan Wang, Viet Thanh Pham, Farhad Moghimifar, Thuy-Trang Vu
Accepted at: acl25

Proverbs Run in Pairs: Evaluating Proverb Translation Capability of Large Language Model

Minghan Wang, Viet-Thanh Pham, Farhad Moghimifar, Thuy-Trang Vu
Department of Data Science & AI, Monash University
{minghan.wang, thanh.pham1, trang.vu1}@monash.edu

Abstract

Despite achieving remarkable performance, machine translation (MT) research remains underexplored in terms of translating cultural elements in languages, such as idioms, proverbs, and colloquial expressions. This paper investigates the capability of state-of-the-art neural machine translation (NMT) and large language models (LLMs) in translating proverbs, which are deeply rooted in cultural contexts. We construct a translation dataset of standalone proverbs and proverbs in conversation for four language pairs. Our experiments show that the studied models can achieve good translation between languages with similar cultural backgrounds, and LLMs generally outperform NMT models in proverb translation. Furthermore, we find that current automatic evaluation metrics such as BLEU, CHRF++ and COMET are inadequate for reliably assessing the quality of proverb translation, highlighting the need for more culturally aware evaluation metrics.1

1 Introduction

Translating multi-word figurative expressions, particularly idioms and proverbs, have long been a challenge in MT due to their meanings diverging from the literal interpretation of individual words (Constant et al., 2017; Zaninello and Birch, 2020). Previous research in neural machine translation (NMT) has primarily focused on translating idiomatic expressions to capture their figurative meanings in the source language and accurately convey them in the target language (Isabelle et al., 2017; Fadaee et al., 2018; Avramidis et al., 2019). The development of large language models (LLMs) for MT has demonstrated a reduction in overly literal translations, improving the quality of idiomatic translations (Raunak et al., 2023). However, the translation of proverbs, which are short popular sayings conveying cultural beliefs, has received comparatively less attention.

Translating proverbs presents additional challenges beyond simply preserving figurative meaning. Proverbs are often deeply rooted in cultural contexts, and their translation requires careful cultural adaptation. This involves translating culture-specific terms, paraphrasing, and considering widely accepted versions of the proverb that resonate in the target language (Shehab and Daragmeh, 2014; Newmark, 2003). Recent research by Liu et al. (2024) has shown that while LLMs possess some degree of knowledge about proverbs, their capability of reasoning with proverbs, especially in dealing with figurative proverbs, remains limited. In light of these challenges and recent developments, using proverbs as a proxy for cultural common grounds, our study examines the ability of current NMT and LLM-based MT models in proverb translation to better understand how existing MT models handle cross-cultural elements. In particular, we aim to answer the following research questions: (i) Can current MT methods, particularly LLM-based MT systems, handle proverb translation?; (ii) What are the roles of conversation contexts and prompts in proverb translation ability of LLM-based MT?; and (iii) Can current automatic evaluation metrics measure the accuracy of translating cultural nuances?

To address these research questions, we first expand the existing multicultural proverbs and sayings dataset (Liu et al., 2024) into an English-centric proverb translation dataset. As proverbs can also appear in conversations, we further mine the Proverb in Conversation (PiC) dataset by extracting proverb usage from movie subtitles. These datasets include five languages representing diverse geographical areas: English, German, Bengali, Indonesian, and Mandarin Chinese. We then conduct extensive evaluations of state-of-the-art NMT systems and multiple LLM families on these datasets

1Dataset and code are available at https://github.com/yuriak/LLMProverbMT.
to assess their proverb translation capabilities.

Our experiments reveal that current MT models demonstrate a certain level of proficiency in handling proverb translations, with notably better performance observed when translating between languages from similar cultural areas. Furthermore, we observe that the reliability of existing automatic evaluation metrics is insufficient for accurately assessing the cultural nuance in proverb translation. Specifically, we make the following contributions

• We construct a proverb translation and proverb in conversation translation dataset in four translation pairs to facilitate the future research in figurative language translation.

• Our experiments provide insights into the roles of context and prompting in the performance of proverb translation tasks. Larger models already learn the meaning of proverbs, hence, adding the proverb interpretation or example does not help. On the other hand, conversation context in dialogue format plays a more important role.

• We also find that the current automatic evaluation metrics such as BLEU and COMET as well as the LLM-as-a-judge are unreliable and very sensitive to small lexical changes when evaluating figurative translation quality.

2 Proverb Translation Dataset

2.1 Standalone Proverb Translation Dataset
Proverbs are fixed expressions that convey traditional wisdom and deeply rooted in lived experiences and socio-cultural contexts. This makes proverbs an excellent lens through which to evaluate how well the current MT model captures and translates cultural information. To facilitate such analysis, we extend the MAPS dataset (Liu et al., 2024) which is a collection of proverbs from multiple languages, including English (EN), German (DE), Bengali (BN), Mandarin Chinese (ZH) and Indonesian (ID).2 These languages come from diverse geographical regions and exhibit varying linguistic structures and resource availability according to Joshi et al. (2020). Each proverb in MAPS dataset is companied with its explanation, the machine translation into English, and a label indication whether the proverb is figurative or literal. The figurative proverbs have meanings that differ from their literal expressions, while the literal ones convey meaning directly. This dataset allows us to evaluate not only the translation quality but also how well models can handle figurative language. Table 1 provides detailed statistics.

<table>
  <tr>
    <th>Language</th>
    <th>#Prov.</th>
    <th>#Fig.</th>
    <th>Region</th>
  </tr>
  <tr>
    <td>English (EN)</td>
    <td>424</td>
    <td>232</td>
    <td>Western Europe</td>
  </tr>
  <tr>
    <td>Bengali (BN)</td>
    <td>340</td>
    <td>272</td>
    <td>South Asia</td>
  </tr>
  <tr>
    <td>German (DE)</td>
    <td>334</td>
    <td>183</td>
    <td>Western Europe</td>
  </tr>
  <tr>
    <td>Indonesian (ID)</td>
    <td>341</td>
    <td>267</td>
    <td>Southeast Asia</td>
  </tr>
  <tr>
    <td>Chinese (ZH)</td>
    <td>334</td>
    <td>143</td>
    <td>East Asia</td>
  </tr>
</table>

Table 1: MAPS dataset statistics including number of proverbs (#Prov.), number of figurative proverbs (#Fig.) and geography region.

To ensure the quality of the dataset, we recruit annotators to verify and post-edit the machine-translated proverbs. These annotators were fluent in both English and native speakers of Chinese, Indonesian, or Bengali, with experience in either professional or volunteer translation work. For German-to-English translations, although the annotators were not native German speakers, they were proficient in both German and English. The annotation process was structured to ensure the cultural and linguistic accuracy of the translations.

Annotators were presented with proverbs in their native language alongside the machine-generated English translations. Their primary task was to evaluate the correctness of these translations and post edit where necessary. Additionally, they are also tasked with identifying context-dependent proverbs whose meanings can shift based on the situation in which they are used. For English proverbs, the annotators are also asked to provide equivalent proverbs in their native languages that convey similar meanings, when possible. The native proverb provided will serve as a translation reference during the evaluation of from English translation. The details of annotation protocols and addition analysis can be found in Appendix A.

2.2 Proverb in Conversation Mining
OpenSubtitles (Lison and Tiedemann, 2016) is a multilingual parallel corpora of movie and TV subtitles. It contains culturally rich conversation and spans multiple languages, making it a good source to mine the parallel corpus of proverb usage. However, as OpenSubtitles is often included in LLM pretraining corpus, we perform data contamination
<table>
  <tr>
    <th rowspan="2">Lang</th>
    <th colspan="3">From-En</th>
    <th colspan="3">To-En</th>
  </tr>
  <tr>
    <th>P1</th>
    <th>P2</th>
    <th>P3</th>
    <th>P1</th>
    <th>P2</th>
    <th>P3</th>
  </tr>
  <tr>
    <td>BN</td>
    <td>89</td>
    <td>69</td>
    <td>42</td>
    <td>511</td>
    <td>76</td>
    <td>8</td>
  </tr>
  <tr>
    <td>DE</td>
    <td>7028</td>
    <td>2000</td>
    <td>1540</td>
    <td>1903</td>
    <td>1755</td>
    <td>1129</td>
  </tr>
  <tr>
    <td>ID</td>
    <td>2459</td>
    <td>1969</td>
    <td>1214</td>
    <td>51</td>
    <td>44</td>
    <td>13</td>
  </tr>
  <tr>
    <td>ZH</td>
    <td>3456</td>
    <td>2000</td>
    <td>272</td>
    <td>1498</td>
    <td>1488</td>
    <td>827</td>
  </tr>
</table>

Table 2: The statistics of the mined subtitles in each phrase: P1: Initial Mining §2.2.1; P2: Fine-grained Filtering §2.2.2; P3: Human Evaluation §2.2.3.

analysis in §5.3 and find that contamination is not significant enough to bias the evaluation.

2.2.1 Initial Mining with Source-side Proverb
We first collect potential translation pairs which containing proverbs from OpenSubtitles dataset. In this step, we preprocess both proverbs and translation pairs with lemmatization3. We then use an edit-distance-based string matching library4 to search for translation pairs where proverbs are contained in the source sentence. When a source sentence contains an exact match of the given proverb, its matching score will be 1.0, any replacement of characters will reduce the score. We set a threshold as 0.8 during the search.

2.2.2 Fine-grained Filtering
Although lemmatization can solve some of the matching errors caused by morphological inflection, the search results will still contain a large number of errors. Therefore we propose two fine-grained methods for further filtering through the semantics of the mined sample.

LLM-based Proverb Usage Filtering  We use an LLM5 to filter those translation pairs that contain the proverb by asking the model: “Whether the proverb is contained in the sentence”. This ensures that the filtering is performed through the semantic meaning of the given text and thus is more accurate. To make sure that the LLM’s output is reliable, we set the temperature to 0 and constrain the model’s output as “Yes” and “No”. For samples labeled as “NO”, we remove them from our candidate set.

Filtering with Quality Estimation  Another filtering process aims to ensure the translation of the collected pairs is good enough, as there are cases where source and target texts are mismatched in the subtitles due to reordering of utterances which have to be excluded. In this step, we use both LLM (LLM-QE) and a dedicated Quality Estimation with Direct Assessment (DA-QE) model to score the collected translation.
• For LLM-QE, we let the model to score the translation from 1-5 and replace the order of source and target, then, score it again to reduce the influence of the order; an average of two values is computed as the label of the candidate pair.
• For DA-QE, we use “Unbabel/wmt23-cometkiwi-da-xxl” as the dedicated DA scorer to score the translation pair in a range between 0 to 1. Then, we compute the overall score for each pair as score = score_{LLM-QE} + score_{DA-QE} × 5.

Filtering is conducted separately across language pairs as the quality of the mined corpus in each direction differs largely. Specifically, we set the maximum required sample size for each direction as 2000, and use it to compute the minimum quantile as \( q_{min} = max(0, 1 - \frac{2000}{|\mathcal{D}_{s\rightarrow t}|}) \), where \( |\mathcal{D}_{s\rightarrow t}| \) stands for the number of samples for a specific direction (e.g. En→De) and the corresponding overall score (\( max(score_{q_{min}}, 4) \)), where score_{q_{min}} is the corresponding score of the quantile \( q_{min}, 4 \) is the minimum score threshold we assigned for all language pairs) as the threshold. Finally, we used this threshold to filter qualified samples. Detailed statistics in each step are presented in Table 2.

Conversation Context Retrieval  While proverbs and sayings are self-contained, they are typically used in conversation. As we aim to study whether the provided context could influence the translation of a sentence containing a proverb, we need to retrieve the prior and proceeding sentences for each filtered translation pair. In this step, we retrieve a maximum of 5 sentences for each direction (prior and proceeding).

2.2.3 Human Evaluation
To validate the collected translation data, the annotators are presented with a proverb in source language, and ask to evaluate the translation in target language given the preceding and following context. Additionally, they are asked whether the proverb is correctly used in the given sentences. The details of annotation protocols and addition analysis can be found in Appendix A. Finally, we collect the parallel pairs which contains correct
<table>
  <tr>
    <th></th>
    <th colspan="2">Proverbs</th>
    <th colspan="2">PiC</th>
  </tr>
  <tr>
    <th></th>
    <th>#lit.</th>
    <th>#fig.</th>
    <th>#lit.</th>
    <th>#fig.</th>
  </tr>
  <tr>
    <td>en -> bn</td>
    <td>130</td>
    <td>163</td>
    <td>29</td>
    <td>13</td>
  </tr>
  <tr>
    <td>bn -> en</td>
    <td>68</td>
    <td>272</td>
    <td>2</td>
    <td>6</td>
  </tr>
  <tr>
    <td>en -> de</td>
    <td>180</td>
    <td>214</td>
    <td>1191</td>
    <td>349</td>
  </tr>
  <tr>
    <td>de -> en</td>
    <td>151</td>
    <td>183</td>
    <td>614</td>
    <td>515</td>
  </tr>
  <tr>
    <td>en -> id</td>
    <td>162</td>
    <td>183</td>
    <td>886</td>
    <td>328</td>
  </tr>
  <tr>
    <td>id -> en</td>
    <td>71</td>
    <td>262</td>
    <td>3</td>
    <td>10</td>
  </tr>
  <tr>
    <td>en -> zh</td>
    <td>134</td>
    <td>161</td>
    <td>227</td>
    <td>45</td>
  </tr>
  <tr>
    <td>zh -> en</td>
    <td>191</td>
    <td>143</td>
    <td>386</td>
    <td>441</td>
  </tr>
</table>

Table 3: Data statistics of the standalone proverb translation (Proverbs) and proverb in conversation translation (PiC). The number of samples contains literal and figurative proverbs are denoted by #lit and #fig respectively.

proverb usage and correct translation. The data statistics are shown in Table 3. Since the sample size of EN->BN, BN->EN, and ID->EN in our final PiC dataset is relatively small, we omit these three translation directions in our experiments.

3 Experimental Setup

In this paper, we investigate the ability of the current MT system, with a particular focus on LLM-based MT models, in translating proverbs. More specifically, we aim to answer the following research questions (RQs):

• RQ1: To what extent can existing MT methods accurately translate proverbs?

• RQ2: What are the roles of conversation contexts and prompts in the proverb translation ability of LLM-based MT?

• RQ3: Are current automatic evaluation metrics reliable and effective in measuring the accuracy of proverb translation?

3.1 Models

We study the proverb translation ability of the current MT methods, including

• State-of-the-art multilingual NMT We experiment with NLLB which are trained on translation data of 200 languages and available with different sets of parameters: 600M, 1.3B and 3.3B (Costa-jussà et al., 2022).6

• Instruction-following LLMs We evaluate instruction-following LLMs with different model parameter size from multiple model families: MISTRAL (7B) (Jiang et al., 2023), QWEN2 (7B) (Yang et al., 2024), LLAMA-3.1 (8B, 70B) (Dubey et al., 2024), GEMMA2 9B (Team et al., 2024) and GPT-4O MINI.7

• LLM-based MT In addition to NMT and off-the-shelf LLMs models, we also evaluate fine-tuned LLM model for MT tasks. Particularly, we consider ALMA-R 13B which is based on Llama2 13B further fine-tuning with contrastive preference optimization on high quality translation data (Xu et al., 2024b).8

3.2 Prompts

We design 5 types of prompt templates to evaluate the performance of LLM under different conditions (see Figure 6 in the Appendix). All 5 prompt templates are used in the evaluation on the PiC test set, but only the zero- and one-shot prompts are used in proverb standalone translation.

• Zero-Shot We only provide a simple system message to set a role (a professional translator) for the LLM and instruct it to translate the given source sentence without returning any irrelevant content.

• One-Shot As smaller LLMs may have limited instruction-following capability, we add an example of translating the sentence “Good morning” in the first round of dialogue. This allows LLM to have access to the dialogue history.9

• Proverb Explanation In the system message, we first signal the LLM that the proverb may contained in the given source text, and the explanation of the proverb, followed by the source sentence.

• Contextualization through Dialogue To study the role of the conversation context in translation performance, we consider previous subtitle sentences as contexts and place

6Model signatures: facebook/nllb-200-distilled-600M, facebook/nllb-200-1.3B, and facebook/nllb-200-3.3B
7Signatures: mistralai/Mistral-7B-Instruct-v0.3, Owen/Qwen2-7B-Instruct, meta-llama/Meta-Llama-3.1-8B-Instruct, meta-llama/Meta-Llama-3.1-70B-Instruct, gpt-4o-mini-2024-07-18
8Model signature: haoranxu/ALMA-13B-R.
9We use the simple sentence for the one-shot case to prevent introducing any bias to the actual translation.
<table>
  <tr>
    <th rowspan="2"></th>
    <th colspan="4">BLEU</th>
    <th colspan="4">CHRF++</th>
    <th colspan="4">COMET</th>
  </tr>
  <tr>
    <th>lit.</th><th>fig.</th><th>lit.</th><th>fig.</th><th>lit.</th><th>fig.</th><th>lit.</th><th>fig.</th>
  </tr>
  <tr>
    <th></th>
    <th>0-shot</th><th>1-shot</th><th>0-shot</th><th>1-shot</th><th>0-shot</th><th>1-shot</th><th>0-shot</th><th>1-shot</th>
  </tr>
  <tr>
    <th colspan="9">From English translation</th>
  </tr>
  <tr>
    <td>NLLB-600M</td><td>9.42</td><td>8.23</td><td>23.14</td><td>21.49</td><td>67.36</td><td>60.30</td><td></td><td></td>
  </tr>
  <tr>
    <td>NLLB-1.3B</td><td>10.66</td><td>8.97</td><td>24.12</td><td>21.88</td><td>67.94</td><td>60.37</td><td></td><td></td>
  </tr>
  <tr>
    <td>NLLB-3.3B</td><td>11.23</td><td>9.17</td><td>24.73</td><td>22.43</td><td>67.99</td><td>60.60</td><td></td><td></td>
  </tr>
  <tr>
    <td>ALMA-R 13B</td><td>6.55</td><td>5.68</td><td>17.97</td><td>16.93</td><td>62.55</td><td>55.56</td><td></td><td></td>
  </tr>
  <tr>
    <td>QWEN2</td><td>9.57</td><td>10.37</td><td>8.17</td><td>7.94</td><td>22.42</td><td>22.93</td><td>20.71</td><td>20.70</td>
  </tr>
  <tr>
    <td>MISTRAL</td><td>6.44</td><td>6.92</td><td>5.24</td><td>5.06</td><td>18.91</td><td>18.74</td><td>16.79</td><td>16.57</td>
  </tr>
  <tr>
    <td>GEMMA2 9B</td><td>13.27</td><td>14.22</td><td>10.23</td><td>10.40</td><td>24.89</td><td>25.54</td><td>22.52</td><td>22.69</td>
  </tr>
  <tr>
    <td>LLAMA-3.1 8B</td><td>10.51</td><td>11.08</td><td>8.22</td><td>8.87</td><td>23.40</td><td>23.68</td><td>20.67</td><td>21.33</td>
  </tr>
  <tr>
    <td>LLAMA-3.1 70B</td><td><b>15.16</b></td><td><b>16.83</b></td><td><b>13.53</b></td><td><b>13.97</b></td><td><b>27.59</b></td><td><b>28.68</b></td><td><b>25.59</b></td><td><b>26.12</b></td>
  </tr>
  <tr>
    <td>GPT-4O MINI</td><td>13.61</td><td>12.82</td><td>11.20</td><td>10.64</td><td>26.97</td><td>26.62</td><td>24.61</td><td>24.04</td>
  </tr>
  <tr>
    <th colspan="9">To English translation</th>
  </tr>
  <tr>
    <td>NLLB-600M</td><td>11.42</td><td>11.42</td><td>28.63</td><td>28.35</td><td>60.79</td><td>57.91</td><td></td><td></td>
  </tr>
  <tr>
    <td>NLLB-1.3B</td><td>14.30</td><td>14.12</td><td>30.83</td><td>30.70</td><td>62.37</td><td>59.38</td><td></td><td></td>
  </tr>
  <tr>
    <td>NLLB-3.3B</td><td>15.84</td><td>14.41</td><td>31.41</td><td>31.35</td><td>62.71</td><td>59.69</td><td></td><td></td>
  </tr>
  <tr>
    <td>ALMA-R 13B</td><td>13.98</td><td>17.42</td><td>29.45</td><td>32.48</td><td>61.33</td><td>61.19</td><td></td><td></td>
  </tr>
  <tr>
    <td>QWEN2</td><td>15.66</td><td>15.07</td><td>16.47</td><td>16.66</td><td>32.25</td><td>32.26</td><td>33.79</td><td>33.50</td>
  </tr>
  <tr>
    <td>MISTRAL</td><td>13.03</td><td>13.58</td><td>14.83</td><td>14.46</td><td>27.98</td><td>28.91</td><td>30.47</td><td>30.34</td>
  </tr>
  <tr>
    <td>GEMMA2 9B</td><td>17.78</td><td>17.43</td><td>18.93</td><td>18.41</td><td>34.97</td><td>34.30</td><td>35.98</td><td>35.33</td>
  </tr>
  <tr>
    <td>LLAMA-3.1 8B</td><td>15.66</td><td>16.72</td><td>16.20</td><td>16.77</td><td>30.84</td><td>31.67</td><td>32.03</td><td>32.69</td>
  </tr>
  <tr>
    <td>LLAMA-3.1 70B</td><td><b>20.85</b></td><td><b>20.10</b></td><td><b>22.35</b></td><td><b>21.55</b></td><td><b>36.73</b></td><td><b>36.57</b></td><td><b>38.22</b></td><td><b>37.26</b></td>
  </tr>
  <tr>
    <td>GPT-4O MINI</td><td>18.84</td><td>17.14</td><td>18.47</td><td>18.22</td><td><b>37.15</b></td><td>36.83</td><td>38.05</td><td>38.10</td>
  </tr>
</table>

Table 4: Results on standalone proverb translation. Bold highlights the best score in each column. The better score among one-shot and zero-shot are underlined.

them in the dialogue history, up to 5 rounds with source and target sentences acting as user input and model responses (essentially becoming a maximum of 5-shot form).

• Contextualization through Concatenation
Although using a dialogue format to provide previous sentences as contexts to the model is an intuitive approach, it may introduce noise when source and target sentence pairs in the context containing reordering. To this end, we design another approach for contextualization by concatenating all source and target sentence pairs in the context into one user input and model response, making it into a one-shot form. Through this, reordered contexts can be placed in the same round and naturally recovered to the correct alignment.

3.3 Evaluation

Evaluation Metrics We evaluate the translation quality with lexical-overlap metrics including BLEU (Papineni et al., 2002) and CHRF++ (Popović, 2017) using SacreBLEU (Post, 2018), and neural evaluation metric such as COMET (Rei et al., 2020).10

Inference We use sampling with the default decoding parameters for GPT-4O MINI, and beam search with a beam size of 5 for NLLB and open-source LLMs.

4 Main Results

4.1 Standalone Proverb Translation

NMT vs LLMs Table 4 presents the performance of various models on both literal and figurative proverb translation with zero-shot and one-shot prompting. Notably, despite being a LLM specialized for MT, ALMA-R 13B consistently underperforms compared to other LLMs across the board and even falls behind the smallest NLLB model in from-English translation direction. We speculate that it can be attributed to the absence of Bengali and Indonesian languages in its fine-tuning data. Similarly, MISTRAL also struggles on this task due to its limited language support and relatively smaller size. In contrast, other LLMs outperform NLLB models, with LLAMA-3.1 70B emerging as the strongest model.

10COMET signature: Unbabe1/wmt22-comet-da.
Zero-shot vs One-shot Prompting  One-shot prompting generally outperforms zero-shot prompting, especially in from-English translation direction. Interestingly, our strongest model LLAMA-3.1 70B achieves slightly higher score in zero-shot prompting in to-English translation tasks.

Literal vs Figurative Proverbs  Overall, the performance on figurative proverbs is consistently lower than on literal proverbs across all metrics in from-English translation directions. It is expected as the figurative proverbs have an underlying meaning different to their literal wording which makes them more challenging to translate. However, in the to-English translation direction, we notice an unexpected trend across different metrics. While NLLB models generally score higher on literal proverbs than figurative one for all metrics, the opposite trend is observed with LLMs. Specifically, LLMs achieve higher BLEU and CHRF++ scores when translating figurative proverbs, but they tend to score better COMET scores on literal proverbs.

Figure 1 breaks down the performance of 4 models on literal and figurative proverb translation in each translation direction. All models perform reasonably well in DE-EN pairs because both languages are high-resource and belong to similar cultural regions. On the other hand, BN-EN are the most challenging tasks. Overall, all models perform better on literal translation in all translation directions, except the ID->EN direction. This leads to the average performance on to-English figurative proverb translation is higher than literal ones.

![Result on proverb translation of each translation direction.](page_1012_98_388_388.png)

Figure 1: Result on proverb translation of each translation direction.

4.2 Proverb in Conversation Translation

NMT vs LLMs  Table 5 reports COMET scores of various models on the translation of proverbs in conversational contexts. Detailed results in BLEU and CHRF++ can be found in the Appendix. Consistent with earlier findings in proverb translation, MISTRAL 7B lags behind other models, while GPT-4O MINI stands out as the strongest LLM model, following by LLAMA-3.1 70B. In this particular task, interestingly, NLLB models show highly competitive results to the LLMs, especially in the literal proverb subset. However, NLLB models fall short in translating figurative proverbs.

Roles of Context and Prompting  Among the different prompting strategies, one-shot prompting consistently improves upon the performance of zero-shot prompting. However, we do not observe any notable improvement when providing explanations of proverbs, particularly in the case of literal proverbs. This may be because LLMs have already exposed to the meanings of proverbs during their pre-training, allowing them to capture these meanings without additional explanation. On the other hand, incorporating conversational context significantly enhances translation performance. This is likely due to the extra context clues in the conversations, which help the models generate more accurate translations. Furthermore, framing the context as a dialogue proves to be more effective than simple concatenation, as it aligns more naturally with the conversational nature of LLMs.

5 Analysis

5.1 Limitations of Evaluation Metrics on Proverb Translation

As proverbs are often highly culture-specific, standard evaluation metrics like BLEU, CHRF, or COMET may fall short in certain cases. In this section, we illustrate the weaknesses of these metrics regarding proverb translation. With all of the hypotheses from different models and prompt templates, we compute the cosine similarity between the sentence embedding\(^{11}\) of the hypothesis and

\(^{11}\)We use sentence-transformers/all-mmpnet-base-v2 model to obtain the embeddings.
<table>
  <tr>
    <th rowspan="2"> </th>
    <th colspan="5">Literal</th>
    <th colspan="5">Figurative</th>
  </tr>
  <tr>
    <th>0-shot</th>
    <th>1-shot</th>
    <th>EXPL.</th>
    <th>DIALOG</th>
    <th>CONCAT</th>
    <th>0-shot</th>
    <th>1-shot</th>
    <th>EXPL.</th>
    <th>DIALOG</th>
    <th>CONCAT</th>
  </tr>
  <tr>
    <td colspan="11"><i>From English translation, incl. EN-DE, EN-ID, EN-ZH</i></td>
  </tr>
  <tr>
    <td>NLLB-600M</td>
    <td>84.93</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td>78.87</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>NLLB-1.3B</td>
    <td>86.04</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td>80.18</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>NLLB-3.3B</td>
    <td>85.34</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td>80.32</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>ALMA-R 13B</td>
    <td>84.94</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td>80.41</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>QWEn2</td>
    <td>84.50</td>
    <td>84.95</td>
    <td>84.31</td>
    <td><b>85.74</b></td>
    <td><b>85.74</b></td>
    <td>80.63</td>
    <td>81.40</td>
    <td>81.24</td>
    <td><b>83.08</b></td>
    <td>82.71</td>
  </tr>
  <tr>
    <td>MISTRAL</td>
    <td>81.63</td>
    <td>81.54</td>
    <td>81.48</td>
    <td><b>83.38</b></td>
    <td>82.88</td>
    <td>76.37</td>
    <td>76.10</td>
    <td>76.66</td>
    <td><b>77.94</b></td>
    <td>77.34</td>
  </tr>
  <tr>
    <td>GEMMA 2 9B</td>
    <td>85.51</td>
    <td>85.31</td>
    <td>85.12</td>
    <td><b>86.19</b></td>
    <td>86.05</td>
    <td>82.05</td>
    <td>82.20</td>
    <td>82.54</td>
    <td><b>82.82</b></td>
    <td>82.62</td>
  </tr>
  <tr>
    <td>LLAMA-3.1 8B</td>
    <td>83.72</td>
    <td>84.69</td>
    <td>83.30</td>
    <td><b>85.75</b></td>
    <td>85.59</td>
    <td>79.16</td>
    <td>78.82</td>
    <td>78.37</td>
    <td><b>80.96</b></td>
    <td>80.57</td>
  </tr>
  <tr>
    <td>LLAMA-3.1 70B</td>
    <td>86.29</td>
    <td>86.42</td>
    <td>86.37</td>
    <td><b>87.30</b></td>
    <td>86.71</td>
    <td>84.26</td>
    <td>83.92</td>
    <td><b>84.81</b></td>
    <td>85.14</td>
    <td>84.44</td>
  </tr>
  <tr>
    <td>GPT-4O MINI</td>
    <td><b>87.22</b></td>
    <td><b>87.36</b></td>
    <td><b>86.91</b></td>
    <td><b>88.06</b></td>
    <td><b>87.95</b></td>
    <td><b>84.60</b></td>
    <td><b>84.70</b></td>
    <td>84.75</td>
    <td><b>85.59</b></td>
    <td><b>84.61</b></td>
  </tr>
  <tr>
    <td colspan="11"><i>To English translation, incl. DE-EN, ZH-EN</i></td>
  </tr>
  <tr>
    <td>NLLB-600M</td>
    <td>60.07</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td>61.61</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>NLLB-1.3B</td>
    <td>63.16</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td>63.17</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>NLLB-3.3B</td>
    <td>62.99</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td>63.64</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>ALMA-R 13B</td>
    <td>64.83</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td>65.18</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>QWEn2</td>
    <td>64.32</td>
    <td>64.04</td>
    <td>64.78</td>
    <td><b>65.86</b></td>
    <td>65.45</td>
    <td>66.41</td>
    <td>64.98</td>
    <td>66.26</td>
    <td><b>68.64</b></td>
    <td>67.74</td>
  </tr>
  <tr>
    <td>MISTRAL</td>
    <td>60.54</td>
    <td>62.86</td>
    <td>61.33</td>
    <td><b>64.83</b></td>
    <td>64.15</td>
    <td>63.14</td>
    <td>62.46</td>
    <td>62.88</td>
    <td><b>65.90</b></td>
    <td>65.10</td>
  </tr>
  <tr>
    <td>GEMMA 2 9B</td>
    <td>64.87</td>
    <td>65.93</td>
    <td>66.05</td>
    <td><b>67.83</b></td>
    <td>67.96</td>
    <td>66.22</td>
    <td>66.98</td>
    <td>66.81</td>
    <td><b>68.98</b></td>
    <td>68.37</td>
  </tr>
  <tr>
    <td>LLAMA-3.1 8B</td>
    <td>62.83</td>
    <td>65.17</td>
    <td>62.67</td>
    <td><b>67.22</b></td>
    <td>66.53</td>
    <td>63.98</td>
    <td>65.55</td>
    <td>65.15</td>
    <td><b>66.44</b></td>
    <td>66.21</td>
  </tr>
  <tr>
    <td>LLAMA-3.1 70B</td>
    <td>65.76</td>
    <td>66.28</td>
    <td><b>66.93</b></td>
    <td><b>68.83</b></td>
    <td>68.01</td>
    <td>65.59</td>
    <td>67.86</td>
    <td><b>67.25</b></td>
    <td>68.48</td>
    <td>68.44</td>
  </tr>
  <tr>
    <td>GPT-4O MINI</td>
    <td><b>66.30</b></td>
    <td><b>66.32</b></td>
    <td>66.30</td>
    <td><b>68.30</b></td>
    <td><b>68.17</b></td>
    <td><b>66.65</b></td>
    <td><b>68.03</b></td>
    <td>66.79</td>
    <td><b>68.87</b></td>
    <td><b>68.46</b></td>
  </tr>
</table>

Table 5: Subtitle Translation (COMET score) beam search. Bold highlights the best score in each column. The better score among different prompting methods is underlined.

<table>
  <tr>
    <th>Example</th>
    <th>BLEU</th>
    <th>CHRF</th>
    <th>COMET</th>
  </tr>
  <tr>
    <td>Reference: “Distance determines the stamina of a horse.”<br>Hypothesis 1: “Distance reveals the strength of a horse”<br><span style="color:red;">Hypothesis 2: “A long journey reveals the strength of a horse”</span></td>
    <td>26.27<br>19.06</td>
    <td>46.19<br>35.42</td>
    <td>82.85<br>71.58</td>
  </tr>
  <tr>
    <td>Reference: “The face of a tiger, the heart of a mouse.”<br>Hypothesis 1: “The face of a tiger, the heart of a mouse”<br><span style="color:red;">Hypothesis 2: “The look of a tiger, the heart of a rat.”</span></td>
    <td>89.32<br>71.03</td>
    <td>95.05<br>80.81</td>
    <td>93.23<br>81.99</td>
  </tr>
  <tr>
    <td>Reference: “Spare the rod and spoil the child.”<br>Hypothesis 1: “Spare the rod and spoil the child.”<br><span style="color:red;">Hypothesis 2: “Discipline brings forth filial children.”</span></td>
    <td>100.00<br>0.0</td>
    <td>100.00<br>13.93</td>
    <td>95.22<br>60.10</td>
  </tr>
</table>

Table 6: Evaluation of Hypotheses with BLEU, CHRF, and COMET Scores. Red highlights the words that make the metrics score the hypotheses lower.

its reference. We focus on cases where the cosine similarity difference of two hypotheses against the same reference is low (< 0.05), while the difference is above a threshold of 10.0, 5.0 and 10.0 for COMET, BLEU and CHRF++, respectively. From the total of 5M hypothesis pairs, we find that 22,704 pairs satisfy these thresholds. Notably, specialized NMT models (NLLB and ALMA-R 13B) have the fewest appearances in these cases, accounting for 1,962 pairs. This may indicate that LLMs may produce more creative translations that the evaluation metrics may have neglected.

Qualitative Analysis  We manually check the detected cases to identify problems of evaluation metrics. Table 6 shows some cases where the metrics are unreliable for evaluating proverb translation. In the first and second examples, the hypothesis 2 scores lower on all metrics due to the usage of different phrases to the reference, even though it conveys the same idiomatic meaning ("a long journey" instead of "distance", and "rat" instead of "mouse"). This indicates that the metrics are overly sensitive to surface-level lexical differences. The third example, "Spare the rod and spoil the child," show-
Figure 2: We present the win rate of 5 pairs of models evaluated by the GPT-4O MINI. Results are separated with From/To-En and Figurative/Literal, with corresponding COMET scores indicated near the model name.

<table>
  <tr>
    <th></th>
    <th colspan="3">Literal</th>
    <th colspan="3">From-En</th>
    <th colspan="3">Figurative</th>
  </tr>
  <tr>
    <th></th>
    <th>Win</th>
    <th>Tie</th>
    <th>Loss</th>
    <th>Win</th>
    <th>Tie</th>
    <th>Loss</th>
    <th>Win</th>
    <th>Tie</th>
    <th>Loss</th>
  </tr>
  <tr>
    <td>NLLB-3.3B-(85.34)</td>
    <td>46</td>
    <td>22</td>
    <td>31</td>
    <td></td>
    <td></td>
    <td></td>
    <td>48</td>
    <td>16</td>
    <td>34</td>
  </tr>
  <tr>
    <td>Gemma-2.9B-(86.19)</td>
    <td>51</td>
    <td>16</td>
    <td>32</td>
    <td></td>
    <td></td>
    <td></td>
    <td>59</td>
    <td>18</td>
    <td>22</td>
  </tr>
  <tr>
    <td>Llama-3.1-70B-(86.71)</td>
    <td>53</td>
    <td>19</td>
    <td>26</td>
    <td></td>
    <td></td>
    <td></td>
    <td>58</td>
    <td>21</td>
    <td>19</td>
  </tr>
  <tr>
    <td>Llama-3.1-70B-(86.71)</td>
    <td>40</td>
    <td>27</td>
    <td>32</td>
    <td></td>
    <td></td>
    <td></td>
    <td>38</td>
    <td>28</td>
    <td>33</td>
  </tr>
  <tr>
    <td>Llama-3.1-70B-(86.71)</td>
    <td>32</td>
    <td>33</td>
    <td>34</td>
    <td></td>
    <td></td>
    <td></td>
    <td>32</td>
    <td>35</td>
    <td>31</td>
  </tr>
  <tr>
    <td>GPT-4o-mini-(88.06)</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>ALMA-(84.94)</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td>ALMA-(80.41)</td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>ALMA-(84.94)</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td>ALMA-(80.41)</td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>ALMA-(84.94)</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td>ALMA-(80.41)</td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>Gemma-2.9B-(86.19)</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td>Gemma-2.9B-(82.62)</td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>Llama-3.1-70B-(84.44)</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td>GPT-4o-mini-(84.61)</td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>Llama-3.1-70B-(84.44)</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>GPT-4o-mini-(88.06)</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
</table>

<table>
  <tr>
    <th>Models</th>
    <th>En</th>
    <th>Bn</th>
    <th>De</th>
    <th>Id</th>
    <th>Zh</th>
  </tr>
  <tr>
    <td>LLAMA-3.1 8B</td>
    <td>4.5</td>
    <td>0.0</td>
    <td>3.8</td>
    <td>1.1</td>
    <td>3.8</td>
  </tr>
  <tr>
    <td>GEMMA2 9B</td>
    <td>1.3</td>
    <td>0.0</td>
    <td>1.9</td>
    <td>0.4</td>
    <td>1.7</td>
  </tr>
  <tr>
    <td>QWEN2</td>
    <td>4.7</td>
    <td>0.0</td>
    <td>1.5</td>
    <td>0.1</td>
    <td>1.2</td>
  </tr>
  <tr>
    <td>MISTRAL</td>
    <td>1.0</td>
    <td>0.0</td>
    <td>0.7</td>
    <td>0.3</td>
    <td>0.8</td>
  </tr>
</table>

Table 7: In this table, we present the percentage of samples with \( \gamma > 0.9 \) as the measurement of contamination.

cases the metrics’ inability to handle significant paraphrasing. Despite conveying the same core meaning, the metrics fail to recognize the equivalence due to their reliance on word overlap rather than deeper semantic understanding. These examples demonstrate that standard metrics can be inadequate when evaluating translations involving idioms and non-literal expressions, as they tend to penalize valid translations that do not strictly adhere to the reference’s lexical choices. Even COMET, while more robust in capturing meaning, struggles with cases of metaphorical language and significant rephrasing.

5.2 LLM-as-a-Judge Evaluation

Realizing the ineffectiveness of traditional evaluation metrics, we further use the LLM-as-a-judge method for evaluation. Here, we primarily evaluate the PiC translation results and follow the setup of Table 5. We selected five representative model pairs and compared their translation results using GPT-4O MINI, calculating the win rates. For NLLB and ALMA-R 13B, we use zero-shot results, while for other LLMs, we use results generated with dialogue prompts. The prompt used for evaluation is shown in Figure 7 in the Appendix. We prompted the model to compare the translations comprehensively based on three aspects: translation accuracy, fluency, and cultural appropriateness. Additionally, we provided the complete contextual information before and after the sample, along with the corresponding reference to assist the model in its judgment. To avoid the bias caused by the positioning of hypothesis A and B, we randomly assigned the results of the two models as A and B, thus eliminating the influence of position. Finally, GPT-4O MINI will return one of three results: A, B, or tie.

In Figure 2, we present the win rates along with the COMET scores of the two groups of models. It can be seen that the win rates and COMET scores generally follow a consistent trend (models with higher COMET scores usually have higher win rates). However, the gap in COMET scores does not strictly correlate linearly with the win rate. For instance, a larger difference in COMET scores does not necessarily mean a higher win rate (e.g. LLAMA-3.1 70B vs ALMA-R 13B, LLAMA-3.1 70B vs GEMMA2 9B). This suggest that evaluation with LLM-as-judge cannot fully solve the limitations of traditional evaluation methods.

5.3 Data Contamination Analysis

Following (Liu et al., 2024), we measure the memorization rate of the Proverb-in-Conversation dataset by assessing the model ability to complete an utterance based on partial context. This is quantified using the longest common subsequence (LCS) rate between the model’s prediction and the actual utterance, given the preceding context. However, since proverbs are often well-known phrases, models might accurately predict them even without relying
on the provided context. To account for this, we introduce the contamination rate \( \gamma \) as the difference between the LCS of the model’s prediction and the reference with and without context, normalized by the utterance length. We provide more details of this metric in Appendix B.1.

A higher value of \( \gamma \) indicates a greater likelihood that the model has been affected by data contamination for a given sample. We present the percentage of samples with \( \gamma > 0.9 \) in Table 7 across each language. A relatively higher correlation between the contamination rate and the resource-level of the language can be found, but the correlation to the translation performance is not significant. This suggests that the contamination issue is not biasing our evaluation.

6 Related works

Figurative Expression in MT  Multi-word expressions (MWEs), including idioms, phrasal verbs, and multi-word named entities, present a unique challenge in natural language processing due to their non-compositional nature, where the meaning of the whole expression cannot be easily inferred from the meanings of its individual words (Constant et al., 2017). Previous research has largely focused on understanding and paraphrasing MWEs, particularly in English (Liu and Hwa, 2016; Wada et al., 2023). In NMT literature, much focuses has been on idiomatic and slang translation, primarily targeting European languages (Fadae et al., 2018; Sun et al., 2022; Baziotis et al., 2023). However, a key obstacle to further progress in this area is the absence of standardized evaluation benchmarks and metrics. Our work addresses this gap by constructing a dataset on the translation of proverbs for four language pairs, each representing distinct geographical and cultural regions.

LLM-based MT  Several studies have explored the application of LLMs for translation tasks, highlighting their impressive performance across multiple high-resource language pairs (Xu et al., 2024a,b; Wu et al., 2024). One notable advantage of LLMs over traditional neural machine translation (NMT) systems is their ability to generate more controlled and nuanced translations, particularly when dealing with idiomatic expressions that require less literal interpretation (Manakhimova et al., 2023; Stap et al., 2024). In this work, we focus on evaluating the capabilities of LLMs in proverb translation, a challenging task due to the cultural and figurative nature of proverbs.

7 Conclusion

We curate a proverb and its usage in conversation to investigate the ability of LLMs on proverb translation. Our experiments reveal that LLMs generally outperform NMT model on this task, showcasing the advantage of LLMs in translating figurative expressions, especially between high-resource languages and languages from similar cultural region. Additionally, our analysis also reveals that current automatic evaluation metrics are unreliable in measuring translation with figurative languages.

Limitation

Although we have constructed the Proverb in Conversation dataset and conducted systematic evaluations of different models, we acknowledge the following two limitations of this study:

• The scale of our dataset is currently relatively small. This is partly due to the limited number of proverbs in each language, which constrains the size of the samples that can be collected. Additionally, our strict filtering and human annotation process further excluded low-quality samples, which may have led to excessive discarding and thus limited the dataset size. In future work, we will consider expanding data sources and ensure that the scale of the collected data meets the needs for more comprehensive evaluations.

• Another limitation lies in the relatively limited variety of models and prompts tested. Although most of them are commonly used models, they still cannot fully represent the capabilities of other models, especially those with more than 70B or fewer than 7B parameters. Therefore, in our future work, we will further expand the selection of models and the variety of prompts to enhance the comprehensiveness of the evaluation.

Acknowledgments

The authors are grateful to the anonymous reviewers for their helpful comments. This research/work was supported by Monash eResearch capabilities, including m3.
References

Eleftherios Avramidis, Vivien Macketanz, Ursula Strohriegl, and Hans Uszkoreit. 2019. Linguistic evaluation of German-English machine translation using a test suite. In Proceedings of the Fourth Conference on Machine Translation (Volume 2: Shared Task Papers, Day 1), pages 445–454, Florence, Italy. Association for Computational Linguistics.

Christos Baziotis, Prashant Mathur, and Eva Hasler. 2023. Automatic evaluation and analysis of idioms in neural machine translation. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pages 3682–3700, Dubrovnik, Croatia. Association for Computational Linguistics.

Mathieu Constant, Gülşen Eryiğit, Johanna Monti, Loneke van der Plas, Carlos Ramisch, Michael Rosner, and Amalia Todorascu. 2017. Multword Expression Processing: A Survey. Computational Linguistics, 43(4):837–892.

Marta R Costa-jussà, James Cross, Onur Çelebi, Maha Elbayad, Kenneth Heafield, Kevin Heffernan, Elahe Kalbassi, Janice Lam, Daniel Licht, Jean Maillard, et al. 2022. No language left behind: Scaling human-centered machine translation. arXiv preprint arXiv:2207.04672.

Abhimanyu Dubey, Abhinav Jauhari, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783.

Marzieh Fadaee, Arianna Bisazza, and Christof Monz. 2018. Examining the tip of the iceberg: A data set for idiom translation. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), Miyazaki, Japan. European Language Resources Association (ELRA).

Pierre Isabelle, Colin Cherry, and George Foster. 2017. A challenge set approach to evaluating machine translation. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 2486–2496, Copenhagen, Denmark. Association for Computational Linguistics.

Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. 2023. Mistral 7b. arXiv preprint arXiv:2310.06825.

Pratik Joshi, Sebastin Santy, Amar Budhiraja, Kalika Bali, and Monojit Choudhury. 2020. The state and fate of linguistic diversity and inclusion in the NLP world. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 6282–6293, Online. Association for Computational Linguistics.

Pierre Lison and Jörg Tiedemann. 2016. OpenSubtitles2016: Extracting large parallel corpora from movie and TV subtitles. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC’16), pages 923–929, Portorož, Slovenia. European Language Resources Association (ELRA).

Changsheng Liu and Rebecca Hwa. 2016. Phrasal substitution of idiomatic expressions. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 363–373, San Diego, California. Association for Computational Linguistics.

Chen Liu, Fajri Koto, Timothy Baldwin, and Iryna Gurevych. 2024. Are multilingual LLMs culturally-diverse reasoners? an investigation into multicultural proverbs and sayings. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 2016–2039, Mexico City, Mexico. Association for Computational Linguistics.

Shushen Manakhimova, Eleftherios Avramidis, Vivien Macketanz, Ekaterina Lapshinova-Koltunski, Sergei Bagdasarov, and Sebastian Möller. 2023. Linguistically motivated evaluation of the 2023 state-of-the-art machine translation: Can ChatGPT outperform NMT? In Proceedings of the Eighth Conference on Machine Translation, pages 224–245, Singapore. Association for Computational Linguistics.

Peter Newmark. 2003. A textbook of translation.

Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pages 311–318, Philadelphia, Pennsylvania, USA. Association for Computational Linguistics.

Maja Popović. 2017. chrF++: words helping character n-grams. In Proceedings of the Second Conference on Machine Translation, pages 612–618, Copenhagen, Denmark. Association for Computational Linguistics.

Matt Post. 2018. A call for clarity in reporting BLEU scores. In Proceedings of the Third Conference on Machine Translation: Research Papers, pages 186–191, Brussels, Belgium. Association for Computational Linguistics.

Vikas Raunak, Arul Menezes, Matt Post, and Hany Hassan. 2023. Do GPTs produce less literal translations? In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 1041–1050, Toronto, Canada. Association for Computational Linguistics.

Ricardo Rei, Craig Stewart, Ana C Farinha, and Alon Lavie. 2020. COMET: A neural framework for MT
evaluation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2685–2702, Online. Association for Computational Linguistics.

Ekrema Shehab and Abdelkarim Daragmeh. 2014. A context-based approach to proverb translation: The case of arabic into english translation. Translation Review, 90(1):51–68.

David Stap, Eva Hasler, Bill Byrne, Christof Monz, and Ke Tran. 2024. The fine-tuning paradox: Boosting translation quality without sacrificing LLM abilities. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 6189–6206, Bangkok, Thailand. Association for Computational Linguistics.

Zhewei Sun, Richard Zemel, and Yang Xu. 2022. Semantically informed slang interpretation. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 5213–5231, Seattle, United States. Association for Computational Linguistics.

Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, Léonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ramé, et al. 2024. Gemma 2: Improving open language models at a practical size. arXiv preprint arXiv:2408.00118.

Takashi Wada, Yuji Matsumoto, Timothy Baldwin, and Jey Han Lau. 2023. Unsupervised paraphrasing of multiword expressions. In Findings of the Association for Computational Linguistics: ACL 2023, pages 4732–4746, Toronto, Canada. Association for Computational Linguistics.

Minghao Wu, Thuy-Trang Vu, Lizhen Qu, George Foster, and Gholamreza Haffari. 2024. Adapting large language models for document-level machine translation. arXiv preprint arXiv:2401.06468.

Haoran Xu, Young Jin Kim, Amr Sharaf, and Hany Hassan Awadalla. 2024a. A paradigm shift in machine translation: Boosting translation performance of large language models. In The Twelfth International Conference on Learning Representations.

Haoran Xu, Amr Sharaf, Yunmo Chen, Weiting Tan, Lingfeng Shen, Benjamin Van Durme, Kenton Murray, and Young Jin Kim. 2024b. Contrastive preference optimization: Pushing the boundaries of LLM performance in machine translation. In Forty-first International Conference on Machine Learning.

An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, et al. 2024. Qwen2 technical report. arXiv preprint arXiv:2407.10671.

Andrea Zaninello and Alexandra Birch. 2020. Multi-word expression aware neural machine translation. In Proceedings of the Twelfth Language Resources and Evaluation Conference, pages 3816–3825, Marseille, France. European Language Resources Association.
A Data Annotation

We use Label Studio12 as our annotation platform. For Bengali, Indonesian and Mandarin Chinese, we recruit four annotators per language. For German-English, two annotators are recruited.

Figure 3 shows the instruction to annotate proverb translation dataset. Figure 4 shows the Label Studio interface for the task.

B Additional Experiment Details

B.1 Data Contamination Analysis

We measure the contamination rate of the LLM using a method similar to Liu et al. (2024). Specifically, for a given sentence and its preceding context \((s_t, s_{<t})\), we create an input prefix by truncating \(s_t\) to a certain proportion of its words (\(\tau\)), denoting the prefix as \(s_t^{<\tau}\) and the remaining part as the suffix \(s_t^{\geq\tau}\). Two types of prompts are then created, one with context (\(X_c = [s_{<t}; s_t^{<\tau}]\)) and one without context (\(X_0 = s_t^{<\tau}\)), where [:] represents string concatenation.

We use the base version of the LLM13 to complete the given prefix in both forms, resulting in hypotheses denoted as \(\hat{Y}^c\) (with context) and \(\hat{Y}^0\) (without context). During the generation, the greedy search strategy (temperature set as 0) is used to ensure the reproduction of the result.

Finally, we compute the length of the longest common subsequence (LCS) between the model’s predictions (with and without context) and the reference suffix, denoted as \(|\mathrm{LCS}(\hat{Y}^c, s_t^{\geq\tau})|\) and \(|\mathrm{LCS}(\hat{Y}^0, s_t^{\geq\tau})|\). The contamination ratio \(\gamma\) is then estimated as:

\[
\gamma = \max(0, \frac{|\mathrm{LCS}(\hat{Y}^c, s_t^{\geq\tau})| - |\mathrm{LCS}(\hat{Y}^0, s_t^{\geq\tau})|}{|s_t^{\geq\tau}|})
\]

(1)

A higher value of \(\gamma\) indicates a greater likelihood that the model has been affected by data contamination for a given sample. The reasoning behind this is as follows: (i) A large \(\gamma\) occurs only when the first term (\(|\mathrm{LCS}(\hat{Y}^c, s_t^{\geq\tau})|\)) is significantly larger than the second term (\(|\mathrm{LCS}(\hat{Y}^0, s_t^{\geq\tau})|\)), indicating that the model can predict the suffix accurately when given context but struggles without it. (ii) When both terms are large, it likely means the sample is a proverb or well-known phrase, which the model can predict accurately even without context, resulting in a small \(\gamma\). (iii) When both terms are small, it suggests that the model lacks sufficient knowledge to predict the suffix, with or without context.

B.2 Prompt Template for Translation

B.3 Prompt Template for LLM-based Evaluation

12https://labelstud.io/
13To reduce costs, we evaluated four LLMs at the 7-9B scale.
Objective
Your task is to analyse a given proverb in a language other than English and its provided English translation. You will evaluate the context dependency of the proverb, assess the accuracy of the translation, and suggest corrections if necessary.

- Please be as clear and concise as possible in your explanations.
- If you are unsure about the context dependency or translation accuracy, provide your best judgement and include a note explaining your reasoning.
- Use the space provided for each question to write your answers.

Steps to Follow

1. Read the Proverb and Translation:
   - Carefully read the provided proverb in the original language.
   - Read the given English translation of the proverb.

2. Context Dependency:
   - Determine whether the meaning of the proverb changes based on different contexts or situations.
   - Question to Answer: Is the meaning of this proverb context-dependent?
   - Options: Yes / No

3. Translation Accuracy:
   - Assess the accuracy of the provided English translation.
   - Question to Answer: Is the provided English translation accurate?
   - Options: Yes / No
   - If you select "No," please provide the correct translation of the proverb in English.

Figure 3: Proverb Translation Annotation Instruction

Proverb Translation Assessment

Chinese proverb
水能载舟，亦能覆舟

English translation
Water can carry a boat, but it can also overturn it

Is this proverb context dependent, i.e its meaning changes depending on the context?
Yes[1]  No[2]

Is the English translation correct?
Yes[3]  No[4]

If the translation is not correct, please provide the correct translation below.

Add

Figure 4: Proverb Translation Annotation Interface
Proverb Translation Assessment

Chinese proverb
水能载舟，亦能覆舟
Preceding context
被人打也不知道痛
但是稍微多喝一点就会失去理智
所以醉酒就让人产生一种威猛的错觉
喝酒的人都知道酒是越喝越多
为了打醉拳变成酒鬼 是不值得的
Chinese sentences
要知酒水能载舟 亦能覆舟
English translations
"Boats can float on water," or so it says, "but they can sink in it, too."
Following context
喝酒也能喝醉 知道吗
真的吗
威士忌加水喝会醉
波本酒加水喝会醉
琴酒加水喝也会醉 都是这样

Is the proverb correctly used in the given sentences?
Yes[1]  No[2]

Is the translation correct?
Yes[1]  No[2]

Figure 5: OpenSubtitle Translation Annotation Interface
<table>
  <tr>
    <th rowspan="2"></th>
    <th colspan="5">Literal</th>
    <th colspan="5">Figurative</th>
  </tr>
  <tr>
    <th>0-shot</th>
    <th>1-shot</th>
    <th>EXPL.</th>
    <th>DIALOG</th>
    <th>CONCAT</th>
    <th>0-shot</th>
    <th>1-shot</th>
    <th>EXPL.</th>
    <th>DIALOG</th>
    <th>CONCAT</th>
  </tr>
  <tr>
    <td colspan="11"><i>From English translation, incl. EN-DE, EN-ID, EN-Zh</i></td>
  </tr>
  <tr>
    <td>NLLB-600M</td><td>27.33</td><td></td><td></td><td></td><td></td><td>27.37</td><td></td><td></td><td></td><td></td>
  </tr>
  <tr>
    <td>NLLB-1.3B</td><td>29.85</td><td></td><td></td><td></td><td></td><td>29.38</td><td></td><td></td><td></td><td></td>
  </tr>
  <tr>
    <td>NLLB-3.3B</td><td>30.70</td><td></td><td></td><td></td><td></td><td>31.01</td><td></td><td></td><td></td><td></td>
  </tr>
  <tr>
    <td>ALMA-R 13B</td><td>20.14</td><td></td><td></td><td></td><td></td><td>23.05</td><td></td><td></td><td></td><td></td>
  </tr>
  <tr>
    <td>Qwen2</td><td>23.63</td><td>24.87</td><td>22.11</td><td>27.31</td><td>26.89</td><td>26.12</td><td>25.88</td><td>26.00</td><td>30.72</td><td>29.08</td>
  </tr>
  <tr>
    <td>MISTRAL</td><td>17.87</td><td>18.66</td><td>16.12</td><td>22.33</td><td>22.09</td><td>19.80</td><td>19.74</td><td>18.70</td><td>22.23</td><td>21.99</td>
  </tr>
  <tr>
    <td>GEMMA2 9B</td><td>27.93</td><td>28.24</td><td>26.85</td><td>30.94</td><td>30.88</td><td>32.01</td><td>31.08</td><td>31.37</td><td>33.81</td><td>33.17</td>
  </tr>
  <tr>
    <td>Llama-3.1 8B</td><td>23.85</td><td>26.62</td><td>23.26</td><td>28.99</td><td>29.13</td><td>25.64</td><td>26.09</td><td>24.98</td><td>29.45</td><td>29.05</td>
  </tr>
  <tr>
    <td>Llama-3.1 70B</td><td>28.59</td><td>29.19</td><td>28.68</td><td>33.30</td><td>32.40</td><td>35.61</td><td>35.72</td><td>35.07</td><td>37.97</td><td>36.54</td>
  </tr>
  <tr>
    <td>GPT-4O MINI</td><td>29.92</td><td>30.37</td><td>27.86</td><td>32.34</td><td>32.57</td><td>34.27</td><td>34.51</td><td>33.91</td><td>36.47</td><td>35.03</td>
  </tr>
  <tr>
    <td colspan="11"><i>To English translation, incl. DE-EN, ZH-EN</i></td>
  </tr>
  <tr>
    <td>NLLB-600M</td><td>13.96</td><td></td><td></td><td></td><td></td><td>18.96</td><td></td><td></td><td></td><td></td>
  </tr>
  <tr>
    <td>NLLB-1.3B</td><td>18.28</td><td></td><td></td><td></td><td></td><td>23.05</td><td></td><td></td><td></td><td></td>
  </tr>
  <tr>
    <td>NLLB-3.3B</td><td>19.31</td><td></td><td></td><td></td><td></td><td>24.16</td><td></td><td></td><td></td><td></td>
  </tr>
  <tr>
    <td>ALMA-R 13B</td><td>16.99</td><td></td><td></td><td></td><td></td><td>22.42</td><td></td><td></td><td></td><td></td>
  </tr>
  <tr>
    <td>Qwen2</td><td>14.77</td><td>15.56</td><td>13.51</td><td>20.05</td><td>18.06</td><td>22.27</td><td>22.44</td><td>19.85</td><td>26.03</td><td>24.48</td>
  </tr>
  <tr>
    <td>MISTRAL</td><td>12.23</td><td>13.40</td><td>11.55</td><td>19.76</td><td>17.89</td><td>18.56</td><td>19.18</td><td>17.52</td><td>23.45</td><td>21.64</td>
  </tr>
  <tr>
    <td>GEMMA2 9B</td><td>18.17</td><td>19.23</td><td>17.19</td><td>24.51</td><td>23.66</td><td>22.97</td><td>23.34</td><td>22.40</td><td>27.29</td><td>26.66</td>
  </tr>
  <tr>
    <td>Llama-3.1 8B</td><td>16.52</td><td>18.42</td><td>16.04</td><td>23.65</td><td>22.48</td><td>20.31</td><td>20.89</td><td>20.91</td><td>24.89</td><td>24.54</td>
  </tr>
  <tr>
    <td>Llama-3.1 70B</td><td>19.28</td><td>20.25</td><td>19.56</td><td>25.96</td><td>24.07</td><td>23.53</td><td>24.13</td><td>23.89</td><td>28.45</td><td>28.08</td>
  </tr>
  <tr>
    <td>GPT-4O MINI</td><td>18.07</td><td>17.78</td><td>17.25</td><td>22.48</td><td>21.88</td><td>22.72</td><td>22.83</td><td>21.68</td><td>26.08</td><td>24.89</td>
  </tr>
</table>

Table 8: BLEU scores on Proverb in Conversation Translation. <b>Bold</b> highlights the best score in each column. The better score among different prompts are <u>underlined</u>.

<table>
  <tr>
    <th rowspan="2"></th>
    <th colspan="5">Literal</th>
    <th colspan="5">Figurative</th>
  </tr>
  <tr>
    <th>0-shot</th>
    <th>1-shot</th>
    <th>EXPL.</th>
    <th>DIALOG</th>
    <th>CONCAT</th>
    <th>0-shot</th>
    <th>1-shot</th>
    <th>EXPL.</th>
    <th>DIALOG</th>
    <th>CONCAT</th>
  </tr>
  <tr>
    <td colspan="11"><i>From English translation, incl. EN-DE, EN-ID, EN-Zh</i></td>
  </tr>
  <tr>
    <td>NLLB-600M</td><td>40.39</td><td></td><td></td><td></td><td></td><td>41.81</td><td></td><td></td><td></td><td></td>
  </tr>
  <tr>
    <td>NLLB-1.3B</td><td>42.45</td><td></td><td></td><td></td><td></td><td>43.31</td><td></td><td></td><td></td><td></td>
  </tr>
  <tr>
    <td>NLLB-3.3B</td><td>43.66</td><td></td><td></td><td></td><td></td><td>44.17</td><td></td><td></td><td></td><td></td>
  </tr>
  <tr>
    <td>ALMA-R 13B</td><td>34.29</td><td></td><td></td><td></td><td></td><td>37.36</td><td></td><td></td><td></td><td></td>
  </tr>
  <tr>
    <td>Qwen2</td><td>37.51</td><td>38.27</td><td>36.39</td><td>40.93</td><td>39.43</td><td>40.40</td><td>40.18</td><td>40.05</td><td>46.46</td><td>41.46</td>
  </tr>
  <tr>
    <td>MISTRAL</td><td>31.65</td><td>31.71</td><td>31.74</td><td>36.79</td><td>35.52</td><td>34.60</td><td>34.34</td><td>33.94</td><td>40.53</td><td>40.16</td>
  </tr>
  <tr>
    <td>GEMMA2 9B</td><td>39.74</td><td>39.47</td><td>38.78</td><td>43.54</td><td>41.76</td><td>44.59</td><td>44.27</td><td>42.58</td><td>44.97</td><td>44.56</td>
  </tr>
  <tr>
    <td>Llama-3.1 8B</td><td>36.24</td><td>38.35</td><td>35.62</td><td>41.26</td><td>41.38</td><td>38.76</td><td>39.91</td><td>38.64</td><td>42.02</td><td>41.35</td>
  </tr>
  <tr>
    <td>Llama-3.1 70B</td><td>41.38</td><td>41.41</td><td>41.23</td><td>45.15</td><td>43.74</td><td>46.77</td><td>46.79</td><td>46.61</td><td>47.97</td><td>47.16</td>
  </tr>
  <tr>
    <td>GPT-4O MINI</td><td>42.82</td><td>42.98</td><td>41.77</td><td>45.89</td><td>44.69</td><td>46.15</td><td>46.19</td><td>46.18</td><td>47.93</td><td>46.81</td>
  </tr>
  <tr>
    <td colspan="11"><i>To English translation, incl. DE-EN, ZH-EN</i></td>
  </tr>
  <tr>
    <td>NLLB-600M</td><td>29.52</td><td></td><td></td><td></td><td></td><td>34.35</td><td></td><td></td><td></td><td></td>
  </tr>
  <tr>
    <td>NLLB-1.3B</td><td>33.11</td><td></td><td></td><td></td><td></td><td>36.83</td><td></td><td></td><td></td><td></td>
  </tr>
  <tr>
    <td>NLLB-3.3B</td><td>33.82</td><td></td><td></td><td></td><td></td><td>37.89</td><td></td><td></td><td></td><td></td>
  </tr>
  <tr>
    <td>ALMA-R 13B</td><td>33.00</td><td></td><td></td><td></td><td></td><td>37.72</td><td></td><td></td><td></td><td></td>
  </tr>
  <tr>
    <td>Qwen2</td><td>31.84</td><td>32.07</td><td>31.43</td><td>35.65</td><td>34.61</td><td>38.19</td><td>38.39</td><td>36.72</td><td>41.01</td><td>39.91</td>
  </tr>
  <tr>
    <td>MISTRAL</td><td>27.26</td><td>29.73</td><td>27.55</td><td>34.19</td><td>33.24</td><td>34.08</td><td>34.68</td><td>33.54</td><td>38.39</td><td>36.94</td>
  </tr>
  <tr>
    <td>GEMMA2 9B</td><td>34.03</td><td>34.96</td><td>33.61</td><td>38.98</td><td>38.74</td><td>38.65</td><td>38.86</td><td>38.35</td><td>42.31</td><td>41.75</td>
  </tr>
  <tr>
    <td>Llama-3.1 8B</td><td>30.71</td><td>33.47</td><td>30.96</td><td>37.50</td><td>36.86</td><td>34.94</td><td>35.59</td><td>35.66</td><td>39.06</td><td>39.03</td>
  </tr>
  <tr>
    <td>Llama-3.1 70B</td><td>34.64</td><td>35.33</td><td>35.86</td><td>40.26</td><td>38.94</td><td>38.09</td><td>38.66</td><td>39.40</td><td>42.39</td><td>42.42</td>
  </tr>
  <tr>
    <td>GPT-4O MINI</td><td>34.57</td><td>34.47</td><td>34.15</td><td>37.87</td><td>37.75</td><td>39.48</td><td>39.35</td><td>38.41</td><td>42.04</td><td>41.36</td>
  </tr>
</table>

Table 9: CHRF++ scores on Proverb in Conversation Translation. <b>Bold</b> highlights the best score in each column. The better score among different prompts are <u>underlined</u>.
<table>
  <tr>
    <th>System Message</th>
    <th>System Message</th>
    <th>System Message</th>
    <th>System Message</th>
  </tr>
  <tr>
    <td>You are a professional translator. Your task is to translate the user input from [src_lang_name] to [tgt_lang_name]. Remember! Don’t return any irrelevant content except from the translation!</td>
    <td>You are a professional translator. Your task is to translate the user input from [src_lang_name] to [tgt_lang_name]. Remember! Don’t return any irrelevant content except from the translation!</td>
    <td>You are a professional translator. Your task is to translate the user input from [src_lang_name] to [tgt_lang_name]. Remember! Don’t return any irrelevant content except from the translation!</td>
    <td>You are a professional translator. Your task is to translate the user input from [src_lang_name] to [tgt_lang_name]. Remember! Don’t return any irrelevant content except from the translation!</td>
  </tr>
  <tr>
    <th>User Message</th>
    <th>User Message</th>
    <th>User Message</th>
    <th>User Message</th>
  </tr>
  <tr>
    <td>[src]</td>
    <td>[src_sample]</td>
    <td>[tgt_sample]</td>
    <td>[src]</td>
  </tr>
  <tr>
    <th>Zero-shot Prompt</th>
    <th>Assistant Message</th>
    <th>User Message</th>
    <th>One-shot Prompt</th>
  </tr>
  <tr>
    <td></td>
    <td>[tgt_sample]</td>
    <td>[src]</td>
    <td></td>
  </tr>
  <tr>
    <th>Proverb-Explanation Prompt</th>
    <th></th>
    <th></th>
    <th></th>
  </tr>
  <tr>
    <td>[src]</td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <th>Contextualization-through-Dialogue Prompt</th>
    <th></th>
    <th></th>
    <th>Contextualization-through-Concatenation Prompt</th>
  </tr>
  <tr>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td colspan="4">For i in (0...5):<br>User Message<br>[src_utterance_history[i]]<br>Assistant Message<br>[tgt_utterance_history[i]]<br>End For<br>User Message<br>[src]</td>
    <td colspan="4">User Message<br>" ".join(src_utterance_history[i] for i in range(5))<br>Assistant Message<br>" ".join(tgt_utterance_history[i] for i in range(5))<br>User Message<br>[src]</td>
  </tr>
</table>

Figure 6: The prompt template of subtitle translation.
You are an expert bilingual translation evaluator in {src_lang} and (tgt_lang), with deep knowledge of both cultures, especially in the use of proverbs and idioms. Your task is to evaluate two translations ({src_lang} -> {tgt_lang}) that may contain proverbs or culturally specific content.

Evaluation Criteria:

1. Accuracy: Does the translation accurately convey the meaning and intent of the source text, considering the context?
2. Fluency: Is the translation grammatically correct and natural-sounding in the target language?
3. Cultural Appropriateness: Has the translation appropriately handled cultural references, proverbs, and idiomatic expressions? Specifically, are proverbs translated into equivalent proverbs or expressions in the target language to maintain cultural resonance?

Context:

• Preceding Context ({src_lang}):
{src_pre_context}

• Preceding Context ({tgt_lang}):
{tgt_pre_context}

• Following Context ({src_lang}):
{src_post_context}

• Following Context ({tgt_lang}):
{tgt_post_context}

Source Text ({src_lang}):
{src}

Target Reference ({tgt_lang}):
{ref}

Translation A:
{hyp1}

Translation B:
{hyp2}

Instructions:

• Analyze both translations based on the criteria above, taking into account the context and the reference provided.
• Conclude by stating which translation is better overall according to the evaluation criteria, or if they are equally good or bad.
• Return your answer in a JSON object: {"result":"A"} or {"result":"B"} or {"result":"tie"}.

Question:

Which translation is better according to the evaluation criteria above?

Figure 7: The prompt template of LLM-based evaluation.

Reviewer paper summaries: This paper focuses on evaluating the proverb translation capabilities of state - of - the - art neural machine translation (NMT) systems and large language models (LLMs). The authors construct a translation dataset for standalone proverbs and proverbs in conversation across four language pairs. They conduct extensive experiments on various models, including NLLB, instruction - following LLMs, and fine - tuned LLM - based MT models. The results show that LLMs generally outperform NMT models in proverb translation, especially between languages with similar cultural backgrounds. However, current automatic evaluation metrics like BLEU, CHRF++, and COMET are found to be inadequate for assessing proverb translation quality.

This paper introduces a proverb translation dataset for four language pairs, designed to benchmark MT systems' ability to handle cross-cultural elements. Experiments on various MT systems offer insights into the roles of context and prompting in proverb translation tasks. Additionally, the authors point out that traditional translation metrics are insufficient for assessing proverb translation quality.
Summary of strengths: - **Dataset Construction**: The construction of a comprehensive dataset for proverb translation, including both standalone proverbs and proverbs in conversation, is a significant strength. The dataset covers multiple languages from diverse geographical regions, which allows for a more comprehensive evaluation of models' ability to handle cross - cultural elements in proverb translation.

- This paper presents a proverb translation dataset with context for four language pairs, collected through a rigorous data filtering process.
- It evaluates a diverse range of MT systems, including both NMT and LLM-based MT, highlighting the strengths of LLM-based MT in proverb translation.
- The paper is well-structured and clearly written.
Summary of weaknesses: - **Insufficient Exploration of Cultural Adaptation Strategies**: Although the paper emphasizes the cultural nature of proverbs and the need for cultural adaptation in translation, it does not deeply explore different cultural adaptation strategies. There is no in-depth analysis of how models can better handle culture-specific terms, or which strategies are most effective for different types of proverbs and cultural contexts. This lack of exploration means that the practical implications for improving proverb translation in real-world scenarios are not fully developed.

- There is no comparison between the proposed dataset and existing proverb translation datasets. While the authors do not claim that theirs is the first, they also do not provide a comparison to demonstrate its strengths. Additionally, the related work section is placed in the appendix and lacks a clear summary of prior research on proverb translation.
- The authors argue that context aids proverb translation. However, when prompting LLMs with context, they include both the source and target sentences, which may serve as few-shot examples for the model. Why not include only the context in the source language?
- The claim that traditional translation metrics are insufficient for proverb translation is primarily based on observation. Although the authors state that the win rate results in Figure 2 do not directly align with COMET differences, they appear to be closely related. Could the authors provide correlation coefficients between the win rate and COMET difference to better quantify the limitations of existing metrics? Additionally, beyond using LLM-as-a-judge, incorporating human evaluation would further strengthen the analysis.
Comments, suggestions and typos: See weakness.

- The related work is placed in the appendix, making it difficult for readers to follow the paper’s motivation.
- In Table 3, the dataset appears highly imbalanced, with some translation directions containing fewer than 10 examples. Can translation performance in these cases be taken seriously?
- The distinction between Contextualization through Dialogue and Contextualization through Concatenation is unclear. Why does reordering occur in Contextualization through Dialogue? Providing an example would help clarify this.
- LLMs may still have a limited understanding of proverbs. How reliable is LLM-as-a-judge in this setting?
Overall assessment scores: 3.5

3
Metareview score: 3
Metareview: The authors expand the existing multicultural proverbs dataset, MAPS, to create an English-centric proverb translation dataset across four language pairs: German, Bengali, Indonesian, and Mandarin Chinese.
They also extend the dataset to include proverb usage extracted from movie subtitles.
They investigate the capabilities of Neural Machine Translation models and Large Language Models for translating proverbs.
Summary of reasons to publish: The proverb translation dataset is a valuable resource for examining the impact of context and cultural differences in machine translation.
The authors performed extensive experiments and revealed notable findings, such as the observation that LLMs generally surpass NMT models in translating proverbs, particularly between languages with comparable cultural contexts.