That was the last straw, we need more: Are Translation Systems Sensitive to Disambiguating Context?
Authors: Jaechan Lee
Alisa Liu
Orevaoghene Ahia
Hila Gonen
Noah A. Smith
Venue: EMNLP 2023 Findings (2023)
Keywords: model evaluation
model analysis
dataset creation
machine translation
LM
TL;DR: We evaluate how MT systems handle ambiguous input and how sensitive they are to disambiguating context, by collecting a new dataset based on idioms with both literal & figurative meanings. We find that LMs are much more context-aware than NMT models.

Abstract: The translation of ambiguous text presents a challenge for translation systems, as it requires using the surrounding context to disambiguate the intended meaning as much as possible. While prior work has studied ambiguities that result from different grammatical features of the source and target language, we study semantic ambiguities that exist in the source (English in this work) itself. In particular, we focus on idioms that are open to both literal and figurative interpretations (e.g., goose egg), and collect TIDE, a dataset of 512 pairs of English sentences containing idioms with disambiguating context such that one is literal (it laid a goose egg) and another is figurative (they scored a goose egg, as in a score of zero). In experiments, we compare MT-specific models and language models for (i) their preference when given an ambiguous subsentence, (ii) their sensitivity to disambiguating context, and (iii) the performance disparity between figurative and literal source sentences. We find that current MT models consistently translate English idioms literally, even when the context suggests a figurative interpretation. On the other hand, LMs are far more context-aware, although there remain disparities across target languages. Our findings underline the potential of LMs as a strong backbone for context-aware translation.

That was the last straw, we need more:
Are Translation Systems Sensitive to Disambiguating Context?
Jaechan Lee' Alisa Liu
Orevaoghene Ahia
Hila Gonen V Noah A. Smith v
Paul G. Allen School of Computer Science & Engineering, University of Washington
Allen Institute for AI {chan0369, alisaliu, oahia, thess. washington. edu, hi hilagnn@gmail.com
Abstract
The translation ambiguous text presents challenge for translation systems, as it requires using the surrounding context to disambiguate the intended meaning as much as possible. While prior work has studied ambiguities that result from different grammatical features of the source and target language, languaa, we study seman- tic umbiguities that exist in the source (English this work) itself. In particular, we focus on idioms that open both literal and figura- tive interpretations (c.g. goose egg), and collect TIDE,' dataset of 512 pairs of English sen- tences containing idioms with disambiguating context such that one is literal (it laid goose egg) and another figurative (they scored goose egg, as in score of zero). In experi- ments, we compare MT-specific models and language models for (i) their preference when given an ambiguous subsentence, (ii) their sen- sitivity to disambiguating context, and (iii) performance disparity between figurative and literal source sentences. We find that current MT models consistently translate English id- ioms literally, even when the context suggests a figurative interpretation. On the other hand, LMS are far context-aware, although there remain disparities across target languages. Our findings underline the potential of LMs strong backbone for context-aware translation.
Egarstins
can't focus this minor have bigger fryat work
can't cook these small ones now, have bigger for dinrer
No puedo concentrarme tengo peces más grandes para freir en el trabajo
No puedo concinar estos pequeños tenge pecas más grandes para frelr para cena
NLLB
No puedo concentrarme problema menor, tango cosas más importantes que resolver trabajo
cocinar pequeños ahora, tengo peces más grandes que freir para
ChatGPT
TIDE consists of pairs of contrastive sentences that contain the same idiomatic expression in different contexts, such that one uses the figurative meaning of the idiom (left), and another uses its literal meaning (right). On this set of inputs, ChatGPT is sensitive to the disambiguating context when translating the idiom, while NLLB is not.
Figure 1:
In machine translation (MT), it has long been recognized that ambiguities arise when the source language does not encode grammatical attributes that the target language requires (Bar-Hillel, 1953; Prates et al., 2019; Savoldi et al., 2021; Gonen and Webster, 2020, i.a.). For instance, the English sentence "I
doctor" would require disam- biguating the doctor's gender for translation Ger- man, which has no gender-neutral word for "doc- tor." Prior work created contrastive test sets for such phenomena, to evaluate whether MT mod- els correctly translate an ambiguous word (here, "doctor") when disambiguating context is available (e.g., "She is a doctor") (Müller et al., 2018; Baw- den et al., 2018; Voita et al., 2019b).
1 Introduction
Natural language is inherently ambiguous due to the competing pressures of efficiency and clarity in communication (Zipf. 1949; Piantadosi al., 2012). As communicators, we disambiguate mean- ings on the basis of a wide range of contextual fac- tors, or ask clarifying questions when such context is not available. Though sometimes overlooked, the role of ambiguity in NLP has gained growing interest in recent work (Min et al., 2020; Liu et al., 2023; Stengel-Eskii et al., 2023).
In contrast with grammatical ambiguity with re- spect to a target language, it is relatively less under- stood how MT systems handle semantic ambiguity present in the source text itself. For instance. "I have bigger fish to fry" " ambiguous between fig- urative ("... at work") and literal ("... for the dinner)) interpretations English, outside of the
Data and code can he found at https: http:::///itiie httg/gitthbb.cco com/ jaechan-repo repo/mt--oooeeeeeeeee weepo//ttambiieeeee bubig.

---

context of translation. Therefore, we extend the line of work on context-aware translation seman- tically ambiguous phrases English.
2 Creating TIDE
Idioms, though commonplace in daily communi- cation, pose a challenge for MT systems due its inherent ambiguity between literal and non-literal meanings. Generating the most appropriate transla- tion among potential disambiguations of the idiom involves an understanding that extends beyond the idiom itself, as an MT system must use broader context clues to discern the most fitting translation.
To this end, we create TIDE, Translations of Idioms in Disambiguating context in English, a dataset of 512 example triples. Each triple con- sists of an ambiguous subsentence and pair of contrastive sentences that contain the subsentence but add disambiguating context: one to produce figurative interpretation of the idiom, and another produce a literal interpretation of (see Fig- ure I for an example). Our creation process for the triples combines automatic text generation with hu- man annotation: we use GPT-4 to draft the triples, which are then scrutinized by human annotators. Following this, we engage native speakers from four languages to craft reference translations for subset of the dataset.
We present TIDE, a dataset of 512 example triples. Each triple consists of an ambiguous sub- sentence, a figurative sentence, and a literal sen- tence in English, all including the same idiom. The ambiguous subsentence permits both figurative and literal interpretations of the idiom, while the figura- tive and literal sentences introduce additional con- text that resolves the ambiguity to figurative and literal readings, respectively. We design subsen- lences (e.g.. "had a card up his sleeve ") to be more than an idiom itself (here, "card up sleeve") "), as idioms alone can often be unnatural standalone input an MT system.
In experiments, we evaluate both traditional neural MT models and language models (LMs). MT-specific models are trained on large corpora of parallel sentences, and have formed the foundation of translation research; LMs are trained without any explicit supervision for translation, yet recently demonstrate impressive translation ability (Hendy et al., 2023). Using TIDE, compare how these two types of systems handle ambiguity, and evalu- ate their sensitivity to disambiguating context. We find that on ambiguous input, LMs demonstrate roughly balanced preference between literal and fig- urative interpretations, whereas MT-specific mod- els consistently prefer literal ones (54.1). Given dis- ambiguating context, LMs are substantially more context-aware, though this sensitivity declines for more low-resource target languages; in contrast, MT-specific models tend to translate idioms liter- ally irrespective of context ($4.2). Finally, MT- specific models are better at translation of literal text than figurative text, whereas this disparity in LMs is much narrower (64.3).
We construct TIDE through a human- AI collab- orative approach following a line of recent work (Liu et al., 2022; Chakrabarty et al., 2022). We first manually select candidate idioms from large idiom corpora ($2.1). Next, we leverage the generative power of GPT-4 to efficiently produce diverse and high-quality text, by prompting it to write a complete triple for each idiom ($2.2). To ensure quality and correctness, we then involve hu- man annotators to filter out invalid triples ($2.3). Finally, we collect gold translations for subset of the dataset among native speakers ($2.4).
2.1 Collection of Idioms
To collect idioms, we scrape THE IDIOMS dic- tionary² to obtain 1409 idioms, and additionally use dataset of 905 idioms from Rabinovich et al. (2020); both sources contain corresponding idiom definitions. We discard duplicate idioms (includ- ing those that appear in different conjugations) and proverbs (e.g., All that glitters is not gold), which are often too self-contained to be disambiguated with context. Then, we manually select idioms that are available to a natural and plausible literal inter- pretation, in addition to their figurative meanings. This results in set of 700 idioms with definitions.
We summarize our contributions as follows: (1) We formalize the challenge of ambiguous idiomatic language in MT; (2) we create a new translation benchmark, TIDE, that includes sentences with id- ioms along with disanbiguating contexts (literal and figurative); (3) we analyze MT systems' be- havior with and without disambiguating contexts, pointing to interesting trends and differences be- tween LMs and MT-specific models.
Phttps:// theidioms.com/

---

Idiom
Figurative Sentence
Literal Sentence
tip of the iceberg
The problems we discovered were just the tip of the iceberg in this company.
As we approached the glacier, we saw just the tip of the iceberg above the water.
to only know a very small part of the problem
fall between the cracks be ignored or wnobserved
His request for promotion fell between the cracks due to the company's restructuring.
The small toy fell between the cracks of the wooden floor.
foam at the mouth be extremely angry
He was foaming at the mouth when he found out about the betrayal.
The rabid dog was foaming at the mouth and needed be isolated.
foot in the door succeed with a first step
By voluntecring at the company, she got foot
When the door was closing, he quickly got a foot in the door to prevent it from shutting.
in the door for full-time position.
Table I:
Examples in TIDE A figurative and literal sentence disambiguates the idiom by adding context that demands figurative and literal interpretations, respectively.
2.2 Generation of Idioms in Context
where the intended intended-figurative sentences received votes for figurative, the intended-literal sen- tences received at least one vote not for literal. This asymmetry the filtering heuristic because we observe that GPT-4 was more reliable at gener- ating figurative uses idioms than literal ones, and therefore we enforce lower bar for retaining figu- rative sentences. We also discard all the triples that contain at least one vote for discard. In this way, obtain the 512 English triples which constitute TIDE.
Next, we draft an example triple for each idiom by prompting GPT-4 with a fixed prompt, contain- ing two in-context examples along with additional guidelines (details in Appendix A). We write a set of heuristics to automatically identify some types of ill-formed output, such when the subsentence is not exact substring of the full sentences. When rule violated, we add an additional turn of dia- logue instructing the model to revise its output to follow the broken rule. We repeat this until all rules are followed, or when two revisions are attempted without success. After this, we have 700 English triples, each associated with a unique idiom.
2.4 Collecting Translations
Finally, for randomly subset of 50 idioms, we gather reference translations for the contrastive pairs of figurative and literal sentences from native speakers of Hebrew, Yoruba, Korean, and Chinese.
2.3 Human Annotation
Of course, the triples collected in $2.2 may not cor- rectly use idioms literally and figuratively, and gen- erated text is susceptible to fluency and coherence issues. To ensure data quality, we recruit crowd- workers on Amazon Mechanical Turk to label each of the full sentences as using either the literal or the figurative sense of an idiom. We present each full sentence independently (not as a pair) to two different crowdworkers. who are asked to label it as figurative, literal, or ambiguous with respect to how it uses the given idiom. They may also indi- cate that the sentence is invalid if it is offensive or has fluency issues (see Appendix B for details).
3 Experimental Setup
In this section we outline the models ($3.1) and lan- guages ($3.2) we evaluate, our automatic metrics ($3.3), and our setup for collecting human evalua- tions of generated translations ($3.4).
3.1 Models
We evaluate two classes of translation systems: MT- specific models and LMs. Here, the MT-specific models use encoder-decoded architecture and are trained on large amounts of parallel data, whereas the LMs are decoder-only models trained max- imize likelihood (i.e., next-token prediction) on predominantly-Fngee text.
The annotators achieved substantial agreement on this task, with Fleiss к score of 0.721. Fur- thermore, for 82.9% of examples, there is com- plete agreement between both annotators and the intended label (the label which we ask GPT-4 to follow when generating triples).
MT-Specific Models We evaluate NLLB (Meta, 2022) and Opus MT (Tiedemann and Thottingal, 2020; Tiedemann, 2020). NLLB is trained on par-
Based on the annotations, we discard triples

---

tially synthetic parallel data, and covers 202 lan- guages." Opus MT collection of models, each with fixed source and target language.* 4 For both models, we decode the translation greedily.
3.3 Automatic Metries
We use different sets of metrics to evaluate transla- tions for their literalness and for the overall transla- tion quality.
Language Models We evaluate ChatGPT (gpt-3.5-turbe) OpenAl 2022) and PALM (text-bison-000 Google et al. 2023). We do not include GPT-4 as it partially authored the examples in the dataset.
Literalness Following Hendy et al. (2023), we use two metrics to assess the literalness of the trans- lation: (I) Unaligned Source Words (USW) repre- sents the number of source words unaligned with words in the translation, and (2) Non-Monotoyicityy (NM; Schioppa et al., 2021) determines the tent of reordering in the word-to-word alignments from the source sentence to its translation. For both metrics, we use the bitext alignments from the awesome-all ign framework (Dou and Neubig, 2021) which extract word alignments from mBERT embeddings.
Both models were trained on a mixture of differ- ent languages, and in particular PaLM 2's training corpus included parallel data for hundreds of lan- guages. However, both LMs trained for the next-token-prediction objective.
We prompt the LM to generate translations zero- shot with the prompt "Translate the following English sentence to [target language] [source sentencel," and greedily decode the continuation. We do not provide in-context amples or further instructions about figurative lan- guage, in order to create a setting comparable to the evaluation of MT-specific models.
Translation quality We evaluate translation quality based on sentence similarity between ref- erence and predicted translations. We use chrF (Popović, 2015),, BERTScore (Sun et al., 2022), and BLEURT (Sellam et al., 2020). chrF measures pre- cision, recall, and F-score of character n-grams. BERTScore a contextual embedding- basedd evalu- ation metric that leverages the pretrained language model. BLEURT is a learned regression metric for automatic evaluation of generated text, which uti- lizes BERT for training on pairwise comparisons of reference and candidate sentences, calibrated on human quality judgments.
Google Translate We also include Google Trans- late" for reference due to its popularity in com- mercial use. We do not classify it either an MT-specific model or LM due the lack of public understanding of how it works.
3.2 Languages
We consider the eight target languages: Spanish (Es), Hindi (Hi),, German (De), Hungarian (Hu), Ko- rean (Ko), Chinese (Zh), Hebrew (He), and Yoruba (Yo), which vary in resource- aviilabiiiyy and are typologically and culturally diverse. When the evaluation requires a gold translation, we focus on the last four languages for which TIDE contains human-writter references.
3.4 Human Evaluation
Due to the documented limitations of automatic evaluation for translation (Kasai et al., 2022), we additionally perform human evaluation of model- generated translations for Chinese, Korean, He- brew, and Yoruba. We recruit one native speaker for each language, who are presented with the source sentences in each triple, along with generated trans- lations from NLLB, Opus MT, ChatGPT, and PaLM 2. The model -ener-eeee translations are presented a random order not shown to the annotator. For each sentence, they are asked: (1) Does the translation use the figurative meaning of the idiom, the literal meaning of the idiom, preserve the ambiguity due to equivalent idiom in their language, or is it too nonsensical to determine? (2) Overall, is the trans- lation perfectly correct, containing slight errors, containing major errors? We use the same subset
"nttps://huggiiggaae http://innnnacecacaaeeeeeeeeeeee to/faacebook/nlb--2200
"The most recent model for each language pair was downloaded from https://gitthen com/HelsinkiiNUPP laaeehhaalenggeereeeeeeeeee /models:: transformer- big for De, Es, and Hu, transformer-align for He, Hi, and Yo. Their most recent English to Chinese models by July 2023 do not produce coherent outputs, so we proceed with the carlier version available on HuggingFace: https:/ httgg/hgggigg face aace. co/Hels Enki-Ni sinki-iiii Mirii-NPP/On P/opus potuus-t--aa -mt-en-ze -n--en zh. English to Korean mod- els are not evaluated due to an issue with their Py Torch imple- mentation, reported by multiple users.
API last accessed on June 18, 2023. API last accessed on June 17, 2023. 'https: h/ttraaslaateeoaaat/o on June 14, 2023.
API last accessed
We use Ma--RoBRRTa-bas emheddings for BERTScore.
(Conneau et al., 2020)

---

18
Non-literaln hnan-litralnnesss (USW)
1/4
1.2
1.0
0.8
0.1
en-de
en-zh
efi-ets:
en-ko
en-hu
en-hi
en-he
en-yyo
Average
ChatGPT
PaLM
Opus M
NLLB
Google Translate
Figure 2: Non-Literalness of Translations of Ambiguous Subsentences, as measured by the number of unaligned source words (USW) between the source sentence and its translation, normalized by the within-language average. Translations from pretrained LMS are less literal than those of MT-specific models, suggesting that LMs prefer less literal tramslations of ambiguous input (i.c., without disambiguating context). En Ko Opus MT models are not evaluated due an issue with their implementation.
Figurative
Ambigusus
4.1 RQ1: How do MT systems translate ambiguous subsentences?
Proportion of Examples %
Nonsense
First, we investigate how MT systems behave on ambiguous subsentences without disambiguating context, in order to measure their preference for translating them figuratively or literally. We hy- pothesize that LMs more likely produce less literal translations of ambiguous subsentences than MT-specific systems, based recent findings in Raunak al. (2023). Unlike their setting, here the source sentences are always ambiguous, so both literal and figurative translations are correct.
25
X 2h to he Opus MT
to yo ChatGPT
2h No ho yo PaLM 2
2h Ko Th NLLB
Figure 3: Human Evaluation of Translations of Ambigu- ous Subsentences. where annotators are asked to evaluate whether each translation is figurative, literal, ambiguous due to an equivalent idiom, or is nonsensical. ChatGPT and PALM 2 are more balanced in their preference between figurative and literal translations; Opus and NLLB overwhelmingly profer literal translations.
Automatic Evaluation We measure the literal- ness of translations using USW and NM, where higher values mean less literal translations ($3.3). Within each language, we normalize values by the average across systems in that language. This is because the metrics are not comparable across target lan- guages, as they depend on linguistic properties each target language. Shown in Figure 2, LMs (in blue) produce translations with higher USW scores than MT-specific models (in orange), across all tar- get languages. In particular, Opus MT is the most literal model across all target languages. More- over, we observe that the differences between LMs and MT-specific models become less pronounced for more reseurees languages (the languages are ordered left to right based on count of pages Common Craw19).
50 triples from $2.4. With sentences per triple and 4 source models for each triple, annotators each evaluate 600 translations.
4 Experimental Results
In experiments, explore MT-specific and LM systems' translation behavior on ambiguous subsentences ($4.1), their sensitivity disam- biguating context ($4.2), and their overall com- petence at translating literal versus figurative input ($4.3).
hhttps://comonnrraee http://ommmnnrawlliiithhbub github. withub.iol. io/ cc- se-ccaal-sttiiie hrrraal-ssaatssticsplloosss/saae

---

Sensitivity
en-de
en-zh
en-e5
en-ko
en-hu
en-hi
en-he
en-yo
Averago
ChatGPT
PaLM
Opus MT
NLLB
Google Translate
Figure 4: Sensitivity to Disambiguating Context, as measured by BERTScore-P, describes how well an MT model adapts disambiguating context an otherwise ambiguous subsentence. The metric is based on how the translation of the ambiguous subsentence changes between the two full sentences, and is normalized by the in-language mean. LMs generally demonstrate greater awaeeess than MT-specific models.
Sensitive
Insensitive (Equiv. Idiom)
the target language Chinese, ChatGPT prefers fig- urative translation 62% of the time; however, that preference declines dramatically the target lan- guage becomes more low-resource, dropping 6% for Yoruba. PaLM 2 demonstrates more robust preferences across target languages, consistently preferring figurative translations 28% to 46% of the time. In contrast, Opus MT and NLLB overwhelm- ingly prefer literal translations, choosing figura- tive translation only 4% to 20% of the time.
Incensitive (Figurative) Insensitive (Literal)
Proportion of Examples de
50
X Opus MT
he ChatGPT
PaLM
NLLB
4.2
RQ2: How sensitive are MT systems disambiguating context?
Figure 5: Human Evaluation of Sensitivity. A set of gener- ated translations considered context-sensitive when uses the figurative literal) of idiom given figurative literal) disamhiguating context. ChatGPT and PaLM much context-sensitive than Opus MT and NLLB, which tend translate idioms literally irrespective of context.
We next explore to what extent the predicted trans- lation of an ambiguous subsentence changes when disambiguating context is available.
Automatic Evaluation Intuitively, if the LM is not sensitive to context, then the translation of the ambiguous subsentence, Pa, should be equally con- tained in the translation pe for the literal sentence, and the translation Py for the figurative sentence. That is, the way the ambiguous subsentence is translated should not be affected by the added con- text. On the other hand, if Pa is more contained Ps than in Py (or vice versa), that would mean how the model handles a changes with the context. Therefore, we operationalize the sensitivity to disambiguating context
Results based on NM (shown in Appendix C) cor- roborate our findings for SVO languages. This met- ric is inherently limited to target languages with the same word order as the source language (English in this work, with SVO order).
Human Evaluation In Figure 3, we show the human judgments of translations of ambiguous subsentences, indicating whether the translation is ambiguous, literal, figurative nonsense. These results corroborate findings from automatic evalu- ation, and show even clearer distinctions. Overall, ChatGPT and PaLM demonstrate much more bal- anced preferences between figurative and literal translations, compared to Opus MT and NLLB. For
[contained_in(p_.. pi) contained_in(22. P/)
where contained_in() is a measure of unidirec- tional sentence similarity. Here, we use chrP and

---

Literal Translation
Figurative Translation
1.2
1.1
Correctness
1.0
0.9
11.8
en-zh
en-ko
en-he
en-yo
en-zh
en-ko
en-he
ChatGPT
PaLM
Opus MT
NLLB
Google Translate
Figure 6: Overall Translation Quality for Literal (left) and Figurative (right) Sentences, measured by BLEURT between the reference and prediction. While LMS and MT-specific models show comparable performance in translating literal sentences, NMT models much weaker figurative source sentences.
BERTScore-P, the precision outputs of chrF and BERTScore, both ranging from O to 1. A higher value of sensitivity (close to 1) indicates high sen- sitivity disambiguating contexts.
training for translation leads to better results on low-resource languages.
4.3
RQ3: Are there performance disparities between figurative and literal translations?
Figure shows the sensitivity results for the dif- ferent models. The LMs, PaLM and ChatGPT, generally exhibit a higher degree of sensitivity across most language pairs. Comparatively, the MT-specific models, Opus MT and NLLB, show less sensitivity. Opus MT, in particular, consistently demonstrates the lowest context sensitivity for all target languages.
Finally, we investigate if translation systems have systematic performance gaps between translating figurative versus literal input.
Automatic Evaluation We the reference translations collected in $2.4, and measure text sim- ilarity between predicted and reference translation with BLEURT.
Human Evaluation In human evaluation, model considered context-sensitive on triple if annotators indicate that the idiom is translated figuratively for the figurative sentence, and literally for the literal sentence. Otherwise, the model is in- sensitive. As shown in Figure 5, both ChatGPT and PALM 2 are very sensitive to context, though there is still room for improvement. For instance, for En- translation, ChatGPT and PaLM translate correctly for 76% and 72% of idioms, respectively. Yet, the sensitivity of both models declines mono- tonically as the target language becomes more low-resource. In particular, for En +Yo translation, ChatGPT translations are entirely nonsensical, and are qualitatively reported as frequently containing hallucinations completely unrelated to the source. Nonetheless, Opus MT and NLLB substantially less context-aware, correctly adapting to disam- biguating context only 11.5% and 34.5% of the time, respectively. Yet, their more consistent per- formance across languages suggests that dedicated
The results are shown in Figure Across the board, models are more capable at literal transla- tion than figurative translation. Yet, the gap is more pronounced for MT-specific models compared to LMs. ChatGPT and PaLM exhibit performance gaps of 2.92% and 4.85%, respectively, between lit- eral (higher) and figurative translations, on average across languages. For OPUS and NLLB this disparity is higher: 16.4% and 1 11.7%, respectively.
Overall, MT-specific models and LMs demon- strate comparable performance literal transla- tions, while NMT models lag behind LMs figu- rative translations.
Human Evaluation In Figure 7, we compare how human annotators evaluate the correctness of translations overall, with the options perfect, minor mistakes, and major mistakes. Consistent with find- ings from automatic evaluation, ChatGPT and PaLM demonstrate more consistent performance across

---

Literal Translation
Figurative Translation
Proportion of Examples %
50
25
2h 1o he
zh ko he
zh ko he
zh ko he NLLB Minor Mistakes
2h ka he ChatGPT
2h ka he PaLM 2
zh ko he Opus MT N/A
2h ko he NLLB
ChatGPT
PaLM
Opus MT
Perfect
Major Mistakes
Figure 7: Human Evaluation of Overall Translation Quality, reported separately for figurative versus literal source sentences. Opus and NLLB are substantially better at literal translation than figurative translation overall, whereas ChatGPT and PaLM exhibit much smaller disparity between literal and figurative translation quality.
literal and figurative translations. However, Opus and NLLB are notably stronger at literal translations than figurative ones.
TIDE joins a family contrastive datasets that test model sensitivity to contextual information (Müller et al., 2018; Bawden et al., 2018; Voita et al., 2019b, i.a.).
We additionally observe that Yoruba, the most low-resource language we study, Opus MT and NLLB actually far outperform ChatGPT and PaLM 2. We speculate that pretrained LMs are particularly strong on languages that were well-represented dur- ing pretraining; when this is not the case, it may produce degenerate text by entirely failing to grasp the translation task.
Translation of figurative language Figurative language has received considerable attention in MT research. Some work has studied the hidden repre- sentations or attention patterns of MT--seeific mod- els when processing multi-word expressions (Rik- ters and Bojar, 2017; Garcia et al., 2021; Dankers et al., 2022), or proposed methods to improve trans- lation of these expressions (Zaninello and Birch, 2020; Gamallo and Garcia, 2019). In particular, Baziotis et (2023) show that monolingual pre- training improves figurative translation, which may explain our finding that pretrained LMs generate less literal translations and are more sensitive to disambiguating context.
5 Related Work
Ambiguity translation Context-aware trans- lation usually focuses on grammatical features that the source language does not encode but the tar- get language requires, such as formality (e.g.. Chi- nese has formal and informal "you": Voita et al.. 2019a), gendered pronouns (e.g., French has male and female "ir": Müller et al., 2018; Yin et al., 2021), verb form (e.g., Spanish has six verb forms for past tense; Fernandes et al.. 2023), and ellipses (e.g.. "We all did" in English cannot be translated to Russian without identifying the elided verb; Voita et al., 2019b). Another well-studied issue is lexi- cal cohesion, where the same phrase in the source sentence (c.g., named entity like "Julia") should be translated consistently each time (Wong and Kit, 2012: Kuang et al., 2018). In contrast, our work extends the study of context-aware translation to expressions which are ambiguous in the source lan- guage alone, focusing on idiomatic expressions.
The most closely related work, Raunak al. (2023), compare how LMs and MT-specific sys- tems translate sentences with idiomatic expressions, and similarly find that LMs produce substantially less literal translations. We go further by evaluat- ing how these models handle ambiguous input and their sensitivity to disambiguating context.
Datasets for idiom translation Fadace et al. (2018) introduced the first extensive dataset for idiom translation, identifying data scarcity as one of core challenges this domain. EPIE (Saxena and Paul, 2020) is a large-scale corpus with 25K potentially idiomatic expressions (PIEs), with rep-

---

resentation of both figurative and literal usages. MAGPIE (Haagsma et al., 2020) is a more expan- sive dataset of 50K samples that also contain genre labels. PECTI (Tang, 2022) curated a parallel En- glish translation dataset of Chinese idioms. While these datasets offer a general-purpose testbed, the contrastive sentence pairs in TIDE enable finer- grained analysis, while the fluency of source sen- tences matches (if not exceeding) that of naturally- occurring datasets.
In addition, we only study the behavior of trans- lation systems when English is the source language. due to the availability of English idiom collections. Yet figurative expressions vary greatly across lan- guages (Kabra et al., 2023), and our conclusions may not necessarily generalize to translation from other languages.
References
Yehoshua Bar- Bar-Hillel. 1953. Some linguistic problems connected with machine translation. Philosophy of Science, 20(3):217-225
6 Conclusion
In this work we focus on semantic ambiguity in ma- chine translation, specifically when using idiomatic language. We introduce new benchmark (TiDe) of sentences that include idioms, along with dis- ambiguating contexts (both literal and figurative). We then use TIDE investigate the behavior of dif- ferent translation systems on ambiguous input and their sensitivity to disambiguating context, uncov- ering new strengths of pretrained LMs compared to MT-specifiic models.
Rachel Bawden, Rico Sennrich, Alexandra Birch, and Barry Haddow. 2018. Evaluating discourse phenom- ena in neural machine translation. In Proceedings of the 2018 Conference of the North American Chap- of the Association for Computational Linguistics: Human Language Technologies, Volume / (Long Pa- pers), pages 1304-1313, New Orleans, Louisiana. Association for Computational Linguisties.
Christos Baziotis, Prashant Mathur, and Eva Hasler. 2023. Automatic evaluation and analysis of idioms in neural machine translation. In Proceedings of the 17th Conference of the European Chapter of the As- sociation for Computational Linguistics, pages 3682- 3700, Dubrovnik, Croatia. Association for Computa- tional Linguistics.
Our findings point to pretrained LMs as promis- ing backbone for translation systems, and we fore- see a future that combines the strong language un- derstanding of LMs with dedicated supervision for translation.
Tuhin Chakrabarty, Arkadiy Saakyan, Debanjan Ghosh, and Smaranda Muresan. 2022. FLUTE: Figurative language understanding through textual explanations. In Proceedings of the 2022 Conference Empiri- cal Methods in Natural Language Processing, pages 7139-7159, Abu Dhabi, United Arab Emirates. As- sociation for Computational Linguistics.
Acknowledgments
We would like to thank the UW NLP community for valuable discussion of this work. We are grate- ful Weijia Shi, Jiacheng (Gary) Liu, and Xi- aochuang Han for their help in writing and evalu- ating Chinese translations, and Zhaofeng Wu for feedback on the draft and figures.
Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettle- moyer, and Veselin Stoyanov. 2020. Unsupervised cross- representation learning at scale.
We thank the reviewers for their valuable feed- back and suggestions, and OpenAl for offering access to their models through the API.
Verna Dankers, Christopher Lucas, and Ivan Titov. 2022. Can transformer be too compositional? analysing id- iom processing in neural machine translation. In Proceedings of the 60th Annual Meeting of the As- sociation for Computational Lingnistics (Volume I: Long Papers), pages 3608-3626, Dublin, Ireland. As- sociation for Computational I. Linguistics.
Limitations
this work we study ambiguous source sentences specifically through idioms that are available to both literal and figurative interpretations. While this allows us to efficiently collect a dataset and perform focused evaluation, ambiguity occurs in more diverse forms, and we encourage future work collect more data in the form of TIDE. Con- temporary work collects dataset of ambiguous sentences (with direct disambiguations, rather than disambiguating context), and a promising start (Liu et al., 2023).
Zi-Yi Dou and Graham Neubig. 2021. Word alignment by fine-tuning embeddings on parallel corpora.
Marzich Fadace, Arianna Bisazza, and Christof Monz. 2018. Examining the tip of the iceberg: A data set for idiom translation. In Proceedings of the Fleventh In- ternational Conference on Language Resources and Evaluation (LREC 2018), Miyazaki. Japan. European Language Resources Association (ELRA).

---

Patrick Fernandes, Kayo Yin, Emmy Liu, André Mar- tins, and Graham Neubig. 2023. When does trans- lation require context? a data-driven, multilingual exploration. In Proceedings of the sl Annual Meet- ing of the Association for Computational Linguisties (Volume 1: Long Pupers), pages 606-626, Toronto, Canada. Association for Computational Linguisties.
Wang, Tao Wang, John Wieting, Yuhuai Wu, Kelvin Xu, Yunhan Xu, Linting Xue, Pengcheng Yin. Jiahui Yu, Qiao Zhang, Steven Zheng, Ce Zheng, Weikang Zhou, Denny Zhou, Slav Petrov, and Yonghui Wu. 2023. Palm technical report.
Hessel Haagsma, Johan Bos, and Malvina Nissim. 2020. MAGPIE: A large corpus of potentially idiomatic expressions. In Proceedings of the Twelfth Lan- guage Resources and Evaluation Conference, pages 279-287, Marseille, France. European Language Re- sources Association.
Pablo Gamallo and Marcos Garcia. 2019. Unsuper- vised compositional translation of multiword expres- sions. In Proceedings of the Joint Workshop on Mul- fiword Expressions and WordNet (MWF-WN 2019), pages 40-48, Florence, Italy Association for Com- putational Linguistics.
Amr Hendy, Mohamed Abdelrchim, Amr Sharaf, Vikas Raunak, Mohamed Gabr, Hitokazu Matsushita, Young Jin Kim, Mohamed Atify, and Hany Has- san Awadalla. 2023, How good are gpt models at machine translation? comprehensive evaluation. arXiv.
Marcos Garcia, Tiago Kramer Vieira, Carolina Scarton, Marco Idiart, and Aline Villavicencio. 2021. Probing for idiomaticity in vector space models. In Proceed- ings of the 16th Conference of the European Chap- ler of the Association for Computational Linguisties: Main Volume, pages 3551-3566 Online. Association for Computational Linguistics.
Anubha Kabra, Emmy Liu, Simran Khamuja, Al- ham Fikri Aji, Genta Winata, Samuel Cahyawijaya, Anuoluwapo Aremu, Perez Ogayo, and Graham Neu- big. 2023. Multi-lingual and culturcll figurative language understanding. In Findings of the Asso- ciation for Computational Linguisties: ACL 2023, pages $269-8284, Toronto, Canada. Association for Computational Linguistics.
Hila Gonen and Kellie Webster. 2020. Automatically identifying gender issues in machine translation us- ing perturbations. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 1991-1995, Online. Association for Computational Linguistics.
Jungo Kasai, Keisuke Sakaguchi, Ronan Le Bras, Lavinia Dunagan, Jacob Morrison, Alexander Fabbri, Yejin Choi, and Noah A. Smith. 2022. Bidimensional leaderboards: Generate and evaluate language hand in hand. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- mologies. pages 3540-3551, Scattle, United States. Association for Computational Linguisties.
Rohan Anil Google, Andrew M. Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexandre Pas- sos, Siamak Shakeri, Emanuel Taropa, Paige Bai- ley, Zhifeng Chen, Eric Chu, Jonathan H. Clark, Laurent EI Shafey. Yanping Huang, Kathy Meier- Hellstern, Gaurav Mishra, Erica Moreira, Mark Omernick, Kevin Robinson, Sehastian Ruder, Yi Tay, Kefan Xiao, Yuanzhong Xu, Yujing Zhang, Gus- tavo Hernandez Abrego, Junwhan Ahn, Jacob Austin, Paul Barham, Jan Botha, James Brad- bury. Siddhartha Brahma, Kevin Brooks, Michele Catasta, Yong Cheng. Colin Cherry, Christopher A. Choquette-Choo, Aakanksha Chowdhery, Clément Crepy, Shachi Dave, Mostafa Dehghami, Sunipa Dev. Jacob Devlin, Mark Díaz, Nan Feinberg, Fangxiaoyu Feng, Vlad Fienber, Markus Freitag, Xavier Garcia, Sebastian Gehrmann, Lu- cas Gonzalez, Guy Gur-Ari, Steven Hand, Hadi Hashemi, Le Hou, Joshua Howland, Andrea Hu, Jef- frey Hui, Jeremy Hurwitz, Michael Isard, Abe Itty- cheriah, Matthew Jagielski, Wenhao Jia, Kathleen Kenealy, Maxim Krikun, Sneha Kudugunta, Chang Lan, Katherine Lee, Benjamin Lee, Eric Li, Music Li. Wei Li, YaGuang Li, Jian Li, Hycontack Lim, Hanzhao Lin, Zhongtao Liu, Frederick Liu, Mar- cello Maggioni, Aroma Mahendru, Joshua Maynez, Vedant Misra, Maysam Moussalem, Zachary Nado, John Nham, Eric Ni. Andrew Nystrom, Alicia Par- rish, Marie Pellat, Martin Polacek, Alex Polozov, Reiner Pope. Siyuan Qiao. Emily Reif, Bryan Richter. Parker Riley, Alex Castro Ros, Aurko Roy, Brennan Saeta, Rajkumar Samuel, Renee Shelhy, Ambrose Slone, Daniel Smilkov, David R. So, Daniel Sohn, Simon Tokumine, Dasha Valter, Vijay Vasudevan. Ki- ran Vodrahalli, Xuezhi Wang, Pidong Wang, Zirui
Shaohui Kuang, Deyi Xiong, Weihua Luo, and Guodong Zhou. 2018 Modeling coherence for neural machine translation with dynamic and topic caches. In Pro- ceedings of the 27th International Conference on Computational Linguistics, pages 596 396-606, Santa Fe, New Mexico, USA. Association for Computational Linguistics.
Du, Ethan Dyer, Vlad
Alisa Liu, Swabha Swayamdipta, Noah A. Smith, and Yejin Choi. 2022. WANLI: Worker and Al collabora- tion for natural language inference dataset creation. In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 6826-6847, Abu Dhabi, United Arab Emirates. Association for Com- putational Linguistics.
Alisa Liu, Zhaofeng Wu, Julian Michael, Alane Suhr, Peter West, Alexander Koller, Swabha Swayamdipta. Noah A. Smith, and Yejin Choi. 2023. We're afraid language models aren't 't modeling ambiguity. arXiv.
Meta. 2022. No language left behind: Scaling human- centered machine translation.
Sewon Min, Julian Michael, Hannanch Hajishirzi, and Luke Zettlemoyer. 2020. AmbigQA: Answering am- biguous open-domain questions. In Proceedings of

---

the 2020 Conference on Empirical Methods Nat- ural Language Processing (EMNLP), pages 5783- 5797, Online. Association for Computational Lin- guistics.
Elias Stengel-Esking Jimena Guallar- Blasco, Yi Zhou, and Benjamin Van Durme. 2023. Why did the chicken cross the road? rephrasing and analyzing ambiguous questions in VQA.
Mathias Müller, Annette Rios, Elena Voita, and Rico Sennrich. 2018. A large-scale test set for the evalua- tion of context-aware pronoun translation in neural machine translation. In Proceedings of the Third Conference on Machine Translation: Research Pa- pers, pages 61-72, Brussels, Belgium. Association for Computational Linguistics.
Tianxiang Sun, Junliang He, Xipeng Qiu, and Xuan- jing Huang. 2022. BERTScore is unfair: On social bias in language model-based metrics for text gen- eration. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing. pages 3726-3739, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
OpenAT. 2022. Introducing chatgpt.
Kenan Tang. 2022. Petci: A parallel english translation dataset of chinese idioms.
Steven T. Piantadosi, Harry Tily, and Edward Gibson. 2012. The communicative function of ambiguity in language. Cognition, 122(3))280-291.
Jörg Tiedemann, 2020. The tatoeba translation chal- lenge realistic data sets for low resource and multi- lingual MT. In Proceedings of the Fifth Conference on Machine Translation, pages 1174-1182, Online. Association for Computational Linguisties
Maja Popović. 2015, chrF: character n- n-gram F-score
for automatic MT evaluation. In Proceedings of the Tenth Workshop on Statistical Machine Translation, pages 392-395, Lisbon, Portugal, Association for Computational Linguistics.
Jörg Tiedemann and Santhosh Thottingal. 2020. OPUS- M building open translation services for the world. In Proceedings of the 22nd Annual Conference of the European Association for Machine Translation, pages 479-480, Lisboa, Portugal. European Associa- tion for Machine Translation.
Marcelo O. R. Prates, Pedro H. Avelar, and Luis C. Lamb, 2019. Assessing gender bias machine trans- lation: a case study with google translate. Newral Computing and Applications.
Ella Rabinovich, Hila Gonen, and Suzanne Stevenson. 2020. Pick a fight or bite your tongue: Investigation of gender differences in idiomatic language usage.
Elena Voita, Rico Sennrich, and Ivan Titov. 2019a. Context-aware monolingual repair for neural ma- chine translation. Proceedings of the 2019 Confer- ence on Empirical Methods in Natural Language Pro- cessing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 877-886, Hong Kong, China. Association for Computational Linguistics.
Vikas Raunak, Arul Menezes, Matt Post, and Hany Has- san. 2023. Do GPTs produce less literal translations? In Proceedings of the sl Annual Meeting of the Association for Computational Linguisties (Volume
2: Short Pupers), pages 1041-1050, Toronto, Canada. Association for Computational I Linguistics.
Elena Voita, Rico Sennrich, and Ivan Titov, 2019b. When a good translation is wrong in context: Context- aware machine translation improves on deixis, ellip- sis, and lexical cohesion. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1 1198-1212 1212, Florence, Italy. Asso- ciation for Computational Linguistics.
Matiss Rikters and Ondfej Bojar. 2017. Paying attention to wordi-word expressions in neural machine transla- tion. In Proceedings of Machine Translation Summit XVI: Research Track, pages 66-95, Nagoya Japan.
Beatrice Savoldi, Marco Gaido, Luisa Bentivogli, Mat- teo Negri, and Marco Turchi. 2021. Gender bias in machine translation. Transactions of the Association for Computational Linguistics, 9:84
Billy T. M. Wong and Chunyu Kit. 2012. Extending machine translation evaluation metries with lexical cohesion to document level. In Proceedings of the 2012 Joint Conference Empirical Methods in Natu- ral Language Processing and Computational Natural Language Learning, pages 1060-1068, Jeju Island, Korea. Association for Computational Linguistics.
Prateck Saxena and Soma Paul. 2020. Epie dataset: A corpus for possible idiomatic expressions.
Andrea Schioppa, David Vilar, Artem Sokolov, and Katja Filippova. 2021. Controlling machine transla- tion for multiple attributes with additive interventions. In Proceedings of the 2021 Conference on Empiri- cal Methods in Natural Language Processing, pages 6676-6696 Online and Punta Cana, Dominican Re- public. Association for Computational Linguistics.
Kayo Yin, Patrick Fernandes, Danish Pruthi, Aditi Chaudhary, André T. Martins, and Graham Neu- big, 2021. Do context-aware tramslation models pay the right attention? In Proceedings of the 59th An- nual Meeting of the Association for Computational Linguistics and the 11th International Joint Confer- ence on Natural Language Processing (Volume I: Long Pupers), pages 788-801, Online, Association for Computational Linguistics.
Thibault Sellam, Dipanjan Das, and Ankur P. Parikh. 2020. Bleurt: Learning robust metrics for text gener- ation.

---

Andrea Zaninello and Alexandra Birch. 2020. Multi- word expression aware neural machine translation. In Proceedings of the Twelfth Language Resources and Evaluation Conference, pages 3816-3825, Marseille, France. European Language Resources Association.
the floodwaters rose, leaving him up to the neck in work to clean the mess
[Dictionary Definition] "up to the neck" : when people very involved in something, especially when it's something wrong
George Kingsley Zipf. 1949. Human behavior and the principle of least effort.
Idiomatic
llteral
amblguous
discard
after harsh criticism, his ego was bruised black blue
A TIDE Creation Details
[Dietionary Definition] "black and blue": hurt, either bodily, emotionally or psychologically
A.1 Sentence generation
idiomatic
ambiguous
We use GPT-4 to generate the 700 triples consist- ing of an ambiguous subsentence, figurative sen- tence, and a literal sentence. The configuration parameters were set as follows: max max_ mox__toees=511 2, temperature=0, and top_p=1. The prompt is shown in Table 2.
computer was brains behind project, controlling all the automated processes
[Dictionary Definition] "the brains behind" : used to refer to the person who originally invented a particular scheme, plan or idea
In addition, the generation process undergoes iterative refinements under a set of criteria, dur- ing which we prompt the GPT-4 instance rewrite the entire triple if the generation included any pro- hibited words: "literally", "figuratively". "ambigu- ously", "physically", "metaphorically" and "be- cause".. These words are observed to potentially degrade sentence quality, as they often prompt the GPT-4 to merely provide working definitions of the idioms instead of generating novel context. We also ensure through these refinements that the am- biguous subsentence is indeed substring of the figurative and literal sentences.
idiomatic
ambiguous
Figure 8: Amazon Mechanical Turk (MTurk) Worker Interface, containing 3 example problems.
B Amazon Mechanical Turk (MTurk) details
We employ Amazon Mechanical Turk (MTurk), crowdsourcing marketplace, to collect well-formed triples, composed of idiom and corresponding ambiguous subsentence, figurative sentence, and literal sentence, generated by GPT-4 as described in $2.2.
A.2 Processing pronouns
We select 30 workers based their scores in qualification test human intelligence task (HIT) that we administer. This test, which typically requires less than 30 minutes to complete, consists of 20 handcrafted problems the exact format as the main HIT. Upon completion, workers receive a payment of S7.
As written in the generation prompt (Table 2), we ban -4 from including subjects in the ambigu- ous subsentence
we observe that GPT-4 fre- quently uses personal pronouns which end up dis- ambiguating the whole subsentence (e.g., g., He is chip off the old block is ambiguous due to the pronoun he). Following the generation stage, we conduct additional rule-based modifications to the sentences to facilitate the translation process for MT models. In cases where the ambiguous subsen- tence begins with a lexical verb, and both the literal and figurative sentences include interchangeable subjects preceding the verb, we make alterations that both the same pronoun, which are then incorporated into the shared subsentence. These al- terations include converting "he" to "she", "she" to "he", and "he"/"she" to "they" to have the pronoun shared between the figurative and literal sentence.
In both the qualificattin test and the main task, each problem presents an English utterance derived from a randomly shuffled pool of 700 ambiguous subsentences, 700 figurative sentences, and 700 literal sentences. The problem also provides the corresponding idiom in use and its dictionary (figu- rative) definition. With this information, workers must ascertain whether the idiom was used in a fig- urative. literal, or ambiguous context, if the terance should be discarded. The option discard is included to climinate nonsensical or offensive generations.

---

For the main task, we gather multiple gold labels for each problem to ensure accuracy and credibility. This means that multiple workers are assigned the same problem. We initially release a batch of first 50 problems of the pool, collecting gold labels for each to examine interannotator agreement. Based on our observation that 2 gold labels are sufficient, we proceed to collect only 2 labels for the remain- ing batches. Workers remunerated at rate of $0.30 for every 5 problems completed, an interval expected to take minute.
For how we utilize these labels, see $2.3.
Additional results
See Figure and Figure 10 for additional results RQI and RQ2, respectively.

---

16
1/4
Literalness (NM)
3.0
0.8
0.4
en-ya
Average
ChatGPT
PaLM
Opus MT
NLLB
Google Translate
Figure 9: Non-Literalness of Ambiguous Subsentences. This graph visualizes non-monee monotonicity (NM) between the translation of each language pair, quantifying the level of word order shifts that occur during translation.
12
Sensitivity (chrP)
1.0
0.8
0.6
0.4
en-de
en-hi
en-yo Google Translate
Average
ChatGPT
PaLM
Opus M
NLLB
Figure 10: Sensitivity, ccontained_ __(_p.nnnp py) contained_ in((p., Pt)). computed using chrP (the precision score of chrF) as the implementation of contained in.

---

System: Your task is to write AMBIGUOUS phrase. FIGURATIVE sentence, and LITERAL sentence that use the given IDIOM. Consider the IDIOM "bur the hridge" example. An AMBIGUOUS phrase would allow for both figurative and literal interpretations of the IDIOM. For example. the phrase "burned the bridge with him" can be interpreted figuratively (in context of interpersonal relationships) literally (burning down the physical bridge). A FIGURATIVE sentence must add extra words to your AMBIGUOUS phrase only allow for figurative, metaphorical interpretation. For example, the sentence "She hurned the bridge with him because he publicly criticized her works" only makes sense in the figurative context human relationships. LITERAL sentence must add extra words your AMBIGUOUS phrase only allow for literal, word-by-worl by-word interpretation. For example, the sentence "She burned the bridge with him after crossing it" only makes in literal context of physical bridge that be crossed. Note these warnings:
System Instruction
Unless the idiom already contains subject, ambiguous phrase must not introduce any subjects.
Do not your ambiguous phrase with preposition.
Do not write more than one sentence for cach figurative or literal sentence.
Surround your AMBIGUOUS phrase with delimiters
When you write the LITERAL and FIGURATIVE sentences, do NOT modify spelling of AMBIGUOUS phrase in any way. The AMBIGUOUS phrase must be an exact substring the FIGURATIVE and LITERAL. sentences.
User: break the ice
Assistant: {
In- In-Context Examples
"AMBIGUOUS": "sbroke the ice with our teammates>",
"FIGURATIVE": "we <broke the ice with our teammates> by initiating a conversation",
"LITERAL": "holding steel ice hammers tightly, <broke the ice with teammates>"
(user: felt pins needles
Assistant: {
"AMBIGUOUS"; "<felt pins and needles>",
"FIGURATIVE": "waiting for boss respond, <felt pins and needles> in the room",
"LITERAL": "she <felt pins and needles> on the floor when she spilled sewing supplies"
User: goose egg
Assistant: {
"AMBIGUOUS"; "<got goose egg>",
Added Messages
"FIGURATIVE": "he <scored a goose egg> after studying so diligently for the test", "LITERAL": "she <got goose egg? from her grandfather's backyard"
User: The AMBIGUOUS phrase must be an exact substring of the FIGURATIVE sentence, but yours isn't. Rewrite your AMBIGUOUS phrase, FIGURATIVE sentence, and LITERAL sentence accordingly.
Assistant: {
"AMBIGUOUS": "<got goose egg>",
"FIGURATIVE": "he <got a goose egg? after studying so diligently for the test",
"LITERAL"; "she <got goose egg? from her grandfather's backyard"
Table 2: Prompt for GPT-4 to generate one triple. The instruction is stated once, followed by two complete in-context examples. Finally, only the idiom provided for the last example. In this case, the generation does not meet the requirement that the ambiguous subsentence must be substring of the figurative sentence. We request new triple, to which GPT responds with triple that meets all the requirements.


Reviews:
[{"review_id": "sYPqT0PB98", "reviewer": "Reviewer_eu4q", "summary": "", "questions": "1. Don't you think using GPT to generate the literal and non-literal examples is a limit on your corpus?  Yes, there's a human review after that, but it seems that's just vetting and does not involve writing any example for those cases that GPT has failed to generate good examples.\n\n2. Continuing with the above question, don't you think that gives an advantage to the LLM-based approaches?   A more comprehensive way of corpus creation would have been to compile authentic examples from a large corpus via search rather than using a generative model", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your insightful feedback on our submission! We are pleased that you recognized that TIDE can be a “good benchmark for future studies” and that we study a “fairly under-studied problem.” We hope the following discussion addresses your concerns.\n\n**Novelty - Translation of figurative language is an important topic:** Figurative language is everywhere in real-world language (consider our community’s recent series of Workshops on Processing Figurative Language, which the reviewer mentions), and for broad applicability, strong MT models must correctly handle figurative language by attending to the disambiguating context. Aside from creating a diverse benchmark for future work on this topic, we also draw conclusions about the qualitative differences between large pre-trained LMs versus dedicated MT models for translation. The reviewer's criticism of the novelty of our work based on the argument that our work is too niche (\"better suited for a focused workshop on MT or figurative languages\") is indeed one of the heuristics discouraged by ACL 2023 peer review policies.\n\n**Creating data using GPT (Q.1)**\n1. **It is hard to collect contrastive pairs from a corpus of natural examples:** For a controlled investigation into model sensitivity to context, we need contrastive pairs with shared subsentences to compare translations before and after disambiguation. Our initial approach was to scrape figurative sentences from BookCorpus and theidioms.com, and then minimally revise them to create a literal counterpart. However, sentences from BookCorpus were often extracted from long dialogues/passages, scarcely understandable on their own. For sentences from theidioms.com, we found that minimal revisions were often insufficient for making the sentences literal, requiring the entire sentence to be changed. Thus we opted for the automatic generation of contrastive pairs.\n2. **We tried crowdworker revision to improve model generations, but it was not helpful:** Indeed, we only removed incoherent LM generations instead of attempting to revise them. In fact we experimented with crowdworker revision, but found that it often failed to fix incoherence, mainly when the underlying issue was that the subsentence was not actually ambiguous. For instance, revising a phrase like “Playing soccer was a piece of cake” (erroneously marked as ambiguous by GPT-4) would require complete rewriting, as the idiom is already used figuratively without marked figurative context. We decided to remove it when complete rewriting is needed. We acknowledge that our corpus may exclude idioms that GPT-4 couldn’t handle, but we believe the decision maintains our dataset’s quality without affecting our findings.\n3. **Human data creation from scratch suffers limited diversity:** Many studies have shown that crowdworkers employ heuristics to write examples efficiently, and resulting datasets suffer a lack of diversity ([Geva et al., 2019](https://aclanthology.org/D19-1107/); [Gururangan et al., 2018](https://aclanthology.org/N18-2017/)). In our case, crowdworkers may write short, prototypical uses of idiom, rather than longer and more natural-sounding sentences like the ones we observe in our dataset. Our work follows a growing body of work that points to the promise of using controlled text generation alongside human review for dataset creation ([Yuan et al., 2021](https://openreview.net/forum?id=Fkpr2RYDvI1); [Liu et al., 2022](https://aclanthology.org/2022.findings-emnlp.508/); [Bartolo et al., 2022](https://aclanthology.org/2022.naacl-main.275/); [Perez et al., 2022](https://aclanthology.org/2023.findings-acl.847/)).\n\n**Q.2 - Discussion of an unfair advantage to GPT-3.5:** We took care to use a different GPT model for generating dataset examples (GPT-4) than in our experiments (GPT-3.5), for which we also use PaLM2. We chose a more robust model for data generation than for evaluation, as we expect there are idioms that GPT-4 can handle, which GPT-3.5 cannot. (This would be consistent with other findings ([Liu et al., 2023](https://arxiv.org/abs/2304.14399)) that GPT-4 is substantially stronger than GPT-3.5 in dealing with ambiguity.) Additionally, translation is not part of the data creation pipeline, while it is the objective for evaluation.\n\n**Replicability of the use of GPT:** GPT models are indeed black box models. But for reasons outlined above, we follow a long line of work in using GPT-3/4 for data generation, and include GPT-3.5 in the set of models we evaluate because they are currently indispensable for characterizing the capacity of LMs for translation. Despite the challenges, we make the evaluation code and dataset publicly available to enhance the replicability of our study. We believe the value of the TIDE dataset for the research community is great.\n\nYour concerns have helped us identify areas for further clarification, and we appreciate your thoughtful critique!", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. MT for figurative text and idioms is fairly under-studied problem and this work is a step to close the gap.\n\n2. The TIDES corpus can be a good benchmark for future studies on this topic.", "reasons_to_reject": "1. The novelty of the work is quite limited.  It's suited for a focused workshop on MT or figurative languages rather than the main track of the tier-1 EMNLP.\n\n2. The fact that an LLM has been used to create the examples of the TIDES corpus is a major limit and can even create a circular limit to the approach.  That limits the scope of examples to what GPT model is capable of generating and to some extent gives a leverage to the GPT-based solution.\n\n3. The main approach is not explained well (some questions to follow)\n\n4. Weak replicability: Due to the black-box nature of the study with the evolving GPT models, it is difficult to replicate these experiments.", "questions_to_authors": "1. Don't you think using GPT to generate the literal and non-literal examples is a limit on your corpus?  Yes, there's a human review after that, but it seems that's just vetting and does not involve writing any example for those cases that GPT has failed to generate good examples.\n\n2. Continuing with the above question, don't you think that gives an advantage to the LLM-based approaches?   A more comprehensive way of corpus creation would have been to compile authentic examples from a large corpus via search rather than using a generative model", "paper_topic_and_main_contributions": "Summary: The paper is a comparison between LLM-based translation vs. the state of the art MT (NMT in this case) in handling translation of phrases that have both literal and figurative meanings.  The work includes semi-manual creation of a corpus of triples of ambiguous phrase along with its literal and figurative (idiom) usage.  Furthermore, authors conduct run the ambigious text through two LLM-based systems (openAI and PaLM) and also two NMT systems and assess the translation quality both automatically and through human judgement.  Result show that LM-based translation outperforms the NMT models, specially around figurative examples.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "IujqDIx3kw", "reviewer": "Reviewer_PHaX", "summary": "", "questions": "A) What is the baseline quality of the used translation systems in easier sentences? I am thinking if English-Yoruba generates non-sensible tranlsations altogether, evaluating on the harder task of idiom translation might not be very effective use of resources.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your insightful feedback on our submission! We are pleased that you recognize “our new approach for evaluation” and that handling figurative language is “important for users of translation systems.”\n\n**Translation of figurative language has universal implications:** Figurative language is everywhere in real-world language use, and for broad application, strong MT models must correctly handle figurative language by attending to the disambiguating context. Aside from creating a diverse benchmark for future work on this topic, we also draw conclusions about the qualitative differences between large pre-trained LMs versus dedicated MT models for translations. The reviewer’s criticism that the work is too niche (“very specific topic and potentially premature for the smaller languages”) is indeed one of the heuristics discouraged by ACL 2023 peer review policies ( https://2023.aclweb.org/blog/review-acl23/#2-check-for-lazy-thinking ).\n\n**Q.A - Translations are generally sensible:** As shown in Figure 7, all models demonstrate commendable translation performance. The only exceptions are ChatGPT and PaLM 2 when handling Yoruba; however, explicitly supervised models such as Opus MT and NLLB exhibit satisfactory performance on translations of Yoruba as well. A native speaker who reviewed their Yoruba translations found them generally reasonable, particularly for the literal sentences.\n\n**Ethical Concerns - We did not use MTurk for data annotation in languages other than English:** Language experts for translation tasks were native speakers who volunteered to do the task (researchers in NLP). Only GPT-4 generation filtering (a fully English task) was completed by English-speaking MTurk annotators from the United States.\n\n**Reproducibility:** We make the evaluation code and dataset publicly available to enhance the replicability of our study. \n\nYour concerns have helped us identify areas for further clarification, and we appreciate your thoughtful critique!", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "* new testing approach for evaluation of idiom translations is interesting for most conference-goers in this venue", "reasons_to_reject": "* very specific topic and potentially premature for the smaller languages", "questions_to_authors": "A) What is the baseline quality of the used translation systems in easier sentences? I am thinking if English-Yoruba generates non-sensible tranlsations altogether, evaluating on the harder task of idiom translation might not be very effective use of resources.", "paper_topic_and_main_contributions": "Article describes testing of neural machine translations of idiomatic phrases in both non-idiomatic and idiomatic contexts. This is an interesting experiment and something that probably is both important for users of translation systems as well as problems for many human translators as well. The idioms and their translations are evaluated on mt-specific and generic large language models. The idiom translations are evaluated based on being literal, figurative or preserving ambiguity across languages.", "excitement": 4, "reproducibility": 3, "ethical_concerns": "Yes", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Justification_for_Ethical_Concerns": "As far as I've understood use of amazon mturk especially with annotators of less-resourced language experts should be justified. There is no indication of this in the paper and no section."}, "extra_text": {}}, {"review_id": "vk19LqK2VV", "reviewer": "Reviewer_ZKwx", "summary": "", "questions": "A. Did you try to constrain GPT-4 to generate contrastive figurative/literal pairs that were minimally different from each other? While the current pairs may prove useful, a more granular analysis of the impact of context might be challenging due to their diversity.", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your insightful feedback on our submission! We are pleased that you recognize the soundness of our evaluation and “interesting” results that “showcase the strengths and weaknesses of each model category.”\n\n**Previous work - Need for a new dataset:** For a controlled investigation into model sensitivity to context, we need contrastive pairs with shared subsentences to compare translations before and after disambiguation. Many existing datasets based on naturally occurring examples, including MAGPIE, do not meet this constraint. Thanks to the reviewer for the references; we will include all of these in the next revision.\n\n**Q.A. Constrain GPT-4 to generate contrastive figurative/literal pairs that were minimally different from each other:** The reviewer’s suggestion to constrain GPT-4 with minimal pairs provides an interesting angle for creating this dataset. From a preliminary investigation following the reviewer’s suggestion, however, we found that efforts to collect minimal pairs led to the model generating subpar contexts (i.e., less contrastive than without the minimality constraint). Thus, it is not feasible to obtain minimally different pairs for most idioms (getting pairs is not always possible even without the minimality constraint). Such pairs of sentences also sound more unnatural and adversarial, rather than reflecting more natural uses of idioms. Moreover, lacking sensitivity even with non-minimal pairs of contexts is a more meaningful finding. This means that despite the extra surface-level differences (that minimal pairs lack) like length, word choices, or syntax, the model still fails to disambiguate the subsentence correctly.\n\n**Reproducibility:** We make the evaluation code and dataset publicly available to enhance the replicability of our study. \n\nThank you for suggesting **presentation improvements** and pointing out the **missing references**, all of which we find constructive and immediately actionable. We will make sure to add them to the final version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The paper is well-written and introduces a valuable resource for studying how figurative language is translated by machine translation systems and prompted LLMs.\n- The experimental part is framed very clearly and involves both an automatic evaluation using appropriate metrics and a human evaluation covering a sizable data sample (when considering all languages and model combinations).\n- Evaluation results are interesting and clearly showcase the strengths and weaknesses of each model category, covering a comprehensive set of state-of-the-art models and typologically-diverse target languages.", "reasons_to_reject": "The main weakness of this work concerns the novelty of the collected data. Indeed, some ad-hoc test sets to evaluate idiom translation already exist (see \"Missing References\" section), although they were not mentioned in the Related Work section. It is unclear whether TIDE would prove more useful than the much larger MAGPIE dataset for studying figurative/literal translations with MT and LMs. For this reason, it would be important to discuss clearly which components make TIDE a relevant complement to existing resources.", "questions_to_authors": "A. Did you try to constrain GPT-4 to generate contrastive figurative/literal pairs that were minimally different from each other? While the current pairs may prove useful, a more granular analysis of the impact of context might be challenging due to their diversity.", "paper_topic_and_main_contributions": "This paper introduces TIDE, a new dataset to evaluate the figurative language capabilities of machine translation (MT) systems. TIDE was obtained via human-AI collaboration and contains 512 ambiguous idioms alongside disambiguating content for one possible figurative and literal usage. Authors use TIDE to analyze the generations of various MT systems and large language models (LLMs), focusing on 1) models' preferences between literal/figurative translations given ambiguous expressions; 2) models' sensitivity to disambiguating context; and 3) evaluating the translation quality gap between figurative and literal sources. The findings of this study corroborate the strong performance of LLMs in producing more context-aware translations compared to regular MT models while also showing the limited abilities of LLMs in handling low-resource settings.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "The paper misses some relevant citations for new resources and/or analysis of MT and LMs for figurative language generation:\n\n- Examining the Tip of the Iceberg: A Data Set for Idiom Translation (https://aclanthology.org/L18-1148, Fadaee et al., LREC 2018)\n- MAGPIE: A Large Corpus of Potentially Idiomatic Expressions (https://aclanthology.org/2020.lrec-1.35, Haagsma et al., LREC 2020)\n- Can Transformer be Too Compositional? Analysing Idiom Processing in Neural Machine Translation (https://aclanthology.org/2022.acl-long.252, Dankers et al., ACL 2022)\n- Understanding Transformer Memorization Recall Through Idioms (https://aclanthology.org/2023.eacl-main.19, Haviv et al., EACL 2023)\n\nThe first two papers introduce larger datasets that were used for previous studies in idiom translation (like the third paper), and hence should be mentioned and compared to TIDE in the present work.", "Typos_Grammar_Style_and_Presentation_Improvements": "236-238: It should be specified which version of NLLB was used (3.3B, 1.3B, Distilled 600M or MoE 54B), possibly linking directly to the Github/Huggingface repository containing the model. For OPUS MT, it would be useful to know from the main body of the text whether a 6-layer (base) or a 12-layer (big) transformer architecture was used, given that both are available for some of the selected language pairs.\n\n246-248, 260-261: It should be specified when LMs and Google Translate APIs were accessed to produce translations.\n\nFigures 2,4 and 6: The gray bars are not explained anywhere. It should be mentioned what they represent. Also, it would be useful to recall that Marian En->Ko is not used in the Figure caption to explain the purple cross.\n\n350-351: Typo, OpusMT is actually the most literal model among those tested.\n\n472-474: The usage of the term \"emerge\" in this passage implicitly suggests that the translation capabilities of LLMs are acquired suddenly and in the absence of related training data. Given the controversial and still, to my best knowledge, unproven \"emergence\" of ability in pretrained language models, an alternative formulation such as \"[...] We speculate that pretrained LMs are particularly strong when translating in languages for whose enough examples were available during pre-training.\""}}]

Author rebuttal:

Meta-review: This work presents a datsaet and evaluations for the ability of neural MT systems and language models to translate idiomatic or non-idiomatic text which is ambiguous alone but disambiguated with context. The reviewers had somewhat ambivalent but overall positive evaluations of the work’s soundness and interest to the field; moderate concerns were raised both with the distinctness of the dataset from prior work and the validation of the LLM-assisted generations that comprise the dataset.

The work merited discussion among the AC and reviewers, and it was agreed that the initial concerns of the reviewers regarding scope were not at issue, and that there remain some concerns as to related work and dataset composition, as stated above. Overall, however, I’m happy to see the community leveraging strong generative models to help ask interesting questions about model behavior. However, concerns as to whether a dataset co-generated by GPT-4 will lead to, e.g., ChatGPT performing better than had the data come from an unrelated distribution are serious and under-explored; see, e.g., the AlpacaEval results that show that strong generative models like GPT-4 and Claude2 prefer their own family of models at a greater rate (https://tatsu-lab.github.io/alpaca_eval/). I encourage the authors to engage with this.
Decision: Accept-Findings