Crossing the Threshold: Idiomatic Machine Translation through Retrieval Augmentation and Loss Weighting
Authors: Emmy Liu
Aditi Chaudhary
Graham Neubig
Venue: EMNLP 2023 Main (2023)
Keywords: machine translation
idioms
multi-word expressions
retrieval-based machine translation
TL;DR: Idioms are hard to translate for neural machine translation systems. We investigate when this is the case, and propose some simple methods to translate them correctly while not degrading normal translation.

Abstract: Idioms are common in everyday language, but often pose a challenge to translators because their meanings do not follow from the meanings of their parts. Despite significant advances, machine translation systems still struggle to translate idiomatic expressions. We provide a simple characterization of idiomatic translation and related issues. This allows us to conduct a synthetic experiment revealing a tipping point at which transformer-based machine translation models correctly default to idiomatic translations. To expand multilingual resources, we compile a dataset of ~4k natural sentences containing idiomatic expressions in French, Finnish, and Japanese. To improve translation of natural idioms, we introduce two straightforward yet effective techniques: the strategic upweighting of training loss on potentially idiomatic sentences, and using retrieval-augmented models. This not only improves the accuracy of a strong pretrained MT model on idiomatic sentences by up to 13\% in absolute accuracy, but also holds potential benefits for non-idiomatic sentences.

Crossing the Threshold: Idiomatic Machine Translation through Retrieval Augmentation and Loss Weighting
Emmy Liu
Aditi Chaudhary' * Graham Neubig
Carnegie Mellon University
emmy cmm. edu, aditichaud@@ooglleooooo com, neubig@cs. cmu. cmu.edu
Abstract
Although idiom translation has been recognized as a problem even before the advent of neural ma- chine translation (Bar-Hillel, 1952; Wehrli, 1998). most work has focused on identifying and evaluat- ing the problem cross rross-ingguisticllyy (Baziotis et al., 2022; Dankers et al., 2022b), or on interpreting the behaviour of transformer-based models in trans- lating memorizing idioms (Haviv et al., 2022; Dankers al., 2022b). Others pose idiom identifi- cation and paraphrasing separate task from ma- chine translation (Pershina et al., 2015). Compara- tively fewer recent works have attempted to remedy this problem. Early work made use of idiom dic- tionaries and direct substitution, or example-based machine translation (Salton et al., 2014; Nagao, 1984). However, would idcally want make use of the contextual translation abilities of neu- ral models. Data augmentation and the creation of new datasets have helped address this problem (Agrawal et al., 2018), but may also be possi- ble to use existing data resources more effectively, especially for higher-resource languages.
Idioms are common in everyday language, but often pose a challenge translators because their meanings do not follow from the mean- ings of their parts. Despite significant advances, machine translation systems still struggle to translate idiomatic expressions. We provide a simple characterization idiomatic trans- lation and related issues. This allows to conduct a synthetic experiment revealing tipping point at which transformer-basee ma- chine translation models correctly default to idiomatic translations. To expand multilingual resources, we compile dataset of 4k natu- ral sentences containing idiomatic expressions French, Finnish, and Japanese. To improve translation of natural idioms, introduce two straightforward yet effective techniques: the strategic upweighting of training loss on poten- tially idiomatic sentences, and using retrieval- augmented models. This not only improves the accuracy of a strong pretrained MT model on idiomatic sentences by up to 13% in absolute accuracy, but also holds potential benefits for non-idiomatic sentences.
We first frame the general problem of non- compositional translation, which encompasses the translation of idioms and other multi-word expres- sions that cannot be translated word-for- word ($2). We then perform synthetic experiments in very simple case, finding that transformer-basse ma- chine translation models generally translate word- for-word until a proportional threshold sentences contain non-com con-cooposiiional expressions, at which point the translations flip to being correct (64.1), We evaluate translations by commercial models in three natural languages, and find a drop in perfor- mance idiomatic sentences and stronger perfor- mance more common idioms (51.2). We hypoth- esize that this may reflect similar trends as exist in processing other long-tail phenomena, and similar tactics to those used to deal with rare phenomena may work (Kandpal et al., 2022).
1 Introduction
An idiom is conventionalized expression in which the intended meaning differs from its literal transla- tion. The translation idioms has remained prob- lem for tfte--of-the-ar research and commercial translation systems, as idioms tend to be translated literally (Dankers et al., 2022b; Shao et al., 2017; Anastasiou, 2010). Failure translate these expres- sions correctly may lead to incomprehensible trans- lations, particularly literary text (Toral and Way, 2018). To illustrate the difficulty understanding mistranslated idioms, we present mistranslations from commercial systems in Table 3.3
Currently works Google Research. 'Code and data available at https:/ /ttpst//giie /gi. con/n. ight i ingal 3/ idi om- zrans lation/
"Translations from commercial systems were collected at the end of 2022.
With this intuition, we improve the idiomatic

---

Source
Tagget
Trussation
Language
Syste
Vou dises avor la deille.
You galnn he surving
You must hove slb.
tr
Deepl.
Whi fant tojoors chercher la petiee bite
He has dot all the 1' cross all t'e
He alonys hax to look for the Tinke heavt.
tr
Doepl.
I'al la pêche à mont
Good as bell.
have the peach to deuth.
Google
The Weak are Meat, Stroeg do Eat.
The Weok are the Strong.
30
Deepl.
falling that.
not gouna eat that hand.
Deepl.
HIS 40.00 1.
Well, sometimes what don't doesn't right?
don't sometimes Buddha
Google
kuin pellessa.
Homegiri clearly never met trashcan..
hea like a field.
Deepl.
Simun olisi phllayt mukain
You should have included in you sbout how leave banth.
You should inclided me decision way
Deepl.
paseskerpi
gonns
speum
Grasta, jolla istmme heitt2 tuskan morkismm
Roger vol tykadd kyttyr32.
Roger may take dim this...
Roger may like hunchbuck.
Google
Table 1: Examples of mistranslated sentences produced by commercial translation systems. Idioms and their corresponding translations are highlighted in red."
translations generated by a strong pretrained ma- chine translation model, ALM (Ma et al., 2021), without harming the translation quality of literal expressions. To contribute resources toward doc- umenting idioms and improving their translation cross-linguistically,, we create dataset of sentences containing idiomatic expressions in three languages (French (fx), Finnish (f (f and Japanese (ja) (53). We propose two simple but effective ways to im- prove translation of idioms, namely upweighting training loss on potentially idiomatic sentences and retrieval augmentation ($5). We find that this can improve the idiomatic translation abilities of the model significantly, by average of 10.4% in abso- lute accuracy ($7.1). Moreover, this does not harm translation of sentences where the literal sense of the idiom is used, and it improves translation of out-of-distribution sentences in French and Finnish as well. We perform human evaluation and eтror analysis, and find that the rate of severe seman- tic errors is reduced by an average of 7.52% ab- solute accuracy ($7.2). The ultimate aim for ma- chine translation is to ensure accessibility for all texts. This requires addressing idiomatic phrases, culturally language, and complex seman- tics. We demonstrate the potential for enhancing idiom translation using existing resources.
often challenging for non-native speakers (Cooper, 1999). For instance, a literal translation of one Por- tuguese idiom is "it is from litrle that you twist the cucumber". This is difficult to understand. How- ever, equivalent English expression is "As the twig is bent, is the tree inclined", which refers to actions during childhood influencing behaviours that people have as adults (Unbabel, 2019). This example illustrates the importance of translating idioms using equivalent idioms from the target cul- ture, or a paraphrase if there is no equivalent.
Idiomatic expressions are heavily shaped by the culture of language speakers, including religious beliefs, history, geography, and cuisine. For in- stance, food-related idioms in English tend to refer to foods such as beef and potatoes, while in Chi- nese, these idioms tend to refer more to rice and tofu (Yang, 2010). Cross-cultural knowledge is important in choosing a translation that conveys the proper intent to readers in the target language (Liu, 2012). Overly-literal translations and lack of broader context lation is still not parity with human translators, particularly when translating literary text (Matusov, 2019; Omar and Gomaa, 2020; Poibeau, 2022).
reasons why machine trans-
2.2 Formal definition
We use the idea of con-coopositionaaliyy to frame idiomatic translation more precisely. Let X' = {I1...., vNH be the set of tokens in the source lan- guage, and y = (un,..., VNr) ) be the set of tokens in the target language. Suppose that have oracle function TRANSLATE X ) that al- ways produces a correct translation. We can imag- ine this to be a helpful speaker who is perfectly familiar with both languages and never misreads text. Then we can say that a multi-token string
2 Non-Compositional Translation
2.1 Background on Idioms
Idioms commonly understood to be fixed ex- pressions that contradict the principle of compo- sitionality in language, which is say that their meaning cannot be predicted from the meanings of their parts (Radford, 2004; Portner, 2005). Idioms occur relatively frequently in all languages, and are

---

requires non-compositional translation if it can be translated correctly by the oracle as whole, but it cannot be translated correctly by individually translating parts of the sentence and joining them (according to the target language's 's word order). In other words, for a string of tokens 21,,.,, Fn. 4
the idiomatic test set were actually idiomatic, we sorted sentences into an idiomatic test set where the idiomatic meaning of phrase was used (e.g. "to die") and a literal test set, where the literal mean- ing of the phrase was used (e.g. kicking physical bucket). The first 100 examples containing cach idiom's 's lemmatized form were collected, and up to the first 3 (for Japanese) or 5 (for Finnish and French) literal and figurative examples in this set were collected create the test set. This was avoid dominance of very common idioms in the test set. This created two test sets related to the id- iom list for each language, the idiomatic and literal test sets.
TRANSLATE(<)) o TRANSLATE a.)
(I)
2m1y
We note that this definition is very general and also includes other phenomena such as multi-word expressions and named entities. However, we can now use this definition to create relevant synthetic task, allowing to observe translation composi- tionality under different settings (54.1).
To validate these judgments, we hired native annotators in French and Finnish. They were pre- sented with examples from the final literal and id- iomatic test sets in a shuffled order, and asked label them with idiomatic, literal, or N/A labels if they didn't think was an instance of either. Agreement (Krippendorff's a (Krippendorff, 1970; Castro, 2017)) in both cases was moderately high (French = 0.5754, Finnish a 0.6154). Details can be found in Appendix D.
3 Idioms and Data Collection
We can use the formal definition from the previous section to generate synthetic data for experiments. However, we ultimately want to improve transla- tion of real idioms. To do so, we collect dataset of natural sentences to evaluate commercial systems and the model we seek to improve.
Although large corpus of potentially idiomatic expressions exists in English (Haagsma et al., 2020), there are no readily accessible equivalents in other languages. Therefore, we collected idioms in French, Finnish, and Japanese from language- learning sites, listed in Appendix B. These lan- guages were chosen for phylogenetic diversity, and due to availability of commercial translation sys- tems. In total, there were 148 French idioms col- lected, 92 in Finnish, and 1336 in Japanese.
Finally, we collect two random test sets, one which is in-domain and another which is out-of- domain. For the in-domain test set, we simply select sentences from the development set of Open- Subtitles (see subsection 6.2 for details on our split of OpenSubtitle).. For the out-of-domain test set, we use the Ted Talks corpus (Reimers and Gurevych, 2020). This is to ensure that translation quality of other, unrelated sentences is not impacted by any modifications meant to improve translation of idioms. Topics discussed and vocabulary used in Ted Talks may be slightly different from what is discussed in movies or TV shows, so training the model on OpenSubtitles and testing on Ted Talks allows us to evaluate model generalization. For both test sets, to control for translation length as source of difficulty, sentences were matthhedd on the target side with corresponding sentences in the idiomatic set. This created the random set, which is the same size as the idiomatic test set. All three test sets are summarized in Table 2.
To collect sentences containing these idioms, we matched on lemmatized forms from the 2018 ver- sion of OpenSubtitles (Lison et al., 2018), where lemmatization was performed with Stanza (Qi et al., 2020). In total, there were 85632 French sentences containing potentially idiomatic expressions, 51811 Finnish sentences, and 23018 Japanese sentences. To filter out unaligned sentences, we scored each source and reference sentence using COMET-QE (Rei et al., 2020) and removed the bottom 10% of each language's ' sentences by COMET-QE scores. Some idioms have a plausible literal meaning (such as "kick the bucket" to mean kicking a phys- ical bucket). To make sure that all examples in
4 Evaluating Non-Compositional Translation
denotes string concatenation given the word order of language X. i.c. if the word order is SVO, the tokens helonging to the subject should be placed in front of the tokens helonging to the verb, and so on.
4.1 Artificial Language Translation
We first use the definition of non-compositional translation in ($2) to create a synthetic task. This

---

Langtage
(oot)
be 10, 20, and 30 in the non-informative case and 15, 15 and 25 in the informative case.² Further training details can be found Appendix A.
Ex
83632
777
79
737
777
2410
41 51811
449
81
449
44p
1428
30 23018
3253
389
3253
3253
10143
Although this may seem like simple task, we found it surprisingly difficult for models to learn this non-compositional pattern. Results in each set- ting, averaged across 5 random sceds, are presented in Figure 1. Especially for the small model, there is a sharp gradation from translating none of the non-compositional expressions correctly to translat- ing them all correctly, which occurs when roughly 10% of training data contains non-compositional pattern. A similar trend exists for larger models, but the threshold is less distinct. This corrobo- rates the tendency for transformers to translate non-compositional phrases literally (Dankers al., 2022b). Comparatively less data is required when the context is informative, but the trends remain similar to the non-informative case. As model size and corpus size increase, the rate of correct trans- lations for non-compositional examples actually drops, contrary to expectation.
Table 2: Size of test sets for each language. The id- iomatic and literal sentences contain strings matching known idioms (after lemmatization), and the in-domain random set contains unrelated sentences from Open- Subtitles, but the out-of-domain random set contains unrelated sentences from the Ted Talks corpus.
allows us to gain an understanding of how much data is required to memorize non-compositional patterns, Although this experiment is not realistic to natural language (notably, there is no token-level ambiguity in this experiment), we note that using synthetic experiments allows us to easily extend the data generation setup and examine model be- haviour along many different conditions, such as informativity.
The source language in these experiments was composed of tokens 0 through 9. X {0, 1, ..., 9). by adding 10 each token, Y (10,..., 19). The translation rule was add 10 to the value of each token in the source language, e.g. + 10. 1 11. We add a single non-compositional rule that doesn't follow this trend, 0 >> 12 (rather than 0 1 + 10 11). We limited the maximum sequence length to 6 tokens.
The target language was produced
It is unlikely that any individual idioms occur in 10% of sentences in natural language. Due to the highly regular translation rules in this synthetic lan- guage, there may be stronger bias toward translat- ing compositionally in this experiment. However, gain the intuition that idioms can be translated effectively if they appear frequently, and that clear context clues reduce data required.
We generated synthetic training corpora of sev- eral sizes containing different numbers of occur- rences of the non-compositional rule 0 1 + 12. The number of training sentences ranged from 100k to 10M, while the number of noncompositional oc- currences ranged from 10 1M. We examined two informativity conditions, corresponding to the case where the context provides no information (to- kens are randomized around the non-compositional expression), and the context being perfectly infor- mative. The perfect informativity condition was achieved by adding the canary token "11" to the source vocabulary, and only inserting this token prior to the non-compositional pattern "0 1".
4.2 Evaluation of Commercial Systems
Although synthetic experiments provide intuition on the difficulty of translating idioms, one might ask whether similar results hold in natural lan- guage. To answer this, we examine the perfor- mance of commercial systems on the test sets in ($3). Namely, examine Google Translate and DeepL on Finnish, French, and Japanese idiomatic, literal, and random sentences. Results are in Ta- ble 3. We observe drops in translation quality on idiomatic sentences in all languages, with lower automatic metrics overall.
We experimented with three different trans- former sizes (Vaswani et al., 2017), each of which had a hidden dimension and embedding size of 512, as well 16 attention heads. Only the num- ber of encoder and decoder layers varied, such that the small transformer had encoder and decoder layers, the medium transformer 8, and the large transformer 16. We fix the number of epochs for the small, medium and large models to respectively
"The numher of training epochs was determined by the number of epochs it took for the validation loss to plateau in the 100k size corpus with 1K composiioii examples, rounded up to multiples of 5. This was done to mimic the typical training process for MT models, which are trained until loss or accuracy plateaus on general dev set. Since idiomatic expressions tend to be uncommon compared to literal ones, there may not be many in the dev or train sets, and so the model's 's performance on idiomatic expressions may not he tracked.

---

Small transformer
Small transformer
20.00
0.00
20.35
20.04
0.01
=0.38
0.00 =0
0.00 10.00
0.98 =:0
s.0.55
0.01 =0.02
56
1,00 =0.00
0,11 =0.15
0,00 +0,00
1,00 =0.00
0,86 =0.23
0,02 #0.03
0.00
1.00 20. 00
0.00 =0.00
0.99 #0.02
0.28 =0.17
67
d
Medium transtormer
Medium transtormer
16
Number of ictomss
Number of idemms
20
0.04 10.00
1.000
0.05
0.74 =0.33
0.16 =0.19
0.05 =0.00
0.94 =:
0.00 =0.01
0.04 =0.02
66
30 318
1.00 =0.00
0.63 35
0.03 =0.05
0.87 =0.07
0.70 #0.56
0.27 =0.37
0:05
00 =:0
0:03
0.31
UAA
50
Large transformer
Large transfermer
02
s0.27
10
000 500
0.08 =0.16
0.05 =0.05
0.04 =0.00
0.62 =0 =0.33
0.36 =0.36
0.23 =0.23
0.07 =0.17
0.37 10.11
0.06 1:0.07
0.76 10.41
0.16 10.36
0.09 10.07
66
0.71 20.25
0.33 =0
0.12 =0.14
0.78 =0.11
0.59 =0.51
0.27 =0.29
64
=0.19
100k
3M Training set size
10M
100k
60
Training set size
Uninformative context
Informative context
Figure I: Accuracy of a transformer in translating a non-conpositional phrase after training on datasets of different sizes, with different numbers of non-compositional patterns (only conpoooposiii translation accuracy is depicted). Results are averaged across 5 seeds, and standard deviation is shown.
Language
System
BLEU
METEOR
Dagiic Scare
0.5053
11
Google
Despl.
0.1001
0.2497
0.2250
0.8966
0.8726
0.5023
ALM-base
0616
0.3592
0.9126
0.35
0.5093
1
Despl.
0. 1488
0.3908
0.9146
0.8073
Google
0.2
0.4
0.6
0.5
2.2
244
2.5
0.5
ALM base
0.2093
0.5050
0.9350
F Cinsionso
DeepL.
0.2052
0.4082
0.9103
0.50
Google
0.2288
0.4357
0.9062
Dgooi Soore
ALMM-ase
0.2365
0.4971
0.9145
0.66
DeepL
0.1575
0.3278
0.9006
Google
0.1261
0.2794
0.8808
ALM-base
0.9211
022
Parcentli
0.5 0.6
3.2
3.4
ofidioms
f fiikiial
Deapl.
0.2219
0.4022
0.9122
Google
0.2034
0.3830
0.9012
AL.M-hase
0.2778
0.5504
0.9377
Figure 2: Automatic metrics Quality of DeepL French translations on idiomatic test set bucketed by idiom frequency. The bottom 20% least common idioms are excluded, as they may occur fewer than 3 times and not be our test set.
Deepl.
0.2854
0.4650
0.9125
Google
0.3103
0.4922
0.9149
ALM-base
0.2778
0.5504
0.9377
1 Fateraii
Daspl.
0.1172
0.2735
0.8932
Google
0.0672
0.1839
0,8644
ALM-base
0.09048
0.2998
0.9234
Deepl.
0.1517
0.3440
0.9059
Although it's impossible for us to determine what data these commercial systems were trained on, we examine the frequency of each idiom within OpenSubtitles as a proxy for its overall frequency in the training data, and bucket idioms into quin- tiles based on their occurrence frequency in source text. As idioms become more frequent, the quality of translations increases. An example of DeepL the French idiom set is shown in Figure 2. Trends for other languages and systems are in Appendix H. This indicates that like in the synthetic experiments, there may be strong frequency effects on translation quality idioms.
Google
0.0937
0.2565
0.8829
Al.M-base
0.4222
0.9222
Deepl. 0.1074
0.2934
0.8878
Google
0.1079
0.2834
0,8829
Al.M-buse
Table Performance of commercial systems id- iomatic, literal, and random test sets. There is clear degradation performance on idiomatic sentences.
Methods to Improve
Non-Coopositional Translation
We explore two methods to improve translation, loss weighting and kNN-MT. These two methods are relatively simple to use, where loss weighting

---

only requires list of potentially idiomatic phrases in the source language, and KNN-MT only requires enough space on disk to save datastores.
simply be their rarity. Upweighting training ex- amples that contain idioms may help with under- representation. Furthermore, retrieving similar ex- amples may find occurrences of the same idiom which were translated correctly.
More formally, we consider the basic case of autoregressive machine translation, with a set of parallel sentences in the source (X (2(0) li=1)
and target (Y ) language: D {(x(1 ())),, (z(M), ). The model pe with parameters is trained by minimizing the loss:
Experimental Settings
6.1 Experimental Settings
We run experiments on ALM-base, a transformer encoder-decoder model with 360M parameters, a larger version of which ranked first in the WMT21 multilingual translation task (Ma et al., 2021; Machine Translation, WMT21). We train ALM model for each language pair. Each model was trained for 2 million steps, and the checkpoint with the best loss on the validation set was kept. Further details are in Appendix C. To decode, we used beam search with a beam size of 5.
((0, Σ byy Po
Upweighting here refers to sentence-level up- weighting, where there is a set of sentences A that we'd d like upweight with weight coefficient a. In this case, A would be potentially idiomatic sen- tences. We keep all other parameters for training the same as the base model.
6.2 Data
1 e(y Ea )o((
Models were trained on OpenSubtitles for each lan- guage pair. Data from test sets were removed, and 10% the remaining data was used as validation set. There were 33.8M sentences in the fr-en. train set, 22.0M in fi-en, and 1.6M in ja-en.
C(0, D)
(3)
kNN-MT augments a translation model with a retrieval component (Khandelwal et al., 2021). Given each sentence (x, y), we construct data- store with keys based on hidden representations constructed from the translation model, and values being the next word in the target sentence.
6.3 Evaluation
We use multiple automatic metrics to evaluate trans- lation quality. However, due to the importance of accurate semantic evaluation, the authors (native English speakers and fluent in French and Japanese) conduct human evaluation inspired by MQM (Lommel al., 2014). Only errors that would fall under the "trminnooog""" and "accuracy" error types are considered, as we are focused on severe semantic errors. We give a score of for severe errors and score of 0.5 for major errors. A score of 1 is given otherwise. Exact evaluation standards are in Appendix E.
During gencration, a probability distribution over next words can be computed based on the retrieved next words and the distance of their keys to the current context. A parameter À controls inter- polation between the distribution over next words predicted by the base model, and the distribution predicted by the retrieved k neighbours."
10 , )) = Ap (1 (2 10, Mi..-1
+ +11 (1 Appo ()) (0) .i1-))
(4)
Results
We also combine loss weighting with KNN-MT. where model is trained with sentence upweighting and interpolated with datastore based repre- sentations from the teweight-trained model.
7.1 Automatic and Human Evaluation
In most cases, as reported in Figure 3, using com- bination of sentence upweighting and KNN-MT led to the greatest increase in automatic metrics on all three test sets, of up 3.08 BLEU points on the idiomatic test set (fr), 2.69 BLEU points on the literal test set (f and 5.75 points on the random test set (fz). In all cases except ja-rand, us- ing one or more of these methods improved over
Intuitively, these methods make sense to use for idiom translation we have previously seen that one problem with non-compositional phrases may
"We run a hyperpurameter search using the validation set to find the best KNN-MT settings for each language. Further details are in Appendix C.

---

Normal
Upweight
upwalghht
BLEU
Fl- RougeL sum
F1 BertScore
Fl Mcteor
622
03A
644
C/m
Literal
Randroi
laoma
Uiteral
Rand sor
Kandiat
eh
Namma
Litrral
Rind
FR BLEU
FR- RougeL-sum
FR BertScore
FR. Meteor
AdC
ARH-
624
A3H
E4c
644
AA
Liberal
Bard
Idlmm
Litr
Aand
JP BLEU
RougeL sum
BertScare
Meteor
620
0:0
MA
021
08
Mc
690
Ltbral
Raneln
Rand
Lttrrl
Randle
Ulbel
Rareta
Litrl
Runsri
Asr Bo
Figure 3: Results of automatic metrics. In most cases, combining loss weighting with KNN-MT improves automatic metrics the most on all three test sets, including the out-of-distribution (Random) test set.
the baseline. Exact numerical results are in Ap- pendix J.
We note that the Japanese model was trained on roughly 1/10th of the data of the French and Finnish models, so its translations are not as high- quality. This also leads to the construction of a much smaller datastore, which may lead to weaker performance on the random set.
We evaluate the statistical significance of the results through one-tailed permutation test (Gra- ham et al., 2014). Further details are in Appendix Exact results are in Appendix G. For Finnish, sig- nificance is achieved for all three test sets, and for French, significance is achieved for the idiomatic and random test sets. For Japanese, values achieved are not significant, but borderline.
knn
upweight
upweight
Ex-idioms
0.6177
0.6659
0.7010
0.7463
Ex-Literal
0,7039
0.7303
0.7105
0.7434
Ex- Ex-rand-oor
0.7526
0.8398
0.7477
0,8232
fi-idions
0,4803
0.5562
0.5604
0.6194
As our focus is on mitigating semantic errors, we mostly focus on the results of human evalua- tion, which are summarized in Table 4. Here, we also find that using both sentence upweighting and kNN is the best condition in most cases, increasing accuracy by roughly 13% in French and Finnish, and 4.5% in Japanese for idiomatic sentences. En- couragingly, this does not overly harm translation of literal sentences, as accuracy on the literal set either increases slightly (by roughly 4% in French and Finnish), or decreases very slightly (by roughly 0.4% in Japanese). For the random set, the combi- nation of sentence upweighting and KNN-MT by around 7% accuracy. However, in Japanese, perfor- mance on the random test set decreases by 4%. In all cases except ja-rand, one or more of these methods improves over the baseline.
Fi-literal 0.7692
0.8462
0.8205
0,8141
fi-rand-out
0.7647
0.8235
0.7771
0.828
1 1a-1dioms
0.4152
0.4286
0.4643
0.4598
18 litteraa
0.6475
0.6516
0.6557
0.6434
ja-rand-out
0.6207
0.5560
0.5776
0,5862
Table 4: Human-judged accuracy on sentence-level se- mantics.
7.2 Error analysis
We repeat the frequency analysis performed on commercial systems (64.2) for ALM, and find that adding upweighting and KNN-MT generally im- proves translations at all frequency levels. These increases are not concentrated in low-frequency idioms, more common idioms continue to be translated better." A representative example (for
"This trend is different in retrieval of long-tail facts in ques- tion answering, in which retricval flattens out the difference

---

French) is in Figure 4. A complete set of plots are in Appendix I.
unclear why this should be the case. We examined sentences where the original translation was incor- rect but the upweight+knn translation was correct, and found that they tended to contain named enti- ties. For instance, for the sentence "La chirurgie à coeur ouvert au Nigeria, c' c'est un gros problême. (Open heart surgery in Nigeria - big trouble.)", the base model incorrectly produced the translation "Open-heart surgery in Forbes, that's a big prob- lem.", while the upweight+knn model translated correctly. In some cases, words with multiple pos- sible translations (e.g. spectre: ghost, spectrum) became correctly translated. "Mais regardez le nombre de lignes noires dans ce spectre. (But look at the number of black lines in that spectrum.)" was originally translated incorrectly as "But look at the number of black lines in that ghost".
F1
III RT Score
0.925
0.25
0.530
0,925
0.20
0.920
upw.ightt kmn
0,915
upwwightsicc
0.15
0.4
0.4
5.4
5.6
Percentiless
Percentiles NETEOR
ROUGE
0,500
Heric Sogre.
0,500
0.475
0.450
0.400
Bade
base
upweightt-kcn ktn
0.400
0.6
Porcentile ofidioms 05
Percentiie odidmsss O5
Figure 4: Automatic metrics for fr-idion sentences, plotted by frequency, for base and upweight+knn. We examine the rate of severe and major errors made in the base model and the upweight+knn model in Table 5. In French and Finnish, the rate of critical errors decreased greatly, particularly in the idiomatic and random test sets, This is true to lesser extent in Japanese. Major errors also decreased to lesser extent. The only test set where errors increase is again the ja- rand test set. We note that it's possible for the rate of major errors to be higher in the upweight+knn model because some severe errors transitioned to major errors.
8 Related Work
Recent work has raised the issue of idiom han- dling in MT (Baziotis et al., 2022; Dankers et al., 2022b,a). There is historical recognition of the problem, including of mordi-word expressions (Sag et al.. 2002; Calzolari et al., 2002; Zaninello and Birch, 2020). This has historically motivated example-based machine translation (Nagao, 1984). Similar motivations underlie the use of KNN-MT. However, neural models may already be capable of translating idiomatic phrases if they appear often enough in training data.
System
Severe (1)
Major (L)
ri-1 ona
hase
0.4258
0,1648
upweight*knn
0.3242
0.0962
Other works focus on data augmentation and cre- ating new data resources (Ho et al., 2014; Fadaee et al., 2018; 8; Agrawal et al., 2018; Haagsma et al., 2020). A related task is detection of convention- alized metaphors (Levin al., 2014). Automatic identification of idiomatic phrases, as well as data augmentation are promising avenues to improve performance in lower-resource languages.
fi literal
base
0.1728
0.1234
upweight-knn
0.1234
0.1358
fi-sandom
0.1317
0.2009
upweight-knn
0.0603
0.2188
#u-1dionn
base
0,3042
0.1528
upweight-knn
0.1092
tr-1itera
0.2326
0.1047
upweight-kma
0.2209
0.04651
dr-sandom
base
0,1624
0,1688
upweight-knn
0.09407
0.1649
Instance weighting has been explored previously in the MT literature, but has been mostly explored in the context of domain adaptation, rather than being used to improve translations of rare or non- compositional phrases in the same domain (Foster et al., 2010; Wang et al., 2017).
ja-1dt tiiddtoon
hase
0.4643
0.241
upweight-knn
0.4464
0.1875
ja literal
0.28667
0,1311
upweight-knn
0.2787
0.1557
ja-zandom
buse
0.2931
0.1724
upweight-knn
0.3190
0.1897
Table 5: Rate of major and severe errors in translations. One question is why the error rate on out-of- distribution sentences drops for French and Finnish. In fi fi-rand, the severe error rate more than halves (0.1317 + 0.603), and in fr-rand, it nearly halves (0.1624- 0.09407). However, it is
Idiomatic phrases are prototypical case of phrases that need to be memorized (Haviv et al., 2022). Many also occur infrequently in training data, which may make it difficult for transformer- based models to translate them (Kandpal et al., 2022). This can be mitigated, as we have shown in this paper. However, more work is needed to effec-
hetween rare and common facts (Kandpal et al., 2022).

---

tively learn idioms and other infrequent linguistic elements with few repetitions.
Next is our use of synthetic data. While syn- thetic data allowed us to control for certain vari- ables, our setting is purposefully simplified, poten- tially limiting the ecological validity of our findings. Although our synthetic language was designed to mimic non non-compositional translation issues, it may not encapsulate the full extent of such complexi- ties in real-world languages. Namely, there is only one non-compositional pattern and the remaining translations are one-to-one mappings.
9
Conclusion
We highlight the challenge idiomatic expressions pose to machine translation systems and provide simple solutions to improve performance. Through synthetic experiments, we identify threshold at which cansformer-based models correctly default to idiomatic translations. We develop a dataset of sentences containing idiomatic expressions in French, Finnish, and Japanese, and introduce two techniques upweighting training loss on poten- tially idiomatic sentences and augmenting models with KNN-MT which enhance the idiomatic trans- lation accuracy of a strong model, while offering potential benefits for non-idiomatic sentences.
Our research also depends on the quality and representativeness of the training and evaluation corpora. For instance, certain idioms may be over- represented or underrepresented, which could af- fect the translation performance.
Lastly, our improvement methods, namely up- weighting and KNN-MT, have inherent limitations. Upweighting could lead to overfitting on idiomatic expressions and may not be effective when id- ioms occur infrequently in the data. On the other hand, kNN-MT might not yield significant improve- ments the idiom or its correct translation rarely appears in the training data, limiting its utility in such scenarios.
Future research could extend these techniques to additional languages, and explore their effective- ness in dealing with other long-tail phenomena. We hope that this work contributes toward increasing the intelligibility of translations containing idioms set phrases. Ultimately, for machine translation to be useful for everyone without causing misunder- standings, "last mile" problems involving cultural knowledge, long-tail phenomena, and complex se- mantic evaluation should be taken into account.
Future work could address these limitations by expanding the linguistic scope of the study, explor- ing more complex methods or architectures. in- vestigating to what extent similar techniques can be applicd to related issues in semantic preservation during machine translation.
Acknowledgements
Thank you to Perez Ogayo for helping with DeltaLM setup, all annotators who validated id- iomatic/literal judgments, as well as to NeuLab members for providing feedback on parts of the draft. This project was funded by the P2020 pro- gram MAIA -1S8OAA-0--024-FEEDDRR
References
Ruchit Agrawal, Vighnesh Chenthil Kumar. Vignesh- waran Muralidharan, and Dipti Sharma. 2018. No more beating about the bush A step towards idiom handling for Indian language NLP. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), Miyazaki, Japan. European Language Resources Association (ELRA).
10 Limitations
Our research provides first step toward capturing non-compositional expressions in machine trans- lation. However, we do not conclusively solve the problem, as ideally a machine translation sys- tem should be able to learn any idiom or non- compositional phrase from a few examples.
Dimitra Anastasiou. 2010. Idiom treatment experiments in machine translation. Ph.D. thesis.
Yehoshua Bar- Hillel 1952. The treatment "idioms" by translating machine. In Proceedings of the Con- ference Mechanical Translation, Massachusetts Institute of Technology.
First, our experiments were conducted on select group of languages (Finnish, French, and Japanese), which do not fully capture the varicty and complexity of languages worldwide. Given the diversity of language structures and idiomatic expressions, the generality of our findings lan- guages with drastically differrnt grammatical struc- tures or idiom usage patterns remains uncertain.
Christos Baziotis, Prashant Mathur, and Eva Hasler. 2022. Automatic evaluation and analysis of idioms in neural machine translation.
Nicoletta Calzolari, Charles J. Fillmore, Ralph Gr- ishman, Nancy Ide, Alessandro Lenci, Catherine MacLeod, and Antonio Zampolli. 2002. Towards

---

best practice for multiword expressions in compu- tational lexicons. In Proceedings of the Third In- ternational Conference on Language Resources and Evaluation (LREC'02), Las Palmas, Canary Islands Spain. European Language Resources Association (ELRA).
Wan Yu Ho, Christine Kng, Shan Wang, and Francis Bond. 2014. Identifying idioms in Chinese trans- lations. In Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC'14), pages 716-721, Reykjavik, Iceland. Eu- ropean Language Resources Association (ELRA).
Jeff Johnson, Matthij Douze, and Hervé Jégou. 2019. Billion-scale similarity search with GPUs. IEEE Transactions on Big Data. 7(3):355-547.
Santiago Castro. 2017. Fast Krippendorff: Fast com- putation of KrippendorfI's alpha agreement measure. https://github: com/ pln- win---inngd---e c/fast-krippendon
Nikhil Kandpal, Haikang Deng, Adam Roberts, Eric Wallace, and Colin Raffel. 2022. Large language models struggle to learn long-tail knowledge.
Thomas C. Cooper. 1999. Processing of Idioms by L. Learners of English. TESOL Quarterly, 33(2):333- 262.
Urvashi Khandelwal, Angela Fan, Dan Jurafsky, Luke Zettlemoyer, and Mike Lewis. 2021. Nearest neigh- bor machine translation.
Verna Dankers, Elia Brumi, and Dieuwke Hupkes. 2022a. The paradox of the compositionality of natural lan- guage: A neural machine translation case study. In Proceedings of the 60th Anmual Meeting of the As- sociation for Computational Linguistics ( (Volume 1: Long Papers), pages 4175, Dublin, Ireland. As- sociation for Computational Linguistics.
Diederik P. Kingma and Jimmy Ba. 2017. Adam: A method for stochastic optimization.
Klaus Krippendorff. 1970. Estimating the reliability, systematic error and random error of interval data. Educational and Psychological Measurement, 30:61 -70.
Verna Dankers, Christopher Lucas, and Ivan Titov. 2022b. Can transformer be too compositional? analysing idiom processing in neural machine trans- lation. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Vol- ume 1: Long Papers), pages 3608-3626, Dublin, Ireland. Association for Computational Linguisties.
Lori Levin, Teruko Mitamura, Brian MacWhinney, Davida Fromm, Jaime Carbonell, Weston Feely, Robert Frederking, Anatole Gershman, and Car- los Ramirez. 2014. Resources for the detection of conventionalized metaphors in four languages. In Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC' 14), pages 501, Reykjavik, Iceland. European Lan- guage Resources Association (ELRA).
Marzich Fadace, Arianna Bisazza, and Christof Monz. 2018. Examining the tip of the iceberg: A data set for idiom translation. In Proceedings of the Eleventh In- ternational Conference on Language Resources and Evaluation (LREC 2018), Miyazaki, Japan. European Language Resources Association (ELRA).
Pierre Lison, Jörg Tiedemann, and Milen Kouylekov. 2018. OpenSubiirles0018: Statistical rescoring of sentence alignments large. noisy parallel corpora. In Proceedings of the Eleventh International Confer- ence on Language Resources and Evaluation (LREC 2018). Miyazaki, Japan. European Language Re- sources Association (ELRA).
George Foster, Cyril Goutte, and Roland Kuhn. 2010. Discriminative instance weighting for domain adap- tation in statistical machine translation. In Proceed- ings of the 2010 Conference Empirical Methods Natural Language Processing, pages 451-459, Cambridge, MA. Association for Computational Lin- guistics.
Dayan Liu. 2012. Translation and culture: Translating idioms between english and chinese from a cultural perspective. Theory and Practice in Language Stud- ies, -262.
Yvette Graham, Nikita Mathur, and Timothy Baldwin. 2014. Randomized significance tests machine translation. In Proceedings of the Ninth Workshop on Statistical Machine Translation, pages 166-274. Association for Computational Linguisties.
Arle Lommel, Hans Uszkoreit, and Aljoscha Burchardt. 2014. Multidimensional quality metrics (mqm): A framework for declaring and describing translation quality metrics. Tradumâtica, (12):0455-46) 463.
Shuming Ma. Li Dong, Shaohan Huang, Dong- dong Zhang, Alexandre Muzio, Saksham Singhal, Hany Hassan Awadalla, Xia Song, and Furu Wei. 2021. Deltalm: Eacoodr-decoder pre-training for language generation and translation by augmenting pretrained multilingual encoders.
Hessel Haagsma, Johan Bos, and Malvina Nissim. 2020. MAGPIE: A large corpus of potentially idiomatic expressions. In Proceedings of the Twelfth Lan- guage Resources and Evaluation Conference, pages 279-287, Marseille, France. European Language Re- sources Association.
Evgeny Matusov. 2019, The challenges of using neural machine translation for literature. In Proceedings of the Qualities of Literary Machine Translation, pages 10-19, Dublin. Ireland. European Association for Machine Translation.
Adi Haviv, Ido Cohen, Jacob Gidron, Roci Schuster, Yoav Goldberg, and Mor Geva. 2022. Understanding transformer memorization recall through idioms.

---

Makoto Nagao. 1984. A framework of mechanical translation between japanese and english by analogy principle. In Proc. of the International NATO Sym- posium on Artificial and Human Intelligence, page 173-180, USA. Elsevier North-Holland, Inc.
Giancarlo Salton, Robert Ross, and John Kelleher. 2014. Evaluation of a substitution method for idiom trans- formation in statistical machine translation. In Pro- ceedings of the 10th Workshop on Multiword Expres- sions (MWE), pages 38-42, Gothenburg, Sweden. Association for Computational Linguisties.
Abdulfattah Omar and Yasser Gomaa. 2020. The ma-
chine translation of literature: Implications for trams- lation pedagogy. International Journal of Emerging Technologies in Learning (UJTT), 15:228.
Yutong Shao, Rico Sennrich, Bonnie Webher, and Fed- erico Fancellu. 2017. Evaluating machine transla- tion performance on chinese idioms with a blacklist method.
Sixth Confrence Machine Translation (WMT21). Shared task: Large-scale multilingual machine trans- lation.
Antonio Toral and Andy Way. 2018. What level qual- ity can neural machine translation attain on literary text? In J Moorkens, S Castilho, F Gaspari, and S Doherty, editors. Translation Quality Assessment: Technologies and Applicatioos. Springer, Cham.
Myle Out, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng. David Grangier, and Michael Auli. 2019. fairseq: A fast, extensible toolkit for sequence modeling.
Unbabel. 2019. Why translating idioms is hard.
Maria Pershina, Yifan Hc, and Ralph Grishman. 2015. Idiom paraphrases: Seventh heaven vs cloud nine. In Proceedings of the First Workshop on Linking Computational Models of Lexical, Sentential and Discourse-leve Semantics, pages 76 82, Lisbon, Por- tugal. Association for Computational Linguisties.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez. Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need.
Rui Wang, Masao Utiyama, Lemao Liu, Kehai Chen, and Eiichiro Sumita. 2017. Instance weighting for neural machine translation domain adaptation. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 1482-1488, Copenhagen, Denmark. Association for Computational Linguistics.
Thierry Poibeau. 2022. On "human parity" and "super human performance" in machine translation evalu- ation. Marseille, France. Language Resource and Evaluation Conference.
Paul H. Portner. 2005. What is meaning?: Funda- mentuls of formal semantics. Blackwell Publishing, Malden, MA.
Eric Wehrli. 1998. Translating idioms. In COLING 1998 Volume 2: The 17th International Conference Computational Linguistics.
Peng Qi, Yuhao Zhang, Yuhui Zhang, Jason Bolton, and Christopher D. Manning. 2020. Stanza: A Python natural language processing toolkit for many human languages. In Proceedings of the 58th Annual Meet- ing of the Association for Computational Linguisties: System Demonstrations.
Chunli Yang. 2010. Cultural differences chinese and english idioms of diet and the translation. English Language Teaching. 3.
Andrea Zaninello and Alexandra Birch. 2020. Multi- word expression aware neural machine translation. Proceedings of the Twelfth Language Resources and Evaluation Conference, pages 3816-3825, Marseille, France. European Language Resources Association.
Andrew Radford. 2004. English Syntax: An Introduc- lion. Cambridge University Press, Cambridge, UK.
Ricardo Rei. Craig Stewart, Ana C Farinha, and Alon Lavie. 2020. Unbabel's 's participation in the WMT20 metrics shared task. In Proceedings of the Fifth Con- ference on Machine Translation, pages 91 1-920, On- line. Association for Computational IiiiaalLinnge Linguistics.
A Synthetic Dataset Training Details
Three encoder/decoder transformer sizes were trained, differing only in number of en- coder/decoder layers. The small size had 3 encoder/decoder layers, the medium size had 8 encoder/decoder layers, and the large size had 16 encoder/decoder layers. For all models, the hidden dimension was 512, the embedding dimension was 5 512, and there were 16 attention heads.
Nils Reimers and Iryna Gurevych. 2020. Making monolingual sentence embeddings multilingual us- ing knowledge distillation. In Proceedings of the 2020 Conference Empirical Methods in Natural Language Processing. Association for Computational Linguistics.
Ivan A. Sag, Timothy Baldwin, Francis Bond, Ann Copestake, and Dan Flickinger. 2002. Multiword expressions: A pain in the neck for nip. In Compu- tational Linguistics and Intelligent Text Processing, pages 1-15, Berlin, Heidelberg. Springer Berlin Hei- delberg.
In the experiments without informative context, the small transformer was trained for 10 epochs, the medium transformer for 20, and the large for 30. In experiments with context, this was changed to 15, 15, and 25 respectively. These values were

---

based on early experimentation with loss plateaus on the validation set.
source and target lengths of 512. Label smoothing of 0.1 was used, and the loss function used was cross-entropy.
Sentences the synthetic dataset were com- posed of tokens as described subsection 4.1. Sentences were constrained to be 1-6 tokens in length.
For upweighting experiments, the upweight co- efficient for each language was found through a hyperparameter search on the validation set over a E {3, 5, 10, 50}. The values for each language were O' = 3, fr = 10, Oja =
Experiments with the synthetic dataset were im- plemented in Py Torch.
B Idiom Sources
For kNN-MT, three datastores were built for ap- proximate kNN search using the training set from OpenSubtitles. These datastores were built with the faiss library (Johnson et al., 2019). The Finnish datastore contained 248M vectors with 507k centroids, while the French datastore con- tained 348M with 713k centroids, and the Japanese datastore 17.8M with 73k centroids. All vectors were stored in ipl with code size of 32. The vectors used as keys in the datastore correspond to the inpul to the last feedforward layer. Ad- ditionally, a hyperparameter search for each lan- guage was carried out on the validation set, over values of À E {0.2, 0.4, 0.6, 0.8}, temperature E {0.1, 1, 10), and number of retrieved neighbours E {5, 10, 15, 20}. Hyperparameters were selected based BLEU score on the validation set.
Sources of idioms were pulled from language- learning websites below:
fi:
https://en.wikkiionrryyoom /Appendix:Fimiish idions
fr:
httpe://geeechl.nngcommmbb/////p
most-common- frencr--feehheheeee idioms-tii o o--ge--oo--tllliig--l-----
ive/
httpss//reenchtogettee...o/////
nch-idioms/
uny-french-idiomee nglish
The best hyperparameters were as follows: fi:
(A 0.4, temp = 10. num neighbours 20).
fr: (A = 0.2, temp 10. num neighbours 20),
ja:
ja: (A 0.4, temp 10, num neighbours 20)
During test time, the best hyperparameters for each language were used, with a probe size of 20 and beam size of 5.
om/japanese expressions and-i dioms/
Experiments with ALM and kNN-MT were implemented fairseq (version from September 2021) (Ott et al., 2019).
http:://en.wiiipeddd..ort Tapanese_proverbbs
http:://ww.ffuentttcooo/bbb
apentese/jaaannee-llioossss
D Native Annotator Recruiting
httpss://kotwwazzjiieeee jp/
OpenSubtitles Training Details
Annotators were hired through Upwork, and con- sisted of professional translators French and Finnish who were also fluent in English. Anno- tators were paid between $10 - 25USD an hour, depending on their individual hourly rate. No per- sonally identifying information collected.
A separate deltalm-base was trained from the pretrained model for 2M steps on each lan- guage. The Adam optimizer was used (Kingma and Ba, 2017), with lcarning rate of Ic-4, betas of (0.9, 0.98). and inverse square root learning rate scheduler with 4000 warmup updates (with a warmup learning rate of le-7, and minimum learning rate of Ic-9). The maximum number of tokens in a batch was set to 1024, with maximum
Annotators were shown the task description be- low, and also had access to compiled list of literal and figurative translations for each idiom.
Description:

---

This job is to create dataset that will allow us (researchers Carnegie Mellon University, Pl Graham Neubig) to study the ability of machine translation systems to translate idioms. We would like you to look translations and decide whether they ure literal or idiomatic translations. These will be English sentences containing the translation of sentence in cither French, Finnish, or Japanese. All the sentences contain phrase that is an idiom in the foreign language, but sometimes the translations may be meant literally. As example, in English "kick the bucket" idiomatic way.
Metric Score
0.32
0.59
0.30
0.00
0.06
0.87
0.2
0.6
0.8
0.8
Percentiles
Percentiles
die when used in an literally kick bucket when used in literal way. You should mark the example with "idiomatic" if you think it's idiomatic example, "literal" you think it's literal example.
ROUGE
METEOR
0:45
0.30
Metric
0.35
0.25
0.30
If you think the translation is good translation, will ask you to mark this entry with the word "None", but this should occur in relatively few cases.
0.20
0.25
Percentile idioomsii 05
Percentile idlioms
We estimate that job will take 2-3 hours overall.
Examples:
Figure 5: Effect of idiom frequency on translations by DeepL in Finnish
Original sentence: Jalkaväkeä oli pilvin pimein. English trans- lation (reference): Arty and infantry just kept on coming. Contains idiom: pilvin pimein Idiom literal meaning: dark clouds Idiom figurative meaning: huge (often excessive) amount of something. Label: idiomatic
in business documents. If part of sentence missing, based the error severity whether not the missing portion was crucial to the intent of the sentence. Examples of sentences labelled with error severity are provided in Table 6:
Original sentence: Harmi, ettei chhistttilaisuudessss. ollut lammasta. Fnglish translation (reference): Too they didn't 't have lamb at your press conference, Francis. Contains idiom: olla lammas Idiom literal meaning: To be lamb Idiom figura- tive meaning: a person who is like lamb does nothing alone. the person does Label: literal
F Statistical Significance Testing
E Standards for Human Evaluation
For each language and test set, examine the null hypothesis that the BLEU scores of the two systems actually have the same distribution, and the difference occurred by chance.
Evaluation standards were based on MQM stan- dards, in particular the major and critical severity levels, which are defined below (Lommel et al.. 2014):
For each source sentence, we shuffled the transla- tions produced by the two systems with probability 0.5, and then recalculated BLEU scores. This was repeated 1000 times, and the number of times the shuffled difference was greater or equal to the ob- served difference was recorded to find the p-value.
Major severity error: severity level of an er- ror that seriously affects the understandability. reliability, or usability of the content for its intended purpose or hinders the proper use of the product or service due to a significant loss or change in meaning or because the error appears in highly visible or important part of the content.
G Statistical Significance Results
Statistical significance results are shown in Table 7.
Critical severity level: severity level of an error that renders the entire content unfit for purpose or poses the risk for serious physical, financial, or reputational harm.
H Effect of Frequency Commercial Translations
The effect of idiom frequency on commercial mod- els' translations is in Figure 5 through Figure 9.
We slightly adapted this for idioms, where if was possible to infer the meaning of a sentence through the existence of similar English idiom, but it would not be something generally said by assigned major severity.
I Effect of Frequency on ALM
The effect of frequency on ALM for Finnish and Japanese is shown in Figure 10 and Figure
English speakers, Due ALM often making errors with numbers and named entities, we assigned these errors a major rather than critical severity in most cases, although these would generally be critical severity errors
J Automatic Metries in Detail
Exact results for automatic metrics shown in Table 8.

---

Language
Ref
Hyp
Severity
Eh oui, l'habit ne fait pas le moine.
Yes, clothes don't 't make monk.
fr
Yes, clothes don't 't make the
Major
11 l'a fait une queue de poisson sur I' T'autoroute.
He made a fish tail the highway.
He cut you on the freeway.
Severe
II pleut des cordes.
It's raining cats and dogs. He does nothing.
It's raining.
Major
pane tikkunkaan ristiin.
Doesn't even cross match.
Severe
Tämä meni yli hilseen.
This just way over my head.
This over top.
Major
Nice speech, Mr. Hobson. Way hit the suicide theme.
Hieno aasinsilta itsemurhaan.
It's nice morning for suicide.
Severe
# < th #
You need learn the element of surprise.
You need to learn to play hard-
ja
Severe
to-get.
I'm not falling for this shit.
I won't tt have to do that again.
Major
You swindled me!
You' 've given me enough to eat!
Severe
Table 6: Examples of categorized errors made by ALM.
BERTScome
Language
Test
p-val
0.34
0.886
ti
idioms
0.0*
30.13
literal
0.003*
Netk
0.882
rand
0.018*
0.12
0.880
fr
idioms
0.0*
literal
0,234
0.2
0.4
Percentiles
0.6
0.8
Percentiles
rand
0.0*
ROUGE
METEOR
ja
idioms 0.058
literal 0.083
Metric Score
0.44
rand
0.088
0.28
0.42
Table 7: p-values obtained using approximate random- ization on translations produced by the base model and the upweight+knn model.
0.40
0.27
0.4
Percentile idiommsii 05
Percentile idlioms
Figure Effect of idiom frequency on translations by Google in French
BLEU
0.90
BLEU
BERTScore
0.175
0.90
0.30
00.150
0.08
0.88
Metric
0.1225
Scote
0.89
0.0000
0.88
0.075
Percentlccs
Percontiles
ROUGE
METEOR
Percentiles
Percentlees
0.30
ROUGE
METEOR
Metric Store
0.300
Matric
0.35
0.25
0.40
0.275
0.20
0.35
0.200
0.25
0.2
Percentile laloms in OS
0.8
0.2
Percentile idionss in 05
0.4
0.6
0.8
0.30
0.2
0.4
0.6
0.8
0.225
0.2
0.4
0.6
0.8
Percentile idioms 05
Percentile of idlimms OS
Figure 6: Effect of idiom frequency on translations by
Google in Finnish
Figure Effect of idiom frequency on translations by Deepl. in Japanese

---

Language
Test Set
Model
BLEU
Rougel.-sum
BenScooe
Metcor
Idioms
Normal
0.1608
0.3737
0.9126
0.3592
Knn
0.179
0.3904
0.9152
0,3745
Upweight
0.1773
0,3895
0.9154
0.3709
Knn upweight
0.182
0,4027
0.9172
0,3833
Literal
Normal
0.2093
0.5053
0,9350
0.5050
Knn
0.2257
0.535
0.937
0.5161
Upweight
0.2314
0.5281
0.9364
0.5258
Knn upweight
0.2362
0.5335
0.9387
0.5217
Random-in
Normal
0.2056
0.5044
0.9321
0.4739
Knn
0.1991
0.5067
0.9316
0,4732
Upweight
0.2044
0,5048
0.9324
0.4728
Knn upweight
0.2120
0.5224
0.9351
0,488
Random-out
Normal 0.2365
0,535
0,9145
0.4971
Knn
0.2557
0,5602
0,9183
0,5178
Upweight
0.2374
0,5458
0,9163
0,5102
Knn + upweight
0.2498
0.564
0.9193
0.5261
Idioms
Normal
0.2001
0.4329
0,9211
0.4393
0.2154
0.4547
0,9235
0,4493
Upweight
0.2174
0.4398
0,9235
0,44199
Knn + upweight
0,2309
0,4568
0,926
0.4581
Literal
Normal
0.2778
0.5261
0,9377
0.5504
Knn
0.2824
0.5318
0.9387
0.5544
Upweight
0.2923
0,5332
0.9384
0,5564
Knn upweight
0.2883
0,5325
0.9399
0.5549
Random-in
Normal
0.3197
0.5942
0.9443
0.5803
Knn
0.3142
0,5949
0.9438
0.5774
Upweight
0.3079
0.5884
0.9427
0.5715
Knn upweight
0.3258
0.6036
0.9461
0.5865
Random-out
Normal
0.2778
0.528
0.9377
0.5504
Knn
0.3396
0.6573
0,928
0.6146
Upweight
0.3078
0.5458
0.9223
0.5856
Knn upweight
0.3353
0.652
0.9272
0.6075
ja
Idioms
Normal
0.09048
0.2826
0.9234
0,2998
Knn
0.09376
0.2767
0,9034
0,2905
Upweight
0.09505
0.285
0.9052
0,3022
Knn + upweight
0.09589
0.284
0.905
0,2982
Literal
Normal 0.1416
0.3951
0,9222
0.4222
0.1418
0.3947
0,9223
0,4183
Upweight
0.1427
0.3923
0,9221
0.4222
Knn + upweight
0.1509
0.3962
0.9226
0.4228
Random-in
Normal
0.1443
0.4008
0.9212
0.3927
Knn
0.1535
0.4036
0,9218
0.3960
Upweight
0.1430
0.3981
0,9212
0.3925
Knn upweight
0.1543
0.407
0.9224
0.4009
Random-out
Normal
0.0948
0.3539
0.8946
0.3436
Knn
0.0905
0.3389
0.8934
0.3284
Upweight
0.09228
0.3522
0.8947
0.3392
Knn upweight
0.09752
0.3545
0.8959
0.341
Table Results automatic metrics. most cases, combining loss weighting with KNN-MT improves automatic metrics the most all three test sets, including the out-of-distribution (Random) test set.

---

BERTScore
0.075
0.87
0.080
0.86
0.000
0.8
0.4
0.6
Percentiles
ROUGE
METEOR
0.250
0.225
0.35
0.30
0.175
0.150
Percentile idlooss
Percentile loloms 0S
Figure 9: Effect of idiom frequency on translations by Google in Japanese

---

BLEU
BERTScore
base
0.93
base
Metric Score
0.20
ukweight+knn
upweight+knn
0.92
Score
0.15
0.91
0.90
0.10
0.2
0.6
0.8
0.2
0.4 Percentiles
0.6
0.8
Percentiles
ROUGE
METEOR
0.5
0.5
base
Metric Score
upweight+knn
Score
0.4
0.4
base
0.3
0.3
upweight+knn
0.2
0.4
0.6
0.8
0.2
0.4
0.6
0.8
Percentile of idioms in OS
Percentile of idioms in OS
Figure 10: Translation metrics for Finnish idiomatic sentences, for base and upweight+knn models.
BLEU
BERTScore
base
0.91
base upweight+knn
0.15
Metric Score
upweight+knn
0.10
Score
0.90
0.05
0.89
0.00
0.2
0.4
0.6
0.8
0.2
0.4
0.6
0.8
Percentiles
Percentiles
ROUGE
METEOR
0.35
0.35
Metric Score
0.30
0.30
Score
0.25
0.25
0.20
base
base upweight+knn
upweight knn
0.20
0.2
0.4
0.6
0.8
0.2
0.4
0.6
0.8
Percentile of idioms in OS
Percentile of idioms in OS
Figure I1: Translation metries for Japanese idiomatic sentences, for base and upweight+knn models.


Reviews:
[{"review_id": "T9k4MJ1Hcl", "reviewer": "Reviewer_3Wyz", "summary": "", "questions": "- A. Can you add references for the up-weighting approach?\n- B. Can you add references for related synthetic data experiments?\n- C. What method is used to lemmatize the idioms?\n- D.  The best softmax temperature values and number of neighbours retrieved per query parameters of the Khandelwal et al., 2021 model are not all in your parameter search space, can you explain how you picked this hyperparameter search space?\n- E. Would it be possible to share the results obtain on the random set coming from the OpenSubtitles dataset that seems to be in the additional material?\n- F. How did you proceed to label the literal instances of the resource?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Hello, thank you for reading the paper so carefully and providing many suggestions! We’re glad that you appreciated the new resources as well as the synthetic experiment! We wrote a general response as well, and pasted relevant sections here. See below for responses to individual questions and comments as well:\n\n### General response \n\nWe thank all reviewers for their detailed responses and suggestions. We plan to update our manuscript based on some suggestions raised, and we also respond to some common concerns below:\n\n1. In-domain random sentences set\n\nSeveral reviewers have raised the fact that seeing the results for an in-domain set of randomly selected sentences would also be helpful. We agree with this. We initially thought that we could fulfill two criteria at once by choosing an out-of-domain random set, but we agree it is also important to see the in-domain results. \n\nBelow are automatic metrics results for a random test set (same size as the one in the paper), selected from the dev set of our opensubtitles data. We used the same process as in the paper to create the random test set, except using opensubtitles sentences. Unfortunately we don’t have access to the Japanese base model checkpoint right now, but we will also update this result with Japanese afterward, as well as conduct human evaluation on all three languages.\n\n|  language | metric | base | upweight | knn | upweight+knn |\n|---|---|---|---|---|---|\n| FI | BLEU | 0.2056 | 0.2044 | 0.1991 | **0.2120** |\n|  | RougeLSum | 0.5044 | 0.5048 | 0.5067 | **0.5224** |\n|  | Meteor | 0.4739 | 0.4728 | 0.4732 | **0.4880** |\n|  | Bertscore | 0.9321 | 0.9324 | 0.9316 | **0.9351** |\n| FR | BLEU | 0.3197 | 0.3079 | 0.3142 | **0.3258** |\n|  | RougeLSum | 0.5942 | 0.5884 | 0.5949 | **0.6036** |\n|  | Meteor | 0.5803 | 0.5715 | 0.5774 | **0.5865** |\n|  | Bertscore | 0.9443 | 0.9427 | 0.9438 | **0.9461** |\n\n2. Applicability of the synthetic experiment \n\nSeveral reviewers also raised concerns about the applicability of the synthetic experiment to real data. We would like to reiterate that this is meant to be a simple experiment by design and is not meant to capture the complexity of idioms in natural language. The motivation behind this experiment was to provide a simple testing ground so that factors such as corpus size, idiom frequency, sentence length, and informativity of the context could be easily manipulated. Past analysis in this area has focused on real idioms already (Shao et al 2018, Dankers et al. 2022), but this is based on a few examples, and it is not as easy to manipulate natural language data. This experiment was meant to be unambiguous and simple in order to complement the other two experiments conducted on natural language data.\n\nWe agree that further extensions can be made in order to make the setting more realistic, including manipulating informativity conditions further, creating multiple non-compositional rules and longer non-compositional ngrams, and manipulating the occurrence rate of tokens in non-compositional phrases outside the phrase. \n\n3. Annotation of literal and idiomatic sentences\n\nAnnotation of literal and idiomatic sentences was conducted by the authors, who had access to the idiom present in that sentence, as well as the literal and figurative translations of that idiom in the target language (english). As stated in section 6.3, the authors were fluent in Japanese and French, but not Finnish. \n\nWe will validate the literal/idiomatic judgments with external annotators in the camera-ready version, and report inter-rater reliability. \n\n### Comments\n#### Comment A\n> The authors do not really deal with potential idiomatic expressions that can be translated word-by-word or non compositionally depending on the context. They focus instead on idioms that are, in their large majority, impossible to translate word-by-word. The word-by-word translations are separated manually and form a small literal set for each language. It would be interesting to extend the experiments to more ambiguous expressions.\n\nThank you for pointing this out. We agree that there are many expressions which can be both idiomatic and ambiguous. This was based on idioms recorded in language-learning sites, which tend towards idioms that are not possible to translate word-for-word, as these may be expressions that language learners are more likely to search for. More ambiguous expressions were not focused on in this work, but we agree that this is an interesting avenue for future work.\n\n> The authors could eventually distinguish word-by-word from literal translations : if the same idiom is understandable in two languages it can be both figurative and word-by-word (page 3, line 197 says that literal instances of the test set only contains literal uses of the expressions, but it is not the case in the fr final literal test set, e.g. \"Moi, j'ai bien une idée... Vous donnez votre langue au chat ? Couper le courant.\").\n\nThanks for catching this example. We agree that some idioms are understandable in both source and target languages, perhaps because the same idiom exists in both languages, or because the expression depends on common-sense knowledge. For this particular example, the translation given as a reference was “Okay, okay, so, then, let me throw this out there and see if the cat licks it up”. Since the translation would be matched against this translation, it was categorized as literal since it didn’t contain a paraphrase of “I give up”. This is not the best translation of this sentence in general, but we chose to just categorize based on english translations for consistency.\n\n> It would be beneficial to study the diversity of translations found in the OpenSubtitles corpus for each given idiom. It would be informative to know if they are always translated the same way or if there is a high variability for this study.\n\nThank you for this suggestion, this is something that we can add to the camera-ready version. We were also concerned about the potential for multiple valid translations, so we emphasize that the human evaluation also considered semantically similar phrases as correct, e.g. for a reference of “This is just way over my head”, “this is confusing” is marked as correct.\n\n\n#### Comment B\n> Some subtitles in this dataset are generated by machine translation systems. The authors do not mention this fact, so I am not sure that they filter them out. If they do I don't know if the metadata indicating which subtitles have been generated automatically are always reliable.\n\nThanks for pointing this out as well! We used the version of opensubtitles from the huggingface datasets library, which does not have this metadata. However, we can take a look at the metadata and see if it is reliable/covers our languages sufficiently. If so, we can also divide the results by references that are machine translated or not.\n\n> I can read the created dataset in French as a native speaker, and the idiom matching function could be improved. It currently doesn't match the word boundaries ( e.g. The entry : \"rendre le âme, I take people's souls before they die., JE prends les âmes des gens avant qu'ils meurent.\" matches the verb \"prendre\" as an instance of the verb \"rendre\")\n\nOh, thank you for pointing this out. We can definitely improve the matching and exclude anything in the current test sets that doesn't match on word boundaries. The current sorting into idiomatic/literal was done by an author and an attempt was made to exclude these cases, but we may have missed a few sentences like this.\n\n#### Comment C\n> The synthetic data framework is interesting but very simplified. I would like to see what happens with more than one compositional rule and what happens when the context decides if the expression must be translated literally or metaphorically.\n\nSee general response, item 2. I think exploring ambiguous expressions and multiple non-compositional rules would be interesting for future work.\n\n#### Comment D\n> The sizes of the test sets of the \"literal\" translations in the 3 languages are a little small to reach a strong significance in the evaluation.\n\nWe agree that the literal test sets are relatively small, but nonetheless we conducted significance tests, which are covered in appendix E and F. Significance (\\alpha \\leq 0.05) was reached for the fi literal set, but not for the other two. It is a bit hard to expand the literal test sets because most of the idioms that we collected do not have a plausible literal interpretation, or the literal interpretation does not appear in OpenSubtitles. \n\n#### Comment E\n\n> In the files shared by the authors, there is a random test set of the OpenSubtitles corpus. I would be curious to know the automatic metrics scores on this one, it would help in the interpretation of the overall results of the experiments. Domain adaptation is interesting but I think that having results on a random set drawn from the same distribution is important here.\n\nSee general response, item 1.\n\n### Answers to Questions\n\n> A, B: Can you add references for the up-weighting approach? Can you add references for related synthetic data experiments?\n\nThank you for the suggestion, we plan to add these references for instance weighting in MT:\n\n- Instance Weighting for Neural machine translation domain adaptation, Wang et al. 2017\n- Discriminative Instance Weighting for Domain Adaptation in Statistical Machine Translation, Foster at al. 2010\n\nFor the synthetic data experiment, we are not aware of any similar experiments in MT. There are experiments on memorization of idioms in the Dankers et al 2022 and Haviv et al 2022 papers cited in the manuscript, but these are on english idioms.\n\n> C: What method is used to lemmatize the idioms?\n\nThe stanza english lemmatizer was used. We will add a citation to stanza in the updated manuscript.\n\n> D: The best softmax temperature values and number of neighbours retrieved per query parameters of the Khandelwal et al., 2021 model are not all in your parameter search space, can you explain how you picked this hyperparameter search space?\n\nThe use case in Khandelwal et al 2021 is quite different from ours, although we were inspired by that paper. Since many idioms occur relatively rarely in the training set, we set the max number of neighbours to be 20, rather than 64 (the default in the original kNN-MT paper). Other hyperparameters were just set to reasonable grid searches, such as $\\lambda$ and temperature, as outlined in Appendix C.\n\n> E: Would it be possible to share the results obtain on the random set coming from the OpenSubtitles dataset that seems to be in the additional material?\n\nSee general response, item 1.\n\n> F. How did you proceed to label the literal instances of the resource?\n\nSee general response, item 3.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The paper is very clearly written, the level of details provided is mostly sufficient, the additional material (code and data) is also a clear contribution, and the references given are solid and interesting. There is enough new material for a long paper.\n- Retrieval augmentation has never been used in relation to idiomatic expressions processing to my knowledge, and is a promising approach.\n- Although I thought this experiment was too limited, experimenting using synthetic data for studying of compositional rules learning is an inspiring idea (I may be unaware of previous work using similar method, cf Questions for Authors).\n- The experiments show that the method improves the translation of idiomatic expressions.\n\n", "reasons_to_reject": "A. Contextual translation : line 053 \"However, we would ideally want to make use of the contextual translation abilities of neural models.\" \n- The authors do not really deal with potential idiomatic expressions that can be translated word-by-word or non compositionally depending on the context. They focus instead on idioms that are, in their large majority, impossible to translate word-by-word. The word-by-word translations are separated manually and form a small literal set for each language. It would be interesting to extend the experiments to more ambiguous expressions.\n- The authors could eventually distinguish word-by-word from literal translations : if the same idiom is understandable in two languages it can be both figurative and word-by-word (page 3, line 197 says that literal instances of the test set only contains literal uses of the expressions, but it is not the case in the fr final literal test set, e.g. \"Moi, j'ai bien une idée... Vous donnez votre langue au chat ? Couper le courant.\").\n- It would be beneficial to study the diversity of translations found in the OpenSubtitles corpus for each given idiom. It would be informative to know if they are always translated the same way or if there is a high variability for this study.\n\nB. Created resource from Open Subtitles (Lison et al., 2018)\n- Some subtitles in this dataset are generated by machine translation systems. The authors do not mention this fact, so I am not sure that they filter them out. If they do I don't know if the metadata indicating which subtitles have been generated automatically are always reliable.\n-  I can read the created dataset in French as a native speaker, and the idiom matching function could be improved. It currently doesn't match the word boundaries ( e.g. The entry : \"rendre le âme, I take people's souls before they die., JE prends les âmes des gens avant qu'ils meurent.\" matches the verb \"prendre\" as an instance of the verb \"rendre\")\n\n\nC. The synthetic data framework is interesting but very simplified. I would like to see what happens with more than one compositional rule and what happens when the context decides if the expression must be translated literally or metaphorically.\n\nD. Evaluation\n- The sizes of the test sets of the \"literal\" translations in the 3 languages are a little small to reach a strong significance in the evaluation.\n- In the files shared by the authors, there is a random test set of the OpenSubtitles corpus. I would be curious to know the automatic metrics scores on this one, it would help in the interpretation of the overall results of the experiments. Domain adaptation is interesting but I think that having results on a random set drawn from the same distribution is important here.\n\n", "questions_to_authors": "- A. Can you add references for the up-weighting approach?\n- B. Can you add references for related synthetic data experiments?\n- C. What method is used to lemmatize the idioms?\n- D.  The best softmax temperature values and number of neighbours retrieved per query parameters of the Khandelwal et al., 2021 model are not all in your parameter search space, can you explain how you picked this hyperparameter search space?\n- E. Would it be possible to share the results obtain on the random set coming from the OpenSubtitles dataset that seems to be in the additional material?\n- F. How did you proceed to label the literal instances of the resource?", "paper_topic_and_main_contributions": "This paper describes methods for improving the idiom translations generated by transformer-based machine translation systems. Idiomatic expressions are challenging for machine translation systems because they often cannot be translated word-by-word and they may occur rather infrequently in training data. The paper approach is inspired by Kandpal et al. (2022) analysis of the long-tail knowledge learning issues shown by language models : it studies the effect of retrieval augmentation and training objective modification on the quality of idiomatic expressions translations.\n\nAfter an initial study of the ability of encoder-decoder models to learn a non compositional rule occurring with various frequencies in the training data using synthetic data, the main experiments are ran on a created resource of 4k natural sentences containing idiomatic expressions in French, Finnish and Japanese, extracted from the OpenSubtitles corpus. \n\nFirst the authors evaluate two commercial systems on the created dataset, and then evaluate their approach.\nModifying a recent strong translation model (Ma et al., 2021), the authors evaluate (1) the effect of the up-weighing of the training loss of sentences containing potential idiomatic sentences, (2) the effect of augmenting the model with a retrieval component (Khandelwal et al., 2021) and (3) and the combined effects of (1) and (2) on the quality of the translation of the idioms. Evaluations are conducted with both automatic scoring metrics and human ratings.\n(3) is found to improve the translation of sentences containing idioms by an average of 10.4% in absolute accuracy, with human ratings, while not harming the performance the model for other sentences. The authors show the effectiveness of the proposed methods for idioms occurring with various relative frequencies in the parallel training data.\n\nThe authors shared the code and data created for the experiments.\n", "excitement": 4, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "-  The reference to Unbabel (2019) lacks a URL\n- Some papers are cited without the conference or journal they were accepted in, in the reference section. The authors could eventually complete those references.", "Typos_Grammar_Style_and_Presentation_Improvements": "- Some information of section 6.2 about the OpenSubtitles corpus size would be interesting in Section 3.\n- Page 3, line 198, I struggled to understand the sentence: \" The first 100 examples containing each idiom’s lemmatized form were collected, and up to the first 3 (for Japanese) or 5 (for Finnish and French) literal and figurative examples in this set were collected.\".  Is \"to create the test set\" missing at the sentence end?\n- Page 11, line 785  : \"without context\" should be \"without informative context\"?\n- Page 11 line 789 : \"15, 15 and 25\" should be \"15, 20 and 25\"?"}}, {"review_id": "79HXc3yB37", "reviewer": "Reviewer_ZCwa", "summary": "", "questions": "Question A: Regarding the use of kNN-MT, was the retrieval limited to the idiomatic phrase or used for every word in the sentence (i.e., vanilla kNN-MT from Khandelwal et al., 2021)?\n\nQuestion B: Regarding Figure 1, do you have thoughts why the large transformer seems to do better with the uninformed context than the informed context (with a high number of idioms)?  Note that in several cases the medium and/or small transformer does better.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review, we are glad that you appreciate the methods used in our study! See below for answers to questions:\n\n### Answers to Questions\n\n> Question A: Regarding the use of kNN-MT, was the retrieval limited to the idiomatic phrase or used for every word in the sentence (i.e., vanilla kNN-MT from Khandelwal et al., 2021)?\n\nkNN-MT was used in the usual way, for every word in the sentence. It may be possible to use kNN-MT only on idiomatic phrases for increased efficiency, but we leave this for future work.\n\n> Question B: Regarding Figure 1, do you have thoughts why the large transformer seems to do better with the uninformed context than the informed context (with a high number of idioms)? Note that in several cases the medium and/or small transformer does better.\n\nI am not sure if this is a general pattern, can you point out the cells that you see this pattern in and maybe I can comment more specifically? In general though, the large transformer tended to have more variance in training runs, especially with a larger number of idioms. This may be due to overfitting which gets worse with a greater number of non-compositional examples, but I’m also not completely sure.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- careful analysis of problem in existing models\n\n- theoretical study with synthetic data pointing toward solution, then use of simple techniques to provide the direction of a solution\n\n- lots of analysis and multiple evaluations of results", "reasons_to_reject": "none\n", "questions_to_authors": "Question A: Regarding the use of kNN-MT, was the retrieval limited to the idiomatic phrase or used for every word in the sentence (i.e., vanilla kNN-MT from Khandelwal et al., 2021)?\n\nQuestion B: Regarding Figure 1, do you have thoughts why the large transformer seems to do better with the uninformed context than the informed context (with a high number of idioms)?  Note that in several cases the medium and/or small transformer does better.", "paper_topic_and_main_contributions": "This paper does an analysis of idiomatic MT using both synthetic language and real language, including constructing a new multilingual corpus for testing the behavior.  The paper observed that very frequent idioms are learned well, but less frequent idioms are more difficult, then suggests two techniques that should help increase the visibility of less frequent idioms (up-weighting and kNN-MT).  The results and subsequent error analysis and informative and convincing that the authors are looking in the right direction. This paper does not present a full solution to problems with idiomatic translation, but is a solid step in the process.\n", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "- line 13: tilde is above 4 in 4k, did you mean ~4k?\n\n- line 95: in improves -> it improves\n\n- equation 3: The notation after the alpha is very hard to interpret due to super/subscripting.  After reading the text (lines 333-338), I believe the alpha term is being raised to the power of 1 if x(i) is in A and 0 if not?  So this is an indicator function for A?  Now I can see it, but I suggest changing it for easier interpretation.\n\n"}}, {"review_id": "qfryk8uWwq", "reviewer": "Reviewer_dV8h", "summary": "", "questions": "- Could you elaborate on the procedure used to label PIE usage in Opensubtitles as idiomatic or literal? How many annotators were involved, were they native speakers of English and Japanese/Finnish/French? What was the inter-annotator disagreement level? UPDATE AFTER REBUTTAL: apologies for missing the comment about speakers being native in the paper\n- Could you elaborate on what, in your opinion, we can learn about real idioms and how frequent they need to be based on the synthetic experiment that you conducted in light of the simplifications I pointed out above?\n- Could you reflect on the extent to which the upweighting and knn-MT methods are idiom-specific?", "limitations": "", "rating": 2, "confidence": 4, "soundness": 2, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your comments, and we’re glad that you’ve taken the time to review our manuscript thoroughly. We are glad that you appreciate the work as a contribution to figurative language processing. We wrote a general response as well, and pasted relevant sections here. See below for responses to individual questions and comments as well:\n\n### General response \n\nWe thank all reviewers for their detailed responses and suggestions. We plan to update our manuscript based on some suggestions raised, and we also respond to some common concerns below:\n\n2. Applicability of the synthetic experiment \n\nSeveral reviewers also raised concerns about the applicability of the synthetic experiment to real data. We would like to reiterate that this is meant to be a simple experiment by design and is not meant to capture the complexity of idioms in natural language. The motivation behind this experiment was to provide a simple testing ground so that factors such as corpus size, idiom frequency, sentence length, and informativity of the context could be easily manipulated. Past analysis in this area has focused on real idioms already (Shao et al 2018, Dankers et al. 2022), but this is based on a few examples, and it is not as easy to manipulate natural language data. This experiment was meant to be unambiguous and simple in order to complement the other two experiments conducted on natural language data.\n\nWe agree that further extensions can be made in order to make the setting more realistic, including manipulating informativity conditions further, creating multiple non-compositional rules and longer non-compositional ngrams, and manipulating the occurrence rate of tokens in non-compositional phrases outside the phrase. \n\n3. Annotation of literal and idiomatic sentences\n\nAnnotation of literal and idiomatic sentences was conducted by the authors, who had access to the idiom present in that sentence, as well as the literal and figurative translations of that idiom in the target language (english). As stated in section 6.3, the authors were fluent in Japanese and French, but not Finnish. \n\nWe will validate the literal/idiomatic judgments with external annotators in the camera-ready version, and report inter-rater reliability. \n\n### Comments\n\n> Regarding the synthetic experiment: It is impossible to tell to what extent the findings from the artificial language translation experiment generalise to natural data, where non-compositional translations are much more complex. To name 3 reasons: 1) idioms have various conventionalities (~ratio between idiomatic vs literal meaning) and the frequency above which models default to the non-compositional translation likely interacts with conventionality, 2) words and n-grams contained in the idioms themselves can appear outside of the idiom, 3) idioms require disambiguation unless the idiomatic meaning is 100% conventional, 4) many idiomatic translations can be partially compositional. The artificial experiment is interesting in itself, but to assume that this is a proxy for idiom processing seems like a stretch.See general response, item 2. To respond to the points more directly, the synthetic experiment is not meant to capture all the nuances of idioms in natural language, as noted in the last paragraph of section 2.2, this experiment is moreso meant to capture non-compositionality in a very controlled setting, and the current definition includes things like multi-word expressions, while not currently dealing with ambiguity in language. The artificial setting is not necessarily as restrictive as we presented it, and can be extended to model other phenomena as outlined below. As there is, to our knowledge, no existing literature on evaluating non-compositionality handling in a fully controlled context, we emphasize that we are starting with a single simple case, but the approach can be extended to capture important factors like the ones that you have mentioned:\n\n1. Conventionality - this is true for natural language idioms, and is something that we could potentially analyze in the opensubtitles results. However, to do this we would potentially have to look at much more data than opensubtitles, since many idioms don’t appear at all in their literal form in opensubtitles. We can update the manuscript with conventionality (figurative/literal counts), but a full investigation may be outside the scope of this paper. \n\n2. This is technically true for the one non-compositional expression as well, as the unigrams “0” and “1” also appear separately outside the phrase. The proportion of times “0” or “1” can appear outside of the phrase “0 1” can also be controlled, and we can consider adding this to the manuscript. The synthetic experiment can also be extended to include longer non-compositional phrases, but a two-token phrase is the simplest case. \n\n3. Yes, this is true in many cases. In the opensubtitles data, we can also add conventionality based on counts for each idiom as stated in point 2. Because the datasets generated are fully controllable, the informativity of the context (ease of disambiguation) can also be manipulated directly, since P(idiomatic translation|ctx) can be calculated directly. \n\n4. This is true, but to reduce ambiguity, we treat partially compositional phrases as non-compositional since they cannot be completely reconstructed. We agree that this is an important factor to consider in future work though.\n\n> Regarding the new dataset: While the newly created dataset could potentially be a very useful resource (idiom analyses are predominantly using English corpora), the paper is not very elaborate about how the authors ensure quality control for this corpus. Who annotates the Opensubtitles sentences for idiomaticity? The paper is slightly vague about this, which suggests the authors may have annotated this. When presenting a new dataset meant to be used by future work, the proper way to construct that would be using external annotators, ideally multiple annotators per example to be able to estimate reliability of the annotations and measures of disagreement. Moreover, the corpus is constructed using idioms from a few websites, instead of from idioms taken from idiom dictionaries.\n\n\nThank you for raising this concern. The authors did annotate whether or not a sentence contained idiomatic statements with respect to the reference translation, while having access to a literal translation of the idiom as well as the idiomatic translation in English. In order to validate the literal/idiomatic split, we plan to hire external annotators to annotate a subset, and measure agreement between our judgments and the external annotators. \n\nFor data, our data sources do already include Wikitionary (for Finnish) and Kotozawa (a japanese dictionary of proverbs and idioms). However, language learning sites also contain some idioms that are not present in dictionaries. A list of data sources is available in the linked repository under `metaphor-translation/data/external/idiom_sources.txt`. If you have suggestions for online idioms dictionaries for any of our languages, please let us know.\n\n> Regarding the proposed upweighing and KNN methods: For the majority of language and score combinations (see Figure 3), the impact that the methods have on idiomatic vs random data is similar; hence the proposed MT modelling methods seem far from idiom-specific. Therefore, the results simply appear to indicate that \"better NMT systems are also better at idiomatic translations\".\n\nAlthough it is true that a better NMT model will likely be better at idiomatic translation, some idiom-specific methods such as sentence weighting improve this further, given a base model. In Table 4, upweighting on its own does improve translation of idiomatic sentences. We expected kNN-MT to benefit idiomatic sentences more than random sentences, but in practice we found that the effect was more uniform across test sets. kNN-MT based on an unweighted model outperforms vanilla kNN-MT in many cases, indicating that idiom-specific upweighting does help the model even when it has been improved with kNN-MT already.\n\n> Across the board, the paper appears to lack focus and a single, clear storyline. The experiments seem somewhat disconnected, which makes it hard to read and understand what the main contributions are.\n\nWe apologize if the paper is somewhat disorganized. We can make the goals of each section more clear and explicit, such as emphasizing that the synthetic experiment is about non-compositionality in a controlled environment. If you have any additional suggestions for clarity, please let us know.\n\n### Answers to Questions\n\n> Could you elaborate on the procedure used to label PIE usage in Opensubtitles as idiomatic or literal? How many annotators were involved, were they native speakers of English and Japanese/Finnish/French? What was the inter-annotator disagreement level?\n\nSee general response, item 3. In addition, the authors are fluent speakers, but not native, in French and Japanese in section 6.3.\n\n> Could you elaborate on what, in your opinion, we can learn about real idioms and how frequent they need to be based on the synthetic experiment that you conducted in light of the simplifications I pointed out above?\n\nAs stated before, the synthetic experiment was not supposed to capture the nuances of real idioms, but to provide insight into non-compositional patterns in a controlled setting. \n\n> Could you reflect on the extent to which the upweighting and knn-MT methods are idiom-specific?\n\nUpweighting is very idiom specific as only potentially idiomatic phrases were upweighted, as stated in the paragraph on upweighting in section 5. \nkNN-MT is not idiom-specific, but is expected to help models more with rare patterns/entities as reported in the Khandelwal et al 2021 paper (on kNN-LM). When combined with upweighting, it does become more idiom specific as the representations used to build the datastore would be those of the model trained with upweighting.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The paper presents a very useful dataset of idiomatic translations for French, Finnish and Japanese. As this paper points out, and as previous papers have done, NMT systems still struggle with idioms, partially due to their low frequency. The majority of the resources currently used is in English, and having data for these three additional languages could support analyses in future work.\n- The paper presents evidence for the role of frequency in the acquisition of idiomatic translations, both in a synthetic experiment and using real data. While the role of frequency has been assumed by previous work, few articles make this explicit, which is a meaningful contribution to have for the subcommunity interested in figurative language processing in NLP.", "reasons_to_reject": "- Regarding the synthetic experiment: It is impossible to tell to what extent the findings from the artificial language translation experiment generalise to natural data, where non-compositional translations are much more complex. To name 3 reasons: 1) idioms have various conventionalities (~ratio between idiomatic vs literal meaning) and the frequency above which models default to the non-compositional translation likely interacts with conventionality, 2) words and n-grams contained in the idioms themselves can appear outside of the idiom, 3) idioms require disambiguation unless the idiomatic meaning is 100% conventional, 4) many idiomatic translations can be partially compositional. The artificial experiment is interesting in itself, but to assume that this is a proxy for idiom processing seems like a stretch.\n- Regarding the new dataset: While the newly created dataset could potentially be a very useful resource (idiom analyses are predominantly using English corpora), the paper is not very elaborate about how the authors ensure quality control for this corpus. Who annotates the Opensubtitles sentences for idiomaticity? The paper is slightly vague about this, which suggests the authors may have annotated this. When presenting a new dataset meant to be used by future work, the proper way to construct that would be using external annotators, ideally multiple annotators per example to be able to estimate reliability of the annotations and measures of disagreement. Moreover, the corpus is constructed using idioms from a few websites, instead of from idioms taken from idiom dictionaries.\n- Regarding the proposed upweighing and KNN methods: For the majority of language and score combinations (see Figure 3), the impact that the methods have on idiomatic vs random data is similar; hence the proposed MT modelling methods seem far from idiom-specific. Therefore, the results simply appear to indicate that \"better NMT systems are also better at idiomatic translations\".\n- Across the board, the paper appears to lack focus and a single, clear storyline. The experiments seem somewhat disconnected, which makes it hard to read and understand what the main contributions are.", "questions_to_authors": "- Could you elaborate on the procedure used to label PIE usage in Opensubtitles as idiomatic or literal? How many annotators were involved, were they native speakers of English and Japanese/Finnish/French? What was the inter-annotator disagreement level? UPDATE AFTER REBUTTAL: apologies for missing the comment about speakers being native in the paper\n- Could you elaborate on what, in your opinion, we can learn about real idioms and how frequent they need to be based on the synthetic experiment that you conducted in light of the simplifications I pointed out above?\n- Could you reflect on the extent to which the upweighting and knn-MT methods are idiom-specific?", "paper_topic_and_main_contributions": "This paper addresses the challenging task of idiom translation, and performs multiple experiments related to that:\n1. Firstly, an experiment is conducted that uses synthetically generated data of strings of numbers, to examine what proportion of non-compositional expressions is required for the system to be able to learn that non-compositional translation. An example 'sentence' would be \"0 3 4 7 0 1\" --> \"10 13 14 17 12\", where \"0 1\" are the non-compositional mapping. Transformer models of varying sizes are trained corpora of varying sizes. Approximately 10% of non-compositional data was needed for the models to learn the non-compositional mapping.\n2. A new dataset is created by taking French, Finnish and Japanese idioms, and matching them with Opensubtitles sentences. The sentences are sorted into literal and figurative examples, using 3-5 examples per idiom, for literal and idiomatic occurrences.\n3. Using the newly created dataset, two commercial translation systems (DeepL and Google Translate) and one LM (LM-base) are evaluated, which provides convincing evidence that these systems, too, have subpar performance on idiomatic inputs (when compared to literal occurrences of PIEs and random data from the TED talks corpus). Idioms that occurred more frequently in Opensubtitles also have better translations.\n4. Finally, the paper proposes two modelling techniques that can improve idiom processing. 1) Upweighting sentences that contain PIEs, 2) knn-MT. For Finnish and French clear improvements are observed for the two methods, and the two methods combined. For Japanese, the results are a bit less consistent: combining the two typically performs similar to normal training, but only applying one of the two slightly decreases performance. Across the board, the results are quite similar for the literal, idiomatic and random data.\n5. The paper ends with an analysis of 1) the frequency of idiomatic translations to show that the upweighting and knn-MT improve performance for both low- and high-frequency idioms, and 2) an error analysis, where \"major\" and \"severe\" errors are annotated", "excitement": 2, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "- Table 5: Perhaps put the tables/figures at the top of the page rather than in the middle of running text?\n- line 469: \"they tended to contain named entities\", This is something that can be easily supported with factual information: perform NER and report the relative frequencies of named entities in the sentences for which the performance did/did not drop.\n- line 465: \"it's\" --> \"it is\""}}, {"review_id": "CuEJ7hghB4", "reviewer": "Reviewer_23HH", "summary": "", "questions": "- Clarification on the synthetic setup. ", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Hello, thank you for your review, we’re glad that you appreciate the results of the paper. We wrote a general response as well, and pasted relevant sections here. See below for responses to individual questions and comments as well:\n\n### General response \n\nWe thank all reviewers for their detailed responses and suggestions. We plan to update our manuscript based on some suggestions raised, and we also respond to some common concerns below:\n\n2. Applicability of the synthetic experiment \n\nSeveral reviewers also raised concerns about the applicability of the synthetic experiment to real data. We would like to reiterate that this is meant to be a simple experiment by design and is not meant to capture the complexity of idioms in natural language. The motivation behind this experiment was to provide a simple testing ground so that factors such as corpus size, idiom frequency, sentence length, and informativity of the context could be easily manipulated. Past analysis in this area has focused on real idioms already (Shao et al 2018, Dankers et al. 2022), but this is based on a few examples, and it is not as easy to manipulate natural language data. This experiment was meant to be unambiguous and simple in order to complement the other two experiments conducted on natural language data.\n\nWe agree that further extensions can be made in order to make the setting more realistic, including manipulating informativity conditions further, creating multiple non-compositional rules and longer non-compositional ngrams, and manipulating the occurrence rate of tokens in non-compositional phrases outside the phrase. Below are responses to individual questions and comments:\n\n### Comments\n\n> The analysis on synthetic translation experiments don't seem valid to me. The authors assume a specific ordering in translated words as well as a paradigm where every word is translated into one other word. I'm suspecting what they analyze in such a case is not non-compositionality in language but a specific and simple case of pattern recognition.\n\nYes, we agree that non-compositionality in language is actually more of a spectrum, and the synthetic experiment in fact is meant to test a case of pattern recognition. Namely, the recognition of a non-compositional example when compositional translations are more common. We do state that this is meant to be a simple case, and is a complement to the later experiments with real idioms.\n\n> While the paper introduces new datasets, the novelty of the work is not acceptable for me. The methods used in this work have already been existing and applying them to a new problem, while useful, might not have enough contribution for a long paper.\n\nAlthough we agree that the methods that we use already existed and had been applied in other settings, we think that the contribution may still provide value to the community. Namely, idiom translation has been a problem for a long time, and providing some simple methods that boost performance is still of value to the community if it has not been done before.\n\n### Answers to Questions\n\n> Clarification on the synthetic setup.\n\nSee general response, item 2.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Idiom translation is an open question in the field. \n- Their proposed approach of using upweight in addition to knn-MT seem to be working quite well.\n- The paper introduces a new dataset for idiom translation for 3 languages.", "reasons_to_reject": "- The analysis on synthetic translation experiments don't seem valid to me. The authors assume a specific ordering in translated words as well as a paradigm where every word is translated into one other word. I'm suspecting what they analyze in such a case is not non-compositionality in language but a specific and simple case of pattern recognition. \n\n- While the paper introduces new datasets, the novelty of the work is not acceptable for me. The methods used in this work have already been existing and applying them to a new problem, while useful, might not have enough contribution for a long paper. ", "questions_to_authors": "- Clarification on the synthetic setup. ", "paper_topic_and_main_contributions": "The paper on idiom translation is a well-written and well executed piece of research. The authors present a comprehensive study that delves into the impact of idiom phrase frequency in training data on model learning. The synthetic evaluation proposed by the authors sheds light on an essential aspect of idiom translation. By examining how the frequency of idiom phrases influences the model's ability to translate them accurately, the authors offer valuable insights into the intricacies of idiomatic language processing.\nThey incorporate two existing methods on their datasets to see if they can improve the translation quality of idioms. The experiments conducted in the study are sound and provide strong evidence to support the conclusions drawn by the authors. \n", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: Four reviewers gave the Soundness scores 3/4/2/4 and Excitement Scores 4/4/2/3. The authors have responded to all four reviewers and all reviewers acknowledged the author responses.
All reviewers recognise the contribution of this paper on addressing idiomatic MT, one of the bottlenecks in MT field, especially  in French, Finnish, and Japanese. 
Majority of the reviewers suggest this is a good and solid work on Soundness, but one reviewer raised many concerns regarding the synthetic experiment, dataset quality control, methods justification, and paper coherence. 
There are active discussions between reviewers and authors, especially on comments from the lower review scores.
Overall this is an interesting paper with solid work. It shall appear in main / findings. 
Appendix content can be reorganised to save spaces and reduce number of pages. I hope the authors can address this.
Decision: Accept-Main