Dictionary-Aided Translation for Handling Multi-Word Expressions in Low-Resource Languages (2024)

Abstract: Multi-word expressions (MWEs) present unique challenges in natural language processing (NLP), particularly within the context of translation systems, due to their inherent scarcity, non-compositional nature, and other distinct lexical and morphosyntactic characteristics, issues that are exacerbated in lowresource settings. In this study, we elucidate and attempt to address these challenges by leveraging a substantial corpus of humanannotated Greek MWEs. To address the complexity of translating such phrases, we propose a novel method leveraging an available outof-context lexicon. We assess the translation capabilities of current state-of-the-art systems on this task, employing both automated metrics and human evaluators. We find that by using our method when applicable, the performance of current systems can be significantly improved. However, these models are still unable to produce translations comparable to those of a human speaker.1

heading: 1 Introduction
text: Multi-word expressions (MWEs) are defined as combinations of at least two words which exhibit lexical, morphological, syntactic, semantic or statistical idiomaticity (Baldwin and Kim, 2009). Processing MWEs has long been considered one of the most challenging tasks in natural language processing (NLP) (Ramisch et al., 2010), especially when it concerns translation systems.
Despite neural models outclassing more traditional NLP techniques in many areas, the distinct nature of MWEs makes it so that the generalizing abilities inherent in these approaches cannot significantly increase performance. There is also currently a significant lack of datasets and benchmarks in this domain, especially for low-resource
1We publicly release all code and datasets produced for this work: github.com/andhmak/dictMWE_MT
languages. Faithful automatic translation of MWEs is particularly important in improving the performance of general translation systems, given that MWEs make up a significant amount (estimates range from 11% to 40%) of the vocabulary in any piece of text (Sag et al., 2002; Candito et al., 2021).
In this work, we use a large corpus of humanannotated MWEs in Greek and we measure the translation ability of current state-of-the-art systems on sentences containing these MWEs, both automatically and with the help of human evaluators. We focus on two tasks; standard "unaided" translation, where the model is only provided with the source sentence, and an "aided" version, where we provide additional guidance on the translation of the MWE used in the source sentence, taking advantage of the available lexicon. An example of both tasks is shown in Figure 1. In short, we make the following contributions:
• We propose a new method for machine translation of in-context MWE using an out-ofcontext lexicon.
• We evaluate the ability of current state-of-theart translation systems in handling text containing MWEs, using a large and general test set.
2588

heading: 2 Related Work
text: Previous research has explored the complexities of processing MWEs in various tasks, including MWE detection (Lai et al., 2023) as well as translation-specific settings (Kabra et al., 2023), such as in contrastive examples or metaphors, highlighting the need for specialized approaches. While neural models have excelled in many NLP tasks, their performance with MWEs remains suboptimal. In certain sub-areas, such as detection, traditional non-neural statistical methods remain remarkably competitive (Constant et al., 2017; Pasquer et al., 2020). This is theorized to be due to MWEs having a lexical quality, meaning that they behave similarly to out-of-vocabulary words when they do not appear in their idiomatic usage in the training data (Savary et al., 2019). This problem is therefore exacerbated disproportionately more for lower-resource languages, such as Greek.
There has also been research in the domain of dictionary-aided neural machine translation in a general setting, concerning single one-word lemmas (Zhang et al., 2021). Older methods have also attempted to leverage MWE dictionaries by incorporating them into the non-neural, statistical translation pipeline (Bungum et al., 2013), while newer ones have tried adding the dictionary to the training data of neural models (Arthur et al., 2016; Zaninello and Birch, 2020). Our work borrows ideas from the efforts towards terminology translation (Alam et al., 2021), which usually incorporates hard (Dinu et al., 2019; Susanto et al., 2020), usually through constrained decoding (Post and Vilar, 2018; Hu et al., 2019), or soft constraints (Bergmanis and Pinnis, 2021) during decoding. Similar techniques have also been employed for translating rare words (Ghazvininejad et al., 2023) and terminology in specific domains (Moslem et al., 2023; Bogoychev and Chen, 2023).
In contrast, our method leverages the prompting capabilities of Large Language Models to imbue them with information from a MWE dictionary (somewhat similar to the soft constraints) while still taking advantage of their generalizing abilities.

heading: 3 Methodology
text: Our evaluation focuses on two tasks: standard ("unaided") and "aided" translation.
Unaided Translation In this task, the model generates an output solely based on the provided
source sentence in a traditional manner.
Aided Translation This task is based on our proposed method. The model receives each lone MWE and its human-generated translations along with the source sentence.
We already know from previous work (Baziotis et al., 2023) that, at least in the current landscape, it is not possible to reasonably translate passages containing MWEs without them appearing in the training data or a lexicon. By providing this lexicon in a simple manner as part of the prompt, we aim to measure the improvement in the output.

heading: 3.1 Dataset
text: We rely on a comprehensive corpus of humanannotated Greek MWEs, providing a diverse and representative set for evaluating translation models on this specific task.
Specifically, we use the IDION dataset of Greek MWEs (Markantonatou et al., 2019). This dataset provides a large set of Greek verb MWEs in dictionary form, as well as (possibly multiple) contextunaware translations of these phrases. It also contains 5750 Greek sentences, without an English translation, matched to one of 916 MWEs. These sentences have all been validated by linguists as being grammatically correct and representative of each MWE, from a larger set of Greek-language data gathered from the internet. These 5750 sentences form our MWE-focused test set.

heading: 4 Experiments
text: We use two state-of-the-art models for our experiments. The first is No Language Left Behind (NLLB; NLLB Team et al., 2022), a state-of-theart multilingual model specializing in low-resource languages. We specifically use the 600M parameter version due to computational resource constraints. The second is GPT-4-0613 (OpenAI et al., 2023), a large language model operating as a chatbot.
We then evaluate the various translations, both in absolute and comparative quality, in different ways, both manually and automatically.
Unaided Translation Under the unaided translation setting, we benchmark both models—NLLB and GPT-4.
The NLLB model was provided simply with the source sentence (in Greek), the source language (Greek), and the target language (English). GPT-4 was provided with the following prompt:
Can you translate the Greek text "[source sentence]" into English?
Aided Translation Our aided translation proposed method relies on the prompting capabilities of LLMs, hence it is only applicable to GPT-4 and not to NLLB. We now provide GPT-4 with the following prompt:
Can you translate the Greek text "[source sentence]" into English? For context, the Greek multi-word expression "[MWE]" can mean "[English meaning]", "[possible other English meaning]", ..., or "[possible other English meaning]".

heading: 4.1 Human Evaluation
text: For human evaluation, we employ 4 human annotators,2 all native Greek and fluent English-asa-second-language (ESL) speakers, to annotate a subset (356 sentences) of the outputs.3 We set up an annotation interface that provided the following information:
1. The Greek source sentence. 2. The MWE in the source sentence in isolation. 3. The "unaided" translation generated by GPT-4
or NLLB. 4. The "aided" translation generated by GPT-4.
We did not tell the annotators which translation was produced by which model. We asked them to (a) choose the better translation of the two; (b) to provide a quality rating from 1 (terrible) to 5 (perfect) for each one; and (c) to indicate whether the MWE was correctly translated in each of the MT hypotheses.
In cases where GPT-4 generated more than a single translation, or provided more context apart from the translated sentence, only the first output sentence was taken into account.
Note that; in that first study, the choice of ESL speakers might lead to untrustworthy results, as the annotators might be unconsciously biased to accept something closer to their native language as correct. Hence, we also repeat the annotation study with non-Greek-speaking English native speakers, asking them only to rate the quality of the English outputs, without any regard to its similarity to the Greek source (which they cannot understand).

heading: 4.2 Automatic Evaluation with Quality Estimation
text: Beyond human evaluation, we use COMET (Rei et al., 2022) to benchmark automatic quality esti-
2One of them is also an author of the paper. 3Due to budget constraints.
mation metrics. COMET tries to predict the quality scores provided by expert annotators. These are scores similar to the direct assessment ones, but in practice COMET is trained to predict an overall MQM score. More specifically, we obtained absolute and relative scores, using the comet-score and comet-compare functions, respectively, for every set of outputs produced, namely the ones generated by NLLB, and those generated by GPT-4, both in the "aided" and "unaided" variants.
Note that reference translations are not available, so we make use of a quality estimation model.4
Since human evaluators showed significant preference for "aided" GPT-4 outputs, we also evaluated the other two translation sets using the former as a reference. We did this in two ways, evaluating first using all data, and then focusing on the subset whose "aided" translations were rated as exemplary (5 stars) by the annotators.

heading: 5 Results
text: Our human evaluations indicated that the "aided" method using GPT performed significantly better than the GPT-based "unaided" one, which in turn outperforms the NLLB-based "unaided" method. Quality estimation metrics generally agreed with the human evaluations when it comes to this order of quality.
In Table 1, we present the results obtained from the Greek-speaking annotators. They rank "aided" GPT as significantly better than "unaided" GPT, with NLLB worse than both. However, even the best model is subjectively rated noticeably worse than a theoretical correct translation. The first col-
4The Unbabel/wmt20-comet-qe-da pretrained model, which is trained on scores provided by human annotators, which are then normalized into z-scores (hence they can be negative).
umn contains the quality of the translations as a whole, while the second one concerns the correctness of the translation of the MWEs exclusively, irrespective of the rest of the sentence.
Additionally, when asked to explicitly choose the best between the "aided" and "unaided" translations, humans judged "aided" GPT outputs to be better than the "unaided" (78% of the time) and than the NLLB (95% of the time).
In Table 2 we show the evaluations of the English-speaking (but not Greek-speaking) annotators, who only judged the quality of the output without taking into account the input. We skipped evaluating NLLB this way, as it was clearly worse than the others. Here we find that our method not only improves general translation quality, but it also slightly increases the fluency of the output.
In Table 3, we depict the results obtained from the COMET quality estimation model (without a reference translation). These results are of limited value, as the correct translations of sentences using MWEs would diverge significantly from anything a model with no lexical information could predict. They also indicate NLLB’s significantly inferior quality to the other models, but the order between the two GPT methods is reversed.
We also attempt to better compare the two baselines, using high-quality (as judged by humans) "aided" GPT outputs as a reference. Table 4 shows the respective reference-based COMET scores. It too indicates the same conclusion as the humans. Under the same settings, we also explicitly found this difference to be statistically significant for both variations (all data and only high-quality ones), with the null hypothesis being rejected when taking p = 0.05. This was also the case when only using 100 samples, during initial testing.
Targetted versus Random Sentences Using any random translation examples in the prompt has been observed to improve translation quality in certain cases, because they can help the LLM focus on the translation task (Zhang et al., 2023). It
is reasonable, then, that the improvements we observe in our "aided" setting are due to this particular phenomenon, and not due to helping the system translate specifically the target MWE. We therefore also conduct a "random" GPT experiment, which is similar to the "aided" one but instead uses an MWE that is unrelated to the sentence being translated. We found that it indeed scores worse than even the "unaided" variant under the above settings (see Table 4), indicating that the improvement in translation quality should indeed be attributed to the information provided by the lexicon.
All human scores are normalised as explained in Appendix B.

heading: 6 Discussion and Future Work
text: Our experimental results (§5) show that, at least for the IDION dataset, our proposed "aided" method for MWE translation through GPT-4 by providing a pre-contextualized lexicon significantly outperforms the naive "unaided" GPT-4 usage, with both producing better results when compared to stateof-the-art MT models (NLLB). This is significant as the latter is tailored specifically for use cases similar to our experiment, namely translation of lower-resource languages.
However, under all settings, current models seem to underperform in MWE translation. As established previously, a lexicon is most likely required to achieve human-level translations, but having a context-agnostic lexicon and simply providing it to the model, even if it has been generated by humans and correctly mapped to each instance, appears to not be enough to reach this level, although it does significantly improve performance.
We hypothesize that English speakers seem to prefer the same system as English-Greek bilinguals because the model avoids awkward word-to-word translations. So while they cannot tell whether the translation is adequate, they can penalize cases where a literal translation is obviously nonsensical.
Moreover, it seems that GPT, on accessing its own evaluation, might amend its answer. We therefore also attempted further improving GPT-4’s output by giving it the chance to evaluate its previous response and possibly change it (details provided in Appendix A). It did not seem to significantly help, as it was found to rate itself highly and barely, if at all, change its translation.
Future Work We believe that it would be worthwhile to professionally translate our dataset and extend it to more languages, thus providing the necessary context in bilingual settings, making it useful for training NMT models on MWEs.5 Moreover, since current automatic evaluation methods present several shortcomings, our translated dataset could generally be used for benchmarking the MT abilities when it comes to MWEs.
In the era of LLMs, perhaps we should reevaluate how we construct dictionaries. LLMs learn more effectively from more context, so rather than providing simple definitions with minimal disambiguating example usages, lexicographers could focus on providing more and longer examples.

heading: 7 Limitations
text: Human and automatic evaluations seem to generally agree with each other, but we acknowledge that, due to the lexical nature of our task, without a reference translation, quality estimation models (such as the Unbabel one we use) cannot be relied upon to offer trustworthy results.
We also acknowledge that since the evaluators are all primarily native Greek speakers, they may have been influenced in their evaluation of wordfor-word translations into English, as they would
5Unfortunately we lack the funds to do so ourselves.
have made sense to them and it could occasionally have been difficult to know how natural they would sound to native speakers. We attempted to mitigate this issue with our English-speakers evaluation experiment, but ideally one would employ native English speakers that also understand Greek.

heading: 8 Ethics Statement
text: We believe that our work does not introduce any significant additional risks other than those inherent in the models used.
We have obtained permission from all annotators to publish the data they produced in the context of this paper. The annotators were volunteer co-authors and co-workers, and no monetary compensation was provided for their involvement.
The content of IDION is available under a CCBY-NC license, in XML format. Their usage in this project is therefore consistent with its intended use. All models we use come with permissive licenses, at least when it comes to research.

heading: Acknowledgments
text: The authors are grateful for the feedback from the anonymous reviewers and area chairs. This work has been partially supported by project MIS 5154714 of the National Recovery and Resilience Plan Greece 2.0 funded by the European Union under the NextGenerationEU Program. Antonios Anastasopoulos is also supported by the US National Science Foundation under award IIS2327143.
This project was also supported by resources provided by the Office of Research Computing at George Mason University (https://orc.gmu. edu).

heading: A GPT-4 self-evaluation and translation amending
text: While maintaining the conversation context after each initial translation, we provided the following prompt:
’Would you evaluate your translation positively? Give a 1-5 score, and change your response to improve it if necessary.
Format your response as: "X/5<new translation (if applicable)>", don’t
include any other explanation.’

heading: B Normalization of human annotations
text: Since not all human annotators reviewed the same sentences for each model (most only annotated "aided"/"unaided" GPT pairs (task I), while some also annotated "aided" GPT/"unaided" NLLB pairs (task II)), we normalized the results so that they are comparable.
Therefore, if the set of all participants is K and Nk is the set of all annotation scores by participant k on task I, the average for task I is defined as:
∑︂
k∈K
∑︂
i∈Nk
i
|K| · |Nk|
If L(⊂ K) is the set of all participants that also annotated task II, and Ml is the set of all annotation scores by participant l on task II, the normalized average for task II is defined as:
∑︂
l∈L
∑︂
i∈Ml
i |L| · |Ml| · ∑︁ k∈K ∑︁ i∈Nk i |K|·|Nk|∑︁ i∈Nl i |L|·|Nl|
These values, rounded to the nearest decimal, are what is displayed in the relevant tables in Section 5.

heading: Future Work
text: Future Work We believe that it would be worthwhile to professionally translate our dataset and extend it to more languages, thus providing the necessary context in bilingual settings, making it useful for training NMT models on MWEs. 5 Moreover, since current automatic evaluation methods present several shortcomings, our translated dataset could generally be used for benchmarking the MT abilities when it comes to MWEs. In the era of LLMs, perhaps we should reevaluate how we construct dictionaries. LLMs learn more effectively from more context, so rather than providing simple definitions with minimal disambiguating example usages, lexicographers could focus on providing more and longer examples