Multilingual Definition Modeling
Authors: Edison Marrese-Taylor, Erica K. Shimomoto, A. Solano, Enrique Reid
Accepted at: acl25

Multilingual Definition Modeling

Edison Marrese-Taylor1,2, Erica K. Shimomoto1, Alfredo Solano2, Enrique Reid2
National Institute of Advanced Industrial Science and Technology1
The University of Tokyo2
edison.marrese@aist.go.jp, kidoshimomoto.e@aist.go.jp
asolano@weblab.t.u-tokyo.ac.jp, erarvin2007@gmail.com

Abstract

In this paper, we propose the first multilingual study on definition modeling. We use monolingual dictionary data for four new languages (Spanish, French, Portuguese, and German) and perform an in-depth empirical study to test the performance of pre-trained multilingual language models on definition modeling of monosemic words when finetuned on this data. Furthermore, we use a zero-shot approach to test the multilingual capabilities of two popular chat-based Large Language Models (LLMs) in the task. Results show that multilingual language models can perform on-par with English but cannot leverage potential cross-lingual synergies, with LLMs generally offering better performance overall. A comprehensive human evaluation of the LLM-generated definition highlights the zero and few-shot capabilities of these models in this new task, also showing their shortcomings. Finally, we show that performance on our task via BERTScore strongly correlates to the performance on multilingual LLM benchmarks, suggesting that our task offers a viable compute-constrained, stable and natural alternative to these.

1 Introduction

Definitions play a key role in effective communication and language understanding. However, creating high-quality and precise definitions requires time and effort given the complexity of languages and how they are constantly evolving.

Aiming to suffice this necessity, the definition modeling task was first proposed by Noraset et al. (2017). The goal is to estimate the probability of a textual definition given a word being defined. This task can be applied to generate definitions for known and new words, assisting language learners, as well as serving as a tool for language preservation and description (Kabiri and Cook, 2020; Dimas Furtado et al., 2024).

Since its conception, several approaches have tackled this task, showing that it gives an arguably more transparent view of how much a model captures syntax and semantics. So far, existing approaches for this task are trained on a corpus of word-definition pairs, to be later tested on how well they generate definitions for words not seen during training. Current approaches (Noraset et al., 2017; Gadetsky et al., 2018; Ni and Wang, 2017; Ishiwatari et al., 2019) are mainly encoder-decoder based, in which a contextual representation for a word/phrase, encoded using a variety of features, is used to generate the definition.

Despite the progress, previous work has predominantly focused on definition modeling in English. Concretely, we find just a handful of papers that work on definition modeling for languages other than English, namely Reid et al. (2020) for French, Dimas Furtado et al. (2024) for Portuguese and Kong et al. (2020) for Chinese. We also find that dictionary data for French and Italian have been released in the past few years, but that there are no approaches that leverage them so far (Hathout et al., 2014; Hathout and Sajous, 2016; Sajous et al., 2020; Calderone et al., 2017). Finally, while Kabiri and Cook (2020) has covered nine different languages in their study, they did not explore multilingual capabilities of models, focusing on performance of monolingual models on the polysemy aspect of words. Moreover, we note that these datasets are mostly based on crowdsourced dictionaries, where quality can be a concern.

In light of these issues, in this paper, we present the first truly multilingual study on definition modeling, introducing datasets obtained from not-crowdsourced dictionaries in French, Spanish, Portuguese, and German, with a total of approximately 460K new terms entries and 730K definitions, thus substantially extending language coverage for the task. We combine our collected data with existing corpora to train models and lay out the first set
of experiments on multilingual definition modeling. As our main focus is to analyze the potential synergy between different languages, we focus our experiments on monosemic words, hoping to explore polysemy in future work.

Furthermore, we also test a selection of Large Language Models (LLMs) on our collected data, probing their multilingual abilities in this task through zero-shot and few-shot approaches (Brown et al., 2020; Kojima et al., 2022). We evaluate performance based on three different automatic metrics, i.e., BLEU, BERTScore and COMET, and through human evaluation.

Our results show that current multilingual language models such as mBART (Liu et al., 2020; Tang et al., 2020) and mT5 (Xue et al., 2021) can perform on-par with English when fine-tuned on single-language dictionaries. They can also deliver consistent performance when trained in multiple languages, but are unable to leverage cross-lingual synergies to consistently improve performance. As for LLMs, we show that prompting techniques, either in the zero-shot or few-shot scenarios, offer generally better performance than multilingual language model fine-tuning, even in the case of LLMs. However, we also observe that the output language can be difficult to control and that techniques such as in-context learning do not lead to significantly better quality output.

Finally, we show empirical evidence suggesting that per-language performance in our task measured via BERTScore is strongly correlated with performance on multilingual benchmarks. Opposite to these, which are often translations of datasets in English, our task can offer a compute-constrained, static and natural alternative to assess the proficiency of LLMs on a given language. We release our code and data on GitHub1.

2 Related Work

Our paper is primarily related to the seminal work by Noraset et al. (2017); Hill et al. (2016), in which a model is tasked with generating a definition for a word given its respective embedding or with mapping dictionary definitions to lexical representations of words, respectively. Later work has proposed improvements and extensions, introducing techniques and datasets to address shortcomings. For example, Gadetsky et al. (2018) address polysemy and presents a dataset from Oxford Dictionaries, where each definition is supplemented with context sentences. Ni and Wang (2017) proposed an approach for automatically explaining slang English terms in a sentence and introduced another dataset extracted from The Urban Dictionary. Ishiwatari et al. (2019) proposed to further rely on local and global contexts to help the model disambiguate and generate better definitions. Different from our approach, their work focused on unfamiliar words and phrases (many of which are polysemous words), rarely used idioms, or emerging entities, relying on Wikipedia as a source.

More recently, Huang et al. (2021) proposed to study the problem of definition specificity, creating a method for tuning a model to account for hyper-focused (over-specific) or highly general (under-specific) definitions. Finally, Chen and Zhao (2022) proposed to unify the seminal ideas of reverse dictionary and definition modeling in a single model, with the goal of helping better understand word sense and embeddings. Periti et al. (2024) proposed to directly finetune an LLM specifically for definition modeling.

Previous work discussed so far has mostly focused on definitions in the English language. There are a few exceptions, such as the work of Reid et al. (2020), who presented the first study on definition modeling for the French language with the release of a dataset collected from Le Petit Robert, and the work of Dimas Furtado et al. (2024), whose main goal was to collect a dataset for Portuguese definition generation, leveraging several models. Their approach is, however, still monolingual.

We also note that over the past few years, dictionary datasets in several languages derived from Wiktionary have been released, including English (Sajous et al., 2020), French (Hathout et al., 2014; Hathout and Sajous, 2016), Italian (Calderone et al., 2017). Ylonen (2022) collected data for all available Wiktionaries. These datasets, were not accompanied by models that leverage them, and we were unable to find other works using them to train or evaluate models on definition modeling. We think one critical point here is that Wiktionaries are built on the base of crowdsourcing, where quality could be a concern.

Finally, we find previous works aiming to generate definitions in specialized fields, e.g., scientific and medical terms (August et al., 2022), biomedical terminology definition (Liu et al., 2021; Huang et al., 2022), or financial terminology (Jhirad et al., 2023).

1https://github.com/epochx/defmod
3 Data

We are interested in assessing the capabilities of multilingual models to describe words in multiple languages. Therefore, we aim to collect high-quality data for “traditional” dictionary definitions, where most common words in a given language are contained. This approach deviates from recent previous work in definition modeling, where the focus has been on domains of “uncommon” words like slang terms (Ni and Wang, 2017) or rarely used idioms and emerging entities (Ishiwatari et al., 2019) and specific domains such as scientific or financial terms (August et al., 2022; Liu et al., 2021; Huang et al., 2021; Jhirad et al., 2023), and only in English.

To support our study, we utilize data for three of the Indo-European languages with the largest number of speakers, according to Wikipedia: Spanish (485 M native speakers), Portuguese (236 M native speakers), and French (80 M native speakers). As we are particularly interested in studying potential performance gains when training on multiple languages, we focus on these in particular, as they all derive from the Romance branch and thus share many similarities. Finally, we also consider German, another Indo-European language but from the Germanic family with 75 M native speakers. We deem German as the ideal candidate to experiment together with existing datasets for the English language.

<table>
  <tr>
    <th>Dataset (language)</th>
    <th>Terms</th>
    <th>Defs.</th>
    <th>Mono.</th>
  </tr>
  <tr>
    <td>OXFORD - en</td>
    <td>36,767</td>
    <td>122,319</td>
    <td>44.07%</td>
  </tr>
  <tr>
    <td>Le Petit Robert - fr</td>
    <td>33,507</td>
    <td>-</td>
    <td>-</td>
  </tr>
  <tr>
    <td>DRAE - es (ours)</td>
    <td>82,386</td>
    <td>159,863</td>
    <td>60.85%</td>
  </tr>
  <tr>
    <td>DICIO - pt (ours)</td>
    <td>191,499</td>
    <td>279,985</td>
    <td>77.41%</td>
  </tr>
  <tr>
    <td>LAROUSSE - fr (ours)</td>
    <td>72,486</td>
    <td>130,408</td>
    <td>70.06%</td>
  </tr>
  <tr>
    <td>DUDEN - de (ours)</td>
    <td>110,444</td>
    <td>164,086</td>
    <td>73.17%</td>
  </tr>
</table>

Table 1: Summary of our collected datasets, compared to prior relevant corpora. In the table, Mono. stands for Monosemic, i.e. terms with a single definition. We compare against OXFORD and Le Petit Robert (data from Reid et al. (2020)), which we regard as existing datasets extracted from “traditional” dictionaries.

For each language, we start by collecting word lists, which are put together from various online sources, including but not limited to MUSE (Lample et al., 2018). We found inflections of verbs (e.g., tenses) and adjectives (e.g., gender) are present and relied on spacy2 to identify word lemmas, utilizing the en_core_web_sm model for English as well as the core_news_sm model for German, Spanish, Portuguese and French. After obtaining the lemmatized version of each entry in our data, we kept the example where the original term matches the lemmatized word, similarly to Ishiwatari et al. (2019).

In order to obtain good quality definitions for the terms collected, we work with traditional, well-known and readily available dictionaries. Concretely, we rely on the Dictionary of the Spanish language (“Diccionario de la lengua española - DLE”) developed and maintained by the Royal Spanish Academy (Real Academia Española - RAE) for Spanish, Dicio for Portuguese, the Larousse Dictionary for French, and the Duden Dictionary for German. Collecting data from online dictionaries can involve accessing copyrighted information. In this sense, our approach follows previous work (Gadetskyy et al., 2018; Ni and Wang, 2017; Ishiwatari et al., 2019; Reid et al., 2020; Dimas Furtado et al., 2024) in making sure that the data can indeed be downloaded/distributed. Based on our analysis of the Terms of Service for the consulted dictionaries, we will not directly distribute our datasets, but instead release recipes on how to reconstruct them based on our compiled word lists. Please see §A for further details about the elaboration of our datasets.

Table 1 summarizes the main characteristics of the data we collected, compared against existing resources in English and French, where we can see that our collected data is substantially richer, both in terms of the number of terms and definitions.

4 Experimental Setup

Data An important distinction between our corpus and recent prior work is that our collected data does not contain examples of word usage. While we acknowledge the importance of such context to disambiguate a specific meaning for a given term in the case of polysemy, we opt to explore monosemic words as a first step into the study of multilinguality for definition modeling. We thus select the subset of terms that exhibit only a single meaning, which in practice we achieve by simply selecting the terms with a single definition entry. The final word lists are split into the 80/10/10 ratio. Table 2 below shows the exact details of our splits.

In addition to our data, we also use the previ-

2https://spacy.io/
ously released dataset built from Oxford Dictionaries (Gadetskyy et al., 2018) (OXFORD). Though we were also interested in using the French data collected by Reid et al. (2020), we found it was not available at the time of writing this paper.

<table>
  <tr>
    <th>Dataset</th>
    <th>Train</th>
    <th>Valid</th>
    <th>Test</th>
  </tr>
  <tr>
    <td>OXFORD (en)</td>
    <td>15,770</td>
    <td>6,884</td>
    <td>6,834</td>
  </tr>
  <tr>
    <td>DRAE (es)</td>
    <td>32,834</td>
    <td>4,104</td>
    <td>4,105</td>
  </tr>
  <tr>
    <td>DICIO (pt)</td>
    <td>118,591</td>
    <td>14,824</td>
    <td>14,824</td>
  </tr>
  <tr>
    <td>LAROUSSE (fr)</td>
    <td>31,224</td>
    <td>3,903</td>
    <td>3,904</td>
  </tr>
  <tr>
    <td>DUDEN (de)</td>
    <td>55,521</td>
    <td>6,940</td>
    <td>6,941</td>
  </tr>
</table>

Table 2: Details of the size of each split for our collected data filtered for polysemy, compared against OXFORD.

Model Finetuning We train models based on mT5 (Xue et al., 2021). This model is a multilingual version of T5 (Raffel et al., 2020), a generative Transformer pre-trained on a Common Crawl-based dataset covering 101 languages and including a data for a variety of tasks which have been converted into a text-to-text format. We begin by training monolingual baselines for each language, finetuning each dataset. These models give us a set of robust baselines. We then train combining data for several languages. We utilize the “large” model (mT5) (approx. 1.5 B parameters) with the prefix “Define in {language}”, where the variable {language} indicates the language in which we would like the model to generate (English, Spanish, Portuguese, French, German). During training, we randomly sample (word, definition) pairs from the data involved and use these examples to create multilingual batches. In preliminary experiments, we also tested alternative multilingual models, but found that mT5 performed better overall.

Model Prompting Given their success in solving several NLP tasks, we also evaluate the ability of instruction-tuned or instruction-tuned LLMs to provide definitions for words. We consider white-box models including Llama 2 (Touvron et al., 2023), Llama 3.1 (Grattafiori et al., 2024), and Mistral v1 models (Jiang et al., 2023). Specifically, we utilize Llama-2-13b-chat-hf (LLAMA-2), Llama-3.1-8B-Instruct (LLAMA-3), Mistral-7B-Instruct-v0.1 (MISTRAL), respectively. Via preliminary experiments using 4-bit quantization with QLoRA (Dettmers et al., 2023) on a subset of languages and models, we observed that the effects of quantization were bounded <1% of the full model performance, and affected the datasets evenly, and thus adopted this strategy for all our LLM experiments.

We test two settings: (1) zero-shot, where the model is directly asked to generate the definition of the word, and (2) few-shot, where we incorporate term-definition examples in the prompt before requesting the definition for the target term. These shots are randomly sampled from the training data and kept constant across examples.

Prompting LLMs can lead to substantially different outputs depending on how instructions are crafted. For our experiments, we kept the input to the model as simple as possible, always using the prompt “Define the {language} word '{term}'. Use only {language} to reply”, where {language} and {term} are variables denoting the target language and the term to define. We repeat each experiment 3 times with different random seeds. In the case of few-shot, we also sample a different set of (term, definition) tuples from the train portion of the dataset to use as context.

Evaluation Previous work in definition modeling has mainly utilized n-gram overall metrics such as BLEU (Papineni et al., 2002) and METEOR (Banerjee and Lavie, 2005). As the latter is language-specific, here we report BLEU, relying on the sacrebleu3 implementation (Post, 2018). Previous work (Reid et al., 2020; Huang et al., 2021) has also shown that metrics based on n-gram overlap cannot capture nuance in the definitions generated by models. As an alternative, approaches have relied on metrics based on Machine Learning, such as BERTScore (Zhang et al., 2019). Following Jhi- rad et al. (2023) we also experiment with COMET (Rei et al., 2020), using the wmt22-comet-da model, which has multilingual support as it is built on top of XLM-R (Conneau et al., 2020).

To evaluate the ability of the LLMs to generate definitions in the target language, we rely on a fasttext-based language classification model (Grave et al., 2018). Here, compare the desired target language with the highest probability language as predicted by this model. We report the percentage of examples where the prediction matches the desired target language (Compliance).

Finally, we also perform human evaluation on a subset of the definitions generated by our models. We split our study into two subsets: Romance and Germanic languages, with particular emphasis on the former, due to their linguistic similarities and relative higher number of first language speak-

3https://github.com/mjpost/sacrebleu
ers. We recruit three volunteer native speakers of Brazilian Portuguese, European French, and Spanish, and one native speaker of American English, and German. To make sure the evaluations are consistent across languages, we designed an evaluation scheme based on a Likert scale with 5 values (1 to 5), which we crafted based on a detailed examination of the outputs of the models.

Please see §B for more details on models, setup, training and inference, including the results of our study on quantization and performance the model used for language identification. For details about our evaluation scheme, please refer to §D.

5 Results

5.1 Automatic Evaluation

Table 3 summarizes the results of our multilingual experiments using mT5. We observe that this model can perform consistently when trained in multiple languages, as measured by COMET and BERTScore. When it comes to BLEU scores, although we observe little to no improvements over the monolingual baselines in the validation set, we see a more consistent behavior on the test portion. However, we note that the definitions generated by our models are often correct but expressed using different surface forms compared to the gold standard, which can lead to low BLEU scores.

Overall, our results show that even for languages from the same family, where we often find that similar words are associated to similar or related meanings (e.g. words sharing the same Latin root in the case of the Romance languages), exposing models such multilingual lexical data did not help them generalize across languages. However, training models in this fashion does not lead to degraded per-language performance either. A possible explanation for these observations is that a significant portion of the cross-lingual term-definition relatedness is rooted in polysemic terms.

Table 4 summarizes the results of our experiments with LLMs. Compared to our finetuning experiments, LLMs generally offer better performance in terms of COMET and BERTScore and are competitive in terms of BLEU. One important issue here is that LLMs are sometimes unable to follow instructions, which, in our case, often leads to the model generating outputs in English.

When it comes to differences due to in-context learning (ICL), although our results may suggest that this leads to improvements, these were not always statistically significant across metrics and languages, showing that ICL may not be as effective for this task as suggested by previous work.

<table>
  <tr>
    <th>Dataset</th>
    <th>Training Data</th>
    <th>BL</th>
    <th>CMT</th>
    <th>BS</th>
  </tr>
  <tr>
    <td rowspan="2">DUDEN (de)</td>
    <td>de</td>
    <td><b>0.939</b></td>
    <td>0.346</td>
    <td>0.664</td>
  </tr>
  <tr>
    <td>de + en</td>
    <td>0.507</td>
    <td>0.345</td>
    <td>0.656</td>
  </tr>
  <tr>
    <td rowspan="3">OXFORD (en)</td>
    <td>en</td>
    <td><b>1.330</b></td>
    <td>0.345</td>
    <td>0.813</td>
  </tr>
  <tr>
    <td>en + fr</td>
    <td>0.038</td>
    <td>0.345</td>
    <td>0.778</td>
  </tr>
  <tr>
    <td>en + de</td>
    <td>0.863</td>
    <td>0.344</td>
    <td><b>0.839</b></td>
  </tr>
  <tr>
    <td></td>
    <td>en + fr + es + pt</td>
    <td>0.532</td>
    <td><b>0.345</b></td>
    <td><b>0.839</b></td>
  </tr>
  <tr>
    <td rowspan="4">DRAE (es)</td>
    <td>es</td>
    <td>7.700</td>
    <td><b>0.384</b></td>
    <td>0.743</td>
  </tr>
  <tr>
    <td>es + fr</td>
    <td>8.084</td>
    <td>0.382</td>
    <td>0.743</td>
  </tr>
  <tr>
    <td>es + pt</td>
    <td>9.158</td>
    <td>0.382</td>
    <td>0.746</td>
  </tr>
  <tr>
    <td>es + pt + fr</td>
    <td><b>9.244</b></td>
    <td>0.381</td>
    <td><b>0.747</b></td>
  </tr>
  <tr>
    <td></td>
    <td>en + fr + es + pt</td>
    <td>5.503</td>
    <td><b>0.384</b></td>
    <td>0.727</td>
  </tr>
  <tr>
    <td rowspan="4">LAROUSSE (fr)</td>
    <td>fr</td>
    <td>0.490</td>
    <td><b>0.300</b></td>
    <td>0.685</td>
  </tr>
  <tr>
    <td>fr + es</td>
    <td>0.936</td>
    <td>0.293</td>
    <td>0.686</td>
  </tr>
  <tr>
    <td>fr + pt</td>
    <td>0.587</td>
    <td>0.298</td>
    <td>0.675</td>
  </tr>
  <tr>
    <td>fr + en</td>
    <td>0.104</td>
    <td>0.294</td>
    <td>0.644</td>
  </tr>
  <tr>
    <td></td>
    <td>fr + es + pt</td>
    <td><b>2.660</b></td>
    <td>0.292</td>
    <td><b>0.705</b></td>
  </tr>
  <tr>
    <td></td>
    <td>fr + en + es + pt</td>
    <td>0.748</td>
    <td>0.293</td>
    <td>0.682</td>
  </tr>
  <tr>
    <td rowspan="4">DICIO (pt)</td>
    <td>pt</td>
    <td>11.539</td>
    <td>0.352</td>
    <td>0.732</td>
  </tr>
  <tr>
    <td>pt + es</td>
    <td>18.626</td>
    <td>0.352</td>
    <td><b>0.748</b></td>
  </tr>
  <tr>
    <td>pt + fr</td>
    <td>4.969</td>
    <td><b>0.353</b></td>
    <td>0.682</td>
  </tr>
  <tr>
    <td>pt + fr + es</td>
    <td><b>18.856</b></td>
    <td>0.352</td>
    <td><b>0.748</b></td>
  </tr>
  <tr>
    <td></td>
    <td>en + fr + es + pt</td>
    <td>4.542</td>
    <td>0.351</td>
    <td>0.685</td>
  </tr>
</table>

Table 3: Results of our multilingual experiments based on mT5, where BL is short for BLEU, CMT is short for COMET, and BS is short for BERTScore.

Role of Prompt Language As explained in §4, the prompt utilized in our experiments with LLMs is provided in English. Since this may bias models toward producing output in this language, we performed a study using a prompt in Spanish on the DRAE dataset. In this case, we use the following prompt template: “Define la palabra ‘{term}’. Utiliza solo español en tu respuesta.”, a loose translation of our English prompt, where ‘{term}’ is a placeholder for the actual word. We follow the same methodology and test on LLAMA-2 and LLAMA-3, on the 0-shot and 5-shot scenarios. Table 5 summarizes our results and compares them against our baseline prompt in English. As shown in the table, results show that prompting models in English leads to consistent superior performance accross all metrics. This validates our choice to prompt models solely in English, and supports our observations during early experimentation. These results also suggest that more recent models offer increased abilities to follow instructions in languages other than English. However, a significant gap still exists.
<table>
  <tr>
    <th>Dataset</th>
    <th>Model</th>
    <th>S</th>
    <th>BL</th>
    <th>BS</th>
    <th>CMT</th>
    <th>Cmp.</th>
  </tr>
  <tr>
    <td rowspan="3">DUDEN (de)</td>
    <td>LLAMA-2</td>
    <td>0</td>
    <td>0.180</td>
    <td>0.612</td>
    <td><b>0.439</b></td>
    <td><b>0.685</b></td>
  </tr>
  <tr>
    <td>MISTRAL</td>
    <td>5</td>
    <td><b>0.758</b></td>
    <td><b>0.637</b></td>
    <td>0.378</td>
    <td>0.604</td>
  </tr>
  <tr>
    <td>MISTRAL</td>
    <td>5</td>
    <td>0.159</td>
    <td>0.621</td>
    <td><b>0.411</b></td>
    <td><b>0.762</b></td>
  </tr>
  <tr>
    <td rowspan="2">LLAMA-3</td>
    <td>5</td>
    <td>0.233</td>
    <td><b>0.623</b></td>
    <td>0.362</td>
    <td>0.628</td>
  </tr>
  <tr>
    <td>5</td>
    <td>3.261</td>
    <td>0.676</td>
    <td>0.580</td>
    <td><b>0.998</b></td>
  </tr>
  <tr>
    <td rowspan="3">OXFORD (en)</td>
    <td>LLAMA-2</td>
    <td>0</td>
    <td>0.949</td>
    <td>0.826</td>
    <td><b>0.553</b></td>
    <td>-</td>
  </tr>
  <tr>
    <td>MISTRAL</td>
    <td>5</td>
    <td><b>3.906</b></td>
    <td><b>0.851</b></td>
    <td>0.537</td>
    <td>-</td>
  </tr>
  <tr>
    <td>MISTRAL</td>
    <td>5</td>
    <td>1.183</td>
    <td>0.837</td>
    <td><b>0.557</b></td>
    <td>0.999</td>
  </tr>
  <tr>
    <td rowspan="2">LLAMA-3</td>
    <td>5</td>
    <td>1.821</td>
    <td><b>0.842</b></td>
    <td>0.529</td>
    <td>-</td>
  </tr>
  <tr>
    <td>5</td>
    <td>1.588</td>
    <td>0.835</td>
    <td>0.568</td>
    <td>-</td>
  </tr>
  <tr>
    <td rowspan="3">DRAE (es)</td>
    <td>LLAMA-2</td>
    <td>0</td>
    <td>0.271</td>
    <td>0.617</td>
    <td><b>0.504</b></td>
    <td><b>0.996</b></td>
  </tr>
  <tr>
    <td>MISTRAL</td>
    <td>5</td>
    <td><b>1.360</b></td>
    <td><b>0.675</b></td>
    <td>0.470</td>
    <td>0.877</td>
  </tr>
  <tr>
    <td>MISTRAL</td>
    <td>5</td>
    <td>0.324</td>
    <td>0.650</td>
    <td><b>0.458</b></td>
    <td>0.703</td>
  </tr>
  <tr>
    <td rowspan="2">LLAMA-3</td>
    <td>5</td>
    <td>0.559</td>
    <td><b>0.663</b></td>
    <td>0.440</td>
    <td><b>0.852</b></td>
  </tr>
  <tr>
    <td>5</td>
    <td>1.855</td>
    <td>0.670</td>
    <td>0.545</td>
    <td><b>0.999</b></td>
  </tr>
  <tr>
    <td rowspan="3">LAROUSSE (fr)</td>
    <td>LLAMA-2</td>
    <td>0</td>
    <td>0.326</td>
    <td>0.631</td>
    <td><b>0.473</b></td>
    <td><b>0.970</b></td>
  </tr>
  <tr>
    <td>MISTRAL</td>
    <td>5</td>
    <td><b>1.350</b></td>
    <td><b>0.676</b></td>
    <td>0.469</td>
    <td>0.966</td>
  </tr>
  <tr>
    <td>MISTRAL</td>
    <td>5</td>
    <td>0.327</td>
    <td>0.648</td>
    <td><b>0.441</b></td>
    <td>0.775</td>
  </tr>
  <tr>
    <td rowspan="2">LLAMA-3</td>
    <td>5</td>
    <td><b>0.626</b></td>
    <td><b>0.664</b></td>
    <td>0.425</td>
    <td><b>0.951</b></td>
  </tr>
  <tr>
    <td>5</td>
    <td>3.443</td>
    <td>0.676</td>
    <td><b>0.543</b></td>
    <td><b>0.999</b></td>
  </tr>
  <tr>
    <td rowspan="3">DICIO (pt)</td>
    <td>LLAMA-2</td>
    <td>0</td>
    <td>0.143</td>
    <td>0.682</td>
    <td><b>0.513</b></td>
    <td><b>0.969</b></td>
  </tr>
  <tr>
    <td>MISTRAL</td>
    <td>5</td>
    <td><b>1.828</b></td>
    <td><b>0.683</b></td>
    <td>0.462</td>
    <td>0.883</td>
  </tr>
  <tr>
    <td>MISTRAL</td>
    <td>5</td>
    <td>0.146</td>
    <td>0.689</td>
    <td>0.475</td>
    <td>0.630</td>
  </tr>
  <tr>
    <td rowspan="2">LLAMA-3</td>
    <td>5</td>
    <td><b>4.030</b></td>
    <td><b>0.709</b></td>
    <td><b>0.501</b></td>
    <td><b>0.926</b></td>
  </tr>
  <tr>
    <td>5</td>
    <td>0.457</td>
    <td>0.683</td>
    <td>0.542</td>
    <td><b>0.998</b></td>
  </tr>
  <tr>
    <td>LLAMA-3</td>
    <td>5</td>
    <td><b>2.298</b></td>
    <td><b>0.715</b></td>
    <td><b>0.549</b></td>
    <td>0.982</td>
  </tr>
</table>

Table 4: Results of our experiments with LLMs, where S indicates the number of shots, and Cmp. is short for compliance, the % of cases where the answer is in the correct language. BL, BS and CMT are short for BLEU, BERTScore and COMET, respectively. Numbers in bold indicate best results for each language-model combination. Underlined results indicate differences against zero-shot are statistically significant at level \( \alpha = 0.05 \).

Dataset Contamination A key aspect to consider when analyzing our results is the possibility that portions of our benchmark have appeared in the training data of the LLMs that we tested (Sainz et al., 2023; Deng et al., 2024; Li et al., 2024). An important point here is to first clarify what “contamination” should mean in the context of our task. We think exposing LLMs to actual word-definition pairs during training can and should be considered a kind of contamination. However, if models are simply exposed to examples of words/terms in context, this should not count as a case of contamination. Moreover, the degree of exposure of the

<table>
  <tr>
    <th>Model</th>
    <th>S</th>
    <th>L</th>
    <th>BL</th>
    <th>BS</th>
    <th>CMT</th>
    <th>Cmp.</th>
  </tr>
  <tr>
    <td rowspan="2">LLAMA-2</td>
    <td>0</td>
    <td>en</td>
    <td>0.271</td>
    <td>0.617</td>
    <td>0.504</td>
    <td>0.996</td>
  </tr>
  <tr>
    <td>5</td>
    <td>es</td>
    <td>0.186</td>
    <td>0.601</td>
    <td>0.502</td>
    <td>0.814</td>
  </tr>
  <tr>
    <td rowspan="2">LLAMA-3</td>
    <td>5</td>
    <td>en</td>
    <td><b>1.360</b></td>
    <td><b>0.675</b></td>
    <td><b>0.470</b></td>
    <td><b>0.877</b></td>
  </tr>
  <tr>
    <td>5</td>
    <td>es</td>
    <td>0.019</td>
    <td>0.526</td>
    <td>0.409</td>
    <td>0.078</td>
  </tr>
  <tr>
    <td rowspan="2">LLAMA-3</td>
    <td>0</td>
    <td>en</td>
    <td><b>1.855</b></td>
    <td><b>0.670</b></td>
    <td><b>0.545</b></td>
    <td>-</td>
  </tr>
  <tr>
    <td>5</td>
    <td>es</td>
    <td>0.303</td>
    <td>0.619</td>
    <td>0.532</td>
    <td>-</td>
  </tr>
  <tr>
    <td rowspan="2">LLAMA-3</td>
    <td>5</td>
    <td>en</td>
    <td><b>3.067</b></td>
    <td>0.683</td>
    <td><b>0.530</b></td>
    <td><b>0.974</b></td>
  </tr>
  <tr>
    <td>5</td>
    <td>es</td>
    <td>2.885</td>
    <td><b>0.711</b></td>
    <td>0.529</td>
    <td>0.908</td>
  </tr>
</table>

Table 5: Performance on Spanish definition modelling when using prompts in English (en) and Spanish (es), tested on DRAE, were S and L indicate the number of shots and prompt language respectively. BL, BS and CMT are short for BLEU, BERTscore and COMET, respectively. Numbers in bold indicate best results per model.

LLMs to these contexts is highly relevant to our task and should be studied. One possibility is to work only with fully open-source LLMs, which would enable us to analyze the training data. This is left for future work.

5.2 Human Evaluation

We further evaluate the output of the 5-shot experiments on all of our languages for LLAMA-2 and MISTRAL via a human study. We use 200 randomly-sampled terms, for one random seed across languages, totaling 400 definitions to be evaluated per language. We additionally require the annotators to provide us with an assessment of the quality of the gold standard definitions we collected via the same 1-5 Likert score defined in §4. During the whole evaluation procedure, we allow the annotators to see gold-standard definitions and consult external sources if needed. Table 6 summarizes our results.

Inter-annotation agreement We report Cohen’s Kappa (\( \kappa \)) scores, which we regard as a lower bound in our case, as it assumes label independence. Since our Likert scores are inherently a ranking, we also report Kendall’s Tau (\( \tau \)), which accounts for this and offers a more realistic agreement measure. Finally, we also report the percentage of examples where the annotators have an exact match (EM) in their answers. For languages where we have more than two evaluators, we report the average value of all possible pairwise combinations.

As shown in Table 6 we first see a clear gap in performance between English and in the rest of the languages, which is in agreement with our automatic evaluation. These results further confirm
<table>
  <tr>
    <th rowspan="2">Dataset</th>
    <th rowspan="2">Model</th>
    <th rowspan="2">Score</th>
    <th colspan="3">IAA</th>
  </tr>
  <tr>
    <th>\( \kappa \)</th>
    <th>\( \tau \)</th>
    <th>EM</th>
  </tr>
  <tr>
    <td rowspan="3">DUDEN (de)</td>
    <td>LLAMA-2</td>
    <td>2.050</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
  </tr>
  <tr>
    <td>MISTRAL</td>
    <td>1.950</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
  </tr>
  <tr>
    <td>Data</td>
    <td>3.825</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
  </tr>
  <tr>
    <td rowspan="3">OXFORD (en)</td>
    <td>LLAMA-2</td>
    <td>4.830</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
  </tr>
  <tr>
    <td>MISTRAL</td>
    <td>4.710</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
  </tr>
  <tr>
    <td>Data</td>
    <td>4.925</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
  </tr>
  <tr>
    <td rowspan="3">DRAE (es)</td>
    <td>LLAMA-2</td>
    <td>2.303</td>
    <td>0.520</td>
    <td>0.788</td>
    <td>0.675</td>
  </tr>
  <tr>
    <td>MISTRAL</td>
    <td>1.815</td>
    <td>0.616</td>
    <td>0.856</td>
    <td>0.785</td>
  </tr>
  <tr>
    <td>Data</td>
    <td>4.565</td>
    <td>0.134</td>
    <td>0.148</td>
    <td>0.600</td>
  </tr>
  <tr>
    <td rowspan="3">DICIO (pt)</td>
    <td>LLAMA-2</td>
    <td>1.812</td>
    <td>0.337</td>
    <td>0.594</td>
    <td>0.633</td>
  </tr>
  <tr>
    <td>MISTRAL</td>
    <td>1.352</td>
    <td>0.281</td>
    <td>0.446</td>
    <td>0.777</td>
  </tr>
  <tr>
    <td>Data</td>
    <td>2.967</td>
    <td>0.164</td>
    <td>0.920</td>
    <td>0.355</td>
  </tr>
  <tr>
    <td rowspan="3">LAROUSSE (fr)</td>
    <td>LLAMA-2</td>
    <td>2.720</td>
    <td>0.449</td>
    <td>0.652</td>
    <td>0.563</td>
  </tr>
  <tr>
    <td>MISTRAL</td>
    <td>2.133</td>
    <td>0.581</td>
    <td>0.731</td>
    <td>0.700</td>
  </tr>
  <tr>
    <td>Data</td>
    <td>4.972</td>
    <td>0.083</td>
    <td>0.125</td>
    <td>0.965</td>
  </tr>
</table>

Table 6: Results of our human evaluation on our best results with LLMs, in terms of average Likert scores, along with their IAA measured via average Cohen’s Kappa (\( \kappa \)), Kendall’s Tau (\( \tau \)) and exact match (EM).

that all three models struggle with Portuguese and German, at least in comparison with the other European languages, despite reportedly being trained on a similar amount of text as is the case of LLAMA-2, where the training mixture contains 0.17% de, 0.16% fr, 0.13% es (Touvron et al., 2023).

Our human evaluation also shows substantial variation in terms of Likert scores for the quality of the gold standard definitions, suggesting that DICIO and DUDEN may offer comparatively lower quality data. In contrast, although results on LAROUSSE exhibit particularly low \( \kappa \) and \( \tau \) values, we see an EM of 0.97. Given the high average score of 4.9, this shows that although the annotators may disagree about the exact score for a given definition, overall they think that definitions are of excellent quality.

Finally, we allowed the annotators to provide us with comments, and one concrete issue that was brought to our attention was that these dictionaries would often offer just a synonym as a definition for a given term. This phenomenon may respond to historical reasons (originally, dictionaries were printed) or be due to the editorial of each company. Nevertheless, if we regard these scores as an upper bound for the human evaluation, we find that in all cases there is still a significant gap between this and the performance of the LLMs. We performed an additional human study to gain insight into the effect of ICL and the effectiveness of COMET as a metric, please see §D for details.

5.3 Multilingual benchmark comparison

We study the relationship between our proposed task and existing datasets used to measure the multilingual performance of LLMs. Concretely, we compare per-language performance for each one of our metrics against multilingual versions of HellaSwag (Zellers et al., 2019), ARC-Challenge (Clark et al., 2018) and MMLU (Hendrycks et al., 2021), provided by Lai et al. (2023) via the lm-evaluation-harness\(^4\) platform (Gao et al., 2023). Together, these datasets contain multiple choice questions spanning a wide variety of tasks, including adversarial commonsense natural language inference, abstraction and reasoning ability acquisition via demonstrations, and questions derived from diverse fields of knowledge.

We found that per-language performance in our task, measured via BERTScore, strongly correlates with all of our selected multilingual benchmarks, with average values of 0.94, 0.95 and 0.87 for LLAMA-2, MISTRAL and LLAMA-3, respectively. Please see Table 12 for the details of all the correlation values we obtained, and §C.3 for the exact benchmark performance we measured. Compared to these, which are machine-translations of the originals in English, we think the definition modeling task can offer a natural, stable alternative to assess the proficiency of LLMs in a given language.

Furthermore, to understand the computational trade-offs of using our task instead of our studied multilingual benchmarks, we compare the correlation between performance on the latter and in our datasets, versus subsamples of different sizes (from 0.1% to 5%). For each size, we sample 10 times and report average correlation and confidence intervals via Gaussian-based asymptotic approximation.

Figure 1 summarizes our findings. The computational budget required for our dataset is roughly equivalent to 0.5% of MMLU, 18% of ARC-Challenge or 36.7% or HellaSwag in number of tokens (more details available in Table 17). This means that definition modeling can offer an efficient alternative to measuring the LLM performance on a specific language, particularly when compared with computationally-intensive benchmarks like MMLU. Our datasets can also be combined with modern sampling approaches for benchmark score approximation such as IRT (Polo et al., 2024), leading to further efficiency gains.

\(^4\)github.com/EleutherAI/lm-evaluation-harness
<table>
  <tr>
    <th>Model</th>
    <th>BL</th>
    <th>CMT</th>
    <th>BS</th>
  </tr>
  <tr>
    <th colspan="4">Full OXFORD</th>
  </tr>
  <tr>
    <td>LoG-CAD (Ishiwatari et al.)</td>
    <td>25.19</td>
    <td>-</td>
    <td>-</td>
  </tr>
  <tr>
    <td>VCDM (Reid et al.)</td>
    <td>27.38</td>
    <td>0.570<sup>◊</sup></td>
    <td>-</td>
  </tr>
  <tr>
    <td>T5 + RR (Huang et al.)</td>
    <td>26.52</td>
    <td>-</td>
    <td>-</td>
  </tr>
  <tr>
    <td>LLama3Dict. (Periti et al.)</td>
    <td>21.98</td>
    <td>-</td>
    <td>0.889</td>
  </tr>
  <tr>
    <th colspan="4">Monosemic only OXFORD</th>
  </tr>
  <tr>
    <td>mT5 + ft (ours)</td>
    <td>1.33</td>
    <td>0.345</td>
    <td>0.839</td>
  </tr>
  <tr>
    <td>FLAN-UL2 (Jhirad et al.)<sup>◊</sup></td>
    <td>2.63</td>
    <td>0.457</td>
    <td>-</td>
  </tr>
  <tr>
    <td>+ examples</td>
    <td>4.03</td>
    <td>0.501</td>
    <td>-</td>
  </tr>
  <tr>
    <td>+ 2 shots</td>
    <td>3.40</td>
    <td>0.475</td>
    <td>-</td>
  </tr>
  <tr>
    <td>+ 2 shots + examples</td>
    <td>4.40</td>
    <td>0.505</td>
    <td>-</td>
  </tr>
  <tr>
    <td>LLAMA-2 (ours)</td>
    <td>0.95</td>
    <td>0.553</td>
    <td>0.826</td>
  </tr>
  <tr>
    <td>+ 5 shots</td>
    <td>3.91</td>
    <td>0.537</td>
    <td>0.851</td>
  </tr>
  <tr>
    <td>MISTRAL (ours)</td>
    <td>1.18</td>
    <td>0.557</td>
    <td>0.837</td>
  </tr>
  <tr>
    <td>+ 5 shots</td>
    <td>1.82</td>
    <td>0.529</td>
    <td>0.842</td>
  </tr>
  <tr>
    <td>LLAMA-3 (ours)</td>
    <td>1.59</td>
    <td>0.568</td>
    <td>0.835</td>
  </tr>
  <tr>
    <td>+ 5 shots</td>
    <td>3.57</td>
    <td>0.847</td>
    <td>0.847</td>
  </tr>
  <tr>
    <td>+ ft</td>
    <td>0.57</td>
    <td>0.511</td>
    <td>0.806</td>
  </tr>
  <tr>
    <td>LLama3Dict (Periti et al.)†</td>
    <td>0.87</td>
    <td>0.455</td>
    <td>0.799</td>
  </tr>
</table>

Table 7: Comparison of our best models against state-of-the-art approaches on OXFORD. Note that for the “Full OXFORD” section of the table, we compare against models trained/tested on the entire dataset, which differs substantially from ours. BL, BS and CMT are short for BLEU, BERTScore and COMET, respectively; † observed results, ◊ reported by Jhirad et al. (2023).

Figure 1: Pearson correlation between performance on multilingual benchmarks and on our task via BERTScore (solid lines), versus the correlation between the same benchmarks and random samples of different sizes for our three tested models LLAMA-2 (top), MISTRAL (center), and LLAMA-3 (bottom) [Improve this! -EMT].

5.4 Comparison with prior work

We compare our results against relevant prior work, focusing on the English language via OXFORD. When it comes to models finetuned on dictionary data, we consider the approaches by Ishiwatari et al. (2019), who proposed a local-and-global context model based on word embeddings, Reid et al. (2020), who leveraged BERT (Devlin et al., 2019) and combined it with a variational inference framework, Huang et al. (2021), who propose a specificity-sensitive approach with based on T5 finetunes and re-ranking (RR), and Periti et al. (2024), who finetuned LLMs to generate definitions of words in context. We point out that all of these approaches work with the complete OXFORD dataset and rely heavily on example sentences for disambiguation.

For zero-shot and few-shot approaches, we compare our results against the values reported by Jhirad et al. (2023). To perform an apples-to-apples comparison against Periti et al. (2024), we replicate their setup using our data by finetuning LLAMA-3 in the training portion of the concatenation of our dictionary data. We utilize the same prompt introduced in §4, which does not substantially differ from what the authors use, save from the fact that we do not provide example sentences. We follow their training procedure, using 4-bit quantization (Dettmers et al., 2023) via PEFT with their best hyperparameter configuration.

Table 7 summarizes our comparison against relevant prior work. We see a substantial gap of BLEU scores exists between models tested on
the full OXFORD dataset, and its monosemic-only subset. These differences are not so significant when looking at COMET or BERTScore. We also note that LLama3Dict, a model trained on dictionary data with example sentences, performs significantly worse in the absence of these examples (BLEU of 21.98 decreased to 0.87). While our similar LLAMA-3 finetune is able to outperform LLama3Dict in this scenario in terms of BERTScore, we see both models perform worse than the base LLMs even when combined with ICL across all metrics. In fact, we observed (Table 13) that finetuning on dictionary data, with our without example sentences, decreased performance with respect to the base model in all our studied languages for the monosemic scenario. Finally, our finetuned mT5, although trained only on monosemic data, offers competitive results with the best models in terms of BERTScore.

Overall, we think these results show that the task of deriving the meaning of a word from example sentences differs substantially from the task of eliciting definitions purely by recalling concepts learned during pretraining. For the latter task, which is the focus of this paper, it is still unclear why LLMs finetuned on dictionary data without context sentences do not perform better than base LLMs with ICL. Based on the characterization recently introduced by Pan et al. (2023), we think it is possible that both finetuning and ICL lead to improvements in terms of Task Recognition (TR) but not in Task Learning (TL): by being exposed dictionary data, models learn to mimic the style in which definitions should be provided, but without actually leading to a better ability to recall definitions.

Our results can also be explained in light of the discussion by Agarwal et al. (2024), who point out important limitations of the next-token prediction loss as an indicator of downstream performance. It is possible that the low performance is due to a lack of exposure to lexically-rich data during pretraining, especially for languages other than English, which naive SFT cannot compensate for.

6 Conclusions

In this paper, we presented the first multilingual study on definition modeling, introducing monolingual dictionary data for German, Spanish, French and Portuguese. Results show that LLMs can provide generally better results compared to smaller multilingual LMs trained on monosemic dictionary data, but they can be unstable and generate outputs outside the target language. Furthermore, techniques such as ICL and finetuning did not lead to significantly better quality output. Finally, we show that our task can serve as a natural, stable alternative to assess the proficiency of LLMs in a language.

For future work, we would like to incorporate more languages and perform further human evaluation to better understand the relationship between human assessments and automatic metrics. We would also like to tackle the disambiguation problem by collecting usage examples to tackle polysemy. Our supplementary material offers preliminary evidence suggesting that using LLMs could be a viable alternative for this (§F).

Acknowledgements

This paper is based on results obtained from the by the National Institute of Advanced Industrial Science and Technology (AIST) policy-based budget project “R&D on Generative AI Foundation Models for the Physical Domain”, and was also partially funded by project JPNP20006, commissioned by the New Energy and Industrial Technology Development Organization (NEDO). The computational resources of the AI Bridging Cloud Infrastructure (ABCi), provided by AIST, were used for most of the experiments in this paper. We are also grateful to the NVIDIA Corporation, which donated one of the GPUs we used. Finally, we would also like to thank our volunteer annotators Daniela Serien, Carolina Dias Alexiou, Mauricio Cordova, Lucie Kunitomo-Jacquin, Antoine Jacquin and Denis Higashi Imoto, for their willingness to help and feedback.

Limitations

There are a few notable limitations of our work. First, while we tackled five different languages, there is no evidence that our results will generalize to other languages, especially those belonging to different linguistic branches. Furthermore, we recognize that despite their remarkable performance, LLMs may be challenging to access for many researchers, as white-box models require powerful computing resources to run them locally, and black-box models are behind a paywall. Finally, we demonstrated how LLMs can help generate use cases for polysemic words, but have not tackled polysemy.
Ethics Statement

Our main objective is to propose a new task to evaluate the abilities of LLMs. One potential use-case is to have a model generate fake definitions that may mislead users that interact with an LLM when deployed.

References

Rishabh Agarwal, Avi Singh, Lei M Zhang, Bernd Bohnet, Luis Rosias, Stephanie C.Y. Chan, Biao Zhang, Aleksandra Faust, and Hugo Larochelle. 2024. Many-shot in-context learning. In ICML 2024 Workshop on In-Context Learning.

Tal August, Katharina Reinecke, and Noah A. Smith. 2022. Generating Scientific Definitions with Controllable Complexity. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 8298–8317, Dublin, Ireland. Association for Computational Linguistics.

Timothy Baldwin and Marco Lui. 2010. Language identification: The long and the short of the matter. In Human Language Technologies: The 2010 Annual Conference of the North American Chapter of the Association for Computational Linguistics, pages 229–237, Los Angeles, California. Association for Computational Linguistics.

Satanjeev Banerjee and Alon Lavie. 2005. METEOR: An automatic metric for MT evaluation with improved correlation with human judgments. In Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization, pages 65–72, Ann Arbor, Michigan. Association for Computational Linguistics.

Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, and 12 others. 2020. Language Models are Few-Shot Learners. In Advances in Neural Information Processing Systems, volume 33, pages 1877–1901. Curran Associates, Inc.

Basilio Calderone, Matteo Pascoli, Franck Sajous, and Nabil Hathout. 2017. Hybrid Method for Stress Prediction Applied to GLAFF-IT, a Large-Scale Italian Lexicon. In Language, Data, and Knowledge, Lecture Notes in Computer Science, pages 26–41, Cham. Springer International Publishing.

Pinzhen Chen and Zheng Zhao. 2022. A Unified Model for Reverse Dictionary and Definition Modelling. In Proceedings of the 2nd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 12th International Joint Conference on Natural Language Processing (Volume 2: Short Papers), pages 8–13, Online only. Association for Computational Linguistics.

Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. 2018. Think you have solved question answering? try arc, the ai2 reasoning challenge. Preprint, arXiv:1803.05457.

Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020. Unsupervised Cross-lingual Representation Learning at Scale. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8440–8451, Online. Association for Computational Linguistics.

Chunyuan Deng, Yilun Zhao, Yuzhao Heng, Yitong Li, Jiannan Cao, Xiangru Tang, and Arman Cohan. 2024. Unveiling the spectrum of data contamination in language model: A survey from detection to remediation. In Findings of the Association for Computational Linguistics: ACL 2024, pages 16078–16092, Bangkok, Thailand. Association for Computational Linguistics.

Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. 2023. Qlora: Efficient finetuning of quantized llms. Preprint, arXiv:2305.14314.

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Volume 1 (Long and Short Papers), pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics.

Anna Beatriz Dimas Furtado, Tharindu Ranasinghe, Frederic Blain, and Ruslan Mitkov. 2024. DORE: A dataset for Portuguese definition generation. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), pages 5315–5322, Torino, Italia. ELRA and ICCL.

Artyom Gadetskyy, Ilya Yakubovskiy, and Dmitry Vetrov. 2018. Conditional generators of words definitions. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 266–271, Melbourne, Australia. Association for Computational Linguistics.

Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac’h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, and
5 others. 2023. A framework for few-shot language model evaluation.

Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, and Ahmad Al-Dahle. 2024. The llama 3 herd of models. Preprint, arXiv:2407.21783.

Edouard Grave, Piotr Bojanowski, Prakhar Gupta, Armand Joulin, and Tomas Mikolov. 2018. Learning word vectors for 157 languages. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), Miyazaki, Japan. European Language Resources Association (ELRA).

Nabil Hathout and Franck Sajous. 2016. Wiktionnaire’s WikiCode GLAWfied: A Workable French Machine-Readable Dictionary. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC’16), pages 1369–1376, Portorož, Slovenia. European Language Resources Association (ELRA).

Nabil Hathout, Franck Sajous, and Basilio Calderone. 2014. GLÄFF, a Large Versatile French Lexicon. In Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC’14), pages 1007–1012, Reykjavik, Iceland. European Language Resources Association (ELRA).

Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021. Measuring massive multitask language understanding. In International Conference on Learning Representations.

Felix Hill, Kyunghyun Cho, Anna Korhonen, and Yoshua Bengio. 2016. Learning to understand phrases by embedding the dictionary. Transactions of the Association for Computational Linguistics, 4:17–30.

Han Huang, Tomoyuki Kajiwara, and Yuki Arase. 2021. Definition Modelling for Appropriate Specificity. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 2499–2509, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.

Jie Huang, Hanyin Shao, Kevin Chen-Chuan Chang, Jinjun Xiong, and Wen-mei Hwu. 2022. Understanding Jargon: Combining Extraction and Generation for Definition Modeling. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 3994–4004, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.

Shonosuke Ishiwatari, Hiroaki Hayashi, Naoki Yoshi-naga, Graham Neubig, Shoetsu Sato, Masashi Toyoda, and Masaru Kitsuregawa. 2019. Learning to Describe Unknown Phrases with Local and Global Contexts. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 3467–3476, Minneapolis, Minnesota. Association for Computational Linguistics.

James Jhirad, Edison Marrese-Taylor, and Yutaka Matsuo. 2023. Evaluating Large Language Models’ Understanding of Financial Terminology via Definition Modeling. In Proceedings of the 13th International Joint Conference on Natural Language Processing and the 3rd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics: Student Research Workshop, pages 93–100, Nusa Dua, Bali. Association for Computational Linguistics.

Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. 2023. Mistral 7b. Preprint, arXiv:2310.06825.

Arman Kabiri and Paul Cook. 2020. Evaluating a multi-sense definition generation model for multiple languages. In Text, Speech, and Dialogue: 23rd International Conference, TSD 2020, Brno, Czech Republic, September 8–11, 2020, Proceedings 23, pages 153–161. Springer.

Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. 2022. Large Language Models are Zero-Shot Reasoners. In Advances in Neural Information Processing Systems.

Cunliang Kong, Liner Yang, Tianzuo Zhang, Qinan Fan, Zhenghao Liu, Yun Chen, and Erhong Yang. 2020. Toward Cross-Lingual Definition Generation for Language Learners. Preprint, arxiv:2010.05533.

Viet Lai, Chien Nguyen, Nghia Ngo, Thuat Nguyen, Franck Demoncourt, Ryan Rossi, and Thien Nguyen. 2023. Okapi: Instruction-tuned large language models in multiple languages with reinforcement learning from human feedback. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 318–327, Singapore. Association for Computational Linguistics.

Guillaume Lample, Alexis Conneau, Marc’Aurelio Ranzato, Ludovic Denoyer, and Hervé Jégou. 2018. Word translation without parallel data. In International Conference on Learning Representations.

Yucheng Li, Yunhao Guo, Frank Guerin, and Chenghua Lin. 2024. An open-source data contamination report for large language models. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 528–541, Miami, Florida, USA. Association for Computational Linguistics.

Yinhan Liu, Jiatao Gu, Naman Goyal, Xian Li, Sergey Edunov, Marjan Ghazvininejad, Mike Lewis, and Luke Zettlemoyer. 2020. Multilingual Denoising
Pre-training for Neural Machine Translation. Transactions of the Association for Computational Linguistics, 8:726–742.

Zequn Liu, Shukai Wang, Yiyang Gu, Ruiyi Zhang, Ming Zhang, and Sheng Wang. 2021. Graphine: A Dataset for Graph-aware Terminology Definition Generation. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 3453–3463, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.

Marco Lui and Timothy Baldwin. 2012. langid.py: An off-the-shelf language identification tool. In Proceedings of the ACL 2012 System Demonstrations, pages 25–30, Jeju Island, Korea. Association for Computational Linguistics.

Ke Ni and William Yang Wang. 2017. Learning to Explain Non-Standard English Words and Phrases. In Proceedings of the Eighth International Joint Conference on Natural Language Processing (Volume 2: Short Papers), pages 413–417, Taipei, Taiwan. Asian Federation of Natural Language Processing.

Thanapon Noraset, Chen Liang, Larry Birnbaum, and Doug Downey. 2017. Definition modeling: Learning to define word embeddings in natural language. In Proceedings of the Thirty-First AAAI Conference on Artificial Intelligence, AAAI’17, pages 3259–3266, San Francisco, California, USA. AAAI Press.

Jane Pan, Tianyu Gao, Howard Chen, and Danqi Chen. 2023. What In-Context Learning “Learns” In-Context: Disentangling Task Recognition and Task Learning. In Findings of the Association for Computational Linguistics: ACL 2023, pages 8298–8319, Toronto, Canada. Association for Computational Linguistics.

Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pages 311–318, Philadelphia, Pennsylvania, USA. Association for Computational Linguistics.

Francesco Periti, David Alfter, and Nina Tahmasebi. 2024. Automatically Generated Definitions and their utility for Modeling Word Meaning. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 14008–14026, Miami, Florida, USA. Association for Computational Linguistics.

Felipe Maia Polo, Lucas Weber, Leshem Choshen, Yuekai Sun, Gongjun Xu, and Mikhail Yurochkin. 2024. tinybenchmarks: evaluating LLMs with fewer examples. In Forty-first International Conference on Machine Learning.

Matt Post. 2018. A call for clarity in reporting BLEU scores. In Proceedings of the Third Conference on Machine Translation: Research Papers, pages 186–191, Brussels, Belgium. Association for Computational Linguistics.

Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer. Journal of Machine Learning Research, 21(140):1–67.

Ricardo Rei, Craig Stewart, Ana C Farinha, and Alon Lavie. 2020. COMET: A Neural Framework for MT Evaluation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2685–2702, Online. Association for Computational Linguistics.

Machel Reid, Edison Marrese-Taylor, and Yutaka Matsuo. 2020. VCDM: Leveraging Variational Bi-encoding and Deep Contextualized Word Representations for Improved Definition Modeling. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6331–6344, Online. Association for Computational Linguistics.

Oscar Sainz, Jon Campos, Iker García-Ferrero, Julen Etxaniz, Oier Lopez de Lacalle, and Eneko Agirre. 2023. NLP evaluation in trouble: On the need to measure LLM data contamination for each benchmark. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 10776–10787, Singapore. Association for Computational Linguistics.

Franck Sajous, Basilio Calderone, and Nabil Hathout. 2020. ENGLAWI: From Human- to Machine-Readable Wiktionary. In 12th International Conference on Language Resources and Evaluation (LREC 2020), page 3016.

Yuqing Tang, Chau Tran, Xian Li, Peng-Jen Chen, Naman Goyal, Vishrav Chaudhary, Jiatao Gu, and Angela Fan. 2020. Multilingual Translation with Extensible Multilingual Pretraining and Finetuning. Preprint, arxiv:2008.00401.

Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esibou, Jude Fernandes, Jeremy Fu, Wenyin Fu, and 49 others. 2023. Llama 2: Open foundation and fine-tuned chat models. Preprint, arXiv:2307.09288.

Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. 2021. mT5: A Massively Multilingual Pre-trained Text-to-Text Transformer. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 483–498, Online. Association for Computational Linguistics.
Tatu Ylonen. 2022. Wiktextextract: Wiktionary as Machine-Readable Structured Data. In Proceedings of the Thirteenth Language Resources and Evaluation Conference, pages 1317–1325, Marseille, France. European Language Resources Association.

Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. 2019. HellaSwag: Can a machine really finish your sentence? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4791–4800, Florence, Italy. Association for Computational Linguistics.

Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2019. BERTScore: Evaluating Text Generation with BERT. In International Conference on Learning Representations.
A Data Sources

A.1 Word Lists
• For German: https://gist.github.com/MarvinJWendt/2f4f4154b8ae218600eb091a5706b5f4
• For Spanish: https://github.com/keepassxreboot/keepassxc/discussions/9854 (Real Academia Española Corpus de Referencia del Español Actua (RAE, CREA))
• For French: https://github.com/kkrypt0nn/wordlists/blob/main/wordlists/languages/french.txt
• For Portuguese: https://natura.di.uminho.pt/download/sources/Dictionaries/wordlists/

A.2 Dictionaries
The list below contains links to the Terms of Service for each one of the dictionaries used in our work, as well as relevant data sources from previous work, together with the specific sections where access/distribution rights are stated.

• Oxford (Gadetskyy et al., 2018)
  – https://www.oed.com/information/about-the-oed/legal-notice
• Urban Dictionary (Ni and Wang, 2017)
  – https://about.urbandictionary.com/tos/
• Cambridge (Ishiwatari et al., 2019)
  – https://www.cambridge.org/legal/copyright
• Le Petit Robert (Reid et al., 2020)
  – https://dictionnaire.lerobert.com/mentions-legales PROPRIÉTÉ INTELLECTUELLE
• Dicio (ours)
  – https://www.dicio.com.br/termos-de-uso.html
• DRAE (ours)
  – https://dle.rae.es/contenido/aviso-legal Point 4
  – https://www.duden.de/service/agb Point 11.1
• Larousse (ours)
  – https://www.larousse.fr/infos/cgu Section PROPRIÉTÉ INTELLECTUELLE

In our paper, we utilize this data in a way that does not necessarily align with the intended use as defined by the providers. As mentioned in §3, in order to make sure we comply with the Terms of Services of our data sources, we will not distribute the definition data we collected, but only the wordlists we compiled. For the same reason, we do not run few-shot experiments with ChatGPT.

B Experimental Setup Details

Model Training We train our models with the AdamW optimizer, with an initial learning rate of 5e-5, using a batch size of 16 for mBART-50 or 8 for mT5 and for a maximum of 20 epochs. During inference, we sample from the output distribution with a maximum length of 256.

Prompting Regarding the system prompt, for LLAMA-2, we follow the approach by the original paper but replace the original one\footnote{https://github.com/huggingface/transformers/blob/ee2a3400f2a7038a23b83a39c5d0e24f7f699561/src/transformers/models/llama/tokenization_llama.py#L57, https://github.com/facebookresearch/llama/blob/main/UPDATES.md} with the phrase “*You are a helpful assistant. Always answer as helpfully and concisely as possible.*”, to help reduce false refusal rates, i.e., when the model incorrectly refuses to answer a question that it should, for example, due to overly broad instructions to be cautious in how it provides responses. In our case, we observed that this often happened whenever we asked the model to define words that could be offensive in certain contexts. In the case of MISTRAL, we simply utilize the original system prompt setting. To generate definitions, we sample from the output distribution using *top_k=40, top_p=0.95*, using a repetition penalty of 1. We generate a maximum number of 256 tokens, which falls well below the length of our gold standard definitions.

Evaluation To compute the compliance of a language model, i.e. its ability to consistently generate definitions in the required language, we use
fasttext-based language classification model (Grave et al., 2018). This is a linear model on top of said embeddings, which recognize 176 languages and was trained on 400 million tokens from Wikipedia, as well as sentences from the Tatoeba website. In terms of performance, this model is superior to previous work, including the readily available ‘langid.py’ by Lui and Baldwin (2012), in the TCL, Wikipedia, and EuroGov benchmarks (Baldwin and Lui, 2010), with accuracies of 94.7, 93.0 and 98.7 respectively.

C Detailed Results

C.1 Model Training Detailed Results

In addition to experimenting with mT5, we also trained models based on mBART (Liu et al., 2020; Tang et al., 2020). The original mBART model was pre-trained in denoising full texts in multiple languages. We worked with the “large” variation (approx. 340 M parameters) pre-trained on 25 languages (mBART-cc25) (Liu et al., 2020). Since Portuguese was not included, we also use mBART-50 (mBART-50) (Tang et al., 2020), which was initialized with the former and trained to cover other 25 languages.

Table 8 summarizes the results of our monolingual experiments when training models. As we compare the performance of other languages against English, we see that our models can attain similar performance in terms of COMET, with Spanish obtaining the best results. In terms of BLEU scores, we observe an unusually high value for Portuguese, which is due to the limitations of our lemmatization-based data cleaning technique and leads to some data leakage problems. Based on our monolingual finetuning experiments, we utilize mT5 as our base model for the multilingual experiments. Table 9 summarized our results with mT5 on the validation split.

<table>
  <tr>
    <th rowspan="2">Dataset</th>
    <th rowspan="2">Model</th>
    <th colspan="2">Valid</th>
    <th colspan="2">Test</th>
  </tr>
  <tr>
    <th>BL</th>
    <th>CMT</th>
    <th>BL</th>
    <th>CMT</th>
  </tr>
  <tr>
    <td rowspan="3">DUDEN (de)</td>
    <td>mBART-cc25</td>
    <td>4.06</td>
    <td>0.338</td>
    <td>1.732</td>
    <td><b>0.348</b></td>
  </tr>
  <tr>
    <td>mBART-50</td>
    <td><b>6.47</b></td>
    <td>0.338</td>
    <td><b>2.477</b></td>
    <td>0.347</td>
  </tr>
  <tr>
    <td>mT5</td>
    <td>4.64</td>
    <td><b>0.343</b></td>
    <td>0.939</td>
    <td>0.346</td>
  </tr>
  <tr>
    <td rowspan="3">OXFORD (en)</td>
    <td>mBART-cc25</td>
    <td>2.08</td>
    <td>0.340</td>
    <td>0.00</td>
    <td><b>0.345</b></td>
  </tr>
  <tr>
    <td>mBART-50</td>
    <td><b>3.19</b></td>
    <td>0.340</td>
    <td><b>2.61</b></td>
    <td>0.341</td>
  </tr>
  <tr>
    <td>mT5</td>
    <td>2.71</td>
    <td><b>0.341</b></td>
    <td>1.33</td>
    <td><b>0.345</b></td>
  </tr>
  <tr>
    <td rowspan="3">DRAE (es)</td>
    <td>mBART-cc25</td>
    <td>7.28</td>
    <td>0.382</td>
    <td>7.07</td>
    <td><b>0.385</b></td>
  </tr>
  <tr>
    <td>mBART-50</td>
    <td>6.11</td>
    <td>0.378</td>
    <td>6.10</td>
    <td>0.383</td>
  </tr>
  <tr>
    <td>mT5</td>
    <td><b>9.19</b></td>
    <td><b>0.379</b></td>
    <td><b>7.70</b></td>
    <td>0.384</td>
  </tr>
  <tr>
    <td rowspan="3">LAROUSSE (fr)</td>
    <td>mBART-cc25</td>
    <td>2.59</td>
    <td><b>0.295</b></td>
    <td>2.05</td>
    <td><b>0.300</b></td>
  </tr>
  <tr>
    <td>mBART-50</td>
    <td>3.01</td>
    <td><b>0.295</b></td>
    <td><b>1.73</b></td>
    <td>0.298</td>
  </tr>
  <tr>
    <td>mT5</td>
    <td><b>3.68</b></td>
    <td><b>0.295</b></td>
    <td>0.49</td>
    <td><b>0.300</b></td>
  </tr>
  <tr>
    <td rowspan="2">DICIO (pt)</td>
    <td>mBART-50</td>
    <td>34.02</td>
    <td>0.316</td>
    <td><b>18.58</b></td>
    <td>0.351</td>
  </tr>
  <tr>
    <td>mT5</td>
    <td><b>44.77</b></td>
    <td><b>0.317</b></td>
    <td>11.53</td>
    <td><b>0.352</b></td>
  </tr>
</table>

Table 8: Results of our monolingual experiments, fine-tuning multilingual language models on each of our datasets. In the table, CMT is short for COMET and BL is short for BLEU.

serve as a reasonable estimate for the overall performance gap due to the effect of quantization. We run each experiments three times with different random seeds, and report average results, which we summarize in Table 11.

Hardware Our mBART-50 and mT5 finetuning experiments were performed on a large cluster, where we usually relied on a node with 4 NVIDIA V100 GPUs, or an instance with a single NVIDIA A100 GPU. For inference with LLMs and LLM finetuning, we used nodes with 1 NVIDIA H200 GPU. We spent a total of approximately 3,000 USD in our experimental setup, most of which is due to improving the robustness with of our study by repeating LLM experiments with multiple seeds and to running the multilingual LLM benchmarks.

C.2 Model Prompting Details

Statistical Significance Details Table 10 below shows the details of the p-values for the experiments with LLMs.

Role of Quantization Quantization has been shown to be detrimental to performance in certain scenarios, we also evaluate the influence of this technique on our task. For this study, we limit ourselves to the zero-shot scenario and to the DRAE, LAROUSSE and DICIO, which we believe should

C.3 Correlation with Benchmarks

C.4 Finetuninig LLMs

D Human Evaluation Details

Table 14 shows the details of our Likert evaluation scheme, which includes example generations we used as guidance to define our score categories. Participants were verbally instructed about the task and informed that their answers would only be used for the purposes of this study, and they all agreed to participate on their on volition. They are all members of an Engineering department of a University, with ages ranging between 20 and 40
<table>
  <tr>
    <th>Dataset</th>
    <th>Training Data</th>
    <th>BLEU</th>
    <th>COMET</th>
  </tr>
  <tr>
    <td rowspan="2">DUDEN (de)</td>
    <td>de</td>
    <td><b>4.640</b></td>
    <td>0.343</td>
  </tr>
  <tr>
    <td>de + en</td>
    <td>0.497</td>
    <td><b>0.345</b></td>
  </tr>
  <tr>
    <td rowspan="4">OXFORD (en)</td>
    <td>en</td>
    <td><b>2.706</b></td>
    <td>0.341</td>
  </tr>
  <tr>
    <td>en + fr</td>
    <td>0.040</td>
    <td><b>0.347</b></td>
  </tr>
  <tr>
    <td>en + de</td>
    <td>0.775</td>
    <td><b>0.347</b></td>
  </tr>
  <tr>
    <td>en + fr + es + pt</td>
    <td>0.539</td>
    <td><b>0.347</b></td>
  </tr>
  <tr>
    <td rowspan="5">DRAE (es)</td>
    <td>es</td>
    <td>9.19</td>
    <td>0.379</td>
  </tr>
  <tr>
    <td>es + fr</td>
    <td>8.589</td>
    <td>0.378</td>
  </tr>
  <tr>
    <td>es + pt</td>
    <td><b>9.519</b></td>
    <td><b>0.380</b></td>
  </tr>
  <tr>
    <td>es + pt + fr</td>
    <td>9.493</td>
    <td>0.379</td>
  </tr>
  <tr>
    <td>en + fr + es + pt</td>
    <td>6.365</td>
    <td><b>0.380</b></td>
  </tr>
  <tr>
    <td rowspan="5">LAROUSSE (fr)</td>
    <td>fr</td>
    <td>3.675</td>
    <td>0.295</td>
  </tr>
  <tr>
    <td>fr + es</td>
    <td>2.603</td>
    <td>0.295</td>
  </tr>
  <tr>
    <td>fr + pt</td>
    <td>0.904</td>
    <td><b>0.299</b></td>
  </tr>
  <tr>
    <td>fr + en</td>
    <td>0.040</td>
    <td>0.298</td>
  </tr>
  <tr>
    <td>fr + es + pt</td>
    <td><b>4.353</b></td>
    <td>0.294</td>
  </tr>
  <tr>
    <td rowspan="5">DICIO (pt)</td>
    <td>pt</td>
    <td><b>44.772</b></td>
    <td>0.317</td>
  </tr>
  <tr>
    <td>pt + es</td>
    <td>41.538</td>
    <td>0.317</td>
  </tr>
  <tr>
    <td>pt + fr</td>
    <td>7.464</td>
    <td><b>0.318</b></td>
  </tr>
  <tr>
    <td>pt + fr + es</td>
    <td>43.229</td>
    <td>0.317</td>
  </tr>
  <tr>
    <td>en + fr + es + pt</td>
    <td>12.050</td>
    <td>0.316</td>
  </tr>
</table>

Table 9: Results of our multilingual experiments based on mT5 on the validation set, where BL is short for BLEU and CMT is short for COMET.

<table>
  <tr>
    <th>Dataset</th>
    <th>Model</th>
    <th>BL</th>
    <th>BS</th>
    <th>CMT</th>
    <th>Cmp.</th>
  </tr>
  <tr>
    <td>DUDEN (de)</td>
    <td>LLAMA-2</td>
    <td>0.003</td>
    <td>0.000</td>
    <td>0.002</td>
    <td>0.382</td>
  </tr>
  <tr>
    <td></td>
    <td>MISTRAL</td>
    <td>0.420</td>
    <td>0.212</td>
    <td>0.004</td>
    <td>0.372</td>
  </tr>
  <tr>
    <td>OXFORD (en)</td>
    <td>LLAMA-2</td>
    <td>0.000</td>
    <td>0.000</td>
    <td>0.037</td>
    <td>-</td>
  </tr>
  <tr>
    <td></td>
    <td>MISTRAL</td>
    <td>0.008</td>
    <td>0.002</td>
    <td>0.002</td>
    <td>-</td>
  </tr>
  <tr>
    <td>DRAE (es)</td>
    <td>LLAMA-2</td>
    <td>0.011</td>
    <td>0.000</td>
    <td>0.001</td>
    <td>0.011</td>
  </tr>
  <tr>
    <td></td>
    <td>MISTRAL</td>
    <td>0.096</td>
    <td>0.037</td>
    <td>0.003</td>
    <td>0.021</td>
  </tr>
  <tr>
    <td>LAROUSSE (fr)</td>
    <td>LLAMA-2</td>
    <td>0.001</td>
    <td>0.000</td>
    <td>0.263</td>
    <td>0.970</td>
  </tr>
  <tr>
    <td></td>
    <td>MISTRAL</td>
    <td>0.003</td>
    <td>0.000</td>
    <td>0.040</td>
    <td>0.775</td>
  </tr>
  <tr>
    <td>DICIO (pt)</td>
    <td>LLAMA-2</td>
    <td>0.185</td>
    <td>0.602</td>
    <td>0.597</td>
    <td>0.295</td>
  </tr>
  <tr>
    <td></td>
    <td>MISTRAL</td>
    <td>0.027</td>
    <td>0.012</td>
    <td>0.035</td>
    <td>0.000</td>
  </tr>
</table>

Table 10: P-values of t-test comparing the model’s performance in zero and five shot scenarios.

<table>
  <tr>
    <th>Dataset</th>
    <th>Model</th>
    <th>Type</th>
    <th>BL</th>
    <th>CMT</th>
  </tr>
  <tr>
    <td rowspan="2">DRAE (es)</td>
    <td>LLAMA-2</td>
    <td>4bit</td>
    <td>0.271</td>
    <td>0.504</td>
  </tr>
  <tr>
    <td>MISTRAL</td>
    <td>full</td>
    <td>0.295</td>
    <td>0.510</td>
  </tr>
  <tr>
    <td></td>
    <td></td>
    <td>4bit</td>
    <td>0.324</td>
    <td>0.458</td>
  </tr>
  <tr>
    <td></td>
    <td></td>
    <td>full</td>
    <td>0.333</td>
    <td>0.469</td>
  </tr>
  <tr>
    <td rowspan="2">LAROUSSE (fr)</td>
    <td>LLAMA-2</td>
    <td>4bit</td>
    <td>0.326</td>
    <td>0.473</td>
  </tr>
  <tr>
    <td>MISTRAL</td>
    <td>full</td>
    <td>0.380</td>
    <td>0.483</td>
  </tr>
  <tr>
    <td></td>
    <td></td>
    <td>4bit</td>
    <td>0.327</td>
    <td>0.441</td>
  </tr>
  <tr>
    <td></td>
    <td></td>
    <td>full</td>
    <td>0.432</td>
    <td>0.463</td>
  </tr>
  <tr>
    <td rowspan="2">DICIO (pt)</td>
    <td>LLAMA-2</td>
    <td>4bit</td>
    <td>0.143</td>
    <td>0.513</td>
  </tr>
  <tr>
    <td>MISTRAL</td>
    <td>full</td>
    <td>0.120</td>
    <td>0.515</td>
  </tr>
  <tr>
    <td></td>
    <td></td>
    <td>4bit</td>
    <td>0.146</td>
    <td>0.475</td>
  </tr>
  <tr>
    <td></td>
    <td></td>
    <td>full</td>
    <td>0.138</td>
    <td>0.489</td>
  </tr>
</table>

Table 11: Results of our study on the impact of low precision (4 bit). In the table, BL and CMT are short for BLEU and COMET, respectively.

years old.

As mentioned above, in order to further understand the effect of in-context learning in our task, and the effectiveness of COMET as a metric, we perform an additional experiment by using this metric to select the 50 best and 50 worst generations, again resulting in a total of 400 definitions to be evaluated per annotator. We choose the same random seed for all models. We exclude English from this study as this language has been studied before for our task, and because our initial evaluation shows that performance substantially superior.

Table 15 summarizes the results of this second experiment. We first notice that the effects of in context learning are again not consistent across models and languages. We often find that results for “Worse” sets contain rare terms, which we assume should also have relative lower frequency in the training data. Given the nature of our task, we think this shows that although few-shot learning may help the model achieve a more dictionary-like style when generating, it does not elicit better task-solving capabilities for those cases.

We also see observe that Likert scores on the “Best” sets are higher than on the “Worst” sets, which suggests that COMET is indeed able to capture the semantic quality of the generated definitions. However, from a qualitative point of few, feedback from our annotators suggest that while LLAMA-2 generations were more precise and correct, MISTRAL generations were rather “Very good”, with rich details, or “Very Poor”, containing wrong information; however, COMET could not detect such differences. These findings are in agreement with previous observations by Jhi-rad et al. (2023), and provide additional evidence to show that the performance of COMET when capturing semantic similarity decreases substantially with length of the inputs.

E Performance on multilingual benchmarks

Table 16 summarizes the result we obtained when running our chosen LLMs on the multilingual benchmarks HellaSwag, MMLU and ARC-Challenge. Following (Lai et al., 2023), experiments on HellaSwag are zero-shot, and we use 25 shots for MMLU and ARC in languages other than
<table>
  <tr>
    <th>Model</th>
    <th>Score</th>
    <th>BL</th>
    <th>BS</th>
    <th>CMT</th>
  </tr>
  <tr>
    <td rowspan="3">LLAMA-2</td>
    <td>HellaSwag</td>
    <td>0.49</td>
    <td>0.95</td>
    <td>0.74</td>
  </tr>
  <tr>
    <td>MMLU</td>
    <td>0.55</td>
    <td>0.97</td>
    <td>0.72</td>
  </tr>
  <tr>
    <td>ARC-Challenge</td>
    <td>0.61</td>
    <td>0.88</td>
    <td>0.87</td>
  </tr>
  <tr>
    <td rowspan="3">MISTRAL</td>
    <td>HellaSwag</td>
    <td>0.13</td>
    <td>0.95</td>
    <td>0.73</td>
  </tr>
  <tr>
    <td>MMLU</td>
    <td>0.04</td>
    <td>0.91</td>
    <td>0.63</td>
  </tr>
  <tr>
    <td>ARC-Challenge</td>
    <td>0.32</td>
    <td>0.99</td>
    <td>0.84</td>
  </tr>
  <tr>
    <td rowspan="3">LLAMA-3</td>
    <td>HellaSwag</td>
    <td>-0.44</td>
    <td>0.93</td>
    <td>-0.18</td>
  </tr>
  <tr>
    <td>MMLU</td>
    <td>-0.24</td>
    <td>0.97</td>
    <td>-0.10</td>
  </tr>
  <tr>
    <td>ARC-Challenge</td>
    <td>-0.83</td>
    <td>0.71</td>
    <td>-0.25</td>
  </tr>
</table>

Table 12: Pearson correlation between performance in our task and multilingual benchmarks for LLMs.

<table>
  <tr>
    <th>Lang.</th>
    <th>Model</th>
    <th>BL</th>
    <th>BS</th>
    <th>CMT</th>
  </tr>
  <tr>
    <td rowspan="3">DUDEN (de)</td>
    <td>LLAMA-3 (ours)</td>
    <td>3.261</td>
    <td>0.676</td>
    <td>0.580</td>
  </tr>
  <tr>
    <td>+ ft (ours)</td>
    <td>0.794</td>
    <td>0.585</td>
    <td>0.411</td>
  </tr>
  <tr>
    <td>+ ft (<a href="https://periti.github.io/">Periti et al.</a>)</td>
    <td>0.147</td>
    <td>0.574</td>
    <td>0.360</td>
  </tr>
  <tr>
    <td rowspan="3">DRAE (es)</td>
    <td>LLAMA-3 (ours)</td>
    <td>1.855</td>
    <td>0.670</td>
    <td>0.545</td>
  </tr>
  <tr>
    <td>+ ft (ours)</td>
    <td>0.634</td>
    <td>0.584</td>
    <td>0.460</td>
  </tr>
  <tr>
    <td>+ ft (<a href="https://periti.github.io/">Periti et al.</a>)</td>
    <td>0.186</td>
    <td>0.570</td>
    <td>0.379</td>
  </tr>
  <tr>
    <td rowspan="3">OXFORD (en)</td>
    <td>LLAMA-3 (ours)</td>
    <td>1.588</td>
    <td>0.568</td>
    <td>0.835</td>
  </tr>
  <tr>
    <td>+ ft (ours)</td>
    <td>0.568</td>
    <td>0.511</td>
    <td>0.806</td>
  </tr>
  <tr>
    <td>+ ft (<a href="https://periti.github.io/">Periti et al.</a>)</td>
    <td>0.867</td>
    <td>0.455</td>
    <td>0.799</td>
  </tr>
  <tr>
    <td rowspan="3">LAROUSSE (fr)</td>
    <td>LLAMA-3 (ours)</td>
    <td>3.443</td>
    <td>0.676</td>
    <td>0.543</td>
  </tr>
  <tr>
    <td>+ ft (ours)</td>
    <td>0.809</td>
    <td>0.608</td>
    <td>0.496</td>
  </tr>
  <tr>
    <td>+ ft (<a href="https://periti.github.io/">Periti et al.</a>)</td>
    <td>0.266</td>
    <td>0.595</td>
    <td>0.388</td>
  </tr>
  <tr>
    <td rowspan="3">DICIO (pt)</td>
    <td>LLAMA-3 (ours)</td>
    <td>0.457</td>
    <td>0.683</td>
    <td>0.542</td>
  </tr>
  <tr>
    <td>+ ft (ours)</td>
    <td>1.648</td>
    <td>0.598</td>
    <td>0.467</td>
  </tr>
  <tr>
    <td>+ ft (<a href="https://periti.github.io/">Periti et al.</a>)</td>
    <td>0.219</td>
    <td>0.592</td>
    <td>0.393</td>
  </tr>
</table>

Table 13: Summary of the performance of our LLAMA-3 model finetuned on monosemic multilingual dictionary data on our datasets, compared against LLamaDict (<a href="https://periti.github.io/">Periti et al., 2024</a>) [Highlight best values –EMT].

English.

F Tackling Polysemy

Our main results are limited to the set of terms for which we only have one definition. Though our results suggest that LLMs can, to some extent, generate plausible definitions for words, our empirical study offers no insight into the more challenging scenario of polysemy.

Given the success of our studied LLMs in generating definitions for our selected languages, we propose to utilize LLMs to obtain example sentences for polysemic terms in our data. We focus on Portuguese given the annotators’ availability.

We run a pilot study with to generate such example sentences. Concretely, we prompt the model as follows: In the Portuguese Language, the word '{term}' can mean "{definition}". Please give me a sentence in that language where this word is used., where {term} and {definition} are placeholders for variables denoting a given pair of a term and its corresponding definition.

For our study, we sample a subset of 59 terms from our data, with a total of 156 different meanings. We utilize the official OpenAI API to feed this data into the model. To assess the viability of this approach, we subject these sentences to a thorough human evaluation. Specifically, we ask a native Brazilian Portuguese speaker to evaluate the generated sentences according to the following simplified Likert scale: 1 - Poor: The sentence uses the term with a different meaning; 2 - Acceptable: The sentence uses the term with the specified meaning, but in the wrong context/sounds unnatural; 3 - Good: The sentence uses the term with the specified meaning.

Our results showed that 75.64% of the examples we obtained were labeled as “Good” by our human evaluator, while only 7.69% and 16.67% were labeled as “Poor” and “Acceptable”, respectively.

The results of this evaluation are in Table 18. We can see that most of the generated examples received the “Good” score, with an average of 2.68, which indicates that using this specific LLM could be an alternative to obtain such example sentences for Brazilian Portuguese.
<table>
  <tr>
    <th>Category</th>
    <th>Details</th>
    <th>Example</th>
  </tr>
  <tr>
    <td>Very poor</td>
    <td>A completely wrong definition, one that contains only the word, or its English translation.</td>
    <td><b>Waterspot (Tromba d’água)</b><br>Definition: Phenomenon defined by the formation of a mass of vapors over rivers, lakes, seas, generally in the shape of a funnel, a cone with the base facing the clouds.<br>Generated: Sure! “Waterspot” is a term used to designate a type of flooding caused by a river flood or storm surge, which can flood coastal or plain areas.</td>
  </tr>
  <tr>
    <td>Poor</td>
    <td>A definition of a related word.</td>
    <td><b>To sharpen (Nitidificar)</b><br>Definition: To make something clear, perceptible, sharp; To clear a substance; the concept became clear with your explanation.<br>Generated: To sharpen is a verb that means “to enhance” or “to beautify.” It is used to describe the action of improving or enriching something, it can be applied to objects, situations or people. Example: “The sun on the beach sharpened the sea with its rays”.</td>
  </tr>
  <tr>
    <td>Acceptable</td>
    <td>It gives an idea of the broad sense of the word</td>
    <td><b>Supralegal (Supralegal)</b><br>Definition: [Legal] Above the law; that, hierarchically, occupies a superior position in relation to legislation.<br>Generated: Of course! The term “supralegal” in Portuguese means “beyond the law” or “outside the law”. It is used to describe acts or situations that are illegal or that are not subject to any type of legal regulation.</td>
  </tr>
  <tr>
    <td>Good</td>
    <td>Gives the correct word definition, but it has some imprecision (e.g. wrong use cases)</td>
    <td><b>To oversize (Superdimensionar)</b><br>Definition: To give large dimensions to; give an exaggerated size, an exaggerated scope to something; to exaggerate the achievements of a government.<br>Generated: Of course! The Portuguese “oversize” is a term that is not very common, but can be used to describe an action or process that increases or exaggerates a characteristic or property of something. Most of the time, it is used to criticize situations in which someone is exaggerating or increasing something disproportionately. For example, if someone is exaggerating their ability in a certain subject, it could be said that they are “oversizing” their ability.</td>
  </tr>
  <tr>
    <td>Very good</td>
    <td>Correct definition with no imprecision.</td>
    <td><b>To sanitize (Sanitizar)</b><br>Definition: To comply with hygiene requirements or recommendations; to clean; To sanitize an environment.<br>Generated: Of course! The Portuguese “sanitize” refers to eliminating or removing anything that could be considered unclean or dangerous to health. For example, it may refer to disinfecting surfaces or objects, or eliminating bacteria or viruses from an environment. In short, “sanitize” means to purify or disinfect.</td>
  </tr>
</table>

Table 14: Explanation of our human evaluation system for generated definitions. Examples are taken from DICIO and manually translated to English by the annotator.

<table>
  <tr>
    <th>Model</th>
    <th>Likert (b/w)</th>
    <th>COMET (b/w)</th>
  </tr>
  <tr>
    <td>DUDEN (de)</td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>LLAMA-2<br>+ 5 shots</td>
    <td>3.060 / 1.100<br>1.400 / 1.040</td>
    <td>0.670 / 0.241<br>0.767 / 0.194</td>
  </tr>
  <tr>
    <td>MISTRAL<br>+ 5 shots</td>
    <td>2.560 / 1.000<br>1.920 / 1.000</td>
    <td>0.699 / 0.211<br>0.700 / 0.197</td>
  </tr>
  <tr>
    <td>DRAE (es)</td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>LLAMA-2<br>+ 5 shots</td>
    <td>4.680 / 1.100<br>4.900 / 1.100</td>
    <td>0.700 / 0.317<br>0.751 / 0.235</td>
  </tr>
  <tr>
    <td>MISTRAL<br>+ 5 shots</td>
    <td>4.600 / 1.080<br>4.440 / 1.140</td>
    <td>0.727 / 0.258<br>0.710 / 0.236</td>
  </tr>
  <tr>
    <td>LAROUSSE (fr)</td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>LLAMA-2<br>+ 5 shots</td>
    <td>4.280 / 1.122<br>4.460 / 1.260</td>
    <td>0.697 / 0.281<br>0.736 / 0.239</td>
  </tr>
  <tr>
    <td>MISTRAL<br>+ 5 shots</td>
    <td>4.061 / 1.000<br>4.040 / 1.000</td>
    <td>0.705 / 0.233<br>0.689 / 0.228</td>
  </tr>
  <tr>
    <td>DICIO (pt)</td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>LLAMA-2<br>+ 5 shots</td>
    <td>3.860 / 1.240<br>4.000 / 1.460</td>
    <td>0.639 / 0.385<br>0.623 / 0.339</td>
  </tr>
  <tr>
    <td>MISTRAL<br>+ 5 shots</td>
    <td>3.260 / 1.140<br>1.780 / 1.180</td>
    <td>0.577 / 0.320<br>0.530 / 0.306</td>
  </tr>
</table>

Table 15: Results of our second human evaluation experiment in terms of average Likert scores for the best 50 / worst 50 denoted as (b/w); and compared to COMET.
<table>
  <tr>
    <th>Language</th>
    <th>Model</th>
    <th>HellaSwag</th>
    <th>MMLU</th>
    <th>ARC</th>
  </tr>
  <tr>
    <td rowspan="3">English (en)</td>
    <td>LLAMA-2</td>
    <td>0.607</td>
    <td>0.533</td>
    <td>0.462</td>
  </tr>
  <tr>
    <td>MISTRAL</td>
    <td>0.563</td>
    <td>0.534</td>
    <td>0.501</td>
  </tr>
  <tr>
    <td>LLAMA-3</td>
    <td>0.591</td>
    <td>0.681</td>
    <td>0.515</td>
  </tr>
  <tr>
    <td rowspan="3">French (fr)</td>
    <td>LLAMA-2</td>
    <td>0.460</td>
    <td>0.439</td>
    <td>0.418</td>
  </tr>
  <tr>
    <td>MISTRAL</td>
    <td>0.426</td>
    <td>0.413</td>
    <td>0.387</td>
  </tr>
  <tr>
    <td>LLAMA-3</td>
    <td>0.488</td>
    <td>0.544</td>
    <td>0.467</td>
  </tr>
  <tr>
    <td rowspan="3">German (de)</td>
    <td>LLAMA-2</td>
    <td>0.431</td>
    <td>0.431</td>
    <td>0.408</td>
  </tr>
  <tr>
    <td>MISTRAL</td>
    <td>0.395</td>
    <td>0.401</td>
    <td>0.358</td>
  </tr>
  <tr>
    <td>LLAMA-3</td>
    <td>0.468</td>
    <td>0.529</td>
    <td>0.442</td>
  </tr>
  <tr>
    <td rowspan="3">Portuguese (pt)</td>
    <td>LLAMA-2</td>
    <td>0.437</td>
    <td>0.434</td>
    <td>0.427</td>
  </tr>
  <tr>
    <td>MISTRAL</td>
    <td>0.415</td>
    <td>0.396</td>
    <td>0.402</td>
  </tr>
  <tr>
    <td>LLAMA-3</td>
    <td>0.487</td>
    <td>0.541</td>
    <td>0.487</td>
  </tr>
  <tr>
    <td rowspan="3">Spanish (es)</td>
    <td>LLAMA-2</td>
    <td>0.474</td>
    <td>0.441</td>
    <td>0.441</td>
  </tr>
  <tr>
    <td>MISTRAL</td>
    <td>0.433</td>
    <td>0.414</td>
    <td>0.393</td>
  </tr>
  <tr>
    <td>LLAMA-3</td>
    <td>0.506</td>
    <td>0.548</td>
    <td>0.492</td>
  </tr>
</table>

Table 16: Observed performance of LLAMA-2 and MISTRAL on our three selected multilingual benchmarks, separated by language.

<table>
  <tr>
    <th rowspan="2">Lang.</th>
    <th colspan="3">Existing Benchmark</th>
    <th colspan="2">Ours</th>
  </tr>
  <tr>
    <th>MMLU</th>
    <th>ARC</th>
    <th>HellaSwag</th>
    <th>Input</th>
    <th>Exp. Output</th>
  </tr>
  <tr>
    <td>fr</td>
    <td>53,431,809</td>
    <td>1,489,775</td>
    <td>735,614</td>
    <td>60,629</td>
    <td>105,753</td>
  </tr>
  <tr>
    <td>es</td>
    <td>54,083,040</td>
    <td>1,460,328</td>
    <td>707,524</td>
    <td>62,971</td>
    <td>64,925</td>
  </tr>
  <tr>
    <td>pt</td>
    <td>54,044,351</td>
    <td>1,459,704</td>
    <td>707,201</td>
    <td>231,425</td>
    <td>285,676</td>
  </tr>
  <tr>
    <td>de</td>
    <td>54,805,912</td>
    <td>1,459,108</td>
    <td>735,190</td>
    <td>139,289</td>
    <td>108,314</td>
  </tr>
  <tr>
    <td>Total</td>
    <td>216,365,112</td>
    <td>5,868,915</td>
    <td>2,885,529</td>
    <td>494,314</td>
    <td>564,668</td>
  </tr>
</table>

Table 17: Length of input (and expected output, where available), in number of tokens (via the LLAMA-2 tokenizer).

<table>
  <tr>
    <th>Poor</th>
    <th>Acceptable</th>
    <th>Good</th>
  </tr>
  <tr>
    <td>7.69</td>
    <td>16.67</td>
    <td>75.64</td>
  </tr>
</table>

Table 18: Results of the human evaluation of the generated example sentences. The numbers indicate the percentage of sentences evaluated in each category

Reviewer paper summaries: This paper extends existing work on definition modeling into a multilingual setting. The authors collect dictionary data for 4 non-English languages (es, fr, de, pt) where entries in the datasets correspond to pairs of lexical terms and sense definitions. Because these datasets do not come with usage context sentences, the authors evaluate the performance of language models on monosemic terms, focusing on two modeling setups: 1) mT5 models finetuned on monolingual or multilingual dictionary data and 2) large language models (LLMs) under zero-shot, few-shot, and finetuned setups. The results show that while the models can perform reasonably well on English, the level of performance does not seem to transfer toward other languages, even though some of these languages belong to the same language family and share lexical terms with similar morphologies and semantics. The authors also show high correlations between the performances of the definition modeling task and existing multilingual benchmarks, arguing that definition modeling makes an efficient and more natural alternative to evaluating multilingual performance of LLMs.

An interesting paper on multilingual definition modeling on four languages (Spanish, French, Portuguese, and German). The study uses two pre-trained multilingual language models on definition modeling. The the zero and few-shot capabilities of the models in this task, as well as their shortcomings, are analysed.
Summary of strengths: - The paper contributes a useful set of multilingual dictionary datasets that could facilitate further research in the field.

- Insightful results on the transferability of lexical semantic knowledge from English to other languages.

Clearly structured and well presented paper.
Objectives are clear and methodology is sound. So are the contributions of the work, mainly the extension to multiple languages and muptiple domains.
The results are promising - although the BLEU scores are not essentially improved, the improved consistency is an important outcome.
Summary of weaknesses: - The work did not discuss important recent work that is highly relevant to the study. For example, Giulianelli et al (2023; https://aclanthology.org/2023.acl-long.176/) studies definition generation under a very similar setup (although with context) and performed extensive testing on the T5 model family.

- The experimental design may affect the validity of the main findings. In particular, having the prompts in English may bias the models. See the question below for more details.

In some cases inter-annotator agreement is very low. These issues should be explained in more depth. 
The main question is whether the methodology, or an extension of it, can be applied in the context of polysemous terms, which is the natural state in the language.
Comments, suggestions and typos: Questions and suggestions:

- Why are the prompts always given in English instead of the respective languages? Would this prompting setup bias the models toward English in terms of performance? You mentioned on line 402 that the LLMs tend to generate outputs in English even when they’re asked about another language.
- The current study focuses on conventional usages only, which means that the models have likely seen these words many times in actual usages (at least for English). In light of this, how do your results distinguish between a model’s ability to generate definition entries (i.e., come up with a definition sentence for something it already knows) or its ability to create new usage (i.e., look at an unseen word form and attach a meaning to it).
- Could you clarify what you mean by “best results for each language-model combination” in Table 4? My understanding implies there would be a bolded number in every row that corresponds to a language-LM combination but that’s not always the case.
- In the human experiment, how do you distinguish whether the model output is being judged based on its semantic content or the writing style?

Typos:
- line 169, wrong citation mode used.
- line 172 & 177, ‘works’ -> ‘work’
- line 557, ‘our’ -> ‘or’
- line 580, incomplete sentence.

Excellent English
Overall assessment scores: 3

4