言語処理学会 第30回年次大会 発表論文集（2024年3月）

Zero-shot Deﬁnition Modelling for Portuguese
Edison Marrese-Taylor1, 2 , Erica Kido Shimomoto1 , Enrique Reid2
National Institute of Advanced Industrial Science and Technology1
Graduate School of Engineering, The University of Tokyo2{edison.marrese,kidoshimomoto.e}@aist.go.jp, erarvin2007@gmail.com

Abstract
In this paper, we propose to study the task of deﬁnition modeling for a new language: Portuguese. To that
end, we collect monolingual dictionary data and perform
an in-depth empirical study to test the multilingual abilities of Large Language Models (LLMs), utilizing zero-shotand few-shot approaches. We analyze the performance of
Llama-2 and Mistral LLMs on monosemic terms, as ourcollected data does not contain context information forpolysemic words. To address this limitation, we further
propose to utilize LLMs to generate usage examples that
can assist deﬁnition modeling of polysemic terms in the future. To validate our ﬁndings, we performed a pilot humanstudy to evaluate the quality of generated deﬁnitions andusage examples. Our results are encouraging and suggestthat LLMs can generate plausible deﬁnitions of words in
Portuguese and that the COMET metric aligns well withthe human evaluation. Finally, our human study indicatesthat using LLMs could be an alternative to obtain context
information of polysemic words in Portuguese.

1

Introduction

Deﬁnition modeling is the task of estimating the probability of a textual deﬁnition given a word [1]. This task has
been shown to give an arguably more transparent view ofthe extent to which a model captures syntax and semantics.
This task can be framed as a conditional generation, in
which the deﬁnition of the word or phrase is generated
given a conditioning variable. So far, existing works havefollowed the traditional approach, where models are trainedon a corpus of word-deﬁnition pairs, to be later tested
on how well they generate deﬁnitions. These approaches[2, 3, 4, 5] are mainly encoder-decoder based, encoding
contextual representation for a word/phrase and using thecontextual representation to generate the deﬁnition.

Despite the progress, previous work has predominantly
focused on the English language. Concretely, we ﬁnd justa handful of papers that work on languages other than Eng
lish, namely VCDM from [5] for French and the work of [6]for Chinese. We also ﬁnd that dictionary data for French
and Italian have been recently released, but there are so far
no approaches that leverage them [7, 8].
In light of these issues, we present a study on deﬁnition modeling from the perspective of a new language,
Portuguese, introducing a dataset total of approximately
191, 499 new terms entries and 279, 985 deﬁnitions, extending language coverage for the task.
Moreover, we note that the recent success of Large
Language Models (LLMs) showed that such models canachieve excellent performance on a wide variety of downstream tasks, utilizing zero-shot or few-shot approaches[9]. Despite these results, we ﬁnd limited work in assessing their multilingual capabilities. To shed light on thisissue, and motivated by the original ideas of [1], we test aselection of such models, namely Llama-2 [10] and Mistral 7B [11], on our collected data in Portuguese, eﬀectively
proposing to use deﬁnition modeling tasks as a probe totest the multilingual abilities of LLMs.
Our results show that prompting techniques on LLMs,
either in the zero-shot or few-shot scenarios, can be usedsuccessfully for deﬁnition modeling in Portuguese. However, we also observe that the output language can be challenging to control. Human evaluation of the generated
deﬁnitions by a native Brazilian Portuguese speaker showsthat while not tailored for this task, COMET scores can bea valid guide in the quality evaluation of such deﬁnitions.
Our experiments focused on monosemic terms, as ourcollected data does not contain context data; To address
this limitation, we further propose to utilize LLMs to generate usage examples that can assist deﬁnition modeling ofpolysemic words. We prompted GPT-3.5 [9] to generate― 206 ―

This work is licensed by the author(s) under CC BY 4.0(https://creativecommons.org/licenses/by/4.0/).

sentences using polysemic terms in diﬀerent contexts andPortuguese is listed among the top-5 most spoken nativeevaluated their quality through a human evaluation. Ourlanguage in the world and has been previously regarded

results indicate that using LLMs could be an alternative to

as one of the ten most inﬂuential languages in the world

obtain such example sentences for Portuguese.

[16]. Portuguese also shows in the top three Indo-European

2

languages with the largest number of speakers according

Related Work

to Wikipedia, with 236 M native speakers.

Our paper is primarily related to the seminal work by

We choose a readily available dictionary to serve as a

Noraset et al. [1] and Hill et al. [12], in which a model

source for our dataset, Dicio1）. We collect the term and theis tasked with generating a deﬁnition for a word given its

available deﬁnitions. Table 1 summarizes the main char-respective embedding, or with mapping dictionary deﬁ-acteristics of the collected data, compared against existing

nitions to lexical representations of words, respectively.

resources in English and French. We can see that our data

Later work has proposed improvements and extensions, in-is substantially richer.

troducing techniques and datasets to address shortcomings.
For example, Gatedesky et al. [2] address polysemy andTable 1: Summary of our collected dataset, compared toprior relevant corpora. In the table, Mono. stands forpresent a dataset from Oxford Dictionaries, where each definition is also supplemented with context sentences. Ni etMonosemic, i.e. terms with a single deﬁnition.

al. [3] proposed an approach for automatically explaining

DatasetTerms Defs.

slang English terms in a sentence and introduced another

Oxford - en

36,767 122,319 44.07%

dataset from Wikipedia. Ishiwatari et al. [4] proposed to

Le Petit Robert - fr 33,507

further rely on local and global contexts to disambiguate

Dicio - pt (ours)

and generate better deﬁnitions, also introducing a dataset

Mono.

-

-

191,499 279,985 77.41%

based on Cambridge Dictionaries and a dataset for French.

An important distinction between our corpus and recent

More recently, Huang et al. [13] studied the problem ofprior work is that our collected data does not contain exam-

deﬁnition speciﬁcity, tuning a model to account for hyper-

ples of word usage. Previous research has shown that thisfocused or highly general deﬁnitions. Finally, Chen et al.

information is critical in allowing models to disambiguate[14] recently proposed to unify the seminal ideas of reverse
dictionary and deﬁnition modeling in a single model to help

a speciﬁc meaning for a given term in the case of polysemy.
With this in mind, for our experiments in this paper, we

better understand word sense and embeddings.

select the subset of terms that exhibit only a single meaning

Previous work discussed so far has mostly focused on
deﬁnitions in the English language. One main exception inand thus experiment in a monosemic scenario.

this context is the work of Reid et al. [5], who presented thethe same term, as inﬂections of verbs (e.g., tenses) andﬁrst study on deﬁnition modelling for the French language

adjectives (e.g., gender) are present in the dictionary. This

with the release of a dataset collected from Le Petit Robert.

problem leads to an artiﬁcially inﬂated amount of available

Finally, we note that over the past few years, dictionary

data. To alleviate this issue, we rely on spacy2）to iden-

datasets in several languages derived from Wikictionary

tify word lemmas, utilizing the “core news sm” model for

have been released, including English (ENGLAWI) [15],Portuguese, keeping only the original term matched theFrench [7], and Italian [8]. We ﬁnd, however, that theselemmatized word. The resulting dataset is split into thedatasets are not accompanied by models that leverage them.

80/10/10 ratio. Table 2 below shows the exact details ofMoreover, an important distinction in this regard is thatthe split sizes.

these dictionaries are built on the base of crowdsourcing,where quality could be a concern.

3

We also note that our data contains multiple entries for4

Experiments
Since our interest is to utilize our task as a proxy to better

Data

understand the multilingual abilities of Large Language
Our interest in the Portuguese language derives from its
importance in terms of the number of native speakers, as1） https://www.dicio.com.br/2） https://spacy.io/― 207 ―

This work is licensed by the author(s) under CC BY 4.0(https://creativecommons.org/licenses/by/4.0/).

Table 2: Details of the size of each split for our collected
data, compared against the Oxford dataset.

a set of 100 generated deﬁnitions by each of the two chosenmodels in both settings, where we picked the 50 best and 50
worst generations based on the COMET metric, resulting

Dataset

Train

Valid

Test

Oxford (en)15,770

6,884

6,834

quality of the generated deﬁnitions based on the following

Dicio (pt)

118,591

14,824

14,824

Likert scale: 1 - Very poor: A deﬁnition of a completely

in a total of 400 deﬁnitions. The speaker has to evaluate thediﬀerent word / Wrong deﬁnition; 2 - Poor: A deﬁnition
models, our approach for deﬁnition modeling is based onof a related term; 3 - Acceptable: A vague deﬁnition ofprompting and in-context learning. Regarding models,

the term; 4 - Good: A deﬁnition of the term, but with few

we consider Llama2 [10], speciﬁcally the chat versions,mistakes; 5 - Very Good: A correct deﬁnition of the term.

and the recently-released Mistral models [11]. For theformer, we utilize the 13B-parameter models. The latter

4.1

Results

we quantize to 4-bits using QLoRA [17] in order to ﬁt into

Table 3: Results of our experiments using LLMs for zero-

our GPU memory. We test two settings, as detailed below:

shot and few-shot deﬁnition modelling in Portuguese. In
the table, Comp. is short for compliance, the % of cases

1. A zero-shot approach, where the model is directly
asked to generate the deﬁnition of the word

where the answer is in the correct language.

2. A 5-shot setting, where we incorporate termdeﬁnition examples in the prompt before requesting

Model

Performance
BLEU COMET Comp.

the deﬁnition for the target term. These shots are
Llama-2-13b-chat+ 5 shotsrandomly sampled from the training data and kept
constant across examples.
In all cases, the input to the model is “Deﬁne the{language} word ’{term}’. Use only {language} to re-

0.162
3.460

0.513
0.527

0.968
0.962

Mistral-7B-Instruct 0.140

0.475

0.625

+ 5 shots

0.488

0.898

3.034

ply.”, where {language} and {term} are variables denoting
Table 3 summarizes the results of our experiments. We

the target language and the term to deﬁne.
Regarding evaluation, previous work in deﬁnition mod-see that providing the model with examples tends to leadeling has mainly used n-gram overall metrics such as BLEU

to better performance, as expected. However, an important[18] and METEOR [19]. As the latter is language-speciﬁc,issue here is that LLMs are sometimes unable to follow

here we report BLEU, relying on the sacrebleu3） imple-

instructions, which in our case often leads to the model

mentation [20]. As metrics based on n-gram overlap do
not capture nuance in the generations [5, 13], we follow

generating outputs in English. We also observed that models often hallucinate deﬁnitions which may “feel” correct,

works that adopted machine learning-based metrics andbut in fact are not. Better mechanisms to control theseexperiment with COMET [21]4）.

kinds of behaviors are required.

Furthermore, to evaluate the ability of the LLMs toTable 4 shows the human evaluation results of the sam-generate deﬁnitions in the target language, we rely on apled generated deﬁnitions. First, we can observe that the

fasttext-based language classiﬁcation model [22], a linear

Likert score on the Best sets is higher than on the Worst

model based on character n-grams. It can recognize 176

sets, which shows that the COMET metric is well aligned

languages and was trained on 400 million tokens fromwith the human evaluation. The overall positive Pearson

Wikipedia and sentences from the Tatoeba website.

correlation between these values supports this point. We

Finally, we perform human evaluation on a subset of

also observe that the correlation in the Worst sets is overall

the deﬁnitions generated by each model and setting. We

smaller. This result is likely because while COMET varied

recruit a native speaker of Brazilian Portuguese to evaluate

between values of 0.2 to 0.4, these deﬁnitions mostly were
tagged as “Very poor”. Furthermore, we observe that the3） https://github.com/mjpost/sacrebleu
4） We use the “wmt22-comet-da’’ model, which has multilingual
support.

correlation in the Best set in the zero-shot scenario is almost
0 for Mistral and negative for Llama2. In this scenario, we― 208 ―

This work is licensed by the author(s) under CC BY 4.0(https://creativecommons.org/licenses/by/4.0/).

Table 4: Results of the human evaluation of the generated
deﬁnitions. Corr. is short for the Pearson correlation

data, with a total of 156 diﬀerent meanings. We utilize

between Likert score and COMET values.

To assess the viability of this approach, we subject theseModel

the oﬃcial OpenAI API to feed this data into the model.
sentences to a thorough human evaluation. Speciﬁcally, we

Set

Likert COMET Corr.

Best

3.860

0.639

-0.224

generated sentences according to the following simpliﬁed

Worst 1.240

0.385

0.118

Likert scale: 1 - Poor: The sentence uses the term with

All

2.550

0.512

0.752

a diﬀerent meaning; 2 - Acceptable: The sentence uses

Best

4.000

0.623

0.623

Worst 1.460

0.339

0.102

All

2.730

0.481

0.721

Table 5: Results of the human evaluation of the generated

Best

3.260

0.577

0.089

example sentences. The numbers indicate the percentage

Mistral-7B-Instruct Worst 1.140

0.320

0.181

of sentences evaluated in each category

Llama-2-13b-chat+ 5 shots+ 5 shots

the term with the speciﬁed meaning, but in the wrong
context/sounds unnatural; 3 - Good: The sentence uses theterm with the speciﬁed meaning.

All

2.200

0.448

0.633

Poor Acceptable Good

Best

1.780

0.530

0.186

7.69

Worst 1.180

0.306-0.053

All

0.418

0.897

1.480

noticed that models generate very long deﬁnitions. For
Mistral, these generations were rather “Very good”, withrich details, or “Very Poor”, containing wrong information; however, COMET could not detect such diﬀerences.
Llama2 generations were more precise and correct, but
we hypothesize that COMET could not correctly evaluate
them due to the length diﬀerence between reference andgenerated deﬁnitions.

4.2

ask a native Brazilian Portuguese speaker to evaluate the

16.67

75.64

The results of this evaluation are in Table 5. We can seethat most of the generated examples received the “Good”
score, with an average of 2.68, which indicates that using
this speciﬁc LLM could be an alternative to obtain such
example sentences for Brazilian Portuguese.

5

Conclusions
In this paper, we present a study on deﬁnition mod-eling for the Portuguese language, which we propose as aproxy to assess the multilingual abilities of Large Language
Models. Our encouraging results suggest that LLMs canTackling Polysemy

generate plausible deﬁnitions of words in Portuguese. Fur-Our results are limited to the set of terms for which we
only have one deﬁnition. Though our results suggest that
LLMs can, to some extent, generate plausible deﬁnitions
for words in Portuguese, our empirical study oﬀers no
insight into the more challenging scenario of polysemy.
Therefore, we propose to utilize LLMs to obtain example
sentences for polysemic terms in our dictionary. We run
a pilot study using GPT-3.5 [9] to generate such example
sentences. Concretely, we prompt the model as follows:

thermore, our human evaluation showed that the COMET
metric could be used as a guide for generation quality andthat prompting LLMs could be an alternative to obtain
context information on polysemic terms in Portuguese.
For future work, we would like to incorporate more languages into our study and broaden our human evaluation
scope to understand better how these models behave. We
also plan to run LLM ﬁnetuning experiments on our data
to test if this leads to better performance and behavior.

In the Portuguese Language, the word ’{term}’ can mean

References“{deﬁnition}”. Please give me a sentence in that language[1] Thanapon Noraset, Chen Liang, Larry Birnbaum, andDoug Downey. Deﬁnition modeling: Learning to deﬁne
word embeddings in natural language. In Proceedings ofwhere this word is used., where {term} and {deﬁnition} areplaceholders for variables denoting a given pair of a term
and its corresponding deﬁnition.
For our study, we sample a subset of 59 terms from our― 209 ―

the Thirty-First AAAI Conference on Artiﬁcial Intelligence, AAAI’17, pp. 3259–3266, San Francisco, Califor-

nia, USA, February 2017. AAAI Press.

This work is licensed by the author(s) under CC BY 4.0(https://creativecommons.org/licenses/by/4.0/).

[2] Artyom Gadetsky, Ilya Yakubovskiy, and Dmitry Vetrov.
Conditional generators of words deﬁnitions. In Proceedings of the 56th Annual Meeting of the Association
for Computational Linguistics (Volume 2: Short Papers), pp. 266–271, Melbourne, Australia, July 2018. As-sociation for Computational Linguistics.
[3] Ke Ni and William Yang Wang. Learning to Explain NonStandard English Words and Phrases. In Proceedings ofthe Eighth International Joint Conference on Natural
Language Processing (Volume 2: Short Papers), pp.

413–417, Taipei, Taiwan, November 2017. Asian Federation of Natural Language Processing.
[4] Shonosuke Ishiwatari, Hiroaki Hayashi, Naoki Yoshinaga, Graham Neubig, Shoetsu Sato, Masashi Toyoda,
and Masaru Kitsuregawa. Learning to Describe Unknown
Phrases with Local and Global Contexts. In Proceedings of the 2019 Conference of the North American
Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1(Long and Short Papers), pp. 3467–3476, Minneapo-lis, Minnesota, June 2019. Association for Computational
Linguistics.
[5] Machel Reid, Edison Marrese-Taylor, and Yutaka Matsuo. VCDM: Leveraging Variational Bi-encoding andDeep Contextualized Word Representations for Improved
Deﬁnition Modeling. In Proceedings of the 2020 Con-[11] Albert Q. Jiang et al. Mistral 7b, 2023.
[12] Felix Hill, Kyunghyun Cho, Anna Korhonen, and Yoshua
Bengio. Learning to understand phrases by embedding thedictionary. Transactions of the Association for Computational Linguistics, Vol. 4, pp. 17–30, 2016.
[13] Han Huang, Tomoyuki Kajiwara, and Yuki Arase. Deﬁnition Modelling for Appropriate Speciﬁcity. In Proceedings of the 2021 Conference on Empirical Methods in
Natural Language Processing, pp. 2499–2509, Online

and Punta Cana, Dominican Republic, November 2021.
Association for Computational Linguistics.
[14] Pinzhen Chen and Zheng Zhao. A Uniﬁed Model for Reverse Dictionary and Deﬁnition Modelling. In Proceedings of the 2nd Conference of the Asia-Paciﬁc Chapter of the Association for Computational Linguistics
and the 12th International Joint Conference on Natural Language Processing (Volume 2: Short Papers),

[15]

[16]

ference on Empirical Methods in Natural Language
Processing (EMNLP), pp. 6331–6344, Online, Novem-[17]

ber 2020. Association for Computational Linguistics.
[6] Cunliang Kong, Liner Yang, Tianzuo Zhang, Qinan Fan,
Zhenghao Liu, Yun Chen, and Erhong Yang. Toward
Cross-Lingual Deﬁnition Generation for Language Learners, October 2020.
[7] Nabil Hathout and Franck Sajous. Wiktionnaire’s Wikicode GLAWIﬁed: A Workable French Machine-Readable
Dictionary. In Proceedings of the Tenth International[18]

Conference on Language Resources and Evaluation(LREC’16), pp. 1369–1376, Portorož, Slovenia, May

2016. European Language Resources Association (ELRA).
[8] Basilio Calderone, Matteo Pascoli, Franck Sajous, andNabil Hathout. Hybrid Method for Stress Prediction Applied to GLAFF-IT, a Large-Scale Italian Lexicon. In Language, Data, and Knowledge, Lecture Notes in Computer Science, pp. 26–41, Cham, 2017. Springer International Publishing.
[9] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell,
Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger,
Tom Henighan, Rewon Child, Aditya Ramesh, Daniel
Ziegler, Jeﬀrey Wu, Clemens Winter, Chris Hesse, MarkChen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin
Chess, Jack Clark, Christopher Berner, Sam McCandlish,
Alec Radford, Ilya Sutskever, and Dario Amodei. Language Models are Few-Shot Learners. In Advances in
Neural Information Processing Systems, Vol. 33, pp.
1877–1901. Curran Associates, Inc., 2020.
[10] Hugo Touvron et al. Llama 2: Open foundation and ﬁnetuned chat models, 2023.

pp. 8–13, Online only, November 2022. Association for
Computational Linguistics.
Franck Sajous, Basilio Calderone, and Nabil Hathout.
ENGLAWI: From Human- to Machine-Readable Wiktionary. In 12th International Conference on Language Resources and Evaluation (LREC 2020), p.
3016, May 2020.
George Weber. The world’s 10 most inﬂuential languages,
May 2011.
Tim Dettmers et al. Qlora: Eﬃcient ﬁnetuning of quantized llms, 2023.
Kishore Papineni, Salim Roukos, Todd Ward, and WeiJing Zhu. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th Annual
Meeting of the Association for Computational Linguistics, pp. 311–318, Philadelphia, Pennsylvania, USA,

July 2002. Association for Computational Linguistics.
[19] Satanjeev Banerjee and Alon Lavie. METEOR: An automatic metric for MT evaluation with improved correlation with human judgments. In Proceedings of the
ACL Workshop on Intrinsic and Extrinsic Evaluation
Measures for Machine Translation and/or Summarization, pp. 65–72, Ann Arbor, Michigan, June 2005.

Association for Computational Linguistics.
[20] Matt Post. A call for clarity in reporting BLEU scores.
In Proceedings of the Third Conference on MachineTranslation: Research Papers, pp. 186–191, Brussels,
Belgium, October 2018. Association for Computational
Linguistics.
[21] Ricardo Rei, Craig Stewart, Ana C Farinha, and Alon
Lavie. COMET: A Neural Framework for MT Evaluation.
In Proceedings of the 2020 Conference on Empirical
Methods in Natural Language Processing (EMNLP),
pp. 2685–2702, Online, November 2020. Association for
Computational Linguistics.
[22] Edouard Grave, Piotr Bojanowski, Prakhar Gupta, Armand Joulin, and Tomas Mikolov. Learning word vectors
for 157 languages. In Proceedings of the Eleventh In-― 210 ―

ternational Conference on Language Resources and
Evaluation (LREC 2018), Miyazaki, Japan, May 2018.

European Language Resources Association (ELRA).

This work is licensed by the author(s) under CC BY 4.0(https://creativecommons.org/licenses/by/4.0/).