PRCs with kinship terms as the possessum were
too challenging for him to process, so he just rated
them as ‘unacceptable’ and admitted that if he had
spent more time reading such sentences, he would
have understood them. We did not ask our partic-
ipants to answer comprehension questions in this
AJT paradigm, but it is possible that their process-
ing was shallow (Sanford and Graesser, 2006). It
would be intriguing to examine if, when a deep
comprehension processing is forced, participants’
ratings for the PRCs with kinship terms as the pos-
sessums would be higher. Further studies could ex-
plore this possible phenomenon by modifying our
tasks, e.g., by including comprehension questions
or asking the participants to ‘think aloud’ while
making the judgements.

Notably, our findings that the body-parts condi-
tion was rated the highest, then the alienable pos-
sessions, and then the kinships, are consistent with
the frequency results of a corpus study on the Man-
darin passive construction with retained objects
(PCRO) (Yue and Wu, 2019). PCROs with body
parts, alienable possessions, and kinship terms as
the retained objects are illustrated in (13-15):

(13) Ta bei daduanle biliang.
he PASS hit-broken-ASP nose
‘His nose was broken (by someone).’

(14) Ta bei ren touzoule
He PASS someone steel-away-ASP
qianbao.
wallet
‘His wallet was stolen by someone.’

(15) Ta bei ren qiangle laogong.
she PASS people rob-ASP husband
‘Her husband was taken away from her by
someone else.’

In a PCRO, the subject and the retained object
formed a possessive relationship, and like the PRC
structure in our study, the possessor and posses-
sum are not adjacent to each other. The results
of corpus analyses (Yue and Wu, 2019) show that
body parts are the most common retained objects,
accounting for 41.7% of all 422 instances, whereas
typical nouns as the alienable possessums made up
29.8% of the data. Although kinship terms are al-
lowed in possessive phrases, there were only nine
instances found in the corpus, making up 1.9%
of the PCROs found in the corpus. Yue and Wu

(2019) argued from a cognitive perspective that
the possessive relationship needs to have inferen-
tial accessibility, and that retained objects should
be included in the semantic framework of the sub-
ject. The animacy and the cognitive prominence of
the kinship terms both make it difficult for them to
be the objects of PCROs. This idea echoes our sug-
gestions that the [+human] feature and the clause-
initial position of kinship terms seem to increase
PRCs' processing loads.

This leaves one to wonder if the (in)alienability
effect would come into force for kinship terms if
the [+human] feature were controlled in the alien-
able and inalienable conditions. Lin (2007) de-
veloped a self-paced reading experiment using hu-
man nouns as the possessums for both the inalien-
able condition (16a) and the alienable condition
(16b), and showed that the reading time for the
head nouns of the inalienable condition (16a) was
significantly faster than that for the alienable con-
dition (16b). This finding supports the facilitation
by inherent inalienable semantics.

(16) a. Fuqin bei jingcha zhuazou
father PASS police take
de zongcai xiande shifen
DE boss appear very
huangzhang.
nervous
‘The boss whose father was taken by
the police appeared very nervous.’

b. Yuangong bei jingcha zhuazou
employee PASS police take
de zongcai xiande shifen
DE boss appear very
huangzhang.
nervous
‘The boss whose father was taken by
the police appeared very nervous.’

It is important to bear in mind that our findings
were based on two AJT experiments. Although
people's explicit judgments can reflect the process-
ing difficulties of sentences to some extent (Chom-
sky and Miller, 1968; Fanselow and Frisch, 2006),
it would be worthwhile in the future to use online
methods, such as the self-paced reading paradigm
and eye-tracking technology, to measure readers'
reaction times and eye-gaze patterns when process-
ing PRCs. We also assumed that readers would
search for a possessor when they found the inalien-
able possessum at the beginning of the sentence.
PRCs with kinship terms as the possessum were
too challenging for him to process, so he just rated
them as ‘unacceptable’ and admitted that if he had
spent more time reading such sentences, he would
have understood them. We did not ask our partic-
ipants to answer comprehension questions in this
AJT paradigm, but it is possible that their process-
ing was shallow (Sanford and Graesser, 2006). It
would be intriguing to examine if, when a deep
comprehension processing is forced, participants’
ratings for the PRCs with kinship terms as the pos-
sessums would be higher. Further studies could ex-
plore this possible phenomenon by modifying our
tasks, e.g., by including comprehension questions
or asking the participants to ‘think aloud’ while
making the judgements.

Notably, our findings that the body-parts condi-
tion was rated the highest, then the alienable pos-
sessions, and then the kinships, are consistent with
the frequency results of a corpus study on the Man-
darin passive construction with retained objects
(PCRO) (Yue and Wu, 2019). PCROs with body
parts, alienable possessions, and kinship terms as
the retained objects are illustrated in (13-15):

(13) Ta bei daduanle biliang.
he PASS hit-broken-ASP nose
‘His nose was broken (by someone).’

(14) Ta bei ren touzoule
He PASS someone steel-away-ASP
qianbao.
wallet
‘His wallet was stolen by someone.’

(15) Ta bei ren qiangle laogong.
she PASS people rob-ASP husband
‘Her husband was taken away from her by
someone else.’

In a PCRO, the subject and the retained object
formed a possessive relationship, and like the PRC
structure in our study, the possessor and posses-
sum are not adjacent to each other. The results
of corpus analyses (Yue and Wu, 2019) show that
body parts are the most common retained objects,
accounting for 41.7% of all 422 instances, whereas
typical nouns as the alienable possessums made up
29.8% of the data. Although kinship terms are al-
lowed in possessive phrases, there were only nine
instances found in the corpus, making up 1.9%
of the PCROs found in the corpus. Yue and Wu

(2019) argued from a cognitive perspective that
the possessive relationship needs to have inferen-
tial accessibility, and that retained objects should
be included in the semantic framework of the sub-
ject. The animacy and the cognitive prominence of
the kinship terms both make it difficult for them to
be the objects of PCROs. This idea echoes our sug-
gestions that the [+human] feature and the clause-
initial position of kinship terms seem to increase
PRCs' processing loads.

This leaves one to wonder if the (in)alienability
effect would come into force for kinship terms if
the [+human] feature were controlled in the alien-
able and inalienable conditions. Lin (2007) de-
veloped a self-paced reading experiment using hu-
man nouns as the possessums for both the inalien-
able condition (16a) and the alienable condition
(16b), and showed that the reading time for the
head nouns of the inalienable condition (16a) was
significantly faster than that for the alienable con-
dition (16b). This finding supports the facilitation
by inherent inalienable semantics.

(16) a. Fuqin bei jingcha zhuazou
father PASS police take
de zongcai xiande shifen
DE boss appear very
huangzhang.
nervous
‘The boss whose father was taken by
the police appeared very nervous.’

b. Yuangong bei jingcha zhuazou
employee PASS police take
de zongcai xiande shifen
DE boss appear very
huangzhang.
nervous
‘The boss whose father was taken by
the police appeared very nervous.’

It is important to bear in mind that our findings
were based on two AJT experiments. Although
people's explicit judgments can reflect the process-
ing difficulties of sentences to some extent (Chom-
sky and Miller, 1968; Fanselow and Frisch, 2006),
it would be worthwhile in the future to use online
methods, such as the self-paced reading paradigm
and eye-tracking technology, to measure readers'
reaction times and eye-gaze patterns when process-
ing PRCs. We also assumed that readers would
search for a possessor when they found the inalien-
able possessum at the beginning of the sentence.
That means a gap would be detected as long as readers encountered the first noun in the two inalienable conditions. Several following-up questions could usefully be asked about this presumably detected gap. For instance, do readers start searching for potential fillers for this gap as soon as they detected it? Would this gap results in a temporary slowdown in reading times, due to the unit being unresolved and needing to be held in mind? Does early preparation for a filler-gap dependency facilitate the processing of the latter part of the sentence? Because answering them will require fine-grained and region-specific data, we leave these questions to our future research.

To conclude, our results demonstrate the following effects of (in)alienability on sentence acceptability. First, semantically inherent whole-part relationships facilitate the resolution of the long-distance dependency between the possessor and possessum in PRCs. Second, the fact that the kinship terms, despite also forming alienable possessions, received the lowest acceptability ratings in both our AJT experiments suggests that the [+human] feature and/or the salient syntactic position led to considerable confusions when the readers only adopted shallow processing. Previous research has found that when the possessed nouns are human nouns for both the inalienable and alienable conditions, the former has a processing advantage (Lin, 2007). Thus, future studies could explore more types of possessive relationships in different syntactic structures, as well as using different experimental paradigms to test the (in)alienability effects on sentence comprehension and processing.

## Acknowledgements

We would like to thank the three anonymous reviewers for their insightful comments. We would also like to thank Bingxian Chen and Chenxin Liu for their technical support. This research was made possible by the department general research fund (G-UALS) from the Department of Chinese and Bilingual Studies at the Hong Kong Polytechnic University.

## References

Artemis Alexiadou. 2003. Some notes on the structure of alienable and inalienable possessors. In Martine Coene and Yves D'Hulst, editors, *From NP to DP: Volume 2: The Expression of Possession in Noun

Phrases, chapter 6, pages 167–188. John Benjamins, Amsterdam.

Noam Chomsky and George A. Miller. 1968. Introduction to the formal analysis of natural languages. *Journal of Symbolic Logic*, 33(2):299–300.

Kiel Christianson, Andrew Hollingworth, John F Halilwell, and Fernanda Ferreira. 2001. Thematic roles assigned along the garden path linger. *Cognitive psychology*, 42(4):368–407.

Robert MW Dixon. 2000. Categories of the noun phrase in jarawara. *Journal of Linguistics*, 36(3):487–510.

Gisbert Fanselow and Stefan Frisch. 2006. Effects of processing difficulty on judgments of acceptability. *Gradience in grammar: Generative perspectives*, pages 291–316.

Fernanda Ferreira, Karl GD Bailey, and Vittoria Ferarraro. 2002. Good-enough representations in language comprehension. *Current directions in psychological science*, 11(1):11–15.

Keffyalew Gebregziabher. 2012. The alienable-inalienable asymmetry: Evidence from tigrinya. In *Selected Proceedings of the 42nd Annual Conference on African Linguistics*, pages 161–182.

Peter C Gordon, Randall Hendrick, and Marcus Johnson. 2001. Memory interference during language processing. *Journal of experimental psychology: learning, memory, and cognition*, 27(6):1411.

Peter C Gordon, Randall Hendrick, and William H Levine. 2002. Memory-load interference in syntactic processing. *Psychological science*, 13(5):425–430.

Yu-Yin Hsu. 2009. Possessor extraction in mandarin chinese. In *University of Pennsylvania Working Papers in Linguistics*, volume 15.

Yu-Yin Hsu and Jen Ting. 2006. On the possessive multiple nominative construction in mandarin chinese. In *14th Annual Conference of the International Association of Chinese Linguistics*.

Idan Landau. 1999. Possessor raising and the structure of vp. *Lingua*, 107(1-2):1–37.

Charles N Li and Sandra A Thompson. 1989. *Mandarin Chinese: A functional reference grammar*, volume 3. Univ of California Press.

Y-H Audrey Li. 2012. de in mandarin e in taiwanese. *Studies in Chinese Linguistics*, 33(1):17–40.

Chien-Jer Charles Lin. 2007. Processing (in)alienable possessions at the syntax-semantics interface. In *Interfaces in linguistics: New research perspectives*. Citeseer, Oxford University Press.
That means a gap would be detected as long as readers encountered the first noun in the two inalienable conditions. Several following-up questions could usefully be asked about this presumably detected gap. For instance, do readers start searching for potential fillers for this gap as soon as they detected it? Would this gap results in a temporary slowdown in reading times, due to the unit being unresolved and needing to be held in mind? Does early preparation for a filler-gap dependency facilitate the processing of the latter part of the sentence? Because answering them will require fine-grained and region-specific data, we leave these questions to our future research.

To conclude, our results demonstrate the following effects of (in)alienability on sentence acceptability. First, semantically inherent whole-part relationships facilitate the resolution of the long-distance dependency between the possessor and possessum in PRCs. Second, the fact that the kinship terms, despite also forming alienable possessions, received the lowest acceptability ratings in both our AJT experiments suggests that the [+human] feature and/or the salient syntactic position led to considerable confusions when the readers only adopted shallow processing. Previous research has found that when the possessed nouns are human nouns for both the inalienable and alienable conditions, the former has a processing advantage (Lin, 2007). Thus, future studies could explore more types of possessive relationships in different syntactic structures, as well as using different experimental paradigms to test the (in)alienability effects on sentence comprehension and processing.

## Acknowledgements

We would like to thank the three anonymous reviewers for their insightful comments. We would also like to thank Bingxian Chen and Chenxin Liu for their technical support. This research was made possible by the department general research fund (G-UALS) from the Department of Chinese and Bilingual Studies at the Hong Kong Polytechnic University.

## References

Artemis Alexiadou. 2003. Some notes on the structure of alienable and inalienable possessors. In Martine Coene and Yves D'Hulst, editors, *From NP to DP: Volume 2: The Expression of Possession in Noun

Phrases, chapter 6, pages 167–188. John Benjamins, Amsterdam.

Noam Chomsky and George A. Miller. 1968. Introduction to the formal analysis of natural languages. *Journal of Symbolic Logic*, 33(2):299–300.

Kiel Christianson, Andrew Hollingworth, John F Halilwell, and Fernanda Ferreira. 2001. Thematic roles assigned along the garden path linger. *Cognitive psychology*, 42(4):368–407.

Robert MW Dixon. 2000. Categories of the noun phrase in jarawara. *Journal of Linguistics*, 36(3):487–510.

Gisbert Fanselow and Stefan Frisch. 2006. Effects of processing difficulty on judgments of acceptability. *Gradience in grammar: Generative perspectives*, pages 291–316.

Fernanda Ferreira, Karl GD Bailey, and Vittoria Ferarraro. 2002. Good-enough representations in language comprehension. *Current directions in psychological science*, 11(1):11–15.

Keffyalew Gebregziabher. 2012. The alienable-inalienable asymmetry: Evidence from tigrinya. In *Selected Proceedings of the 42nd Annual Conference on African Linguistics*, pages 161–182.

Peter C Gordon, Randall Hendrick, and Marcus Johnson. 2001. Memory interference during language processing. *Journal of experimental psychology: learning, memory, and cognition*, 27(6):1411.

Peter C Gordon, Randall Hendrick, and William H Levine. 2002. Memory-load interference in syntactic processing. *Psychological science*, 13(5):425–430.

Yu-Yin Hsu. 2009. Possessor extraction in mandarin chinese. In *University of Pennsylvania Working Papers in Linguistics*, volume 15.

Yu-Yin Hsu and Jen Ting. 2006. On the possessive multiple nominative construction in mandarin chinese. In *14th Annual Conference of the International Association of Chinese Linguistics*.

Idan Landau. 1999. Possessor raising and the structure of vp. *Lingua*, 107(1-2):1–37.

Charles N Li and Sandra A Thompson. 1989. *Mandarin Chinese: A functional reference grammar*, volume 3. Univ of California Press.

Y-H Audrey Li. 2012. de in mandarin e in taiwanese. *Studies in Chinese Linguistics*, 33(1):17–40.

Chien-Jer Charles Lin. 2007. Processing (in)alienable possessions at the syntax-semantics interface. In *Interfaces in linguistics: New research perspectives*. Citeseer, Oxford University Press.
Willem M Mak, Wietske Vonk, and Herbert Schriefers. 2002. The influence of animacy on relative clause processing. *Journal of memory and language*, 47(1):50–68.

Miriam Meyerhoff. 2002. Formal and cultural constraints on optional objects in bislama. *Language Variation and Change*, 14(3):323–346.

Zhiying Qian, Susan Garnsey, and Kiel Christianson. 2018. A comparison of online and offline measures of good-enough processing in garden-path sentences. *Language, Cognition and Neuroscience*, 33(2):227–254.

Anthony J Sanford and Arthur C Graesser. 2006. Shallow processing and underspecification. *Discourse Processes*, 42(2):99–108.

Hansjakob Seiler. 1983. Possessivity, subject and object. *Studies in Language. International Journal sponsored by the Foundation "Foundations of Language"*, 7(1):89–117.

Jean-Roger Vergnaud and Maria Luisa Zubizarreta. 1992. The definite determiner and the inalienable constructions in french and in english. *Linguistic inquiry*, 23(4):595–652.

Hui Yue and Heng Wu. 2019. Types of possessive relationship and cognitive analysis on the licensing of prco. *Chinese Teaching in the World*, 33:58–69.
Willem M Mak, Wietske Vonk, and Herbert Schriefers. 2002. The influence of animacy on relative clause processing. *Journal of memory and language*, 47(1):50–68.

Miriam Meyerhoff. 2002. Formal and cultural constraints on optional objects in bislama. *Language Variation and Change*, 14(3):323–346.

Zhiying Qian, Susan Garnsey, and Kiel Christianson. 2018. A comparison of online and offline measures of good-enough processing in garden-path sentences. *Language, Cognition and Neuroscience*, 33(2):227–254.

Anthony J Sanford and Arthur C Graesser. 2006. Shallow processing and underspecification. *Discourse Processes*, 42(2):99–108.

Hansjakob Seiler. 1983. Possessivity, subject and object. *Studies in Language. International Journal sponsored by the Foundation "Foundations of Language"*, 7(1):89–117.

Jean-Roger Vergnaud and Maria Luisa Zubizarreta. 1992. The definite determiner and the inalienable constructions in french and in english. *Linguistic inquiry*, 23(4):595–652.

Hui Yue and Heng Wu. 2019. Types of possessive relationship and cognitive analysis on the licensing of prco. *Chinese Teaching in the World*, 33:58–69.
# Do age of acquisition and orthographic transparency have the same effects in different modalities?

Mohammad Momenian

Department of Chinese and Bilingual Studies, Faculty of Humanities
The Hong Kong Polytechnic University
Yuk Choi Road 11, Hung Hom, Kowloon, Hong Kong
mohammad.momenian@polyu.edu.hk

## Abstract

This paper is intended to study the effects of age of acquisition (AoA) and orthographic transparency on word retrieval in Persian, which is an understudied language. A naming task (both pictures and words) and a recall task (both pictures and words) were used to explore how lexical retrieval and verbal memory are affected by AoA and transparency. Seventy two native speakers of Persian were recruited to participate in two experiments. The results showed that early acquired words are processed faster than late acquired words only when pictures were used as stimuli. Transparency of the words was not an influential factor. However, in the recall experiment a three-way interaction was observed: early acquired pictures and words were processed faster than late acquired stimuli except the words in the transparent condition. The findings speak to the fact that language-specific properties of languages are very important.

<table><caption>Table 1: Summary of significant effects in the reaction time experiment</caption><thead><tr><th>Fixed effects</th><th>Estimate</th><th>Std. Error</th><th>t value</th></tr></thead><tbody><tr><td>(Intercept)</td><td>915.32</td><td>31.44</td><td>29.11</td></tr><tr><td>Condition</td><td>265.35</td><td>-37.34</td><td>-7.10</td></tr><tr><td>AoA</td><td>169.14</td><td>17.16</td><td>9.86</td></tr><tr><td>Condition*AoA</td><td>-147.16</td><td>15.59</td><td>-9.44</td></tr></tbody></table><table><thead><tr><th>Random effects</th><th>Variance</th><th>Standard Deviation</th></tr></thead><tbody><tr><td>Items</td><td>2833</td><td>53.22</td></tr><tr><td>Subjects</td><td>10540</td><td>102.66</td></tr><tr><td>Residual</td><td>31399</td><td>177.20</td></tr></tbody></table>

<figure><img src="image_11.png" /><figcaption>Figure 1: Interaction between Condition and AoA</figcaption></figure>

## 1 Introduction

The majority of research on word retrieval and recall are done only on few languages of the word such as English (Cycowicz et al.,1997; Snodgrass & Vanderwart,1980), Dutch (Shao & Stiegert, 2016), French (Alario & Ferrand,1999; Bonin et al.,2003), Spanish (Cuetos et al.,1999; Manoiloff et al.,2010) and Italian (Dell'Acqua et al., 2000; Navarrete et al., 2019). There are only few reports available in other languages of the world such as Persian.

Age of acquisition (AoA) is known to have a strong effect on word retrieval. Words which are learned earlier are usually processed faster than late acquired words (Alario et al.,2004). This finding has been replicated across several languages in
# Do age of acquisition and orthographic transparency have the same effects in different modalities?

Mohammad Momenian

Department of Chinese and Bilingual Studies, Faculty of Humanities
The Hong Kong Polytechnic University
Yuk Choi Road 11, Hung Hom, Kowloon, Hong Kong
mohammad.momenian@polyu.edu.hk

## Abstract

This paper is intended to study the effects of age of acquisition (AoA) and orthographic transparency on word retrieval in Persian, which is an understudied language. A naming task (both pictures and words) and a recall task (both pictures and words) were used to explore how lexical retrieval and verbal memory are affected by AoA and transparency. Seventy two native speakers of Persian were recruited to participate in two experiments. The results showed that early acquired words are processed faster than late acquired words only when pictures were used as stimuli. Transparency of the words was not an influential factor. However, in the recall experiment a three-way interaction was observed: early acquired pictures and words were processed faster than late acquired stimuli except the words in the transparent condition. The findings speak to the fact that language-specific properties of languages are very important.

## 1 Introduction

The majority of research on word retrieval and recall are done only on few languages of the word such as English (Cycowicz et al.,1997; Snodgrass & Vanderwart,1980), Dutch (Shao & Stiegert, 2016), French (Alario & Ferrand,1999; Bonin et al.,2003), Spanish (Cuetos et al.,1999; Manoiloff et al.,2010) and Italian (Dell'Acqua et al., 2000; Navarrete et al., 2019). There are only few reports available in other languages of the world such as Persian.

Age of acquisition (AoA) is known to have a strong effect on word retrieval. Words which are learned earlier are usually processed faster than late acquired words (Alario et al.,2004). This finding has been replicated across several languages in

<table><caption>Table 1: Summary of significant effects in the reaction time experiment</caption><thead><tr><th>Fixed effects</th><th>Estimate</th><th>Std. Error</th><th>t value</th></tr></thead><tbody><tr><td>(Intercept)</td><td>915.32</td><td>31.44</td><td>29.11</td></tr><tr><td>Condition</td><td>265.35</td><td>-37.34</td><td>-7.10</td></tr><tr><td>AoA</td><td>169.14</td><td>17.16</td><td>9.86</td></tr><tr><td>Condition*AoA</td><td>-147.16</td><td>15.59</td><td>-9.44</td></tr></tbody></table><table><thead><tr><th>Random effects</th><th>Variance</th><th>Standard Deviation</th></tr></thead><tbody><tr><td>Items</td><td>2833</td><td>53.22</td></tr><tr><td>Subjects</td><td>10540</td><td>102.66</td></tr><tr><td>Residual</td><td>31399</td><td>177.20</td></tr></tbody></table>

<figure><img src="image_11.png" /><figcaption>Figure 1: Interaction between Condition and AoA</figcaption></figure>
picture naming. However, when it comes to word recall, the findings are mixed. For example, Cortese et al. (2010, 2015) showed that later acquired stimuli were recalled better than early acquired ones. Raman et al., (2018) found no effects of AoA.

Where AoA effect interacts with orthographic transparency is another question. Orthographic transparency refers to the level of consistency in grapheme to morpheme correspondence. For instance, the word car is orthographically transparent, while the word night is not. Arbitrary Mapping Hypothesis (AMH) (Ellis & Lambon Ralph, 2000) posits that AoA effect is only observable in words where print to sound correspondence is inconsistent (opaque). They believe that AoA effects is diminished in the transparent orthographies. Findings from highly transparent orthographies in other languages such as Turkish failed to replicate the predictions of AMH (Raman, 2006).

The aim of this paper is to see if previous findings could be replicated in Persian which is an understudied language. Persian orthography is both opaque and transparent thus allowing researchers to test AoA effects within one single language. This report is intended to see if AoA interacts with transparency similarly across both naming and recall tasks.

If predictions of ARH are correct, it is hypothesized that in both modalities (words vs. picture) of naming and recall tasks AoA should only show an effect in the opaque condition. In the transparent condition, AoA effect should disappear based on ARH predictions.

# 2 Methods

## 2.1 Participants

Number of participants in this study included 72 native speakers of Farsi. Thirty six people participated in the naming experiment (mean age: 23.27, gender: 22 male) and 36 in the recall experiment (mean age: 21.56, gender: 17 male). They were all undergraduate university students. The participants had normal or corrected to normal visual acuity, and reported no history of neurological or psychiatric disorders. They received course credit and gave informed consent before their participation.

## 2.2 Materials

Sixty words and their pictures were selected from Farsi Snodgrass and Vanderwart naming battery (Bakhtiar, Nilipour, & Weekes, 2013). These words and pictures were divided, on the basis of a 3.8 cut-off point, into two categories: early acquired and late acquired words. Each category of words was then divided into 15 opaque and 15 transparent ones. The stimuli included both tools and animals. Orthographic transparency was defined based on how well the letters in a word were matched with the sounds in the same words. For instance, the word ‘car’ in English is transparent, but ‘psychology’ is not as transparent because not all the letters have a phonological representation in the production stage of the word.

## 2.3 Procedure

Recall experiment: in this experiment the participants were required to recall the words and pictures that were just presented to them. Half of the participants recalled the words and the other half recalled the pictures. It should be noted that a distractor task was performed by the participants for about two minutes after the presentation of the words or pictures was finished in order to create a delay in the recall process. After this, the participants were required to write down as many names as possible they could remember. There was no time limit on finishing the task.

Reaction time experiment: This experiment follows the same procedure adopted in the recall one. Using DMDX software, half of the participants named the pictures. In the same session, the other half of the participants read words for the same pictures. Ten words or pictures were used as practice items so that the participants would get familiar with the type of the task. Each word was shown to the participants for 1500ms and each picture was presented for 2000ms. If participants could not answer in the time provided, the software moved on to the next item automatically. Participants were instructed not to cough or make any unnecessary noises during this task particularly at the beginning of each picture. Any response which did not match with the correct most
picture naming. However, when it comes to word recall, the findings are mixed. For example, Cortese et al. (2010, 2015) showed that later acquired stimuli were recalled better than early acquired ones. Raman et al., (2018) found no effects of AoA.

Where AoA effect interacts with orthographic transparency is another question. Orthographic transparency refers to the level of consistency in grapheme to morpheme correspondence. For instance, the word car is orthographically transparent, while the word night is not. Arbitrary Mapping Hypothesis (AMH) (Ellis & Lambon Ralph, 2000) posits that AoA effect is only observable in words where print to sound correspondence is inconsistent (opaque). They believe that AoA effects is diminished in the transparent orthographies. Findings from highly transparent orthographies in other languages such as Turkish failed to replicate the predictions of AMH (Raman, 2006).

The aim of this paper is to see if previous findings could be replicated in Persian which is an understudied language. Persian orthography is both opaque and transparent thus allowing researchers to test AoA effects within one single language. This report is intended to see if AoA interacts with transparency similarly across both naming and recall tasks.

If predictions of ARH are correct, it is hypothesized that in both modalities (words vs. picture) of naming and recall tasks AoA should only show an effect in the opaque condition. In the transparent condition, AoA effect should disappear based on ARH predictions.

# 2 Methods

## 2.1 Participants

Number of participants in this study included 72 native speakers of Farsi. Thirty six people participated in the naming experiment (mean age: 23.27, gender: 22 male) and 36 in the recall experiment (mean age: 21.56, gender: 17 male). They were all undergraduate university students. The participants had normal or corrected to normal visual acuity, and reported no history of neurological or psychiatric disorders. They received course credit and gave informed consent before their participation.

## 2.2 Materials

Sixty words and their pictures were selected from Farsi Snodgrass and Vanderwart naming battery (Bakhtiar, Nilipour, & Weekes, 2013). These words and pictures were divided, on the basis of a 3.8 cut-off point, into two categories: early acquired and late acquired words. Each category of words was then divided into 15 opaque and 15 transparent ones. The stimuli included both tools and animals. Orthographic transparency was defined based on how well the letters in a word were matched with the sounds in the same words. For instance, the word ‘car’ in English is transparent, but ‘psychology’ is not as transparent because not all the letters have a phonological representation in the production stage of the word.

## 2.3 Procedure

Recall experiment: in this experiment the participants were required to recall the words and pictures that were just presented to them. Half of the participants recalled the words and the other half recalled the pictures. It should be noted that a distractor task was performed by the participants for about two minutes after the presentation of the words or pictures was finished in order to create a delay in the recall process. After this, the participants were required to write down as many names as possible they could remember. There was no time limit on finishing the task.

Reaction time experiment: This experiment follows the same procedure adopted in the recall one. Using DMDX software, half of the participants named the pictures. In the same session, the other half of the participants read words for the same pictures. Ten words or pictures were used as practice items so that the participants would get familiar with the type of the task. Each word was shown to the participants for 1500ms and each picture was presented for 2000ms. If participants could not answer in the time provided, the software moved on to the next item automatically. Participants were instructed not to cough or make any unnecessary noises during this task particularly at the beginning of each picture. Any response which did not match with the correct most
dominant name of the picture was considered incorrect.

## 3 Results

Linear Mixed Effect (LME) Modeling is gaining popularity in psycholinguistic research. LME modeling offers several advantages over the classic statistical analyses. LME modeling takes into account item and subject random effects which leads to higher generalizability of findings to the larger population and stimuli (Baayen, Davidson, & Bates, 2008). In this study, lme4 package (https://cran.r-project.org/web/packages/lme4/) was used in R software (R Development Core Team, 2012) in order to analyze the data in both experiments.

### 3.1 Reaction time experiment

Our dependent variable was transformed reaction time (RT) using common log trans-formation. The model tested included all fixed variables such as AoA (early vs. late), transparency (transparent vs. opaque), and condition (picture vs. print) and their interactions along with the random effects for subjects and items. Random intercepts were not included because the variables had less than 5 levels which could result in singularity. To test the collinearity among the variables, a variance inflation factor (VIF) was used. Variables with a VIF above 5 should be removed from the analysis based on the recommendation by Craney and Surles (2002). In order to find which variables and interaction had significant effects, conditional F-tests were adopted because doing Lilehood Ratio Tests (LRT) on the fixed effects is anti-conservative and could result in misleading findings (Pinheiro & Bates, 2000). Kenward-Roger approximations were used to calculate denominator degrees of freedom which have shown more acceptable type 1 error rates in comparison with LRT and Wald tests (Kuznetsova, Brockhoff, & Christensen, 2017).

The results of the analysis for this section are presented in Table 1. Since the interaction between Condition and AoA is significant, it doesn't make sense to look into main effects. See Figure 1 for the interaction patterns.

### 3.2 Recall experiment

Since the response variable in this experiment was a binomial variable, a generalized linear mixed effect model (GLMER) was used. First, a full model was created including AoA, transparency, and condition as main effects, AoA*transparency*condition as the interaction effect, and random effects of subjects and items. Conditional F-tests were used to find the significant effects just like the reaction time experiment.

The results of the analysis for the recall experiment are presented in Table 2. For the interactions, see Figure 2.

## 4 Discussion and Conclusion

Regardless of the interaction patterns, the significant effect of AoA in this report is in line with many previous picture naming studies in other languages (Alario et al., 2004) and Persian (Nilipour, Bakhtiar, Momenian, & Weekes, 2017). Words and pictures which were learned earlier were processed faster and recalled more accurately than the late acquired stimuli regardless of the modality. However, the existence of interactions in both the reaction time and recall analyses reveals that AoA effect is more complicated than a simple main effect.

The results of the reaction time experiment are not consistent with previous studies. First, there was no interaction between AoA and transparency predicted by AMH (Morrison & Ellis, 2000). Second, AoA had a significant effect only in the picture naming modality, while the effect disappeared in the word reading modality. It is believed that AoA is a fundamental property of lexical retrieval and is independent of the modality. In other words, no matter whether the stimuli are presented as print or picture, the effect should be there. This is a counterintuitive finding which needs further investigation in the future studies.

The results from the recall experiment are partially consistent with AHM. Although, transparency and AoA did not have any interaction in the picture recall, the interaction observed in the word recall is consistent with AHM predictions. Based on AHM predictions, the AoA effect disappeared in the transparent
dominant name of the picture was considered incorrect.

## 3 Results

Linear Mixed Effect (LME) Modeling is gaining popularity in psycholinguistic research. LME modeling offers several advantages over the classic statistical analyses. LME modeling takes into account item and subject random effects which leads to higher generalizability of findings to the larger population and stimuli (Baayen, Davidson, & Bates, 2008). In this study, lme4 package (https://cran.r-project.org/web/packages/lme4/) was used in R software (R Development Core Team, 2012) in order to analyze the data in both experiments.

### 3.1 Reaction time experiment

Our dependent variable was transformed reaction time (RT) using common log trans-formation. The model tested included all fixed variables such as AoA (early vs. late), transparency (transparent vs. opaque), and condition (picture vs. print) and their interactions along with the random effects for subjects and items. Random intercepts were not included because the variables had less than 5 levels which could result in singularity. To test the collinearity among the variables, a variance inflation factor (VIF) was used. Variables with a VIF above 5 should be removed from the analysis based on the recommendation by Craney and Surles (2002). In order to find which variables and interaction had significant effects, conditional F-tests were adopted because doing Lilehood Ratio Tests (LRT) on the fixed effects is anti-conservative and could result in misleading findings (Pinheiro & Bates, 2000). Kenward-Roger approximations were used to calculate denominator degrees of freedom which have shown more acceptable type 1 error rates in comparison with LRT and Wald tests (Kuznetsova, Brockhoff, & Christensen, 2017).

The results of the analysis for this section are presented in Table 1. Since the interaction between Condition and AoA is significant, it doesn't make sense to look into main effects. See Figure 1 for the interaction patterns.

### 3.2 Recall experiment

Since the response variable in this experiment was a binomial variable, a generalized linear mixed effect model (GLMER) was used. First, a full model was created including AoA, transparency, and condition as main effects, AoA*transparency*condition as the interaction effect, and random effects of subjects and items. Conditional F-tests were used to find the significant effects just like the reaction time experiment.

The results of the analysis for the recall experiment are presented in Table 2. For the interactions, see Figure 2.

## 4 Discussion and Conclusion

Regardless of the interaction patterns, the significant effect of AoA in this report is in line with many previous picture naming studies in other languages (Alario et al., 2004) and Persian (Nilipour, Bakhtiar, Momenian, & Weekes, 2017). Words and pictures which were learned earlier were processed faster and recalled more accurately than the late acquired stimuli regardless of the modality. However, the existence of interactions in both the reaction time and recall analyses reveals that AoA effect is more complicated than a simple main effect.

The results of the reaction time experiment are not consistent with previous studies. First, there was no interaction between AoA and transparency predicted by AMH (Morrison & Ellis, 2000). Second, AoA had a significant effect only in the picture naming modality, while the effect disappeared in the word reading modality. It is believed that AoA is a fundamental property of lexical retrieval and is independent of the modality. In other words, no matter whether the stimuli are presented as print or picture, the effect should be there. This is a counterintuitive finding which needs further investigation in the future studies.

The results from the recall experiment are partially consistent with AHM. Although, transparency and AoA did not have any interaction in the picture recall, the interaction observed in the word recall is consistent with AHM predictions. Based on AHM predictions, the AoA effect disappeared in the transparent
<table><thead><tr><th>Fixed effects</th><th>Estimate</th><th>Std. Error</th><th>z value</th></tr></thead><tbody><tr><td>(Intercept)</td><td>-1.11</td><td>0.10</td><td>-10.2</td></tr><tr><td>AoA</td><td>-0.62</td><td>0.17</td><td>-3.66</td></tr><tr><td>Condition*AoA</td><td>1.02</td><td>0.40</td><td>2.49</td></tr><tr><td>*Transparency</td><td></td><td></td><td></td></tr></tbody></table>

<table><caption>Table 2: Summary of significant effects in the recall experiment</caption><thead><tr><th>Random effects</th><th>Variance</th><th>Standard Deviation</th></tr></thead><tbody><tr><td>Items</td><td>0.27</td><td>0.52</td></tr><tr><td>Subjects</td><td>0.15</td><td>0.39</td></tr></tbody></table>

condition. What is still counterintuitive is that why this effect is only observed in word recall and not in picture recall. If AoA effect is independent of modality, a similar effect should have been witnessed in both modalities.

our findings should be interpreted with caution. However, the counterintuitive findings could pave the way for future studies in other languages. We need more studies with null or counterintuitive effects indeed.

The data and codes for this manuscript are available at the following DOI 10.17605/OSF.IO/RTPH6.

## References

Alario, F. X., & Ferrand, L. 1999. A set of 400 pictures standardized for French: Norms for name agreement, image agreement, familiarity, visual complexity, image variability, and age of acquisition. *Behavior Research Methods, Instruments, & Computers: A Journal of the Psychonomic Society, 31*(3), 531–552.

<figure><img src="image_12.png" /><figcaption>Figure 2: The interactions in the recall experiment</figcaption></figure>

We did not control for other variables such as imageability, visual complexity, familiarity, and frequency. It's possible that the effects observed in this study could be attributed to lack of control over these variables. The number of items was not too many limiting the power of the study and hence generalizability of the findings. Moreover, the participants in the reaction time and recall experiments were different which could be another limitation due to lack of control over inter-individual variability. For these reasons, we believe

Alario, F. X., Ferrand, L., Laganaro, M., New, B., Frauenfelder, U. H., & Segui, J. 2004. Predictors of picture naming speed. *Behavior Research Methods, 36*(1), 140–155.

Bakhtiar, M., Nilipour, R., & Weekes, B. S. 2013. Predictors of timed picture naming in Persian. *Behavior Research Methods, 45*(3), 834–841.

Baayen, R. H., Davidson, D. J., & Bates, D. M. 2008. Mixed-effects modeling with crossed random effects for subjects and items. *Journal of memory and language, 59*(4), 390–412.
<table><thead><tr><th>Fixed effects</th><th>Estimate</th><th>Std. Error</th><th>z value</th></tr></thead><tbody><tr><td>(Intercept)</td><td>-1.11</td><td>0.10</td><td>-10.2</td></tr><tr><td>AoA</td><td>-0.62</td><td>0.17</td><td>-3.66</td></tr><tr><td>Condition*AoA</td><td>1.02</td><td>0.40</td><td>2.49</td></tr><tr><td>*Transparency</td><td></td><td></td><td></td></tr></tbody></table>

<table><caption>Table 2: Summary of significant effects in the recall experiment</caption><thead><tr><th>Random effects</th><th>Variance</th><th>Standard Deviation</th></tr></thead><tbody><tr><td>Items</td><td>0.27</td><td>0.52</td></tr><tr><td>Subjects</td><td>0.15</td><td>0.39</td></tr></tbody></table>

condition. What is still counterintuitive is that why this effect is only observed in word recall and not in picture recall. If AoA effect is independent of modality, a similar effect should have been witnessed in both modalities.

our findings should be interpreted with caution. However, the counterintuitive findings could pave the way for future studies in other languages. We need more studies with null or counterintuitive effects indeed.

The data and codes for this manuscript are available at the following DOI 10.17605/OSF.IO/RTPH6.

## References

Alario, F. X., & Ferrand, L. 1999. A set of 400 pictures standardized for French: Norms for name agreement, image agreement, familiarity, visual complexity, image variability, and age of acquisition. *Behavior Research Methods, Instruments, & Computers: A Journal of the Psychonomic Society, 31*(3), 531-552.

<figure><img src="image_12.png" /><figcaption>Figure 2: The interactions in the recall experiment</figcaption></figure>

We did not control for other variables such as imageability, visual complexity, familiarity, and frequency. It's possible that the effects observed in this study could be attributed to lack of control over these variables. The number of items was not too many limiting the power of the study and hence generalizability of the findings. Moreover, the participants in the reaction time and recall experiments were different which could be another limitation due to lack of control over inter-individual variability. For these reasons, we believe

Alario, F. X., Ferrand, L., Laganaro, M., New, B., Frauenfelder, U. H., & Segui, J. 2004. Predictors of picture naming speed. *Behavior Research Methods, 36*(1), 140-155.

Bakhtiar, M., Nilipour, R., & Weekes, B. S. 2013. Predictors of timed picture naming in Persian. *Behavior Research Methods, 45*(3), 834-841.

Baayen, R. H., Davidson, D. J., & Bates, D. M. 2008. Mixed-effects modeling with crossed random effects for subjects and items. *Journal of memory and language, 59*(4), 390-412.
Bonin, P., Meot, A., Lagarrigue, A., & Roux, S. 2015. Written object naming, spelling to dictation, and immediate copying: Different tasks, different pathways? *Quarterly Journal of Experimental Psychology, 68*(7), 1268–1294.

Cortese, M. J., Khanna, M. M., & Hacker, S. 2010. Recognition memory for 2,578 monosyllabic words. *Memory, 18*(6), 595-609.

Cortese, M. J., McCarty, D. P., & Schock, J. 2015. A mega recognition memory study of 2897 disyllabic words. *The Quarterly Journal of Experimental Psychology, 68*(8), 1489-1501.

Craney, T. A., & Surles, J. G. 2002. Model-Dependent Variance Inflation Factor Cutoff Values. *Quality Engineering, 14*(3), 391-403.

Cuetos, F., Ellis, A. W., & Alvarez, B. 1999. Naming times for the Snodgrass and Vanderwart pictures in Spanish. *Behavior Research Methods, Instruments, & Computers: A Journal of the Psychonomic Society, 31*(4), 650–658.

Cycowicz, Y.M., Friedman, D., Rothstein, M.,& Snodgrass, J.G. 1997. Picture naming by young children: Norms for name agreement, familiarity, and visual complexity. *Journal of Experimental Child Psychology, 65*(2), 171-237.

Dell'Acqua, R., Lotto, L., & Job, R. 2000. Naming times and standardized norms for the Italian PD/DPSS set of 266 pictures: Direct comparisons with American, English, French, and Spanish published databases. *Behavior Research Methods, Instruments, & Computers: A Journal of the Psychonomic Society, 32*(4), 588–615.

Ellis, A. W., & Lambon Ralph, M. A. 2000. Age of acquisition effects in adult lexical processing reflect loss of plasticity in maturing systems: Insights from connectionist networks. *Journal of Experimental Psychology: Learning, Memory, and Cognition, 26*(5), 1103/1123.

Kuznetsova, A., Brockhoff, P. B., & Christensen, R. H. B. 2017. lmerTest Package: Tests in Linear Mixed Effects Models. *Journal of Statistical Software, 82*(13), 26.

Manoiloff, L., Artstein, M., Canavoso, M. B., Fernandez, L., & Segui, J. 2010. Expanded norms for 400 experimental pictures in an Argentinean Spanish-speaking population. *Behavior Research Methods, 42*(2), 452–460.

Morrison, C. M., & Ellis, A. W. 2000. Real age of acquisition effects in word naming and lexical decision. *Br J Psychol, 91* (*Pt 2*), 167-180.

Navarrete, E., Arcara, G., Mondini, S., & Penolazzi, B. (2019). Italian norms and naming latencies for 357 high quality color images. *PLoSOne, 14*(2), e0209524.

Nilipour, R., Bakhtiar, M., Momenian, M., & Weekes, B. S. 2017. Object and action picture naming in brain-damaged Persian speakers with aphasia. *Aphasiology, 31*(4), 388-405.

Pinheiro, J., & Bates, D. 2000. *Mixed-Effects Models in S and S-PLUS*.USA: Springer.

Raman, I. (2006). On the age-of-acquisition effects in word naming and orthographic transparency: Mapping specific or universal? *Visual cognition, 13*(7-8), 1044-1053.

Raman, I., Raman, E., İkier, S., Kilecioğlu, E., Uzun Eroğlu, D., & Zeyveli, Ş. 2018. Differential effects of age of acquisition and frequency on memory: evidence from free recall of pictures and words in Turkish. *Writing Systems Research, 10*(1), 1-14.

Snodgrass, J. G., & Vanderwart, M. 1980. A standardized set of 260 pictures: Norms for name agreement, image agreement, familiarity, and visual complexity. *Journal of Experimental Psychology: Human Learning and Memory, 6*(2), 174.
Bonin, P., Meot, A., Lagarrigue, A., & Roux, S. 2015. Written object naming, spelling to dictation, and immediate copying: Different tasks, different pathways? *Quarterly Journal of Experimental Psychology, 68*(7), 1268–1294.

Cortese, M. J., Khanna, M. M., & Hacker, S. 2010. Recognition memory for 2,578 monosyllabic words. *Memory, 18*(6), 595-609.

Cortese, M. J., McCarty, D. P., & Schock, J. 2015. A mega recognition memory study of 2897 disyllabic words. *The Quarterly Journal of Experimental Psychology, 68*(8), 1489-1501.

Craney, T. A., & Surles, J. G. 2002. Model-Dependent Variance Inflation Factor Cutoff Values. *Quality Engineering, 14*(3), 391-403.

Cuetos, F., Ellis, A. W., & Alvarez, B. 1999. Naming times for the Snodgrass and Vanderwart pictures in Spanish. *Behavior Research Methods, Instruments, & Computers: A Journal of the Psychonomic Society, 31*(4), 650–658.

Cycowicz, Y.M., Friedman, D., Rothstein, M.,& Snodgrass, J.G. 1997. Picture naming by young children: Norms for name agreement, familiarity, and visual complexity. *Journal of Experimental Child Psychology, 65*(2), 171-237.

Dell'Acqua, R., Lotto, L., & Job, R. 2000. Naming times and standardized norms for the Italian PD/DPSS set of 266 pictures: Direct comparisons with American, English, French, and Spanish published databases. *Behavior Research Methods, Instruments, & Computers: A Journal of the Psychonomic Society, 32*(4), 588–615.

Ellis, A. W., & Lambon Ralph, M. A. 2000. Age of acquisition effects in adult lexical processing reflect loss of plasticity in maturing systems: Insights from connectionist networks. *Journal of Experimental Psychology: Learning, Memory, and Cognition, 26*(5), 1103/1123.

Kuznetsova, A., Brockhoff, P. B., & Christensen, R. H. B. 2017. lmerTest Package: Tests in Linear Mixed Effects Models. *Journal of Statistical Software, 82*(13), 26.

Manoiloff, L., Artstein, M., Canavoso, M. B., Fernandez, L., & Segui, J. 2010. Expanded norms for 400 experimental pictures in an Argentinean Spanish-speaking population. *Behavior Research Methods, 42*(2), 452–460.

Morrison, C. M., & Ellis, A. W. 2000. Real age of acquisition effects in word naming and lexical decision. *Br J Psychol, 91* (*Pt 2*), 167-180.

Navarrete, E., Arcara, G., Mondini, S., & Penolazzi, B. (2019). Italian norms and naming latencies for 357 high quality color images. *PLoSOne, 14*(2), e0209524.

Nilipour, R., Bakhtiar, M., Momenian, M., & Weekes, B. S. 2017. Object and action picture naming in brain-damaged Persian speakers with aphasia. *Aphasiology, 31*(4), 388-405.

Pinheiro, J., & Bates, D. 2000. *Mixed-Effects Models in S and S-PLUS*.USA: Springer.

Raman, I. (2006). On the age-of-acquisition effects in word naming and orthographic transparency: Mapping specific or universal? *Visual cognition, 13*(7-8), 1044-1053.

Raman, I., Raman, E., İkier, S., Kilecioğlu, E., Uzun Eroğlu, D., & Zeyveli, Ş. 2018. Differential effects of age of acquisition and frequency on memory: evidence from free recall of pictures and words in Turkish. *Writing Systems Research, 10*(1), 1-14.

Snodgrass, J. G., & Vanderwart, M. 1980. A standardized set of 260 pictures: Norms for name agreement, image agreement, familiarity, and visual complexity. *Journal of Experimental Psychology: Human Learning and Memory, 6*(2), 174.
Response to reviewers’ comments:

I included figures in the manuscript which clearly show
the RTs and accuracy for each condition.

I updated the analysis pipeline adding a few points
which were missing in the previous manuscript.

The codes and data have been made available online at
the following DOI 10.17605/OSF.IO/RTPH6.

I added extensive information to the introduction and
discussion making the paper fitter for CogAlex
objectives and audience.
Response to reviewers’ comments:

I included figures in the manuscript which clearly show
the RTs and accuracy for each condition.

I updated the analysis pipeline adding a few points
which were missing in the previous manuscript.

The codes and data have been made available online at
the following DOI 10.17605/OSF.IO/RTPH6.

I added extensive information to the introduction and
discussion making the paper fitter for CogAlex
objectives and audience.
# CAT ManyNames: a New Dataset for Object Naming in Catalan

Mar Domínguez Orfila

Maite Melero Nogués

Universitat Pompeu Fabra

Barcelona Supercomputing Center

Barcelona Supercomputing Center

maite.melero@bsc.es

mar.dominguez01@estudiant.upf.edu

Gemma Boleda Torrent

Universitat Pompeu Fabra

gemma.boleda@upf.edu

## Abstract

Object Naming is an important task within the field of Language and Vision that consists of generating a correct and appropriate name for an object given an image. The ManyNames dataset uses real-world human annotated images with multiple labels, instead of just one. In this work, we describe the adaptation of this dataset (originally in English) to Catalan, by (i) machine-translating the English labels and (ii) collecting human annotations for a subset of the original corpus and comparing both resources. Analyses reveal divergences in the lexical variation of the two sets showing potential problems of directly translated resources, particularly when there is no resource to a proper context, which in this case is conveyed by the image. The analysis also points to the impact of cultural factors in the naming task, which should be accounted for in future cross-lingual naming tasks.

## 1 Introduction

Most NLP resources are only available for a small percentage of languages (Joshi et al., 2020), being the rest of the languages spoken in the world left behind. This affects also Catalan, which can be considered a moderately under-resourced language (Armengol-Estapé et al., 2021). In the Language and Vision area, although significantly large datasets of annotated images have been created for a variety of tasks for English, to date no resources of this kind exist for Catalan. In this work we present CAT ManyNames<fnref n="1" />, the Catalan version of the ManyNames dataset, which is the first available resource for the task of Object Naming in Catalan. The dataset has been translated from the English version and its test set has been human annotated to assess the quality of the translation. We also provide analyses of the sources of variation between

the human annotated dataset and its translated counterpart.

## 2 Background

### 2.1 Object Naming: an Interdisciplinary Task

Naming an object accounts for picking out a nominal to refer to it (Silberer et al., 2020a), and is a linguistic phenomenon that can show lexical variation. On the one hand, objects can belong to different semantic categories at the same time (i.e., a baby boy belongs to the categories PERSON, CHILD, BOY, HUMAN, etc.), which, according to Brown (Brown, 1958) could all be valid alternatives for naming that object. On the other hand, the three different levels within semantic categorization<fnref n="2" /> identified by Rosch et al. (Rosch et al., 1976) can all be valid alternatives for naming the same object as well. Although the basic-level categories are considered to be the most natural terms for speakers when referring to objects (Hajibayova, 2013; Jolicoeur et al., 1984; Rosch et al., 1976), these are not universal categories since they are restricted by perceptive, cognitive and environmental factors that can result in lexical variation (Berlin, 2014; Graf et al., 2016; Malt, 1995; Wierzbicka, 1996).

While the task of Object Naming has been studied in both Language and Vision and Psycholinguistics, and it is related to Object Recognition tasks in Computer Vision, each field has a different approach to the task:

Within the field of Language and Vision, datasets typically collect free and natural referential utterances<fnref n="3" /> produced by annotators for a given real-world image. Some relevant datasets are RefCOCO

<footnote n="1">¹Available at https://huggingface.co/datasets/projecte-aina/cat manynames</footnote>

<footnote n="2">²The superordinate level (i.e., animal), the basic level (i.e., dog), and the subordinate level (i.e., Chihuahua)</footnote>

<footnote n="3">³In semantics, a referring expression is a piece of language (typically a noun phrase) used with a particular referent in mind that refers to something or someone, or a clearly delimited collection of things or people (Hurford et al., 2007).</footnote>
# CAT ManyNames: a New Dataset for Object Naming in Catalan

Mar Domínguez Orfila

Maite Melero Nogués

Universitat Pompeu Fabra

Barcelona Supercomputing Center

Barcelona Supercomputing Center

maite.melero@bsc.es

mar.dominguez01@estudiant.upf.edu

Gemma Boleda Torrent

Universitat Pompeu Fabra

gemma.boleda@upf.edu

## Abstract

Object Naming is an important task within the field of Language and Vision that consists of generating a correct and appropriate name for an object given an image. The ManyNames dataset uses real-world human annotated images with multiple labels, instead of just one. In this work, we describe the adaptation of this dataset (originally in English) to Catalan, by (i) machine-translating the English labels and (ii) collecting human annotations for a subset of the original corpus and comparing both resources. Analyses reveal divergences in the lexical variation of the two sets showing potential problems of directly translated resources, particularly when there is no resource to a proper context, which in this case is conveyed by the image. The analysis also points to the impact of cultural factors in the naming task, which should be accounted for in future cross-lingual naming tasks.

## 1 Introduction

Most NLP resources are only available for a small percentage of languages (Joshi et al., 2020), being the rest of the languages spoken in the world left behind. This affects also Catalan, which can be considered a moderately under-resourced language (Armengol-Estapé et al., 2021). In the Language and Vision area, although significantly large datasets of annotated images have been created for a variety of tasks for English, to date no resources of this kind exist for Catalan. In this work we present CAT ManyNames<fnref n="1" />, the Catalan version of the ManyNames dataset, which is the first available resource for the task of Object Naming in Catalan. The dataset has been translated from the English version and its test set has been human annotated to assess the quality of the translation. We also provide analyses of the sources of variation between

the human annotated dataset and its translated counterpart.

## 2 Background

### 2.1 Object Naming: an Interdisciplinary Task

Naming an object accounts for picking out a nominal to refer to it (Silberer et al., 2020a), and is a linguistic phenomenon that can show lexical variation. On the one hand, objects can belong to different semantic categories at the same time (i.e., a baby boy belongs to the categories PERSON, CHILD, BOY, HUMAN, etc.), which, according to Brown (Brown, 1958) could all be valid alternatives for naming that object. On the other hand, the three different levels within semantic categorization<fnref n="2" /> identified by Rosch et al. (Rosch et al., 1976) can all be valid alternatives for naming the same object as well. Although the basic-level categories are considered to be the most natural terms for speakers when referring to objects (Hajibayova, 2013; Jolicoeur et al., 1984; Rosch et al., 1976), these are not universal categories since they are restricted by perceptive, cognitive and environmental factors that can result in lexical variation (Berlin, 2014; Graf et al., 2016; Malt, 1995; Wierzbicka, 1996).

While the task of Object Naming has been studied in both Language and Vision and Psycholinguistics, and it is related to Object Recognition tasks in Computer Vision, each field has a different approach to the task:

Within the field of Language and Vision, datasets typically collect free and natural referential utterances<fnref n="3" /> produced by annotators for a given real-world image. Some relevant datasets are RefCOCO

<footnote n="1">¹Available at https://huggingface.co/datasets/projecte-aina/catManyNames</footnote>

<footnote n="2">²The superordinate level (i.e., animal), the basic level (i.e., dog), and the subordinate level (i.e., Chihuahua)</footnote>

<footnote n="3">³In semantics, a referring expression is a piece of language (typically a noun phrase) used with a particular referent in mind that refers to something or someone, or a clearly delimited collection of things or people (Hurford et al., 2007).</footnote>
(and its newer variant RefCOCO+) (Yu et al., 2016), Flickr30k Entities (Plummer et al., 2015), and VisualGenome (Krishna et al., 2017). Although naming occurs within those datasets, it is not normally marked up and linked to its corresponding image regions.

The task of picture naming constitutes an important experimental paradigm on research in Cognitive Science and Psycholinguistics, and has been traditionally used to assess language impairments and difficulties recalling general knowledge from semantic memory (Snodgrass and Vanderwart, 1980). Subjects reach a high agreement in this task, but it must be taken into account that participants are normally shown line drawing pictures that depict a prototypical category rather than real-world images that show objects in a context.

In Computer Vision, the task of Object Recognition identifies and classifies objects into several different categories (Russakovsky et al., 2015). Nevertheless, current recognition benchmarks use labels and images from ImageNet (Deng et al., 2009) that assume a single ground-truth label, ignoring linguistic variation.

As we can see, the task of Object Naming is addressed differently in Cognitive Science, Language and Vision and Computer Vision, but it would highly benefit from bringing together the particularities of each field so as to generate and provide quality resources.

## 2.2 The ManyNames Dataset

The ManyNames dataset (Silberer et al., 2020a) provides up to 36 crowd-sourced names for 25K object instances extracted from VisualGenome (Krishna et al., 2017). Unlike other Language and Vision datasets, it focuses on Object Naming rather than collecting complete utterances. Data collection was inspired by the picture naming norms developed in Psycholinguistics (Snodgrass and Vanderwart, 1980) but using real-world images of objects in a visual context, making it suitable for analysis and modeling of object naming, as well as for research in Language and Vision.

Images were selected from seven domains<fnref n="4" /> (ANIMALS_PLANTS, BUILDINGS, CLOTHING, FOOD, HOME, PEOPLE, VEHICLES) by defining 52 synsets from VisualGenome in order to collect instances from different taxonomic levels. In-

stances were sampled depending on the size of the number of names obtained per synset in order to balance the collection. The annotations were collected by setting a crowdsourcing elicitation task on Amazon Mechanical Turk (AMT). The procedure required several annotation rounds, in which problematic cases such as unclear bounding boxes or occluded objects were discarded. Because of noise in the data, a second version of ManyNames (MN v2) was released (Silberer et al., 2020b), which is a verified dataset that contains consistent response sets with adequate responses that refer to the same object only. The resulting dataset contained substantial variation (2.2 names per object on average in MN v2). ANIMALS_PLANTS obtained the highest agreement, whereas PEOPLE reached a particularly low agreement. The analysis performed on the Bottom-Up model (Anderson et al., 2018) using the ManyNames dataset (Silberer et al., 2020b) showed that single-label data underestimated model effectiveness against multi-label data, obtaining a lower accuracy. This demonstrates that, compared to single-label resources for Object Naming, the ManyNames dataset provides a more accurate picture of human naming preferences by taking into account linguistic variation.

## 3 A New Dataset for Object Naming in Catalan

The main motivations for using the ManyNames dataset as source data are (i) its consideration of linguistic variation in Object Naming, which is widely ignored up to now in Computer Vision, and (ii) the better accuracy that has shown to perform against single-label datasets in Language and Vision modelling. In order to obtain a Catalan version of ManyNames, we decided to automatically translate all the annotations in the original English dataset to Catalan using a state-of-the-art Machine Translation (MT) tool. To assess the quality of the resulting resource, we collected real human annotations for a subset of the dataset, consisting of around 1K images. Although the size of the manually annotated subset may seem small, it can be considered standard for a test set with the purpose of evaluating the quality of automatic annotations. Table 1 shows an overview of the columns contained in the CAT ManyNames dataset.

<footnote n="4">⁴All domains are based on McRae's feature norms (McRae et al., 2005) except PEOPLE, which was considered to be salient due to its prominence for humans.</footnote>
(and its newer variant RefCOCO+) (Yu et al., 2016), Flickr30k Entities (Plummer et al., 2015), and VisualGenome (Krishna et al., 2017). Although naming occurs within those datasets, it is not normally marked up and linked to its corresponding image regions.

The task of picture naming constitutes an important experimental paradigm on research in Cognitive Science and Psycholinguistics, and has been traditionally used to assess language impairments and difficulties recalling general knowledge from semantic memory (Snodgrass and Vanderwart, 1980). Subjects reach a high agreement in this task, but it must be taken into account that participants are normally shown line drawing pictures that depict a prototypical category rather than real-world images that show objects in a context.

In Computer Vision, the task of Object Recognition identifies and classifies objects into several different categories (Russakovsky et al., 2015). Nevertheless, current recognition benchmarks use labels and images from ImageNet (Deng et al., 2009) that assume a single ground-truth label, ignoring linguistic variation.

As we can see, the task of Object Naming is addressed differently in Cognitive Science, Language and Vision and Computer Vision, but it would highly benefit from bringing together the particularities of each field so as to generate and provide quality resources.

## 2.2 The ManyNames Dataset

The ManyNames dataset (Silberer et al., 2020a) provides up to 36 crowd-sourced names for 25K object instances extracted from VisualGenome (Krishna et al., 2017). Unlike other Language and Vision datasets, it focuses on Object Naming rather than collecting complete utterances. Data collection was inspired by the picture naming norms developed in Psycholinguistics (Snodgrass and Vanderwart, 1980) but using real-world images of objects in a visual context, making it suitable for analysis and modeling of object naming, as well as for research in Language and Vision.

Images were selected from seven domains<fnref n="4" /> (ANIMALS_PLANTS, BUILDINGS, CLOTHING, FOOD, HOME, PEOPLE, VEHICLES) by defining 52 synsets from VisualGenome in order to collect instances from different taxonomic levels. In-

stances were sampled depending on the size of the number of names obtained per synset in order to balance the collection. The annotations were collected by setting a crowdsourcing elicitation task on Amazon Mechanical Turk (AMT). The procedure required several annotation rounds, in which problematic cases such as unclear bounding boxes or occluded objects were discarded. Because of noise in the data, a second version of ManyNames (MN v2) was released (Silberer et al., 2020b), which is a verified dataset that contains consistent response sets with adequate responses that refer to the same object only. The resulting dataset contained substantial variation (2.2 names per object on average in MN v2). ANIMALS_PLANTS obtained the highest agreement, whereas PEOPLE reached a particularly low agreement. The analysis performed on the Bottom-Up model (Anderson et al., 2018) using the ManyNames dataset (Silberer et al., 2020b) showed that single-label data underestimated model effectiveness against multi-label data, obtaining a lower accuracy. This demonstrates that, compared to single-label resources for Object Naming, the ManyNames dataset provides a more accurate picture of human naming preferences by taking into account linguistic variation.

## 3 A New Dataset for Object Naming in Catalan

The main motivations for using the ManyNames dataset as source data are (i) its consideration of linguistic variation in Object Naming, which is widely ignored up to now in Computer Vision, and (ii) the better accuracy that has shown to perform against single-label datasets in Language and Vision modelling. In order to obtain a Catalan version of ManyNames, we decided to automatically translate all the annotations in the original English dataset to Catalan using a state-of-the-art Machine Translation (MT) tool. To assess the quality of the resulting resource, we collected real human annotations for a subset of the dataset, consisting of around 1K images. Although the size of the manually annotated subset may seem small, it can be considered standard for a test set with the purpose of evaluating the quality of automatic annotations. Table 1 shows an overview of the columns contained in the CAT ManyNames dataset.

<footnote n="4">⁴All domains are based on McRae's feature norms (McRae et al., 2005) except PEOPLE, which was considered to be salient due to its prominence for humans.</footnote>
<table><caption>Table 1: Description of the columns in the CAT ManyNames</caption><thead><tr><th>Column</th><th>Type</th><th>Description</th></tr></thead><tbody><tr><td>responses</td><td>dict</td><td>Correct responses and their counts</td></tr><tr><td>topname</td><td>str</td><td>The most frequent name of the object</td></tr><tr><td>domain</td><td>str</td><td>The ManyNames domain of the object</td></tr><tr><td>incorrect<sup>5</sup></td><td>dict</td><td>Incorrect responses and their counts</td></tr><tr><td>singletons<sup>6</sup></td><td>dict</td><td>All responses which were given only once</td></tr><tr><td>totalResponses</td><td>int</td><td>Sum count of correct responses</td></tr><tr><td>split</td><td>str</td><td>Use of the images in training, test and validation</td></tr><tr><td>vg_object_id</td><td>int</td><td>The VisualGenome id of the object</td></tr><tr><td>vg_image_id</td><td>int</td><td>The VisualGenome id of the image</td></tr><tr><td>topname_agreement<sup>7</sup></td><td>int</td><td>Top name responses divided by total responses</td></tr><tr><td>jaccard_similarity<sup>8</sup></td><td>int</td><td>Jaccard similarity index of the responses column</td></tr><tr><td>rawOLA responses<sup>9</sup></td><td>dict</td><td>Uncorrected responses in the human annotated test set</td></tr></tbody></table>

## 3.1 Translated Annotations

Two different neural MT systems were considered before carrying out the translation of the ManyNames dataset: SoftCatalà and Google Translate.

Softcatalà is an open-source initiative <fnref n="10" /> that, among other free NLP tools, offers automatic translation services between Catalan and several languages based on neural network technology (Mas, 2021). The popular Google Translate engine, which provides translation services between more than 100 language pairs (Caswell and Liang, 2020), was also considered.

Given the lack of linguistic context in the annotations to be translated (which were, in most cases, a single word), sense disambiguation was a major linguistic issue that needed to be solved before carrying out the automatic translation. Since no current MT system is yet able to take advantage of images as context <fnref n="11" />, ad-hoc linguistic contexts were automatically inserted in each input string in order to compensate for this. The linguistic patterns were added using regular expressions depending on the domain. For example, in the domain HOME, the following pattern was used: "I bought a/an [word] for my home.".

Once the linguistic contexts were added, the resulting sentences from the training split of the data were translated with both SoftCatalà and Google Translate. In order to evaluate which system performed a better translation, a random sample of 500 sentences out of the total translated sentences

was collected and its quality was manually evaluated. 403 sentences out of 500 had an identical translation in both systems, but in 74 cases Google Translate got a more accurate translation than SoftCatalà (which only surpassed Google Translate in 23 examples), probably due to having been trained with larger amounts of data. As a result, Google Translate was considered as a better option for performing the automatic translation of the dataset. The linguistic patterns added in order to disambiguate were removed after the translation of the whole dataset, and repeated words, as well as their counts, were merged.

## 3.2 Manual Annotation of the Test Set

In order to gather as many manual annotations as possible for the test set, an annotation campaign was launched for a subset of 1,072 images. For this, we used 22 different Google Forms<fnref n="12" />, each containing 50 images<fnref n="13" />. Participants were asked to fill one of the Google forms (picked at random) and name the object, animal or person inside the bounding box with the first name that came to their mind. Demographic information about participants was collected during the survey, such as age, gender and region of origin. Statistics show that they were quite balanced in terms of age and gender, but in terms of geographical variation, the Central Catalan dialect was largely over-represented. At the end of the campaign, a total of 220 native Catalan speakers had participated, gathering a total of 10,072 annotations, corresponding to 10 annota-

<footnote n="10">¹⁰Visit the following link for further information: https://github.com/Softcatala/nmt-softcatala</footnote>

<footnote n="11">¹¹Please note that in order to carry out the automatic translation, images were not considered</footnote>

<footnote n="12">¹²Among the main reasons to use Google forms are its simplicity of use and the possibility to fill in surveys from a mobile device.</footnote>

<footnote n="13">¹³Except the last one, which contained 22 images.</footnote>
<table><caption>Table 1: Description of the columns in the CAT ManyNames</caption><thead><tr><th>Column</th><th>Type</th><th>Description</th></tr></thead><tbody><tr><td>responses</td><td>dict</td><td>Correct responses and their counts</td></tr><tr><td>topname</td><td>str</td><td>The most frequent name of the object</td></tr><tr><td>domain</td><td>str</td><td>The ManyNames domain of the object</td></tr><tr><td>incorrect<sup>5</sup></td><td>dict</td><td>Incorrect responses and their counts</td></tr><tr><td>singletons<sup>6</sup></td><td>dict</td><td>All responses which were given only once</td></tr><tr><td>totalResponses</td><td>int</td><td>Sum count of correct responses</td></tr><tr><td>split</td><td>str</td><td>Use of the images in training, test and validation</td></tr><tr><td>vg_object_id</td><td>int</td><td>The VisualGenome id of the object</td></tr><tr><td>vg_image_id</td><td>int</td><td>The VisualGenome id of the image</td></tr><tr><td>topname_agreement<sup>7</sup></td><td>int</td><td>Top name responses divided by total responses</td></tr><tr><td>jaccard_similarity<sup>8</sup></td><td>int</td><td>Jaccard similarity index of the responses column</td></tr><tr><td>rawOLA</td><td>dict</td><td>Uncorrected responses in the human annotated test set</td></tr></tbody></table>

## 3.1 Translated Annotations

Two different neural MT systems were considered before carrying out the translation of the ManyNames dataset: SoftCatalà and Google Translate.

Softcatalà is an open-source initiative <fnref n="10" /> that, among other free NLP tools, offers automatic translation services between Catalan and several languages based on neural network technology (Mas, 2021). The popular Google Translate engine, which provides translation services between more than 100 language pairs (Caswell and Liang, 2020), was also considered.

Given the lack of linguistic context in the annotations to be translated (which were, in most cases, a single word), sense disambiguation was a major linguistic issue that needed to be solved before carrying out the automatic translation. Since no current MT system is yet able to take advantage of images as context <fnref n="11" />, ad-hoc linguistic contexts were automatically inserted in each input string in order to compensate for this. The linguistic patterns were added using regular expressions depending on the domain. For example, in the domain HOME, the following pattern was used: "I bought a/an [word] for my home.".

Once the linguistic contexts were added, the resulting sentences from the training split of the data were translated with both SoftCatalà and Google Translate. In order to evaluate which system performed a better translation, a random sample of 500 sentences out of the total translated sentences

was collected and its quality was manually evaluated. 403 sentences out of 500 had an identical translation in both systems, but in 74 cases Google Translate got a more accurate translation than SoftCatalà (which only surpassed Google Translate in 23 examples), probably due to having been trained with larger amounts of data. As a result, Google Translate was considered as a better option for performing the automatic translation of the dataset. The linguistic patterns added in order to disambiguate were removed after the translation of the whole dataset, and repeated words, as well as their counts, were merged.

## 3.2 Manual Annotation of the Test Set

In order to gather as many manual annotations as possible for the test set, an annotation campaign was launched for a subset of 1,072 images. For this, we used 22 different Google Forms<fnref n="12" />, each containing 50 images<fnref n="13" />. Participants were asked to fill one of the Google forms (picked at random) and name the object, animal or person inside the bounding box with the first name that came to their mind. Demographic information about participants was collected during the survey, such as age, gender and region of origin. Statistics show that they were quite balanced in terms of age and gender, but in terms of geographical variation, the Central Catalan dialect was largely over-represented. At the end of the campaign, a total of 220 native Catalan speakers had participated, gathering a total of 10,072 annotations, corresponding to 10 annota-

<footnote n="10">¹⁰Visit the following link for further information: https://github.com/Softcatala/nmt-softcatala</footnote>

<footnote n="11">¹¹Please note that in order to carry out the automatic translation, images were not considered</footnote>

<footnote n="12">¹²Among the main reasons to use Google forms are its simplicity of use and the possibility to fill in surveys from a mobile device.</footnote>

<footnote n="13">¹³Except the last one, which contained 22 images.</footnote>
tions per image<fnref n="14" />.

Post-processing steps for the human annotations included spellchecking the responses. After this step, possible erroneous responses were filtered out by comparing the corrected responses to the *incorrect* translated column of the ManyNames dataset and were also manually revised. In the process, possible offensive and/or inadequate content were also eliminated. Counts were added once the filtering process was finished. The resulting manually annotated subset has been published with an open license¹⁵.

# 4 Analysis and Discussion

The purpose of the analysis was, on the one hand, to assess the quality of the automatic translation in the subset that had been human annotated by comparing both the translation and the human annotations, and on the other, to explore possible differences in lexical choices based on cultural biases. To this end, the accuracy of the top name, the degree of variation per image, the average number of different responses per image and the agreement on the top name were computed for both test sets.

The most immediate measure to evaluate the quality of the translated test set was to compute the accuracy of the translation of the most frequent response per image (aka the top name) by comparing it with the corresponding top name in the human annotated set. This accuracy only reached 67,91%, which is a clear indication of how different both resources are.

Another interesting metric to be computed was the degree of lexical variation in the two sets. Despite the difference in the number of annotations (36 for the translated vs 10 for the human annotated), the average number of types in the translated test set was 2.1 responses per image, whereas in the human annotated test set, it was 3.1¹⁶, showing greater lexical variation in the human annotated test set. To account for this clear divergence, we could hypothesize that often two different names get conflated into one in the translation process. However, the ratio of the translated dataset (2.1) is

very close to the 2.2 names per object on average in the original ManyNames dataset.

A related metric that was also applied is agreement on the top name per image, which is computed by dividing the number of responses for the top name by the number of total responses. Since more variation is observed in the human annotated set, we expect a higher agreement in the translated set. Indeed, the median is higher in the translated data (0.93) than in the human annotated data (0.7).

A qualitative analysis was performed by sorting both test sets by domain and top name and manually inspecting them to spot divergent cases of translation between English and Catalan. Several findings account for the observed richer lexical variation in Catalan: it was found that Catalan speakers tended to choose a subordinate name (*portaveu*, *esportista*, *tennista*, *etc.*) rather than a taxonomic name (*dona*, *noi*, *noia*, *etc.*) in the PEOPLE domain, the exception being images that involved specific terminology of an activity or a sport not specific to the Catalan culture, i.e. baseball or skateboarding. In those cases, Catalan speakers tended to choose the basic level (*jugador*, *noi*) rather than the subordinate level (*batedor*, *patinador*). Certain domains, such as CLOTHING showed Catalan to be more specific than English (which had repercussions in the translation). For example, *jacket* can be translated as *americana* or *jaqueta*, depending on the formality of the event. In addition, Catalan speakers may opt for the use of a diminutive (*trenet* vs *tren*), but this is a lexical option that English speakers do not have.

Our analyses show major divergences between the automatically translated dataset and the manually annotated subset, both in terms of degree of internal lexical variation and accuracy of the translated top names. Manual inspection of the results further confirms that these divergences can be attributed to linguistic and even cultural differences. Automatic translation of language resources from well-resourced languages to less-resourced ones is a common practice in NLP and related fields. Our results show that linguistic and cultural differences may affect the quality of automatically translated resources, such as the one presented here.

# 5 Conclusions

In this paper, we have presented a new dataset for the task of Object Naming in Catalan, namely CAT ManyNames. The new resource is the result of

<footnote n="14">¹⁴Time constraints prevented us from gathering more annotations per image, but for the purposes of the present exercise, 10 annotations looks like an acceptable number

¹⁵Available at https://huggingface.co/datasets/projecteaina/cat manynames

¹⁶As for the types by domain, the human annotated test set has more types in all domains except in FOOD and CLOTHING, where both test sets have the same number of types.</footnote>
tions per image<fnref n="14" />.

Post-processing steps for the human annotations included spellchecking the responses. After this step, possible erroneous responses were filtered out by comparing the corrected responses to the *incorrect* translated column of the ManyNames dataset and were also manually revised. In the process, possible offensive and/or inadequate content were also eliminated. Counts were added once the filtering process was finished. The resulting manually annotated subset has been published with an open license¹⁵.

# 4 Analysis and Discussion

The purpose of the analysis was, on the one hand, to assess the quality of the automatic translation in the subset that had been human annotated by comparing both the translation and the human annotations, and on the other, to explore possible differences in lexical choices based on cultural biases. To this end, the accuracy of the top name, the degree of variation per image, the average number of different responses per image and the agreement on the top name were computed for both test sets.

The most immediate measure to evaluate the quality of the translated test set was to compute the accuracy of the translation of the most frequent response per image (aka the top name) by comparing it with the corresponding top name in the human annotated set. This accuracy only reached 67,91%, which is a clear indication of how different both resources are.

Another interesting metric to be computed was the degree of lexical variation in the two sets. Despite the difference in the number of annotations (36 for the translated vs 10 for the human annotated), the average number of types in the translated test set was 2.1 responses per image, whereas in the human annotated test set, it was 3.1¹⁶, showing greater lexical variation in the human annotated test set. To account for this clear divergence, we could hypothesize that often two different names get conflated into one in the translation process. However, the ratio of the translated dataset (2.1) is

very close to the 2.2 names per object on average in the original ManyNames dataset.

A related metric that was also applied is agreement on the top name per image, which is computed by dividing the number of responses for the top name by the number of total responses. Since more variation is observed in the human annotated set, we expect a higher agreement in the translated set. Indeed, the median is higher in the translated data (0.93) than in the human annotated data (0.7).

A qualitative analysis was performed by sorting both test sets by domain and top name and manually inspecting them to spot divergent cases of translation between English and Catalan. Several findings account for the observed richer lexical variation in Catalan: it was found that Catalan speakers tended to choose a subordinate name (*portaveu, esportista, tennista, etc.*) rather than a taxonomic name (*dona, noi, noia, etc.*) in the PEOPLE domain, the exception being images that involved specific terminology of an activity or a sport not specific to the Catalan culture, i.e. baseball or skateboarding. In those cases, Catalan speakers tended to choose the basic level (*jugador, noi*) rather than the subordinate level (*batedor, patinador*). Certain domains, such as CLOTHING showed Catalan to be more specific than English (which had repercussions in the translation). For example, *jacket* can be translated as *americana* or *jaqueta*, depending on the formality of the event. In addition, Catalan speakers may opt for the use of a diminutive (*trenet* vs *tren*), but this is a lexical option that English speakers do not have.

Our analyses show major divergences between the automatically translated dataset and the manually annotated subset, both in terms of degree of internal lexical variation and accuracy of the translated top names. Manual inspection of the results further confirms that these divergences can be attributed to linguistic and even cultural differences. Automatic translation of language resources from well-resourced languages to less-resourced ones is a common practice in NLP and related fields. Our results show that linguistic and cultural differences may affect the quality of automatically translated resources, such as the one presented here.

# 5 Conclusions

In this paper, we have presented a new dataset for the task of Object Naming in Catalan, namely CAT ManyNames. The new resource is the result of

<footnote n="14">¹⁴Time constraints prevented us from gathering more annotations per image, but for the purposes of the present exercise, 10 annotations looks like an acceptable number

¹⁵Available at https://huggingface.co/datasets/projecteaina/cat manynames

¹⁶As for the types by domain, the human annotated test set has more types in all domains except in FOOD and CLOTHING, where both test sets have the same number of types.</footnote>
the machine translation of the English ManyNames dataset, with some pre- and post-processing steps. It also includes a subset of 1,072 images which has been entirely human annotated with 10 annotations per image. The comparison between the translated and the human annotated subsets reveals cultural-based divergences in lexical choices that can affect the quality of the machine-translated resource. Our results shows potential weaknesses in resources built up by translating annotations, particularly in the Language and Vision field, where context is provided by the image and thus is not available to the machine translation system. Since current literature on Object Naming within the Language and Vision field is scarce, these findings could serve as a starting point for research on cross-lingual Object Naming, and on the impact of automatic translation in the annotation of multilingual resources.

# 6 Acknowledgements

This research was supported by the Generalitat de Catalunya through the AINA project. We thank our colleagues in the Text Mining Unit at the Barcelona Supercomputing Center, who provided insight and expertise that greatly assisted the research. We would also like to thank the anonymous reviewers for their insightful comments.

# References

Peter Anderson, Xiaodong He, Chris Buehler, Damien Teney, Mark Johnson, Stephen Gould, and Lei Zhang. 2018. Bottom-up and top-down attention for image captioning and visual question answering. In *Proceedings of the IEEE conference on computer vision and pattern recognition*, pages 6077–6086.

Jordi Armengol-Estapé, Casimiro Pio Carrino, Carlos Rodriguez-Penagos, Ona de Gibert Bonet, Carme Armentano-Oller, Aitor Gonzalez-Agirre, Maite Melero, and Marta Villegas. 2021. Are multilingual models the best choice for moderately underresourced languages? A comprehensive assessment for Catalan. In *Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021*, pages 4933–4946. Online. Association for Computational Linguistics.

Brent Berlin. 2014. *Ethnobiological classification: Principles of categorization of plants and animals in traditional societies*, volume 185. Princeton University Press.

Roger Brown. 1958. How shall a thing be called? *Psychological review*, 65(1):14.

Isaac Caswell and Bowen Liang. 2020. Recent advances in *google translate*.

Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. 2009. Imagenet: A large-scale hierarchical image database. In *2009 IEEE Conference on Computer Vision and Pattern Recognition*, pages 248–255.

Caroline Graf, Judith Degen, Robert XD Hawkins, and Noah D Goodman. 2016. Animal, dog, or dalmatian? level of abstraction in nominal referring expressions. In *CogSci*.

Lala Hajibayova. 2013. Basic-level categories: A review. *Journal of Information Science*, 39(5):676–687.

James R Hurford, Brendan Heasley, and Michael B Smith. 2007. *Semantics: a coursebook*. Cambridge university press.

Pierre Jolicoeur, Mark A Gluck, and Stephen M Kosslyn. 1984. Pictures and names: Making the connection. *Cognitive psychology*, 16(2):243–275.

Pratik Joshi, Sebastin Santy, Amar Budhiraja, Kalika Bali, and Monojit Choudhury. 2020. The state and fate of linguistic diversity and inclusion in the NLP world. In *Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics*, pages 6282–6293, Online. Association for Computational Linguistics.

Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A Shamma, et al. 2017. Visual genome: Connecting language and vision using crowdsourced dense image annotations. *International journal of computer vision*, 123(1):32–73.

Barbara C Malt. 1995. Category coherence in cross-cultural perspective. *Cognitive psychology*, 29(2):85–148.

Jordi Mas. 2021. Resum de l'any 2020 a softcatalà.

Ken McRae, George S Cree, Mark S Seidenberg, and Chris McNorgan. 2005. Semantic feature production norms for a large set of living and nonliving things. *Behavior research methods*, 37(4):547–559.

Bryan A Plummer, Liwei Wang, Chris M Cervantes, Juan C Caicedo, Julia Hockenmaier, and Svetlana Lazebnik. 2015. Flickr30k entities: Collecting region-to-phrase correspondences for richer image-to-sentence models. In *Proceedings of the IEEE international conference on computer vision*, pages 2641–2649.

Eleanor Rosch, Carol Simpson, and R Scott Miller. 1976. Structural bases of typicality effects. *Journal of Experimental Psychology: Human perception and performance*, 2(4):491.

Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein,
the machine translation of the English ManyNames dataset, with some pre- and post-processing steps. It also includes a subset of 1,072 images which has been entirely human annotated with 10 annotations per image. The comparison between the translated and the human annotated subsets reveals cultural-based divergences in lexical choices that can affect the quality of the machine-translated resource. Our results shows potential weaknesses in resources built up by translating annotations, particularly in the Language and Vision field, where context is provided by the image and thus is not available to the machine translation system. Since current literature on Object Naming within the Language and Vision field is scarce, these findings could serve as a starting point for research on cross-lingual Object Naming, and on the impact of automatic translation in the annotation of multilingual resources.

# 6 Acknowledgements

This research was supported by the Generalitat de Catalunya through the AINA project. We thank our colleagues in the Text Mining Unit at the Barcelona Supercomputing Center, who provided insight and expertise that greatly assisted the research. We would also like to thank the anonymous reviewers for their insightful comments.

# References

Peter Anderson, Xiaodong He, Chris Buehler, Damien Teney, Mark Johnson, Stephen Gould, and Lei Zhang. 2018. Bottom-up and top-down attention for image captioning and visual question answering. In *Proceedings of the IEEE conference on computer vision and pattern recognition*, pages 6077–6086.

Jordi Armengol-Estapé, Casimiro Pio Carrino, Carlos Rodriguez-Penagos, Ona de Gibert Bonet, Carme Armentano-Oller, Aitor Gonzalez-Agirre, Maite Melero, and Marta Villegas. 2021. Are multilingual models the best choice for moderately underresourced languages? A comprehensive assessment for Catalan. In *Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021*, pages 4933–4946. Online. Association for Computational Linguistics.

Brent Berlin. 2014. *Ethnobiological classification: Principles of categorization of plants and animals in traditional societies*, volume 185. Princeton University Press.

Roger Brown. 1958. How shall a thing be called? *Psychological review*, 65(1):14.

Isaac Caswell and Bowen Liang. 2020. Recent advances in *google translate*.

Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. 2009. Imagenet: A large-scale hierarchical image database. In *2009 IEEE Conference on Computer Vision and Pattern Recognition*, pages 248–255.

Caroline Graf, Judith Degen, Robert XD Hawkins, and Noah D Goodman. 2016. Animal, dog, or dalmatian? level of abstraction in nominal referring expressions. In *CogSci*.

Lala Hajibayova. 2013. Basic-level categories: A review. *Journal of Information Science*, 39(5):676–687.

James R Hurford, Brendan Heasley, and Michael B Smith. 2007. *Semantics: a coursebook*. Cambridge university press.

Pierre Jolicoeur, Mark A Gluck, and Stephen M Kosslyn. 1984. Pictures and names: Making the connection. *Cognitive psychology*, 16(2):243–275.

Pratik Joshi, Sebastin Santy, Amar Budhiraja, Kalika Bali, and Monojit Choudhury. 2020. The state and fate of linguistic diversity and inclusion in the NLP world. In *Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics*, pages 6282–6293, Online. Association for Computational Linguistics.

Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A Shamma, et al. 2017. Visual genome: Connecting language and vision using crowdsourced dense image annotations. *International journal of computer vision*, 123(1):32–73.

Barbara C Malt. 1995. Category coherence in cross-cultural perspective. *Cognitive psychology*, 29(2):85–148.

Jordi Mas. 2021. Resum de l'any 2020 a softcatalà.

Ken McRae, George S Cree, Mark S Seidenberg, and Chris McNorgan. 2005. Semantic feature production norms for a large set of living and nonliving things. *Behavior research methods*, 37(4):547–559.

Bryan A Plummer, Liwei Wang, Chris M Cervantes, Juan C Caicedo, Julia Hockenmaier, and Svetlana Lazebnik. 2015. Flickr30k entities: Collecting region-to-phrase correspondences for richer image-to-sentence models. In *Proceedings of the IEEE international conference on computer vision*, pages 2641–2649.

Eleanor Rosch, Carol Simpson, and R Scott Miller. 1976. Structural bases of typicality effects. *Journal of Experimental Psychology: Human perception and performance*, 2(4):491.

Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein,
et al. 2015. Imagenet large scale visual recognition
challenge. *International journal of computer vision*,
115(3):211–252.

Carina Silberer, Sina Zarrieß, and Gemma Boleda. 2020a. Object naming in language and vision: A survey and a new dataset. In *Calzolari N, Béchet F, Blache P, Choukri K, Cieri C, Declerck T, Goggi S, Isahara H, Maegaard B, Mariani J, Mazo H, Moreno A, Odijk J, Piperidis S, editors. Proceedings of the 12th Language Resources and Evaluation Conference; 2020 May 13-15; Marseilles, France*. Stroudsburg (PA): ACL; 2020. p. 5792-801. ACL (Association for Computational Linguistics).

Carina Silberer, Sina Zarrieß, Matthijs Westera, and Gemma Boleda. 2020b. Humans meet models on object naming: A new dataset and analysis. In *Proceedings of the 28th International Conference on Computational Linguistics*, pages 1893–1905.

Joan G Snodgrass and Mary Vanderwart. 1980. A standardized set of 260 pictures: norms for name agreement, image agreement, familiarity, and visual complexity. *Journal of experimental psychology: Human learning and memory*, 6(2):174.

Anna Wierzbicka. 1996. *Semantics: Primes and universals: Primes and universals*. Oxford University Press, UK.

Licheng Yu, Patrick Poirson, Shan Yang, Alexander C Berg, and Tamara L Berg. 2016. Modeling context in referring expressions. In *European Conference on Computer Vision*, pages 69–85. Springer.
et al. 2015. Imagenet large scale visual recognition
challenge. *International journal of computer vision*,
115(3):211–252.

Carina Silberer, Sina Zarrieß, and Gemma Boleda. 2020a. Object naming in language and vision: A survey and a new dataset. In *Calzolari N, Béchet F, Blache P, Choukri K, Cieri C, Declerck T, Goggi S, Isahara H, Maegaard B, Mariani J, Mazo H, Moreno A, Odijk J, Piperidis S, editors. Proceedings of the 12th Language Resources and Evaluation Conference; 2020 May 13-15; Marseilles, France*. Stroudsburg (PA): ACL; 2020. p. 5792-801. ACL (Association for Computational Linguistics).

Carina Silberer, Sina Zarrieß, Matthijs Westera, and Gemma Boleda. 2020b. Humans meet models on object naming: A new dataset and analysis. In *Proceedings of the 28th International Conference on Computational Linguistics*, pages 1893–1905.

Joan G Snodgrass and Mary Vanderwart. 1980. A standardized set of 260 pictures: norms for name agreement, image agreement, familiarity, and visual complexity. *Journal of experimental psychology: Human learning and memory*, 6(2):174.

Anna Wierzbicka. 1996. *Semantics: Primes and universals: Primes and universals*. Oxford University Press, UK.

Licheng Yu, Patrick Poirson, Shan Yang, Alexander C Berg, and Tamara L Berg. 2016. Modeling context in referring expressions. In *European Conference on Computer Vision*, pages 69–85. Springer.
# Finetuning Latin BERT for Word Sense Disambiguation
on the *Thesaurus Linguae Latinae*

Piroska Lendvai

Department for Digital Humanities Research & Development
Bavarian Academy of Sciences and Humanities
Munich, Germany
piroska.lendvai@badw.de

Claudia Wick

Thesaurus Linguae Latinae
Bavarian Academy of Sciences and Humanities
Munich, Germany
claudia.wick@thesaurus.badw.de

## Abstract

The *Thesaurus Linguae Latinae* (TLL) is a comprehensive monolingual dictionary that records contextualized meanings and usages of Latin words in antique sources at an unprecedented scale. We created a new dataset based on a subset of sense representations in the TLL, with which we finetuned the Latin-BERT neural language model (Bamman and Burns, 2020) on a supervised Word Sense Disambiguation task. We observe that the contextualized BERT representations finetuned on TLL data score better than static embeddings used in a bidirectional LSTM classifier on the same dataset, and that our per-lemma BERT models achieve higher and more robust performance than reported by Bamman and Burns (2020) based on data from a bilingual Latin dictionary. We discuss the differences in sense organizational principles between these two lexical resources, and report about our dataset construction and improved evaluation methodology.

## 1 Introduction

In the field of Natural Language Processing (NLP), there is a growing amount of languages for which contextualized representation models are created. For Latin, a pretrained BERT model (cf. Devlin et al., 2018) was published by Bamman and Burns (2020), which they finetuned for four classical NLP tasks, among others for Word Sense Disambiguation (WSD). WSD, an area of computational semantics, has been approached in NLP by several machine learning setups (for an overview cf. Navigation, 2009 and Bevilacqua et al., 2021), and recent works (e.g. Scarlini et al., 2020) have also targeted

the use of neural models and architectures in combination with lexical knowledge bases and encyclopaedic resources.

WSD is typically cast as supervised classification, where the learning task consists of predicting the appropriate sense label for one or more focus tokens in their context unit, e.g. within a sentence. Based on the application end task, sense labels can be defined in a variety of ways, e.g. aiming to distinguish coarse or fine granularity of senses, binary or multiple sense distinctions, etc. Creating labeled data for a supervised WSD application is nontrivial. Large, sense-annotated benchmark datasets are scarce, especially in languages other than English. A promising resource to be utilized for Latin WSD could be the Latin Wordnet<fnref n="1" />; for its evaluation and references cf. Franzini et al. (2019). Seeking proxy resources and methods to leverage WSD resources is important, since it is expensive to manually produce a sense labeled corpus from scratch that captures contextual information for several senses of a word. Therefore, our study aims to contribute insights into methods that use dictionaries for automatically assigning sense labels.

Bamman and Burns (2020) (henceforth: B&B) constructed WSD data for BERT, a transformer-based language model, by taking the textual examples (i.e., quotes from antique sources) inventorized for a particular headword (aka 'lemma') in the bilingual Latin Dictionary of Lewis and Short (1879): to each quote snippet in the first two sense groups of each lemma, they assigned its sense category (i.e., I or II) as gold standard label.

Inspired by this sense inventory creation (i.e., bi-

<footnote n="1">¹https://latinwordnet.exeter.ac.uk</footnote>
# Finetuning Latin BERT for Word Sense Disambiguation
on the *Thesaurus Linguae Latinae*

Piroska Lendvai

Department for Digital Humanities Research & Development
Bavarian Academy of Sciences and Humanities
Munich, Germany
piroska.lendvai@badw.de

Claudia Wick

Thesaurus Linguae Latinae
Bavarian Academy of Sciences and Humanities
Munich, Germany
claudia.wick@thesaurus.badw.de

## Abstract

The *Thesaurus Linguae Latinae* (TLL) is a comprehensive monolingual dictionary that records contextualized meanings and usages of Latin words in antique sources at an unprecedented scale. We created a new dataset based on a subset of sense representations in the TLL, with which we finetuned the Latin-BERT neural language model (Bamman and Burns, 2020) on a supervised Word Sense Disambiguation task. We observe that the contextualized BERT representations finetuned on TLL data score better than static embeddings used in a bidirectional LSTM classifier on the same dataset, and that our per-lemma BERT models achieve higher and more robust performance than reported by Bamman and Burns (2020) based on data from a bilingual Latin dictionary. We discuss the differences in sense organizational principles between these two lexical resources, and report about our dataset construction and improved evaluation methodology.

## 1 Introduction

In the field of Natural Language Processing (NLP), there is a growing amount of languages for which contextualized representation models are created. For Latin, a pretrained BERT model (cf. Devlin et al., 2018) was published by Bamman and Burns (2020), which they finetuned for four classical NLP tasks, among others for Word Sense Disambiguation (WSD). WSD, an area of computational semantics, has been approached in NLP by several machine learning setups (for an overview cf. Navigation, 2009 and Bevilacqua et al., 2021), and recent works (e.g. Scarlini et al., 2020) have also targeted

the use of neural models and architectures in combination with lexical knowledge bases and encyclopaedic resources.

WSD is typically cast as supervised classification, where the learning task consists of predicting the appropriate sense label for one or more focus tokens in their context unit, e.g. within a sentence. Based on the application end task, sense labels can be defined in a variety of ways, e.g. aiming to distinguish coarse or fine granularity of senses, binary or multiple sense distinctions, etc. Creating labeled data for a supervised WSD application is nontrivial. Large, sense-annotated benchmark datasets are scarce, especially in languages other than English. A promising resource to be utilized for Latin WSD could be the Latin Wordnet<fnref n="1" />; for its evaluation and references cf. Franzini et al. (2019). Seeking proxy resources and methods to leverage WSD resources is important, since it is expensive to manually produce a sense labeled corpus from scratch that captures contextual information for several senses of a word. Therefore, our study aims to contribute insights into methods that use dictionaries for automatically assigning sense labels.

Bamman and Burns (2020) (henceforth: B&B) constructed WSD data for BERT, a transformer-based language model, by taking the textual examples (i.e., quotes from antique sources) inventorized for a particular headword (aka 'lemma') in the bilingual Latin Dictionary of Lewis and Short (1879): to each quote snippet in the first two sense groups of each lemma, they assigned its sense category (i.e., I or II) as gold standard label.

Inspired by this sense inventory creation (i.e., bi-

<footnote n="1">¹https://latinwordnet.exeter.ac.uk</footnote>
nary class labeling) method of B&B, we requested
data for the same lemmas that B&B presented,
from a currently proprietary resource: the *The-
saurus Linguae Latinae*<fnref n="2" /> (TLL)<fnref n="3" />. The TLL is a
comprehensive monolingual Latin dictionary that
aims to record all meanings of all ancient Latin
words, citing all (or a representative sample) of its
seen attestations. The TLL is vast: it is estimated
to comprise cca. 53k-56k entries as of now<fnref n="4" />, so it
likely holds a major part of the quotes that occur in
L&S and thus in the B&B WSD dataset.

The prospect of comparing WSD performance
across datasets constructed from two dictionaries
– one bilingual, another monolingual – was in-
triguing in several scholarly respects, a.o. for gain-
ing quantitative insights into dictionary structuring
practices, or even for attempting to validate sense
structuring in an empirical way. After inspecting
the data, we realized that a direct comparison of
machine learning performance based on data con-
structed from the TLL resp. from L&S would be
methodologically flawed:

1. We made pilot analyses of the quotes across the B&B and TLL sense labeled data, and noted that sense categorization in TLL and L&S draws on very different semantic principles: for one and the same lemma, the subset of quotes labeled with sense I in B&B can be distributed across both sense class I and II in TLL, and/or vice versa.

2. Working with the methodology of B&B of constructing sense-balanced data would not allow unleashing the full potential of the TLL data size. As we chose not to discard quotes (i.e., did not match the amount of quotes in the smaller sense label set), our TLL dataset became orders of magnitude larger and sense-label-wise possibly more aggregative, thus likely coarser-grained.

Our aims in the current contribution were thus:

* Investigating methods and challenges for experimentally validating sense representations and their WSD distinction

* Giving account of joint work between the Humanities and the NLP communities that deliver complementary expertise

* Reusing a pretrained contextual representation model for Latin, released by Bamman and Burns (2020)

* Reproducing the WSD experiment of Bamman and Burns (2020) via the benchmark data, code, and baseline classifier they released<fnref n="5" />

* Repeating the WSD experiment by finetuning Latin BERT on new WSD data that we constructed from the TLL

* Observing sense organization principles and scale across the two datasets

* Improving experimental methodology by providing a detailed evaluation in terms of F-macro scoring in a per-lemma-WSD-setup.

The paper is structured as follows: first, a short
exploratory analysis is given for the B&B resp. the
TLL data in terms of the original resources and
their construction principles. Afterwards we report
on the finetuning experiments and we summarize
the study with a conclusion section.

Pres definitae:

① specimina pœuo ad illustrandus notiones selecta:

② tolerandi, sustinendi (sc. fortiter sim.)

① in universam: PLAVT. Men. 721 videlicet esse mavelim, quam istaec flagitia tua -i
(779 perpetuli) 378 magni multo -or facilis weba, webera egoque -TEL. Enim
24c, neque ridiculus esse neque plagas -i possunt. PACVV. trag. 279 -or facile
inluriam, si est vacua a contumelia. CIC. Verr. II 3, 95 quem contumelium aculeum -i...
viri boni difficillime possunt 3, 201 si hoc rectigal atroaria tolerare, hoc est Sicilia ferre
ac -i potest. Phil. 6, 10 aliae nationes servitutem -i possunt, populi Romani est
propria libertas. (in. 3, 42 si dolores eosdem tolerabilissi -untur qui excipiant eos pro
patria quam qui leviore de causa (item in philosophic: 4, 3) Paoneuri cum. de dolore
-endo scribeter. Tusc. 4, 60 qui non turbulente humana -antur. sim. al.). BRVT. Cic. ad
Brut. 24, 6 W. servire et -i contumelias... odena. VARBO rust. 2, 10, 3 senes calium
difficillitatem ac montium aruitatem... non facerit, quod -undum est porstibus.
et possim.

② -untur qui quid incolumes, sine noxa sim. sustinet (exempla porstio; cf. p. 725,
3): 09 trist. 1, 3, 7 nec caeium -or nec equis adsubvemum hest. CELS. 2, 18, 3 paces -
qui saltem non -untur. SEN. epi. 51, 10 quamlibet viam iumena -untur, quorum
durata... ungula est. COLVM. 8, 17, 8 nullus raro... vivori claustra -itur. PLIN. nat. 31,
23 fluvii culosum gurgitem periuri negantur -i velut flammas.

③ -untur qui pondera corporea sustinet (propriet et in imagine; cf. e.g. p. 724, 7);
SEN. contr. 3 praef. 9 quidem equi melius equiten -antur, quidem iugum (admis
imagines vol. VII 2, 641, 70 sq; alter p. 722, 4) suas. 2, (1onic) insulta... arma non-
surae manus (STAT. Theb. 11, 551 [Polyynes of ad tribus] executa... membra vides me;
disse a. -i). SEN. Thy. 931 (in imag) pondera regni non inflexa cervice -i (SIL. 16,
90).

④ subheundi, experiendi (sc. mult, quibus quis afficitur negectus respectu fortiter,
laboriosi sim. perpetuli; bona v. sub Bj). CIC. rep. 1, 23 cum de tribus unum est
optandum, aut facere inluriam nec accipere, aut et facere et accipere, aut neutrum,
optimum est facere impune... secundum nec facere nec -i, miserrimum digladiari,
semper tum faciendis tum accependi inluriam. NICID. Gell. v. 12, 6 immentia fraudis,
...

<figure><img src="image_13.png" /><figcaption>Figure 2: Flattened, sense-labeled WSD data for BERT, derived from the TLL article and its sense inventory, for the lemma *patior*.</figcaption></figure>

<footnote n="2">²https://tll.degruyter.com/about</footnote>

<footnote n="3">³The Bavarian Academy of Sciences and Humanities plans to make the complete TLL data open source by 2030.</footnote>

<footnote n="4">⁴Currently headwords are prepared till letter R.</footnote>

<footnote n="5">⁵https://github.com/dbamman/latin-bert</footnote>
nary class labeling) method of B&B, we requested
data for the same lemmas that B&B presented,
from a currently proprietary resource: the *The-
saurus Linguae Latinae*<fnref n="2" /> (TLL)<fnref n="3" />. The TLL is a
comprehensive monolingual Latin dictionary that
aims to record all meanings of all ancient Latin
words, citing all (or a representative sample) of its
seen attestations. The TLL is vast: it is estimated
to comprise cca. 53k-56k entries as of now<fnref n="4" />, so it
likely holds a major part of the quotes that occur in
L&S and thus in the B&B WSD dataset.

The prospect of comparing WSD performance
across datasets constructed from two dictionaries
– one bilingual, another monolingual – was in-
triguing in several scholarly respects, a.o. for gain-
ing quantitative insights into dictionary structuring
practices, or even for attempting to validate sense
structuring in an empirical way. After inspecting
the data, we realized that a direct comparison of
machine learning performance based on data con-
structed from the TLL resp. from L&S would be
methodologically flawed:

1. We made pilot analyses of the quotes across the B&B and TLL sense labeled data, and noted that sense categorization in TLL and L&S draws on very different semantic principles: for one and the same lemma, the subset of quotes labeled with sense I in B&B can be distributed across both sense class I and II in TLL, and/or vice versa.

2. Working with the methodology of B&B of constructing sense-balanced data would not allow unleashing the full potential of the TLL data size. As we chose not to discard quotes (i.e., did not match the amount of quotes in the smaller sense label set), our TLL dataset became orders of magnitude larger and sense-label-wise possibly more aggregative, thus likely coarser-grained.

Our aims in the current contribution were thus:

* Investigating methods and challenges for experimentally validating sense representations and their WSD distinction

* Giving account of joint work between the Humanities and the NLP communities that deliver complementary expertise

* Reusing a pretrained contextual representation model for Latin, released by Bamman and Burns (2020)

* Reproducing the WSD experiment of Bamman and Burns (2020) via the benchmark data, code, and baseline classifier they released<fnref n="5" />

* Repeating the WSD experiment by finetuning Latin BERT on new WSD data that we constructed from the TLL

* Observing sense organization principles and scale across the two datasets

* Improving experimental methodology by providing a detailed evaluation in terms of F-macro scoring in a per-lemma-WSD-setup.

The paper is structured as follows: first, a short
exploratory analysis is given for the B&B resp. the
TLL data in terms of the original resources and
their construction principles. Afterwards we report
on the finetuning experiments and we summarize
the study with a conclusion section.

Pres definitae:

① specimina pœuo ad illustrandus notiones selecta:

② tolerandi, sustendi (sc. fortiter sit).

③ in universam: PLAVT. Men. 721 videlicet esse mavelim, quam istaec flagitia tua -i
(779 perpetuli) 778 magni multo -or facilis weba, webera egoque -TEL. Enim
24c, neque ridiculus esse neque plagas -i possunt. PACVV. trag. 279 -or facile
inluriam, si est vacua a contumelia. CIC. Verr. II 3, 95 quem contumelium aculeum -i...
viri boni difficillime possunt 3, 201 si hoc rectigal atroaria tolerare, hoc est Sicilia ferre
ac -i potest. Phil. 6, 10 aliae nationes servitutem -i possunt, populi Romani est
propria libertas. (in. 3, 42 si dolores eosdem tolerabilis -untur qui excipiet eos pro
patria quam qui leviore de causa (item in philosophic: 4, 3) Paoneuri cum. de dolore
-endo scribeter. Tusc. 4, 60 qui non turbulente humana -antur. sim. al.). BRVT. Cic. ad
Brut. 24, 6 W. servire et -i contumelias -odero. VARBO rust. 2, 10, 3 senes calium
difficillitatem ac montium aruitatem... non facerit, quod -undum est porstibus.
et possim.

④ -untur qui quid incolumes, sine noxa sim. sustinet (exempla porstio; cf. p. 725,
3): 09 trist. 1, 3, 7 nec caeium -or nec equis adsubvemum hest. CELS. 2, 18, 3 paces -
qui saltem non -untur. SEN. epi. 51, 10 quamlibet viam iumena -untur, quorum
durata... ungula est. COLVM. 8, 17, 8 nullus raro... vivori claustra -itur. PLIN. nat. 31,
23 fluxi culosum gurgitem periuri negantur -i velut flammas.

⑤ -untur qui pondera corporea sustinet (propriet et in imagine; cf. e.g. p. 724, 7);
SEN. contr. 3 praef. 9 quidem equi melius equiten -antur, quidem iugum (admis
imagines vol. VII 2, 641, 70 sq; alter p. 722, 4) suas. 2, (Inonic) insulta... arma non-
surae manus (STAT. Theb. 11, 551 [Polyynes of ad tribus] executa... membra vides me;
disse a. -i). SEN. Thy. 931 (in imag) pondera regni non inflexa cervice -i (SIL. 16,
90).

⑥ subheundi, experiendi (sc. mult, quibus quis afficitur negecto respectus foriter,
laboriosi sim. perpetuli; bona v. sub Bj). CIC. rep. 1, 23 cum de tribus unum est
optandum, aut facere inluriam nec accipere, aut et facere et accipere, aut neutrum,
optimum est facere impune... secundum nec facere nec -i, miserrimum digladiari
semper tum faciendis tum accependi inluriam. NICID. Gell. v. 12, 6 immentia fraudis,
...

<figure><img src="image_13.png" /><figcaption>Figure 2: Flattened, sense-labeled WSD data for BERT, derived from the TLL article and its sense inventory, for the lemma *patior*.</figcaption></figure>

<footnote n="2">²https://tll.degruyter.com/about</footnote>

<footnote n="3">³The Bavarian Academy of Sciences and Humanities plans to make the complete TLL data open source by 2030.</footnote>

<footnote n="4">⁴Currently headwords are prepared till letter R.</footnote>

<footnote n="5">⁵https://github.com/dbamman/latin-bert</footnote>
# 2 Exploratory Data Analysis

## 2.1 B&B Data

The B&B dataset comprises 8,354 instances for a total of 201 dictionary headwords (lemmas). The source of B&B data is the bilingual dictionary of L&S<fnref n="6" /> that is a translation of Freund’s dictionary from the 19th century, reflecting edition techniques from 200 years ago.

## 2.2 TLL Data

The ongoing TLL compilation project started in 1900; its editorial principles have changed every once in a while<fnref n="7" />. Within each TLL article, a contrastive, nested (thus: semantically additive) structure is pursued that can descend as deep as 10+ levels.

Sense groups on the same level are aimed to be of the same sense granularity but to feature mutually exclusive parameters of syntactic or semantic nature or their combination. This implies that the TLL structure does not reflect sense distinctions that depend on their translatability to another language, but its goal is a dichotomic arrangement (which is not always limited to two sense groups on the highest level) of word attestations (quotes from Latin texts) while staying within the same language.

The TLL data was available to us in TEI XML format. Just like B&B, we generated the data from within a single dictionary entry, by definition excluding homonymy, and we only considered the first two main senses of a lemma, labeling all text snippets that are longer than 4 words with the corresponding highest-level sense label (see Figure 2), by recursively descending into – thus flattening – the nested structure of the printed article (see Figure 1). Our TLL data points correspond to 25,227 text snippets for the subset of 40 lemmas, whose part-of-speech distributions are: 40% verbs, 22.5% adjectives, 10% nouns, 27.5% others (adverb, pronoun, preposition, conjunction, particle).

Starting from letter C, in the articles a large amount of words – by definition the lemma onset itself, but also other tokens – are heavily and somewhat irregularly abbreviated, which we had to resolve by extensive human-in-the-loop procedures, e.g. by identifying patterns and writing replacement rules for omitted subword material in a per-lemma

<figure><img src="image_14.png" /><figcaption>Figure 3: Data size per lemma per dataset. Blue: B&B data. Orange: TLL data.</figcaption></figure>

fashion. Reconstructing the omitted subword parts of the inflected lemma forms was mandatory for running meaningful WSD experiments because the lemma forms supply a core piece of information to the learning algorithms.

## 2.3 Analysis of the Derived Sense Classes

We observed a number of important phenomena about sense classes as derived from the dictionaries.

1. **Semantically motivated separation between senses** Often, sense separation is mutually exclusive, e.g. the lemma *relinquo* demonstrates that out of the first two main TLL sense groups, I pertains to 'relocation in physical space', as opposed to II that describes 'movement in a figurative sense'.

2. **Artificial dichotomy of senses** The separation of senses can often be rather artificially constructed, e.g. in *ratio*, and L&S uses such separation practices, e.g. by container labels such as "in general" vs. "in particular", even though the latter split oftentimes does not yield a semantically or syntactically homogeneous group.

3. **Lemma vs. Sublemma** Classes can also be split on certain grammatical phenomena in L&S, e.g. on participle perfect used as an adjective (cf. *remitto* where this usage makes up class II for B&B), whereas the TLL renders such usage as a so-called sublemma and treats it structurally elsewhere than in the main article, thus the Latin quotes in it do not get extracted into the TLL WSD data.

4. **Temporal and domain diversity** The TLL has a uniquely wide temporal scope spanning nearly 1000 years from Old and Classical Latin till late antiquity and Christian Latin (cca. AD 700),

<footnote n="6">⁶http://www.perseus.tufts.edu/hopper/</footnote>

<footnote n="7">⁷For an impression see https://publikationen.badw.de/de/thesaurus/lemmata</footnote>
# 2 Exploratory Data Analysis

## 2.1 B&B Data

The B&B dataset comprises 8,354 instances for a total of 201 dictionary headwords (lemmas). The source of B&B data is the bilingual dictionary of L&S<fnref n="6" /> that is a translation of Freund’s dictionary from the 19th century, reflecting edition techniques from 200 years ago.

## 2.2 TLL Data

The ongoing TLL compilation project started in 1900; its editorial principles have changed every once in a while<fnref n="7" />. Within each TLL article, a contrastive, nested (thus: semantically additive) structure is pursued that can descend as deep as 10+ levels.

Sense groups on the same level are aimed to be of the same sense granularity but to feature mutually exclusive parameters of syntactic or semantic nature or their combination. This implies that the TLL structure does not reflect sense distinctions that depend on their translatability to another language, but its goal is a dichotomic arrangement (which is not always limited to two sense groups on the highest level) of word attestations (quotes from Latin texts) while staying within the same language.

The TLL data was available to us in TEI XML format. Just like B&B, we generated the data from within a single dictionary entry, by definition excluding homonymy, and we only considered the first two main senses of a lemma, labeling all text snippets that are longer than 4 words with the corresponding highest-level sense label (see Figure 2), by recursively descending into – thus flattening – the nested structure of the printed article (see Figure 1). Our TLL data points correspond to 25,227 text snippets for the subset of 40 lemmas, whose part-of-speech distributions are: 40% verbs, 22.5% adjectives, 10% nouns, 27.5% others (adverb, pronoun, preposition, conjunction, particle).

Starting from letter C, in the articles a large amount of words – by definition the lemma onset itself, but also other tokens – are heavily and somewhat irregularly abbreviated, which we had to resolve by extensive human-in-the-loop procedures, e.g. by identifying patterns and writing replacement rules for omitted subword material in a per-lemma

<figure><img src="image_14.png" /><figcaption>Figure 3: Data size per lemma per dataset. Blue: B&B data. Orange: TLL data.</figcaption></figure>

fashion. Reconstructing the omitted subword parts of the inflected lemma forms was mandatory for running meaningful WSD experiments because the lemma forms supply a core piece of information to the learning algorithms.

## 2.3 Analysis of the Derived Sense Classes

We observed a number of important phenomena about sense classes as derived from the dictionaries.

1. **Semantically motivated separation between senses** Often, sense separation is mutually exclusive, e.g. the lemma *relinquo* demonstrates that out of the first two main TLL sense groups, I pertains to 'relocation in physical space', as opposed to II that describes 'movement in a figurative sense'.

2. **Artificial dichotomy of senses** The separation of senses can often be rather artificially constructed, e.g. in *ratio*, and L&S uses such separation practices, e.g. by container labels such as "in general" vs. "in particular", even though the latter split oftentimes does not yield a semantically or syntactically homogeneous group.

3. **Lemma vs. Sublemma** Classes can also be split on certain grammatical phenomena in L&S, e.g. on participle perfect used as an adjective (cf. *remitto* where this usage makes up class II for B&B), whereas the TLL renders such usage as a so-called sublemma and treats it structurally elsewhere than in the main article, thus the Latin quotes in it do not get extracted into the TLL WSD data.

4. **Temporal and domain diversity** The TLL has a uniquely wide temporal scope spanning nearly 1000 years from Old and Classical Latin till late antiquity and Christian Latin (cca. AD 700),

<footnote n="6">⁶http://www.perseus.tufts.edu/hopper/</footnote>

<footnote n="7">⁷For an impression see https://publikationen.badw.de/de/thesaurus/lemmata</footnote>
and encompasses genres beyond the domain of literature, such as legal and medical texts and inscriptions. Thereby, it delivers markedly different semantic representation proportions than (a) the pretrained BERT that saw texts spanning cca. 2000 years, seeing attestation from Middle Latin and Humanism, or (b) the B&B finetuned BERT that saw texts from cca. 200 years, focused on a subset of canonical classical authors. As an example: *religio* in the contemporary sense of *religion* as 'a dogmatic system of faith based on revelation' did not exist before the rise of Christianity; for "pagan" Romans, *religio* denoted 'feelings of awe, fear, respect towards the gods or strictly defined forms of (liturgical) worship'.<fnref n="8" />

**5. Truthfulness to sources** In both the B&B and the TLL data, their antique sources are not always literally cited, but the quotes are often edited. The TLL maintains more strictness, e.g. no syntactic changes are allowed. In the B&B (aka L&S) data, one regularly finds modified or artificially inserted constructions that diverge from the sources.

## 3 WSD Experiments

Finetuning BERT is a technique that takes its pretrained language model and explicitly trains it for the WSD task, i.e., in our case on Latin quote – typically on the subsentential level – that are labeled to have class I or class II, as assigned based on the TLL sense inventory. This yields a classification model that can distinguish exactly two meanings for the token that designates the focus lemma. This is certainly a simplified WSD setup, nevertheless helpful for pilot studies to assess the power of newly constructed data for disambiguating between two major senses (or usage contexts) of words. The finetuning task is in contrast with what already took place in the first phase of creating lexical representations, the so-called pretraining. There the task was that BERT’s Latin language model learns as many senses of a word as possible.

### 3.1 Training and Testing Setup

The setup across our WSD experiments on a machine with GPU running Linux Ubuntu 18 is listed below. Splitting the data into partitions for training, development, and testing was done by the method and Github code of Bamman and Burns (2020).

<table><caption>Table 1: Mean performance scores over 40 lemmas.</caption><thead><tr><th>Dataset</th><th>Model</th><th>mean F-macro</th><th>stdev</th></tr></thead><tbody><tr><td rowspan="2">B&B</td><td>biLSTM</td><td>.613</td><td>.205</td></tr><tr><td>BERT</td><td>.695</td><td>.213</td></tr><tr><td rowspan="2">TLL</td><td>biLSTM</td><td>.705</td><td>.132</td></tr><tr><td>BERT</td><td>.794</td><td>.143</td></tr></tbody></table>

* 100 epochs (training rounds) per lemma

* Training and testing performed per lemma

* B&B used cross entropy loss without class weights for training. Since in our data the two classes are imbalanced per lemma, we calculated the weights for each class for the cross entropy loss function

* Performance was evaluated in terms of the unweighted macro F1-score per lemma using Pedregosa et al. (2011). Accuracy would be suboptimal to use as it does not transparently express how well we perform on the two classes and it does not correct for class imbalance

* For each epoch, macro F1 was calculated on the development set

* For each lemma, the best performing development epoch's parameters were used to measure macro F1 on the heldout test set

* As baseline model we used from B&B<fnref n="9" /> 200-dimensional static word2vec embeddings (Mikolov et al., 2013) in a biLSTM classifier

* Enclitica were not separated from words since BERT’s wordpiece tokenizer¹⁰ was assumed to account for these.

### 3.2 Evaluation

**B&B Dataset** We reproduced the B&B WSD study with a similar accuracy score as they report (.737). Next, we derived from the B&B aggregated dataset a per-lemma dataset, on which we trained both classifier models, using the B&B code that we amended with the settings listed in Section 3.1. The results are shown in Table 1. We observe that the B&B per-lemma data are small (cf. Figure 3) and yield statistically unreliable results as standard deviation values are large; this variability is also illustrated by the whiskers of the boxplot (cf. Figure 4). While Table 1 reports the means and the standard deviations, the boxplots show the median.

<footnote n="8">⁸We aim to utilize TLL data for chronological analyses, characterizing and training the recognition of e.g. semantic drift, but this goes beyond the scope of the current paper.</footnote>

<footnote n="9">⁹https://github.com/dbamman/latin-bert/blob/cd6bea9f7ff84ff4b18c172f3d5719d1d3198e69/case_studies/pos_tagging/scripts/download_static vectors.sh
¹⁰https://ai.googleblog.com/2021/12/a-fast-wordpiece-tokenization-system.html</footnote>
and encompasses genres beyond the domain of literature, such as legal and medical texts and inscriptions. Thereby, it delivers markedly different semantic representation proportions than (a) the pretrained BERT that saw texts spanning cca. 2000 years, seeing attestation from Middle Latin and Humanism, or (b) the B&B finetuned BERT that saw texts from cca. 200 years, focused on a subset of canonical classical authors. As an example: *religio* in the contemporary sense of religion as 'a dogmatic system of faith based on revelation' did not exist before the rise of Christianity; for "pagan" Romans, *religio* denoted 'feelings of awe, fear, respect towards the gods or strictly defined forms of (liturgical) worship'.<fnref n="8" />

**5. Truthfulness to sources** In both the B&B and the TLL data, their antique sources are not always literally cited, but the quotes are often edited. The TLL maintains more strictness, e.g. no syntactic changes are allowed. In the B&B (aka L&S) data, one regularly finds modified or artificially inserted constructions that diverge from the sources.

## 3 WSD Experiments

Finetuning BERT is a technique that takes its pretrained language model and explicitly trains it for the WSD task, i.e., in our case on Latin quote – typically on the subsentential level – that are labeled to have class I or class II, as assigned based on the TLL sense inventory. This yields a classification model that can distinguish exactly two meanings for the token that designates the focus lemma. This is certainly a simplified WSD setup, nevertheless helpful for pilot studies to assess the power of newly constructed data for disambiguating between two major senses (or usage contexts) of words. The finetuning task is in contrast with what already took place in the first phase of creating lexical representations, the so-called pretraining. There the task was that BERT’s Latin language model learns as many senses of a word as possible.

### 3.1 Training and Testing Setup

The setup across our WSD experiments on a machine with GPU running Linux Ubuntu 18 is listed below. Splitting the data into partitions for training, development, and testing was done by the method and Github code of Bamman and Burns (2020).

<table><caption>Table 1: Mean performance scores over 40 lemmas.</caption><thead><tr><th>Dataset</th><th>Model</th><th>mean F-macro</th><th>stdev</th></tr></thead><tbody><tr><td rowspan="2">B&B</td><td>biLSTM</td><td>.613</td><td>.205</td></tr><tr><td>BERT</td><td>.695</td><td>.213</td></tr><tr><td rowspan="2">TLL</td><td>biLSTM</td><td>.705</td><td>.132</td></tr><tr><td>BERT</td><td>.794</td><td>.143</td></tr></tbody></table>

* 100 epochs (training rounds) per lemma

* Training and testing performed per lemma

* B&B used cross entropy loss without class weights for training. Since in our data the two classes are imbalanced per lemma, we calculated the weights for each class for the cross entropy loss function

* Performance was evaluated in terms of the unweighted macro F1-score per lemma using Pedregosa et al. (2011). Accuracy would be suboptimal to use as it does not transparently express how well we perform on the two classes and it does not correct for class imbalance

* For each epoch, macro F1 was calculated on the development set

* For each lemma, the best performing development epoch's parameters were used to measure macro F1 on the heldout test set

* As baseline model we used from B&B<fnref n="9" /> 200-dimensional static word2vec embeddings (Mikolov et al., 2013) in a biLSTM classifier

* Enclitica were not separated from words since BERT’s wordpiece tokenizer¹⁰ was assumed to account for these.

### 3.2 Evaluation

**B&B Dataset** We reproduced the B&B WSD study with a similar accuracy score as they report (.737). Next, we derived from the B&B aggregated dataset a per-lemma dataset, on which we trained both classifier models, using the B&B code that we amended with the settings listed in Section 3.1. The results are shown in Table 1. We observe that the B&B per-lemma data are small (cf. Figure 3) and yield statistically unreliable results as standard deviation values are large; this variability is also illustrated by the whiskers of the boxplot (cf. Figure 4). While Table 1 reports the means and the standard deviations, the boxplots show the median.

<footnote n="8">⁸We aim to utilize TLL data for chronological analyses, characterizing and training the recognition of e.g. semantic drift, but this goes beyond the scope of the current paper.</footnote>

<footnote n="9">⁹https://github.com/dbamman/latin-bert/blob/cd6bea9f7ff84ff4b18c172f3d5719d1d3198e69/case_studies/pos_tagging/scripts/download_static vectors.sh
¹⁰https://ai.googleblog.com/2021/12/a-fast-wordpiece-tokenization-system.html</footnote>
<figure><img src="image_15.png" /><figcaption>Figure 4: Performance distribution boxplots: F-macro and accuracy across lemmas per dataset per model.</figcaption></figure>

**TLL Dataset** Due to the data preparation over-
head, thus far we processed a subset of 40 lemmas.
The WSD performance scores on TLL data are also
listed in Table 1: BERT attains a nearly .80 F-score
and outperforms the baseline biLSTM model with
a large margin (for both datasets). Figure 4 also
indicates that the median of the scores for TLL data
is higher than for B&B data.

4 Summary and Conclusion

Our study aimed to confirm the impact of Latin
BERT (Bamman and Burns, 2020) and to point
out an important new Latin WSD resource. We
constructed a large dataset from the TLL that
holds quotes labeled with the first two highest-level
senses of a headword. These likely incorporate
senses that the B&B dataset did not include. We
experimentally validated that the nested dictionary
structure of the TLL is able to deliver WSD data
for finetuning contextual representations in a trans-
former architecture. The WSD models yielded a
large improvement above the static embeddings
baseline, when evaluated on held-out data from
our new, TLL-based dataset. We plan to scale up
this study and to release a benchmark dataset and
trained models for Latin WSD in future work.

References

David Bamman and Patrick J. Burns. 2020. Latin BERT: A Contextual Language Model for Classical Philology. CoRR, abs/2009.10053.

Michele Bevilacqua, Tommaso Pasini, Alessandro Ra-
ganato, Roberto Navigli, et al. 2021. Recent trends
in word sense disambiguation: A survey. In *Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence, IJCAI-21*. Inter-

national Joint Conference on Artificial Intelligence,
Inc.

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. BERT: Pre-training of deep bidirectional transformers for language understanding. *arXiv preprint arXiv:1810.04805*.

Greta Franzini, Andrea Peverelli, Paolo Ruffolo, Marco Passarotti, Helena Sanna, Edoardo Signoroni, Vi-viana Ventura, and Federica Zampedri. 2019. Nunc est aestimandum: Towards an evaluation of the Latin WordNet. In CLiC-it.

Tomas Mikolov, Kai Chen, Greg Corrado, and Jef-
frey Dean. 2013. Efficient estimation of word
representations in vector space. arXiv preprint
arXiv:1301.3781.

Roberto Navigli. 2009. Word Sense Disambiguation: A Survey. *ACM Computing Surveys*, 41(2).

F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. 2011. Scikit-learn: Machine learning in Python. *Journal of Machine Learning Research*, 12:2825–2830.

Bianca Scarlini, Tommaso Pasini, and Roberto Navigli.
2020. SENSEMBERT: Context-enhanced sense em-
beddings for multilingual word sense disambigua-
tion. In *Proceedings of The Thirty-Fourth AAAI
Conference on Artificial Intelligence (AAAI-20)*.
<figure><img src="image_15.png" /><figcaption>Figure 4: Performance distribution boxplots: F-macro and accuracy across lemmas per dataset per model.</figcaption></figure>

**TLL Dataset** Due to the data preparation over-
head, thus far we processed a subset of 40 lemmas.
The WSD performance scores on TLL data are also
listed in Table 1: BERT attains a nearly .80 F-score
and outperforms the baseline biLSTM model with
a large margin (for both datasets). Figure 4 also
indicates that the median of the scores for TLL data
is higher than for B&B data.

# 4 Summary and Conclusion

Our study aimed to confirm the impact of Latin
BERT (Bamman and Burns, 2020) and to point
out an important new Latin WSD resource. We
constructed a large dataset from the TLL that
holds quotes labeled with the first two highest-level
senses of a headword. These likely incorporate
senses that the B&B dataset did not include. We
experimentally validated that the nested dictionary
structure of the TLL is able to deliver WSD data
for finetuning contextual representations in a trans-
former architecture. The WSD models yielded a
large improvement above the static embeddings
baseline, when evaluated on held-out data from
our new, TLL-based dataset. We plan to scale up
this study and to release a benchmark dataset and
trained models for Latin WSD in future work.

References

David Bamman and Patrick J. Burns. 2020. Latin BERT: A Contextual Language Model for Classical Philology. CoRR, abs/2009.10053.

Michele Bevilacqua, Tommaso Pasini, Alessandro Ra-
ganato, Roberto Navigli, et al. 2021. Recent trends
in word sense disambiguation: A survey. In *Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence, IJCAI-21*. Inter-

national Joint Conference on Artificial Intelligence,
Inc.

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. BERT: Pre-training of deep bidirectional transformers for language understanding. *arXiv preprint arXiv:1810.04805*.

Greta Franzini, Andrea Peverelli, Paolo Ruffolo, Marco Passarotti, Helena Sanna, Edoardo Signoroni, Vi-viana Ventura, and Federica Zampedri. 2019. Nunc est aestimandum: Towards an evaluation of the Latin WordNet. In CLiC-it.

Tomas Mikolov, Kai Chen, Greg Corrado, and Jef-
frey Dean. 2013. Efficient estimation of word
representations in vector space. arXiv preprint
arXiv:1301.3781.

Roberto Navigli. 2009. Word Sense Disambiguation: A Survey. *ACM Computing Surveys*, 41(2).

F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. 2011. Scikit-learn: Machine learning in Python. *Journal of Machine Learning Research*, 12:2825–2830.

Bianca Scarlini, Tommaso Pasini, and Roberto Navigli.
2020. SENSEMBERT: Context-enhanced sense em-
beddings for multilingual word sense disambigua-
tion. In *Proceedings of The Thirty-Fourth AAAI
Conference on Artificial Intelligence (AAAI-20)*.
# Putting WordNet's Dictionary Examples in the Context of Definition Modelling: An Empirical Analysis

Fatemah Almeman* Luis Espinosa-Anke*◊

*CardiffNLP, School of Computer Science and Informatics, Cardiff University, UK

◊AMPLYFI, UK

{almemanf, espinosa-ankel}@cardiff.ac.uk

## Abstract

Definition modeling is the task to generate a valid definition for a given input term. This relatively novel task has been approached either with no context (i.e., given a word embedding alone) and, more recently, as word-in-context modeling. Despite their success, most works make little to no distinction between resources and their specific features (e.g., type and style of definitions, or quality of examples) when used for training. Given the high diversity lexicographic resources exhibit in terms of topic coverage, style and formal structure, it is desirable for downstream definition modeling to better understand which of them are better suited for the task. In this paper, we propose an empirical evaluation of the well-known lexical database WordNet, and specifically, its dictionary examples. We evaluate them both directly, by matching them against criteria for good dictionary writing, and indirectly, in the task of definition modeling. Our results suggest that WordNet's dictionary examples could be improved by extending them in length, and incorporating prototypicality.

## 1 Introduction

Definition modeling (DM), as introduced by Noraset et al. (2017), is the task of generating a dictionary definition for a given word. This task was made possible by the adoption in NLP of sequence-to-sequence architectures based on RNNs (Gardner et al., 2022). Recently, DM systems have shown impressive performance in several intrinsic and downstream tasks, mostly thanks to being able to go from context-less (Noraset et al. only used the definition-dum<fnref n="1" /> as a conditioning token at all timesteps) to a contextually richer setting, e.g., by conditioning the generated definition to an example of usage of the target word (Ni and Wang, 2017; Gadetsky

et al., 2018; Chang et al., 2018; Zhu et al., 2019; Mickus et al., 2019; Ishiwatari et al., 2019).

Recently, a notable leap in DM was achieved in Bevilacqua et al. (2020), who fine-tuned BART (Lewis et al., 2019) on example-definition pairs, and reported high results in intrinsic benchmarks and, more importantly, used their DM system for downstream NLP, specifically word sense dis-ambiguation (WSD) and word-in-context classification. DM has also been explored from other perspectives, e.g., generating definitions with appropriate specificity using re-ranking mechanisms (Huang et al., 2021), or extending the generation cross-entropy loss with a reconstruction objective (Kong et al., 2022) (reminiscent of works that used dictionary definitions for improving word embeddings via autoencoders (Bosc and Vincent, 2018) or LSTMs (Hill et al., 2016)). Moreover, Barba et al. (2021) explore a BART-based model for performing the reverse task to DM, i.e., exemplification modeling, or generating a dictionary example given a term and its definition. Other applications of DM range from the aforementioned lexical semantics tasks to reverse dictionary (predict a word given a definition), interpretability, or for clarifying technical and medical terminology (Chen and Zhao, 2022; August et al., 2022), whereas recent applications of BART to tasks not originally designed to be solved generatively are semantic role labeling (Bevilacqua et al., 2020), relation extraction (Cabot and Navigli, 2021) or entity linking (De Cao et al., 2020).

Despite the above successes, little attention has been paid so far to the quality of the dictionary examples (or *contexts*) used for fine-tuning these models. In fact, most existing DM systems train on WordNet (WN) (Miller, 1995), which is the de-facto lexical database for English. However, we are not aware of previous work that has explored the quality (and hence, suitability for DM) of WN examples. Therefore, in this paper, we first inves-

<footnote n="1">¹The genus-et-differentia Aristotelian definitions follows an A is a B which Z structure, with A being the definiendum, B the genus and Z the definiens or differentia specifica.</footnote>
# Putting WordNet's Dictionary Examples in the Context of Definition Modelling: An Empirical Analysis

Fatemah Almeman* Luis Espinosa-Anke*◊

*CardiffNLP, School of Computer Science and Informatics, Cardiff University, UK

◊AMPLYFI, UK

{almemanf, espinosa-ankel}@cardiff.ac.uk

## Abstract

Definition modeling is the task to generate a valid definition for a given input term. This relatively novel task has been approached either with no context (i.e., given a word embedding alone) and, more recently, as word-in-context modeling. Despite their success, most works make little to no distinction between resources and their specific features (e.g., type and style of definitions, or quality of examples) when used for training. Given the high diversity lexicographic resources exhibit in terms of topic coverage, style and formal structure, it is desirable for downstream definition modeling to better understand which of them are better suited for the task. In this paper, we propose an empirical evaluation of the well-known lexical database WordNet, and specifically, its dictionary examples. We evaluate them both directly, by matching them against criteria for good dictionary writing, and indirectly, in the task of definition modeling. Our results suggest that WordNet's dictionary examples could be improved by extending them in length, and incorporating prototypicality.

## 1 Introduction

Definition modeling (DM), as introduced by Noraset et al. (2017), is the task of generating a dictionary definition for a given word. This task was made possible by the adoption in NLP of sequence-to-sequence architectures based on RNNs (Gardner et al., 2022). Recently, DM systems have shown impressive performance in several intrinsic and downstream tasks, mostly thanks to being able to go from context-less (Noraset et al. only used the definition-dum<fnref n="1" /> as a conditioning token at all timesteps) to a contextually richer setting, e.g., by conditioning the generated definition to an example of usage of the target word (Ni and Wang, 2017; Gadetsky

et al., 2018; Chang et al., 2018; Zhu et al., 2019; Mickus et al., 2019; Ishiwatari et al., 2019).

Recently, a notable leap in DM was achieved in Bevilacqua et al. (2020), who fine-tuned BART (Lewis et al., 2019) on example-definition pairs, and reported high results in intrinsic benchmarks and, more importantly, used their DM system for downstream NLP, specifically word sense dis-ambiguation (WSD) and word-in-context classification. DM has also been explored from other perspectives, e.g., generating definitions with appropriate specificity using re-ranking mechanisms (Huang et al., 2021), or extending the generation cross-entropy loss with a reconstruction objective (Kong et al., 2022) (reminiscent of works that used dictionary definitions for improving word embeddings via autoencoders (Bosc and Vincent, 2018) or LSTMs (Hill et al., 2016)). Moreover, Barba et al. (2021) explore a BART-based model for performing the reverse task to DM, i.e., exemplification modeling, or generating a dictionary example given a term and its definition. Other applications of DM range from the aforementioned lexical semantics tasks to reverse dictionary (predict a word given a definition), interpretability, or for clarifying technical and medical terminology (Chen and Zhao, 2022; August et al., 2022), whereas recent applications of BART to tasks not originally designed to be solved generatively are semantic role labeling (Bevilacqua et al., 2020), relation extraction (Cabot and Navigli, 2021) or entity linking (De Cao et al., 2020).

Despite the above successes, little attention has been paid so far to the quality of the dictionary examples (or *contexts*) used for fine-tuning these models. In fact, most existing DM systems train on WordNet (WN) (Miller, 1995), which is the de-facto lexical database for English. However, we are not aware of previous work that has explored the quality (and hence, suitability for DM) of WN examples. Therefore, in this paper, we first inves-

<footnote n="1">¹The genus-et-differentia Aristotelian definitions follows an A is a B which Z structure, with A being the definiendum, B the genus and Z the definiens or differentia specifica.</footnote>
tigate the quality of WN examples by evaluating against the GDEX (Good Dictionary Examples) set of criteria (Kilgarriff et al., 2008), and use as a point of comparison a widely adopted open dataset used in DM, which is primarily based on the Oxford Dictionary (Chang and Chen, 2019) (CHA). It is worth noting, however, that these two resources were built for different objectives, as the initial purpose behind creating WN was to explain how lexical meaning is stored in the mind (Broda et al., 2009), and its primary use may be as a sense inventory (Agirre and Edmonds, 2007). However, with this caveat in mind, and given how lexicographic resources are currently converging into useful pre-training and fine-tuning datasets for lexical semantics, we also propose to extrinsically test these two resources in the DM task. Specifically, in our second set of experiments we fine-tune a BART-based model on WN and CHA, and show that generally speaking, results of models fine-tuned on WN perform slightly worse than if fine-tuned on CHA. Our preliminary results suggest that WN's examples sometimes do not provide enough context, making it difficult to learn a good representation for the word being contextualized. We also report an experiment comparing DM modeling results on WN nouns vs. WN verbs; which suggests that a DM model trained on WN nouns performs slightly better.

## 2 Data

WordNet (WN) is an electronic lexical dictionary for English that describes words (11,7097 nouns, 11,488 verbs, 22,141 adjectives, and 4,601 adverbs) organized in groups of synonyms called “synsets” (Miller, 1995; Fellbaum, 2013). Each synset is described by its definition, lemmas, examples of usage (for some but not all words), and the relations between synsets, e.g., hypernymy (is-a), meronymy (is-part) or troponymy (manner-of). WN has typically been used in lexicographic and language learning settings (Morato et al., 2004), but more importantly, also in NLP, e.g., as a natural language interface for optimizing the precision of search engines, WSD or query expansion (Moldovan and Mihalcea, 2000; Banerjee and Pedersen, 2002). Moreover, relations in WN have been used extensively, for example for improving word embeddings via retrofitting (Faruqui et al., 2014; Espinosa-Anke et al., 2016; Vulić and Mrkšić, 2017; Mrkšić et al., 2017).

CHA (Chang and Chen, 2019), the other resource we consider in this paper, is based on Oxford Dictionaries. It was released with two splits, namely seen, where definitions in the training set also exist in the test set, and unseen, which contains a set of words not available in the training set (Bevilacqua et al., 2020). This is similar to the lexical splits (as opposed to random splits) present in other analogous tasks such as graded lexical entailment (Shwartz et al., 2016; Vulić et al., 2017). In this paper, we are concerned with the quality of examples in WN (and how they compare with CHA), i.e., sentences where a target word appears, and which should be informative enough to convey the necessary contextual information to clarify fully or partially the word's meaning (encoded in a natural language definition or gloss, instead of e.g., a word embedding).

We show in Table 1 examples from WN and CHA, where it becomes apparent that WN examples have a different pattern, e.g., they are much shorter, and are crucially limited in the contextual information they provide, as opposed to the examples in CHA, which features, first, full-fledged grammatical examples, and second, associated vocabularies that help position the target word in the mental lexicon, which is crucial for word access (Zock et al., 2010).

<table><caption>Table 1: WN vs CHA definitions and examples for a given lemma (in bold).</caption><thead><tr><th>Data</th><th>Lemma</th><th>Definition</th><th>Example</th></tr></thead><tbody><tr><td>WN</td><td><strong>people</strong></td><td>(plural) any group of human beings (men or women or children) collectively</td><td>old people</td></tr><tr><td>CHA</td><td><strong>people</strong></td><td>human beings in general or considered collectively</td><td>each day he has looked at a key issue facing us as a nation as a people as frail human beings</td></tr><tr><td>WN</td><td><strong>sheet</strong></td><td>any broad thin expanse or surface</td><td>a sheet of ice</td></tr><tr><td>CHA</td><td><strong>sheet</strong></td><td>a large rectangular piece of cotton or other fabric used on a bed to cover the mattress and as a layer beneath blankets when these are used</td><td>Mary quietly got off the bed and covered him with the sheet and blanket</td></tr><tr><td>WN</td><td><strong>tall</strong></td><td>great in vertical dimension; high in stature</td><td>tall people</td></tr><tr><td>CHA</td><td><strong>tall</strong></td><td>of great or more than average height especially with reference to an object relative to width</td><td>the elevator came to a stop and the doors slid open revealing the sixth floor of the tall building</td></tr></tbody></table>
tigate the quality of WN examples by evaluating against the GDEX (Good Dictionary Examples) set of criteria (Kilgarriff et al., 2008), and use as a point of comparison a widely adopted open dataset used in DM, which is primarily based on the Oxford Dictionary (Chang and Chen, 2019) (CHA). It is worth noting, however, that these two resources were built for different objectives, as the initial purpose behind creating WN was to explain how lexical meaning is stored in the mind (Broda et al., 2009), and its primary use may be as a sense inventory (Agirre and Edmonds, 2007). However, with this caveat in mind, and given how lexicographic resources are currently converging into useful pre-training and fine-tuning datasets for lexical semantics, we also propose to extrinsically test these two resources in the DM task. Specifically, in our second set of experiments we fine-tune a BART-based model on WN and CHA, and show that generally speaking, results of models fine-tuned on WN perform slightly worse than if fine-tuned on CHA. Our preliminary results suggest that WN's examples sometimes do not provide enough context, making it difficult to learn a good representation for the word being contextualized. We also report an experiment comparing DM modeling results on WN nouns vs. WN verbs; which suggests that a DM model trained on WN nouns performs slightly better.

## 2 Data

WordNet (WN) is an electronic lexical dictionary for English that describes words (11,7097 nouns, 11,488 verbs, 22,141 adjectives, and 4,601 adverbs) organized in groups of synonyms called “synsets” (Miller, 1995; Fellbaum, 2013). Each synset is described by its definition, lemmas, examples of usage (for some but not all words), and the relations between synsets, e.g., hypernymy (is-a), meronymy (is-part) or troponymy (manner-of). WN has typically been used in lexicographic and language learning settings (Morato et al., 2004), but more importantly, also in NLP, e.g., as a natural language interface for optimizing the precision of search engines, WSD or query expansion (Moldovan and Mihalcea, 2000; Banerjee and Pedersen, 2002). Moreover, relations in WN have been used extensively, for example for improving word embeddings via retrofitting (Faruqui et al., 2014; Espinosa-Anke et al., 2016; Vulić and Mrkšić, 2017; Mrkšić et al., 2017).

CHA (Chang and Chen, 2019), the other resource we consider in this paper, is based on Oxford Dictionaries. It was released with two splits, namely seen, where definitions in the training set also exist in the test set, and unseen, which contains a set of words not available in the training set (Bevilacqua et al., 2020). This is similar to the lexical splits (as opposed to random splits) present in other analogous tasks such as graded lexical entailment (Shwartz et al., 2016; Vulić et al., 2017). In this paper, we are concerned with the quality of examples in WN (and how they compare with CHA), i.e., sentences where a target word appears, and which should be informative enough to convey the necessary contextual information to clarify fully or partially the word's meaning (encoded in a natural language definition or gloss, instead of e.g., a word embedding).

We show in Table 1 examples from WN and CHA, where it becomes apparent that WN examples have a different pattern, e.g., they are much shorter, and are crucially limited in the contextual information they provide, as opposed to the examples in CHA, which features, first, full-fledged grammatical examples, and second, associated vocabularies that help position the target word in the mental lexicon, which is crucial for word access (Zock et al., 2010).

<table><caption>Table 1: WN vs CHA definitions and examples for a given lemma (in bold).</caption><thead><tr><th>Data</th><th>Lemma</th><th>Definition</th><th>Example</th></tr></thead><tbody><tr><td>WN</td><td><strong>people</strong></td><td>(plural) any group of human beings (men or women or children) collectively</td><td>old people</td></tr><tr><td>CHA</td><td><strong>people</strong></td><td>human beings in general or considered collectively</td><td>each day he has looked at a key issue facing us as a nation as a people as frail human beings</td></tr><tr><td>WN</td><td><strong>sheet</strong></td><td>any broad thin expanse or surface</td><td>a sheet of ice</td></tr><tr><td>CHA</td><td><strong>sheet</strong></td><td>a large rectangular piece of cotton or other fabric used on a bed to cover the mattress and as a layer beneath blankets when these are used</td><td>Mary quietly got off the bed and covered him with the sheet and blanket</td></tr><tr><td>WN</td><td><strong>tall</strong></td><td>great in vertical dimension; high in stature</td><td>tall people</td></tr><tr><td>CHA</td><td><strong>tall</strong></td><td>of great or more than average height especially with reference to an object relative to width</td><td>the elevator came to a stop and the doors slid open revealing the sixth floor of the tall building</td></tr></tbody></table>
# 3 Experiments

In this section, we introduce the two sets of experiments we perform. First, the descriptive comparison between WN and CHA examples using GDEX as a proxy (Section 3.1). Second, we describe the setting for the DM experiment, where we test WN as supervision signal (Section 3.2).

## 3.1 GDEX-based comparison

As a proxy for determining the quality of dictionary examples in WN, and given that there is no manually annotated dataset for this purpose, we used GDEX (Good Dictionary Examples) criteria. GDEX is a system that added around 8,000 new example sentences to Macmillan English Dictionary by automatically finding good examples in corpora using a set of rules of thumb (Kilgarriff et al., 2008; Bejoint, 2014).

In our work, we used some of the features that are introduced in GDEX, specifically:

* **sentence length:** according to Kilgarriff et al. (2008), good dictionary examples should range between 10 and 25 words, and thus we penalize shorter or longer dictionary examples proportionally (the more an example deviates from the acceptable minimum or maximum, the more it is penalized).

* **word frequency:** a sentence is penalized for each non-frequent word that is not in the list of the top 20,000 most frequent words in English Wikipedia.

* **anaphoric references:** we penalize the number of pronouns in the dictionary example, normalized by sentence length.

* **sentence probability:** we use the GPT-2 (Radford et al., 2019) language model to score the probability of dictionary examples. Intuitively, this can be a useful metric for semantic coherence and fluency.

## 3.2 Definition Modeling

The general formulation of DM is as follows. To generate a gloss g that defines a target lemma t in a context c, the standard sequence-to-sequence conditional generation probability is computed by factorising it auto-regressively (Bevilacqua et al., 2020):

$$P(g|c, t) = \prod_{k=1}^{|g|} P(g_k|g_{0:k-1}, c, t) \quad (1)$$

where $g_k$ is the $k$th token of g and $g_0$ is a special start token (Bevilacqua et al., 2020). We fine-tune BART, a pre-trained encoder-decoder system, to perform the definition generation task by taking the pair (context, target lemma) as an input to produce the corresponding definition. The dataset includes (c, t, g) triples where t is the target word (lemma) in a context c (example) and g is the gold gloss which defines t in c (definition). We encode the input as (t, c) pairs and special tokens are used to identify the target lemma in each context such as *The cherry tree <target> bloomed <target>*., with the lemma “bloom” as the target word in this context.

**Exp. 1 (WN vs CHA)** Since we are concerned with using WN in definition modeling, we trained and tested the definition generation model (BART) on WN lemmas that have examples (44,351 lemmas) using an 80/20 split for training and testing. Additionally, we trained the same model using a CHA-derived training set of the same size as our WN training set, and tested it on the same WN test set. We ensured that no duplicates/leakage occured between sets in both experiments. We train both models with a maximum of 50 epochs with early stopping<fnref n="2" />.

**Exp. 2 (WN Nouns vs WN Verbs)** We trained and tested the same BART model with same hyperparameters as in the WN vs CHA experiment on random 10k noun lemmas and 10k verb lemmas from WN separately (using again an 80/20 ratio for training and testing) to evaluate whether there are noticeable differences between these two grammatical categories.

# 4 Analysis

In this section, we discuss the results of our two experiments, namely GDEX-wise comparison between WN and CHA, and WN's stress test in the DM task.

<footnote n="2">²We implemented our experiments using the simpletransformers (http://simpletransformers.ai/) library, a wrapper on top of transformers (Wolf et al., 2020).</footnote>
# 3 Experiments

In this section, we introduce the two sets of experiments we perform. First, the descriptive comparison between WN and CHA examples using GDEX as a proxy (Section 3.1). Second, we describe the setting for the DM experiment, where we test WN as supervision signal (Section 3.2).

## 3.1 GDEX-based comparison

As a proxy for determining the quality of dictionary examples in WN, and given that there is no manually annotated dataset for this purpose, we used GDEX (Good Dictionary Examples) criteria. GDEX is a system that added around 8,000 new example sentences to Macmillan English Dictionary by automatically finding good examples in corpora using a set of rules of thumb (Kilgarriff et al., 2008; Bejoint, 2014).

In our work, we used some of the features that are introduced in GDEX, specifically:

* **sentence length:** according to Kilgarriff et al. (2008), good dictionary examples should range between 10 and 25 words, and thus we penalize shorter or longer dictionary examples proportionally (the more an example deviates from the acceptable minimum or maximum, the more it is penalized).

* **word frequency:** a sentence is penalized for each non-frequent word that is not in the list of the top 20,000 most frequent words in English Wikipedia.

* **anaphoric references:** we penalize the number of pronouns in the dictionary example, normalized by sentence length.

* **sentence probability:** we use the GPT-2 (Radford et al., 2019) language model to score the probability of dictionary examples. Intuitively, this can be a useful metric for semantic coherence and fluency.

## 3.2 Definition Modeling

The general formulation of DM is as follows. To generate a gloss *g* that defines a target lemma *t* in a context *c*, the standard sequence-to-sequence conditional generation probability is computed by factorising it auto-regressively (Bevilacqua et al., 2020):

$$P(g|c, t) = \prod_{k=1}^{|g|} P(g_k|g_{0:k-1}, c, t) \quad (1)$$

where $g_k$ is the $k^{th}$ token of *g* and $g_0$ is a special start token (Bevilacqua et al., 2020). We fine-tune BART, a pre-trained encoder-decoder system, to perform the definition generation task by taking the pair (context, target lemma) as an input to produce the corresponding definition. The dataset includes $(c, t, g)$ triples where *t* is the target word (lemma) in a context *c* (example) and *g* is the gold gloss which defines *t* in *c* (definition). We encode the input as $(t, c)$ pairs and special tokens are used to identify the target lemma in each context such as *The cherry tree <target> bloomed <target>*, with the lemma “bloom” as the target word in this context.

**Exp. 1 (WN vs CHA)** Since we are concerned with using WN in definition modeling, we trained and tested the definition generation model (BART) on WN lemmas that have examples (44,351 lemmas) using an 80/20 split for training and testing. Additionally, we trained the same model using a CHA-derived training set of the same size as our WN training set, and tested it on the same WN test set. We ensured that no duplicates/leakage occured between sets in both experiments. We train both models with a maximum of 50 epochs with early stopping<fnref n="2" />.

**Exp. 2 (WN Nouns vs WN Verbs)** We trained and tested the same BART model with same hyper-parameters as in the WN vs CHA experiment on random 10k noun lemmas and 10k verb lemmas from WN separately (using again an 80/20 ratio for training and testing) to evaluate whether there are noticeable differences between these two grammatical categories.

# 4 Analysis

In this section, we discuss the results of our two experiments, namely GDEX-wise comparison between WN and CHA, and WN's stress test in the DM task.

<footnote n="2">²We implemented our experiments using the simpletransformers (http://simpletransformers.ai/) library, a wrapper on top of transformers (Wolf et al., 2020).</footnote>
<figure><img src="image_16.png" /><figcaption>Figure 1: Empirical distribution functions between WN (blue) and CHA (orange) for length (a) and frequency (b) penalties.</figcaption></figure>

<figure><img src="image_17.png" /><figcaption>Figure 2: Violin plot showing the difference in log-likelihood assigned by GPT-2 to WN vs CHA examples (higher is better).</figcaption></figure>

## 4.1 GDEX score

Since Kilgarriff et al. did not specify an optimal weighting for the different factors they took into account in the GDEX metric, we look individually at each of the four factors discussed in Section 3.1. We leave for future work investigating optimal weighting for these and other metrics, for example, by tuning them on downstream applications. When comparing these scores for both WN and CHA examples, Figure 1 (lower is better in both metrics) shows that WN has generally higher penalties both for example length and for usage of infrequent words. Specifically, for instance, we found that 80% of CHA's examples have a length penalty of .6 or less, whereas for the same proportion, the length penalty reaches more than .8 in WN. In a subsequent analysis, we found that these differences, if studied between WN's nouns and verbs, clearly favour nouns, that is, WN's nouns are in general accompanied by better examples. Specifically, we found that, on average, the length penalty is .49 for nouns, and .62 for verbs, and that

the frequency penalty is .10 for nouns and .15 for verbs.

Finally, while the **sentence probability** is a valid metric, we observe that it is more likely that shorter sentences exhibit lower perplexity, and therefore will be scored higher by a language model. To further investigate this, we conduct an analysis where we split WN's and CHA's examples into 4 bins, namely *short*, *mid-short*, *mid-long* and *long*, with *short* examples containing between 1 and 15 tokens, *mid-short* up to 30 tokens, *mid-long* up to 45 tokens, and *long* above 45 tokens. Then, we compared the probability assigned by the language model to these examples, and verified that, indeed, WN has better short and mid-short examples, but worse mid-long examples. It also important to note that among the *long* examples, most of them were close to 45 tokens for WN, while for CHA they are much longer. To (perhaps anecdotally) illustrate this point, the longest dictionary example in WN is only 46 tokens long, while the longest in CHA is 141. Finally, in terms of usage of anaphoric references, we did not find significantly different results between WN and CHA.

## 4.2 Definition Modeling

Evaluating the quality of the generated definitions is a subjective matter, as delivering the meaning of words can take many forms. Table 2 shows examples of the predicted definitions generated by a WN-trained model and a CHA-trained model. When analysing these definitions and annotating the error types (following the typification proposed in Noraset et al. (2017)), it seems that the predicted definitions generated by the WN-trained model show evidence of under-specificity (first and second rows), since in each case the definition represents the general idea, but where part of the meaning of the target lemma in context is lost. In the third row, the generated definition falls into the self-reference type of error, since it refers to the same lemma in a circular way.

We also noticed that, generally speaking, the CHA-trained model learned to explicitly mention the prototypical concept or the idea to which a definition applies, and this is interesting from a commonsense learning point of view, which has recently received considerable attention (Gajbhiye et al., 2022; Nguyen et al., 2022). Therefore, given that CHA has many definitions that start with the prototypical concept/entity that embodies that prop-
<figure><img src="image_16.png" /><figcaption>Figure 1: Empirical distribution functions between WN (blue) and CHA (orange) for length (a) and frequency (b) penalties.</figcaption></figure>

<figure><img src="image_17.png" /><figcaption>Figure 2: Violin plot showing the difference in log-likelihood assigned by GPT-2 to WN vs CHA examples (higher is better).</figcaption></figure>

## 4.1 GDEX score

Since Kilgarriff et al. did not specify an optimal weighting for the different factors they took into account in the GDEX metric, we look individually at each of the four factors discussed in Section 3.1. We leave for future work investigating optimal weighting for these and other metrics, for example, by tuning them on downstream applications. When comparing these scores for both WN and CHA examples, Figure 1 (lower is better in both metrics) shows that WN has generally higher penalties both for example length and for usage of infrequent words. Specifically, for instance, we found that 80% of CHA's examples have a length penalty of .6 or less, whereas for the same proportion, the length penalty reaches more than .8 in WN. In a subsequent analysis, we found that these differences, if studied between WN's nouns and verbs, clearly favour nouns, that is, WN's nouns are in general accompanied by better examples. Specifically, we found that, on average, the length penalty is .49 for nouns, and .62 for verbs, and that

the frequency penalty is .10 for nouns and .15 for verbs.

Finally, while the **sentence probability** is a valid metric, we observe that it is more likely that shorter sentences exhibit lower perplexity, and therefore will be scored higher by a language model. To further investigate this, we conduct an analysis where we split WN's and CHA's examples into 4 bins, namely *short*, *mid-short*, *mid-long* and *long*, with *short* examples containing between 1 and 15 tokens, *mid-short* up to 30 tokens, *mid-long* up to 45 tokens, and *long* above 45 tokens. Then, we compared the probability assigned by the language model to these examples, and verified that, indeed, WN has better short and mid-short examples, but worse mid-long examples. It also important to note that among the *long* examples, most of them were close to 45 tokens for WN, while for CHA they are much longer. To (perhaps anecdotally) illustrate this point, the longest dictionary example in WN is only 46 tokens long, while the longest in CHA is 141. Finally, in terms of usage of anaphoric references, we did not find significantly different results between WN and CHA.

## 4.2 Definition Modeling

Evaluating the quality of the generated definitions is a subjective matter, as delivering the meaning of words can take many forms. Table 2 shows examples of the predicted definitions generated by a WN-trained model and a CHA-trained model. When analysing these definitions and annotating the error types (following the typification proposed in Noraset et al. (2017)), it seems that the predicted definitions generated by the WN-trained model show evidence of under-specificity (first and second rows), since in each case the definition represents the general idea, but where part of the meaning of the target lemma in context is lost. In the third row, the generated definition falls into the self-reference type of error, since it refers to the same lemma in a circular way.

We also noticed that, generally speaking, the CHA-trained model learned to explicitly mention the prototypical concept or the idea to which a definition applies, and this is interesting from a commonsense learning point of view, which has recently received considerable attention (Gajbhiye et al., 2022; Nguyen et al., 2022). Therefore, given that CHA has many definitions that start with the prototypical concept/entity that embodies that prop-
<table><caption>Table 2: Sample of predicted definitions generated by WN-trained model and CHA-trained model. PD_WN: predicted definition by WN-trained model, PD_CHA: predicted definition by CHA-trained model</caption><thead><tr><th>No. Lemma</th><th>Example</th><th>Gold definition</th><th>PD_WN</th><th>PD_CHA</th></tr></thead><tbody><tr><td>(1) accelerate</td><td>The car accelerated</td><td>move faster</td><td>become more powerful or efficient</td><td>of a vehicle or aircraft move forward at a high rate of speed</td></tr><tr><td>(2) appear</td><td>Did your latest book appear yet?</td><td>be issued or published</td><td>have a physical form or appearance</td><td>of a book or other product reach the shelves of a bookstore or other store</td></tr><tr><td>(3) immigrate</td><td>Many people immigrated at the beginning of the 20th century</td><td>come into a new country and change residency</td><td>become immigratory</td><td>of a person move to a foreign country to settle permanently</td></tr></tbody></table>

<table><caption>Table 3: DM evaluation results for WN and CHA</caption><thead><tr><th></th><th>WN</th><th>CHA</th></tr></thead><tbody><tr><td>BLEU</td><td>0.18</td><td>00.16</td></tr><tr><td>METEOR</td><td>12.28</td><td>14.89</td></tr><tr><td>ROUGE-L</td><td>16.49</td><td>17.37</td></tr></tbody></table>

<table><caption>Table 4: DM evaluation results for WN Nouns vs WN Verbs</caption><thead><tr><th></th><th>Nouns</th><th>Verbs</th></tr></thead><tbody><tr><td>BLEU</td><td>3.67</td><td>0.47</td></tr><tr><td>METEOR</td><td>20.66</td><td>14.13</td></tr><tr><td>ROUGE-L</td><td>26.85</td><td>18.72</td></tr><tr><td>Average</td><td>17.06</td><td>11.12</td></tr></tbody></table>

erty (e.g., “accelerate” having a definition starting with “of a vehicle”), for the future, this resource could be helpful to map prototypical features to concepts, using dictionary examples as additional contexts.

We evaluated the definitions intrinsically using automatic string matching measures, specifically BLEU, ROUGE-L and METEOR. BLEU is a metric used for machine translation evaluation and compares n-grams matches of the candidate sentence with the reference sentence (Papineni et al., 2002) (we used the default BLEU-4). Rouge-L measures the longest common sub-sequence between the candidate sentence with the reference sentence (Lin, 2004). METEOR is another improved machine translation evaluation metric that matches uni-grams based on their surface forms, stemmed forms, and meanings (Lavie and Agarwal, 2007).

**Exp. 1 (WN vs CHA)** Table 3 shows the average BLEU, METEOR and ROUGE-L scores for the definitions generated by WN-trained model and CHA-trained model. The results show that the over-

all scores for evaluating the definition generation model that uses WN examples are low in general, even when comparing it with the model that uses CHA examples for training.

**Exp. 2 (WN Nouns vs WN Verbs)** Finally, with regards to the WN nouns vs WN verbs experiment, Table 4 shows the results of the three metrics used for evaluating the generated definitions. When comparing these results and the average of the scores, we can see that the quality of generated definitions of nouns is generally better than that of verbs. We leave for future work to further explore the differences between WN’s noun vs verb examples, and why nouns seem to be easier to learn.

## 5 Conclusion

Definition modeling is the task to generate a dictionary definition given an input word and, optionally, some context. While different lexicographic resources are used as supervision for DM systems, there is little work analyzing their intrinsic quality. Our evaluation is focused on the examples available in WordNet and the Oxford Dictionary, where we train a sequence-to-sequence definition modeling architecture based on BART using these two dictionaries. We found that WN’s dictionary examples are written in a style that may make them hard to learn (especially verbs), and that they are, generally, (perhaps too) short. For the future, we would like to explore extrinsic evaluations and perform additional experiments with other datasets and language models.

## 6 Acknowledgements

We thank the anonymous reviewers for their insightful comments. The experiments were executed using the computational facilities of the Advanced Research Computing @Cardiff (ARCCA) at Cardiff University.
<table><caption>Table 2: Sample of predicted definitions generated by WN-trained model and CHA-trained model. PD_WN: predicted definition by WN-trained model, PD_CHA: predicted definition by CHA-trained model</caption><thead><tr><th>No. Lemma</th><th>Example</th><th>Gold definition</th><th>PD_WN</th><th>PD_CHA</th></tr></thead><tbody><tr><td>(1) accelerate</td><td>The car accelerated</td><td>move faster</td><td>become more powerful or efficient</td><td>of a vehicle or aircraft move forward at a high rate of speed</td></tr><tr><td>(2) appear</td><td>Did your latest book appear yet?</td><td>be issued or published</td><td>have a physical form or appearance</td><td>of a book or other product reach the shelves of a bookstore or other store</td></tr><tr><td>(3) immigrate</td><td>Many people immigrated at the beginning of the 20th century</td><td>come into a new country and change residency</td><td>become immigratory</td><td>of a person move to a foreign country to settle permanently</td></tr></tbody></table>

<table><caption>Table 3: DM evaluation results for WN and CHA</caption><thead><tr><th></th><th>WN</th><th>CHA</th></tr></thead><tbody><tr><td>BLEU</td><td>0.18</td><td>00.16</td></tr><tr><td>METEOR</td><td>12.28</td><td>14.89</td></tr><tr><td>ROUGE-L</td><td>16.49</td><td>17.37</td></tr></tbody></table>

<table><caption>Table 4: DM evaluation results for WN Nouns vs WN Verbs</caption><thead><tr><th></th><th>Nouns</th><th>Verbs</th></tr></thead><tbody><tr><td>BLEU</td><td>3.67</td><td>0.47</td></tr><tr><td>METEOR</td><td>20.66</td><td>14.13</td></tr><tr><td>ROUGE-L</td><td>26.85</td><td>18.72</td></tr><tr><td>Average</td><td>17.06</td><td>11.12</td></tr></tbody></table>

erty (e.g., “accelerate” having a definition starting with “of a vehicle”), for the future, this resource could be helpful to map prototypical features to concepts, using dictionary examples as additional contexts.

We evaluated the definitions intrinsically using automatic string matching measures, specifically BLEU, ROUGE-L and METEOR. BLEU is a metric used for machine translation evaluation and compares n-grams matches of the candidate sentence with the reference sentence (Papineni et al., 2002) (we used the default BLEU-4). Rouge-L measures the longest common sub-sequence between the candidate sentence with the reference sentence (Lin, 2004). METEOR is another improved machine translation evaluation metric that matches uni-grams based on their surface forms, stemmed forms, and meanings (Lavie and Agarwal, 2007).

**Exp. 1 (WN vs CHA)** Table 3 shows the average BLEU, METEOR and ROUGE-L scores for the definitions generated by WN-trained model and CHA-trained model. The results show that the over-

all scores for evaluating the definition generation model that uses WN examples are low in general, even when comparing it with the model that uses CHA examples for training.

**Exp. 2 (WN Nouns vs WN Verbs)** Finally, with regards to the WN nouns vs WN verbs experiment, Table 4 shows the results of the three metrics used for evaluating the generated definitions. When comparing these results and the average of the scores, we can see that the quality of generated definitions of nouns is generally better than that of verbs. We leave for future work to further explore the differences between WN’s noun vs verb examples, and why nouns seem to be easier to learn.

## 5 Conclusion

Definition modeling is the task to generate a dictionary definition given an input word and, optionally, some context. While different lexicographic resources are used as supervision for DM systems, there is little work analyzing their intrinsic quality. Our evaluation is focused on the examples available in WordNet and the Oxford Dictionary, where we train a sequence-to-sequence definition modeling architecture based on BART using these two dictionaries. We found that WN’s dictionary examples are written in a style that may make them hard to learn (especially verbs), and that they are, generally, (perhaps too) short. For the future, we would like to explore extrinsic evaluations and perform additional experiments with other datasets and language models.

## 6 Acknowledgements

We thank the anonymous reviewers for their insightful comments. The experiments were executed using the computational facilities of the Advanced Research Computing @Cardiff (ARCCA) at Cardiff University.
## References

Eneko Agirre and Philip Edmonds. 2007. *Word sense disambiguation: Algorithms and applications*, volume 33. Springer Science & Business Media.

Tal August, Catharina Reinecke, and Noah Smith. 2022. generating scientific definitions with controllable complexity.

Satanjeev Banerjee and Ted Pedersen. 2002. An adapted lesk algorithm for word sense disambiguation using wordnet. volume 2276, pages 136–145.

Edoardo Barba, Luigi Procopio, Caterina Lacerra, Tommaso Pasini, and Roberto Navigli. 2021. Exemplification modeling: Can you give me an example, please? In *Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence, IJCAI-21*, pages 3779–3785. International Joint Conferences on Artificial Intelligence Organization.

Henri Bejoint. 2014. *The bloomsbury companion to lexicography edited by howard jackson*. *Dictionaries: Journal of the Dictionary Society of North America*, 35:374–381.

Michele Bevilacqua, Marco Maru, and Roberto Navigli. 2020. Generationary or “how we went beyond word sense inventories and learned to gloss”. In *Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)*, pages 7207–7221, Online. Association for Computational Linguistics.

Tom Bosc and Pascal Vincent. 2018. Auto-encoding dictionary definitions into consistent word embeddings. In *EMNLP*, pages 1522–1532.

Bartosz Broda, Maciej Piasecki, and Stan Szpakowicz. 2009. A wordnet from the ground up. *Oficyna Wydawnicza Politechniki Wrocławskiej*.

Pere-Lluís Huguet Cabot and Roberto Navigli. 2021. Rebel: Relation extraction by end-to-end language generation. In *Findings of the Association for Computational Linguistics: EMNLP 2021*, pages 2370–2381.

Ting-Yun Chang and Yun-Nung Chen. 2019. What does this word mean? explaining contextualized embeddings with natural language definition. In *Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)*, pages 6064–6070, Hong Kong, China. Association for Computational Linguistics.

Ting-Yun Chang, Ta-Chung Chi, Shang-Chi Tsai, and Yun-Nung Chen. 2018. xsense: Learning sense-separated sparse representations and textual definitions for explainable word sense networks.

Pinzhen Chen and Zheng Zhao. 2022. A unified model for reverse dictionary and definition modelling. *arXiv preprint arXiv:2205.04602*.

Nicola De Cao, Gautier Izacard, Sebastian Riedel, and Fabio Petroni. 2020. Autoregressive entity retrieval. In *International Conference on Learning Representations*.

Luis Espinosa-Anke, Jose Camacho-Collados, Sara Rodríguez Fernández, Horacio Saggion, and Leo Wan-ner. 2016. Extending wordnet with fine-grained col-locational information via supervised distributional learning. In *Proceedings of COLING 2016: Technical Papers. The 26th International Conference on Computational Linguistics; 2016 Dec. 11-16; Osaka (Japan). [place unknown]: COLING; 2016. p. 900-10. COLING*.

Manaal Faruqui, Jesse Dodge, Sujay Jauhar, Chris Dyer, Eduard Hovy, and Noah Smith. 2014. Retrofitting word vectors to semantic lexicons.

Christiane Fellbaum. 2013. Wordnet. In Carol Chapelle, editor, *The encyclopedia of applied linguistics*, pages 6739–6746. Blackwell Publishing Ltd.

Artyom Gadetsky, Ilya Yakubovskiy, and Dmitry Vetrov. 2018. Conditional generators of words definitions. In *Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)*, pages 266–271, Melbourne, Australia. Association for Computational Linguistics.

Amit Gajbhiye, Luis Espinosa-Anke, and Steven Schockaert. 2022. Modelling commonsense properties using pre-trained bi-encoders.

Noah Gardner, Hafiz Khan, and Chih-Cheng Hung. 2022. Definition modeling: literature review and dataset analysis. *Applied Computing and Intelligence*, 2(1):83–98.

Felix Hill, Kyunghyun Cho, Anna Korhonen, and Yoshua Bengio. 2016. Learning to understand phrases by embedding the dictionary. *Transactions of the Association for Computational Linguistics*, 4:17–30.

Han Huang, Tomoyuki Kajiwara, and Yuki Arase. 2021. Definition modelling for appropriate specificity. In *Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing*, pages 2499–2509, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.

Shonosuke Ishiwatari, Hiroaki Hayashi, Naoki Yoshi-naga, Graham Neubig, Shoetsu Sato, Masashi Toyoda, and Masaru Kitsuregawa. 2019. Learning to describe unknown phrases with local and global contexts. In *Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)*, pages 3467–3476, Minneapolis, Minnesota. Association for Computational Linguistics.

Adam Kilgarriff, Miloš Husák, Katy McAdam, Michael Rundell, and Pavel Rychlý. 2008. Gdex: Automatically finding good dictionary examples in a corpus.
## References

Eneko Agirre and Philip Edmonds. 2007. *Word sense disambiguation: Algorithms and applications*, volume 33. Springer Science & Business Media.

Tal August, Catharina Reinecke, and Noah Smith. 2022. generating scientific definitions with controllable complexity.

Satanjeev Banerjee and Ted Pedersen. 2002. An adapted lesk algorithm for word sense disambiguation using wordnet. volume 2276, pages 136–145.

Edoardo Barba, Luigi Procopio, Caterina Lacerra, Tommaso Pasini, and Roberto Navigli. 2021. Exemplification modeling: Can you give me an example, please? In *Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence, IJCAI-21*, pages 3779–3785. International Joint Conferences on Artificial Intelligence Organization.

Henri Bejoint. 2014. *The bloomsbury companion to lexicography edited by howard jackson*. *Dictionaries: Journal of the Dictionary Society of North America*, 35:374–381.

Michele Bevilacqua, Marco Maru, and Roberto Navigli. 2020. Generationary or “how we went beyond word sense inventories and learned to gloss”. In *Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)*, pages 7207–7221, Online. Association for Computational Linguistics.

Tom Bosc and Pascal Vincent. 2018. Auto-encoding dictionary definitions into consistent word embeddings. In *EMNLP*, pages 1522–1532.

Bartosz Broda, Maciej Piasecki, and Stan Szpakowicz. 2009. A wordnet from the ground up. *Oficyna Wydawnicza Politechniki Wrocławskiej*.

Pere-Lluís Huguet Cabot and Roberto Navigli. 2021. Rebel: Relation extraction by end-to-end language generation. In *Findings of the Association for Computational Linguistics: EMNLP 2021*, pages 2370–2381.

Ting-Yun Chang and Yun-Nung Chen. 2019. What does this word mean? explaining contextualized embeddings with natural language definition. In *Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)*, pages 6064–6070, Hong Kong, China. Association for Computational Linguistics.

Ting-Yun Chang, Ta-Chung Chi, Shang-Chi Tsai, and Yun-Nung Chen. 2018. xsense: Learning sense-separated sparse representations and textual definitions for explainable word sense networks.

Pinzhen Chen and Zheng Zhao. 2022. A unified model for reverse dictionary and definition modelling. *arXiv preprint arXiv:2205.04602*.

Nicola De Cao, Gautier Izacard, Sebastian Riedel, and Fabio Petroni. 2020. Autoregressive entity retrieval. In *International Conference on Learning Representations*.

Luis Espinosa-Anke, Jose Camacho-Collados, Sara Rodríguez Fernández, Horacio Saggion, and Leo Wan-ner. 2016. Extending wordnet with fine-grained col-locational information via supervised distributional learning. In *Proceedings of COLING 2016: Technical Papers. The 26th International Conference on Computational Linguistics; 2016 Dec. 11-16; Osaka (Japan). [place unknown]: COLING; 2016. p. 900-10. COLING*.

Manaal Faruqui, Jesse Dodge, Sujay Jauhar, Chris Dyer, Eduard Hovy, and Noah Smith. 2014. Retrofitting word vectors to semantic lexicons.

Christiane Fellbaum. 2013. Wordnet. In Carol Chapelle, editor, *The encyclopedia of applied linguistics*, pages 6739–6746. Blackwell Publishing Ltd.

Artyom Gadetsky, Ilya Yakubovskiy, and Dmitry Vetrov. 2018. Conditional generators of words definitions. In *Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)*, pages 266–271, Melbourne, Australia. Association for Computational Linguistics.

Amit Gajbhiye, Luis Espinosa-Anke, and Steven Schockaert. 2022. Modelling commonsense properties using pre-trained bi-encoders.

Noah Gardner, Hafiz Khan, and Chih-Cheng Hung. 2022. Definition modeling: literature review and dataset analysis. *Applied Computing and Intelligence*, 2(1):83–98.

Felix Hill, Kyunghyun Cho, Anna Korhonen, and Yoshua Bengio. 2016. Learning to understand phrases by embedding the dictionary. *Transactions of the Association for Computational Linguistics*, 4:17–30.

Han Huang, Tomoyuki Kajiwara, and Yuki Arase. 2021. Definition modelling for appropriate specificity. In *Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing*, pages 2499–2509, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.

Shonosuke Ishiwatari, Hiroaki Hayashi, Naoki Yoshi-naga, Graham Neubig, Shoetsu Sato, Masashi Toyoda, and Masaru Kitsuregawa. 2019. Learning to describe unknown phrases with local and global contexts. In *Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)*, pages 3467–3476, Minneapolis, Minnesota. Association for Computational Linguistics.

Adam Kilgarriff, Miloš Husák, Katy McAdam, Michael Rundell, and Pavel Rychlý. 2008. Gdex: Automatically finding good dictionary examples in a corpus.