# Towards Sense to Sense Linking across DBnary Languages

Gilles Serasset

<figure><img src="image_1.png" /></figure>

## To cite this version:

Gilles Serasset. Towards Sense to Sense Linking across DBnary Languages. Proceedings of the 5th Conference on Language, Data and Knowledge, Sep 2025, Naples, Italy. pp.318–327. hal-05267573

HAL Id: hal-05267573

https://hal.science/hal-05267573v1

Submitted on 18 Sep 2025

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_2.png" /></figure>
# Towards Sense to Sense Linking across DBnary Languages

Gilles Sérasset

Univ. Grenoble Alpes, CNRS, Grenoble INP, LIG
GETALP team
38000 Grenoble, France
gilles.serasset@imag.fr

## Abstract

Since 2012, the DBnary project extracts lexical information from different Wiktionary language editions (26 editions in 2025) and makes it available to the community as queryable RDF data (modelled using the Ontolex-lemon ontology). This dataset contains more than 12M translations linking languages at the level of Lexical Entries. This paper presents an effort to automatically link the DBnary languages at the Lexical Sense level. For this, we explore different ways to compute cross-lingual semantic similarity, using multilingual language models.

## 1 Introduction

Even in the era of Large Language Models pretrained in unsupervised settings, Lexical Resources (LR) are still in use and have proved useful for advancing various natural language processing (NLP) tasks. For instance, such resources may enhance the quality of machine translation by providing accurate cross-lingual mappings, thus improving translation fidelity (Jones et al., 2023). They are also of importance for end users that easily refer to them through on-line browsing or mobile dictionary apps.

Since 2012, the DBnary dataset<fnref n="1" /> extracts lexical data from 26 Wiktionary language editions² and makes it available as an RDF dataset. Extracted from one of the most important community built lexical resource, it contains lexical entries in many languages, along with definitions, lexico-semantic relations, translation, among other lexical information. One of the shortcomings of the current dataset lies in the lack of semantic alignment between language editions.

The final objective of this work will be twofold:
1. providing cross-lingual links at the lexical sense level, based on the translations available at the entry or surface form level, and 2. associating each DBnary lexical sense with an embedding in a unique multilingual vector space.

In this paper, we explore the use of Multilingual Neural Language Models for the computation of a cross-lingual semantic similarity measure that we use to align existing translation pairs at the semantic level. After describing the current way DBnary dataset models cross-lingual links (section 2), we will define the task at hand and the related work we borrow from (section 3), then describe a gold standard dataset we built to evaluate different approaches (section 3.2). We proceed with the experiments (section 4) and results (section 5) and discuss shortcomings of the approaches for the systematic modelling of translations at the semantic level and for the distribution of sense embeddings to the end users (section 6).

## 2 The DBnary Dataset

DBnary (Sérasset, 2012, 2015) is a large multilingual lexical dataset extracted from 26 language editions of the Wiktionary project. It is made available following the Lexical Linked Open Data principles using Ontolex-lemon model in RDF format, following Chiarcos et al. (2011). Overall, this dataset describes 7.9M Lexical Entries, accounting for 6.5M Lexical Senses, usually described with a textual definition. Additionally, it contains 12.3M translation pairs.

The OntoLex<fnref n="3" /> model, is a community standard for machine-readable lexical resources that has been adopted by many data providers for its ability to ensure FAIR principles,⁴ and the dominant

<footnote n="1">¹https://kaiko.getalp.org/about-dbnary
²Just like Wikipedia, there are different editions of Wiktionary (that differ by their URL (e.g http://en.wiktionary.org refers to the English edition and http://fr.wiktionary.org to the French edition). Following Meyer and Gurevych (2012), we call each of these independent web sites a language edition.</footnote>

<footnote n="3">³https://www.w3.org/2016/05/ontolex
⁴FAIR (for Findable Accessible Interoperable and Reusable) refers to a set of principles dedicated to allow for re-use of any research object.</footnote>
vocabulary for modeling machine-readable dictionaries as Linked Data. The goal of OntoLex is to represent lexical resources as a knowledge graph, allowing the integration of information from different dictionaries and to facilitate the exchange, storage, and reusability of lexical information. The Ontolex-lemon model is a W3C community report consisting of a core model, along with additional modules (mainly, **lime** for metadata, **synsem** for the description of syntax and semantics, **decomp** for decomposition of terms into subterms and **vartrans** to represent lexico-semantic and translation relations).

<figure><img src="image_3.png" /><figcaption>Figure 1: The Ontolex model along with DBnary extension used to represent Wiktionary pages and translations.</figcaption></figure>

The atomic unit of information of Wiktionary is a *page*, where several lexical *entries* may be described. Such entries usually share their canonical form (which usually corresponds to the name of the page). The organisation of entries in the page and the structure and content of such entries differ according to the Wiktionary language editions, but usually contain definitions of the senses and a set of lexical information (etymology, morphology, lexico-semantic derivations, ...), along with translations in other languages. As an example, the *cat*<sub>eng</sub><fnref n="5" /> page in the English edition describes 6 entries (3 nouns, 2 verbs and 1 adjective). Other lexical entries in other languages are also described and extracted in DBnary; however, for this article, we will focus only on the lexical entries of the language edition (what we call the *endolex*).

DBnary uses Ontolex core model to represent the extracted data, hence, it gives a common representation of lexical data, despite the differences in the way such lexical information is organised in

each edition. Ad-hoc classes were added to be able to represent pages (dbnary:Page) and also to represent translations pairs (dbnary:Translation), along with glosses (dbnary:Gloss) that are often used to provide users with context to disambiguate the lexical sense for which a lexical information stands. The DBnary resulting model is shown in Figure 1.

<figure><img src="image_4.png" /><figcaption>(a) The **vartrans** module allows for the representation of cross-lingual links either at the Lexical Entry level or at the Lexical Sense level.</figcaption></figure>

<figure><img src="image_5.png" /><figcaption>(b) The Wiktionary data is often insufficient to decide if the target of a translation is a Lexical Entry in the target language, and, if it is, which one (in case of homonymy), also, there are usually no information allowing to decide on the target sense that is involved in a translation.</figcaption></figure>

<caption>Figure 2: A graphical representation of the cross-lingual linking strategies, according to the semantic level of the links and inspired by the Vauquois triangle (Vauquois, 1968) according to Gracia et al. (2025).</caption>

The Ontolex model defines the **vartrans** extension to encode relations in general and translations in particular. However, DBnary authors chose a nonstandard representation. The reason is illustrated in Figure 2. Figure 2a shows that the **vartrans** module may be used either to link two Lexical Entries or two Lexical Senses together. However, in Wiktionary, each language edition is independent of the other, and the available translations are given as strings, with no guarantee that they correspond to a lexical entry in the target edition. For example, *persignar*<sub>cat</sub> is translated in English as “cross oneself on forehead, lips, and heart”, which does not correspond to a valid English lexical entry.

<footnote n="5">⁵In this article we will use *entry*<sub>lg</sub> to denote the page named “*entry*” in the edition for language *lg*, this particular page is available at https://en.wiktionary.org/wiki/cat</footnote>
Also, since Wiktionary is an ongoing collaborative dictionary, the target page may not exist while being a perfectly valid lexical entry in the target lexicon. Finally, even if the page exists, it is not possible to systematically decide which lexical entry is the target of the translation. For all these reasons, DBnary decided to reify each translation pair using dbnary:Translation class where the source lexical entry is usually known and the target is represented as a surface form rather than as a link to a lexical entry in the target lexicon (in blue in Figure 2b).

Many translations are also associated with their source lexical sense (in dotted green in Figure 2b) that are selected using Tchechmedjiev et al. (2014) with an accuracy of 0.82 to 0.96 F1 score.

# 3 Linking Language Editions at the Lexical Sense Level

The task we address in this paper is the following: "How can we efficiently identify the correct source and target lexical sense(s) that are involved in available translation pairs?"

The main objective of this work is to include such links using the **vartrans** module in DBnary and, as a by-product, provide the multilingual lexical sense embeddings that allowed for this linking. With 12M translations and more than 6M lexical senses with definitions that continuously change while the language communities collaboratively correct and expand the editions, DBnary extracts a new version twice a month. So, links must be recomputed at each extraction, and the efficiency of the method should be assessed both in terms of performance and in terms of frugality in computing resources.

## 3.1 Related Work

This work follows on Tchechmedjiev et al. (2014), which attempted to identify the lexical sense of translation sources by leveraging monolingual similarity measures using a two-level string distance based on Tversky index (Tversky, 1977) (sentence similarity distance computed on a sequence of tokens, with token similarity computed with a character-level string distance).

In this initial work, approaches using statistical measures like Jimenez et al. (2012) were disregarded as they were requiring too much computation times for the statistical model computation in a multilingual setting where the number of languages

was growing. However, times have changed, and today, many language models are available for semantic similarity measure computation.

The task at hand implies being able to compute the similarity between lexical sense definitions in different languages. For this, we tried several strategies.

### 3.1.1 Token similarity measure

The first strategy modified the two-level similarity measure used in Tchechmedjiev et al. (2014) by substituting the token-level character-based string distance with cosine similarity between *fastText* non contextual token embeddings (Bojanowski et al., 2016) trained and aligned on multilingual texts (Joulin et al., 2018).

### 3.1.2 Sentence similarity measures

Many models are now trained to directly compute sentence similarity. For the monolingual task, we could use monolingual models; however, in multilingual settings we need the model to be multilingual and compute similarity between sentences in different languages. In this work, we focus on multilingual sentence similarity models, as the final objective is to align definitions in as many language pairs as possible. We evaluated Multilingual Universal Sentence Encoder (MUSE) (Yang et al., 2020), Language-agnostic BERT sentence embedding (LaBSe) (Feng et al., 2022), Language-Agnostic SEntence Representations (LASER) (Artetxe and Schwenk, 2019a,b), Sentence-Level Multimodal and Language-Agnostic Representations (SONAR) (Duquenne et al., 2023), Multilingual E5 Text Embeddings (Wang et al., 2024), mGTE (Zhang et al., 2024), and original multilingual pretrained models from sentence BERT (Reimers and Gurevych, 2019, 2020).<fnref n="6" /> OpenAI text embedding models (via openai API) (Neelakantan et al., 2022) have also been used for comparison purposes only.

### 3.1.3 Machine Translation based similarity measures

Another approach for computing the cross-lingual similarity measure is to rely on a machine translation system to compute similarity on two texts in the same language. In this work, we used Opus-MT

<footnote n="6">⁶Namely paraphrase-multilingual-mpnet-base-v2 (paraphrase), stsb-xlm-r-multilingual (stsb), static-retrieval-mrl-en-v1 (static) and static-similarity-mrl-multilingual-v1 (static-similarity).</footnote>
models from Tiedemann et al. (2023) and Tiedemann and Thottingal (2020) to translate target languages into English before computing monolingual similarity with sentence similarity measures.

## 3.2 Sense to Sense Linking Gold Standard

We evaluated the different strategies on a gold standard dataset created from translation pairs and lexical sense definitions extracted from DBnary. To create this dataset we extracted 96 English pages from DBnary, chosen among frequent and highly ambiguous English terms. These pages described 232 different lexical entries and 2646 different English lexical senses.

Translation pairs from English to Chinese, French, German, Italian, Russian and Spanish were extracted. For this experiment, we only selected translations that were associated to a textual gloss helping to disambiguate the source sense. Table 1a shows the resulting number of entries and translation pairs per Part of Speech.

For each translation pair, we extracted target lexical entries and lexical senses. Each target lexical entry was also associated to a fake lexical sense [NAWS] (Not A Word Sense) created to identify lexical entries that are a valid translation target, but for which none of the described lexical senses was a valid target. Table 2 shows an example of the resulting data that is presented to the annotators.

Six annotators identified the source and target sense(s) involved in each translation pair, given the associated gloss. This implied two successive tasks: (1) *Monolingual task*: selecting the English definition associated to the English gloss, and (2) *Cross-lingual task*: selecting the target definition(s) associated to the translation pair, given the English selected sense definition. The annotator may select more than one target sense if necessary, and if no target sense is to be found, select [NAWS] definition for the appropriate lexical entry.

Agreement is measured using Kripendorff alpha ($\alpha$ = 0.966, which is considered high agreement, while $\alpha$ = 0.674 for cross-lingual task, which is just above the $\alpha$ value (0.667) considered as minimal for data to be used to draw tentative conclusions. This agreement value is coherent with previous observation drawn when creating word sense disambiguation (WSD) datasets with fine-grain word sense definitions (Véronis, 1998; Mur-ray and Green, 2004) as it is the case here.

After cleanup, systematic errors correction,

<table><thead><tr><th></th><th>N.</th><th>Vb.</th><th>Adj.</th><th>Adv.</th><th>Int.</th></tr></thead><tbody><tr><td>entries</td><td>116</td><td>82</td><td>30</td><td>3</td><td>1</td></tr><tr><td>pairs</td><td>1711</td><td>907</td><td>263</td><td>25</td><td>3</td></tr></tbody></table>

(a) Repartition of part of speech for English source entries and translation pairs.

<table><thead><tr><th>deu</th><th>fra</th><th>ita</th><th>rus</th><th>spa</th><th>zho</th></tr></thead><tbody><tr><td>873</td><td>622</td><td>639</td><td>147</td><td>530</td><td>98</td></tr></tbody></table>

(b) Number of annotated translation pairs per target language.

<caption>Table 1: Insights on the annotated dataset built for the task.</caption>

and majority vote for disagreeing annotations, the dataset contains 2927 annotated translation pairs. Table 1b gives the number of annotated translation per target language. The annotated dataset is available at https://gricad-gitlab.univ-grenoble-alpes.fr/getalp/dbnary-translations-disambiguation

# 4 Experiments

## 4.1 Monolingual Task

The purpose of the monolingual task is to identify the source sense that is denoted by a gloss associated to a translation pair. The first column of table 2 shows the set of English definitions among which to choose the one that is referred to by the gloss “device made of flexible material”. In Wiktionary, glosses are usually a shorter version of the intended lexical sense definition. In our example, the gloss refers to lexical sense 5.4 of spring eng noun.

In a first approach, one could try to select the most similar definition in the set of available definitions $D$, based on a similarity measure (Sim) with the gloss $g$ (Equation 1), hence selecting only one lexical sense per gloss.

$$ \arg \max_{d \in D} \mathrm{Sim}(d, g) \qquad (1) $$

However, in practice, the gloss may refer to several lexical senses of the entry. Hence, we allow the selection of multiple senses if their similarity is within a window size $\delta$ (Equation 2).

$$ \left\{ d \in D \mid \mathrm{Sim}(d, g) \geq \max_{d' \in D} \mathrm{Sim}(d', g) - \delta \right\} \qquad (2) $$

We chose to address three strategies for the computation of the similarity measure.
<table><caption>Table 2: Example of the spring<sub>eng</sub> to muelle<sub>spa</sub> translation pair which is associated with the gloss: “device made of flexible material”. presented with this extract, the annotator has to select the correct word sense in English (monolingual task) and in Spanish (cross-lingual task) taking the gloss into account.</caption><thead><tr><th>spring<sub>eng</sub>, noun</th><th>muelle<sub>spa</sub></th></tr></thead><tbody><tr><td>1. (countable) An act of springing: a leap, a jump.</td><td>adj. 1</td></tr><tr><td>2. (countable) The season of the year in temperate regions in which plants spring from the ground and into bloom and dormant animals spring to life.</td><td>1. Delicado, suave, blando.</td></tr><tr><td>2.1. (astronomy) The period from the moment of vernal equinox (around March 21 in the Northern Hemisphere) to the moment of the summer solstice ...</td><td>2. Voluptuoso.</td></tr><tr><td>2.2. (meteorology) The three months of March, April, and May in the Northern Hemisphere and September, October, and November in the Southern Hemisphere.</td><td>NAWS</td></tr><tr><td>3. (uncountable, figurative) The time of something's growth; the early stages of some process.</td><td>sust. 1</td></tr><tr><td>3.1. (figurative, politics) a period of political liberalization and democratization</td><td>1. Pieza elástica, usualmente de metal, colocada de modo que pueda utilizarse la fuerza que hace para recobrar su posición natural cuando ha sido separada de ella.</td></tr><tr><td>4. (countable, fashion) Someone with ivory or peach skin tone and eyes and hair that are not extremely dark, seen as best suited to certain colors of clothing.</td><td>2. Adorno compuesto de varios relicarios o dijes, que las mujeres de distinción llevaban pendiente a un lado de la cintura.</td></tr><tr><td>5. (countable) Something which springs, springs forth, springs up, or springs back, particularly</td><td>3. En plural Tenazas grandes que se usan en las casas de moneda para agarrar los rieles y tejos durante la fundición y echarlos en la copela.</td></tr><tr><td>5.1. (geology) A spray or body of water springing from the ground.</td><td>NAWS</td></tr><tr><td>5.2. (oceanography, obsolete) The rising of the sea at high tide.</td><td>sust. 2</td></tr><tr><td>5.3. (oceanography) Short for spring tide, the especially high tide shortly after full and new moons.</td><td>1. Náutica. Obra de piedra, hierro o madera, construida en dirección conveniente en la orilla del mar o de un río navegable, y que sirve para facilitar el embarque y ...</td></tr><tr><td>5.4. A mechanical device made of flexible or coiled material that exerts force and attempts to spring back when bent, compressed, or stretched.</td><td>2. Transporte. Andén alto, cubierto o descubierto, que en las estaciones de ferrocarriles sirve para la carga y descarga de mercancías.</td></tr><tr><td>5.5. (nautical) A line from a vessel's end or side to its anchor cable used to diminish or control its movement.</td><td>NAWS</td></tr><tr><td>5.6. (nautical) A line laid out from a vessel's end to the opposite end of an adjacent vessel or mooring to diminish or control its movement.</td><td></td></tr><tr><td>5.7. (figurative) A race, a lineage.</td><td></td></tr><tr><td>5.8. (figurative) A youth.</td><td></td></tr><tr><td>5.9. A shoot, a young tree.</td><td></td></tr><tr><td>5.10. A grove of trees; a forest.</td><td></td></tr><tr><td>6. (countable, slang) An erection of the penis.</td><td></td></tr><tr><td>7. (countable, nautical, obsolete) A crack which has sprung up in a mast, spar, or (rare) a plank or seam.</td><td></td></tr><tr><td>8. (uncountable) Springiness: an attribute or quality of springing, springing up, or springing back, particularly</td><td></td></tr><tr><td>8.1. Elasticity: the property of a body springing back to its original form after compression, stretching, etc.</td><td></td></tr><tr><td>8.2. Elastic energy, power, or force.</td><td></td></tr><tr><td>9. (countable) The source from which an action or supply of something springs.</td><td></td></tr><tr><td>10. (countable) Something which causes others or another to spring forth or spring into action, particularly</td><td></td></tr><tr><td>10.1. A cause, a motive, etc.</td><td></td></tr><tr><td>10.2. (obsolete) A lively piece of music.</td><td></td></tr><tr><td>NAWS</td><td></td></tr></tbody></table>

We first reproduced the results from Tchechmedjiev et al. (2014) as a baseline to compare with the other similarity measures.

Then, we evaluated token similarity measure (see 3.1.1) borrowing the sentence similarity computation from Tchechmedjiev et al. (2014) and replacing the token similarity computation with cosine similarity on aligned fastText vectors.

Our third strategy uses sentence similarity measures (see 3.1.2), with cosine similarity on sentence embedding models that compute an unique vector for each definition.

For better interpretation of the results, we also provide two heuristics that were frequently used in WSD tasks: 1. random selection of a word sense and 2. systematic selection of the word sense described first in the lexical entry.

## 4.2 Multilingual Task

The input of the multilingual task is a source sense (supposedly identified by the monolingual task) and a surface form in a target language. The surface

form is used to query DBnary for Lexical Entries on their canonical form (lemma) and their associated Lexical Senses. A pseudo sense labelled “NAWS” is added to each lexical entry.

The purpose of the task is to choose the lexical senses that are the adequate target for this specific translation of the given source lexical sense. Also, if none of the given target lexical senses are adequate, the multilingual task should identify the target lexical entry by selecting its associated “NAWS” pseudo sense.

For example, the second column of table 2 shows the lexical entries and senses associated to the Spanish translation of spring<sub>eng</sub> sense 5.4. Among those, the task should choose sense 1. of the first nominal entry of muelle<sub>spa</sub>.

The approach used in this task is borrowed from the monolingual task Equation 2, where the gloss is substituted by the source sense definition and the similarity measures are multilingual.

The approach should also be able to decide that none of the lexical senses are fit to be selected
as targets and should choose the fallback “NAMS” pseudo word sense. For this, we introduce the Ω hyper-parameter which is the minimum similarity for which a lexical sense is eligible to be selected (Equation 3).

$$ \text{let } m = \max_{d' \in D} \text{Sim}(d', g) \text{ in} \\ \left\{ d \in D \mid \begin{array}{l} \text{Sim}(d, g) \ge m - \delta \ \& \\ \text{Sim}(d, g) \ge \Omega \end{array} \right\} \quad (3) $$

Tchechmedjiev et al. (2014) cannot be used in this setting as the string distance used as token similarity measure is not multilingual. Token and sentence similarity measures (see 3.1.1 and 3.1.2) may be used provided that the models embed token or sentences into the same vector space regardless of the token or sentence language.

For this task, we also experimented with Machine Translation based similarity measures (see 3.1.3).

## 4.3 Hyper-parameters Optimisation

All hyper-parameters were optimised using a grid search on 20% of the evaluation data. All results below are computed using optimised hyper-parameters.

Among the hyper-parameters, $\delta$ represents the ability of the method to select more than one target word-senses if their respective similarity are close enough, while $\Omega$ represents the ability of the model to decide that the most similar target sense is a valid choice or if none of the target senses are valid.

Of course, these hyper-parameters depend on the model, however, their values and behaviours follow common tendency. As an example Figure 3 shows the hyper-parameter influence for the sentence embeddings paraphrase model.

The source code for the experiments is available at https://gricad.gitlab.univ-grenoble-alpes.fr/getalp/sense2sense_translations.

# 5 Results

All tasks are evaluated using standard set-matching metrics, i.e. Recall, Precision and F1 score. All scores are computed using the optimal hyper-parameters setting.

## 5.1 Monolingual Task

Table 3 shows the performance of the different approaches. For better understanding of the task, we

<table><caption>Table 3: Results of the monolingual task, with F1, precision and recall scores. Bold values are maximum, and non significantly different scores are underlined.</caption><thead><tr><th>Model</th><th>F1</th><th>precision</th><th>recall</th></tr></thead><tbody><tr><td>random</td><td>0.131</td><td>0.131</td><td>0.131</td></tr><tr><td>first-sense</td><td>0.214</td><td>0.214</td><td>0.214</td></tr><tr><td>tchechmedjiev</td><td>0.914</td><td>0.908</td><td>0.929</td></tr><tr><td>fasttext</td><td>0.899</td><td>0.890</td><td>0.921</td></tr><tr><td>e5-instruct</td><td>0.899</td><td>0.899</td><td>0.899</td></tr><tr><td>gte-multilingual-base</td><td>0.928</td><td>0.928</td><td>0.928</td></tr><tr><td>labse</td><td>0.901</td><td>0.882</td><td>0.946</td></tr><tr><td>laser</td><td>0.763</td><td>0.763</td><td>0.763</td></tr><tr><td>muse</td><td>0.889</td><td>0.875</td><td>0.923</td></tr><tr><td>paraphrase</td><td>0.930</td><td>0.930</td><td>0.930</td></tr><tr><td>sonar</td><td>0.783</td><td>0.760</td><td>0.843</td></tr><tr><td>static</td><td>0.929</td><td>0.922</td><td>0.944</td></tr><tr><td>static-similarity</td><td>0.932</td><td>0.926</td><td>0.946</td></tr><tr><td>stsb</td><td>0.877</td><td>0.877</td><td>0.877</td></tr><tr><td>text-embedding-3-large</td><td><strong>0.946</strong></td><td><strong>0.946</strong></td><td>0.946</td></tr><tr><td>text-embedding-3-small</td><td><u>0.944</u></td><td>0.936</td><td><strong>0.962</strong></td></tr><tr><td>text-embedding-ada-002</td><td><u>0.944</u></td><td>0.944</td><td>0.944</td></tr></tbody></table>

evaluated the heuristic consisting in selecting the first sense as the predicted answer that is frequently used in WSD tasks, based on the hypothesis that Wiktionary senses are given in order of usage frequency. Unlike usual WSD tasks where the first sense is also the most frequent sense (hence the most frequent answer), this heuristic is not significantly better than the random baseline.

In Tchechmedjiev et al. (2014) the reported F1-scores were 0.826 for French, 0.865 for Portuguese and 0.968 for Finnish. The evaluation was not available for English, due to the way the gold standard was (automatically) generated using glosses that were both given as a short text summarising a definition and as a word sense number (that were taken as the ground truth). The original string distance based monolingual word sense identification performs significantly better when evaluated on our gold standard than when evaluated using the original automatic gold standard generation. The reason for this difference comes from the fact that Wiktionary is an ever changing resources, and when word senses are edited, added, removed or re-ordered, the numerical glosses that refer to them are sometimes not updated and become out of sync with the set of definitions. Hence the original performance of this method was underestimated and one can see that it outperforms many sentence similarity models despite its very efficient computational cost.

The best-performing models are the OpenAI em-
<figure><img src="image_6.png" /><figcaption>Figure 3: Influence of the hyper-parameters on the system performance for the paraphrase embedding in English to Italian task. In most models, $\delta = .05$, which shows it is a good choice to slightly take the risk to accept several lexical sense as valid targets, while $\Omega$ is much more fluctuant and highly depend on the model.</figcaption></figure>

bedding models accessible through the OpenAI API. However, it is unclear what the actual energy cost of such models is, and using these models for the entire DBnary is likely to incur a cost that is not sustainable in our context. Moreover, some open-source model results may be considered almost as good as those of the OpenAI models, even if they are significantly lower.

## 5.2 Multilingual Task

Table 4 presents the results of the multilingual task, using F1 score. The random and first-sense baselines are provided.

For this task, OpenAI models also give the best results. However, we chose not to rely on these models in our use cases for reasons explained in section 6. For this reason, OpenAI models are only evaluated for reference, and are not included in best result and significance computations. These models are given in an independent table section were maximum values and significances are computed independently.

From these results, we can see that no model wins it all. The performance of sentence embeddings depends on the target language. Using a translation step plus monolingual sentence embedding seems to give slightly better results, but has a higher energy cost as the translation cost is added to embedding cost. However, this approach could allow for the use of a better monolingual sentence embedding model. Further evaluation is needed on this aspect.

## 6 Discussions and Limitations

Monolingual evaluation shows that it is still difficult to surpass Tchechmedjiev et al. (2014) which is based solely on string distance computation, with a very small energy cost for all languages. In this

task, the added value of some models may not justify the energy cost. Thanks to the manually built gold standard, we also showed that the original results were underestimated.

In multilingual settings, sentence embedding models yield the best scores. However, it is not entirely clear whether the overall performance is sufficient to create links of high quality for inclusion in a distributed dataset. More detailed analysis should be conducted to understand what makes this task so challenging. The structure of definitions (frequently structured as genus-differencia) may differ significantly from the structure of sentences used to train the sentence embedding models. Moreover, lexical senses of the same entry are expected to share most of their semantics, and the task at hand should focus more on their differences rather than on their similarities.

OpenAI models delivered the best results in both monolingual and multilingual tasks. However, we cannot rely on these embeddings for our use case. The first reason is a general concern that arises when OpenAI models or chat services are used in a research setting. The fact that OpenAI embeddings, architectures, and datasets are closed source and largely unexplained prevents us from gaining any understanding of the reasons behind their success, nor can we determine whether the success on the gold standard is generalisable to real-world data. Although we are relatively confident that our (newly created) dataset was not part of those model training, we also have a particular reason to exclude OpenAI from our work.

Our objective is twofold: 1) to provide sense-to-sense translation links and 2) to distribute the embeddings associated with each lexical sense, so that end users can compute similarities at the lexical sense level. For the first objective, the best
<table><caption>Table 4: F1 measure (higher is better) for optimal hyper-parameters for each language and averaged over languages. Maximum scores are given in bold and values that do not differ significantly (i.e. when p-values > .05) from best results are underlined. In all but latest section, OpenAI models are disregarded for maximum and significance computation. The latest section gather results using OpenAI API text embedding models. In this section, maximums are given in italics and significance is computed taking OpenAI models into account.</caption><thead><tr><th></th><th>deu</th><th>fra</th><th>ita</th><th>rus</th><th>spa</th><th>zho</th><th>Average</th></tr></thead><tbody><tr><td>random</td><td>0.307</td><td>0.222</td><td>0.254</td><td>0.329</td><td>0.238</td><td>0.510</td><td>0.310</td></tr><tr><td>first-sense</td><td>0.391</td><td>0.383</td><td>0.328</td><td>0.522</td><td>0.358</td><td><u>0.544</u></td><td>0.421</td></tr><tr><td>fasttext</td><td>0.444</td><td>0.511</td><td>0.511</td><td>0.558</td><td>0.502</td><td>0.513</td><td>0.507</td></tr><tr><td>translation+fasttext</td><td>0.446</td><td>0.412</td><td>0.368</td><td>0.427</td><td>0.400</td><td>0.526</td><td>0.430</td></tr><tr><td>e5-instruct</td><td>0.513</td><td><u>0.595</u></td><td>0.393</td><td><u>0.652</u></td><td>0.472</td><td><u>0.609</u></td><td>0.539</td></tr><tr><td>gte-multilingual-base</td><td>0.535</td><td><u>0.585</u></td><td><u>0.569</u></td><td>0.595</td><td>0.521</td><td><u>0.614</u></td><td>0.570</td></tr><tr><td>labse</td><td>0.507</td><td>0.524</td><td>0.503</td><td><u>0.632</u></td><td>0.539</td><td><u>0.595</u></td><td>0.550</td></tr><tr><td>laser</td><td>0.441</td><td>0.435</td><td>0.483</td><td>0.468</td><td>0.443</td><td><u>0.560</u></td><td>0.472</td></tr><tr><td>muse</td><td>0.511</td><td>0.545</td><td>0.527</td><td><u>0.641</u></td><td>0.551</td><td><u>0.609</u></td><td>0.564</td></tr><tr><td>paraphrase</td><td><u>0.570</u></td><td><u>0.611</u></td><td>0.514</td><td><u>0.663</u></td><td><u>0.597</u></td><td><u>0.591</u></td><td>0.591</td></tr><tr><td>sonar</td><td>0.516</td><td>0.519</td><td>0.510</td><td><u>0.649</u></td><td>0.494</td><td><u>0.597</u></td><td>0.547</td></tr><tr><td>static</td><td>0.329</td><td>0.355</td><td>0.447</td><td>0.236</td><td>0.327</td><td>0.339</td><td>0.339</td></tr><tr><td>static-similarity</td><td>0.485</td><td>0.529</td><td><u>0.549</u></td><td>0.540</td><td>0.535</td><td><u>0.605</u></td><td>0.540</td></tr><tr><td>stsb</td><td>0.520</td><td><u>0.593</u></td><td>0.545</td><td><u>0.675</u></td><td>0.546</td><td><u>0.588</u></td><td>0.578</td></tr><tr><td>translation+e5-instruct</td><td>0.504</td><td>0.577</td><td>0.380</td><td>0.584</td><td>0.487</td><td><u>0.593</u></td><td>0.521</td></tr><tr><td>translation+gte-multilingual-base</td><td>0.527</td><td><u>0.595</u></td><td><u>0.555</u></td><td><u>0.610</u></td><td>0.547</td><td><u>0.604</u></td><td>0.573</td></tr><tr><td>translation+labse</td><td>0.497</td><td>0.509</td><td>0.501</td><td>0.574</td><td>0.522</td><td><u>0.611</u></td><td>0.536</td></tr><tr><td>translation+laser</td><td>0.425</td><td>0.421</td><td>0.496</td><td>0.469</td><td>0.404</td><td><u>0.543</u></td><td>0.460</td></tr><tr><td>translation+muse</td><td>0.529</td><td>0.557</td><td><u>0.556</u></td><td><u>0.640</u></td><td>0.577</td><td><u>0.614</u></td><td>0.579</td></tr><tr><td>translation+paraphrase</td><td><u>0.570</u></td><td><u>0.604</u></td><td>0.523</td><td><u>0.628</u></td><td><u>0.617</u></td><td><u>0.611</u></td><td><u>0.592</u></td></tr><tr><td>translation+sonar</td><td>0.498</td><td>0.530</td><td>0.513</td><td>0.591</td><td>0.471</td><td><u>0.586</u></td><td>0.531</td></tr><tr><td>translation+static</td><td>0.536</td><td>0.549</td><td>0.537</td><td>0.563</td><td>0.539</td><td><u>0.569</u></td><td>0.549</td></tr><tr><td>translation+static-similarity</td><td>0.498</td><td>0.571</td><td><u>0.568</u></td><td>0.604</td><td>0.561</td><td><u>0.565</u></td><td>0.561</td></tr><tr><td>translation+stsb</td><td>0.529</td><td>0.576</td><td><u>0.574</u></td><td>0.603</td><td>0.569</td><td><u>0.599</u></td><td>0.575</td></tr><tr><td>text-embedding-3-large</td><td><u>0.606</u></td><td><u>0.667</u></td><td>0.588</td><td><u>0.693</u></td><td><u>0.623</u></td><td><u>0.677</u></td><td><u>0.642</u></td></tr><tr><td>text-embedding-3-small</td><td><u>0.591</u></td><td>0.616</td><td><u>0.594</u></td><td>0.629</td><td><u>0.629</u></td><td><u>0.646</u></td><td>0.617</td></tr><tr><td>text-embedding-ada-002</td><td>0.539</td><td>0.621</td><td>0.405</td><td>0.601</td><td>0.494</td><td>0.590</td><td>0.542</td></tr><tr><td>translation+text-embedding-3-large</td><td><u>0.588</u></td><td>0.639</td><td><u>0.611</u></td><td>0.625</td><td><u>0.624</u></td><td>0.619</td><td>0.618</td></tr><tr><td>translation+text-embedding-3-small</td><td><u>0.581</u></td><td><u>0.644</u></td><td><u>0.590</u></td><td><u>0.657</u></td><td><u>0.612</u></td><td>0.605</td><td>0.615</td></tr><tr><td>translation+text-embedding-ada-002</td><td>0.538</td><td>0.616</td><td>0.407</td><td>0.598</td><td>0.496</td><td><u>0.636</u></td><td>0.548</td></tr></tbody></table>

model should be used if its energy cost is reasonable on the scale of the full DBnary dataset. However, providing embeddings tied to a closed-source, proprietary model would tie DBnary users to the model provider for their own use case. This would force any user of the DBnary embeddings to pay OpenAI to exploit them in their use cases.

Finally we should also note two limitations of this preliminary work. The first limitation comes from the dataset that only shows translation from English to other languages. Other sources should be added to the dataset. Moreover, more entries should be added to get more Chinese and Russian translations in order to have more significant results.

The second limitation comes from the main methodology. In this preliminary work, we take the glosses as a starting point for sense-to-sense link computation. The monolingual task identify the source lexical sense from the gloss, then the multilingual task is performed using the identified

source lexical sense to identified the target one. This means that we can only deal with translations that are associated to a gloss. In English this represents 96.8% of the available translations. For all other translations, the sense-to-sense cross-lingual link cannot be computed with this process and 3.2% of the available translations will be disregarded (see Appendix A for full statistics on the availability of glosses in all DBnary extracted languages). Among all DBnary languages, the proportion of translations that are associated to a textual gloss are very imbalanced from 0% to 99.6%. As an example, with this methodology we will only handle 34.6% of the translations of French words.

## 7 Conclusion

This preliminary study is a first step towards a better cross-lingual link modelling in the DBnary dataset. Although some results are encouraging, additional work is required to achieve our goal. Focussing on our first objective (identifying lexical
senses involved in translation pairs), definition embeddings bring a lot but will certainly benefit of other approaches.

If we succeed in computing a cross-lingual semantic similarity measure between DBnary lexical senses, we will be able to provide such cross-lingual links, but also distribute embeddings for each lexical sense in the dataset. Such embeddings could be used to query for semantically related senses and could be reused by end-users in downstream tasks. Many applications would benefit from these: browsing DBnary with direct access to close senses, bootstrapping/aligning models with lexical sense embedding rather than embeddings associated to surface forms, or linking such senses with several ontologies.

However, more work should be done before achieving results that are good enough for the computed data to be distributed along with the original DBnary data. For this, we need to further study the similarity measures we may apply for such definitions that could take into account the specificities of definition and better discriminate between definitions that share much semantics but differ on specific aspects.

Other solutions should also be investigated to handle all definitions that are not associated with a source gloss, in order to benefit from the richness of the original Wiktionary data.

# 8 Bibliographical References

## References

Mikel Artetxe and Holger Schwenk. 2019a. Margin-based parallel corpus mining with multilingual sentence embeddings. In *Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics*, pages 3197–3203, Florence, Italy. Association for Computational Linguistics.

Mikel Artetxe and Holger Schwenk. 2019b. Massively multilingual sentence embeddings for zero-shot cross-lingual transfer and beyond. *Transactions of the Association for Computational Linguistics*, 7:597–610.

Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. 2016. Enriching word vectors with subword information. *arXiv preprint arXiv:1607.04606*.

Christian Chiarcos, Sebastian Hellmann, and Sebastian Nordhoff. 2011. Towards a linguistic linked open data cloud: The open linguistics working group. *Traité Automatique des Langues*, 52(3):245–275.

Paul-Ambroise Duquenne, Holger Schwenk, and Benoît Sagot. 2023. Sonar: Sentence-level multimodal and language-agnostic representations. *Preprint*, arXiv:2308.11466.

Fangxiaoyu Feng, Yinfei Yang, Daniel Cer, Naveen Arivazhagan, and Wei Wang. 2022. Language-agnostic BERT sentence embedding. In *Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)*, pages 878–891, Dublin, Ireland. Association for Computational Linguistics.

Jorge Gracia, Gilles Sérasset, Michael Rosner, Ilan Kernerman, and Katerina Gkirtzou. 2025. Cross-lingual linking representation levels on the web of data. Submitted to NLP journal, preprint at https://zenodo.org/.

Sergio Jimenez, Claudia Becerra, and Alexander Gelbukh. 2012. Soft Cardinality: A Parameterized Similarity Function for Text Comparison. In *SEM 2012: The First Joint Conference on Lexical and Computational Semantics – Volume 2: Proceedings of the Sixth International Workshop on Semantic Evaluation (SemEval 2012)*.

Alex Jones, Isaac Caswell, Ishank Saxena, and Orhan Firat. 2023. Bilex rx: Lexical data augmentation for massively multilingual machine translation. *Preprint*, arXiv:2303.15265.

Armand Joulin, Piotr Bojanowski, Tomas Mikolov, Hervé Jégou, and Edouard Grave. 2018. Loss in translation: Learning bilingual word mapping with a retrieval criterion. In *Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing*.

Klaus Krippendorff. 2025. Content Analysis: An Introduction to Its Methodology, fourth edition. SAGE Publications, Inc., Thousand Oaks, California. Especially chapter 12.

Christian M. Meyer and Iryna Gurevych. 2012. Wiktionary: A new rival for expert-built lexicons? Exploring the possibilities of collaborative lexicography. In Sylviane Granger and Magali Paquot, editors, *Electronic Lexicography*, chapter 13, pages 259–291. Oxford: Oxford University Press.

G. Craig Murray and Rebecca Green. 2004. Lexical knowledge and human disagreement on a wsd task. *Computer Speech & Language*, 18(3):209–222. Word Sense Disambiguation.

Arvind Neelakantan, Tao Xu, Raul Puri, Alec Radford, Jesse Michael Han, Jerry Tworek, Qiming Yuan, Nikolas Tezak, Jong Wook Kim, Chris Hallacy, Johannes Heidecke, Pranav Shyam, Boris Power, Tyna Eloundou Nekoul, Girish Sastry, Gretchen Krueger, David Schnurr, Felipe Petroski Such, Kenny Hsu, and 6 others. 2022. Text and code embeddings by contrastive pre-training. *Preprint*, arXiv:2201.10005.
Nils Reimers and Iryna Gurevych. 2019. Sentence-BERT: Sentence embeddings using Siamese BERT-networks. In *Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP)-IJCNLP*, pages 3982–3992, Hong Kong, China. Association for Computational Linguistics.

Nils Reimers and Iryna Gurevych. 2020. Making monolingual sentence embeddings multilingual using knowledge distillation. In *Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)*, pages 4512–4525, Online. Association for Computational Linguistics.

Gilles Sérasset. 2012. Dbnary: Wiktionary as a LMF based Multilingual RDF network. In *Language Resources and Evaluation Conference, LREC 2012*, Istanbul, Turkey. Nicoletta Calzolari and Khalid Choukri and Thierry Declerck and Mehmet Uğur Doğan and Bente Maegaard and Joseph Mariani and Jan Odijk and Stelios Piperidis.

Gilles Sérasset. 2015. DBnary: Wiktionary as a Lemon-Based Multilingual Lexical Resource in RDF. *Semantic Web – Interoperability, Usability, Applicability*, 6(4):355–361.

Andon Tchechmedjiev, Gilles Sérasset, Jérôme Goulian, and Didier Schwab. 2014. Attaching Translations to Proper Lexical Senses in DBnary. In *3rd Workshop on Linked Data in Linguistics: Multilingual Knowledge Resources and Natural Language Processing*, pages 5–12, Reykjavik, Iceland.

Jörg Tiedemann, Mikko Aulamo, Daria Bakshandaeva, Michele Boggia, Stig-Arne Grönroos, Tommi Nieminen, Alessandro Raganato Yves Scherrer, Raul Vazquez, and Sami Virpioja. 2023. Democratizing neural machine translation with OPUS-MT. *Language Resources and Evaluation*, 58(2):713–755.

Jörg Tiedemann and Santhosh Thottingal. 2020. OPUS-MT — Building open translation services for the World. In *Proceedings of the 22nd Annual Conference of the European Association for Machine Translation (EAMT)*, Lisbon, Portugal.

Amos Tversky. 1977. Features of Similarity. *Psychological Review*, 84(2):327–352.

Bernard Vauquois. 1968. A survey of formal grammars and algorithms for recognition and transformation in machine translation. In *IFIP Congress-68*, pages 254–260, Edinburgh.

Jean Véronis. 1998. A study of polysemy judgements and inter-annotator agreement. In *Proceedings of the Senseval workshop*. Citeseer.

Liang Wang, Nan Yang, Xiaolong Huang, Linjun Yang, Rangan Majumder, and Furu Wei. 2024. Multilingual e5 text embeddings: A technical report. *arXiv preprint arXiv:2402.05672*.

Yinfei Yang, Daniel Cer, Amin Ahmad, Mandy Guo, Jax Law, Noah Constant, Gustavo Hernandez Abrego, Steve Yuan, Chris Tar, Yun-hsuan Sung, Brian Strope, and Ray Kurzweil. 2020. Multilingual universal sentence encoder for semantic retrieval. In *Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations*, pages 87–94, Online. Association for Computational Linguistics.

Xin Zhang, Yanzhao Zhang, Dingkun Long, Wen Xie, Ziqi Dai, Jialong Tang, Huan Lin, Baosong Yang, Pengjun Xie, Fei Huang, and 1 others. 2024. mgte: Generalized long-context text representation and reranking models for multilingual text retrieval. In *Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing: Industry Track*, pages 1393–1412.

## A Availability of Glosses in DBnary Translations

Table 5 gives statistics on the availability of textual glosses in translations in DBnary, by source language. These statistics are computed on the 20250320 version of DBnary.

<table><caption>Table 5: Statistics on the availability of textual glosses (short form designating a lexical sense definition), compared with numeric glosses (giving the lexical sense number) and redundant glosses (glossing giving both a sense number AND a short form of a definition), along with the total number of translations, by languages.</caption><thead><tr><th>Language</th><th>#transl</th><th>#text gloss</th><th>%text gloss</th></tr></thead><tbody><tr><td>Bulgarian</td><td>30166</td><td>1393</td><td>(4.6%)</td></tr><tr><td>Catalan</td><td>455532</td><td>195919</td><td>(43%)</td></tr><tr><td>Danish</td><td>47669</td><td>6852</td><td>(14.4%)</td></tr><tr><td>German</td><td>1066903</td><td>854695</td><td>(80.1%)</td></tr><tr><td>Greek (modern)</td><td>221162</td><td>26236</td><td>(11.9%)</td></tr><tr><td>English</td><td>3315080</td><td>3210642</td><td>(96.8%)</td></tr><tr><td>Finnish</td><td>232696</td><td>232113</td><td>(99.7%)</td></tr><tr><td>French</td><td>1337057</td><td>462564</td><td>(34.6%)</td></tr><tr><td>Irish (Gaeilge)</td><td>10655</td><td>6264</td><td>(58.8%)</td></tr><tr><td>Serbo Croat</td><td>607</td><td>0</td><td>(0%)</td></tr><tr><td>Indonesian</td><td>10207</td><td>0</td><td>(0%)</td></tr><tr><td>Italian</td><td>162006</td><td>112117</td><td>(69.2%)</td></tr><tr><td>Japanese</td><td>216387</td><td>55494</td><td>(25.6%)</td></tr><tr><td>Kurdish</td><td>750368</td><td>73549</td><td>(9.8%)</td></tr><tr><td>Latin</td><td>25049</td><td>5472</td><td>(21.8%)</td></tr><tr><td>Lithuanian</td><td>156364</td><td>156232</td><td>(99.9%)</td></tr><tr><td>Malagasy</td><td>148776</td><td>0</td><td>(0%)</td></tr><tr><td>Dutch</td><td>311370</td><td>249509</td><td>(80.1%)</td></tr><tr><td>Norwegian</td><td>70061</td><td>63482</td><td>(90.6%)</td></tr><tr><td>Polish</td><td>687603</td><td>0</td><td>(0%)</td></tr><tr><td>Portuguese</td><td>312860</td><td>76335</td><td>(24.4%)</td></tr><tr><td>Russian</td><td>760150</td><td>317534</td><td>(41.8%)</td></tr><tr><td>Spanish</td><td>227402</td><td>8289</td><td>(3.6%)</td></tr><tr><td>Swedish</td><td>416744</td><td>335938</td><td>(80.6%)</td></tr><tr><td>Turkish</td><td>196182</td><td>31037</td><td>(15.8%)</td></tr><tr><td>Chinese</td><td>1167362</td><td>276028</td><td>(23.6%)</td></tr></tbody><tfoot><tr><td>Total</td><td>12336418</td><td>6757694</td><td>(54.8%)</td></tr></tfoot></table>