ASOS at KSAA-CAD 2024: One Embedding is All You Need for Your Dictionary (2024)

Abstract: Semantic search tasks have grown extremely fast following the advancements in large language models, including the Reverse Dictionary and Word Sense Disambiguation in Arabic. This paper describes our participation in the Contemporary Arabic Dictionary Shared Task. We propose two models that achieved first place in both tasks. We conducted comprehensive experiments on the latest five multilingual sentence transformers and the Arabic BERT model for semantic embedding extraction. We achieved a ranking score of 0.06 for the reverse dictionary task, which is double than last year’s winner. We had an accuracy score of 0.268 for the Word Sense Disambiguation task.

heading: 1 Introduction
text: A Reverse Dictionary (RD) takes a phrase that describes a specific concept as input and provides words whose definitions match the entered phrase. This is different from the traditional function of a forward dictionary, which maps words to their meanings or definitions, helping users understand unfamiliar terms. For example, a forward dictionary would tell the user that (YJ. Ë @) means ( ªK. úÎ« é ªK. Zú æ Ë @ I. »P - things are superimposed on top of each other) whereas an RD allows the user to input the phrase ( ªK. úÎ« é ªK. Zú æ
Ë @ I. »P - things are superimposed on top of each other) and would likely produce the word (YJ. Ë @) along with other words having similar meanings as the output. While numerous RDs exist for languages like English, Japanese, and French, Arabic has seen limited development in this area, often due to the absence of substantial datasets with word definitions.
RDs are extremely helpful in overcoming the tipof-the-tongue phenomenon, where people struggle to find the right word to express their thoughts. Additionally, RDs are essential in applications of NLP
such as evaluating sentence representations, mapping text to specific entities, answering questions, and retrieving information. It is crucial to understand the precise meanings of words in context. This understanding is improved by Word Sense Disambiguation (WSD), which helps to identify the most relevant meaning of a word in a given context, thus enhancing the effectiveness and accuracy of reverse dictionaries in handling complex queries. For example, the phrase "to come together" could correspond to various options like "meet," "gather," "assemble," and more. Each of these words, while synonyms, may be more appropriate in different contexts, highlighting the importance of WSD. WSD plays a significant role in improving the accuracy of RD systems by helping to determine which word meanings are most relevant based on contextual cues.
This paper describes our participation in the Contemporary Arabic Dictionary Shared Task, organized by the King Salman Global Academy for Arabic Language (KSAA-CAD). The task aims to tackle challenges in the Arabic linguistic domain, focusing on two main areas: RD and WSD. In this study, we extend the methodologies previously developed in our work (Sibaee et al., 2023), utilizing an expanded model further to enhance the performance and capabilities of our approach. By participating in both tasks, we aim to contribute valuable insights and advancements to the fields of RD and WSD, particularly within the context of the Arabic language, where such resources have historically been scarce.

heading: 2 Related Work
text: Previously, researchers have tackled the RD problem using a traditional approach called semantic analysis, utilizing WordNet (Méndez et al., 2013). They employed semantic similarity measurements to determine how similar two words are, using
697
distance-based similarity measures to establish connections between the term and the input words in a graph (Thorat and Choudhari, 2016). Recently, many researchers have been utilizing embedding techniques alongside advanced neural networks to enhance the generation of reverse dictionaries. Pilehvar (2019) implemented a combination of Bidirectional Long Short-Term Memory (BiLSTM) and Cascade Forward Neural Network (CFNN) to improve the performance of neural RDs. To assess whether a proposed neural network framework is universally effective across all languages, Bendahman et al. (2022) employed sequential models with a variety of neural networks, including embedding networks, denser networks, and Long Short-Term Memory (LSTM) networks.
For Arabic RD, Al-Matham et al. (2023) proposed the first shared task focused on developing RD systems for the Arabic language. This encompassed two subtasks: Arabic RD and Crosslingual RD. The aim was to enable users to locate words based on their meanings or definitions in both Arabic and English. Sibaee et al. (2023), proposed a method that converts word definitions into multidimensional vectors and trains them using the SemiDecoder model. The proposed system secured the 2nd place based on the rank metric for target embeddings (Electra and SGNS). Elbakry et al. (2023) proposed a method utilizing an ensemble of fine-tuned Arabic BERT-based models to predict the word embedding for a given definition. The final representation is obtained by averaging the output embeddings from each model within the ensemble.
In studies on the WSD task, Barba et al. (2021) introduced CONtinuous SEnse Comprehension (CONSEC), a novel approach designed to address limitations in traditional supervised neural WSD systems. In conventional systems, each word is disambiguated independently without considering the explicit senses assigned to nearby words. Focusing on eight languages, CONSEC introduced a feedback loop strategy that conditions the disambiguation process on both the context and possible meanings of a word, as well as the senses assigned to adjacent words. Kaddoura and Nassar (2024) proposed the development of a dataset for Arabic WSD, utilizing an ensemble learning architecture that combined multiple BERT models. The ensemble model aimed to address word ambiguity effectively and enhance classification performance by integrating various models, each equipped with
unique feature sets. These feature sets included part-of-speech (POS) information, word frequency counts, and weak supervision annotations, which contributed valuable contextual information to the disambiguation process. Saidi and Jarray (2023) tackled the task of Word Sense Induction (WSI), a similar NLP task that plays a pivotal role in understanding and interpreting the nuanced meanings of language. Their proposed approach for Arabic WSI seeked to group sentences based on their semantic similarity. This approach involved a two-stage clustering method. In the first stage, sentences were encoded using BERT or DistilBERT to generate sentence embeddings. In the second stage, clustering algorithms like K-means and hierarchical agglomerative clustering were used to partition sentences based on their semantic similarity. The model specifically employed the K-means algorithm to cluster the sentence embedding vectors.

heading: 3 Methodology
text: In this section we will describe the used methodology that was used in the paper by starting of by the description of the dataset and our approach to expand subset of it, Then the explanation about the used model and how we expanded our approach in our past paper (Sibaee et al., 2023), and for the second task WSD we will mention it in end of each section.

heading: 3.1 Data Description

heading: 3.1.1 RD Task
text: This year, the task dataset was provided as follows: 31,372 samples for training, 3,921 samples for development, and 3,922 samples for testing. Each training and development data sample includes the definition (gloss) and the targeted word in both text and embedding formats. Moreover, the given embeddings were from three different models: Electra (256-d) (Antoun et al., 2021), camelBERT-MSA (768-d) (Inoue et al., 2021), and AraBERTv2 (768- d) (Antoun et al., 2020). We expanded the dataset using the previous year’s dataset and only included the ArElectra embeddings, which were provided last year and comprised approximately 84000 samples, including training and development data. Due to the limited data provided this year, we combined the training and development datasets. This decision was based on our analysis and insights from the previous year, which will be discussed in the
following subsection. For this reason, we will not report results based on the development subset.

heading: 3.1.2 WSD Task
text: The WSD dataset is provided in JSON format and consists of two main components: the WSD context gloss mapping and the dictionary data. The WSD context gloss mapping includes elements such as word, context (the sentence or phrase providing semantic clues), context ID, and corresponding gloss ID. The dictionary data comprises word, gloss (the definition of the word aiding in disambiguation), and gloss ID. The dataset is distributed across three subsets: 22,404 entries for training, 2,801 for development, and 2,801 for testing, with a total of 15,865 gloss entries in the WSD dictionary.

heading: 3.2 Models
text: In previous work (Sibaee et al., 2023), our approach centred on utilizing a consistent sentence transformer model while exploring various neural network architectures. In contrast, the current study adopts an inverse strategy, where we standardized the neural network architecture. Firstly, we will explain the neural network architecture for both tasks and then discuss the sentence transformer models used in our experiments.

heading: 3.2.1 Neural Network Architecture for RD
text: After conducting extensive experiments, we optimized our neural network architecture by selecting a semi-encoder structure with four hidden layers. These layers decrease in size by a factor of two, starting from eight times the input size down to the input size itself (input=s, h1=s*8, ..., h4=s, output). We used GELU activation and adjustable dropout rates ranging from 0.2 to 0.4. This configuration was standardized for all experiments, with the only modification being the optimizer, which was changed to AdamW. Additionally, we adjusted the learning rate from 1e-3, used in our previous work, to 1e-4 after finding the former to be too high for this architecture. We maintained this learning rate across all experiments. For abstract architecture please look at figure 1.

heading: 3.2.2 Neural Network Architecture for WSD
text: In this task, we carefully designed a two-stage neural network architecture. Initially, each input is processed independently (where the targeted word and the context are both embedded using ArBERTv2 model) through its dedicated neural network layer,
consisting of an input layer, a single hidden layer, and an output layer. The outputs from these individual networks are then concatenated and fed into a subsequent neural network that mirrors the structure of the first, including an input layer, a hidden layer, and an output layer (which has the real output embedding using the ArBERTv2 model also). The final output is a precisely defined 768- dimensional vector. Detailed discussion on the input encoding will be presented in the upcoming section, providing deeper insight into the methodology employed. Worth mentioning: the texts are embedded using ArBERTv2 model in the section 3.2.3 in ArBERTv2 paragraph.

heading: 3.2.3 Encoding models for the both tasks
text: Instead of only using the ’distiluse-basemultilingual-cased-v1’ model (Reimers and Gurevych, 2019) to encode texts, the creators of this model have recently developed additional sentence transformer models, now available on the Hugging Face platform. We conducted a comprehensive evaluation of these models to assess their performance enhancements; details of these models are in Table 1.
Sentence transformers model Dimension
We employed various sentence transformer models in our experiments. The LaBSE model (Feng et al., 2022), a language-agnostic BERT sentence embedding model, encodes text into highdimensional vectors. This model is trained and optimized to produce similar representations exclusively for bilingual sentence pairs that are transla-
tions of each other, and it supports 109 different languages.
The MiniLM-L12-v2 model1 is another sentence transformer that maps sentences and paragraphs to a 384-dimensional dense vector space, making it suitable for tasks such as clustering and semantic search. Similarly, the mpnet-base-v2 model2 maps sentences and paragraphs to a 768-dimensional dense vector space and can be used for similar tasks.
The Distiluse-base-multilingual-cased models3, both v1 and v2, map sentences and paragraphs to a 512-dimensional dense vector space. We specifically used the v1 model because it was utilized in our previous research.
Additionally, in our final experiments, we incorporated ArBERTv2 (Abdul-Mageed et al., 2021), (Elmadany et al., 2022), a BERT model (Devlin et al., 2019) trained on approximately 250 GB of Modern Standard Arabic (MSA) text. To represent an entire sentence, we calculated the mean of the token embeddings produced by the model:
x = 1
N
N∑
1
ei
where x is the input in the neural network and ei ∈ R768 is the embedding of each token from the ArBERTv2 model, Also for the WSD task we used the same model to represent the context, targeted word, and the output from the dictionary.

heading: 4 Result
text: In this section we will describe our results from training the model mentioned in section 3.2.1 but before that we will mention an experiment that we did while working on the task.

heading: 4.1 RD results

heading: 4.1.1 Sentence Transformers Results
text: This section presents the results of the sentence transformer encoders method, focusing on the RANK metric, as shown in Table 2. For the Electra model, LaBSE achieved the best RANK score (0.17), followed by mpnet and L12 with scores of 0.22, and CasedV2 and CasedV1 scoring 0.25 and
1https://huggingface.co/sentence-transformers/allMiniLM-L12-v2
2https://huggingface.co/sentencetransformers/paraphrase-multilingual-mpnet-base-v2
3https://huggingface.co/sentence-transformers/distilusebase-multilingual-cased-v2
0.21, respectively. The bertMSA model demonstrated improved RANK scores, with CasedV2 leading at 0.403, followed by mpnet (0.371), L12 (0.368), and LaBSE (0.367). In the BERTseg model, CasedV2 again achieved the highest RANK score (0.416), with mpnet and LaBSE both scoring 0.399, and L12 and CasedV1 scoring 0.397 and 0.406, respectively. Overall, CasedV2 and LaBSE consistently performed well on the RANK metric across all models, demonstrating their effectiveness in this task.

heading: 4.1.2 ArBERTv2 Results
text: This section presents the best results from our final proposed method using the ArBERTv2 embedding, as displayed in Table 3. The Electra model achieved the highest RANK score (0.0644) after 500 epochs. Following closely, the bertMSA model attained a RANK score of 0.1484 after 1000 epochs. Although the bertSEG model secured the highest COS score (0.830) and a noteworthy MSE score (0.059), it obtained a RANK score of 0.198 with 500 epochs. Overall, the ArBERTv2 embedding exhibited strong performance across different models, with Electra delivering the best results based on the RANK metric.

heading: 4.2 WSD Results
text: Due to limited time, we conducted only a few experiments with the ArBERTv2 embedding for the WSD task, as shown in Table 4. Using the same model described in Section 3.3.2, we varied the number of epochs and the learning rate. Model 2 achieved the highest accuracy (0.268) and mrr@2 (0.297) with a learning rate of 1e-3 and 75 epochs. These preliminary results indicate promising directions for future research, warranting further investigation.

heading: 4.3 Error Analysis
text: In this section, we will analyze the outcomes and results by discussing two different aspects. First, we will examine the reasons why the ARBERT model outperformed other sentence transformer models. Second, we will evaluate the provided dataset, considering the common rule that "the quality of the data determines the quality of the
model."

heading: 4.3.1 Models Analysis
text: The models used in this paper are the Arabic BERT model (ArBERTv2) and sentence transformers. BERT models, in general, have shown significant results in understanding language semantics. As noted by (Wang et al., 2024), "The bidirectional property of BERT-based models enables the learning of the context (i.e., semantic information) of unlabeled texts from both the left and the right sides. It lays the foundation for obtaining contextualized embeddings."
We believe this bidirectional nature is the main advantage of BERT models over sentence transformers, as their underlying targets and training approaches differ. According to (Reimers and Gurevych, 2019), the primary focus in training sentence transformers is to fine-tune BERT and RoBERTa models for sentence similarity tasks using cosine similarity with a pooling layer at the end. Additionally, the multilingual models provided by sentence transformers were trained in different languages, leading to a somewhat noisy embedding distribution due to language mixing. In contrast, the ArBERTv2 model was trained exclusively on a large and clean Modern Standard Arabic (MSA) text dataset, resulting in more expressive embeddings in our experiments.
Moreover, it is worth mentioning that the BERTseg model (AraBERTv2) has shown better results in the MSE and cosine similarity criteria. These improved results can be attributed to the underlying ArBERTv2 model, which was used to extract the embeddings from the dataset.

heading: 4.3.2 Dataset Analysis
text: Research in the field of large language models (LLMs) has consistently shown that data quality is a major factor in the performance of these models (Naveed et al., 2024). To evaluate the quality of the given dataset, we conducted an analysis by randomly sampling 100 entries from each dataset segment. The main criticisms identified are as follows:
1. Lack of a systematic and logical methodology for definitions.
The definitions vary significantly in their formulation. Some definitions focus on morphological forms, while others highlight a specific meaning derived from the general mean-
ing of the Arabic root of the word. Additionally, some definitions are extracted from scientific contexts without specifying the targeted field, and some definitions use the word(s) intended to be defined within the definition itself, which is not a clear method of defining. examples are given in the next criticisms.
2. Reliance on word morphological forms rather than defining their meanings. Number of examples could be mentioned like ÉÒ where it only mentioned the éK. É Ò Ó Èñª ®ÖÏ @ð É Ò Ó ñê ̄ , C Ò where
non of these are a meaning. Also some examples: ¨ñ JK , Yª .
3. Mentioning specific definitions without emphasizing the general absolute meaning. A lot of the definitions are written from a specific fields. For example: ú
̄ Q« word they mentioned the legal definition. some other definitions the chosen meanings are very specific and narrowing the general meaning for example: àYK. the definition H. ñ JË @ àYK. instead of the famous meaning ’body’, Õæ m.Ì'@. Also, Q Kð the definition is about the mathematical one IÊ JÖÏ @ Q Kð istead of the abstract famous meaning of the bow " éÊJ.k ñ ®Ë@ Q Kð"
It is important for the dataset to be created in accordance with the logical rules of Arabic definitions, as described in both contemporary and classical texts on logic. Notable examples include "Tahzeeb Al Mantiq" (Refinement of Logic) by Al Taftazani and "Ta’rifaat" (Definitions) by Al Jurjani (Jurjānı̄, 1969). These works offer a systematic and logical framework for creating definitions, ensuring consistency and precision in the dataset, and it worth mentioning the field of the definition so the extraction get more detailed and specific For example in the "Ta’rifaat" book the word H. ñk. ð he mentioned four aspects of the definitions ú æê ® ̄ , ú Î ®« ,ù K @X @ , ú «Qå .

heading: 5 Conclusion
text: In this paper, we conducted a comprehensive experimental analysis on sentence transformers and ArBERTv2 to create semantic embeddings for RD and WSD tasks, training a fixed neural network
with four hidden layers. Our models achieved first place in both tasks, with a RANK score of 0.06 for the RD task and an accuracy score of 0.268 for the WSD task. Additionally, we performed a dataset analysis to develop a linguistic, logical methodology for building structured definitions for both tasks.
For future work, we plan to explore multiple methods for generating embeddings from the ArBERTv2 model and aim to build a larger and more comprehensive dataset to enhance our understanding of task semantics.

heading: Acknowledgments
text: We would like to thank RIOTU labs at Prince Sultan University in Saudi Arabia for allowing us to use the lab server.

heading: Future Work
text: For future work, we plan to explore multiple methods for generating embeddings from the ArBERTv2 model and aim to build a larger and more comprehensive dataset to enhance our understanding of task semantics