Baleegh at KSAA-CAD 2024: Towards Enhancing Arabic Reverse Dictionaries (2024)

Abstract: The domain of reverse dictionaries (RDs), while advancing in languages like English and Chinese, remains underdeveloped for Arabic. This study attempts to explore a data-driven approach to enhance word retrieval processes in Arabic RDs. The research focuses on the ArabicNLP 2024 Shared Task, named KSAACAD, which provides a dictionary dataset of 39,214 word-gloss pairs, each with a corresponding target word embedding. The proposed solution aims to surpass the baseline performance by employing SOTA deep learning models and innovative data expansion techniques. The methodology involves enriching the dataset with contextually relevant examples, training a T5 model to align the words to their glosses in the space, and evaluating the results on the shared task metrics. We find that our model is closely aligned with the baseline performance on bertseg and bertmsa targets, however does not perform well on electra target, suggesting the need for further exploration.

heading: 1 Introduction
text: While reverse dictionaries have witnessed advancements in languages like English and Chinese e.g. WantWords (Qi et al., 2020), they remain less developed and explored for Arabic. This gap is particularly concerning for a language with a rich linguistic heritage and widespread use. The only ongoing effort in this domain is the First Arabic RD shared task launched in 2023 by King Salman Global Academy for Arabic Language (KSAA)1 (Al-Matham et al., 2023), and the recent KSAACAD (Contemporary Arabic Dictionary) Shared Task for 20242.
Reverse dictionaries are a form of dictionaries where a description yields a set of words from the dictionary that semantically matches the description. A prime use-case is their application in data
1https://arai.ksaa.gov.sa/sharedTask/ 2https://arai.ksaa.gov.sa/sharedTask2024/
exploration and analysis, where reverse dictionaries facilitate the identification of relevant features within complex textual datasets by generating key terms aligned with the text meaning. This enhances the efficiency of data mining and fosters the discovery of new insights (Chen and Zhao, 2022).
The Arabic language, with its intricate morphology and diverse dialects, presents unique challenges for Natural Language Processing (NLP) tasks. RDs are crucial tools for language learners, translators, and researchers, enabling them to identify words based on their meanings or descriptions.
Recent studies have explored various approaches to RD. In (Elbakry et al., 2023), the authors demonstrated success as the winning solution in the 2023 shared task (Al-Matham et al., 2023) using an ensemble of fine-tuned BERT models. Their results formed the SOTA baseline for this 2024 shared task. In (Qaddoumi, 2023), authors focused on enhancing Arabic word embeddings through a modified BERT model and data augmentation while in (Sibaee et al., 2023), authors utilized a SemiDecoder architecture with an SBERT encoder for effective word definition encoding.
In other languages, Authors of (Mane et al., 2022) proposed a unique approach using mT5 for Indian languages, while authors in (Ardoiz et al., 2022) emphasized the importance of high-quality lexicographic data for optimal RD model performance. For English, the authors in (Chen and Zhao, 2022) embedded both the definitions and words into the same shared space using transformer-based architectures to optimize the model across both tasks simultaneously. The model demonstrated superior performance in RD tasks, achieving high accuracy and consistency over previous methods.
In this paper, the use of a RD is cast as a supervised learning task and the main questions addressed are:
704
1. What is the impact of enriching the definitions in the KSAA-CAD dataset with contextual examples on the performance of the proposed Arabic RD model?
2. Are there any pre-trained architectures other than BERT that have similar good performance on Arabic RD?
We leverage the new KSAA-CAD dataset, which is shared exclusively with the participants of the 2024 RD Shared Task (Alshammari et al., 2024), comprising word-gloss pairs with corresponding target word embedding, alongside a SOTA baseline results. Our goal is to surpass or match the baseline performance. The methodology goes through 2 steps:
1. Data Enrichment: The dataset will be enriched with contextually relevant examples to enhance the model’s understanding of word meanings. This will be achieved by leveraging a large Arabic text corpus to curate examples for each word-gloss pair.
2. Pre-trained T5 Model Adaption: A pretrained T5 model (Xue et al., 2021) will be fine-tuned on both the original and enriched KSAA-CAD dataset. Given the novelty of the T5 model and the scarcity of research on it within the Arabic RD literature, we set out to explore its capabilities in this specific domain.
The code utilized in this study has been made available on GitHub3 to ensure the reproducibility of the experimental results.

heading: 2 Dataset
text: The KSAA-CAD dataset is an Arabic dictionary dataset containing 39,214 entries, collected from various Arabic dictionaries.
The dataset is split into train, dev and test sets, as seen in Table 1, with 3 features named: word, gloss, pos, and 3 target embeddings named: electra, bertseg, bertmsa. Table 2 demonstrates a sample entry from the dataset.
3https://github.com/pr-Mais/ksaa-cad-2024

heading: 2.1 Features engineering: enriching the dataset
text: As noticed while performing data exploration, the glosses are often short and formal descriptions written by expert linguists. Table 3 shows 2 words with short and concise glosses, making its usage unclear, and might result in a vague understanding of the word.
Average users are unlikely to provide such precise descriptions, on the contrary, user queries might lack any key words that could identify the target word or set of words. Humans exhibit remarkable facility in acquiring new vocabulary from context early in childhood (Kilian et al., 1995). Therefore, and inspired by that capability, we propose that in order to enhance the model’s ability to align words and descriptions from user queries, we would need to provide the model with more contextually relevant examples for each word-gloss pair.
The manual retrieval of contextually relevant examples is a laborious and resource-demanding task, which, given the short time frame of this experiment, is not a feasible solution, therefore the need to find an automatic way to curate examples from publicly available Arabic datasets.
We used the Arabic Wikipedia Embeddings4
dataset from the Embedding Archives project by CohereAI, which contains 3.1 million entries from Wikipedia, each entry containing a text, and the embedding of that text, alongside other metadata. Text embeddings in the dataset are achieved through CohereAI’s multilingual-22-125 semantic embeddings model, trained for multilingual comprehension encompassing 101 languages including Arabic. This closed-source model is accessible via Cohere’s API (Kamalloo et al., 2023).
To curate a number of examples for each word, firstly we append the definition to the word in a single input string, seprating them by a colon, then we embed the resulting string using multilingual-2212 model, which is the same model that was used to embed Wikipedia’s text, then perform a vector search using cosine similarity 1, to look up the top
4Cohere/wikipedia-22-12-ar-embeddings 5https://cohere.com/blog/multilingual
5 closest entries to the given word:gloss pairs. Finally, the enriched dataset will encompass a total of 156,860 examples, distributed over 31,372 words in the training set, each word having 5 examples as a single feature.
CosineSimilarity = A ·B
∥A∥ × ∥B∥ (1)
Table 3 demonstrates a data point with a retrieved example. The example may not directly contain the lemma H. @
Y», however the surrounding context establishes a clear semantic relationship with the word’s meaning.

heading: 2.2 Data pre-processing
text: The KSAA-CAD dataset was prepared for training with two steps. Firstly, we created a new input by combining the glosses and their corresponding 1st example into a single input string, to enrich the training data. Secondly, each of the inputs (gloss only, and gloss+example) were tokenized using a pre-trained SentencePiece tokenizer. Figure 1 visualizes the result of tokenizing an Arabic sentence from the dataset.

heading: 3 System
text: Recent studies on RDs have utilized pre-trained transformers, as seen in literature, most of which focused on BERT-based transformers for Arabic. Consequently, the potential for exploring other architectures and pre-trained models for the Arabic language remains intact.
Our model architecture is based on AraT5 V26 (Nagoudi et al., 2021), a fine-tuned T5 transformer model for Arabic. The input is processed by the AraT5 V2 encoder, producing a hidden state matrix. A pooling layer 3 then converts this matrix into a fixed-length vector, which is subsequently fed into a linear layer for the final prediction. This architecture enables effective utilization of AraT5 V2’s language understanding capabilities for RD tasks.
6https://huggingface.co/UBC-NLP/ AraT5-base-1024
pool = ∑
j,k(Oijk ·Aijk)∑ j Aij
We train 2 models for 2 epochs with the same setup but different inputs, the number of epochs being low due to the over-fitting noticed when training for higher than 2 epochs. The first model is trained on only glosses, while the other on the glosses merged with example retrieved in step 2.1. Mean Square Error is used as a loss function, and Adam as an optimizer with a learning rate value of 3e-5.

heading: 4 Results and Discussion
text: Table 4 presents the final performance across the shared task metrics on the development and test sets. Notably, incorporating example input alongside gloss definitions resulted in a marginal decrease in performance compared to utilizing gloss alone. This observation may be attributed to the automated example retrieval process, which could potentially introduce contextually irrelevant instances, thereby confounding the model’s ability to differentiate between the true meaning of a word and its unrelated example. Furthermore, the finite availability of resources for example extraction might have led to instances where certain words were absent from the Wikipedia dataset, resulting in the retrieval of contextually inappropriate text.
Overall, our models did not surpass the baseline results across all embedding types. Nonetheless, we achieved competitive performance, particularly in predicting bertseg and bertmsa embeddings, which indicate the model is capable of aligning the words from both spaces to glosses, suggesting
AraT5 V2 has a similar vocabulary to the models which produced these embeddings.
While incorporating contextual examples did not improve the results on this task, we believe this is regarded to several reasons including the model single-layer architecture and shared task targets, which were obtained from the definitions only, causing the model to over-fit on the retrieved examples. Another reason may be attributed to the quality of the examples, which may not always provide meaningful context for the word:gloss pairs.
The model’s performance could be enhanced through hyper parameter tuning, exploring advanced architectures, and potentially training some layers from the base encoder model.

heading: 5 Conclusion
text: Although not exceeding the baseline on all metrics, the initial results show promise for our approach. Further development could lead to exceeding baseline performance across all metrics. AraT5 V2 generally performed well on the given RD task,
and further experimentation could unveil more of it potential.
Future work could involve several enhancements to the current model. One avenue for improvement is refining the data enrichment process, potentially by incorporating diverse resources beyond Wikipedia to enhance the model’s knowledge base and adaptability. Additionally, exploring more sophisticated architectures and fine-tuning hyperparameters could further optimize the model’s performance and accuracy. Finally, a promising direction would be to train the model’s encoder-decoder architecture to directly predict word sequences instead of relying on target embeddings, potentially improving the generation of coherent and contextually relevant text.

heading: Future Work
text: Future work could involve several enhancements to the current model. One avenue for improvement is refining the data enrichment process, potentially by incorporating diverse resources beyond Wikipedia to enhance the model’s knowledge base and adaptability. Additionally, exploring more sophisticated architectures and fine-tuning hyperparameters could further optimize the model’s performance and accuracy. Finally, a promising direction would be to train the model’s encoder-decoder architecture to directly predict word sequences instead of relying on target embeddings, potentially improving the generation of coherent and contextually relevant text