rematchka at ArabicNLU2024: Evaluating Large Language Models for Arabic Word Sense and Location Sense Disambiguation (2024)

Abstract: Natural Language Understanding (NLU) plays a vital role in Natural Language Processing (NLP) by facilitating semantic interactions. Arabic, with its diverse morphology, poses a challenge as it allows multiple interpretations of words, leading to potential misunderstandings and errors in NLP applications. In this paper, we present our approach for tackling Arabic NLU shared tasks for word sense disambiguation (WSD) and location mention disambiguation (LMD). Various approaches have been investigated from zero-shot inference of large language models (LLMs) to fine-tuning of pre-trained language models (PLMs). The best approach achieved 57% on WSD task ranking third place, while for the LMD task, our best systems achieved 94% MRR@1 ranking first place.

heading: 1 Introduction
text: Natural Language Understanding (NLU) is a crucial field in language processing that aims to comprehend and interpret human language. Word Sense Disambiguation (WSD) plays a crucial role in resolving multiple meanings and nuances of words, but applying WSD to the Arabic language presents unique challenges due to its rich morphology, dialectal variations across different regions, and polysemy. A special case within WSD is Location Mention Disambiguation (LMD), which involves disambiguating location references in Arabic texts considering diverse geographical, historical, and cultural contexts. Developing effective NLU techniques for Arabic, particularly for WSD and LMD tasks, is of great significance for various applications such as information retrieval, machine translation, and natural language understanding systems. However, there are not enough good linguistic resources and datasets specifically designed for these tasks in Arabic, making it difficult to train and test models, affecting their accuracy and reliability. This paper outlines our solution to the ArabicNLU Shared-Task (Khalilia et al., 2024), which involves two tasks. The first task is Word Sense Disambiguation (WSD), where the objective is to determine the correct meaning of a target word within a given context by selecting from a set of candidate senses (i.e., glosses or definitions) associated with the target word. The second task is Location Mention Disambiguation (LMD). In LMD, the goal is to match each location mention within a post or tweet to the correct toponym from a geo-positioning database that contains a list of toponyms. This matching process aims to find the toponym in the database that accurately represents the mentioned location. To address the WSD challenge, we conducted experiments with large language models (LLMs) in a zero-shot setting. In this approach, we have instructed the model to select the appropriate sense from a list of senses given the context, and target word. Additionally, we explored the effectiveness of fine-tuning models like MARBERT and AraBERT for improved performance in WSD. For the LMD task, we have experimented with LLMs in a zero-shot setting and in the context of the Retrieval-Augmented Generation (RAG) framework. In the Zero-shot setting, we provided instructions to the model to translate the location and provide the corresponding country name as a means to obtain the accurate toponym from the Geopy framework. In the RAG framework, we maintained a record of the context, location, country, and the valid toponyms that were available. Retrieving the toponym involved evaluating the similarity score between the new context and location, and comparing it with the knowledge base to select the highest-scoring match. The experiments aimed to determine which approach yielded better results and provided insights into the effectiveness of LLMs in solving WSD and LMD challenges in Arabic natural language understanding.
383
The rest of the paper goes as follows: section 3 gives an overview of the dataset, section 4 discusses the proposed methods, section 5 shows experimental results, and section 6 concludes the paper.

heading: 2 Related Work
text: This section of the paper provides a literature review on Arabic word sense disambiguation and location mention disambiguation. It discusses the importance of these tasks in accurate and contextaware natural language processing in Arabic. The review focuses on existing approaches, resources, and evaluation methodologies employed in the field. (Malaysha et al., 2023) proposed ArabGlossBERT dataset, which consists of 167K context-gloss pairs collected from Arabic dictionaries, is commonly used but relatively small. The authors present an enriched version of the dataset achieved through machine back-translation, increasing its size to 352K pairs. They evaluate the impact of augmentation using various data configurations to fine-tune BERT for the target sense verification (TSV) task. The accuracy ranges from 78% to 84% across different data configurations, with some improvements observed for specific parts-of-speech (POS) in certain experiments. (Al-Hajj and Jarrar, 2021) focuses on fine-tuning BERT models for Arabic WSD. The authors constructed a dataset of labeled Arabic context-gloss pairs, extracted from the Arabic Ontology and a lexicographic database. Each pair was labeled as true or false, and target words in each context were identified and annotated. Three pretrained Arabic BERT models were then fine-tuned using this dataset. The authors also experimented with different supervised signals to emphasize target words in context. The experiments yielded promising results, achieving an accuracy of 84%, even when dealing with a large set of senses in the experiment. (Kaddoura and Nassar, 2024) introduces a new dataset for Arabic WSD and proposes a novel approach using BERT. The proposed WSD system outperforms existing methods, achieving an approximate F1-score of 96%. The effectiveness of WSD is demonstrated in a case study involving sentiment analysis as a downstream task. (Saidi et al., 2023) addresses the challenges of WSD in Arabic due to limited resources and semantic sparsity. The authors propose WSDTN, a manually annotated corpus consisting of 27,530 sentences collected from diverse sources. Each sentence includes a target word and its corresponding sense.
They also present a transformer-based model for disambiguating new words and evaluate the corpus’s performance. The baseline approach achieves an accuracy of approximately 90%. (Saidi and Jarray, 2023) presents a hybrid approach for Arabic Word Sense Disambiguation (AWSD) by combining a single-layer Convolutional Neural Network (CNN) with contextual representation using BERT. The proposed approach utilizes a concatenation of BERT models as word embeddings to capture both target and context representations. Experimental results demonstrate that the proposed model surpasses state-of-the-art approaches, achieving an accuracy of 96.42% on the Arabic WordNet dataset, thereby improving the performance of WSD in Arabic languages.

heading: 3 Data
text: In subtask 1, the development dataset provided by the organizers is a subset of the SALMA corpus (Jarrar et al., 2023). The training dataset consisted of 100 sentences, while the test set comprised 1,340 sentences. As for subtask 2, the training dataset used in this task was derived from the IDRISI-D dataset (Suwaileh et al., 2023), which contained 2,170 sentences. The validation and test sets were composed of 333 and 791 sentences, respectively. Within the training dataset, there were a total of 3,893 location mentions, with 763 unique mentions.

heading: 4 System Overview
text: LLMs have proven to excel in a wide range of linguistic tasks, due to their extensive training on vast amounts of internet data. With their high capabilities in tasks such as generation and reasoning. LLMs have extensively been studied and utilized in languages like English. However, their potential in the Arabic language has not been fully explored, presenting a significant opportunity to investigate and harness LLMs for Arabic NLU tasks. To seize this opportunity, we conduct experiments on several large language models that have been trained on data of multiple languages including Arabic, namely LLama3, WizardLM-2 (Xu et al., 2023), AceGPT (Huang et al., 2023), and openchat (Wang et al., 2023), to assess their performance in word sense disambiguation and location mention disambiguation tasks. In addition, we compared their performance with pre-trained language models (PLM) such as MARBERT (Abdul-Mageed et al., 2021)
and AraBERT (Antoun et al.). In the following section, we provide detailed information about their training and inference setup.

heading: 4.1 Subtask 1
text: To address this task, we explored two main approaches: the zero-shot setting with large language models (LLMs) and fine-tuning using pre-trained language models (PLMs). Extensive evidence has shown that by finetuning LLMs to adhere to natural language instructions, their performance can be significantly enhanced on tasks that were previously unseen, particularly in zero-shot and fewshot settings (Li et al., 2023; Liu et al., 2023). In our study on WSD task, we employed zeroshot settings for LLama3, WizardLM2, LLama3Instruct, and AceGPT-7B. Different instructions and pre-processing techniques have been explored to improve LLMs performance. Refer to the Appendix A for the prompt/instructions and architecture. Moreover, previous studies have indicated that pre-trained language models (PLM) can outperform LLMs in certain specific tasks (Liusie et al., 2024; Kwon et al., 2023; Almazrouei et al., 2023). Building upon this insight, we decided to fine-tune two PLMs, namely MARBERT and AraBERT, in three distinct settings to evaluate their performance in word sense disambiguation. The first setting utilizes the dataset provided by the organizers of the shared task. In this setting, the development dataset is divided into training and dev subsets using an 80%-20% split. The second setting incorporates the dataset mentioned in the work by (Kaddoura and Nassar, 2024), along with the dev-dataset provided by the shared task organizers. In this setting, the validation set is 20% of both datasets. However, the validation set is carefully curated to ensure uniqueness based on words and senses that have not been encountered in the training split. The third setting involves the combination of three datasets: 1- provided by organizers, 2- in (Kaddoura and Nassar, 2024) work, 3- Razzaz 1. In this setting, the validation set constitutes 20% of the entire combined dataset. The training for MARBERT and AraBERT goes as follows, the problem is formulated as a binary classification problem, for a given sentence, the word and sense/meaning model has to predict 1 if it is the correct meaning or zero if it is not. The input format is defined as follows: a window size of 10 words surrounding the target
1data available at https://github.com/MElrazzaz/ Arabic-word-sense-disambiguation-bench-mark
word, followed by a separator, the target word itself, another separator, and finally, the sense being conditioned on for prediction. During inference as shown in Figure 1, we select the sense/meaning that gives a high probability score to be true.

heading: 4.2 Subtask 2
text: In this task, the geo-positioning database contains a collection of location data with associated coordinates or geographic information. The toponyms refer to the names or labels assigned to these specific locations within the database. The task involves retrieving and ranking these location toponyms accurately based on the content or context provided in a given post. The goal is to identify the most relevant and appropriate location names that align with the information mentioned in the post. The architecture in this subtask is inspired by retrieval augmented systems (RAG). We first build a database from the training dataset composed of text provided, location mentioned in Arabic and English, country location in Arabic and English, and toponyms. To build this database, at first, we query LLama3 whether the given word is most likely a city or country name. If it’s a country name we ask for its English equivalent and save it in the database. If it is not a country name, we query LLama3 to give the name of the most likely country, where this location would be in the given correct context (post). Then we ask LLama3 to translate both location and country names in English. The reason behind it is that GeoPy (tool) that is responsible for retrieving toponyms given names of cities or locations is more sensitive to the English language than Arabic. For instance, it could not retrieve the location of àA JJ. Ë éK Y JºË@ èPA ® Ë@ however, was able to retrieve it when given “Canadian Embassy, Lebanon". Another example toponyms retired by Geopy for location á£AJ. Ë @Q ®k doesn’t match any of toponyms in the training set for this location, however, when presented with its English equivalent “al-Hafar alBatin, Saudi Arabia" the tool was able to retrieve all corresponding toponyms correctly. During inference, as shown in Figure 2, at first for each location mentioned, we determine whether the location is a country name or a city name, or a location in a country by querying LLama3. If it’s a city/location name we ask LLama3 to give the most likely country that this location should be in. Then we match with what is in the database based on context, country, and location relevance. The cosine similarity
score is calculated between the target query, location, and information in the database. If there is relevance (similarity score higher than 0.8), then we return back toponyms stored in the database. If there is no relevance, Geopy is queried based on the English version, if there is no response, then it is queried by the Arabic name. The query contains the location and country name. Refer to the Appendix B for the prompt/instructions and architecture.

heading: 5 Results and Discussion
text: This section presents the model’s performance during the development and testing phases using the official metrics. Additionally, an error analysis is carried out to pinpoint the shortcomings of the presented models. For subtask 1 the official metric is the macro average F1-score, while for subtask 2 the official metric is the mean reciprocal rank (MRR@k), where k indicates the cutoff of the retrieved ranked list of candidate toponyms. For devphase results refer to Appendix C.

heading: 5.1 Test-phase results
text: Table 1 shows the results of developed models for subtask1. LLama3-Instruct model with a limited context window of ±10 words around target words showed the best performance with 57.52% accuracy. While comes in second place LLama3 with a limited context window of ±10 words around target words, with 57.34% accuracy. As shown from the results, limited context windows matter when inference with LLMS. However, size of context window is a hyperparameter that needs to be adjusted. PLMs struggle in performance for WSD tasks. Most of the models struggle to outperform the baseline by a large margin. The proposed approach archives third place in the leaderboard with a 20% margin from the first-place solution. Table 2 shows the results of developed models for subtask2. Refer to Appendix B for details about pre-processing and post-processing. The proposed
approach outperforms the baseline by a margin of 37% and lands in first place. In addition, it outperforms the second place solution by a margin of 35%.

heading: 5.2 Error analysis
text: In this section, we will explore the limitations and challenges of the model, with the goal of identifying areas for improvement and enhancing its overall performance and reliability. Specifically, for subtask 1, we encountered the following issues:
• LLMs struggled to infer word meaning from
Model Approach Accuracy
LLama3 Zero-shot
No pre-processing or No post-processing
30.06
LLama3 Zero-shot
Pre-processing 30.74
LLama3 Data Base 92.2 LLama3 Data Base
post-processing 94.97
Baseline - 62.70
number. • Some senses were too large for the model
to handle effectively, causing difficulties in responding to the original task. • Certain models struggled to understand the task and continued providing context instead of solving the task itself. • PLMs faced issues due to the limited number of training datasets and a small context window of 512 tokens, making it challenging to correctly infer meaning for large senses, which was more common in the test set than in the training set. For subtask 2, one of the most common errors was the LLMs (LLama3) inability to correctly associate locations with countries, for instance it would associate ÉJ JÖÏ @ with Iraq rather than Egypt. Another problem is the translation of location from Arabic to English for instance 1/3 of the time it would translate àA JJ. Ë , HðQ K. A ̄QÓ to “Market in Beirut, Lebanon". Refer to the Appendix D for more analysis.

heading: 6 Conclusion
text: This paper presented the work conducted during the ArabicNLU Shared Task. The best solution for both subtasks was based on LLama3, which demonstrated superior performance compared to MARBERT and AraBERT. However, it is important to note a limitation observed during the experiments, namely the issue of hallucination, where the LLama3 model occasionally provided incorrect output formats, or would continue text rather than solving task. Future work will focus on enhancing these models to address this limitation and further improve their accuracy and reliability

heading: A Subtask-1 Methodology
text: To address the first subtask, the LLMs were instructed to select the most suitable definition. The input was the target word, context, and list of sense choices. Various instructions have been investigated to improve model performance. The figure 3 shows the whole pipeline.
A.1 Instruction Designs The following set of instructions has been investigated, Including both the Arabic version of the instruction as well as the English version
1. ð @ ú æªÖÏ @ É JÖ ß ú æË @ éjJ j Ë@ éK. Ag. B@ Q g@
éÊÒm.Ì'@ ú ̄ èA¢ªÖÏ @ éÒÊ¾ÊË ø ñj
JË @ Q ® JË @ éJ ËA JË @
[] : éÊÒm.Ì'@ [] : éÒÊ¾Ë@
áÓ [] éÒÊ¾Ë iJ j Ë@ ú æªÖÏ @ Q g@ : H@PAJ J kB@ (tÌ'@ ,3 ,2 ,1) ÉK. A ®ÖÏ @ ȬQË @ Õç' Y
® K ÈC g .hQå àðX ¡ ® ̄
Choose the correct answer that represents the meaning or grammatical interpretation of the word given in the following sentence
Sentence: []
Word: []
Choices: Choose the correct meaning of the word [] by providing only the corresponding number (1, 2, 3, etc.) without explanation.
2. . [] éÒÊ¾Ë@ úÎ« ø ñ Jm ' éÊÔg. Õç' Y ® K Õ æJ Q ® JË @ ð @ iJ j Ë@ ú æªÖÏ @ YK Ym ' ù ë ½ JÒêÓ
Ȭ . ù¢ªÖÏ @ AJ Ë@ úÎ« ZA JK. iJ j Ë@ ø ñj JË @ Q º ® JË @ ÐY j J @ð éK A JªK. AJ Ë@ ÉJ Êj JK. .I. B@ PAJ J kB@ YK Yj JË èñ¢ m '.
èñ¢ mÌ'@ [] : éÒÊ¾Ë@ [] : éÊÒm.Ì'@
ð @ ú æªÖÏ @ É JÖ ß ú æË @ éjJ j Ë@ éK. Ag. B@ Q g@ . ù¢ªÖÏ @ AJ Ë@ ú ̄ [] éÒÊ¾ÊË ø ñj JË @ Q ® JË @
¡ ® ̄ ÉK. A ®ÖÏ @ éÔ ̄P Q ̄ñ JK. iJ j Ë@ PAJ mÌ'@ XYg É ®Ó Q ® K ZA¢«@ àðX áÓ (tÌ'@ ,3 ,2 ,1) . éÊÒm.Ì'@ ÈAÒ» @ ð @
[] : H@PAJ J kB@ A sentence containing the word [] will be presented. Your task is to determine the correct meaning or grammatical interpretation based on the given context. Carefully analyze the context and use step-by-step reasoning to determine the most appropriate choice.
Sentence: [] Word: []
Choose the correct answer that represents the meaning or grammatical interpretation of the word [] in the given context. Select the correct option by providing only its corresponding number (1, 2, 3, etc.) without giving a detailed explanation or completing the sentence.
Choices: []
3. Instructions: You will be presented with a text containing the word "[]". Your task is to determine the correct meaning or grammatical pattern based on the given context. Carefully analyze the context and use step-by-step reasoning to select the most appropriate choice.
Text: "[]" Word: "[]"
Which of the following choices best represents the meaning or grammatical pattern of the word "" in the given context? Choose the correct option based on careful analysis.
Choices: "[]" Please provide the number corresponding to the correct choice.

heading: A.2 Pre-processing
text: To handle large contexts and inconsistencies in answers, various pre-processing techniques have been used. At first, we remove any punctuation or numbers from sentences. Then we limit input sentence/context/text to be ±5 or ±10 words around the target word. Post-processing is applied to get the answer, if the model answers in numeric values we extract it. Otherwise, the model could answer with textual values of the number of another number included in the sense, in this case, we don’t handle it. But if the model returned the whole sentences of the chosen sense, we match it with the current sense list and return the number of the sense.

heading: A.3 Hyperparameters
text: The hyperparameter used to to fine-tune PLMs is listed in Table 3.

heading: B Subtask-2
text: B.1 Instruction The following set of instructions was used to get information from LLama3. For detecting whether a given word is a country or location we use “According to this text: [] Does this word [] refer to a country name The answer should be yes/no." For translating words we use the following instruction: “Translate this word to English:[]. The answer must not contain any Explanation and should be structured as "Answer: " " " For querying location we use the following instruction “ Given this context: "[]"
According to the previous text, Which country is the location "[]" located in? If you could not get it from context, which country is highly likely to have this location/place "[]" in? The answer should be in the English language, with no explanation needed only, and should be structured as "Answer: Country ". "

heading: B.2 Pre-processing and Post-processing
text: In this approach, the impact of pre-processing and post-processing on results has been explored. In the pre-processing step, both tweet/text and location mention are clean, that is to remove hashtags, mentions, emojis, and underscores. In post-processing, various steps are applied
1. If Geopy returns nothing for the correct text, we try to remove one of the words in the text. For example, Geopy could not retrieve any information about àA JJ. Ë , àñ JÊj. « éK P@
YªË@ é PYÓ however, by removing the last word àA JJ. Ë , éK P@
YªË@ é PYÓ it was able to retrieve the correct information.
2. Another step in applying filtering based on retrieved toponyms, is if country and location are not explicitly present we query LLama3 whether both the location and country are the most likely to be in toponyms. If it answered no then we remove it else we keep it.
3. For ranking, if the country name or location name is explicitly in toponyms then it has high ranking than those its not in it.

heading: C Dev-phase results
text: Table 4 shows the performance of various LLMs models in zero-shot setting inference on the training dataset in subtask 1. LLama showed the best performance with around 70% accuracy compared to other models. While WizardLM2 comes in second place. Table 5 shows the performance of MARBERT and AraBERT during dev-phase in subtask 1 on various settings. Both Models showed comparable results and improvement when having a large dataset during training. In subtask 2, LLama3Instruct with Database, pre and post-processing steps achieved 90% F1-score.

heading: D Error Anaylsis
text: Extensive error analysis has been conducted on LLMs for both subtasks.

heading: D.1 Subtask-1
text: The following section will discuss errors in each model.
D.1.1 AceGPT-7B AceGPT-7B fails most of the time to solve the original problem, for instance, for the following instruction . J » @ éÒÊ¾Ë@ úÎ« ø ñ Jm ' éÊÔg. Õç' Y ® K Õ æJ Q ® JË @ ð @ iJ j Ë@ ú æªÖÏ @ YK Ym ' ù ë ½ JÒêÓ Ȭ . ù¢ªÖÏ @ AJ Ë@ úÎ« ZA JK. iJ j Ë@ ø ñj JË @
èñ¢ mÌ'@ Q º ® JË @ ÐY j J @ð éK A JªK. AJ Ë@ ÉJ Êj JK. .I. B@ PAJ J kB@ YK Yj JË èñ¢ m '. èY J ÖÏ @ éJ »Q Ó B@ é AJ Ë@ IÒëA J »
@ : éÊÒm.Ì'@ J » @ : éÒÊ¾Ë@ Zñ ú
̄ éK Xñª éK ðP úÍ@ ð @ ú æªÖÏ @ É JÖ ß ú æË @ éjJ j Ë@ éK. Ag. B@ Q g@ . ù¢ªÖÏ @ AJ Ë@ ú ̄ J » @ éÒÊ¾ÊË ø ñj
JË @ Q ® JË @ ,1) ¡ ® ̄ ÉK. A ®ÖÏ @ éÔ ̄P Q ̄ñ JK. iJ j Ë@ PAJ mÌ'@ XYg
ÈAÒ» @ ð @ É ®Ó Q ® K ZA¢«@ àðX áÓ (tÌ'@ ,3 ,2 . éÊÒm.Ì'@ ÉÒª J i J ®Ë @ úÎ« ú
æJ.Ó Õæ @ .1 : H@PAJ J kB@ ( ø PA¾
K @ ú æ. j. ª K) ù
®J ®mÌ'@ Q « ð @ ù ®J ®mÌ'@ ÐAê ® J CË - é K Q . mÌ'@ úÎ« © ̄P Ém× ú ̄ A Jë J » : ? YK P J » @ úÎ« I. Ém× ú ̄ A Jë J » : ? Y Òm× ZAg. J »
- 260 èQ ®J. Ë @ - é ®Ê¢ÖÏ @ é J Ëñª ®ÖÏ @ ð @ é J ËAmÌ'@
. ø PA¾ K @ ú æ.j. ª K ÐAê ® J @ úÎ« ÈY K : 28 èQ ®J. Ë @
? B J » - ? ÈAmÌ'@ àA¿ J » - ?½ ËAg J »
. @ © J @ © J J » @ Ð PAg. Q «  Qå Õæ @ .2
it would return úÍ@ èY J ÖÏ @ éJ »Q Ó B@ é AJ Ë@ IÒëA J »
, B J » @ ø PA¾ K @ @Y JJ.Ó .3 Zñ ú ̄ éK Xñª éK ðP . éJ £Qå Ë @ éÊÒm.Ì'@ áÓ IËA JË @ Õæ ®Ë @ ñëð
ù ëð @? . . . IÒëA J » @ éJ J.j. ª JË @ èPAJ.ªË @ .4
Qê ¢ IJ k , J » @ Ë èñ̄ Q » B@ Ð@Y j J B@
. È@ ñ Ë@ð é ëYË@ ? J » - ú æ
@Q «B@ È@ ñ Ë@ áÓ Z Qk. .5 IJ k ,
J » @ XZ éJ £Qå éÊÔg. ð @ Q . g Õæ @ .6
. @ X @ " Ë éJ @Q «B@ é ªJ Ë@ ù ë J » @
: éÊ JÓ B@ áÓð , A J ®J » - éJ Ë @ ñ Ë@ éÊÒm.Ì'@ éK @YK. .7
. ¼ è A ¿ P ¬ Qå J » ð ú G ñ Ü Ï @ úm ' J » @
? J » - I I J» ñË , AÓY J« , @ X @ Q . g .8

heading: D.1.2 Opengpt
text: For openGPT it fails when Quran verses are presented in text. It would either return them all or answer “ These are some of the meanings and
contexts in which "" can be used. Remember that Arabic is a rich language with many nuances, so it’s essential to consider the context when using words to ensure the intended meaning is conveyed. "

heading: D.1.3 WizardLM2
text: The failure cases occurred because the model attempted to provide additional information and simplify the text while elaborating on the chosen sense. As a result, it became challenging to automatically determine the correct answer. For instance it would return . éJ Êm× lÌ'A Óð ú Îm × t ' PA K AêË éJ Êm× é«ñÒm.×
However choices are É m × úÍ@ H. ñ Ó Õæ @ and é ®¢ JÖß. A g ð @ á ªÓ © ñÖß. Êª JÓ , ú Î g@X
D.1.4 LLama3 The failure cases observed in LLama3 share similarities with WizardLM2. Both models tend to simplify the content, and in some instances, LLama3 mistakenly selects the number mentioned inside the choice text instead of the actual number corresponding to the choice itself. These challenges highlight the need for further improvements in the models’ understanding of context and accurate interpretation of the provided choices.

heading: D.2 Subtask-2
text: LLama3 struggled in correctly associating locations with countries and correctly translating the name of the location. For instance 1/3 of the time it would translate HðQ K. A ̄QÓ to “Market in Beirut" or “Marina Beirut". Another example ÉJ JÖÏ @ would translate it to “The key" or “The Treasure". Therefore we ask the model 10 times the same question and take the most repetitive answer. Another problem is associating location to country for instance location ÉJ JÖÏ @ or Ð@PðB@ YêªÓ would be associated 1/3 time with Lebanon, 1/3 of the time with Iraq, and 1/3 of the time with Egypt. However, this was solved by adding “most probable country and is usually a known place " in the prompt. Another Example location èPñ J K @P 1/3 percent of the time
it will associated with “’Sudan’, 1/3 of the time with “Lebanon" and 1/3 of the time with “Saudi Arabia". Another problem is in Geopy itself, incorrectly retrieving toponyms itself. For instance, when given àA JË éK Y JºË@ èPA ® Ë@, it would retrieve a lot of places not related to the current query itself.
Another problem with Geopy, is that if a location is presented in multiple countries it would return them all, even if the query does contain the correct country name.

heading: Future Work
text: Future work will focus on enhancing these models to address this limitation and further improve their accuracy and reliability