## 5 Model Evaluation
### Analogy Test  
The primary objective of the analogy test is to determine the term D based on the given terms A, B, and C. Essentially, given that A:B are related by a specific type of relationship, C would be associated with D using the same relationship. Therefore, the test is structured as follows: if (A,B), then (C-7). The model should respond to this question by predicting the term D. This equation is primarily employed to ascertain the answer:  
\[A-B+C=D \tag{1}\]  
One of the famous examples of an analogy test question is: \"king - man + woman = queen\". To conduct this test, a predefined analogy is utilized, containing pairs of items that are related in a specific manner, such as country-capital, country-currency, and singular-plural. One widely used analogy benchmark is the Google Analogy Test developed by [PERSON] et al. [3]. It encompasses 14 types of relations (nine morphological and five semantic) and serves as a popular benchmark for evaluating word embedding models. Figure 4 depicts a subset of the pairs from the Google Analogy Test. Despite its popularity, there have been doubts about the structure of the Google Analogy Test in terms of relations. The Google Analogy Test contains limited relations. When assessing non-English word embedding models, many studies have utilized a translated version of the Google Analogy Test. However, its effectiveness in evaluating non-English word embedding models has been criticized [10]. For languages with rich morphology, such as Arabic and German, using the Google Analogy Test could result in shortcomings, as it does not initially offer a comprehensive representation of these languages. One major issue with the Google Analogy Test is the inclusion of the \"city-in-state\" test section, which is irrelevant to certain languages, such as Arabic, due to cultural differences. Furthermore, some vocabulary used in the test lacks an exact translation, leading to challenges for the model in producing the intended results. For example, in the \"family\" test section, the words \"rephew\" and \"niece\" cannot be accurately translated as single words in Arabic. It is noteworthy that limited efforts have been made to provide a suitable version of the Analogy Test dataset for Arabic. Most of these efforts have involved translating the Google Analogy Test, as observed in [21, 22], wherein it was argued that translating an English benchmark is challenging because it could result in translation drift. Additionally, the Google Analogy Test was specifically designed for English, and some  
Figure 4: A subset of the Google Analogy Test. of its pairs may not be applicable to Arabic. Moreover, [PERSON] et al. [10] asserted that the existing analogies are insufficient for other languages that have unique characteristics. For instance, the Arabic language is root-based, morphologically rich, and has complex syntax. These limitations have led some researchers to adapt and expand the Google Analogy Test using further adjustments. For instance, [PERSON] et al. [23] contended that the Google Analogy Test has limited relations. Consequently, they created the Bigger Analogy Test Set, in which they added 40 relations, resulting in a total of 99,280 different pairs [10]. Moreover, other researchers have made modifications to the translated version to enhance its usefulness for evaluating non-English models. These modifications have involved either removing certain relations [10] or adding more relations, as in [24]. For Arabic, [PERSON] et al. [25] developed an Arabic Analogy Test. Their analogy contains approximately 100,000 tuples with nine relations. Another attempt was made by [PERSON] et al. [26]; they developed an analogy dataset for dialectal Arabic, encompassing Algerian, Egyptian, Lebanese, Syrian, and Tunisian dialects.