# Arabic Gloss WSD Using BERT
## 1 Introduction  
Words in spoken and written languages can have multiple meanings for each word and, thus, could be ambiguous. The intended meaning of a word (_sense_) depends strongly on its context. Word Sense Disambiguation (WSD) is the task of predicting the correct sense of a word that has multiple senses given its context [1]. Many Computational Linguistics' applications depend on the performance of WSD, such as Machine Translation, Text Summarization, Information Retrieval, and Question Answering [2]. The Arabic language is particularly challenging for WSD. It was shown in [3] that 43% of Arabic words that have diacritic marks are ambiguous. Moreover, that percentage increases to 72% when the diacritic marks are missing, which is the case for most written Arabic text [4]. This is compared to 32% for the English language [5]. We note here three main challenges in Arabic WSD. First, most written text does not have diacritic marks. As a result, the level of ambiguity for each word can increase. For example, the word \"\(\frac{1}{2}\)\" (meaning flag) and the word \"\(\frac{1}{2}\)\" (meaning science) are both written as \"\(\frac{1}{2}\)\". Secondly, even when diacritic marks are present, many words can have several possible part-of-speech (POS) tags that lead to different meanings. For example, the word \"\(\frac{1}{2}\)\" can have the POS tag _verb_ (leading to the meaning of \"showing something\"). It can also have the POS tag _noun_ (leading to the meaning of \"impermanent\"). Third, a word with both a POS tag and diacritic marks can still have several senses depending on its context. For example, the word \"\(\frac{1}{2}\)\" with POS tag _verb_ has the meaning of \"parade\" in the context \"\(\frac{1}{2}\)\" while having the meaning of \"inspect\" in the context\"_\"_\"_\"_\"_\"_\"_\"_\"_. In addition, Arabic is an agglutinative language that adds parts of speech to the stems or roots of words to give new meanings [6]. For instance, the word \"_\"_\"_\"_ (in the street) has a stem \"_\"_\"_ (a street) and two prefixes, \"_\"_\"_ (in and \"_\"_\"_ (the), which is the definite article. This problem has been addressed in the preprocessing phase in our work, which will be discussed shortly. Bidirectional Encoder Representation from Transformers (BERT) [7] is a neural language model based on transformer encoders [8]. Recent Arabic Computational Linguistics' studies show a large performance enhancement when pretrained BERT models are fine-tuned to perform many Arabic Natural Language Processing (NLP) tasks, for example, Question Answering (QA), Sentiment Analysis (SA), and Named Entity Recognition (NER) [9; 10]. This performance enhancement stems from the fact that BERT can generate contextual word embeddings for each word in a context. These embeddings were shown to encode syntactic, semantic, and long-distance dependencies of the sentences [11], thus making them ideal for many NLP tasks. Contextual embedding methods (e.g., BERT [7], Embeddings from Language Models (ELMO) [12], and Generative Pre-trained Transformer (GPT) [13]) learn sequence-level semantics by considering the sequence of all the words in the input sentence. In particular, BERT is trained to predict the masked word(s) of the input sentence; to do this, BERT learns _self-attention_ to weigh the relationship between each word in the input sentence and the other words in the same sentence. Each word then has a vector that represents the relationship with other words in the input sentence. Those vectors are used to generate word embedding, so the generated embeddings for each word depends on the other words in a given sentence. This is unlike Glove [14] and Word2 Vec [15] that represent the word using a fixed embedding, regardless of its context. These traditional models depend on the cooccurrence of these words in the hole corpus. If two words usually have similar words in different contexts, they will have a similar representation. WSD is classified into three main categories: supervised WSD, unsupervised WSD, and knowledge-based WSD.