## 2 Related Literature Review
### Transformer-Based Language Models  
This section briefly discusses the key characteristics of each model relevant to the literature review. This study delineates the differences among each of these models. The transformer is a non-recurrent design including an encoder and a decoder for sequence-to-sequence (seq2 seq) communication [26]. The encoder and decoder of a transformer are situated next to each other, consisting of several identical blocks. Each encoder block consists of a position-wise feed-forward layer and a multi-head self-attention layer. Each decoder block has an additional cross-attention layer compared to the encoder block, as the decoder requires the encoder's output as contextual information for generation [26]. Transformers were introduced in the 2017 article \"Attention Is All You Need\". Their objective is to rectify the shortcomings of RNNs. The self-attention mechanism represents the primary innovation of transformers. Transformers exceed RNNs in their capacity to capture long-range dependencies in phrases, exhibiting considerable effectiveness in neural machine translation (NMT) [27].  
The domain of natural language processing (NLP) has gained significance over time. Transformer-based pre-trained models, particularly BERT, have demonstrated remarkable efficacy in many NLP tasks; yet, their substantial parameter count and extended processing demands complicate their deployment on resource-limited embedded platforms. To address this problem, researchers [28] have employed the ALBERT model with the enhanced early exit technique (ELBERT) and developed an efficient VLSI architecture via an algorithmic and hardware co-design methodology. Initially, through the implementation of quantization and encoder-level parameter sharing approaches, the storage capacity for BERT is diminished from 1208.88 MB to 20.99 MB without any degradation in accuracy [28].  
Word sense disambiguation (WSD) in Arabic [29] is related to WSD in Sesotho sa Leboa because BERT uses contextualized word embeddings to resolve lexical ambiguity in a low-resource, morphologically complicated language. Arabic, like Sesotho sa Leboa, has extensive morphological features, polysemy, and homonymy, which reduces the effectiveness of classic WSD techniques. This study investigates how to enhance WSD resolution in Sesotho sa Leboa by combining transformer-based embeddings (BERT, RoBERTa, and DeBERTa) with BiGRU and attention processes. BERT's success in Arabic WSD shows that multilingual BERT models could be enhanced, or Sesotho sa Leboa-specific embeddings could be created for better WSD performance.  
1.1. 2.1.  
(MLM)--are applied for pre-training, while WordPiece tokenization is implemented to decompose words into their constituent character-level components [23; 26; 30; 32]. The researchers in [29] introduced a dataset including one hundred polysemous Arabic phrases, each demonstrating three to eight unique interpretations, accompanied by ten illustrative utterances for each term. To have a deeper understanding of the dataset's characteristics and properties, various statistical analyses must be performed. BERT, an innovative method for word sense disambiguation, was developed to determine the relationship between dictionary meanings and contextual information using similarity metrics, and the suggested pre-trained language model enabled effective Arabic word disambiguation [25]. New attributes were integrated during training to improve the model's capacity to distinguish between distinct meanings of words. A composite model architecture was created by incorporating the proposed BERT model [33], leading to the WSD system attaining an F1-score of 96%, exceeding the latest systems [29].