## 2 Related Literature Review
### Transformer-Based Language Models
#### 2.1.7 DeBERTa  
DeBERTa is a sophisticated evolution of the RoBERTa and BERT models, refined in adversarial environments subsequent to pre-training for masked language modeling. A disconnected component attention layer is integrated into DeBERTa [25; 40].  
Word sense disambiguation (WSD) significantly depends on DeBERTa's disentangled attention mechanism. Unlike traditional self-attention mechanisms, exemplified by BERT and RoBERTa, DeBERTa separates two critical representations--\"Content Embeddings\" and \"Position Embeddings\"--through disentangled attention. The disentangled attention mechanism of DeBERTa enhances contextual representations and captures subtle semantic distinctions [40].  
DeBERTa introduces two unique strategies: disentangled attention and enhanced masked decoding, rendering it a more decoding-centric variant of BERT with disentangled attention. In the input layer, each word or token is represented by two vectors that signify its content and position within the corpus, exemplifying the principle of disentangled attention. This is indicated by the dependence of content extraction on the word's position [40]. DeBERTa enhances RoBERTa and BERT through the implementation of disentangled attention processes that independently encode word content and location information, leading to superior performance in word sense disambiguation tests [25].  
The attention mechanism operates with two types of embeddings in DeBERTa:  
* Content embedding \(c_{i}\) for the word content;
* Positional embedding \(p_{i}\) for the word's position in the sequence.  
The attention score for word \(i\) and word \(j\) is computed using Equation (2):  
\[Attention(i,j)=\frac{\bigg{(}W_{Q_{c_{i}}+W_{Qp_{i}}^{p}}\bigg{)}\bigg{(}W_{K _{c_{j}}}+W_{K_{p_{j}}}^{p}\bigg{)}^{T}}{\sqrt{d_{k}}} \tag{2}\]  
where  
* \(W_{Q}\), \(W_{Q}^{p}\) are the weight matrices for the content and positional query;
* \(W_{K}\), \(W_{K}^{p}\) are the weight matrices for the content and positional key;
* \(d_{k}\) is the dimension of the key vectors.  
DeBERTa enhances the expressiveness of the attention mechanism by separating positional and content embeddings, which results in improved performance on tasks such as word sense disambiguation (WSD).  
By distinguishing between positional information and word content, DeBERTa improves on BERT and makes it possible to model word meanings in context more accurately. Performance on a variety of NLP tasks is enhanced using this method, particularly in WSD. A summary table including the paper title, authors, research techniques/methodology, research gaps, summary of findings, and interventions is presented in Table 1 for studies on low-resource languages.  
This synopsis offers a succinct synopsis of the important facets of research on low-resource languages, highlighting the difficulties and cutting-edge strategies being investigated.  
\begin{table}
\begin{tabular}{c c c c c c c} \hline \hline
**Dataset** & **\# Sentences** & **\# Tokens** & **\# Annotations** & **\# Sense Types** & **\# Lemmas** & **Ambiguity** \\ \hline Augmented dataset & 2859 & 3289 & 3288 & 162 & 22 & 4.85 \\ \hline \hline \end{tabular}
\end{table}
Table 1: Summary of dataset description.