# Contemporary Approaches in Evolving Language Models
## 3 Neural Network Models
### Pretrained Language Models
#### 3.2.1 Bert
In the 2022 [52], the BERT model was enhanced by feeding it with two sentences: the first one containing word expressing sentiment and the second one providing information about the aspect, essentially assisting in aspect-based sentiment analysis. Experiments conducted on three Arabic datasets reached a peak accuracy of 89.51% for hotel reviews. [PERSON], in 2022, presented a systematic review of BERT models for Arabic-language text classification, including models like Multilingual BERT, AraBERT, MARBERT, ArabicBERT, ArabicBERT, ARBERT, XLM-RoBERTa, QARiB, GigaBERT, and ArabALBERT. Based on the analysis, the author concluded that MARBERT, QARiB, ARBERT, AraBERT, and ArabicBERT are the most effective models. A plausible explanation for this finding is the size of the pre-training corpora used for training these models [53]. As evident from the review, research on BERT-like models for the Arabic language is advancing and increasingly addressing diverse NLP tasks. [PERSON] et al. [54] introduced two methods in the context of sentiment analysis: (a) fine-tuning the BERT model and (b) applying the base BERT model after translating Turkish texts into English. Their findings indicated that (a) using the BERT model for sentiment analyses yields better results than other methods, and (b) English-specific BERT models outperform the multilingual BERT model when there are ample training data available. [PERSON] et al. [55] developed a manually annotated dataset of Turkish tweets containing sentences of varying sentiment and polarity. They implemented base models and adapted models named TBERTmarked, TBERTmarked-TS, and T-BERTmarked-MP. As a result, the proposed models demonstrated superior performance in a target sentiment analysis. In Ref. [56], a language model capable of understanding linguistic phenomena of the Korean language was developed and trained on smaller datasets. By leveraging linguistic features and part-of-speech (PoS) tagging during pre-training, models based on transformer architecture and bidirectional language representations proved effective not just for Korean but also for other agglutinative languages. In Ref. [57], a clinical BERT model tailored to Japanese clinical narratives was introduced. The model was pre-trained on a vast domain-specific dataset, facilitating accurate phrase generation in complex tasks like question-answering or recognizing causal relationships using genuine clinical texts. In Ref. [58], an enhanced version of the multilingual BERT, termed E-MBERT, was proposed. It integrates a target language by expanding the vocabulary, encoders, and decoders and then continues pre-training. This approach can be applied to any language with limited datasets not supported by multilingual BERT, especially for named-entity recognition tasks. [PERSON] et al. [59] developed a language-agnostic BERT sentence-embedding model supporting 109 languages. Various methods were examined, including masked language modeling (MLM), translation language modeling (TLM), dual-encoder translation ranking, and additive margin softmax. The introduction of a pre-trained multilingual language model significantly reduces the need for parallel training data, achieving an 80% reduction while attaining 83.7% bi-text retrieval accuracy over 112 languages on Tateboa. The combined methods outperform previous benchmarks and remain competitive in monolingual transfer learning. Additionally, parallel data mined from CommonCrawl using their model proved effective for training NMT models for English-Chinese and English-German. [PERSON] et al. [60] introduced a BERT-based model, named UzBERT, pre-trained on news websites for the Uzbek language. The proposed model was trained on a small corpus of approximately 140 million words. Based on experiments, its accuracy on the masked language model task was significantly higher than that of the multilingual BERT. UzBERT has a smaller vocabulary size compared to mBERT, which means it demands fewer resources for fine-tuning.