# Contemporary Approaches in Evolving Language Models
## 3 Neural Network Models
### Pretrained Language Models
#### 3.2.1 Bert

Table 1: Brief comparison of BERT models and variants based on various characteristics. remains a more resource-conservative version, even though it still demands substantial computational resources [45]. These advancements show how researchers consistently aim to refine and expand concepts associated with BERT, targeting higher performance and versatility in various natural language processing tasks. After its preliminary training, the BERT model can undergo further training for specific tasks through fine-tuning. This enables BERT to grasp more intricate and higher-level semantic nuances in the text, making it a potent tool for diverse NLP tasks. The BERT model boasts numerous advantages. However, like all technologies, it also has its downsides:  
* BERT, especially its larger versions like BERT Large, encompasses a considerable number of parameters. This leads to high computational demands during training and deployment, presenting challenges for systems with restricted computational capacities. Additionally, BERT has a set maximum sequence length it can handle, which can be an issue for texts with long contexts or documents. * BERT processes words and symbols alike, potentially leading to unnecessary processing of punctuation and other non-essential text elements. Moreover, BERT does not consider the internal order of characters within words (morphological structure), which can be a limitation for languages with rich morphology. * Effectively training BERT requires vast and varied data volumes. This can be challenging for languages with limited available data, such as agglutinative languages. BERT also views context within a specific window, possibly overlooking more distant dependencies between words. * BERT might capture noisy data during pre-training, which could affect its generalization ability on new data. It is important to note that many of these drawbacks can be mitigated or overcome with improved model versions, additional fine-tuning and adaptation methods, and the integration of BERT with other techniques and architectures. Enhancements and Modifications for Agglutinative Languages  
In Ref. [46], two models, ARBERT and MARBERT, were introduced for data classification tasks focusing on the Arabic language. Both ARBERT and MARBERT utilized the same architecture as BERTBase; however, MARBERT did not incorporate NSP (Next Sentence Prediction) and used the WordPiece tokenizer. The dataset was expanded with Twitter data for pre-training the new MARBERT model. [PERSON] and his team introduced a combined ARABERT and GPT2 model using transfer learning to detect text produced by various AI bots in Arabic. For generating fake sentences, GPT2-Small-Arabic was used. The proposed model was found to significantly outperform RNN-based models. In Ref. [47], a BERT-BGRU model was introduced for the task of recognizing Arabic-named entities. Here, BERT combined multiple attention layers, and fine-tuning BERT as a base input representation model effectively reduced the training time for the subsequent deep-neuralnetwork-based model. [PERSON] et al. [48] implemented a Turkish specialized transformer, BerTURK, to determine the authenticity of Turkish COVID-19 news on social networks. The model's efficiency was achieved through a minimal fine-tuning strategy, boasting a prediction accuracy of 98.5%. [PERSON] et al. [49] developed AraLegal-BERT, a bidirectional encoder transformer trained on legal documents. It followed the same procedure as in AraBERT by [PERSON] et al. (2020) [50]. For the development of the AraLegal-BERT model, a lexicon of around 64,000 words was used which was constructed based on a subword segmentation approach to divide all tokens into prefixes, suffixes, etc. [51]. Comparative analyses showed that a model pre-trained based on a specific domain performed better than the standard BERT models.