# Contemporary Approaches in Evolving Language Models
## 3 Neural Network Models
### Pretrained Language Models
#### 3.2.2 Gpt
Enhanced linguistic capabilities and multilingual support augment its proficiency in producing higher-quality and coherent responses in various interactions. Other improvements and modifications of the GPT model for agglutinative languages are described below. In recent research efforts, various adaptations and evaluations of large language models, such as GPT-3, for non-English languages have been undertaken. This section provides an overview of studies that explore the performance, challenges, and applications of GPT-based models in linguistic contexts beyond English. 1. For the Arabic Language:  
[PERSON] et al. [64] developed the first refined model for Arabic language generation, AraGPT2, trained from scratch on a large corpus of Arabic Internet texts and news articles. Their larger model, ARAGPT2-MEGA, which has 1.46 billion parameters, has shown success in various tasks including synthetic news generation and zero-shot question-answering. Regarding text generation, their best model achieves a perplexity of 29.8 on held-out Wikipedia articles. A human evaluation study showed that AraGPT2-MEGA significantly excels in creating news articles that are hard to differentiate from those written by humans. In [65], researchers evaluated the efficacy of 24 models for detecting sentiment and sarcasm in Arabic texts. The findings indicated that the AraGPT2 variants show relatively lower performance, especially when compared to BERT-based models, suggesting a potential unsuitability for classification tasks, i.e., revealing potential limitations of the GPT-2 architecture for Arabic in text-classification tasks. [PERSON] et al. [66] created a set of five autoregressive language models named JASMINE, based on GPT-3 for the Arabic language, with parameters ranging from 300 million to 13 billion. The models were trained on approximately 400 GB of texts representing various domains and dialects of the Arabic language. JASMINE underwent rigorous testing on various natural language processing (NLP) tasks across different training regimes, and the results showed that the models can generate text indistinguishable from that written by a human. In Ref. [67], an ensemble learning model for Arabic sentiment analysis was proposed which incorporates a bidirectional long short-term memory (BiLSTM) model and a Generative Pre-trained Transformer (GPT) model. Experiments were conducted that involved separate applications of BiLSTM and GPT and a comparison with the ensemble model. The proposed ensemble model demonstrated an accuracy improvement of approximately 7\(\%\) compared to individually trained models, indicating that model accuracy can be enhanced when combining different architecture\". 2. For the Korean Language:  
[PERSON] et al. [68] emphasized the challenges and opportunities associated with using large-scale language models, like GPT-3, especially in the context of non-English data such \begin{table}
\begin{tabular}{c c c c c} \hline \hline
**GPT Versions** & **Year of Publication** & **Parameters** & **Architecture Improvements** & **Training Dataset** \\ \hline GPT-1 & 2018 & 117 million & 12 decoder layers and 768 hidden & \\  & & & layers, including softmax and & BookCorpus \\  & & & linear layers & \\  & & & 48 decoder layers, 1600 hidden & \\ GPT-2 & 2019 & 1.5 billion & layers, and the normalization layer & WebText \\  & & & was extended & \\ GPT-3 & 2020 & 175 billion & 96 decoder layers and & \\  & & & 12,288 hidden layers & Common Crawl \\ GPT-3.5 & First version: 2021 & 175 billion & WebText2 & \\ GPT-4 & Last version: March & & Combines Transformer and & \\  & & & CNN+RNN models; it can process & Undefined \\  & & & and analyze images & \\ \hline \hline \end{tabular}
\end{table} 
Table 2: Evolution of the GPT model [63]. as the Korean language, and various aspects of model performance. The authors introduced HyperCLOVA, a Korean version of the 82B GPT-3, trained on a Korean-centric corpus of 560B tokens.