# Contemporary Approaches in Evolving Language Models
## 3 Neural Network Models
### Pretrained Language Models
#### 3.2.2 Gpt
GPT-2 demonstrated that training on larger datasets and having a higher number of parameters enhanced the language model's ability to comprehend tasks, surpassing the current state of the art in many zero-shot conditions. Additionally, the decrease in the perplexity of language models showed no signs of plateauing and continued to drop as the number of parameters increased. An intriguing capability of GPT-2 is its zero-shot task transfer. Zero-shot learning is a specific instance of task transfer in which no examples are provided and the model understands the task based solely on the provided instruction. Instead of reshuffling sequences, as was carried out with GPT-1 for fine-tuning, inputs in GPT-2 were provided in a format expecting the model to grasp the task's nature and supply answers. This was designed to mimic behavior in zero-shot task transfer scenarios. For instance, for an English-to-French translation task, the model was presented with an English sentence followed by the word \"French\" and a hint (:). The model was expected to recognize this as a translation task and provide the French equivalent of the English sentence. 2. GPT-3 boasts a considerable number of parameters, standing at 175 billion, making it one of the most extensive neural models to date. It was trained on a combination of five distinct corpora, each assigned a specific weight. High-quality datasets were prioritized, with the model being trained on them for more than a single epoch. The datasets used include Common Crawl, WebText2, Books1, Books2, and Wikipedia. The architecture of GPT-3 remains consistent with GPT-2, featuring 96 layers, each containing 96 attention heads. The word-embedding count was increased to 12,888, and the context window size was expanded to 2048 tokens. The Adam optimizer was utilized with \(\beta_{1}=0.9\), \(\beta_{2}=0.95\), and \(\varepsilon=10^{-8}\). Additionally, alternating dense and locally striped sparse attention patterns were implemented. GPT-3 was evaluated on an array of language models and NLP datasets. It performed admirably on NLP tasks such as closed-book question-answering, schema resolution, and translation, often outpacing the most contemporary or comparably fine-tuned models. However, certain shortcomings of this version were identified:  
* Its unidirectional nature struggles with tasks like natural language inference, gap-filling, and some reading comprehension tasks and loses coherence when framing long sentences, often repeating text sequences [62]. * The model weighs each token equally and lacks task-awareness or goal-directed token forecasting. To mitigate this flaw, suggestions included enhancing training objectives, employing reinforcement learning for model fine-tuning, and incorporating additional modalities. * The model's vast architecture complicates inferencing and offers reduced language and result interpretability. * Like other model versions, it can generate phishing emails, spam, and texts with racial or ethnic biases. 3. The GPT-3.5 model comes in three variants with parameters of 1.3B, 6B, and 175B. While this model is trained based on GPT-3 data, it undergoes an additional fine-tuning process with human feedback to adjust the model's behavior. The model was trained on data up to the fourth quarter of 2021 and serves as the foundation for the ChatGPT dialogue bot. 4. GPT-4 (March 2023) represents the evolutionary stage in the GPT series, surpassing its predecessor, GPT-3.5, by enhancing capabilities like processing longer text segments and images. The essence of GPT-4's training encompasses transferring the rich knowledge acquired during preliminary training to diverse new tasks. Relying on large-scale pre-training, this approach enables the model to grasp linguistic patterns and context, which can then be fine-tuned for optimal performance in specific tasks. Memory and attention mechanisms play a crucial role, ensuring GPT-4's effective learning even with limited data.