# Contemporary Approaches in Evolving Language Models
## 3 Neural Network Models
### Pretrained Language Models
#### 3.2.2 Gpt  
GPT (Generative Pre-trained Transformer) is a series of neural language models developed by OpenAI. GPT basically uses the goal of training a language model with an autoregressive approach, predicting the next word in a sequence based on the context of the preceding words. Unlike BERT, GPT does not use MLM or NSP tasks. The models utilize the transformer architecture but only employ the decoder blocks. One of GPT's defining features is its ability to generate texts as opposed to merely predicting the likelihood of subsequent words. The original GPT-1 model was trained on the BooksCorpus dataset (Table 2). The BooksCorpus contained about 7000 unpublished books, aiding in training the language model on unseen data. GPT employed a 12-layer decoder, utilizing only the transformer structure with masked self-attention for language model training. The masking ensured that the language model could not access subsequent words to the right of the current word during training. Model Training Details:  
* A Byte-Pair Encoding (BPE) vocabulary with 40,000 merges was used. * The model utilized a 768-dimensional state to encode tokens into word embeddings. Positional embeddings were also learned during training. * A 12-layer model with 12 attention heads per self-attention layer was employed. * A 3072-dimensional state was used for the positional feedforward layer. * The Adam optimizer was used with a learning rate of \(2.5\times 10^{-4}\). * Attention, residual, and embedding dropout were applied for regularization, with a dropout rate of 0.1. A modified L2 regularization was also applied to weights without bias. * The GELU activation function was used. * The model was trained for 100 epochs on mini-batches of a size of 64 and a sequence length of 512. In total, the model had 117 million parameters. One of the most significant achievements of this model is its impressive zero-shot performance across various tasks. Studies have shown that the model achieved zero-shot capabilities in various NLP tasks such as question-answering, coreference resolution, and sentiment analysis, all thanks to its pre-training. GPT has proven that a language model serves as an effective pre-training objective that can aid the model in generalizing well. The architecture facilitates transfer learning and can execute a myriad of NLP tasks with minimal fine-tuning. This model demonstrated the power of generative pre-training and paved the way for subsequent models that could further harness this potential with larger datasets and more parameters. Since the debut of the original GPT model, several enhancements and variations have been introduced to optimize its performance and capabilities. Some of the improvements include the following:  
1. GPT-2 is the successor of the original GPT model. GPT-2 was trained on a significantly larger dataset and boasts a higher capacity (1.5 billion parameters). It showcased impressive results in text generation. However, due to concerns about its potential misuse, its release was restricted. To compile a comprehensive and high-quality dataset, the authors scraped the Reddit platform and extracted data from outgoing links in highly rated articles. The resultant dataset, named WebText, consisted of 40 GB of textual data from over 8 million documents. This dataset was considerably larger than the Book Corpus used to train the original GPT model. All Wikipedia articles were excluded from WebText since many test sets incorporate Wikipedia articles. GPT-2 had 1.5 billion parameters; this is tenfold the number in GPT-1, which had 117 million parameters. The primary differences from GPT-1 included the following:  
* GPT-2 featured 48 layers and utilized 1600-dimensional vectors for word embeddings. It employed a larger vocabulary of 50,257 tokens. * A more substantial batch size of 512 and a larger context window of 1024 tokens were used. * Layer normalization was moved to the input of each sub-block, and additional layer normalization was added after the final self-attention block [61].