# Contemporary Approaches in Evolving Language Models
## 3 Neural Network Models
### Pretrained Language Models
#### 3.2.1 Bert  
In 2018, Google introduced BERT, which continues to be a frequently used modeling architecture due to its high efficacy in various NLP tasks. This model is a pre-trained one based on unlabeled textual data from open sources like BookCorpus and Wikipedia. Rooted in the transformer framework, its unique bidirectional nature allows the model to capture intricate dependencies between words. By fine-tuning the model on labeled data from specific tasks, BERT achieves competitive performance. In its structure, BERT employs the encoder component of the transformer architecture. There are both base and large versions of the model, differing in the number of layers and the number of self-attention heads they have, with 12 and 24 layers, respectively. The total count of adjustable parameters reaches several hundred million, necessitating significant computational resources. Although most of the high achievements initially came from using the 24-layer BERT, issues often arose due to the limited computational power of machines. To address this challenge, the ELECTRA model was proposed. ELECTRA is designed to be more efficient than BERT, optimizing both training time and model size while retaining, or in some cases even surpassing, the performance of larger models like BERT. It rethinks the pre-training process and utilizes a replaced token detection task, distinguishing it from BERT's masked language model approach [38]. Like GPT, BERT uses contextual embedding. For each word, a fixed number of 768-dimensional embedding vectors are determined, serving as the model's input sequences. BERT outputs a vector with 768 parameters for each word/phrase based on its internal calculations. These output sequences then become embeddings, representing the sentence sequence in the context of the data. In simpler terms, an initial input to BERT has a basic vector layer, but after processing through BERT, a contextual vector is produced using the entire word vector. The model comprises input formatting and embedding extraction parts (Figure 5). Special tokens like [CLS] and [SEP] are used to denote the beginning and end of a sentence, respectively. [PAD] is used for padding, and [UNK] indicates an unknown word. The embedding layers include token embeddings, segment embeddings, and position embeddings. For token embeddings in BERT, the WordPiece method is commonly used for tokenization. Segment embedding examines the semantic relationship of up to two sentences. Position embedding retains the position information of a word to create a word sequence. Even though BERT uses bidirectional encoding, it operates quite differently from recurrent neural networks. This is because BERT maps the obtained position embeddings of each token to segment embeddings to represent a word's property and its location. In other words, all that is needed is to format the input text using the embedding layer and let the encoder provide the appropriate output. BERT's Pre-Training Algorithm  
There are two main approaches, with one being the Masked Language Model (MLM). The MLM is trained to predict words that are randomly masked in a sentence. For instance, consider the sentence: \"I [MASK] in Almaty\". Here, the word \"live\" has been masked. The model then tries to predict this masked word, and its training is optimized based on the associated loss. This method enables the model to learn to capture semantic relationships and the context within a sentence. This specific technique is central to language modeling in BERT. Next sentence prediction (NSP): to further train the model, pairs of sentences are presented, and the model must determine whether the second sentence is a continuation of the first (Figure 6). One of BERT's main advantages is its multilingual capabilities. It is trained on a vast volume of polysemous texts, allowing it to work with data in multiple languages. Figure 5: BERT masked language model example. Figure 6: Example of a next sentence prediction model.