# Methods of Annotating and Identifying Metaphors in the Field of Natural Language Processing
## 4 Identifying Metaphors
### Neural Networks and Word Embeddings
#### 4.3.2 Overview of Contextual Word Embeddings  
The first contextual word embedding, ELMo (Embeddings from Language Models), was presented in [PERSON] et al. [68]. In its architecture, ELMo takes over and modifies biLM (Bidirectional Language Model), presented in [PERSON] et al. [72]. biLM consists of LSTM (Long Short Term Memory) [73] layers and a CNN (Convolutional Neural Network) [74] layer in which character convolution is performed, and ELMo enables training in both directions while it receives the entire sentence as input and learns the word representation depending on the context. The authors tested ELMo on a series of natural language processing tasks and achieved significantly better results than the previous best results in 6 natural language processing tasks.  
The BERT (Bidirectional Encoder Representation from Transformers) language model [69] is the first word embedding trained on the encoder component of the Transformer neural network. Besides the fact that the BERT architecture is bidirectional, an important component is also the Masked Language Model (MLM), which masks or hides, by random selection, some of the tokens in the input data, with the aim of predicting the masked token based on the context.  
BERT was trained in two sizes, the basic BERTBASE and the large BERTLARGE, which differ in the number of layers with Transformer blocks (L), the number of hidden layers (H) and the number of attention heads (A), which ultimately results in a difference in the number of parameters (BERTBASE--L = 12, H = 768, A = 12, Parameters = 110 M, BERTLARGE--L = 24, H = 1024, A = 16, Parameters = 340 M). With the BERT language model, better results were achieved than in previous papers in eleven natural language processing tasks, with BERT representing the best achievement at the time of publication.  
BERT was also trained in a multilingual variant ([[https://github.com/google-research/bert/blob/master/multilingual.md](https://github.com/google-research/bert/blob/master/multilingual.md)]([https://github.com/google-research/bert/blob/master/multilingual.md](https://github.com/google-research/bert/blob/master/multilingual.md)); last accessed on 2 February 2023), mBERT, which includes 104 languages and is the _base_ model in terms of size.  
The RoBERTa [70] language model, as the authors state, is a proposal to improve BERT by correcting four areas that the authors believe are wrong in the original BERT architecture, i.e., training process--(1) longer training of the model and with larger batches, (2) removing the goal of predicting the next sentence, (3) training on longer sequences, and (4) dynamically changing the token masking pattern in the training data. An additional difference is the dataset used for training, where the authors use an additional 5 datasets in English, a total of 160 GB of uncompressed text data. The RoBERTa model achieves better results than the BERT model in three tasks (GLUE, RACE and SQuAD).  
While the RoBERTa model itself is not multilingual, XLM-RoBERTa [75], abbreviated XML-R, is its multilingual variant, trained on texts in 100 languages for which a corpus called CC-100 is created. The authors report that the XLM-R model gives significantly better results than the mBERT model, up to 23% improved accuracy for low-resource languages. In its architecture, the XLM-R model follows the XLM (cross-language model) presented in [PERSON] and Conneau [76] and is available in two sizes (following the two sizes of the BERT model), XLM-RBASE and XLM-R.