## 4 Machine Learning Methods
### Features Used for Text Representation
#### 4.2.4 Language Models  
Language models, pre-trained on large bodies of unlabeled texts, represent words by vectors in a space where words with similar meanings are close to each other. We use the following models: FastText [45], Embeddings from Language Model (ELMo) [26], Bidirectional Encoder Representations from Transformer (BERT) [27] and XLM-RoBERTa [46].  
The approach of FastText is based on the Word2 Vec model principles, where word distributions are predicted by their context, but FastText uses character trigrams as its basis vector representation. Each word is represented as a sum of its trigram vectors, which are then used as the base for continuous bag of words or skip-grams algorithms [52]. Such a model is simpler to train due to decreased dictionary size: the number of character n-grams is less than the number of unique words. Another advantage of this approach is that morphology is accounted automatically, which is important for the Russian language.  
Instead of using fixed vectors for every word similar to how FastText does, ELMo word vectors are sentence-dependent. ELMo is based on the bidirectional language model (BiLM), which learns to predict the next word in a word sequence. Vectors obtained with ELMo are contextualized by means of grouping the hidden states (and initial embedding) in a certain way (concatenation followed by weighed summation). However, predicting the next word in a sequence is a directional approach and therefore is limited in taking the context into account. This is a common problem in training NLP models, and is addressed in BERT.  
BERT is based on the transformer mechanism, which analyzes contextual relations between words in a text. The BERT model consists of an encoder extracting information from a text and a decoder which gives output predictions. In order to address the context accounting problem, BERT uses two learning strategies: word masking and logic check of the next sentence. In the first strategy, 15% of the words are replaced with a token \"MASK\", the original words later being the target for the neural network to predict. In the second learning strategy, the neural network is used to determine whether two input sentencesare a logical sequence or just a random set of unrelated phrases. In BERT training, both strategies are used simultaneously by minimizing their combined loss function.  
XLM-RoBERTa is a model similar to a masked BERT language model based on Transformers [53]. The main differences between XLM-RoBERTa and BERT are the following. Firstly, XLM-RoBERTa was trained on a larger multilingual corpus from the CommonCrawl project which contains 2.5 TB of texts. Russian is the second language by texts count in this corpus after English. Minibatches during model training included texts in different languages. Secondly, XLM-RoBERTa was trained only for the masked token prediction task; its loss function did not involve the next sentence prediction learning strategy. Thirdly, it used a different tokenization algorithm: while BERT used WordPiece [54], XLM-RoBERTa used SentencePiece [55]. The vocabulary size in XLM-RoBERTa is 250,000 unique tokens for all languages.  
There are two versions of the model: XLM-RoBERTa-base (with 270M parameters) and XLM-RoBERTa-large (with 550M), of which we use the latter.