## 3 Datasets and Methods
### Methods  
Transformer-based deep neural network architectures are well known for their effectiveness in different natural language processing problems, including machine translation and text generation [13]. Bidirectional encoder representations from transformers (BERT) recently [40] have disrupted the fields of language modeling and learning representations from unstructured text. Such representations are derived as a result of semi-supervised pre-training on a large corpus, stored as a set of weights of a neural network (usually, hundreds of millions or larger) and later can be used for fine-tuning to solve a down-stream task. In our study, we consider three pre-trained transformer-based neural architectures capable of processing texts in different languages. The selected state-of-the-art multilingual pre-trained models include the following (the checkpoints of the transformer-based models are available at [[https://huggingface.co](https://huggingface.co)]([https://huggingface.co](https://huggingface.co)) (accessed on 4 May 2022)):  
* M1: distilled version of multilingual BERT (distil. mBERT) [41],
* M2: base multilingual BERT (mBERT) [40],
* M3: Microsoft's Multilingual-MiniLM-L12-H384 (MS-MiniLM) [42]  
These three multilingual models were pre-trained on masked language modeling tasks and are able to represent text written in many languages. Before applying them in C/A rating prediction for words, we fine-tuned them using a simple architecture (see Figure 1).  
We proposed to fine-tune each of the models on a training subset of words from the expert C/A dictionary with ratings and then test the fine-tuned model on the C/A rating prediction (regression) task. Typically, pre-trained models can be fine-tuned on a classification task using the [CLS] token. In our experimental setup, each word was provided with a real number in the range [1; 5] as a label. Therefore, in fine-tuning, we used concreteness rating prediction as a numeric output (Figure 1). To this end, the loss function and the training dataset should be modified accordingly; we used mean squared error as a loss function. Training sets for each language should be scaled in the following manner: ratings from the dictionary are linearly transformed from a range [1; 5] to a range between 0.0 ('abstract') and 1.0 ('concrete'). On the one hand, this linear transformation does not affect the performance measures (e.g., Pearson's correlation coefficient is not sensitive to scaling). On the other hand, the training process becomes more stable if target labels are in the [0.0, 1.0] range.  
The next step before fine-tuning is setting the maximum input sequence length. We set the maximum length of the sequence to 10 tokens as most of the words should not be longer than ten tokens from BERT's vocabulary. All other input tokens are padded and are not used in calculating the loss function. During fine-tuning, we use typical settings for learning rate (\(2\times 10^{-5}\)) and batch size (bs = 32). We also use 2 to 4 epochs for fine-tuning, as more epochs led to overfitting on the validation dataset. The architecture is shown in Figure 1. Here, one can see the input sequence that starts with a special [CLS] token along with internal token representations (in 'Transformer layers') and the output of a model. This schema is the same for the three models we selected for testing: MS-MiniLM, mBERT, and distill mBERT.