## 5 Modeling
### BERT-Based Classifier Model  
BERT, which stands for Bidirectional Encoder Representations from Transformers, is a language model [6]. This is a masked model trained for language understanding and solving GLUE tasks [2]. Generally, the idea of masking was applied in the following way. A text corpus was initialized with random masks replacing word tokens. After that, the model was trained to predict the missing token based on its context. In addition, BERT was trained on the next sentence prediction task. One of the main advantages of BERT is its bidirectional design, meaning that the model is trained to analyze both left and right context, unlike only left-to-right analysis [6].  
BERT is defined as a framework that consists of two parts: pre-training and fine-tuning. The process described before is the pre-training process for language understanding. The fine-tuning stage is presented as adding additional layers to the model and training on the specific task.  
The authors followed the fine-tuning approach. The authors of BERT supply a family of pre-trained models of different sizes and on cased/uncased data. The model size is defined as a number of layers (L), hidden units (H), and attention heads (A). Considering that during the text standardization, all text was converted to lowercase, the authors used an uncased model, which means that the model was pre-trained to ignore the case of words. Models are available to use via the TensorFlow HUB portal [24]; each model contains its own pre-process layer.  
Although the BERT is recommended to be fine-tuned with only one additional layer, it is mentioned that additional layers can be applied according to specific tasks.  
The authors propose the following structure of the classifier:  
Standardizer: text standardization layer described in Section 4.3;  
Pre-process: text pre-process layer added according to the selected BERT model;  
BERT encoder: BERT model selected;  
Dropout: random dropout of units;  
Dense layer: internal dense layer with ReLU activation function;  
Dropout: random dropout of units;  
Dense layer: output classifier layer with output units for each class. Considering that each output solves the binary classification problem, the Sigmoid activation function is applied.  
The BERT model contains several outputs available for solving different tasks. In this study, the authors use pooled output [6]. The output shape is (batch size \(\times\) BERT hidden units). As a result, each text sequence submitted is passed to output and encoded.  
This architecture is defined in bert_model.BertSentimentModel class. The BERT model name is submitted as a parameter; the appropriate model will be used from the TensorFlow HUB.  
In this study, the authors used a model from the small BERT family with 2 layers, 128 hidden units, and 2 attention heads, uncased. The reason for this is that the authors have limited computation resources since the training process was established on a laptop having the following configuration: Intel Core i5-7200U, 8 Gbs of RAM, and SSD drive (Intel, Santa Clara, CA, USA).  
As for the other parameters, both dropout rates were set to 0.4 (randomly drop 40% of units), internal dense units were set to 256, and the output dense units were set to 27.