# Detection of Chinese Deceptive Reviews Based on Pre-Trained Language Model
## 3 Methodology
### Phase 3: Model Designing
#### 3.3.2 BERT Classification Model  
BERT provides two training models with different sizes, namely, 12-layer, 768-hidden BERTBASE and 24-layer, 1024-hidden BERTLARGE. In this study, the BERT-base for Chinese was used. The BERT models adopt two approaches: fine-tuning and feature approach. Fine-tuning allows users to use their own datasets and fine-tune the BERT model to complete specific NLP tasks. The advantage of this approach is that many parameters of the model do not need re-learning. Hence, the training and learning efficiency is high. Moreover, it indicates that even if the size of the training-data samples is limited, the classification output remains satisfactory. The feature approach uses the BERT model to extract text features. The first token [CLS] that is output by the last layer of the BERT model is used as the vector representation of the corresponding input sequence, and the output vectors that follow [CLS] are used as the word vectors of each word in the corresponding input sequence. In addition, the output vector of the last layer of the BERT model can be input into other models for training. This study used the BERT feature-extraction method to construct four classification models. In addition, BertForSequenceClassification (a text classification model), which was developed by [PERSON] [34], was employed, and the main dataset in this study was input to fine-tune the model. Please note that different variants of our proposed model could deal with different datasets. In other words, we could choose the variant according to the validation result in a given dataset. Fine-tuning  
BertForSequenceClassification [34] was employed in the present study to fine-tune the dataset, which was composed of a BERT model with a single linear classification layer. When data were input into the model, the pre-training model and linear classification layer were trained together. Adam was used as the optimizer. The learning rate was set at  
Figure 6: Architecture of the Word2 Vec + LSTM model. 2 \(\times\) e\({}^{-5}\) with three epochs and a batch size of eight. Figure 7a shows the architecture of the BERT fine-tuning model. BERT + SVM  
The first feature model built by this study used the first token [CLS] (a set of 768-dimensional sentence vectors) that was output by the last layer of the BERT model as an input to the SVM model. The radial basis function kernel was used as the kernel function. Figure 7: BERT classification models. The SVM classifier output was either \"0\" or \"1\" (\"0\" represented real reviews and \"1\" represented deceptive reviews). Figure 7b shows the architecture of the BERT + SVM model. * BERT + DNN  
The second feature model used the first token [CLS] that was output by the last layer of the BERT model as an input to the DNN model. The input layer contained 768 nodes with two fully connected hidden layers (with 484 and 128 hidden nodes). The final output node was one. The activation function of each layer was performed by the rectified linear unit (ReLU), and the activation function of the last layer was compressed using the sigmoid function. The dropout of each layer was set to 0.1. The optimizer used was AdaMax. The learning rate was 0.0001, the number of epochs was set to 20, the batch size was split to 1024, and the final output was either \"0\" or \"1\" (\"0\" represented a real review and \"1\" represented a deceptive review). Figure 7c shows the architecture of the BERT + DNN model. * BERT + LSTM  
The third feature model used all the character vectors that were output by the last layer of the BERT model as inputs to the LSTM model. However, because the length of the majority of the reviews was less than 100 characters, when the LSTM input length was fixed at 512 characters, a large number of null values had to be added. To avoid the effect of the null values on the training performance, the maximum input length was set to 100. Each character vector contained 768 dimensions. The character-vector matrix for a single review input was (100, 768), which was the LSTM input shape.