# Detection of Chinese Deceptive Reviews Based on Pre-Trained Language Model
## 4 Experimental Evaluations  
The dataset was divided into training, validation, and test sets. First, the dataset was split into training and test sets at a ratio of 70:30. Then, the training set was further divided into training and validation sets at a ratio of 90:10. Thus, 63% of the data were used to train the models, 7% were used to adjust the hyperparameters of the models and to preliminarily evaluate their classification ability, and 30% were used to test the accuracy of the models. To facilitate the performance evaluation, a confusion matrix was employed to calculate the accuracy, precision, recall, F1-score, and area under ROC (AUC) of the models. This study put more emphasis on accuracy, which was defined as the ratio of correctly classified samples to the total samples, to demonstrate the reliability of the classification results. A total of 15,089 reviews were collected from the Dianping and Samsung datasets. The number of valid reviews that were retained after the removal of the null values, reviews with a length of fewer than 30 characters, and unusable reviews was 14,954. The real and deceptive reviews from the Dianping and Samsung cases are listed in Table 4.  
The three different datasets (Dianping, Samsung, and combined datasets) were used to verify the classification efficiency of the models, and the classification results are listed in Tables 5-7, respectively. In addition, the aforementioned classification results were compared with the results of past studies that also used the same Samsung dataset (Table 8). \begin{table}
\begin{tabular}{c c c c c c c c} \hline \hline \multicolumn{1}{c}{\multirow{2}{*}{Static Vector}} & \multicolumn{1}{c}{**Methods**} & \multicolumn{1}{c}{**Length**} & \multicolumn{1}{c}{**Accuracy**} & \multicolumn{1}{c}{**Precision**} & \multicolumn{1}{c}{**Recall**} & \multicolumn{1}{c}{**F1-Score**} & \multicolumn{1}{c}{**AUC**} \\ \hline \multicolumn{1}{c}{Static Vector} & Word2 Vec + LSTM & 100 & 0.82 & 0.84 & 0.83 & 0.83 & 0.83 \\ \hline \multirow{8}{*}{\ \begin{tabular}{c} Samsung \\ \end{tabular}  } & BERT + SVM & 512 & 0.90 & 0.91 & 0.87 & 0.89 & 0.89 \\ \cline{2-8}  & BERT + DNN & 512 & 0.92 & 0.92 & 0.91 & 0.91 & 0.92 \\ \cline{2-8}  & BERT + LSTM & 100 & 0.89 & 0.89 & 0.89 & 0.89 & 0.90 \\ \cline{2-8}  & BERT + CNN & 100 & 0.86 & 0.86 & 0.85 & 0.85 & 0.85 \\ \hline \multirow{8}{*}{ \begin{tabular}{c} Samsung \\ \end{tabular}  } & BERT & \multirow{2}{*}{512} & BERT & \multirow{2}{*}{0.88} & \multirow{2}{*}{0.88} & \multirow{2}{*}{0.88} & \multirow{2}{*}{0.88} & \multirow{2}{*}{0.89} \\  & fine-tuned & & & & & & \\ \cline{1-1} \cline{3-8}  & BERT & \multirow{2}{*}{512} & \multirow{2}{*}{0.75} & \multirow{2}{*}{0.77} & \multirow{2}{*}{0.76} & \multirow{2}{*}{0.75} & \multirow{2}{*}{0. The lists in Tables 5 and 6 indicate that regardless of whether the reviews originated from Dianping or Samsung, both feature- and fine-tuning-based BERT models exhibited better overall performance than the Word2 Vec + LSTM model, demonstrating an 8-10% increase in accuracy, precision, recall, and F1-score. In addition to the individual testing of the models using the Dianping and Samsung datasets, the two datasets were combined to investigate whether the models proposed by the present study could effectively distinguish real from deceptive reviews from different industries. The results are listed in Table 7 which indicates that the highest detection accuracy of the models that used the combined dataset was 80%. The results listed in Tables 5-7 indicated that the models that used all the character vectors that were output by the last layer of BERT as training features (BERT + LSTM and BERT + CNN) did not significantly perform better than those that used [CLS] outputs by the last layer of BERT as training features (BERT + SVM and BERT + DNN). However, the model that used only the sentence vectors for classification yielded overall good results as evidenced by the accuracy of the SVM model scoring more than 80%.