# Automated Scoring of Translations with BERT Models: Chinese and English Language Case Study
## 4 Results
### C-E Translations  
#### 4.1.1 Results of Model Selection Indicators  
Table 3 demonstrates the overall results of model selection indicators for the BERT-based system (C-E). In Table 3, the values of \(r\) are all greater than 0.99, with a corresponding \(p<2.2\times 10^{-16}\), indicating a strong correlation between the expected ranking and the predicted ranking. For all the BERT models, the mean absolute error rates of ranking are all below 0.0229, and the maximum absolute error rates of ranking do not exceed 0.1400. These results suggest that the average and maximum fluctuations are under 2.29% and 14%, respectively, indicating that high-quality translations are not being filtered out. To be specific, there are four English BERT models: base and large variations, for both cased and uncased input text. The BERT (large cased)-based system outperforms the other models, with a correlation coefficient (\(r\)) of 0.999039925 and a corresponding \(p<2.2\times 10^{-16}\). The mean absolute error rate of ranking is 0.0066, and the maximum absolute error rate of ranking is 0.0600. The BERT (large uncased)-based system performs relatively poorly in comparison to the other three systems, but still exhibits a correlation coefficient of over 0.99 with a corresponding \(p<2.2\times 10^{-16}\). The mean absolute error rate of ranking is 0.0229, and the maximum absolute error rate of ranking is 0.1400. These results suggest that the BERT-based system is reliable enough to predict the ranking of C-E translations. Additionally, the absolute error rate curves in Figures 1-4 reveal that the error rates are remarkably low. Figure 1 illustrates the absolute error rate of ranking of synthetic texts generated based on the four expert translations in the BERT-based system (C-E, bert-base-uncased), with four sets of curves in (a-d). Overall, the figure shows a low absolute error rate, with a mean absolute error rate of ranking of 0.0137. Specifically, among the 400 synthetic texts, 49.5% of them have predicted rankings that match their expected rankings, resulting in an absolute error rate of 0. For 46% of the texts, the absolute error rates fall in the range of 0.01 to 0.05. Only 4.5% of the texts have absolute error rates of more than 0.05, with a maximum \begin{table}
\begin{tabular}{l c c c c} \hline \hline \multirow{2}{*}{**Measures**} & **[PERSON]’s Rank Correlation** & **Significance (\(p\))** & **Mean Absolute Error Rate of Ranking** & **Maximum Absolute Error Rate of Ranking** \\ \hline bert-base-uncased & 0.996504650 & \(<2.2\times 10^{-16}\) & 0.0137 & 0.1100 \\ bert-base-cased & 0.998805875 & \(<2.2\times 10^{-16}\) & 0.0077 & 0.0700 \\ bert-large-uncased & 0.991977225 & \(<2.2\times 10^{-16}\) & 0.0229 & 0.1400 \\ bert-large-cased & 0.999039925 & \(<2.2\times 10^{-16}\) & 0.0066 & 0.0600 \\ \hline \hline \end{tabular}
\end{table} 
Table 3: Results of model selection indicators for BERT-based systems (C–E). of 0.11. These results indicate that the errors of the BERT (base uncased)-based system do not significantly impact the selection of the top-ranked C-E translations. Figure 1: Absolute error rate of ranking of BERT-based system (C–E, bert-base-uncased). Figure 2: Absolute error rate of ranking of BERT-based system (C–E, bert-base-cased). absolute error rate of 0. For 40.25% of the texts, the absolute error rates fall in the range of 0.01 to 0.05. Only 0.5% of the texts have absolute error rates of more than 0.05, with a maximum of 0.07. These results suggest that the errors of the BERT (base cased)-based system do not have a significant impact on the selection of the top-ranked C-E translations. Figure 3 illustrates the absolute error rate of ranking of synthetic texts generated using the BERT-based system (C-E, bert-large-uncased), with four sets of curves in (a-d). The results show a low absolute error rate, with a mean absolute error rate of ranking of 0.0229. Specifically, among the 400 synthetic texts, 28.25% of them had predicted rankings that matched their expected rankings with an absolute error rate of 0.