# Automated Scoring of Translations with BERT Models: Chinese and English Language Case Study
## 3 Materials and Methods
### Procedures
#### 3.3.2 Model Evaluation  
After generating enough translations with known expected rankings, a model evaluation is necessary to ensure that the ranks generated by the system are reliable. This evaluation process includes analysis of the results of model evaluation indicators. * Indicators for model evaluation  
Given that the system built in this study is based on \"picking out the good translations and leaving the bad ones\", the following three indicators were used to evaluate the models: (1) [PERSON]'s rank correlation between the expected ranking (actual ranking) and the \begin{table}
\begin{tabular}{c c c} \hline \hline
**Synthetic Texts** & **Score** & **Rank** \\ \hline
1\% Expert translation text & 1 & 100 \\
2\% Expert translation text & 2 & 99 \\
3\% Expert translation text & 3 & 98 \\ \(\cdots\) & \(\cdots\) & \(\cdots\) \\
99\% Expert translation text & 99 & 2 \\
100\% Expert translation text & 100 & 1 \\ \hline \hline \end{tabular}
\end{table} 
Table 2: Synthetic texts generated by an expert translation. predicted ranking (automated assessment done by the system); (2) mean absolute error rate of ranking; and (3) maximum absolute error rate of ranking. 1. [PERSON]'s Rank Correlation Coefficient  
According to [PERSON] [40], the correlation coefficient is a standard measure for inter-rater reliability between Rater 1 and Rater 2, which can be computed using the popular Equation (2):  
\[r=1-\frac{6\sum d_{i}^{2}}{n(n^{2}-1)} \tag{2}\]  
where \(r\) represents [PERSON]'s rank correlation coefficient, \(d_{i}\) represents difference between the two ranks of each observation, \(n\) represents number of observations. In this study, the expected ranking is considered to have been done by Rater 1, while the predicted ranking was considered to have been done by Rater 2. As ranking is the focus, [PERSON]'s rank correlation was used for the analysis since it effectively tests the association between two ranked variables. The [PERSON]'s rank correlation coefficient, \(r\), ranges from \(-1\) to \(+1\). A \(r\) of \(+1\) indicates a perfect association of ranks, a \(r\) of zero indicates no association between ranks, and a \(r\) of \(-1\) indicates a perfect negative association of ranks. The closer \(r\) is to zero, the weaker the association between the ranked variables. Thus, if \(r\) is close to \(+1\), it indicates a strong positive correlation between the expected ranking and the predicted ranking, implying that the system is a highly reliable rater. Conversely, if \(r\) is low, the system is deemed to be of low reliability. Then, the four sets of synthetic texts are assessed by the designed system, and four predicted ranking lists can be obtained. The overall reliability of the system is shown in Equation (3):  
\[r_{final}=\sum_{i=1}^{4}r_{i} \tag{3}\]  
where \(r_{final}\) is the average of the four Spearman's rank correlation coefficients (\(r\)) of the four predicted ranking lists generated by the designed system compared with their expected ranking lists. 2. Absolute error rate of ranking  
The mean absolute error rate of ranking and the maximum absolute error rate of ranking are also included in the reliability analysis as they can reflect the prediction error rate of the automated assessment system in two dimensions. Let \(D_{1,j}\) denotes the synthetic text generated with the first \(j\%\) of \"Expert Translation 1\". The absolute ranking error, denoted by \(\mid E_{1,j}\mid\), is defined as the absolute value of the difference between the expected ranking and the predicted ranking of \(D_{1,j}\). The absolute ranking error rate, denoted by \(P_{1,j}\), is calculated as \(\frac{E_{1,j}}{100}\). For example, suppose \(D_{1,1}\) is expected to rank 100 th, but is ranked 98 th by the automated assessment system. Then \(\mid E_{1,1}\mid\) is equal to \(\mid 100-98\mid\) = 2, and \(P_{1,1}\) is equal to \(\frac{\mid\mid\mid}{100}\) = 0.02.