# Automated Scoring of Translations with BERT Models: Chinese and English Language Case Study
## 3 Materials and Methods
### Procedures
#### 3.3.4 Measures of Reliability  
The final step is the reliability analysis. Reliability entails a comprehensive comparison between scores assigned by the automated scoring systems and human raters based on accuracy, recall, and F0.5-Score. Given that automated scoring systems are designed to emulate human scoring, it is systematically evaluated against human scoring metrics. This evaluation serves as a benchmark for assessing the alignment and effectiveness of system scoring in replicating human scoring. A set of randomly selected translations, including 138 C-E participant translations and 262 E-C participant translations, which have been manually assessed, was used for the reliability analysis.  
* Accuracy is a metric used to measure how well a model performs across all categories. It is calculated by dividing the number of correct predictions by the total number of predictions. Mathematically, it can be expressed as shown in Equation (9):  
\[Accuracy=\frac{True_{positive}+True_{negative}}{True_{positive}+True_{ negative}+False_{positive}+False_{negative}} \tag{9}\]  
where \(True_{positive}\) are the number of positive samples correctly identified by the system, \(True_{negative}\) are the number of negative samples correctly rejected by the system, \(False_{positive}\) are the number of negative samples incorrectly identified by the system, and \(False_{negative}\) are the number of positive samples that incorrectly rejected by the system.  
* Recall is a metric used to measure the ability of a system to correctly identify all positive samples. It is calculated as the ratio of the number of true positives to the sum of true positives and false negatives, as shown in Equation (10):  
\[Recall=\frac{True_{positive}}{True_{positive}+False_{negative}} \tag{10}\]  
where \(True_{positive}\) are the number of positive samples correctly identified by the system, and \(False_{negative}\) are the number of positive samples that incorrectly rejected by the system.  
* F-measure  
F-measure is a metric that combines both precision and recall into a single measure. It is often used to assess the overall performance of a classification system. The \(F0.5\)-Score is a variant of the F-measure that places more emphasis on precision than recall, as shown in Equation (11):  
\[F0.5=(1+0.5^{2})\frac{Precision\times Recall}{(0.5^{2}\times Precision)+Recall} \tag{11}\]  
Precision is calculated as the ratio of the number of positive samples correctly classified to the total number of samples classified as positive (correct or incorrect). Precisionmeasures the accuracy of the system in classifying samples as positive. The higher the value of precision, the fewer false positives are predicted. It can be calculated as shown in Equation (12):  
\[Precision=\frac{True_{positive}}{True_{positive}+False_{positive}} \tag{12}\]  
where _Precision_ and _Recall_ are as defined previously, the \(F0.5\)-Score ranges between 0 and 1, with higher values indicating better performance. A score of 1 indicates perfect precision and recall, while a score of 0 indicates no correct predictions.