## 3 Materials and Methods
### Analysis  
To assess the quality of the translations generated by each GenAI and machine translator, a comprehensive evaluation was conducted using a combination of various indicators. For measuring the quality of each translation, including those that will serve as the reference later, descriptive statistics were computed based on the application of word embeddings. The translator yielding the highest average similarity (the closest cosine similarity to 1) produces translations that are most faithful to the original meaning. A higher arithmetic mean of the embedding similarity indicates superior translation quality, as it suggests that the translations are, on average, more aligned with the original text's semantic content. However, a high degree of variability in similarity scores (indicating a broad distribution of results) suggests a less reliable translator, as some translations may exhibit high quality while others diverge significantly from the intended meaning. A \begin{table}
\begin{tabular}{c c} \hline \hline
**Metric** & **Implementation** \\ \hline BLEU [43], GLEU [44], METEOR [46], CHRF [45] & NLTK library [60] available in Python \\ \hline YSi [49] & \
\begin{tabular}{c} GitHub repository [61] in C++ \\ and Python _subprocess_ library \\ \end{tabular} \\ \hline TER [48] & _Torchmetrics_ library [62], in Python \\ \hline RIBES [47] &
\begin{tabular}{c} GitHub repository [63] \\ and Python _subprocess_ library \\ \end{tabular} \\ \hline COMET [50] & COMET model in Hugging Face [64] \\ \hline \hline \end{tabular}
\end{table} 
Table 9: Evaluation metrics. lower standard deviation signifies greater consistency in translation quality. If a translator demonstrates a high mean similarity but also a high standard deviation, it suggests that while certain translations may be of high quality, others may significantly differ from the original. In the presence of extreme values, such as when the median is lower than the mean, the median is employed as a more robust measure to avoid the influence of outliers. Finally, the maximum and minimum values also provide crucial insights into the system's performance. A low minimum score for a translator may indicate that some of its translations are of suboptimal quality. Given the three distinct word embedding models used (the INCIBE dataset, a public corpus, and pretrained models), it is critical to assess whether the results remain consistent across all models. If a translator performs well in all three models, it provides strong evidence that its translations are more accurate. Conversely, if a translator excels in one model but performs poorly in another, it may indicate that its translation style aligns more closely with specific types of texts (e.g., aligning better with INCIBE but less so with pretrained models). Therefore, the individual performance of each translator was evaluated based on the number of cases (vulnerabilities) in which it achieved the best result compared to other models. This metric, referred to as Victories, indicates the reliability of the translator. A translator with a high number of victories can be considered more reliable, and a consistently superior performance in the majority of cases strongly suggests better translation quality. After confirming the quality of the reference translation provided by INCIBE and establishing that no tool can be excluded (as all translators achieved sufficiently high translation quality), we propose the following assessment framework: The evaluation system is designed to highlight translations that exhibit statistically significant differences and do not receive similar evaluations across different metrics, thereby indicating notable performance variations among the translation tools. This aspect is referred to as Consistency. To evaluate consistency, potential correlations and significant differences between translation evaluations were analyzed. The correlation analysis provides insights into whether translations are generally aligned in terms of overall quality.