# Cascade or Direct Speech Translation? A Case Study +
## 7 Targeted Evaluations of Cascade and Advanced Direct Models
### Manual Ranking Task  
This task consisted of a manual evaluation of the translations generated by the different selected systems, previously described, where users were presented the source transcription and two competing translations and had to indicate whether one was preferred or whether they could be considered similar in accuracy and fluency. As evaluation data for the task, we extracted new data from the 2019 Sessions of the Basque Parliament, not covered by the mintzai-ST corpus, and sampled 100 representative audio inputs. To select the samples, the content of a complete Session was first translated with each of the four selected ST models. The data were then log-normalised and split into quartiles according to the length of the transcriptions, with 25 samples randomly selected from each quartile to provide a representation of different types of input in terms of duration. Table 13 indicates the BLEU scores obtained by each model on the selected samples, to be taken as a first indication of the relative translation quality on the samples. The results were in line with the previous indications of relative model strength in terms of metrics, with slightly better scores overall obtained by the cascade models. For the manual three-way ranking task, two separate groups of users were defined to handle, with the following characteristics: \begin{table}
\begin{tabular}{c l l l l} \hline \hline
**Lang** & **Model** & **Data** & **WER** & **BLEU** \\ \hline EU–ES & CAS & IND & 12.07 & 29.2 \\ EU–ES & S-TRF.PT.KD & IND & - & 28.4 \\ EU–ES & CAS & ALL & **11.78** & **34.7** \\ EU–ES & WAV2 Vec-ST.PT & ALL & - & 31.4 \\ \hline ES–EU & CAS & IND & 7.23 & 20.9 \\ ES–EU & S-TRF.PT.KD & IND & - & 20.4 \\ ES–EU & CAS & ALL & **7.21** & 22.7 \\ ES–EU & WAV2 Vec-ST.PT & ALL & - & **24.7** \\ \hline \hline \end{tabular}
\end{table} 
Table 12: Summary of the results with cascade and advanced end-to-end models. Best-performing systems are indicated in bold. * Group A was tasked with comparing the translations between models trained only on in-domain data, namely CAS IND and S-TRF.PT.KD. For Spanish to Basque, nine users completed the full evaluation and one user only partially completed the task (forty out of one-hundred). For Basque to Spanish, eight users completed the full evaluation and one user only partially completed the task (twenty-three out of one-hundred);
* Group B was tasked with comparing the translations between models trained on additional data, namely CAS all and wav2 vec-st.pt. For Spanish to Basque, 12 users completed the full evaluation; for Basque to Spanish, 11 users completed the full evaluation. The evaluators who volunteered for the task were native speakers of Basque and Spanish, but not professional translators. They were provided guidelines on the task itself and on the use of the evaluation environment, which is based on the Apprraise system [64] and provides inter-annotator agreement statistics. The results of the manual evaluation are shown in Table 14, with inter-annotator agreement measured in terms of [PERSON]'s alpha [65], [PERSON]'s Kappa [66], [PERSON]' S [67], and [PERSON]'s Pi [68] (omitted from the table are the number of cases that were skipped by users, which were none in all cases for Group B and, for Group A, amounted to 2.66% in ES-EU and 0.12% in EU-ES). Overall, translations from the cascade systems were preferred by a large margin for systems trained only on in-domain data, in both translation directions, and in Basque to Spanish for the systems trained on additional data. Only in the latter case, for Spanish to Basque, were the systems considered relatively equal. It also worth noting that between 30% and 40% of the translations, approximately, were considered of equal quality overall. Inter-annotator agreement was moderate overall and significantly higher in both cases of Basque to Spanish translation, where the differences were larger between the compared systems.