# Cascade or Direct Speech Translation? A Case Study +
## 8 Conclusions  
In this work, we presented the results of a case study in Basque-Spanish speech translation, centred on comparative evaluations of cascade and direct approaches to the task. Figure 3: BLEU score differences on WER-based quartile subsets of the mintzai-ST test sets, between: (**a**)cas ind and s-trf.pt.kd ind in EU-ES (**b**)cas ind and s-trf.pt.kd ind in ES-EU, (**c**)cas all and wav2 vec-st.pt in EU2 ES, and (**d**)cas all and wav2 vec-st.pt in ES2 EU. The red line indicates the average difference over joint subsets. Quartiles range from lower WER (Q4) to higher WER (Q1). We first described the mintzai-ST corpus, based on the proceedings of the sessions of the Basque Parliament between 2011 and 2018. We extended the initial description of the corpus in [14] with a detailed examination of the different alignment and filtering steps, along with an analysis of the data distribution in the corpus. Different ST models were compared, based on cascade and end-to-end approaches, starting with baseline results that included end-to-end models trained strictly on the source-target data, which resulted in cascade systems significantly outperforming their direct counterparts, with or without additional data. Several variants of advanced end-to-end models were then prepared, exploiting pre-training and knowledge distillation techniques in particular. As was the case in other studies exploiting these techniques [18; 34], all advanced variants significantly closed the gap with their cascaded counterparts, in terms of automated metrics, including a variant based on wav2 vec[63], which outperformed all other models in Spanish to Basque translation. This latter model proved the most efficient in terms of metrics, providing further support to the usefulness of self-supervised training on audio data prior to performing speech translation. The comparison with other models in this study was less direct, however, given the use of large amounts of audio data to pretrain the models. Among other variants, pretraining and knowledge distillation both proved critical to drastically increase the performance of end-to-end models, with the S-Transformer model outperforming similarly trained model variants that featured architectural differences. To further evaluate the differences between the two main ST approaches, cascade and direct, a manual evaluation was carried out on the translations from the best-performing models in each case. Although between 30% and 40% of the translations overall were considered of similar quality by a panel of native speakers of the two languages, the translations generated by the cascade models were preferred by a significant margin in all but one case, where the preferences were equally distributed. These results complement other comparative manual evaluations such as [24], though reaching differing conclusions, as in our study and specific evaluation protocol, cascade translations were preferred overall. We also conducted targeted evaluations along two axes. First, we evaluated divergent translations under the cascade and direct approaches, where the latter approach performed better on numbers and punctuation against the specific cascade models in this study, although in both cases, improvements could be straightforwardly achieved in a cascade approach. Additionally, we compared model result differences according to the WER scores, to measure the potential impact of ASR errors on MT results in a cascade approach and the eventual higher robustness of a direct approach in this respect. The results were inconclusive, with better relative scores for direct models on more difficult input in Basque to Spanish translation, but a reverse tendency in Spanish to Basque. From this study, it appears that the gap between cascade and direct approaches has been reduced significantly with recent approaches to direct ST modelling, in line with similar findings [19; 23; 24].