# Cascade or Direct Speech Translation? A Case Study +
## 6 Advanced End-to-End Models
### Comparative Direct Models' Results  
We applied each of the previously described variants on the mintzai-ST test sets, with the results shown in Tables 10 and 11, for Spanish to Basque and Basque to Spanish, respectively.  
In both translation directions, the wav2 vec-ST models performed significantly better than the alternatives, overall. Although these results illustrate the positive impact of wav2 vec 2.0 cross-lingual speech representations for the ST task as well, it is worth noting that all other models were trained only on the in-domain mintzai-ST data, which feature significantly lower volumes than those used to pretrain wav2 vec 2.0.  
Another notable result is the significant impact of pretraining, with a 9.8 BLEU point increase for the S-Transformer model for Basque to Spanish,and 4.9 points for Spanish to Basque. Knowledge distillation was also impactful across the board, with overall increases of at least two BLEU points over pretrained models.  
Turning to architecture variants, in both translation directions, the default S-Transformer model outperformed the variant with eleven SA encoder layers, four decoder layers, and without 2D SA layers, which in turn performed better than the variant with a linear layer replacing both the CNN and SA layers. These comparative metrics' results were consistent for the variants with pretraining only and knowledge distillation in addition to pretraining.  
Table 12 summarises the comparative results between cascade and advanced end-to-end models, trained on either in-domain data only or on additional data (as previously indicated, for the wav2 vec-ST model, additional data refer to the datasets used to pretrain  
\begin{table}
\begin{tabular}{c c c c} \hline \hline
**Lang** & **Model** & **BLEU** & **CHRF** \\ \hline EU–ES & S-TRF & 17.0 & 42.5 \\ EU–ES & S-TRF.PT & 26.8 & 52.6 \\ EU–ES & S-TRF.PT.KD & 28.4 & 53.2 \\ EU–ES & CNN.11E-4D.TRF.PT & 25.0 & 53.5 \\ EU–ES & CNN.11E-4D.TRF.PT.KD & 28.1 & 53.7 \\ EU–ES & LINEAR.TRF.PT & 24.6 & 51.4 \\ EU–ES & LINEAR.TRF.PT.KD & 26.8 & 52.2 \\ EU–ES & wav2 vec-ST & 29.8 & 57.6 \\ EU–ES & wav2 vec-ST.PT & **31.4** & **58.5** \\ \hline \hline \end{tabular}
\end{table}
Table 10: Comparative results for end-to-end model variants on Basque to Spanish translation. The best results, indicated in bold, were statistically significant against all other results, for \(p<0.05\); differences between top-performing systems in bold were not statistically significant.  
\begin{table}
\begin{tabular}{c c c} \hline \hline
**Lang** & **Model** & **BLEU** & **CHRF** \\ \hline ES–EU & S-TRF & 12.9 & 48.4 \\ ES–EU & S-TRF.PT & 17.8 & 54.7 \\ ES–EU & S-TRF.PT.KD & 20.4 & 57.3 \\ ES–EU & CNN.11E-4D.TRF.PT & 15.6 & 53.3 \\ ES–EU & CNN.11E-4D.TRF.PT.KD & 19.2 & 56.2 \\ ES–EU & LINEAR.TRF.PT & 17.0 & 53.7 \\ ES–EU & LINEAR.TRF.PT.KD & 18.9 & 55.9 \\ ES–EU & wav2 vec-ST & **24.5** & **61.9** \\ ES–EU & wav2 vec-ST.PT & **24.7** & **62.2** \\ \hline \hline \end{tabular}
\end{table}
Table 11: Comparative results for end-to-end model variants on Spanish to Basque translation. The best results, indicated in bold, were statistically significant against all other results, for \(p<0.05\); differences between top-performing systems in bold were not statistically significant.  
the model). In all cases, the gap between the direct and cascade models was significantly reduced, compared to the baseline results of Section 5. End-to-end models obtained only slightly lower results overall in both directions among systems trained only on in-domain data. In Spanish to Basque, the end-to-end model based on WAV2 vec-ST even outperformed the cascade model trained on all data. For Basque to Spanish though, the cascade model still outperformed the best end-to-end model on the test sets.