# Cascade or Direct Speech Translation? A Case Study +
## 7 Targeted Evaluations of Cascade and Advanced Direct Models
### Error Propagation  
As previously noted, one of the expected advantages of direct ST models is avoiding the propagation of ASR errors to the MT component, an aspect that was one of the characteristics of earlier cascade systems. To measure this effect, albeit indirectly, we evaluated the translation results on input classified according to the WER scores obtained with the top-performing ASR component described in Section 4.1.1, with WER scores assumed to reflect the percentage of ASR errors that may impact a cascaded MT model. Clearly, since higher WER scores may be due to audio input that is intrinsically difficult to process for any automated recognition system, cascaded or direct, this evaluation was an approximation explored to determine eventual differences in behaviour between cascade and direct models. For these evaluations, we used the four systems selected in Section 7. We first measured the correlation between the WER and BLEU scores for each considered model, to determine the linear relationship between the two scores, if any. Additionally, we measured the correlation between WER and the difference of BLEU scores obtained by the compared cascade and end-to-end models, to determine the relation between WER and performance variation for the two types of models. To account for the fact that WER scores are less reliable on short audio, we filtered all audio whose corresponding transcription consisted of 10 words or less; WER scores were also normalised in the 0-100 range. The results in terms of Pearson correlation coefficients are shown in Table 17. It is worth noting that negative coefficients between WER and BLEU indicate a positive correlation, since the metrics are reversed in terms of rank interpretation, with a higher WER indicating worse results, while a higher BLEU indicates better results. For both cascade and direct models, the correlation between the WER and BLEU scores was small overall, with lower marks for ES-EU than for EU-ES. The correlation was comparatively higher between the WER and BLEU scores' differences, being moderate across the board. To further evaluate the impact of input difficulty as indicated by the BLEU scores, the test data were further first divided into quartiles according to the WER scores obtained by the ASR component. We then computed the difference in the BLEU scores between paired models on the data in each quartile. The results are shown in Figure 3. \begin{table}
\begin{tabular}{l l l} \hline \hline
**Subset** & **Translation** & **Example** \\ \hline \multirow{4}{*}{Name} & \multirow{2}{*}{Reference EN} & Pasando al turno de réplica, tiene used la palabra, serhora [PERSON]. \\  & & _Turning to the reply, you have the floor, Mrs. [PERSON]_ \\ \cline{2-3}  & & \\ \cline{1-1}  & & \\ \cline{1-1}  & & \\ \cline{1-1}  & & \\ \cline{1-1}  & & \\ \cline{1-1}  & & \\ \cline{1-1}  & & \\ \hline \hline \end{tabular}
\end{table} 
Table 16: _Cont._ \begin{table}
\begin{tabular}{c c c c c} \hline \hline
**Lang** & **Data** & \(\boldsymbol{\rho(\text{w},\text{b CAS})}\) & \(\boldsymbol{\rho(\text{w},\text{b E2E})}\) & \(\boldsymbol{\rho(\text{w},\text{(b CAS})\) **– (\text{b E2E}))** \\ \hline ES–EU & IND & \(-\)0.14 & \(-\)0.13 & 0.40 \\ ES–EU & ALL & \(-\)0.15 & \(-\)0.10 & 0.35 \\ EU–ES & IND & \(-\)0.26 & \(-\)0.25 & 0.32 \\ EU–ES & ALL & \(-\)0.24 & \(-\)0.20 & 0.43 \\ \hline \hline \end{tabular}
\end{table} 
Table 17: Pearson correlation between WER\(\downarrow\) (W) and BLEU\(\uparrow\) (B) scores for the selected Cascade (CAS) and End-to-End (E2E) models. All results were statistically significant (\(p<0.0001\)). Under the assumption that higher WER scores would correlate with a larger number of errors that would impact the MT component of cascade models more than they would an end-to-end model, the expectation would be that the difference in BLEU scores is lower than the average where cascade models outperform direct models and higher than average where direct models score better than their cascade counterpart.