# Cascade or Direct Speech Translation? A Case Study +
## 7 Targeted Evaluations of Cascade and Advanced Direct Models
### Divergence on Specific Phenomena  
To assess more detailed differences between the cascade and direct approaches, we selected and evaluated translation subsets addressing three different phenomena, after a preliminary manual evaluation of the data to identify aspects that led to divergent translations. We thus identified all source transcriptions in the test sets that contained: numbers; a specific subset of punctuation marks, namely question, exclamation, and ellipsis marks; named entities introduced by markers in Basque or Spanish corresponding to _Sir_, _Madam_, or similar, which are almost systematically used to refer to other participants by name in \begin{table}
\begin{tabular}{c c c c c c} \hline \hline
**Lang** & **CAS IND** & **s-trf.pt.kd** & **CAS All** & **wav2 vec-st.kd** \\ \hline ES–EU & 18.4 & 16.7 & 22.0 & 21.7 \\ EU–ES & 30.7 & 28.2 & 35.5 & 33.0 \\ \hline \hline \end{tabular}
\end{table} 
Table 13: BLEU scores on manual evaluation samples. the sessions of the Basque Parliament. The number of identified samples, and BLEU scores for the four selected systems on each subset, are indicated in Table 15.  
On the numbers subset, the end-to-end models performed better overall, although this was mainly due to the fact that numbers were provided in word rather than numeral form by ASR in the cascade system. Additional processes could perform this conversion in a cascade setup, similar to the use of additional processes to insert punctuation marks. The differences in translations of numbers were thus mainly significant to indicate that some of the gains obtained by direct models were related to this variable way of representing numbers in the final translations. Translations also differed in terms of punctuation, in this case also with higher marks obtained overall by end-to-end models, except for Basque to Spanish with additional data. One of the main reasons for this divergence comes from the limitations of the specific punctuation model used for the cascade system, which only covered commas and periods. In contrast, the end-to-end models exploited the source-target punctuation data directly and could model the whole spectrum of punctuation marks in the datasets. Finally, the divergences in terms of names were less marked, with slightly better scores with cascade models overall for Basque to Spanish. For Spanish to Basque, nearly identical results were obtained with models trained only on in-domain data and slightly better scores for wav2 vec-st.pt against CAS ALL. Table 16 provides examples of divergent translations between cascade and direct models. Except the punctuation case, which is only correct in the direct translation example, the other examples illustrated divergences that may all be considered correct. The differences nonetheless impacted the BLEU scores on the single references used in these evaluations, an aspect that needs to be factored in when comparing translation models [16; 24]. \begin{table}
\begin{tabular}{l l l} \hline \hline
**Subset** & **Translation** & **Example** \\ \hline \multirow{4}{*}{Number} & Reference EN & Lean, si no, el articulo 6. \\  & _If not, read article 6._ \\ \cline{2-3}  & CAS ALL EN & Vease, de lo contrario, el articulo sexto. \\ \cline{2-3}  & WAV2 VEC.PT EN & Si no, se ha visto el articulo 6. \\ \cline{2-3}  & ER & _If not, article 6 has been seen._ \\ \hline \multirow{4}{*}{Punct} & Reference EN & Lan-erreforma indargabetu da? \\  & EN & _Is job reform repealed?_ \\ \cline{2-3}  & CAS IND EN & Lan-erreforma indargabetu egin da. \\ \cline{1-1} \cline{2-3}  & EN & _job reform is repealed._ \\ \cline{1-1} \cline{2-3}  & S-TF.PT.KD & Indargabetu al da lan-erreforma? \\ \cline{1-1} \cline{2-3}  & EN & _Is job reform repealed?_ \\ \hline \hline \end{tabular}
\end{table} 
Table 16: Examples of diverging translations on specific phenomena, with English translations.