SlowGuess/ACL_2020_OCR huggingface dataset:
Model	HIGH	MID	LOW	Δ
Baseline End-to-End	118hr	40hr	22hr	-
Salesky et al. (2019)	41hr	13hr	10hr	0.4×
Baseline Cascade	76hr	19hr	12hr	0.6×
Phone Cascade	57hr	39hr	27hr	0.7×
Phone End-to-End	42hr	20hr	13hr	0.4×
Hybrid Cascade	47hr	34hr	24hr	0.6×
Table 4: Total training time  $\boxed{\vee}$  for all models (including time to generate phone features) on 3 resource conditions. The ASR and MT models in the baseline cascade can be trained in parallel, reflected here, while phone featured models may not as the MT requires phone features from ASR.
Comparing to previous work using additional data. Previous work used the parallel speech transcripts in this dataset for auxiliary tasks with gains of up to 2 BLEU; we show using the same data to generate phone supervision is far more effective. We note that our phone models further outperform previous work trained with additional corpora. The attention-passing model of Sperber et al. (2019) trained on additional parallel Spanish-English text yields 38.8 on test on the full dataset, which Salesky et al. (2019) matches on the full dataset and our proposed models exceed, with the phone cascade yielding a similar result (37.4) trained on only 40 hours. Pre-training with 300 hours of English ASR data and fine-tuning on 20 hours of Spanish-English data, Stoian et al. (2020); Bansal et al. (2019) improve their end-to-end models from  $\approx 10$  BLEU to 20.2. All three of our proposed models exceed this mark trained on 20 hours of Fisher.
8 Model Robustness & Further Analysis
In this section, we analyze the robustness of each of our models by varying the quality of our phone features, and further explore the strengths and limitations of each model.
8.1 Phone Cascade
Phone cascades use a representation for translation which may be more robust to non-phonetic aspects of orthography. However, as a cascaded model, this still requires hard decisions between ASR and MT, and so we may expect lower phone quality to lead to unrecoverable errors. Figure 3 compares the impact of phone quality on the performance of phone cascades trained on our high, medium, and low-resource conditions. We use alignments produced with gold transcripts as an upper bound on performance. We note that with gold alignments, translation performance is similar to text-based translation (see Section 6). We see that phone quality does have a significant impact on performance, with the MT model trained on low phone quality yielding similar translation performance using the full 160 hour dataset to the MT model with the highest quality phones trained on only 20 hours. However, we also see significantly more data-efficiency with this model, with less reduction in performance between  $160hr \rightarrow 40hr \rightarrow 20hr$  training conditions than previous models.
Figure 3: Phone Cascade Robustness: using phone labels in place of BPE as the text source for downstream MT. Comparing performance across our three data conditions and phone label qualities.
Redundancy. For the phone cascade models compared in Figure 3, we collapse adjacent consecutive phones with the same label, i.e. when three consecutive frames have been aligned to the same phone label  $B$ $B$  we have reduced the sequence to a single phone  $B'$  for translation. We additionally compared translating non-uniqued phone sequences (e.g. the same sequence length as the number of frames) as a more controlled proxy for our model's handling of longer frame-based feature vector sequences compared to Salesky et al. (2019)'s downsampled feature vector sequences. The redundant phones caused consistent decreases in BLEU,
2393
with much greater impact in lower-resource conditions. Translating the full sequence of redundant frame-level phone labels, for the full 160hr dataset, all models performed on average 0.6 BLEU worse; for 40hr, 1.8 BLEU worse; and with 20 hours, 4.1 BLEU worse - a  $13\%$  decrease in performance solely from non-uniqued sequences.
Phones correspond to a variable-length number of speech frames depending on context, speaker, and other semantic information. When translating speech feature vectors, speech features within a phone are similar but uniquely valued; using instead phone labels in a phone cascade, the labels are identical though still redundant. These results suggest our LSTM-based models are better able to handle redundancy and variable phone length at higher resource conditions with sufficient examples, but are less able to handle redundancy with less training data.
8.2 Phone End-to-End
Our phone end-to-end model concatenates trainable embeddings for phone labels to frame-level filterbank features, associating similar feature vectors globally across the corpus, as opposed to locally within an utterance as with the phone-averaged embeddings (Section 8.3). Figure 4 compares the results of these factored models using phone features of differing qualities, with 'gold' alignments as an upper bound. The phone end-to-end models compared do not reach the same upper performance as the phone cascades: comparing gold phone labels, the phone end-to-end model performs slightly worse at 160hr with more degradation in performance at 40hr and 20hr. While this comparison is even more pronounced for 'low' phone quality than 'gold,' the phone end-to-end model has more similar performance between 'gold' and 'high' phone quality than the cascade.
This model's input contains both the phone features used in the phone cascade and speech features of the baseline end-to-end model, but unlike the phone cas
Figure 4: Phone End-to-End Robustness: trainable embeddings for phone labels are concatenated to frame-level filterbank features. Comparing performance across three data conditions and phone label qualities.
cade or Salesky et al. (2019) the input sequence has not been reduced in length. That the end-to-end phone model achieves top performance and converges much faster than end-to-end baseline is unsurprising, as access to both speech feature vectors and phone labels mitigates the effects of long noisy input sequences. The significant performance improvements over Salesky et al. (2019), however, are more interesting, as these models make use of the similar information in different ways – the use of discrete embeddings seems to aid the phone end-to-end model, though the sequence length is not reduced. The model's performance degradation compared to the phone cascade in lower-resource conditions is likely due in part to these sequence lengths, as shown by our additional experiments with input redundancy for the cascade. The greater reduction in performance here using lower quality phones suggests the noise of the labels and concatenated filterbank features compound, further detracting from performance. Perhaps further investigation into the relative weights placed on the two embedding factors over the training process could close this additional gap.
8.3 Phone Segmentation: Salesky et al. (2019)
We also compare to the models from Salesky et al. (2019) as a strong end-to-end baseline. That work introduced downsampling informed by phone segmentation – unlike our other models, the value of the phone label is not used, but rather, phone alignments are used only to determine the boundary between adjacent phones for variable-length downsampling. Their model provides considerable training and decoding time improvements due to the reduced source sequence length, and shows consistent improvements over the baseline end-to-end model using the original filterbank feature sequences which increase with the amount of training data. However, their model has lower overall performance and with much smaller performance improvements over our baselines in lower-resource conditions than the phone featured models we propose here. We hypothesize that the primary reason for their BLEU improvements is the reduction in local redundancy between similar frames, as discovered in the previous section. We refer readers to their paper for further analysis.
8.4 Quality of Phone Labels
We show two examples of phone sequences produced with each overall model quality in Figure 5, unique within consecutive frame sequences with the same label for space constraints. Individual phones are typically 5-20 frames. We see the primary difference in produced phones between different models is the label values, rather than the boundaries. While we do see some cases where the boundaries shift, they chiefly vary by only 1-3 frames. It is not the case that there are significantly more or fewer phone segments aligned per utterance by quality, though there are outlying utterances (Example 2 - 'Low').
Figure 5: Two examples of phone sequences demonstrating differences across qualities of phone features.
Relating our observed trends to the differences between our phone cascades and phone end-to-end models, we note that differences in frame-level phone boundaries would not affect our phone cascaded models, where the speech features are discarded, while they would affect our phone end-to-end models, where the phone labels are concatenated to speech feature vectors and associate them across the corpus. While errors in phone labels may be seen as 'unrecoverable' in a cascade, for the end-to-end model, they add noise to distribution of filterbank feature associated with each phone label embedding, which appears to have a more negative impact on performance than the hard decisions in cascades. Though the concatenated filterbank features may allow our end-to-end models to recover from discrete label errors, our results testing various phone qualities suggest this may only be the case under higher-resource settings with sufficient examples.
Speech translation was initially performed by cascading separately trained ASR and MT models, allowing each model to be trained on larger data sources without parallel speech, transcriptions, and translations, but potentially yielding unrecoverable errors between models. Linking models through lattices with both phrase-based (Kumar et al., 2014) and neural MT (Sperber et al., 2017) reduced many such errors. Using one model to directly translate speech was later enabled by attentional encoder-decoder models.
Direct end-to-end speech translation was first explored as a way to reduce both error propagation, and also the need for high quality intermediate transcriptions (e.g. for unwritten languages). The first such models were investigated in Bérard et al. (2016); Duong et al. (2016), but these used, respectively, a small synthetic corpus and evaluated on speech-to-text alignments rather than translation. Subsequently Weiss et al. (2017)
extended these neural attentional models to deep, multitask models with excellent results on Fisher Spanish-English, exceeding a cascade for the first time. However, efforts from the community have not yet replicated their success (Stoian et al., 2020; Sperber et al., 2019; Salesky et al., 2019). End-to-end models have performed inconsistently compared to cascades on other corpora: Bérard et al. (2018) perform well on high-resource audiobooks but do not exceed a cascade; Anastasopoulos and Chiang (2018) found 'triangle' models performed better than cascades for 2 of 3 very low-resource language pairs; and in the most recent IWSLT evaluation campaigns, cascades have remained the highest-performing systems (Niehues et al., 2018, 2019).
Similarly-motivated work exists in speech translation. In addition to Salesky et al. (2019); Sperber et al. (2019) addressed above, preliminary cascades using phone-like units have been explored for low-resource speech translation, motivated by translation of unwritten languages where a traditional cascade would not be possible. To this end, Bansal et al. (2018) utilized unsupervised term discovery, and Wilkinson et al. (2016) synthesized speech; but these approaches were only evaluated in terms of precision and recall and were not tested on both 'higher-resource' and natural speech data conditions.
We show that phone features significantly improve the performance and data efficiency of neural speech translation models. We study the existing performance gap between cascaded and end-to-end models, and introduce two methods to use phoneme-level features in both architectures. Our improvements hold across high, medium, and low-resource conditions. Our greatest improvements are seen in our lowest-resource settings (20 hours), where our end-to-end model outperforms a strong baseline cascade by  $\approx 5$  BLEU, and our cascade outperforms prior work by  $\approx 9$ BLEU. Generating phone features uses the same data as auxiliary speech recognition tasks from prior work; our experiments suggest these features are a more effective use of this data, with our models matching the performance from previous works' performance without additional training data. We hope that these model comparisons and results inform development of more robust end-to-end models, and provide a stronger benchmark for performance on low-resource settings.
The authors thank Andrew Runge, Carlos Aguirre, Carol Edwards, Eleanor Chodroff, Florian's cluster, Huda Khayrallah, Matthew Wiesner, Nikolai Vogler, Rachel Wicks, Ryan Cotterell, and the anonymous reviewers for helpful feedback and resources.
2395
Antonios Anastasopoulos and David Chiang. 2018. Tied multitask learning for neural speech translation. Proc. of NAACL. arXiv:1802.06655.
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural machine translation by jointly learning to align and translate. Proc. of ICLR. arXiv:1409.0473.
Sameer Bansal, Herman Kamper, Karen Livescu, Adam Lopez, and Sharon Goldwater. 2018. Low-resource speech-to-text translation. Proc. of Interspeech. arXiv:1803.09164.
Sameer Bansal, Herman Kamper, Karen Livescu, Adam Lopez, and Sharon Goldwater. 2019. Pre-training on high-resource speech recognition improves low-resource speech-to-text translation. Proc. of NAACL. arXiv:1809.01431.
Alexandre Bérard, Laurent Besacier, Ali Can Kocabiyikoglu, and Olivier Pietquin. 2018. End-to-end automatic speech translation of audiobooks. In Proc. of ICASSP.
Alexandre Bérard, Olivier Pietquin, Christophe Servan, and Laurent Besacier. 2016. Listen and translate: A proof of concept for end-to-end speech-to-text translation. NIPS Workshop on End-to-end Learning for Speech and Audio Processing. arXiv:1612.01744.
Aditi Chaudhary, Chunting Zhou, Lori Levin, Graham Neubig, David R Mortensen, and Jaime G Carbonell. 2018. Adapting word embeddings to new languages with morphological and phonological subword representations. Proc. of EMNLP. arXiv:1808.09500.
Long Duong, Antonios Anastasopoulos, David Chiang, Steven Bird, and Trevor Cohn. 2016. An attentional model for speech translation without transcription. In Proc. of NAACL.
Yarin Gal and Zoubin Ghahramani. 2016. A theoretically grounded application of dropout in recurrent neural networks. Proc. of NeurIPS.
D Jurafsky and J Martin. 2000. Speech and Language Processing, 3rd edition. Prentice Hall.
Diederik P Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. Proc. of ICLR. arXiv:1412.6980.
Gaurav Kumar, Matt Post, Daniel Povey, and Sanjeev Khudanpur. 2014. Some insights from translating conversational telephone speech. Proc. of ICASSP.
Minh-Thang Luong, Hieu Pham, and Christopher D. Manning. 2015. Effective approaches to attention-based neural machine translation. Proc. of EMNLP.
Graham Neubig, Matthias Sperber, Xinyi Wang, Matthieu Felix, Austin Matthews, Sarguna Padmanabhan, Ye Qi, Devendra Singh Sachan, Philip Arthur, Pierre Godard, et al. 2018. XNMT: The eXtensible neural machine translation toolkit. Proc. of AMTA. arXiv:1803.00188.
Toan Q Nguyen and David Chiang. 2018. Improving lexical choice in neural machine translation. Proc. of NAACL. arXiv:1710.01329.
Jan Niehues, Ronaldo Cattoni, Sebastian Stüker, Mauro Cettolo, Marco Turchi, and Marcello Federico. 2018. The iwslt 2018 evaluation campaign.
Jan Niehues, Ronaldo Cattoni, Sebastian Stüker, Matteo Negri, Marco Turchi, Thanh-Le Ha, Elizabeth Salesky, Ramon Sanabria, Loic Barrault, Lucia Specia, and Marcello Federico. 2019. The iwslt 2019 evaluation campaign.
Matt Post, Gaurav Kumar, Adam Lopez, Damianos Karakos, Chris Callison-Burch, and Sanjeev Khudanpur. 2013. Improved speech-to-text translation with the Fisher and Callhome Spanish-English speech translation corpus.
Daniel Povey, Arnab Ghoshal, Gilles Boulianne, Lukas Burget, Ondrej Glembek, Nagendra Goel, Mirko Hannemann, Petr Motlicek, Yanmin Qian, Petr Schwarz, et al. 2011. The Kaldi speech recognition toolkit. Proc. of ASRU.
Elizabeth Salesky, Matthias Sperber, and Alan Black. 2019. Exploring phoneme-level speech representations for end-to-end speech translation. Proc. of ACL.
Rico Sennrich and Barry Haddow. 2016. Linguistic input features improve neural machine translation. Proc. of WMT. arXiv:1606.02892.
Matthias Sperber, Graham Neubig, Jan Niehues, and Alex Waibel. 2017. Neural lattice-to-sequence models for uncertain inputs. Proc. of EMNLP. arXiv:1704.00559.
Matthias Sperber, Graham Neubig, Jan Niehues, and Alex Waibel. 2019. Attention-passing models for robust and data-efficient end-to-end speech translation. Proc. of TACL. arXiv:1904.07209.
Mihaela C Stoian, Sameer Bansal, and Sharon Goldwater. 2020. Analyzing ASR pretraining for low-resource speech-to-text translation. Proc. of ICASSP.
Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and Zbigniew Wojna. 2016. Rethinking the inception architecture for computer vision. Proc. of CVPR.
Ron J Weiss, Jan Chorowski, Navdeep Jaitly, Yonghui Wu, and Zhifeng Chen. 2017. Sequence-to-sequence models can directly transcribe foreign speech. Proc. of INTERSPEECH. arXiv:1703.08581.
Andrew Wilkinson, Tiancheng Zhao, and Alan W Black. 2016. Deriving phonetic transcriptions and discovering word segmentations for speech-to-speech translation in low-resource settings. Proc. of INTER-SPEECH.
Yu Zhang, William Chan, and Navdeep Jaitly. 2017. Very deep convolutional networks for end-to-end speech recognition. Proc. of ICASSP.
A Single-Reference BLEU Scores
These tables contain the same results as our tables and figures as in the main paper, but show average single-reference BLEU scores in place of multi-reference (4-reference) BLEU. WER for ASR is unchanged: the dataset contains a single reference transcript for ASR. Results from prior work report only multi-reference BLEU and so are not included below.
Data	ASR↓	MT↑	Cascade	End-to-End
dev	test	dev	test	dev	test	dev	test
Full	33.3	30.4	34.5	33.6	23.2	23.7	19.0	19.6
40hr	44.8	46.7	29.9	28.3	17.4	15.7	11.5	10.4
20hr	56.3	59.1	22.4	22.6	13.2	11.8	5.9	5.3
Table 8: Baseline results for end-to-end and cascaded speech translation models, with component ASR and MT model performance for cascades (blue). ASR results in  $\mathrm{{WER}} \downarrow$  and translation results in  $\mathrm{{BLEU}} \uparrow$  .
Phone Quality	160hr	40hr	20hr
dev	test	dev	test	dev	test
Gold	33.3	33.2	29.3	28.5	24.4	23.0
High	24.1	25.1	21.6	21.7	18.9	18.3
Med	23.1	23.4	20.6	20.7	17.6	17.2
Low	18.2	19.1	16.4	17.0	14.1	14.2
Table 5: Phone Cascades. We use frame-level phone labels as the text source for downstream MT. Comparing method robustness to phone quality and resource conditions.
Phone Quality	160hr	40hr	20hr
dev	test	dev	test	dev	test
Gold	34.1	31.3	27.9	23.4	20.5	17.2
Med	24.0	23.7	20.8	18.4	16.5	14.6
Low	20.5	18.3	17.0	13.0	12.2	8.7
Table 6: Phone End-to-End. Trainable embeddings for phone labels are concatenated to frame-level filterbank features. Comparing method robustness to phone quality and resource conditions.
Model	Full (160hr)	40hr	20hr
dev	test	Δ	dev	test	Δ	dev	test	Δ
Baseline	Baseline End-to-End	19.0	19.6	-	11.5	10.4	-	5.9	5.3	-
Salesky et al. (2019)	22.0	21.9	+2.7	12.6	11.6	+1.2	6.7	6.2	+0.9
Baseline Cascade	23.2	23.7	+4.2	17.4	15.7	+5.6	13.2	11.8	+6.9
Proposed	Phone End-to-End	24.0	23.7	+4.6	20.8	18.4	+8.7	16.5	14.6	+10.0
Phone Cascade	24.1	25.1	+5.3	21.6	21.7	+10.7	18.9	18.3	+13.0
Hybrid Cascade	24.9	25.9	+6.1	19.6	18.2	+8.0	13.6	12.6	+7.5
Table 7: Results in BLEU↑ comparing our proposed phone featured models to baselines. We compare three resource conditions, and show average improvement for dev and test (Δ). Best performance bolded by column.