SlowGuess/ACL_2020_OCR huggingface dataset:
Elizabeth Salesky and Alan W Black
esalesky@jhu.edu, awb@cs.cmu.edu
End-to-end models for speech translation (ST) more tightly couple speech recognition (ASR) and machine translation (MT) than a traditional cascade of separate ASR and MT models, with simpler model architectures and the potential for reduced error propagation. Their performance is often assumed to be superior, though in many conditions this is not yet the case. We compare cascaded and end-to-end models across high, medium, and low-resource conditions, and show that cascades remain stronger baselines. Further, we introduce two methods to incorporate phone features into ST models. We show that these features improve both architectures, closing the gap between end-to-end models and cascades, and outperforming previous academic work – by up to 9 BLEU on our low-resource setting.
End-to-end models have become the common approach for speech translation (ST), but the performance gap between these models and a cascade of separately trained speech recognition (ASR) and machine translation (MT) remains, particularly in low-resource conditions. Models for low-resource ASR leverage phone<sup>1</sup> information, but this information is not typically leveraged by current sequence-to-sequence ASR or speech translation models. We propose two methods to incorporate phone features into current neural speech translation models. We explore the existing performance gap between end-to-end and cascaded models, and show that incorporating phone features not only closes this gap, but greatly improves the performance and training efficiency of both model architectures, particularly in lower-resource conditions.
The sequences of speech features used as input for ST are  $\approx 10$  times longer than the equivalent sequence of characters in e.g. a text-based MT model. This impacts memory usage, the number of model parameters, and
training time. Multiple consecutive feature vectors can belong to the same phone, but the exact number depends on the phone and local context. Further, these speech features are continuously valued rather than discrete, such that a given phone will have many different instantiations across a corpus. Neural models learn to associate ranges of similarly valued feature vectors in a data-driven way, impacting performance in lower-resource conditions. Using phoneme-level information provides explicit links about local and global similarities between speech features, allowing models to learn the task at hand more efficiently and yielding greater robustness to lower-resource conditions.
We propose two simple heuristics to integrate phoneme-level information into neural speech translation models: (1) as a more robust intermediate representation in a cascade; and (2) as a concatenated embedding factor. We use the common Fisher Spanish-English dataset to compare with previous work, and simulate high-, mid-, and low-resource conditions to compare model performance across different data conditions. We compare to recent work using phone segmentation for end-to-end speech translation (Salesky et al., 2019), and show that our methods outperform this model by up to 20 BLEU on our lowest-resource condition. Further, our models outperform all previous academic work on this dataset, achieving similar performance trained on 20 hours as a baseline end-to-end model trained on the full 160 hour dataset. Finally, we test model robustness by varying the quality of our phone features, which may indicate which models will better generalize across differently-resourced conditions.
2 Models with Phone Supervision
We add higher-level phone features to low-level speech features to improve our models' robustness across data conditions and training efficiency. We propose two methods to incorporate phone information into cascaded and end-to-end models, depicted in Figure 1. Our phone cascade uses phone labels as the machine translation input, in place of the output transcription from a speech recognition model. Our phone end-to-end model uses
24-reference BLEU scores are used for this dataset.
<sup>3</sup>Our code is public: github.com/esalesky/xnmt-devel
<sup>1</sup>The term 'phone' refers to segments corresponding to a collection of fine-grained phonetic units, but which may separate allophonic variation: see Jurafsky and Martin (2000).
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 2388-2397 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
Figure 1: Comparison between traditional cascaded and end-to-end models, and our proposed methods using phone features as (1) the intermediate representation in a cascaded model; and (2) a concatenated embedding factor in an end-to-end model. We additionally compare to previous work; (3) where phone segmentation is used for feature vector downsampling in time (Salesky et al., 2019).
phone labels to augment source speech feature vectors in end-to-end models. We call these end-to-end or 'direct' because they utilize a single model with access to the source speech features, though they additionally use phone features generated by an external model. We additionally compare to a recent end-to-end model proposed by Salesky et al. (2019).
Model 1: Phone Cascade. In a cascade, the intermediate representation between ASR and MT is the final output of a speech recognition model, e.g. characters, subwords, or words. Using separate models for ASR and MT means that errors made in ASR are likely to propagate through MT. Common errors include substitution of phonetically similar words, or misspellings due to irregularities in a language's orthography, the latter of which may be addressed by using phone labels in place of ASR output. By not committing to orthographic targets, we believe this model will propagate fewer errors to downstream MT.
Model 2: Phone End-to-End. Our final model uses phone-factored embeddings, where trainable embeddings for phone features are concatenated to typical speech feature vector input. Because phone durations are variable and typically span more than one filterbank feature (or frame), adjacent filterbank features may have the predicted phone label; in the example shown in Figure 1, /R/ spans three frames or filterbank features. We note that this method maintains the same source sequence length as the original speech feature sequence. This method associates similar feature vectors at the corpus level, because all filterbank features with the same phone alignment (e.g. /OH/) will have the same trainable phone embedding concatenated. In MT and NER, concatenating trainable embeddings for linguistic features to words, such as morphemes and phones, has improved models' ability to generalize (Sennrich and Haddow, 2016; Chaudhary et al., 2018). While these
works appended finer-grained information to associate words with similar lower-level structure, we use phone embeddings to associate higher-level structure to similar but unique speech feature vectors globally across a corpus.
Model 3: Phone Segmentation. We compare to the method from Salesky et al. (2019) as a strong end-to-end baseline. Here, phone boundaries are used to segment and compress speech feature vector sequences. Within each utterance, the feature vectors of consecutive speech frames with the same phone label are averaged to produce one feature vector for translation from a variable number of frames. This significantly reduces source sequence lengths (by  $\sim 80\%$ ), reducing the number of model parameters and memory. Rather than having a variable number of feature vectors per phone-like unit, each has one representation, more similar in granularity to character-based MT. The averaged feature vectors remain continuously-valued, and are locally summarized: a given phone across the corpus will still have different representations in each instance.
We use the Fisher Spanish-English corpus, which consists of parallel speech, transcripts, and translations, enabling comparisons between cascaded and direct models on the same data and allowing us to generate phone supervision using matched data. The dataset contains 160 hours of Spanish telephone speech, split into 138K utterances, which were translated via crowdsourcing by Post et al. (2013). We use the standard dev and test sets, each with  $\sim 4\mathrm{k}$  utterances. Because we are particularly interested in how our methods will affect training across differently-resourced conditions, we compare results using randomly selected 40 hour and 20 hour subsets of the data.
<sup>4</sup> joshua. incubator.apache.org/data/fisher-callhome-corpus
4 Generating Phone Supervision
To generate phoneme-level labels for sequences of speech features, we generate frame-level alignments using a trained speech recognizer. Specifically, we extract 40-dimensional Mel filterbank features with per-speaker mean and variance normalization using Kaldi (Povey et al., 2011). We train an HMM/GMM system on the full Fisher Spanish dataset with the Kaldi recipe (Povey et al., 2011), using the Spanish CALLHOME Lexicon (LDC96L16), and compute per-frame phone alignments with the triphone model (tri3a) with LDA+MLLT features. This yields 50 phone labels, including silence (<sil>), noise, and laughter.
Producing phone alignments uses supervision from a transcript, which inherently does not exist at inference time. While phones can be extracted from Kaldi lattices at inference time, we found that our HMM/GMM model was not our best performing ASR model on this dataset – by greater than 10 WER. To leverage our better-performing neural ASR models for phone generation, we create essentially a ‘2-pass’ alignment procedure: first, generating a transcript, and second, using this transcript to force align phones. Table 1 shows the mapping between phone quality and the ASR models used for phone feature generation. This procedure enables us to both improve phone
Alignment Quality	WER	ASR Supervision
Gold	-	Gold transcript
High	23.2	Salesky et al. (2019)
Med	30.4	Seq2Seq ASR
Low	35.5	Kaldi HMM/GMM
Table 1: Mapping between phone quality and the ASR models used for alignment generation, with the models' WER on Fisher Spanish test.
alignment quality and also match training and inference procedures for phone generation for our translation models. In Section 8, we compare the impact of phone alignment quality on our translation models utilizing phone features, and show higher quality phone features can improve downstream results by  $>10$  BLEU.
Producing phone features in this way uses the same data (source speech and transcripts) as the ASR task in a cascade, and auxiliary ASR tasks from multi-task end-to-end models, but as we show, to far greater effect. Further, auxiliary tasks as used in previous work rely on three-way parallel data, while it is possible to generate effective phoneme-level supervision using a recognizer trained on other corpora or languages (Salesky et al., 2019), though we do not do this here.
5 Model & Training Procedure
As in previous academic work on this corpus (Bansal et al., 2018; Sperber et al., 2019; Salesky et al., 2019), we use a sequence-to-sequence architecture inspired
by Weiss et al. (2017) modified to train within lower resources; specifically, each model converges within  $\approx 5$  days on one GPU. We build encoder-decoder models with attention in xnmt (Neubig et al., 2018) with 512 hidden units. Our pyramidal encoder uses 3-layer BiLSTMs with linear network-in-network (NiN) projections and batch normalization between layers (Sperber et al., 2019; Zhang et al., 2017). The NiN projections are used to downsample by a factor of 2 between layers, resulting in the same total  $4\times$  downsampling in time as the additional convolutional layers from Weiss et al. (2017); Bansal et al. (2019): They give us the benefit of added depth with fewer additional parameters. We use single layer MLP attention (Bahdanau et al., 2015) with 128 units and 1 decoder layer as opposed to 3 or 4 in previous work - we did not see consistent benefits from additional depth.
In line with previous work on this dataset, all experiments preprocess target text by lowercasing and removing punctuation aside from apostrophes. We use 40-dimensional Mel filterbank features as previous work did not see significant difference with higher-dimensional features (Salesky et al., 2019). We use 1k BPE units for translation text, shown in Salesky et al. (2019) to have both better performance and training efficiency than characters (Weiss et al., 2017; Sperber et al., 2019) or words (Bansal et al., 2018). For both text and phones, we use 64-dimensional embeddings.
For the MT component in cascaded speech translation models, we compared using the pyramidal speech architecture above (3 encoder, 1 decoder layers) to the traditional BiLSTM text model (2 layers each for encoder and decoder). Using the pyramidal architecture resulted in the same performance as the BiLSTM model when translating BPE transcriptions from ASR, but gave us consistent improvements of up to 1.5 BLEU when instead translating phone sequences; we posit this is because phone sequences are longer than BPE equivalents. Accordingly, we use the same model architecture for all our ASR, MT, and ST models.
We use layer dropout with  $p = 0.2$  and target embedding dropout with  $p = 0.1$  (Gal and Ghahramani, 2016). We apply label smoothing with  $p = 0.1$  (Szegedy et al., 2016) and fix the target embedding norm to 1 (Nguyen and Chiang, 2018). For inference, we use beam of size 15 and length normalization with exponent 1.5. We set the batch size dynamically depending on the input sequence length with average batch size was 36. We use Adam (Kingma and Ba, 2015) with initial learning rate 0.0003, decayed by 0.5 when validation BLEU did not improve for 10 epochs initially and subsequently 5 epochs. We do not use L2 weight decay or Gaussian noise, and use a single model replica. We use input feeding (Luong et al., 2015), and exclude utterances longer than 1500 frames in training for memory.
6 Prior Work: Cascaded vs End-to-End Models on Fisher Spanish-English
The large body of research on the Fisher Spanish-English dataset, including both cascaded and end-to-end models, makes it a good benchmark to compare these architectures. Not all previous work has compared across multiple resource settings or compared to cascaded models, which we address in this section. We summarize best previous results on this dataset on high, medium, and low-resource conditions in Table 2.
Best Results. The cascade of traditional HMM/DNN ASR and Joshua MT models from Kumar et al. (2014) set a competitive baseline on the full dataset (40.4 test BLEU) which no subsequent academic models have been able to match until this work; subsequent exploration of end-to-end models has produced notable relative improvements but the best end-to-end academic number (Salesky et al., 2019) remains 1.6 BLEU behind this traditional cascade.
Industry models from Weiss et al. (2017) achieved exceptional performance with very deep end-to-end models on the full dataset (47.3 test BLEU), exceeding a cascade for the first time. They additionally show results with an updated cascade using neural models, improving over Kumar et al. (2014). Their results have been previously unmet by the rest of the community. This is likely in part due to the computational resources required to fully explore training schedules and hyperparameters with models of their depth. While their ASR models took  $\sim 4$  days to converge, their ST models took another 2 weeks, compared to the lighter-weight models of recent academic work which converged in  $< 5$  days (Sperber et al., 2019; Salesky et al., 2019; Bansal et al., 2019).
This dataset is challenging: improving ASR WER from 35 (Post et al.) to 23 (Kumar et al.) only resulted in 4 BLEU ST improvement: see Components in Table 2. We believe this to be in part because the multi-reference scoring masks some model differences, and the conver
sational phenomena (like disfluencies) are challenging.
Lower-Resource. While deep end-to-end models have become competitive at higher-resource conditions, previous work on this dataset has showed they are not as data-efficient as cascades under lower-resource conditions. While some works have tested multiple resource conditions, only Sperber et al. (2019) compared against cascades across multiple conditions. Their end-to-end baseline outperformed their cascades on the full dataset, but not under lower-resource conditions, while their end-to-end but multi-stage attention-passing model is more data-efficient than previous models and shows the best previous results under lower-resource condition. Sperber et al. do not report results without auxiliary ASR, MT, and autoencoding tasks, which they state add up to 2 BLEU.
Additional Data. Stoian et al. (2020); Bansal et al. (2019); Sperber et al. (2019) investigate speech translation performance using additional corpora through transfer learning from ASR and auxiliary MT tasks. The ability to leverage non-parallel corpora was previously a strength of cascades and had not been explored with end-to-end models. We do not use additional data here, but show these numbers as context for our results with phone supervision, and refer readers to Sperber et al. for discussion of cascaded and end-to-end models' capacity to make use of more data.
Parameter Tuning. We find cascaded model performance can be impacted significantly by model settings such as beam size and choice of ASR target preprocessing. While Weiss et al. (2017); Sperber et al. (2019) use character targets for ASR, we use BPE, which gave us an average increase of 2 BLEU. Further, we note that search space in decoding has significant impact on cascaded model performance. In cascaded models, errors produced by ASR can be unrecoverable, as the MT component has access only to ASR output. While Sperber et al. (2019) use a beam of size 1 for the ASR component of their cascade to compare with their two-stage end-to
Model	Source	HIGH (160hr)	MID (40hr)	LOW (20hr)	Components
dev	test	dev	test	dev	test	ASR↓	MT↑
Cascaded	Weiss et al. (2017)	45.1	45.5	-	-	-	-	23.2	57.9
Kumar et al. (2014)	-	\( 40.4^† \)	-	-	-	-	25.3	62.9
Sperber et al. (2019)	-	32.5	-	16.8	-	6.6	40.9	58.1
End-to-End	Weiss et al. (2017)	46.5	\( 47.3^* \)	-	-	-	-
Salesky et al. (2019)	37.6	38.8	21.0	19.8	11.1	10.0
Sperber et al. (2019)	-	36.7	-	31.9	-	22.8
Stoian et al. (2020)	34.1	34.6	-	-	10.3	10.2
+ Add'l Data	Sperber et al. (2019)	-	38.8	-	-	-	-
Stoian et al. (2020)	37.9	37.8	-	-	20.1	20.2
Table 2: End-to-end vs cascaded speech translation model performance in BLEU↑ on Fisher Spanish-English data from the literature. (†) denotes the best previous academic result on the full dataset, (*) the best from industry. Component models for cascades reported on test on full dataset: ASR reported inWER↓ and MT in BLEU↑.
end models, we find that using equal beam sizes of 15 for both ASR and MT improves cascaded performance with the same model by 4-8 BLEU; combining these two parameter changes makes the same cascaded model a much more competitive baseline (compare lines 3 in both Table 2 and Table 3). In contrast, widening beam size to yield an equivalent search space for end-to-end models has diminishing returns after a certain point; we did not see further benefits with a larger beam ( $>15$ ).
Our Baselines. We report best numbers from previous work in Table 2 for comparison (which may use multi-task training), but use single-task models in our work. We report our baseline results in Table 3. On the full dataset, our baseline cascade improves slightly over Kumar et al. (2014) with 41.0 compared to 40.4 on test, a mark most recent work has not matched primarily due to model choices noted above, with component ASR performance of WER 30.4 and 58.6 BLEU for MT. Our end-to-end baseline is comparable to the baselines in Salesky et al. (2019); Sperber et al. (2019); Stoian et al. (2020). This suggests we have competitive baselines for both end-to-end and cascaded models.
7 Results Using Phone Features
We compare our two ways to leverage phone features to our cascaded and end-to-end baselines across three resource conditions. Table 3 shows our results; following previous work, all BLEU scores are multi-reference. Average single reference scores may be found in Appendix A. All models using phone supervision outperform the end-to-end baseline on all three resource conditions, while our proposed models also exceed the cascaded baseline and previous work at lower-resource conditions.
Phone features. Salesky et al. (2019) performs most similarly to the end-to-end baseline, but nonetheless represents an average relative improvement of  $13\%$  across the three data sizes with a significant reduction in training time. Our phone featured models use not just the phone segmentation, but also the phone labels, and perform significantly better. Our phone end-to-end model not only shows less of a decrease in performance across
Figure 2: Performance of all models relative to 'Baseline Cascade' ( $\Delta = 0$ ) across our 3 resource conditions. Cascaded models in orange, end-to-end models in purple. Our proposed models yield improvements across all three conditions, with a widening margin under low-resource conditions for the phone cascade.
resource conditions than Salesky et al. (2019), but further improves by 4 BLEU over the baseline cascade on our two lower-resource conditions. This suggests augmenting embeddings with discrete phone features is more effective than improved downsampling. The phone cascade performs still better, with marked improvements across all conditions over all other models (see Figure 2). On the full dataset, using phones as the source for MT in a cascade performs  $\sim 2$  BLEU better than using BPE, while at 40 and 20 hours this increases to up to 10 BLEU. We analyze the robustness of phone models further in Section 8.
Hybrid cascade. We additionally use a 'hybrid cascade' model to compare using phone features to improving ASR. Our hybrid cascade uses an ASR model with phone-informed downsampling and BPE targets (Salesky et al., 2019). This improves the WER of our ASR model to 28.1 on dev and 23.2 on test, matching Weiss et al. (2017)'s state-of-the-art on test (23.2) and approaching it on dev (25.7). Our hybrid cascade performs more similarly to Weiss et al.'s cascade on the full dataset, with 45.0 to their 45.5 on test, and is our best-performing ST model on the full dataset. However, at lower-resource conditions, it does not perform as favor-
Model	HIGH (160hr)	MID (40hr)	LOW (20hr)
dev	test	Δ	dev	test	Δ	dev	test	Δ
Baseline	Baseline End-to-End	32.4	33.7	-	19.5	17.4	-	9.8	9.8	-
Salesky et al. (2019)	37.6	38.8	+5.2	21.0	19.8	+2.0	11.1	10.0	+0.8
Baseline Cascade	39.7	41.0	+7.3	29.8	27.1	+10.0	22.6	20.2	+11.6
Proposed	Phone End-to-End	40.5	42.1	+8.3	34.5	33.0	+15.3	26.7	26.2	+16.7
Phone Cascade	41.6	43.3	+9.4	37.2	37.4	+18.9	32.2	31.5	+22.1
Hybrid Cascade	42.9	45.0	+10.9	33.3	31.2	+13.8	23.2	21.5	+12.6
Table 3: Results in BLEU↑ comparing our proposed phone featured models to baselines. We compare three resource conditions, and show average improvement for dev and test (Δ). Best performance bolded by column.
ably compared to phone featured models – as shown in Figure 2, both the phone cascade and phone end-to-end models outperform the hybrid cascade at lower-resource conditions, by up to 10 BLEU at 20 hours. This suggests improving ASR may enable cascades to perform better at high-resource conditions, but under lower-resource conditions it is not as effective as utilizing phone features.
Training time. In addition to performance improvements, our models with phone features are typically more efficient with respect to training time, shown in Table 4. The fixed time to produce phone labels, which must be performed before translation, becomes a greater proportion of overall training time at lower-resource settings. In particular, the phone end-to-end model offers similar training time reduction over the baseline to Salesky et al. (2019), where downsampling reduces sequence lengths by up to  $60\%$ , with unreduced sequence lengths through earlier convergence; this model offers a better trade-off between time and performance.