Venue: NAACL 2019

# MuST-C: a Multilingual Speech Translation Corpus

Mattia Antonino Di Gangi\({}^{1,2}\), Roldano Cattoni\({}^{1}\), Luisa Bentivogli\({}^{1}\), Matteo Negri\({}^{1}\) and Marco Turchi\({}^{1}\)

\({}^{1}\)Fondazione Bruno Kessler

\({}^{2}\)University of Trento

Trento, Italy {digangi,cattoni,bentivo,negri,turchi}@fbk.eu

###### Abstract

Current research on spoken language translation (SLT) has to confront with the scarcity of sizeable and publicly available training corpora. This problem hinders the adoption of neural end-to-end approaches, which represent the state of the art in the two parent tasks of SLT: automatic speech recognition and machine translation. To fill this gap, we created MuST-C, a multilingual speech translation corpus whose size and quality will facilitate the training of end-to-end systems for SLT from English into 8 languages. For each target language, MuST-C comprises at least \(385\) hours of audio recordings from English TED Takls, which are automatically aligned at the sentence level with their manual transcriptions and translations. Together with a description of the corpus creation methodology (scalable to add new data and cover new languages), we provide an empirical verification of its quality and SLT results computed with strong baseline system on each language direction.

## 1 Introduction

Besides the increased computing power, the recent surge of neural end-to-end approaches to natural language processing tasks has been stoked by the increased availability of data. For instance, when supported by sizeable training corpora, the robustness and the strong generalization capabilities of neural networks led to their dominance over previous paradigms both in automatic speech recognition (ASR Chiu et al., 2018) and machine translation (MT Bojar et al., 2018)).

Compared to its two parent research areas, spoken language translation (SLT) has not shown such a steady progress yet. Despite recent claims by big industry players about the effectiveness of end-to-end learning Weiss et al. (2017); Jia et al. (2018), its adoption does not yet represent the mainstream solution to the SLT task. One of the main obstacles to a stable dominance of the end-to-end paradigm also in this area is the scarcity of training corpora. While cascade ASR+MT solutions can exploit the wealth of task-specific data available for each of the two tasks,1 the situation for end-to-end model training is much less favourable. As shown in Table 1, few publicly available corpora exist, their language coverage is rather limited and, most importantly, their size is often too small (less than \(100\) hours of translated audio) for training data-hungry neural models.2

Footnote 1: In resource-rich conditions, ASR and MT training often builds on thousands of hours of transcribed speech and tens of millions of parallel sentences, respectively.

Footnote 2: Besides the corpora reported in Table 1, several smaller (\(<4\) hours) freely-available datasets have been created (_e.g._ the IWSLT evaluation campaign development and test sets from 2010 to 2017 and the Griko-Italian corpus by Boito et al. (2018)).

To circumvent the problem, neural SLT approaches currently rely on: _i)_ large proprietary corpora Jia et al. (2018), _ii)_ multitask learning

\begin{table}
\begin{tabular}{l|c|c} Corpus & Languages & Hours \\ \hline Niehues et al. (2018) & En\(\rightarrow\)De & 273 \\ Kocabyikoglu et al. (2018) & En\(\rightarrow\)Fr & 236 \\ Tohyama et al. (2005) & En\(\leftrightarrow\)Jp & 182 \\ Paulik and Waibel (2009) & En\(\rightarrow\)Es & 111 \\  & Es\(\rightarrow\)En & 105 \\ Post et al. (2013) & En\(\rightarrow\)ES & 38 \\ Stuker et al. (2012) & De\(\rightarrow\)En & 37 \\ Shimizu et al. (2014) & En\(\leftrightarrow\)JP & 22 \\ Federmann and Lewis (2017) & En\(\leftrightarrow\)JpZh & 22 \\ Bendazzoli and Sandrellis (2005) & E\(\leftrightarrow\)Ufres & 18 \\  & It\(\leftrightarrow\)Es & 17 \\ Berard et al. (2016) & Fr\(\rightarrow\)En & 17 \\ Federmann and Lewis (2016) & En\(\leftrightarrow\)Fr/De & 8 \\ Woldeyohannis et al. (2017) & Am\(\rightarrow\)En & 7 \\ Godard et al. (2017) & Mboshi\(\rightarrow\)Fr & 4 \\ \end{tabular}
\end{table}
Table 1: Publicly available SLT corpora. The two most recent resources (also known as _IWSLT18_ and _Augmented LibriSpeech_) are also the largest ones. Though considerably smaller, the _Fisher and Callhome_ corpus described in Post et al. (2013) is among the most widely used ones in previous research.



(Weiss et al., 2017; Anastasopoulos and Chiang, 2018; Berard et al., 2018), _iii)_ encoder/decoder pre-training (Bansal et al., 2018; Berard et al., 2018), _iv)_ synthesized speech data (Berard et al., 2016), or _v)_ machine-translated target text data (Berard et al., 2018). Though effective, solutions _ii)_ and _iii)_ assume the availability of ASR and MT data, which is not always guaranteed (especially in low-resource language settings). Solutions _iv)_ and _v)_, instead, rely on training material derived from sub-optimal automatic data creation/augmentation procedures. This situation calls for initiatives towards the creation of large, high-quality multilingual corpora suitable to explore end-to-end SLT in more favorable conditions similar to condition _i)_. Along this direction, our contributions are:

* A large (\(\sim\)400 hours of speech per language) multilingual corpus for SLT from English into \(8\) languages (German, Spanish, French, Italian, Dutch, Portuguese, Romanian and Russian);
* An empirical verification of its quality;
* ASR, MT and SLT results computed with strong baseline systems on each language direction.

MuST-C is released under a Creative Commons license, Attribution - Non Commercial - No Derivatives (CC BY NC ND 4.0 International), and is freely downloadable at mustc.fbk.eu

## 2 Corpus Creation Methodology

Must-C was created pursuing high quality as well as large size, speaker variety (male/female, native/non-native) and coverage in terms of topics and languages. To achieve these objectives, similar to (Niehues et al., 2018), we started from English TED Talks, in which a variety of speakers discuss topics spanning from business to science and entertainment. Most importantly, the fact that TED talks are often manually transcribed and translated sets ideal conditions for creating an SLT corpus from high-quality text material. Although the initial data are similar to those used to build the IWSLT18 corpus, our methodology is different. Inspired by Kocabiyikoglu et al. (2018), it exploits automatic alignment procedures, first at the text level (between transcriptions and translations) and then with the corresponding audio segments.

More in detail, for each target language \(L_{i}\), the (English-\(L_{i}\)) section of MuST-C is created as follows. First, for all the English talks available from the TED website,3 we download the videos and the HTML files containing the manual transcriptions and their translation into \(L_{i}\).4

Footnote 3: www.ted.com – dump of April 2018.

Footnote 4: All talks have manual captions, which were also translated into many languages by volunteers. The language coverage of the translations depends on several factors like the age of the talk (the old ones often have more translations), the popularity of its topic and the availability of volunteer translators for a given language.

Then, the plain text transcription and the translation of each talk are split at the sentence level based on strong punctuation marks and aligned using the Gargantua sentence alignment tool (Braune and Fraser, 2010). This step produces a bilingual text corpus aligned at the sentence level.

In the third step, the English side of this bilingual corpus is aligned to the corresponding audio track extracted from the video. This is done using Gentle,5 an off-the-shelf English forced-aligner built on the Kaldi ASR toolkit (Povey et al., 2011).

Footnote 5: github.com/lowerquality/gentle

Next, the audio-text alignments are processed to create a YAML file containing time information (_i.e._ start and duration) for each sentence. In this processing step, two filters are applied to weed out potentially noisy segments, or entire talks, based on the number of words that were not aligned by Gentle. First, entire talks are discarded if the proportion of unrecognized words is equal or greater than \(15\%\) of the total. This threshold was determined after a manual analysis of \(73\) talks (those with the highest percentage of unrecognized words). The analysis showed that these cases are representative of different types of noise like: _i)_ non-English speech, _ii)_ long silences, _iii)_ music, non-transcribed songs and videos played during the talk, and _iv)_ wrong transcriptions (_e.g._ captions from other talks in the material downloaded from the TED website). The second rule applies to the single sentences of the talks that passed the first filter, and removes those in which none of the words was aligned by Gentle.6

Footnote 6: The effectiveness of this filtering criterion was manually verified on random samples. More aggressive solutions will be explored for future releases of the corpus.

In the last step, the log Mel 40-dimensional filter-bank features - commonly used as input representation for ASR (Graves et al., 2013) and SLT (Weiss et al., 2 

aligned audio using the XNMT tool (Neubig et al., 2018).7

Footnote 6: [https://github.com/neulab/xnmt](https://github.com/neulab/xnmt)

Table 2 provides basic statistics for the \(8\) sections of the MuST-C corpus. Comparing the \(4^{th}\) column with the numbers reported in Table 1, it is worth noting that, in terms of hours of transcribed/translated speech, each section is larger than any existing publicly available SLT resource.

## 3 Experiments

In this section we present two sets of experiments, which are respectively aimed to: _i)_ empirically assess the quality of the MuST-C corpus (Section 3.3) and _ii)_ compute baseline ASR, MT, and SLT results for future comparisons (Section 3.4).

In these experiments, the _audio-transcription_ alignments of MuST-C are used to train and evaluate ASR models, _transcription-translation_ alignments are used for the MT models, and _audio-translation_ alignments are used for the SLT models.

### ASR, MT and SLT Models

#### ASR and SLT.

For our experiments in ASR and SLT we use the same neural architecture. This setting allows us to use the encoder of the ASR models to initialize the weights of the SLT encoders and achieve a faster convergence (Bansal et al., 2018). Our SLT architecture is a variant of the system proposed by Berard et al. (2018), which we re-implemented in the fairseq toolkit (Gehring et al., 2017). The system relies on an attentional encoder-decoder model that takes in input sequences of audio features and outputs the target sequence at the character level. The encoder processes the input with two consecutive fully-connected layers to expand the size of the representation, followed by two 2D strided convolutional layers that reduce the sequence length. The output of the convolutions is then processed by three stacked LSTMs (Hochreiter and Schmidhuber, 1997). The decoder consists of a two-layered deep transition (Pascanu et al., 2014) LSTM with an attention network based on the general soft attention score (Luong et al., 2015). The final output of the decoder is a function of the concatenation of the LSTM output, the context vector and the previous-character embedding.

#### Mt.

For the MT experiments we use the open source version of ModernMT.8 The system is based on the Transformer (Vaswani et al., 2017) architecture, which represents the state of the art in NMT (Bojar et al., 2018). The encoder consists of a stack of \(6\) layers, each containing a sequence of two sub-layers, a self-attention network based on multi-head attention, and a position-wise feed-forward layer. The decoder layers have an additional sub-layer: between the self attention and the position-wise feed-forward layer they have an encoder-decoder multi-head attention. All the sub-layers in both the encoder and decoder are preceded by layer normalization and are followed by residual connections.

Footnote 8: www.modernmt.eu

### Data Processing and Evaluation Metrics

In our experiments, texts are tokenized and punctuation is normalized. Furthermore, the English texts are lowercased, while the target language texts are split into characters still preserving the word boundaries. For MT, we segment the English words with the BPE algorithm (Sennrich et al., 2015) using a maximum of \(30K\) merge operations. The output generation of all models is performed using beam search with a beam size of \(5\).

ASR performance is measured with word error rate (WER) computed on lower-cased, tokenized texts without punctuation. MT and SLT results are computed with BLEU (Papineni et al., 2002).

### Experiment 1: Corpus Quality

As observed in Section 2, each section of MuST-C is larger than any other existing publicly available SLT corpus. The usefulness of a resource, however, is not only a matter of size but also of quality (in this case, the quality of the _audio-transcription-translation_ alignments). For an empirical verification of this aspect, we experimented with two comparable datasets. One is

\begin{table}
\begin{tabular}{l|c|c|c|c|c}
**Tgt** & **\#Ialk** & **\#Sent** & **Hours** & **src w** & **tgt w** \\ \hline
**De** & 2,093 & 234K & 408 & 4.3M & 4.0M \\ \hline
**Es** & 2,564 & 270K & 504 & 5.3M & 5.1M \\ \hline
**Fr** & 2,510 & 280K & 492 & 5.2M & 5.4M \\ \hline
**It** & 2,374 & 258K & 465 & 4.9M & 4.6M \\ \hline
**NI** & 2,267 & 253K & 442 & 4.7M & 4.3M \\ \hline
**Pt** & 2,050 & 211K & 385 & 4.0M & 3.8M \\ \hline
**Ro** & 2,216 & 240K & 432 & 4.6M & 4.3M \\ \hline
**Ru** & 2,498 & 270K & 489 & 5.1M & 4.3M \\ \end{tabular}
\end{table}
Table 2: Statistics for each section of MuST-C.

 

the TED-derived English-German IWSLT18 corpus (Niehues et al., 2018), which is built following a pipeline that performs segment extraction and alignment based on time information (_i.e._ start and end position of each segment in the SubRip Text (SRT) files) instead of text-level alignments. The other is the English-German subset of MuST-C derived from the same TED Talks used to build the IWSLT18 corpus. On one side (MuST-C), the number of segments, their length, and the overall corpus quality depend on text-level alignments. On the other side (IWSLT18), they depend on matching time stamps. This strategy, however, has some drawbacks. First, as pointed out by (Niehues et al., 2018; Liu et al., 2018; Di Gangi et al., 2018), the use of time information brings some noise in the corpus. Second, it often results in utterance-level alignment (based on speakers' pauses in the original audio). Compared to sentence-level alignment, this level of granularity can be sub-optimal during model training (_e.g._ for MT and SLT, learning from complete sentences is easier than learning from phrases). Finally, time information about the recorded speech is not always available: bypassing this need would make the method replicable on other data (not only TED-like).

Though initialized with the same set of \(1,619\) talks, the two pipelines produce different corpora. As shown in Table 3, our approach filters out \(58\) entire talks (\(\sim\)3.6% of the total) but the final number of segments, their corresponding audio duration and their average length (in words) are larger.

Each corpus was divided into training, development and test. Development and test contain segments from randomly selected common talks (_i.e._ those preserved by the MuST-C pipeline). Their size is respectively \(2.3K\) (from \(28\) talks) and \(2.1K\) segments (from \(26\) talks). The test portions were concatenated to create a balanced test set (\(4.2K\) segments) containing half of the instances from the IWSLT18 corpus and half from MuST-C. The remaining material was used to separately train ASR, MT and SLT models on homogeneous data from either of the two corpora (_i.e._ three systems per corpus). All the systems are evaluated on the common test set.

Table 4 shows that the models trained on MuST-C data achieve better results on the balanced test set in all the three tasks. In particular: _i)_ a reduction of \(10.1\) WER points in ASR indicates a higher quality of _audio-transcription_ alignments, _ii)_ a BLEU increase of \(0.56\) points in MT indicates a similar quality for _transcription-translation_ alignments, and _iii)_ a BLEU increase of \(3.31\) points in SLT indicates a higher quality of _audio-translation_ alignments. We consider these results as evidence of the reliability of our corpus creation methodology. Being the same for all the language pairs, we expect this procedure to end up in comparable quality for all the \(8\) sections of MuST-C.

### Experiment 2: Baseline Results

We finally present baseline results computed, for all the three tasks, on each section of MuST-C. Also for these experiments, development and test data are created with segments from talks that are common to all the languages. Their size is respectively \(1.4K\) (from \(11\) talks) and \(2.5K\) segments (from \(27\) talks). The remaining data (of variable size depending on the language pairs) are used for training. For the sake of replicability, these splits are preserved in the released version of MuST-C.

The results in Table 5 lead to the following observations. First, though not directly comparable since they are computed on different test sets, English-German results are in line (actually

\begin{table}
\begin{tabular}{l|c|c|c}
**Training set** & **ASR (\(\downarrow\))** & **MT (\(\uparrow\))** & **SLT (\(\uparrow\))** \\ \hline
**IWSLT18** & 42.15 & 24.90 & 8.94 \\ \hline
**MuST-C** & 32.05 & 25.46 & 12.25 \\ \hline \end{tabular}
\end{table}
Table 4: Performance of ASR, MT and SLT systems trained with En-De IWSLT18 and MuST-C data.

\begin{table}
\begin{tabular}{l|c|c|c|c|c}
**Corpus** & **\#Talk** & **\#Sent** & **Hours** & **src w** & **tgt w** \\ \hline
**IWSLT18** & 1,619 & 176K & 280 & 2.7M & 2.5M \\ \hline
**MuST-C** & 1,561 & 179K & 313 & 3.3M & 3.1M \\ \hline  & \multicolumn{1}{c}{} & \multicolumn{1}{c}{} & \multicolumn{1}{c}{} & \multicolumn{1}{c}{} & \\ \end{tabular}
\end{table}
Table 3: Statistics of the English-German corpora created by applying the IWSLT18 and MuST-C pipelines to the same initial set of \(1,619\) TED Talks.

 

higher, since they are produced by models built on larger training data) with those presented in Section 3.3. This indicates that the level of quality observed in the previous experiments with a subset of the training data is preserved by the whole material released for this language pair. Second, looking at the other language pairs, ASR, MT and SLT results are comparable with the English-German scores. Besides normal fluctuations in the optimization of the neural models, performance differences are coherent with: _i)_ the relative difficulty of each target language (_e.g._ Russian is more difficult due to high inflection) and _ii)_ the variable quantity of training data available (_e.g._ French has the largest training set, see Table 2). Overall, these explainable differences suggest that our corpus creation methodology yields homogeneous quality for all the languages covered by MuST-C.

## 4 Conclusion and Future Work

We presented MuST-C, a _Mu_lilingual _S_peech _T_ranslation _C_orpus built to address the need of resources for training data-hungry neural SLT models. To the best of our knowledge, to date MuST-C is the largest publicly available corpus of this kind. In its current version, it comprises the English transcription and the translations into \(8\) target languages of at least \(385\) hours of speech (up to \(504\)) per language. Thanks to a scalable corpus creation procedure initialized with constantly expanding TED talks data, future extensions will increase the coverage of the already present target languages and introduce new ones.

MuST-C is released under a Creative Commons license, Attribution - Non Commercial - No Derivatives (CC BY NC ND 4.0 International), and is freely downloadable at mustc.fbk.eu

## Acknowledgments

The authors gratefully acknowledge NVIDIA Corporation for the donation of the Tesla K80 and GeForce GTX 1080 Ti GPUs used for this research.

## References

* Anastasopoulos and Chiang (2018) Antonios Anastasopoulos and David Chiang. 2018. Tied Multitask Learning for Neural Speech Translation. In _Proceedings of NAACL-HLT 2018_, pages 82-91, New Orleans, Louisiana.
* Bansal et al. (2018) Sameer Bansal, Herman Kamper, Karen Livescu, Adam Lopez, and Sharon Goldwater. 2018. Pre-Training on High-Resource Speech Recognition Improves Low-Resource Speech-to-Text Translation. _arXiv preprint arXiv:1809.01431_.
* Bendazzoli and Sandrelli (2005) Claudio Bendazzoli and Annalisa Sandrelli. 2005. An Approach to Corpus-based Interpreting Studies: Developing EPIC (European Parliament Interpreting Corpus). In _Proceedings of the EU-High-Level Scientific Conference Series MuTra 2005Challenges of Multidimensional Translation_, pages 149-160, Saarbrucken, Germany.
* Berard et al. (2018) Alexandre Berard, Laurent Besacier, Ali Can Kocabiyikoglu, and Olivier Pietquin. 2018. End-to-End Automatic Speech Translation of Audiobooks. In _Proceedings of ICASSP 2018_, pages 6224-6228, Calgary, Alberta, Canada.
* Berard et al. (2016) Alexandre Berard, Olivier Pietquin, Laurent Besacier, and Christophe Servan. 2016. Listen and Translate: A Proof of Concept for End-to-End Speech-to-Text Translation. In _Proceedings of the NIPS Workshop on end-to-end learning for speech and audio processing_, Barcelona, Spain.
* Boito et al. (2018) Marcely Zanon Boito, Antonios Anastasopoulos, Marika Lekkou, Aline Villavicencio, and Laurent Besacier. 2018. A Small Griko-Italian Speech Translation Corpus. _CoRR_, abs/1807.10740.
* Bojar et al. (2018) Ondrey Bojar, Christian Federmann, Mark Fishel, Yvette Graham, Barry Haddow, Matthias Huck, Philipp Koehn, and Christof Monz. 2018. Findings of the 2018 Conference on Machine Translation (WMT'18). In _Proceedings of the Third Conference on Machine Translation, Volume 2: Shared Task Papers_, pages 272-307, Belgium, Brussels.
* Braune and Fraser (2010) Fabienne Braune and Alexander Fraser. 2010. Improved Unsupervised Sentence Alignment for Symmetrical and Asymmetrical Parallel Corpora. In _Proceedings of COLING 2010_, pages 81-89, Beijing, China.
* Chiu et al. (2018) Chung-Cheng Chiu, Tara Sainath, Yonghui Wu, Rohit Prabhavalkar, Patrick Nguyen, et al. 2018. State-of-the-art Speech Recognition With Sequence-to-Sequence Models. In _Proceedings of ICASSP 2018_, pages 4774-4778, Calgary, Alberta, Canada.
* Gangi et al. (2018) Mattia Antonino Di Gangi, Roberto Dessi, Roldano Cattoni, Matteo Negri, and Marco Turchi. 2018. Fine-tuning on Clean Data for End-to-End Speech Translation: FB&@ IWSLT 2018. In _Proceedings of IWSLT 2018_, Bruges, Belgium.
* Federmann and Lewis (2016) Christian Federmann and William D Lewis. 2016. Microsoft Speech Language Translation (MSLT) Corpus: The IWSLT 2016 Release for English, French and German. In _Proceedings of IWSLT 2016_, Seattle, USA.
* Federmann and Lewis (2017) Christian Federmann and William D Lewis. 2017. The Microsoft Speech Language Translation (MSLT) Corpus for Chinese and Japanese: Conversational Test data for Machine Translation and Speech 

Recognition. In _Proceedings of the 16th Machine Translation Summit_, Nagoya, Japan.
* Gehring et al. (2017) Jonas Gehring, Michael Auli, David Grangier, Denis Yarats, and Yann N Dauphin. 2017. Convolutional Sequence to Sequence Learning. In _Proceedings of ICML 2017_, pages 1243-1252, Sydney, Australia.
* Godard et al. (2017) Pierre Godard, Gilles Adda, Martine Adda-Decker, Juan Benjumea, Laurent Besacier, et al. 2017. A Very Low Resource Language Speech Corpus for Computational Language Documentation Experiments. _arXiv preprint arXiv:1710.03501_.
* Graves et al. (2013) Alex Graves, Abdel-rahman Mohamed, and Geoffrey E. Hinton. 2013. Speech Recognition with Deep Recurrent Neural Networks. In _Proceedings of ICASSP 2018_, pages 6645-6649, Vancouver, BC, Canada.
* Hochreiter and Schmidhuber (1997) Sepp Hochreiter and Jurgen Schmidhuber. 1997. Long Short-Term Memory. _Neural computation_, 9(8).
* Jia et al. (2018) Ye Jia, Melvin Johnson, Wolfgang Macherey, Ron-J. Weiss, Yuan Cao, Chung-Cheng Chiu, Stella-Laurenzo Ari, and Yonghui Wu. 2018. Leveraging Weakly Supervised Data to Improve End-to-End Speech-to-Text Translation. _ArXiv e-prints arXiv:1811.02050_.
* Kocabiyikoglu et al. (2018) Ali Can Kocabiyikoglu, Laurent Besacier, and Olivier Kraif. 2018. Augmenting Librispeech with French Translations: A Multimodal Corpus for Direct Speech Translation Evaluation. In _Proceedings of LFEC 2018_, Miyazaki, Japan.
* Liu et al. (2018) Dan Liu, Junhua Liu, Wu Guo, Shifu Xiong, Zhiqiang Ma, Rui Song, Chongliang Wu, and Quan Liu. 2018. The USTC-NEL Speech Translation system at IWSLT 2018. In _Proceedings of IWSLT 2018_, Bruges, Belgium.
* Luong et al. (2015) Thang Luong, Hieu Pham, and Christopher D Manning. 2015. Effective Approaches to Attention-based Neural Machine Translation. In _Proceedings of EMNLP 2015_, pages 1412-1421, Lisbon, Portugal.
* Neubig et al. (2018) Graham Neubig, Matthias Sperber, Xinyi Wang, Matthieu Felix, Austin Matthews, et al. 2018. XNNT: The eXtensible Neural Machine Translation Toolkit. In _Proceedings of AMTA 2018_, pages 185-192, Boston, MA.
* Niehus et al. (2018) Jan Niehus, Roldano Cattoni, Sebastian Stuker, Mauro Cettolo, Marco Turchi, and Marcello Federico. 2018. The IWSLT 2018 Evaluation Campaign. In _Proceedings of IWSLT 2018_, Bruges, Belgium.
* Papineni et al. (2002) Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. BLEU: a Method for Automatic Evaluation of Machine Translation. In _Proceedings of ACL 2002_, Philadelphia, PA, USA.
* Pascanu et al. (2014) Razvan Pascanu, Caglar Gulcehre, Kyunghyun Cho, and Yoshua Bengio. 2014. How to Construct Deep Recurrent Neural Networks. In _Proceedings of ICLR 2014_, Banff, Canada.
* Paulik and Waibel (2009) Matthias Paulik and Alex Waibel. 2009. Automatic Translation from Parallel Speech: Simultaneous Interpretation as MT Training Data. In _Proceedings of ASRU 2009_, pages 496-501, Merano, Italy.
* Post et al. (2013) Matt Post, Gaurav Kumar, Adam Lopez, Damianos Karakos, Chris Callison-Burch, and Sanjeev Khudanpur. 2013. Improved Speech-to-Text Translation with the Fisher and Callhome Spanish-English Speech Translation Corpus. In _Proceedings of JWSLT 2013_, Heidelberg, Germany.
* Povey et al. (2011) Daniel Povey, Arnab Ghoshal, Gilles Boulianne, Lukas Burget, Ondrej Glembek, K. Nagendra Goel, Mirko Hannemann, Petr Motlicek, Yammin Qian, Petr Schwarz, Jan Silovsky, Georg Stemmer, and Karel Vesely. 2011. The Kaldi Speech Recognition Toolkit. In _Proceedings of ASRU 2011_, pages 1-4, Big Island, Hawaii, USA.
* Sennrich et al. (2015) Rico Sennrich, Barry Haddow, and Alexandra Birch. 2015. Neural Machine Translation of Rare Words with Subword Units. _arXiv preprint arXiv:1508.07909_.
* Shimizu et al. (2014) Hiroaki Shimizu, Graham Neubig, Sakriani Sakti, Tomoki Toda, and Satoshi Nakamura. 2014. Collection of a Simultaneous Translation Corpus for Comparative Analysis. In _Proceedings of LIREC 2014_, Reykjayvik, Iceland.
* Stuker et al. (2012) Sebastian Stuker, Florian Kraft, Christian Mohr, Teresa Herrmann, Eunah Cho, and Alex Waibel. 2012. The KIT Lecture Corpus for Speech Translation. In _Proceedings of LREC-2012_, Istanbul, Turkey.
* Tohyama et al. (2005) Hitomi Tohyama, Shigeki Matsubara, Nobuo Kawaguchi, and Yasuyoshi Inagaki. 2005. Construction and utilization of bilingual speech corpus for simultaneous machine interpretation research. In _Proceedings of INTERSPEECH_, pages 1585-1588, Lisbon, Portugal.
* Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is All you Need. In _Proceedings of NIPS 2017_, pages 5998-6008, Long Beach, CA, USA.
* Weiss et al. (2017) Ron J. Weiss, Jan Chorowski, Navdeep Jaitly, Yonghui Wu, and Zhifeng Chen. 2017. Sequence-to-Sequence Models Can Directly Translate Foreign Speech. In _Proceedings of Interspeech 2017_, pages 2625-2629, Stockholm, Sweden.
* Woldeyohamnis et al. (2017) Michael Melease Woldeyohamnis, Laurent Besacier, and Million Meshesha. 2017. A Corpus for Amharic-English Speech Translation: the Case of Tourism Domain. In _Proceedings of ICT4DA 2017_, pages 129-139, Bahir Dar, Ethiopia