Venue: NAACL 2018

# Tied Multitask Learning for Neural Speech Translation

Antonios Anastasopoulos

David Chiang

Department of Computer Science and Engineeering

University of Notre Dame

{aanastas,dchiang}@nd.edu

###### Abstract

We explore multitask models for neural translation of speech, augmenting them in order to reflect two intuitive notions. First, we introduce a model where the second task decoder receives information from the decoder of the first task, since higher-level intermediate representations should provide useful information. Second, we apply regularization that encourages _transitivity_ and _invertibility_. We show that the application of these notions on jointly trained models improves performance on the tasks of low-resource speech transcription and translation. It also leads to better performance when using attention information for word discovery over unsegmented input.

## 1 Introduction

Recent efforts in endangered language documentation focus on collecting spoken language resources, accompanied by spoken translations in a high resource language to make the resource interpretable Bird et al. (2014). For example, the BULB project Adda et al. (2016) used the LIGAikuma mobile app Bird et al. (2014); Blachon et al. (2016) to collect parallel speech corpora between three Santu languages and French. Since it's common for speakers of endangered languages to speak one or more additional languages, collection of such a resource is a realistic goal.

Speech can be interpreted either by transcription in the original language or translation to another language. Since the size of the data is extremely small, multitask models that jointly train a model for both tasks can take advantage of both signals. Our contribution lies in improving the sequence-to-sequence multitask learning paradigm, by drawing on two intuitive notions: that higher-level representations are more useful than lower-level representations, and that translation should be both transitive and invertible.

_Higher-level intermediate representations_, such as transcriptions, should in principle carry information useful for an end task like speech translation. A typical multitask setup Weiss et al. (2017) shares information at the level of encoded frames, but intuitively, a human translating speech must work from a higher level of representation, at least at the level of phonemes if not syntax or semantics. Thus, we present a novel architecture for _tied_ multitask learning with sequence-to-sequence models, in which the decoder of the second task receives information not only from the encoder, but also from the decoder of the first task.

In addition, _transitivity_ and _invertibility_ are two properties that should hold when mapping between levels of representation or across languages. We demonstrate how these two notions can be implemented through regularization of the attention matrices, and how they lead to further improved performance.

We evaluate our models in three experiment settings: low-resource speech transcription and translation, word discovery on unsegmented input, and high-resource text translation. Our high-resource experiments are performed on English, French, and German. Our low-resource speech experiments cover a wider range of linguistic diversity: Spanish-English, Mboshi-French, and Aniuc English.

In the speech transcription and translation tasks, our proposed model leads to improved performance against all baselines as well as previous multitask architectures. We observe improvements of up to 5% character error rate in the transcription task, and up to 2.8% character-level BLEU in the translation task. However, we didn't observe similar improvements in the text translation experiments. Finally, on the word discovery task, we improve upon previous work by about 3% F-score on both tokens and types.



Model

Our models are based on a sequence-to-sequence model with attention (Bahdanau et al., 2015). In general, this type of model is composed of three parts: a recurrent encoder, the attention, and a recurrent decoder (see Figure 0(a)).1

Footnote 1: For simplicity, we have assumed only a single layer for both the encoder and decoder. It is possible to use multiple stacked RNNs; typically, the output of the encoder and decoder (\({\bf c}_{m}\) and \(P({\bf y}_{n})\), respectively) would be computed from the top layer only.

The encoder transforms an input sequence of words or feature frames \({\bf x}_{1},\ldots,{\bf x}_{N}\) into a sequence of _input states_\({\bf h}_{1},\ldots,{\bf h}_{N}\):

\[{\bf h}_{n}=\operatorname{enc}({\bf h}_{n-1},{\bf x}_{n}).\]

The attention transforms the input states into a sequence of _context vectors_ via a matrix of _attention weights_:

\[{\bf c}_{m}=\sum_{n}\alpha_{mn}{\bf h}_{n}.\]

Finally, the decoder computes a sequence of _output states_ from which a probability distribution over output words can be computed.

\[{\bf s}_{m} =\operatorname{dec}({\bf s}_{m-1},{\bf c}_{m},{\bf y}_{m-1})\] \[P({\bf y}_{m}) =\operatorname{softmax}({\bf s}_{m}).\]

In a standard encoder-decoder _multitask_ model (Figure 0(b)(Dong et al., 2015; Weiss et al., 2017), we jointly model two output sequences using a shared encoder, but separate attentions and decoders:

\[{\bf c}^{1}_{m} =\sum_{n}\alpha^{1}_{mn}{\bf h}_{n}\] \[{\bf s}^{1}_{m} =\operatorname{dec}^{1}({\bf s}^{1}_{m-1},{\bf c}^{1}_{m},{\bf y }^{1}_{m-1})\] \[P({\bf y}^{1}_{m}) =\operatorname{softmax}({\bf s}^{1}_{m})\]

and

\[{\bf c}^{2}_{m} =\sum_{n}\alpha^{2}_{mn}{\bf h}_{n}\] \[{\bf s}^{2}_{m} =\operatorname{dec}^{2}({\bf s}^{2}_{m-1},{\bf c}^{2}_{m},{\bf y }^{2}_{m-1})\] \[P({\bf y}^{2}_{m}) =\operatorname{softmax}({\bf s}^{2}_{m}).\]

We can also arrange the decoders in a _cascade_ (Figure 0(c)), in which the second decoder attends only to the output states of the first decoder:

\[{\bf c}^{2}_{m} =\sum_{m^{\prime}}\alpha^{12}_{mm^{\prime}}{\bf s}^{1}_{m^{\prime}}\] \[{\bf s}^{2}_{m} =\operatorname{dec}^{2}({\bf s}^{2}_{m-1},{\bf c}^{2}_{m},{\bf y }^{2}_{m-1})\] \[P({\bf y}^{2}_{m}) =\operatorname{softmax}({\bf s}^{2}_{m}).\]

Tu et al. (2017) use exactly this architecture to train on bitext by setting the second output sequence to be equal to the input sequence (\({\bf y}^{2}_{i}={\bf x}_{i}\)).

In our proposed _triangle_ model (Figure 0(d)), the first decoder is as above, but the second decoder has two attentions, one for the input states of the encoder and one for the output states of the first decoder:

\[{\bf c}^{2}_{m} =\left[\sum_{m^{\prime}}\alpha^{12}_{mm^{\prime}}{\bf s}^{1}_{m^{ \prime}}\quad\sum_{n}\alpha^{2}_{mn}{\bf h}_{n}\right]\] \[{\bf s}^{2}_{m} =\operatorname{dec}^{2}({\bf s}^{2}_{m-1},{\bf c}^{2}_{m},{\bf y }^{2}_{m-1})\] \[P({\bf y}^{2}_{m}) =\operatorname{softmax}({\bf s}^{2}_{m}).\]

Note that the context vectors resulting from the two attentions are concatenated, not added.

## 3 Learning and Inference

For compactness, we will write \({\bf X}\) for the matrix whose rows are the \({\bf x}_{n}\), and similarly \({\bf H}\), \({\bf C}\), and so on. We also write \({\bf A}\) for the matrix of attention weights: \([{\bf A}]_{ij}=\alpha_{ij}\).

Let \(\theta\) be the parameters of our model, which we train on sentence triples \(({\bf X},{\bf Y}^{1},{\bf Y}^{2})\).

### Maximum likelihood estimation

Define the score of a sentence triple to be a log-linear interpolation of the two decoders' probabilities:

\[\operatorname{score}({\bf Y}^{1},{\bf Y}^{2}\mid{\bf X};\theta) =\lambda\log P({\bf Y}^{1}\mid{\bf X};\theta)+\] \[(1-\lambda)\log P({\bf Y}^{2}\mid{\bf X},{\bf S}^{1};\theta)\]

where \(\lambda\) is a parameter that controls the importance of each sub-task. In all our experiments, we set \(\lambda\) to 0.5. We then train the model to maximize

\[\mathcal{L}(\theta)=\sum\operatorname{score}({\bf Y}^{1},{\bf Y}^{2}\mid{\bf X };\theta),\]

where the summation is over all sentence triples in the training data.

### Regularization

We can optionally add a regularization term to the objective function, in order to encourage our attention mechanisms to conform to two intuitive principles of machine translation: _transitivity_ and _invertibility_.

#### Transitivity attention regularizer

To a first approximation, the translation relation should be transitive (Wang et al., 2006; Levinboim and Chiang, 2015): If source word \({\bf x}_{i}\) aligns to target word 

\({\bf y}_{j}^{1}\) and \({\bf y}_{j}^{1}\) aligns to target word \({\bf y}_{k}^{2}\), then \({\bf x}_{i}\) should also probably align to \({\bf y}_{k}^{2}\). To encourage the model to preserve this relationship, we add the following _transitivity_ regularizer to the loss function of the _triangle_ models with a small weight \(\lambda_{\text{trans}}=0.2\):

Invertibility attention regularizerThe translation relation also ought to be roughly invertible Levinboim et al. (2015): if, in the _reconstruction_ version of the _cascade_ model, source word \({\bf x}_{i}\) aligns to target word \({\bf y}_{j}^{1}\), then it stands to reason that \({\bf y}_{j}\) is likely to align to \({\bf x}_{i}\). So, whereas Tu et al. (2017) let the attentions of the translator and the reconstructor be unrelated, we try adding the following _invertibility_ regularizer to encourage the attentions to each be the inverse of the other, again with a weight \(\lambda_{\text{inv}}=0.2\):

### Decoding

Since we have two decoders, we now need to employ a two-phase beam search, following Tu et al. (2017):

1. The _first decoder_ produces, through standard beam search, a set of triples each consisting of a candidate transcription \(\hat{\mathbf{Y}}^{1}\), a score \(P(\hat{\mathbf{Y}}^{1})\), and a hidden state sequence \(\hat{\mathbf{S}}\).
2. For each transcription candidate from the _first decoder_, the _second decoder_ now produces through beam search a set of candidate translations \(\hat{\mathbf{Y}}^{2}\), each with a score \(P(\hat{\mathbf{Y}}^{2})\).
3. We then output the combination that yields the highest total score(\(\mathbf{Y}^{1},\mathbf{Y}^{2}\)).

### Implementation

All our models are implemented in DyNet Neubig et al. (2017).2 We use a dropout of 0.2, and train using Adam with initial learning rate of 0.0002 for a maximum of 500 epochs. For testing, we select the model with the best performance on dev. At inference time, we use a beam size of 4 for each decoder (due to GPU memory constraints), and the beam scores include length normalization Wu et al. (2016) with a weight of 0.8, which Nguyen and Chiang (2017) found to work well for low-resource NMT.

Footnote 2: Our code is available at: [https://bitbucket.org/antonis/dynet-multitask-models](https://bitbucket.org/antonis/dynet-multitask-models).

## 4 Speech Transcription and Translation

We focus on speech transcription and translation of endangered languages, using three different cor

Figure 1: Variations on the standard attentional model. In the standard _single-task_ model, the decoder attends to the encoder’s states. In a typical _multitask_ setup, two decoders attend to the encoder’s states. In the _cascade_Tu et al. (2017), the second decoder attends to the first decoder’s states. In our proposed _triangle_ model, the second decoder attends to both the encoder’s states and the first decoder’s states. Note that for clarity’s sake there are dependencies not shown.

 

pora on three different language directions: Spanish (es) to English (en), Ainu (ai) to English, and Mboshi (mb) to French (fr).

### Data

Spanish is, of course, not an endangered language, but the availability of the CALLHOME Spanish Speech dataset (LDC2014T23) with English translations [57] makes it a convenient language to work with, as has been done in almost all previous work in this area. It consists of telephone conversations between relatives (about 20 total hours of audio) with more than 240 speakers. We use the original train-dev-test split, with the training set comprised of 80 conversations and dev and test of 20 conversations each.

Hokkaido Ainu is the sole surviving member of the Ainu language family and is generally considered a language isolate. As of 2007, only ten native speakers were alive. The Glossed Audio Corpus of Ainu Folklore provides 10 narratives with audio (about 2.5 hours of audio) and translations in Japanese and English.3 Since there does not exist a standard train-dev-test split, we employ a cross validation scheme for evaluation purposes. In each fold, one of the 10 narratives becomes the test set, with the previous one (mod 10) becoming the dev set, and the remaining 8 narratives becoming the training set. The models for each of the 10 folds are trained and tested separately. On average, for each fold, we train on about 2000 utterances; the dev and test sets consist of about 270 utterances. We report results on the concatenation of all folds. The Ainu text is split into characters, except for the equals (=) and underscore (_) characters, which are used as phonological or structural markers and are thus merged with the following character.4

Footnote 3: [http://ainucorpus.ninjal.ac.jp/corpus/en/](http://ainucorpus.ninjal.ac.jp/corpus/en/)

Mboshi [69] is a language spoken in Congo-Brazzaville, without standard orthography. We use a corpus [10] of 5517 parallel utterances (about 4.4 hours of audio) collected from three native speakers. The corpus provides non-standard grapheme transcriptions (close to the language phonology) produced by linguists, as well as French translations. We sampled 100 segments from the training set to be our dev set, and used the original dev set (514 sentences) as our test set.

### Implementation

We employ a 3-layer speech encoding scheme similar to that of Duong et al. duong 

output the sequences at the grapheme level, so the output embedding size is set to 64.

We found that this simpler speech encoder works well for our extremely small datasets. Applying our models to larger datasets with many more speakers would most likely require a more sophisticated speech encoder, such as the one used by Weiss et al. (2017).

### Results

In Table 2, we present results on three small datasets that demonstrate the efficacy of our models. We compare our proposed models against three baselines and one "skyline." The first baseline is a traditional pivot approach (line 1), where the ASR output, a sequence of characters, is the input to a character-based NMT system (trained on gold transcriptions). The "skyline" model (line 2) is the same NMT system, but tested on gold transcriptions instead of ASR output. The second baseline is translation directly from source speech to target text (line 3). The last baseline is the standard _multitask_ model (line 4), which is similar to the model of Weiss et al. (2017).

The _cascade_ model (line 5) outperforms the baselines on the translation task, while only falling behind the _multitask_ model in the transcription task. On all three datasets, the _triangle_ model (lines 6, 7) outperforms all baselines, including the standard _multitask_ model. On Ainu-English, we even obtain translations that are comparable to the "skyline" model, which is tested on gold Ainu transcriptions.

Comparing the performance of all models across the three datasets, there are two notable trends that verify common intuitions regarding the speech transcription and translation tasks. First, an increase in the number of speakers hurts the performance of the speech transcription tasks. The character error rates for Ainu are smaller than the CER in Mboshi, which in turn are smaller than the CER in CALLHOME. Second, the character-level BLEU scores increase as the amount of training data increases, with our smallest dataset (Ainu) having the lowest BLEU scores, and the largest dataset (CALLHOME) having the highest BLEU scores. This is expected, as more training data means that the translation decoder learns a more informed character-level language model for the target language.

Note that Weiss et al. (2017) report much higher BLEU scores on CALLHOME: our model underperforms theirs by almost 9 _word-level_ BLEU points. However, their model has significantly more parameters and is trained on 10 times more data than ours. Such an amount of data would never be available in our endangered languages scenario. When calculated on the word-level, all our models' BLEU scores are between 3 and 7 points for the extremely low resource datasets (Mboshi-French and Ainu-English), and between 7 and 10 for CALLHOME. Clearly, the size of the training data in our experiments is not enough for producing high quality speech translations, but we plan to investigate the performance of our proposed models on larger datasets as part of our future work.

To evaluate the effect of using the combined score from both decoders at decoding time, we evaluated the _triangle_ models using only the 1-best output from the speech model (lines 8, 9). One would expect that this would favor speech at the expense of translation. In transcription accuracy, we indeed observed improvements across the board. In translation accuracy, we observed a surprisingly large drop on Mboshi-French, but surprisingly little effect on the other language pairs - in fact, BLEU scores tended to go up slightly, but not significantly.

Finally, Figure 2 visualizes the attention matrices for one utterance from the baseline multitask model and our proposed _triangle_ model. It is clear that our intuition was correct: the translation decoder receives most of its context from the transcription decoder, as indicated by the higher attention weights of \({\bf A}^{12}\). Ideally, the area under the red squares (gold alignments) would account for 100% of the attention mass of \({\bf A}^{12}\). In our triangle model, the total mass under the red squares is 34%, whereas the multitask model's correct attentions amount to only 21% of the attention mass.

## 5 Word Discovery

Although the above results show that our model gives large performance improvements, in absolute terms, its performance on such low-resource tasks leaves a lot of room for future improvement. A possible more realistic application of our methods is word discovery, that is, finding word boundaries in unsegmented phonetic transcriptions.

After training an attentional encoder-decoder model between Mboshi unsegmented phonetic se 

quences and French word sequences, the attention weights can be thought of as soft alignments, which allow us to project the French word boundaries onto Mboshi. Although we could in principle perform word discovery directly on speech, we leave this for future work, and only explore single-task and reconstruction models.

### Data

We use the same Mboshi-French corpus as in Section 4, but with the original training set of 4617 utterances and the dev set of 514 utterances. Our parallel data consist of the unsegmented phonetic Mboshi transcriptions, along with the word-level French translations.

### Implementation

We first replicate the model of Boito et al. (2017), with a single-layer bidirectional encoder and single layer decoder, using an embedding and hidden size of 12 for the base model, and an embedding and hidden state size of 64 for the reverse model. In our own models, we set the embedding size to 32 for Mboshi characters, 64 for French words, and the hidden state size to 64. We smooth the attention weights **A** using the method of Duong et al. (2016) with a temperature \(T=10\) for the softmax computation of the attention mechanism.

Following Boito et al. (2017), we train models both on the _base_ Mboshi-to-French direction, as well as the _reverse_ (French-to-Mboshi) direction, with and without this smoothing operation. We further smooth the computed soft alignments of all models so that \(a_{mn}=(a_{nm-1}+a_{mn}+a_{mm+1})/3\) as a post-processing step. From the _single-task_ models we extract the \(\textbf{A}^{1}\) attention matrices. We also train _reconstruction_ models on both directions, with and without the _invertibility_ regularizer, extracting both \(\textbf{A}^{1}\) and \(\textbf{A}^{12}\) matrices. The two matrices are then combined so that \(\textbf{A}=\textbf{A}^{1}+(\textbf{A}^{12})^{T}\).

### Results

Evaluation is done both at the token and the type level, by computing precision, recall, and F-score over the discovered segmentation, with the best results shown in Table 3. We reimplemented the base (Mboshi-French) and reverse (French-Mboshi) models from Boito et al. (2017), and the performance of the base model was comparable to the one reported. However, we were unable to

Figure 2: Attentions in an Mboshi-French sentence, extracted from two of our models. The red squares denote gold alignments. The second decoder of the _triangle_ model receives most of its context from the first decoder through \(\textbf{A}^{12}\) instead of the source. The \(\textbf{A}^{2}\) matrix of the _triangle_ model is more informed (34% correct attention mass) than the _multitask_ one (21% correct), due to the _transitivity_ regularizer.

 

reproduce the significant gains that were reported when using the reverse model (_italicized_ in Table 3). Also, our version of both the base and reverse singletask models performed better than our reimplementation of the baseline.

Furthermore, we found that we were able to obtain even better performance at the type level by combining the attention matrices of a reconstruction model trained with the _invertibility_ regularizer. Boito et al. (2017) reported that combining the attention matrices of a base and a reverse model significantly reduced performance, but they trained the two models separately. In contrast, we obtain the base (\({\bf A}^{1}\)) and the reverse attention matrices (\({\bf A}^{12}\)) from a model that trains them jointly, while also tying them together through the _invertibility_ regularizer. Using the regularizer is key to the improvements; in fact, we did not observe any improvements when we trained the reconstruction models without the regularizer.

## 6 Negative Results: High-Resource Text Translation

### Data

For evaluating our models on text translation, we use the Europarl corpus which provides parallel sentences across several European languages. We extracted 1,450,890 three-way parallel sentences on English, French, and German. The concatenation of the newstest 2011-2013 sets (8,017 sentences) is our dev set, and our test set is the concatenation of the newstest 2014 and 2015 sets (6,003 sentences). We test all architectures on the six possible translation directions between English (en), French (fr) and German (de). All the sequences are represented by subword units with byte-pair encoding (BPE) (Sennrich et al., 2016) trained on each language with 32000 operations.

### Experimental Setup

On all experiments, the encoder and the decoder(s) have 2 layers of LSTM units with hidden state size and attention size of 1024, and embedding size of 1024. For this high resource scenario, we only train for a maximum of 40 epochs.

### Results

The accuracy of all the models on all six language pair directions is shown in Table 4. In all cases, the best models are the baseline single-task or simple multitask models. There are some instances, such as English-German, where the _reconstruction_ or the _triangle_ models are not statistically significantly different from the best model. The reason for this, we believe, is that in the case of text translation between so linguistically close languages, the lower level representations (the output of the encoder) provide as much information as the higher level ones, without the search errors that are introduced during inference.

A notable outcome of this experiment is that we do not observe the significant improvements with the _reconstruction_ models that Tu et al. (2017) observed. A few possible differences between our experiment and theirs are: our models are BPE-based, theirs are word-based; we use Adam for optimization, they use Adadelta; our model has slightly fewer parameters than theirs; we test on less typologically different language pairs than

\begin{table}
\begin{tabular}{p{42.7pt} p{28.5pt}|r r r|r r r} \hline \hline  & & \multicolumn{3}{c|}{Tokens} & \multicolumn{3}{c}{Types} \\ Model (with smoothing) & Precision & Recall & F-score & Precision & Recall & F-score \\ \hline Boito et al. 2017 & _base_ & _5.85_ & _6.82_ & _6.30_ & _6.76_ & _15.00_ & _9.32_ \\ (reported) & _reverse_ & _21.44_ & _16.49_ & _18.64_ & _27.23_ & _15.02_ & _19.36_ \\ \hline Boito et al. 2017 & base & 6.87 & 6.33 & 6.59 & 6.17 & 13.02 & 8.37 \\ (reimplementation) & reverse & 7.58 & 8.16 & 7.86 & 9.22 & 11.97 & 10.42 \\ \hline  & base & 7.99 & 7.57 & 7.78 & 7.59 & **16.41** & 10.38 \\  & reverse & **11.31** & **11.82** & **11.56** & 9.29 & 14.75 & 11.40 \\ \hline  & \multicolumn{1}{c|}{} & \multicolumn{1}{c|}{} & \multicolumn{1}{c|}{} & \multicolumn{1}{c|}{} & \multicolumn{1}{c|}{} & \\ reconstruction + 0.2\(\mathcal{L}_{\text{inv}}\) & 8.93 & 9.78 & 9.33 & 8.66 & 15.48 & 11.02 \\ reconstruction + 0.5\(\mathcal{L}_{\text{inv}}\) & 7.42 & 10.00 & 8.52 & **10.46** & 16.36 & **12.76** \\ \hline \hline \end{tabular}
\end{table}
Table 3: The reconstruction model with the _invertibility_ regularizer produces more informed attentions that result in better word discovery for Mboshi with an Mboshi-French model. Scores reported by previous work are in _italics_ and best scores from our experiments are in **bold**.

 

English-Chinese.

However, we also observe that in most cases our proposed regularizers lead to increased performance. The _invertibility_ regularizer aids the _reconstruction_ models in achieve slightly higher BLEU scores in 3 out of the 6 cases. The _transitivity_ regularizer is even more effective: in 9 out the 12 source-target language combinations, the _triangle_ models achieve higher performance when trained using the regularizer. Some of them are statistical significant improvements, as in the case of French to English where English is the intermediate target language and German is the final target.

## 7 Related Work

The speech translation problem has been traditionally approached by using the output of an ASR system as input to a MT system. For example, Ney (1999) and Matusov et al. (2005) use ASR output lattices as input to translation models, integrating speech recognition uncertainty into the translation model. Recent work has focused more on modelling speech translation without explicit access to transcriptions. Duong et al. (2016) introduced a sequence-to-sequence model for speech translation without transcriptions but only evaluated on alignment, while Anastasopoulos et al. (2016) presented an unsupervised alignment method for speech-to-translation alignment. Bansal et al. (2017) used an unsupervised term discovery system Jansen et al. (2010) to cluster recurring audio segments into pseudowords and translate speech using a bag-of-words model. Berard et al. (2016) translated synthesized speech data using a model similar to the Listen Attend and Spell model Chan et al. (2016). A larger-scale study Berard et al. (2018) used an end-to-end neural system system for translating audio books between French and English. On a different line of work, Boito et al. (2017) used the attentions of a sequence-to-sequence model for word discovery.

Multitask learning Caruana (1998) has found extensive use across several machine learning and NLP fields. For example, Luong et al. (2016) and Eriguchi et al. (2017) jointly learn to parse and translate; Kim et al. (2017) combine CTC- and attention-based models using multitask models for speech transcription; Dong et al. (2015) use multitask learning for multiple language translation. Toshniwal et al. (2017) apply multitask learning to neural speech recognition in a less traditional fashion: the lower-level outputs of the speech encoder are used for fine-grained auxiliary tasks such as predicting HMM states or phonemes, while the final output of the encoder is passed to a character-level decoder.

Our work is most similar to the work of Weiss et al. (2017). They used sequence-to-sequence models to transcribe Spanish speech and translate it in English, by jointly training the two tasks in a multitask scenario where the decoders share the encoder. In contrast to our work, they use a large corpus for training the model on roughly 163 hours of data, using the Spanish Fisher and CALL

\begin{table}
\begin{tabular}{l|c c c c c c} \hline \hline Model & \multicolumn{4}{c}{\(s\to t\)} \\  & en\(\rightarrow\)fr & en\(\rightarrow\)de & fr\(\rightarrow\)en & fr\(\rightarrow\)de & de\(\leftarrow\)en & de\(\rightarrow\)fr \\ \hline singletask & **20.92** & **12.69** & **20.96** & **11.24** & **16.10** & **15.29** \\ multitask \(s\to x,t\) & 20.54 & **12.79** & 20.01 & **11.18** & **16.31** & **15.07** \\ cascade \(s\to x\to t\) & 15.93 & 11.31 & 16.58 & 7.60 & 13.46 & 13.24 \\ cascade \(s\to t\to x\) & 20.34 & 12.27 & 19.17 & **11.09** & 15.24 & 14.78 \\ reconstruction & 20.19 & **12.44** & 20.63 & 10.88 & 15.66 & 13.44 \\ reconstruction +\(\mathcal{L}_{\text{inv}}\) & **20.72** & **12.64** & 20.11 & 10.46 & 15.43 & 12.64 \\ triangle \(s\xrightarrow[]{x\to}t\) & 20.39 & **12.70** & 17.93 & 10.17 & 14.94 & 14.07 \\ triangle \(s\xrightarrow[]{x\to}t+\mathcal{L}_{\text{trans}}\) & 20.52 & **12.64** & 18.34 & 10.42 & 15.22 & 14.37 \\ triangle \(s\xrightarrow[]{t\to}x\) & 20.38 & **12.40** & 18.50 & 10.22 & 15.62 & 14.77 \\ triangle \(s\xrightarrow[]{t\to}x\) & 20.64 & **12.42** & 19.20 & 10.21 & **15.87** & 14.89 \\ \hline \hline \end{tabular}
\end{table}
Table 4: BLEU scores for each model and translation direction \(s\to t\). In the multitask, cascade, and triangle models, \(x\) stands for the third language, other than \(s\) and \(t\). In each column, the best results are highlighted. The non-highlighted results are statistically significantly worse than the single-task baseline.

 

HOME conversational speech corpora. The parameter number of their model is significantly larger than ours, as they use 8 encoder layers, and 4 layers for each decoder. This allows their model to adequately learn from such a large amount of data and deal well with speaker variation. However, training such a large model on endangered language datasets would be infeasible.

Our model also bears similarities to the architecture of the model proposed by Tu et al. (2017). They report significant gains in Chinese-English translation by adding an additional _reconstruction_ decoder that attends on the last states of the _translation_ decoder, mainly inspired by auto-encoders.

## 8 Conclusion

We presented a novel architecture for multitask learning that provides the second task with higher-level representations produced from the first task decoder. Our model outperforms both the single-task models as well as traditional multitask architectures. Evaluating on extremely low-resource settings, our model improves on both speech transcription and translation. By augmenting our models with regularizers that implement transitivity and invertibility, we obtain further improvements on all low-resource tasks.

These results will hopefully lead to new tools for endangered language documentation. Projects like BULB aim to collect about 100 hours of audio with translations, but it may be impractical to transcribe this much audio for many languages. For future work, we aim to extend these methods to settings where we don't necessarily have sentence triples, but where some audio is only transcribed and some audio is only translated.

AcknowledgementsThis work was generously supported by NSF Award 1464553. We are grateful to the anonymous reviewers for their useful comments.

## References

* Adda et al. (2016) Gilles Adda, Sebastian Stuker, Martine AdaA-Decker, Odette Ambouroue, Laurent Besacier, David Blachon, Helene Bonneau-Maynard, Pierre Godard, Fatima Hamlaoui, Dmitry Idiatov, et al. 2016. Breaking the unwritten language barrier: The BULB project. _Procedia Computer Science_, 81:8-14.
* Anastasopoulos et al. (2016) Antonios Anastasopoulos, David Chiang, and Long Duong. 2016. An unsupervised probability model for speech-to-translation alignment of low-resource languages. In _Proc. EMNLP_.
* Bahdanau et al. (2015) Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural machine translation by jointly learning to align and translate. In _Proc. ICLR_.
* Bansal et al. (2017) Sameer Bansal, Herman Kamper, Adam Lopez, and Sharon Goldwater. 2017. Towards speech-to-text translation without speech recognition. In _Proc. EACL_.
* Berard et al. (2018) Alexandre Berard, Laurent Besacier, Ali Can Kocabiyikoglu, and Olivier Pietquin. 2018. End-to-end automatic speech translation of audiobooks. arXiv:1802.04200.
* Berard et al. (2016) Alexandre Berard, Olivier Pietquin, Christophe Servan, and Laurent Besacier. 2016. Listen and translate: A proof of concept for end-to-end speech-to-text translation. In _Proc. NIPS Workshop on End-to-end Learning for Speech and Audio Processing_.
* Bird et al. (2014a) Steven Bird, Lauren Gawne, Katie Gelbart, and Isaac McAlister. 2014a. Collecting bilingual audio in remote indigenous communities. In _Proc. COLING_.
* Bird et al. (2014b) Steven Bird, Florian R. Hanke, Oliver Adams, and Haejoong Lee. 2014b. Aikuma: A mobile app for collaborative language documentation. In _Proc. of the 2014 Workshop on the Use of Computational Methods in the Study of Endangered Languages_.
* Blachon et al. (2016) David Blachon, Elodie Gauthier, Laurent Besacier, Guy-Noel Kouarata, Martime Adda-Decker, and Annie Rialland. 2016. Parallel speech collection for under-resourced language studies using the LIG-Aikuma mobile device app. In _Proc. SLTU (Spoken Language Technologies for Under-Resourced Languages)_, volume 81.
* Boito et al. (2017) Marcely Zanon Boito, Alexandre Berard, Aline Villavicencio, and Laurent Besacier. 2017. Unwritten languages demand attention too! word discovery with encoder-decoder models. arXiv:1709.05631.
* Caruana (1998) Rich Caruana. 1998. Multitask learning. In _Learning to learn_, pages 95-133. Springer.
* Chan et al. (2016) William Chan, Navdeep Jaitly, Quoc Le, and Oriol Vinyals. 2016. Listen, attend and spell: A neural network for large vocabulary conversational speech recognition. In _Proc. ICASSP_, pages 4960-4964. IEEE.
* Dong et al. (2015) Daxiang Dong, Hua Wu, Wei He, Dianhai Yu, and Haifeng Wang. 2015. Multi-task learning for multiple language translation. In _Proc. ACL-IJCNLP_.
* Duong et al. (2016) Long Duong, Antonios Anastasopoulos, David Chiang, Steven Bird, and Trevor Cohn. 2016. An attentional model for speech translation without transcription. In _Proc. NAACL HLT_.
* Eriguchi et al. (2017) Akiko Eriguchi, Yoshimasa Tsuruoka, and Kyunghyun Cho. 2017. Learning to parse and translate improves neural machine translation. In _Proc. ACL_ 

P. Godard, G. Adda, M. Adda-Decker, J. Benjumea, L. Besacier, J. Cooper-Leavitt, G-N. Kouarata, L. Lamel, H. Maynard, M. Mueller, et al. 2017. A very low resource language speech corpus for computational language documentation experiments. arXiv:1710.03501.
* Hermansky (1990) Hynek Hermansky. 1990. Perceptual linear predictive (PLP) analysis of speech. _J. Acoustical Society of America_, 87(4):1738-1752.
* Jansen et al. (2010) Aren Jansen, Kenneth Church, and Hynek Hermansky. 2010. Towards spoken term discovery at scale with zero resources. In _Proc. INTERSPEECH_.
* Kim et al. (2017) Suyoun Kim, Takaaki Hori, and Shinji Watanabe. 2017. Joint CTC-attention based end-to-end speech recognition using multi-task learning. In _Proc. ICASSP_.
* Levinboim and Chiang (2015) Tomer Levinboim and David Chiang. 2015. Multi-task word alignment triangulation for low-resource languages. In _Proc. NAACL HLT_.
* Levinboim et al. (2015) Tomer Levinboim, Ashish Vaswani, and David Chiang. 2015. Model invertibility regularization: Sequence alignment with or without parallel data. In _Proc. NAACL HLT_.
* Luong et al. (2016) Minh-Thang Luong, Quoc V Le, Ilya Sutskever, Oriol Vinyals, and Lukasz Kaiser. 2016. Multi-task sequence to sequence learning. In _Proc. ICLR_.
* Matusov et al. (2005) Evgeny Matusov, Stephan Kanthak, and Hermann Ney. 2005. On the integration of speech recognition and statistical machine translation. In _Ninth European Conference on Speech Communication and Technology_.
* Neubig et al. (2017) Graham Neubig, Chris Dyer, Yoav Goldberg, Austin Matthews, Waleed Ammar, Antonios Anastasopoulos, Miguel Ballesteros, David Chiang, Daniel Clothiaux, Trevor Cohn, et al. 2017. DyNet: The dynamic neural network toolkit. arXiv:1701.03980.
* Ney (1999) Hermann Ney. 1999. Speech translation: Coupling of recognition and translation. In _Proc. ICASSP_, volume 1.
* Nguyen and Chiang (2017) Toan Q. Nguyen and David Chiang. 2017. Transfer learning across low-resource related languages for neural machine translation. In _Proc. IJCNLP_.
* Post et al. (2013) Matt Post, Gaurav Kumar, Adam Lopez, Damianos Karakos, Chris Callison-Burch, and Sanjeev Khudanpur. 2013. Improved speech-to-text translation with the Fisher and Callhome Spanish-English speech translation corpus. In _Proc. IWSLT_.
* Sennrich et al. (2016) Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Neural machine translation of rare words with subword units. In _Proc. ACL_.
* Toshniwal et al. (2017) Shubham Toshniwal, Hao Tang, Liang Lu, and Karen Livescu. 2017. Multitask learning with low-level auxiliary tasks for encoder-decoder based speech recognition. In _Proc. Interspeech_.
* Tu et al. (2017) Zhaopeng Tu, Yang Liu, Lifeng Shang, Xiaohua Liu, and Hang Li. 2017. Neural machine translation with reconstruction. In _Proc. AAAI_.
* Wang et al. (2006) Haifeng Wang, Hua Wu, and Zhanyi Liu. 2006. Word alignment for languages with scarce resources using bilingual corpora of other language pairs. In _Proc. COLING/ACL_, pages 874-881.
* Weiss et al. (2017) Ron J. Weiss, Jan Chorowski, Navdeep Jaitly, Yonghui Wu, and Zhifeng Chen. 2017. Sequence-to-sequence models can directly transcribe foreign speech. In _Proc. INTERSPEECH_.
* Wu et al. (2016) Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V. Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, et al. 2016. Google's neural machine translation system: Bridging the gap between human and machine translation. arXiv:1609.08144.