prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. What was the previous state of the art for this task?
Context 1: Introduction
Massive Open Online Courses (MOOCs) have strived to bridge the social gap in higher education by bringing quality education from reputed universities to students at large  Such massive scaling through online classrooms, however, disrupt co-located, synchronous two-way communication between the students and the instructor 
MOOC platforms provide discussion forums for students to talk to their classmates about the lectures, homeworks, quizzes and provide a venue to socialise  Instructors (defined here as the course instructors, their teaching assistants and the MOOC platform's technical staff) monitor the discussion forum to post (reply to their message) in discussion threads among students  We refer to this posting as intervention, following prior work BIBREF0   However, due to large student enrolment, the student–instructor ratio in MOOCs is very high Therefore, instructors are not able to monitor and participate in all student discussions  To address this problem, a number of works have proposed systems e g , BIBREF0 , BIBREF1 to aid instructors to selectively intervene on student discussions where they are needed the most 
In this paper, we improve the state-of-the-art for instructor intervention in MOOC forums  We propose the first neural models for this prediction problem  We show that modelling the thread structure and the sequence of posts explicitly improves performance  Instructors in different MOOCs from different subject areas intervene differently  For example, on a Science, Technology, Engineering and Mathematics (STEM) MOOC, instructors may often intervene early as possible to resolve misunderstanding of the subject material and prevent confusion  However, in a Humanities MOOC, instructors allow for the students to explore open-ended discussions and debate among themselves  Such instructors may prefer to intervene later in the discussion to encourage further discussion or resolve conflicts among students  We therefore propose attention models to infer the latent context, i e , the series of posts that trigger an intervention  Earlier studies on MOOC forum intervention either model the entire context or require the context size to be specified explicitly 
Problem Statement
A thread INLINEFORM0 consists of a series of posts INLINEFORM1 through INLINEFORM2 where INLINEFORM3 is an instructor's post when INLINEFORM4 is intervened, if applicable  INLINEFORM5 is considered intervened if an instructor had posted at least once  The problem of predicting instructor intervention is cast as a binary classification problem  Intervened threads are predicted as 1 given while non-intervened threads are predicted as 0 given posts INLINEFORM6 through INLINEFORM7  
The primary problem leads to a secondary problem of inferring the appropriate amount of context to intervene  We define a context INLINEFORM0 of a post INLINEFORM1 as a series of linear contiguous posts INLINEFORM2 through INLINEFORM3 where INLINEFORM4   The problem of inferring context is to identify context INLINEFORM5 from a set of candidate contexts INLINEFORM6  
Modelling Context in Forums
Context has been used and modelled in various ways for different problems in discussion forums  In a work on a closely related problem of forum thread retrieval BIBREF2 models context using inter-post discourse e g , Question-Answer  BIBREF3 models the structural dependencies and relationships between forum posts using a conditional random field in their problem to infer the reply structure  Unlike BIBREF2 , BIBREF3 can be used to model any structural dependency and is, therefore, more general  In this paper, we seek to infer general dependencies between a reply and its previous context whereas BIBREF3 inference is limited to pairs of posts  More recently BIBREF4 proposed a context based model which factorises attention over threads of different lengths  Differently, we do not model length but the context before a post  However, our attention models cater to threads of all lengths 
BIBREF5 proposed graph structured LSTM to model the explicit reply structure in Reddit forums  Our work does not assume access to such a reply structure because 1) Coursera forums do not provide one and 2) forum participants often err by posting their reply to a different post than that they intended  At the other end of the spectrum are document classification models that do not assume structure in the document layout but try to infer inherent structure in the natural language, viz, words, sentences, paragraphs and documents  Hierarchical attention BIBREF6 is a well know recent work that classifies documents using a multi-level LSTMs with attention mechanism to select important units at each hierarchical level  Differently, we propose a hierarchical model that encodes layout hierarchy between a post and a thread but also infers reply structure using a attention mechanism since the layout does not reliably encode it 
Instructor Intervention in MOOC forums
The problem of predicting instructor intervention in MOOCs was proposed by BIBREF0   Later BIBREF7 evaluated baseline models by BIBREF0 over a larger corpus and found the results to vary widely across MOOCs  Since then subsequent works have used similar diverse evaluations on the same prediction problem BIBREF1 , BIBREF8   BIBREF1 proposed models with discourse features to enable better prediction over unseen MOOCs  BIBREF8 recently showed interventions on Coursera forums to be biased by the position at which a thread appears to an instructor viewing the forum interface and proposed methods for debiased prediction 
While all works since BIBREF0 address key limitations in this line of research, they have not investigated the role of structure and sequence in the threaded discussion in predicting instructor interventions  BIBREF0 proposed probabilistic graphical models to model structure and sequence  They inferred vocabulary dependent latent post categories to model the thread sequence and infer states that triggered intervention  Their model, however, requires a hyperparameter for the number of latent states  It is likely that their empirically reported setting will not generalise due to their weak evaluation BIBREF7   In this paper, we propose models to infer the context that triggers instructor intervention that does not require context lengths to be set apriori  All our proposed models generalise over modelling assumptions made by BIBREF0  
For the purpose of comparison against a state-of-the-art and competing baselines we choose BIBREF7 since BIBREF0 's system and data are not available for replication 
Data and Preprocessing
We evaluate our proposed models over a corpus of 12 MOOC iterations (offerings) on Coursera org In partnership with Coursera and in line with its Terms of Service, we obtained the data for use in our academic research  Following prior work BIBREF7 we evaluate over a diverse dataset to represent MOOCs of varying sizes, instructor styles, instructor team sizes and number of threads intervened  We only include threads from sub-forums on Lecture, Homework, Quiz and Exam  We also normalise and label sub-forums with other non-standard names (e g , Assignments instead of Homework) into of the four said sub-forums  Threads on general discussion, meet and greet and other custom sub-forums for social chitchat are omitted as our focus is to aid instructors on intervening on discussion on the subject matter  We also exclude announcement threads and other threads started by instructors since they are not interventions  We preprocess each thread by replacing URLs, equations and other mathematical formulae and references to timestamps in lecture videos by tokens INLINEFORM0 URL INLINEFORM1 , INLINEFORM2 MATH INLINEFORM3 , INLINEFORM4 TIMEREF INLINEFORM5 respectively  We also truncate intervened threads to only include posts before the first instructor post since the instructor's and subsequent posts will bias the prediction due to the instructor's post 
Model
The key innovation of our work is to decompose the intervention prediction problem into a two-stage model that first explicitly tries to discover the proper context to which a potential intervention could be replying to, and then, predict the intervention status  This model implicitly assesses the importance (or urgency) of the existing thread's context to decide whether an intervention is necessary  For example in Figure SECREF1 , prior to the instructor's intervention, the ultimate post (Post #6) by Student 2 already acknowledged the OP's gratitude for his answer  In this regard, the instructor may have decided to use this point to summarize the entire thread to consolidate all the pertinent positions  Here, we might assume that the instructor's reply takes the entire thread (Posts #1–6) as the context for her reply 
This subproblem of inferring the context scope is where our innovation centers on  To be clear, in order to make the prediction that a instruction intervention is now necessary on a thread, the instructor's reply is not yet available — the model predicts whether a reply is necessary — so in the example, only Posts #1–6 are available in the problem setting  To infer the context, we have to decide which subsequence of posts are the most plausible motivation for an intervention 
Recent work in deep neural modeling has used an attention mechanism as a focusing query to highlight specific items within the input history that significantly influence the current decision point  Our work employs this mechanism – but with a twist: due to the fact that the actual instructor intervention is not (yet) available at the decision timing, we cannot use any actual intervention to decide the context  To employ attention, we must then employ a surrogate text as the query to train our prediction model  Our model variants model assess the suitability of such surrogate texts for the attention mechanism basis 
Congruent with the representation of the input forums, in all our proposed models, we encode the discussion thread hierarchically  We first build representations for each post by passing pre-trained word vector representations from GloVe BIBREF9 for each word through an LSTM BIBREF10 , INLINEFORM0   We use the last layer output of the LSTM as a representation of the post  We refer this as the post vector INLINEFORM1  
Then each post INLINEFORM0 is passed through another LSTM, INLINEFORM1 , whose last layer output forms the encoding of the entire thread  Hidden unit outputs of INLINEFORM2 represent the contexts INLINEFORM3 ; that is, snapshots of the threads after each post, as shown in Figure FIGREF1  
The INLINEFORM0 and INLINEFORM1 together constitute the hierarchical LSTM (hLSTM) model  This general hLSTM model serves as the basis for our model exploration in the rest of this section 
Contextual Attention Models
When they intervene, instructors either pay attention to a specific post or a series of posts, which trigger their reply  However, instructors rarely explicitly indicate to which post(s) their intervention is in relation to  This is the case in our corpus, party due to Coursera's user interface which only allows for single level comments (see Figure FIGREF2 )  Based solely on the binary, thread-level intervention signal, our secondary objective seeks to infer the appropriate context – represented by a sequence of posts – as the basis for the intervention 
We only consider linear contiguous series of posts starting with the thread's original post to constitute to a context; e g , INLINEFORM0   This is a reasonable as MOOC forum posts always reply to the original post or to a subsequent post, which in turn replies to the original post  This is in contrast to forums such as Reddit that have a tree or graph-like structure that require forum structure to be modelled explicitly, such as in BIBREF5  
We propose three neural attention BIBREF11 variants based on how an instructor might attend and reply to a context in a thread: the ultimate, penultimate and any post attention models  We review each of these in turn 
Ultimate Post Attention (UPA) Model  In this model we attend to the context represented by hidden state of the INLINEFORM0   We use the post prior to the instructor's reply as a query over the contexts INLINEFORM1 to compute attention weights INLINEFORM2 , which are then used to compute the attended context representation INLINEFORM3 (recall again that the intervention text itself is not available for this purpose)  This attention formulation makes an equivalence between the final INLINEFORM4 post and the prospective intervention, using Post INLINEFORM5 as the query for finding the appropriate context INLINEFORM6 , inclusive of itself INLINEFORM7   Said in another way, UPA uses the most recent content in the thread as the attentional query for context 
For example, if post INLINEFORM0 is the instructor's reply, post INLINEFORM1 will query over the contexts INLINEFORM2 and INLINEFORM3   The model schematic is shown in Figure FIGREF12  
The attended context representations are computed as: DISPLAYFORM0
The INLINEFORM0 representation is then passed through a fully connected softmax layer to yield the binary prediction 
Penultimate Post Attention (PPA) Model  While the UPA model uses the most recent text and makes the ultimate post itself available as potential context, our the ultimate post may be better modeled as having any of its prior posts as potential context  Penultimate Post Attention (PPA) variant does this  The schematic and the equations for the PPA model are obtained by summing over contexts INLINEFORM0 in Equation EQREF10 and Figure FIGREF12   While we could properly model such a context inference decision with any post INLINEFORM1 and prospective contexts INLINEFORM2 (where INLINEFORM3 is a random post), it makes sense to use the penultimate post, as we can make the most information available to the model for the context inference 
The attended context representations are computed as: DISPLAYFORM0
Any Post Attention (APA) Model  APA further relaxes both UPA and PPA, allowing APA to generalize and hypothesize that the prospective instructor intervention is based on the context that any previous post INLINEFORM0 replied to  In this model, each post INLINEFORM1 is set as a query to attend to its previous context INLINEFORM2   For example, INLINEFORM3 will attend to INLINEFORM4   Different from standard attention mechanisms, APA attention weights INLINEFORM5 are obtained by normalising interaction matrix over the different queries 
In APA, the attention context INLINEFORM0 is computed via: DISPLAYFORM0
Evaluation
The baseline and the models are evaluated on a corpus of 12 MOOC discussion forums  We train on 80% of the training data and report evaluation results on the held-out 20% of test data  We report INLINEFORM0 scores on the positive class (interventions), in line with prior work  We also argue that recall of the positive class is more important than precision, since it is costlier for instructors to miss intervening on a thread than spending irrelevant time intervening on a less critical threads due to false positives 
Model hyperpameter settings  All proposed and baseline neural models are trained using Adam optimizer with a learning rate of 0 001  We used cross-entropy as loss function  Importantly we updated the model parameters during training after each instance as in vanilla stochastic gradient descent; this setting was practical since data on most courses had only a few hundred instances enabling convergence within a reasonable training time of a few hours (see Table TABREF15 , column 2)  Models were trained for a single epoch as most of our courses with a few hundred thread converged after a single epoch  We used 300-dimensional GloVe vectors and permitted the embeddings to be updated during the model's end-to-end training  The hidden dimension size of both INLINEFORM0 and INLINEFORM1 are set to 128 for all the models 
Baselines  We compare our models against a neural baseline models, hierarchical LSTM (hLSTM), with the attention ablated but with access to the complete context, and a strong, open-sourced feature-rich baseline BIBREF7   We choose BIBREF7 over other prior works such as BIBREF0 since we do not have access to the dataset or the system used in their papers for replication  BIBREF7 is a logistic regression classifier with features inclusive of bag-of-words representation of the unigrams and thread length, normalised counts of agreements to previous posts, counts of non-lexical reference items such as URLs, and the Coursera forum type in which a thread appeared  We also report aggregated results from a hLSTM model with access only to the last post as context for comparison  Table TABREF17 compares the performance of these baselines against our proposed methods 
Results
Table TABREF15 shows performance of all our proposed models and the neural baseline over our 12 MOOC dataset  Our models of UPA, PPA individually better the baseline by 5 and 2% on INLINEFORM0 and 3 and 6% on recall respectively  UPA performs the best in terms of INLINEFORM1 on average while PPA performs the best in terms of recall on average  At the individual course level, however, the results are mixed  UPA performs the best on INLINEFORM2 on 5 out of 12 courses, PPA on 3 out 12 courses, APA 1 out of 12 courses and the baseline hLSTM on 1  PPA performs the best on recall on 7 out of the 12 courses  We also note that course level performance differences correlate with the course size and intervention ratio (hereafter, i ratio), which is the ratio of intervened to non-intervened threads  UPA performs better than PPA and APA on low intervention courses (i ratio INLINEFORM3 0 25) mainly because PPA and APA's performance drops steeply when i ratio drops (see col 2 parenthesis and INLINEFORM4 of PPA and APA)  While all the proposed models beat the baseline on every course except casebased-2  On medicalneuro-2 and compilers-4 which have the lowest i ratio among the 12 courses none of the neural models better the reported baseline BIBREF7 (course level not scores not shown in this paper)  The effect is pronounced in compilers-4 course where none of the neural models were able to predict any intervened threads  This is due to the inherent weakness of standard neural models, which are unable to learn features well enough when faced with sparse data 
The best performance of UPA indicates that the reply context of the instructor's post INLINEFORM0 correlates strongly with that of the previous post INLINEFORM1   This is not surprising since normal conversations are typically structured that way 
Discussion
In order to further understand the models' ability to infer the context and its effect on intervention prediction, we further investigate the following research questions 
RQ1  Does context inference help intervention prediction?
In order to understand if context inference is useful to intervention prediction, we ablate the attention components and experiment with the vanilla hierarchical LSTM model  Row 3 of Table TABREF17 shows the macro averaged result from this experiment  The UPA and PPA attention models better the vanilla hLSTM by 5% and 2% on average in INLINEFORM0 respectively  Recall that the vanilla hLSTM already has access to a context consisting of all posts (from INLINEFORM1 through INLINEFORM2 )  In contrast, the UPA and PPA models selectively infers a context for INLINEFORM3 and INLINEFORM4 posts, respectively, and use it to predict intervention  The improved performance of our attention models that actively select their optimal context, over a model with the complete thread as context, hLSTM, shows that the context inference improves intervention prediction over using the default full context 
RQ2  How well do the models perform across threads of different lengths? To understand the models' prediction performance across threads of different lengths, we bin threads by length and study the models' recall  We choose three courses, ml-5, rprog-3 and calc-1, from our corpus of 12 with the highest number of positive instances ( INLINEFORM0 100 threads)  We limit our analysis to these since binning renders courses with fewer positive instances sparse  Figure FIGREF18 shows performance across thread lengths from 1 through 7 posts and INLINEFORM1 posts  Clearly, the UPA model performs much better on shorter threads than on longer threads while PPA and APA works better on longer threads  Although, UPA is the best performing model in terms of overall INLINEFORM2 its performance drops steeply on threads of length INLINEFORM3   UPA's overall best performance is because most of the interventions in the corpus happen after one post  To highlight the performance of APA we show an example from smac-1 in Figure FIGREF22 with nine posts which was predicted correctly as intervened by APA but not by other models  Threads shows students confused over a missing figure in a homework  The instructor finally shows up, though late, to resolve the confusion 
RQ3  Do models trained with different context lengths perform better than when trained on a single context length?
We find that context length has a regularising effect on the model's performance at test time  This is not surprising since models trained with threads of single context length will not generalise to infer different context lengths  Row 4 of Table TABREF17 shows a steep performance drop in training by classifier with all threads truncated to a context of just one post, INLINEFORM0 , the post immediately preceding the intervened post  We also conducted an experiment with a multi-objective loss function with an additive cross-entropy term where each term computes loss from a model with context limited to a length of 3  We chose 3 since intervened threads in all the courses had a median length between 3 and 4  We achieved an INLINEFORM1 of 0 45 with a precision of 0 47 and recall of 0 43  This achieves a performance comparable to that of the BIBREF7 with context length set to only to 3  This approach of using infinitely many loss terms for each context length from 1 through the maximum thread length in a course is naive and not practical  We only use this model to show the importance of training the model with loss from threads of different lengths to prevent models overfitting to threads of specific context lengths 
Conclusion
We predict instructor intervention on student discussions by first inferring the optimal size of the context needed to decide on the intervention decision for the intervened post  We first show that a structured representation of the complete thread as the context is better than a bag-of-words, feature-rich representation  We then propose attention-based models to infer and select a context – defined as a contiguous subsequence of student posts – to improve over a model that always takes the complete thread as a context to prediction intervention  Our Any Post Attention (APA) model enables instructors to tune the model to predict intervention early or late  We posit our APA model will enable MOOC instructors employing varying pedagogical styles to use the model equally well  We introspect the attention models' performance across threads of varying lengths and show that APA predicts intervention on longer threads, which possesses more candidate contexts, better 
We note that the recall of the predictive models for longer threads (that is, threads of length greater 2) can still be improved  Models perform differently between shorter and longer length  An ensemble model or a multi-objective loss function is thus planned in our future work to better prediction on such longer threads 
Context 2: Introduction
Speech-to-Text translation (ST) is essential for a wide range of scenarios: for example in emergency calls, where agents have to respond emergent requests in a foreign language BIBREF0; or in online courses, where audiences and speakers use different languages BIBREF1  To tackle this problem, existing approaches can be categorized into cascaded method BIBREF2, BIBREF3, where a machine translation (MT) model translates outputs of an automatic speech recognition (ASR) system into target language, and end-to-end method BIBREF4, BIBREF5, where a single model learns acoustic frames to target word sequence mappings in one step towards the final objective of interest  Although the cascaded model remains the dominant approach due to its better performance, the end-to-end method becomes more and more popular because it has lower latency by avoiding inferences with two models and rectifies the error propagation in theory 
Since it is hard to obtain a large-scale ST dataset, multi-task learning BIBREF5, BIBREF6 and pre-training techniques BIBREF7 have been applied to end-to-end ST model to leverage large-scale datasets of ASR and MT  A common practice is to pre-train two encoder-decoder models for ASR and MT respectively, and then initialize the ST model with the encoder of the ASR model and the decoder of the MT model  Subsequently, the ST model is optimized with the multi-task learning by weighing the losses of ASR, MT, and ST  This approach, however, causes a huge gap between pre-training and fine-tuning, which are summarized into three folds:
Subnet Waste: The ST system just reuses the ASR encoder and the MT decoder, while discards other pre-trained subnets, such as the MT encoder  Consequently, valuable semantic information captured by the MT encoder cannot be inherited by the final ST system 
Role Mismatch: The speech encoder plays different roles in pre-training and fine-tuning  The encoder is a pure acoustic model in pre-training, while it has to extract semantic and linguistic features additionally in fine-tuning, which significantly increases the learning difficulty 
Non-pre-trained Attention Module: Previous work BIBREF6 trains attention modules for ASR, MT and ST respectively, hence, the attention module of ST does not benefit from the pre-training 
To address these issues, we propose a Tandem Connectionist Encoding Network (TCEN), which is able to reuse all subnets in pre-training, keep the roles of subnets consistent, and pre-train the attention module  Concretely, the TCEN consists of three components, a speech encoder, a text encoder, and a target text decoder  Different from the previous work that pre-trains an encoder-decoder based ASR model, we only pre-train an ASR encoder by optimizing the Connectionist Temporal Classification (CTC) BIBREF8 objective function  In this way, the additional decoder of ASR is not required while keeping the ability to read acoustic features into the source language space by the speech encoder  Besides, the text encoder and decoder can be pre-trained on a large MT dataset  After that, we employ common used multi-task learning method to jointly learn ASR, MT and ST tasks 
Compared to prior works, the encoder of TCEN is a concatenation of an ASR encoder and an MT encoder and our model does not have an ASR decoder, so the subnet waste issue is solved  Furthermore, the two encoders work at tandem, disentangling acoustic feature extraction and linguistic feature extraction, ensuring the role consistency between pre-training and fine-tuning  Moreover, we reuse the pre-trained MT attention module in ST, so we can leverage the alignment information learned in pre-training 
Since the text encoder consumes word embeddings of plausible texts in MT task but uses speech encoder outputs in ST task, another question is how one guarantees the speech encoder outputs are consistent with the word embeddings  We further modify our model to achieve semantic consistency and length consistency  Specifically, (1) the projection matrix at the CTC classification layer for ASR is shared with the word embedding matrix, ensuring that they are mapped to the same latent space, and (2) the length of the speech encoder output is proportional to the length of the input frame, so it is much longer than a natural sentence  To bridge the length gap, source sentences in MT are lengthened by adding word repetitions and blank tokens to mimic the CTC output sequences 
We conduct comprehensive experiments on the IWSLT18 speech translation benchmark BIBREF1, demonstrating the effectiveness of each component  Our model is significantly better than previous methods by 3 6 and 2 2 BLEU scores for the subword-level decoding and character-level decoding strategies, respectively 
Our contributions are three-folds: 1) we shed light on why previous ST models cannot sufficiently utilize the knowledge learned from the pre-training process; 2) we propose a new ST model, which alleviates shortcomings in existing methods; and 3) we empirically evaluate the proposed model on a large-scale public dataset 
Background ::: Problem Formulation
End-to-end speech translation aims to translate a piece of audio into a target-language translation in one step  The raw speech signals are usually converted to sequences of acoustic features, e g  Mel filterbank features  Here, we define the speech feature sequence as $\mathbf {x} = (x_1, \cdots , x_{T_x})$ The transcription and translation sequences are denoted as $\mathbf {y^{s}} = (y_1^{s}, \cdots , y_{T_s}^{s})$, and $\mathbf {y^{t}} = (y_1^{t}, \cdots , y_{T_t}^{t})$ repectively  Each symbol in $\mathbf {y^{s}}$ or $\mathbf {y^{t}}$ is an integer index of the symbol in a vocabulary $V_{src}$ or $V_{trg}$ respectively (e g  $y^s_i=k, k\in [0, |V_{src}|-1]$)  In this work, we suppose that an ASR dataset, an MT dataset, and a ST dataset are available, denoted as $\mathcal {A} = \lbrace (\mathbf {x_i}, \mathbf {y^{s}_i})\rbrace _{i=0}^I$, $\mathcal {M} =\lbrace (\mathbf {y^{s}_j}, \mathbf {y^{t}_j})\rbrace _{j=0}^J$ and $ \mathcal {S} =\lbrace (\mathbf {x_l}, \mathbf {y^{t}_l})\rbrace _{l=0}^L$ respectively  Given a new piece of audio $\mathbf {x}$, our goal is to learn an end to end model to generate a translation sentence $\mathbf {y^{t}}$ without generating an intermediate result $\mathbf {y^{s}}$ 
Background ::: Multi-Task Learning and Pre-training for ST
To leverage large scale ASR and MT data, multi-task learning and pre-training techniques are widely employed to improve the ST system  As shown in Figure FIGREF4, there are three popular multi-task strategies for ST, including 1) one-to-many setting, in which a speech encoder is shared between ASR and ST tasks; 2) many-to-one setting in which a decoder is shared between MT and ST tasks; and 3) many-to-many setting where both the encoder and decoder are shared 
A many-to-many multi-task model contains two encoders as well as two decoders  It can be jointly trained on ASR, MT, and ST tasks  As the attention module is task-specific, three attentions are defined 
Usually, the size of $\mathcal {A}$ and $\mathcal {M}$ is much larger than $\mathcal {S}$  Therefore, the common training practice is to pre-train the model on ASR and MT tasks and then fine-tune it with a multi-task learning manner  However, as aforementioned, this method suffers from subnet waste, role mismatch and non-pre-trained attention issues, which severely limits the end-to-end ST performance 
Our method
In this section, we first introduce the architecture of TCEN, which consists of two encoders connected in tandem, and one decoder with an attention module  Then we give the pre-training and fine-tuning strategy for TCEN  Finally, we propose our solutions for semantic and length inconsistency problems, which are caused by multi-task learning 
Our method ::: TCEN Architecture
Figure FIGREF5 sketches the overall architecture of TCEN, including a speech encoder $enc_s$, a text encoder $enc_t$ and a decoder $dec$ with an attention module $att$  During training, the $enc_s$ acts like an acoustic model which reads the input $\mathbf {x}$ to word or subword representations $\mathbf {h^s}$, then $enc_t$ learns high-level linguistic knowledge into hidden representations $\mathbf {h^t}$  Finally, the $dec$ defines a distribution probability over target words  The advantage of our architecture is that two encoders disentangle acoustic feature extraction and linguistic feature extraction, making sure that valuable knowledge learned from ASR and MT tasks can be effectively leveraged for ST training  Besides, every module in pre-training can be utilized in fine-tuning, alleviating the subnet waste problem 
Follow BIBREF9 inaguma2018speech, we use CNN-BiLSTM architecture to build our model  Specifically, the input features $\mathbf {x}$ are organized as a sequence of feature vectors in length $T_x$  Then, $\mathbf {x}$ is passed into a stack of two convolutional layers followed by max-pooling:
where $\mathbf {v}^{(l-1)}$ is feature maps in last layer and $\mathbf {W}^{(l)}$ is the filter  The max-pooling layers downsample the sequence in length by a total factor of four  The down-sampled feature sequence is further fed into a stack of five bidirectional $d$-dimensional LSTM layers:
where $[;]$ denotes the vector concatenation  The final output representation from the speech encoder is denoted as $\mathbf {h^s}=(h^s_1, \cdots , h^s_{\frac{T_x}{4}})$, where $h_i^s \in \mathbb {R}^d$ 
The text encoder $enc_t$ consists of two bidirectional LSTM layers  In ST task, $enc_t$ accepts speech encoder output $\mathbf {h}^s$ as input  While in MT, $enc_t$ consumes the word embedding representation $\mathbf {e^s}$ derived from $\mathbf {y^s}$, where each element $e^s_i$ is computed by choosing the $y_i^s$-th vector from the source embedding matrix $W_{E^s}$  The goal of $enc_t$ is to extract high-level linguistic features like syntactic features or semantic features from lower level subword representations $\mathbf {h}^s$ or $\mathbf {e}^s$  Since $\mathbf {h}^s$ and $\mathbf {e}^s$ belong to different latent space and have different lengths, there remain semantic and length inconsistency problems  We will provide our solutions in Section SECREF21  The output sequence of $enc_t$ is denoted as $\mathbf {h}^t$ 
The decoder is defined as two unidirectional LSTM layers with an additive attention $att$  It predicts target sequence $\mathbf {y^{t}}$ by estimating conditional probability $P(\mathbf {y^{t}}|\mathbf {x})$:
Here, $z_k$ is the the hidden state of the deocder RNN at $k$ step and $c_k$ is a time-dependent context vector computed by the attention $att$ 
Our method ::: Training Procedure
Following previous work, we split the training procedure to pre-training and fine-tuning stages  In pre-training stage, the speech encoder $enc_s$ is trained towards CTC objective using dataset $\mathcal {A}$, while the text encoder $enc_t$ and the decoder $dec$ are trained on MT dataset $\mathcal {M}$  In fine-tuning stage, we jointly train the model on ASR, MT, and ST tasks 
Our method ::: Training Procedure ::: Pre-training
To sufficiently utilize the large dataset $\mathcal {A}$ and $\mathcal {M}$, the model is pre-trained on CTC-based ASR task and MT task in the pre-training stage 
For ASR task, in order to get rid of the requirement for decoder and enable the $enc_s$ to generate subword representation, we leverage connectionist temporal classification (CTC) BIBREF8 loss to train the speech encoder 
Given an input $\mathbf {x}$, $enc_s$ emits a sequence of hidden vectors $\mathbf {h^s}$, then a softmax classification layer predicts a CTC path $\mathbf {\pi }$, where $\pi _t \in V_{src} \cup $ {`-'} is the observing label at particular RNN step $t$, and `-' is the blank token representing no observed labels:
where $W_{ctc} \in \mathbb {R}^{d \times (|V_{src}|+1)}$ is the weight matrix in the classification layer and $T$ is the total length of encoder RNN 
A legal CTC path $\mathbf {\pi }$ is a variation of the source transcription $\mathbf {y}^s$ by allowing occurrences of blank tokens and repetitions, as shown in Table TABREF14  For each transcription $\mathbf {y}$, there exist many legal CTC paths in length $T$  The CTC objective trains the model to maximize the probability of observing the golden sequence $\mathbf {y}^s$, which is calculated by summing the probabilities of all possible legal paths:
where $\Phi _T(y)$ is the set of all legal CTC paths for sequence $\mathbf {y}$ with length $T$  The loss can be easily computed using forward-backward algorithm  More details about CTC are provided in supplementary material 
For MT task, we use the cross-entropy loss as the training objective  During training, $\mathbf {y^s}$ is converted to embedding vectors $\mathbf {e^s}$ through embedding layer $W_{E^s}$, then $enc_t$ consumes $\mathbf {e^s}$ and pass the output $\mathbf {h^t}$ to decoder  The objective function is defined as:
Our method ::: Training Procedure ::: Fine-tune
In fine-tune stage, we jointly update the model on ASR, MT, and ST tasks  The training for ASR and MT follows the same process as it was in pre-training stage 
For ST task, the $enc_s$ reads the input $\mathbf {x}$ and generates $\mathbf {h^s}$, then $enc_t$ learns high-level linguistic knowledge into $\mathbf {h^t}$  Finally, the $dec$ predicts the target sentence  The ST loss function is defined as:
Following the update strategy proposed by BIBREF11 luong2015multi, we allocate a different training ratio $\alpha _i$ for each task  When switching between tasks, we select randomly a new task $i$ with probability $\frac{\alpha _i}{\sum _{j}\alpha _{j}}$ 
Our method ::: Subnet-Consistency
Our model keeps role consistency between pre-training and fine-tuning by connecting two encoders for ST task  However, this leads to some new problems: 1) The text encoder consumes $\mathbf {e^s}$ during MT training, while it accepts $\mathbf {h^s}$ during ST training  However, $\mathbf {e^s}$ and $\mathbf {h^s}$ may not follow the same distribution, resulting in the semantic inconsistency  2) Besides, the length of $\mathbf {h^s}$ is not the same order of magnitude with the length of $\mathbf {e^s}$, resulting in the length inconsistency 
In response to the above two challenges, we propose two countermeasures: 1) We share weights between CTC classification layer and source-end word embedding layer during training of ASR and MT, encouraging $\mathbf {e^s}$ and $\mathbf {h^s}$ in the same space  2)We feed the text encoder source sentences in the format of CTC path, which are generated from a seq2seq model, making it more robust toward long inputs 
Our method ::: Subnet-Consistency ::: Semantic Consistency
As shown in Figure FIGREF5, during multi-task training, two different hidden features will be fed into the text encoder $enc_t$: the embedding representation $\mathbf {e}^s$ in MT task, and the $enc_s$ output $\mathbf {h^s}$ in ST task  Without any regularization, they may belong to different latent spaces  Due to the space gap, the $enc_t$ has to compromise between two tasks, limiting its performance on individual tasks 
To bridge the space gap, our idea is to pull $\mathbf {h^s}$ into the latent space where $\mathbf {e}^s$ belong  Specifically, we share the weight $W_{ctc}$ in CTC classification layer with the source embedding weights $W_{E^s}$, which means $W_{ctc} = W_{E^s}$  In this way, when predicting the CTC path $\mathbf {\pi }$, the probability of observing the particular label $w_i \in V_{src}\cup ${`-'} at time step $t$, $p(\pi _t=w_i|\mathbf {x})$, is computed by normalizing the product of hidden vector $h_t^s$ and the $i$-th vector in $W_{E^s}$:
The loss function closes the distance between $h^s_t$ and golden embedding vector, encouraging $\mathbf {h}^s$ have the same distribution with $\mathbf {e}^s$ 
Our method ::: Subnet-Consistency ::: Length Consistency
Another existing problem is length inconsistency  The length of the sequence $\mathbf {h^s}$ is proportional to the length of the input frame $\mathbf {x}$, which is much longer than the length of $\mathbf {e^s}$  To solve this problem, we train an RNN-based seq2seq model to transform normal source sentences to noisy sentences in CTC path format, and replace standard MT with denoising MT for multi-tasking 
Specifically, we first train a CTC ASR model based on dataset $\mathcal {A} = \lbrace (\mathbf {x}_i, \mathbf {y}^s_i)\rbrace _{i=0}^{I}$, and generate a CTC-path $\mathbf {\pi }_i$ for each audio $\mathbf {x}_i$ by greedy decoding  Then we define an operation $S(\cdot )$, which converts a CTC path $\mathbf {\pi }$ to a sequence of the unique tokens $\mathbf {u}$ and a sequence of repetition times for each token $\mathbf {l}$, denoted as $S(\mathbf {\pi }) = (\mathbf {u}, \mathbf {l})$  Notably, the operation is reversible, meaning that $S^{-1} (\mathbf {u}, \mathbf {l})=\mathbf {\pi }$  We use the example $\mathbf {\pi _1}$ in Table TABREF14 and show the corresponding $\mathbf {u}$ and $\mathbf {l}$ in Table TABREF24 
Then we build a dataset $\mathcal {P} = \lbrace (\mathbf {y^s}_i, \mathbf {u}_i, \mathbf {l}_i)\rbrace _{i=0}^{I}$ by decoding all the audio pieces in $\mathcal {A}$ and transform the resulting path by the operation $S(\cdot )$  After that, we train a seq2seq model, as shown in Figure FIGREF25, which takes $ \mathbf {y^s}_i$ as input and decodes $\mathbf {u}_i, \mathbf {l}_i$ as outputs  With the seq2seq model, a noisy MT dataset $\mathcal {M}^{\prime }=\lbrace (\mathbf {\pi }_l, \mathbf {y^t}_l)\rbrace _{l=0}^{L}$ is obtained by converting every source sentence $\mathbf {y^s}_i \in \mathcal {M}$ to $\mathbf {\pi _i}$, where $\mathbf {\pi }_i = S^{-1}(\mathbf {u}_i, \mathbf {l}_i)$  We did not use the standard seq2seq model which takes $\mathbf {y^s}$ as input and generates $\mathbf {\pi }$ directly, since there are too many blank tokens `-' in $\mathbf {\pi }$ and the model tends to generate a long sequence with only blank tokens  During MT training, we randomly sample text pairs from $\mathcal {M}^{\prime }$ and $\mathcal {M}$ according to a hyper-parameter $k$  After tuning on the validation set, about $30\%$ pairs are sampled from $\mathcal {M}^{\prime }$  In this way, the $enc_t$ is more robust toward the longer inputs given by the $enc_s$ 
Experiments
We conduct experiments on the IWSLT18 speech translation task BIBREF1  Since IWSLT participators use different data pre-processing methods, we reproduce several competitive baselines based on the ESPnet BIBREF12 for a fair comparison 
Experiments ::: Dataset ::: Speech translation data:
The organizer provides a speech translation corpus extracting from the TED talk (ST-TED), which consists of raw English wave files, English transcriptions, and aligned German translations  The corpus contains 272 hours of English speech with 171k segments  We split 2k segments from the corpus as dev set and tst2010, tst2013, tst2014, tst2015 are used as test sets 
Speech recognition data: Aside from ST-TED, TED-LIUM2 corpus BIBREF13 is provided as speech recognition data, which contains 207 hours of English speech and 93k transcript sentences 
Text translation data: We use transcription and translation pairs in the ST-TED corpus and WIT3 as in-domain MT data, which contains 130k and 200k sentence pairs respectively  WMT2018 is used as out-of-domain training data which consists of 41M sentence pairs 
Data preprocessing: For speech data, the utterances are segmented into multiple frames with a 25 ms window size and a 10 ms step size  Then we extract 80-channel log-Mel filter bank and 3-dimensional pitch features using Kaldi BIBREF14, resulting in 83-dimensional input features  We normalize them by the mean and the standard deviation on the whole training set  The utterances with more than 3000 frames are discarded  The transcripts in ST-TED are in true-case with punctuation while in TED-LIUM2, transcripts are in lower-case and unpunctuated  Thus, we lowercase all the sentences and remove the punctuation to keep consistent  To increase the amount of training data, we perform speed perturbation on the raw signals with speed factors 0 9 and 1 1  For the text translation data, sentences longer than 80 words or shorter than 10 words are removed  Besides, we discard pairs whose length ratio between source and target sentence is smaller than 0 5 or larger than 2 0  Word tokenization is performed using the Moses scripts and both English and German words are in lower-case 
We use two different sets of vocabulary for our experiments  For the subword experiments, both English and German vocabularies are generated using sentencepiece BIBREF15 with a fixed size of 5k tokens  BIBREF9 inaguma2018speech show that increasing the vocabulary size is not helpful for ST task  For the character experiments, both English and German sentences are represented in the character level 
For evaluation, we segment each audio with the LIUM SpkDiarization tool BIBREF16 and then perform MWER segmentation with RWTH toolkit BIBREF17  We use lowercase BLEU as evaluation metric 
Experiments ::: Baseline Models and Implementation
We compare our method with following baselines 
Vanilla ST baseline: The vanilla ST BIBREF9 has only a speech encoder and a decoder  It is trained from scratch on the ST-TED corpus 
Pre-training baselines: We conduct three pre-training baseline experiments: 1) encoder pre-training, in which the ST encoder is initialized from an ASR model; 2) decoder pre-training, in which the ST decoder is initialized from an MT model; and 3) encoder-decoder pre-training, where both the encoder and decoder are pre-trained  The ASR model has the same architecture with vanilla ST model, trained on the mixture of ST-TED and TED-LIUM2 corpus  The MT model has a text encoder and decoder with the same architecture of which in TCEN  It is first trained on WMT data (out-of-domain) and then fine-tuned on in-domain data 
Multi-task baselines: We also conduct three multi-task baseline experiments including one-to-many setting, many-to-one setting, and many-to-many setting  In the first two settings, we train the model with $\alpha _{st}=0 75$ while $\alpha _{asr}=0 25$ or $\alpha _{mt}=0 25$  For many-to-many setting, we use $\alpha _{st}=0 6, \alpha _{asr}=0 2$ and $\alpha _{mt}=0 2$   For MT task, we use only in-domain data 
Many-to-many+pre-training: We train a many-to-many multi-task model where the encoders and decoders are derived from pre-trained ASR and MT models  Triangle+pre-train: BIBREF18 DBLP:conf/naacl/AnastasopoulosC18 proposed a triangle multi-task strategy for speech translation  Their model solves the subnet waste issue by concatenating an ST decoder to an ASR encoder-decoder model  Notably, their ST decoder can consume representations from the speech encoder as well as the ASR decoder  For a fair comparison, the speech encoder and the ASR decoder are initialized from the pre-trained ASR model  The Triangle model is fine-tuned under their multi-task manner 
All our baselines as well as TCEN are implemented based on ESPnet BIBREF12, the RNN size is set as $d=1024$ for all models  We use a dropout of 0 3 for embeddings and encoders, and train using Adadelta with initial learning rate of 1 0 for a maximum of 10 epochs 
For training of TCEN, we set $\alpha _{asr}=0 2$ and $\alpha _{mt}=0 8$ in the pre-training stage, since the MT dataset is much larger than ASR dataset  For fine-tune, we use $\alpha _{st}=0 6, \alpha _{asr}=0 2$ and $\alpha _{mt}=0 2$, same as the `many-to-many' baseline 
For testing, we select the model with the best accuracy on speech translation task on dev set  At inference time, we use a beam size of 10, and the beam scores include length normalization with a weight of 0 2 
Experiments ::: Experimental Results
Table TABREF29 shows the results on four test sets as well as the average performance  Our method significantly outperforms the strong `many-to-many+pretrain' baseline by 3 6 and 2 2 BLEU scores respectively, indicating the proposed method is very effective that substantially improves the translation quality  Besides, both pre-training and multi-task learning can improve translation quality, and the pre-training settings (2nd-4th rows) are more effective compared to multi-task settings (5th-8th rows)  We observe a performance degradation in the `triangle+pretrain' baseline  Compared to our method, where the decoder receives higher-level syntactic and semantic linguistic knowledge extracted from text encoder, their ASR decoder can only provide lower word-level linguistic information  Besides, since their model lacks text encoder and the architecture of ST decoder is different from MT decoder, their model cannot utilize the large-scale MT data in all the training stages  Interestingly, we find that the char-level models outperform the subword-level models in all settings, especially in vanilla baseline  A similar phenomenon is observed by BIBREF6 berard2018end  A possible explanation is that learning the alignments between speech frames and subword units in another language is notoriously difficult  Our method can bring more gains in the subword setting since our model is good at learning the text-to-text alignment and the subword-level alignment is more helpful to the translation quality 
Experiments ::: Discussion ::: Ablation Study
To better understand the contribution of each component, we perform an ablation study on subword-level experiments  The results are shown in Table TABREF37  In `-MT noise' setting, we do not add noise to source sentences for MT  In `-weight sharing' setting, we use different parameters in CTC classification layer and source embedding layer  These two experiments prove that both weight sharing and using noisy MT input benefit to the final translation quality  Performance degrades more in `-weight sharing', indicating the semantic consistency contributes more to our model  In the `-pretrain' experiment, we remove the pre-training stage and directly update the model on three tasks, leading to a dramatic decrease on BLEU score, indicating the pre-training is an indispensable step for end-to-end ST 
Experiments ::: Discussion ::: Learning Curve
It is interesting to investigate why our method is superior to baselines  We find that TCEN achieves a higher final result owing to a better start-point in fine-tuning  Figure FIGREF39 provides learning curves of subword accuracy on validation set  The x-axis denotes the fine-tuning training steps  The vanilla model starts at a low accuracy, because its networks are not pre-trained on the ASR and MT data  The trends of our model and `many-to-many+pretrain' are similar, but our model outperforms it about five points in the whole fine-tuning process  It indicates that the gain comes from bridging the gap between pre-training and fine-tuning rather than a better fine-tuning process 
Experiments ::: Discussion ::: Compared with a Cascaded System
Table TABREF29 compares our model with end-to-end baselines  Here, we compare our model with cascaded systems  We build a cascaded system by combining the ASR model and MT model used in pre-training baseline  Word error rate (WER) of the ASR system and BLEU score of the MT system are reported in the supplementary material  In addition to a simple combination of the ASR and MT systems, we also re-segment the ASR outputs before feeding to the MT system, denoted as cascaded+re-seg  Specifically, we train a seq2seq model BIBREF19 on the MT dataset, where the source side is a no punctuation sentence and the target side is a natural sentence  After that, we use the seq2seq model to add sentence boundaries and punctuation on ASR outputs  Experimental results are shown in Table TABREF41  Our end-to-end model outperforms the simple cascaded model over 2 BLEU scores, and it achieves a comparable performance with the cascaded model combining with a sentence re-segment model 
Related Work
Early works conduct speech translation in a pipeline manner BIBREF2, BIBREF20, where the ASR output lattices are fed into an MT system to generate target sentences  HMM BIBREF21, DenseNet BIBREF22, TDNN BIBREF23 are commonly used ASR systems, while RNN with attention BIBREF19 and Transformer BIBREF10 are top choices for MT  To enhance the robustness of the NMT model towards ASR errors, BIBREF24 DBLP:conf/eacl/TsvetkovMD14 and BIBREF25 DBLP:conf/asru/ChenHHL17 propose to simulate the noise in training and inference 
To avoid error propagation and high latency issues, recent works propose translating the acoustic speech into text in target language without yielding the source transcription BIBREF4  Since ST data is scarce, pre-training BIBREF7, multi-task learning BIBREF4, BIBREF6, curriculum learning BIBREF26, attention-passing BIBREF27, and knowledge distillation BIBREF28, BIBREF29 strategies have been explored to utilize ASR data and MT data  Specifically, BIBREF5 DBLP:conf/interspeech/WeissCJWC17 show improvements of performance by training the ST model jointly with the ASR and the MT model  BIBREF6 berard2018end observe faster convergence and better results due to pre-training and multi-task learning on a larger dataset  BIBREF7 DBLP:conf/naacl/BansalKLLG19 show that pre-training a speech encoder on one language can improve ST quality on a different source language  All of them follow the traditional multi-task training strategies  BIBREF26 DBLP:journals/corr/abs-1802-06003 propose to use curriculum learning to improve ST performance on syntactically distant language pairs  To effectively leverage transcriptions in ST data, BIBREF18 DBLP:conf/naacl/AnastasopoulosC18 augment the multi-task model where the target decoder receives information from the source decoder and they show improvements on low-resource speech translation  Their model just consumes ASR and ST data, in contrast, our work sufficiently utilizes the large-scale MT data to capture the rich semantic knowledge  BIBREF30 DBLP:conf/icassp/JiaJMWCCALW19 use pre-trained MT and text-to-speech (TTS) synthesis models to convert weakly supervised data into ST pairs and demonstrate that an end-to-end MT model can be trained using only synthesised data 
Conclusion
This paper has investigated the end-to-end method for ST  It has discussed why there is a huge gap between pre-training and fine-tuning in previous methods  To alleviate these issues, we have proposed a method, which is capable of reusing every sub-net and keeping the role of sub-net consistent between pre-training and fine-tuning  Empirical studies have demonstrated that our model significantly outperforms baselines 

ground_truth: hLSTM

hLSTM

prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. What are the baselines?
Context 3: Introduction
End-to-end speech-to-text translation (ST) has attracted much attention recently BIBREF2, BIBREF3, BIBREF4, BIBREF5, BIBREF6 given its simplicity against cascading automatic speech recognition (ASR) and machine translation (MT) systems  The lack of labeled data, however, has become a major blocker for bridging the performance gaps between end-to-end models and cascading systems  Several corpora have been developed in recent years  post2013improved introduced a 38-hour Spanish-English ST corpus by augmenting the transcripts of the Fisher and Callhome corpora with English translations  di-gangi-etal-2019-must created the largest ST corpus to date from TED talks but the language pairs involved are out of English only  beilharz2019librivoxdeen created a 110-hour German-English ST corpus from LibriVox audiobooks  godard-etal-2018-low created a Moboshi-French ST corpus as part of a rare language documentation effort  woldeyohannis provided an Amharic-English ST corpus in the tourism domain  boito2019mass created a multilingual ST corpus involving 8 languages from a multilingual speech corpus based on Bible readings BIBREF7  Previous work either involves language pairs out of English, very specific domains, very low resource languages or a limited set of language pairs  This limits the scope of study, including the latest explorations on end-to-end multilingual ST BIBREF8, BIBREF9  Our work is mostly similar and concurrent to iranzosnchez2019europarlst who created a multilingual ST corpus from the European Parliament proceedings  The corpus we introduce has larger speech durations and more translation tokens  It is diversified with multiple speakers per transcript/translation  Finally, we provide additional out-of-domain test sets 
In this paper, we introduce CoVoST, a multilingual ST corpus based on Common Voice BIBREF10 for 11 languages into English, diversified with over 11,000 speakers and over 60 accents  It includes a total 708 hours of French (Fr), German (De), Dutch (Nl), Russian (Ru), Spanish (Es), Italian (It), Turkish (Tr), Persian (Fa), Swedish (Sv), Mongolian (Mn) and Chinese (Zh) speeches, with French and German ones having the largest durations among existing public corpora  We also collect an additional evaluation corpus from Tatoeba for French, German, Dutch, Russian and Spanish, resulting in a total of 9 3 hours of speech  Both corpora are created at the sentence level and do not require additional alignments or segmentation  Using the official Common Voice train-development-test split, we also provide baseline models, including, to our knowledge, the first end-to-end many-to-one multilingual ST models  CoVoST is released under CC0 license and free to use  The Tatoeba evaluation samples are also available under friendly CC licenses  All the data can be acquired at https://github com/facebookresearch/covost 
Data Collection and Processing ::: Common Voice (CoVo)
Common Voice BIBREF10 is a crowdsourcing speech recognition corpus with an open CC0 license  Contributors record voice clips by reading from a bank of donated sentences  Each voice clip was validated by at least two other users  Most of the sentences are covered by multiple speakers, with potentially different genders, age groups or accents 
Raw CoVo data contains samples that passed validation as well as those that did not  To build CoVoST, we only use the former one and reuse the official train-development-test partition of the validated data  As of January 2020, the latest CoVo 2019-06-12 release includes 29 languages  CoVoST is currently built on that release and covers the following 11 languages: French, German, Dutch, Russian, Spanish, Italian, Turkish, Persian, Swedish, Mongolian and Chinese 
Validated transcripts were sent to professional translators  Note that the translators had access to the transcripts but not the corresponding voice clips since clips would not carry additional information  Since transcripts were duplicated due to multiple speakers, we deduplicated the transcripts before sending them to translators  As a result, different voice clips of the same content (transcript) will have identical translations in CoVoST for train, development and test splits 
In order to control the quality of the professional translations, we applied various sanity checks to the translations BIBREF11  1) For German-English, French-English and Russian-English translations, we computed sentence-level BLEU BIBREF12 with the NLTK BIBREF13 implementation between the human translations and the automatic translations produced by a state-of-the-art system BIBREF14 (the French-English system was a Transformer big BIBREF15 separately trained on WMT14)  We applied this method to these three language pairs only as we are confident about the quality of the corresponding systems  Translations with a score that was too low were manually inspected and sent back to the translators when needed  2) We manually inspected examples where the source transcript was identical to the translation  3) We measured the perplexity of the translations using a language model trained on a large amount of clean monolingual data BIBREF14  We manually inspected examples where the translation had a high perplexity and sent them back to translators accordingly  4) We computed the ratio of English characters in the translations  We manually inspected examples with a low ratio and sent them back to translators accordingly  5) Finally, we used VizSeq BIBREF16 to calculate similarity scores between transcripts and translations based on LASER cross-lingual sentence embeddings BIBREF17  Samples with low scores were manually inspected and sent back for translation when needed 
We also sanity check the overlaps of train, development and test sets in terms of transcripts and voice clips (via MD5 file hashing), and confirm they are totally disjoint 
Data Collection and Processing ::: Tatoeba (TT)
Tatoeba (TT) is a community built language learning corpus having sentences aligned across multiple languages with the corresponding speech partially available  Its sentences are on average shorter than those in CoVoST (see also Table TABREF2) given the original purpose of language learning  Sentences in TT are licensed under CC BY 2 0 FR and part of the speeches are available under various CC licenses 
We construct an evaluation set from TT (for French, German, Dutch, Russian and Spanish) as a complement to CoVoST development and test sets  We collect (speech, transcript, English translation) triplets for the 5 languages and do not include those whose speech has a broken URL or is not CC licensed  We further filter these samples by sentence lengths (minimum 4 words including punctuations) to reduce the portion of short sentences  This makes the resulting evaluation set closer to real-world scenarios and more challenging 
We run the same quality checks for TT as for CoVoST but we do not find poor quality translations according to our criteria  Finally, we report the overlap between CoVo transcripts and TT sentences in Table TABREF5  We found a minimal overlap, which makes the TT evaluation set a suitable additional test set when training on CoVoST 
Data Analysis ::: Basic Statistics
Basic statistics for CoVoST and TT are listed in Table TABREF2 including (unique) sentence counts, speech durations, speaker demographics (partially available) as well as vocabulary and token statistics (based on Moses-tokenized sentences by sacreMoses) on both transcripts and translations  We see that CoVoST has over 327 hours of German speeches and over 171 hours of French speeches, which, to our knowledge, corresponds to the largest corpus among existing public ST corpora (the second largest is 110 hours BIBREF18 for German and 38 hours BIBREF19 for French)  Moreover, CoVoST has a total of 18 hours of Dutch speeches, to our knowledge, contributing the first public Dutch ST resource  CoVoST also has around 27-hour Russian speeches, 37-hour Italian speeches and 67-hour Persian speeches, which is 1 8 times, 2 5 times and 13 3 times of the previous largest public one BIBREF7  Most of the sentences (transcripts) in CoVoST are covered by multiple speakers with potentially different accents, resulting in a rich diversity in the speeches  For example, there are over 1,000 speakers and over 10 accents in the French and German development / test sets  This enables good coverage of speech variations in both model training and evaluation 
Data Analysis ::: Speaker Diversity
As we can see from Table TABREF2, CoVoST is diversified with a rich set of speakers and accents  We further inspect the speaker demographics in terms of sample distributions with respect to speaker counts, accent counts and age groups, which is shown in Figure FIGREF6, FIGREF7 and FIGREF8  We observe that for 8 of the 11 languages, at least 60% of the sentences (transcripts) are covered by multiple speakers  Over 80% of the French sentences have at least 3 speakers  And for German sentences, even over 90% of them have at least 5 speakers  Similarly, we see that a large portion of sentences are spoken in multiple accents for French, German, Dutch and Spanish  Speakers of each language also spread widely across different age groups (below 20, 20s, 30s, 40s, 50s, 60s and 70s) 
Baseline Results
We provide baselines using the official train-development-test split on the following tasks: automatic speech recognition (ASR), machine translation (MT) and speech translation (ST) 
Baseline Results ::: Experimental Settings ::: Data Preprocessing
We convert raw MP3 audio files from CoVo and TT into mono-channel waveforms, and downsample them to 16,000 Hz  For transcripts and translations, we normalize the punctuation, we tokenize the text with sacreMoses and lowercase it  For transcripts, we further remove all punctuation markers except for apostrophes  We use character vocabularies on all the tasks, with 100% coverage of all the characters  Preliminary experimentation showed that character vocabularies provided more stable training than BPE  For MT, the vocabulary is created jointly on both transcripts and translations  We extract 80-channel log-mel filterbank features, computed with a 25ms window size and 10ms window shift using torchaudio  The features are normalized to 0 mean and 1 0 standard deviation  We remove samples having more than 3,000 frames or more than 256 characters for GPU memory efficiency (less than 25 samples are removed for all languages) 
Baseline Results ::: Experimental Settings ::: Model Training
Our ASR and ST models follow the architecture in berard2018end, but have 3 decoder layers like that in pino2019harnessing  For MT, we use a Transformer base architecture BIBREF15, but with 3 encoder layers, 3 decoder layers and 0 3 dropout  We use a batch size of 10,000 frames for ASR and ST, and a batch size of 4,000 tokens for MT  We train all models using Fairseq BIBREF20 for up to 200,000 updates  We use SpecAugment BIBREF21 for ASR and ST to alleviate overfitting 
Baseline Results ::: Experimental Settings ::: Inference and Evaluation
We use a beam size of 5 for all models  We use the best checkpoint by validation loss for MT, and average the last 5 checkpoints for ASR and ST  For MT and ST, we report case-insensitive tokenized BLEU BIBREF22 using sacreBLEU BIBREF23  For ASR, we report word error rate (WER) and character error rate (CER) using VizSeq 
Baseline Results ::: Automatic Speech Recognition (ASR)
For simplicity, we use the same model architecture for ASR and ST, although we do not leverage ASR models to pretrain ST model encoders later  Table TABREF18 shows the word error rate (WER) and character error rate (CER) for ASR models  We see that French and German perform the best given they are the two highest resource languages in CoVoST  The other languages are relatively low resource (especially Turkish and Swedish) and the ASR models are having difficulties to learn from this data 
Baseline Results ::: Machine Translation (MT)
MT models take transcripts (without punctuation) as inputs and outputs translations (with punctuation)  For simplicity, we do not change the text preprocessing methods for MT to correct this mismatch  Moreover, this mismatch also exists in cascading ST systems, where MT model inputs are the outputs of an ASR model  Table TABREF20 shows the BLEU scores of MT models  We notice that the results are consistent with what we see from ASR models  For example thanks to abundant training data, French has a decent BLEU score of 29 8/25 4  German doesn't perform well, because of less richness of content (transcripts)  The other languages are low resource in CoVoST and it is difficult to train decent models without additional data or pre-training techniques 
Baseline Results ::: Speech Translation (ST)
CoVoST is a many-to-one multilingual ST corpus  While end-to-end one-to-many and many-to-many multilingual ST models have been explored very recently BIBREF8, BIBREF9, many-to-one multilingual models, to our knowledge, have not  We hence use CoVoST to examine this setting  Table TABREF22 and TABREF23 show the BLEU scores for both bilingual and multilingual end-to-end ST models trained on CoVoST  We observe that combining speeches from multiple languages is consistently bringing gains to low-resource languages (all besides French and German)  This includes combinations of distant languages, such as Ru+Fr, Tr+Fr and Zh+Fr  Moreover, some combinations do bring gains to high-resource language (French) as well: Es+Fr, Tr+Fr and Mn+Fr  We simply provide the most basic many-to-one multilingual baselines here, and leave the full exploration of the best configurations to future work  Finally, we note that for some language pairs, absolute BLEU numbers are relatively low as we restrict model training to the supervised data  We encourage the community to improve upon those baselines, for example by leveraging semi-supervised training 
Baseline Results ::: Multi-Speaker Evaluation
In CoVoST, large portion of transcripts are covered by multiple speakers with different genders, accents and age groups  Besides the standard corpus-level BLEU scores, we also want to evaluate model output variance on the same content (transcript) but different speakers  We hence propose to group samples (and their sentence BLEU scores) by transcript, and then calculate average per-group mean and average coefficient of variation defined as follows:
and
where $G$ is the set of sentence BLEU scores grouped by transcript and $G^{\prime } = \lbrace g | g\in G, |g|>1, \textrm {Mean}(g) > 0 \rbrace $ 
$\textrm {BLEU}_{MS}$ provides a normalized quality score as oppose to corpus-level BLEU or unnormalized average of sentence BLEU  And $\textrm {CoefVar}_{MS}$ is a standardized measure of model stability against different speakers (the lower the better)  Table TABREF24 shows the $\textrm {BLEU}_{MS}$ and $\textrm {CoefVar}_{MS}$ of our ST models on CoVoST test set  We see that German and Persian have the worst $\textrm {CoefVar}_{MS}$ (least stable) given their rich speaker diversity in the test set and relatively small train set (see also Figure FIGREF6 and Table TABREF2)  Dutch also has poor $\textrm {CoefVar}_{MS}$ because of the lack of training data  Multilingual models are consistantly more stable on low-resource languages  Ru+Fr, Tr+Fr, Fa+Fr and Zh+Fr even have better $\textrm {CoefVar}_{MS}$ than all individual languages 
Conclusion
We introduce a multilingual speech-to-text translation corpus, CoVoST, for 11 languages into English, diversified with over 11,000 speakers and over 60 accents  We also provide baseline results, including, to our knowledge, the first end-to-end many-to-one multilingual model for spoken language translation  CoVoST is free to use with a CC0 license, and the additional Tatoeba evaluation samples are also CC-licensed 

ground_truth: Vanilla ST baseline, encoder pre-training, in which the ST encoder is initialized from an ASR model, decoder pre-training, in which the ST decoder is initialized from an MT model, encoder-decoder pre-training, where both the encoder and decoder are pre-trained, many-to-many multi-task model where the encoders and decoders are derived from pre-trained ASR and MT models, Triangle+pre-train: BIBREF18 DBLP:conf/naacl/AnastasopoulosC18 proposed a triangle multi-task strategy for speech translation

Vanilla ST baseline, Pre-training baselines, Multi-task baselines, Many-to-many+pre-training, Triangle+pre-train

Vanilla ST baseline: The vanilla ST BIBREF9 has only a speech encoder and a decoder. It is trained from scratch on the ST-TED corpus.

Pre-training baselines: We conduct three pre-training baseline experiments: 1) encoder pre-training, in which the ST encoder is initialized from an ASR model; 2) decoder pre-training, in which the ST decoder is initialized from an MT model; and 3) encoder-decoder pre-training, where both the encoder and decoder are pre-trained. The ASR model has the same architecture with vanilla ST model, trained on the mixture of ST-TED and TED-LIUM2 corpus. The MT model has a text encoder and decoder with the same architecture of which in TCEN. It is first trained on WMT data (out-of-domain) and then fine-tuned on in-domain data.

Multi-task baselines: We also conduct three multi-task baseline experiments including one-to-many setting, many-to-one setting, and many-to-many setting. In the first two settings, we train the model with $\alpha _{st}=0.75$ while $\alpha _{asr}=0.25$ or $\alpha _{mt}=0.25$. For many-to-many setting, we use $\alpha _{st}=0.6, \alpha _{asr}=0.2$ and $\alpha _{mt}=0.2$.. For MT task, we use only in-domain data.

Many-to-many+pre-training: We train a many-to-many multi-task model where the encoders and decoders are derived from pre-trained ASR and MT models. 

prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. Is Arabic one of the 11 languages in CoVost?
Context 4:  To enforce structural correlations between labels in sequence decoding, INLINEFORM1 is then passed to a conditional random field (CRF) to produce a label for each token maximizing the following objective  y* = y p(y|h; WCRF)
p(y|h; WCRF) = t t (yt-1,yt;h) y' t t (y't-1,y't;h)
where INLINEFORM0 is a potential function, INLINEFORM1 is a set of parameters that defines the potential functions and weight vectors for label pairs ( INLINEFORM2 )  Bias terms are omitted for brevity of formulation 
The model can be trained via log-likelihood maximization for the training set INLINEFORM0 :
L(WCRF) = i p(y|h; W)
Modality Attention
The modality attention module learns a unified representation space for multiple available modalities (words, characters, images, etc ), and produces a single vector representation with aggregated knowledge among multiple modalities, based on their weighted importance  We motivate this module from the following observations 
A majority of the previous literature combine the word and character-level contexts by simply concatenating the word and character embeddings at each decoding step, e g  INLINEFORM0 in Eq  SECREF4   However, this naive concatenation of two modalities (word and characters) results in inaccurate decoding, specifically for unknown word token embeddings (an all-zero vector INLINEFORM1 or a random vector INLINEFORM2 is assigned for any unknown token INLINEFORM3 , thus INLINEFORM4 or INLINEFORM5 )  While this concatenation approach does not cause significant errors for well-formatted text, we observe that it induces performance degradation for our social media post datasets which contain a significant number of missing tokens 
Similarly, naive merging of textual and visual information ( INLINEFORM0 ) yields suboptimal results as each modality is treated equally informative, whereas in our datasets some of the images may contain irrelevant contexts to textual modalities  Hence, ideally there needs a mechanism in which the model can effectively turn the switch on and off the modalities adaptive to each sample 
To this end, we propose a general modality attention module, which adaptively attenuates or emphasizes each modality as a whole at each decoding step INLINEFORM0 , and produces a soft-attended context vector INLINEFORM1 as an input token for the entity LSTM  [at(w),at(c),at(v)] = (Wm[xt(w); xt(c); xt(v)] + bm )
t(m) = (at(m))m'{w,c,v}(at(m')) m {w,c,v}
xt = m{w,c,v} t(m)xt(m)
where INLINEFORM0 is an attention vector at each decoding step INLINEFORM1 , and INLINEFORM2 is a final context vector at INLINEFORM3 that maximizes information gain for INLINEFORM4   Note that the optimization of the objective function (Eq  SECREF4 ) with modality attention (Eq  SECREF5 ) requires each modality to have the same dimension ( INLINEFORM5 ), and that the transformation via INLINEFORM6 essentially enforces each modality to be mapped into the same unified subspace, where the weighted average of which encodes discrimitive features for recognition of named entities 
When visual context is not provided with each token (as in the traditional NER task), we can define the modality attention for word and character embeddings only in a similar way: [at(w),at(c)] = (Wm[xt(w); xt(c)] + bm )
t(m) = (at(m))m'{w,c}(at(m')) m {w,c}
xt = m{w,c} t(m)xt(m)
Note that while we apply this modality attention module to the Bi-LSTM+CRF architecture (Section SECREF4 ) for its empirical superiority, the module itself is flexible and thus can work with other NER architectures or for other multimodal applications 
SnapCaptions Dataset
The SnapCaptions dataset is composed of 10K user-generated image (snap) and textual caption pairs where named entities in captions are manually labeled by expert human annotators (entity types: PER, LOC, ORG, MISC)  These captions are collected exclusively from snaps submitted to public and crowd-sourced stories (aka Snapchat Live Stories or Our Stories)  Examples of such public crowd-sourced stories are “New York Story” or “Thanksgiving Story”, which comprise snaps that are aggregated for various public events, venues, etc  All snaps were posted between year 2016 and 2017, and do not contain raw images or other associated information (only textual captions and obfuscated visual descriptor features extracted from the pre-trained InceptionNet are available)  We split the dataset into train (70%), validation (15%), and test sets (15%)  The captions data have average length of 30 7 characters (5 81 words) with vocabulary size 15,733, where 6,612 are considered unknown tokens from Stanford GloVE embeddings BIBREF22   Named entities annotated in the SnapCaptions dataset include many of new and emerging entities, and they are found in various surface forms (various nicknames, typos, etc ) To the best of our knowledge, SnapCaptions is the only dataset that contains natural image-caption pairs with expert-annotated named entities 
Baselines
Task: given a caption and a paired image (if used), the goal is to label every token in a caption in BIO scheme (B: beginning, I: inside, O: outside) BIBREF27   We report the performance of the following state-of-the-art NER models as baselines, as well as several configurations of our proposed approach to examine contributions of each component (W: word, C: char, V: visual) 
Bi-LSTM/CRF (W only): only takes word token embeddings (Stanford GloVE) as input  The rest of the architecture is kept the same 
Bi-LSTM/CRF + Bi-CharLSTM (C only): only takes a character sequence of each word token as input  (No word embeddings)
Bi-LSTM/CRF + Bi-CharLSTM (W+C) BIBREF0 : takes as input both word embeddings and character embeddings extracted from a Bi-CharLSTM  Entity LSTM takes concatenated vectors of word and character embeddings as input tokens 
Bi-LSTM/CRF + CharCNN (W+C) BIBREF1 : uses character embeddings extracted from a CNN instead 
Bi-LSTM/CRF + CharCNN (W+C) + Multi-task BIBREF8 : trains the model to perform both recognition (into multiple entity types) as well as segmentation (binary) tasks 
(proposed) Bi-LSTM/CRF + Bi-CharLSTM with modality attention (W+C): uses the modality attention to merge word and character embeddings 
(proposed) Bi-LSTM/CRF + Bi-CharLSTM + Inception (W+C+V): takes as input visual contexts extracted from InceptionNet as well, concatenated with word and char vectors 
(proposed) Bi-LSTM/CRF + Bi-CharLSTM + Inception with modality attention (W+C+V): uses the modality attention to merge word, character, and visual embeddings as input to entity LSTM 
Results: SnapCaptions Dataset
Table TABREF6 shows the NER performance on the Snap Captions dataset  We report both entity types recognition (PER, LOC, ORG, MISC) and named entity segmentation (named entity or not) results 
Parameters: We tune the parameters of each model with the following search space (bold indicate the choice for our final model): character embeddings dimension: {25, 50, 100, 150, 200, 300}, word embeddings size: {25, 50, 100, 150, 200, 300}, LSTM hidden states: {25, 50, 100, 150, 200, 300}, and INLINEFORM0 dimension: {25, 50, 100, 150, 200, 300}  We optimize the parameters with Adagrad BIBREF28 with batch size 10, learning rate 0 02, epsilon INLINEFORM1 , and decay 0 0 
Main Results: When visual context is available (W+C+V), we see that the model performance greatly improves over the textual models (W+C), showing that visual contexts are complimentary to textual information in named entity recognition tasks  In addition, it can be seen that the modality attention module further improves the entity type recognition performance for (W+C+V)  This result indicates that the modality attention is able to focus on the most effective modality (visual, words, or characters) adaptive to each sample to maximize information gain  Note that our text-only model (W+C) with the modality attention module also significantly outperform the state-of-the-art baselines BIBREF8 , BIBREF1 , BIBREF0 that use the same textual modalities (W+C), showing the effectiveness of the modality attention module for textual models as well 
Error Analysis: Table TABREF17 shows example cases where incorporation of visual contexts affects prediction of named entities  For example, the token `curry' in the caption “The curry's " is polysemous and may refer to either a type of food or a famous basketball player `Stephen Curry', and the surrounding textual contexts do not provide enough information to disambiguate it  On the other hand, visual contexts (visual tags: `parade', `urban area',    ) provide similarities to the token's distributional semantics from other training examples (snaps from “NBA Championship Parade Story"), and thus the model successfully predicts the token as a named entity  Similarly, while the text-only model erroneously predicts `Apple' in the caption “Grandma w dat lit Apple Crisp" as an organization (Apple Inc ), the visual contexts (describing objects related to food) help disambiguate the token, making the model predict it correctly as a non-named entity (a fruit)  Trending entities (musicians or DJs such as `CID', `Duke Dumont', `Marshmello', etc ) are also recognized correctly with strengthened contexts from visual information (describing concert scenes) despite lack of surrounding textual contexts  A few cases where visual contexts harmed the performance mostly include visual tags that are unrelated to a token or its surrounding textual contexts 
Visualization of Modality Attention: Figure FIGREF19 visualizes the modality attention module at each decoding step (each column), where amplified modality is represented with darker color, and attenuated modality is represented with lighter color 
For the image-aided model (W+C+V; upper row in Figure FIGREF19 ), we confirm that the modality attention successfully attenuates irrelevant signals (selfies, etc ) and amplifies relevant modality-based contexts in prediction of a given token  In the example of “disney word essential = coffee" with visual tags selfie, phone, person, the modality attention successfully attenuates distracting visual signals and focuses on textual modalities, consequently making correct predictions  The named entities in the examples of “Beautiful night atop The Space Needle" and “Splash Mountain" are challenging to predict because they are composed of common nouns (space, needle, splash, mountain), and thus they often need additional contexts to correctly predict  In the training data, visual contexts make stronger indicators for these named entities (space needle, splash mountain), and the modality attention module successfully attends more to stronger signals 
For text-only model (W+C), we observe that performance gains mostly come from the modality attention module better handling tokens unseen during training or unknown tokens from the pre-trained word embeddings matrix  For example, while WaRriOoOrs and Kooler Matic are missing tokens in the word embeddings matrix, it successfully amplifies character-based contexts (capitalized first letters, similarity to known entities `Golden State Warriors') and suppresses word-based contexts (word embeddings for unknown tokens `WaRriOoOrs'), leading to correct predictions  This result is significant because it shows performance of the model, with an almost identical architecture, can still improve without having to scale the word embeddings matrix indefinitely 
Figure FIGREF19 (b) shows the cases where the modality attention led to incorrect predictions  For example, the model predicts missing tokens HUUUGE and Shampooer incorrectly as named entities by amplifying misleading character-based contexts (capitalized first letters) or visual contexts (concert scenes, associated contexts of which often include named entities in the training dataset) 
Sensitivity to Word Embeddings Vocabulary Size: In order to isolate the effectiveness of the modality attention module on textual models in handling missing tokens, we report the performance with varying word embeddings vocabulary sizes in Table TABREF20   By increasing the number of missing tokens artificially by randomly removing words from the word embeddings matrix (original vocab size: 400K), we observe that while the overall performance degrades, the modality attention module is able to suppress the peformance degradation  Note also that the performance gap generally gets bigger as we decrease the vocabulary size of the word embeddings matrix  This result is significant in that the modality attention is able to improve the model more robust to missing tokens without having to train an indefinitely large word embeddings matrix for arbitrarily noisy social media text datasets 
Conclusions
We proposed a new multimodal NER (MNER: image + text) task on short social media posts  We demonstrated for the first time an effective MNER system, where visual information is combined with textual information to outperform traditional text-based NER baselines  Our work can be applied to myriads of social media posts or other articles across multiple platforms which often include both text and accompanying images  In addition, we proposed the modality attention module, a new neural mechanism which learns optimal integration of different modes of correlated information  In essence, the modality attention learns to attenuate irrelevant or uninformative modal information while amplifying the primary modality to extract better overall representations  We showed that the modality attention based model outperforms other state-of-the-art baselines when text was the only modality available, by better combining word and character level information 

ground_truth: No

No