prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. What metrics are used for evaluation?
Context 1: Introduction
In the kitchen, we increasingly rely on instructions from cooking websites: recipes  A cook with a predilection for Asian cuisine may wish to prepare chicken curry, but may not know all necessary ingredients apart from a few basics  These users with limited knowledge cannot rely on existing recipe generation approaches that focus on creating coherent recipes given all ingredients and a recipe name BIBREF0  Such models do not address issues of personal preference (e g  culinary tastes, garnish choices) and incomplete recipe details  We propose to approach both problems via personalized generation of plausible, user-specific recipes using user preferences extracted from previously consumed recipes 
Our work combines two important tasks from natural language processing and recommender systems: data-to-text generation BIBREF1 and personalized recommendation BIBREF2  Our model takes as user input the name of a specific dish, a few key ingredients, and a calorie level  We pass these loose input specifications to an encoder-decoder framework and attend on user profiles—learned latent representations of recipes previously consumed by a user—to generate a recipe personalized to the user's tastes  We fuse these `user-aware' representations with decoder output in an attention fusion layer to jointly determine text generation  Quantitative (perplexity, user-ranking) and qualitative analysis on user-aware model outputs confirm that personalization indeed assists in generating plausible recipes from incomplete ingredients 
While personalized text generation has seen success in conveying user writing styles in the product review BIBREF3, BIBREF4 and dialogue BIBREF5 spaces, we are the first to consider it for the problem of recipe generation, where output quality is heavily dependent on the content of the instructions—such as ingredients and cooking techniques 
To summarize, our main contributions are as follows:
We explore a new task of generating plausible and personalized recipes from incomplete input specifications by leveraging historical user preferences;
We release a new dataset of 180K+ recipes and 700K+ user reviews for this task;
We introduce new evaluation strategies for generation quality in instructional texts, centering on quantitative measures of coherence  We also show qualitatively and quantitatively that personalized models generate high-quality and specific recipes that align with historical user preferences 
Related Work
Large-scale transformer-based language models have shown surprising expressivity and fluency in creative and conditional long-text generation BIBREF6, BIBREF7  Recent works have proposed hierarchical methods that condition on narrative frameworks to generate internally consistent long texts BIBREF8, BIBREF9, BIBREF10  Here, we generate procedurally structured recipes instead of free-form narratives 
Recipe generation belongs to the field of data-to-text natural language generation BIBREF1, which sees other applications in automated journalism BIBREF11, question-answering BIBREF12, and abstractive summarization BIBREF13, among others  BIBREF14, BIBREF15 model recipes as a structured collection of ingredient entities acted upon by cooking actions  BIBREF0 imposes a `checklist' attention constraint emphasizing hitherto unused ingredients during generation  BIBREF16 attend over explicit ingredient references in the prior recipe step  Similar hierarchical approaches that infer a full ingredient list to constrain generation will not help personalize recipes, and would be infeasible in our setting due to the potentially unconstrained number of ingredients (from a space of 10K+) in a recipe  We instead learn historical preferences to guide full recipe generation 
A recent line of work has explored user- and item-dependent aspect-aware review generation BIBREF3, BIBREF4  This work is related to ours in that it combines contextual language generation with personalization  Here, we attend over historical user preferences from previously consumed recipes to generate recipe content, rather than writing styles 
Approach
Our model's input specification consists of: the recipe name as a sequence of tokens, a partial list of ingredients, and a caloric level (high, medium, low)  It outputs the recipe instructions as a token sequence: $\mathcal {W}_r=\lbrace w_{r,0}, \dots , w_{r,T}\rbrace $ for a recipe $r$ of length $T$  To personalize output, we use historical recipe interactions of a user $u \in \mathcal {U}$ 
Encoder: Our encoder has three embedding layers: vocabulary embedding $\mathcal {V}$, ingredient embedding $\mathcal {I}$, and caloric-level embedding $\mathcal {C}$  Each token in the (length $L_n$) recipe name is embedded via $\mathcal {V}$; the embedded token sequence is passed to a two-layered bidirectional GRU (BiGRU) BIBREF17, which outputs hidden states for names $\lbrace \mathbf {n}_{\text{enc},j} \in \mathbb {R}^{2d_h}\rbrace $, with hidden size $d_h$  Similarly each of the $L_i$ input ingredients is embedded via $\mathcal {I}$, and the embedded ingredient sequence is passed to another two-layered BiGRU to output ingredient hidden states as $\lbrace \mathbf {i}_{\text{enc},j} \in \mathbb {R}^{2d_h}\rbrace $  The caloric level is embedded via $\mathcal {C}$ and passed through a projection layer with weights $W_c$ to generate calorie hidden representation $\mathbf {c}_{\text{enc}} \in \mathbb {R}^{2d_h}$ 
Ingredient Attention: We apply attention BIBREF18 over the encoded ingredients to use encoder outputs at each decoding time step  We define an attention-score function $\alpha $ with key $K$ and query $Q$:
with trainable weights $W_{\alpha }$, bias $\mathbf {b}_{\alpha }$, and normalization term $Z$  At decoding time $t$, we calculate the ingredient context $\mathbf {a}_{t}^{i} \in \mathbb {R}^{d_h}$ as:
Decoder: The decoder is a two-layer GRU with hidden state $h_t$ conditioned on previous hidden state $h_{t-1}$ and input token $w_{r, t}$ from the original recipe text  We project the concatenated encoder outputs as the initial decoder hidden state:
To bias generation toward user preferences, we attend over a user's previously reviewed recipes to jointly determine the final output token distribution  We consider two different schemes to model preferences from user histories: (1) recipe interactions, and (2) techniques seen therein (defined in data)  BIBREF19, BIBREF20, BIBREF21 explore similar schemes for personalized recommendation 
Prior Recipe Attention: We obtain the set of prior recipes for a user $u$: $R^+_u$, where each recipe can be represented by an embedding from a recipe embedding layer $\mathcal {R}$ or an average of the name tokens embedded by $\mathcal {V}$  We attend over the $k$-most recent prior recipes, $R^{k+}_u$, to account for temporal drift of user preferences BIBREF22  These embeddings are used in the `Prior Recipe' and `Prior Name' models, respectively 
Given a recipe representation $\mathbf {r} \in \mathbb {R}^{d_r}$ (where $d_r$ is recipe- or vocabulary-embedding size depending on the recipe representation) the prior recipe attention context $\mathbf {a}_{t}^{r_u}$ is calculated as
Prior Technique Attention: We calculate prior technique preference (used in the `Prior Tech` model) by normalizing co-occurrence between users and techniques seen in $R^+_u$, to obtain a preference vector $\rho _{u}$  Each technique $x$ is embedded via a technique embedding layer $\mathcal {X}$ to $\mathbf {x}\in \mathbb {R}^{d_x}$  Prior technique attention is calculated as
where, inspired by copy mechanisms BIBREF23, BIBREF24, we add $\rho _{u,x}$ for technique $x$ to emphasize the attention by the user's prior technique preference 
Attention Fusion Layer: We fuse all contexts calculated at time $t$, concatenating them with decoder GRU output and previous token embedding:
We then calculate the token probability:
and maximize the log-likelihood of the generated sequence conditioned on input specifications and user preferences  fig:ex shows a case where the Prior Name model attends strongly on previously consumed savory recipes to suggest the usage of an additional ingredient (`cilantro') 
Recipe Dataset: Food com
We collect a novel dataset of 230K+ recipe texts and 1M+ user interactions (reviews) over 18 years (2000-2018) from Food com  Here, we restrict to recipes with at least 3 steps, and at least 4 and no more than 20 ingredients  We discard users with fewer than 4 reviews, giving 180K+ recipes and 700K+ reviews, with splits as in tab:recipeixnstats 
Our model must learn to generate from a diverse recipe space: in our training data, the average recipe length is 117 tokens with a maximum of 256  There are 13K unique ingredients across all recipes  Rare words dominate the vocabulary: 95% of words appear $<$100 times, accounting for only 1 65% of all word usage  As such, we perform Byte-Pair Encoding (BPE) tokenization BIBREF25, BIBREF26, giving a training vocabulary of 15K tokens across 19M total mentions  User profiles are similarly diverse: 50% of users have consumed $\le $6 recipes, while 10% of users have consumed $>$45 recipes 
We order reviews by timestamp, keeping the most recent review for each user as the test set, the second most recent for validation, and the remainder for training (sequential leave-one-out evaluation BIBREF27)  We evaluate only on recipes not in the training set 
We manually construct a list of 58 cooking techniques from 384 cooking actions collected by BIBREF15; the most common techniques (bake, combine, pour, boil) account for 36 5% of technique mentions  We approximate technique adherence via string match between the recipe text and technique list 
Experiments and Results
For training and evaluation, we provide our model with the first 3-5 ingredients listed in each recipe  We decode recipe text via top-$k$ sampling BIBREF7, finding $k=3$ to produce satisfactory results  We use a hidden size $d_h=256$ for both the encoder and decoder  Embedding dimensions for vocabulary, ingredient, recipe, techniques, and caloric level are 300, 10, 50, 50, and 5 (respectively)  For prior recipe attention, we set $k=20$, the 80th %-ile for the number of user interactions  We use the Adam optimizer BIBREF28 with a learning rate of $10^{-3}$, annealed with a decay rate of 0 9 BIBREF29  We also use teacher-forcing BIBREF30 in all training epochs 
In this work, we investigate how leveraging historical user preferences can improve generation quality over strong baselines in our setting  We compare our personalized models against two baselines  The first is a name-based Nearest-Neighbor model (NN)  We initially adapted the Neural Checklist Model of BIBREF0 as a baseline; however, we ultimately use a simple Encoder-Decoder baseline with ingredient attention (Enc-Dec), which provides comparable performance and lower complexity  All personalized models outperform baseline in BPE perplexity (tab:metricsontest) with Prior Name performing the best  While our models exhibit comparable performance to baseline in BLEU-1/4 and ROUGE-L, we generate more diverse (Distinct-1/2: percentage of distinct unigrams and bigrams) and acceptable recipes  BLEU and ROUGE are not the most appropriate metrics for generation quality  A `correct' recipe can be written in many ways with the same main entities (ingredients)  As BLEU-1/4 capture structural information via n-gram matching, they are not correlated with subjective recipe quality  This mirrors observations from BIBREF31, BIBREF8 
We observe that personalized models make more diverse recipes than baseline  They thus perform better in BLEU-1 with more key entities (ingredient mentions) present, but worse in BLEU-4, as these recipes are written in a personalized way and deviate from gold on the phrasal level  Similarly, the `Prior Name' model generates more unigram-diverse recipes than other personalized models and obtains a correspondingly lower BLEU-1 score 
Qualitative Analysis: We present sample outputs for a cocktail recipe in tab:samplerecipes, and additional recipes in the appendix  Generation quality progressively improves from generic baseline output to a blended cocktail produced by our best performing model  Models attending over prior recipes explicitly reference ingredients  The Prior Name model further suggests the addition of lemon and mint, which are reasonably associated with previously consumed recipes like coconut mousse and pork skewers 
Personalization: To measure personalization, we evaluate how closely the generated text corresponds to a particular user profile  We compute the likelihood of generated recipes using identical input specifications but conditioned on ten different user profiles—one `gold' user who consumed the original recipe, and nine randomly generated user profiles  Following BIBREF8, we expect the highest likelihood for the recipe conditioned on the gold user  We measure user matching accuracy (UMA)—the proportion where the gold user is ranked highest—and Mean Reciprocal Rank (MRR) BIBREF32 of the gold user  All personalized models beat baselines in both metrics, showing our models personalize generated recipes to the given user profiles  The Prior Name model achieves the best UMA and MRR by a large margin, revealing that prior recipe names are strong signals for personalization  Moreover, the addition of attention mechanisms to capture these signals improves language modeling performance over a strong non-personalized baseline 
Recipe Level Coherence: A plausible recipe should possess a coherent step order, and we evaluate this via a metric for recipe-level coherence  We use the neural scoring model from BIBREF33 to measure recipe-level coherence for each generated recipe  Each recipe step is encoded by BERT BIBREF34  Our scoring model is a GRU network that learns the overall recipe step ordering structure by minimizing the cosine similarity of recipe step hidden representations presented in the correct and reverse orders  Once pretrained, our scorer calculates the similarity of a generated recipe to the forward and backwards ordering of its corresponding gold label, giving a score equal to the difference between the former and latter  A higher score indicates better step ordering (with a maximum score of 2)  tab:coherencemetrics shows that our personalized models achieve average recipe-level coherence scores of 1 78-1 82, surpassing the baseline at 1 77 
Recipe Step Entailment: Local coherence is also crucial to a user following a recipe: it is crucial that subsequent steps are logically consistent with prior ones  We model local coherence as an entailment task: predicting the likelihood that a recipe step follows the preceding  We sample several consecutive (positive) and non-consecutive (negative) pairs of steps from each recipe  We train a BERT BIBREF34 model to predict the entailment score of a pair of steps separated by a [SEP] token, using the final representation of the [CLS] token  The step entailment score is computed as the average of scores for each set of consecutive steps in each recipe, averaged over every generated recipe for a model, as shown in tab:coherencemetrics 
Human Evaluation: We presented 310 pairs of recipes for pairwise comparison BIBREF8 (details in appendix) between baseline and each personalized model, with results shown in tab:metricsontest  On average, human evaluators preferred personalized model outputs to baseline 63% of the time, confirming that personalized attention improves the semantic plausibility of generated recipes  We also performed a small-scale human coherence survey over 90 recipes, in which 60% of users found recipes generated by personalized models to be more coherent and preferable to those generated by baseline models 
Conclusion
In this paper, we propose a novel task: to generate personalized recipes from incomplete input specifications and user histories  On a large novel dataset of 180K recipes and 700K reviews, we show that our personalized generative models can generate plausible, personalized, and coherent recipes preferred by human evaluators for consumption  We also introduce a set of automatic coherence measures for instructional texts as well as personalization metrics to support our claims  Our future work includes generating structured representations of recipes to handle ingredient properties, as well as accounting for references to collections of ingredients (e g  “dry mix") 
Acknowledgements  This work is partly supported by NSF #1750063  We thank all reviewers for their constructive suggestions, as well as Rei M , Sujoy P , Alicia L , Eric H , Tim S , Kathy C , Allen C , and Micah I  for their feedback 
Appendix ::: Food com: Dataset Details
Our raw data consists of 270K recipes and 1 4M user-recipe interactions (reviews) scraped from Food com, covering a period of 18 years (January 2000 to December 2018)  See tab:int-stats for dataset summary statistics, and tab:samplegk for sample information about one user-recipe interaction and the recipe involved 
Appendix ::: Generated Examples
See tab:samplechx for a sample recipe for chicken chili and tab:samplewaffle for a sample recipe for sweet waffles 
Human Evaluation
We prepared a set of 15 pairwise comparisons per evaluation session, and collected 930 pairwise evaluations (310 per personalized model) over 62 sessions  For each pair, users were given a partial recipe specification (name and 3-5 key ingredients), as well as two generated recipes labeled `A' and `B'  One recipe is generated from our baseline encoder-decoder model and one recipe is generated by one of our three personalized models (Prior Tech, Prior Name, Prior Recipe)  The order of recipe presentation (A/B) is randomly selected for each question  A screenshot of the user evaluation interface is given in fig:exeval  We ask the user to indicate which recipe they find more coherent, and which recipe best accomplishes the goal indicated by the recipe name  A screenshot of this survey interface is given in fig:exeval2 
Context 2: Introduction
Massive Open Online Courses (MOOCs) have strived to bridge the social gap in higher education by bringing quality education from reputed universities to students at large  Such massive scaling through online classrooms, however, disrupt co-located, synchronous two-way communication between the students and the instructor 
MOOC platforms provide discussion forums for students to talk to their classmates about the lectures, homeworks, quizzes and provide a venue to socialise  Instructors (defined here as the course instructors, their teaching assistants and the MOOC platform's technical staff) monitor the discussion forum to post (reply to their message) in discussion threads among students  We refer to this posting as intervention, following prior work BIBREF0   However, due to large student enrolment, the student–instructor ratio in MOOCs is very high Therefore, instructors are not able to monitor and participate in all student discussions  To address this problem, a number of works have proposed systems e g , BIBREF0 , BIBREF1 to aid instructors to selectively intervene on student discussions where they are needed the most 
In this paper, we improve the state-of-the-art for instructor intervention in MOOC forums  We propose the first neural models for this prediction problem  We show that modelling the thread structure and the sequence of posts explicitly improves performance  Instructors in different MOOCs from different subject areas intervene differently  For example, on a Science, Technology, Engineering and Mathematics (STEM) MOOC, instructors may often intervene early as possible to resolve misunderstanding of the subject material and prevent confusion  However, in a Humanities MOOC, instructors allow for the students to explore open-ended discussions and debate among themselves  Such instructors may prefer to intervene later in the discussion to encourage further discussion or resolve conflicts among students  We therefore propose attention models to infer the latent context, i e , the series of posts that trigger an intervention  Earlier studies on MOOC forum intervention either model the entire context or require the context size to be specified explicitly 
Problem Statement
A thread INLINEFORM0 consists of a series of posts INLINEFORM1 through INLINEFORM2 where INLINEFORM3 is an instructor's post when INLINEFORM4 is intervened, if applicable  INLINEFORM5 is considered intervened if an instructor had posted at least once  The problem of predicting instructor intervention is cast as a binary classification problem  Intervened threads are predicted as 1 given while non-intervened threads are predicted as 0 given posts INLINEFORM6 through INLINEFORM7  
The primary problem leads to a secondary problem of inferring the appropriate amount of context to intervene  We define a context INLINEFORM0 of a post INLINEFORM1 as a series of linear contiguous posts INLINEFORM2 through INLINEFORM3 where INLINEFORM4   The problem of inferring context is to identify context INLINEFORM5 from a set of candidate contexts INLINEFORM6  
Modelling Context in Forums
Context has been used and modelled in various ways for different problems in discussion forums  In a work on a closely related problem of forum thread retrieval BIBREF2 models context using inter-post discourse e g , Question-Answer  BIBREF3 models the structural dependencies and relationships between forum posts using a conditional random field in their problem to infer the reply structure  Unlike BIBREF2 , BIBREF3 can be used to model any structural dependency and is, therefore, more general  In this paper, we seek to infer general dependencies between a reply and its previous context whereas BIBREF3 inference is limited to pairs of posts  More recently BIBREF4 proposed a context based model which factorises attention over threads of different lengths  Differently, we do not model length but the context before a post  However, our attention models cater to threads of all lengths 
BIBREF5 proposed graph structured LSTM to model the explicit reply structure in Reddit forums  Our work does not assume access to such a reply structure because 1) Coursera forums do not provide one and 2) forum participants often err by posting their reply to a different post than that they intended  At the other end of the spectrum are document classification models that do not assume structure in the document layout but try to infer inherent structure in the natural language, viz, words, sentences, paragraphs and documents  Hierarchical attention BIBREF6 is a well know recent work that classifies documents using a multi-level LSTMs with attention mechanism to select important units at each hierarchical level  Differently, we propose a hierarchical model that encodes layout hierarchy between a post and a thread but also infers reply structure using a attention mechanism since the layout does not reliably encode it 
Instructor Intervention in MOOC forums
The problem of predicting instructor intervention in MOOCs was proposed by BIBREF0   Later BIBREF7 evaluated baseline models by BIBREF0 over a larger corpus and found the results to vary widely across MOOCs  Since then subsequent works have used similar diverse evaluations on the same prediction problem BIBREF1 , BIBREF8   BIBREF1 proposed models with discourse features to enable better prediction over unseen MOOCs  BIBREF8 recently showed interventions on Coursera forums to be biased by the position at which a thread appears to an instructor viewing the forum interface and proposed methods for debiased prediction 
While all works since BIBREF0 address key limitations in this line of research, they have not investigated the role of structure and sequence in the threaded discussion in predicting instructor interventions  BIBREF0 proposed probabilistic graphical models to model structure and sequence  They inferred vocabulary dependent latent post categories to model the thread sequence and infer states that triggered intervention  Their model, however, requires a hyperparameter for the number of latent states  It is likely that their empirically reported setting will not generalise due to their weak evaluation BIBREF7   In this paper, we propose models to infer the context that triggers instructor intervention that does not require context lengths to be set apriori  All our proposed models generalise over modelling assumptions made by BIBREF0  
For the purpose of comparison against a state-of-the-art and competing baselines we choose BIBREF7 since BIBREF0 's system and data are not available for replication 
Data and Preprocessing
We evaluate our proposed models over a corpus of 12 MOOC iterations (offerings) on Coursera org In partnership with Coursera and in line with its Terms of Service, we obtained the data for use in our academic research  Following prior work BIBREF7 we evaluate over a diverse dataset to represent MOOCs of varying sizes, instructor styles, instructor team sizes and number of threads intervened  We only include threads from sub-forums on Lecture, Homework, Quiz and Exam  We also normalise and label sub-forums with other non-standard names (e g , Assignments instead of Homework) into of the four said sub-forums  Threads on general discussion, meet and greet and other custom sub-forums for social chitchat are omitted as our focus is to aid instructors on intervening on discussion on the subject matter  We also exclude announcement threads and other threads started by instructors since they are not interventions  We preprocess each thread by replacing URLs, equations and other mathematical formulae and references to timestamps in lecture videos by tokens INLINEFORM0 URL INLINEFORM1 , INLINEFORM2 MATH INLINEFORM3 , INLINEFORM4 TIMEREF INLINEFORM5 respectively  We also truncate intervened threads to only include posts before the first instructor post since the instructor's and subsequent posts will bias the prediction due to the instructor's post 
Model
The key innovation of our work is to decompose the intervention prediction problem into a two-stage model that first explicitly tries to discover the proper context to which a potential intervention could be replying to, and then, predict the intervention status  This model implicitly assesses the importance (or urgency) of the existing thread's context to decide whether an intervention is necessary  For example in Figure SECREF1 , prior to the instructor's intervention, the ultimate post (Post #6) by Student 2 already acknowledged the OP's gratitude for his answer  In this regard, the instructor may have decided to use this point to summarize the entire thread to consolidate all the pertinent positions  Here, we might assume that the instructor's reply takes the entire thread (Posts #1–6) as the context for her reply 
This subproblem of inferring the context scope is where our innovation centers on  To be clear, in order to make the prediction that a instruction intervention is now necessary on a thread, the instructor's reply is not yet available — the model predicts whether a reply is necessary — so in the example, only Posts #1–6 are available in the problem setting  To infer the context, we have to decide which subsequence of posts are the most plausible motivation for an intervention 
Recent work in deep neural modeling has used an attention mechanism as a focusing query to highlight specific items within the input history that significantly influence the current decision point  Our work employs this mechanism – but with a twist: due to the fact that the actual instructor intervention is not (yet) available at the decision timing, we cannot use any actual intervention to decide the context  To employ attention, we must then employ a surrogate text as the query to train our prediction model  Our model variants model assess the suitability of such surrogate texts for the attention mechanism basis 
Congruent with the representation of the input forums, in all our proposed models, we encode the discussion thread hierarchically  We first build representations for each post by passing pre-trained word vector representations from GloVe BIBREF9 for each word through an LSTM BIBREF10 , INLINEFORM0   We use the last layer output of the LSTM as a representation of the post  We refer this as the post vector INLINEFORM1  
Then each post INLINEFORM0 is passed through another LSTM, INLINEFORM1 , whose last layer output forms the encoding of the entire thread  Hidden unit outputs of INLINEFORM2 represent the contexts INLINEFORM3 ; that is, snapshots of the threads after each post, as shown in Figure FIGREF1  
The INLINEFORM0 and INLINEFORM1 together constitute the hierarchical LSTM (hLSTM) model  This general hLSTM model serves as the basis for our model exploration in the rest of this section 
Contextual Attention Models
When they intervene, instructors either pay attention to a specific post or a series of posts, which trigger their reply  However, instructors rarely explicitly indicate to which post(s) their intervention is in relation to  This is the case in our corpus, party due to Coursera's user interface which only allows for single level comments (see Figure FIGREF2 )  Based solely on the binary, thread-level intervention signal, our secondary objective seeks to infer the appropriate context – represented by a sequence of posts – as the basis for the intervention 
We only consider linear contiguous series of posts starting with the thread's original post to constitute to a context; e g , INLINEFORM0   This is a reasonable as MOOC forum posts always reply to the original post or to a subsequent post, which in turn replies to the original post  This is in contrast to forums such as Reddit that have a tree or graph-like structure that require forum structure to be modelled explicitly, such as in BIBREF5  
We propose three neural attention BIBREF11 variants based on how an instructor might attend and reply to a context in a thread: the ultimate, penultimate and any post attention models  We review each of these in turn
Context 3: Introduction
Speech-to-Text translation (ST) is essential for a wide range of scenarios: for example in emergency calls, where agents have to respond emergent requests in a foreign language BIBREF0; or in online courses, where audiences and speakers use different languages BIBREF1  To tackle this problem, existing approaches can be categorized into cascaded method BIBREF2, BIBREF3, where a machine translation (MT) model translates outputs of an automatic speech recognition (ASR) system into target language, and end-to-end method BIBREF4, BIBREF5, where a single model learns acoustic frames to target word sequence mappings in one step towards the final objective of interest  Although the cascaded model remains the dominant approach due to its better performance, the end-to-end method becomes more and more popular because it has lower latency by avoiding inferences with two models and rectifies the error propagation in theory 
Since it is hard to obtain a large-scale ST dataset, multi-task learning BIBREF5, BIBREF6 and pre-training techniques BIBREF7 have been applied to end-to-end ST model to leverage large-scale datasets of ASR and MT  A common practice is to pre-train two encoder-decoder models for ASR and MT respectively, and then initialize the ST model with the encoder of the ASR model and the decoder of the MT model  Subsequently, the ST model is optimized with the multi-task learning by weighing the losses of ASR, MT, and ST  This approach, however, causes a huge gap between pre-training and fine-tuning, which are summarized into three folds:
Subnet Waste: The ST system just reuses the ASR encoder and the MT decoder, while discards other pre-trained subnets, such as the MT encoder  Consequently, valuable semantic information captured by the MT encoder cannot be inherited by the final ST system 
Role Mismatch: The speech encoder plays different roles in pre-training and fine-tuning  The encoder is a pure acoustic model in pre-training, while it has to extract semantic and linguistic features additionally in fine-tuning, which significantly increases the learning difficulty 
Non-pre-trained Attention Module: Previous work BIBREF6 trains attention modules for ASR, MT and ST respectively, hence, the attention module of ST does not benefit from the pre-training 
To address these issues, we propose a Tandem Connectionist Encoding Network (TCEN), which is able to reuse all subnets in pre-training, keep the roles of subnets consistent, and pre-train the attention module  Concretely, the TCEN consists of three components, a speech encoder, a text encoder, and a target text decoder  Different from the previous work that pre-trains an encoder-decoder based ASR model, we only pre-train an ASR encoder by optimizing the Connectionist Temporal Classification (CTC) BIBREF8 objective function  In this way, the additional decoder of ASR is not required while keeping the ability to read acoustic features into the source language space by the speech encoder  Besides, the text encoder and decoder can be pre-trained on a large MT dataset  After that, we employ common used multi-task learning method to jointly learn ASR, MT and ST tasks 
Compared to prior works, the encoder of TCEN is a concatenation of an ASR encoder and an MT encoder and our model does not have an ASR decoder, so the subnet waste issue is solved  Furthermore, the two encoders work at tandem, disentangling acoustic feature extraction and linguistic feature extraction, ensuring the role consistency between pre-training and fine-tuning  Moreover, we reuse the pre-trained MT attention module in ST, so we can leverage the alignment information learned in pre-training 
Since the text encoder consumes word embeddings of plausible texts in MT task but uses speech encoder outputs in ST task, another question is how one guarantees the speech encoder outputs are consistent with the word embeddings  We further modify our model to achieve semantic consistency and length consistency  Specifically, (1) the projection matrix at the CTC classification layer for ASR is shared with the word embedding matrix, ensuring that they are mapped to the same latent space, and (2) the length of the speech encoder output is proportional to the length of the input frame, so it is much longer than a natural sentence  To bridge the length gap, source sentences in MT are lengthened by adding word repetitions and blank tokens to mimic the CTC output sequences 
We conduct comprehensive experiments on the IWSLT18 speech translation benchmark BIBREF1, demonstrating the effectiveness of each component  Our model is significantly better than previous methods by 3 6 and 2 2 BLEU scores for the subword-level decoding and character-level decoding strategies, respectively 
Our contributions are three-folds: 1) we shed light on why previous ST models cannot sufficiently utilize the knowledge learned from the pre-training process; 2) we propose a new ST model, which alleviates shortcomings in existing methods; and 3) we empirically evaluate the proposed model on a large-scale public dataset 
Background ::: Problem Formulation
End-to-end speech translation aims to translate a piece of audio into a target-language translation in one step  The raw speech signals are usually converted to sequences of acoustic features, e g  Mel filterbank features  Here, we define the speech feature sequence as $\mathbf {x} = (x_1, \cdots , x_{T_x})$ The transcription and translation sequences are denoted as $\mathbf {y^{s}} = (y_1^{s}, \cdots , y_{T_s}^{s})$, and $\mathbf {y^{t}} = (y_1^{t}, \cdots , y_{T_t}^{t})$ repectively  Each symbol in $\mathbf {y^{s}}$ or $\mathbf {y^{t}}$ is an integer index of the symbol in a vocabulary $V_{src}$ or $V_{trg}$ respectively (e g  $y^s_i=k, k\in [0, |V_{src}|-1]$)  In this work, we suppose that an ASR dataset, an MT dataset, and a ST dataset are available, denoted as $\mathcal {A} = \lbrace (\mathbf {x_i}, \mathbf {y^{s}_i})\rbrace _{i=0}^I$, $\mathcal {M} =\lbrace (\mathbf {y^{s}_j}, \mathbf {y^{t}_j})\rbrace _{j=0}^J$ and $ \mathcal {S} =\lbrace (\mathbf {x_l}, \mathbf {y^{t}_l})\rbrace _{l=0}^L$ respectively  Given a new piece of audio $\mathbf {x}$, our goal is to learn an end to end model to generate a translation sentence $\mathbf {y^{t}}$ without generating an intermediate result $\mathbf {y^{s}}$ 
Background ::: Multi-Task Learning and Pre-training for ST
To leverage large scale ASR and MT data, multi-task learning and pre-training techniques are widely employed to improve the ST system  As shown in Figure FIGREF4, there are three popular multi-task strategies for ST, including 1) one-to-many setting, in which a speech encoder is shared between ASR and ST tasks; 2) many-to-one setting in which a decoder is shared between MT and ST tasks; and 3) many-to-many setting where both the encoder and decoder are shared 
A many-to-many multi-task model contains two encoders as well as two decoders  It can be jointly trained on ASR, MT, and ST tasks  As the attention module is task-specific, three attentions are defined 
Usually, the size of $\mathcal {A}$ and $\mathcal {M}$ is much larger than $\mathcal {S}$  Therefore, the common training practice is to pre-train the model on ASR and MT tasks and then fine-tune it with a multi-task learning manner  However, as aforementioned, this method suffers from subnet waste, role mismatch and non-pre-trained attention issues, which severely limits the end-to-end ST performance 
Our method
In this section, we first introduce the architecture of TCEN, which consists of two encoders connected in tandem, and one decoder with an attention module  Then we give the pre-training and fine-tuning strategy for TCEN  Finally, we propose our solutions for semantic and length inconsistency problems, which are caused by multi-task learning 
Our method ::: TCEN Architecture
Figure FIGREF5 sketches the overall architecture of TCEN, including a speech encoder $enc_s$, a text encoder $enc_t$ and a decoder $dec$ with an attention module $att$  During training, the $enc_s$ acts like an acoustic model which reads the input $\mathbf {x}$ to word or subword representations $\mathbf {h^s}$, then $enc_t$ learns high-level linguistic knowledge into hidden representations $\mathbf {h^t}$  Finally, the $dec$ defines a distribution probability over target words  The advantage of our architecture is that two encoders disentangle acoustic feature extraction and linguistic feature extraction, making sure that valuable knowledge learned from ASR and MT tasks can be effectively leveraged for ST training  Besides, every module in pre-training can be utilized in fine-tuning, alleviating the subnet waste problem 
Follow BIBREF9 inaguma2018speech, we use CNN-BiLSTM architecture to build our model  Specifically, the input features $\mathbf {x}$ are organized as a sequence of feature vectors in length $T_x$  Then, $\mathbf {x}$ is passed into a stack of two convolutional layers followed by max-pooling:
where $\mathbf {v}^{(l-1)}$ is feature maps in last layer and $\mathbf {W}^{(l)}$ is the filter  The max-pooling layers downsample the sequence in length by a total factor of four  The down-sampled feature sequence is further fed into a stack of five bidirectional $d$-dimensional LSTM layers:
where $[;]$ denotes the vector concatenation  The final output representation from the speech encoder is denoted as $\mathbf {h^s}=(h^s_1, \cdots , h^s_{\frac{T_x}{4}})$, where $h_i^s \in \mathbb {R}^d$ 
The text encoder $enc_t$ consists of two bidirectional LSTM layers  In ST task, $enc_t$ accepts speech encoder output $\mathbf {h}^s$ as input  While in MT, $enc_t$ consumes the word embedding representation $\mathbf {e^s}$ derived from $\mathbf {y^s}$, where each element $e^s_i$ is computed by choosing the $y_i^s$-th vector from the source embedding matrix $W_{E^s}$  The goal of $enc_t$ is to extract high-level linguistic features like syntactic features or semantic features from lower level subword representations $\mathbf {h}^s$ or $\mathbf {e}^s$  Since $\mathbf {h}^s$ and $\mathbf {e}^s$ belong to different latent space and have different lengths, there remain semantic and length inconsistency problems  We will provide our solutions in Section SECREF21  The output sequence of $enc_t$ is denoted as $\mathbf {h}^t$ 
The decoder is defined as two unidirectional LSTM layers with an additive attention $att$  It predicts target sequence $\mathbf {y^{t}}$ by estimating conditional probability $P(\mathbf {y^{t}}|\mathbf {x})$:
Here, $z_k$ is the the hidden state of the deocder RNN at $k$ step and $c_k$ is a time-dependent context vector computed by the attention $att$ 
Our method ::: Training Procedure
Following previous work, we split the training procedure to pre-training and fine-tuning stages  In pre-training stage, the speech encoder $enc_s$ is trained towards CTC objective using dataset $\mathcal {A}$, while the text encoder $enc_t$ and the decoder $dec$ are trained on MT dataset $\mathcal {M}$  In fine-tuning stage, we jointly train the model on ASR, MT, and ST tasks 
Our method ::: Training Procedure ::: Pre-training
To sufficiently utilize the large dataset $\mathcal {A}$ and $\mathcal {M}$, the model is pre-trained on CTC-based ASR task and MT task in the pre-training stage 
For ASR task, in order to get rid of the requirement for decoder and enable the $enc_s$ to generate subword representation, we leverage connectionist temporal classification (CTC) BIBREF8 loss to train the speech encoder 
Given an input $\mathbf {x}$, $enc_s$ emits a sequence of hidden vectors $\mathbf {h^s}$, then a softmax classification layer predicts a CTC path $\mathbf {\pi }$, where $\pi _t \in V_{src} \cup $ {`-'} is the observing label at particular RNN step $t$, and `-' is the blank token representing no observed labels:
where $W_{ctc} \in \mathbb {R}^{d \times (|V_{src}|+1)}$ is the weight matrix in the classification layer and $T$ is the total length of encoder RNN 
A legal CTC path $\mathbf {\pi }$ is a variation of the source transcription $\mathbf {y}^s$ by allowing occurrences of blank tokens and repetitions, as shown in Table TABREF14

ground_truth: Byte-Pair Encoding perplexity  (BPE PPL),
BLEU-1,
BLEU-4,
ROUGE-L,
percentage of distinct unigram (D-1),
percentage of distinct bigrams(D-2),
user matching accuracy(UMA),
Mean Reciprocal Rank(MRR)
Pairwise preference over baseline(PP)

BLEU-1/4 and ROUGE-L, likelihood of generated recipes using identical input specifications but conditioned on ten different user profiles, user matching accuracy (UMA), Mean Reciprocal Rank (MRR), neural scoring model from BIBREF33 to measure recipe-level coherence

 Distinct-1/2, UMA = User Matching Accuracy, MRR
= Mean Reciprocal Rank, PP = Pairwise preference over baseline (evaluated for 310 recipe pairs per model)

prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. What was the previous state of the art for this task?
Context 2: 
Ultimate Post Attention (UPA) Model  In this model we attend to the context represented by hidden state of the INLINEFORM0   We use the post prior to the instructor's reply as a query over the contexts INLINEFORM1 to compute attention weights INLINEFORM2 , which are then used to compute the attended context representation INLINEFORM3 (recall again that the intervention text itself is not available for this purpose)  This attention formulation makes an equivalence between the final INLINEFORM4 post and the prospective intervention, using Post INLINEFORM5 as the query for finding the appropriate context INLINEFORM6 , inclusive of itself INLINEFORM7   Said in another way, UPA uses the most recent content in the thread as the attentional query for context 
For example, if post INLINEFORM0 is the instructor's reply, post INLINEFORM1 will query over the contexts INLINEFORM2 and INLINEFORM3   The model schematic is shown in Figure FIGREF12  
The attended context representations are computed as: DISPLAYFORM0
The INLINEFORM0 representation is then passed through a fully connected softmax layer to yield the binary prediction 
Penultimate Post Attention (PPA) Model  While the UPA model uses the most recent text and makes the ultimate post itself available as potential context, our the ultimate post may be better modeled as having any of its prior posts as potential context  Penultimate Post Attention (PPA) variant does this  The schematic and the equations for the PPA model are obtained by summing over contexts INLINEFORM0 in Equation EQREF10 and Figure FIGREF12   While we could properly model such a context inference decision with any post INLINEFORM1 and prospective contexts INLINEFORM2 (where INLINEFORM3 is a random post), it makes sense to use the penultimate post, as we can make the most information available to the model for the context inference 
The attended context representations are computed as: DISPLAYFORM0
Any Post Attention (APA) Model  APA further relaxes both UPA and PPA, allowing APA to generalize and hypothesize that the prospective instructor intervention is based on the context that any previous post INLINEFORM0 replied to  In this model, each post INLINEFORM1 is set as a query to attend to its previous context INLINEFORM2   For example, INLINEFORM3 will attend to INLINEFORM4   Different from standard attention mechanisms, APA attention weights INLINEFORM5 are obtained by normalising interaction matrix over the different queries 
In APA, the attention context INLINEFORM0 is computed via: DISPLAYFORM0
Evaluation
The baseline and the models are evaluated on a corpus of 12 MOOC discussion forums  We train on 80% of the training data and report evaluation results on the held-out 20% of test data  We report INLINEFORM0 scores on the positive class (interventions), in line with prior work  We also argue that recall of the positive class is more important than precision, since it is costlier for instructors to miss intervening on a thread than spending irrelevant time intervening on a less critical threads due to false positives 
Model hyperpameter settings  All proposed and baseline neural models are trained using Adam optimizer with a learning rate of 0 001  We used cross-entropy as loss function  Importantly we updated the model parameters during training after each instance as in vanilla stochastic gradient descent; this setting was practical since data on most courses had only a few hundred instances enabling convergence within a reasonable training time of a few hours (see Table TABREF15 , column 2)  Models were trained for a single epoch as most of our courses with a few hundred thread converged after a single epoch  We used 300-dimensional GloVe vectors and permitted the embeddings to be updated during the model's end-to-end training  The hidden dimension size of both INLINEFORM0 and INLINEFORM1 are set to 128 for all the models 
Baselines  We compare our models against a neural baseline models, hierarchical LSTM (hLSTM), with the attention ablated but with access to the complete context, and a strong, open-sourced feature-rich baseline BIBREF7   We choose BIBREF7 over other prior works such as BIBREF0 since we do not have access to the dataset or the system used in their papers for replication  BIBREF7 is a logistic regression classifier with features inclusive of bag-of-words representation of the unigrams and thread length, normalised counts of agreements to previous posts, counts of non-lexical reference items such as URLs, and the Coursera forum type in which a thread appeared  We also report aggregated results from a hLSTM model with access only to the last post as context for comparison  Table TABREF17 compares the performance of these baselines against our proposed methods 
Results
Table TABREF15 shows performance of all our proposed models and the neural baseline over our 12 MOOC dataset  Our models of UPA, PPA individually better the baseline by 5 and 2% on INLINEFORM0 and 3 and 6% on recall respectively  UPA performs the best in terms of INLINEFORM1 on average while PPA performs the best in terms of recall on average  At the individual course level, however, the results are mixed  UPA performs the best on INLINEFORM2 on 5 out of 12 courses, PPA on 3 out 12 courses, APA 1 out of 12 courses and the baseline hLSTM on 1  PPA performs the best on recall on 7 out of the 12 courses  We also note that course level performance differences correlate with the course size and intervention ratio (hereafter, i ratio), which is the ratio of intervened to non-intervened threads  UPA performs better than PPA and APA on low intervention courses (i ratio INLINEFORM3 0 25) mainly because PPA and APA's performance drops steeply when i ratio drops (see col 2 parenthesis and INLINEFORM4 of PPA and APA)  While all the proposed models beat the baseline on every course except casebased-2  On medicalneuro-2 and compilers-4 which have the lowest i ratio among the 12 courses none of the neural models better the reported baseline BIBREF7 (course level not scores not shown in this paper)  The effect is pronounced in compilers-4 course where none of the neural models were able to predict any intervened threads  This is due to the inherent weakness of standard neural models, which are unable to learn features well enough when faced with sparse data 
The best performance of UPA indicates that the reply context of the instructor's post INLINEFORM0 correlates strongly with that of the previous post INLINEFORM1   This is not surprising since normal conversations are typically structured that way 
Discussion
In order to further understand the models' ability to infer the context and its effect on intervention prediction, we further investigate the following research questions 
RQ1  Does context inference help intervention prediction?
In order to understand if context inference is useful to intervention prediction, we ablate the attention components and experiment with the vanilla hierarchical LSTM model  Row 3 of Table TABREF17 shows the macro averaged result from this experiment  The UPA and PPA attention models better the vanilla hLSTM by 5% and 2% on average in INLINEFORM0 respectively  Recall that the vanilla hLSTM already has access to a context consisting of all posts (from INLINEFORM1 through INLINEFORM2 )  In contrast, the UPA and PPA models selectively infers a context for INLINEFORM3 and INLINEFORM4 posts, respectively, and use it to predict intervention  The improved performance of our attention models that actively select their optimal context, over a model with the complete thread as context, hLSTM, shows that the context inference improves intervention prediction over using the default full context 
RQ2  How well do the models perform across threads of different lengths? To understand the models' prediction performance across threads of different lengths, we bin threads by length and study the models' recall  We choose three courses, ml-5, rprog-3 and calc-1, from our corpus of 12 with the highest number of positive instances ( INLINEFORM0 100 threads)  We limit our analysis to these since binning renders courses with fewer positive instances sparse  Figure FIGREF18 shows performance across thread lengths from 1 through 7 posts and INLINEFORM1 posts  Clearly, the UPA model performs much better on shorter threads than on longer threads while PPA and APA works better on longer threads  Although, UPA is the best performing model in terms of overall INLINEFORM2 its performance drops steeply on threads of length INLINEFORM3   UPA's overall best performance is because most of the interventions in the corpus happen after one post  To highlight the performance of APA we show an example from smac-1 in Figure FIGREF22 with nine posts which was predicted correctly as intervened by APA but not by other models  Threads shows students confused over a missing figure in a homework  The instructor finally shows up, though late, to resolve the confusion 
RQ3  Do models trained with different context lengths perform better than when trained on a single context length?
We find that context length has a regularising effect on the model's performance at test time  This is not surprising since models trained with threads of single context length will not generalise to infer different context lengths  Row 4 of Table TABREF17 shows a steep performance drop in training by classifier with all threads truncated to a context of just one post, INLINEFORM0 , the post immediately preceding the intervened post  We also conducted an experiment with a multi-objective loss function with an additive cross-entropy term where each term computes loss from a model with context limited to a length of 3  We chose 3 since intervened threads in all the courses had a median length between 3 and 4  We achieved an INLINEFORM1 of 0 45 with a precision of 0 47 and recall of 0 43  This achieves a performance comparable to that of the BIBREF7 with context length set to only to 3  This approach of using infinitely many loss terms for each context length from 1 through the maximum thread length in a course is naive and not practical  We only use this model to show the importance of training the model with loss from threads of different lengths to prevent models overfitting to threads of specific context lengths 
Conclusion
We predict instructor intervention on student discussions by first inferring the optimal size of the context needed to decide on the intervention decision for the intervened post  We first show that a structured representation of the complete thread as the context is better than a bag-of-words, feature-rich representation  We then propose attention-based models to infer and select a context – defined as a contiguous subsequence of student posts – to improve over a model that always takes the complete thread as a context to prediction intervention  Our Any Post Attention (APA) model enables instructors to tune the model to predict intervention early or late  We posit our APA model will enable MOOC instructors employing varying pedagogical styles to use the model equally well  We introspect the attention models' performance across threads of varying lengths and show that APA predicts intervention on longer threads, which possesses more candidate contexts, better 
We note that the recall of the predictive models for longer threads (that is, threads of length greater 2) can still be improved  Models perform differently between shorter and longer length  An ensemble model or a multi-objective loss function is thus planned in our future work to better prediction on such longer threads 
Context 3:  For each transcription $\mathbf {y}$, there exist many legal CTC paths in length $T$  The CTC objective trains the model to maximize the probability of observing the golden sequence $\mathbf {y}^s$, which is calculated by summing the probabilities of all possible legal paths:
where $\Phi _T(y)$ is the set of all legal CTC paths for sequence $\mathbf {y}$ with length $T$  The loss can be easily computed using forward-backward algorithm  More details about CTC are provided in supplementary material 
For MT task, we use the cross-entropy loss as the training objective  During training, $\mathbf {y^s}$ is converted to embedding vectors $\mathbf {e^s}$ through embedding layer $W_{E^s}$, then $enc_t$ consumes $\mathbf {e^s}$ and pass the output $\mathbf {h^t}$ to decoder  The objective function is defined as:
Our method ::: Training Procedure ::: Fine-tune
In fine-tune stage, we jointly update the model on ASR, MT, and ST tasks  The training for ASR and MT follows the same process as it was in pre-training stage 
For ST task, the $enc_s$ reads the input $\mathbf {x}$ and generates $\mathbf {h^s}$, then $enc_t$ learns high-level linguistic knowledge into $\mathbf {h^t}$  Finally, the $dec$ predicts the target sentence  The ST loss function is defined as:
Following the update strategy proposed by BIBREF11 luong2015multi, we allocate a different training ratio $\alpha _i$ for each task  When switching between tasks, we select randomly a new task $i$ with probability $\frac{\alpha _i}{\sum _{j}\alpha _{j}}$ 
Our method ::: Subnet-Consistency
Our model keeps role consistency between pre-training and fine-tuning by connecting two encoders for ST task  However, this leads to some new problems: 1) The text encoder consumes $\mathbf {e^s}$ during MT training, while it accepts $\mathbf {h^s}$ during ST training  However, $\mathbf {e^s}$ and $\mathbf {h^s}$ may not follow the same distribution, resulting in the semantic inconsistency  2) Besides, the length of $\mathbf {h^s}$ is not the same order of magnitude with the length of $\mathbf {e^s}$, resulting in the length inconsistency 
In response to the above two challenges, we propose two countermeasures: 1) We share weights between CTC classification layer and source-end word embedding layer during training of ASR and MT, encouraging $\mathbf {e^s}$ and $\mathbf {h^s}$ in the same space  2)We feed the text encoder source sentences in the format of CTC path, which are generated from a seq2seq model, making it more robust toward long inputs 
Our method ::: Subnet-Consistency ::: Semantic Consistency
As shown in Figure FIGREF5, during multi-task training, two different hidden features will be fed into the text encoder $enc_t$: the embedding representation $\mathbf {e}^s$ in MT task, and the $enc_s$ output $\mathbf {h^s}$ in ST task  Without any regularization, they may belong to different latent spaces  Due to the space gap, the $enc_t$ has to compromise between two tasks, limiting its performance on individual tasks 
To bridge the space gap, our idea is to pull $\mathbf {h^s}$ into the latent space where $\mathbf {e}^s$ belong  Specifically, we share the weight $W_{ctc}$ in CTC classification layer with the source embedding weights $W_{E^s}$, which means $W_{ctc} = W_{E^s}$  In this way, when predicting the CTC path $\mathbf {\pi }$, the probability of observing the particular label $w_i \in V_{src}\cup ${`-'} at time step $t$, $p(\pi _t=w_i|\mathbf {x})$, is computed by normalizing the product of hidden vector $h_t^s$ and the $i$-th vector in $W_{E^s}$:
The loss function closes the distance between $h^s_t$ and golden embedding vector, encouraging $\mathbf {h}^s$ have the same distribution with $\mathbf {e}^s$ 
Our method ::: Subnet-Consistency ::: Length Consistency
Another existing problem is length inconsistency  The length of the sequence $\mathbf {h^s}$ is proportional to the length of the input frame $\mathbf {x}$, which is much longer than the length of $\mathbf {e^s}$  To solve this problem, we train an RNN-based seq2seq model to transform normal source sentences to noisy sentences in CTC path format, and replace standard MT with denoising MT for multi-tasking 
Specifically, we first train a CTC ASR model based on dataset $\mathcal {A} = \lbrace (\mathbf {x}_i, \mathbf {y}^s_i)\rbrace _{i=0}^{I}$, and generate a CTC-path $\mathbf {\pi }_i$ for each audio $\mathbf {x}_i$ by greedy decoding  Then we define an operation $S(\cdot )$, which converts a CTC path $\mathbf {\pi }$ to a sequence of the unique tokens $\mathbf {u}$ and a sequence of repetition times for each token $\mathbf {l}$, denoted as $S(\mathbf {\pi }) = (\mathbf {u}, \mathbf {l})$  Notably, the operation is reversible, meaning that $S^{-1} (\mathbf {u}, \mathbf {l})=\mathbf {\pi }$  We use the example $\mathbf {\pi _1}$ in Table TABREF14 and show the corresponding $\mathbf {u}$ and $\mathbf {l}$ in Table TABREF24 
Then we build a dataset $\mathcal {P} = \lbrace (\mathbf {y^s}_i, \mathbf {u}_i, \mathbf {l}_i)\rbrace _{i=0}^{I}$ by decoding all the audio pieces in $\mathcal {A}$ and transform the resulting path by the operation $S(\cdot )$  After that, we train a seq2seq model, as shown in Figure FIGREF25, which takes $ \mathbf {y^s}_i$ as input and decodes $\mathbf {u}_i, \mathbf {l}_i$ as outputs  With the seq2seq model, a noisy MT dataset $\mathcal {M}^{\prime }=\lbrace (\mathbf {\pi }_l, \mathbf {y^t}_l)\rbrace _{l=0}^{L}$ is obtained by converting every source sentence $\mathbf {y^s}_i \in \mathcal {M}$ to $\mathbf {\pi _i}$, where $\mathbf {\pi }_i = S^{-1}(\mathbf {u}_i, \mathbf {l}_i)$  We did not use the standard seq2seq model which takes $\mathbf {y^s}$ as input and generates $\mathbf {\pi }$ directly, since there are too many blank tokens `-' in $\mathbf {\pi }$ and the model tends to generate a long sequence with only blank tokens  During MT training, we randomly sample text pairs from $\mathcal {M}^{\prime }$ and $\mathcal {M}$ according to a hyper-parameter $k$  After tuning on the validation set, about $30\%$ pairs are sampled from $\mathcal {M}^{\prime }$  In this way, the $enc_t$ is more robust toward the longer inputs given by the $enc_s$ 
Experiments
We conduct experiments on the IWSLT18 speech translation task BIBREF1  Since IWSLT participators use different data pre-processing methods, we reproduce several competitive baselines based on the ESPnet BIBREF12 for a fair comparison 
Experiments ::: Dataset ::: Speech translation data:
The organizer provides a speech translation corpus extracting from the TED talk (ST-TED), which consists of raw English wave files, English transcriptions, and aligned German translations  The corpus contains 272 hours of English speech with 171k segments  We split 2k segments from the corpus as dev set and tst2010, tst2013, tst2014, tst2015 are used as test sets 
Speech recognition data: Aside from ST-TED, TED-LIUM2 corpus BIBREF13 is provided as speech recognition data, which contains 207 hours of English speech and 93k transcript sentences 
Text translation data: We use transcription and translation pairs in the ST-TED corpus and WIT3 as in-domain MT data, which contains 130k and 200k sentence pairs respectively  WMT2018 is used as out-of-domain training data which consists of 41M sentence pairs 
Data preprocessing: For speech data, the utterances are segmented into multiple frames with a 25 ms window size and a 10 ms step size  Then we extract 80-channel log-Mel filter bank and 3-dimensional pitch features using Kaldi BIBREF14, resulting in 83-dimensional input features  We normalize them by the mean and the standard deviation on the whole training set  The utterances with more than 3000 frames are discarded  The transcripts in ST-TED are in true-case with punctuation while in TED-LIUM2, transcripts are in lower-case and unpunctuated  Thus, we lowercase all the sentences and remove the punctuation to keep consistent  To increase the amount of training data, we perform speed perturbation on the raw signals with speed factors 0 9 and 1 1  For the text translation data, sentences longer than 80 words or shorter than 10 words are removed  Besides, we discard pairs whose length ratio between source and target sentence is smaller than 0 5 or larger than 2 0  Word tokenization is performed using the Moses scripts and both English and German words are in lower-case 
We use two different sets of vocabulary for our experiments  For the subword experiments, both English and German vocabularies are generated using sentencepiece BIBREF15 with a fixed size of 5k tokens  BIBREF9 inaguma2018speech show that increasing the vocabulary size is not helpful for ST task  For the character experiments, both English and German sentences are represented in the character level 
For evaluation, we segment each audio with the LIUM SpkDiarization tool BIBREF16 and then perform MWER segmentation with RWTH toolkit BIBREF17  We use lowercase BLEU as evaluation metric 
Experiments ::: Baseline Models and Implementation
We compare our method with following baselines 
Vanilla ST baseline: The vanilla ST BIBREF9 has only a speech encoder and a decoder  It is trained from scratch on the ST-TED corpus 
Pre-training baselines: We conduct three pre-training baseline experiments: 1) encoder pre-training, in which the ST encoder is initialized from an ASR model; 2) decoder pre-training, in which the ST decoder is initialized from an MT model; and 3) encoder-decoder pre-training, where both the encoder and decoder are pre-trained  The ASR model has the same architecture with vanilla ST model, trained on the mixture of ST-TED and TED-LIUM2 corpus  The MT model has a text encoder and decoder with the same architecture of which in TCEN  It is first trained on WMT data (out-of-domain) and then fine-tuned on in-domain data 
Multi-task baselines: We also conduct three multi-task baseline experiments including one-to-many setting, many-to-one setting, and many-to-many setting  In the first two settings, we train the model with $\alpha _{st}=0 75$ while $\alpha _{asr}=0 25$ or $\alpha _{mt}=0 25$

ground_truth: hLSTM

hLSTM

prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. What is the attention module pretrained on?
Context 3:  For many-to-many setting, we use $\alpha _{st}=0 6, \alpha _{asr}=0 2$ and $\alpha _{mt}=0 2$   For MT task, we use only in-domain data 
Many-to-many+pre-training: We train a many-to-many multi-task model where the encoders and decoders are derived from pre-trained ASR and MT models  Triangle+pre-train: BIBREF18 DBLP:conf/naacl/AnastasopoulosC18 proposed a triangle multi-task strategy for speech translation  Their model solves the subnet waste issue by concatenating an ST decoder to an ASR encoder-decoder model  Notably, their ST decoder can consume representations from the speech encoder as well as the ASR decoder  For a fair comparison, the speech encoder and the ASR decoder are initialized from the pre-trained ASR model  The Triangle model is fine-tuned under their multi-task manner 
All our baselines as well as TCEN are implemented based on ESPnet BIBREF12, the RNN size is set as $d=1024$ for all models  We use a dropout of 0 3 for embeddings and encoders, and train using Adadelta with initial learning rate of 1 0 for a maximum of 10 epochs 
For training of TCEN, we set $\alpha _{asr}=0 2$ and $\alpha _{mt}=0 8$ in the pre-training stage, since the MT dataset is much larger than ASR dataset  For fine-tune, we use $\alpha _{st}=0 6, \alpha _{asr}=0 2$ and $\alpha _{mt}=0 2$, same as the `many-to-many' baseline 
For testing, we select the model with the best accuracy on speech translation task on dev set  At inference time, we use a beam size of 10, and the beam scores include length normalization with a weight of 0 2 
Experiments ::: Experimental Results
Table TABREF29 shows the results on four test sets as well as the average performance  Our method significantly outperforms the strong `many-to-many+pretrain' baseline by 3 6 and 2 2 BLEU scores respectively, indicating the proposed method is very effective that substantially improves the translation quality  Besides, both pre-training and multi-task learning can improve translation quality, and the pre-training settings (2nd-4th rows) are more effective compared to multi-task settings (5th-8th rows)  We observe a performance degradation in the `triangle+pretrain' baseline  Compared to our method, where the decoder receives higher-level syntactic and semantic linguistic knowledge extracted from text encoder, their ASR decoder can only provide lower word-level linguistic information  Besides, since their model lacks text encoder and the architecture of ST decoder is different from MT decoder, their model cannot utilize the large-scale MT data in all the training stages  Interestingly, we find that the char-level models outperform the subword-level models in all settings, especially in vanilla baseline  A similar phenomenon is observed by BIBREF6 berard2018end  A possible explanation is that learning the alignments between speech frames and subword units in another language is notoriously difficult  Our method can bring more gains in the subword setting since our model is good at learning the text-to-text alignment and the subword-level alignment is more helpful to the translation quality 
Experiments ::: Discussion ::: Ablation Study
To better understand the contribution of each component, we perform an ablation study on subword-level experiments  The results are shown in Table TABREF37  In `-MT noise' setting, we do not add noise to source sentences for MT  In `-weight sharing' setting, we use different parameters in CTC classification layer and source embedding layer  These two experiments prove that both weight sharing and using noisy MT input benefit to the final translation quality  Performance degrades more in `-weight sharing', indicating the semantic consistency contributes more to our model  In the `-pretrain' experiment, we remove the pre-training stage and directly update the model on three tasks, leading to a dramatic decrease on BLEU score, indicating the pre-training is an indispensable step for end-to-end ST 
Experiments ::: Discussion ::: Learning Curve
It is interesting to investigate why our method is superior to baselines  We find that TCEN achieves a higher final result owing to a better start-point in fine-tuning  Figure FIGREF39 provides learning curves of subword accuracy on validation set  The x-axis denotes the fine-tuning training steps  The vanilla model starts at a low accuracy, because its networks are not pre-trained on the ASR and MT data  The trends of our model and `many-to-many+pretrain' are similar, but our model outperforms it about five points in the whole fine-tuning process  It indicates that the gain comes from bridging the gap between pre-training and fine-tuning rather than a better fine-tuning process 
Experiments ::: Discussion ::: Compared with a Cascaded System
Table TABREF29 compares our model with end-to-end baselines  Here, we compare our model with cascaded systems  We build a cascaded system by combining the ASR model and MT model used in pre-training baseline  Word error rate (WER) of the ASR system and BLEU score of the MT system are reported in the supplementary material  In addition to a simple combination of the ASR and MT systems, we also re-segment the ASR outputs before feeding to the MT system, denoted as cascaded+re-seg  Specifically, we train a seq2seq model BIBREF19 on the MT dataset, where the source side is a no punctuation sentence and the target side is a natural sentence  After that, we use the seq2seq model to add sentence boundaries and punctuation on ASR outputs  Experimental results are shown in Table TABREF41  Our end-to-end model outperforms the simple cascaded model over 2 BLEU scores, and it achieves a comparable performance with the cascaded model combining with a sentence re-segment model 
Related Work
Early works conduct speech translation in a pipeline manner BIBREF2, BIBREF20, where the ASR output lattices are fed into an MT system to generate target sentences  HMM BIBREF21, DenseNet BIBREF22, TDNN BIBREF23 are commonly used ASR systems, while RNN with attention BIBREF19 and Transformer BIBREF10 are top choices for MT  To enhance the robustness of the NMT model towards ASR errors, BIBREF24 DBLP:conf/eacl/TsvetkovMD14 and BIBREF25 DBLP:conf/asru/ChenHHL17 propose to simulate the noise in training and inference 
To avoid error propagation and high latency issues, recent works propose translating the acoustic speech into text in target language without yielding the source transcription BIBREF4  Since ST data is scarce, pre-training BIBREF7, multi-task learning BIBREF4, BIBREF6, curriculum learning BIBREF26, attention-passing BIBREF27, and knowledge distillation BIBREF28, BIBREF29 strategies have been explored to utilize ASR data and MT data  Specifically, BIBREF5 DBLP:conf/interspeech/WeissCJWC17 show improvements of performance by training the ST model jointly with the ASR and the MT model  BIBREF6 berard2018end observe faster convergence and better results due to pre-training and multi-task learning on a larger dataset  BIBREF7 DBLP:conf/naacl/BansalKLLG19 show that pre-training a speech encoder on one language can improve ST quality on a different source language  All of them follow the traditional multi-task training strategies  BIBREF26 DBLP:journals/corr/abs-1802-06003 propose to use curriculum learning to improve ST performance on syntactically distant language pairs  To effectively leverage transcriptions in ST data, BIBREF18 DBLP:conf/naacl/AnastasopoulosC18 augment the multi-task model where the target decoder receives information from the source decoder and they show improvements on low-resource speech translation  Their model just consumes ASR and ST data, in contrast, our work sufficiently utilizes the large-scale MT data to capture the rich semantic knowledge  BIBREF30 DBLP:conf/icassp/JiaJMWCCALW19 use pre-trained MT and text-to-speech (TTS) synthesis models to convert weakly supervised data into ST pairs and demonstrate that an end-to-end MT model can be trained using only synthesised data 
Conclusion
This paper has investigated the end-to-end method for ST  It has discussed why there is a huge gap between pre-training and fine-tuning in previous methods  To alleviate these issues, we have proposed a method, which is capable of reusing every sub-net and keeping the role of sub-net consistent between pre-training and fine-tuning  Empirical studies have demonstrated that our model significantly outperforms baselines 

ground_truth: the model is pre-trained on CTC-based ASR task and MT task in the pre-training stage.