prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. What are two datasets model is applied to?
Context 1: Introduction
“Ché saetta previsa vien più lenta ”
– Dante Alighieri, Divina Commedia, Paradiso
Antisocial behavior is a persistent problem plaguing online conversation platforms; it is both widespread BIBREF0 and potentially damaging to mental and emotional health BIBREF1, BIBREF2  The strain this phenomenon puts on community maintainers has sparked recent interest in computational approaches for assisting human moderators 
Prior work in this direction has largely focused on post-hoc identification of various kinds of antisocial behavior, including hate speech BIBREF3, BIBREF4, harassment BIBREF5, personal attacks BIBREF6, and general toxicity BIBREF7  The fact that these approaches only identify antisocial content after the fact limits their practicality as tools for assisting pre-emptive moderation in conversational domains 
Addressing this limitation requires forecasting the future derailment of a conversation based on early warning signs, giving the moderators time to potentially intervene before any harm is done (BIBREF8 BIBREF8, BIBREF9 BIBREF9, see BIBREF10 BIBREF10 for a discussion)  Such a goal recognizes derailment as emerging from the development of the conversation, and belongs to the broader area of conversational forecasting, which includes future-prediction tasks such as predicting the eventual length of a conversation BIBREF11, whether a persuasion attempt will eventually succeed BIBREF12, BIBREF13, BIBREF14, whether team discussions will eventually lead to an increase in performance BIBREF15, or whether ongoing counseling conversations will eventually be perceived as helpful BIBREF16 
Approaching such conversational forecasting problems, however, requires overcoming several inherent modeling challenges  First, conversations are dynamic and their outcome might depend on how subsequent comments interact with each other  Consider the example in Figure FIGREF2: while no individual comment is outright offensive, a human reader can sense a tension emerging from their succession (e g , dismissive answers to repeated questioning)  Thus a forecasting model needs to capture not only the content of each individual comment, but also the relations between comments  Previous work has largely relied on hand-crafted features to capture such relations—e g , similarity between comments BIBREF16, BIBREF12 or conversation structure BIBREF17, BIBREF18—, though neural attention architectures have also recently shown promise BIBREF19 
The second modeling challenge stems from the fact that conversations have an unknown horizon: they can be of varying lengths, and the to-be-forecasted event can occur at any time  So when is it a good time to make a forecast? Prior work has largely proposed two solutions, both resulting in important practical limitations  One solution is to assume (unrealistic) prior knowledge of when the to-be-forecasted event takes place and extract features up to that point BIBREF20, BIBREF8  Another compromising solution is to extract features from a fixed-length window, often at the start of the conversation BIBREF21, BIBREF15, BIBREF16, BIBREF9  Choosing a catch-all window-size is however impractical: short windows will miss information in comments they do not encompass (e g , a window of only two comments would miss the chain of repeated questioning in comments 3 through 6 of Figure FIGREF2), while longer windows risk missing the to-be-forecasted event altogether if it occurs before the end of the window, which would prevent early detection 
In this work we introduce a model for forecasting conversational events that overcomes both these inherent challenges by processing comments, and their relations, as they happen (i e , in an online fashion)  Our main insight is that models with these properties already exist, albeit geared toward generation rather than prediction: recent work in context-aware dialog generation (or “chatbots”) has proposed sequential neural models that make effective use of the intra-conversational dynamics BIBREF22, BIBREF23, BIBREF24, while concomitantly being able to process the conversation as it develops (see BIBREF25 for a survey) 
In order for these systems to perform well in the generative domain they need to be trained on massive amounts of (unlabeled) conversational data  The main difficulty in directly adapting these models to the supervised domain of conversational forecasting is the relative scarcity of labeled data: for most forecasting tasks, at most a few thousands labeled examples are available, insufficient for the notoriously data-hungry sequential neural models 
To overcome this difficulty, we propose to decouple the objective of learning a neural representation of conversational dynamics from the objective of predicting future events  The former can be pre-trained on large amounts of unsupervised data, similarly to how chatbots are trained  The latter can piggy-back on the resulting representation after fine-tuning it for classification using relatively small labeled data  While similar pre-train-then-fine-tune approaches have recently achieved state-of-the-art performance in a number of NLP tasks—including natural language inference, question answering, and commonsense reasoning (discussed in Section SECREF2)—to the best of our knowledge this is the first attempt at applying this paradigm to conversational forecasting 
To test the effectiveness of this new architecture in forecasting derailment of online conversations, we develop and distribute two new datasets  The first triples in size the highly curated `Conversations Gone Awry' dataset BIBREF9, where civil-starting Wikipedia Talk Page conversations are crowd-labeled according to whether they eventually lead to personal attacks; the second relies on in-the-wild moderation of the popular subreddit ChangeMyView, where the aim is to forecast whether a discussion will later be subject to moderator action due to “rude or hostile” behavior  In both datasets, our model outperforms existing fixed-window approaches, as well as simpler sequential baselines that cannot account for inter-comment relations  Furthermore, by virtue of its online processing of the conversation, our system can provide substantial prior notice of upcoming derailment, triggering on average 3 comments (or 3 hours) before an overtly toxic comment is posted 
To summarize, in this work we:
introduce the first model for forecasting conversational events that can capture the dynamics of a conversation as it develops;
build two diverse datasets (one entirely new, one extending prior work) for the task of forecasting derailment of online conversations;
compare the performance of our model against the current state-of-the-art, and evaluate its ability to provide early warning signs 
Our work is motivated by the goal of assisting human moderators of online communities by preemptively signaling at-risk conversations that might deserve their attention  However, we caution that any automated systems might encode or even amplify the biases existing in the training data BIBREF26, BIBREF27, BIBREF28, so a public-facing implementation would need to be exhaustively scrutinized for such biases BIBREF29 
Further Related Work
Antisocial behavior  Antisocial behavior online comes in many forms, including harassment BIBREF30, cyberbullying BIBREF31, and general aggression BIBREF32  Prior work has sought to understand different aspects of such behavior, including its effect on the communities where it happens BIBREF33, BIBREF34, the actors involved BIBREF35, BIBREF36, BIBREF37, BIBREF38 and connections to the outside world BIBREF39 
Post-hoc classification of conversations  There is a rich body of prior work on classifying the outcome of a conversation after it has concluded, or classifying conversational events after they happened  Many examples exist, but some more closely related to our present work include identifying the winner of a debate BIBREF40, BIBREF41, BIBREF42, identifying successful negotiations BIBREF21, BIBREF43, as well as detecting whether deception BIBREF44, BIBREF45, BIBREF46 or disagreement BIBREF47, BIBREF48, BIBREF49, BIBREF50, BIBREF51 has occurred 
Our goal is different because we wish to forecast conversational events before they happen and while the conversation is still ongoing (potentially allowing for interventions)  Note that some post-hoc tasks can also be re-framed as forecasting tasks (assuming the existence of necessary labels); for instance, predicting whether an ongoing conversation will eventually spark disagreement BIBREF18, rather than detecting already-existing disagreement 
Conversational forecasting  As described in Section SECREF1, prior work on forecasting conversational outcomes and events has largely relied on hand-crafted features to capture aspects of conversational dynamics  Example feature sets include statistical measures based on similarity between utterances BIBREF16, sentiment imbalance BIBREF20, flow of ideas BIBREF20, increase in hostility BIBREF8, reply rate BIBREF11 and graph representations of conversations BIBREF52, BIBREF17  By contrast, we aim to automatically learn neural representations of conversational dynamics through pre-training 
Such hand-crafted features are typically extracted from fixed-length windows of the conversation, leaving unaddressed the problem of unknown horizon  While some work has trained multiple models for different window-lengths BIBREF8, BIBREF18, they consider these models to be independent and, as such, do not address the issue of aggregating them into a single forecast (i e , deciding at what point to make a prediction)  We implement a simple sliding windows solution as a baseline (Section SECREF5) 
Pre-training for NLP  The use of pre-training for natural language tasks has been growing in popularity after recent breakthroughs demonstrating improved performance on a wide array of benchmark tasks BIBREF53, BIBREF54  Existing work has generally used a language modeling objective as the pre-training objective; examples include next-word prediction BIBREF55, sentence autoencoding, BIBREF56, and machine translation BIBREF57  BERT BIBREF58 introduces a variation on this in which the goal is to predict the next sentence in a document given the current sentence  Our pre-training objective is similar in spirit, but operates at a conversation level, rather than a document level  We hence view our objective as conversational modeling rather than (only) language modeling  Furthermore, while BERT's sentence prediction objective is framed as a multiple-choice task, our objective is framed as a generative task 
Derailment Datasets
We consider two datasets, representing related but slightly different forecasting tasks  The first dataset is an expanded version of the annotated Wikipedia conversations dataset from BIBREF9  This dataset uses carefully-controlled crowdsourced labels, strictly filtered to ensure the conversations are civil up to the moment of a personal attack  This is a useful property for the purposes of model analysis, and hence we focus on this as our primary dataset  However, we are conscious of the possibility that these strict labels may not fully capture the kind of behavior that moderators care about in practice  We therefore introduce a secondary dataset, constructed from the subreddit ChangeMyView (CMV) that does not use post-hoc annotations  Instead, the prediction task is to forecast whether the conversation will be subject to moderator action in the future 
Wikipedia data  BIBREF9's `Conversations Gone Awry' dataset consists of 1,270 conversations that took place between Wikipedia editors on publicly accessible talk pages  The conversations are sourced from the WikiConv dataset BIBREF59 and labeled by crowdworkers as either containing a personal attack from within (i e , hostile behavior by one user in the conversation directed towards another) or remaining civil throughout 
A series of controls are implemented to prevent models from picking up on trivial correlations  To prevent models from capturing topic-specific information (e g , political conversations are more likely to derail), each attack-containing conversation is paired with a clean conversation from the same talk page, where the talk page serves as a proxy for topic  To force models to actually capture conversational dynamics rather than detecting already-existing toxicity, human annotations are used to ensure that all comments preceding a personal attack are civil 
To the ends of more effective model training, we elected to expand the `Conversations Gone Awry' dataset, using the original annotation procedure  Since we found that the original data skewed towards shorter conversations, we focused this crowdsourcing run on longer conversations: ones with 4 or more comments preceding the attack  Through this additional crowdsourcing, we expand the dataset to 4,188 conversations, which we are publicly releasing as part of the Cornell Conversational Analysis Toolkit (ConvoKit) 
We perform an 80-20-20 train/dev/test split, ensuring that paired conversations end up in the same split in order to preserve the topic control  Finally, we randomly sample another 1 million conversations from WikiConv to use for the unsupervised pre-training of the generative component 
Reddit CMV data  The CMV dataset is constructed from conversations collected via the Reddit API  In contrast to the Wikipedia-based dataset, we explicitly avoid the use of post-hoc annotation  Instead, we use as our label whether a conversation eventually had a comment removed by a moderator for violation of Rule 2: “Don't be rude or hostile to other users” 
Though the lack of post-hoc annotation limits the degree to which we can impose controls on the data (e g , some conversations may contain toxic comments not flagged by the moderators) we do reproduce as many of the Wikipedia data's controls as we can  Namely, we replicate the topic control pairing by choosing pairs of positive and negative examples that belong to the same top-level post, following BIBREF12; and enforce that the removed comment was made by a user who was previously involved in the conversation  This process results in 6,842 conversations, to which we again apply a pair-preserving 80-20-20 split  Finally, we gather over 600,000 conversations that do not include any removed comment, for unsupervised pre-training 
Conversational Forecasting Model
We now describe our general model for forecasting future conversational events  Our model integrates two components: (a) a generative dialog model that learns to represent conversational dynamics in an unsupervised fashion; and (b) a supervised component that fine-tunes this representation to forecast future events  Figure FIGREF13 provides an overview of the proposed architecture, henceforth CRAFT (Conversational Recurrent Architecture for ForecasTing) 
Terminology  For modeling purposes, we treat a conversation as a sequence of $N$ comments $C = \lbrace c_1,\dots ,c_N\rbrace $  Each comment, in turn, is a sequence of tokens, where the number of tokens may vary from comment to comment  For the $n$-th comment ($1 \le n \le N)$, we let $M_n$ denote the number of tokens  Then, a comment $c_n$ can be represented as a sequence of $M_n$ tokens: $c_n = \lbrace w_1,\dots ,w_{M_n}\rbrace $ 
Generative component  For the generative component of our model, we use a hierarchical recurrent encoder-decoder (HRED) architecture BIBREF60, a modified version of the popular sequence-to-sequence (seq2seq) architecture BIBREF61 designed to account for dependencies between consecutive inputs  BIBREF23 showed that HRED can successfully model conversational context by encoding the temporal structure of previously seen comments, making it an ideal fit for our use case  Here, we provide a high-level summary of the HRED architecture, deferring deeper technical discussion to BIBREF60 and BIBREF23 
An HRED dialog model consists of three components: an utterance encoder, a context encoder, and a decoder  The utterance encoder is responsible for generating semantic vector representations of comments  It consists of a recurrent neural network (RNN) that reads a comment token-by-token, and on each token $w_m$ updates a hidden state $h^{\text{enc}}$ based on the current token and the previous hidden state:
where $f^{\text{RNN}}$ is a nonlinear gating function (our implementation uses GRU BIBREF62)  The final hidden state $h^{\text{enc}}_M$ can be viewed as a vector encoding of the entire comment 
Running the encoder on each comment $c_n$ results in a sequence of $N$ vector encodings  A second encoder, the context encoder, is then run over this sequence:
Each hidden state $h^{\text{con}}_n$ can then be viewed as an encoding of the full conversational context up to and including the $n$-th comment  To generate a response to comment $n$, the context encoding $h^{\text{con}}_n$ is used to initialize the hidden state $h^{\text{dec}}_{0}$ of a decoder RNN  The decoder produces a response token by token using the following recurrence:
where $f^{\text{out}}$ is some function that outputs a probability distribution over words; we implement this using a simple feedforward layer  In our implementation, we further augment the decoder with attention BIBREF63, BIBREF64 over context encoder states to help capture long-term inter-comment dependencies  This generative component can be pre-trained using unlabeled conversational data 
Prediction component  Given a pre-trained HRED dialog model, we aim to extend the model to predict from the conversational context whether the to-be-forecasted event will occur  Our predictor consists of a multilayer perceptron (MLP) with 3 fully-connected layers, leaky ReLU activations between layers, and sigmoid activation for output  For each comment $c_n$, the predictor takes as input the context encoding $h^{\text{con}}_n$ and forwards it through the MLP layers, resulting in an output score that is interpreted as a probability $p_{\text{event}}(c_{n+1})$ that the to-be-forecasted event will happen (e g , that the conversation will derail) 
Training the predictive component starts by initializing the weights of the encoders to the values learned in pre-training  The main training loop then works as follows: for each positive sample—i e , a conversation containing an instance of the to-be-forecasted event (e g , derailment) at comment $c_e$—we feed the context $c_1,\dots ,c_{e-1}$ through the encoder and classifier, and compute cross-entropy loss between the classifier output and expected output of 1  Similarly, for each negative sample—i e , a conversation where none of the comments exhibit the to-be-forecasted event and that ends with $c_N$—we feed the context $c_1,\dots ,c_{N-1}$ through the model and compute loss against an expected output of 0 
Note that the parameters of the generative component are not held fixed during this process; instead, backpropagation is allowed to go all the way through the encoder layers  This process, known as fine-tuning, reshapes the representation learned during pre-training to be more directly useful to prediction BIBREF55 
We implement the model and training code using PyTorch, and we are publicly releasing our implementation and the trained models together with the data as part of ConvoKit 
Forecasting Derailment
We evaluate the performance of CRAFT in the task of forecasting conversational derailment in both the Wikipedia and CMV scenarios  To this end, for each of these datasets we pre-train the generative component on the unlabeled portion of the data and fine-tune it on the labeled training split (data size detailed in Section SECREF3) 
In order to evaluate our sequential system against conversational-level ground truth, we need to aggregate comment level predictions  If any comment in the conversation triggers a positive prediction—i e , $p_{\text{event}}(c_{n+1})$ is greater than a threshold learned on the development split—then the respective conversation is predicted to derail  If this forecast is triggered in a conversation that actually derails, but before the derailment actually happens, then the conversation is counted as a true positive; otherwise it is a false positive  If no positive predictions are triggered for a conversation, but it actually derails then it counts as a false negative; if it does not derail then it is a true negative 
Fixed-length window baselines  We first seek to compare CRAFT to existing, fixed-length window approaches to forecasting  To this end, we implement two such baselines: Awry, which is the state-of-the-art method proposed in BIBREF9 based on pragmatic features in the first comment-reply pair, and BoW, a simple bag-of-words baseline that makes a prediction using TF-IDF weighted bag-of-words features extracted from the first comment-reply pair 
Online forecasting baselines  Next, we consider simpler approaches for making forecasts as the conversations happen (i e , in an online fashion)  First, we propose Cumulative BoW, a model that recomputes bag-of-words features on all comments seen thus far every time a new comment arrives  While this approach does exhibit the desired behavior of producing updated predictions for each new comment, it fails to account for relationships between comments 
This simple cumulative approach cannot be directly extended to models whose features are strictly based on a fixed number of comments, like Awry  An alternative is to use a sliding window: for a feature set based on a window of $W$ comments, upon each new comment we can extract features from a window containing that comment and the $W-1$ comments preceding it  We apply this to the Awry method and call this model Sliding Awry  For both these baselines, we aggregate comment-level predictions in the same way as in our main model 
CRAFT ablations  Finally, we consider two modified versions of the CRAFT model in order to evaluate the impact of two of its key components: (1) the pre-training step, and (2) its ability to capture inter-comment dependencies through its hierarchical memory 
To evaluate the impact of pre-training, we train the prediction component of CRAFT on only the labeled training data, without first pre-training the encoder layers with the unlabeled data  We find that given the relatively small size of labeled data, this baseline fails to successfully learn, and ends up performing at the level of random guessing  This result underscores the need for the pre-training step that can make use of unlabeled data 
To evaluate the impact of the hierarchical memory, we implement a simplified version of CRAFT where the memory size of the context encoder is zero (CRAFT $-$ CE), thus effectively acting as if the pre-training component is a vanilla seq2seq model  In other words, this model cannot capture inter-comment dependencies, and instead at each step makes a prediction based only on the utterance encoding of the latest comment 
Results  Table TABREF17 compares CRAFT to the baselines on the test splits (random baseline is 50%) and illustrates several key findings  First, we find that unsurprisingly, accounting for full conversational context is indeed helpful, with even the simple online baselines outperforming the fixed-window baselines  On both datasets, CRAFT outperforms all baselines (including the other online models) in terms of accuracy and F1  Furthermore, although it loses on precision (to CRAFT $-$ CE) and recall (to Cumulative BoW) individually on the Wikipedia data, CRAFT has the superior balance between the two, having both a visibly higher precision-recall curve and larger area under the curve (AUPR) than the baselines (Figure FIGREF20)  This latter property is particularly useful in a practical setting, as it allows moderators to tune model performance to some desired precision without having to sacrifice as much in the way of recall (or vice versa) compared to the baselines and pre-existing solutions 
Analysis
We now examine the behavior of CRAFT in greater detail, to better understand its benefits and limitations  We specifically address the following questions: (1) How much early warning does the the model provide? (2) Does the model actually learn an order-sensitive representation of conversational context?
Early warning, but how early? The recent interest in forecasting antisocial behavior has been driven by a desire to provide pre-emptive, actionable warning to moderators  But does our model trigger early enough for any such practical goals?
For each personal attack correctly forecasted by our model, we count the number of comments elapsed between the time the model is first triggered and the attack  Figure FIGREF22 shows the distribution of these counts: on average, the model warns of an attack 3 comments before it actually happens (4 comments for CMV)  To further evaluate how much time this early warning would give to the moderator, we also consider the difference in timestamps between the comment where the model first triggers and the comment containing the actual attack  Over 50% of conversations get at least 3 hours of advance warning (2 hours for CMV)  Moreover, 39% of conversations get at least 12 hours of early warning before they derail 
Does order matter? One motivation behind the design of our model was the intuition that comments in a conversation are not independent events; rather, the order in which they appear matters (e g , a blunt comment followed by a polite one feels intuitively different from a polite comment followed by a blunt one)  By design, CRAFT has the capacity to learn an order-sensitive representation of conversational context, but how can we know that this capacity is actually used? It is conceivable that the model is simply computing an order-insensitive “bag-of-features”  Neural network models are notorious for their lack of transparency, precluding an analysis of how exactly CRAFT models conversational context  Nevertheless, through two simple exploratory experiments, we seek to show that it does not completely ignore comment order 
The first experiment for testing whether the model accounts for comment order is a prefix-shuffling experiment, visualized in Figure FIGREF23  For each conversation that the model predicts will derail, let $t$ denote the index of the triggering comment, i e , the index where the model first made a derailment forecast  We then construct synthetic conversations by taking the first $t-1$ comments (henceforth referred to as the prefix) and randomizing their order  Finally, we count how often the model no longer predicts derailment at index $t$ in the synthetic conversations  If the model were ignoring comment order, its prediction should remain unchanged (as it remains for the Cumulative BoW baseline), since the actual content of the first $t$ comments has not changed (and CRAFT inference is deterministic)  We instead find that in roughly one fifth of cases (12% for CMV) the model changes its prediction on the synthetic conversations  This suggests that CRAFT learns an order-sensitive representation of context, not a mere “bag-of-features” 
To more concretely quantify how much this order-sensitive context modeling helps with prediction, we can actively prevent the model from learning and exploiting any order-related dynamics  We achieve this through another type of shuffling experiment, where we go back even further and shuffle the comment order in the conversations used for pre-training, fine-tuning and testing  This procedure preserves the model's ability to capture signals present within the individual comments processed so far, as the utterance encoder is unaffected, but inhibits it from capturing any meaningful order-sensitive dynamics  We find that this hurts the model's performance (65% accuracy for Wikipedia, 59 5% for CMV), lowering it to a level similar to that of the version where we completely disable the context encoder 
Taken together, these experiments provide evidence that CRAFT uses its capacity to model conversational context in an order-sensitive fashion, and that it makes effective use of the dynamics within  An important avenue for future work would be developing more transparent models that can shed light on exactly what kinds of order-related features are being extracted and how they are used in prediction 
Conclusions and Future Work
In this work, we introduced a model for forecasting conversational events that processes comments as they happen and takes the full conversational context into account to make an updated prediction at each step  This model fills a void in the existing literature on conversational forecasting, simultaneously addressing the dual challenges of capturing inter-comment dynamics and dealing with an unknown horizon  We find that our model achieves state-of-the-art performance on the task of forecasting derailment in two different datasets that we release publicly  We further show that the resulting system can provide substantial prior notice of derailment, opening up the potential for preemptive interventions by human moderators BIBREF65 
While we have focused specifically on the task of forecasting derailment, we view this work as a step towards a more general model for real-time forecasting of other types of emergent properties of conversations  Follow-up work could adapt the CRAFT architecture to address other forecasting tasks mentioned in Section SECREF2—including those for which the outcome is extraneous to the conversation  We expect different tasks to be informed by different types of inter-comment dynamics, and further architecture extensions could add additional supervised fine-tuning in order to direct it to focus on specific dynamics that might be relevant to the task (e g , exchange of ideas between interlocutors or stonewalling) 
With respect to forecasting derailment, there remain open questions regarding what human moderators actually desire from an early-warning system, which would affect the design of a practical system based on this work  For instance, how early does a warning need to be in order for moderators to find it useful? What is the optimal balance between precision, recall, and false positive rate at which such a system is truly improving moderator productivity rather than wasting their time through false positives? What are the ethical implications of such a system? Follow-up work could run a user study of a prototype system with actual moderators to address these questions 
A practical limitation of the current analysis is that it relies on balanced datasets, while derailment is a relatively rare event for which a more restrictive trigger threshold would be appropriate  While our analysis of the precision-recall curve suggests the system is robust across multiple thresholds ($AUPR=0 7$), additional work is needed to establish whether the recall tradeoff would be acceptable in practice 
Finally, one major limitation of the present work is that it assigns a single label to each conversation: does it derail or not? In reality, derailment need not spell the end of a conversation; it is possible that a conversation could get back on track, suffer a repeat occurrence of antisocial behavior, or any number of other trajectories  It would be exciting to consider finer-grained forecasting of conversational trajectories, accounting for the natural—and sometimes chaotic—ebb-and-flow of human interactions 
Acknowledgements  We thank Caleb Chiam, Liye Fu, Lillian Lee, Alexandru Niculescu-Mizil, Andrew Wang and Justine Zhang for insightful conversations (with unknown horizon), Aditya Jha for his great help with implementing and running the crowd-sourcing tasks, Thomas Davidson and Claire Liang for exploratory data annotation, as well as the anonymous reviewers for their helpful comments  This work is supported in part by the NSF CAREER award IIS-1750615 and by the NSF Grant SES-1741441 
Context 2: Introduction
Machine translation has made remarkable progress, and studies claiming it to reach a human parity are starting to appear BIBREF0  However, when evaluating translations of the whole documents rather than isolated sentences, human raters show a stronger preference for human over machine translation BIBREF1  These findings emphasize the need to shift towards context-aware machine translation both from modeling and evaluation perspective 
Most previous work on context-aware NMT assumed that either all the bilingual data is available at the document level BIBREF2, BIBREF3, BIBREF4, BIBREF5, BIBREF6, BIBREF7, BIBREF8, BIBREF9, BIBREF10 or at least its fraction BIBREF11  But in practical scenarios, document-level parallel data is often scarce, which is one of the challenges when building a context-aware system 
We introduce an approach to context-aware machine translation using only monolingual document-level data  In our setting, a separate monolingual sequence-to-sequence model (DocRepair) is used to correct sentence-level translations of adjacent sentences  The key idea is to use monolingual data to imitate typical inconsistencies between context-agnostic translations of isolated sentences  The DocRepair model is trained to map inconsistent groups of sentences into consistent ones  The consistent groups come from the original training data; the inconsistent groups are obtained by sampling round-trip translations for each isolated sentence 
To validate the performance of our model, we use three kinds of evaluation: the BLEU score, contrastive evaluation of translation of several discourse phenomena BIBREF11, and human evaluation  We show strong improvements for all metrics 
We analyze which discourse phenomena are hard to capture using monolingual data only  Using contrastive test sets for targeted evaluation of several contextual phenomena, we compare the performance of the models trained on round-trip translations and genuine document-level parallel data  Among the four phenomena in the test sets we use (deixis, lexical cohesion, VP ellipsis and ellipsis which affects NP inflection) we find VP ellipsis to be the hardest phenomenon to be captured using round-trip translations 
Our key contributions are as follows:
we introduce the first approach to context-aware machine translation using only monolingual document-level data;
our approach shows substantial improvements in translation quality as measured by BLEU, targeted contrastive evaluation of several discourse phenomena and human evaluation;
we show which discourse phenomena are hard to capture using monolingual data only 
Our Approach: Document-level Repair
We propose a monolingual DocRepair model to correct inconsistencies between sentence-level translations of a context-agnostic MT system  It does not use any states of a trained MT model whose outputs it corrects and therefore can in principle be trained to correct translations from any black-box MT system 
The DocRepair model requires only monolingual document-level data in the target language  It is a monolingual sequence-to-sequence model that maps inconsistent groups of sentences into consistent ones  Consistent groups come from monolingual document-level data  To obtain inconsistent groups, each sentence in a group is replaced with its round-trip translation produced in isolation from context  More formally, forming a training minibatch for the DocRepair model involves the following steps (see also Figure FIGREF9):
sample several groups of sentences from the monolingual data;
for each sentence in a group, (i) translate it using a target-to-source MT model, (ii) sample a translation of this back-translated sentence in the source language using a source-to-target MT model;
using these round-trip translations of isolated sentences, form an inconsistent version of the initial groups;
use inconsistent groups as input for the DocRepair model, consistent ones as output 
At test time, the process of getting document-level translations is two-step (Figure FIGREF10):
produce translations of isolated sentences using a context-agnostic MT model;
apply the DocRepair model to a sequence of context-agnostic translations to correct inconsistencies between translations 
In the scope of the current work, the DocRepair model is the standard sequence-to-sequence Transformer  Sentences in a group are concatenated using a reserved token-separator between sentences  The Transformer is trained to correct these long inconsistent pseudo-sentences into consistent ones  The token-separator is then removed from corrected translations 
Evaluation of Contextual Phenomena
We use contrastive test sets for evaluation of discourse phenomena for English-Russian by BIBREF11  These test sets allow for testing different kinds of phenomena which, as we show, can be captured from monolingual data with varying success  In this section, we provide test sets statistics and briefly describe the tested phenomena  For more details, the reader is referred to BIBREF11 
Evaluation of Contextual Phenomena ::: Test sets
There are four test sets in the suite  Each test set contains contrastive examples  It is specifically designed to test the ability of a system to adapt to contextual information and handle the phenomenon under consideration  Each test instance consists of a true example (a sequence of sentences and their reference translation from the data) and several contrastive translations which differ from the true one only in one specific aspect  All contrastive translations are correct and plausible translations at the sentence level, and only context reveals the inconsistencies between them  The system is asked to score each candidate translation, and we compute the system accuracy as the proportion of times the true translation is preferred to the contrastive ones  Test set statistics are shown in Table TABREF15  The suites for deixis and lexical cohesion are split into development and test sets, with 500 examples from each used for validation purposes and the rest for testing  Convergence of both consistency scores on these development sets and BLEU score on a general development set are used as early stopping criteria in models training  For ellipsis, there is no dedicated development set, so we evaluate on all the ellipsis data and do not use it for development 
Evaluation of Contextual Phenomena ::: Phenomena overview
Deixis Deictic words or phrases, are referential expressions whose denotation depends on context  This includes personal deixis (“I”, “you”), place deixis (“here”, “there”), and discourse deixis, where parts of the discourse are referenced (“that's a good question”)  The test set examples are all related to person deixis, specifically the T-V distinction between informal and formal you (Latin “tu” and “vos”) in the Russian translations, and test for consistency in this respect 
Ellipsis Ellipsis is the omission from a clause of one or more words that are nevertheless understood in the context of the remaining elements  In machine translation, elliptical constructions in the source language pose a problem in two situations  First, if the target language does not allow the same types of ellipsis, requiring the elided material to be predicted from context  Second, if the elided material affects the syntax of the sentence  For example, in Russian the grammatical function of a noun phrase, and thus its inflection, may depend on the elided verb, or, conversely, the verb inflection may depend on the elided subject 
There are two different test sets for ellipsis  One contains examples where a morphological form of a noun group in the last sentence can not be understood without context beyond the sentence level (“ellipsis (infl )” in Table TABREF15)  Another includes cases of verb phrase ellipsis in English, which does not exist in Russian, thus requires predicting the verb when translating into Russian (“ellipsis (VP)” in Table TABREF15) 
Lexical cohesion The test set focuses on reiteration of named entities  Where several translations of a named entity are possible, a model has to prefer consistent translations over inconsistent ones 
Experimental Setup ::: Data preprocessing
We use the publicly available OpenSubtitles2018 corpus BIBREF12 for English and Russian  For a fair comparison with previous work, we train the baseline MT system on the data released by BIBREF11  Namely, our MT system is trained on 6m instances  These are sentence pairs with a relative time overlap of subtitle frames between source and target language subtitles of at least $0 9$ 
We gathered 30m groups of 4 consecutive sentences as our monolingual data  We used only documents not containing groups of sentences from general development and test sets as well as from contrastive test sets  The main results we report are for the model trained on all 30m fragments 
We use the tokenization provided by the corpus and use multi-bleu perl on lowercased data to compute BLEU score  We use beam search with a beam of 4 
Sentences were encoded using byte-pair encoding BIBREF13, with source and target vocabularies of about 32000 tokens  Translation pairs were batched together by approximate sequence length  Each training batch contained a set of translation pairs containing approximately 15000 source tokens  It has been shown that Transformer's performance depends heavily on batch size BIBREF14, and we chose a large batch size to ensure the best performance  In training context-aware models, for early stopping we use both convergence in BLEU score on the general development set and scores on the consistency development sets  After training, we average the 5 latest checkpoints 
Experimental Setup ::: Models
The baseline model, the model used for back-translation, and the DocRepair model are all Transformer base models BIBREF15  More precisely, the number of layers is $N=6$ with $h = 8$ parallel attention layers, or heads  The dimensionality of input and output is $d_{model} = 512$, and the inner-layer of a feed-forward networks has dimensionality $d_{ff}=2048$  We use regularization as described in BIBREF15 
As a second baseline, we use the two-pass CADec model BIBREF11  The first pass produces sentence-level translations  The second pass takes both the first-pass translation and representations of the context sentences as input and returns contextualized translations  CADec requires document-level parallel training data, while DocRepair only needs monolingual training data 
Experimental Setup ::: Generating round-trip translations
On the selected 6m instances we train sentence-level translation models in both directions  To create training data for DocRepair, we proceed as follows  The Russian monolingual data is first translated into English, using the Russian$\rightarrow $English model and beam search with beam size of 4  Then, we use the English$\rightarrow $Russian model to sample translations with temperature of $0{ }5$  For each sentence, we precompute 20 sampled translations and randomly choose one of them when forming a training minibatch for DocRepair  Also, in training, we replace each token in the input with a random one with the probability of $10\%$ 
Experimental Setup ::: Optimizer
As in BIBREF15, we use the Adam optimizer BIBREF16, the parameters are $\beta _1 = 0{ }9$, $\beta _2 = 0{ }98$ and $\varepsilon = 10^{-9}$  We vary the learning rate over the course of training using the formula:
where $warmup\_steps = 16000$ and $scale=4$ 
Results ::: General results
The BLEU scores are provided in Table TABREF24 (we evaluate translations of 4-sentence fragments)  To see which part of the improvement is due to fixing agreement between sentences rather than simply sentence-level post-editing, we train the same repair model at the sentence level  Each sentence in a group is now corrected separately, then they are put back together in a group  One can see that most of the improvement comes from accounting for extra-sentential dependencies  DocRepair outperforms the baseline and CADec by 0 7 BLEU, and its sentence-level repair version by 0 5 BLEU 
Results ::: Consistency results
Scores on the phenomena test sets are provided in Table TABREF26  For deixis, lexical cohesion and ellipsis (infl
Context 3: Introduction
In the kitchen, we increasingly rely on instructions from cooking websites: recipes  A cook with a predilection for Asian cuisine may wish to prepare chicken curry, but may not know all necessary ingredients apart from a few basics  These users with limited knowledge cannot rely on existing recipe generation approaches that focus on creating coherent recipes given all ingredients and a recipe name BIBREF0  Such models do not address issues of personal preference (e g  culinary tastes, garnish choices) and incomplete recipe details  We propose to approach both problems via personalized generation of plausible, user-specific recipes using user preferences extracted from previously consumed recipes 
Our work combines two important tasks from natural language processing and recommender systems: data-to-text generation BIBREF1 and personalized recommendation BIBREF2  Our model takes as user input the name of a specific dish, a few key ingredients, and a calorie level  We pass these loose input specifications to an encoder-decoder framework and attend on user profiles—learned latent representations of recipes previously consumed by a user—to generate a recipe personalized to the user's tastes  We fuse these `user-aware' representations with decoder output in an attention fusion layer to jointly determine text generation  Quantitative (perplexity, user-ranking) and qualitative analysis on user-aware model outputs confirm that personalization indeed assists in generating plausible recipes from incomplete ingredients 
While personalized text generation has seen success in conveying user writing styles in the product review BIBREF3, BIBREF4 and dialogue BIBREF5 spaces, we are the first to consider it for the problem of recipe generation, where output quality is heavily dependent on the content of the instructions—such as ingredients and cooking techniques 
To summarize, our main contributions are as follows:
We explore a new task of generating plausible and personalized recipes from incomplete input specifications by leveraging historical user preferences;
We release a new dataset of 180K+ recipes and 700K+ user reviews for this task;
We introduce new evaluation strategies for generation quality in instructional texts, centering on quantitative measures of coherence  We also show qualitatively and quantitatively that personalized models generate high-quality and specific recipes that align with historical user preferences 
Related Work
Large-scale transformer-based language models have shown surprising expressivity and fluency in creative and conditional long-text generation BIBREF6, BIBREF7  Recent works have proposed hierarchical methods that condition on narrative frameworks to generate internally consistent long texts BIBREF8, BIBREF9, BIBREF10  Here, we generate procedurally structured recipes instead of free-form narratives 
Recipe generation belongs to the field of data-to-text natural language generation BIBREF1, which sees other applications in automated journalism BIBREF11, question-answering BIBREF12, and abstractive summarization BIBREF13, among others  BIBREF14, BIBREF15 model recipes as a structured collection of ingredient entities acted upon by cooking actions  BIBREF0 imposes a `checklist' attention constraint emphasizing hitherto unused ingredients during generation  BIBREF16 attend over explicit ingredient references in the prior recipe step  Similar hierarchical approaches that infer a full ingredient list to constrain generation will not help personalize recipes, and would be infeasible in our setting due to the potentially unconstrained number of ingredients (from a space of 10K+) in a recipe  We instead learn historical preferences to guide full recipe generation 
A recent line of work has explored user- and item-dependent aspect-aware review generation BIBREF3, BIBREF4  This work is related to ours in that it combines contextual language generation with personalization  Here, we attend over historical user preferences from previously consumed recipes to generate recipe content, rather than writing styles 
Approach
Our model's input specification consists of: the recipe name as a sequence of tokens, a partial list of ingredients, and a caloric level (high, medium, low)  It outputs the recipe instructions as a token sequence: $\mathcal {W}_r=\lbrace w_{r,0}, \dots , w_{r,T}\rbrace $ for a recipe $r$ of length $T$  To personalize output, we use historical recipe interactions of a user $u \in \mathcal {U}$ 
Encoder: Our encoder has three embedding layers: vocabulary embedding $\mathcal {V}$, ingredient embedding $\mathcal {I}$, and caloric-level embedding $\mathcal {C}$  Each token in the (length $L_n$) recipe name is embedded via $\mathcal {V}$; the embedded token sequence is passed to a two-layered bidirectional GRU (BiGRU) BIBREF17, which outputs hidden states for names $\lbrace \mathbf {n}_{\text{enc},j} \in \mathbb {R}^{2d_h}\rbrace $, with hidden size $d_h$  Similarly each of the $L_i$ input ingredients is embedded via $\mathcal {I}$, and the embedded ingredient sequence is passed to another two-layered BiGRU to output ingredient hidden states as $\lbrace \mathbf {i}_{\text{enc},j} \in \mathbb {R}^{2d_h}\rbrace $  The caloric level is embedded via $\mathcal {C}$ and passed through a projection layer with weights $W_c$ to generate calorie hidden representation $\mathbf {c}_{\text{enc}} \in \mathbb {R}^{2d_h}$ 
Ingredient Attention: We apply attention BIBREF18 over the encoded ingredients to use encoder outputs at each decoding time step  We define an attention-score function $\alpha $ with key $K$ and query $Q$:
with trainable weights $W_{\alpha }$, bias $\mathbf {b}_{\alpha }$, and normalization term $Z$  At decoding time $t$, we calculate the ingredient context $\mathbf {a}_{t}^{i} \in \mathbb {R}^{d_h}$ as:
Decoder: The decoder is a two-layer GRU with hidden state $h_t$ conditioned on previous hidden state $h_{t-1}$ and input token $w_{r, t}$ from the original recipe text  We project the concatenated encoder outputs as the initial decoder hidden state:
To bias generation toward user preferences, we attend over a user's previously reviewed recipes to jointly determine the final output token distribution  We consider two different schemes to model preferences from user histories: (1) recipe interactions, and (2) techniques seen therein (defined in data)  BIBREF19, BIBREF20, BIBREF21 explore similar schemes for personalized recommendation 
Prior Recipe Attention: We obtain the set of prior recipes for a user $u$: $R^+_u$, where each recipe can be represented by an embedding from a recipe embedding layer $\mathcal {R}$ or an average of the name tokens embedded by $\mathcal {V}$  We attend over the $k$-most recent prior recipes, $R^{k+}_u$, to account for temporal drift of user preferences BIBREF22  These embeddings are used in the `Prior Recipe' and `Prior Name' models, respectively 
Given a recipe representation $\mathbf {r} \in \mathbb {R}^{d_r}$ (where $d_r$ is recipe- or vocabulary-embedding size depending on the recipe representation) the prior recipe attention context $\mathbf {a}_{t}^{r_u}$ is calculated as
Prior Technique Attention: We calculate prior technique preference (used in the `Prior Tech` model) by normalizing co-occurrence between users and techniques seen in $R^+_u$, to obtain a preference vector $\rho _{u}$  Each technique $x$ is embedded via a technique embedding layer $\mathcal {X}$ to $\mathbf {x}\in \mathbb {R}^{d_x}$  Prior technique attention is calculated as
where, inspired by copy mechanisms BIBREF23, BIBREF24, we add $\rho _{u,x}$ for technique $x$ to emphasize the attention by the user's prior technique preference 
Attention Fusion Layer: We fuse all contexts calculated at time $t$, concatenating them with decoder GRU output and previous token embedding:
We then calculate the token probability:
and maximize the log-likelihood of the generated sequence conditioned on input specifications and user preferences  fig:ex shows a case where the Prior Name model attends strongly on previously consumed savory recipes to suggest the usage of an additional ingredient (`cilantro')

ground_truth:  `Conversations Gone Awry' dataset, subreddit ChangeMyView

An expanded version of the existing 'Conversations Gone Awry' dataset and the ChangeMyView dataset, a subreddit whose only annotation is whether the conversation required action by the Reddit moderators. 

prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. what was the baseline?
Context 2: ) we see substantial improvements over both the baseline and CADec  The largest improvement over CADec (22 5 percentage points) is for lexical cohesion  However, there is a drop of almost 5 percentage points for VP ellipsis  We hypothesize that this is because it is hard to learn to correct inconsistencies in translations caused by VP ellipsis relying on monolingual data alone  Figure FIGREF27(a) shows an example of inconsistency caused by VP ellipsis in English  There is no VP ellipsis in Russian, and when translating auxiliary “did” the model has to guess the main verb  Figure FIGREF27(b) shows steps of generating round-trip translations for the target side of the previous example  When translating from Russian, main verbs are unlikely to be translated as the auxiliary “do” in English, and hence the VP ellipsis is rarely present on the English side  This implies the model trained using the round-trip translations will not be exposed to many VP ellipsis examples in training  We discuss this further in Section SECREF34 
Table TABREF28 provides scores for deixis and lexical cohesion separately for different distances between sentences requiring consistency  It can be seen, that the performance of DocRepair degrades less than that of CADec when the distance between sentences requiring consistency gets larger 
Results ::: Human evaluation
We conduct a human evaluation on random 700 examples from our general test set  We picked only examples where a DocRepair translation is not a full copy of the baseline one 
The annotators were provided an original group of sentences in English and two translations: baseline context-agnostic one and the one corrected by the DocRepair model  Translations were presented in random order with no indication which model they came from  The task is to pick one of the three options: (1) the first translation is better, (2) the second translation is better, (3) the translations are of equal quality  The annotators were asked to avoid the third answer if they are able to give preference to one of the translations  No other guidelines were given 
The results are provided in Table TABREF30  In about $52\%$ of the cases annotators marked translations as having equal quality  Among the cases where one of the translations was marked better than the other, the DocRepair translation was marked better in $73\%$ of the cases  This shows a strong preference of the annotators for corrected translations over the baseline ones 
Varying Training Data
In this section, we discuss the influence of the training data chosen for document-level models  In all experiments, we used the DocRepair model 
Varying Training Data ::: The amount of training data
Table TABREF33 provides BLEU and consistency scores for the DocRepair model trained on different amount of data  We see that even when using a dataset of moderate size (e g , 5m fragments) we can achieve performance comparable to the model trained on a large amount of data (30m fragments)  Moreover, we notice that deixis scores are less sensitive to the amount of training data than lexical cohesion and ellipsis scores  The reason might be that, as we observed in our previous work BIBREF11, inconsistencies in translations due to the presence of deictic words and phrases are more frequent in this dataset than other types of inconsistencies  Also, as we show in Section SECREF7, this is the phenomenon the model learns faster in training 
Varying Training Data ::: One-way vs round-trip translations
In this section, we discuss the limitations of using only monolingual data to model inconsistencies between sentence-level translations  In Section SECREF25 we observed a drop in performance on VP ellipsis for DocRepair compared to CADec, which was trained on parallel data  We hypothesized that this is due to the differences between one-way and round-trip translations, and now we test this hypothesis  To do so, we fix the dataset and vary the way in which the input for DocRepair is generated: round-trip or one-way translations  The latter assumes that document-level data is parallel, and translations are sampled from the source side of the sentences in a group rather than from their back-translations  For parallel data, we take 1 5m parallel instances which were used for CADec training and add 1m instances from our monolingual data  For segments in the parallel part, we either sample translations from the source side or use round-trip translations  The results are provided in Table TABREF35 
The model trained on one-way translations is slightly better than the one trained on round-trip translations  As expected, VP ellipsis is the hardest phenomena to be captured using round-trip translations, and the DocRepair model trained on one-way translated data gains 6% accuracy on this test set  This shows that the DocRepair model benefits from having access to non-synthetic English data  This results in exposing DocRepair at training time to Russian translations which suffer from the same inconsistencies as the ones it will have to correct at test time 
Varying Training Data ::: Filtering: monolingual (no filtering) or parallel
Note that the scores of the DocRepair model trained on 2 5m instances randomly chosen from monolingual data (Table TABREF33) are different from the ones for the model trained on 2 5m instances combined from parallel and monolingual data (Table TABREF35)  For convenience, we show these two in Table TABREF36 
The domain, the dataset these two data samples were gathered from, and the way we generated training data for DocRepair (round-trip translations) are all the same  The only difference lies in how the data was filtered  For parallel data, as in the previous work BIBREF6, we picked only sentence pairs with large relative time overlap of subtitle frames between source-language and target-language subtitles  This is necessary to ensure the quality of translation data: one needs groups of consecutive sentences in the target language where every sentence has a reliable translation 
Table TABREF36 shows that the quality of the model trained on data which came from the parallel part is worse than the one trained on monolingual data  This indicates that requiring each sentence in a group to have a reliable translation changes the distribution of the data, which might be not beneficial for translation quality and provides extra motivation for using monolingual data 
Learning Dynamics
Let us now look into how the process of DocRepair training progresses  Figure FIGREF38 shows how the BLEU scores with the reference translation and with the baseline context-agnostic translation (i e  the input for the DocRepair model) are changing during training  First, the model quickly learns to copy baseline translations: the BLEU score with the baseline is very high  Then it gradually learns to change them, which leads to an improvement in BLEU with the reference translation and a drop in BLEU with the baseline  Importantly, the model is reluctant to make changes: the BLEU score between translations of the converged model and the baseline is 82 5  We count the number of changed sentences in every 4-sentence fragment in the test set and plot the histogram in Figure FIGREF38  In over than 20$\%$ of the cases the model has not changed base translations at all  In almost $40\%$, it modified only one sentence and left the remaining 3 sentences unchanged  The model changed more than half sentences in a group in only $14\%$ of the cases  Several examples of the DocRepair translations are shown in Figure FIGREF43 
Figure FIGREF42 shows how consistency scores are changing in training  For deixis, the model achieves the final quality quite quickly; for the rest, it needs a large number of training steps to converge 
Related Work
Our work is most closely related to two lines of research: automatic post-editing (APE) and document-level machine translation 
Related Work ::: Automatic post-editing
Our model can be regarded as an automatic post-editing system – a system designed to fix systematic MT errors that is decoupled from the main MT system  Automatic post-editing has a long history, including rule-based BIBREF17, statistical BIBREF18 and neural approaches BIBREF19, BIBREF20, BIBREF21 
In terms of architectures, modern approaches use neural sequence-to-sequence models, either multi-source architectures that consider both the original source and the baseline translation BIBREF19, BIBREF20, or monolingual repair systems, as in BIBREF21, which is concurrent work to ours  True post-editing datasets are typically small and expensive to create BIBREF22, hence synthetic training data has been created that uses original monolingual data as output for the sequence-to-sequence model, paired with an automatic back-translation BIBREF23 and/or round-trip translation as its input(s) BIBREF19, BIBREF21 
While previous work on automatic post-editing operated on the sentence level, the main novelty of this work is that our DocRepair model operates on groups of sentences and is thus able to fix consistency errors caused by the context-agnostic baseline MT system  We consider this strategy of sentence-level baseline translation and context-aware monolingual repair attractive when parallel document-level data is scarce 
For training, the DocRepair model only requires monolingual document-level data  While we create synthetic training data via round-trip translation similarly to earlier work BIBREF19, BIBREF21, note that we purposefully use sentence-level MT systems for this to create the types of consistency errors that we aim to fix with the context-aware DocRepair model  Not all types of consistency errors that we want to fix emerge from a round-trip translation, so access to parallel document-level data can be useful (Section SECREF34) 
Related Work ::: Document-level NMT
Neural models of MT that go beyond the sentence-level are an active research area BIBREF2, BIBREF3, BIBREF4, BIBREF5, BIBREF6, BIBREF7, BIBREF8, BIBREF10, BIBREF9, BIBREF11  Typically, the main MT system is modified to take additional context as its input  One limitation of these approaches is that they assume that parallel document-level training data is available 
Closest to our work are two-pass models for document-level NMT BIBREF24, BIBREF11, where a second, context-aware model takes the translation and hidden representations of the sentence-level first-pass model as its input  The second-pass model can in principle be trained on a subset of the parallel training data BIBREF11, somewhat relaxing the assumption that all training data is at the document level 
Our work is different from this previous work in two main respects  Firstly, we show that consistency can be improved with only monolingual document-level training data  Secondly, the DocRepair model is decoupled from the first-pass MT system, which improves its portability 
Conclusions
We introduce the first approach to context-aware machine translation using only monolingual document-level data  We propose a monolingual DocRepair model to correct inconsistencies between sentence-level translations  The model performs automatic post-editing on a sequence of sentence-level translations, refining translations of sentences in context of each other  Our approach results in substantial improvements in translation quality as measured by BLEU, targeted contrastive evaluation of several discourse phenomena and human evaluation  Moreover, we perform error analysis and detect which discourse phenomena are hard to capture using only monolingual document-level data  While in the current work we used text fragments of 4 sentences, in future work we would like to consider longer contexts 
Acknowledgments
We would like to thank the anonymous reviewers for their comments  The authors also thank David Talbot and Yandex Machine Translation team for helpful discussions and inspiration  Ivan Titov acknowledges support of the European Research Council (ERC StG BroadSem 678254) and the Dutch National Science Foundation (NWO VIDI 639 022 518)  Rico Sennrich acknowledges support from the Swiss National Science Foundation (105212_169888), the European Union’s Horizon 2020 research and innovation programme (grant agreement no 825460), and the Royal Society (NAF\R1\180122) 
Context 3: 
Recipe Dataset: Food com
We collect a novel dataset of 230K+ recipe texts and 1M+ user interactions (reviews) over 18 years (2000-2018) from Food com  Here, we restrict to recipes with at least 3 steps, and at least 4 and no more than 20 ingredients  We discard users with fewer than 4 reviews, giving 180K+ recipes and 700K+ reviews, with splits as in tab:recipeixnstats 
Our model must learn to generate from a diverse recipe space: in our training data, the average recipe length is 117 tokens with a maximum of 256  There are 13K unique ingredients across all recipes  Rare words dominate the vocabulary: 95% of words appear $<$100 times, accounting for only 1 65% of all word usage  As such, we perform Byte-Pair Encoding (BPE) tokenization BIBREF25, BIBREF26, giving a training vocabulary of 15K tokens across 19M total mentions  User profiles are similarly diverse: 50% of users have consumed $\le $6 recipes, while 10% of users have consumed $>$45 recipes 
We order reviews by timestamp, keeping the most recent review for each user as the test set, the second most recent for validation, and the remainder for training (sequential leave-one-out evaluation BIBREF27)  We evaluate only on recipes not in the training set 
We manually construct a list of 58 cooking techniques from 384 cooking actions collected by BIBREF15; the most common techniques (bake, combine, pour, boil) account for 36 5% of technique mentions  We approximate technique adherence via string match between the recipe text and technique list 
Experiments and Results
For training and evaluation, we provide our model with the first 3-5 ingredients listed in each recipe  We decode recipe text via top-$k$ sampling BIBREF7, finding $k=3$ to produce satisfactory results  We use a hidden size $d_h=256$ for both the encoder and decoder  Embedding dimensions for vocabulary, ingredient, recipe, techniques, and caloric level are 300, 10, 50, 50, and 5 (respectively)  For prior recipe attention, we set $k=20$, the 80th %-ile for the number of user interactions  We use the Adam optimizer BIBREF28 with a learning rate of $10^{-3}$, annealed with a decay rate of 0 9 BIBREF29  We also use teacher-forcing BIBREF30 in all training epochs 
In this work, we investigate how leveraging historical user preferences can improve generation quality over strong baselines in our setting  We compare our personalized models against two baselines  The first is a name-based Nearest-Neighbor model (NN)  We initially adapted the Neural Checklist Model of BIBREF0 as a baseline; however, we ultimately use a simple Encoder-Decoder baseline with ingredient attention (Enc-Dec), which provides comparable performance and lower complexity  All personalized models outperform baseline in BPE perplexity (tab:metricsontest) with Prior Name performing the best  While our models exhibit comparable performance to baseline in BLEU-1/4 and ROUGE-L, we generate more diverse (Distinct-1/2: percentage of distinct unigrams and bigrams) and acceptable recipes  BLEU and ROUGE are not the most appropriate metrics for generation quality  A `correct' recipe can be written in many ways with the same main entities (ingredients)  As BLEU-1/4 capture structural information via n-gram matching, they are not correlated with subjective recipe quality  This mirrors observations from BIBREF31, BIBREF8 
We observe that personalized models make more diverse recipes than baseline  They thus perform better in BLEU-1 with more key entities (ingredient mentions) present, but worse in BLEU-4, as these recipes are written in a personalized way and deviate from gold on the phrasal level  Similarly, the `Prior Name' model generates more unigram-diverse recipes than other personalized models and obtains a correspondingly lower BLEU-1 score 
Qualitative Analysis: We present sample outputs for a cocktail recipe in tab:samplerecipes, and additional recipes in the appendix  Generation quality progressively improves from generic baseline output to a blended cocktail produced by our best performing model  Models attending over prior recipes explicitly reference ingredients  The Prior Name model further suggests the addition of lemon and mint, which are reasonably associated with previously consumed recipes like coconut mousse and pork skewers 
Personalization: To measure personalization, we evaluate how closely the generated text corresponds to a particular user profile  We compute the likelihood of generated recipes using identical input specifications but conditioned on ten different user profiles—one `gold' user who consumed the original recipe, and nine randomly generated user profiles  Following BIBREF8, we expect the highest likelihood for the recipe conditioned on the gold user  We measure user matching accuracy (UMA)—the proportion where the gold user is ranked highest—and Mean Reciprocal Rank (MRR) BIBREF32 of the gold user  All personalized models beat baselines in both metrics, showing our models personalize generated recipes to the given user profiles  The Prior Name model achieves the best UMA and MRR by a large margin, revealing that prior recipe names are strong signals for personalization  Moreover, the addition of attention mechanisms to capture these signals improves language modeling performance over a strong non-personalized baseline

ground_truth:  MT system on the data released by BIBREF11

Transformer base, two-pass CADec model

prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. What were their results on the new dataset?
Context 3: 
Recipe Level Coherence: A plausible recipe should possess a coherent step order, and we evaluate this via a metric for recipe-level coherence  We use the neural scoring model from BIBREF33 to measure recipe-level coherence for each generated recipe  Each recipe step is encoded by BERT BIBREF34  Our scoring model is a GRU network that learns the overall recipe step ordering structure by minimizing the cosine similarity of recipe step hidden representations presented in the correct and reverse orders  Once pretrained, our scorer calculates the similarity of a generated recipe to the forward and backwards ordering of its corresponding gold label, giving a score equal to the difference between the former and latter  A higher score indicates better step ordering (with a maximum score of 2)  tab:coherencemetrics shows that our personalized models achieve average recipe-level coherence scores of 1 78-1 82, surpassing the baseline at 1 77 
Recipe Step Entailment: Local coherence is also crucial to a user following a recipe: it is crucial that subsequent steps are logically consistent with prior ones  We model local coherence as an entailment task: predicting the likelihood that a recipe step follows the preceding  We sample several consecutive (positive) and non-consecutive (negative) pairs of steps from each recipe  We train a BERT BIBREF34 model to predict the entailment score of a pair of steps separated by a [SEP] token, using the final representation of the [CLS] token  The step entailment score is computed as the average of scores for each set of consecutive steps in each recipe, averaged over every generated recipe for a model, as shown in tab:coherencemetrics 
Human Evaluation: We presented 310 pairs of recipes for pairwise comparison BIBREF8 (details in appendix) between baseline and each personalized model, with results shown in tab:metricsontest  On average, human evaluators preferred personalized model outputs to baseline 63% of the time, confirming that personalized attention improves the semantic plausibility of generated recipes  We also performed a small-scale human coherence survey over 90 recipes, in which 60% of users found recipes generated by personalized models to be more coherent and preferable to those generated by baseline models 
Conclusion
In this paper, we propose a novel task: to generate personalized recipes from incomplete input specifications and user histories  On a large novel dataset of 180K recipes and 700K reviews, we show that our personalized generative models can generate plausible, personalized, and coherent recipes preferred by human evaluators for consumption  We also introduce a set of automatic coherence measures for instructional texts as well as personalization metrics to support our claims  Our future work includes generating structured representations of recipes to handle ingredient properties, as well as accounting for references to collections of ingredients (e g  “dry mix") 
Acknowledgements  This work is partly supported by NSF #1750063  We thank all reviewers for their constructive suggestions, as well as Rei M , Sujoy P , Alicia L , Eric H , Tim S , Kathy C , Allen C , and Micah I  for their feedback 
Appendix ::: Food com: Dataset Details
Our raw data consists of 270K recipes and 1 4M user-recipe interactions (reviews) scraped from Food com, covering a period of 18 years (January 2000 to December 2018)  See tab:int-stats for dataset summary statistics, and tab:samplegk for sample information about one user-recipe interaction and the recipe involved 
Appendix ::: Generated Examples
See tab:samplechx for a sample recipe for chicken chili and tab:samplewaffle for a sample recipe for sweet waffles 
Human Evaluation
We prepared a set of 15 pairwise comparisons per evaluation session, and collected 930 pairwise evaluations (310 per personalized model) over 62 sessions  For each pair, users were given a partial recipe specification (name and 3-5 key ingredients), as well as two generated recipes labeled `A' and `B'  One recipe is generated from our baseline encoder-decoder model and one recipe is generated by one of our three personalized models (Prior Tech, Prior Name, Prior Recipe)  The order of recipe presentation (A/B) is randomly selected for each question  A screenshot of the user evaluation interface is given in fig:exeval  We ask the user to indicate which recipe they find more coherent, and which recipe best accomplishes the goal indicated by the recipe name  A screenshot of this survey interface is given in fig:exeval2 

ground_truth: average recipe-level coherence scores of 1.78-1.82, human evaluators preferred personalized model outputs to baseline 63% of the time