prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. by how much did their model improve?
Context 1: Introduction
Text simplification aims to reduce the lexical and structural complexity of a text, while still retaining the semantic meaning, which can help children, non-native speakers, and people with cognitive disabilities, to understand text better  One of the methods of automatic text simplification can be generally divided into three categories: lexical simplification (LS) BIBREF0 , BIBREF1 , rule-based BIBREF2 , and machine translation (MT) BIBREF3 , BIBREF4   LS is mainly used to simplify text by substituting infrequent and difficult words with frequent and easier words  However, there are several challenges for the LS approach: a great number of transformation rules are required for reasonable coverage and should be applied based on the specific context; third, the syntax and semantic meaning of the sentence is hard to retain  Rule-based approaches use hand-crafted rules for lexical and syntactic simplification, for example, substituting difficult words in a predefined vocabulary  However, such approaches need a lot of human-involvement to manually define these rules, and it is impossible to give all possible simplification rules  MT-based approach has attracted great attention in the last several years, which addresses text simplification as a monolingual machine translation problem translating from 'ordinary' and 'simplified' sentences 
In recent years, neural Machine Translation (NMT) is a newly-proposed deep learning approach and achieves very impressive results BIBREF5 , BIBREF6 , BIBREF7   Unlike the traditional phrased-based machine translation system which operates on small components separately, NMT system is being trained end-to-end, without the need to have external decoders, language models or phrase tables  Therefore, the existing architectures in NMT are used for text simplification BIBREF8 , BIBREF4   However, most recent work using NMT is limited to the training data that are scarce and expensive to build  Language models trained on simplified corpora have played a central role in statistical text simplification BIBREF9 , BIBREF10   One main reason is the amount of available simplified corpora typically far exceeds the amount of parallel data  The performance of models can be typically improved when trained on more data  Therefore, we expect simplified corpora to be especially helpful for NMT models 
In contrast to previous work, which uses the existing NMT models, we explore strategy to include simplified training corpora in the training process without changing the neural network architecture  We first propose to pair simplified training sentences with synthetic ordinary sentences during training, and treat this synthetic data as additional training data  We obtain synthetic ordinary sentences through back-translation, i e  an automatic translation of the simplified sentence into the ordinary sentence BIBREF11   Then, we mix the synthetic data into the original (simplified-ordinary) data to train NMT model  Experimental results on two publicly available datasets show that we can improve the text simplification quality of NMT models by mixing simplified sentences into the training set over NMT model only using the original training data 
Related Work
Automatic TS is a complicated natural language processing (NLP) task, which consists of lexical and syntactic simplification levels BIBREF12   It has attracted much attention recently as it could make texts more accessible to wider audiences, and used as a pre-processing step, improve performances of various NLP tasks and systems BIBREF13 , BIBREF14 , BIBREF15   Usually, hand-crafted, supervised, and unsupervised methods based on resources like English Wikipedia and Simple English Wikipedia (EW-SEW) BIBREF10 are utilized for extracting simplification rules  It is very easy to mix up the automatic TS task and the automatic summarization task BIBREF3 , BIBREF16 , BIBREF6   TS is different from text summarization as the focus of text summarization is to reduce the length and redundant content 
At the lexical level, lexical simplification systems often substitute difficult words using more common words, which only require a large corpus of regular text to obtain word embeddings to get words similar to the complex word BIBREF1 , BIBREF9   Biran et al  BIBREF0 adopted an unsupervised method for learning pairs of complex and simpler synonyms from a corpus consisting of Wikipedia and Simple Wikipedia  At the sentence level, a sentence simplification model was proposed by tree transformation based on statistical machine translation (SMT) BIBREF3   Woodsend and Lapata BIBREF17 presented a data-driven model based on a quasi-synchronous grammar, a formalism that can naturally capture structural mismatches and complex rewrite operations  Wubben et al  BIBREF18 proposed a phrase-based machine translation (PBMT) model that is trained on ordinary-simplified sentence pairs  Xu et al  BIBREF19 proposed a syntax-based machine translation model using simplification-specific objective functions and features to encourage simpler output 
Compared with SMT, neural machine translation (NMT) has shown to produce state-of-the-art results BIBREF5 , BIBREF7   The central approach of NMT is an encoder-decoder architecture implemented by recurrent neural networks, which can represent the input sequence as a vector, and then decode that vector into an output sequence  Therefore, NMT models were used for text simplification task, and achieved good results BIBREF8 , BIBREF4 , BIBREF20   The main limitation of the aforementioned NMT models for text simplification depended on the parallel ordinary-simplified sentence pairs  Because ordinary-simplified sentence pairs are expensive and time-consuming to build, the available largest data is EW-SEW that only have 296,402 sentence pairs  The dataset is insufficiency for NMT model if we want to NMT model can obtain the best parameters  Considering simplified data plays an important role in boosting fluency for phrase-based text simplification, and we investigate the use of simplified data for text simplification  We are the first to show that we can effectively adapt neural translation models for text simplifiation with simplified corpora 
Simplified Corpora
We collected a simplified dataset from Simple English Wikipedia that are freely available, which has been previously used for many text simplification methods BIBREF0 , BIBREF10 , BIBREF3   The simple English Wikipedia is pretty easy to understand than normal English Wikipedia  We downloaded all articles from Simple English Wikipedia  For these articles, we removed stubs, navigation pages and any article that consisted of a single sentence  We then split them into sentences with the Stanford CorNLP BIBREF21 , and deleted these sentences whose number of words are smaller than 10 or large than 40  After removing repeated sentences, we chose 600K sentences as the simplified data with 11 6M words, and the size of vocabulary is 82K 
Text Simplification using Neural Machine Translation
Our work is built on attention-based NMT BIBREF5 as an encoder-decoder network with recurrent neural networks (RNN), which simultaneously conducts dynamic alignment and generation of the target simplified sentence 
The encoder uses a bidirectional RNN that consists of forward and backward RNN  Given a source sentence INLINEFORM0 , the forward RNN and backward RNN calculate forward hidden states INLINEFORM1 and backward hidden states INLINEFORM2 , respectively  The annotation vector INLINEFORM3 is obtained by concatenating INLINEFORM4 and INLINEFORM5  
The decoder is a RNN that predicts a target simplificated sentence with Gated Recurrent Unit (GRU) BIBREF22   Given the previously generated target (simplified) sentence INLINEFORM0 , the probability of next target word INLINEFORM1 is DISPLAYFORM0
where INLINEFORM0 is a non-linear function, INLINEFORM1 is the embedding of INLINEFORM2 , and INLINEFORM3 is a decoding state for time step INLINEFORM4  
State INLINEFORM0 is calculated by DISPLAYFORM0
where INLINEFORM0 is the activation function GRU 
The INLINEFORM0 is the context vector computed as a weighted annotation INLINEFORM1 , computed by DISPLAYFORM0
where the weight INLINEFORM0 is computed by DISPLAYFORM0 DISPLAYFORM1
where INLINEFORM0 , INLINEFORM1 and INLINEFORM2 are weight matrices  The training objective is to maximize the likelihood of the training data  Beam search is employed for decoding 
Synthetic Simplified Sentences
We train an auxiliary system using NMT model from the simplified sentence to the ordinary sentence, which is first trained on the available parallel data  For leveraging simplified sentences to improve the quality of NMT model for text simplification, we propose to adapt the back-translation approach proposed by Sennrich et al  BIBREF11 to our scenario  More concretely, Given one sentence in simplified sentences, we use the simplified-ordinary system in translate mode with greedy decoding to translate it to the ordinary sentences, which is denoted as back-translation  This way, we obtain a synthetic parallel simplified-ordinary sentences  Both the synthetic sentences and the available parallel data are used as training data for the original NMT system 
Evaluation
We evaluate the performance of text simplification using neural machine translation on available parallel sentences and additional simplified sentences 
Dataset  We use two simplification datasets (WikiSmall and WikiLarge)  WikiSmall consists of ordinary and simplified sentences from the ordinary and simple English Wikipedias, which has been used as benchmark for evaluating text simplification BIBREF17 , BIBREF18 , BIBREF8   The training set has 89,042 sentence pairs, and the test set has 100 pairs  WikiLarge is also from Wikipedia corpus whose training set contains 296,402 sentence pairs BIBREF19 , BIBREF20   WikiLarge includes 8 (reference) simplifications for 2,359 sentences split into 2,000 for development and 359 for testing 
Metrics  Three metrics in text simplification are chosen in this paper  BLEU BIBREF5 is one traditional machine translation metric to assess the degree to which translated simplifications differed from reference simplifications  FKGL measures the readability of the output BIBREF23   A small FKGL represents simpler output  SARI is a recent text-simplification metric by comparing the output against the source and reference simplifications BIBREF20  
We evaluate the output of all systems using human evaluation  The metric is denoted as Simplicity BIBREF8   The three non-native fluent English speakers are shown reference sentences and output sentences  They are asked whether the output sentence is much simpler (+2), somewhat simpler (+1), equally (0), somewhat more difficult (-1), and much more difficult (-2) than the reference sentence 
Methods  We use OpenNMT BIBREF24 as the implementation of the NMT system for all experiments BIBREF5   We generally follow the default settings and training procedure described by Klein et al (2017)  We replace out-of-vocabulary words with a special UNK symbol  At prediction time, we replace UNK words with the highest probability score from the attention layer  OpenNMT system used on parallel data is the baseline system  To obtain a synthetic parallel training set, we back-translate a random sample of 100K sentences from the collected simplified corpora  OpenNMT used on parallel data and synthetic data is our model  The benchmarks are run on a Intel(R) Core(TM) i7-5930K CPU@3 50GHz, 32GB Mem, trained on 1 GPU GeForce GTX 1080 (Pascal) with CUDA v  8 0 
We choose three statistical text simplification systems  PBMT-R is a phrase-based method with a reranking post-processing step BIBREF18   Hybrid performs sentence splitting and deletion operations based on discourse representation structures, and then simplifies sentences with PBMT-R BIBREF25   SBMT-SARI BIBREF19 is syntax-based translation model using PPDB paraphrase database BIBREF26 and modifies tuning function (using SARI)  We choose two neural text simplification systems  NMT is a basic attention-based encoder-decoder model which uses OpenNMT framework to train with two LSTM layers, hidden states of size 500 and 500 hidden units, SGD optimizer, and a dropout rate of 0 3 BIBREF8   Dress is an encoder-decoder model coupled with a deep reinforcement learning framework, and the parameters are chosen according to the original paper BIBREF20   For the experiments with synthetic parallel data, we back-translate a random sample of 60 000 sentences from the collected simplified sentences into ordinary sentences  Our model is trained on synthetic data and the available parallel data, denoted as NMT+synthetic 
Results  Table 1 shows the results of all models on WikiLarge dataset  We can see that our method (NMT+synthetic) can obtain higher BLEU, lower FKGL and high SARI compared with other models, except Dress on FKGL and SBMT-SARI on SARI  It verified that including synthetic data during training is very effective, and yields an improvement over our baseline NMF by 2 11 BLEU, 1 7 FKGL and 1 07 SARI  We also substantially outperform Dress, who previously reported SOTA result  The results of our human evaluation using Simplicity are also presented in Table 1  NMT on synthetic data is significantly better than PBMT-R, Dress, and SBMT-SARI on Simplicity  It indicates that our method with simplified data is effective at creating simpler output 
Results on WikiSmall dataset are shown in Table 2  We see substantial improvements (6 37 BLEU) than NMT from adding simplified training data with synthetic ordinary sentences  Compared with statistical machine translation models (PBMT-R, Hybrid, SBMT-SARI), our method (NMT+synthetic) still have better results, but slightly worse FKGL and SARI  Similar to the results in WikiLarge, the results of our human evaluation using Simplicity outperforms the other models  In conclusion, Our method produces better results comparing with the baselines, which demonstrates the effectiveness of adding simplified training data 
Conclusion
In this paper, we propose one simple method to use simplified corpora during training of NMT systems, with no changes to the network architecture  In the experiments on two datasets, we achieve substantial gains in all tasks, and new SOTA results, via back-translation of simplified sentences into the ordinary sentences, and treating this synthetic data as additional training data  Because we do not change the neural network architecture to integrate simplified corpora, our method can be easily applied to other Neural Text Simplification (NTS) systems  We expect that the effectiveness of our method not only varies with the quality of the NTS system used for back-translation, but also depends on the amount of available parallel and simplified corpora  In the paper, we have only utilized data from Wikipedia for simplified sentences  In the future, many other text sources are available and the impact of not only size, but also of domain should be investigated 
Context 3: Introduction
In the kitchen, we increasingly rely on instructions from cooking websites: recipes  A cook with a predilection for Asian cuisine may wish to prepare chicken curry, but may not know all necessary ingredients apart from a few basics  These users with limited knowledge cannot rely on existing recipe generation approaches that focus on creating coherent recipes given all ingredients and a recipe name BIBREF0  Such models do not address issues of personal preference (e g  culinary tastes, garnish choices) and incomplete recipe details  We propose to approach both problems via personalized generation of plausible, user-specific recipes using user preferences extracted from previously consumed recipes 
Our work combines two important tasks from natural language processing and recommender systems: data-to-text generation BIBREF1 and personalized recommendation BIBREF2  Our model takes as user input the name of a specific dish, a few key ingredients, and a calorie level  We pass these loose input specifications to an encoder-decoder framework and attend on user profiles—learned latent representations of recipes previously consumed by a user—to generate a recipe personalized to the user's tastes  We fuse these `user-aware' representations with decoder output in an attention fusion layer to jointly determine text generation  Quantitative (perplexity, user-ranking) and qualitative analysis on user-aware model outputs confirm that personalization indeed assists in generating plausible recipes from incomplete ingredients 
While personalized text generation has seen success in conveying user writing styles in the product review BIBREF3, BIBREF4 and dialogue BIBREF5 spaces, we are the first to consider it for the problem of recipe generation, where output quality is heavily dependent on the content of the instructions—such as ingredients and cooking techniques 
To summarize, our main contributions are as follows:
We explore a new task of generating plausible and personalized recipes from incomplete input specifications by leveraging historical user preferences;
We release a new dataset of 180K+ recipes and 700K+ user reviews for this task;
We introduce new evaluation strategies for generation quality in instructional texts, centering on quantitative measures of coherence  We also show qualitatively and quantitatively that personalized models generate high-quality and specific recipes that align with historical user preferences 
Related Work
Large-scale transformer-based language models have shown surprising expressivity and fluency in creative and conditional long-text generation BIBREF6, BIBREF7  Recent works have proposed hierarchical methods that condition on narrative frameworks to generate internally consistent long texts BIBREF8, BIBREF9, BIBREF10  Here, we generate procedurally structured recipes instead of free-form narratives 
Recipe generation belongs to the field of data-to-text natural language generation BIBREF1, which sees other applications in automated journalism BIBREF11, question-answering BIBREF12, and abstractive summarization BIBREF13, among others  BIBREF14, BIBREF15 model recipes as a structured collection of ingredient entities acted upon by cooking actions  BIBREF0 imposes a `checklist' attention constraint emphasizing hitherto unused ingredients during generation  BIBREF16 attend over explicit ingredient references in the prior recipe step  Similar hierarchical approaches that infer a full ingredient list to constrain generation will not help personalize recipes, and would be infeasible in our setting due to the potentially unconstrained number of ingredients (from a space of 10K+) in a recipe  We instead learn historical preferences to guide full recipe generation 
A recent line of work has explored user- and item-dependent aspect-aware review generation BIBREF3, BIBREF4  This work is related to ours in that it combines contextual language generation with personalization  Here, we attend over historical user preferences from previously consumed recipes to generate recipe content, rather than writing styles 
Approach
Our model's input specification consists of: the recipe name as a sequence of tokens, a partial list of ingredients, and a caloric level (high, medium, low)  It outputs the recipe instructions as a token sequence: $\mathcal {W}_r=\lbrace w_{r,0}, \dots , w_{r,T}\rbrace $ for a recipe $r$ of length $T$  To personalize output, we use historical recipe interactions of a user $u \in \mathcal {U}$ 
Encoder: Our encoder has three embedding layers: vocabulary embedding $\mathcal {V}$, ingredient embedding $\mathcal {I}$, and caloric-level embedding $\mathcal {C}$  Each token in the (length $L_n$) recipe name is embedded via $\mathcal {V}$; the embedded token sequence is passed to a two-layered bidirectional GRU (BiGRU) BIBREF17, which outputs hidden states for names $\lbrace \mathbf {n}_{\text{enc},j} \in \mathbb {R}^{2d_h}\rbrace $, with hidden size $d_h$  Similarly each of the $L_i$ input ingredients is embedded via $\mathcal {I}$, and the embedded ingredient sequence is passed to another two-layered BiGRU to output ingredient hidden states as $\lbrace \mathbf {i}_{\text{enc},j} \in \mathbb {R}^{2d_h}\rbrace $  The caloric level is embedded via $\mathcal {C}$ and passed through a projection layer with weights $W_c$ to generate calorie hidden representation $\mathbf {c}_{\text{enc}} \in \mathbb {R}^{2d_h}$ 
Ingredient Attention: We apply attention BIBREF18 over the encoded ingredients to use encoder outputs at each decoding time step  We define an attention-score function $\alpha $ with key $K$ and query $Q$:
with trainable weights $W_{\alpha }$, bias $\mathbf {b}_{\alpha }$, and normalization term $Z$  At decoding time $t$, we calculate the ingredient context $\mathbf {a}_{t}^{i} \in \mathbb {R}^{d_h}$ as:
Decoder: The decoder is a two-layer GRU with hidden state $h_t$ conditioned on previous hidden state $h_{t-1}$ and input token $w_{r, t}$ from the original recipe text  We project the concatenated encoder outputs as the initial decoder hidden state:
To bias generation toward user preferences, we attend over a user's previously reviewed recipes to jointly determine the final output token distribution  We consider two different schemes to model preferences from user histories: (1) recipe interactions, and (2) techniques seen therein (defined in data)  BIBREF19, BIBREF20, BIBREF21 explore similar schemes for personalized recommendation 
Prior Recipe Attention: We obtain the set of prior recipes for a user $u$: $R^+_u$, where each recipe can be represented by an embedding from a recipe embedding layer $\mathcal {R}$ or an average of the name tokens embedded by $\mathcal {V}$  We attend over the $k$-most recent prior recipes, $R^{k+}_u$, to account for temporal drift of user preferences BIBREF22  These embeddings are used in the `Prior Recipe' and `Prior Name' models, respectively 
Given a recipe representation $\mathbf {r} \in \mathbb {R}^{d_r}$ (where $d_r$ is recipe- or vocabulary-embedding size depending on the recipe representation) the prior recipe attention context $\mathbf {a}_{t}^{r_u}$ is calculated as
Prior Technique Attention: We calculate prior technique preference (used in the `Prior Tech` model) by normalizing co-occurrence between users and techniques seen in $R^+_u$, to obtain a preference vector $\rho _{u}$  Each technique $x$ is embedded via a technique embedding layer $\mathcal {X}$ to $\mathbf {x}\in \mathbb {R}^{d_x}$  Prior technique attention is calculated as
where, inspired by copy mechanisms BIBREF23, BIBREF24, we add $\rho _{u,x}$ for technique $x$ to emphasize the attention by the user's prior technique preference 
Attention Fusion Layer: We fuse all contexts calculated at time $t$, concatenating them with decoder GRU output and previous token embedding:
We then calculate the token probability:
and maximize the log-likelihood of the generated sequence conditioned on input specifications and user preferences  fig:ex shows a case where the Prior Name model attends strongly on previously consumed savory recipes to suggest the usage of an additional ingredient (`cilantro') 
Recipe Dataset: Food com
We collect a novel dataset of 230K+ recipe texts and 1M+ user interactions (reviews) over 18 years (2000-2018) from Food com  Here, we restrict to recipes with at least 3 steps, and at least 4 and no more than 20 ingredients  We discard users with fewer than 4 reviews, giving 180K+ recipes and 700K+ reviews, with splits as in tab:recipeixnstats 
Our model must learn to generate from a diverse recipe space: in our training data, the average recipe length is 117 tokens with a maximum of 256  There are 13K unique ingredients across all recipes  Rare words dominate the vocabulary: 95% of words appear $<$100 times, accounting for only 1 65% of all word usage  As such, we perform Byte-Pair Encoding (BPE) tokenization BIBREF25, BIBREF26, giving a training vocabulary of 15K tokens across 19M total mentions  User profiles are similarly diverse: 50% of users have consumed $\le $6 recipes, while 10% of users have consumed $>$45 recipes 
We order reviews by timestamp, keeping the most recent review for each user as the test set, the second most recent for validation, and the remainder for training (sequential leave-one-out evaluation BIBREF27)  We evaluate only on recipes not in the training set 
We manually construct a list of 58 cooking techniques from 384 cooking actions collected by BIBREF15; the most common techniques (bake, combine, pour, boil) account for 36 5% of technique mentions  We approximate technique adherence via string match between the recipe text and technique list 
Experiments and Results
For training and evaluation, we provide our model with the first 3-5 ingredients listed in each recipe  We decode recipe text via top-$k$ sampling BIBREF7, finding $k=3$ to produce satisfactory results  We use a hidden size $d_h=256$ for both the encoder and decoder  Embedding dimensions for vocabulary, ingredient, recipe, techniques, and caloric level are 300, 10, 50, 50, and 5 (respectively)  For prior recipe attention, we set $k=20$, the 80th %-ile for the number of user interactions  We use the Adam optimizer BIBREF28 with a learning rate of $10^{-3}$, annealed with a decay rate of 0 9 BIBREF29  We also use teacher-forcing BIBREF30 in all training epochs

ground_truth: For the WikiLarge dataset, the improvement over baseline NMT is 2.11 BLEU, 1.7 FKGL and 1.07 SARI.
For the WikiSmall dataset, the improvement over baseline NMT is  8.37 BLEU.

6.37 BLEU

prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. What are the datasets used for evaluation?
Context 2: Introduction
Language model pretraining has advanced the state of the art in many NLP tasks ranging from sentiment analysis, to question answering, natural language inference, named entity recognition, and textual similarity  State-of-the-art pretrained models include ELMo BIBREF1, GPT BIBREF2, and more recently Bidirectional Encoder Representations from Transformers (Bert; BIBREF0)  Bert combines both word and sentence representations in a single very large Transformer BIBREF3; it is pretrained on vast amounts of text, with an unsupervised objective of masked language modeling and next-sentence prediction and can be fine-tuned with various task-specific objectives 
In most cases, pretrained language models have been employed as encoders for sentence- and paragraph-level natural language understanding problems BIBREF0 involving various classification tasks (e g , predicting whether any two sentences are in an entailment relationship; or determining the completion of a sentence among four alternative sentences)  In this paper, we examine the influence of language model pretraining on text summarization  Different from previous tasks, summarization requires wide-coverage natural language understanding going beyond the meaning of individual words and sentences  The aim is to condense a document into a shorter version while preserving most of its meaning  Furthermore, under abstractive modeling formulations, the task requires language generation capabilities in order to create summaries containing novel words and phrases not featured in the source text, while extractive summarization is often defined as a binary classification task with labels indicating whether a text span (typically a sentence) should be included in the summary 
We explore the potential of Bert for text summarization under a general framework encompassing both extractive and abstractive modeling paradigms  We propose a novel document-level encoder based on Bert which is able to encode a document and obtain representations for its sentences  Our extractive model is built on top of this encoder by stacking several inter-sentence Transformer layers to capture document-level features for extracting sentences  Our abstractive model adopts an encoder-decoder architecture, combining the same pretrained Bert encoder with a randomly-initialized Transformer decoder BIBREF3  We design a new training schedule which separates the optimizers of the encoder and the decoder in order to accommodate the fact that the former is pretrained while the latter must be trained from scratch  Finally, motivated by previous work showing that the combination of extractive and abstractive objectives can help generate better summaries BIBREF4, we present a two-stage approach where the encoder is fine-tuned twice, first with an extractive objective and subsequently on the abstractive summarization task 
We evaluate the proposed approach on three single-document news summarization datasets representative of different writing conventions (e g , important information is concentrated at the beginning of the document or distributed more evenly throughout) and summary styles (e g , verbose vs  more telegraphic; extractive vs  abstractive)  Across datasets, we experimentally show that the proposed models achieve state-of-the-art results under both extractive and abstractive settings  Our contributions in this work are three-fold: a) we highlight the importance of document encoding for the summarization task; a variety of recently proposed techniques aim to enhance summarization performance via copying mechanisms BIBREF5, BIBREF6, BIBREF7, reinforcement learning BIBREF8, BIBREF9, BIBREF10, and multiple communicating encoders BIBREF11  We achieve better results with a minimum-requirement model without using any of these mechanisms; b) we showcase ways to effectively employ pretrained language models in summarization under both extractive and abstractive settings; we would expect any improvements in model pretraining to translate in better summarization in the future; and c) the proposed models can be used as a stepping stone to further improve summarization performance as well as baselines against which new proposals are tested 
Background ::: Pretrained Language Models
Pretrained language models BIBREF1, BIBREF2, BIBREF0, BIBREF12, BIBREF13 have recently emerged as a key technology for achieving impressive gains in a wide variety of natural language tasks  These models extend the idea of word embeddings by learning contextual representations from large-scale corpora using a language modeling objective  Bidirectional Encoder Representations from Transformers (Bert; BIBREF0) is a new language representation model which is trained with a masked language modeling and a “next sentence prediction” task on a corpus of 3,300M words 
The general architecture of Bert is shown in the left part of Figure FIGREF2  Input text is first preprocessed by inserting two special tokens  [cls] is appended to the beginning of the text; the output representation of this token is used to aggregate information from the whole sequence (e g , for classification tasks)  And token [sep] is inserted after each sentence as an indicator of sentence boundaries  The modified text is then represented as a sequence of tokens $X=[w_1,w_2,\cdots ,w_n]$  Each token $w_i$ is assigned three kinds of embeddings: token embeddings indicate the meaning of each token, segmentation embeddings are used to discriminate between two sentences (e g , during a sentence-pair classification task) and position embeddings indicate the position of each token within the text sequence  These three embeddings are summed to a single input vector $x_i$ and fed to a bidirectional Transformer with multiple layers:
where $h^0=x$ are the input vectors; $\mathrm {LN}$ is the layer normalization operation BIBREF14; $\mathrm {MHAtt}$ is the multi-head attention operation BIBREF3; superscript $l$ indicates the depth of the stacked layer  On the top layer, Bert will generate an output vector $t_i$ for each token with rich contextual information 
Pretrained language models are usually used to enhance performance in language understanding tasks  Very recently, there have been attempts to apply pretrained models to various generation problems BIBREF15, BIBREF16  When fine-tuning for a specific task, unlike ELMo whose parameters are usually fixed, parameters in Bert are jointly fine-tuned with additional task-specific parameters 
Background ::: Extractive Summarization
Extractive summarization systems create a summary by identifying (and subsequently concatenating) the most important sentences in a document  Neural models consider extractive summarization as a sentence classification problem: a neural encoder creates sentence representations and a classifier predicts which sentences should be selected as summaries  SummaRuNNer BIBREF7 is one of the earliest neural approaches adopting an encoder based on Recurrent Neural Networks  Refresh BIBREF8 is a reinforcement learning-based system trained by globally optimizing the ROUGE metric  More recent work achieves higher performance with more sophisticated model structures  Latent BIBREF17 frames extractive summarization as a latent variable inference problem; instead of maximizing the likelihood of “gold” standard labels, their latent model directly maximizes the likelihood of human summaries given selected sentences  Sumo BIBREF18 capitalizes on the notion of structured attention to induce a multi-root dependency tree representation of the document while predicting the output summary  NeuSum BIBREF19 scores and selects sentences jointly and represents the state of the art in extractive summarization 
Background ::: Abstractive Summarization
Neural approaches to abstractive summarization conceptualize the task as a sequence-to-sequence problem, where an encoder maps a sequence of tokens in the source document $\mathbf {x} = [x_1,    , x_n]$ to a sequence of continuous representations $\mathbf {z} = [z_1,    , z_n]$, and a decoder then generates the target summary $\mathbf {y} = [y_1,    , y_m]$ token-by-token, in an auto-regressive manner, hence modeling the conditional probability: $p(y_1,    , y_m|x_1,    , x_n)$ 
BIBREF20 and BIBREF21 were among the first to apply the neural encoder-decoder architecture to text summarization  BIBREF6 enhance this model with a pointer-generator network (PTgen) which allows it to copy words from the source text, and a coverage mechanism (Cov) which keeps track of words that have been summarized  BIBREF11 propose an abstractive system where multiple agents (encoders) represent the document together with a hierarchical attention mechanism (over the agents) for decoding  Their Deep Communicating Agents (DCA) model is trained end-to-end with reinforcement learning  BIBREF9 also present a deep reinforced model (DRM) for abstractive summarization which handles the coverage problem with an intra-attention mechanism where the decoder attends over previously generated words  BIBREF4 follow a bottom-up approach (BottomUp); a content selector first determines which phrases in the source document should be part of the summary, and a copy mechanism is applied only to preselected phrases during decoding  BIBREF22 propose an abstractive model which is particularly suited to extreme summarization (i e , single sentence summaries), based on convolutional neural networks and additionally conditioned on topic distributions (TConvS2S) 
Fine-tuning Bert for Summarization ::: Summarization Encoder
Although Bert has been used to fine-tune various NLP tasks, its application to summarization is not as straightforward  Since Bert is trained as a masked-language model, the output vectors are grounded to tokens instead of sentences, while in extractive summarization, most models manipulate sentence-level representations  Although segmentation embeddings represent different sentences in Bert, they only apply to sentence-pair inputs, while in summarization we must encode and manipulate multi-sentential inputs  Figure FIGREF2 illustrates our proposed Bert architecture for Summarization (which we call BertSum) 
In order to represent individual sentences, we insert external [cls] tokens at the start of each sentence, and each [cls] symbol collects features for the sentence preceding it  We also use interval segment embeddings to distinguish multiple sentences within a document  For $sent_i$ we assign segment embedding $E_A$ or $E_B$ depending on whether $i$ is odd or even  For example, for document $[sent_1, sent_2, sent_3, sent_4, sent_5]$, we would assign embeddings $[E_A, E_B, E_A,E_B, E_A]$  This way, document representations are learned hierarchically where lower Transformer layers represent adjacent sentences, while higher layers, in combination with self-attention, represent multi-sentence discourse 
Position embeddings in the original Bert model have a maximum length of 512; we overcome this limitation by adding more position embeddings that are initialized randomly and fine-tuned with other parameters in the encoder 
Fine-tuning Bert for Summarization ::: Extractive Summarization
Let $d$ denote a document containing sentences $[sent_1, sent_2, \cdots , sent_m]$, where $sent_i$ is the $i$-th sentence in the document  Extractive summarization can be defined as the task of assigning a label $y_i \in \lbrace 0, 1\rbrace $ to each $sent_i$, indicating whether the sentence should be included in the summary  It is assumed that summary sentences represent the most important content of the document 
With BertSum, vector $t_i$ which is the vector of the $i$-th [cls] symbol from the top layer can be used as the representation for $sent_i$  Several inter-sentence Transformer layers are then stacked on top of Bert outputs, to capture document-level features for extracting summaries:
where $h^0=\mathrm {PosEmb}(T)$; $T$ denotes the sentence vectors output by BertSum, and function $\mathrm {PosEmb}$ adds sinusoid positional embeddings BIBREF3 to $T$, indicating the position of each sentence 
The final output layer is a sigmoid classifier:
where $h^L_i$ is the vector for $sent_i$ from the top layer (the $L$-th layer ) of the Transformer  In experiments, we implemented Transformers with $L=1, 2, 3$ and found that a Transformer with $L=2$ performed best  We name this model BertSumExt 
The loss of the model is the binary classification entropy of prediction $\hat{y}_i$ against gold label $y_i$  Inter-sentence Transformer layers are jointly fine-tuned with BertSum  We use the Adam optimizer with $\beta _1=0 9$, and $\beta _2=0 999$)  Our learning rate schedule follows BIBREF3 with warming-up ($ \operatorname{\operatorname{warmup}}=10,000$):
Fine-tuning Bert for Summarization ::: Abstractive Summarization
We use a standard encoder-decoder framework for abstractive summarization BIBREF6  The encoder is the pretrained BertSum and the decoder is a 6-layered Transformer initialized randomly  It is conceivable that there is a mismatch between the encoder and the decoder, since the former is pretrained while the latter must be trained from scratch  This can make fine-tuning unstable; for example, the encoder might overfit the data while the decoder underfits, or vice versa  To circumvent this, we design a new fine-tuning schedule which separates the optimizers of the encoder and the decoder 
We use two Adam optimizers with $\beta _1=0 9$ and $\beta _2=0 999$ for the encoder and the decoder, respectively, each with different warmup-steps and learning rates:
where $\tilde{lr}_{\mathcal {E}}=2e^{-3}$, and $\operatorname{\operatorname{warmup}}_{\mathcal {E}}=20,000$ for the encoder and $\tilde{lr}_{\mathcal {D}}=0 1$, and $\operatorname{\operatorname{warmup}}_{\mathcal {D}}=10,000$ for the decoder  This is based on the assumption that the pretrained encoder should be fine-tuned with a smaller learning rate and smoother decay (so that the encoder can be trained with more accurate gradients when the decoder is becoming stable) 
In addition, we propose a two-stage fine-tuning approach, where we first fine-tune the encoder on the extractive summarization task (Section SECREF8) and then fine-tune it on the abstractive summarization task (Section SECREF13)  Previous work BIBREF4, BIBREF23 suggests that using extractive objectives can boost the performance of abstractive summarization  Also notice that this two-stage approach is conceptually very simple, the model can take advantage of information shared between these two tasks, without fundamentally changing its architecture  We name the default abstractive model BertSumAbs and the two-stage fine-tuned model BertSumExtAbs 
Experimental Setup
In this section, we describe the summarization datasets used in our experiments and discuss various implementation details 
Experimental Setup ::: Summarization Datasets
We evaluated our model on three benchmark datasets, namely the CNN/DailyMail news highlights dataset BIBREF24, the New York Times Annotated Corpus (NYT; BIBREF25), and XSum BIBREF22  These datasets represent different summary styles ranging from highlights to very brief one sentence summaries  The summaries also vary with respect to the type of rewriting operations they exemplify (e g , some showcase more cut and paste operations while others are genuinely abstractive)  Table TABREF12 presents statistics on these datasets (test set); example (gold-standard) summaries are provided in the supplementary material 
Experimental Setup ::: Summarization Datasets ::: CNN/DailyMail
contains news articles and associated highlights, i e , a few bullet points giving a brief overview of the article  We used the standard splits of BIBREF24 for training, validation, and testing (90,266/1,220/1,093 CNN documents and 196,961/12,148/10,397 DailyMail documents)  We did not anonymize entities  We first split sentences with the Stanford CoreNLP toolkit BIBREF26 and pre-processed the dataset following BIBREF6  Input documents were truncated to 512 tokens 
Experimental Setup ::: Summarization Datasets ::: NYT
contains 110,540 articles with abstractive summaries  Following BIBREF27, we split these into 100,834/9,706 training/test examples, based on the date of publication (the test set contains all articles published from January 1, 2007 onward)  We used 4,000 examples from the training as validation set  We also followed their filtering procedure, documents with summaries less than 50 words were removed from the dataset  The filtered test set (NYT50) includes 3,452 examples  Sentences were split with the Stanford CoreNLP toolkit BIBREF26 and pre-processed following BIBREF27  Input documents were truncated to 800 tokens 
Experimental Setup ::: Summarization Datasets ::: XSum
contains 226,711 news articles accompanied with a one-sentence summary, answering the question “What is this article about?”  We used the splits of BIBREF22 for training, validation, and testing (204,045/11,332/11,334) and followed the pre-processing introduced in their work  Input documents were truncated to 512 tokens 
Aside from various statistics on the three datasets, Table TABREF12 also reports the proportion of novel bi-grams in gold summaries as a measure of their abstractiveness  We would expect models with extractive biases to perform better on datasets with (mostly) extractive summaries, and abstractive models to perform more rewrite operations on datasets with abstractive summaries  CNN/DailyMail and NYT are somewhat abstractive, while XSum is highly abstractive 
Experimental Setup ::: Implementation Details
For both extractive and abstractive settings, we used PyTorch, OpenNMT BIBREF28 and the `bert-base-uncased' version of Bert to implement BertSum  Both source and target texts were tokenized with Bert's subwords tokenizer 
Experimental Setup ::: Implementation Details ::: Extractive Summarization
All extractive models were trained for 50,000 steps on 3 GPUs (GTX 1080 Ti) with gradient accumulation every two steps  Model checkpoints were saved and evaluated on the validation set every 1,000 steps  We selected the top-3 checkpoints based on the evaluation loss on the validation set, and report the averaged results on the test set  We used a greedy algorithm similar to BIBREF7 to obtain an oracle summary for each document to train extractive models  The algorithm generates an oracle consisting of multiple sentences which maximize the ROUGE-2 score against the gold summary 
When predicting summaries for a new document, we first use the model to obtain the score for each sentence  We then rank these sentences by their scores from highest to lowest, and select the top-3 sentences as the summary 
During sentence selection we use Trigram Blocking to reduce redundancy BIBREF9  Given summary $S$ and candidate sentence $c$, we skip $c$ if there exists a trigram overlapping between $c$ and $S$  The intuition is similar to Maximal Marginal Relevance (MMR; BIBREF29); we wish to minimize the similarity between the sentence being considered and sentences which have been already selected as part of the summary 
Experimental Setup ::: Implementation Details ::: Abstractive Summarization
In all abstractive models, we applied dropout (with probability $0 1$) before all linear layers; label smoothing BIBREF30 with smoothing factor $0 1$ was also used  Our Transformer decoder has 768 hidden units and the hidden size for all feed-forward layers is 2,048  All models were trained for 200,000 steps on 4 GPUs (GTX 1080 Ti) with gradient accumulation every five steps  Model checkpoints were saved and evaluated on the validation set every 2,500 steps  We selected the top-3 checkpoints based on their evaluation loss on the validation set, and report the averaged results on the test set 
During decoding we used beam search (size 5), and tuned the $\alpha $ for the length penalty BIBREF31 between $0 6$ and 1 on the validation set; we decode until an end-of-sequence token is emitted and repeated trigrams are blocked BIBREF9  It is worth noting that our decoder applies neither a copy nor a coverage mechanism BIBREF6, despite their popularity in abstractive summarization  This is mainly because we focus on building a minimum-requirements model and these mechanisms may introduce additional hyper-parameters to tune  Thanks to the subwords tokenizer, we also rarely observe issues with out-of-vocabulary words in the output; moreover, trigram-blocking produces diverse summaries managing to reduce repetitions 
Results ::: Automatic Evaluation
We evaluated summarization quality automatically using ROUGE BIBREF32  We report unigram and bigram overlap (ROUGE-1 and ROUGE-2) as a means of assessing informativeness and the longest common subsequence (ROUGE-L) as a means of assessing fluency  Table TABREF23 summarizes our results on the CNN/DailyMail dataset  The first block in the table includes the results of an extractive Oracle system as an upper bound  We also present the Lead-3 baseline (which simply selects the first three sentences in a document)  The second block in the table includes various extractive models trained on the CNN/DailyMail dataset (see Section SECREF5 for an overview)  For comparison to our own model, we also implemented a non-pretrained Transformer baseline (TransformerExt) which uses the same architecture as BertSumExt, but with fewer parameters  It is randomly initialized and only trained on the summarization task  TransformerExt has 6 layers, the hidden size is 512, and the feed-forward filter size is 2,048  The model was trained with same settings as in BIBREF3  The third block in Table TABREF23 highlights the performance of several abstractive models on the CNN/DailyMail dataset (see Section SECREF6 for an overview)  We also include an abstractive Transformer baseline (TransformerAbs) which has the same decoder as our abstractive BertSum models; the encoder is a 6-layer Transformer with 768 hidden size and 2,048 feed-forward filter size  The fourth block reports results with fine-tuned Bert models: BertSumExt and its two variants (one without interval embeddings, and one with the large version of Bert), BertSumAbs, and BertSumExtAbs  Bert-based models outperform the Lead-3 baseline which is not a strawman; on the CNN/DailyMail corpus it is indeed superior to several extractive BIBREF7, BIBREF8, BIBREF19 and abstractive models BIBREF6  Bert models collectively outperform all previously proposed extractive and abstractive systems, only falling behind the Oracle upper bound  Among Bert variants, BertSumExt performs best which is not entirely surprising; CNN/DailyMail summaries are somewhat extractive and even abstractive models are prone to copying sentences from the source document when trained on this dataset BIBREF6  Perhaps unsurprisingly we observe that larger versions of Bert lead to performance improvements and that interval embeddings bring only slight gains  Table TABREF24 presents results on the NYT dataset  Following the evaluation protocol in BIBREF27, we use limited-length ROUGE Recall, where predicted summaries are truncated to the length of the gold summaries  Again, we report the performance of the Oracle upper bound and Lead-3 baseline  The second block in the table contains previously proposed extractive models as well as our own Transformer baseline  Compress BIBREF27 is an ILP-based model which combines compression and anaphoricity constraints  The third block includes abstractive models from the literature, and our Transformer baseline  Bert-based models are shown in the fourth block  Again, we observe that they outperform previously proposed approaches  On this dataset, abstractive Bert models generally perform better compared to BertSumExt, almost approaching Oracle performance 
Table TABREF26 summarizes our results on the XSum dataset  Recall that summaries in this dataset are highly abstractive (see Table TABREF12) consisting of a single sentence conveying the gist of the document  Extractive models here perform poorly as corroborated by the low performance of the Lead baseline (which simply selects the leading sentence from the document), and the Oracle (which selects a single-best sentence in each document) in Table TABREF26  As a result, we do not report results for extractive models on this dataset  The second block in Table TABREF26 presents the results of various abstractive models taken from BIBREF22 and also includes our own abstractive Transformer baseline  In the third block we show the results of our Bert summarizers which again are superior to all previously reported models (by a wide margin) 
Results ::: Model Analysis ::: Learning Rates
Recall that our abstractive model uses separate optimizers for the encoder and decoder  In Table TABREF27 we examine whether the combination of different learning rates ($\tilde{lr}_{\mathcal {E}}$ and $\tilde{lr}_{\mathcal {D}}$) is indeed beneficial  Specifically, we report model perplexity on the CNN/DailyMail validation set for varying encoder/decoder learning rates  We can see that the model performs best with $\tilde{lr}_{\mathcal {E}}=2e-3$ and $\tilde{lr}_{\mathcal {D}}=0 1$ 
Results ::: Model Analysis ::: Position of Extracted Sentences
In addition to the evaluation based on ROUGE, we also analyzed in more detail the summaries produced by our model  For the extractive setting, we looked at the position (in the source document) of the sentences which were selected to appear in the summary  Figure FIGREF31 shows the proportion of selected summary sentences which appear in the source document at positions 1, 2, and so on  The analysis was conducted on the CNN/DailyMail dataset for Oracle summaries, and those produced by BertSumExt and the TransformerExt  We can see that Oracle summary sentences are fairly smoothly distributed across documents, while summaries created by TransformerExt mostly concentrate on the first document sentences  BertSumExt outputs are more similar to Oracle summaries, indicating that with the pretrained encoder, the model relies less on shallow position features, and learns deeper document representations 
Results ::: Model Analysis ::: Novel N-grams
We also analyzed the output of abstractive systems by calculating the proportion of novel n-grams that appear in the summaries but not in the source texts  The results are shown in Figure FIGREF33  In the CNN/DailyMail dataset, the proportion of novel n-grams in automatically generated summaries is much lower compared to reference summaries, but in XSum, this gap is much smaller  We also observe that on CNN/DailyMail, BertExtAbs produces less novel n-ngrams than BertAbs, which is not surprising  BertExtAbs is more biased towards selecting sentences from the source document since it is initially trained as an extractive model  The supplementary material includes examples of system output and additional ablation studies 
Results ::: Human Evaluation
In addition to automatic evaluation, we also evaluated system output by eliciting human judgments  We report experiments following a question-answering (QA) paradigm BIBREF33, BIBREF8 which quantifies the degree to which summarization models retain key information from the document  Under this paradigm, a set of questions is created based on the gold summary under the assumption that it highlights the most important document content  Participants are then asked to answer these questions by reading system summaries alone without access to the article  The more questions a system can answer, the better it is at summarizing the document as a whole  Moreover, we also assessed the overall quality of the summaries produced by abstractive systems which due to their ability to rewrite content may produce disfluent or ungrammatical output  Specifically, we followed the Best-Worst Scaling BIBREF34 method where participants were presented with the output of two systems (and the original document) and asked to decide which one was better according to the criteria of Informativeness, Fluency, and Succinctness 
Both types of evaluation were conducted on the Amazon Mechanical Turk platform  For the CNN/DailyMail and NYT datasets we used the same documents (20 in total) and questions from previous work BIBREF8, BIBREF18  For XSum, we randomly selected 20 documents (and their questions) from the release of BIBREF22  We elicited 3 responses per HIT  With regard to QA evaluation, we adopted the scoring mechanism from BIBREF33; correct answers were marked with a score of one, partially correct answers with 0 5, and zero otherwise  For quality-based evaluation, the rating of each system was computed as the percentage of times it was chosen as better minus the times it was selected as worse  Ratings thus range from -1 (worst) to 1 (best) 
Results for extractive and abstractive systems are shown in Tables TABREF37 and TABREF38, respectively  We compared the best performing BertSum model in each setting (extractive or abstractive) against various state-of-the-art systems (whose output is publicly available), the Lead baseline, and the Gold standard as an upper bound  As shown in both tables participants overwhelmingly prefer the output of our model against comparison systems across datasets and evaluation paradigms  All differences between BertSum and comparison models are statistically significant ($p<0 05$), with the exception of TConvS2S (see Table TABREF38; XSum) in the QA evaluation setting 
Conclusions
In this paper, we showcased how pretrained Bert can be usefully applied in text summarization  We introduced a novel document-level encoder and proposed a general framework for both abstractive and extractive summarization  Experimental results across three datasets show that our model achieves state-of-the-art results across the board under automatic and human-based evaluation protocols  Although we mainly focused on document encoding for summarization, in the future, we would like to take advantage the capabilities of Bert for language generation 
Acknowledgments
This research is supported by a Google PhD Fellowship to the first author  We gratefully acknowledge the support of the European Research Council (Lapata, award number 681760, “Translating Multiple Modalities into Text”)  We would also like to thank Shashi Narayan for providing us with the XSum dataset 

ground_truth: CNN/DailyMail news highlights, New York Times Annotated Corpus, XSum

the CNN/DailyMail news highlights dataset BIBREF24, the New York Times Annotated Corpus (NYT; BIBREF25), XSum BIBREF22

prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. What metrics are used for evaluation?
Context 3: 
In this work, we investigate how leveraging historical user preferences can improve generation quality over strong baselines in our setting  We compare our personalized models against two baselines  The first is a name-based Nearest-Neighbor model (NN)  We initially adapted the Neural Checklist Model of BIBREF0 as a baseline; however, we ultimately use a simple Encoder-Decoder baseline with ingredient attention (Enc-Dec), which provides comparable performance and lower complexity  All personalized models outperform baseline in BPE perplexity (tab:metricsontest) with Prior Name performing the best  While our models exhibit comparable performance to baseline in BLEU-1/4 and ROUGE-L, we generate more diverse (Distinct-1/2: percentage of distinct unigrams and bigrams) and acceptable recipes  BLEU and ROUGE are not the most appropriate metrics for generation quality  A `correct' recipe can be written in many ways with the same main entities (ingredients)  As BLEU-1/4 capture structural information via n-gram matching, they are not correlated with subjective recipe quality  This mirrors observations from BIBREF31, BIBREF8 
We observe that personalized models make more diverse recipes than baseline  They thus perform better in BLEU-1 with more key entities (ingredient mentions) present, but worse in BLEU-4, as these recipes are written in a personalized way and deviate from gold on the phrasal level  Similarly, the `Prior Name' model generates more unigram-diverse recipes than other personalized models and obtains a correspondingly lower BLEU-1 score 
Qualitative Analysis: We present sample outputs for a cocktail recipe in tab:samplerecipes, and additional recipes in the appendix  Generation quality progressively improves from generic baseline output to a blended cocktail produced by our best performing model  Models attending over prior recipes explicitly reference ingredients  The Prior Name model further suggests the addition of lemon and mint, which are reasonably associated with previously consumed recipes like coconut mousse and pork skewers 
Personalization: To measure personalization, we evaluate how closely the generated text corresponds to a particular user profile  We compute the likelihood of generated recipes using identical input specifications but conditioned on ten different user profiles—one `gold' user who consumed the original recipe, and nine randomly generated user profiles  Following BIBREF8, we expect the highest likelihood for the recipe conditioned on the gold user  We measure user matching accuracy (UMA)—the proportion where the gold user is ranked highest—and Mean Reciprocal Rank (MRR) BIBREF32 of the gold user  All personalized models beat baselines in both metrics, showing our models personalize generated recipes to the given user profiles  The Prior Name model achieves the best UMA and MRR by a large margin, revealing that prior recipe names are strong signals for personalization  Moreover, the addition of attention mechanisms to capture these signals improves language modeling performance over a strong non-personalized baseline 
Recipe Level Coherence: A plausible recipe should possess a coherent step order, and we evaluate this via a metric for recipe-level coherence  We use the neural scoring model from BIBREF33 to measure recipe-level coherence for each generated recipe  Each recipe step is encoded by BERT BIBREF34  Our scoring model is a GRU network that learns the overall recipe step ordering structure by minimizing the cosine similarity of recipe step hidden representations presented in the correct and reverse orders  Once pretrained, our scorer calculates the similarity of a generated recipe to the forward and backwards ordering of its corresponding gold label, giving a score equal to the difference between the former and latter  A higher score indicates better step ordering (with a maximum score of 2)  tab:coherencemetrics shows that our personalized models achieve average recipe-level coherence scores of 1 78-1 82, surpassing the baseline at 1 77 
Recipe Step Entailment: Local coherence is also crucial to a user following a recipe: it is crucial that subsequent steps are logically consistent with prior ones  We model local coherence as an entailment task: predicting the likelihood that a recipe step follows the preceding  We sample several consecutive (positive) and non-consecutive (negative) pairs of steps from each recipe  We train a BERT BIBREF34 model to predict the entailment score of a pair of steps separated by a [SEP] token, using the final representation of the [CLS] token  The step entailment score is computed as the average of scores for each set of consecutive steps in each recipe, averaged over every generated recipe for a model, as shown in tab:coherencemetrics 
Human Evaluation: We presented 310 pairs of recipes for pairwise comparison BIBREF8 (details in appendix) between baseline and each personalized model, with results shown in tab:metricsontest  On average, human evaluators preferred personalized model outputs to baseline 63% of the time, confirming that personalized attention improves the semantic plausibility of generated recipes  We also performed a small-scale human coherence survey over 90 recipes, in which 60% of users found recipes generated by personalized models to be more coherent and preferable to those generated by baseline models 
Conclusion
In this paper, we propose a novel task: to generate personalized recipes from incomplete input specifications and user histories  On a large novel dataset of 180K recipes and 700K reviews, we show that our personalized generative models can generate plausible, personalized, and coherent recipes preferred by human evaluators for consumption  We also introduce a set of automatic coherence measures for instructional texts as well as personalization metrics to support our claims  Our future work includes generating structured representations of recipes to handle ingredient properties, as well as accounting for references to collections of ingredients (e g  “dry mix") 
Acknowledgements  This work is partly supported by NSF #1750063  We thank all reviewers for their constructive suggestions, as well as Rei M , Sujoy P , Alicia L , Eric H , Tim S , Kathy C , Allen C , and Micah I  for their feedback 
Appendix ::: Food com: Dataset Details
Our raw data consists of 270K recipes and 1 4M user-recipe interactions (reviews) scraped from Food com, covering a period of 18 years (January 2000 to December 2018)  See tab:int-stats for dataset summary statistics, and tab:samplegk for sample information about one user-recipe interaction and the recipe involved 
Appendix ::: Generated Examples
See tab:samplechx for a sample recipe for chicken chili and tab:samplewaffle for a sample recipe for sweet waffles 
Human Evaluation
We prepared a set of 15 pairwise comparisons per evaluation session, and collected 930 pairwise evaluations (310 per personalized model) over 62 sessions  For each pair, users were given a partial recipe specification (name and 3-5 key ingredients), as well as two generated recipes labeled `A' and `B'  One recipe is generated from our baseline encoder-decoder model and one recipe is generated by one of our three personalized models (Prior Tech, Prior Name, Prior Recipe)  The order of recipe presentation (A/B) is randomly selected for each question  A screenshot of the user evaluation interface is given in fig:exeval  We ask the user to indicate which recipe they find more coherent, and which recipe best accomplishes the goal indicated by the recipe name  A screenshot of this survey interface is given in fig:exeval2 

ground_truth: Byte-Pair Encoding perplexity  (BPE PPL),
BLEU-1,
BLEU-4,
ROUGE-L,
percentage of distinct unigram (D-1),
percentage of distinct bigrams(D-2),
user matching accuracy(UMA),
Mean Reciprocal Rank(MRR)
Pairwise preference over baseline(PP)

BLEU-1/4 and ROUGE-L, likelihood of generated recipes using identical input specifications but conditioned on ten different user profiles, user matching accuracy (UMA), Mean Reciprocal Rank (MRR), neural scoring model from BIBREF33 to measure recipe-level coherence

 Distinct-1/2, UMA = User Matching Accuracy, MRR
= Mean Reciprocal Rank, PP = Pairwise preference over baseline (evaluated for 310 recipe pairs per model)