prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. How much data is needed to train the task-specific encoder?
Context 1: Introduction
Assembling training corpora of annotated natural language examples in specialized domains such as biomedicine poses considerable challenges  Experts with the requisite domain knowledge to perform high-quality annotation tend to be expensive, while lay annotators may not have the necessary knowledge to provide high-quality annotations  A practical approach for collecting a sufficiently large corpus would be to use crowdsourcing platforms like Amazon Mechanical Turk (MTurk)  However, crowd workers in general are likely to provide noisy annotations BIBREF0 , BIBREF1 , BIBREF2 , an issue exacerbated by the technical nature of specialized content  Some of this noise may reflect worker quality and can be modeled BIBREF0 , BIBREF1 , BIBREF3 , BIBREF4 , but for some instances lay people may simply lack the domain knowledge to provide useful annotation 
In this paper we report experiments on the EBM-NLP corpus comprising crowdsourced annotations of medical literature BIBREF5   We operationalize the concept of annotation difficulty and show how it can be exploited during training to improve information extraction models  We then obtain expert annotations for the abstracts predicted to be most difficult, as well as for a similar number of randomly selected abstracts  The annotation of highly specialized data and the use of lay and expert annotators allow us to examine the following key questions related to lay and expert annotations in specialized domains:
Can we predict item difficulty? We define a training instance as difficult if a lay annotator or an automated model disagree on its labeling  We show that difficulty can be predicted, and that it is distinct from inter-annotator agreement  Further, such predictions can be used during training to improve information extraction models 
Are there systematic differences between expert and lay annotations? We observe decidedly lower agreement between lay workers as compared to domain experts  Lay annotations have high precision but low recall with respect to expert annotations in the new data that we collected  More generally, we expect lay annotations to be lower quality, which may translate to lower precision, recall, or both, compared to expert annotations  Can one rely solely on lay annotations? Reasonable models can be trained using lay annotations alone, but similar performance can be achieved using markedly less expert data  This suggests that the optimal ratio of expert to crowd annotations for specialized tasks will depend on the cost and availability of domain experts  Expert annotations are preferable whenever its collection is practical  But in real-world settings, a combination of expert and lay annotations is better than using lay data alone 
Does it matter what data is annotated by experts? We demonstrate that a system trained on combined data achieves better predictive performance when experts annotate difficult examples rather than instances selected at i i d  random 
Our contributions in this work are summarized as follows  We define a task difficulty prediction task and show how this is related to, but distinct from, inter-worker agreement  We introduce a new model for difficulty prediction combining learned representations induced via a pre-trained `universal' sentence encoder BIBREF6 , and a sentence encoder learned from scratch for this task  We show that predicting annotation difficulty can be used to improve the task routing and model performance for a biomedical information extraction task  Our results open up a new direction for ensuring corpus quality  We believe that item difficulty prediction will likely be useful in other, non-specialized tasks as well, and that the most effective data collection in specialized domains requires research addressing the fundamental questions we examine here 
Related Work
Crowdsourcing annotation is now a well-studied problem BIBREF7 , BIBREF0 , BIBREF1 , BIBREF2   Due to the noise inherent in such annotations, there have also been considerable efforts to develop aggregation models that minimize noise BIBREF0 , BIBREF1 , BIBREF3 , BIBREF4  
There are also several surveys of crowdsourcing in biomedicine specifically BIBREF8 , BIBREF9 , BIBREF10   Some work in this space has contrasted model performance achieved using expert vs  crowd annotated training data BIBREF11 , BIBREF12 , BIBREF13   Dumitrache et al  Dumitrache:2018:CGT:3232718 3152889 concluded that performance is similar under these supervision types, finding no clear advantage from using expert annotators  This differs from our findings, perhaps owing to differences in design  The experts we used already hold advanced medical degrees, for instance, while those in prior work were medical students  Furthermore, the task considered here would appear to be of greater difficulty: even a system trained on $\sim $ 5k instances performs reasonably, but far from perfect  By contrast, in some of the prior work where experts and crowd annotations were deemed equivalent, a classifier trained on 300 examples can achieve very high accuracy BIBREF12  
More relevant to this paper, prior work has investigated methods for `task routing' in active learning scenarios in which supervision is provided by heterogeneous labelers with varying levels of expertise BIBREF14 , BIBREF15 , BIBREF16 , BIBREF17 , BIBREF14   The related question of whether effort is better spent collecting additional annotations for already labeled (but potentially noisily so) examples or novel instances has also been addressed BIBREF18   What distinguishes the work here is our focus on providing an operational definition of instance difficulty, showing that this can be predicted, and then using this to inform task routing 
Application Domain
Our specific application concerns annotating abstracts of articles that describe the conduct and results of randomized controlled trials (RCTs)  Experimentation in this domain has become easy with the recent release of the EBM-NLP BIBREF5 corpus, which includes a reasonably large training dataset annotated via crowdsourcing, and a modest test set labeled by individuals with advanced medical training  More specifically, the training set comprises 4,741 medical article abstracts with crowdsourced annotations indicating snippets (sequences) that describe the Participants (p), Interventions (i), and Outcome (o) elements of the respective RCT, and the test set is composed of 191 abstracts with p, i, o sequence annotations from three medical experts 
Table 1 shows an example of difficult and easy examples according to our definition of difficulty  The underlined text demarcates the (consensus) reference label provided by domain experts  In the difficult examples, crowd workers marked text distinct from these reference annotations; whereas in the easy cases they reproduced them with reasonable fidelity  The difficult sentences usually exhibit complicated structure and feature jargon 
An abstract may contain some `easy' and some `difficult' sentences  We thus perform our analysis at the sentence level  We split abstracts into sentences using spaCy  We excluded sentences that comprise fewer than two tokens, as these are likely an artifact of errors in sentence splitting  In total, this resulted in 57,505 and 2,428 sentences in the train and test set abstracts, respectively 
Quantifying Task Difficulty
The test set includes annotations from both crowd workers and domain experts  We treat the latter as ground truth and then define the difficulty of sentences in terms of the observed agreement between expert and lay annotators  Formally, for annotation task $t$ and instance $i$ :
$$\text{Difficulty}_{ti} = \frac{\sum _{j=1}^n{f(\text{label}_{ij}, y_i})}{n}$$   (Eq  3)
where $f$ is a scoring function that measures the quality of the label from worker $j$ for sentence $i$ , as compared to a ground truth annotation, $y_i$   The difficulty score of sentence $i$ is taken as an average over the scores for all $n$ layworkers  We use Spearmans' correlation coefficient as a scoring function  Specifically, for each sentence we create two vectors comprising counts of how many times each token was annotated by crowd and expert workers, respectively, and calculate the correlation between these  Sentences with no labels are treated as maximally easy; those with only either crowd worker or expert label(s) are assumed maximally difficult 
The training set contains only crowdsourced annotations  To label the training data, we use a 10-fold validation like setting  We iteratively retrain the LSTM-CRF-Pattern sequence tagger of Patel et al  patel2018syntactic on 9 folds of the training data and use that trained model to predict labels for the 10th  In this way we obtain predictions on the full training set  We then use predicted spans as proxy `ground truth' annotations to calculate the difficulty score of sentences as described above; we normalize these to the [ $0, 1$ ] interval  We validate this approximation by comparing the proxy scores against reference scores over the test set, the Pearson's correlation coefficients are 0 57 for Population, 0 71 for Intervention and 0 68 for Outcome 
There exist many sentences that contain neither manual nor predicted annotations  We treat these as maximally easy sentences (with difficulty scores of 0)  Such sentences comprise 51%, 42% and 36% for Population, Interventions and Outcomes data respectively, indicating that it is easier to identify sentences that have no Population spans, but harder to identify sentences that have no Interventions or Outcomes spans  This is intuitive as descriptions of the latter two tend to be more technical and dense with medical jargon 
We show the distribution of the automatically labeled scores for sentences that do contain spans in Figure 1   The mean of the Population (p) sentence scores is significantly lower than that for other types of sentences (i and o), again indicating that they are easier on average to annotate  This aligns with a previous finding that annotating Interventions and Outcomes is more difficult than annotating Participants BIBREF5  
Many sentences contain spans tagged by the LSTM-CRF-Pattern model, but missed by all crowd workers, resulting in a maximally difficult score (1)  Inspection of such sentences revealed that some are truly difficult examples, but others are tagging model errors  In either case, such sentences have confused workers and/or the model, and so we retain them all as `difficult' sentences 
Content describing the p, i and o, respectively, is quite different  As such, one sentence usually contains (at most) only one of these three content types  We thus treat difficulty prediction for the respective label types as separate tasks 
Difficulty is not Worker Agreement
Our definition of difficulty is derived from agreement between expert and crowd annotations for the test data, and agreement between a predictive model and crowd annotations in the training data  It is reasonable to ask if these measures are related to inter-annotator agreement, a metric often used in language technology research to identify ambiguous or difficult items  Here we explicitly verify that our definition of difficulty only weakly correlates with inter-annotator agreement 
We calculate inter-worker agreement between crowd and expert annotators using Spearman's correlation coefficient  As shown in Table 2 , average agreement between domain experts are considerably higher than agreements between crowd workers for all three label types  This is a clear indication that the crowd annotations are noisier 
Furthermore, we compare the correlation between inter-annotator agreement and difficulty scores in the training data  Given that the majority of sentences do not contain a PICO span, we only include in these calculations those that contain a reference label  Pearson's r are 0 34, 0 30 and 0 31 for p, i and o, respectively, confirming that inter-worker agreement and our proposed difficulty score are quite distinct 
Predicting Annotation Difficulty
We treat difficulty prediction as a regression problem, and propose and evaluate neural model variants for the task  We first train RNN BIBREF19 and CNN BIBREF20 models 
We also use the universal sentence encoder (USE) BIBREF6 to induce sentence representations, and train a model using these as features  Following BIBREF6 , we then experiment with an ensemble model that combines the `universal' and task-specific representations to predict annotation difficulty  We expect these universal embeddings to capture general, high-level semantics, and the task specific representations to capture more granular information  Figure 2 depicts the model architecture  Sentences are fed into both the universal sentence encoder and, separately, a task specific neural encoder, yielding two representations  We concatenate these and pass the combined vector to the regression layer 
Experimental Setup and Results
We trained models for each label type separately  Word embeddings were initialized to 300d GloVe vectors BIBREF21 trained on common crawl data; these are fine-tuned during training  We used the Adam optimizer BIBREF22 with learning rate and decay set to 0 001 and 0 99, respectively  We used batch sizes of 16 
We used the large version of the universal sentence encoder with a transformer BIBREF23   We did not update the pretrained sentence encoder parameters during training  All hyperparamaters for all models (including hidden layers, hidden sizes, and dropout) were tuned using Vizier BIBREF24 via 10-fold cross validation on the training set maximizing for F1 
As a baseline, we also trained a linear Support-Vector Regression BIBREF25 model on $n$ -gram features ( $n$ ranges from 1 to 3) 
Table 3 reports Pearson correlation coefficients between the predictions with each of the neural models and the ground truth difficulty scores  Rows 1-4 correspond to individual models, and row 5 reports the ensemble performance  Columns correspond to label type  Results from all models outperform the baseline SVR model: Pearson's correlation coefficients range from 0 550 to 0 622  The regression correlations are the lowest 
The RNN model realizes the strongest performance among the stand-alone (non-ensemble) models, outperforming variants that exploit CNN and USE representations  Combining the RNN and USE further improves results  We hypothesize that this is due to complementary sentence information encoded in universal representations 
For all models, correlations for Intervention and Outcomes are higher than for Population, which is expected given the difficulty distributions in Figure 1   In these, the sentences are more uniformly distributed, with a fair number of difficult and easier sentences  By contrast, in Population there are a greater number of easy sentences and considerably fewer difficult sentences, which makes the difficulty ranking task particularly challenging 
Better IE with Difficulty Prediction
We next present experiments in which we attempt to use the predicted difficulty during training to improve models for information extraction of descriptions of Population, Interventions and Outcomes from medical article abstracts  We investigate two uses: (1) simply removing the most difficult sentences from the training set, and, (2) re-weighting the most difficult sentences 
We again use LSTM-CRF-Pattern as the base model and experimenting on the EBM-NLP corpus BIBREF5   This is trained on either (1) the training set with difficult sentences removed, or (2) the full training set but with instances re-weighted in proportion to their predicted difficulty score  Following BIBREF5 , we use the Adam optimizer with learning rate of 0 001, decay 0 9, batch size 20 and dropout 0 5  We use pretrained 200d GloVe vectors BIBREF21 to initialize word embeddings, and use 100d hidden char representations  Each word is thus represented with 300 dimensions in total  The hidden size is 100 for the LSTM in the character representation component, and 200 for the LSTM in the information extraction component  We train for 15 epochs, saving parameters that achieve the best F1 score on a nested development set 
Removing Difficult Examples
We first evaluate changes in performance induced by training the sequence labeling model using less data by removing difficult sentences prior to training  The hypothesis here is that these difficult instances are likely to introduce more noise than signal  We used a cross-fold approach to predict sentence difficulties, training on 9/10ths of the data and scoring the remaining 1/10th at a time  We then sorted sentences by predicted difficulty scores, and experimented with removing increasing numbers of these (in order of difficulty) prior to training the LSTM-CRF-Pattern model 
Figure 3 shows the results achieved by the LSTM-CRF-Pattern model after discarding increasing amounts of the training data: the $x$ and $y$ axes correspond to the the percentage of data removed and F1 scores, respectively  We contrast removing sentences predicted to be difficult with removing them (a) randomly (i i d ), and, (b) in inverse order of predicted inter-annotator agreement  The agreement prediction model is trained exactly the same like difficult prediction model, with simply changing the difficult score to annotation agreement  F1 scores actually improve (marginally) when we remove the most difficult sentences, up until we drop 4% of the data for Population and Interventions, and 6% for Outcomes  Removing training points at i i d  random degrades performance, as expected  Removing sentences in order of disagreement seems to have similar effect as removing them by difficulty score when removing small amount of the data, but the F1 scores drop much faster when removing more data  These findings indicate that sentences predicted to be difficult are indeed noisy, to the extent that they do not seem to provide the model useful signal 
Re-weighting by Difficulty
We showed above that removing a small number of the most difficult sentences does not harm, and in fact modestly improves, medical IE model performance  However, using the available data we are unable to test if this will be useful in practice, as we would need additional data to determine how many difficult sentences should be dropped 
We instead explore an alternative, practical means of exploiting difficulty predictions: we re-weight sentences during training inversely to their predicted difficulty  Formally, we weight sentence $i$ with difficulty scores above $\tau $ according to: $1-a\cdot (d_i-\tau )/(1-\tau )$ , where $d_i$ is the difficulty score for sentence $i$ , and $a$ is a parameter codifying the minimum weight value  We set $\tau $ to 0 8 so as to only re-weight sentences with difficulty in the top 20th percentile, and we set $a$ to 0 5  The re-weighting is equivalent to down-sampling the difficult sentences  LSTM-CRF-Pattern is our base model 
Table 4 reports the precision, recall and F1 achieved both with and without sentence re-weighting  Re-weighting improves all metrics modestly but consistently  All F1 differences are statistically significant under a sign test ( $p<0 01$ )  The model with best precision is different for Patient, Intervention and Outcome labels  However re-weighting by difficulty does consistently yield the best recall for all three extraction types, with the most notable improvement for i and o, where recall improved by 10 percentage points  This performance increase translated to improvements in F1 across all types, as compared to the base model and to re-weighting by agreement 
Involving Expert Annotators
The preceding experiments demonstrate that re-weighting difficult sentences annotated by the crowd generally improves the extraction models  Presumably the performance is influenced by the annotation quality 
We now examine the possibility that the higher quality and more consistent annotations of domain experts on the difficult instances will benefit the extraction model  This simulates an annotation strategy in which we route difficult instances to domain experts and easier ones to crowd annotators  We also contrast the value of difficult data to that of an i i d  random sample of the same size, both annotated by experts 
Expert annotations of Random and Difficult Instances
We re-annotate by experts a subset of most difficult instances and the same number of random instances  As collecting annotations from experts is slow and expensive, we only re-annotate the difficult instances for the interventions extraction task  We re-annotate the abstracts which cover the sentences with predicted difficulty scores in the top 5 percentile  We rank the abstracts from the training set by the count of difficult sentences, and re-annotate the abstracts that contain the most difficult sentences  Constrained by time and budget, we select only 2000 abstracts for re-annotation; 1000 of these are top-ranked, and 1000 are randomly sampled  This re-annotation cost $3,000  We have released the new annotation data at: https://github com/bepnye/EBM-NLP 
Following BIBREF5 , we recruited five medical experts via Up-work with advanced medical training and strong technical reading/writing skills  The expert annotator were asked to read the entire abstract and highlight, using the BRAT toolkit BIBREF26 , all spans describing medical Interventions  Each abstract is only annotated by one expert  We examined 30 re-annotated abstracts to ensure the annotation quality before hiring the annotator 
Table 5 presents the results of LSTM-CRF-Pattern model trained on the reannotated difficult subset and the random subset  The first two rows show the results for models trained with expert annotations  The model trained on random data has a slightly better F1 than that trained on the same amount of difficult data  The model trained on random data has higher precision but lower recall 
Rows 3 and 4 list the results for models trained on the same data but with crowd annotation  Models trained with expert-annotated data are clearly superior to those trained with crowd labels with respect to F1, indicating that the experts produced higher quality annotations  For crowdsourced annotations, training the model with data sampled at i i d  random achieves 2% higher F1 than when difficult instances are used  When expert annotations are used, this difference is less than 1%  This trend in performance may be explained by differences in annotation quality: the randomly sampled set was more consistently annotated by both experts and crowd because the difficult set is harder  However, in both cases expert annotations are better, with a bigger difference between the expert and crowd models on the difficult set 
The last row is the model trained on all 5k abstracts with crowd annotations  Its F1 score is lower than either expert model trained on only 20% of data, suggesting that expert annotations should be collected whenever possible  Again the crowd model on complete data has higher precision than expert models but its recall is much lower 
Routing To Experts or Crowd
So far a system was trained on one type of data, either labeled by crowd or experts  We now examine the performance of a system trained on data that was routed to either experts or crowd annotators depending on their predicted difficult  Given the results presented so far mixing annotators may be beneficial given their respective trade-offs of precision and recall  We use the annotations from experts for an abstract if it exists otherwise use crowd annotations  The results are presented in Table 6  
Rows 1 and 2 repeat the performance of the models trained on difficult subset and random subset with expert annotations only respectively  The third row is the model trained by combining difficult and random subsets with expert annotations  There are around 250 abstracts in the overlap of these two sets, so there are total 1 75k abstracts used for training the D+R model  Rows 4 to 6 are the models trained on all 5k abstracts with mixed annotations, where Other means the rest of the abstracts with crowd annotation only 
The results show adding more training data with crowd annotation still improves at least 1 point F1 score in all three extraction tasks  The improvement when the difficult subset with expert annotations is mixed with the remaining crowd annotation is 3 5 F1 score, much larger than when a random set of expert annotations are added  The model trained with re-annotating the difficult subset (D+Other) also outperforms the model with re-annotating the random subset (R+Other) by 2 points in F1  The model trained with re-annotating both of difficult and random subsets (D+R+Other), however, achieves only marginally higher F1 than the model trained with the re-annotated difficult subset (D+Other)  In sum, the results clearly indicate that mixing expert and crowd annotations leads to better models than using solely crowd data, and better than using expert data alone  More importantly, there is greater gain in performance when instances are routed according to difficulty, as compared to randomly selecting the data for expert annotators  These findings align with our motivating hypothesis that annotation quality for difficult instances is important for final model performance  They also indicate that mixing annotations from expert and crowd could be an effective way to achieve acceptable model performance given a limited budget 
How Many Expert Annotations?
We established that crowd annotation are still useful in supplementing expert annotations for medical IE  Obtaining expert annotations for the one thousand most difficult instances greatly improved the model performance  However the choice of how many difficult instances to annotate was an uninformed choice  Here we check if less expert data would have yielded similar gains  Future work will need to address how best to choose this parameter for a routing system 
We simulate a routing scenario in which we send consecutive batches of the most difficult examples to the experts for annotation  We track changes in performance as we increase the number of most-difficult-articles sent to domain experts  As shown in Figure 4 , adding expert annotations for difficult articles consistently increases F1 scores  The performance gain is mostly from increased recall; the precision changes only a bit with higher quality annotation  This observation implies that crowd workers often fail to mark target tokens, but do not tend to produce large numbers of false positives  We suspect such failures to identify relevant spans/tokens are due to insufficient domain knowledge possessed by crowd workers 
The F1 score achieved after re-annotating the 600 most-difficult articles reaches 68 1%, which is close to the performance when re-annotating 1000 random articles  This demonstrates the effectiveness of recognizing difficult instances  The trend when we use up all expert data is still upward, so adding even more expert data is likely to further improve performance  Unfortunately we exhausted our budget and were not able to obtain additional expert annotations  It is likely that as the size of the expert annotations increases, the value of crowd annotations will diminish  This investigation is left for future work 
Conclusions
We have introduced the task of predicting annotation difficulty for biomedical information extraction (IE)  We trained neural models using different learned representations to score texts in terms of their difficulty  Results from all models were strong with Pearson’s correlation coefficients higher than 0 45 in almost all evaluations, indicating the feasibility of this task  An ensemble model combining universal and task specific feature sentence vectors yielded the best results 
Experiments on biomedical IE tasks show that removing up to $\sim $ 10% of the sentences predicted to be most difficult did not decrease model performance, and that re-weighting sentences inversely to their difficulty score during training improves predictive performance  Simulations in which difficult examples are routed to experts and other instances to crowd annotators yields the best results, outperforming the strategy of randomly selecting data for expert annotation, and substantially improving upon the approach of relying exclusively on crowd annotations  In future work, routing strategies based on instance difficulty could be further investigated for budget-quality trade-off 
Acknowledgements
This work has been partially supported by NSF1748771 grant  Wallace was support in part by NIH/NLM R01LM012086 
Context 2: Introduction
In the kitchen, we increasingly rely on instructions from cooking websites: recipes  A cook with a predilection for Asian cuisine may wish to prepare chicken curry, but may not know all necessary ingredients apart from a few basics  These users with limited knowledge cannot rely on existing recipe generation approaches that focus on creating coherent recipes given all ingredients and a recipe name BIBREF0  Such models do not address issues of personal preference (e g  culinary tastes, garnish choices) and incomplete recipe details  We propose to approach both problems via personalized generation of plausible, user-specific recipes using user preferences extracted from previously consumed recipes 
Our work combines two important tasks from natural language processing and recommender systems: data-to-text generation BIBREF1 and personalized recommendation BIBREF2  Our model takes as user input the name of a specific dish, a few key ingredients, and a calorie level  We pass these loose input specifications to an encoder-decoder framework and attend on user profiles—learned latent representations of recipes previously consumed by a user—to generate a recipe personalized to the user's tastes  We fuse these `user-aware' representations with decoder output in an attention fusion layer to jointly determine text generation  Quantitative (perplexity, user-ranking) and qualitative analysis on user-aware model outputs confirm that personalization indeed assists in generating plausible recipes from incomplete ingredients 
While personalized text generation has seen success in conveying user writing styles in the product review BIBREF3, BIBREF4 and dialogue BIBREF5 spaces, we are the first to consider it for the problem of recipe generation, where output quality is heavily dependent on the content of the instructions—such as ingredients and cooking techniques 
To summarize, our main contributions are as follows:
We explore a new task of generating plausible and personalized recipes from incomplete input specifications by leveraging historical user preferences;
We release a new dataset of 180K+ recipes and 700K+ user reviews for this task;
We introduce new evaluation strategies for generation quality in instructional texts, centering on quantitative measures of coherence  We also show qualitatively and quantitatively that personalized models generate high-quality and specific recipes that align with historical user preferences 
Related Work
Large-scale transformer-based language models have shown surprising expressivity and fluency in creative and conditional long-text generation BIBREF6, BIBREF7  Recent works have proposed hierarchical methods that condition on narrative frameworks to generate internally consistent long texts BIBREF8, BIBREF9, BIBREF10  Here, we generate procedurally structured recipes instead of free-form narratives 
Recipe generation belongs to the field of data-to-text natural language generation BIBREF1, which sees other applications in automated journalism BIBREF11, question-answering BIBREF12, and abstractive summarization BIBREF13, among others  BIBREF14, BIBREF15 model recipes as a structured collection of ingredient entities acted upon by cooking actions  BIBREF0 imposes a `checklist' attention constraint emphasizing hitherto unused ingredients during generation  BIBREF16 attend over explicit ingredient references in the prior recipe step  Similar hierarchical approaches that infer a full ingredient list to constrain generation will not help personalize recipes, and would be infeasible in our setting due to the potentially unconstrained number of ingredients (from a space of 10K+) in a recipe  We instead learn historical preferences to guide full recipe generation 
A recent line of work has explored user- and item-dependent aspect-aware review generation BIBREF3, BIBREF4  This work is related to ours in that it combines contextual language generation with personalization  Here, we attend over historical user preferences from previously consumed recipes to generate recipe content, rather than writing styles 
Approach
Our model's input specification consists of: the recipe name as a sequence of tokens, a partial list of ingredients, and a caloric level (high, medium, low)  It outputs the recipe instructions as a token sequence: $\mathcal {W}_r=\lbrace w_{r,0}, \dots , w_{r,T}\rbrace $ for a recipe $r$ of length $T$  To personalize output, we use historical recipe interactions of a user $u \in \mathcal {U}$ 
Encoder: Our encoder has three embedding layers: vocabulary embedding $\mathcal {V}$, ingredient embedding $\mathcal {I}$, and caloric-level embedding $\mathcal {C}$  Each token in the (length $L_n$) recipe name is embedded via $\mathcal {V}$; the embedded token sequence is passed to a two-layered bidirectional GRU (BiGRU) BIBREF17, which outputs hidden states for names $\lbrace \mathbf {n}_{\text{enc},j} \in \mathbb {R}^{2d_h}\rbrace $, with hidden size $d_h$  Similarly each of the $L_i$ input ingredients is embedded via $\mathcal {I}$, and the embedded ingredient sequence is passed to another two-layered BiGRU to output ingredient hidden states as $\lbrace \mathbf {i}_{\text{enc},j} \in \mathbb {R}^{2d_h}\rbrace $  The caloric level is embedded via $\mathcal {C}$ and passed through a projection layer with weights $W_c$ to generate calorie hidden representation $\mathbf {c}_{\text{enc}} \in \mathbb {R}^{2d_h}$ 
Ingredient Attention: We apply attention BIBREF18 over the encoded ingredients to use encoder outputs at each decoding time step  We define an attention-score function $\alpha $ with key $K$ and query $Q$:
with trainable weights $W_{\alpha }$, bias $\mathbf {b}_{\alpha }$, and normalization term $Z$  At decoding time $t$, we calculate the ingredient context $\mathbf {a}_{t}^{i} \in \mathbb {R}^{d_h}$ as:
Decoder: The decoder is a two-layer GRU with hidden state $h_t$ conditioned on previous hidden state $h_{t-1}$ and input token $w_{r, t}$ from the original recipe text  We project the concatenated encoder outputs as the initial decoder hidden state:
To bias generation toward user preferences, we attend over a user's previously reviewed recipes to jointly determine the final output token distribution  We consider two different schemes to model preferences from user histories: (1) recipe interactions, and (2) techniques seen therein (defined in data)  BIBREF19, BIBREF20, BIBREF21 explore similar schemes for personalized recommendation 
Prior Recipe Attention: We obtain the set of prior recipes for a user $u$: $R^+_u$, where each recipe can be represented by an embedding from a recipe embedding layer $\mathcal {R}$ or an average of the name tokens embedded by $\mathcal {V}$  We attend over the $k$-most recent prior recipes, $R^{k+}_u$, to account for temporal drift of user preferences BIBREF22  These embeddings are used in the `Prior Recipe' and `Prior Name' models, respectively 
Given a recipe representation $\mathbf {r} \in \mathbb {R}^{d_r}$ (where $d_r$ is recipe- or vocabulary-embedding size depending on the recipe representation) the prior recipe attention context $\mathbf {a}_{t}^{r_u}$ is calculated as
Prior Technique Attention: We calculate prior technique preference (used in the `Prior Tech` model) by normalizing co-occurrence between users and techniques seen in $R^+_u$, to obtain a preference vector $\rho _{u}$  Each technique $x$ is embedded via a technique embedding layer $\mathcal {X}$ to $\mathbf {x}\in \mathbb {R}^{d_x}$  Prior technique attention is calculated as
where, inspired by copy mechanisms BIBREF23, BIBREF24, we add $\rho _{u,x}$ for technique $x$ to emphasize the attention by the user's prior technique preference 
Attention Fusion Layer: We fuse all contexts calculated at time $t$, concatenating them with decoder GRU output and previous token embedding:
We then calculate the token probability:
and maximize the log-likelihood of the generated sequence conditioned on input specifications and user preferences  fig:ex shows a case where the Prior Name model attends strongly on previously consumed savory recipes to suggest the usage of an additional ingredient (`cilantro') 
Recipe Dataset: Food com
We collect a novel dataset of 230K+ recipe texts and 1M+ user interactions (reviews) over 18 years (2000-2018) from Food com  Here, we restrict to recipes with at least 3 steps, and at least 4 and no more than 20 ingredients  We discard users with fewer than 4 reviews, giving 180K+ recipes and 700K+ reviews, with splits as in tab:recipeixnstats 
Our model must learn to generate from a diverse recipe space: in our training data, the average recipe length is 117 tokens with a maximum of 256  There are 13K unique ingredients across all recipes  Rare words dominate the vocabulary: 95% of words appear $<$100 times, accounting for only 1 65% of all word usage  As such, we perform Byte-Pair Encoding (BPE) tokenization BIBREF25, BIBREF26, giving a training vocabulary of 15K tokens across 19M total mentions  User profiles are similarly diverse: 50% of users have consumed $\le $6 recipes, while 10% of users have consumed $>$45 recipes 
We order reviews by timestamp, keeping the most recent review for each user as the test set, the second most recent for validation, and the remainder for training (sequential leave-one-out evaluation BIBREF27)  We evaluate only on recipes not in the training set 
We manually construct a list of 58 cooking techniques from 384 cooking actions collected by BIBREF15; the most common techniques (bake, combine, pour, boil) account for 36 5% of technique mentions  We approximate technique adherence via string match between the recipe text and technique list 
Experiments and Results
For training and evaluation, we provide our model with the first 3-5 ingredients listed in each recipe  We decode recipe text via top-$k$ sampling BIBREF7, finding $k=3$ to produce satisfactory results  We use a hidden size $d_h=256$ for both the encoder and decoder  Embedding dimensions for vocabulary, ingredient, recipe, techniques, and caloric level are 300, 10, 50, 50, and 5 (respectively)  For prior recipe attention, we set $k=20$, the 80th %-ile for the number of user interactions  We use the Adam optimizer BIBREF28 with a learning rate of $10^{-3}$, annealed with a decay rate of 0 9 BIBREF29  We also use teacher-forcing BIBREF30 in all training epochs 
In this work, we investigate how leveraging historical user preferences can improve generation quality over strong baselines in our setting  We compare our personalized models against two baselines  The first is a name-based Nearest-Neighbor model (NN)  We initially adapted the Neural Checklist Model of BIBREF0 as a baseline; however, we ultimately use a simple Encoder-Decoder baseline with ingredient attention (Enc-Dec), which provides comparable performance and lower complexity  All personalized models outperform baseline in BPE perplexity (tab:metricsontest) with Prior Name performing the best  While our models exhibit comparable performance to baseline in BLEU-1/4 and ROUGE-L, we generate more diverse (Distinct-1/2: percentage of distinct unigrams and bigrams) and acceptable recipes  BLEU and ROUGE are not the most appropriate metrics for generation quality  A `correct' recipe can be written in many ways with the same main entities (ingredients)  As BLEU-1/4 capture structural information via n-gram matching, they are not correlated with subjective recipe quality  This mirrors observations from BIBREF31, BIBREF8 
We observe that personalized models make more diverse recipes than baseline  They thus perform better in BLEU-1 with more key entities (ingredient mentions) present, but worse in BLEU-4, as these recipes are written in a personalized way and deviate from gold on the phrasal level  Similarly, the `Prior Name' model generates more unigram-diverse recipes than other personalized models and obtains a correspondingly lower BLEU-1 score 
Qualitative Analysis: We present sample outputs for a cocktail recipe in tab:samplerecipes, and additional recipes in the appendix  Generation quality progressively improves from generic baseline output to a blended cocktail produced by our best performing model  Models attending over prior recipes explicitly reference ingredients  The Prior Name model further suggests the addition of lemon and mint, which are reasonably associated with previously consumed recipes like coconut mousse and pork skewers 
Personalization: To measure personalization, we evaluate how closely the generated text corresponds to a particular user profile  We compute the likelihood of generated recipes using identical input specifications but conditioned on ten different user profiles—one `gold' user who consumed the original recipe, and nine randomly generated user profiles  Following BIBREF8, we expect the highest likelihood for the recipe conditioned on the gold user  We measure user matching accuracy (UMA)—the proportion where the gold user is ranked highest—and Mean Reciprocal Rank (MRR) BIBREF32 of the gold user  All personalized models beat baselines in both metrics, showing our models personalize generated recipes to the given user profiles  The Prior Name model achieves the best UMA and MRR by a large margin, revealing that prior recipe names are strong signals for personalization  Moreover, the addition of attention mechanisms to capture these signals improves language modeling performance over a strong non-personalized baseline 
Recipe Level Coherence: A plausible recipe should possess a coherent step order, and we evaluate this via a metric for recipe-level coherence  We use the neural scoring model from BIBREF33 to measure recipe-level coherence for each generated recipe  Each recipe step is encoded by BERT BIBREF34  Our scoring model is a GRU network that learns the overall recipe step ordering structure by minimizing the cosine similarity of recipe step hidden representations presented in the correct and reverse orders  Once pretrained, our scorer calculates the similarity of a generated recipe to the forward and backwards ordering of its corresponding gold label, giving a score equal to the difference between the former and latter  A higher score indicates better step ordering (with a maximum score of 2)  tab:coherencemetrics shows that our personalized models achieve average recipe-level coherence scores of 1 78-1 82, surpassing the baseline at 1 77 
Recipe Step Entailment: Local coherence is also crucial to a user following a recipe: it is crucial that subsequent steps are logically consistent with prior ones  We model local coherence as an entailment task: predicting the likelihood that a recipe step follows the preceding  We sample several consecutive (positive) and non-consecutive (negative) pairs of steps from each recipe  We train a BERT BIBREF34 model to predict the entailment score of a pair of steps separated by a [SEP] token, using the final representation of the [CLS] token  The step entailment score is computed as the average of scores for each set of consecutive steps in each recipe, averaged over every generated recipe for a model, as shown in tab:coherencemetrics 
Human Evaluation: We presented 310 pairs of recipes for pairwise comparison BIBREF8 (details in appendix) between baseline and each personalized model, with results shown in tab:metricsontest  On average, human evaluators preferred personalized model outputs to baseline 63% of the time, confirming that personalized attention improves the semantic plausibility of generated recipes  We also performed a small-scale human coherence survey over 90 recipes, in which 60% of users found recipes generated by personalized models to be more coherent and preferable to those generated by baseline models 
Conclusion
In this paper, we propose a novel task: to generate personalized recipes from incomplete input specifications and user histories  On a large novel dataset of 180K recipes and 700K reviews, we show that our personalized generative models can generate plausible, personalized, and coherent recipes preferred by human evaluators for consumption  We also introduce a set of automatic coherence measures for instructional texts as well as personalization metrics to support our claims  Our future work includes generating structured representations of recipes to handle ingredient properties, as well as accounting for references to collections of ingredients (e g  “dry mix") 
Acknowledgements  This work is partly supported by NSF #1750063  We thank all reviewers for their constructive suggestions, as well as Rei M , Sujoy P , Alicia L , Eric H , Tim S , Kathy C , Allen C , and Micah I  for their feedback 
Appendix ::: Food com: Dataset Details
Our raw data consists of 270K recipes and 1 4M user-recipe interactions (reviews) scraped from Food com, covering a period of 18 years (January 2000 to December 2018)  See tab:int-stats for dataset summary statistics, and tab:samplegk for sample information about one user-recipe interaction and the recipe involved 
Appendix ::: Generated Examples
See tab:samplechx for a sample recipe for chicken chili and tab:samplewaffle for a sample recipe for sweet waffles 
Human Evaluation
We prepared a set of 15 pairwise comparisons per evaluation session, and collected 930 pairwise evaluations (310 per personalized model) over 62 sessions  For each pair, users were given a partial recipe specification (name and 3-5 key ingredients), as well as two generated recipes labeled `A' and `B'  One recipe is generated from our baseline encoder-decoder model and one recipe is generated by one of our three personalized models (Prior Tech, Prior Name, Prior Recipe)  The order of recipe presentation (A/B) is randomly selected for each question  A screenshot of the user evaluation interface is given in fig:exeval  We ask the user to indicate which recipe they find more coherent, and which recipe best accomplishes the goal indicated by the recipe name  A screenshot of this survey interface is given in fig:exeval2 
Context 3: Introduction
This work is licenced under a Creative Commons Attribution 4 0 International License  License details: http://creativecommons org/licenses/by/4 0/ Deep neural networks have been widely used in text classification and have achieved promising results BIBREF0 , BIBREF1 , BIBREF2   Most focus on content information and use models such as convolutional neural networks (CNN) BIBREF3 or recursive neural networks BIBREF4   However, for user-generated posts on social media like Facebook or Twitter, there is more information that should not be ignored  On social media platforms, a user can act either as the author of a post or as a reader who expresses his or her comments about the post 
In this paper, we classify posts taking into account post authorship, likes, topics, and comments  In particular, users and their “likes” hold strong potential for text mining  For example, given a set of posts that are related to a specific topic, a user's likes and dislikes provide clues for stance labeling  From a user point of view, users with positive attitudes toward the issue leave positive comments on the posts with praise or even just the post's content; from a post point of view, positive posts attract users who hold positive stances  We also investigate the influence of topics: different topics are associated with different stance labeling tendencies and word usage  For example we discuss women's rights and unwanted babies on the topic of abortion, but we criticize medicine usage or crime when on the topic of marijuana BIBREF5   Even for posts on a specific topic like nuclear power, a variety of arguments are raised: green energy, radiation, air pollution, and so on  As for comments, we treat them as additional text information  The arguments in the comments and the commenters (the users who leave the comments) provide hints on the post's content and further facilitate stance classification 
In this paper, we propose the user-topic-comment neural network (UTCNN), a deep learning model that utilizes user, topic, and comment information  We attempt to learn user and topic representations which encode user interactions and topic influences to further enhance text classification, and we also incorporate comment information  We evaluate this model on a post stance classification task on forum-style social media platforms  The contributions of this paper are as follows: 1  We propose UTCNN, a neural network for text in modern social media channels as well as legacy social media, forums, and message boards — anywhere that reveals users, their tastes, as well as their replies to posts  2  When classifying social media post stances, we leverage users, including authors and likers  User embeddings can be generated even for users who have never posted anything  3  We incorporate a topic model to automatically assign topics to each post in a single topic dataset  4  We show that overall, the proposed method achieves the highest performance in all instances, and that all of the information extracted, whether users, topics, or comments, still has its contributions 
Extra-Linguistic Features for Stance Classification
In this paper we aim to use text as well as other features to see how they complement each other in a deep learning model  In the stance classification domain, previous work has showed that text features are limited, suggesting that adding extra-linguistic constraints could improve performance BIBREF6 , BIBREF7 , BIBREF8   For example, Hasan and Ng as well as Thomas et al  require that posts written by the same author have the same stance BIBREF9 , BIBREF10   The addition of this constraint yields accuracy improvements of 1–7% for some models and datasets  Hasan and Ng later added user-interaction constraints and ideology constraints BIBREF7 : the former models the relationship among posts in a sequence of replies and the latter models inter-topic relationships, e g , users who oppose abortion could be conservative and thus are likely to oppose gay rights 
For work focusing on online forum text, since posts are linked through user replies, sequential labeling methods have been used to model relationships between posts  For example, Hasan and Ng use hidden Markov models (HMMs) to model dependent relationships to the preceding post BIBREF9 ; Burfoot et al  use iterative classification to repeatedly generate new estimates based on the current state of knowledge BIBREF11 ; Sridhar et al  use probabilistic soft logic (PSL) to model reply links via collaborative filtering BIBREF12   In the Facebook dataset we study, we use comments instead of reply links  However, as the ultimate goal in this paper is predicting not comment stance but post stance, we treat comments as extra information for use in predicting post stance 
Deep Learning on Extra-Linguistic Features
In recent years neural network models have been applied to document sentiment classification BIBREF13 , BIBREF4 , BIBREF14 , BIBREF15 , BIBREF2   Text features can be used in deep networks to capture text semantics or sentiment  For example, Dong et al  use an adaptive layer in a recursive neural network for target-dependent Twitter sentiment analysis, where targets are topics such as windows 7 or taylor swift BIBREF16 , BIBREF17 ; recursive neural tensor networks (RNTNs) utilize sentence parse trees to capture sentence-level sentiment for movie reviews BIBREF4 ; Le and Mikolov predict sentiment by using paragraph vectors to model each paragraph as a continuous representation BIBREF18   They show that performance can thus be improved by more delicate text models 
Others have suggested using extra-linguistic features to improve the deep learning model  The user-word composition vector model (UWCVM) BIBREF19 is inspired by the possibility that the strength of sentiment words is user-specific; to capture this they add user embeddings in their model  In UPNN, a later extension, they further add a product-word composition as product embeddings, arguing that products can also show different tendencies of being rated or reviewed BIBREF20   Their addition of user information yielded 2–10% improvements in accuracy as compared to the above-mentioned RNTN and paragraph vector methods  We also seek to inject user information into the neural network model  In comparison to the research of Tang et al  on sentiment classification for product reviews, the difference is two-fold  First, we take into account multiple users (one author and potentially many likers) for one post, whereas only one user (the reviewer) is involved in a review  Second, we add comment information to provide more features for post stance classification  None of these two factors have been considered previously in a deep learning model for text stance classification  Therefore, we propose UTCNN, which generates and utilizes user embeddings for all users — even for those who have not authored any posts — and incorporates comments to further improve performance 
Method
In this section, we first describe CNN-based document composition, which captures user- and topic-dependent document-level semantic representation from word representations  Then we show how to add comment information to construct the user-topic-comment neural network (UTCNN) 
User- and Topic-dependent Document Composition
As shown in Figure FIGREF4 , we use a general CNN BIBREF3 and two semantic transformations for document composition   We are given a document with an engaged user INLINEFORM0 , a topic INLINEFORM1 , and its composite INLINEFORM2 words, each word INLINEFORM3 of which is associated with a word embedding INLINEFORM4 where INLINEFORM5 is the vector dimension  For each word embedding INLINEFORM6 , we apply two dot operations as shown in Equation EQREF6 : DISPLAYFORM0
where INLINEFORM0 models the user reading preference for certain semantics, and INLINEFORM1 models the topic semantics; INLINEFORM2 and INLINEFORM3 are the dimensions of transformed user and topic embeddings respectively  We use INLINEFORM4 to model semantically what each user prefers to read and/or write, and use INLINEFORM5 to model the semantics of each topic  The dot operation of INLINEFORM6 and INLINEFORM7 transforms the global representation INLINEFORM8 to a user-dependent representation  Likewise, the dot operation of INLINEFORM9 and INLINEFORM10 transforms INLINEFORM11 to a topic-dependent representation 
After the two dot operations on INLINEFORM0 , we have user-dependent and topic-dependent word vectors INLINEFORM1 and INLINEFORM2 , which are concatenated to form a user- and topic-dependent word vector INLINEFORM3   Then the transformed word embeddings INLINEFORM4 are used as the CNN input  Here we apply three convolutional layers on the concatenated transformed word embeddings INLINEFORM5 : DISPLAYFORM0
where INLINEFORM0 is the index of words; INLINEFORM1 is a non-linear activation function (we use INLINEFORM2 ); INLINEFORM5 is the convolutional filter with input length INLINEFORM6 and output length INLINEFORM7 , where INLINEFORM8 is the window size of the convolutional operation; and INLINEFORM9 and INLINEFORM10 are the output and bias of the convolution layer INLINEFORM11 , respectively  In our experiments, the three window sizes INLINEFORM12 in the three convolution layers are one, two, and three, encoding unigram, bigram, and trigram semantics accordingly 
After the convolutional layer, we add a maximum pooling layer among convolutional outputs to obtain the unigram, bigram, and trigram n-gram representations  This is succeeded by an average pooling layer for an element-wise average of the three maximized convolution outputs 
UTCNN Model Description
Figure FIGREF10 illustrates the UTCNN model  As more than one user may interact with a given post, we first add a maximum pooling layer after the user matrix embedding layer and user vector embedding layer to form a moderator matrix embedding INLINEFORM0 and a moderator vector embedding INLINEFORM1 for moderator INLINEFORM2 respectively, where INLINEFORM3 is used for the semantic transformation in the document composition process, as mentioned in the previous section  The term moderator here is to denote the pseudo user who provides the overall semantic/sentiment of all the engaged users for one document  The embedding INLINEFORM4 models the moderator stance preference, that is, the pattern of the revealed user stance: whether a user is willing to show his preference, whether a user likes to show impartiality with neutral statements and reasonable arguments, or just wants to show strong support for one stance  Ideally, the latent user stance is modeled by INLINEFORM5 for each user  Likewise, for topic information, a maximum pooling layer is added after the topic matrix embedding layer and topic vector embedding layer to form a joint topic matrix embedding INLINEFORM6 and a joint topic vector embedding INLINEFORM7 for topic INLINEFORM8 respectively, where INLINEFORM9 models the semantic transformation of topic INLINEFORM10 as in users and INLINEFORM11 models the topic stance tendency  The latent topic stance is also modeled by INLINEFORM12 for each topic 
As for comments, we view them as short documents with authors only but without likers nor their own comments  Therefore we apply document composition on comments although here users are commenters (users who comment)  It is noticed that the word embeddings INLINEFORM0 for the same word in the posts and comments are the same, but after being transformed to INLINEFORM1 in the document composition process shown in Figure FIGREF4 , they might become different because of their different engaged users  The output comment representation together with the commenter vector embedding INLINEFORM2 and topic vector embedding INLINEFORM3 are concatenated and a maximum pooling layer is added to select the most important feature for comments  Instead of requiring that the comment stance agree with the post, UTCNN simply extracts the most important features of the comment contents; they could be helpful, whether they show obvious agreement or disagreement  Therefore when combining comment information here, the maximum pooling layer is more appropriate than other pooling or merging layers  Indeed, we believe this is one reason for UTCNN's performance gains 
Finally, the pooled comment representation, together with user vector embedding INLINEFORM0 , topic vector embedding INLINEFORM1 , and document representation are fed to a fully connected network, and softmax is applied to yield the final stance label prediction for the post 
Experiment
We start with the experimental dataset and then describe the training process as well as the implementation of the baselines  We also implement several variations to reveal the effects of features: authors, likers, comment, and commenters  In the results section we compare our model with related work 
Dataset
We tested the proposed UTCNN on two different datasets: FBFans and CreateDebate  FBFans is a privately-owned, single-topic, Chinese, unbalanced, social media dataset, and CreateDebate is a public, multiple-topic, English, balanced, forum dataset  Results using these two datasets show the applicability and superiority for different topics, languages, data distributions, and platforms 
The FBFans dataset contains data from anti-nuclear-power Chinese Facebook fan groups from September 2013 to August 2014, including posts and their author and liker IDs  There are a total of 2,496 authors, 505,137 likers, 33,686 commenters, and 505,412 unique users  Two annotators were asked to take into account only the post content to label the stance of the posts in the whole dataset as supportive, neutral, or unsupportive (hereafter denoted as Sup, Neu, and Uns)  Sup/Uns posts were those in support of or against anti-reconstruction; Neu posts were those evincing a neutral standpoint on the topic, or were irrelevant  Raw agreement between annotators is 0 91, indicating high agreement  Specifically, Cohen’s Kappa for Neu and not Neu labeling is 0 58 (moderate), and for Sup or Uns labeling is 0 84 (almost perfect)  Posts with inconsistent labels were filtered out, and the development and testing sets were randomly selected from what was left  Posts in the development and testing sets involved at least one user who appeared in the training set  The number of posts for each stance is shown on the left-hand side of Table TABREF12   About twenty percent of the posts were labeled with a stance, and the number of supportive (Sup) posts was much larger than that of the unsupportive (Uns) ones: this is thus highly skewed data, which complicates stance classification  On average, 161

ground_truth: 57,505 sentences

57,505 sentences

prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. What are the baseline models?
Context 3: 1 users were involved in one post  The maximum was 23,297 and the minimum was one (the author)  For comments, on average there were 3 comments per post  The maximum was 1,092 and the minimum was zero 
To test whether the assumption of this paper – posts attract users who hold the same stance to like them – is reliable, we examine the likes from authors of different stances  Posts in FBFans dataset are used for this analysis  We calculate the like statistics of each distinct author from these 32,595 posts  As the numbers of authors in the Sup, Neu and Uns stances are largely imbalanced, these numbers are normalized by the number of users of each stance  Table TABREF13 shows the results  Posts with stances (i e , not neutral) attract users of the same stance  Neutral posts also attract both supportive and neutral users, like what we observe in supportive posts, but just the neutral posts can attract even more neutral likers  These results do suggest that users prefer posts of the same stance, or at least posts of no obvious stance which might cause annoyance when reading, and hence support the user modeling in our approach 
The CreateDebate dataset was collected from an English online debate forum discussing four topics: abortion (ABO), gay rights (GAY), Obama (OBA), and marijuana (MAR)  The posts are annotated as for (F) and against (A)  Replies to posts in this dataset are also labeled with stance and hence use the same data format as posts  The labeling results are shown in the right-hand side of Table TABREF12   We observe that the dataset is more balanced than the FBFans dataset  In addition, there are 977 unique users in the dataset  To compare with Hasan and Ng's work, we conducted five-fold cross-validation and present the annotation results as the average number of all folds BIBREF9 , BIBREF5  
The FBFans dataset has more integrated functions than the CreateDebate dataset; thus our model can utilize all linguistic and extra-linguistic features  For the CreateDebate dataset, on the other hand, the like and comment features are not available (as there is a stance label for each reply, replies are evaluated as posts as other previous work) but we still implemented our model using the content, author, and topic information 
Settings
In the UTCNN training process, cross-entropy was used as the loss function and AdaGrad as the optimizer  For FBFans dataset, we learned the 50-dimensional word embeddings on the whole dataset using GloVe BIBREF21 to capture the word semantics; for CreateDebate dataset we used the publicly available English 50-dimensional word embeddings, pre-trained also using GloVe  These word embeddings were fixed in the training process  The learning rate was set to 0 03  All user and topic embeddings were randomly initialized in the range of [-0 1 0 1]  Matrix embeddings for users and topics were sized at 250 ( INLINEFORM0 ); vector embeddings for users and topics were set to length 10 
We applied the LDA topic model BIBREF22 on the FBFans dataset to determine the latent topics with which to build topic embeddings, as there is only one general known topic: nuclear power plants  We learned 100 latent topics and assigned the top three topics for each post  For the CreateDebate dataset, which itself constitutes four topics, the topic labels for posts were used directly without additionally applying LDA 
For the FBFans data we report class-based f-scores as well as the macro-average f-score ( INLINEFORM0 ) shown in equation EQREF19   DISPLAYFORM0
where INLINEFORM0 and INLINEFORM1 are the average precision and recall of the three class  We adopted the macro-average f-score as the evaluation metric for the overall performance because (1) the experimental dataset is severely imbalanced, which is common for contentious issues; and (2) for stance classification, content in minor-class posts is usually more important for further applications  For the CreateDebate dataset, accuracy was adopted as the evaluation metric to compare the results with related work BIBREF7 , BIBREF9 , BIBREF12  
Baselines
We pit our model against the following baselines: 1) SVM with unigram, bigram, and trigram features, which is a standard yet rather strong classifier for text features; 2) SVM with average word embedding, where a document is represented as a continuous representation by averaging the embeddings of the composite words; 3) SVM with average transformed word embeddings (the INLINEFORM0 in equation EQREF6 ), where a document is represented as a continuous representation by averaging the transformed embeddings of the composite words; 4) two mature deep learning models on text classification, CNN BIBREF3 and Recurrent Convolutional Neural Networks (RCNN) BIBREF0 , where the hyperparameters are based on their work; 5) the above SVM and deep learning models with comment information; 6) UTCNN without user information, representing a pure-text CNN model where we use the same user matrix and user embeddings INLINEFORM1 and INLINEFORM2 for each user; 7) UTCNN without the LDA model, representing how UTCNN works with a single-topic dataset; 8) UTCNN without comments, in which the model predicts the stance label given only user and topic information  All these models were trained on the training set, and parameters as well as the SVM kernel selections (linear or RBF) were fine-tuned on the development set  Also, we adopt oversampling on SVMs, CNN and RCNN because the FBFans dataset is highly imbalanced 
Results on FBFans Dataset
In Table TABREF22 we show the results of UTCNN and the baselines on the FBFans dataset  Here Majority yields good performance on Neu since FBFans is highly biased to the neutral class  The SVM models perform well on Sup and Neu but perform poorly for Uns, showing that content information in itself is insufficient to predict stance labels, especially for the minor class  With the transformed word embedding feature, SVM can achieve comparable performance as SVM with n-gram feature  However, the much fewer feature dimension of the transformed word embedding makes SVM with word embeddings a more efficient choice for modeling the large scale social media dataset  For the CNN and RCNN models, they perform slightly better than most of the SVM models but still, the content information is insufficient to achieve a good performance on the Uns posts  As to adding comment information to these models, since the commenters do not always hold the same stance as the author, simply adding comments and post contents together merely adds noise to the model 
Among all UTCNN variations, we find that user information is most important, followed by topic and comment information  UTCNN without user information shows results similar to SVMs — it does well for Sup and Neu but detects no Uns  Its best f-scores on both Sup and Neu among all methods show that with enough training data, content-based models can perform well; at the same time, the lack of user information results in too few clues for minor-class posts to either predict their stance directly or link them to other users and posts for improved performance  The 17 5% improvement when adding user information suggests that user information is especially useful when the dataset is highly imbalanced  All models that consider user information predict the minority class successfully  UCTNN without topic information works well but achieves lower performance than the full UTCNN model  The 4 9% performance gain brought by LDA shows that although it is satisfactory for single topic datasets, adding that latent topics still benefits performance: even when we are discussing the same topic, we use different arguments and supporting evidence  Lastly, we get 4 8% improvement when adding comment information and it achieves comparable performance to UTCNN without topic information, which shows that comments also benefit performance  For platforms where user IDs are pixelated or otherwise hidden, adding comments to a text model still improves performance  In its integration of user, content, and comment information, the full UTCNN produces the highest f-scores on all Sup, Neu, and Uns stances among models that predict the Uns class, and the highest macro-average f-score overall  This shows its ability to balance a biased dataset and supports our claim that UTCNN successfully bridges content and user, topic, and comment information for stance classification on social media text  Another merit of UTCNN is that it does not require a balanced training data  This is supported by its outperforming other models though no oversampling technique is applied to the UTCNN related experiments as shown in this paper  Thus we can conclude that the user information provides strong clues and it is still rich even in the minority class 
We also investigate the semantic difference when a user acts as an author/liker or a commenter  We evaluated a variation in which all embeddings from the same user were forced to be identical (this is the UTCNN shared user embedding setting in Table TABREF22 )  This setting yielded only a 2 5% improvement over the model without comments, which is not statistically significant  However, when separating authors/likers and commenters embeddings (i e , the UTCNN full model), we achieved much greater improvements (4 8%)  We attribute this result to the tendency of users to use different wording for different roles (for instance author vs commenter)  This is observed when the user, acting as an author, attempts to support her argument against nuclear power by using improvements in solar power; when acting as a commenter, though, she interacts with post contents by criticizing past politicians who supported nuclear power or by arguing that the proposed evacuation plan in case of a nuclear accident is ridiculous  Based on this finding, in the final UTCNN setting we train two user matrix embeddings for one user: one for the author/liker role and the other for the commenter role 
Results on CreateDebate Dataset
Table TABREF24 shows the results of UTCNN, baselines as we implemented on the FBFans datset and related work on the CreateDebate dataset  We do not adopt oversampling on these models because the CreateDebate dataset is almost balanced  In previous work, integer linear programming (ILP) or linear-chain conditional random fields (CRFs) were proposed to integrate text features, author, ideology, and user-interaction constraints, where text features are unigram, bigram, and POS-dependencies; the author constraint tends to require that posts from the same author for the same topic hold the same stance; the ideology constraint aims to capture inferences between topics for the same author; the user-interaction constraint models relationships among posts via user interactions such as replies BIBREF7 , BIBREF9  
The SVM with n-gram or average word embedding feature performs just similar to the majority  However, with the transformed word embedding, it achieves superior results  It shows that the learned user and topic embeddings really capture the user and topic semantics  This finding is not so obvious in the FBFans dataset and it might be due to the unfavorable data skewness for SVM  As for CNN and RCNN, they perform slightly better than most SVMs as we found in Table TABREF22 for FBFans 
Compared to the ILP BIBREF7 and CRF BIBREF9 methods, the UTCNN user embeddings encode author and user-interaction constraints, where the ideology constraint is modeled by the topic embeddings and text features are modeled by the CNN  The significant improvement achieved by UTCNN suggests the latent representations are more effective than overt model constraints 
The PSL model BIBREF12 jointly labels both author and post stance using probabilistic soft logic (PSL) BIBREF23 by considering text features and reply links between authors and posts as in Hasan and Ng's work  Table TABREF24 reports the result of their best AD setting, which represents the full joint stance/disagreement collective model on posts and is hence more relevant to UTCNN  In contrast to their model, the UTCNN user embeddings represent relationships between authors, but UTCNN models do not utilize link information between posts  Though the PSL model has the advantage of being able to jointly label the stances of authors and posts, its performance on posts is lower than the that for the ILP or CRF models  UTCNN significantly outperforms these models on posts and has the potential to predict user stances through the generated user embeddings 
For the CreateDebate dataset, we also evaluated performance when not using topic embeddings or user embeddings; as replies in this dataset are viewed as posts, the setting without comment embeddings is not available  Table TABREF24 shows the same findings as Table TABREF22 : the 21% improvement in accuracy demonstrates that user information is the most vital  This finding also supports the results in the related work: user constraints are useful and can yield 11 2% improvement in accuracy BIBREF7   Further considering topic information yields 3 4% improvement, suggesting that knowing the subject of debates provides useful information  In sum, Table TABREF22 together with Table TABREF24 show that UTCNN achieves promising performance regardless of topic, language, data distribution, and platform 
Conclusion
We have proposed UTCNN, a neural network model that incorporates user, topic, content and comment information for stance classification on social media texts  UTCNN learns user embeddings for all users with minimum active degree, i e , one post or one like  Topic information obtained from the topic model or the pre-defined labels further improves the UTCNN model  In addition, comment information provides additional clues for stance classification  We have shown that UTCNN achieves promising and balanced results  In the future we plan to explore the effectiveness of the UTCNN user embeddings for author stance classification 
Acknowledgements
Research of this paper was partially supported by Ministry of Science and Technology, Taiwan, under the contract MOST 104-2221-E-001-024-MY2 

ground_truth: name-based Nearest-Neighbor model (NN), Encoder-Decoder baseline with ingredient attention (Enc-Dec)

prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. What are the baselines?
Context 4:  Further, the lexical gap between the sentences in AFS is much larger  Hence, simple unsupervised methods as well as state-of-the-art STS systems perform badly on this dataset BIBREF24 
We evaluate SBERT on this dataset in two scenarios: 1) As proposed by Misra et al , we evaluate SBERT using 10-fold cross-validation  A draw-back of this evaluation setup is that it is not clear how well approaches generalize to different topics  Hence, 2) we evaluate SBERT in a cross-topic setup  Two topics serve for training and the approach is evaluated on the left-out topic  We repeat this for all three topics and average the results 
SBERT is fine-tuned using the Regression Objective Function  The similarity score is computed using cosine-similarity based on the sentence embeddings  We also provide the Pearson correlation $r$ to make the results comparable to Misra et al  However, we showed BIBREF22 that Pearson correlation has some serious drawbacks and should be avoided for comparing STS systems  The results are depicted in Table TABREF12 
Unsupervised methods like tf-idf, average GloVe embeddings or InferSent perform rather badly on this dataset with low scores  Training SBERT in the 10-fold cross-validation setup gives a performance that is nearly on-par with BERT 
However, in the cross-topic evaluation, we observe a performance drop of SBERT by about 7 points Spearman correlation  To be considered similar, arguments should address the same claims and provide the same reasoning  BERT is able to use attention to compare directly both sentences (e g  word-by-word comparison), while SBERT must map individual sentences from an unseen topic to a vector space such that arguments with similar claims and reasons are close  This is a much more challenging task, which appears to require more than just two topics for training to work on-par with BERT 
Evaluation - Semantic Textual Similarity ::: Wikipedia Sections Distinction
ein-dor-etal-2018-learning use Wikipedia to create a thematically fine-grained train, dev and test set for sentence embeddings methods  Wikipedia articles are separated into distinct sections focusing on certain aspects  Dor et al  assume that sentences in the same section are thematically closer than sentences in different sections  They use this to create a large dataset of weakly labeled sentence triplets: The anchor and the positive example come from the same section, while the negative example comes from a different section of the same article  For example, from the Alice Arnold article: Anchor: Arnold joined the BBC Radio Drama Company in 1988 , positive: Arnold gained media attention in May 2012 , negative: Balding and Arnold are keen amateur golfers 
We use the dataset from Dor et al  We use the Triplet Objective, train SBERT for one epoch on the about 1 8 Million training triplets and evaluate it on the 222,957 test triplets  Test triplets are from a distinct set of Wikipedia articles  As evaluation metric, we use accuracy: Is the positive example closer to the anchor than the negative example?
Results are presented in Table TABREF14  Dor et al  fine-tuned a BiLSTM architecture with triplet loss to derive sentence embeddings for this dataset  As the table shows, SBERT clearly outperforms the BiLSTM approach by Dor et al 
Evaluation - SentEval
SentEval BIBREF6 is a popular toolkit to evaluate the quality of sentence embeddings  Sentence embeddings are used as features for a logistic regression classifier  The logistic regression classifier is trained on various tasks in a 10-fold cross-validation setup and the prediction accuracy is computed for the test-fold 
The purpose of SBERT sentence embeddings are not to be used for transfer learning for other tasks  Here, we think fine-tuning BERT as described by devlin2018bert for new tasks is the more suitable method, as it updates all layers of the BERT network  However, SentEval can still give an impression on the quality of our sentence embeddings for various tasks 
We compare the SBERT sentence embeddings to other sentence embeddings methods on the following seven SentEval transfer tasks:
MR: Sentiment prediction for movie reviews snippets on a five start scale BIBREF25 
CR: Sentiment prediction of customer product reviews BIBREF26 
SUBJ: Subjectivity prediction of sentences from movie reviews and plot summaries BIBREF27 
MPQA: Phrase level opinion polarity classification from newswire BIBREF28 
SST: Stanford Sentiment Treebank with binary labels BIBREF29 
TREC: Fine grained question-type classification from TREC BIBREF30 
MRPC: Microsoft Research Paraphrase Corpus from parallel news sources BIBREF31 
The results can be found in Table TABREF15  SBERT is able to achieve the best performance in 5 out of 7 tasks  The average performance increases by about 2 percentage points compared to InferSent as well as the Universal Sentence Encoder  Even though transfer learning is not the purpose of SBERT, it outperforms other state-of-the-art sentence embeddings methods on this task 
It appears that the sentence embeddings from SBERT capture well sentiment information: We observe large improvements for all sentiment tasks (MR, CR, and SST) from SentEval in comparison to InferSent and Universal Sentence Encoder 
The only dataset where SBERT is significantly worse than Universal Sentence Encoder is the TREC dataset  Universal Sentence Encoder was pre-trained on question-answering data, which appears to be beneficial for the question-type classification task of the TREC dataset 
Average BERT embeddings or using the CLS-token output from a BERT network achieved bad results for various STS tasks (Table TABREF6), worse than average GloVe embeddings  However, for SentEval, average BERT embeddings and the BERT CLS-token output achieves decent results (Table TABREF15), outperforming average GloVe embeddings  The reason for this are the different setups  For the STS tasks, we used cosine-similarity to estimate the similarities between sentence embeddings  Cosine-similarity treats all dimensions equally  In contrast, SentEval fits a logistic regression classifier to the sentence embeddings  This allows that certain dimensions can have higher or lower impact on the classification result 
We conclude that average BERT embeddings / CLS-token output from BERT return sentence embeddings that are infeasible to be used with cosine-similarity or with Manhatten / Euclidean distance  For transfer learning, they yield slightly worse results than InferSent or Universal Sentence Encoder  However, using the described fine-tuning setup with a siamese network structure on NLI datasets yields sentence embeddings that achieve a new state-of-the-art for the SentEval toolkit 
Ablation Study
We have demonstrated strong empirical results for the quality of SBERT sentence embeddings  In this section, we perform an ablation study of different aspects of SBERT in order to get a better understanding of their relative importance 
We evaluated different pooling strategies (MEAN, MAX, and CLS)  For the classification objective function, we evaluate different concatenation methods  For each possible configuration, we train SBERT with 10 different random seeds and average the performances 
The objective function (classification vs  regression) depends on the annotated dataset  For the classification objective function, we train SBERT-base on the SNLI and the Multi-NLI dataset  For the regression objective function, we train on the training set of the STS benchmark dataset  Performances are measured on the development split of the STS benchmark dataset  Results are shown in Table TABREF23 
When trained with the classification objective function on NLI data, the pooling strategy has a rather minor impact  The impact of the concatenation mode is much larger  InferSent BIBREF4 and Universal Sentence Encoder BIBREF5 both use $(u, v, |u-v|, u*v)$ as input for a softmax classifier  However, in our architecture, adding the element-wise $u*v$ decreased the performance 
The most important component is the element-wise difference $|u-v|$  Note, that the concatenation mode is only relevant for training the softmax classifier  At inference, when predicting similarities for the STS benchmark dataset, only the sentence embeddings $u$ and $v$ are used in combination with cosine-similarity  The element-wise difference measures the distance between the dimensions of the two sentence embeddings, ensuring that similar pairs are closer and dissimilar pairs are further apart 
When trained with the regression objective function, we observe that the pooling strategy has a large impact  There, the MAX strategy perform significantly worse than MEAN or CLS-token strategy  This is in contrast to BIBREF4, who found it beneficial for the BiLSTM-layer of InferSent to use MAX instead of MEAN pooling 
Computational Efficiency
Sentence embeddings need potentially be computed for Millions of sentences, hence, a high computation speed is desired  In this section, we compare SBERT to average GloVe embeddings, InferSent BIBREF4, and Universal Sentence Encoder BIBREF5 
For our comparison we use the sentences from the STS benchmark BIBREF10  We compute average GloVe embeddings using a simple for-loop with python dictionary lookups and NumPy  InferSent is based on PyTorch  For Universal Sentence Encoder, we use the TensorFlow Hub version, which is based on TensorFlow  SBERT is based on PyTorch  For improved computation of sentence embeddings, we implemented a smart batching strategy: Sentences with similar lengths are grouped together and are only padded to the longest element in a mini-batch  This drastically reduces computational overhead from padding tokens 
Performances were measured on a server with Intel i7-5820K CPU @ 3 30GHz, Nvidia Tesla V100 GPU, CUDA 9 2 and cuDNN  The results are depicted in Table TABREF26 
On CPU, InferSent is about 65% faster than SBERT  This is due to the much simpler network architecture  InferSent uses a single BiLSTM layer, while BERT uses 12 stacked transformer layers  However, an advantage of transformer networks is the computational efficiency on GPUs  There, SBERT with smart batching is about 9% faster than InferSent and about 55% faster than Universal Sentence Encoder  Smart batching achieves a speed-up of 89% on CPU and 48% on GPU  Average GloVe embeddings is obviously by a large margin the fastest method to compute sentence embeddings 
Conclusion
We showed that BERT out-of-the-box maps sentences to a vector space that is rather unsuitable to be used with common similarity measures like cosine-similarity  The performance for seven STS tasks was below the performance of average GloVe embeddings 
To overcome this shortcoming, we presented Sentence-BERT (SBERT)  SBERT fine-tunes BERT in a siamese / triplet network architecture  We evaluated the quality on various common benchmarks, where it could achieve a significant improvement over state-of-the-art sentence embeddings methods  Replacing BERT with RoBERTa did not yield a significant improvement in our experiments 
SBERT is computationally efficient  On a GPU, it is about 9% faster than InferSent and about 55% faster than Universal Sentence Encoder  SBERT can be used for tasks which are computationally not feasible to be modeled with BERT  For example, clustering of 10,000 sentences with hierarchical clustering requires with BERT about 65 hours, as around 50 Million sentence combinations must be computed  With SBERT, we were able to reduce the effort to about 5 seconds 
Acknowledgments
This work has been supported by the German Research Foundation through the German-Israeli Project Cooperation (DIP, grant DA 1600/1-1 and grant GU 798/17-1)  It has been co-funded by the German Federal Ministry of Education and Research (BMBF) under the promotional references 03VP02540 (ArgumenText) 

ground_truth: SVM with unigram, bigram, and trigram features, SVM with average word embedding, SVM with average transformed word embeddings, CNN, ecurrent Convolutional Neural Networks, SVM and deep learning models with comment information

SVM with unigram, bigram, trigram features, with average word embedding, with average transformed word embeddings, CNN and RCNN, SVM, CNN, RCNN with comment information