prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. Do they use attention?
Context 1: Background
Teaching machine to read and comprehend a given passage/paragraph and answer its corresponding questions is a challenging task  It is also one of the long-term goals of natural language understanding, and has important applications in e g , building intelligent agents for conversation and customer service support  In a real world setting, it is necessary to judge whether the given questions are answerable given the available knowledge, and then generate correct answers for the ones which are able to infer an answer in the passage or an empty answer (as an unanswerable question) otherwise 
In comparison with many existing MRC systems BIBREF0 , BIBREF1 , BIBREF2 , BIBREF3 , BIBREF4 , which extract answers by finding a sub-string in the passages/paragraphs, we propose a model that not only extracts answers but also predicts whether such an answer should exist  Using a multi-task learning approach (c f  BIBREF5 ), we extend the Stochastic Answer Network (SAN) BIBREF1 for MRC answer span detector to include a classifier that whether the question is unanswerable  The unanswerable classifier is a pair-wise classification model BIBREF6 which predicts a label indicating whether the given pair of a passage and a question is unanswerable  The two models share the same lower layer to save the number of parameters, and separate the top layers for different tasks (the span detector and binary classifier) 
Our model is pretty simple and intuitive, yet efficient  Without relying on the large pre-trained language models (ELMo) BIBREF7 , the proposed model achieves competitive results to the state-of-the-art on Stanford Question Answering Dataset (SQuAD) 2 0 
The contribution of this work is summarized as follows  First, we propose a simple yet efficient model for MRC that handles unanswerable questions and is optimized jointly  Second, our model achieves competitive results on SQuAD v2 0 
Model
The Machine Reading Comprehension is a task which takes a question INLINEFORM0 and a passage/paragraph INLINEFORM1 as inputs, and aims to find an answer span INLINEFORM2 in INLINEFORM3   We assume that if the question is answerable, the answer INLINEFORM4 exists in INLINEFORM5 as a contiguous text string; otherwise, INLINEFORM6 is an empty string indicating an unanswerable question  Note that to handle the unanswerable questions, we manually append a dumpy text string NULL at the end of each corresponding passage/paragraph  Formally, the answer is formulated as INLINEFORM7   In case of unanswerable questions, INLINEFORM8 points to the last token of the passage 
Our model is a variation of SAN BIBREF1 , as shown in Figure FIGREF2   The main difference is the additional binary classifier added in the model justifying whether the question is unanswerable  Roughly, the model includes two different layers: the shared layer and task specific layer  The shared layer is almost identical to the lower layers of SAN, which has a lexicon encoding layer, a contextual layer and a memory generation layer  On top of it, there are different answer modules for different tasks  We employ the SAN answer module for the span detector and a one-layer feed forward neural network for the binary classification task  It can also be viewed as a multi-task learning BIBREF8 , BIBREF5 , BIBREF9   We will briefly describe the model from ground up as follows  Detailed descriptions can be found in BIBREF1  
Lexicon Encoding Layer  We map the symbolic/surface feature of INLINEFORM0 and INLINEFORM1 into neural space via word embeddings , 16-dim part-of-speech (POS) tagging embeddings, 8-dim named-entity embeddings and 4-dim hard-rule features  Note that we use small embedding size of POS and NER to reduce model size and they mainly serve the role of coarse-grained word clusters  Additionally, we use question enhanced passages word embeddings which can viewwed as soft matching between questions and passages  At last, we use two separate two-layer position-wise Feed-Forward Networks (FFN) BIBREF11 , BIBREF1 to map both question and passage encodings into the same dimension  As results, we obtain the final lexicon embeddings for the tokens for INLINEFORM2 as a matrix INLINEFORM3 , and tokens in INLINEFORM4 as INLINEFORM5  
Contextual Encoding Layer  A shared two-layers BiLSTM is used on the top to encode the contextual information of both passages and questions  To avoid overfitting, we concatenate a pre-trained 600-dimensional CoVe vectors BIBREF12 trained on German-English machine translation dataset, with the aforementioned lexicon embeddings as the final input of the contextual encoding layer, and also with the output of the first contextual encoding layer as the input of its second encoding layer  Thus, we obtain the final representation of the contextual encoding layer by a concatenation of the outputs of two BiLSTM: INLINEFORM0 for questions and INLINEFORM1 for passages 
Memory Generation Layer  In this layer, we generate a working memory by fusing information from both passages INLINEFORM0 and questions INLINEFORM1   The attention function BIBREF11 is used to compute the similarity score between passages and questions as: INLINEFORM2
Note that INLINEFORM0 and INLINEFORM1 is transformed from INLINEFORM2 and INLINEFORM3 by one layer neural network INLINEFORM4 , respectively  A question-aware passage representation is computed as INLINEFORM5   After that, we use the method of BIBREF13 to apply self attention to the passage: INLINEFORM6
where INLINEFORM0 means that we only drop diagonal elements on the similarity matrix (i e , attention with itself)  At last, INLINEFORM1 and INLINEFORM2 are concatenated and are passed through a BiLSTM to form the final memory: INLINEFORM3  
Span detector  We adopt a multi-turn answer module for the span detector BIBREF1   Formally, at time step INLINEFORM0 in the range of INLINEFORM1 , the state is defined by INLINEFORM2   The initial state INLINEFORM3 is the summary of the INLINEFORM4 : INLINEFORM5 , where INLINEFORM6   Here, INLINEFORM7 is computed from the previous state INLINEFORM8 and memory INLINEFORM9 : INLINEFORM10 and INLINEFORM11   Finally, a bilinear function is used to find the begin and end point of answer spans at each reasoning step INLINEFORM12 : DISPLAYFORM0 DISPLAYFORM1
The final prediction is the average of each time step: INLINEFORM0   We randomly apply dropout on the step level in each time step during training, as done in BIBREF1  
Unanswerable classifier  We adopt a one-layer neural network as our unanswerable binary classifier: DISPLAYFORM0
, where INLINEFORM0 is the summary of the memory: INLINEFORM1 , where INLINEFORM2   INLINEFORM3 denotes the probability of the question which is unanswerable 
Objective The objective function of the joint model has two parts: DISPLAYFORM0
Following BIBREF0 , the span loss function is defined: DISPLAYFORM0
The objective function of the binary classifier is defined: DISPLAYFORM0
where INLINEFORM0 is a binary variable: INLINEFORM1 indicates the question is unanswerable and INLINEFORM2 denotes the question is answerable 
Setup
We evaluate our system on SQuAD 2 0 dataset BIBREF14 , a new MRC dataset which is a combination of Stanford Question Answering Dataset (SQuAD) 1 0 BIBREF15 and additional unanswerable question-answer pairs  The answerable pairs are around 100K; while the unanswerable questions are around 53K  This dataset contains about 23K passages and they come from approximately 500 Wikipedia articles  All the questions and answers are obtained by crowd-sourcing  Two evaluation metrics are used: Exact Match (EM) and Macro-averaged F1 score (F1) BIBREF14  
Implementation details
We utilize spaCy tool to tokenize the both passages and questions, and generate lemma, part-of-speech and named entity tags  The word embeddings are initialized with pre-trained 300-dimensional GloVe BIBREF10   A 2-layer BiLSTM is used encoding the contextual information of both questions and passages  Regarding the hidden size of our model, we search greedily among INLINEFORM0   During training, Adamax BIBREF16 is used as our optimizer  The min-batch size is set to 32  The learning rate is initialized to 0 002 and it is halved after every 10 epochs  The dropout rate is set to 0 1  To prevent overfitting, we also randomly set 0 5% words in both passages and questions as unknown words during the training  Here, we use a special token unk to indicate a word which doesn't appear in GloVe  INLINEFORM1 in Eq EQREF9 is set to 1 
Results
We would like to investigate effectiveness the proposed joint model  To do so, the same shared layer/architecture is employed in the following variants of the proposed model:
The results in terms of EM and F1 is summarized in Table TABREF20   We observe that Joint SAN outperforms the SAN baseline with a large margin, e g , 67 89 vs 69 27 (+1 38) and 70 68 vs 72 20 (+1 52) in terms of EM and F1 scores respectively, so it demonstrates the effectiveness of the joint optimization  By incorporating the output information of classifier into Joint SAN, it obtains a slight improvement, e g , 72 2 vs 72 66 (+0 46) in terms of F1 score  By analyzing the results, we found that in most cases when our model extract an NULL string answer, the classifier also predicts it as an unanswerable question with a high probability 
Table TABREF21 reports comparison results in literature published   Our model achieves state-of-the-art on development dataset in setting without pre-trained large language model (ELMo)  Comparing with the much complicated model R M -Reader + Verifier, which includes several components, our model still outperforms by 0 7 in terms of F1 score  Furthermore, we observe that ELMo gives a great boosting on the performance, e g , 2 8 points in terms of F1 for DocQA  This encourages us to incorporate ELMo into our model in future 
Analysis  To better understand our model, we analyze the accuracy of the classifier in our joint model  We obtain 75 3 classification accuracy on the development with the threshold 0 5  By increasing value of INLINEFORM0 in Eq EQREF9 , the classification accuracy reached to 76 8 ( INLINEFORM1 ), however the final results of our model only have a small improvement (+0 2 in terms of F1 score)  It shows that it is important to make balance between these two components: the span detector and unanswerable classifier 
Conclusion
To sum up, we proposed a simple yet efficient model based on SAN  It showed that the joint learning algorithm boosted the performance on SQuAD 2 0  We also would like to incorporate ELMo into our model in future 
Acknowledgments
We thank Yichong Xu, Shuohang Wang and Sheng Zhang for valuable discussions and comments  We also thank Robin Jia for the help on SQuAD evaluations 
Context 2: Introduction
Headline generation is the process of creating a headline-style sentence given an input article  The research community has been regarding the task of headline generation as a summarization task BIBREF1, ignoring the fundamental differences between headlines and summaries  While summaries aim to contain most of the important information from the articles, headlines do not necessarily need to  Instead, a good headline needs to capture people's attention and serve as an irresistible invitation for users to read through the article  For example, the headline “$2 Billion Worth of Free Media for Trump”, which gives only an intriguing hint, is considered better than the summarization style headline “Measuring Trump’s Media Dominance” , as the former gets almost three times the readers as the latter  Generating headlines with many clicks is especially important in this digital age, because many of the revenues of journalism come from online advertisements and getting more user clicks means being more competitive in the market  However, most existing websites naively generate sensational headlines using only keywords or templates  Instead, this paper aims to learn a model that generates sensational headlines based on an input article without labeled data 
To generate sensational headlines, there are two main challenges  Firstly, there is a lack of sensationalism scorer to measure how sensational a headline is  Some researchers have tried to manually label headlines as clickbait or non-clickbait BIBREF2, BIBREF3  However, these human-annotated datasets are usually small and expensive to collect  To capture a large variety of sensationalization patterns, we need a cheap and easy way to collect a large number of sensational headlines  Thus, we propose a distant supervision strategy to collect a sensationalism dataset  We regard headlines receiving lots of comments as sensational samples and the headlines generated by a summarization model as non-sensational samples  Experimental results show that by distinguishing these two types of headlines, we can partially teach the model a sense of being sensational 
Secondly, after training a sensationalism scorer on our sensationalism dataset, a natural way to generate sensational headlines is to maximize the sensationalism score using reinforcement learning (RL)  However, the following shows an example of a RL model maximizing the sensationalism score by generating a very unnatural sentence, while its sensationalism scorer gave a very high score of 0 99996: UTF8gbsn 十个可穿戴产品的设计原则这消息消息可惜说明 Ten design principles for wearable devices, this message message pity introduction  This happens because the sensationalism scorer can make mistakes and RL can generate unnatural phrases which fools our sensationalism scorer  Thus, how to effectively leverage RL with noisy rewards remains an open problem  To deal with the noisy reward, we introduce Auto-tuned Reinforcement Learning (ARL)  Our model automatically tunes the ratio between MLE and RL based on how sensational the training headline is  In this way, we effectively take advantage of RL with a noisy reward to generate headlines that are both sensational and fluent 
The major contributions of this paper are as follows: 1) To the best of our knowledge, we propose the first-ever model that tackles the sensational headline generation task with reinforcement learning techniques  2) Without human-annotated data, we propose a distant supervision strategy to train a sensationalism scorer as a reward function 3) We propose a novel loss function, Auto-tuned Reinforcement Learning, to give dynamic weights to balance between MLE and RL  Our code will be released  
Sensationalism Scorer
To evaluate the sensationalism intensity score $\alpha _{\text{sen}}$ of a headline, we collect a sensationalism dataset and then train a sensationalism scorer  For the sensationalism dataset collection, we choose headlines with many comments from popular online websites as positive samples  For the negative samples, we propose to use the generated headlines from a sentence summarization model  Intuitively, the summarization model, which is trained to preserve the semantic meaning, will lose the sensationalization ability and thus the generated negative samples will be less sensational than the original one, similar to the obfuscation of style after back-translation BIBREF4  For example, an original headline like UTF8gbsn“一趟挣10万？铁总增开申通、顺丰专列" (One trip to earn 100 thousand? China Railway opens new Shentong and Shunfeng special lines) will become UTF8gbsn“中铁总将增开京广两列快递专列" (China Railway opens two special lines for express) from the baseline model, which loses the sensational phrases of UTF8gbsn“一趟挣10万？" (One trip to earn 100 thousand?)   We then train the sensationalism scorer by classifying sensational and non-sensational headlines using a one-layer CNN with a binary cross entropy loss $L_{\text{sen}}$  Firstly, 1-D convolution is used to extract word features from the input embeddings of a headline  This is followed by a ReLU activation layer and a max-pooling layer along the time dimension  All features from different channels are concatenated together and projected to the sensationalism score by adding another fully connected layer with sigmoid activation  Binary cross entropy is used to compute the loss $L_{\text{sen}}$ 
Sensationalism Scorer ::: Training Details and Dataset
For the CNN model, we choose filter sizes of 1, 3, and 5 respectively  Adam is used to optimize $L_{sen}$ with a learning rate of 0 0001  We set the embedding size as 300 and initialize it from qiu2018revisiting trained on the Weibo corpus with word and character features  We fix the embeddings during training  For dataset collection, we utilize the headlines collected in qin2018automatic, lin2019learning from Tencent News, one of the most popular Chinese news websites, as the positive samples  We follow the same data split as the original paper  As some of the links are not available any more, we get 170,754 training samples and 4,511 validation samples  For the negative training samples collection, we randomly select generated headlines from a pointer generator BIBREF0 model trained on LCSTS dataset BIBREF5 and create a balanced training corpus which includes 351,508 training samples and 9,022 validation samples  To evaluate our trained classifier, we construct a test set by randomly sampling 100 headlines from the test split of LCSTS dataset and the labels are obtained by 11 human annotators  Annotations show that 52% headlines are labeled as positive and 48% headlines as negative by majority voting (The detail on the annotation can be found in Section SECREF26) 
Sensationalism Scorer ::: Results and Discussion
Our classifier achieves 0 65 accuracy and 0 65 averaged F1 score on the test set while a random classifier would only achieve 0 50 accuracy and 0 50 averaged F1 score  This confirms that the predicted sensationalism score can partially capture the sensationalism of headlines  On the other hand, a more natural choice is to take headlines with few comments as negative examples  Thus, we train another baseline classifier on a crawled balanced sensationalism corpus of 84k headlines where the positive headlines have at least 28 comments and the negative headlines have less than 5 comments  However, the results on the test set show that the baseline classifier gets 60% accuracy, which is worse than the proposed classifier (which achieves 65%)  The reason could be that the balanced sensationalism corpus are sampled from different distributions from the test set and it is hard for the trained model to generalize  Therefore, we choose the proposed one as our sensationalism scorer  Therefore, our next challenge is to show that how to leverage this noisy sensationalism reward to generate sensational headlines 
Sensational Headline Generation
Our sensational headline generation model takes an article as input and output a sensational headline  The model consists of a Pointer-Gen headline generator and is trained by ARL  The diagram of ARL can be found in Figure FIGREF6 
We denote the input article as $x=\lbrace x_1,x_2,x_3,\cdots ,x_M\rbrace $, and the corresponding headline as $y^*=\lbrace y_1^*,y_2^*,y_3^*,\cdots ,y_T^*\rbrace $, where $M$ is the number of tokens in an article and $T$ is the number of tokens in a headline 
Sensational Headline Generation ::: Pointer-Gen Headline Generator
We choose Pointer Generator (Pointer-Gen) BIBREF0, a widely used summarization model, as our headline generator for its ability to copy words from the input article  It takes a news article as input and generates a headline  Firstly, the tokens of each article, $\lbrace x_1,x_2,x_3,\cdots ,x_M\rbrace $, are fed into the encoder one-by-one and the encoder generates a sequence of hidden states $h_i$  For each decoding step $t$, the decoder receives the embedding for each token of a headline $y_t$ as input and updates its hidden states $s_t$  An attention mechanism following luong2015effective is used:
where $v$, $W_h$, $W_s$, and $b_{attn}$ are the trainable parameters and $h_t^*$ is the context vector  $s_t$ and $h_t^*$ are then combined to give a probability distribution over the vocabulary through two linear layers:
where $V$, $b$, $V^{^{\prime }}$, and $b^{^{\prime }}$ are trainable parameters  We use a pointer generator network to enable our model to copy rare/unknown words from the input article, giving the following final word probability:
where $x^t$ is the embedding of the input word of the decoder, $w_{h^*}^T$, $w_s^T$, $w_x^T$, and $b_{ptr}$ are trainable parameters, and $\sigma $ is the sigmoid function 
Sensational Headline Generation ::: Training Methods
We first briefly introduce MLE and RL objective functions, and a naive way to mix these two by a hyper-parameter $\lambda $  Then we point out the challenge of training with noisy reward, and propose ARL to address this issue 
Sensational Headline Generation ::: Training Methods ::: MLE and RL
A headline generation model can be trained with MLE, RL or a combination of MLE and RL  MLE training is to minimize the negative log likelihood of the training headlines  We feed $y^*$ into the decoder word by word and maximize the likelihood of $y^*$  The loss function for MLE becomes
For RL training, we choose the REINFORCE algorithm BIBREF6  In the training phase, after encoding an article, a headline $y^s = \lbrace y_1^s, y_2^s, y_3^s, \cdots , y_T^s\rbrace $ is obtained by sampling from $P(w)$ from our generator, and then a reward of sensationalism or ROUGE(RG) is calculated 
We use the baseline reward $\hat{R_t}$ to reduce the variance of the reward, similar to ranzato2015sequence  To elaborate, a linear model is deployed to estimate the baseline reward $\hat{R_t}$ based on $t$-th state $o_t$ for each timestep $t$  The parameters of the linear model are trained by minimizing the mean square loss between $R$ and $\hat{R_t}$:
where $W_r$ and $b_r$ are trainable parameters  To maximize the expected reward, our loss function for RL becomes
A naive way to mix these two objective functions using a hyper-parameter $\lambda $ has been successfully incorporated in the summarization task BIBREF7  It includes the MLE training as a language model to mitigate the readability and quality issues in RL  The mixed loss function is shown as follows:
where $*$ is the reward type  Usually $\lambda $ is large, and paulus2017deep used 0 9984 
Sensational Headline Generation ::: Training Methods ::: Auto-tuned Reinforcement Learning
Applying the naive mixed training method using sensationalism score as the reward is not obvious/trivial in our task  The main reason is that our sensationalism reward is notably more noisy and more fragile than the ROUGE-L reward or abstractive reward used in the summarization task BIBREF7, BIBREF8  A higher ROUGE-L F1 reward in summarization indicates higher overlapping ratio between generation and true summary statistically, but our sensationalism reward is a learned score which is fragile to be fooled with unnatural samples 
To effectively train the model with RL under noisy sensationalism reward, our idea is to balance RL with MLE  However, we argue that the weighted ratio between MLE and RL should be sample-dependent, instead of being fixed for all training samples as in paulus2017deep, kryscinski2018improving  The reason is that, RL and MLE have inconsistent optimization objectives  When the training headline is non-sensational, MLE training will encourage our model to imitate the training headline (thus generating non-sensational headlines), which counteracts the effects of RL training to generate sensational headlines 
The sensationalism score is, therefore, used to give dynamic weight to MLE and RL  Our ARL loss function becomes:
If $\alpha _{\text{sen}}(y^*)$ is high, meaning the training headline is sensational, our loss function encourages our model to imitate the sample more using the MLE training  If $\alpha _{\text{sen}}(y^*)$ is low, our loss function replies on RL training to improve the sensationalism  Note that the weight $\alpha _{\text{sen}}(y^*)$ is different from our sensationalism reward $\alpha _{\text{sen}}(y^s)$ and we call the loss function Auto-tuned Reinforcement Learning, because the ratio between MLE and RL are well “tuned” towards different samples 
Sensational Headline Generation ::: Dataset
We use LCSTS BIBREF5 as our dataset to train the summarization model  The dataset is collected from the Chinese microblogging website Sina Weibo  It contains over 2 million Chinese short texts with corresponding headlines given by the author of each text  The dataset is split into 2,400,591 samples for training, 10,666 samples for validation and 725 samples for testing  We tokenize each sentence with Jieba and a vocabulary size of 50000 is saved 
Sensational Headline Generation ::: Baselines and Our Models
We experiment and compare with the following models 
Pointer-Gen is the baseline model trained by optimizing $L_\text{MLE}$ in Equation DISPLAY_FORM13 
Pointer-Gen+Pos is the baseline model by training Pointer-Gen only on positive examples whose sensationalism score is larger than 0 5
Pointer-Gen+Same-FT is the model which fine-tunes Pointer-Gen on the training samples whose sensationalism score is larger than 0 1
Pointer-Gen+Pos-FT is the model which fine-tunes Pointer-Gen on the training samples whose sensationalism score is larger than 0 5
Pointer-Gen+RL-ROUGE is the baseline model trained by optimizing $L_\text{RL-ROUGE}$ in Equation DISPLAY_FORM17, with ROUGE-L BIBREF9 as the reward 
Pointer-Gen+RL-SEN is the baseline model trained by optimizing $L_\text{RL-SEN}$ in Equation DISPLAY_FORM17, with $\alpha _\text{sen}$ as the reward 
Pointer-Gen+ARL-SEN is our model trained by optimizing $L_\text{ARL-SEN}$ in Equation DISPLAY_FORM19, with $\alpha _\text{sen}$ as the reward 
Test set is the headlines from the test set 
Note that we didn't compare to Pointer-Gen+ARL-ROUGE as it is actually Pointer-GEN  Recall that $\alpha _{\text{sen}}(y^*)$ in Equation DISPLAY_FORM19 measures how good (based on reward function) is $y^*$  Then the loss function for Pointer-Gen+ARL-ROUGE will be
We also tried text style transfer baseline BIBREF10, but the generated headlines were very poor (many unknown words and irrelevant) 
Sensational Headline Generation ::: Training Details
MLE training: An Adam optimizer is used with the learning rate of 0 0001 to optimize $L_{\text{MLE}}$  The batch size is set as 128 and a one-layer, bi-directional Long Short-Term Memory (bi-LSTM) model with 512 hidden sizes and a 350 embedding size is utilized  Gradients with the l2 norm larger than 2 0 are clipped  We stop training when the ROUGE-L f-score stops increasing 
Hybrid training: An Adam optimizer with a learning rate of 0 0001 is used to optimize $L_{\text{RL-*}}$ (Equation DISPLAY_FORM17) and $L_\text{{ARL-SEN}}$ (Equation DISPLAY_FORM19)  When training Pointer-Gen+RL-ROUGE, the best $\lambda $ is chosen based on the ROUGE-L score on the validation set  In our experiment, $\lambda $ is set as 0 95  An Adam optimizer with a learning rate of 0 001 is used to optimize $L_b$  When training Pointer-Gen+ARL-SEN, we don't use the full LCSTS dataset, but only headlines with a sensationalism score larger than 0 1 as we observe that Pointer-Gen+ARL-SEN will generate a few unnatural phrases when using full dataset  We believe the reason is the high ratio of RL during training  Figure FIGREF23 shows that the probability density near 0 is very high, meaning that in each batch, many of the samples will have a very low sensationalism score  On expectation, each sample will receive 0 239 MLE training and 0 761 RL training  This leads to RL dominanting the loss  Thus, we propose to filter samples with a minimum sensationalism score with 0 1 and it works very well  For Pointer-Gen+RL-SEN, we also set the minimum sensationalism score as 0 1, and $\lambda $ is set as 0 5 to remove unnatural phrases, making a fair comparison to Pointer-Gen+ARL-SEN 
We stop training Pointer-Gen+Same-FT, Pointer-Gen+Pos-FT, Pointer-Gen+RL-SEN and Pointer-Gen+ARL-SEN, when $\alpha _\text{sen}$ stops increasing on the validation set  Beam-search with a beam size of 5 is adopted for decoding in all models 
Sensational Headline Generation ::: Evaluation Metrics
We briefly describe the evaluation metrics below 
ROUGE: ROUGE is a commonly used evaluation metric for summarization  It measures the N-gram overlap between generated and training headlines  We use it to evaluate the relevance of generated headlines  The widely used pyrouge toolkit is used to calculate ROUGE-1 (RG-1), ROUGE-2 (RG-2), and ROUGE-L (RG-L) 
Human evaluation: We randomly sample 50 articles from the test set and send the generated headlines from all models and corresponding headlines in the test set to human annotators  We evaluate the sensationalism and fluency of the headlines by setting up two independent human annotation tasks  We ask 10 annotators to label each headline for each task  For the sensationalism annotation, each annotator is asked one question, “Is the headline sensational?”, and he/she has to choose either `yes' or `no'  The annotators were not told which system the headline is from  The process of distributing samples and recruiting annotators is managed by Crowdflower  After annotation, we define the sensationalism score as the proportion of annotations on all generated headlines from one model labeled as `yes'  For the fluency annotation, we repeat the same procedure as for the sensationalism annotation, except that we ask each annotator the question “Is the headline fluent?” We define the fluency score as the proportion of annotations on all headlines from one specific model labeled as `yes'  We put human annotation instructions in the supplemental material 
UTF8gbsn
Results
We first compare all four models, Pointer-Gen, Pointer-Gen-RL+ROUGE, Pointer-Gen-RL-SEN, and Pointer-Gen-ARL-SEN, to existing models with ROUGE in Table TABREF25 to establish that our model produces relevant headlines and we leave the sensationalism for human evaluation  Note that we only compare our models to commonly used strong summarization baselines, to validate that our implementation achieves comparable performance to existing work  In our implementation, Pointer-Gen achieves a 34 51 RG-1 score, 22 21 RG-2 score, and 31 68 RG-L score, which is similar to the results of gu2016incorporating  Pointer-Gen+ARL-SEN, although optimized for the sensationalism reward, achieves similar performance to our Pointer-Gen baseline, which means that Pointer-Gen+ARL-SEN still keeps its summarization ability  An example of headlines generated from different models in Table TABREF29 shows that Pointer-Gen and Pointer-Gen+RL-ROUGE learns to summarize the main point of the article: “The Nikon D600 camera is reported to have black spots when taking photos”  Pointer-Gen+RL-SEN makes the headline more sensational by blaming Nikon for attributing the damage to the smog  Pointer-Gen+ARL-SEN generates the most sensational headline by exaggerating the result “Getting a serious trouble!” to maximize user's attention 
We then compare different models using the sensationalism score in Table TABREF30  The Pointer-Gen baseline model achieves a 42 6% sensationalism score, which is the minimum that a typical summarization model achieves  By filtering out low-sensational headlines, Pointer-Gen+Same-FT and Pointer-Gen+Pos-FT achieves higher sensationalism scores, which implies the effectiveness of our sensationalism scorer  Our Pointer-Gen+ARL-SEN model achieves the best performance of 60 8%  This is an absolute improvement of 18 2% over the Pointer-Gen baseline  The Chi-square test on the results confirms that Pointer-Gen+ARL-SEN is statistically significantly more sensational than all the other baseline models, with the largest p-value less than 0 01  Also, we find that the test set headlines achieves 57 8% sensationalism score, much larger than Pointer-Gen baseline, which also supports our intuition that generated headlines will be less sensational than the original one  On the other hand, we found that Pointer-Gen+Pos is much worse than other baselines  The reason is that training on sensational samples alone discards around 80% of the whole training set that is also helpful for maintaining relevance and a good language model  It shows the necessity of using RL 
UTF8gbsn
In addition, both Pointer-Gen+RL-SEN and Pointer-Gen+ARL-SEN, which use the sensationalism score as the reward, obtain statistically better results than Pointer-Gen+RL-ROUGE and Pointer-Gen, with a p-value less than 0 05 by a Chi-square test  This result shows the effectiveness of RL to generate more sensational headlines  The reason is that even though our noisy classifier could also learn to classify domains, the generator during RL training is not allowed to increase the reward by shifting domains but encouraged to generate more sensational headlines, due to the consistency constraint on the domains of the headline and the article  Furthermore, Poiner-Gen+ARL-SEN gets better performance than Pointer-Gen+RL-SEN, which confirms the superiority of the ARL loss function  We also visualize in Figure FIGREF31 a comparison between Pointer-Gen+ARL-SEN and Pointer-Gen+RL-SEN according to how sensational the test set headlines are  The blue bars denote the smaller scores between the two models  For example, if the blue bar is 0 6, it means that the worse model between Pointer-Gen+RL-SEN and Pointer-Gen+ARL-SEN achieves 0 6  And the color of orange/black further indicates the better model and its score  We find that Pointer-Gen+ARL-SEN outperforms Pointer-Gen+RL-SEN for most cases  The improvement is higher when the test set headlines are not sensational (the sensationalism score is less than 0 5), which may be attributed to the higher ratio of RL training on non-sensational headlines 
Apart from the sensationalism evaluation, we measure the fluency of the headlines generated from different models  Fluency scores in Table TABREF30 show that Pointer-Gen+RL-SEN and Pointer-Gen+ARL-SEN achieve comparable fluency performance to Pointer-Gen and Pointer-Gen+RL-ROUGE  Test set headlines achieve the best performance among all models, but the difference is not statistically significant  Also, we observe that fine-tuning on sensational headlines will hurt the performance, both in sensationalism and fluency 
After manually checking the outputs, we observe that our model is able to generate sensational headlines using diverse sensationalization strategies  These strategies include, but are not limited to, creating a curiosity gap, asking questions, highlighting numbers, being emotional and emphasizing the user  Examples can be found in Table TABREF32 
Related Work
Our work is related to summarization tasks  An encoder-decoder model was first applied to two sentence-level abstractive summarization tasks on the DUC-2004 and Gigaword datasets BIBREF12  This model was later extended by selective encoding BIBREF13, a coarse to fine approach BIBREF14, minimum risk training BIBREF1, and topic-aware models BIBREF15  As long summaries were recognized as important, the CNN/Daily Mail dataset was used in nallapati2016abstractive  Graph-based attention BIBREF16, pointer-generator with coverage loss BIBREF0 are further developed to improve the generated summaries  celikyilmaz2018deep proposed deep communicating agents for representing a long document for abstractive summarization  In addition, many papers BIBREF17, BIBREF18, BIBREF19 use extractive methods to directly select sentences from articles  However, none of these work considered the sensationalism of generated outputs 
RL is also gaining popularity as it can directly optimize non-differentiable metrics BIBREF20, BIBREF21, BIBREF22  paulus2017deep proposed an intra-decoder model and combined RL and MLE to deal with summaries with bad qualities  RL has also been explored with generative adversarial networks (GANs) BIBREF23  liu2017generative applied GANs on summarization task and achieved better performance  niu2018polite tackles the problem of polite generation with politeness reward  Our work is different in that we propose a novel function to balance RL and MLE 
Our task is also related to text style transfer  Implicit methods BIBREF10, BIBREF24, BIBREF4 transfer the styles by separating sentence representations into content and style, for example using back-translationBIBREF4  However, these methods cannot guarantee the content consistency between the original sentence and transferred output BIBREF25  Explicit methods BIBREF26, BIBREF25 transfer the style by directly identifying style related keywords and modifying them  However, sensationalism is not always restricted to keywords, but the full sentence  By leveraging small human labeled English dataset, clickbait detection has been well investigated BIBREF2, BIBREF27, BIBREF3  However, these human labeled dataset are not available for other languages, such as Chinese 
Modeling sensationalism is also related to modeling emotion  Emotion has been well investigated in both word levelBIBREF28, BIBREF29 and sentence levelBIBREF30, BIBREF31, BIBREF32, BIBREF33, BIBREF34  It has also been considered an important factor in engaging interactive systemsBIBREF35, BIBREF36, BIBREF37  Although we observe that sensational headlines contain emotion, it is still not clear which emotion and how emotions will influence the sensationalism 
Conclusion and Future Work
In this paper, we propose a model that generates sensational headlines without labeled data using Reinforcement Learning  Firstly, we propose a distant supervision strategy to train the sensationalism scorer  As a result, we achieve 65% accuracy between the predicted sensationalism score and human evaluation  To effectively leverage this noisy sensationalism score as the reward for RL, we propose a novel loss function, ARL, to automatically balance RL with MLE  Human evaluation confirms the effectiveness of both our sensationalism scorer and ARL to generate more sensational headlines  Future work can be improving the sensationalism scorer and investigating the applications of dynamic balancing methods between RL and MLE in textGANBIBREF23  Our work also raises the ethical questions about generating sensational headlines, which can be further explored 
Acknowledgments
Thanks to ITS/319/16FP of Innovation Technology Commission, HKUST 16248016 of Hong Kong Research Grants Council for funding  In addition, we thank Zhaojiang Lin for helpful discussion and Yan Xu, Zihan Liu for the data collection 
Context 3: Introduction
We posses a wealth of prior knowledge about many natural language processing tasks  For example, in text categorization, we know that words such as NBA, player, and basketball are strong indicators of the sports category BIBREF0 , and words like terrible, boring, and messing indicate a negative polarity while words like perfect, exciting, and moving suggest a positive polarity in sentiment classification 
A key problem arisen here, is how to leverage such knowledge to guide the learning process, an interesting problem for both NLP and machine learning communities  Previous studies addressing the problem fall into several lines  First, to leverage prior knowledge to label data BIBREF1 , BIBREF2   Second, to encode prior knowledge with a prior on parameters, which can be commonly seen in many Bayesian approaches BIBREF3 , BIBREF4   Third, to formalise prior knowledge with additional variables and dependencies BIBREF5   Last, to use prior knowledge to control the distributions over latent output variables BIBREF6 , BIBREF7 , BIBREF8 , which makes the output variables easily interpretable 
However, a crucial problem, which has rarely been addressed, is the bias in the prior knowledge that we supply to the learning model  Would the model be robust or sensitive to the prior knowledge? Or, which kind of knowledge is appropriate for the task? Let's see an example: we may be a baseball fan but unfamiliar with hockey so that we can provide a few number of feature words of baseball, but much less of hockey for a baseball-hockey classification task  Such prior knowledge may mislead the model with heavy bias to baseball  If the model cannot handle this situation appropriately, the performance may be undesirable 
In this paper, we investigate into the problem in the framework of Generalized Expectation Criteria BIBREF7   The study aims to reveal the factors of reducing the sensibility of the prior knowledge and therefore to make the model more robust and practical  To this end, we introduce auxiliary regularization terms in which our prior knowledge is formalized as distribution over output variables  Recall the example just mentioned, though we do not have enough knowledge to provide features for class hockey, it is easy for us to provide some neutral words, namely words that are not strong indicators of any class, like player here  As one of the factors revealed in this paper, supplying neutral feature words can boost the performance remarkably, making the model more robust 
More attractively, we do not need manual annotation to label these neutral feature words in our proposed approach 
More specifically, we explore three regularization terms to address the problem: (1) a regularization term associated with neutral features; (2) the maximum entropy of class distribution regularization term; and (3) the KL divergence between reference and predicted class distribution  For the first manner, we simply use the most common features as neutral features and assume the neutral features are distributed uniformly over class labels  For the second and third one, we assume we have some knowledge about the class distribution which will be detailed soon later 
To summarize, the main contributions of this work are as follows:
The rest of the paper is structured as follows: In Section 2, we briefly describe the generalized expectation criteria and present the proposed regularization terms  In Section 3, we conduct extensive experiments to justify the proposed methods  We survey related work in Section 4, and summarize our work in Section 5 
Method
We address the robustness problem on top of GE-FL BIBREF0 , a GE method which leverages labeled features as prior knowledge  A labeled feature is a strong indicator of a specific class and is manually provided to the classifier  For example, words like amazing, exciting can be labeled features for class positive in sentiment classification 
Generalized Expectation Criteria
Generalized expectation (GE) criteria BIBREF7 provides us a natural way to directly constrain the model in the preferred direction  For example, when we know the proportion of each class of the dataset in a classification task, we can guide the model to predict out a pre-specified class distribution 
Formally, in a parameter estimation objective function, a GE term expresses preferences on the value of some constraint functions about the model's expectation  Given a constraint function $G({\rm x}, y)$ , a conditional model distribution $p_\theta (y|\rm x)$ , an empirical distribution $\tilde{p}({\rm x})$ over input samples and a score function $S$ , a GE term can be expressed as follows:
$$S(E_{\tilde{p}({\rm x})}[E_{p_\theta (y|{\rm x})}[G({\rm x}, y)]])$$   (Eq  4)
Learning from Labeled Features
Druck et al  ge-fl proposed GE-FL to learn from labeled features using generalized expectation criteria  When given a set of labeled features $K$ , the reference distribution over classes of these features is denoted by $\hat{p}(y| x_k), k \in K$   GE-FL introduces the divergence between this reference distribution and the model predicted distribution $p_\theta (y | x_k)$ , as a term of the objective function:
$$\mathcal {O} = \sum _{k \in K} KL(\hat{p}(y|x_k) || p_\theta (y | x_k)) + \sum _{y,i} \frac{\theta _{yi}^2}{2 \sigma ^2}$$   (Eq  6)
where $\theta _{yi}$ is the model parameter which indicates the importance of word $i$ to class $y$   The predicted distribution $p_\theta (y | x_k)$ can be expressed as follows: $ p_\theta (y | x_k) = \frac{1}{C_k} \sum _{\rm x} p_\theta (y|{\rm x})I(x_k) $
in which $I(x_k)$ is 1 if feature $k$ occurs in instance ${\rm x}$ and 0 otherwise, $C_k = \sum _{\rm x} I(x_k)$ is the number of instances with a non-zero value of feature $k$ , and $p_\theta (y|{\rm x})$ takes a softmax form as follows: $ p_\theta (y|{\rm x}) = \frac{1}{Z(\rm x)}\exp (\sum _i \theta _{yi}x_i)  $
To solve the optimization problem, L-BFGS can be used for parameter estimation 
In the framework of GE, this term can be obtained by setting the constraint function $G({\rm x}, y) = \frac{1}{C_k} \vec{I} (y)I(x_k)$ , where $\vec{I}(y)$ is an indicator vector with 1 at the index corresponding to label $y$ and 0 elsewhere 
Regularization Terms
GE-FL reduces the heavy load of instance annotation and performs well when we provide prior knowledge with no bias  In our experiments, we observe that comparable numbers of labeled features for each class have to be supplied  But as mentioned before, it is often the case that we are not able to provide enough knowledge for some of the classes  For the baseball-hockey classification task, as shown before, GE-FL will predict most of the instances as baseball  In this section, we will show three terms to make the model more robust 
Neutral features are features that are not informative indicator of any classes, for instance, word player to the baseball-hockey classification task  Such features are usually frequent words across all categories  When we set the preference distribution of the neutral features to be uniform distributed, these neutral features will prevent the model from biasing to the class that has a dominate number of labeled features 
Formally, given a set of neutral features $K^{^{\prime }}$ , the uniform distribution is $\hat{p}_u(y|x_k) = \frac{1}{|C|}, k \in K^{^{\prime }}$ , where $|C|$ is the number of classes  The objective function with the new term becomes
$$\mathcal {O}_{NE} = \mathcal {O} + \sum _{k \in K^{^{\prime }}} KL(\hat{p}_u(y|x_k) || p_\theta (y | x_k)) $$   (Eq  9)
Note that we do not need manual annotation to provide neutral features  One simple way is to take the most common features as neutral features  Experimental results show that this strategy works successfully 
Another way to prevent the model from drifting from the desired direction is to constrain the predicted class distribution on unlabeled data  When lacking knowledge about the class distribution of the data, one feasible way is to take maximum entropy principle, as below:
$$\mathcal {O}_{ME} = \mathcal {O} + \lambda \sum _{y} p(y) \log p(y)$$   (Eq  11)
where $p(y)$ is the predicted class distribution, given by $ p(y) = \frac{1}{|X|} \sum _{\rm x} p_\theta (y | \rm x)  $ To control the influence of this term on the overall objective function, we can tune $\lambda $ according to the difference in the number of labeled features of each class  In this paper, we simply set $\lambda $ to be proportional to the total number of labeled features, say $\lambda = \beta |K|$  
This maximum entropy term can be derived by setting the constraint function to $G({\rm x}, y) = \vec{I}(y)$   Therefore, $E_{p_\theta (y|{\rm x})}[G({\rm x}, y)]$ is just the model distribution $p_\theta (y|{\rm x})$ and its expectation with the empirical distribution $\tilde{p}(\rm x)$ is simply the average over input samples, namely $p(y)$   When $S$ takes the maximum entropy form, we can derive the objective function as above 
Sometimes, we have already had much knowledge about the corpus, and can estimate the class distribution roughly without labeling instances  Therefore, we introduce the KL divergence between the predicted and reference class distributions into the objective function 
Given the preference class distribution $\hat{p}(y)$ , we modify the objective function as follows:
$$\mathcal {O}_{KL} &= \mathcal {O} + \lambda KL(\hat{p}(y) || p(y))$$   (Eq  13)
Similarly, we set $\lambda = \beta |K|$  
This divergence term can be derived by setting the constraint function to $G({\rm x}, y) = \vec{I}(y)$ and setting the score function to $S(\hat{p}, p) = \sum _i \hat{p}_i \log \frac{\hat{p}_i}{p_i}$ , where $p$ and $\hat{p}$ are distributions  Note that this regularization term involves the reference class distribution which will be discussed later 
Experiments
In this section, we first justify the approach when there exists unbalance in the number of labeled features or in class distribution  Then, to test the influence of $\lambda $ , we conduct some experiments with the method which incorporates the KL divergence of class distribution  Last, we evaluate our approaches in 9 commonly used text classification datasets  We set $\lambda = 5|K|$ by default in all experiments unless there is explicit declaration  The baseline we choose here is GE-FL BIBREF0 , a method based on generalization expectation criteria 
Data Preparation
We evaluate our methods on several commonly used datasets whose themes range from sentiment, web-page, science to medical and healthcare  We use bag-of-words feature and remove stopwords in the preprocess stage  Though we have labels of all documents, we do not use them during the learning process, instead, we use the label of features 
The movie dataset, in which the task is to classify the movie reviews as positive or negtive, is used for testing the proposed approaches with unbalanced labeled features, unbalanced datasets or different $\lambda $ parameters  All unbalanced datasets are constructed based on the movie dataset by randomly removing documents of the positive class  For each experiment, we conduct 10-fold cross validation 
As described in BIBREF0 , there are two ways to obtain labeled features  The first way is to use information gain  We first calculate the mutual information of all features according to the labels of the documents and select the top 20 as labeled features for each class as a feature pool  Note that using information gain requires the document label, but this is only to simulate how we human provide prior knowledge to the model  The second way is to use LDA BIBREF9 to select features  We use the same selection process as BIBREF0 , where they first train a LDA on the dataset, and then select the most probable features of each topic (sorted by $P(w_i|t_j)$ , the probability of word $w_i$ given topic $t_j$ ) 
Similar to BIBREF10 , BIBREF0 , we estimate the reference distribution of the labeled features using a heuristic strategy  If there are $|C|$ classes in total, and $n$ classes are associated with a feature $k$ , the probability that feature $k$ is related with any one of the $n$ classes is $\frac{0 9}{n}$ and with any other class is $\frac{0 1}{|C| - n}$  
Neutral features are the most frequent words after removing stop words, and their reference distributions are uniformly distributed  We use the top 10 frequent words as neutral features in all experiments 
With Unbalanced Labeled Features
In this section, we evaluate our approach when there is unbalanced knowledge on the categories to be classified  The labeled features are obtained through information gain  Two settings are chosen:
(a) We randomly select $t \in [1, 20]$ features from the feature pool for one class, and only one feature for the other  The original balanced movie dataset is used (positive:negative=1:1) 
(b) Similar to (a), but the dataset is unbalanced, obtained by randomly removing 75% positive documents (positive:negative=1:4) 
As shown in Figure 1 , Maximum entropy principle shows improvement only on the balanced case

ground_truth: Yes

Yes

prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. Which baselines are used for evaluation?
Context 3:  An obvious reason is that maximum entropy only favors uniform distribution 
Incorporating Neutral features performs similarly to maximum entropy since we assume that neutral words are uniformly distributed  Its accuracy decreases slowly when the number of labeled features becomes larger ( $t>4$ ) (Figure 1 (a)), suggesting that the model gradually biases to the class with more labeled features, just like GE-FL 
Incorporating the KL divergence of class distribution performs much better than GE-FL on both balanced and unbalanced datasets  This shows that it is effective to control the unbalance in labeled features and in the dataset 
With Balanced Labeled Features
We also compare with the baseline when the labeled features are balanced  Similar to the experiment above, the labeled features are obtained by information gain  Two settings are experimented with:
(a) We randomly select $t \in [1, 20]$ features from the feature pool for each class, and conduct comparisons on the original balanced movie dataset (positive:negtive=1:1) 
(b) Similar to (a), but the class distribution is unbalanced, by randomly removing 75% positive documents (positive:negative=1:4) 
Results are shown in Figure 2   When the dataset is balanced (Figure 2 (a)), there is little difference between GE-FL and our methods  The reason is that the proposed regularization terms provide no additional knowledge to the model and there is no bias in the labeled features  On the unbalanced dataset (Figure 2 (b)), incorporating KL divergence is much better than GE-FL since we provide additional knowledge(the true class distribution), but maximum entropy and neutral features are much worse because forcing the model to approach the uniform distribution misleads it 
With Unbalanced Class Distributions
Our methods are also evaluated on datasets with different unbalanced class distributions  We manually construct several movie datasets with class distributions of 1:2, 1:3, 1:4 by randomly removing 50%, 67%, 75% positive documents  The original balanced movie dataset is used as a control group  We test with both balanced and unbalanced labeled features  For the balanced case, we randomly select 10 features from the feature pool for each class, and for the unbalanced case, we select 10 features for one class, and 1 feature for the other  Results are shown in Figure 3  
Figure 3 (a) shows that when the dataset and the labeled features are both balanced, there is little difference between our methods and GE-FL(also see Figure 2 (a))  But when the class distribution becomes more unbalanced, the difference becomes more remarkable  Performance of neutral features and maximum entropy decrease significantly but incorporating KL divergence increases remarkably  This suggests if we have more accurate knowledge about class distribution, KL divergence can guide the model to the right direction 
Figure 3 (b) shows that when the labeled features are unbalanced, our methods significantly outperforms GE-FL  Incorporating KL divergence is robust enough to control unbalance both in the dataset and in labeled features while the other three methods are not so competitive 
The Influence of λ\lambda 
We present the influence of $\lambda $ on the method that incorporates KL divergence in this section  Since we simply set $\lambda = \beta |K|$ , we just tune $\beta $ here  Note that when $\beta = 0$ , the newly introduced regularization term is disappeared, and thus the model is actually GE-FL  Again, we test the method with different $\lambda $ in two settings:
(a) We randomly select $t \in [1, 20]$ features from the feature pool for one class, and only one feature for the other class  The original balanced movie dataset is used (positive:negative=1:1) 
(b) Similar to (a), but the dataset is unbalanced, obtained by randomly removing 75% positive documents (positive:negative=1:4) 
Results are shown in Figure 4   As expected, $\lambda $ reflects how strong the regularization is  The model tends to be closer to our preferences with the increasing of $\lambda $ on both cases 
Using LDA Selected Features
We compare our methods with GE-FL on all the 9 datasets in this section  Instead of using features obtained by information gain, we use LDA to select labeled features  Unlike information gain, LDA does not employ any instance labels to find labeled features  In this setting, we can build classification models without any instance annotation, but just with labeled features 
Table 1 shows that our three methods significantly outperform GE-FL  Incorporating neutral features performs better than GE-FL on 7 of the 9 datasets, maximum entropy is better on 8 datasets, and KL divergence better on 7 datasets 
LDA selects out the most predictive features as labeled features without considering the balance among classes  GE-FL does not exert any control on such an issue, so the performance is severely suffered  Our methods introduce auxiliary regularization terms to control such a bias problem and thus promote the model significantly 
Related Work
There have been much work that incorporate prior knowledge into learning, and two related lines are surveyed here  One is to use prior knowledge to label unlabeled instances and then apply a standard learning algorithm  The other is to constrain the model directly with prior knowledge 
Liu et al text manually labeled features which are highly predictive to unsupervised clustering assignments and use them to label unlabeled data  Chang et al guiding proposed constraint driven learning  They first used constraints and the learned model to annotate unlabeled instances, and then updated the model with the newly labeled data  Daumé daume2008cross proposed a self training method in which several models are trained on the same dataset, and only unlabeled instances that satisfy the cross task knowledge constraints are used in the self training process 
MaCallum et al gec proposed generalized expectation(GE) criteria which formalised the knowledge as constraint terms about the expectation of the model into the objective function Graça et al pr proposed posterior regularization(PR) framework which projects the model's posterior onto a set of distributions that satisfy the auxiliary constraints  Druck et al ge-fl explored constraints of labeled features in the framework of GE by forcing the model's predicted feature distribution to approach the reference distribution  Andrzejewski et al andrzejewski2011framework proposed a framework in which general domain knowledge can be easily incorporated into LDA  Altendorf et al altendorf2012learning explored monotonicity constraints to improve the accuracy while learning from sparse data  Chen et al chen2013leveraging tried to learn comprehensible topic models by leveraging multi-domain knowledge 
Mann and McCallum simple,generalized incorporated not only labeled features but also other knowledge like class distribution into the objective function of GE-FL  But they discussed only from the semi-supervised perspective and did not investigate into the robustness problem, unlike what we addressed in this paper 
There are also some active learning methods trying to use prior knowledge  Raghavan et al feedback proposed to use feedback on instances and features interlacedly, and demonstrated that feedback on features boosts the model much  Druck et al active proposed an active learning method which solicits labels on features rather than on instances and then used GE-FL to train the model 
Conclusion and Discussions
This paper investigates into the problem of how to leverage prior knowledge robustly in learning models  We propose three regularization terms on top of generalized expectation criteria  As demonstrated by the experimental results, the performance can be considerably improved when taking into account these factors  Comparative results show that our proposed methods is more effective and works more robustly against baselines  To the best of our knowledge, this is the first work to address the robustness problem of leveraging knowledge, and may inspire other research 
We then present more detailed discussions about the three regularization methods  Incorporating neutral features is the simplest way of regularization, which doesn't require any modification of GE-FL but just finding out some common features  But as Figure 1 (a) shows, only using neutral features are not strong enough to handle extremely unbalanced labeled features 
The maximum entropy regularization term shows the strong ability of controlling unbalance 
This method doesn't need any extra knowledge, and is thus suitable when we know nothing about the corpus  But this method assumes that the categories are uniformly distributed, which may not be the case in practice, and it will have a degraded performance if the assumption is violated (see Figure 1 (b), Figure 2 (b), Figure 3 (a)) 
The KL divergence performs much better on unbalanced corpora than other methods  The reason is that KL divergence utilizes the reference class distribution and doesn't make any assumptions  The fact suggests that additional knowledge does benefit the model 
However, the KL divergence term requires providing the true class distribution  Sometimes, we may have the exact knowledge about the true distribution, but sometimes we may not  Fortunately, the model is insensitive to the true distribution and therefore a rough estimation of the true distribution is sufficient  In our experiments, when the true class distribution is 1:2, where the reference class distribution is set to 1:1 5/1:2/1:2 5, the accuracy is 0 755/0 756/0 760 respectively  This provides us the possibility to perform simple computing on the corpus to obtain the distribution in reality  Or, we can set the distribution roughly with domain expertise 

ground_truth: Pointer-Gen, Pointer-Gen+Pos, Pointer-Gen+Same-FT, Pointer-Gen+Pos-FT, Pointer-Gen+RL-ROUGE, Pointer-Gen+RL-SEN

prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. How do they define robustness of a model?
Context 4:  Since set 2 is directly sampled from millions of reviews, it better reflects real-life sentiment distribution 
Large-scale datasets: We further conduct experiments on four much larger datasets: IMDB (I), Yelp2014 (Y), Cell Phone (C), and Baby (B)  IMDB and Yelp2014 were previously used in BIBREF25 , BIBREF26   Cell phone and Baby are from the large-scale Amazon dataset BIBREF24 , BIBREF27   Detailed statistics are summarized in Table TABREF9   We keep all reviews in the original datasets and consider a transductive setting where all target examples are used for both training (without label information) and evaluation  We perform sampling to balance the classes of labeled source data in each minibatch INLINEFORM3 during training 
Selection of Development Set
Ideally, the development set should be drawn from the same distribution as the test set  However, under the unsupervised domain adaptation setting, we do not have any labeled target data at training phase which could be used as development set  In all of our experiments, for each pair of domains, we instead sample 1000 examples from the training set of the source domain as development set  We train the network for a fixed number of epochs, and the model with the minimum classification error on this development set is saved for evaluation  This approach works well on most of the problems since the target domain is supposed to behave like the source domain if the domain difference is effectively reduced 
Another problem is how to select the values for hyper-parameters  If we tune INLINEFORM0 and INLINEFORM1 directly on the development set from the source domain, most likely both of them will be set to 0, as unlabeled target data is not helpful for improving in-domain accuracy of the source domain  Other neural network models also have the same problem for hyper-parameter tuning  Therefore, our strategy is to use the development set from the target domain to optimize INLINEFORM2 and INLINEFORM3 for one problem (e g , we only do this on E INLINEFORM4 BK), and fix their values on the other problems  This setting assumes that we have at least two labeled domains such that we can optimize the hyper-parameters, and then we fix them for other new unlabeled domains to transfer to 
Training Details and Hyper-parameters
We initialize word embeddings using the 300-dimension GloVe vectors supplied by Pennington et al , ( BIBREF28 ), which were trained on 840 billion tokens from the Common Crawl  For each pair of domains, the vocabulary consists of the top 10000 most frequent words  For words in the vocabulary but not present in the pre-trained embeddings, we randomly initialize them 
We set hyper-parameters of the CNN encoder following previous works BIBREF22 , BIBREF4 without specific tuning on our datasets  The window size is set to 3 and the size of the hidden layer is set to 300  The nonlinear activation function is Relu  For regularization, we also follow their settings and employ dropout with probability set to 0 5 on INLINEFORM0 before feeding it to the output layer INLINEFORM1 , and constrain the INLINEFORM2 -norm of the weight vector INLINEFORM3 , setting its max norm to 3 
On the small-scale datasets and the Aamzon benchmark, INLINEFORM0 and INLINEFORM1 are set to 200 and 1, respectively, tuned on the development set of task E INLINEFORM2 BK under setting 1  On the large-scale datasets, INLINEFORM3 and INLINEFORM4 are set to 500 and 0 2, respectively, tuned on I INLINEFORM5 Y  We use a Gaussian curve INLINEFORM6 to ramp up the weight of the bootstrapping loss INLINEFORM7 from 0 to INLINEFORM8 , where INLINEFORM9 denotes the maximum number of training epochs  We train 30 epochs for all experiments  We set INLINEFORM10 to 3 and INLINEFORM11 to 0 5 for all experiments 
The batch size is set to 50 on the small-scale datasets and the Amazon benchmark  We increase the batch size to 250 on the large-scale datasets to reduce the number of iterations  RMSProp optimizer with learning rate set to 0 0005 is used for all experiments 
Models for Comparison
We compare with the following baselines:
(1) Naive: A non-domain-adaptive baseline with bag-of-words representations and SVM classifier trained on the source domain 
(2) mSDA BIBREF7 : This is the state-of-the-art method based on discrete input features  Top 1000 bag-of-words features are kept as pivot features  We set the number of stacked layers to 3 and the corruption probability to 0 5 
(3) NaiveNN: This is a non-domain-adaptive CNN trained on source domain, which is a variant of our model by setting INLINEFORM0 , INLINEFORM1 , and INLINEFORM2 to zeros 
(4) AuxNN BIBREF4 : This is a neural model that exploits auxiliary tasks, which has achieved state-of-the-art results on cross-domain sentiment classification  The sentence encoder used in this model is the same as ours 
(5) ADAN BIBREF16 : This method exploits adversarial training to reduce representation difference between domains  The original paper uses a simple feedforward network as encoder  For fair comparison, we replace it with our CNN-based encoder  We train 5 iterations on the discriminator per iteration on the encoder and sentiment classifier as suggested in their paper 
(6) MMD: MMD has been widely used for minimizing domain discrepancy on images  In those works BIBREF9 , BIBREF13 , variants of deep CNNs are used for encoding images and the MMDs of multiple layers are jointly minimized  In NLP, adding more layers of CNNs may not be very helpful and thus those models from image-related tasks can not be directly applied to our problem  To compare with MMD-based method, we train a model that jointly minimize the classification loss INLINEFORM0 on the source domain and MMD between INLINEFORM1 and INLINEFORM2   For computing MMD, we use a Gaussian RBF which is a common choice for characteristic kernel 
In addition to the above baselines, we also show results of different variants of our model  DAS as shown in Algorithm SECREF6 denotes our full model  DAS-EM denotes the model with only entropy minimization for semi-supervised learning (set INLINEFORM0 )  DAS-SE denotes the model with only self-ensemble bootstrapping for semi-supervised learning (set INLINEFORM1 )  FANN (feature-adaptation neural network) denotes the model without semi-supervised learning performed (set both INLINEFORM2 and INLINEFORM3 to zeros) 
Main Results
Figure FIGREF17 shows the comparison of adaptation results (see Appendix SECREF7 for the exact numerical numbers)  We report classification accuracy on the small-scale dataset  For the large-scale dataset, macro-F1 is instead used since the label distribution in the test set is extremely unbalanced  Key observations are summarized as follows  (1) Both DAS-EM and DAS-SE perform better in most cases compared with ADAN, MDD, and FANN, in which only feature adaptation is performed  This demonstrates the effectiveness of the proposed domain adaptive semi-supervised learning framework  DAS-EM is more effective than DAS-SE in most cases, and the full model DAS with both techniques jointly employed overall has the best performance  (2) When comparing the two settings on the small-scale dataset, all domain-adaptive methods generally perform better under setting 1  In setting 1, the target examples are balanced in classes, which can provide more diverse opinion-related features  However, when considering unsupervised domain adaptation, we should not presume the label distribution of the unlabeled data  Thus, it is necessary to conduct experiments using datasets that reflect real-life sentiment distribution as what we did on setting2 and the large-scale dataset  Unfortunately, this is ignored by most of previous works  (3) Word-embeddings are very helpful, as we can see even NaiveNN can substantially outperform mSDA on most tasks 
To see the effect of semi-supervised learning alone, we also conduct experiments by setting INLINEFORM0 to eliminate the effect of feature adaptation  Both entropy minimization and bootstrapping perform very badly in this setting  Entropy minimization gives almost random predictions with accuracy below 0 4, and the results of bootstrapping are also much lower compared to NaiveNN  This suggests that the feature adaptation component is essential  Without it, the learned target representations are less meaningful and discriminative  Applying semi-supervised learning in this case is likely to worsen the results 
Further Analysis
In Figure FIGREF23 , we show the change of accuracy with respect to the percentage of unlabeled data used for training on three particular problems under setting 1  The value at INLINEFORM0 denotes the accuracies of NaiveNN which does not utilize any target data  For DAS, we observe a nonlinear increasing trend where the accuracy quickly improves at the beginning, and then gradually stabilizes  For other methods, this trend is less obvious, and adding more unlabeled data sometimes even worsen the results  This finding again suggests that the proposed approach can better exploit the information from unlabeled data 
We also conduct experiments under a setting with a small number of labeled target examples available  Figure FIGREF27 shows the change of accuracy with respect to the number of labeled target examples added for training  We can observe that DAS is still more effective under this setting, while the performance differences to other methods gradually decrease with the increasing number of labeled target examples 
CNN Filter Analysis
In this subsection, we aim to better understand DAS by analyzing sentiment-related CNN filters  To do that, 1) we first select a list of the most related CNN filters for predicting each sentiment label (positive, negative neutral)  Those filters can be identified according to the learned weights INLINEFORM0 of the output layer INLINEFORM1   Higher weight indicates stronger relatedness  2) Recall that in our implementation, each CNN filter has a window size of 3 with Relu activation  We can thus represent each selected filter as a ranked list of trigrams with highest activation values 
We analyze the CNN filters learned by NaiveNN, FANN and DAS respectively on task E INLINEFORM0 BT under setting 1  We focus on E INLINEFORM1 BT for study because electronics and beauty are very different domains and each of them has a diverse set of domain-specific sentiment expressions  For each method, we identify the top 10 most related filters for each sentiment label, and extract the top trigrams of each selected filter on both source and target domains  Since labeled source examples are used for training, we find the filters learned by the three methods capture similar expressions on the source domain, containing both domain-invariant and domain-specific trigrams  On the target domain, DAS captures more target-specific expressions compared to the other two methods  Due to space limitation, we only present a small subset of positive-sentiment-related filters in Table TABREF34   The complete results are provided in Appendix SECREF8   From Table TABREF34 , we can observe that the filters learned by NaiveNN are almost unable to capture target-specific sentiment expressions, while FANN is able to capture limited target-specific words such as “clean” and “scent”  The filters learned by DAS are more domain-adaptive, capturing diverse sentiment expressions in the target domain 
Conclusion
In this work, we propose DAS, a novel framework that jointly performs feature adaptation and semi-supervised learning  We have demonstrated through multiple experiments that DAS can better leverage unlabeled data, and achieve substantial improvements over baseline methods  We have also shown that feature adaptation is an essential component, without which, semi-supervised learning is not able to function properly  The proposed framework could be potentially adapted to other domain adaptation tasks, which is the focus of our future studies 
Results on Amazon Benchmark
Most previous works BIBREF0 , BIBREF1 , BIBREF6 , BIBREF7 , BIBREF29 carried out experiments on the Amazon benchmark released by Blitzer et al  ( BIBREF0 )  The dataset contains 4 different domains: Book (B), DVDs (D), Electronics (E), and Kitchen (K)  Following their experimental settings, we consider the binary classification task to predict whether a review is positive or negative on the target domain  Each domain consists of 1000 positive and 1000 negative reviews respectively  We also allow 4000 unlabeled reviews to be used for both the source and the target domains, of which the positive and negative reviews are balanced as well, following the settings in previous works  We construct 12 cross-domain sentiment classification tasks and split the labeled data in each domain into a training set of 1600 reviews and a test set of 400 reviews  The classifier is trained on the training set of the source domain and is evaluated on the test set of the target domain  The comparison results are shown in Table TABREF37  
Numerical Results of Figure 
Due to space limitation, we only show results in figures in the paper  All numerical numbers used for plotting Figure FIGREF17 are presented in Table TABREF38   We can observe that DAS-EM, DAS-SE, and DAS all achieve substantial improvements over baseline methods under different settings 
CNN Filter Analysis Full Results
As mentioned in Section SECREF36 , we conduct CNN filter analysis on NaiveNN, FANN, and DAS  For each method, we identify the top 10 most related filters for positive, negative, neutral sentiment labels respectively, and then represent each selected filter as a ranked list of trigrams with the highest activation values on it  Table TABREF39 , TABREF40 , TABREF41 in the following pages illustrate the trigrams from the target domain (beauty) captured by the selected filters learned on E INLINEFORM0 BT for each method 
We can observe that compared to NaiveNN and FANN, DAS is able to capture a more diverse set of relevant sentiment expressions on the target domain for each sentiment label  This observation is consistent with our motivation  Since NaiveNN, FANN and other baseline methods solely train the sentiment classifier on the source domain, the learned encoder is not able to produce discriminative features on the target domain  DAS addresses this problem by refining the classifier on the target domain with semi-supervised learning, and the overall objective forces the encoder to learn feature representations that are not only domain-invariant but also discriminative on both domains 

ground_truth: ability to accurately classify texts even when the amount of prior knowledge for different classes is unbalanced, and when the class distribution of the dataset is unbalanced

Low sensitivity to bias in prior knowledge