prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. Which baselines are used for evaluation?
Context 1: Introduction
Headline generation is the process of creating a headline-style sentence given an input article  The research community has been regarding the task of headline generation as a summarization task BIBREF1, ignoring the fundamental differences between headlines and summaries  While summaries aim to contain most of the important information from the articles, headlines do not necessarily need to  Instead, a good headline needs to capture people's attention and serve as an irresistible invitation for users to read through the article  For example, the headline “$2 Billion Worth of Free Media for Trump”, which gives only an intriguing hint, is considered better than the summarization style headline “Measuring Trump’s Media Dominance” , as the former gets almost three times the readers as the latter  Generating headlines with many clicks is especially important in this digital age, because many of the revenues of journalism come from online advertisements and getting more user clicks means being more competitive in the market  However, most existing websites naively generate sensational headlines using only keywords or templates  Instead, this paper aims to learn a model that generates sensational headlines based on an input article without labeled data 
To generate sensational headlines, there are two main challenges  Firstly, there is a lack of sensationalism scorer to measure how sensational a headline is  Some researchers have tried to manually label headlines as clickbait or non-clickbait BIBREF2, BIBREF3  However, these human-annotated datasets are usually small and expensive to collect  To capture a large variety of sensationalization patterns, we need a cheap and easy way to collect a large number of sensational headlines  Thus, we propose a distant supervision strategy to collect a sensationalism dataset  We regard headlines receiving lots of comments as sensational samples and the headlines generated by a summarization model as non-sensational samples  Experimental results show that by distinguishing these two types of headlines, we can partially teach the model a sense of being sensational 
Secondly, after training a sensationalism scorer on our sensationalism dataset, a natural way to generate sensational headlines is to maximize the sensationalism score using reinforcement learning (RL)  However, the following shows an example of a RL model maximizing the sensationalism score by generating a very unnatural sentence, while its sensationalism scorer gave a very high score of 0 99996: UTF8gbsn 十个可穿戴产品的设计原则这消息消息可惜说明 Ten design principles for wearable devices, this message message pity introduction  This happens because the sensationalism scorer can make mistakes and RL can generate unnatural phrases which fools our sensationalism scorer  Thus, how to effectively leverage RL with noisy rewards remains an open problem  To deal with the noisy reward, we introduce Auto-tuned Reinforcement Learning (ARL)  Our model automatically tunes the ratio between MLE and RL based on how sensational the training headline is  In this way, we effectively take advantage of RL with a noisy reward to generate headlines that are both sensational and fluent 
The major contributions of this paper are as follows: 1) To the best of our knowledge, we propose the first-ever model that tackles the sensational headline generation task with reinforcement learning techniques  2) Without human-annotated data, we propose a distant supervision strategy to train a sensationalism scorer as a reward function 3) We propose a novel loss function, Auto-tuned Reinforcement Learning, to give dynamic weights to balance between MLE and RL  Our code will be released  
Sensationalism Scorer
To evaluate the sensationalism intensity score $\alpha _{\text{sen}}$ of a headline, we collect a sensationalism dataset and then train a sensationalism scorer  For the sensationalism dataset collection, we choose headlines with many comments from popular online websites as positive samples  For the negative samples, we propose to use the generated headlines from a sentence summarization model  Intuitively, the summarization model, which is trained to preserve the semantic meaning, will lose the sensationalization ability and thus the generated negative samples will be less sensational than the original one, similar to the obfuscation of style after back-translation BIBREF4  For example, an original headline like UTF8gbsn“一趟挣10万？铁总增开申通、顺丰专列" (One trip to earn 100 thousand? China Railway opens new Shentong and Shunfeng special lines) will become UTF8gbsn“中铁总将增开京广两列快递专列" (China Railway opens two special lines for express) from the baseline model, which loses the sensational phrases of UTF8gbsn“一趟挣10万？" (One trip to earn 100 thousand?)   We then train the sensationalism scorer by classifying sensational and non-sensational headlines using a one-layer CNN with a binary cross entropy loss $L_{\text{sen}}$  Firstly, 1-D convolution is used to extract word features from the input embeddings of a headline  This is followed by a ReLU activation layer and a max-pooling layer along the time dimension  All features from different channels are concatenated together and projected to the sensationalism score by adding another fully connected layer with sigmoid activation  Binary cross entropy is used to compute the loss $L_{\text{sen}}$ 
Sensationalism Scorer ::: Training Details and Dataset
For the CNN model, we choose filter sizes of 1, 3, and 5 respectively  Adam is used to optimize $L_{sen}$ with a learning rate of 0 0001  We set the embedding size as 300 and initialize it from qiu2018revisiting trained on the Weibo corpus with word and character features  We fix the embeddings during training  For dataset collection, we utilize the headlines collected in qin2018automatic, lin2019learning from Tencent News, one of the most popular Chinese news websites, as the positive samples  We follow the same data split as the original paper  As some of the links are not available any more, we get 170,754 training samples and 4,511 validation samples  For the negative training samples collection, we randomly select generated headlines from a pointer generator BIBREF0 model trained on LCSTS dataset BIBREF5 and create a balanced training corpus which includes 351,508 training samples and 9,022 validation samples  To evaluate our trained classifier, we construct a test set by randomly sampling 100 headlines from the test split of LCSTS dataset and the labels are obtained by 11 human annotators  Annotations show that 52% headlines are labeled as positive and 48% headlines as negative by majority voting (The detail on the annotation can be found in Section SECREF26) 
Sensationalism Scorer ::: Results and Discussion
Our classifier achieves 0 65 accuracy and 0 65 averaged F1 score on the test set while a random classifier would only achieve 0 50 accuracy and 0 50 averaged F1 score  This confirms that the predicted sensationalism score can partially capture the sensationalism of headlines  On the other hand, a more natural choice is to take headlines with few comments as negative examples  Thus, we train another baseline classifier on a crawled balanced sensationalism corpus of 84k headlines where the positive headlines have at least 28 comments and the negative headlines have less than 5 comments  However, the results on the test set show that the baseline classifier gets 60% accuracy, which is worse than the proposed classifier (which achieves 65%)  The reason could be that the balanced sensationalism corpus are sampled from different distributions from the test set and it is hard for the trained model to generalize  Therefore, we choose the proposed one as our sensationalism scorer  Therefore, our next challenge is to show that how to leverage this noisy sensationalism reward to generate sensational headlines 
Sensational Headline Generation
Our sensational headline generation model takes an article as input and output a sensational headline  The model consists of a Pointer-Gen headline generator and is trained by ARL  The diagram of ARL can be found in Figure FIGREF6 
We denote the input article as $x=\lbrace x_1,x_2,x_3,\cdots ,x_M\rbrace $, and the corresponding headline as $y^*=\lbrace y_1^*,y_2^*,y_3^*,\cdots ,y_T^*\rbrace $, where $M$ is the number of tokens in an article and $T$ is the number of tokens in a headline 
Sensational Headline Generation ::: Pointer-Gen Headline Generator
We choose Pointer Generator (Pointer-Gen) BIBREF0, a widely used summarization model, as our headline generator for its ability to copy words from the input article  It takes a news article as input and generates a headline  Firstly, the tokens of each article, $\lbrace x_1,x_2,x_3,\cdots ,x_M\rbrace $, are fed into the encoder one-by-one and the encoder generates a sequence of hidden states $h_i$  For each decoding step $t$, the decoder receives the embedding for each token of a headline $y_t$ as input and updates its hidden states $s_t$  An attention mechanism following luong2015effective is used:
where $v$, $W_h$, $W_s$, and $b_{attn}$ are the trainable parameters and $h_t^*$ is the context vector  $s_t$ and $h_t^*$ are then combined to give a probability distribution over the vocabulary through two linear layers:
where $V$, $b$, $V^{^{\prime }}$, and $b^{^{\prime }}$ are trainable parameters  We use a pointer generator network to enable our model to copy rare/unknown words from the input article, giving the following final word probability:
where $x^t$ is the embedding of the input word of the decoder, $w_{h^*}^T$, $w_s^T$, $w_x^T$, and $b_{ptr}$ are trainable parameters, and $\sigma $ is the sigmoid function 
Sensational Headline Generation ::: Training Methods
We first briefly introduce MLE and RL objective functions, and a naive way to mix these two by a hyper-parameter $\lambda $  Then we point out the challenge of training with noisy reward, and propose ARL to address this issue 
Sensational Headline Generation ::: Training Methods ::: MLE and RL
A headline generation model can be trained with MLE, RL or a combination of MLE and RL  MLE training is to minimize the negative log likelihood of the training headlines  We feed $y^*$ into the decoder word by word and maximize the likelihood of $y^*$  The loss function for MLE becomes
For RL training, we choose the REINFORCE algorithm BIBREF6  In the training phase, after encoding an article, a headline $y^s = \lbrace y_1^s, y_2^s, y_3^s, \cdots , y_T^s\rbrace $ is obtained by sampling from $P(w)$ from our generator, and then a reward of sensationalism or ROUGE(RG) is calculated 
We use the baseline reward $\hat{R_t}$ to reduce the variance of the reward, similar to ranzato2015sequence  To elaborate, a linear model is deployed to estimate the baseline reward $\hat{R_t}$ based on $t$-th state $o_t$ for each timestep $t$  The parameters of the linear model are trained by minimizing the mean square loss between $R$ and $\hat{R_t}$:
where $W_r$ and $b_r$ are trainable parameters  To maximize the expected reward, our loss function for RL becomes
A naive way to mix these two objective functions using a hyper-parameter $\lambda $ has been successfully incorporated in the summarization task BIBREF7  It includes the MLE training as a language model to mitigate the readability and quality issues in RL  The mixed loss function is shown as follows:
where $*$ is the reward type  Usually $\lambda $ is large, and paulus2017deep used 0 9984 
Sensational Headline Generation ::: Training Methods ::: Auto-tuned Reinforcement Learning
Applying the naive mixed training method using sensationalism score as the reward is not obvious/trivial in our task  The main reason is that our sensationalism reward is notably more noisy and more fragile than the ROUGE-L reward or abstractive reward used in the summarization task BIBREF7, BIBREF8  A higher ROUGE-L F1 reward in summarization indicates higher overlapping ratio between generation and true summary statistically, but our sensationalism reward is a learned score which is fragile to be fooled with unnatural samples 
To effectively train the model with RL under noisy sensationalism reward, our idea is to balance RL with MLE  However, we argue that the weighted ratio between MLE and RL should be sample-dependent, instead of being fixed for all training samples as in paulus2017deep, kryscinski2018improving  The reason is that, RL and MLE have inconsistent optimization objectives  When the training headline is non-sensational, MLE training will encourage our model to imitate the training headline (thus generating non-sensational headlines), which counteracts the effects of RL training to generate sensational headlines 
The sensationalism score is, therefore, used to give dynamic weight to MLE and RL  Our ARL loss function becomes:
If $\alpha _{\text{sen}}(y^*)$ is high, meaning the training headline is sensational, our loss function encourages our model to imitate the sample more using the MLE training  If $\alpha _{\text{sen}}(y^*)$ is low, our loss function replies on RL training to improve the sensationalism  Note that the weight $\alpha _{\text{sen}}(y^*)$ is different from our sensationalism reward $\alpha _{\text{sen}}(y^s)$ and we call the loss function Auto-tuned Reinforcement Learning, because the ratio between MLE and RL are well “tuned” towards different samples 
Sensational Headline Generation ::: Dataset
We use LCSTS BIBREF5 as our dataset to train the summarization model  The dataset is collected from the Chinese microblogging website Sina Weibo  It contains over 2 million Chinese short texts with corresponding headlines given by the author of each text  The dataset is split into 2,400,591 samples for training, 10,666 samples for validation and 725 samples for testing  We tokenize each sentence with Jieba and a vocabulary size of 50000 is saved 
Sensational Headline Generation ::: Baselines and Our Models
We experiment and compare with the following models 
Pointer-Gen is the baseline model trained by optimizing $L_\text{MLE}$ in Equation DISPLAY_FORM13 
Pointer-Gen+Pos is the baseline model by training Pointer-Gen only on positive examples whose sensationalism score is larger than 0 5
Pointer-Gen+Same-FT is the model which fine-tunes Pointer-Gen on the training samples whose sensationalism score is larger than 0 1
Pointer-Gen+Pos-FT is the model which fine-tunes Pointer-Gen on the training samples whose sensationalism score is larger than 0 5
Pointer-Gen+RL-ROUGE is the baseline model trained by optimizing $L_\text{RL-ROUGE}$ in Equation DISPLAY_FORM17, with ROUGE-L BIBREF9 as the reward 
Pointer-Gen+RL-SEN is the baseline model trained by optimizing $L_\text{RL-SEN}$ in Equation DISPLAY_FORM17, with $\alpha _\text{sen}$ as the reward 
Pointer-Gen+ARL-SEN is our model trained by optimizing $L_\text{ARL-SEN}$ in Equation DISPLAY_FORM19, with $\alpha _\text{sen}$ as the reward 
Test set is the headlines from the test set 
Note that we didn't compare to Pointer-Gen+ARL-ROUGE as it is actually Pointer-GEN  Recall that $\alpha _{\text{sen}}(y^*)$ in Equation DISPLAY_FORM19 measures how good (based on reward function) is $y^*$  Then the loss function for Pointer-Gen+ARL-ROUGE will be
We also tried text style transfer baseline BIBREF10, but the generated headlines were very poor (many unknown words and irrelevant) 
Sensational Headline Generation ::: Training Details
MLE training: An Adam optimizer is used with the learning rate of 0 0001 to optimize $L_{\text{MLE}}$  The batch size is set as 128 and a one-layer, bi-directional Long Short-Term Memory (bi-LSTM) model with 512 hidden sizes and a 350 embedding size is utilized  Gradients with the l2 norm larger than 2 0 are clipped  We stop training when the ROUGE-L f-score stops increasing 
Hybrid training: An Adam optimizer with a learning rate of 0 0001 is used to optimize $L_{\text{RL-*}}$ (Equation DISPLAY_FORM17) and $L_\text{{ARL-SEN}}$ (Equation DISPLAY_FORM19)  When training Pointer-Gen+RL-ROUGE, the best $\lambda $ is chosen based on the ROUGE-L score on the validation set  In our experiment, $\lambda $ is set as 0 95  An Adam optimizer with a learning rate of 0 001 is used to optimize $L_b$  When training Pointer-Gen+ARL-SEN, we don't use the full LCSTS dataset, but only headlines with a sensationalism score larger than 0 1 as we observe that Pointer-Gen+ARL-SEN will generate a few unnatural phrases when using full dataset  We believe the reason is the high ratio of RL during training  Figure FIGREF23 shows that the probability density near 0 is very high, meaning that in each batch, many of the samples will have a very low sensationalism score  On expectation, each sample will receive 0 239 MLE training and 0 761 RL training  This leads to RL dominanting the loss  Thus, we propose to filter samples with a minimum sensationalism score with 0 1 and it works very well  For Pointer-Gen+RL-SEN, we also set the minimum sensationalism score as 0 1, and $\lambda $ is set as 0 5 to remove unnatural phrases, making a fair comparison to Pointer-Gen+ARL-SEN 
We stop training Pointer-Gen+Same-FT, Pointer-Gen+Pos-FT, Pointer-Gen+RL-SEN and Pointer-Gen+ARL-SEN, when $\alpha _\text{sen}$ stops increasing on the validation set  Beam-search with a beam size of 5 is adopted for decoding in all models 
Sensational Headline Generation ::: Evaluation Metrics
We briefly describe the evaluation metrics below 
ROUGE: ROUGE is a commonly used evaluation metric for summarization  It measures the N-gram overlap between generated and training headlines  We use it to evaluate the relevance of generated headlines  The widely used pyrouge toolkit is used to calculate ROUGE-1 (RG-1), ROUGE-2 (RG-2), and ROUGE-L (RG-L) 
Human evaluation: We randomly sample 50 articles from the test set and send the generated headlines from all models and corresponding headlines in the test set to human annotators  We evaluate the sensationalism and fluency of the headlines by setting up two independent human annotation tasks  We ask 10 annotators to label each headline for each task  For the sensationalism annotation, each annotator is asked one question, “Is the headline sensational?”, and he/she has to choose either `yes' or `no'  The annotators were not told which system the headline is from  The process of distributing samples and recruiting annotators is managed by Crowdflower  After annotation, we define the sensationalism score as the proportion of annotations on all generated headlines from one model labeled as `yes'  For the fluency annotation, we repeat the same procedure as for the sensationalism annotation, except that we ask each annotator the question “Is the headline fluent?” We define the fluency score as the proportion of annotations on all headlines from one specific model labeled as `yes'  We put human annotation instructions in the supplemental material 
UTF8gbsn
Results
We first compare all four models, Pointer-Gen, Pointer-Gen-RL+ROUGE, Pointer-Gen-RL-SEN, and Pointer-Gen-ARL-SEN, to existing models with ROUGE in Table TABREF25 to establish that our model produces relevant headlines and we leave the sensationalism for human evaluation  Note that we only compare our models to commonly used strong summarization baselines, to validate that our implementation achieves comparable performance to existing work  In our implementation, Pointer-Gen achieves a 34 51 RG-1 score, 22 21 RG-2 score, and 31 68 RG-L score, which is similar to the results of gu2016incorporating  Pointer-Gen+ARL-SEN, although optimized for the sensationalism reward, achieves similar performance to our Pointer-Gen baseline, which means that Pointer-Gen+ARL-SEN still keeps its summarization ability  An example of headlines generated from different models in Table TABREF29 shows that Pointer-Gen and Pointer-Gen+RL-ROUGE learns to summarize the main point of the article: “The Nikon D600 camera is reported to have black spots when taking photos”  Pointer-Gen+RL-SEN makes the headline more sensational by blaming Nikon for attributing the damage to the smog  Pointer-Gen+ARL-SEN generates the most sensational headline by exaggerating the result “Getting a serious trouble!” to maximize user's attention 
We then compare different models using the sensationalism score in Table TABREF30  The Pointer-Gen baseline model achieves a 42 6% sensationalism score, which is the minimum that a typical summarization model achieves  By filtering out low-sensational headlines, Pointer-Gen+Same-FT and Pointer-Gen+Pos-FT achieves higher sensationalism scores, which implies the effectiveness of our sensationalism scorer  Our Pointer-Gen+ARL-SEN model achieves the best performance of 60 8%  This is an absolute improvement of 18 2% over the Pointer-Gen baseline  The Chi-square test on the results confirms that Pointer-Gen+ARL-SEN is statistically significantly more sensational than all the other baseline models, with the largest p-value less than 0 01  Also, we find that the test set headlines achieves 57 8% sensationalism score, much larger than Pointer-Gen baseline, which also supports our intuition that generated headlines will be less sensational than the original one  On the other hand, we found that Pointer-Gen+Pos is much worse than other baselines  The reason is that training on sensational samples alone discards around 80% of the whole training set that is also helpful for maintaining relevance and a good language model  It shows the necessity of using RL 
UTF8gbsn
In addition, both Pointer-Gen+RL-SEN and Pointer-Gen+ARL-SEN, which use the sensationalism score as the reward, obtain statistically better results than Pointer-Gen+RL-ROUGE and Pointer-Gen, with a p-value less than 0 05 by a Chi-square test  This result shows the effectiveness of RL to generate more sensational headlines  The reason is that even though our noisy classifier could also learn to classify domains, the generator during RL training is not allowed to increase the reward by shifting domains but encouraged to generate more sensational headlines, due to the consistency constraint on the domains of the headline and the article  Furthermore, Poiner-Gen+ARL-SEN gets better performance than Pointer-Gen+RL-SEN, which confirms the superiority of the ARL loss function  We also visualize in Figure FIGREF31 a comparison between Pointer-Gen+ARL-SEN and Pointer-Gen+RL-SEN according to how sensational the test set headlines are  The blue bars denote the smaller scores between the two models  For example, if the blue bar is 0 6, it means that the worse model between Pointer-Gen+RL-SEN and Pointer-Gen+ARL-SEN achieves 0 6  And the color of orange/black further indicates the better model and its score  We find that Pointer-Gen+ARL-SEN outperforms Pointer-Gen+RL-SEN for most cases  The improvement is higher when the test set headlines are not sensational (the sensationalism score is less than 0 5), which may be attributed to the higher ratio of RL training on non-sensational headlines 
Apart from the sensationalism evaluation, we measure the fluency of the headlines generated from different models  Fluency scores in Table TABREF30 show that Pointer-Gen+RL-SEN and Pointer-Gen+ARL-SEN achieve comparable fluency performance to Pointer-Gen and Pointer-Gen+RL-ROUGE  Test set headlines achieve the best performance among all models, but the difference is not statistically significant  Also, we observe that fine-tuning on sensational headlines will hurt the performance, both in sensationalism and fluency 
After manually checking the outputs, we observe that our model is able to generate sensational headlines using diverse sensationalization strategies  These strategies include, but are not limited to, creating a curiosity gap, asking questions, highlighting numbers, being emotional and emphasizing the user  Examples can be found in Table TABREF32 
Related Work
Our work is related to summarization tasks  An encoder-decoder model was first applied to two sentence-level abstractive summarization tasks on the DUC-2004 and Gigaword datasets BIBREF12  This model was later extended by selective encoding BIBREF13, a coarse to fine approach BIBREF14, minimum risk training BIBREF1, and topic-aware models BIBREF15  As long summaries were recognized as important, the CNN/Daily Mail dataset was used in nallapati2016abstractive  Graph-based attention BIBREF16, pointer-generator with coverage loss BIBREF0 are further developed to improve the generated summaries  celikyilmaz2018deep proposed deep communicating agents for representing a long document for abstractive summarization  In addition, many papers BIBREF17, BIBREF18, BIBREF19 use extractive methods to directly select sentences from articles  However, none of these work considered the sensationalism of generated outputs 
RL is also gaining popularity as it can directly optimize non-differentiable metrics BIBREF20, BIBREF21, BIBREF22  paulus2017deep proposed an intra-decoder model and combined RL and MLE to deal with summaries with bad qualities  RL has also been explored with generative adversarial networks (GANs) BIBREF23  liu2017generative applied GANs on summarization task and achieved better performance  niu2018polite tackles the problem of polite generation with politeness reward  Our work is different in that we propose a novel function to balance RL and MLE 
Our task is also related to text style transfer  Implicit methods BIBREF10, BIBREF24, BIBREF4 transfer the styles by separating sentence representations into content and style, for example using back-translationBIBREF4  However, these methods cannot guarantee the content consistency between the original sentence and transferred output BIBREF25  Explicit methods BIBREF26, BIBREF25 transfer the style by directly identifying style related keywords and modifying them  However, sensationalism is not always restricted to keywords, but the full sentence  By leveraging small human labeled English dataset, clickbait detection has been well investigated BIBREF2, BIBREF27, BIBREF3  However, these human labeled dataset are not available for other languages, such as Chinese 
Modeling sensationalism is also related to modeling emotion  Emotion has been well investigated in both word levelBIBREF28, BIBREF29 and sentence levelBIBREF30, BIBREF31, BIBREF32, BIBREF33, BIBREF34  It has also been considered an important factor in engaging interactive systemsBIBREF35, BIBREF36, BIBREF37  Although we observe that sensational headlines contain emotion, it is still not clear which emotion and how emotions will influence the sensationalism 
Conclusion and Future Work
In this paper, we propose a model that generates sensational headlines without labeled data using Reinforcement Learning  Firstly, we propose a distant supervision strategy to train the sensationalism scorer  As a result, we achieve 65% accuracy between the predicted sensationalism score and human evaluation  To effectively leverage this noisy sensationalism score as the reward for RL, we propose a novel loss function, ARL, to automatically balance RL with MLE  Human evaluation confirms the effectiveness of both our sensationalism scorer and ARL to generate more sensational headlines  Future work can be improving the sensationalism scorer and investigating the applications of dynamic balancing methods between RL and MLE in textGANBIBREF23  Our work also raises the ethical questions about generating sensational headlines, which can be further explored 
Acknowledgments
Thanks to ITS/319/16FP of Innovation Technology Commission, HKUST 16248016 of Hong Kong Research Grants Council for funding  In addition, we thank Zhaojiang Lin for helpful discussion and Yan Xu, Zihan Liu for the data collection 
Context 3: Introduction
Microblogging such as Twitter and Weibo is a popular social networking service, which allows users to post messages up to 140 characters  There are millions of active users on the platform who stay connected with friends  Unfortunately, spammers also use it as a tool to post malicious links, send unsolicited messages to legitimate users, etc  A certain amount of spammers could sway the public opinion and cause distrust of the social platform  Despite the use of rigid anti-spam rules, human-like spammers whose homepages having photos, detailed profiles etc  have emerged  Unlike previous "simple" spammers, whose tweets contain only malicious links, those "smart" spammers are more difficult to distinguish from legitimate users via content-based features alone BIBREF0  
There is a considerable amount of previous work on spammer detection on social platforms  Researcher from Twitter Inc  BIBREF1 collect bot accounts and perform analysis on the user behavior and user profile features  Lee et al  lee2011seven use the so-called social honeypot by alluring social spammers' retweet to build a benchmark dataset, which has been extensively explored in our paper  Some researchers focus on the clustering of urls in tweets and network graph of social spammers BIBREF2 , BIBREF3 , BIBREF4 , BIBREF5 , showing the power of social relationship features As for content information modeling, BIBREF6 apply improved sparse learning methods  However, few studies have adopted topic-based features  Some researchers BIBREF7 discuss topic characteristics of spamming posts, indicating that spammers are highly likely to dwell on some certain topics such as promotion  But this may not be applicable to the current scenario of smart spammers 
In this paper, we propose an efficient feature extraction method  In this method, two new topic-based features are extracted and used to discriminate human-like spammers from legitimate users  We consider the historical tweets of each user as a document and use the Latent Dirichlet Allocation (LDA) model to compute the topic distribution for each user  Based on the calculated topic probability, two topic-based features, the Local Outlier Standard Score (LOSS) which captures the user's interests on different topics and the Global Outlier Standard Score (GOSS) which reveals the user's interests on specific topic in comparison with other users', are extracted  The two features contain both local and global information, and the combination of them can distinguish human-like spammers effectively 
To the best of our knowledge, it is the first time that features based on topic distributions are used in spammer classification  Experimental results on one public dataset and one self-collected dataset further validate that the two sets of extracted topic-based features get excellent performance on human-like spammer classification problem compared with other state-of-the-art methods  In addition, we build a Weibo dataset, which contains both legitimate users and spammers 
To summarize, our major contributions are two-fold:
In the following sections, we first propose the topic-based features extraction method in Section 2, and then introduce the two datasets in Section 3  Experimental results are discussed in Section 4, and we conclude the paper in Section 5  Future work is presented in Section 6 
Methodology
In this section, we first provide some observations we obtained after carefully exploring the social network, then the LDA model is introduced  Based on the LDA model, the ways to obtain the topic probability vector for each user and the two topic-based features are provided 
Observation
After exploring the homepages of a substantial number of spammers, we have two observations  1) social spammers can be divided into two categories  One is content polluters, and their tweets are all about certain kinds of advertisement and campaign  The other is fake accounts, and their tweets resemble legitimate users' but it seems they are simply random copies of others to avoid being detected by anti-spam rules  2) For legitimate users, content polluters and fake accounts, they show different patterns on topics which interest them 
Legitimate users mainly focus on limited topics which interest him  They seldom post contents unrelated to their concern 
Content polluters concentrate on certain topics 
Fake accounts focus on a wide range of topics due to random copying and retweeting of other users' tweets 
Spammers and legitimate users show different interests on some topics e g  commercial, weather, etc 
To better illustrate our observation, Figure  1 shows the topic distribution of spammers and legitimate users in two employed datasets(the Honeypot dataset and Weibo dataset)  We can see that on both topics (topic-3 and topic-11) there exists obvious difference between the red bars and green bars, representing spammers and legitimate users  On the Honeypot dataset, spammers have a narrower shape of distribution (the outliers on the red bar tail are not counted) than that of legitimate users  This is because there are more content polluters than fake accounts  In other word, spammers in this dataset tend to concentrate on limited topics  While on the Weibo dataset, fake accounts who are interested in different topics take large proportion of spammers  Their distribution is more flat (i e  red bars) than that of the legitimate users  Therefore we can detect spammers by means of the difference of their topic distribution patterns 
LDA model
Blei et al blei2003latent first presented Latent Dirichlet Allocation(LDA) as an example of topic model 
Each document $i$ is deemed as a bag of words $W=\left\lbrace  w_{i1},w_{i2},   ,w_{iM}\right\rbrace $ and $M$ is the number of words  Each word is attributable to one of the document's topics $Z=\left\lbrace  z_{i1},z_{i2},   ,z_{iK}\right\rbrace $ and $K$ is the number of topics  $\psi _{k}$ is a multinomial distribution over words for topic $k$   $\theta _i$ is another multinomial distribution over topics for document $i$   The smoothed generative model is illustrated in Figure  2   $\alpha $ and $W=\left\lbrace  w_{i1},w_{i2},   ,w_{iM}\right\rbrace $0 are hyper parameter that affect scarcity of the document-topic and topic-word distributions  In this paper, $W=\left\lbrace  w_{i1},w_{i2},   ,w_{iM}\right\rbrace $1 , $W=\left\lbrace  w_{i1},w_{i2},   ,w_{iM}\right\rbrace $2 and $W=\left\lbrace  w_{i1},w_{i2},   ,w_{iM}\right\rbrace $3 are empirically set to 0 3, 0 01 and 15  The entire content of each Twitter user is regarded as one document  We adopt Gibbs Sampling BIBREF8 to speed up the inference of LDA  Based on LDA, we can get the topic probabilities for all users in the employed dataset as: $W=\left\lbrace  w_{i1},w_{i2},   ,w_{iM}\right\rbrace $4 , where $W=\left\lbrace  w_{i1},w_{i2},   ,w_{iM}\right\rbrace $5 is the number of users  Each element $W=\left\lbrace  w_{i1},w_{i2},   ,w_{iM}\right\rbrace $6 is a topic probability vector for the $W=\left\lbrace  w_{i1},w_{i2},   ,w_{iM}\right\rbrace $7 document  $W=\left\lbrace  w_{i1},w_{i2},   ,w_{iM}\right\rbrace $8 is the raw topic probability vector and our features are developed on top of it 
Topic-based Features
Using the LDA model, each person in the dataset is with a topic probability vector $X_i$   Assume $x_{ik}\in X_{i}$ denotes the likelihood that the $\emph {i}^{th}$ tweet account favors $\emph {k}^{th}$ topic in the dataset  Our topic based features can be calculated as below 
Global Outlier Standard Score measures the degree that a user's tweet content is related to a certain topic compared to the other users  Specifically, the "GOSS" score of user $i$ on topic $k$ can be calculated as Eq ( 12 ):
$$\centering \begin{array}{ll} \mu \left(x_{k}\right)=\frac{\sum _{i=1}^{n} x_{ik}}{n},\\ GOSS\left(x_{ik}\right)=\frac{x_{ik}-\mu \left(x_k\right)}{\sqrt{\underset{i}{\sum }\left(x_{ik}-\mu \left(x_{k}\right)\right)^{2}}}  \end{array}$$   (Eq  12)
The value of $GOSS\left(x_{ik}\right)$ indicates the interesting degree of this person to the $\emph {k}^{th}$ topic  Specifically, if $GOSS\left(x_{ik}\right)$ > $GOSS\left(x_{jk}\right)$ , it means that the $\emph {i}^{th}$ person has more interest in topic $k$ than the $\emph {j}^{th}$ person  If the value $GOSS\left(x_{ik}\right)$ is extremely high or low, the $\emph {i}^{th}$ person showing extreme interest or no interest on topic $k$ which will probably be a distinctive pattern in the fowllowing classfication  Therefore, the topics interested or disliked by the $\emph {k}^{th}$0 person can be manifested by $\emph {k}^{th}$1 , from which the pattern of the interested topics with regarding to this person is found  Denote $\emph {k}^{th}$2 our first topic-based feature, and it hopefully can get good performance on spammer detection 
Local Outlier Standard Score measures the degree of interest someone shows to a certain topic by considering his own homepage content only  For instance, the "LOSS" score of account $i$ on topic $k$ can be calculated as Eq ( 13 ):
$$\centering \begin{array}{ll} \mu \left(x_{i}\right)=\frac{\sum _{k=1}^{K} x_{ik}}{K},\\ LOSS\left(x_{ik}\right)=\frac{x_{ik}-\mu \left(x_i\right)}{\sqrt{\underset{k}{\sum }\left(x_{ik}-\mu \left(x_{i}\right)\right)^{2}}}  \end{array}$$   (Eq  13)
$\mu (x_i)$ represents the averaged interesting degree for all topics with regarding to $\emph {i}^{th}$ user and his tweet content  Similarly to $GOSS$ , the topics interested or disliked by the $\emph {i}^{th}$ person via considering his single post information can be manifested by $f_{LOSS}^{i}=[LOSS(x_{i1})\cdots LOSS(x_{iK})]$ , and $LOSS$ becomes our second topic-based features for the $\emph {i}^{th}$ person 
Dataset
We use one public dataset Social Honeypot dataset and one self-collected dataset Weibo dataset to validate the effectiveness of our proposed features 
Social Honeypot Dataset: Lee et al  lee2010devils created and deployed 60 seed social accounts on Twitter to attract spammers by reporting back what accounts interact with them  They collected 19,276 legitimate users and 22,223 spammers in their datasets along with their tweet content in 7 months  This is our first test dataset 
Our Weibo Dataset: Sina Weibo is one of the most famous social platforms in China  It has implemented many features from Twitter  The 2197 legitimate user accounts in this dataset are provided by the Tianchi Competition held by Sina Weibo  The spammers are all purchased commercially from multiple vendors on the Internet  We checked them manually and collected 802 suitable "smart" spammers accounts 
Preprocessing: Before directly performing the experiments on the employed datasets, we first delete some accounts with few posts in the two employed since the number of tweets is highly indicative of spammers  For the English Honeypot dataset, we remove stopwords, punctuations, non-ASCII words and apply stemming  For the Chinese Weibo dataset, we perform segmentation with "Jieba", a Chinese text segmentation tool  After preprocessing steps, the Weibo dataset contains 2197 legitimate users and 802 spammers, and the honeypot dataset contains 2218 legitimate users and 2947 spammers  It is worth mentioning that the Honeypot dataset has been slashed because most of the Twitter accounts only have limited number of posts, which are not enough to show their interest inclination 
Evaluation Metrics
The evaluating indicators in our model are show in 2   We calculate precision, recall and F1-score (i e  F1 score) as in Eq  ( 19 )  Precision is the ratio of selected accounts that are spammers  Recall is the ratio of spammers that are detected so  F1-score is the harmonic mean of precision and recall 
$$precision =\frac{TP}{TP+FP}, recall =\frac{TP}{TP+FN}\nonumber \\ F1-score = \frac{2\times precision \times recall}{precision + recall}$$   (Eq  19)
Performance Comparisons with Baseline
Three baseline classification methods: Support Vector Machines (SVM), Adaboost, and Random Forests are adopted to evaluate our extracted features  We test each classification algorithm with scikit-learn BIBREF9 and run a 10-fold cross validation  On each dataset, the employed classifiers are trained with individual feature first, and then with the combination of the two features  From 1 , we can see that GOSS+LOSS achieves the best performance on F1-score among all others  Besides, the classification by combination of LOSS and GOSS can increase accuracy by more than 3% compared with raw topic distribution probability 
Comparison with Other Features
To compare our extracted features with previously used features for spammer detection, we use three most discriminative feature sets according to Lee et al  lee2011seven( 4 )  Two classifiers (Adaboost and SVM) are selected to conduct feature performance comparisons  Using Adaboost, our LOSS+GOSS features outperform all other features except for UFN which is 2% higher than ours with regard to precision on the Honeypot dataset  It is caused by the incorrectly classified spammers who are mostly news source after our manual check  They keep posting all kinds of news pieces covering diverse topics, which is similar to the behavior of fake accounts  However, UFN based on friendship networks is more useful for public accounts who possess large number of followers  The best recall value of our LOSS+GOSS features using SVM is up to 6% higher than the results by other feature groups  Regarding F1-score, our features outperform all other features  To further show the advantages of our proposed features, we compare our combined LOSS+GOSS with the combination of all the features from Lee et al  lee2011seven (UFN+UC+UH)  It's obvious that LOSS+GOSS have a great advantage over UFN+UC+UH in terms of recall and F1-score  Moreover, by combining our LOSS+GOSS features and UFN+UC+UH features together, we obtained another 7 1% and 2 3% performance gain with regard to precision and F1-score by Adaboost  Though there is a slight decline in terms of recall  By SVM, we get comparative results on recall and F1-score but about 3 5% improvement on precision 
Conclusion
In this paper, we propose a novel feature extraction method to effectively detect "smart" spammers who post seemingly legitimate tweets and are thus difficult to identify by existing spammer classification methods  Using the LDA model, we obtain the topic probability for each Twitter user  By utilizing the topic probability result, we extract our two topic-based features: GOSS and LOSS which represent the account with global and local information  Experimental results on a public dataset and a self-built Chinese microblog dataset validate the effectiveness of the proposed features 
Future Work
In future work, the combination method of local and global information can be further improved to maximize their individual strengths  We will also apply decision theory to enhancing the performance of our proposed features  Moreover, we are also building larger datasets on both Twitter and Weibo to validate our method  Moreover, larger datasets on both Twitter and Weibo will be built to further validate our method 

ground_truth: Pointer-Gen, Pointer-Gen+Pos, Pointer-Gen+Same-FT, Pointer-Gen+Pos-FT, Pointer-Gen+RL-ROUGE, Pointer-Gen+RL-SEN

prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. On what datasets are experiments performed?
Context 2: Introduction
Question Generation (QG) is the task of automatically creating questions from a range of inputs, such as natural language text BIBREF0, knowledge base BIBREF1 and image BIBREF2  QG is an increasingly important area in NLP with various application scenarios such as intelligence tutor systems, open-domain chatbots and question answering dataset construction  In this paper, we focus on question generation from reading comprehension materials like SQuAD BIBREF3  As shown in Figure FIGREF1, given a sentence in the reading comprehension paragraph and the text fragment (i e , the answer) that we want to ask about, we aim to generate a question that is asked about the specified answer 
Question generation for reading comprehension is firstly formalized as a declarative-to-interrogative sentence transformation problem with predefined rules or templates BIBREF4, BIBREF0  With the rise of neural models, Du2017LearningTA propose to model this task under the sequence-to-sequence (Seq2Seq) learning framework BIBREF5 with attention mechanism BIBREF6  However, question generation is a one-to-many sequence generation problem, i e , several aspects can be asked given a sentence  Zhou2017NeuralQG propose the answer-aware question generation setting which assumes the answer, a contiguous span inside the input sentence, is already known before question generation  To capture answer-relevant words in the sentence, they adopt a BIO tagging scheme to incorporate the answer position embedding in Seq2Seq learning  Furthermore, Sun2018AnswerfocusedAP propose that tokens close to the answer fragments are more likely to be answer-relevant  Therefore, they explicitly encode the relative distance between sentence words and the answer via position embedding and position-aware attention 
Although existing proximity-based answer-aware approaches achieve reasonable performance, we argue that such intuition may not apply to all cases especially for sentences with complex structure  For example, Figure FIGREF1 shows such an example where those approaches fail  This sentence contains a few facts and due to the parenthesis (i e  “the area's coldest month”), some facts intertwine: “The daily mean temperature in January is 0 3$^\circ $C” and “January is the area's coldest month”  From the question generated by a proximity-based answer-aware baseline, we find that it wrongly uses the word “coldest” but misses the correct word “mean” because “coldest” has a shorter distance to the answer “0 3$^\circ $C” 
In summary, their intuition that “the neighboring words of the answer are more likely to be answer-relevant and have a higher chance to be used in the question” is not reliable  To quantitatively show this drawback of these models, we implement the approach proposed by Sun2018AnswerfocusedAP and analyze its performance under different relative distances between the answer and other non-stop sentence words that also appear in the ground truth question  The results are shown in Table TABREF2  We find that the performance drops at most 36% when the relative distance increases from “$0\sim 10$” to “$>10$”  In other words, when the useful context is located far away from the answer, current proximity-based answer-aware approaches will become less effective, since they overly emphasize neighboring words of the answer 
To address this issue, we extract the structured answer-relevant relations from sentences and propose a method to jointly model such structured relation and the unstructured sentence for question generation  The structured answer-relevant relation is likely to be to the point context and thus can help keep the generated question to the point  For example, Figure FIGREF1 shows our framework can extract the right answer-relevant relation (“The daily mean temperature in January”, “is”, “32 6$^\circ $F (0 3$^\circ $C)”) among multiple facts  With the help of such structured information, our model is less likely to be confused by sentences with a complex structure  Specifically, we firstly extract multiple relations with an off-the-shelf Open Information Extraction (OpenIE) toolbox BIBREF7, then we select the relation that is most relevant to the answer with carefully designed heuristic rules 
Nevertheless, it is challenging to train a model to effectively utilize both the unstructured sentence and the structured answer-relevant relation because both of them could be noisy: the unstructured sentence may contain multiple facts which are irrelevant to the target question, while the limitation of the OpenIE tool may produce less accurate extracted relations  To explore their advantages simultaneously and avoid the drawbacks, we design a gated attention mechanism and a dual copy mechanism based on the encoder-decoder framework, where the former learns to control the information flow between the unstructured and structured inputs, while the latter learns to copy words from two sources to maintain the informativeness and faithfulness of generated questions 
In the evaluations on the SQuAD dataset, our system achieves significant and consistent improvement as compared to all baseline methods  In particular, we demonstrate that the improvement is more significant with a larger relative distance between the answer and other non-stop sentence words that also appear in the ground truth question  Furthermore, our model is capable of generating diverse questions for a single sentence-answer pair where the sentence conveys multiple relations of its answer fragment 
Framework Description
In this section, we first introduce the task definition and our protocol to extract structured answer-relevant relations  Then we formalize the task under the encoder-decoder framework with gated attention and dual copy mechanism 
Framework Description ::: Problem Definition
We formalize our task as an answer-aware Question Generation (QG) problem BIBREF8, which assumes answer phrases are given before generating questions  Moreover, answer phrases are shown as text fragments in passages  Formally, given the sentence $S$, the answer $A$, and the answer-relevant relation $M$, the task of QG aims to find the best question $\overline{Q}$ such that,
where $A$ is a contiguous span inside $S$ 
Framework Description ::: Answer-relevant Relation Extraction
We utilize an off-the-shelf toolbox of OpenIE to the derive structured answer-relevant relations from sentences as to the point contexts  Relations extracted by OpenIE can be represented either in a triple format or in an n-ary format with several secondary arguments, and we employ the latter to keep the extractions as informative as possible and avoid extracting too many similar relations in different granularities from one sentence  We join all arguments in the extracted n-ary relation into a sequence as our to the point context  Figure FIGREF5 shows n-ary relations extracted from OpenIE  As we can see, OpenIE extracts multiple relations for complex sentences  Here we select the most informative relation according to three criteria in the order of descending importance: (1) having the maximal number of overlapped tokens between the answer and the relation; (2) being assigned the highest confidence score by OpenIE; (3) containing maximum non-stop words  As shown in Figure FIGREF5, our criteria can select answer-relevant relations (waved in Figure FIGREF5), which is especially useful for sentences with extraneous information  In rare cases, OpenIE cannot extract any relation, we treat the sentence itself as the to the point context 
Table TABREF8 shows some statistics to verify the intuition that the extracted relations can serve as more to the point context  We find that the tokens in relations are 61% more likely to be used in the target question than the tokens in sentences, and thus they are more to the point  On the other hand, on average the sentences contain one more question token than the relations (1 86 v s  2 87)  Therefore, it is still necessary to take the original sentence into account to generate a more accurate question 
Framework Description ::: Our Proposed Model ::: Overview 
As shown in Figure FIGREF10, our framework consists offour components (1) Sentence Encoder and Relation Encoder, (2) Decoder, (3) Gated Attention Mechanism and (4) Dual Copy Mechanism  The sentence encoder and relation encoder encode the unstructured sentence and the structured answer-relevant relation, respectively  To select and combine the source information from the two encoders, a gated attention mechanism is employed to jointly attend both contextualized information sources, and a dual copy mechanism copies words from either the sentence or the relation 
Framework Description ::: Our Proposed Model ::: Answer-aware Encoder 
We employ two encoders to integrate information from the unstructured sentence $S$ and the answer-relevant relation $M$ separately  Sentence encoder takes in feature-enriched embeddings including word embeddings $\mathbf {w}$, linguistic embeddings $\mathbf {l}$ and answer position embeddings $\mathbf {a}$  We follow BIBREF9 to transform POS and NER tags into continuous representation ($\mathbf {l}^p$ and $\mathbf {l}^n$) and adopt a BIO labelling scheme to derive the answer position embedding (B: the first token of the answer, I: tokens within the answer fragment except the first one, O: tokens outside of the answer fragment)  For each word $w_i$ in the sentence $S$, we simply concatenate all features as input: $\mathbf {x}_i^s= [\mathbf {w}_i; \mathbf {l}^p_i; \mathbf {l}^n_i; \mathbf {a}_i]$  Here $[\mathbf {a};\mathbf {b}]$ denotes the concatenation of vectors $\mathbf {a}$ and $\mathbf {b}$ 
We use bidirectional LSTMs to encode the sentence $(\mathbf {x}_1^s, \mathbf {x}_2^s,    , \mathbf {x}_n^s)$ to get a contextualized representation for each token:
where $\overrightarrow{\mathbf {h}}^{s}_i$ and $\overleftarrow{\mathbf {h}}^{s}_i$ are the hidden states at the $i$-th time step of the forward and the backward LSTMs  The output state of the sentence encoder is the concatenation of forward and backward hidden states: $\mathbf {h}^{s}_i=[\overrightarrow{\mathbf {h}}^{s}_i;\overleftarrow{\mathbf {h}}^{s}_i]$  The contextualized representation of the sentence is $(\mathbf {h}^{s}_1, \mathbf {h}^{s}_2,    , \mathbf {h}^{s}_n)$ 
For the relation encoder, we firstly join all items in the n-ary relation $M$ into a sequence  Then we only take answer position embedding as an extra feature for the sequence: $\mathbf {x}_i^m= [\mathbf {w}_i; \mathbf {a}_i]$  Similarly, we take another bidirectional LSTMs to encode the relation sequence and derive the corresponding contextualized representation $(\mathbf {h}^{m}_1, \mathbf {h}^{m}_2,    , \mathbf {h}^{m}_n)$ 
Framework Description ::: Our Proposed Model ::: Decoder 
We use an LSTM as the decoder to generate the question  The decoder predicts the word probability distribution at each decoding timestep to generate the question  At the t-th timestep, it reads the word embedding $\mathbf {w}_{t}$ and the hidden state $\mathbf {u}_{t-1}$ of the previous timestep to generate the current hidden state:
Framework Description ::: Our Proposed Model ::: Gated Attention Mechanism 
We design a gated attention mechanism to jointly attend the sentence representation and the relation representation  For sentence representation $(\mathbf {h}^{s}_1, \mathbf {h}^{s}_2,    , \mathbf {h}^{s}_n)$, we employ the Luong2015EffectiveAT's attention mechanism to obtain the sentence context vector $\mathbf {c}^s_t$,
where $\mathbf {W}_a$ is a trainable weight  Similarly, we obtain the vector $\mathbf {c}^m_t$ from the relation representation $(\mathbf {h}^{m}_1, \mathbf {h}^{m}_2,    , \mathbf {h}^{m}_n)$  To jointly model the sentence and the relation, a gating mechanism is designed to control the information flow from two sources:
where $\odot $ represents element-wise dot production and $\mathbf {W}_g, \mathbf {W}_h$ are trainable weights  Finally, the predicted probability distribution over the vocabulary $V$ is computed as:
where $\mathbf {W}_V$ and $\mathbf {b}_V$ are parameters 
Framework Description ::: Our Proposed Model ::: Dual Copy Mechanism 
To deal with the rare and unknown words, the decoder applies the pointing method BIBREF10, BIBREF11, BIBREF12 to allow copying a token from the input sentence at the $t$-th decoding step  We reuse the attention score $\mathbf {\alpha }_{t}^s$ and $\mathbf {\alpha }_{t}^m$ to derive the copy probability over two source inputs:
Different from the standard pointing method, we design a dual copy mechanism to copy from two sources with two gates  The first gate is designed for determining copy tokens from two sources of inputs or generate next word from $P_V$, which is computed as $g^v_t = \text{sigmoid}(\mathbf {w}^v_g \tilde{\mathbf {h}}_t + b^v_g)$  The second gate takes charge of selecting the source (sentence or relation) to copy from, which is computed as $g^c_t = \text{sigmoid}(\mathbf {w}^c_g [\mathbf {c}_t^s;\mathbf {c}_t^m] + b^c_g)$  Finally, we combine all probabilities $P_V$, $P_S$ and $P_M$ through two soft gates $g^v_t$ and $g^c_t$  The probability of predicting $w$ as the $t$-th token of the question is:
Framework Description ::: Our Proposed Model ::: Training and Inference 
Given the answer $A$, sentence $S$ and relation $M$, the training objective is to minimize the negative log-likelihood with regard to all parameters:
where $\mathcal {\lbrace }Q\rbrace $ is the set of all training instances, $\theta $ denotes model parameters and $\text{log} P(Q|A,S,M;\theta )$ is the conditional log-likelihood of $Q$ 
In testing, our model targets to generate a question $Q$ by maximizing:
Experimental Setting ::: Dataset & Metrics
We conduct experiments on the SQuAD dataset BIBREF3  It contains 536 Wikipedia articles and 100k crowd-sourced question-answer pairs  The questions are written by crowd-workers and the answers are spans of tokens in the articles  We employ two different data splits by following Zhou2017NeuralQG and Du2017LearningTA   In Zhou2017NeuralQG, the original SQuAD development set is evenly divided into dev and test sets, while Du2017LearningTA treats SQuAD development set as its development set and splits original SQuAD training set into a training set and a test set  We also filter out questions which do not have any overlapped non-stop words with the corresponding sentences and perform some preprocessing steps, such as tokenization and sentence splitting  The data statistics are given in Table TABREF27 
We evaluate with all commonly-used metrics in question generation BIBREF13: BLEU-1 (B1), BLEU-2 (B2), BLEU-3 (B3), BLEU-4 (B4) BIBREF17, METEOR (MET) BIBREF18 and ROUGE-L (R-L) BIBREF19  We use the evaluation script released by Chen2015MicrosoftCC 
Experimental Setting ::: Baseline Models
We compare with the following models 
[leftmargin=*]
s2s BIBREF13 proposes an attention-based sequence-to-sequence neural network for question generation 
NQG++ BIBREF9 takes the answer position feature and linguistic features into consideration and equips the Seq2Seq model with copy mechanism 
M2S+cp BIBREF14 conducts multi-perspective matching between the answer and the sentence to derive an answer-aware sentence representation for question generation 
s2s+MP+GSA BIBREF8 introduces a gated self-attention into the encoder and a maxout pointer mechanism into the decoder  We report their sentence-level results for a fair comparison 
Hybrid BIBREF15 is a hybrid model which considers the answer embedding for the question word generation and the position of context words for modeling the relative distance between the context words and the answer 
ASs2s BIBREF16 replaces the answer in the sentence with a special token to avoid its appearance in the generated questions 
Experimental Setting ::: Implementation Details
We take the most frequent 20k words as our vocabulary and use the GloVe word embeddings BIBREF20 for initialization  The embedding dimensions for POS, NER, answer position are set to 20  We use two-layer LSTMs in both encoder and decoder, and the LSTMs hidden unit size is set to 600 
We use dropout BIBREF21 with the probability $p=0 3$  All trainable parameters, except word embeddings, are randomly initialized with the Xavier uniform in $(-0 1, 0 1)$ BIBREF22  For optimization in the training, we use SGD as the optimizer with a minibatch size of 64 and an initial learning rate of 1 0  We train the model for 15 epochs and start halving the learning rate after the 8th epoch  We set the gradient norm upper bound to 3 during the training 
We adopt the teacher-forcing for the training  In the testing, we select the model with the lowest perplexity and beam search with size 3 is employed for generating questions  All hyper-parameters and models are selected on the validation dataset 
Results and Analysis ::: Main Results
Table TABREF30 shows automatic evaluation results for our model and baselines (copied from their papers)  Our proposed model which combines structured answer-relevant relations and unstructured sentences achieves significant improvements over proximity-based answer-aware models BIBREF9, BIBREF15 on both dataset splits  Presumably, our structured answer-relevant relation is a generalization of the context explored by the proximity-based methods because they can only capture short dependencies around answer fragments while our extractions can capture both short and long dependencies given the answer fragments  Moreover, our proposed framework is a general one to jointly leverage structured relations and unstructured sentences  All compared baseline models which only consider unstructured sentences can be further enhanced under our framework 
Recall that existing proximity-based answer-aware models perform poorly when the distance between the answer fragment and other non-stop sentence words that also appear in the ground truth question is large (Table TABREF2)  Here we investigate whether our proposed model using the structured answer-relevant relations can alleviate this issue or not, by conducting experiments for our model under the same setting as in Table TABREF2  The broken-down performances by different relative distances are shown in Table TABREF40  We find that our proposed model outperforms Hybrid (our re-implemented version for this experiment) on all ranges of relative distances, which shows that the structured answer-relevant relations can capture both short and long term answer-relevant dependencies of the answer in sentences  Furthermore, comparing the performance difference between Hybrid and our model, we find the improvements become more significant when the distance increases from “$0\sim 10$” to “$>10$”  One reason is that our model can extract relations with distant dependencies to the answer, which greatly helps our model ignore the extraneous information  Proximity-based answer-aware models may overly emphasize the neighboring words of answers and become less effective as the useful context becomes further away from the answer in the complex sentences  In fact, the breakdown intervals in Table TABREF40 naturally bound its sentence length, say for “$>10$”, the sentences in this group must be longer than 10  Thus, the length variances in these two intervals could be significant  To further validate whether our model can extract long term dependency words  We rerun the analysis of Table TABREF40 only for long sentences (length $>$ 20) of each interval  The improvement percentages over Hybrid are shown in Table TABREF40, which become more significant when the distance increases from “$0\sim 10$” to “$>10$” 
Results and Analysis ::: Case Study
Figure FIGREF42 provides example questions generated by crowd-workers (ground truth questions), the baseline Hybrid BIBREF15, and our model  In the first case, there are two subsequences in the input and the answer has no relation with the second subsequence  However, we see that the baseline model prediction copies irrelevant words “The New York Times” while our model can avoid using the extraneous subsequence “The New York Times noted    ” with the help of the structured answer-relevant relation  Compared with the ground truth question, our model cannot capture the cross-sentence information like “her fifth album”, where the techniques in paragraph-level QG models BIBREF8 may help  In the second case, as discussed in Section SECREF1, this sentence contains a few facts and some facts intertwine  We find that our model can capture distant answer-relevant dependencies such as “mean temperature” while the proximity-based baseline model wrongly takes neighboring words of the answer like “coldest” in the generated question 
Results and Analysis ::: Diverse Question Generation
Another interesting observation is that for the same answer-sentence pair, our model can generate diverse questions by taking different answer-relevant relations as input  Such capability improves the interpretability of our model because the model is given not only what to be asked (i e , the answer) but also the related fact (i e , the answer-relevant relation) to be covered in the question  In contrast, proximity-based answer-aware models can only generate one question given the sentence-answer pair regardless of how many answer-relevant relations in the sentence  We think such capability can also validate our motivation: questions should be generated according to the answer-aware relations instead of neighboring words of answer fragments  Figure FIGREF45 show two examples of diverse question generation  In the first case, the answer fragment `Hugh L  Dryden' is the appositive to `NASA Deputy Administrator' but the subject to the following tokens `announced the Apollo program    '  Our framework can extract these two answer-relevant relations, and by feeding them to our model separately, we can receive two questions asking different relations with regard to the answer 
Related Work
The topic of question generation, initially motivated for educational purposes, is tackled by designing many complex rules for specific question types BIBREF4, BIBREF23  Heilman2010GoodQS improve rule-based question generation by introducing a statistical ranking model  First, they remove extraneous information in the sentence to transform it into a simpler one, which can be transformed easily into a succinct question with predefined sets of general rules  Then they adopt an overgenerate-and-rank approach to select the best candidate considering several features 
With the rise of dominant neural sequence-to-sequence learning models BIBREF5, Du2017LearningTA frame question generation as a sequence-to-sequence learning problem  Compared with rule-based approaches, neural models BIBREF24 can generate more fluent and grammatical questions  However, question generation is a one-to-many sequence generation problem, i e , several aspects can be asked given a sentence, which confuses the model during train and prevents concrete automatic evaluation  To tackle this issue, Zhou2017NeuralQG propose the answer-aware question generation setting which assumes the answer is already known and acts as a contiguous span inside the input sentence  They adopt a BIO tagging scheme to incorporate the answer position information as learned embedding features in Seq2Seq learning  Song2018LeveragingCI explicitly model the information between answer and sentence with a multi-perspective matching model  Kim2019ImprovingNQ also focus on the answer information and proposed an answer-separated Seq2Seq model by masking the answer with special tokens  All answer-aware neural models treat question generation as a one-to-one mapping problem, but existing models perform poorly for sentences with a complex structure (as shown in Table TABREF2) 
Our work is inspired by the process of extraneous information removing in BIBREF0, BIBREF25  Different from Heilman2010GoodQS which directly use the simplified sentence for generation and cao2018faithful which only consider aggregate two sources of information via gated attention in summarization, we propose to combine the structured answer-relevant relation and the original sentence  Factoid question generation from structured text is initially investigated by Serban2016GeneratingFQ, but our focus here is leveraging structured inputs to help question generation over unstructured sentences  Our proposed model can take advantage of unstructured sentences and structured answer-relevant relations to maintain informativeness and faithfulness of generated questions  The proposed model can also be generalized in other conditional sequence generation tasks which require multiple sources of inputs, e g , distractor generation for multiple choice questions BIBREF26 
Conclusions and Future Work
In this paper, we propose a question generation system which combines unstructured sentences and structured answer-relevant relations for generation  The unstructured sentences maintain the informativeness of generated questions while structured answer-relevant relations keep the faithfulness of questions  Extensive experiments demonstrate that our proposed model achieves state-of-the-art performance across several metrics  Furthermore, our model can generate diverse questions with different structured answer-relevant relations  For future work, there are some interesting dimensions to explore, such as difficulty levels BIBREF27, paragraph-level information BIBREF8 and conversational question generation BIBREF28 
Acknowledgments
This work is supported by the Research Grants Council of the Hong Kong Special Administrative Region, China (No  CUHK 14208815 and No  CUHK 14210717 of the General Research Fund)  We would like to thank the anonymous reviewers for their comments  We would also like to thank Department of Computer Science and Engineering, The Chinese University of Hong Kong for the conference grant support 

ground_truth: SQuAD

SQuAD

prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. LDA is an unsupervised method; is this paper introducing an unsupervised approach to spam detection?
Context 4:  In selecting relevant datasets, we do not consider those that are considered “solved”, i e  where the state of the art performance surpasses human performance, as is the case with SQuAD BIBREF28, BIBREF7  Concretely, we selected gold standards that fit our problem definition and were published in the years 2016 to 2019, have at least $(2019 - publication\ year) \times 20$ citations, and bucket them according to the answer selection styles as described in Section SECREF4 We randomly draw one from each bucket and add two randomly drawn datasets from the candidate pool  This leaves us with the datasets described in Table TABREF19  For a more detailed description, we refer to Appendix  
Application of the Framework ::: Annotation Task
We randomly select 50 distinct question, answer and passage triples from the publicly available development sets of the described datasets  Training, development and the (hidden) test set are drawn from the same distribution defined by the data collection method of the respective dataset  For those collections that contain multiple questions over a single passage, we ensure that we are sampling unique paragraphs in order to increase the variety of investigated texts 
The samples were annotated by the first author of this paper, using the proposed schema  In order to validate our findings, we further take 20% of the annotated samples and present them to a second annotator (second author)  Since at its core, the annotation is a multi-label task, we report the inter-annotator agreement by computing the (micro-averaged) F1 score, where we treat the first annotator's labels as gold  Table TABREF21 reports the agreement scores, the overall (micro) average F1 score of the annotations is 0 82, which means that on average, more than two thirds of the overall annotated labels were agreed on by both annotators  We deem this satisfactory, given the complexity of the annotation schema 
Application of the Framework ::: Qualitative Analysis
We present a concise view of the annotation results in Figure FIGREF23  The full annotation results can be found in Appendix   We centre our discussion around the following main points:
Application of the Framework ::: Qualitative Analysis ::: Linguistic Features
As observed in Figure FIGREF23 the gold standards feature a high degree of Redundancy, peaking at 76% of the annotated HotpotQA samples and synonyms and paraphrases (labelled Synonym), with ReCoRd samples containing 58% of them, likely to be attributed to the elaborating type of discourse of the dataset sources (encyclopedia and newswire)  This is, however, not surprising, as it is fairly well understood in the literature that current state-of-the-art models perform well on distinguishing relevant words and phrases from redundant ones BIBREF32  Additionally, the representational capability of synonym relationships of word embeddings has been investigated and is well known BIBREF33  Finally, we observe the presence of syntactic features, such as ambiguous relative clauses, appositions and adverbial phrases, (RelAdvApp 40% in HotpotQA and ReCoRd) and those introducing variance, concretely switching between verbal and nominal styles (e g  Nominalisation 10% in HotpotQA) and from passive to active voice (Voice, 8% in HotpotQA) 
Syntactic features contributing to variety and ambiguity that we did not observe in our samples are the exploitation of verb symmetry, the use of dative and genitive cases or ambiguous prepositions and coordination scope (respectively Symmetry, Dative, Genitive, Prepositions, Scope)  Therefore we cannot establish whether models are capable of dealing with those features by evaluating them on those gold standards 
Application of the Framework ::: Qualitative Analysis ::: Factual Correctness
We identify three common sources that surface in different problems regarding an answer's factual correctness, as reported in Figure FIGREF23 and illustrate their instantiations in Table TABREF31:
Design Constraints: Choosing the task design and the data collection method introduces some constraints that lead to factually debatable examples  For example, a span might have been arbitrarily selected from multiple spans that potentially answer a question, but only a single continuous answer span per question is allowed by design, as observed in the NewsQA and MsMarco samples (32% and 34% examples annotated as Debatable with 16% and 53% thereof exhibiting arbitrary selection, respectively)  Sometimes, when additional passages are added after the annotation step, they can by chance contain passages that answer the question more precisely than the original span, as seen in HotpotQA (16% Debatable samples, 25% of them due to arbitrary selection)  In the case of MultiRC it appears to be inconsistent, whether multiple correct answer choices are expected to be correct in isolation or in conjunction (28% Debatable with 29% of them exhibiting this problem)  This might provide an explanation to its relatively weak human baseline performance of 84% F1 score BIBREF31 
Weak Quality assurance: When the (typically crowd-sourced) annotations are not appropriately validated, incorrect examples will find their way into the gold standards  This typically results in factually wrong expected answers (i e  when a more correct answer is present in the context) or a question is expected to be Unanswerable, but is actually answerable from the provided context  The latter is observed in MsMarco (83% of examples annotated as Wrong) and NewsQA, where 60% of the examples annotated as Wrong are Unanswerable with an answer present 
Arbitrary Precision: There appears to be no clear guideline on how precise the answer is expected to be, when the passage expresses the answer in varying granularities  We annotated instances as Debatable when the expected answer was not the most precise given the context (44% and 29% of Debatable instances in NewsQA and MultiRC, respectively) 
Application of the Framework ::: Qualitative Analysis ::: Semantics-altering grammatical modifiers
We took interest in whether any of the benchmarks contain what we call distracting lexical features (or distractors): grammatical modifiers that alter the semantics of a sentence for the final task of answering the given question while preserving a similar lexical form  An example of such features are cues for (double) Negation (e g , “no”, “not”), which when introduced in a sentence, reverse its meaning  Other examples include modifiers denoting Restrictivity, Factivity and Reasoning (such as Monotonicity and Conditional cues)  Examples of question-answer pairs containing a distractor are shown in Table FIGREF37 
We posit that the presence of such distractors would allow for evaluating reading comprehension beyond potential simple word matching  However, we observe no presence of such features in the benchmarks (beyond Negation in DROP, ReCoRd and HotpotQA, with 4%, 4% and 2% respectively)  This results in gold standards that clearly express the evidence required to obtain the answer, lacking more challenging, i e , distracting, sentences that can assess whether a model can truly understand meaning 
Application of the Framework ::: Qualitative Analysis ::: Other
In the Figure FIGREF23 we observe that Operational and Arithmetic reasoning moderately (6% to 8% combined) appears “in the wild”, i e  when not enforced by the data design as is the case with HotpotQA (80% Operations combined) or DROP (68% Arithmetic combined)  Causal reasoning is (exclusively) present in MultiRC (32%), whereas Temporal and Spatial reasoning requirements seem to not naturally emerge in gold standards  In ReCoRd, a fraction of 38% questions can only be answered By Exclusion of every other candidate, due to the design choice of allowing questions where the required information to answer them is not fully expressed in the accompanying paragraph 
Therefore, it is also a little surprising to observe that ReCoRd requires external resources with regard to knowledge, as seen in Figure FIGREF23  MultiRC requires technical or more precisely basic scientific knowledge (6% Technical/Scientific), as a portion of paragraphs is extracted from elementary school science textbooks BIBREF31  Other benchmarks moderately probe for factual knowledge (0% to 4% across all categories), while Intuitive knowledge is required to derive answers in each gold standard 
It is also worth pointing out, as done in Figure FIGREF23, that although MultiRC and MsMarco are not modelled as a span selection problem, their samples still contain 50% and 66% of answers that are directly taken from the context  DROP contains the biggest fraction of generated answers (60%), due to the requirement of arithmetic operations 
To conclude our analysis, we observe similar distributions of linguistic features and reasoning patterns, except where there are constraints enforced by dataset design, annotation guidelines or source text choice  Furthermore, careful consideration of design choices (such as single-span answers) is required, to avoid impairing the factual correctness of datasets, as pure crowd-worker agreement seems not sufficient in multiple cases 
Application of the Framework ::: Quantitative Results ::: Lexical overlap
We used the scores assigned by our proposed set of metrics (discussed in Section SECREF11 Dimensions of Interest: Complexity) to predict the supporting facts in the gold standard samples (that we included in our manual annotation)  Concretely, we used the following five features capturing lexical overlap: (i) the number of words occurring in sentence and question, (ii) the length of the longest n-gram shared by sentence and question, whether a (iii) uni- and (iv) bigram from the question is unique to a sentence, and (v) the sentence index, as input to a logistic regression classifier  We optimised on each sample leaving one example for evaluation  We compute the average Precision, Recall and F1 score by means of leave-one-out validation with every sample entry  The averaged results after 5 runs are reported in Table TABREF41 
We observe that even by using only our five features based lexical overlap, the simple logistic regression baseline is able to separate out the supporting facts from the context to a varying degree  This is in line with the lack of semantics-altering grammatical modifiers discussed in the qualitative analysis section above  The classifier performs best on DROP (66% F1) and MultiRC (40% F1), which means that lexical cues can considerably facilitate the search for the answer in those gold standards  On MultiRC, yadav2019quick come to a similar conclusion, by using a more sophisticated approach based on overlap between question, sentence and answer choices 
Surprisingly, the classifier is able to pick up a signal from supporting facts even on data that has been pruned against lexical overlap heuristics by populating the context with additional documents that have high overlap scores with the question  This results in significantly higher scores than when guessing randomly (HotpotQA 26% F1, and MsMarco 11% F1)  We observe similar results in the case the length of the question leaves few candidates to compute overlap with $6 3$ and $7 3$ tokens on average for MsMarco and NewsQA (26% F1), compared to $16 9$ tokens on average for the remaining four dataset samples 
Finally, it is worth mentioning that although the queries in ReCoRd are explicitly independent from the passage, the linear classifier is still capable of achieving 34% F1 score in predicting the supporting facts 
However, neural networks perform significantly better than our admittedly crude baseline (e g  66% F1 for supporting facts classification on HotpotQA BIBREF14), albeit utilising more training examples, and a richer sentence representation  This facts implies that those neural models are capable of solving more challenging problems than simple “text matching” as performed by the logistic regression baseline  However, they still circumvent actual reading comprehension as the respective gold standards are of limited suitability to evaluate this BIBREF34, BIBREF35  This suggests an exciting future research direction, that is categorising the scale between text matching and reading comprehension more precisely and respectively positioning state-of-the-art models thereon 
Related Work
Although not as prominent as the research on novel architecture, there has been steady progress in critically investigating the data and evaluation aspects of NLP and machine learning in general and MRC in particular 
Related Work ::: Adversarial Evaluation
The authors of the AddSent algorithm BIBREF11 show that MRC models trained and evaluated on the SQuAD dataset pay too little attention to details that might change the semantics of a sentence, and propose a crowd-sourcing based method to generate adversary examples to exploit that weakness  This method was further adapted to be fully automated BIBREF36 and applied to different gold standards BIBREF35  Our proposed approach differs in that we aim to provide qualitative justifications for those quantitatively measured issues 
Related Work ::: Sanity Baselines
Another line of research establishes sane baselines to provide more meaningful context to the raw performance scores of evaluated models  When removing integral parts of the task formulation such as question, the textual passage or parts thereof BIBREF37 or restricting model complexity by design in order to suppress some required form of reasoning BIBREF38, models are still able to perform comparably to the state-of-the-art  This raises concerns about the perceived benchmark complexity and is related to our work in a broader sense as one of our goals is to estimate the complexity of benchmarks 
Related Work ::: Benchmark evaluation in NLP
Beyond MRC, efforts similar to ours that pursue the goal of analysing the evaluation of established datasets exist in Natural Language Inference BIBREF13, BIBREF12  Their analyses reveal the existence of biases in training and evaluation data that can be approximated with simple majority-based heuristics  Because of these biases, trained models fail to extract the semantics that are required for the correct inference  Furthermore, a fair share of work was done to reveal gender bias in coreference resolution datasets and models BIBREF39, BIBREF40, BIBREF41 
Related Work ::: Annotation Taxonomies
Finally, related to our framework are works that introduce annotation categories for gold standards evaluation  Concretely, we build our annotation framework around linguistic features that were introduced in the GLUE suite BIBREF42 and the reasoning categories introduced in the WorldTree dataset BIBREF19  A qualitative analysis complementary to ours, with focus on the unanswerability patterns in datasets that feature unanswerable questions was done by Yatskar2019 
Conclusion
In this paper, we introduce a novel framework to characterise machine reading comprehension gold standards  This framework has potential applications when comparing different gold standards, considering the design choices for a new gold standard and performing qualitative error analyses for a proposed approach 
Furthermore we applied the framework to analyse popular state-of-the-art gold standards for machine reading comprehension: We reveal issues with their factual correctness, show the presence of lexical cues and we observe that semantics-altering grammatical modifiers are missing in all of the investigated gold standards  Studying how to introduce those modifiers into gold standards and observing whether state-of-the-art MRC models are capable of performing reading comprehension on text containing them, is a future research goal 
A future line of research is to extend the framework to be able to identify the different types of exploitable cues such as question or entity typing and concrete overlap patterns  This will allow the framework to serve as an interpretable estimate of reading comprehension complexity of gold standards  Finally, investigating gold standards under this framework where MRC models outperform the human baseline (e g  SQuAD) will contribute to a deeper understanding of the seemingly superb performance of deep learning approaches on them 

ground_truth: No

No