prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. How big is imbalance in analyzed corpora?
Context 1: Introduction
In recent years, gender has become a hot topic within the political, societal and research spheres  Numerous studies have been conducted in order to evaluate the presence of women in media, often revealing their under-representation, such as the Global Media Monitoring Project BIBREF0  In the French context, the CSA BIBREF1 produces a report on gender representation in media on a yearly basis  The 2017 report shows that women represent 40% of French media speakers, with a significant drop during high-audience hours (6:00-8:00pm) reaching a value of only 29%  Another large scale study confirmed this trend with an automatic analysis of gender in French audiovisuals streams, highlighting a huge variation across type of shows BIBREF2 
Besides the social impact of gender representation, broadcast recordings are also a valuable source of data for the speech processing community  Indeed, automatic speech recognition (ASR) systems require large amount of annotated speech data to be efficiently trained, which leaves us facing the emerging concern about the fact that "AI artifacts tend to reflect the goals, knowledge and experience of their creators" BIBREF3  Since we know that women are under-represented in media and that the AI discipline has retained a male-oriented focus BIBREF4, we can legitimately wonder about the impact of using such data as a training set for ASR technologies  This concern is strengthened by the recent works uncovering gender bias in several natural language processing (NLP) tools such as BIBREF5, BIBREF6, BIBREF7, BIBREF8 
In this paper, we first highlight the importance of TV and radio broadcast as a source of data for ASR, and the potential impact it can have  We then perform a statistical analysis of gender representation in a data set composed of four state-of-the-art corpora of French broadcast, widely used within the speech community  Finally we question the impact of such a representation on the systems developed on this data, through the perspective of an ASR system 
From gender representation in data to gender bias in AI ::: On the importance of data
The ever growing use of machine learning in science has been enabled by several progresses among which the exponential growth of data available  The quality of a system now depends mostly on the quality and quantity of the data it has been trained on  If it does not discard the importance of an appropriate architecture, it reaffirms the fact that rich and large corpora are a valuable resource  Corpora are research contributions which do not only allow to save and observe certain phenomena or validate a hypothesis or model, but are also a mandatory part of the technology development  This trend is notably observable within the NLP field, where industrial technologies, such as Apple, Amazon or Google vocal assistants now reach high performance level partly due to the amount of data possessed by these companies BIBREF9 
Surprisingly, as data is said to be “the new oil", few data sets are available for ASR systems  The best known are corpora like TIMIT BIBREF10, Switchboard BIBREF11 or Fisher BIBREF12 which date back to the early 1990s  The scarceness of available corpora is justified by the fact that gathering and annotating audio data is costly both in terms of money and time  Telephone conversations and broadcast recordings have been the primary source of spontaneous speech used  Out of all the 130 audio resources proposed by LDC to train automatic speech recognition systems in English, approximately 14% of them are based on broadcast news and conversation  For French speech technologies, four corpora containing radio and TV broadcast are the most widely used: ESTER1 BIBREF13, ESTER2 BIBREF14, ETAPE BIBREF15 and REPERE BIBREF16  These four corpora have been built alongside evaluation campaigns and are still, to our knowledge, the largest French ones of their type available to date 
From gender representation in data to gender bias in AI ::: From data to bias
The gender issue has returned to the forefront of the media scene in recent years and with the emergence of AI technologies in our daily lives, gender bias has become a scientific topic that researchers are just beginning to address  Several studies revealed the existence of gender bias in AI technologies such as face recognition (GenderShades BIBREF17), NLP (word embeddings BIBREF5 and semantics BIBREF6) and machine translation (BIBREF18, BIBREF7)  The impact of the training data used within these deep-learning algorithms is therefore questioned 
Bias can be found at different levels as pointed out by BIBREF19  BIBREF20 defines bias as a skew that produces a type of harm  She distinguishes two types of harms that are allocation harm and representation harm  The allocation harm occurs when a system is performing better or worse for a certain group while representational harm contributes to the perpetuation of stereotypes  Both types of harm are the results of bias in machine learning that often comes from the data systems are trained on  Disparities in representation in our social structures is captured and reflected by the training data, through statistical patterns  The GenderShades study is a striking example of what data disparity and lack of representation can produce: the authors tested several gender recognition modules used by facial recognition tools and found difference in error-rate as high as 34 percentage points between recognition of white male and black female faces  The scarce presence of women and colored people in training set resulted in bias in performance towards these two categories, with a strong intersectional bias  As written by BIBREF21 "A data set may have many millions of pieces of data, but this does not mean it is random or representative  To make statistical claims about a data set, we need to know where data is coming from; it is similarly important to know and account for the weaknesses in that data " (p 668) 
Regarding ASR technology, little work has explored the presence of gender bias within the systems and no consensus has been reached  BIBREF22 found that speech recognizers perform better on female voice on a broadcast news and telephone corpus  They proposed several explanations to this observation, such as the larger presence of non-professional male speech in the broadcast data, implying a less prepared speech for these speakers or a more normative language and standard pronunciation for women linked to the traditional role of women in language acquisition and education  The same trend was observed by BIBREF23  More recently, BIBREF24 discovered a gender bias within YouTube's automatic captioning system but this bias was not observed in a second study evaluating Bing Speech system and YouTube Automatic Captions on a larger data set BIBREF8  However race and dialect bias were found  General American speakers and white speakers had the lowest error rate for both systems  If the better performance on General American speakers could be explained by the fact that they are all voice professionals, producing clear and articulated speech, but no explanation is provided for biases towards non-white speakers 
Gender bias in ASR technology is still an open research question as no clear answer has been reached so far  It seems that many parameters are to take into account to achieve a general agreement  As we established the importance of TV and radio broadcast as a source of data for ASR, and the potential impact it can have, the following content of this paper is structured as this: we first describe statistically the gender representation of a data set composed of four state-of-the-art corpora of French broadcast, widely used within the speech community, introducing the notion of speaker's role to refine our analysis in terms of voice professionalism  We then question the impact of such a representation on a ASR system trained on these data  BIBREF25
Methodology
This section is organized as follows: we first present the data we are working on  In a second time we explain how we proceed to describe the gender representation in our corpus and introduce the notion of speaker's role  The third subsection introduces the ASR system and metrics used to evaluate gender bias in performance 
Methodology ::: Data presentation
Our data consists of two sets used to train and evaluate our automatic speech recognition system  Four major evaluation campaigns have enabled the creation of wide corpora of French broadcast speech: ESTER1 BIBREF13, ESTER2 BIBREF14, ETAPE BIBREF15 and REPERE BIBREF16  These four collections contain radio and/or TV broadcasts aired between 1998 and 2013 which are used by most academic researchers in ASR  Show duration varies between 10min and an hour  As years went by and speech processing research was progressing, the difficulty of the tasks augmented and the content of these evaluation corpora changed  ESTER1 and ESTER2 mainly contain prepared speech such as broadcast news, whereas ETAPE and REPERE consists also of debates and entertainment shows, spontaneous speech introducing more difficulty in its recognition 
Our training set contains 27,085 speech utterances produced by 2,506 speakers, accounting for approximately 100 hours of speech  Our evaluation set contains 74,064 speech utterances produced by 1,268 speakers for a total of 70 hours of speech  Training data by show, medium and speech type is summarized in Table and evaluation data in Table   Evaluation data has a higher variety of shows with both prepared (P) and spontaneous (S) speech type (accented speech from African radio broadcast is also included in the evaluation set) 
Methodology ::: Methodology for descriptive analysis of gender representation in training data
We first describe the gender representation in training data  Gender representation is measured in terms of number of speakers, number of utterances (or speech turns), and turn lengths (descriptive statistics are given in Section SECREF16)  Each speech turn was mapped to its speaker in order to associate it with a gender 
As pointed out by the CSA report BIBREF1, women presence tends to be marginal within the high-audience hours, showing that women are represented but less than men and within certain given conditions  It is clear that a small number of speakers is responsible for a large number of speech turns  Most of these speakers are journalists, politicians, presenters and such, who are representative of a show  Therefore, we introduce the notion of speaker's role to refine our exploration of gender disparity, following studies which quantified women's presence in terms of role  Within our work, we define the notion of speaker role by two criteria specifying the speaker's on-air presence, namely the number of speech turns and the cumulative duration of his or her speaking time in a show  Based on the available speech transcriptions and meta-data, we compute for each speaker the number of speech turns uttered as well as their total length  We then use the following criteria to define speaker's role: a speaker is considered as speaking often (respectively seldom) if he/she accumulates a total of turns higher (respectively lower) than 1% of the total number of speech turns in a given show  The same process is applied to identify speakers talking for a long period from those who do not  We end up with two salient roles called Anchors and Punctual speakers:
the Anchor speakers (A) are above the threshold of 1% for both criteria, meaning they are intervening often and for a long time thus holding an important place in interaction;
the Punctual speakers (PS) on the contrary are below the threshold of 1% for both the total number of turns and the total speech time 
These roles are defined at the show level  They could be roughly assimilated to the categorization “host/guest” in radio and TV shows  Anchors could be described as professional speakers, producing mostly prepared speech, whereas Punctual speakers are more likely to be “everyday people"  The concept of speaker's role makes sense at both sociological and technical levels  An Anchor speaker is more likely to be known from the audience (society), but he or she will also likely have a professional (clear) way of speaking (as mentioned by BIBREF22 and BIBREF8), as well as a high number of utterances, augmenting the amount of data available for a given gender category 
Methodology ::: Gender bias evaluation procedure of an ASR system performance ::: ASR system
The ASR system used in this work is described in BIBREF25  It uses the KALDI toolkit BIBREF26, following a standard Kaldi recipe  The acoustic model is based on a hybrid HMM-DNN architecture and trained on the data summarized in Table   Acoustic training data correspond to 100h of non-spontaneous speech type (mostly broadcast news) coming from both radio and TV shows  A 5-gram language model is trained from several French corpora (3,323M words in total) using SRILM toolkit BIBREF27  The pronunciation model is developed using the lexical resource BDLEX BIBREF28 as well as automatic grapheme-to-phoneme (G2P) transcription to find pronunciation variants of our vocabulary (limited to 80K)  It is important to re-specify here, for further analysis, that our Kaldi pipeline follows speaker adaptive training (SAT) where we train and decode using speaker adapted features (fMLLR-adapted features) in per-speaker mode  It is well known that speaker adaptation acts as an effective procedure to reduce mismatch between training and evaluation conditions BIBREF29, BIBREF26 
Methodology ::: Gender bias evaluation procedure of an ASR system performance ::: Evaluation
Word Error Rate (WER) is a common metric to evaluate ASR performance  It is measured as the sum of errors (insertions, deletions and substitutions) divided by the total number of words in the reference transcription  As we are investigating the impact on performance of speaker's gender and role, we computed the WER for each speaker at the episode (show occurrence) level  Analyzing at such granularity allows us to avoid large WER variation that could be observed at utterance level (especially for short speech turns) but also makes possible to get several WER values for a given speaker, one for each occurrence of a show in which he/she appears on  Speaker's gender was provided by the meta-data and role was obtained using the criteria from Section SECREF6 computed for each show  This enables us to analyze our results across gender and role categories which was done using Wilcoxon rank sum tests also called Mann-Whitney U test (with $\alpha $= 0 001) BIBREF30  The choice of a Wilcoxon rank sum test and not the commonly used t-test is motivated by the non-normality of our data 
Results ::: Descriptive analysis of gender representation in training data ::: Gender representation
As expected, we observe a disparity in terms of gender representation in our data (see Table )  Women represent 33 16% of the speakers, confirming the figures given by the GMMP report BIBREF0  However, it is worth noticing that women account for only 22 57% of the total speech time, which leads us to conclude that women also speak less than men 
Results ::: Descriptive analysis of gender representation in training data ::: Speaker's role representation
Table presents roles' representation in training data and shows that despite the small number of Anchor speakers in our data (3 79%), they nevertheless concentrate 35 71 % of the total speech time 
Results ::: Descriptive analysis of gender representation in training data ::: Role and gender interaction
When crossing both parameters, we can observe that the gender distribution is not constant throughout roles  Women represent 29 47% of the speakers within the Anchor category, even less than among the Punctual speakers  Their percentage of speech is also smaller  When calculating the average speech time uttered by a female Anchor, we obtain a value of 15 9 min against 25 2 min for a male Anchor, which suggests that even within the Anchor category men tend to speak more  This confirms the existence of gender disparities within French media  It corroborates with the analysis of the CSA BIBREF1, which shows that women were less present during high-audience hours  Our study shows that they are also less present in important roles  These results legitimate our initial questioning on the impact of gender balance on ASR performance trained on broadcast recordings 
Results ::: Performance (WER) analysis on evaluation data ::: Impact of gender on WER
As explained in Section SECREF13, WER is the sum of errors divided by the number of words in the transcription reference  The higher the WER, the poorer the system performance  Our 70h evaluation data contains a large amount of spontaneous speech and is very challenging for the ASR system trained on prepared speech: we observe an overall average WER of 42 9% for women and 34 3% for men  This difference of WER between men and women is statistically significant (med(M) = 25%; med(F) = 29%; U = 709040; p-value < 0 001) 
However, when observing gender differences across shows, no clear trend can be identified, as shown in Figure FIGREF21  For shows like Africa1 Infos or La Place du Village, we find an average WER lower for women than for men, while the trend is reversed for shows such as Un Temps de Pauchon or Le Masque et la Plume  The disparity of the results depending on the show leads us to believe that other factors may be entangled within the observed phenomenon 
Results ::: Performance (WER) analysis on evaluation data ::: Impact of role on WER
Speaker's role seems to have an impact on WER: we obtain an average WER of 30 8% for the Anchor speakers and 42 23% for the Punctual speakers  This difference is statistically significant with a p-value smaller than $10^{-14}$ (med(A) = 21%; med(P) = 31%; U = 540,430; p-value < 0 001)  
Results ::: Performance (WER) analysis on evaluation data ::: Role and gender interaction
Figure FIGREF25 presents the WER distribution (WER being obtained for each speaker in a show occurrence) according to the speaker's role and gender  It is worth noticing that the gender difference is only significant within the Punctual speakers group  The average WER is of 49 04% for the women and 38 56% for the men with a p-value smaller than $10^{-6}$ (med(F) = 39%; med(M) = 29%; U = 251,450; p-value < 0 001), whereas it is just a trend between male and female Anchors (med(F) = 21%; med(M) = 21%; U = 116,230; p-value = 0 173)  This could be explained by the quantity of data available per speaker 
Results ::: Performance (WER) analysis on evaluation data ::: Speech type as a third entangled factor?
In order to try to explain the observed variation in our results depending on shows and gender (Figure FIGREF21), we add the notion of speech type to shed some light on our results  BIBREF22 and BIBREF24 suggested that the speaker professionalism, associated with clear and hyper-articulated speech could be an explaining factor for better performance 
Based on our categorization in prepared speech (mostly news reports) and spontaneous speech (mostly debates and entertainment shows), we cross this parameter in our performance analysis  As shown on Figure FIGREF26, these results confirm the inherent challenge of spontaneous speech compared to prepared speech  WER scores are similar between men and women when considering prepared speech (med(F) = 18%; med(M) = 21%; U = 217,160; p-value = 0 005) whereas they are worse for women (61 29%) than for men (46 51%) with p-value smaller than $10^{-14}$ for the spontaneous speech type (med(F) = 61%; med(M) = 37%; U = 153,580; p-value < 0 001) 
Discussion
We find a clear disparity in terms of women presence and speech quantity in French media  Our data being recorded between 1998 and 2013, we can expect this disparity to be smaller on more recent broadcast recordings, especially since the French government displays efforts toward parity in media representation  One can also argue that even if our analysis was conducted on a large amount of data it does not reach the exhaustiveness of large-scale studies such as the one of BIBREF2  Nonetheless it does not affect the relevance of our findings, because if real-world gender representation might be more balanced today, these corpora are still used as training data for AI systems 
The performance difference across gender we observed corroborates (on a larger quantity and variety of language data produced by more than 2400 speakers) the results obtained by BIBREF24 on isolated words recognition  However the following study on read speech does not replicate these results  Yet a performance degradation is observed across dialect and race BIBREF8  BIBREF22 found lower WER for women than men on broadcast news and conversational telephone speech for both English and French  The authors suggest that gender stereotypes associated with women role in education and language acquisition induce a more normative elocution  We observed that the higher the degree of normativity of speech the smaller the gender difference  No significant gender bias is observed for prepared speech nor within the Anchor category  Even if we do not find similar results with lower WER for women than men, we obtained a median WER smaller for women on prepared speech and equal to the male median WER for the Anchor speakers 
Another explanation could be the use of adaptation within the pipeline  Most broadcast programs transcription systems have a speaker adaptation step within their decoding pipeline, which is the case for our system  An Anchor speaker intervening more often would have a larger quantity of data to realize such adaptation of the acoustic model  On the contrary, Punctual speakers who appear scarcely in the data are not provided with the same amount of adaptation data  Hence we can hypothesize that gender performance difference observed for Punctual speakers is due to the fact that female speech is further from the (initial non-adapted) acoustic model as it was trained on unbalanced data (as shown in Table )  Considering that Punctual speakers represent 92 78% of the speakers, this explains why gender difference is significant over our entire data set  A way to confirm our hypothesis would be to reproduce our analysis on WER values obtained without using speaker adapted features at the decoding step 
When decoding prepared speech (hence similar to the training data), no significant difference is found in WER between men and women, revealing that the speaker adaptation step could be sufficient to reach same performance for both genders  But when decoding more spontaneous speech, there is a mismatch with the initial acoustic model (trained on prepared speech)  Consequently, the speaker adaptation step might not be enough to recover good ASR performance, especially for women for whom less adaptation data is available (see Section 4 2 3) 
Conclusion
This paper has investigated gender bias in ASR performance through the following research questions: i) what is the proportion of men and women in French radio and TV media data ? ii) what is the impact of the observed disparity on ASR performance ? iii) is this as simple as a problem of gender proportion in the training data or are other factors entangled ? Our contributions are the following:
Descriptive analysis of the broadcast data used to train our ASR system confirms the already known disparity, where 65% of the speakers are men, speaking more than 75% of the time 
When investigating WER scores according to gender, speaker's role and speech type, huge variations are observed  We conclude that gender is clearly a factor of variation in ASR performance, with a WER increase of 24% for women compared to men, exhibiting a clear gender bias 
Gender bias varies across speaker's role and speech spontaneity level  Performance for Punctual speakers respectively spontaneous speech seems to reinforce this gender bias with a WER increase of 27 2% respectively 31 8% between male and female speakers 
We found that an ASR system trained on unbalanced data regarding gender produces gender bias performance  Therefore, in order to create fair systems it is necessary to take into account the representation problems in society that are going to be encapsulated in the data  Understanding how women under-representation in broadcast data can lead to bias in ASR performances is the key to prevent re-implementing and reinforcing discrimination already existing in our societies  This is in line with the concept of “Fairness by Design" proposed by BIBREF31 
Gender, race, religion, nationality are all characteristics that we deem unfair to classify on, and these ethical standpoints needs to be taken into account in systems' design  Characteristics that are not considered as relevant in a given task can be encapsulated in data nonetheless, and lead to bias performance  Being aware of the demographic skews our data set might contain is a first step to track the life cycle of a training data set and a necessary step to control the tools we develop 
Context 3: Introduction
Headline generation is the process of creating a headline-style sentence given an input article  The research community has been regarding the task of headline generation as a summarization task BIBREF1, ignoring the fundamental differences between headlines and summaries  While summaries aim to contain most of the important information from the articles, headlines do not necessarily need to  Instead, a good headline needs to capture people's attention and serve as an irresistible invitation for users to read through the article  For example, the headline “$2 Billion Worth of Free Media for Trump”, which gives only an intriguing hint, is considered better than the summarization style headline “Measuring Trump’s Media Dominance” , as the former gets almost three times the readers as the latter  Generating headlines with many clicks is especially important in this digital age, because many of the revenues of journalism come from online advertisements and getting more user clicks means being more competitive in the market  However, most existing websites naively generate sensational headlines using only keywords or templates  Instead, this paper aims to learn a model that generates sensational headlines based on an input article without labeled data 
To generate sensational headlines, there are two main challenges  Firstly, there is a lack of sensationalism scorer to measure how sensational a headline is  Some researchers have tried to manually label headlines as clickbait or non-clickbait BIBREF2, BIBREF3  However, these human-annotated datasets are usually small and expensive to collect  To capture a large variety of sensationalization patterns, we need a cheap and easy way to collect a large number of sensational headlines  Thus, we propose a distant supervision strategy to collect a sensationalism dataset  We regard headlines receiving lots of comments as sensational samples and the headlines generated by a summarization model as non-sensational samples  Experimental results show that by distinguishing these two types of headlines, we can partially teach the model a sense of being sensational 
Secondly, after training a sensationalism scorer on our sensationalism dataset, a natural way to generate sensational headlines is to maximize the sensationalism score using reinforcement learning (RL)  However, the following shows an example of a RL model maximizing the sensationalism score by generating a very unnatural sentence, while its sensationalism scorer gave a very high score of 0 99996: UTF8gbsn 十个可穿戴产品的设计原则这消息消息可惜说明 Ten design principles for wearable devices, this message message pity introduction  This happens because the sensationalism scorer can make mistakes and RL can generate unnatural phrases which fools our sensationalism scorer  Thus, how to effectively leverage RL with noisy rewards remains an open problem  To deal with the noisy reward, we introduce Auto-tuned Reinforcement Learning (ARL)  Our model automatically tunes the ratio between MLE and RL based on how sensational the training headline is  In this way, we effectively take advantage of RL with a noisy reward to generate headlines that are both sensational and fluent 
The major contributions of this paper are as follows: 1) To the best of our knowledge, we propose the first-ever model that tackles the sensational headline generation task with reinforcement learning techniques  2) Without human-annotated data, we propose a distant supervision strategy to train a sensationalism scorer as a reward function 3) We propose a novel loss function, Auto-tuned Reinforcement Learning, to give dynamic weights to balance between MLE and RL  Our code will be released  
Sensationalism Scorer
To evaluate the sensationalism intensity score $\alpha _{\text{sen}}$ of a headline, we collect a sensationalism dataset and then train a sensationalism scorer  For the sensationalism dataset collection, we choose headlines with many comments from popular online websites as positive samples  For the negative samples, we propose to use the generated headlines from a sentence summarization model  Intuitively, the summarization model, which is trained to preserve the semantic meaning, will lose the sensationalization ability and thus the generated negative samples will be less sensational than the original one, similar to the obfuscation of style after back-translation BIBREF4  For example, an original headline like UTF8gbsn“一趟挣10万？铁总增开申通、顺丰专列" (One trip to earn 100 thousand? China Railway opens new Shentong and Shunfeng special lines) will become UTF8gbsn“中铁总将增开京广两列快递专列" (China Railway opens two special lines for express) from the baseline model, which loses the sensational phrases of UTF8gbsn“一趟挣10万？" (One trip to earn 100 thousand?)   We then train the sensationalism scorer by classifying sensational and non-sensational headlines using a one-layer CNN with a binary cross entropy loss $L_{\text{sen}}$  Firstly, 1-D convolution is used to extract word features from the input embeddings of a headline  This is followed by a ReLU activation layer and a max-pooling layer along the time dimension  All features from different channels are concatenated together and projected to the sensationalism score by adding another fully connected layer with sigmoid activation  Binary cross entropy is used to compute the loss $L_{\text{sen}}$ 
Sensationalism Scorer ::: Training Details and Dataset
For the CNN model, we choose filter sizes of 1, 3, and 5 respectively  Adam is used to optimize $L_{sen}$ with a learning rate of 0 0001  We set the embedding size as 300 and initialize it from qiu2018revisiting trained on the Weibo corpus with word and character features  We fix the embeddings during training  For dataset collection, we utilize the headlines collected in qin2018automatic, lin2019learning from Tencent News, one of the most popular Chinese news websites, as the positive samples  We follow the same data split as the original paper  As some of the links are not available any more, we get 170,754 training samples and 4,511 validation samples  For the negative training samples collection, we randomly select generated headlines from a pointer generator BIBREF0 model trained on LCSTS dataset BIBREF5 and create a balanced training corpus which includes 351,508 training samples and 9,022 validation samples  To evaluate our trained classifier, we construct a test set by randomly sampling 100 headlines from the test split of LCSTS dataset and the labels are obtained by 11 human annotators  Annotations show that 52% headlines are labeled as positive and 48% headlines as negative by majority voting (The detail on the annotation can be found in Section SECREF26) 
Sensationalism Scorer ::: Results and Discussion
Our classifier achieves 0 65 accuracy and 0 65 averaged F1 score on the test set while a random classifier would only achieve 0 50 accuracy and 0 50 averaged F1 score  This confirms that the predicted sensationalism score can partially capture the sensationalism of headlines  On the other hand, a more natural choice is to take headlines with few comments as negative examples  Thus, we train another baseline classifier on a crawled balanced sensationalism corpus of 84k headlines where the positive headlines have at least 28 comments and the negative headlines have less than 5 comments  However, the results on the test set show that the baseline classifier gets 60% accuracy, which is worse than the proposed classifier (which achieves 65%)  The reason could be that the balanced sensationalism corpus are sampled from different distributions from the test set and it is hard for the trained model to generalize  Therefore, we choose the proposed one as our sensationalism scorer  Therefore, our next challenge is to show that how to leverage this noisy sensationalism reward to generate sensational headlines 
Sensational Headline Generation
Our sensational headline generation model takes an article as input and output a sensational headline  The model consists of a Pointer-Gen headline generator and is trained by ARL  The diagram of ARL can be found in Figure FIGREF6 
We denote the input article as $x=\lbrace x_1,x_2,x_3,\cdots ,x_M\rbrace $, and the corresponding headline as $y^*=\lbrace y_1^*,y_2^*,y_3^*,\cdots ,y_T^*\rbrace $, where $M$ is the number of tokens in an article and $T$ is the number of tokens in a headline 
Sensational Headline Generation ::: Pointer-Gen Headline Generator
We choose Pointer Generator (Pointer-Gen) BIBREF0, a widely used summarization model, as our headline generator for its ability to copy words from the input article  It takes a news article as input and generates a headline  Firstly, the tokens of each article, $\lbrace x_1,x_2,x_3,\cdots ,x_M\rbrace $, are fed into the encoder one-by-one and the encoder generates a sequence of hidden states $h_i$  For each decoding step $t$, the decoder receives the embedding for each token of a headline $y_t$ as input and updates its hidden states $s_t$  An attention mechanism following luong2015effective is used:
where $v$, $W_h$, $W_s$, and $b_{attn}$ are the trainable parameters and $h_t^*$ is the context vector  $s_t$ and $h_t^*$ are then combined to give a probability distribution over the vocabulary through two linear layers:
where $V$, $b$, $V^{^{\prime }}$, and $b^{^{\prime }}$ are trainable parameters  We use a pointer generator network to enable our model to copy rare/unknown words from the input article, giving the following final word probability:
where $x^t$ is the embedding of the input word of the decoder, $w_{h^*}^T$, $w_s^T$, $w_x^T$, and $b_{ptr}$ are trainable parameters, and $\sigma $ is the sigmoid function 
Sensational Headline Generation ::: Training Methods
We first briefly introduce MLE and RL objective functions, and a naive way to mix these two by a hyper-parameter $\lambda $  Then we point out the challenge of training with noisy reward, and propose ARL to address this issue 
Sensational Headline Generation ::: Training Methods ::: MLE and RL
A headline generation model can be trained with MLE, RL or a combination of MLE and RL  MLE training is to minimize the negative log likelihood of the training headlines  We feed $y^*$ into the decoder word by word and maximize the likelihood of $y^*$  The loss function for MLE becomes
For RL training, we choose the REINFORCE algorithm BIBREF6  In the training phase, after encoding an article, a headline $y^s = \lbrace y_1^s, y_2^s, y_3^s, \cdots , y_T^s\rbrace $ is obtained by sampling from $P(w)$ from our generator, and then a reward of sensationalism or ROUGE(RG) is calculated 
We use the baseline reward $\hat{R_t}$ to reduce the variance of the reward, similar to ranzato2015sequence  To elaborate, a linear model is deployed to estimate the baseline reward $\hat{R_t}$ based on $t$-th state $o_t$ for each timestep $t$  The parameters of the linear model are trained by minimizing the mean square loss between $R$ and $\hat{R_t}$:
where $W_r$ and $b_r$ are trainable parameters  To maximize the expected reward, our loss function for RL becomes
A naive way to mix these two objective functions using a hyper-parameter $\lambda $ has been successfully incorporated in the summarization task BIBREF7  It includes the MLE training as a language model to mitigate the readability and quality issues in RL  The mixed loss function is shown as follows:
where $*$ is the reward type  Usually $\lambda $ is large, and paulus2017deep used 0 9984 
Sensational Headline Generation ::: Training Methods ::: Auto-tuned Reinforcement Learning
Applying the naive mixed training method using sensationalism score as the reward is not obvious/trivial in our task  The main reason is that our sensationalism reward is notably more noisy and more fragile than the ROUGE-L reward or abstractive reward used in the summarization task BIBREF7, BIBREF8  A higher ROUGE-L F1 reward in summarization indicates higher overlapping ratio between generation and true summary statistically, but our sensationalism reward is a learned score which is fragile to be fooled with unnatural samples 
To effectively train the model with RL under noisy sensationalism reward, our idea is to balance RL with MLE  However, we argue that the weighted ratio between MLE and RL should be sample-dependent, instead of being fixed for all training samples as in paulus2017deep, kryscinski2018improving  The reason is that, RL and MLE have inconsistent optimization objectives  When the training headline is non-sensational, MLE training will encourage our model to imitate the training headline (thus generating non-sensational headlines), which counteracts the effects of RL training to generate sensational headlines 
The sensationalism score is, therefore, used to give dynamic weight to MLE and RL  Our ARL loss function becomes:
If $\alpha _{\text{sen}}(y^*)$ is high, meaning the training headline is sensational, our loss function encourages our model to imitate the sample more using the MLE training  If $\alpha _{\text{sen}}(y^*)$ is low, our loss function replies on RL training to improve the sensationalism  Note that the weight $\alpha _{\text{sen}}(y^*)$ is different from our sensationalism reward $\alpha _{\text{sen}}(y^s)$ and we call the loss function Auto-tuned Reinforcement Learning, because the ratio between MLE and RL are well “tuned” towards different samples 
Sensational Headline Generation ::: Dataset
We use LCSTS BIBREF5 as our dataset to train the summarization model  The dataset is collected from the Chinese microblogging website Sina Weibo  It contains over 2 million Chinese short texts with corresponding headlines given by the author of each text  The dataset is split into 2,400,591 samples for training, 10,666 samples for validation and 725 samples for testing  We tokenize each sentence with Jieba and a vocabulary size of 50000 is saved 
Sensational Headline Generation ::: Baselines and Our Models
We experiment and compare with the following models 
Pointer-Gen is the baseline model trained by optimizing $L_\text{MLE}$ in Equation DISPLAY_FORM13 
Pointer-Gen+Pos is the baseline model by training Pointer-Gen only on positive examples whose sensationalism score is larger than 0 5
Pointer-Gen+Same-FT is the model which fine-tunes Pointer-Gen on the training samples whose sensationalism score is larger than 0 1
Pointer-Gen+Pos-FT is the model which fine-tunes Pointer-Gen on the training samples whose sensationalism score is larger than 0 5
Pointer-Gen+RL-ROUGE is the baseline model trained by optimizing $L_\text{RL-ROUGE}$ in Equation DISPLAY_FORM17, with ROUGE-L BIBREF9 as the reward 
Pointer-Gen+RL-SEN is the baseline model trained by optimizing $L_\text{RL-SEN}$ in Equation DISPLAY_FORM17, with $\alpha _\text{sen}$ as the reward 
Pointer-Gen+ARL-SEN is our model trained by optimizing $L_\text{ARL-SEN}$ in Equation DISPLAY_FORM19, with $\alpha _\text{sen}$ as the reward 
Test set is the headlines from the test set 
Note that we didn't compare to Pointer-Gen+ARL-ROUGE as it is actually Pointer-GEN  Recall that $\alpha _{\text{sen}}(y^*)$ in Equation DISPLAY_FORM19 measures how good (based on reward function) is $y^*$  Then the loss function for Pointer-Gen+ARL-ROUGE will be
We also tried text style transfer baseline BIBREF10, but the generated headlines were very poor (many unknown words and irrelevant) 
Sensational Headline Generation ::: Training Details
MLE training: An Adam optimizer is used with the learning rate of 0 0001 to optimize $L_{\text{MLE}}$  The batch size is set as 128 and a one-layer, bi-directional Long Short-Term Memory (bi-LSTM) model with 512 hidden sizes and a 350 embedding size is utilized  Gradients with the l2 norm larger than 2

ground_truth: Women represent 33.16% of the speakers

prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. What were their results on the new dataset?
Context 2: Introduction
In the kitchen, we increasingly rely on instructions from cooking websites: recipes  A cook with a predilection for Asian cuisine may wish to prepare chicken curry, but may not know all necessary ingredients apart from a few basics  These users with limited knowledge cannot rely on existing recipe generation approaches that focus on creating coherent recipes given all ingredients and a recipe name BIBREF0  Such models do not address issues of personal preference (e g  culinary tastes, garnish choices) and incomplete recipe details  We propose to approach both problems via personalized generation of plausible, user-specific recipes using user preferences extracted from previously consumed recipes 
Our work combines two important tasks from natural language processing and recommender systems: data-to-text generation BIBREF1 and personalized recommendation BIBREF2  Our model takes as user input the name of a specific dish, a few key ingredients, and a calorie level  We pass these loose input specifications to an encoder-decoder framework and attend on user profiles—learned latent representations of recipes previously consumed by a user—to generate a recipe personalized to the user's tastes  We fuse these `user-aware' representations with decoder output in an attention fusion layer to jointly determine text generation  Quantitative (perplexity, user-ranking) and qualitative analysis on user-aware model outputs confirm that personalization indeed assists in generating plausible recipes from incomplete ingredients 
While personalized text generation has seen success in conveying user writing styles in the product review BIBREF3, BIBREF4 and dialogue BIBREF5 spaces, we are the first to consider it for the problem of recipe generation, where output quality is heavily dependent on the content of the instructions—such as ingredients and cooking techniques 
To summarize, our main contributions are as follows:
We explore a new task of generating plausible and personalized recipes from incomplete input specifications by leveraging historical user preferences;
We release a new dataset of 180K+ recipes and 700K+ user reviews for this task;
We introduce new evaluation strategies for generation quality in instructional texts, centering on quantitative measures of coherence  We also show qualitatively and quantitatively that personalized models generate high-quality and specific recipes that align with historical user preferences 
Related Work
Large-scale transformer-based language models have shown surprising expressivity and fluency in creative and conditional long-text generation BIBREF6, BIBREF7  Recent works have proposed hierarchical methods that condition on narrative frameworks to generate internally consistent long texts BIBREF8, BIBREF9, BIBREF10  Here, we generate procedurally structured recipes instead of free-form narratives 
Recipe generation belongs to the field of data-to-text natural language generation BIBREF1, which sees other applications in automated journalism BIBREF11, question-answering BIBREF12, and abstractive summarization BIBREF13, among others  BIBREF14, BIBREF15 model recipes as a structured collection of ingredient entities acted upon by cooking actions  BIBREF0 imposes a `checklist' attention constraint emphasizing hitherto unused ingredients during generation  BIBREF16 attend over explicit ingredient references in the prior recipe step  Similar hierarchical approaches that infer a full ingredient list to constrain generation will not help personalize recipes, and would be infeasible in our setting due to the potentially unconstrained number of ingredients (from a space of 10K+) in a recipe  We instead learn historical preferences to guide full recipe generation 
A recent line of work has explored user- and item-dependent aspect-aware review generation BIBREF3, BIBREF4  This work is related to ours in that it combines contextual language generation with personalization  Here, we attend over historical user preferences from previously consumed recipes to generate recipe content, rather than writing styles 
Approach
Our model's input specification consists of: the recipe name as a sequence of tokens, a partial list of ingredients, and a caloric level (high, medium, low)  It outputs the recipe instructions as a token sequence: $\mathcal {W}_r=\lbrace w_{r,0}, \dots , w_{r,T}\rbrace $ for a recipe $r$ of length $T$  To personalize output, we use historical recipe interactions of a user $u \in \mathcal {U}$ 
Encoder: Our encoder has three embedding layers: vocabulary embedding $\mathcal {V}$, ingredient embedding $\mathcal {I}$, and caloric-level embedding $\mathcal {C}$  Each token in the (length $L_n$) recipe name is embedded via $\mathcal {V}$; the embedded token sequence is passed to a two-layered bidirectional GRU (BiGRU) BIBREF17, which outputs hidden states for names $\lbrace \mathbf {n}_{\text{enc},j} \in \mathbb {R}^{2d_h}\rbrace $, with hidden size $d_h$  Similarly each of the $L_i$ input ingredients is embedded via $\mathcal {I}$, and the embedded ingredient sequence is passed to another two-layered BiGRU to output ingredient hidden states as $\lbrace \mathbf {i}_{\text{enc},j} \in \mathbb {R}^{2d_h}\rbrace $  The caloric level is embedded via $\mathcal {C}$ and passed through a projection layer with weights $W_c$ to generate calorie hidden representation $\mathbf {c}_{\text{enc}} \in \mathbb {R}^{2d_h}$ 
Ingredient Attention: We apply attention BIBREF18 over the encoded ingredients to use encoder outputs at each decoding time step  We define an attention-score function $\alpha $ with key $K$ and query $Q$:
with trainable weights $W_{\alpha }$, bias $\mathbf {b}_{\alpha }$, and normalization term $Z$  At decoding time $t$, we calculate the ingredient context $\mathbf {a}_{t}^{i} \in \mathbb {R}^{d_h}$ as:
Decoder: The decoder is a two-layer GRU with hidden state $h_t$ conditioned on previous hidden state $h_{t-1}$ and input token $w_{r, t}$ from the original recipe text  We project the concatenated encoder outputs as the initial decoder hidden state:
To bias generation toward user preferences, we attend over a user's previously reviewed recipes to jointly determine the final output token distribution  We consider two different schemes to model preferences from user histories: (1) recipe interactions, and (2) techniques seen therein (defined in data)  BIBREF19, BIBREF20, BIBREF21 explore similar schemes for personalized recommendation 
Prior Recipe Attention: We obtain the set of prior recipes for a user $u$: $R^+_u$, where each recipe can be represented by an embedding from a recipe embedding layer $\mathcal {R}$ or an average of the name tokens embedded by $\mathcal {V}$  We attend over the $k$-most recent prior recipes, $R^{k+}_u$, to account for temporal drift of user preferences BIBREF22  These embeddings are used in the `Prior Recipe' and `Prior Name' models, respectively 
Given a recipe representation $\mathbf {r} \in \mathbb {R}^{d_r}$ (where $d_r$ is recipe- or vocabulary-embedding size depending on the recipe representation) the prior recipe attention context $\mathbf {a}_{t}^{r_u}$ is calculated as
Prior Technique Attention: We calculate prior technique preference (used in the `Prior Tech` model) by normalizing co-occurrence between users and techniques seen in $R^+_u$, to obtain a preference vector $\rho _{u}$  Each technique $x$ is embedded via a technique embedding layer $\mathcal {X}$ to $\mathbf {x}\in \mathbb {R}^{d_x}$  Prior technique attention is calculated as
where, inspired by copy mechanisms BIBREF23, BIBREF24, we add $\rho _{u,x}$ for technique $x$ to emphasize the attention by the user's prior technique preference 
Attention Fusion Layer: We fuse all contexts calculated at time $t$, concatenating them with decoder GRU output and previous token embedding:
We then calculate the token probability:
and maximize the log-likelihood of the generated sequence conditioned on input specifications and user preferences  fig:ex shows a case where the Prior Name model attends strongly on previously consumed savory recipes to suggest the usage of an additional ingredient (`cilantro') 
Recipe Dataset: Food com
We collect a novel dataset of 230K+ recipe texts and 1M+ user interactions (reviews) over 18 years (2000-2018) from Food com  Here, we restrict to recipes with at least 3 steps, and at least 4 and no more than 20 ingredients  We discard users with fewer than 4 reviews, giving 180K+ recipes and 700K+ reviews, with splits as in tab:recipeixnstats 
Our model must learn to generate from a diverse recipe space: in our training data, the average recipe length is 117 tokens with a maximum of 256  There are 13K unique ingredients across all recipes  Rare words dominate the vocabulary: 95% of words appear $<$100 times, accounting for only 1 65% of all word usage  As such, we perform Byte-Pair Encoding (BPE) tokenization BIBREF25, BIBREF26, giving a training vocabulary of 15K tokens across 19M total mentions  User profiles are similarly diverse: 50% of users have consumed $\le $6 recipes, while 10% of users have consumed $>$45 recipes 
We order reviews by timestamp, keeping the most recent review for each user as the test set, the second most recent for validation, and the remainder for training (sequential leave-one-out evaluation BIBREF27)  We evaluate only on recipes not in the training set 
We manually construct a list of 58 cooking techniques from 384 cooking actions collected by BIBREF15; the most common techniques (bake, combine, pour, boil) account for 36 5% of technique mentions  We approximate technique adherence via string match between the recipe text and technique list 
Experiments and Results
For training and evaluation, we provide our model with the first 3-5 ingredients listed in each recipe  We decode recipe text via top-$k$ sampling BIBREF7, finding $k=3$ to produce satisfactory results  We use a hidden size $d_h=256$ for both the encoder and decoder  Embedding dimensions for vocabulary, ingredient, recipe, techniques, and caloric level are 300, 10, 50, 50, and 5 (respectively)  For prior recipe attention, we set $k=20$, the 80th %-ile for the number of user interactions  We use the Adam optimizer BIBREF28 with a learning rate of $10^{-3}$, annealed with a decay rate of 0 9 BIBREF29  We also use teacher-forcing BIBREF30 in all training epochs 
In this work, we investigate how leveraging historical user preferences can improve generation quality over strong baselines in our setting  We compare our personalized models against two baselines  The first is a name-based Nearest-Neighbor model (NN)  We initially adapted the Neural Checklist Model of BIBREF0 as a baseline; however, we ultimately use a simple Encoder-Decoder baseline with ingredient attention (Enc-Dec), which provides comparable performance and lower complexity  All personalized models outperform baseline in BPE perplexity (tab:metricsontest) with Prior Name performing the best  While our models exhibit comparable performance to baseline in BLEU-1/4 and ROUGE-L, we generate more diverse (Distinct-1/2: percentage of distinct unigrams and bigrams) and acceptable recipes  BLEU and ROUGE are not the most appropriate metrics for generation quality  A `correct' recipe can be written in many ways with the same main entities (ingredients)  As BLEU-1/4 capture structural information via n-gram matching, they are not correlated with subjective recipe quality  This mirrors observations from BIBREF31, BIBREF8 
We observe that personalized models make more diverse recipes than baseline  They thus perform better in BLEU-1 with more key entities (ingredient mentions) present, but worse in BLEU-4, as these recipes are written in a personalized way and deviate from gold on the phrasal level  Similarly, the `Prior Name' model generates more unigram-diverse recipes than other personalized models and obtains a correspondingly lower BLEU-1 score 
Qualitative Analysis: We present sample outputs for a cocktail recipe in tab:samplerecipes, and additional recipes in the appendix  Generation quality progressively improves from generic baseline output to a blended cocktail produced by our best performing model  Models attending over prior recipes explicitly reference ingredients  The Prior Name model further suggests the addition of lemon and mint, which are reasonably associated with previously consumed recipes like coconut mousse and pork skewers 
Personalization: To measure personalization, we evaluate how closely the generated text corresponds to a particular user profile  We compute the likelihood of generated recipes using identical input specifications but conditioned on ten different user profiles—one `gold' user who consumed the original recipe, and nine randomly generated user profiles  Following BIBREF8, we expect the highest likelihood for the recipe conditioned on the gold user  We measure user matching accuracy (UMA)—the proportion where the gold user is ranked highest—and Mean Reciprocal Rank (MRR) BIBREF32 of the gold user  All personalized models beat baselines in both metrics, showing our models personalize generated recipes to the given user profiles  The Prior Name model achieves the best UMA and MRR by a large margin, revealing that prior recipe names are strong signals for personalization  Moreover, the addition of attention mechanisms to capture these signals improves language modeling performance over a strong non-personalized baseline 
Recipe Level Coherence: A plausible recipe should possess a coherent step order, and we evaluate this via a metric for recipe-level coherence  We use the neural scoring model from BIBREF33 to measure recipe-level coherence for each generated recipe  Each recipe step is encoded by BERT BIBREF34  Our scoring model is a GRU network that learns the overall recipe step ordering structure by minimizing the cosine similarity of recipe step hidden representations presented in the correct and reverse orders  Once pretrained, our scorer calculates the similarity of a generated recipe to the forward and backwards ordering of its corresponding gold label, giving a score equal to the difference between the former and latter  A higher score indicates better step ordering (with a maximum score of 2)  tab:coherencemetrics shows that our personalized models achieve average recipe-level coherence scores of 1 78-1 82, surpassing the baseline at 1 77 
Recipe Step Entailment: Local coherence is also crucial to a user following a recipe: it is crucial that subsequent steps are logically consistent with prior ones  We model local coherence as an entailment task: predicting the likelihood that a recipe step follows the preceding  We sample several consecutive (positive) and non-consecutive (negative) pairs of steps from each recipe  We train a BERT BIBREF34 model to predict the entailment score of a pair of steps separated by a [SEP] token, using the final representation of the [CLS] token  The step entailment score is computed as the average of scores for each set of consecutive steps in each recipe, averaged over every generated recipe for a model, as shown in tab:coherencemetrics 
Human Evaluation: We presented 310 pairs of recipes for pairwise comparison BIBREF8 (details in appendix) between baseline and each personalized model, with results shown in tab:metricsontest  On average, human evaluators preferred personalized model outputs to baseline 63% of the time, confirming that personalized attention improves the semantic plausibility of generated recipes  We also performed a small-scale human coherence survey over 90 recipes, in which 60% of users found recipes generated by personalized models to be more coherent and preferable to those generated by baseline models 
Conclusion
In this paper, we propose a novel task: to generate personalized recipes from incomplete input specifications and user histories  On a large novel dataset of 180K recipes and 700K reviews, we show that our personalized generative models can generate plausible, personalized, and coherent recipes preferred by human evaluators for consumption  We also introduce a set of automatic coherence measures for instructional texts as well as personalization metrics to support our claims  Our future work includes generating structured representations of recipes to handle ingredient properties, as well as accounting for references to collections of ingredients (e g  “dry mix") 
Acknowledgements  This work is partly supported by NSF #1750063  We thank all reviewers for their constructive suggestions, as well as Rei M , Sujoy P , Alicia L , Eric H , Tim S , Kathy C , Allen C , and Micah I  for their feedback 
Appendix ::: Food com: Dataset Details
Our raw data consists of 270K recipes and 1 4M user-recipe interactions (reviews) scraped from Food com, covering a period of 18 years (January 2000 to December 2018)  See tab:int-stats for dataset summary statistics, and tab:samplegk for sample information about one user-recipe interaction and the recipe involved 
Appendix ::: Generated Examples
See tab:samplechx for a sample recipe for chicken chili and tab:samplewaffle for a sample recipe for sweet waffles 
Human Evaluation
We prepared a set of 15 pairwise comparisons per evaluation session, and collected 930 pairwise evaluations (310 per personalized model) over 62 sessions  For each pair, users were given a partial recipe specification (name and 3-5 key ingredients), as well as two generated recipes labeled `A' and `B'  One recipe is generated from our baseline encoder-decoder model and one recipe is generated by one of our three personalized models (Prior Tech, Prior Name, Prior Recipe)  The order of recipe presentation (A/B) is randomly selected for each question  A screenshot of the user evaluation interface is given in fig:exeval  We ask the user to indicate which recipe they find more coherent, and which recipe best accomplishes the goal indicated by the recipe name  A screenshot of this survey interface is given in fig:exeval2 
Context 3: 0 are clipped  We stop training when the ROUGE-L f-score stops increasing 
Hybrid training: An Adam optimizer with a learning rate of 0 0001 is used to optimize $L_{\text{RL-*}}$ (Equation DISPLAY_FORM17) and $L_\text{{ARL-SEN}}$ (Equation DISPLAY_FORM19)  When training Pointer-Gen+RL-ROUGE, the best $\lambda $ is chosen based on the ROUGE-L score on the validation set  In our experiment, $\lambda $ is set as 0 95  An Adam optimizer with a learning rate of 0 001 is used to optimize $L_b$  When training Pointer-Gen+ARL-SEN, we don't use the full LCSTS dataset, but only headlines with a sensationalism score larger than 0 1 as we observe that Pointer-Gen+ARL-SEN will generate a few unnatural phrases when using full dataset  We believe the reason is the high ratio of RL during training  Figure FIGREF23 shows that the probability density near 0 is very high, meaning that in each batch, many of the samples will have a very low sensationalism score  On expectation, each sample will receive 0 239 MLE training and 0 761 RL training  This leads to RL dominanting the loss  Thus, we propose to filter samples with a minimum sensationalism score with 0 1 and it works very well  For Pointer-Gen+RL-SEN, we also set the minimum sensationalism score as 0 1, and $\lambda $ is set as 0 5 to remove unnatural phrases, making a fair comparison to Pointer-Gen+ARL-SEN 
We stop training Pointer-Gen+Same-FT, Pointer-Gen+Pos-FT, Pointer-Gen+RL-SEN and Pointer-Gen+ARL-SEN, when $\alpha _\text{sen}$ stops increasing on the validation set  Beam-search with a beam size of 5 is adopted for decoding in all models 
Sensational Headline Generation ::: Evaluation Metrics
We briefly describe the evaluation metrics below 
ROUGE: ROUGE is a commonly used evaluation metric for summarization  It measures the N-gram overlap between generated and training headlines  We use it to evaluate the relevance of generated headlines  The widely used pyrouge toolkit is used to calculate ROUGE-1 (RG-1), ROUGE-2 (RG-2), and ROUGE-L (RG-L) 
Human evaluation: We randomly sample 50 articles from the test set and send the generated headlines from all models and corresponding headlines in the test set to human annotators  We evaluate the sensationalism and fluency of the headlines by setting up two independent human annotation tasks  We ask 10 annotators to label each headline for each task  For the sensationalism annotation, each annotator is asked one question, “Is the headline sensational?”, and he/she has to choose either `yes' or `no'  The annotators were not told which system the headline is from  The process of distributing samples and recruiting annotators is managed by Crowdflower  After annotation, we define the sensationalism score as the proportion of annotations on all generated headlines from one model labeled as `yes'  For the fluency annotation, we repeat the same procedure as for the sensationalism annotation, except that we ask each annotator the question “Is the headline fluent?” We define the fluency score as the proportion of annotations on all headlines from one specific model labeled as `yes'  We put human annotation instructions in the supplemental material 
UTF8gbsn
Results
We first compare all four models, Pointer-Gen, Pointer-Gen-RL+ROUGE, Pointer-Gen-RL-SEN, and Pointer-Gen-ARL-SEN, to existing models with ROUGE in Table TABREF25 to establish that our model produces relevant headlines and we leave the sensationalism for human evaluation  Note that we only compare our models to commonly used strong summarization baselines, to validate that our implementation achieves comparable performance to existing work  In our implementation, Pointer-Gen achieves a 34 51 RG-1 score, 22 21 RG-2 score, and 31 68 RG-L score, which is similar to the results of gu2016incorporating  Pointer-Gen+ARL-SEN, although optimized for the sensationalism reward, achieves similar performance to our Pointer-Gen baseline, which means that Pointer-Gen+ARL-SEN still keeps its summarization ability  An example of headlines generated from different models in Table TABREF29 shows that Pointer-Gen and Pointer-Gen+RL-ROUGE learns to summarize the main point of the article: “The Nikon D600 camera is reported to have black spots when taking photos”  Pointer-Gen+RL-SEN makes the headline more sensational by blaming Nikon for attributing the damage to the smog  Pointer-Gen+ARL-SEN generates the most sensational headline by exaggerating the result “Getting a serious trouble!” to maximize user's attention 
We then compare different models using the sensationalism score in Table TABREF30  The Pointer-Gen baseline model achieves a 42 6% sensationalism score, which is the minimum that a typical summarization model achieves  By filtering out low-sensational headlines, Pointer-Gen+Same-FT and Pointer-Gen+Pos-FT achieves higher sensationalism scores, which implies the effectiveness of our sensationalism scorer  Our Pointer-Gen+ARL-SEN model achieves the best performance of 60 8%  This is an absolute improvement of 18 2% over the Pointer-Gen baseline  The Chi-square test on the results confirms that Pointer-Gen+ARL-SEN is statistically significantly more sensational than all the other baseline models, with the largest p-value less than 0 01  Also, we find that the test set headlines achieves 57 8% sensationalism score, much larger than Pointer-Gen baseline, which also supports our intuition that generated headlines will be less sensational than the original one  On the other hand, we found that Pointer-Gen+Pos is much worse than other baselines  The reason is that training on sensational samples alone discards around 80% of the whole training set that is also helpful for maintaining relevance and a good language model  It shows the necessity of using RL 
UTF8gbsn
In addition, both Pointer-Gen+RL-SEN and Pointer-Gen+ARL-SEN, which use the sensationalism score as the reward, obtain statistically better results than Pointer-Gen+RL-ROUGE and Pointer-Gen, with a p-value less than 0 05 by a Chi-square test  This result shows the effectiveness of RL to generate more sensational headlines  The reason is that even though our noisy classifier could also learn to classify domains, the generator during RL training is not allowed to increase the reward by shifting domains but encouraged to generate more sensational headlines, due to the consistency constraint on the domains of the headline and the article  Furthermore, Poiner-Gen+ARL-SEN gets better performance than Pointer-Gen+RL-SEN, which confirms the superiority of the ARL loss function  We also visualize in Figure FIGREF31 a comparison between Pointer-Gen+ARL-SEN and Pointer-Gen+RL-SEN according to how sensational the test set headlines are  The blue bars denote the smaller scores between the two models  For example, if the blue bar is 0 6, it means that the worse model between Pointer-Gen+RL-SEN and Pointer-Gen+ARL-SEN achieves 0 6  And the color of orange/black further indicates the better model and its score  We find that Pointer-Gen+ARL-SEN outperforms Pointer-Gen+RL-SEN for most cases  The improvement is higher when the test set headlines are not sensational (the sensationalism score is less than 0 5), which may be attributed to the higher ratio of RL training on non-sensational headlines 
Apart from the sensationalism evaluation, we measure the fluency of the headlines generated from different models  Fluency scores in Table TABREF30 show that Pointer-Gen+RL-SEN and Pointer-Gen+ARL-SEN achieve comparable fluency performance to Pointer-Gen and Pointer-Gen+RL-ROUGE  Test set headlines achieve the best performance among all models, but the difference is not statistically significant  Also, we observe that fine-tuning on sensational headlines will hurt the performance, both in sensationalism and fluency 
After manually checking the outputs, we observe that our model is able to generate sensational headlines using diverse sensationalization strategies  These strategies include, but are not limited to, creating a curiosity gap, asking questions, highlighting numbers, being emotional and emphasizing the user  Examples can be found in Table TABREF32 
Related Work
Our work is related to summarization tasks  An encoder-decoder model was first applied to two sentence-level abstractive summarization tasks on the DUC-2004 and Gigaword datasets BIBREF12  This model was later extended by selective encoding BIBREF13, a coarse to fine approach BIBREF14, minimum risk training BIBREF1, and topic-aware models BIBREF15  As long summaries were recognized as important, the CNN/Daily Mail dataset was used in nallapati2016abstractive  Graph-based attention BIBREF16, pointer-generator with coverage loss BIBREF0 are further developed to improve the generated summaries  celikyilmaz2018deep proposed deep communicating agents for representing a long document for abstractive summarization  In addition, many papers BIBREF17, BIBREF18, BIBREF19 use extractive methods to directly select sentences from articles  However, none of these work considered the sensationalism of generated outputs 
RL is also gaining popularity as it can directly optimize non-differentiable metrics BIBREF20, BIBREF21, BIBREF22  paulus2017deep proposed an intra-decoder model and combined RL and MLE to deal with summaries with bad qualities  RL has also been explored with generative adversarial networks (GANs) BIBREF23  liu2017generative applied GANs on summarization task and achieved better performance  niu2018polite tackles the problem of polite generation with politeness reward  Our work is different in that we propose a novel function to balance RL and MLE 
Our task is also related to text style transfer  Implicit methods BIBREF10, BIBREF24, BIBREF4 transfer the styles by separating sentence representations into content and style, for example using back-translationBIBREF4  However, these methods cannot guarantee the content consistency between the original sentence and transferred output BIBREF25  Explicit methods BIBREF26, BIBREF25 transfer the style by directly identifying style related keywords and modifying them  However, sensationalism is not always restricted to keywords, but the full sentence  By leveraging small human labeled English dataset, clickbait detection has been well investigated BIBREF2, BIBREF27, BIBREF3  However, these human labeled dataset are not available for other languages, such as Chinese 
Modeling sensationalism is also related to modeling emotion  Emotion has been well investigated in both word levelBIBREF28, BIBREF29 and sentence levelBIBREF30, BIBREF31, BIBREF32, BIBREF33, BIBREF34  It has also been considered an important factor in engaging interactive systemsBIBREF35, BIBREF36, BIBREF37  Although we observe that sensational headlines contain emotion, it is still not clear which emotion and how emotions will influence the sensationalism 
Conclusion and Future Work
In this paper, we propose a model that generates sensational headlines without labeled data using Reinforcement Learning  Firstly, we propose a distant supervision strategy to train the sensationalism scorer  As a result, we achieve 65% accuracy between the predicted sensationalism score and human evaluation  To effectively leverage this noisy sensationalism score as the reward for RL, we propose a novel loss function, ARL, to automatically balance RL with MLE  Human evaluation confirms the effectiveness of both our sensationalism scorer and ARL to generate more sensational headlines  Future work can be improving the sensationalism scorer and investigating the applications of dynamic balancing methods between RL and MLE in textGANBIBREF23  Our work also raises the ethical questions about generating sensational headlines, which can be further explored 
Acknowledgments
Thanks to ITS/319/16FP of Innovation Technology Commission, HKUST 16248016 of Hong Kong Research Grants Council for funding  In addition, we thank Zhaojiang Lin for helpful discussion and Yan Xu, Zihan Liu for the data collection 

ground_truth: average recipe-level coherence scores of 1.78-1.82, human evaluators preferred personalized model outputs to baseline 63% of the time

prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. Which baselines are used for evaluation?
Context 4: 5 after embedding layer, the output between LSTM layers, and the output of final LSTM layer  (2) Advanced dropout: We use the same dropout techniques with the same dropout values as in BIBREF0   We call this model as AWD-PRU 
Results
Table TABREF23 compares the performance of the PRU with state-of-the-art methods  We can see that the PRU achieves the best performance with fewer parameters 
PRUs achieve either the same or better performance than LSTMs  In particular, the performance of PRUs improves with the increasing value of INLINEFORM0   At INLINEFORM1 , PRUs outperform LSTMs by about 4 points on the PTB dataset and by about 3 points on the WT-2 dataset  This is explained in part by the regularization effect of the grouped linear transformation (Figure FIGREF1 )  With grouped linear and pyramidal transformations, PRUs learn rich representations at very high dimensional space while learning fewer parameters  On the other hand, LSTMs overfit to the training data at such high dimensions and learn INLINEFORM2 to INLINEFORM3 more parameters than PRUs 
With the advanced dropouts, the performance of PRUs improves by about 4 points on the PTB dataset and 7 points on the WT-2 dataset  This further improves with finetuning on the PTB (about 2 points) and WT-2 (about 1 point) datasets 
For similar number of parameters, the PRU with standard dropout outperforms most of the state-of-the-art methods by large margin on the PTB dataset (e g  RAN BIBREF7 by 16 points with 4M less parameters, QRNN BIBREF33 by 16 points with 1M more parameters, and NAS BIBREF31 by 1 58 points with 6M less parameters)  With advanced dropouts, the PRU delivers the best performance  On both datasets, the PRU improves the perplexity by about 1 point while learning 15-20% fewer parameters 
PRU is a drop-in replacement for LSTM, therefore, it can improve language models with modern inference techniques such as dynamic evaluation BIBREF21   When we evaluate PRU-based language models (only with standard dropout) with dynamic evaluation on the PTB test set, the perplexity of PRU ( INLINEFORM0 ) improves from 62 42 to 55 23 while the perplexity of an LSTM ( INLINEFORM1 ) with similar settings improves from 66 29 to 58 79; suggesting that modern inference techniques are equally applicable to PRU-based language models 
Analysis
It is shown above that the PRU can learn representations at higher dimensionality with more generalization power, resulting in performance gains for language modeling  A closer analysis of the impact of the PRU in a language modeling system reveals several factors that help explain how the PRU achieves these gains 
As exemplified in Table TABREF34 , the PRU tends toward more confident decisions, placing more of the probability mass on the top next-word prediction than the LSTM  To quantify this effect, we calculate the entropy of the next-token distribution for both the PRU and the LSTM using 3687 contexts from the PTB validation set  Figure FIGREF32 shows a histogram of the entropies of the distribution, where bins of size 0 23 are used to effect categories  We see that the PRU more often produces lower entropy distributions corresponding to higher confidences for next-token choices  This is evidenced by the mass of the red PRU curve lying in the lower entropy ranges compared to the blue LSTM's curve  The PRU can produce confident decisions in part because more information is encoded in the higher dimensional context vectors 
The PRU has the ability to model individual words at different resolutions through the pyramidal transform; which provides multiple paths for the gradient to the embedding layer (similar to multi-task learning) and improves the flow of information  When considering the embeddings by part of speech, we find that the pyramid level 1 embeddings exhibit higher variance than the LSTM across all POS categories (Figure FIGREF33 ), and that pyramid level 2 embeddings show extremely low variance  We hypothesize that the LSTM must encode both coarse group similarities and individual word differences into the same vector space, reducing the space between individual words of the same category  The PRU can rely on the subsampled embeddings to account for coarse-grained group similarities, allowing for finer individual word distinctions in the embedding layer  This hypothesis is strengthened by the entropy results described above: a model which can make finer distinctions between individual words can more confidently assign probability mass  A model that cannot make these distinctions, such as the LSTM, must spread its probability mass across a larger class of similar words 
Saliency analysis using gradients help identify relevant words in a test sequence that contribute to the prediction BIBREF34 , BIBREF35 , BIBREF36   These approaches compute the relevance as the squared norm of the gradients obtained through back-propagation  Table TABREF34 visualizes the heatmaps for different sequences  PRUs, in general, give more relevance to contextual words than LSTMs, such as southeast (sample 1), cost (sample 2), face (sample 4), and introduced (sample 5), which help in making more confident decisions  Furthermore, when gradients during back-propagation are visualized BIBREF37 (Table TABREF34 ), we find that PRUs have better gradient coverage than LSTMs, suggesting PRUs use more features than LSTMs that contributes to the decision  This also suggests that PRUs update more parameters at each iteration which results in faster training  Language model in BIBREF0 takes 500 and 750 epochs to converge with PRU and LSTM as a recurrent unit, respectively 
Ablation studies
In this section, we provide a systematic analysis of our design choices  Our training methodology is the same as described in Section SECREF19 with the standard dropouts  For a thorough understanding of our design choices, we use a language model with a single layer of PRU and fix the size of embedding and hidden layers to 600  The word-level perplexities are reported on the validation sets of the PTB and the WT-2 datasets 
The two hyper-parameters that control the trade-off between performance and number of parameters in PRUs are the number of pyramidal levels INLINEFORM0 and groups INLINEFORM1   Figure FIGREF35 provides a trade-off between perplexity and recurrent unit (RU) parameters 
Variable INLINEFORM0 and fixed INLINEFORM1 : When we increase the number of pyramidal levels INLINEFORM2 at a fixed value of INLINEFORM3 , the performance of the PRU drops by about 1 to 4 points while reducing the total number of recurrent unit parameters by up to 15%  We note that the PRU with INLINEFORM4 at INLINEFORM5 delivers similar performance as the LSTM while learning about 15% fewer recurrent unit parameters 
Fixed INLINEFORM0 and variable INLINEFORM1 : When we vary the value of INLINEFORM2 at fixed number of pyramidal levels INLINEFORM3 , the total number of recurrent unit parameters decreases significantly with a minimal impact on the perplexity  For example, PRUs with INLINEFORM4 and INLINEFORM5 learns 77% fewer recurrent unit parameters while its perplexity (lower is better) increases by about 12% in comparison to LSTMs  Moreover, the decrease in number of parameters at higher value of INLINEFORM6 enables PRUs to learn the representations in high dimensional space with better generalizability (Table TABREF23 ) 
Table TABREF43 shows the impact of different transformations of the input vector INLINEFORM0 and the context vector INLINEFORM1   We make following observations: (1) Using the pyramidal transformation for the input vectors improves the perplexity by about 1 point on both the PTB and WT-2 datasets while reducing the number of recurrent unit parameters by about 14% (see R1 and R4)  We note that the performance of the PRU drops by up to 1 point when residual connections are not used (R4 and R6)  (2) Using the grouped linear transformation for context vectors reduces the total number of recurrent unit parameters by about 75% while the performance drops by about 11% (see R3 and R4)  When we use the pyramidal transformation instead of the linear transformation, the performance drops by up to 2% while there is no significant drop in the number of parameters (R4 and R5) 
We set sub-sampling kernel INLINEFORM0 (Eq  EQREF12 ) with stride INLINEFORM1 and size of 3 ( INLINEFORM2 ) in four different ways: (1) Skip: We skip every other element in the input vector  (2) Convolution: We initialize the elements of INLINEFORM3 randomly from normal distribution and learn them during training the model  We limit the output values between -1 and 1 using INLINEFORM4 activation function to make training stable  (3) Avg  pool: We initialize the elements of INLINEFORM5 to INLINEFORM6   (4) Max pool: We select the maximum value in the kernel window INLINEFORM7  
Table TABREF45 compares the performance of the PRU with different sampling methods  Average pooling performs the best while skipping give comparable performance  Both of these methods enable the network to learn richer word representations while representing the input vector in different forms, thus delivering higher performance  Surprisingly, a convolution-based sub-sampling method does not perform as well as the averaging method  The INLINEFORM0 function used after convolution limits the range of output values which are further limited by the LSTM gating structure, thereby impeding in the flow of information inside the cell  Max pooling forces the network to learn representations from high magnitude elements, thus distinguishing features between elements vanishes, resulting in poor performance 
Conclusion
We introduce the Pyramidal Recurrent Unit, which better model contextual information by admitting higher dimensional representations with good generalizability  When applied to the task of language modeling, PRUs improve perplexity across several settings, including recent state-of-the-art systems  Our analysis shows that the PRU improves the flow of gradient and expand the word embedding subspace, resulting in more confident decisions  Here we have shown improvements for language modeling  In future, we plan to study the performance of PRUs on different tasks, including machine translation and question answering  In addition, we will study the performance of the PRU on language modeling with more recent inference techniques, such as dynamic evaluation and mixture of softmax 
Acknowledgments
This research was supported by NSF (IIS 1616112, III 1703166), Allen Distinguished Investigator Award, and gifts from Allen Institute for AI, Google, Amazon, and Bloomberg  We are grateful to Aaron Jaech, Hannah Rashkin, Mandar Joshi, Aniruddha Kembhavi, and anonymous reviewers for their helpful comments 

ground_truth: Pointer-Gen, Pointer-Gen+Pos, Pointer-Gen+Same-FT, Pointer-Gen+Pos-FT, Pointer-Gen+RL-ROUGE, Pointer-Gen+RL-SEN