prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. Which baselines are used for evaluation?
Context 1: Introduction
Headline generation is the process of creating a headline-style sentence given an input article  The research community has been regarding the task of headline generation as a summarization task BIBREF1, ignoring the fundamental differences between headlines and summaries  While summaries aim to contain most of the important information from the articles, headlines do not necessarily need to  Instead, a good headline needs to capture people's attention and serve as an irresistible invitation for users to read through the article  For example, the headline “$2 Billion Worth of Free Media for Trump”, which gives only an intriguing hint, is considered better than the summarization style headline “Measuring Trump’s Media Dominance” , as the former gets almost three times the readers as the latter  Generating headlines with many clicks is especially important in this digital age, because many of the revenues of journalism come from online advertisements and getting more user clicks means being more competitive in the market  However, most existing websites naively generate sensational headlines using only keywords or templates  Instead, this paper aims to learn a model that generates sensational headlines based on an input article without labeled data 
To generate sensational headlines, there are two main challenges  Firstly, there is a lack of sensationalism scorer to measure how sensational a headline is  Some researchers have tried to manually label headlines as clickbait or non-clickbait BIBREF2, BIBREF3  However, these human-annotated datasets are usually small and expensive to collect  To capture a large variety of sensationalization patterns, we need a cheap and easy way to collect a large number of sensational headlines  Thus, we propose a distant supervision strategy to collect a sensationalism dataset  We regard headlines receiving lots of comments as sensational samples and the headlines generated by a summarization model as non-sensational samples  Experimental results show that by distinguishing these two types of headlines, we can partially teach the model a sense of being sensational 
Secondly, after training a sensationalism scorer on our sensationalism dataset, a natural way to generate sensational headlines is to maximize the sensationalism score using reinforcement learning (RL)  However, the following shows an example of a RL model maximizing the sensationalism score by generating a very unnatural sentence, while its sensationalism scorer gave a very high score of 0 99996: UTF8gbsn 十个可穿戴产品的设计原则这消息消息可惜说明 Ten design principles for wearable devices, this message message pity introduction  This happens because the sensationalism scorer can make mistakes and RL can generate unnatural phrases which fools our sensationalism scorer  Thus, how to effectively leverage RL with noisy rewards remains an open problem  To deal with the noisy reward, we introduce Auto-tuned Reinforcement Learning (ARL)  Our model automatically tunes the ratio between MLE and RL based on how sensational the training headline is  In this way, we effectively take advantage of RL with a noisy reward to generate headlines that are both sensational and fluent 
The major contributions of this paper are as follows: 1) To the best of our knowledge, we propose the first-ever model that tackles the sensational headline generation task with reinforcement learning techniques  2) Without human-annotated data, we propose a distant supervision strategy to train a sensationalism scorer as a reward function 3) We propose a novel loss function, Auto-tuned Reinforcement Learning, to give dynamic weights to balance between MLE and RL  Our code will be released  
Sensationalism Scorer
To evaluate the sensationalism intensity score $\alpha _{\text{sen}}$ of a headline, we collect a sensationalism dataset and then train a sensationalism scorer  For the sensationalism dataset collection, we choose headlines with many comments from popular online websites as positive samples  For the negative samples, we propose to use the generated headlines from a sentence summarization model  Intuitively, the summarization model, which is trained to preserve the semantic meaning, will lose the sensationalization ability and thus the generated negative samples will be less sensational than the original one, similar to the obfuscation of style after back-translation BIBREF4  For example, an original headline like UTF8gbsn“一趟挣10万？铁总增开申通、顺丰专列" (One trip to earn 100 thousand? China Railway opens new Shentong and Shunfeng special lines) will become UTF8gbsn“中铁总将增开京广两列快递专列" (China Railway opens two special lines for express) from the baseline model, which loses the sensational phrases of UTF8gbsn“一趟挣10万？" (One trip to earn 100 thousand?)   We then train the sensationalism scorer by classifying sensational and non-sensational headlines using a one-layer CNN with a binary cross entropy loss $L_{\text{sen}}$  Firstly, 1-D convolution is used to extract word features from the input embeddings of a headline  This is followed by a ReLU activation layer and a max-pooling layer along the time dimension  All features from different channels are concatenated together and projected to the sensationalism score by adding another fully connected layer with sigmoid activation  Binary cross entropy is used to compute the loss $L_{\text{sen}}$ 
Sensationalism Scorer ::: Training Details and Dataset
For the CNN model, we choose filter sizes of 1, 3, and 5 respectively  Adam is used to optimize $L_{sen}$ with a learning rate of 0 0001  We set the embedding size as 300 and initialize it from qiu2018revisiting trained on the Weibo corpus with word and character features  We fix the embeddings during training  For dataset collection, we utilize the headlines collected in qin2018automatic, lin2019learning from Tencent News, one of the most popular Chinese news websites, as the positive samples  We follow the same data split as the original paper  As some of the links are not available any more, we get 170,754 training samples and 4,511 validation samples  For the negative training samples collection, we randomly select generated headlines from a pointer generator BIBREF0 model trained on LCSTS dataset BIBREF5 and create a balanced training corpus which includes 351,508 training samples and 9,022 validation samples  To evaluate our trained classifier, we construct a test set by randomly sampling 100 headlines from the test split of LCSTS dataset and the labels are obtained by 11 human annotators  Annotations show that 52% headlines are labeled as positive and 48% headlines as negative by majority voting (The detail on the annotation can be found in Section SECREF26) 
Sensationalism Scorer ::: Results and Discussion
Our classifier achieves 0 65 accuracy and 0 65 averaged F1 score on the test set while a random classifier would only achieve 0 50 accuracy and 0 50 averaged F1 score  This confirms that the predicted sensationalism score can partially capture the sensationalism of headlines  On the other hand, a more natural choice is to take headlines with few comments as negative examples  Thus, we train another baseline classifier on a crawled balanced sensationalism corpus of 84k headlines where the positive headlines have at least 28 comments and the negative headlines have less than 5 comments  However, the results on the test set show that the baseline classifier gets 60% accuracy, which is worse than the proposed classifier (which achieves 65%)  The reason could be that the balanced sensationalism corpus are sampled from different distributions from the test set and it is hard for the trained model to generalize  Therefore, we choose the proposed one as our sensationalism scorer  Therefore, our next challenge is to show that how to leverage this noisy sensationalism reward to generate sensational headlines 
Sensational Headline Generation
Our sensational headline generation model takes an article as input and output a sensational headline  The model consists of a Pointer-Gen headline generator and is trained by ARL  The diagram of ARL can be found in Figure FIGREF6 
We denote the input article as $x=\lbrace x_1,x_2,x_3,\cdots ,x_M\rbrace $, and the corresponding headline as $y^*=\lbrace y_1^*,y_2^*,y_3^*,\cdots ,y_T^*\rbrace $, where $M$ is the number of tokens in an article and $T$ is the number of tokens in a headline 
Sensational Headline Generation ::: Pointer-Gen Headline Generator
We choose Pointer Generator (Pointer-Gen) BIBREF0, a widely used summarization model, as our headline generator for its ability to copy words from the input article  It takes a news article as input and generates a headline  Firstly, the tokens of each article, $\lbrace x_1,x_2,x_3,\cdots ,x_M\rbrace $, are fed into the encoder one-by-one and the encoder generates a sequence of hidden states $h_i$  For each decoding step $t$, the decoder receives the embedding for each token of a headline $y_t$ as input and updates its hidden states $s_t$  An attention mechanism following luong2015effective is used:
where $v$, $W_h$, $W_s$, and $b_{attn}$ are the trainable parameters and $h_t^*$ is the context vector  $s_t$ and $h_t^*$ are then combined to give a probability distribution over the vocabulary through two linear layers:
where $V$, $b$, $V^{^{\prime }}$, and $b^{^{\prime }}$ are trainable parameters  We use a pointer generator network to enable our model to copy rare/unknown words from the input article, giving the following final word probability:
where $x^t$ is the embedding of the input word of the decoder, $w_{h^*}^T$, $w_s^T$, $w_x^T$, and $b_{ptr}$ are trainable parameters, and $\sigma $ is the sigmoid function 
Sensational Headline Generation ::: Training Methods
We first briefly introduce MLE and RL objective functions, and a naive way to mix these two by a hyper-parameter $\lambda $  Then we point out the challenge of training with noisy reward, and propose ARL to address this issue 
Sensational Headline Generation ::: Training Methods ::: MLE and RL
A headline generation model can be trained with MLE, RL or a combination of MLE and RL  MLE training is to minimize the negative log likelihood of the training headlines  We feed $y^*$ into the decoder word by word and maximize the likelihood of $y^*$  The loss function for MLE becomes
For RL training, we choose the REINFORCE algorithm BIBREF6  In the training phase, after encoding an article, a headline $y^s = \lbrace y_1^s, y_2^s, y_3^s, \cdots , y_T^s\rbrace $ is obtained by sampling from $P(w)$ from our generator, and then a reward of sensationalism or ROUGE(RG) is calculated 
We use the baseline reward $\hat{R_t}$ to reduce the variance of the reward, similar to ranzato2015sequence  To elaborate, a linear model is deployed to estimate the baseline reward $\hat{R_t}$ based on $t$-th state $o_t$ for each timestep $t$  The parameters of the linear model are trained by minimizing the mean square loss between $R$ and $\hat{R_t}$:
where $W_r$ and $b_r$ are trainable parameters  To maximize the expected reward, our loss function for RL becomes
A naive way to mix these two objective functions using a hyper-parameter $\lambda $ has been successfully incorporated in the summarization task BIBREF7  It includes the MLE training as a language model to mitigate the readability and quality issues in RL  The mixed loss function is shown as follows:
where $*$ is the reward type  Usually $\lambda $ is large, and paulus2017deep used 0 9984 
Sensational Headline Generation ::: Training Methods ::: Auto-tuned Reinforcement Learning
Applying the naive mixed training method using sensationalism score as the reward is not obvious/trivial in our task  The main reason is that our sensationalism reward is notably more noisy and more fragile than the ROUGE-L reward or abstractive reward used in the summarization task BIBREF7, BIBREF8  A higher ROUGE-L F1 reward in summarization indicates higher overlapping ratio between generation and true summary statistically, but our sensationalism reward is a learned score which is fragile to be fooled with unnatural samples 
To effectively train the model with RL under noisy sensationalism reward, our idea is to balance RL with MLE  However, we argue that the weighted ratio between MLE and RL should be sample-dependent, instead of being fixed for all training samples as in paulus2017deep, kryscinski2018improving  The reason is that, RL and MLE have inconsistent optimization objectives  When the training headline is non-sensational, MLE training will encourage our model to imitate the training headline (thus generating non-sensational headlines), which counteracts the effects of RL training to generate sensational headlines 
The sensationalism score is, therefore, used to give dynamic weight to MLE and RL  Our ARL loss function becomes:
If $\alpha _{\text{sen}}(y^*)$ is high, meaning the training headline is sensational, our loss function encourages our model to imitate the sample more using the MLE training  If $\alpha _{\text{sen}}(y^*)$ is low, our loss function replies on RL training to improve the sensationalism  Note that the weight $\alpha _{\text{sen}}(y^*)$ is different from our sensationalism reward $\alpha _{\text{sen}}(y^s)$ and we call the loss function Auto-tuned Reinforcement Learning, because the ratio between MLE and RL are well “tuned” towards different samples 
Sensational Headline Generation ::: Dataset
We use LCSTS BIBREF5 as our dataset to train the summarization model  The dataset is collected from the Chinese microblogging website Sina Weibo  It contains over 2 million Chinese short texts with corresponding headlines given by the author of each text  The dataset is split into 2,400,591 samples for training, 10,666 samples for validation and 725 samples for testing  We tokenize each sentence with Jieba and a vocabulary size of 50000 is saved 
Sensational Headline Generation ::: Baselines and Our Models
We experiment and compare with the following models 
Pointer-Gen is the baseline model trained by optimizing $L_\text{MLE}$ in Equation DISPLAY_FORM13 
Pointer-Gen+Pos is the baseline model by training Pointer-Gen only on positive examples whose sensationalism score is larger than 0 5
Pointer-Gen+Same-FT is the model which fine-tunes Pointer-Gen on the training samples whose sensationalism score is larger than 0 1
Pointer-Gen+Pos-FT is the model which fine-tunes Pointer-Gen on the training samples whose sensationalism score is larger than 0 5
Pointer-Gen+RL-ROUGE is the baseline model trained by optimizing $L_\text{RL-ROUGE}$ in Equation DISPLAY_FORM17, with ROUGE-L BIBREF9 as the reward 
Pointer-Gen+RL-SEN is the baseline model trained by optimizing $L_\text{RL-SEN}$ in Equation DISPLAY_FORM17, with $\alpha _\text{sen}$ as the reward 
Pointer-Gen+ARL-SEN is our model trained by optimizing $L_\text{ARL-SEN}$ in Equation DISPLAY_FORM19, with $\alpha _\text{sen}$ as the reward 
Test set is the headlines from the test set 
Note that we didn't compare to Pointer-Gen+ARL-ROUGE as it is actually Pointer-GEN  Recall that $\alpha _{\text{sen}}(y^*)$ in Equation DISPLAY_FORM19 measures how good (based on reward function) is $y^*$  Then the loss function for Pointer-Gen+ARL-ROUGE will be
We also tried text style transfer baseline BIBREF10, but the generated headlines were very poor (many unknown words and irrelevant) 
Sensational Headline Generation ::: Training Details
MLE training: An Adam optimizer is used with the learning rate of 0 0001 to optimize $L_{\text{MLE}}$  The batch size is set as 128 and a one-layer, bi-directional Long Short-Term Memory (bi-LSTM) model with 512 hidden sizes and a 350 embedding size is utilized  Gradients with the l2 norm larger than 2 0 are clipped  We stop training when the ROUGE-L f-score stops increasing 
Hybrid training: An Adam optimizer with a learning rate of 0 0001 is used to optimize $L_{\text{RL-*}}$ (Equation DISPLAY_FORM17) and $L_\text{{ARL-SEN}}$ (Equation DISPLAY_FORM19)  When training Pointer-Gen+RL-ROUGE, the best $\lambda $ is chosen based on the ROUGE-L score on the validation set  In our experiment, $\lambda $ is set as 0 95  An Adam optimizer with a learning rate of 0 001 is used to optimize $L_b$  When training Pointer-Gen+ARL-SEN, we don't use the full LCSTS dataset, but only headlines with a sensationalism score larger than 0 1 as we observe that Pointer-Gen+ARL-SEN will generate a few unnatural phrases when using full dataset  We believe the reason is the high ratio of RL during training  Figure FIGREF23 shows that the probability density near 0 is very high, meaning that in each batch, many of the samples will have a very low sensationalism score  On expectation, each sample will receive 0 239 MLE training and 0 761 RL training  This leads to RL dominanting the loss  Thus, we propose to filter samples with a minimum sensationalism score with 0 1 and it works very well  For Pointer-Gen+RL-SEN, we also set the minimum sensationalism score as 0 1, and $\lambda $ is set as 0 5 to remove unnatural phrases, making a fair comparison to Pointer-Gen+ARL-SEN 
We stop training Pointer-Gen+Same-FT, Pointer-Gen+Pos-FT, Pointer-Gen+RL-SEN and Pointer-Gen+ARL-SEN, when $\alpha _\text{sen}$ stops increasing on the validation set  Beam-search with a beam size of 5 is adopted for decoding in all models 
Sensational Headline Generation ::: Evaluation Metrics
We briefly describe the evaluation metrics below 
ROUGE: ROUGE is a commonly used evaluation metric for summarization  It measures the N-gram overlap between generated and training headlines  We use it to evaluate the relevance of generated headlines  The widely used pyrouge toolkit is used to calculate ROUGE-1 (RG-1), ROUGE-2 (RG-2), and ROUGE-L (RG-L) 
Human evaluation: We randomly sample 50 articles from the test set and send the generated headlines from all models and corresponding headlines in the test set to human annotators  We evaluate the sensationalism and fluency of the headlines by setting up two independent human annotation tasks  We ask 10 annotators to label each headline for each task  For the sensationalism annotation, each annotator is asked one question, “Is the headline sensational?”, and he/she has to choose either `yes' or `no'  The annotators were not told which system the headline is from  The process of distributing samples and recruiting annotators is managed by Crowdflower  After annotation, we define the sensationalism score as the proportion of annotations on all generated headlines from one model labeled as `yes'  For the fluency annotation, we repeat the same procedure as for the sensationalism annotation, except that we ask each annotator the question “Is the headline fluent?” We define the fluency score as the proportion of annotations on all headlines from one specific model labeled as `yes'  We put human annotation instructions in the supplemental material 
UTF8gbsn
Results
We first compare all four models, Pointer-Gen, Pointer-Gen-RL+ROUGE, Pointer-Gen-RL-SEN, and Pointer-Gen-ARL-SEN, to existing models with ROUGE in Table TABREF25 to establish that our model produces relevant headlines and we leave the sensationalism for human evaluation  Note that we only compare our models to commonly used strong summarization baselines, to validate that our implementation achieves comparable performance to existing work  In our implementation, Pointer-Gen achieves a 34 51 RG-1 score, 22 21 RG-2 score, and 31 68 RG-L score, which is similar to the results of gu2016incorporating  Pointer-Gen+ARL-SEN, although optimized for the sensationalism reward, achieves similar performance to our Pointer-Gen baseline, which means that Pointer-Gen+ARL-SEN still keeps its summarization ability  An example of headlines generated from different models in Table TABREF29 shows that Pointer-Gen and Pointer-Gen+RL-ROUGE learns to summarize the main point of the article: “The Nikon D600 camera is reported to have black spots when taking photos”  Pointer-Gen+RL-SEN makes the headline more sensational by blaming Nikon for attributing the damage to the smog  Pointer-Gen+ARL-SEN generates the most sensational headline by exaggerating the result “Getting a serious trouble!” to maximize user's attention 
We then compare different models using the sensationalism score in Table TABREF30  The Pointer-Gen baseline model achieves a 42 6% sensationalism score, which is the minimum that a typical summarization model achieves  By filtering out low-sensational headlines, Pointer-Gen+Same-FT and Pointer-Gen+Pos-FT achieves higher sensationalism scores, which implies the effectiveness of our sensationalism scorer  Our Pointer-Gen+ARL-SEN model achieves the best performance of 60 8%  This is an absolute improvement of 18 2% over the Pointer-Gen baseline  The Chi-square test on the results confirms that Pointer-Gen+ARL-SEN is statistically significantly more sensational than all the other baseline models, with the largest p-value less than 0 01  Also, we find that the test set headlines achieves 57 8% sensationalism score, much larger than Pointer-Gen baseline, which also supports our intuition that generated headlines will be less sensational than the original one  On the other hand, we found that Pointer-Gen+Pos is much worse than other baselines  The reason is that training on sensational samples alone discards around 80% of the whole training set that is also helpful for maintaining relevance and a good language model  It shows the necessity of using RL 
UTF8gbsn
In addition, both Pointer-Gen+RL-SEN and Pointer-Gen+ARL-SEN, which use the sensationalism score as the reward, obtain statistically better results than Pointer-Gen+RL-ROUGE and Pointer-Gen, with a p-value less than 0 05 by a Chi-square test  This result shows the effectiveness of RL to generate more sensational headlines  The reason is that even though our noisy classifier could also learn to classify domains, the generator during RL training is not allowed to increase the reward by shifting domains but encouraged to generate more sensational headlines, due to the consistency constraint on the domains of the headline and the article  Furthermore, Poiner-Gen+ARL-SEN gets better performance than Pointer-Gen+RL-SEN, which confirms the superiority of the ARL loss function  We also visualize in Figure FIGREF31 a comparison between Pointer-Gen+ARL-SEN and Pointer-Gen+RL-SEN according to how sensational the test set headlines are  The blue bars denote the smaller scores between the two models  For example, if the blue bar is 0 6, it means that the worse model between Pointer-Gen+RL-SEN and Pointer-Gen+ARL-SEN achieves 0 6  And the color of orange/black further indicates the better model and its score  We find that Pointer-Gen+ARL-SEN outperforms Pointer-Gen+RL-SEN for most cases  The improvement is higher when the test set headlines are not sensational (the sensationalism score is less than 0 5), which may be attributed to the higher ratio of RL training on non-sensational headlines 
Apart from the sensationalism evaluation, we measure the fluency of the headlines generated from different models  Fluency scores in Table TABREF30 show that Pointer-Gen+RL-SEN and Pointer-Gen+ARL-SEN achieve comparable fluency performance to Pointer-Gen and Pointer-Gen+RL-ROUGE  Test set headlines achieve the best performance among all models, but the difference is not statistically significant  Also, we observe that fine-tuning on sensational headlines will hurt the performance, both in sensationalism and fluency 
After manually checking the outputs, we observe that our model is able to generate sensational headlines using diverse sensationalization strategies  These strategies include, but are not limited to, creating a curiosity gap, asking questions, highlighting numbers, being emotional and emphasizing the user  Examples can be found in Table TABREF32 
Related Work
Our work is related to summarization tasks  An encoder-decoder model was first applied to two sentence-level abstractive summarization tasks on the DUC-2004 and Gigaword datasets BIBREF12  This model was later extended by selective encoding BIBREF13, a coarse to fine approach BIBREF14, minimum risk training BIBREF1, and topic-aware models BIBREF15  As long summaries were recognized as important, the CNN/Daily Mail dataset was used in nallapati2016abstractive  Graph-based attention BIBREF16, pointer-generator with coverage loss BIBREF0 are further developed to improve the generated summaries  celikyilmaz2018deep proposed deep communicating agents for representing a long document for abstractive summarization  In addition, many papers BIBREF17, BIBREF18, BIBREF19 use extractive methods to directly select sentences from articles  However, none of these work considered the sensationalism of generated outputs 
RL is also gaining popularity as it can directly optimize non-differentiable metrics BIBREF20, BIBREF21, BIBREF22  paulus2017deep proposed an intra-decoder model and combined RL and MLE to deal with summaries with bad qualities  RL has also been explored with generative adversarial networks (GANs) BIBREF23  liu2017generative applied GANs on summarization task and achieved better performance  niu2018polite tackles the problem of polite generation with politeness reward  Our work is different in that we propose a novel function to balance RL and MLE 
Our task is also related to text style transfer  Implicit methods BIBREF10, BIBREF24, BIBREF4 transfer the styles by separating sentence representations into content and style, for example using back-translationBIBREF4  However, these methods cannot guarantee the content consistency between the original sentence and transferred output BIBREF25  Explicit methods BIBREF26, BIBREF25 transfer the style by directly identifying style related keywords and modifying them  However, sensationalism is not always restricted to keywords, but the full sentence  By leveraging small human labeled English dataset, clickbait detection has been well investigated BIBREF2, BIBREF27, BIBREF3  However, these human labeled dataset are not available for other languages, such as Chinese 
Modeling sensationalism is also related to modeling emotion  Emotion has been well investigated in both word levelBIBREF28, BIBREF29 and sentence levelBIBREF30, BIBREF31, BIBREF32, BIBREF33, BIBREF34  It has also been considered an important factor in engaging interactive systemsBIBREF35, BIBREF36, BIBREF37  Although we observe that sensational headlines contain emotion, it is still not clear which emotion and how emotions will influence the sensationalism 
Conclusion and Future Work
In this paper, we propose a model that generates sensational headlines without labeled data using Reinforcement Learning  Firstly, we propose a distant supervision strategy to train the sensationalism scorer  As a result, we achieve 65% accuracy between the predicted sensationalism score and human evaluation  To effectively leverage this noisy sensationalism score as the reward for RL, we propose a novel loss function, ARL, to automatically balance RL with MLE  Human evaluation confirms the effectiveness of both our sensationalism scorer and ARL to generate more sensational headlines  Future work can be improving the sensationalism scorer and investigating the applications of dynamic balancing methods between RL and MLE in textGANBIBREF23  Our work also raises the ethical questions about generating sensational headlines, which can be further explored 
Acknowledgments
Thanks to ITS/319/16FP of Innovation Technology Commission, HKUST 16248016 of Hong Kong Research Grants Council for funding  In addition, we thank Zhaojiang Lin for helpful discussion and Yan Xu, Zihan Liu for the data collection 
Context 2: Introduction
Recently, people have started looking at online forums either as a primary or secondary source of counseling services BIBREF0  BIBREF1 reported that over the first five years of operation (2011-2016), ReachOut com – Ireland's online youth mental health service – 62% of young people would visit a website for support when going through a tough time  With the expansion of the Internet, there has been a substantial growth in the number of users looking for psychological support online 
The importance of the on-line life of patients has been recognized in research as well  BIBREF2 stated that the online life of patients constitutes a major influence on their self-definition  Furthermore, according to BIBREF3, the social networking activities of an individual, offer an important reflection of their personality  While dealing with patients suffering from psychological problems, it is important that therapists do not ignore this pivotal source of information which can provide deep insights into their patients' mental conditions 
Acceptance of on-line support groups (OSG) by Mental Health Professionals is still not established BIBREF4  Since OSG can have double-edged effects on patients and the presence of professionals is often limited, we argue that their properties should be further studied  According to BIBREF5 OSG effectiveness is hard to assess, while some studies showed OSG's potential to change participants' attitudes, no such effect was observed in other studies (see Related Work Section for more details)  Furthermore the scope of previous work on analysis of users' behaviour in OSG has been limited by the fact that they relied on expert annotation of posts and comments BIBREF6 
We present a novel approach for automatically analysing online conversations for the presence of therapeutic factors of group therapy defined by BIBREF7 as “the actual mechanisms of effecting change in the patient”  The authors have identified 11 therapeutic factors in group therapy: Universality, Altruism, Instillation of Hope, Guidance, Imparting information, Developing social skills, Interpersonal learning, Cohesion, Catharsis, Existential factors, Imitative behavior and Corrective recapitulation of family of origin issues  In this paper, we focus on 3 therapeutic factors: Universality, Altruism and Instillation of Hope (listed below), as we believe that these can be approximated by using established NLP techniques (e g  Sentiment Analysis, Dialogue Act tagging etc ) 
Universality: the disconfirmation of a user's feelings of uniqueness of their mental health condition 
Altruism: others offer support, reassurance, suggestions and insight 
Instillation of Hope: inspiration provided to participants by their peers 
The selected therapeutic factors are analysed in terms of illocutionary force and attitude  Due to the multi-party and asynchronous nature of on-line social media conversations, prior to the analysis, we extract conversation threads among users – an essential prerequisite for any kind of higher-level dialogue analysis BIBREF10  Afterwards, the illocutionary force is identified using Dialogue Act tagging, whereas the attitude by using Sentiment Analysis  The quantitative analysis is then performed on these processed conversations 
Ideally, the analysis would require experts to annotate each post and comment on the presence of therapeutic factors  However, due to time and cost demands of this task, it is feasible to analyse only a small fraction of the available data  Compared to previous studies (e g  BIBREF6) that analysed few tens of conversations and several thousand lines of chat; using the proposed approach – application of Dialogue Acts and Sentiment Analysis – we were able to automatically analyse approximately 300 thousands conversations (roughly 1 5 million comments)  The rest of the paper is structured as follows  In Section 2 we introduce related work  Next, in Section 3 we describe the pre-processing pipeline and the methodology to perform thread extraction on asynchronous multi-party conversations  In Section 4 we provide the describe the final dataset used for the analysis, and in Section 5 we present the results of our analysis  Finally, in Section 6 we provide concluding remarks and future research directions 
Related Work
On-line support groups have been analyzed for various factors before  For instance, BIBREF11 analysed stress reduction in on-line support group chat-rooms, and the effects of on-line social interactions  Such studies mostly relied on questionnaires and were based on a small number of users  Nevertheless, in BIBREF11, the author showed that social support facilitates coping with distress, improves mood and expedites recovery from it  These findings highlight that, overall, on-line discussion boards appear to be therapeutic and constructive for individuals suffering alcohol-abuse 
Application of NLP to the analysis of mental health-related conversation has been studied as well (e g  BIBREF12, BIBREF13)  BIBREF6 applied sentiment-analysis combined with extensive turn-level annotation to investigate stress reduction in on-line support group chat-rooms, showing that sentiment-analysis is a good predictor of entrance stress level  Furthermore, similar to our setting, they applied automatic thread-extraction to determine conversation threads 
BIBREF14 have shown that on-line support group therapy increased the quality of life of patients with metastatic breast cancer  Since many original posters reported the benefits of group therapy on patients BIBREF15, BIBREF2, BIBREF16, BIBREF17, BIBREF18, BIBREF7, we evaluate the effect of the user interaction using sentiment scores of comments in on-line support groups 
According to BIBREF6, users with high incoming stress tend to request less information from others, as a percentage of their time, and share much more information, in absolute terms  In addition, high information sharing has been shown to be a good predictor of stress reduction at the end of the chat BIBREF6  Regarding information sharing, we rely on Dialogue Acts BIBREF19 to model the speaker's intention in producing an utterance  In particular, we are interested in Dialogue Act label that is defined to represent descriptive, narrative, or personal information – the statement 
Dialogue Acts have been applied to the analysis of spoken BIBREF20, BIBREF21 as well as on-line written synchronous conversations BIBREF22  We apply Dialogue Act tag set defined in BIBREF22 to the analysis of our on-line asynchronous conversations  We argue that Dialogue Acts can be used to analyse user behaviour in social media and verify the presence of therapeutic factors 
Methodology
We select the three therapeutic factors – Universality, Altruism and Instillation of Hope – that can be best approximated using NLP techniques: Sentiment Analysis and Dialogue Act tagging  We discuss each one of the selected therapeutic factors and the identified necessary conditions  The listed conditions, however, are not sufficient to attribute the presence of a therapeutic factor with high confidence, which only can be obtained using expert annotation  Our analysis focuses on the structure of conversations; though content plays an important role as well 
Universality consists in the disconfirmation of patients' belief of uniqueness of their disease  This therapeutic factor is shown to be a powerful source of relief for the patient, according to BIBREF7  From this definition, we can draw the following conditions that are applicable to our environment:
improvement of original poster's sentiment: we hypothesize that the discovery that other people passed through similar issues leads to a higher sentiment score;
posts containing negative personal experiences: to disconfirm the belief of uniqueness users have to share their story;
comments containing negative statements: to disconfirm the patient's feelings of uniqueness, the commenting user must tell a similar negative personal experience  This condition requires two sub-conditions: high presence of statements in comments and the presence of negative comments replying to negative posts 
Instillation of Hope is based on inspiration provided to participants by their peers  Through the inspiration provided by their peers, patients can increase their expectation on the therapy outcome  BIBREF7 in several studies have demonstrated that a high expectation of help before the start of a therapy is significantly correlated with a positive therapy outcome  The author states that many patients pointed out the importance of having observed the improvement of others  Therefore, the three main conditions are the following:
improvement of original poster's sentiment: we hypothesize that instillation of hope leads to a higher sentiment score;
posts containing negative personal experiences: hope can be instilled in someone who shares a negative personal experience;
comments containing positive personal experiences: in order to instill hope, commenting posters must show to original posters an overall positive personal experience  To detect positive personal experience, we require the presence of statements in comments and a positive sentiment of comments replying to negative posts 
Altruism consists of peers offering support, reassurance, suggestions and insight, since they share similar problems with one another BIBREF7  The experience of finding that a patient can be of value to others is refreshing and boosts self-esteem BIBREF7  However, in the current study we focus on testing whether commenting posters are altruists or not  We do not test whether the altruistic behavior leads to an improvement on the altruist itself  For these reasons, we define three main conditions:
improvement of original poster's sentiment: we hypothesize that supportive and reassuring statements improve the sentiment score of the original poster;
posts contains negative personal experiences: users offer support, reassurance and suggestion when facing a negative personal experience of the original poster;
comments containing positive statements: either supportive or reassuring statements show by definition a positive intended emotional communication  Thus comments to the post should consist of positive sentiment statements 
Consequently, a conversation containing the aforementioned therapeutic factors should satisfy the following conditions in terms of NLP: Sentiment Analysis and Dialogue Acts 
original posters have a higher sentiment score at the end of the thread than at the beginning;
the original post consists mostly of polarised statements;
the presence of a significant amount of statements in comments, since both support and sharing similar negative experiences can be represented as statements;
both negative and positive statements in comments lead to higher final sentiment score of the original poster 
Datasets
We verify the presence of therapeutic factors in two social media datasets: OSG and Twitter  The first dataset is crawled from an on-line support groups website, and the second dataset consists of a small sample of Twitter conversation threads  Since the former consists of multi-threaded conversations, we apply a pre-processing to extract conversation threads to provide a fair comparison with the Twitter dataset  An example conversation from each data source is presented in Figure FIGREF19 
Datasets ::: Twitter
We have downloaded 1,873 Twitter conversation threads, roughly 14k tweets, from a publicly available resource that were previously pre-processed and have conversation threads extracted  A conversation in the dataset consists of at least 4 tweets  Even though, according to BIBREF23, Twitter is broadly applicable to public health research, our expectation is that it contains less therapeutic conversations in comparison to specialized on-line support forums 
Datasets ::: OSG
Our data has been developed by crawling and pre-processing an OSG web forum  The forum has a great variety of different groups such as depression, anxiety, stress, relationship, cancer, sexually transmitted diseases, etc  Each conversation starts with one post and can contain multiple comments  Each post or comment is represented by a poster, a timestamp, a list of users it is referencing to, thread id, a comment id and a conversation id  The thread id is the same for comments replying to each other, otherwise it is different  The thread id is increasing with time  Thus, it provides ordering among threads; whereas the timestamp provides ordering in the thread 
Each conversation can belong to multiple groups  Consequently, the dataset needs to be processed to remove duplicates  The dataset resulting after de-duplication contains 295 thousand conversations, each conversation contains on average 6 comments  In total, there are 1 5 million comments  Since the created dataset is multi-threaded, we need to extract conversation threads, to eliminate paths not relevant to the original post 
Datasets ::: OSG ::: Conversation Thread Extraction
The thread extraction algorithm is heuristic-based and consists of two steps: (1) creation of a tree, based on a post written by a user and the related comments and (2) transformation of the tree into a list of threads 
The tree creation is an extension of the approach of BIBREF24, where first a graph of conversation is constructed  In the approach, direct replies to a post are attached to the first nesting level and subsequent comments to increasing nesting levels  In our approach, we also exploit comments' features

ground_truth: Pointer-Gen, Pointer-Gen+Pos, Pointer-Gen+Same-FT, Pointer-Gen+Pos-FT, Pointer-Gen+RL-ROUGE, Pointer-Gen+RL-SEN

prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. How large is the Twitter dataset?
Context 2: 
The tree creation is performed without processing the content of comments, which allows us to process posts and comments of any length efficiently  The heuristic used in the process is based on three simplifying assumptions:
Unless there is a specific reference to another comment or a user, comments are attached to the original post 
When replying, the commenting poster is always replying to the original post or some other comment  Unless specified otherwise, it is assumed that it is a response to the previous (in time) post/comment 
Subsequent comments by the same poster are part of the same thread 
To evaluate the performance of the thread extraction algorithm, 2 annotators have manually constructed the trees for 100 conversations  The performance of the algorithm on this set of 100 conversations is evaluated using accuracy and standard Information Retrieval evaluation metrics of precision, recall, and F$_1$ measure  The results are reported in Table TABREF28 together with random and majority baselines  The turn-level percent agreement between the 2 annotators is 97 99% and Cohen's Kappa Coefficient is 83 80% 
Datasets ::: Data Representation
For both data sources, Twitter and OSG with extracted threads, posts and comments are tokenized and sentence split  Each sentence is passed through Sentiment Analysis and Dialogue Act tagging  Since a post or a comment can contain multiple sentences, therefore multiple Dialogue Acts, it is represented as as a one-hot encoding, where each position represents a Dialogue Act 
For Sentiment Analysis we use a lexicon-based sentiment analyser introduced by BIBREF25  For Dialogue Act tagging, on the other hand, we make use of a model trained on NPSChat corpus BIBREF22 following the approach of BIBREF26 
Analysis
As we mentioned in Section 3, the presence of each of the therapeutic conditions under analysis is a necessary for a conversation to be considered to have therapeutic factors  In this section we present the results of our analysis with respect to these conditions 
Analysis ::: Change in Sentiment score of Original Posters
The first condition which we test is the sentiment change in conversation threads, comparing the initial and final sentiment scores (i e  posts' scores) of the original poster  The results of the analysis are presented in Figure FIGREF33  In the figure we can observe that the distribution of the sentiment change in the two datasets is different  While in Twitter the amount of conversations that lead to the increase of sentiment score is roughly equal to the amount of conversations that lead to the decrease of sentiment score; the situation is different for OSG  In OSG, the amount of conversations that lead to the increase of sentiment score is considerably higher 
Figure FIGREF34 provides a more fine grained analysis, where we additionally analyse the sentiment change in nominal polarity terms – negative and positive  In OSG, the number of users that changed polarity from negative to positive is more than the double of the users that have changed the polarity from positive to negative  In Twitter, on the other hand, the users mostly changed polarity from positive to negative  Results of the analysis suggest that in OSG , sentiment increases and users tend to change polarity from negative to positive, whereas in Twitter sentiment tends to decrease  Verification of this condition alone indicates that the ratio of potentially therapeutic conversations in Twitter is lower 
Analysis ::: Structure of Posts and Comments
Table TABREF36 presents the distribution of automatically predicted per-sentence Dialogue Acts in the datasets  The most frequent tag is statement in both  In Table TABREF37, on the other hand, we present the distribution of post and comment structures in terms of automatically predicted Dialogue Act tags  The structure is an unordered set of tags in the post or comment  From the table we can observe that the distribution of tag sets is similar between posts and comments  In both cases the most common set is statement only  However, conversations containing only statement, emphasis or question posts and comments predominantly appear in Twitter  Which is expected due to the shorter length of Twitter posts and comments 
We can also observe that the original posters tend to ask more questions than the commenting posters – 19 83% for posts vs  11 21% for comments (summed)  This suggests that the original posters frequently ask either for suggestion or confirmation of their points of view or their disconfirmation  However, the high presence of personal experiences is supported by the high number of posts containing only statements 
High number of statement tags in comments suggests that users reply either with supporting or empathic statements or personal experience  However, 6 39% of comments contain accept and reject tags, which mark the degree to which a speaker accepts some previous proposal, plan, opinion, or statement BIBREF20  The described Dialogue Act tags are often used when commenting posters discuss original poster's point of view  For instance, “It's true  I felt the same ” – {Accept, Statement} or “Well no  You're not alone” – {Reject, Statement}  The datasets differ with respect to the distribution of these Dialogue Acts tags, they appear more frequently in OSG 
Analysis ::: Sentiment of Posts and Comments
Table TABREF39 presents the distribution of sentiment polarity in post and comment statements (i e  sentences tagged as statement)  For OSG, the predominant sentiment label of statements is positive and it is the highest for both posts and comments  However, the difference between the amounts of positive and negative statements is higher for the replying comments (34 5% vs  42 5%)  For Twitter, on the other hand, the predominant sentiment label of statements is neutral and the polarity distribution between posts and comments is very close  One particular observation is that the ratio of negative statements is higher in OSG for both posts and comments than in Twitter, which supports the idea of sharing negative experiences 
Further we analyze whether the sentiment of a comment (i e  the replying user) is affected by the sentiment of the original post (i e  the user being replied to), which will imply that the users adapt their behaviour with respect to the post's sentiment  For the analysis, we split the datasets into three buckets according to the posts' sentiment score – negative, neutral, or positive, and represent each conversation in terms of percentages of comments (replies) with each sentiment label  The buckets are then compared using t-test for statistically significant differences 
Table TABREF40 presents the distribution of sentiment labels with respect to the post's sentiment score  The patterns of distribution are similar across the datasets  We can observe that overall, replies tend to have a positive sentiment, which suggests that replying posters tend to have a positive attitude  However, the ratio of positive comments is higher for OSG than for Twitter 
The results of the Welch's t-test on OSG data reveal that there are statistically significant differences in the distribution of replying comments' sentiment between conversations with positive and negative starting posts  A positive post tends to get significantly more positive replies  Similarly, a negative post tends to get significantly more negative replies (both with $p < 0 01$) 
Table TABREF41 presents the distribution of the sentiment labels of the final text provided by the original poster with respect to the sentiment polarity of the comments  The results indicate that OSG participants are more supportive, as the majority of conversations end in a positive final sentiment regardless of the sentiment of comments  We can also observe that negative comments in OSG lead to positive sentiment, which supports the idea of sharing the negative experiences, thus presence of therapeutic factors  For Twitter, on the other hand, only positive comments lead to the positive final sentiments, whereas other comments lead predominantly to neutral final sentiments 
Our analysis in terms of sentiment and Dialogue Acts supports the presence of the three selected therapeutic factors – Universality, Altruism and Instillation of Hope – in OSG more than in Twitter  The main contributors to this conclusion are the facts that there is more positive change in the sentiment of the original posters in OSG (people seeking support) and that in OSG even negative and neutral comments are likely to lead to positive changes 
Conclusion
In this work, we propose a methodology to automatically analyse online social platforms for the presence of therapeutic factors (i e  Universality, Altruism and Instillation of Hope)  We evaluate our approach on two on-line platforms, Twitter and an OSG web forum  We apply NLP techniques of Sentiment Analysis and Dialogue Act tagging to automatically verify the presence of therapeutic factors, which allows us to analyse larger amounts of conversational data (as compared to previous studies) 
Our analysis indicates that OSG conversations satisfy higher number of conditions approximating therapeutic factors than Twitter conversations  Given this outcome, we postulate that users who join support group websites spontaneously seem to benefit from it  Indeed, as shown in Section SECREF5, the original posters who interact with others by replying to comments, have benefited from an improvement of their emotional state 
We would like to reemphasise that the conditions for the therapeutic factors are necessary but not sufficient; since our analysis focuses on the structure of conversations, being agnostic to the content  NLP, however, allows us to strengthen our approximations even further  Thus, the further extension of our work is also augmentation of our study with other language analysis metrics and their correlation with human annotation 
It should be noted that the proposed approach is an approximation of the tedious tasks of annotation of conversations by experts versed in the therapeutic factors and their associated theories  Even though we can use Sentiment Analysis to detect the existence of therapeutic factors, we cannot differentiate between Altruism and Instillation of Hope, as this requires differentiation between emotional state of the user and the intended emotional communication  Thus, the natural extensions of this work are differentiation between different therapeutic factors and comparison of the proposed analysis to the human evaluation 
Although we acknowledge that the proposed methodology does not serve as a replacement of manual analysis of OSG for the presence of therapeutic factors, we believe that it could facilitate and supplement this process  The method can serve as a tool for general practitioners and psychologists who can use it as an additional source of information regarding their patients’ condition and, in turn, offer a more personalised support that is better tailored to individual therapeutic needs 
Context 3: Introduction
Data imbalance is a common issue in a variety of NLP tasks such as tagging and machine reading comprehension  Table TABREF3 gives concrete examples: for the Named Entity Recognition (NER) task BIBREF2, BIBREF3, most tokens are backgrounds with tagging class $O$  Specifically, the number of tokens tagging class $O$ is 5 times as many as those with entity labels for the CoNLL03 dataset and 8 times for the OntoNotes5 0 dataset; Data-imbalanced issue is more severe for MRC tasks BIBREF4, BIBREF5, BIBREF6, BIBREF7, BIBREF8 with the value of negative-positive ratio being 50-200 
Data imbalance results in the following two issues: (1) the training-test discrepancy: Without balancing the labels, the learning process tends to converge to a point that strongly biases towards class with the majority label  This actually creates a discrepancy between training and test: at training time, each training instance contributes equally to the objective function while at test time, F1 score concerns more about positive examples; (2) the overwhelming effect of easy-negative examples  As pointed out by meng2019dsreg, significantly large number of negative examples also means that the number of easy-negative example is large  The huge number of easy examples tends to overwhelm the training, making the model not sufficiently learned to distinguish between positive examples and hard-negative examples  The cross-entropy objective (CE for short) or maximum likelihood (MLE) objective, which is widely adopted as the training objective for data-imbalanced NLP tasks BIBREF9, BIBREF10, BIBREF11, BIBREF12, BIBREF13, BIBREF14, BIBREF15, handles neither of the issues 
To handle the first issue, we propose to replace CE or MLE with losses based on the Sørensen–Dice coefficient BIBREF0 or Tversky index BIBREF1  The Sørensen–Dice coefficient, dice loss for short, is the harmonic mean of precision and recall  It attaches equal importance to false positives (FPs) and false negatives (FNs) and is thus more immune to data-imbalanced datasets  Tversky index extends dice loss by using a weight that trades precision and recall, which can be thought as the approximation of the $F_{\beta }$ score, and thus comes with more flexibility  Therefore, We use dice loss or Tversky index to replace CE loss to address the first issue 
Only using dice loss or Tversky index is not enough since they are unable to address the dominating influence of easy-negative examples  This is intrinsically because dice loss is actually a hard version of the F1 score  Taking the binary classification task as an example, at test time, an example will be classified as negative as long as its probability is smaller than 0 5, but training will push the value to 0 as much as possible  This gap isn't a big issue for balanced datasets, but is extremely detrimental if a big proportion of training examples are easy-negative ones: easy-negative examples can easily dominate training since their probabilities can be pushed to 0 fairly easily  Meanwhile, the model can hardly distinguish between hard-negative examples and positive ones  Inspired by the idea of focal loss BIBREF16 in computer vision, we propose a dynamic weight adjusting strategy, which associates each training example with a weight in proportion to $(1-p)$, and this weight dynamically changes as training proceeds  This strategy helps to deemphasize confident examples during training as their $p$ approaches the value of 1, makes the model attentive to hard-negative examples, and thus alleviates the dominating effect of easy-negative examples 
Combing both strategies, we observe significant performance boosts on a wide range of data imbalanced NLP tasks  Notably, we are able to achieve SOTA results on CTB5 (97 92, +1 86), CTB6 (96 57, +1 80) and UD1 4 (96 98, +2 19) for the POS task; SOTA results on CoNLL03 (93 33, +0 29), OntoNotes5 0 (92 07, +0 96)), MSRA 96 72(+0 97) and OntoNotes4 0 (84 47,+2 36) for the NER task; along with competitive results on the tasks of machine reading comprehension and paraphrase identification 
The rest of this paper is organized as follows: related work is presented in Section 2  We describe different training objectives in Section 3  Experimental results are presented in Section 4  We perform ablation studies in Section 5, followed by a brief conclusion in Section 6 
Related Work ::: Data Resample
The idea of weighting training examples has a long history  Importance sampling BIBREF17 assigns weights to different samples and changes the data distribution  Boosting algorithms such as AdaBoost BIBREF18 select harder examples to train subsequent classifiers  Similarly, hard example mining BIBREF19 downsamples the majority class and exploits the most difficult examples  Oversampling BIBREF20, BIBREF21 is used to balance the data distribution  Another line of data resampling is to dynamically control the weights of examples as training proceeds  For example, focal loss BIBREF16 used a soft weighting scheme that emphasizes harder examples during training  In self-paced learning BIBREF22, example weights are obtained through optimizing the weighted training loss which encourages learning easier examples first  At each training step, self-paced learning algorithm optimizes model parameters and example weights jointly  Other works BIBREF23, BIBREF24 adjusted the weights of different training examples based on training loss  Besides, recent work BIBREF25, BIBREF26 proposed to learn a separate network to predict sample weights 
Related Work ::: Data Imbalance Issue in Object Detection
The background-object label imbalance issue is severe and thus well studied in the field of object detection BIBREF27, BIBREF28, BIBREF29, BIBREF30, BIBREF31  The idea of hard negative mining (HNM) BIBREF30 has gained much attention recently  shrivastava2016ohem proposed the online hard example mining (OHEM) algorithm in an iterative manner that makes training progressively more difficult, and pushes the model to learn better  ssd2016liu sorted all of the negative samples based on the confidence loss and picking the training examples with the negative-positive ratio at 3:1  pang2019rcnn proposed a novel method called IoU-balanced sampling and aploss2019chen designed a ranking model to replace the conventional classification task with a average-precision loss to alleviate the class imbalance issue  The efforts made on object detection have greatly inspired us to solve the data imbalance issue in NLP 
Losses ::: Notation
For illustration purposes, we use the binary classification task to demonstrate how different losses work  The mechanism can be easily extended to multi-class classification 
Let $\lbrace x_i\rbrace $ denote a set of instances  Each $x_i$ is associated with a golden label vector $y_i = [y_{i0},y_{i1} ]$, where $y_{i1}\in \lbrace 0,1\rbrace $ and $y_{i0}\in \lbrace 0,1\rbrace $ respectively denote the positive and negative classes, and thus $y_i$ can be either $[0,1]$ or $[0,1]$  Let $p_i = [p_{i0},p_{i1} ]$ denote the probability vector, and $p_{i1}$ and $p_{i0}$ respectively denote the probability that a model assigns the positive and negative label to $x_i$ 
Losses ::: Cross Entropy Loss
The vanilla cross entropy (CE) loss is given by:
As can be seen from Eq DISPLAY_FORM8, each $x_i$ contributes equally to the final objective  Two strategies are normally used to address the the case where we wish that not all $x_i$ are treated equal: associating different classes with different weighting factor $\alpha $ or resampling the datasets  For the former, Eq DISPLAY_FORM8 is adjusted as follows:
where $\alpha _i\in [0,1]$ may be set by the inverse class frequency or treated as a hyperparameter to set by cross validation  In this work, we use $\lg (\frac{n-n_t}{n_t}+K)$ to calculate the coefficient $\alpha $, where $n_t$ is the number of samples with class $t$ and $n$ is the total number of samples in the training set  $K$ is a hyperparameter to tune  The data resampling strategy constructs a new dataset by sampling training examples from the original dataset based on human-designed criteria, e g , extract equal training samples from each class  Both strategies are equivalent to changing the data distribution and thus are of the same nature  Empirically, these two methods are not widely used due to the trickiness of selecting $\alpha $ especially for multi-class classification tasks and that inappropriate selection can easily bias towards rare classes BIBREF32 
Losses ::: Dice coefficient and Tversky index
Sørensen–Dice coefficient BIBREF0, BIBREF33, dice coefficient (DSC) for short, is a F1-oriented statistic used to gauge the similarity of two sets  Given two sets $A$ and $B$, the dice coefficient between them is given as follows:
In our case, $A$ is the set that contains of all positive examples predicted by a specific model, and $B$ is the set of all golden positive examples in the dataset  When applied to boolean data with the definition of true positive (TP), false positive (FP), and false negative (FN), it can be then written as follows:
For an individual example $x_i$, its corresponding DSC loss is given as follows:
As can be seen, for a negative example with $y_{i1}=0$, it does not contribute to the objective  For smoothing purposes, it is common to add a $\gamma $ factor to both the nominator and the denominator, making the form to be as follows:
As can be seen, negative examples, with $y_{i1}$ being 0 and DSC being $\frac{\gamma }{ p_{i1}+\gamma }$, also contribute to the training  Additionally, milletari2016v proposed to change the denominator to the square form for faster convergence, which leads to the following dice loss (DL):
Another version of DL is to directly compute set-level dice coefficient instead of the sum of individual dice coefficient  We choose the latter due to ease of optimization 
Tversky index (TI), which can be thought as the approximation of the $F_{\beta }$ score, extends dice coefficient to a more general case  Given two sets $A$ and $B$, tversky index is computed as follows:
Tversky index offers the flexibility in controlling the tradeoff between false-negatives and false-positives  It degenerates to DSC if $\alpha =\beta =0 5$  The Tversky loss (TL) for the training set $\lbrace x_i,y_i\rbrace $ is thus as follows:
Losses ::: Self-adusting Dice Loss
Consider a simple case where the dataset consists of only one example $x_i$, which is classified as positive as long as $p_{i1}$ is larger than 0 5  The computation of $F1$ score is actually as follows:
Comparing Eq DISPLAY_FORM14 with Eq DISPLAY_FORM22, we can see that Eq DISPLAY_FORM14 is actually a soft form of $F1$, using a continuous $p$ rather than the binary $\mathbb {I}( p_{i1}>0 5)$  This gap isn't a big issue for balanced datasets, but is extremely detrimental if a big proportion of training examples are easy-negative ones: easy-negative examples can easily dominate training since their probabilities can be pushed to 0 fairly easily  Meanwhile, the model can hardly distinguish between hard-negative examples and positive ones, which has a huge negative effect on the final F1 performance 
To address this issue, we propose to multiply the soft probability $p$ with a decaying factor $(1-p)$, changing Eq DISPLAY_FORM22 to the following form:
One can think $(1-p_{i1})$ as a weight associated with each example, which changes as training proceeds  The intuition of changing $p_{i1}$ to $(1-p_{i1}) p_{i1}$ is to push down the weight of easy examples  For easy examples whose probability are approaching 0 or 1, $(1-p_{i1}) p_{i1}$ makes the model attach significantly less focus to them  Figure FIGREF23 gives gives an explanation from the perspective in derivative: the derivative of $\frac{(1-p)p}{1+(1-p)p}$ with respect to $p$ approaches 0 immediately after $p$ approaches 0, which means the model attends less to examples once they are correctly classified 
A close look at Eq DISPLAY_FORM14 reveals that it actually mimics the idea of focal loss (FL for short) BIBREF16 for object detection in vision  Focal loss was proposed for one-stage object detector to handle foreground-background tradeoff encountered during training  It down-weights the loss assigned to well-classified examples by adding a $(1-p)^{\beta }$ factor, leading the final loss to be $(1-p)^{\beta }\log p$ 
In Table TABREF18, we show the losses used in our experiments, which is described in the next section 
Experiments
We evaluate the proposed method on four NLP tasks: part-of-speech tagging, named entity recognition, machine reading comprehension and paraphrase identification  Baselines in our experiments are optimized by using the standard cross-entropy training objective 
Experiments ::: Part-of-Speech Tagging
Part-of-speech tagging (POS) is the task of assigning a label (e g , noun, verb, adjective) to each word in a given text  In this paper, we choose BERT as the backbone and conduct experiments on three Chinese POS datasets  We report the span-level micro-averaged precision, recall and F1 for evaluation  Hyperparameters are tuned on the corresponding development set of each dataset 
Experiments ::: Part-of-Speech Tagging ::: Datasets
We conduct experiments on the widely used Chinese Treebank 5 0, 6 0 as well as UD1 4 
CTB5 is a Chinese dataset for tagging and parsing, which contains 507,222 words, 824,983 characters and 18,782 sentences extracted from newswire sources 
CTB6 is an extension of CTB5, containing 781,351 words, 1,285,149 characters and 28,295 sentences 
UD is the abbreviation of Universal Dependencies, which is a framework for consistent annotation of grammar (parts of speech, morphological features, and syntactic dependencies) across different human languages  In this work, we use UD1 4 for Chinese POS tagging 
Experiments ::: Part-of-Speech Tagging ::: Baselines
We use the following baselines:
Joint-POS: shao2017character jointly learns Chinese word segmentation and POS 
Lattice-LSTM: lattice2018zhang constructs a word-character lattice 
Bert-Tagger: devlin2018bert treats part-of-speech as a tagging task

ground_truth: 1,873 Twitter conversation threads, roughly 14k tweets

1,873 Twitter conversation threads, roughly 14k tweets

prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. How are weights dynamically adjusted?
Context 3: 
Experiments ::: Part-of-Speech Tagging ::: Results
Table presents the experimental results on the POS task  As can be seen, the proposed DSC loss outperforms the best baseline results by a large margin, i e , outperforming BERT-tagger by +1 86 in terms of F1 score on CTB5, +1 80 on CTB6 and +2 19 on UD1 4  As far as we are concerned, we are achieving SOTA performances on the three datasets  Weighted cross entropy and focal loss only gain a little performance improvement on CTB5 and CTB6, and the dice loss obtains huge gain on CTB5 but not on CTB6, which indicates the three losses are not consistently robust in resolving the data imbalance issue  The proposed DSC loss performs robustly on all the three datasets 
Experiments ::: Named Entity Recognition
Named entity recognition (NER) refers to the task of detecting the span and semantic category of entities from a chunk of text  Our implementation uses the current state-of-the-art BERT-MRC model proposed by xiaoya2019ner as a backbone  For English datasets, we use BERT$_\text{Large}$ English checkpoints, while for Chinese we use the official Chinese checkpoints  We report span-level micro-averaged precision, recall and F1-score  Hyperparameters are tuned on the development set of each dataset 
Experiments ::: Named Entity Recognition ::: Datasets
For the NER task, we consider both Chinese datasets, i e , OntoNotes4 0 BIBREF34 and MSRA BIBREF35, and English datasets, i e , CoNLL2003 BIBREF36 and OntoNotes5 0 BIBREF37 
CoNLL2003 is an English dataset with 4 entity types: Location, Organization, Person and Miscellaneous  We followed data processing protocols in BIBREF14 
English OntoNotes5 0 consists of texts from a wide variety of sources and contains 18 entity types  We use the standard train/dev/test split of CoNLL2012 shared task 
Chinese MSRA performs as a Chinese benchmark dataset containing 3 entity types  Data in MSRA is collected from news domain  Since the development set is not provided in the original MSRA dataset, we randomly split the training set into training and development splits by 9:1  We use the official test set for evaluation 
Chinese OntoNotes4 0 is a Chinese dataset and consists of texts from news domain, which has 18 entity types  In this paper, we take the same data split as wu2019glyce did 
Experiments ::: Named Entity Recognition ::: Baselines
We use the following baselines:
ELMo: a tagging model from peters2018deep 
Lattice-LSTM: lattice2018zhang constructs a word-character lattice, only used in Chinese datasets 
CVT: from kevin2018cross, which uses Cross-View Training(CVT) to improve the representations of a Bi-LSTM encoder 
Bert-Tagger: devlin2018bert treats NER as a tagging task 
Glyce-BERT: wu2019glyce combines glyph information with BERT pretraining 
BERT-MRC: The current SOTA model for both Chinese and English NER datasets proposed by xiaoya2019ner, which formulate NER as machine reading comprehension task 
Experiments ::: Named Entity Recognition ::: Results
Table shows experimental results on NER datasets  For English datasets including CoNLL2003 and OntoNotes5 0, our proposed method outperforms BERT-MRCBIBREF38 by +0 29 and +0 96 respectively  We observe huge performance boosts on Chinese datasets, achieving F1 improvements by +0 97 and +2 36 on MSRA and OntoNotes4 0, respectively  As far as we are concerned, we are setting new SOTA performances on all of the four NER datasets 
Experiments ::: Machine Reading Comprehension
Machine reading comprehension (MRC) BIBREF39, BIBREF40, BIBREF41, BIBREF40, BIBREF42, BIBREF15 has become a central task in natural language understanding  MRC in the SQuAD-style is to predict the answer span in the passage given a question and the passage  In this paper, we choose the SQuAD-style MRC task and report Extract Match (EM) in addition to F1 score on validation set  All hyperparameters are tuned on the development set of each dataset 
Experiments ::: Machine Reading Comprehension ::: Datasets
The following five datasets are used for MRC task: SQuAD v1 1, SQuAD v2 0 BIBREF4, BIBREF6 and Quoref BIBREF8 
SQuAD v1 1 and SQuAD v2 0 are the most widely used QA benchmarks  SQuAD1 1 is a collection of 100K crowdsourced question-answer pairs, and SQuAD2 0 extends SQuAD1 1 allowing no short answer exists in the provided passage 
Quoref is a QA dataset which tests the coreferential reasoning capability of reading comprehension systems, containing 24K questions over 4 7K paragraphs from Wikipedia 
Experiments ::: Machine Reading Comprehension ::: Baselines
We use the following baselines:
QANet: qanet2018 builds a model based on convolutions and self-attention  Convolution to model local interactions and self-attention to model global interactions 
BERT: devlin2018bert treats NER as a tagging task 
XLNet: xlnet2019 proposes a generalized autoregressive pretraining method that enables learning bidirectional contexts 
Experiments ::: Machine Reading Comprehension ::: Results
Table shows the experimental results for MRC tasks  With either BERT or XLNet, our proposed DSC loss obtains significant performance boost on both EM and F1  For SQuADv1 1, our proposed method outperforms XLNet by +1 25 in terms of F1 score and +0 84 in terms of EM and achieves 87 65 on EM and 89 51 on F1 for SQuAD v2 0  Moreover, on QuoRef, the proposed method surpasses XLNet results by +1 46 on EM and +1 41 on F1  Another observation is that, XLNet outperforms BERT by a huge margin, and the proposed DSC loss can obtain further performance improvement by an average score above 1 0 in terms of both EM and F1, which indicates the DSC loss is complementary to the model structures 
Experiments ::: Paraphrase Identification
Paraphrases are textual expressions that have the same semantic meaning using different surface words  Paraphrase identification (PI) is the task of identifying whether two sentences have the same meaning or not  We use BERT BIBREF11 and XLNet BIBREF43 as backbones and report F1 score for comparison  Hyperparameters are tuned on the development set of each dataset 
Experiments ::: Paraphrase Identification ::: Datasets
We conduct experiments on two widely used datasets for PI task: MRPC BIBREF44 and QQP 
MRPC is a corpus of sentence pairs automatically extracted from online news sources, with human annotations of whether the sentence pairs are semantically equivalent  The MRPC dataset has imbalanced classes (68% positive, 32% for negative) 
QQP is a collection of question pairs from the community question-answering website Quora  The class distribution in QQP is also unbalanced (37% positive, 63% negative) 
Experiments ::: Paraphrase Identification ::: Results
Table shows the results for PI task  We find that replacing the training objective with DSC introduces performance boost for both BERT and XLNet  Using DSC loss improves the F1 score by +0 58 for MRPC and +0 73 for QQP 
Ablation Studies ::: The Effect of Dice Loss on Accuracy-oriented Tasks
We argue that the most commonly used cross-entropy objective is actually accuracy-oriented, whereas the proposed dice loss (DL) performs as a hard version of F1-score  To explore the effect of the dice loss on accuracy-oriented tasks such as text classification, we conduct experiments on the Stanford Sentiment Treebank sentiment classification datasets including SST-2 and SST-5  We fine-tune BERT$_\text{Large}$ with different training objectives  Experiment results for SST are shown in   For SST-5, BERT with CE achieves 55 57 in terms of accuracy, with DL and DSC losses slightly degrade the accuracy performance and achieve 54 63 and 55 19, respectively  For SST-2, BERT with CE achieves 94 9 in terms of accuracy  The same as SST-5, we observe a slight performance drop with DL and DSC, which means that the dice loss actually works well for F1 but not for accuracy 
Ablation Studies ::: The Effect of Hyperparameters in Tversky index
As mentioned in Section SECREF10, Tversky index (TI) offers the flexibility in controlling the tradeoff between false-negatives and false-positives  In this subsection, we explore the effect of hyperparameters (i e , $\alpha $ and $\beta $) in TI to test how they manipulate the tradeoff  We conduct experiments on the Chinese OntoNotes4 0 NER dataset and English QuoRef MRC dataset to examine the influence of tradeoff between precision and recall  Experiment results are shown in Table   The highest F1 for Chinese OntoNotes4 0 is 84 67 when $\alpha $ is set to 0 6 while for QuoRef, the highest F1 is 68 44 when $\alpha $ is set to 0 4  In addition, we can observe that the performance varies a lot as $\alpha $ changes in distinct datasets, which shows that the hyperparameters $\alpha ,\beta $ play an important role in the proposed method 
Conclusion
In this paper, we alleviate the severe data imbalance issue in NLP tasks  We propose to use dice loss in replacement of the standard cross-entropy loss, which performs as a soft version of F1 score  Using dice loss can help narrow the gap between training objectives and evaluation metrics  Empirically, we show that the proposed training objective leads to significant performance boost for part-of-speech, named entity recognition, machine reading comprehension and paraphrase identification tasks 

ground_truth: One can think $(1-p_{i1})$ as a weight associated with each example, which changes as training proceeds. The intuition of changing $p_{i1}$ to $(1-p_{i1}) p_{i1}$ is to push down the weight of easy examples. For easy examples whose probability are approaching 0 or 1, $(1-p_{i1}) p_{i1}$ makes the model attach significantly less focus to them. Figure FIGREF23 gives gives an explanation from the perspective in derivative: the derivative of $\frac{(1-p)p}{1+(1-p)p}$ with respect to $p$ approaches 0 immediately after $p$ approaches 0, which means the model attends less to examples once they are correctly classified.

associates each training example with a weight in proportion to $(1-p)$, and this weight dynamically changes as training proceeds