prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. How much improvement is gained from Adversarial Reward Augmented Maximum Likelihood (ARAML)?
Context 1: Introduction
Natural text generation, as a key task in NLP, has been advanced substantially thanks to the flourish of neural models BIBREF0 , BIBREF1   Typical frameworks such as sequence-to-sequence (seq2seq) have been applied to various generation tasks, including machine translation BIBREF2 and dialogue generation BIBREF3   The standard paradigm to train such neural models is maximum likelihood estimation (MLE), which maximizes the log-likelihood of observing each word in the text given the ground-truth proceeding context BIBREF4  
Although widely used, MLE suffers from the exposure bias problem BIBREF5 , BIBREF6 : during test, the model sequentially predicts the next word conditioned on its previous generated words while during training conditioned on ground-truth words  To tackle this problem, generative adversarial networks (GAN) with reinforcement learning (RL) training approaches have been introduced to text generation tasks BIBREF7 , BIBREF8 , BIBREF9 , BIBREF10 , BIBREF11 , BIBREF12 , BIBREF13 , where the discriminator is trained to distinguish real and generated text samples to provide reward signals for the generator, and the generator is optimized via policy gradient BIBREF7  
However, recent studies have shown that potential issues of training GANs on discrete data are more severe than exposure bias BIBREF14 , BIBREF15   One of the fundamental issues when generating discrete text samples with GANs is training instability  Updating the generator with policy gradient always leads to an unstable training process because it's difficult for the generator to derive positive and stable reward signals from the discriminator even with careful pre-training BIBREF8   As a result, the generator gets lost due to the high variance of reward signals and the training process may finally collapse BIBREF16  
In this paper, we propose a novel adversarial training framework called Adversarial Reward Augmented Maximum Likelihood (ARAML) to deal with the instability issue of training GANs for text generation  At each iteration of adversarial training, we first train the discriminator to assign higher rewards to real data than to generated samples  Then, inspired by reward augmented maximum likelihood (RAML) BIBREF17 , the generator is updated on the samples acquired from a stationary distribution with maximum likelihood estimation (MLE), weighted by the discriminator's rewards  This stationary distribution is designed to guarantee that training samples are surrounding the real data, thus the exploration space of our generator is indeed restricted by the MLE training objective, resulting in more stable training  Compared to other text GANs with RL training techniques, our framework acquires samples from the stationary distribution rather than the generator's distribution, and uses RAML training paradigm to optimize the generator instead of policy gradient  Our contributions are mainly as follows:
Related Work
Recently, text generation has been widely studied with neural models trained with maximum likelihood estimation BIBREF4   However, MLE tends to generate universal text BIBREF18   Various methods have been proposed to enhance the generation quality by refining the objective function BIBREF18 , BIBREF19 or modifying the generation distribution with external information like topic BIBREF20 , sentence type BIBREF21 , emotion BIBREF22 and knowledge BIBREF23  
As mentioned above, MLE suffers from the exposure bias problem BIBREF5 , BIBREF6   Thus, reinforcement learning has been introduced to text generation tasks such as policy gradient BIBREF6 and actor-critic BIBREF24   BIBREF17 proposed an efficient and stable approach called Reward Augmented Maximum Likelihood (RAML), which connects the log-likelihood and expected rewards to incorporate MLE training objective into RL framework 
Since some text generation tasks have no explicit metrics to be directly optimized, adversarial training has been applied to generating discrete text samples with a discriminator to learn a proper reward  For instance, SeqGAN BIBREF7 devised a discriminator to distinguish the real data and generated samples, and a generator to maximize the reward from the discriminator via policy gradient  Other variants of GANs have been proposed to improve the generator or the discriminator  To improve the generator, MaliGAN BIBREF8 developed a normalized maximum likelihood optimization target for the generator to stably model the discrete sequences  LeakGAN BIBREF11 guided the generator with reward signals leaked from the discriminator at all generation steps to deal with long text generation task  MaskGAN BIBREF10 employed an actor-critic architecture to make the generator fill in missing text conditioned on the surrounding context, which is expected to mitigate the problem of mode collapse  As for the discriminator, RankGAN BIBREF9 replaced traditional discriminator with a ranker to learn the relative ranking information between the real texts and generated ones  Inverse reinforcement learning BIBREF12 used a trainable reward approximator as the discriminator to provide dense reward signals at each generation step  DPGAN BIBREF13 introduced a language model based discriminator and regarded cross-entropy as rewards to promote the diversity of generation results 
The most similar works to our model are RAML BIBREF17 and MaliGAN BIBREF8 : 1) Compared with RAML, our model adds a discriminator to learn the reward signals instead of choosing existing metrics as rewards  We believe that our model can adapt to various text generation tasks, particularly those without explicit evaluation metrics  2) Unlike MaliGAN, we acquire samples from a fixed distribution near the real data rather than the generator's distribution, which is expected to make the training process more stable 
Task Definition and Model Overview
Text generation can be formulated as follows: given the real data distribution INLINEFORM0 , the task is to train a generative model INLINEFORM1 where INLINEFORM2 can fit INLINEFORM3 well  In this formulation, INLINEFORM4 and INLINEFORM5 denotes a word in the vocabulary INLINEFORM6  
Figure FIGREF3 shows the overview of our model ARAML  This adversarial training framework consists of two phases: 1) The discriminator is trained to assign higher rewards to real data than to generated data  2) The generator is trained on the samples acquired from a stationary distribution with reward augmented MLE training objective  This training paradigm of the generator indeed constrains the search space with the MLE training objective, which alleviates the issue of unstable training 
Discriminator
The discriminator INLINEFORM0 aims to distinguish real data and generated data like other GANs  Inspired by Least-Square GAN BIBREF25 , we devise the loss function as follows: DISPLAYFORM0
This loss function forces the discriminator to assign higher rewards to real data than to generated data, so the discriminator can learn to provide more proper rewards as the training proceeds 
Generator
The training objective of our generator INLINEFORM0 is derived from the objective of other discrete GANs with RL training method: DISPLAYFORM0
where INLINEFORM0 denotes the rewards from the discriminator INLINEFORM1 and the entropy regularized term INLINEFORM2 encourages INLINEFORM3 to generate diverse text samples  INLINEFORM4 is a temperature hyper-parameter to balance these two terms 
As mentioned above, discrete GANs suffer from the instability issue due to policy gradient, thus they are consequently difficult to train  Inspired by RAML BIBREF17 , we introduce an exponential payoff distribution INLINEFORM0 to connect RL loss with RAML loss: DISPLAYFORM0
where INLINEFORM0   Thus, we can rewrite INLINEFORM1 with INLINEFORM2 and INLINEFORM3 as follows: DISPLAYFORM0
Following RAML, we remove the constant term and optimize the KL divergence in the opposite direction: DISPLAYFORM0
where INLINEFORM0 is a constant in the training phase of the generator  It has been proved that INLINEFORM1 and INLINEFORM2 are equivalent up to their first order Taylor approximations, and they have the same global optimum BIBREF17   INLINEFORM3 can be trained in a MLE-like fashion but sampling from the distribution INLINEFORM4 is intractable in the adversarial setting, because INLINEFORM5 varies with the discriminator INLINEFORM6   Thus, we introduce importance sampling to separate sampling process from INLINEFORM7 and obtain the final loss function: DISPLAYFORM0
where INLINEFORM0 denotes a stationary distribution and INLINEFORM1   To optimize this loss function, we first construct the fixed distribution INLINEFORM2 to get samples, and devise the proper reward function INLINEFORM3 to train the generator in a stable and effective way 
We construct the distribution INLINEFORM0 based on INLINEFORM1 : DISPLAYFORM0
In this way, INLINEFORM0 can be designed to guarantee that INLINEFORM1 is near INLINEFORM2 , leading to a more stable training process  To obtain a new sample INLINEFORM3 from a real data sample INLINEFORM4 , we can design three steps which contain sampling an edit distance INLINEFORM5 , the positions INLINEFORM6 for substitution and the new words INLINEFORM7 filled into the corresponding positions  Thus, INLINEFORM8 can be decomposed into three terms: DISPLAYFORM0
The first step is to sample an edit distance based on a real data sample INLINEFORM0 , where INLINEFORM1 is a sequence of length INLINEFORM2   The number of sentences which have the edit distance INLINEFORM3 to some input sentence can be computed approximately as below: DISPLAYFORM0
where INLINEFORM0 denotes the number of sentences which have an edit distance INLINEFORM1 to a sentence of length INLINEFORM2 , and INLINEFORM3 indicates the size of vocabulary  We then follow BIBREF17 to re-scale the counts by INLINEFORM4 and do normalization, so that we can sample an edit distance INLINEFORM5 from: DISPLAYFORM0
where INLINEFORM0 , as a temperature hyper-parameter, restricts the search space surrounding the original sentence  Larger INLINEFORM1 brings more samples with long edit distances 
The next step is to select positions for substitution based on the sampled edit distance INLINEFORM0   Intuitively, we can randomly choose INLINEFORM1 distinct positions in INLINEFORM2 to be replaced by new words  The probability of choosing the position INLINEFORM3 is calculated as follows: DISPLAYFORM0
Following this sampling strategy, we can obtain the position set INLINEFORM0   This strategy approximately guarantees that the edit distance between a new sentence and the original sentence is INLINEFORM1  
At the final step, our model determines new words for substitution at each sampled position INLINEFORM0   We can formulate this sampling process from the original sequence INLINEFORM1 to a new sample INLINEFORM2 as a sequential transition INLINEFORM3   At each step from INLINEFORM4 to INLINEFORM5 INLINEFORM6 , we first sample a new word INLINEFORM7 from the distribution INLINEFORM8 , then replace the old word at position INLINEFORM9 of INLINEFORM10 to obtain INLINEFORM11   The whole sampling process can be decomposed as follows: DISPLAYFORM0
There are two common sampling strategies to model INLINEFORM0 , i e  random sampling and constrained sampling  Random sampling strategy samples a new word INLINEFORM1 according to the uniform distribution over the vocabulary INLINEFORM2 BIBREF17 , while constrained sampling strategy samples INLINEFORM3 to maximize the language model score of the target sentence INLINEFORM4 BIBREF26 , BIBREF27   Here, we adopt constrained sampling in our model and compare the performances of two strategies in the experiment 
We devise the reward function INLINEFORM0 according to the discriminator's output INLINEFORM1 and the stationary distribution INLINEFORM2 : DISPLAYFORM0
Intuitively, this reward function encourages the generator to generate sentences with large sampling probability and high rewards from the discriminator  Thus, the weight of samples INLINEFORM0 can be calculated as follows: DISPLAYFORM0
So far, we can successfully optimize the generator's loss INLINEFORM0 via Equation EQREF12   This training paradigm makes our generator avoid possible variances caused by policy gradient and get more stable reward signals from the discriminator, because our generator is restricted to explore the training samples near the real data 
[htb] Adversarial Reward Augmented Maximum Likelihood [1]
Total adversarial training iterations: INLINEFORM0
Steps of training generator: INLINEFORM0
Steps of training discriminator: INLINEFORM0
Pre-train the generator INLINEFORM0 with MLE loss Generate samples from INLINEFORM1 Pre-train the discriminator INLINEFORM2 via Eq ( EQREF6 ) Construct INLINEFORM3 via Eq ( EQREF14 ) - Eq ( EQREF19 ) each INLINEFORM4 each INLINEFORM5 Update INLINEFORM6 via Eq ( EQREF12 ) each INLINEFORM7 Update INLINEFORM8 via Eq ( EQREF6 )
Extension to Conditional Text Generation
We have shown our adversarial training framework for text generation tasks without an input  Actually, it can also be extended to conditional text generation tasks like dialogue generation  Given the data distribution INLINEFORM0 where INLINEFORM1 denote contexts and responses respectively, the objective function of ARAML's generator can be modified as below: DISPLAYFORM0
where INLINEFORM0 and INLINEFORM1 is trained to distinguish whether INLINEFORM2 is the true response to INLINEFORM3  
Comparison with RAML and MaliGAN
The most similar works to our framework are RAML BIBREF17 and MaliGAN BIBREF8   The main difference among them is the training objective of their generators  We have shown different objective functions in Table TABREF26   For comparison, we use the form with no input for all the three models 
Our model is greatly inspired by RAML, which gets samples from a non-parametric distribution INLINEFORM0 constructed based on a specific reward  Compared to RAML, our reward comes from a learnable discriminator which varies as the adversarial training proceeds rather than a specific reward function  This difference equips our framework with the ability to adapt to the text generation tasks with no explicit evaluation metrics as rewards 
Our model is also similar to MaliGAN, which gets samples from the generator's distribution  In MaliGAN's training objective, INLINEFORM0 also indicates the generator's distribution but it's used in the sampling phase and fixed at each optimization step  The weight of samples INLINEFORM1   Different from our model, MaliGAN acquires samples from the generator's distribution INLINEFORM2 , which usually brings samples with low rewards even with careful pre-training for the generator, leading to training instability  Instead, our framework gets samples from a stationary distribution INLINEFORM3 around real data, thus our training process is more stable 
Datasets
We evaluated ARAML on three datasets: COCO image caption dataset BIBREF28 , EMNLP2017 WMT dataset and WeiboDial single-turn dialogue dataset BIBREF29   COCO and EMNLP2017 WMT are the common benchmarks with no input to evaluate the performance of discrete GANs, and we followed the existing works to preprocess these datasets BIBREF12 , BIBREF11   WeiboDial, as a dialogue dataset, was applied to test the performance of our model with input trigger  We simply removed post-response pairs containing low-frequency words and randomly selected a subset for our training/test set  The statistics of three datasets are presented in Table TABREF28  
Baselines
We compared our model with MLE, RL and GAN baselines  Since COCO and EMNLP2017 WMT don't have input while WeiboDial regards posts as input, we chose the following baselines respectively:
MLE: a RNN model trained with MLE objective BIBREF4   Its extension, Seq2Seq, can work on the dialogue dataset BIBREF2  
SeqGAN: The first text GAN model that updates the generator with policy gradient based on the rewards from the discriminator BIBREF7  
LeakGAN: A variant of SeqGAN that provides rewards based on the leaked information of the discriminator for the generator BIBREF11  
MaliGAN: A variant of SeqGAN that optimizes the generator with a normalized maximum likelihood objective BIBREF8  
IRL: This inverse reinforcement learning method replaces the discriminator with a reward approximator to provide dense rewards BIBREF12  
RAML: A RL approach to incorporate MLE objective into RL training framework, which regards BLEU as rewards BIBREF17  
DialogGAN: An extension of SeqGAN tuned to dialogue generation task with MLE objective added to the adversarial objective BIBREF16  
DPGAN: A variant of DialogGAN which uses a language model based discriminator and regards cross-entropy as rewards BIBREF13  
Note that MLE, SeqGAN, LeakGAN, MaliGAN and IRL are the baselines on COCO and EMNLP2017 WMT, while MLE, RAML, DialogGAN, and DPGAN on WeiboDial  The original codes are used to test the baselines 
Implementation Details
The implementation details of our model are shown in Table TABREF31   For COCO / EMNLP2017, the generator is a LSTM unit BIBREF30 with 128 cells, and the discriminator is implemented based on BIBREF7   For WeiboDial, the generator is an encoder-decoder structure with attention mechanism, where both the encoder and the decoder consist of a two-layer GRU BIBREF31 with 128 cells  The discriminator is implemented based on BIBREF32   The language model used in the constrained sampling of ARAML is implemented in the same setting as the generators, and is pre-trained on the training set of each dataset  The codes and the datasets are available at https://github com/kepei1106/ARAML 
As for the details of the baselines, the generators of all the baselines except LeakGAN are the same as ours  Note that the generator of LeakGAN consists of a hierarchical LSTM unit, thus we followed the implementation in the original paper  In terms of the differences, the discriminators of GAN baselines are implemented based on the original papers  Other hyper-parameters of baselines including batch size, learning rate, and pre-training epochs, were set based on the original codes, because the convergence of baselines is sensitive to these hyper-parameters 
Language Generation on COCO and EMNLP2017 WMT
We adopted forward/reverse perplexity BIBREF33 and Self-BLEU BIBREF34 to evaluate the quality of generated texts  Forward perplexity (PPL-F) indicates the perplexity on the generated data provided by a language model trained on real data to measure the fluency of generated samples  Reverse perplexity (PPL-R) switches the roles of generated data and real data to reflect the discrepancy between the generated distribution and the data distribution  Self-BLEU (S-BLEU) regards each sentence in the generated collection as hypothesis and the others as reference to obtain BLEU scores, which evaluates the diversity of generated results 
Results are shown in Table TABREF33   LeakGAN performs best on forward perplexity because it can generate more fluent samples  As for reverse perplexity, our model ARAML beats other baselines, showing that our model can fit the data distribution better  Other GANs, particularly LeakGAN, obtain high reverse perplexity due to mode collapse BIBREF12 , thus they only capture limited fluent expressions, resulting in large discrepancy between the generated distribution and data distribution  ARAML also outperforms the baselines in terms of Self-BLEU, indicating that our model doesn't fall into mode collapse with the help of the MLE training objective and has the ability to generate more diverse sentences 
We also provide standard deviation of each metric in Table TABREF33 , reflecting the stability of each model's performance  Our model ARAML nearly achieves the smallest standard deviation in all the metrics, indicating that our framework outperforms policy gradient in the stability of adversarial training 
Dialogue Generation on WeiboDial
Dialogue evaluation is an open problem and existing works have found that automatic metrics have low correlation to human evaluation BIBREF35 , BIBREF36 , BIBREF37   Thus, we resorted to manual evaluation to assess the generation quality on WeiboDial  We randomly sampled 200 posts from the test set and collected the generated results from all the models  For each pair of responses (one from ARAML and the other from a baseline, given the same input post), five annotators were hired to label which response is better (i e  win, lose or tie) in terms of grammaticality (whether a response itself is grammatical and logical) and relevance (whether a response is appropriate and relevant to the post)  The two metrics were evaluated independently 
The evaluation results are shown in Table TABREF35   To measure the inter-annotator agreement, we calculated Fleiss' kappa BIBREF38 for each pair-wise comparison where results show moderate agreement ( INLINEFORM0 )  We also conducted sign test to check the significance of the differences 
As shown in Table TABREF35 , ARAML performs significantly better than other baselines in all the cases  This result indicates that the samples surrounding true responses provide stable rewards for the generator, and stable RAML training paradigm significantly enhances the performance in both metrics 
Further Analysis on Stability
To verify the training stability, we conducted experiments on COCO many times and chose the best 5 trials for SeqGAN, LeakGAN, IRL, MaliGAN and ARAML, respectively  Then, we presented the forward/reverse perplexity in the training process in Figure FIGREF38   We can see that our model with smaller standard deviation is more stable than other GAN baselines in both metrics  Although LeakGAN reaches the best forward perplexity, its standard deviation is extremely large and it performs badly in reverse perplexity, indicating that it generates limited expressions that are grammatical yet divergent from the data distribution 
Ablation Study
The temperature INLINEFORM0 controls the search space surrounding the real data as we analyze in Section UID13   To investigate its impact on the performance of our model, we fixed all the other hyper-parameters and test ARAML with different temperatures on COCO 
The experimental results are shown in Figure FIGREF41   We can see that as the temperature becomes larger, forward perplexity increases gradually while Self-BLEU decreases  As mentioned in Section UID13 , large temperatures encourage our generator to explore the samples that are distant from real data distribution, thus the diversity of generated results will be improved  However, these samples distant from the data distribution are more likely to be poor in fluency, leading to worse forward perplexity  Reverse perplexity is influenced by both generation quality and diversity, so the correlation between temperature and reverse perplexity is not intuitive  We can observe that the model with INLINEFORM0 reaches the best reverse perplexity 
We have mentioned two common sampling strategies in Section UID13 , i e  random sampling and constrained sampling  To analyze their impact, we keep all the model structures and hyper-parameters fixed and test ARAML with these two strategies on COCO 
Table TABREF45 shows the results  It's obvious that random sampling hurts the model performance except Self-BLEU-1, because it indeed allows low-quality samples available to the generator  Exploring these samples degrades the quality and diversity of generated results  Despite the worse performance on automatic metrics, random sampling doesn't affect the training stability of our framework  The standard deviation of ARAML-R is still smaller than other GAN baselines 
Case Study
Table TABREF47 presents the examples generated by the models on COCO  We can find that other baselines suffer from grammatical errors (e g  “in front of flying her kite" from MLE), repetitive expressions (e g  “A group of people" from IRL) and incoherent statements (e g  “A group of people sitting on a cell phone” from IRL)  By contrast, our model performs well in these sentences and has the ability to generate grammatical and coherent results 
Table TABREF48 shows the generated examples on WeiboDial  It's obvious that other baselines don't capture the topic word “late" in the post, thus generate irrelevant responses  ARAML can provide a response that is grammatical and closely relevant to the post 
Conclusion
We propose a novel adversarial training framework to deal with the instability problem of current GANs for text generation  To address the instability issue caused by policy gradient, we incorporate RAML into the advesarial training paradigm to make our generator acquire stable rewards  Experiments show that our model performs better than several state-of-the-art GAN baselines with lower training variance, yet producing better performance on three text generation tasks 
Acknowledgments
This work was supported by the National Science Foundation of China (Grant No  61936010/61876096) and the National Key R&D Program of China (Grant No  2018YFC0830200)  We would like to thank THUNUS NExT Joint-Lab for the support 
Context 3: Introduction
There is a recent spark of interest in the task of Question Answering (QA) over unstructured textual data, also referred to as Machine Reading Comprehension (MRC)  This is mostly due to wide-spread success of advances in various facets of deep learning related research, such as novel architectures BIBREF0, BIBREF1 that allow for efficient optimisation of neural networks consisting of multiple layers, hardware designed for deep learning purposes and software frameworks BIBREF2, BIBREF3 that allow efficient development and testing of novel approaches  These factors enable researchers to produce models that are pre-trained on large scale corpora and provide contextualised word representations BIBREF4 that are shown to be a vital component towards solutions for a variety of natural language understanding tasks, including MRC BIBREF5  Another important factor that led to the recent success in MRC-related tasks is the widespread availability of various large datasets, e g , SQuAD BIBREF6, that provide sufficient examples for optimising statistical models  The combination of these factors yields notable results, even surpassing human performance BIBREF7 
MRC is a generic task format that can be used to probe for various natural language understanding capabilities BIBREF8  Therefore it is crucially important to establish a rigorous evaluation methodology in order to be able to draw reliable conclusions from conducted experiments  While increasing effort is put into the evaluation of novel architectures, such as keeping the evaluation data from public access to prevent unintentional overfitting to test data, performing ablation and error studies and introducing novel metrics BIBREF9, surprisingly little is done to establish the quality of the data itself  Additionally, recent research arrived at worrisome findings: the data of those gold standards, which is usually gathered involving a crowd-sourcing step, suffers from flaws in design BIBREF10 or contains overly specific keywords BIBREF11  Furthermore, these gold standards contain “annotation artefacts”, cues that lead models into focusing on superficial aspects of text, such as lexical overlap and word order, instead of actual language understanding BIBREF12, BIBREF13  These weaknesses cast some doubt on whether the data can reliably evaluate the reading comprehension performance of the models they evaluate, i e  if the models are indeed being assessed for their capability to read 
Figure FIGREF3 shows an example from HotpotQA BIBREF14, a dataset that exhibits the last kind of weakness mentioned above, i e , the presence of unique keywords in both the question and the passage (in close proximity to the expected answer) 
An evaluation methodology is vital to the fine-grained understanding of challenges associated with a single gold standard, in order to understand in greater detail which capabilities of MRC models it evaluates  More importantly, it allows to draw comparisons between multiple gold standards and between the results of respective state-of-the-art models that are evaluated on them 
In this work, we take a step back and propose a framework to systematically analyse MRC evaluation data, typically a set of questions and expected answers to be derived from accompanying passages  Concretely, we introduce a methodology to categorise the linguistic complexity of the textual data and the reasoning and potential external knowledge required to obtain the expected answer  Additionally we propose to take a closer look at the factual correctness of the expected answers, a quality dimension that appears under-explored in literature 
We demonstrate the usefulness of the proposed framework by applying it to precisely describe and compare six contemporary MRC datasets  Our findings reveal concerns about their factual correctness, the presence of lexical cues that simplify the task of reading comprehension and the lack of semantic altering grammatical modifiers  We release the raw data comprised of 300 paragraphs, questions and answers richly annotated under the proposed framework as a resource for researchers developing natural language understanding models and datasets to utilise further 
To the best of our knowledge this is the first attempt to introduce a common evaluation methodology for MRC gold standards and the first across-the-board qualitative evaluation of MRC datasets with respect to the proposed categories 
Framework for MRC Gold Standard Analysis ::: Problem definition
We define the task of machine reading comprehension, the target application of the proposed methodology as follows: Given a paragraph $P$ that consists of tokens (words) $p_1, \ldots , p_{n_P}$ and a question $Q$ that consists of tokens $q_1 \ldots q_{n_Q}$, the goal is to retrieve an answer $A$ with tokens $a_1 \ldots a_{n_A}$  $A$ is commonly constrained to be one of the following cases BIBREF15, illustrated in Figure FIGREF9:
Multiple choice, where the goal is to predict $A$ from a given set of choices $\mathcal {A}$ 
Cloze-style, where $S$ is a sentence, and $A$ and $Q$ are obtained by removing a sequence of words such that $Q = S - A$  The task is to fill in the resulting gap in $Q$ with the expected answer $A$ to form $S$ 
Span, where is a continuous subsequence of tokens from the paragraph ($A \subseteq P$)  Flavours include multiple spans as the correct answer or $A \subseteq Q$ 
Free form, where $A$ is an unconstrained natural language string 
A gold standard $G$ is composed of $m$ entries $(Q_i, A_i, P_i)_{i\in \lbrace 1,\ldots , m\rbrace }$ 
The performance of an approach is established by comparing its answer predictions $A^*_{i}$ on the given input $(Q_i, T_i)$ (and $\mathcal {A}_i$ for the multiple choice setting) against the expected answer $A_i$ for all $i\in \lbrace 1,\ldots , m\rbrace $ under a performance metric  Typical performance metrics are exact match (EM) or accuracy, i e  the percentage of exactly predicted answers, and the F1 score – the harmonic mean between the precision and the recall of the predicted tokens compared to expected answer tokens  The overall F1 score can either be computed by averaging the F1 scores for every instance or by first averaging the precision and recall and then computing the F1 score from those averages (macro F1)  Free-text answers, meanwhile, are evaluated by means of text generation and summarisation metrics such as BLEU BIBREF16 or ROUGE-L BIBREF17 
Framework for MRC Gold Standard Analysis ::: Dimensions of Interest
In this section we describe a methodology to categorise gold standards according to linguistic complexity, required reasoning and background knowledge, and their factual correctness  Specifically, we use those dimensions as high-level categories of a qualitative annotation schema for annotating question, expected answer and the corresponding context  We further enrich the qualitative annotations by a metric based on lexical cues in order to approximate a lower bound for the complexity of the reading comprehension task  By sampling entries from each gold standard and annotating them, we obtain measurable results and thus are able to make observations about the challenges present in that gold standard data 
Framework for MRC Gold Standard Analysis ::: Dimensions of Interest ::: Problem setting
We are interested in different types of the expected answer  We differentiate between Span, where an answer is a continuous span taken from the passage, Paraphrasing, where the answer is a paraphrase of a text span, Unanswerable, where there is no answer present in the context, and Generated, if it does not fall into any of the other categories  It is not sufficient for an answer to restate the question or combine multiple Span or Paraphrasing answers to be annotated as Generated  It is worth mentioning that we focus our investigations on answerable questions  For a complementary qualitative analysis that categorises unanswerable questions, the reader is referred to Yatskar2019 
Furthermore, we mark a sentence as Supporting Fact if it contains evidence required to produce the expected answer, as they are used further in the complexity analysis 
Framework for MRC Gold Standard Analysis ::: Dimensions of Interest ::: Factual Correctness
An important factor for the quality of a benchmark is its factual correctness, because on the one hand, the presence of factually wrong or debatable examples introduces an upper bound for the achievable performance of models on those gold standards  On the other hand, it is hard to draw conclusions about the correctness of answers produced by a model that is evaluated on partially incorrect data 
One way by which developers of modern crowd-sourced gold standards ensure quality is by having the same entry annotated by multiple workers BIBREF18 and keeping only those with high agreement  We investigate whether this method is enough to establish a sound ground truth answer that is unambiguously correct  Concretely we annotate an answer as Debatable when the passage features multiple plausible answers, when multiple expected answers contradict each other, or an answer is not specific enough with respect to the question and a more specific answer is present  We annotate an answer as Wrong when it is factually wrong and a correct answer is present in the context 
Framework for MRC Gold Standard Analysis ::: Dimensions of Interest ::: Required Reasoning
It is important to understand what types of reasoning the benchmark evaluates, in order to be able to accredit various reasoning capabilities to the models it evaluates  Our proposed reasoning categories are inspired by those found in scientific question answering literature BIBREF19, BIBREF20, as research in this area focuses on understanding the required reasoning capabilities  We include reasoning about the Temporal succession of events, Spatial reasoning about directions and environment, and Causal reasoning about the cause-effect relationship between events  We further annotate (multiple-choice) answers that can only be answered By Exclusion of every other alternative 
We further extend the reasoning categories by operational logic, similar to those required in semantic parsing tasks BIBREF21, as solving those tasks typically requires “multi-hop” reasoning BIBREF14, BIBREF22  When an answer can only be obtained by combining information from different sentences joined by mentioning a common entity, concept, date, fact or event (from here on called entity), we annotate it as Bridge  We further annotate the cases, when the answer is a concrete entity that satisfies a Constraint specified in the question, when it is required to draw a Comparison of multiple entities' properties or when the expected answer is an Intersection of their properties (e g  “What do Person A and Person B have in common?”)
We are interested in the linguistic reasoning capabilities probed by a gold standard, therefore we include the appropriate category used by Wang2019  Specifically, we annotate occurrences that require understanding of Negation, Quantifiers (such as “every”, “some”, or “all”), Conditional (“if    then”) statements and the logical implications of Con-/Disjunction (i e  “and” and “or”) in order to derive the expected answer 
Finally, we investigate whether arithmetic reasoning requirements emerge in MRC gold standards as this can probe for reasoning that is not evaluated by simple answer retrieval BIBREF23  To this end, we annotate the presence of of Addition and Subtraction, answers that require Ordering of numerical values, Counting and Other occurrences of simple mathematical operations 
An example can exhibit multiple forms of reasoning  Notably, we do not annotate any of the categories mentioned above if the expected answer is directly stated in the passage  For example, if the question asks “How many total points were scored in the game?” and the passage contains a sentence similar to “The total score of the game was 51 points”, it does not require any reasoning, in which case we annotate it as Retrieval 
Framework for MRC Gold Standard Analysis ::: Dimensions of Interest ::: Knowledge
Worthwhile knowing is whether the information presented in the context is sufficient to answer the question, as there is an increase of benchmarks deliberately designed to probe a model's reliance on some sort of background knowledge BIBREF24  We seek to categorise the type of knowledge required  Similar to Wang2019, on the one hand we annotate the reliance on factual knowledge, that is (Geo)political/Legal, Cultural/Historic, Technical/Scientific and Other Domain Specific knowledge about the world that can be expressed as a set of facts  On the other hand, we denote Intuitive knowledge requirements, which is challenging to express as a set of facts, such as the knowledge that a parenthetic numerical expression next to a person's name in a biography usually denotes his life span 
Framework for MRC Gold Standard Analysis ::: Dimensions of Interest ::: Linguistic Complexity
Another dimension of interest is the evaluation of various linguistic capabilities of MRC models BIBREF25, BIBREF26, BIBREF27  We aim to establish which linguistic phenomena are probed by gold standards and to which degree  To that end, we draw inspiration from the annotation schema used by Wang2019, and adapt it around lexical semantics and syntax 
More specifically, we annotate features that introduce variance between the supporting facts and the question  With regard to lexical semantics, we focus on the use of redundant words that do not alter the meaning of a sentence for the task of retrieving the expected answer (Redundancy), requirements on the understanding of words' semantic fields (Lexical Entailment) and the use of Synonyms and Paraphrases with respect to the question wording  Furthermore we annotate cases where supporting facts contain Abbreviations of concepts introduced in the question (and vice versa) and when a Dative case substitutes the use of a preposition (e g  “I bought her a gift” vs “I bought a gift for her”)  Regarding syntax, we annotate changes from passive to active Voice, the substitution of a Genitive case with a preposition (e g  “of”) and changes from nominal to verbal style and vice versa (Nominalisation) 
We recognise features that add ambiguity to the supporting facts, for example when information is only expressed implicitly by using an Ellipsis  As opposed to redundant words, we annotate Restrictivity and Factivity modifiers, words and phrases whose presence does change the meaning of a sentence with regard to the expected answer, and occurrences of intra- or inter-sentence Coreference in supporting facts (that is relevant to the question)  Lastly, we mark ambiguous syntactic features, when their resolution is required in order to obtain the answer  Concretely, we mark argument collection with con- and disjunctions (Listing) and ambiguous Prepositions, Coordination Scope and Relative clauses/Adverbial phrases/Appositions 
Framework for MRC Gold Standard Analysis ::: Dimensions of Interest ::: Complexity
Finally, we want to approximate the presence of lexical cues that might simplify the reading required in order to arrive at the answer  Quantifying this allows for more reliable statements about and comparison of the complexity of gold standards, particularly regarding the evaluation of comprehension that goes beyond simple lexical matching  We propose the use of coarse metrics based on lexical overlap between question and context sentences  Intuitively, we aim to quantify how much supporting facts “stand out” from their surrounding passage context  This can be used as proxy for the capability to retrieve the answer BIBREF10  Specifically, we measure (i) the number of words jointly occurring in a question and a sentence, (ii) the length of the longest n-gram shared by question and sentence and (iii) whether a word or n-gram from the question uniquely appears in a sentence 
The resulting taxonomy of the framework is shown in Figure FIGREF10  The full catalogue of features, their description, detailed annotation guideline as well as illustrating examples can be found in Appendix  
Application of the Framework ::: Candidate Datasets
We select contemporary MRC benchmarks to represent all four commonly used problem definitions BIBREF15  In selecting relevant datasets, we do not consider those that are considered “solved”, i e  where the state of the art performance surpasses human performance, as is the case with SQuAD BIBREF28, BIBREF7  Concretely, we selected gold standards that fit our problem definition and were published in the years 2016 to 2019, have at least $(2019 - publication\ year) \times 20$ citations, and bucket them according to the answer selection styles as described in Section SECREF4 We randomly draw one from each bucket and add two randomly drawn datasets from the candidate pool  This leaves us with the datasets described in Table TABREF19  For a more detailed description, we refer to Appendix  
Application of the Framework ::: Annotation Task
We randomly select 50 distinct question, answer and passage triples from the publicly available development sets of the described datasets  Training, development and the (hidden) test set are drawn from the same distribution defined by the data collection method of the respective dataset  For those collections that contain multiple questions over a single passage, we ensure that we are sampling unique paragraphs in order to increase the variety of investigated texts 
The samples were annotated by the first author of this paper, using the proposed schema  In order to validate our findings, we further take 20% of the annotated samples and present them to a second annotator (second author)  Since at its core, the annotation is a multi-label task, we report the inter-annotator agreement by computing the (micro-averaged) F1 score, where we treat the first annotator's labels as gold  Table TABREF21 reports the agreement scores, the overall (micro) average F1 score of the annotations is 0 82, which means that on average, more than two thirds of the overall annotated labels were agreed on by both annotators  We deem this satisfactory, given the complexity of the annotation schema 
Application of the Framework ::: Qualitative Analysis
We present a concise view of the annotation results in Figure FIGREF23  The full annotation results can be found in Appendix   We centre our discussion around the following main points:
Application of the Framework ::: Qualitative Analysis ::: Linguistic Features
As observed in Figure FIGREF23 the gold standards feature a high degree of Redundancy, peaking at 76% of the annotated HotpotQA samples and synonyms and paraphrases (labelled Synonym), with ReCoRd samples containing 58% of them, likely to be attributed to the elaborating type of discourse of the dataset sources (encyclopedia and newswire)  This is, however, not surprising, as it is fairly well understood in the literature that current state-of-the-art models perform well on distinguishing relevant words and phrases from redundant ones BIBREF32  Additionally, the representational capability of synonym relationships of word embeddings has been investigated and is well known BIBREF33  Finally, we observe the presence of syntactic features, such as ambiguous relative clauses, appositions and adverbial phrases, (RelAdvApp 40% in HotpotQA and ReCoRd) and those introducing variance, concretely switching between verbal and nominal styles (e g  Nominalisation 10% in HotpotQA) and from passive to active voice (Voice, 8% in HotpotQA) 
Syntactic features contributing to variety and ambiguity that we did not observe in our samples are the exploitation of verb symmetry, the use of dative and genitive cases or ambiguous prepositions and coordination scope (respectively Symmetry, Dative, Genitive, Prepositions, Scope)  Therefore we cannot establish whether models are capable of dealing with those features by evaluating them on those gold standards 
Application of the Framework ::: Qualitative Analysis ::: Factual Correctness
We identify three common sources that surface in different problems regarding an answer's factual correctness, as reported in Figure FIGREF23 and illustrate their instantiations in Table TABREF31:
Design Constraints: Choosing the task design and the data collection method introduces some constraints that lead to factually debatable examples  For example, a span might have been arbitrarily selected from multiple spans that potentially answer a question, but only a single continuous answer span per question is allowed by design, as observed in the NewsQA and MsMarco samples (32% and 34% examples annotated as Debatable with 16% and 53% thereof exhibiting arbitrary selection, respectively)  Sometimes, when additional passages are added after the annotation step, they can by chance contain passages that answer the question more precisely than the original span, as seen in HotpotQA (16% Debatable samples, 25% of them due to arbitrary selection)  In the case of MultiRC it appears to be inconsistent, whether multiple correct answer choices are expected to be correct in isolation or in conjunction (28% Debatable with 29% of them exhibiting this problem)  This might provide an explanation to its relatively weak human baseline performance of 84% F1 score BIBREF31 
Weak Quality assurance: When the (typically crowd-sourced) annotations are not appropriately validated, incorrect examples will find their way into the gold standards  This typically results in factually wrong expected answers (i e  when a more correct answer is present in the context) or a question is expected to be Unanswerable, but is actually answerable from the provided context  The latter is observed in MsMarco (83% of examples annotated as Wrong) and NewsQA, where 60% of the examples annotated as Wrong are Unanswerable with an answer present 
Arbitrary Precision: There appears to be no clear guideline on how precise the answer is expected to be, when the passage expresses the answer in varying granularities  We annotated instances as Debatable when the expected answer was not the most precise given the context (44% and 29% of Debatable instances in NewsQA and MultiRC, respectively) 
Application of the Framework ::: Qualitative Analysis ::: Semantics-altering grammatical modifiers
We took interest in whether any of the benchmarks contain what we call distracting lexical features (or distractors): grammatical modifiers that alter the semantics of a sentence for the final task of answering the given question while preserving a similar lexical form  An example of such features are cues for (double) Negation (e g , “no”, “not”), which when introduced in a sentence, reverse its meaning  Other examples include modifiers denoting Restrictivity, Factivity and Reasoning (such as Monotonicity and Conditional cues)  Examples of question-answer pairs containing a distractor are shown in Table FIGREF37 
We posit that the presence of such distractors would allow for evaluating reading comprehension beyond potential simple word matching  However, we observe no presence of such features in the benchmarks (beyond Negation in DROP, ReCoRd and HotpotQA, with 4%, 4% and 2% respectively)  This results in gold standards that clearly express the evidence required to obtain the answer, lacking more challenging, i e , distracting, sentences that can assess whether a model can truly understand meaning 
Application of the Framework ::: Qualitative Analysis ::: Other
In the Figure FIGREF23 we observe that Operational and Arithmetic reasoning moderately (6% to 8% combined) appears “in the wild”, i e  when not enforced by the data design as is the case with HotpotQA (80% Operations combined) or DROP (68% Arithmetic combined)  Causal reasoning is (exclusively) present in MultiRC (32%), whereas Temporal and Spatial reasoning requirements seem to not naturally emerge in gold standards  In ReCoRd, a fraction of 38% questions can only be answered By Exclusion of every other candidate, due to the design choice of allowing questions where the required information to answer them is not fully expressed in the accompanying paragraph 
Therefore, it is also a little surprising to observe that ReCoRd requires external resources with regard to knowledge, as seen in Figure FIGREF23  MultiRC requires technical or more precisely basic scientific knowledge (6% Technical/Scientific), as a portion of paragraphs is extracted from elementary school science textbooks BIBREF31  Other benchmarks moderately probe for factual knowledge (0% to 4% across all categories), while Intuitive knowledge is required to derive answers in each gold standard 
It is also worth pointing out, as done in Figure FIGREF23, that although MultiRC and MsMarco are not modelled as a span selection problem, their samples still contain 50% and 66% of answers that are directly taken from the context  DROP contains the biggest fraction of generated answers (60%), due to the requirement of arithmetic operations 
To conclude our analysis, we observe similar distributions of linguistic features and reasoning patterns, except where there are constraints enforced by dataset design, annotation guidelines or source text choice  Furthermore, careful consideration of design choices (such as single-span answers) is required, to avoid impairing the factual correctness of datasets, as pure crowd-worker agreement seems not sufficient in multiple cases 
Application of the Framework ::: Quantitative Results ::: Lexical overlap
We used the scores assigned by our proposed set of metrics (discussed in Section SECREF11 Dimensions of Interest: Complexity) to predict the supporting facts in the gold standard samples (that we included in our manual annotation)  Concretely, we used the following five features capturing lexical overlap: (i) the number of words occurring in sentence and question, (ii) the length of the longest n-gram shared by sentence and question, whether a (iii) uni- and (iv) bigram from the question is unique to a sentence, and (v) the sentence index, as input to a logistic regression classifier  We optimised on each sample leaving one example for evaluation  We compute the average Precision, Recall and F1 score by means of leave-one-out validation with every sample entry  The averaged results after 5 runs are reported in Table TABREF41 
We observe that even by using only our five features based lexical overlap, the simple logistic regression baseline is able to separate out the supporting facts from the context to a varying degree  This is in line with the lack of semantics-altering grammatical modifiers discussed in the qualitative analysis section above  The classifier performs best on DROP (66% F1) and MultiRC (40% F1), which means that lexical cues can considerably facilitate the search for the answer in those gold standards  On MultiRC, yadav2019quick come to a similar conclusion, by using a more sophisticated approach based on overlap between question, sentence and answer choices 
Surprisingly, the classifier is able to pick up a signal from supporting facts even on data that has been pruned against lexical overlap heuristics by populating the context with additional documents that have high overlap scores with the question  This results in significantly higher scores than when guessing randomly (HotpotQA 26% F1, and MsMarco 11% F1)  We observe similar results in the case the length of the question leaves few candidates to compute overlap with $6 3$ and $7 3$ tokens on average for MsMarco and NewsQA (26% F1), compared to $16 9$ tokens on average for the remaining four dataset samples 
Finally, it is worth mentioning that although the queries in ReCoRd are explicitly independent from the passage, the linear classifier is still capable of achieving 34% F1 score in predicting the supporting facts 
However, neural networks perform significantly better than our admittedly crude baseline (e g  66% F1 for supporting facts classification on HotpotQA BIBREF14), albeit utilising more training examples, and a richer sentence representation  This facts implies that those neural models are capable of solving more challenging problems than simple “text matching” as performed by the logistic regression baseline  However, they still circumvent actual reading comprehension as the respective gold standards are of limited suitability to evaluate this BIBREF34, BIBREF35  This suggests an exciting future research direction, that is categorising the scale between text matching and reading comprehension more precisely and respectively positioning state-of-the-art models thereon 
Related Work
Although not as prominent as the research on novel architecture, there has been steady progress in critically investigating the data and evaluation aspects of NLP and machine learning in general and MRC in particular 
Related Work ::: Adversarial Evaluation
The authors of the AddSent algorithm BIBREF11 show that MRC models trained and evaluated on the SQuAD dataset pay too little attention to details that might change the semantics of a sentence, and propose a crowd-sourcing based method to generate adversary examples to exploit that weakness  This method was further adapted to be fully automated BIBREF36 and applied to different gold standards BIBREF35  Our proposed approach differs in that we aim to provide qualitative justifications for those quantitatively measured issues 
Related Work ::: Sanity Baselines
Another line of research establishes sane baselines to provide more meaningful context to the raw performance scores of evaluated models  When removing integral parts of the task formulation such as question, the textual passage or parts thereof BIBREF37 or restricting model complexity by design in order to suppress some required form of reasoning BIBREF38, models are still able to perform comparably to the state-of-the-art  This raises concerns about the perceived benchmark complexity and is related to our work in a broader sense as one of our goals is to estimate the complexity of benchmarks 
Related Work ::: Benchmark evaluation in NLP
Beyond MRC, efforts similar to ours that pursue the goal of analysing the evaluation of established datasets exist in Natural Language Inference BIBREF13, BIBREF12  Their analyses reveal the existence of biases in training and evaluation data that can be approximated with simple majority-based heuristics  Because of these biases, trained models fail to extract the semantics that are required for the correct inference  Furthermore, a fair share of work was done to reveal gender bias in coreference resolution datasets and models BIBREF39, BIBREF40, BIBREF41 
Related Work ::: Annotation Taxonomies
Finally, related to our framework are works that introduce annotation categories for gold standards evaluation  Concretely, we build our annotation framework around linguistic features that were introduced in the GLUE suite BIBREF42 and the reasoning categories introduced in the WorldTree dataset BIBREF19  A qualitative analysis complementary to ours, with focus on the unanswerability patterns in datasets that feature unanswerable questions was done by Yatskar2019 
Conclusion
In this paper, we introduce a novel framework to characterise machine reading comprehension gold standards  This framework has potential applications when comparing different gold standards, considering the design choices for a new gold standard and performing qualitative error analyses for a proposed approach 
Furthermore we applied the framework to analyse popular state-of-the-art gold standards for machine reading comprehension: We reveal issues with their factual correctness, show the presence of lexical cues and we observe that semantics-altering grammatical modifiers are missing in all of the investigated gold standards  Studying how to introduce those modifiers into gold standards and observing whether state-of-the-art MRC models are capable of performing reading comprehension on text containing them, is a future research goal 
A future line of research is to extend the framework to be able to identify the different types of exploitable cues such as question or entity typing and concrete overlap patterns  This will allow the framework to serve as an interpretable estimate of reading comprehension complexity of gold standards  Finally, investigating gold standards under this framework where MRC models outperform the human baseline (e g  SQuAD) will contribute to a deeper understanding of the seemingly superb performance of deep learning approaches on them 

ground_truth: ARAM has achieved improvement over all baseline methods using reverese perplexity and  slef-BLEU metric.  The maximum reverse perplexity improvement 936,16 is gained for EMNLP2017  WMT  dataset and  48,44 for COCO dataset.

Compared to the baselines, ARAML does not do better in terms of perplexity on COCO and EMNLP 2017 WMT datasets, but it does by up to 0.27 Self-BLEU points on COCO and 0.35 Self-BLEU on EMNLP 2017 WMT. In terms of Grammaticality and Relevance, it scores better than the baselines on up to 75.5% and 73% of the cases respectively.

prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. What are the sources of the datasets?
Context 2: Introduction
Emotion detection has long been a topic of interest to scholars in natural language processing (NLP) domain  Researchers aim to recognize the emotion behind the text and distribute similar ones into the same group  Establishing an emotion classifier can not only understand each user's feeling but also be extended to various application, for example, the motivation behind a user's interests BIBREF0  Based on releasing of large text corpus on social media and the emotion categories proposed by BIBREF1, BIBREF2, numerous models have provided and achieved fabulous precision so far  For example, DeepMoji BIBREF3 which utilized transfer learning concept to enhance emotions and sarcasm understanding behind the target sentence  CARER BIBREF4 learned contextualized affect representations to make itself more sensitive to rare words and the scenario behind the texts 
As methods become mature, text-based emotion detecting applications can be extended from a single utterance to a dialogue contributed by a series of utterances  Table TABREF2 illustrates the difference between single utterance and dialogue emotion recognition  The same utterances in Table TABREF2, even the same person said the same sentence, the emotion it convey may be various, which may depend on different background of the conversation, tone of speaking or personality  Therefore, for emotion detection, the information from preceding utterances in a conversation is relatively critical 
In SocialNLP 2019 EmotionX, the challenge is to recognize emotions for all utterances in EmotionLines dataset, a dataset consists of dialogues  According to the needs for considering context at the same time, we develop two classification models, inspired by bidirectional encoder representations from transformers (BERT) BIBREF5, FriendsBERT and ChatBERT  In this paper, we introduce our approaches including causal utterance modeling, model pre-training, and fine-turning 
Dataset
EmotionLines BIBREF6 is a dialogue dataset composed of two subsets, Friends and EmotionPush, according to the source of the dialogues  The former comes from the scripts of the Friends TV sitcom  The other is made up of Facebook messenger chats  Each subset includes $1,000$ English dialogues, and each dialogue can be further divided into a few consecutive utterances  All the utterances are annotated by five annotators on a crowd-sourcing platform (Amazon Mechanical Turk), and the labeling work is only based on the textual content  Annotator votes for one of the seven emotions, namely Ekman’s six basic emotions BIBREF1, plus the neutral  If none of the emotion gets more than three votes, the utterance will be marked as “non-neutral” 
For the datasets, there are properties worth additional mentioning  Although Friends and EmotionPush share the same data format, they are quite different in nature  Friends is a speech-based dataset which is annotated dialogues from the TV sitcom  It means most of the utterances are generated by the a few main characters  The personality of a character often affects the way of speaking, and therefore “who is the speaker" might provide extra clues for emotion prediction  In contrast, EmotionPush does not have this trait due to the anonymous mechanism  In addition, features such as typo, hyperlink, and emoji that only appear in chat-based data will need some domain-specific techniques to process 
Incidentally, the objective of the challenge is to predict the emotion for each utterance  Just, according to EmotionX 2019 specification, there are only four emotions be selected as our label candidates, which are Joy, Sadness, Anger, and Neutral  These emotions will be considered during performance evaluation  The technical detail will also be introduced and discussed in following Section SECREF13 and Section SECREF26 
Model Description
For this challenge, we adapt BERT which is proposed by BIBREF5 to help understand the context at the same time  Technically, BERT, designed on end-to-end architecture, is a deep pre-trained transformer encoder that dynamically provides language representation and BERT already achieved multiple state-of-the-art results on GLUE benchmark BIBREF7 and many tasks  A quick recap for BERT's architecture and its pre-training tasks will be illustrated in the following subsections 
Model Description ::: Model Architecture
BERT, the Bidirectional Encoder Representations from Transformers, consists of several transformer encoder layers that enable the model to extract very deep language features on both token-level and sentence-level  Each transformer encoder contains multi-head self-attention layers that provide ability to learn multiple attention feature of each word from their bidirectional context  The transformer and its self-attention mechanism are proposed by BIBREF8  This self-attention mechanism can be interpreted as a key-value mapping given query  By given the embedding vector for token input, the query ($Q$), key ($K$) and value ($V$) are produced by the projection from each three parameter matrices where $W^Q \in \mathbb {R}^{d_{{\rm model}} \times d_{k}}, W^K \in \mathbb {R}^{d_{\rm model} \times d_{k}}$ and $W^V \in \mathbb {R}^{d_{\rm model} \times d_{v}}$  The self-attention BIBREF8 is formally represented as:
The $ d_k = d_v = d_{\rm model} = 1024$ in BERT large version and 768 in BERT base version  Once model can extract attention feature, we can extend one self-attention into multi-head self-attention, this extension makes sub-space features can be extracted in same time by this multi-head configuration  Overall, the multi-attention mechanism is adopt for each transformer encoder, and several of encoder layer will be stacked together to form a deep transformer encoder 
For the model input, BERT allow us take one sentence as input sequence or two sentences together as one input sequence, and the maximum length of input sequence is 512  The way that BERT was designed is for giving model the sentence-level and token-level understanding  In two sentences case, a special token ([SEP]) will be inserted between two sentences  In addition, the first input token is also a special token ([CLS]), and its corresponding ouput will be vector place for classification during fine-tuning  The outputs of the last encoder layer corresponding to each input token can be treated as word representations for each token, and the word representation of the first token ([CLS]) will be consider as classification (output) representation for further fine-tuning tasks  In BERT, this vector is denoted as $C \in \mathbb {R}^{d_{\rm model}} $, and a classification layer is denoted as $ W \in \mathbb {R}^{K \times d_{\rm model}}$, where $K$ is number of classification labels  Finally, the prediction $P$ of BERT is represented as $P = {\rm softmax}(CW^T)$ 
Model Description ::: Pre-training Tasks
In pre-training, intead of using unidirectional language models, BERT developed two pre-training tasks: (1) Masked LM (cloze test) and (2) Next Sentence Prediction  At the first pre-training task, bidirectional language modeling can be done at this cloze-like pre-training  In detail, 15% tokens of input sequence will be masked at random and model need to predict those masked tokens  The encoder will try to learn contextual representations from every given tokens due to masking tokens at random  Model will not know which part of the input is going to be masked, so that the information of each masked tokens should be inferred by remaining tokens  At Next Sentence Prediction, two sentences concatenated together will be considered as model input  In order to give model a good nature language understanding, knowing relationship between sentence is one of important abilities  When generating input sequences, 50% of time the sentence B is actually followed by sentence A, and rest 50% of the time the sentence B will be picked randomly from dataset, and model need to predict if the sentence B is next sentence of sentence A  That is, the attention information will be shared between sentences  Such sentence-level understanding may have difficulties to be learned at first pre-training task (Masked LM), therefore, the pre-training task (NSP) is developed as second training goal to capture the cross sentence relationship 
In this competition, limited by the size of dataset and the challenge in contextual emotion recognition, we consider BERT with both two pre-training tasks can give a good starting point to extract emotion changing during dialogue-like conversation  Especially the second pre-training task, it might be more important for dialogue-like conversation where the emotion may various by the context of continuous utterances  That is, given a set of continues conversations, the emotion of current utterance might be influenced by previous utterance  By this assumption and with supporting from the experiment results of BERT, we can take sentence A as one-sentence context and consider sentence B as the target sentence for emotion prediction  The detail will be described in Section SECREF4 
Methodology
The main goal of the present work is to predict the emotion of utterance within the dialogue  Following are four major difficulties we concern about:
The emotion of the utterances depends not only on the text but also on the interaction happened earlier 
The source of the two datasets are different  Friends is speech-based dialogues and EmotionPush is chat-based dialogues  It makes datasets possess different characteristics 
There are only $1,000$ dialogues in both training datasets which are not large enough for the stability of training a complex neural-based model 
The prediction targets (emotion labels) are highly unbalanced 
The proposed approach is summarized in Figure FIGREF3, which aims to overcome these challenges  The framework could be separated into three steps and described as follow:
Methodology ::: Causal Utterance Modeling
Given a dialogue $D^{(i)}$ which includes sequence of utterances denoted as $D^{(i)}=(u^{(i)}_{1}, u^{(i)}_{2},    , u^{(i)}_{n})$, where $i$ is the index in dataset and $n$ is the number of utterances in the given dialogue  In order to conserve the emotional information of both utterance and conversation, we rearrange each two consecutive utterances $u_{t}, u_{t-1}$ into a single sentence representation $x_{t}$ as
The corresponding sentence representation corpus $X^{(i)}$ are denoted as $X^{(i)}=(x^{(i)}_{1}, x^{(i)}_{2},    , x^{(i)}_{n})$  Note that the first utterance within a conversation does not have its causal utterance (previous sentence), therefore, the causal utterance will be set as [None]  A practical example of sentence representation is shown in Table TABREF11 
Since the characteristics of two datasets are not identical, we customize different causal utterance modeling strategies to refine the information in text 
For Friends, there are two specific properties  The first one is that most dialogues are surrounding with the six main characters, including Rachel, Monica, Phoebe, Joey, Chandler, and Ross  The utterance ratio of given by the six roles is up to $83 4\%$  Second, the personal characteristics of the six characters are very clear  Each leading role has its own emotion undulated rule  To make use of these features, we introduce the personality tokenization which help learning the personality of the six characters  Personality tokenization concatenate the speaker and says tokens before the input utterance if the speaker is one of the six characters  The example is shown in Table TABREF12 
For EmotionPush, the text are informal chats which including like slang, acronym, typo, hyperlink, and emoji  Another characteristic is that the specific name entities are tokenized with random index  (e g  “organization_80”, “person_01”, and “time_12”)  We consider some of these informal text are related to expressing emotion such as repeated typing, purposed capitalization, and emoji (e g  “:D”, “:(”, and “<3”))  Therefore, we keep most informal expressions but only process hyperlinks, empty utterance, and name entities by unifying the tokens 
Methodology ::: Model Pre-training
Since the size of both datasets are not large enough for complex neural-based model training as well as BERT model is only pre-train on formal text datasets, the issues of overfitting and domain bias are important considerations for design the pre-training process 
To avoid our model overfitting on the training data and increase the understanding of informal text, we adapted BERT and derived two models, namely FriendsBERT and ChatBERT, with different pre-training tasks before the formal training process for Friends and EmotionPush dataset, respectively  The pre-training strategies are described below 
For pre-training FriendsBERT, we collect the completed scripts of all ten seasons of Friends TV shows from emorynlp which includes 3,107 scenes within 61,309 utterances  All the utterances are followed the preprocessing methods mentions above to compose the corpus for Masked language model pre-training task  The consequent utterances in the same scenes are considered as the consequent sentences to pre-train the Next Sentence Prediction task  In the pre-training process, the training loss is the sum of the mean likelihood of two pre-train tasks 
For pre-training ChatBERT, we pre-train our model on the Twitter dataset, since the text and writing style on Twitter are close to the chat text where both may involved with many informal words or emoticons as well  The Twitter emotion dataset, 8 basic emotions from emotion wheel BIBREF1, was collected by twitter streaming API with specific emotion-related hashtags, such as #anger, #joy, #cry, #sad and etc  The hashtags in tweets are treated as emotion label for model fine-tuning  The tweets were fine-grined processing followed the rules in BIBREF9, BIBREF4, including duplicate tweets removing, the emotion hashtags must appearing in the last position of a tweet, and etc  The statis of tweets were summarized in Table TABREF17  Each tweet and corresponding emotion label composes an emotion classification dataset for pre-training 
Methodology ::: Fine-tuning
Since our emotion recognition task is treated as a sequence-level classification task, the model would be fine-tuned on the processed training data  Following the BERT construction, we take the first embedding vector which corresponds to the special token [CLS] from the final hidden state of the Transformer encoder  This vector represents the embedding vector of the corresponding conversation utterances which is denoted as $\mathbf {C} \in \mathbb {R}^{H}$, where $H$ is the embedding size  A dense neural layer is treated as a classification layer which consists of parameters $\mathbf {W} \in \mathbb {R}^{K\times H}$ and $\mathbf {b} \in \mathbb {R}^{K}$, where $K$ is the number of emotion class  The emotion prediction probabilities $\mathbf {P} \in \mathbb {R}^{K}$ are computed by a softmax activation function as
All the parameters in BERT and the classification layer would be fine-turned together to minimize the Negative Log Likelihood (NLL) loss function, as Equation (DISPLAY_FORM22), based on the ground truth emotion label $c$ 
In order to tackle the problem of highly unbalanced emotion labels, we apply weighted balanced warming on NLL loss function, as Equation (DISPLAY_FORM23), in the first epoch of fine-tuning procedure 
where $\mathbf {w}$ are the weights of corresponding emotion label $c$ which are computed and normalize by the frequency as
By adding the weighted balanced warming on NLL loss, the model could learn to predict the minor emotions (e g  anger and sadness) earlier and make the training process more stable  Since the major evaluation metrics micro F1-score is effect by the number of each label, we only apply the weighted balanced warming in first epoch to optimize the performance 
Experiments
Since the EmotionX challenge only provided the gold labels in training data, we pick the best performance model (weights) to predict the testing data  In this section, we present the experiment and evaluation results 
Experiments ::: Experimental Setup
The EmotionX challenge consists of $1,000$ dialogues for both Friends and EmotionPush  In all of our experiments, each dataset is separated into top 800 dialogues for training and last 200 dialogues for validation  Since the EmotionX challenge considers only the four emotions (anger, joy, neutral, and sadness) in the evaluation stage, we ignore all the data point corresponding to other emotions directly  The details of emotions distribution are shown in Table TABREF18 
The hyperparameters and training setup of our models (FriendsBERT and ChatBERT) are shown in Table TABREF25  Some common and easily implemented methods are selected as the baselines embedding methods and classification models  The baseline embedding methods are including bag-of-words (BOW), term frequency–inverse document frequency (TFIDF), and neural-based word embedding  The classification models are including Logistic Regression (LR), Random Forest (RF), TextCNN BIBREF10 with initial word embedding as GloVe BIBREF11, and our proposed model  All the experiment results are based on the best performances of validation results 
Experiments ::: Performance
The experiment results of validation on Friends are shown in Table TABREF19  The proposed model and baselines are evaluated based on the Precision (P ), Recall (R ), and F1-measure (F1) 
For the traditional baselines, namely BOW and TFIDF, we observe that they achieve surprising high F1 scores around $0 81$, however, the scores for Anger and Sadness are lower  This explains that traditional approaches tend to predict the labels with large sample size, such as Joy and Neutral, but fail to take of scarce samples even when an ensemble random forest classifier is adopted  In order to prevent the unbalanced learning, we choose the weighted loss mechanism for both TextCNN and causal modeling TextCNN (C-TextCNN), these models suffer less than the traditional baselines and achieve a slightly balance performance, where there are around 15% and 7% improvement on Anger and Sadness, respectively  We following adopt the casual utterance modeling to original TextCNN, mapping previous utterance as well as target utterance into model  The causal utterance modeling improve the C-TextCNN over TextCNN for 6%, 2% and 1% on Anger, Joy and overall F1 score  Motivated from these preliminary experiments, the proposed FriendsBERT also adopt the ideas of both weighted loss and causal utterance modeling  As compared to the original BERT, single sentence BERT (FriendsBERT-base-s), the proposed FriendsBERT-base improve 1% for Joy and overall F1, and 2% for Sadness  For the final validation performance, our proposed approach achieves the highest scores, which are $0 85$ and $0 86$ for FriendsBERT-base and FriendsBERT-large, respectively 
Overall, the proposed FriendsBERT successfully captures the sentence-level context-awarded information and outperforms all the baselines, which not only achieves high performance on large sample labels, but also on small sample labels  The similar settings are also adapted to EmotionPush dataset for the final evaluation 
Experiments ::: Evaluation Results
The testing dataset consists of 240 dialogues including $3,296$ and $3,536$ utterances in Friends and EmotionPush respectively  We re-train our FriendsBERT and ChatBERT with top 920 training dialogues and predict the evaluation results using the model performing the best validation results  The results are shown in Table TABREF29 and Table TABREF30  The present method achieves $81 5\%$ and $88 5\%$ micro F1-score on the testing dataset of Friends and EmotionPush, respectively 
Conclusion and Future work
In the present work, we propose FriendsBERT and ChatBERT for the multi-utterance emotion recognition task on EmotionLines dataset  The proposed models are adapted from BERT BIBREF5 with three main improvement during the model training procedure, which are the causal utterance modeling mechanism, specific model pre-training, and adapt weighted loss  The causal utterance modeling takes the advantages of the sentence-level context information during model inference  The specific model pre-training helps to against the bias in different text domain  The weighted loss avoids our model to only predict on large size sample  The effectiveness and generalizability of the proposed methods are demonstrated from the experiments 
In future work, we consider to include the conditional probabilistic constraint $P ({\rm Emo}_{B} | \hat{\rm Emo}_{A})$  Model should predict the emotion based on a certain understanding about context emotions  This might be more reasonable for guiding model than just predicting emotion of ${\rm Sentence}_B$ directly  In addition, due to the limitation of BERT input format, ambiguous number of input sentences is now becoming an important design requirement for our future work  Also, personality embedding development will be another future work of the emotion recognition  The personality embedding will be considered as sentence embedding injected into word embedding, and it seems this additional information can contribute some improvement potentially 

ground_truth: Friends TV sitcom, Facebook messenger chats

prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. How does proposed qualitative annotation schema looks like?
Context 4:  Finally, for B2 we achieve much poorer performance, with average scores of P=0 21, R=0 20 and F1=0 21 
Robustness  In Table TABREF58 , we show the overall performance for the years between 2009 and 2013  An interesting observation we make is that we have a very robust performance and the results are stable across the years  If we consider the experimental setup, where for year INLINEFORM0 we optimize the learning objective with only 74k training instances and evaluate on the rest of the instances, it achieves a very good performance  We predict with F1=0 68 the remaining 469k instances for the years INLINEFORM1  
The results are particularly promising considering the fact that the distribution between our two classes is highly skewed  On average the number of `relevant' pairs account for only around INLINEFORM0 of all pairs  A good indicator to support such a statement is the kappa (denoted by INLINEFORM1 ) statistic  INLINEFORM2 measures agreement between the algorithm and the gold standard on both labels while correcting for chance agreement (often expected due to extreme distributions)  The INLINEFORM3 scores for B1 across the years is on average INLINEFORM4 , while for INLINEFORM5 we achieve a score of INLINEFORM6 (the maximum score for INLINEFORM7 is 1) 
In Figure FIGREF60 we show the impact of the individual feature groups that contribute to the superior performance in comparison to the baselines  Relative entity frequency from the salience feature, models the entity salience as an exponentially decaying function based on the positional index of the paragraph where the entity appears  The performance of INLINEFORM0 with relative entity frequency from the salience feature group is close to that of all the features combined  The authority and novelty features account to a further improvement in terms of precision, by adding roughly a 7%-10% increase  However, if both feature groups are considered separately, they significantly outperform the baseline B1 
Article–Section Placement
We model the ASP placement task as a successor of the AEP task  For all the `relevant' news entity pairs, the task is to determine the correct entity section  Each section in a Wikipedia entity page represents a different topic  For example, Barack Obama has the sections `Early Life', `Presidency', `Family and Personal Life' etc  However, many entity pages have an incomplete section structure  Incomplete or missing sections are due to two Wikipedia properties  First, long-tail entities miss information and sections due to their lack of popularity  Second, for all entities whether popular or not, certain sections might occur for the first time due to real world developments  As an example, the entity Germanwings did not have an `Accidents' section before this year's disaster, which was the first in the history of the airline 
Even if sections are missing for certain entities, similar sections usually occur in other entities of the same class (e g  other airlines had disasters and therefore their pages have an accidents section)  We exploit such homogeneity of section structure and construct templates that we use to expand entity profiles  The learning objective for INLINEFORM0 takes into account the following properties:
Section-templates: account for incomplete section structure for an entity profile INLINEFORM0 by constructing section templates INLINEFORM1 from an entity class INLINEFORM2
Overall fit: measures the overall fit of a news article to sections in the section templates INLINEFORM0
Given the fact that entity profiles are often incomplete, we construct section templates for every entity class  We group entities based on their class INLINEFORM0 and construct section templates INLINEFORM1   For different entity classes, e g  Person and Location, the section structure and the information represented in those section varies heavily  Therefore, the section templates are with respect to the individual classes in our experimental setup (see Figure FIGREF42 )  DISPLAYFORM0
Generating section templates has two main advantages  Firstly, by considering class-based profiles, we can overcome the problem of incomplete individual entity profiles and thereby are able to suggest news articles to sections that do not yet exist in a specific entity INLINEFORM0   The second advantage is that we are able to canonicalize the sections, i e  `Early Life' and `Early Life and Childhood' would be treated similarly 
To generate the section template INLINEFORM0 , we extract all sections from entities of a given type INLINEFORM1 at year INLINEFORM2   Next, we cluster the entity sections, based on an extended version of k–means clustering BIBREF18 , namely x–means clustering introduced in Pelleg et al  which estimates the number of clusters efficiently BIBREF19   As a similarity metric we use the cosine similarity computed based on the tf–idf models of the sections  Using the x–means algorithm we overcome the requirement to provide the number of clusters k beforehand  x–means extends the k–means algorithm, such that a user only specifies a range [ INLINEFORM3 , INLINEFORM4 ] that the number of clusters may reasonably lie in 
The learning objective of INLINEFORM0 is to determine the overall fit of a news article INLINEFORM1 to one of the sections in a given section template INLINEFORM2   The template is pre-determined by the class of the entity for which the news is suggested as relevant by INLINEFORM3   In all cases, we measure how well INLINEFORM4 fits each of the sections INLINEFORM5 as well as the specific entity section INLINEFORM6   The section profiles in INLINEFORM7 represent the aggregated entity profiles from all entities of class INLINEFORM8 at year INLINEFORM9  
To learn INLINEFORM0 we rely on a variety of features that consider several similarity aspects as shown in Table TABREF31   For the sake of simplicity we do not make the distinction in Table TABREF31 between the individual entity section and class-based section similarities, INLINEFORM1 and INLINEFORM2 , respectively  Bear in mind that an entity section INLINEFORM3 might be present at year INLINEFORM4 but not at year INLINEFORM5 (see for more details the discussion on entity profile expansion in Section UID69 ) 
Topic  We use topic similarities to ensure (i) that the content of INLINEFORM0 fits topic-wise with a specific section text and (ii) that it has a similar topic to previously referred news articles in that section  In a pre-processing stage we compute the topic models for the news articles, entity sections INLINEFORM1 and the aggregated class-based sections in INLINEFORM2   The topic models are computed using LDA BIBREF20   We only computed a single topic per article/section as we are only interested in topic term overlaps between article and sections  We distinguish two main features: the first feature measures the overlap of topic terms between INLINEFORM3 and the entity section INLINEFORM4 and INLINEFORM5 , and the second feature measures the overlap of the topic model of INLINEFORM6 against referred news articles in INLINEFORM7 at time INLINEFORM8  
Syntactic  These features represent a mechanism for conveying the importance of a specific text snippet, solely based on the frequency of specific POS tags (i e  NNP, CD etc ), as commonly used in text summarization tasks  Following the same intuition as in BIBREF8 , we weigh the importance of articles by the count of specific POS tags  We expect that for different sections, the importance of POS tags will vary  We measure the similarity of POS tags in a news article against the section text  Additionally, we consider bi-gram and tri-gram POS tag overlap  This exploits similarity in syntactical patterns between the news and section text 
Lexical  As lexical features, we measure the similarity of INLINEFORM0 against the entity section text INLINEFORM1 and the aggregate section text INLINEFORM2   Further, we distinguish between the overall similarity of INLINEFORM3 and that of the different news paragraphs ( INLINEFORM4 which denotes the paragraphs of INLINEFORM5 up to the 5th paragraph)  A higher similarity on the first paragraphs represents a more confident indicator that INLINEFORM6 should be suggested to a specific section INLINEFORM7   We measure the similarity based on two metrics: (i) the KL-divergence between the computed language models and (ii) cosine similarity of the corresponding paragraph text INLINEFORM8 and section text 
Entity-based  Another feature set we consider is the overlap of named entities and their corresponding entity classes  For different entity sections, we expect to find a particular set of entity classes that will correlate with the section, e g  `Early Life' contains mostly entities related to family, school, universities etc 
Frequency  Finally, we gather statistics about the number of entities, paragraphs, news article length, top– INLINEFORM0 entities and entity classes, and the frequency of different POS tags  Here we try to capture patterns of articles that are usually cited in specific sections 
Evaluation Plan
In this section we outline the evaluation plan to verify the effectiveness of our learning approaches  To evaluate the news suggestion problem we are faced with two challenges 
What comprises the ground truth for such a task ?
How do we construct training and test splits given that entity pages consists of text added at different points in time ?
Consider the ground truth challenge  Evaluating if an arbitrary news article should be included in Wikipedia is both subjective and difficult for a human if she is not an expert  An invasive approach, which was proposed by Barzilay and Sauper BIBREF8 , adds content directly to Wikipedia and expects the editors or other users to redact irrelevant content over a period of time  The limitations of such an evaluation technique is that content added to long-tail entities might not be evaluated by informed users or editors in the experiment time frame  It is hard to estimate how much time the added content should be left on the entity page  A more non-invasive approach could involve crowdsourcing of entity and news article pairs in an IR style relevance assessment setup  The problem of such an approach is again finding knowledgeable users or experts for long-tail entities  Thus the notion of relevance of a news recommendation is challenging to evaluate in a crowd setup 
We take a slightly different approach by making an assumption that the news articles already present in Wikipedia entity pages are relevant  To this extent, we extract a dataset comprising of all news articles referenced in entity pages (details in Section SECREF40 )  At the expense of not evaluating the space comprising of news articles absent in Wikipedia, we succeed in (i) avoiding restrictive assumptions about the quality of human judgments, (ii) being invasive and polluting Wikipedia, and (iii) deriving a reusable test bed for quicker experimentation 
The second challenge of construction of training and test set separation is slightly easier and is addressed in Section SECREF46  
Datasets
The datasets we use for our experimental evaluation are directly extracted from the Wikipedia entity pages and their revision history  The generated data represents one of the contributions of our paper  The datasets are the following:
Entity Classes  We focus on a manually predetermined set of entity classes for which we expect to have news coverage  The number of analyzed entity classes is 27, including INLINEFORM0 entities with at least one news reference  The entity classes were selected from the DBpedia class ontology  Figure FIGREF42 shows the number of entities per class for the years (2009-2014) 
News Articles  We extract all news references from the collected Wikipedia entity pages  The extracted news references are associated with the sections in which they appear  In total there were INLINEFORM0 news references, and after crawling we end up with INLINEFORM1 successfully crawled news articles  The details of the news article distribution, and the number of entities and sections from which they are referred are shown in Table TABREF44  
Article-Entity Ground-truth  The dataset comprises of the news and entity pairs INLINEFORM0   News-entity pairs are relevant if the news article is referenced in the entity page  Non-relevant pairs (i e  negative training examples) consist of news articles that contain an entity but are not referenced in that entity's page  If a news article INLINEFORM1 is referred from INLINEFORM2 at year INLINEFORM3 , the features are computed taking into account the entity profiles at year INLINEFORM4  
Article-Section Ground-truth  The dataset consists of the triple INLINEFORM0 , where INLINEFORM1 , where we assume that INLINEFORM2 has already been determined as relevant  We therefore have a multi-class classification problem where we need to determine the section of INLINEFORM3 where INLINEFORM4 is cited  Similar to the article-entity ground truth, here too the features compute the similarity between INLINEFORM5 , INLINEFORM6 and INLINEFORM7  
Data Pre-Processing
We POS-tag the news articles and entity profiles INLINEFORM0 with the Stanford tagger BIBREF21   For entity linking the news articles, we use TagMe! BIBREF22 with a confidence score of 0 3  On a manual inspection of a random sample of 1000 disambiguated entities, the accuracy is above 0 9  On average, the number of entities per news article is approximately 30  For entity linking the entity profiles, we simply follow the anchor text that refers to Wikipedia entities 
Train and Testing Evaluation Setup
We evaluate the generated supervised models for the two tasks, AEP and ASP, by splitting the train and testing instances  It is important to note that for the pairs INLINEFORM0 and the triple INLINEFORM1 , the news article INLINEFORM2 is referenced at time INLINEFORM3 by entity INLINEFORM4 , while the features take into account the entity profile at time INLINEFORM5   This avoids any `overlapping' content between the news article and the entity page, which could affect the learning task of the functions INLINEFORM6 and INLINEFORM7   Table TABREF47 shows the statistics of train and test instances  We learn the functions at year INLINEFORM8 and test on instances for the years greater than INLINEFORM9   Please note that we do not show the performance for year 2014 as we do not have data for 2015 for evaluation 
Article-Section Placement
Here we show the evaluation setup for ASP task and discuss the results with a focus on three main aspects, (i) the overall performance across the years, (ii) the entity class specific performance, and (iii) the impact on entity profile expansion by suggesting missing sections to entities based on the pre-computed templates 
Baselines  To the best of our knowledge, we are not aware of any comparable approach for this task  Therefore, the baselines we consider are the following:
S1: Pick the section from template INLINEFORM0 with the highest lexical similarity to INLINEFORM1 : S1 INLINEFORM2
S2: Place the news into the most frequent section in INLINEFORM0
Learning Models  We use Random Forests (RF) BIBREF23 and Support Vector Machines (SVM) BIBREF24   The models are optimized taking into account the features in Table TABREF31   In contrast to the AEP task, here the scale of the number of instances allows us to learn the SVM models  The SVM model is optimized using the INLINEFORM0 loss function and uses the Gaussian kernels 
Metrics  We compute precision P as the ratio of news for which we pick a section INLINEFORM0 from INLINEFORM1 and INLINEFORM2 conforms to the one in our ground-truth (see Section SECREF40 )  The definition of recall R and F1 score follows from that of precision 
Figure FIGREF66 shows the overall performance and a comparison of our approach (when INLINEFORM0 is optimized using SVM) against the best performing baseline S2  With the increase in the number of training instances for the ASP task the performance is a monotonically non-decreasing function  For the year 2009, we optimize the learning objective of INLINEFORM1 with around 8% of the total instances, and evaluate on the rest  The performance on average is around P=0 66 across all classes  Even though for many classes the performance is already stable (as we will see in the next section), for some classes we improve further  If we take into account the years between 2010 and 2012, we have an increase of INLINEFORM2 P=0 17, with around 70% of instances used for training and the remainder for evaluation  For the remaining years the total improvement is INLINEFORM3 P=0 18 in contrast to the performance at year 2009 
On the other hand, the baseline S1 has an average precision of P=0 12  The performance across the years varies slightly, with the year 2011 having the highest average precision of P=0 13  Always picking the most frequent section as in S2, as shown in Figure FIGREF66 , results in an average precision of P=0 17, with a uniform distribution across the years 
Here we show the performance of INLINEFORM0 decomposed for the different entity classes  Specifically we analyze the 27 classes in Figure FIGREF42   In Table TABREF68 , we show the results for a range of years (we omit showing all years due to space constraints)  For illustration purposes only, we group them into four main classes ( INLINEFORM1 Person, Organization, Location, Event INLINEFORM2 ) and into the specific sub-classes shown in the second column in Table TABREF68   For instance, the entity classes OfficeHolder and Politician are aggregated into Person–Politics 
It is evident that in the first year the performance is lower in contrast to the later years  This is due to the fact that as we proceed, we can better generalize and accurately determine the correct fit of an article INLINEFORM0 into one of the sections from the pre-computed templates INLINEFORM1   The results are already stable for the year range INLINEFORM2   For a few Person sub-classes, e g  Politics, Entertainment, we achieve an F1 score above 0 9  These additionally represent classes with a sufficient number of training instances for the years INLINEFORM3   The lowest F1 score is for the Criminal and Television classes  However, this is directly correlated with the insufficient number of instances 
The baseline approaches for the ASP task perform poorly  S1, based on lexical similarity, has a varying performance for different entity classes  The best performance is achieved for the class Person – Politics, with P=0 43  This highlights the importance of our feature choice and that the ASP cannot be considered as a linear function, where the maximum similarity yields the best results  For different entity classes different features and combination of features is necessary  Considering that S2 is the overall best performing baseline, through our approach INLINEFORM0 we have a significant improvement of over INLINEFORM1 P=+0 64 
The models we learn are very robust and obtain high accuracy, fulfilling our pre-condition for accurate news suggestions into the entity sections  We measure the robustness of INLINEFORM0 through the INLINEFORM1 statistic  In this case, we have a model with roughly 10 labels (corresponding to the number of sections in a template INLINEFORM2 )  The score we achieve shows that our model predicts with high confidence with INLINEFORM3  
The last analysis is the impact we have on expanding entity profiles INLINEFORM0 with new sections  Figure FIGREF70 shows the ratio of sections for which we correctly suggest an article INLINEFORM1 to the right section in the section template INLINEFORM2   The ratio here corresponds to sections that are not present in the entity profile at year INLINEFORM3 , that is INLINEFORM4   However, given the generated templates INLINEFORM5 , we can expand the entity profile INLINEFORM6 with a new section at time INLINEFORM7   In details, in the absence of a section at time INLINEFORM8 , our model trains well on similar sections from the section template INLINEFORM9 , hence we can predict accurately the section and in this case suggest its addition to the entity profile  With time, it is obvious that the expansion rate decreases at later years as the entity profiles become more `complete' 
This is particularly interesting for expanding the entity profiles of long-tail entities as well as updating entities with real-world emerging events that are added constantly  In many cases such missing sections are present at one of the entities of the respective entity class INLINEFORM0   An obvious case is the example taken in Section SECREF16 , where the `Accidents' is rather common for entities of type Airline  However, it is non-existent for some specific entity instances, i e Germanwings airline 
Through our ASP approach INLINEFORM0 , we are able to expand both long-tail and trunk entities  We distinguish between the two types of entities by simply measuring their section text length  The real distribution in the ground truth (see Section SECREF40 ) is 27% and 73% are long-tail and trunk entities, respectively  We are able to expand the entity profiles for both cases and all entity classes without a significant difference, with the only exception being the class Creative Work, where we expand significantly more trunk entities 
Conclusion and Future Work
In this work, we have proposed an automated approach for the novel task of suggesting news articles to Wikipedia entity pages to facilitate Wikipedia updating  The process consists of two stages  In the first stage, article–entity placement, we suggest news articles to entity pages by considering three main factors, such as entity salience in a news article, relative authority and novelty of news articles for an entity page  In the second stage, article–section placement, we determine the best fitting section in an entity page  Here, we remedy the problem of incomplete entity section profiles by constructing section templates for specific entity classes  This allows us to add missing sections to entity pages  We carry out an extensive experimental evaluation on 351,983 news articles and 73,734 entities coming from 27 distinct entity classes  For the first stage, we achieve an overall performance with P=0 93, R=0 514 and F1=0 676, outperforming our baseline competitors significantly  For the second stage, we show that we can learn incrementally to determine the correct section for a news article based on section templates  The overall performance across different classes is P=0 844, R=0 885 and F1=0 860 
In the future, we will enhance our work by extracting facts from the suggested news articles  Results suggest that the news content cited in entity pages comes from the first paragraphs  However, challenging task such as the canonicalization and chronological ordering of facts, still remain 

ground_truth: The resulting taxonomy of the framework is shown in Figure FIGREF10

FIGREF10