prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. Is it possible to convert a cloze-style questions to a naturally-looking questions?
Context 1: Introduction
Deep learning systems have shown a lot of promise for extractive Question Answering (QA), with performance comparable to humans when large scale data is available  However, practitioners looking to build QA systems for specific applications may not have the resources to collect tens of thousands of questions on corpora of their choice  At the same time, state-of-the-art machine reading systems do not lend well to low-resource QA settings where the number of labeled question-answer pairs are limited (c f  Table 2 )  Semi-supervised QA methods like BIBREF0 aim to improve this performance by leveraging unlabeled data which is easier to collect 
In this work, we present a semi-supervised QA system which requires the end user to specify a set of base documents and only a small set of question-answer pairs over a subset of these documents  Our proposed system consists of three stages  First, we construct cloze-style questions (predicting missing spans of text) from the unlabeled corpus; next, we use the generated clozes to pre-train a powerful neural network model for extractive QA BIBREF1 , BIBREF2 ; and finally, we fine-tune the model on the small set of provided QA pairs 
Our cloze construction process builds on a typical writing phenomenon and document structure: an introduction precedes and summarizes the main body of the article  Many large corpora follow such a structure, including Wikipedia, academic papers, and news articles  We hypothesize that we can benefit from the un-annotated corpora to better answer various questions – at least ones that are lexically similar to the content in base documents and directly require factual information 
We apply the proposed system on three datasets from different domains – SQuAD BIBREF3 , TriviaQA-Web BIBREF4 and the BioASQ challenge BIBREF5   We observe significant improvements in a low-resource setting across all three datasets  For SQuAD and TriviaQA, we attain an F1 score of more than 50% by merely using 1% of the training data  Our system outperforms the approaches for semi-supervised QA presented in BIBREF0 , and a baseline which uses the same unlabeled data but with a language modeling objective for pretraining  In the BioASQ challenge, we outperform the best performing system from previous year's challenge, improving over a baseline which does transfer learning from the SQuAD dataset  Our analysis reveals that questions which ask for factual information and match to specific parts of the context documents benefit the most from pretraining on automatically constructed clozes 
Related Work
Semi-supervised learning augments the labeled dataset $L$ with a potentially larger unlabeled dataset $U$   BIBREF0 presented a model, GDAN, which trained an auxiliary neural network to generate questions from passages by reinforcement learning, and augment the labeled dataset with the generated questions to train the QA model  Here we use a much simpler heuristic to generate the auxiliary questions, which also turns out to be more effective as we show superior performance compared to GDAN  Several approaches have been suggested for generating natural questions BIBREF6 , BIBREF7 , BIBREF8 , however none of them show a significant improvement of using the generated questions in a semi-supervised setting  Recent papers also use unlabeled data for QA by training large language models and extracting contextual word vectors from them to input to the QA model BIBREF9 , BIBREF10 , BIBREF11   The applicability of this method in the low-resource setting is unclear as the extra inputs increase the number of parameters in the QA model, however, our pretraining can be easily applied to these models as well 
Domain adaptation (and Transfer learning) leverage existing large scale datasets from a source domain (or task) to improve performance on a target domain (or task)  For deep learning and QA, a common approach is to pretrain on the source dataset and then fine-tune on the target dataset BIBREF12 , BIBREF13   BIBREF14 used SQuAD as a source for the target BioASQ dataset, and BIBREF15 used Book Test BIBREF16 as source for the target SQuAD dataset  BIBREF17 transfer learned model layers from the tasks of sequence labeling, text classification and relation classification to show small improvements on SQuAD  All these works use manually curated source datatset, which in themselves are expensive to collect  Instead, we show that it is possible to automatically construct the source dataset from the same domain as the target, which turns out to be more beneficial in terms of performance as well (c f  Section "Experiments & Results" )  Several cloze datasets have been proposed in the literature which use heuristics for construction BIBREF18 , BIBREF19 , BIBREF20   We further see the usability of such a dataset in a semi-supervised setting 
Methodology
Our system comprises of following three steps:
Cloze generation: Most of the documents typically follow a template, they begin with an introduction that provides an overview and a brief summary for what is to follow  We assume such a structure while constructing our cloze style questions  When there is no clear demarcation, we treat the first $K\%$ (hyperparameter, in our case 20%) of the document as the introduction  While noisy, this heuristic generates a large number of clozes given any corpus, which we found to be beneficial for semi-supervised learning despite the noise 
We use a standard NLP pipeline based on Stanford CoreNLP (for SQuAD, TrivaQA and PubMed) and the BANNER Named Entity Recognizer (only for PubMed articles) to identify entities and phrases  Assume that a document comprises of introduction sentences $\lbrace q_1, q_2,     q_n\rbrace $ , and the remaining passages $\lbrace p_1, p_2,    p_m\rbrace $   Additionally, let's say that each sentence $q_i$ in introduction is composed of words $\lbrace w_1, w_2,     w_{l_{q_i}}\rbrace $ , where $l_{q_i}$ is the length of $q_i$   We consider a $\text{match} (q_i, p_j)$ , if there is an exact string match of a sequence of words $\lbrace w_k, w_{k+1},    w_{l_{q_i}}\rbrace $ between the sentence $q_i$ and passage $p_j$   If this sequence is either a noun phrase, verb phrase, adjective phrase or a named entity in $\lbrace p_1, p_2,    p_m\rbrace $0 , as recognized by CoreNLP or BANNER, we select it as an answer span $\lbrace p_1, p_2,    p_m\rbrace $1   Additionally, we use $\lbrace p_1, p_2,    p_m\rbrace $2 as the passage $\lbrace p_1, p_2,    p_m\rbrace $3 and form a cloze question $\lbrace p_1, p_2,    p_m\rbrace $4 from the answer bearing sentence $\lbrace p_1, p_2,    p_m\rbrace $5 by replacing $\lbrace p_1, p_2,    p_m\rbrace $6 with a placeholder  As a result, we obtain passage-question-answer ( $\lbrace p_1, p_2,    p_m\rbrace $7 ) triples (Table 1 shows an example)  As a post-processing step, we prune out $\lbrace p_1, p_2,    p_m\rbrace $8 triples where the word overlap between the question (Q) and passage (P) is less than 2 words (after excluding the stop words) 
The process relies on the fact that answer candidates from the introduction are likely to be discussed in detail in the remainder of the article  In effect, the cloze question from the introduction and the matching paragraph in the body forms a question and context passage pair  We create two cloze datasets, one each from Wikipedia corpus (for SQuAD and TriviaQA) and PUBMed academic papers (for the BioASQ challenge), consisting of 2 2M and 1M clozes respectively  From analyzing the cloze data manually, we were able to answer 76% times for the Wikipedia set and 80% times for the PUBMed set using the information in the passage  In most cases the cloze paraphrased the information in the passage, which we hypothesized to be a useful signal for the downstream QA task 
We also investigate the utility of forming subsets of the large cloze corpus, where we select the top passage-question-answer triples, based on the different criteria, like i) jaccard similarity of answer bearing sentence in introduction and the passage ii) the tf-idf scores of answer candidates and iii) the length of answer candidates  However, we empirically find that we were better off using the entire set rather than these subsets 
Pre-training: We make use of the generated cloze dataset to pre-train an expressive neural network designed for the task of reading comprehension  We work with two publicly available neural network models – the GA Reader BIBREF2 (to enable comparison with prior work) and BiDAF + Self-Attention (SA) model from BIBREF1 (which is among the best performing models on SQuAD and TriviaQA)  After pretraining, the performance of BiDAF+SA on a dev set of the (Wikipedia) cloze questions is 0 58 F1 score and 0 55 Exact Match (EM) score  This implies that the cloze corpus is neither too easy, nor too difficult to answer 
Fine Tuning: We fine tune the pre-trained model, from the previous step, over a small set of labelled question-answer pairs  As we shall later see, this step is crucial, and it only requires a handful of labelled questions to achieve a significant proportion of the performance typically attained by training on tens of thousands of questions 
Datasets
We apply our system to three datasets from different domains  SQuAD BIBREF3 consists of questions whose answers are free form spans of text from passages in Wikipedia articles  We follow the same setting as in BIBREF0 , and split $10\%$ of training questions as the test set, and report performance when training on subsets of the remaining data ranging from $1\%$ to $90\%$ of the full set  We also report the performance on the dev set when trained on the full training set ( $1^\ast $ in Table 2 )  We use the same hyperparameter settings as in prior work  We compare and study four different settings: 1) the Supervised Learning (SL) setting, which is only trained on the supervised data, 2) the best performing GDAN model from BIBREF0 , 3) pretraining on a Language Modeling (LM) objective and fine-tuning on the supervised data, and 4) pretraining on the Cloze dataset and fine-tuning on the supervised data  The LM and Cloze methods use exactly the same data for pretraining, but differ in the loss functions used  We report F1 and EM scores on our test set using the official evaluation scripts provided by the authors of the dataset 
TriviaQA BIBREF4 comprises of over 95K web question-answer-evidence triples  Like SQuAD, the answers are spans of text  Similar to the setting in SQuAD, we create multiple smaller subsets of the entire set  For our semi-supervised QA system, we use the BiDAF+SA model BIBREF1 – the highest performing publicly available system for TrivaQA  Here again, we compare the supervised learning (SL) settings against the pretraining on Cloze set and fine tuning on the supervised set  We report F1 and EM scores on the dev set 
We also test on the BioASQ 5b dataset, which consists of question-answer pairs from PubMed abstracts  We use the publicly available system from BIBREF14 , and follow the exact same setup as theirs, focusing only on factoid and list questions  For this setting, there are only 899 questions for training  Since this is already a low-resource problem we only report results using 5-fold cross-validation on all the available data  We report Mean Reciprocal Rank (MRR) on the factoid questions, and F1 score for the list questions 
Main Results
Table 2 shows a comparison of the discussed settings on both SQuAD and TriviaQA  Without any fine-tuning (column 0) the performance is low, probably because the model never saw a real question, but we see significant gains with Cloze pretraining even with very little labeled data  The BiDAF+SA model, exceeds an F1 score of $50\%$ with only $1\%$ of the training data (454 questions for SQuAD, and 746 questions for TriviaQA), and approaches $90\%$ of the best performance with only $10\%$ labeled data  The gains over the SL setting, however, diminish as the size of the labeled set increases and are small when the full dataset is available 
Cloze pretraining outperforms the GDAN baseline from BIBREF0 using the same SQuAD dataset splits  Additionally, we show improvements in the $90\%$ data case unlike GDAN  Our approach is also applicable in the extremely low-resource setting of $1\%$ data, which we suspect GDAN might have trouble with since it uses the labeled data to do reinforcement learning  Furthermore, we are able to use the same cloze dataset to improve performance on both SQuAD and TriviaQA datasets  When we use the same unlabeled data to pre-train with a language modeling objective, the performance is worse, showing the bias we introduce by constructing clozes is important 
On the BioASQ dataset (Table 3 ) we again see a significant improvement when pretraining with the cloze questions over the supervised baseline  The improvement is smaller than what we observe with SQuAD and TriviaQA datasets – we believe this is because questions are generally more difficult in BioASQ  BIBREF14 showed that pretraining on SQuAD dataset improves the downstream performance on BioASQ  Here, we show a much larger improvement by pretraining on cloze questions constructed in an unsupervised manner from the same domain 
Analysis
Regression Analysis: To understand which types of questions benefit from pre-training, we pre-specified certain features (see Figure 1 right) for each of the dev set questions in SQuAD, and then performed linear regression to predict the F1 score for that question from these features  We predict the F1 scores from the cloze pretrained model ( $y^{\text{cloze}}$ ), the supervised model ( $y^{\text{sl}}$ ), and the difference of the two ( $y^{\text{cloze}}-y^{\text{sl}}$ ), when using $10\%$ of labeled data  The coefficients of the fitted model are shown in Figure 1 (left) along with their std errors  Positive coefficients indicate that a high value of that feature is predictive of a high F1 score, and a negative coefficient indicates that a small value of that feature is predictive of a high F1 score (or a high difference of F1 scores from the two models in the case of $y^{\text{cloze}}-y^{\text{sl}}$ ) 
The two strongest effects we observe are that a high lexical overlap between the question and the sentence containing the answer is indicative of high boost with pretraining, and that a high lexical overlap between the question and the whole passage is indicative of the opposite  This is hardly surprising, since our cloze construction process is biased towards questions which have a similar phrasing to the answer sentences in context  Hence, test questions with a similar property are answered correctly after pretraining, whereas those with a high overlap with the whole passage tend to have lower performance  The pretraining also favors questions with short answers because the cloze construction process produces short answer spans  Also passages and questions which consist of tokens infrequent in the SQuAD training corpus receive a large boost after pretraining, since the unlabeled data covers a larger domain 
Performance on question types: Figure 2 shows the average gain in F1 score for different types of questions, when we pretrain on the clozes compared to the supervised case  This analysis is done on the $10\%$ split of the SQuAD training set  We consider two classifications of each question – one determined on the first word (usually a wh-word) of the question (Figure 2 (bottom)) and one based on the output of a separate question type classifier adapted from BIBREF21   We use the coarse grain labels namely Abbreviation (ABBR), Entity (ENTY), Description (DESC), Human (HUM), Location (LOC), Numeric (NUM) trained on a Logistic Regression classification system   While there is an improvement across the board, we find that abbreviation questions in particular receive a large boost  Also, "why" questions show the least improvement, which is in line with our expectation, since these usually require reasoning or world knowledge which cloze questions rarely require 
Conclusion
In this paper, we show that pre-training QA models with automatically constructed cloze questions improves the performance of the models significantly, especially when there are few labeled examples  The performance of the model trained only on the cloze questions is poor, validating the need for fine-tuning  Through regression analysis, we find that pretraining helps with questions which ask for factual information located in a specific part of the context  For future work, we plan to explore the active learning setup for this task – specifically, which passages and / or types of questions can we select to annotate, such that there is a maximum performance gain from fine-tuning  We also want to explore how to adapt cloze style pre-training to NLP tasks other than QA 
Acknowledgments
Bhuwan Dhingra is supported by NSF under grants CCF-1414030 and IIS-1250956 and by grants from Google  Danish Pruthi and Dheeraj Rajagopal are supported by the DARPA Big Mechanism program under ARO contract W911NF-14-1-0436 
Context 2: Introduction
Writing errors can occur in many different forms – from relatively simple punctuation and determiner errors, to mistakes including word tense and form, incorrect collocations and erroneous idioms  Automatically identifying all of these errors is a challenging task, especially as the amount of available annotated data is very limited  Rei2016 showed that while some error detection algorithms perform better than others, it is additional training data that has the biggest impact on improving performance 
Being able to generate realistic artificial data would allow for any grammatically correct text to be transformed into annotated examples containing writing errors, producing large amounts of additional training examples  Supervised error generation systems would also provide an efficient method for anonymising the source corpus – error statistics from a private corpus can be aggregated and applied to a different target text, obscuring sensitive information in the original examination scripts  However, the task of creating incorrect data is somewhat more difficult than might initially appear – naive methods for error generation can create data that does not resemble natural errors, thereby making downstream systems learn misleading or uninformative patterns 
Previous work on artificial error generation (AEG) has focused on specific error types, such as prepositions and determiners BIBREF0 , BIBREF1 , or noun number errors BIBREF2   Felice2014a investigated the use of linguistic information when generating artificial data for error correction, but also restricting the approach to only five error types  There has been very limited research on generating artificial data for all types, which is important for general-purpose error detection systems  For example, the error types investigated by Felice2014a cover only 35 74% of all errors present in the CoNLL 2014 training dataset, providing no additional information for the majority of errors 
In this paper, we investigate two supervised approaches for generating all types of artificial errors  We propose a framework for generating errors based on statistical machine translation (SMT), training a model to translate from correct into incorrect sentences  In addition, we describe a method for learning error patterns from an annotated corpus and transplanting them into error-free text  We evaluate the effect of introducing artificial data on two error detection benchmarks  Our results show that each method provides significant improvements over using only the available training set, and a combination of both gives an absolute improvement of 4 3% in INLINEFORM0 , without requiring any additional annotated data 
Error Generation Methods
We investigate two alternative methods for AEG  The models receive grammatically correct text as input and modify certain tokens to produce incorrect sequences  The alternative versions of each sentence are aligned using Levenshtein distance, allowing us to identify specific words that need to be marked as errors  While these alignments are not always perfect, we found them to be sufficient for practical purposes, since alternative alignments of similar sentences often result in the same binary labeling  Future work could explore more advanced alignment methods, such as proposed by felice-bryant-briscoe 
In Section SECREF4 , this automatically labeled data is then used for training error detection models 
Machine Translation
We treat AEG as a translation task – given a correct sentence as input, the system would learn to translate it to contain likely errors, based on a training corpus of parallel data  Existing SMT approaches are already optimised for identifying context patterns that correspond to specific output sequences, which is also required for generating human-like errors  The reverse of this idea, translating from incorrect to correct sentences, has been shown to work well for error correction tasks BIBREF2 , BIBREF3 , and round-trip translation has also been shown to be promising for correcting grammatical errors BIBREF4  
Following previous work BIBREF2 , BIBREF5 , we build a phrase-based SMT error generation system  During training, error-corrected sentences in the training data are treated as the source, and the original sentences written by language learners as the target  Pialign BIBREF6 is used to create a phrase translation table directly from model probabilities  In addition to default features, we add character-level Levenshtein distance to each mapping in the phrase table, as proposed by Felice:2014-CoNLL  Decoding is performed using Moses BIBREF7 and the language model used during decoding is built from the original erroneous sentences in the learner corpus  The IRSTLM Toolkit BIBREF8 is used for building a 5-gram language model with modified Kneser-Ney smoothing BIBREF9  
Pattern Extraction
We also describe a method for AEG using patterns over words and part-of-speech (POS) tags, extracting known incorrect sequences from a corpus of annotated corrections  This approach is based on the best method identified by Felice2014a, using error type distributions; while they covered only 5 error types, we relax this restriction and learn patterns for generating all types of errors 
The original and corrected sentences in the corpus are aligned and used to identify short transformation patterns in the form of (incorrect phrase, correct phrase)  The length of each pattern is the affected phrase, plus up to one token of context on both sides  If a word form changes between the incorrect and correct text, it is fully saved in the pattern, otherwise the POS tags are used for matching 
For example, the original sentence `We went shop on Saturday' and the corrected version `We went shopping on Saturday' would produce the following pattern:
(VVD shop_VV0 II, VVD shopping_VVG II)
After collecting statistics from the background corpus, errors can be inserted into error-free text  The learned patterns are now reversed, looking for the correct side of the tuple in the input sentence  We only use patterns with frequency INLINEFORM0 , which yields a total of 35,625 patterns from our training data  For each input sentence, we first decide how many errors will be generated (using probabilities from the background corpus) and attempt to create them by sampling from the collection of applicable patterns  This process is repeated until all the required errors have been generated or the sentence is exhausted  During generation, we try to balance the distribution of error types as well as keeping the same proportion of incorrect and correct sentences as in the background corpus BIBREF10   The required POS tags were generated with RASP BIBREF11 , using the CLAWS2 tagset 
Error Detection Model
We construct a neural sequence labeling model for error detection, following the previous work BIBREF12 , BIBREF13   The model receives a sequence of tokens as input and outputs a prediction for each position, indicating whether the token is correct or incorrect in the current context  The tokens are first mapped to a distributed vector space, resulting in a sequence of word embeddings  Next, the embeddings are given as input to a bidirectional LSTM BIBREF14 , in order to create context-dependent representations for every token  The hidden states from forward- and backward-LSTMs are concatenated for each word position, resulting in representations that are conditioned on the whole sequence  This concatenated vector is then passed through an additional feedforward layer, and a softmax over the two possible labels (correct and incorrect) is used to output a probability distribution for each token  The model is optimised by minimising categorical cross-entropy with respect to the correct labels  We use AdaDelta BIBREF15 for calculating an adaptive learning rate during training, which accounts for a higher baseline performance compared to previous results 
Evaluation
We trained our error generation models on the public FCE training set BIBREF16 and used them to generate additional artificial training data  Grammatically correct text is needed as the starting point for inserting artificial errors, and we used two different sources: 1) the corrected version of the same FCE training set on which the system is trained (450K tokens), and 2) example sentences extracted from the English Vocabulary Profile (270K tokens)   While there are other text corpora that could be used (e g , Wikipedia and news articles), our development experiments showed that keeping the writing style and vocabulary close to the target domain gives better results compared to simply including more data 
We evaluated our detection models on three benchmarks: the FCE test data (41K tokens) and the two alternative annotations of the CoNLL 2014 Shared Task dataset (30K tokens) BIBREF3   Each artificial error generation system was used to generate 3 different versions of the artificial data, which were then combined with the original annotated dataset and used for training an error detection system  Table TABREF1 contains example sentences from the error generation systems, highlighting each of the edits that are marked as errors 
The error detection results can be seen in Table TABREF4   We use INLINEFORM0 as the main evaluation measure, which was established as the preferred measure for error correction and detection by the CoNLL-14 shared task BIBREF3   INLINEFORM1 calculates a weighted harmonic mean of precision and recall, which assigns twice as much importance to precision – this is motivated by practical applications, where accurate predictions from an error detection system are more important compared to coverage  For comparison, we also report the performance of the error detection system by Rei2016, trained using the same FCE dataset 
The results show that error detection performance is substantially improved by making use of artificially generated data, created by any of the described methods  When comparing the error generation system by Felice2014a (FY14) with our pattern-based (PAT) and machine translation (MT) approaches, we see that the latter methods covering all error types consistently improve performance  While the added error types tend to be less frequent and more complicated to capture, the added coverage is indeed beneficial for error detection  Combining the pattern-based approach with the machine translation system (Ann+PAT+MT) gave the best overall performance on all datasets  The two frameworks learn to generate different types of errors, and taking advantage of both leads to substantial improvements in error detection 
We used the Approximate Randomisation Test BIBREF17 , BIBREF18 to calculate statistical significance and found that the improvement for each of the systems using artificial data was significant over using only manual annotation  In addition, the final combination system is also significantly better compared to the Felice2014a system, on all three datasets  While Rei2016 also report separate experiments that achieve even higher performance, these models were trained on a considerably larger proprietary corpus  In this paper we compare error detection frameworks trained on the same publicly available FCE dataset, thereby removing the confounding factor of dataset size and only focusing on the model architectures 
The error generation methods can generate alternative versions of the same input text – the pattern-based method randomly samples the error locations, and the SMT system can provide an n-best list of alternative translations  Therefore, we also investigated the combination of multiple error-generated versions of the input files when training error detection models  Figure FIGREF6 shows the INLINEFORM0 score on the development set, as the training data is increased by using more translations from the n-best list of the SMT system  These results reveal that allowing the model to see multiple alternative versions of the same file gives a distinct improvement – showing the model both correct and incorrect variations of the same sentences likely assists in learning a discriminative model 
Related Work
Our work builds on prior research into AEG  Brockett2006 constructed regular expressions for transforming correct sentences to contain noun number errors  Rozovskaya2010a learned confusion sets from an annotated corpus in order to generate preposition errors  Foster2009 devised a tool for generating errors for different types using patterns provided by the user or collected automatically from an annotated corpus  However, their method uses a limited number of edit operations and is thus unable to generate complex errors  Cahill2013 compared different training methodologies and showed that artificial errors helped correct prepositions  Felice2014a learned error type distributions for generating five types of errors, and the system in Section SECREF3 is an extension of this model  While previous work focused on generating a specific subset of error types, we explored two holistic approaches to AEG and showed that they are able to significantly improve error detection performance 
Conclusion
This paper investigated two AEG methods, in order to create additional training data for error detection  First, we explored a method using textual patterns learned from an annotated corpus, which are used for inserting errors into correct input text  In addition, we proposed formulating error generation as an MT framework, learning to translate from grammatically correct to incorrect sentences 
The addition of artificial data to the training process was evaluated on three error detection annotations, using the FCE and CoNLL 2014 datasets  Making use of artificial data provided improvements for all data generation methods  By relaxing the type restrictions and generating all types of errors, our pattern-based method consistently outperformed the system by Felice2014a  The combination of the pattern-based method with the machine translation approach gave further substantial improvements and the best performance on all datasets 
Context 3: Introduction
Multi-document summarization (MDS), the transformation of a set of documents into a short text containing their most important aspects, is a long-studied problem in NLP  Generated summaries have been shown to support humans dealing with large document collections in information seeking tasks BIBREF0 , BIBREF1 , BIBREF2   However, when exploring a set of documents manually, humans rarely write a fully-formulated summary for themselves  Instead, user studies BIBREF3 , BIBREF4 show that they note down important keywords and phrases, try to identify relationships between them and organize them accordingly  Therefore, we believe that the study of summarization with similarly structured outputs is an important extension of the traditional task 
A representation that is more in line with observed user behavior is a concept map BIBREF5 , a labeled graph showing concepts as nodes and relationships between them as edges (Figure FIGREF2 )  Introduced in 1972 as a teaching tool BIBREF6 , concept maps have found many applications in education BIBREF7 , BIBREF8 , for writing assistance BIBREF9 or to structure information repositories BIBREF10 , BIBREF11   For summarization, concept maps make it possible to represent a summary concisely and clearly reveal relationships  Moreover, we see a second interesting use case that goes beyond the capabilities of textual summaries: When concepts and relations are linked to corresponding locations in the documents they have been extracted from, the graph can be used to navigate in a document collection, similar to a table of contents  An implementation of this idea has been recently described by BIBREF12  
The corresponding task that we propose is concept-map-based MDS, the summarization of a document cluster in the form of a concept map  In order to develop and evaluate methods for the task, gold-standard corpora are necessary, but no suitable corpus is available  The manual creation of such a dataset is very time-consuming, as the annotation includes many subtasks  In particular, an annotator would need to manually identify all concepts in the documents, while only a few of them will eventually end up in the summary 
To overcome these issues, we present a corpus creation method that effectively combines automatic preprocessing, scalable crowdsourcing and high-quality expert annotations  Using it, we can avoid the high effort for single annotators, allowing us to scale to document clusters that are 15 times larger than in traditional summarization corpora  We created a new corpus of 30 topics, each with around 40 source documents on educational topics and a summarizing concept map that is the consensus of many crowdworkers (see Figure FIGREF3 ) 
As a crucial step of the corpus creation, we developed a new crowdsourcing scheme called low-context importance annotation  In contrast to traditional approaches, it allows us to determine important elements in a document cluster without requiring annotators to read all documents, making it feasible to crowdsource the task and overcome quality issues observed in previous work BIBREF13   We show that the approach creates reliable data for our focused summarization scenario and, when tested on traditional summarization corpora, creates annotations that are similar to those obtained by earlier efforts 
To summarize, we make the following contributions: (1) We propose a novel task, concept-map-based MDS (§ SECREF2 ), (2) present a new crowdsourcing scheme to create reference summaries (§ SECREF4 ), (3) publish a new dataset for the proposed task (§ SECREF5 ) and (4) provide an evaluation protocol and baseline (§ SECREF7 )  We make these resources publicly available under a permissive license 
Task
Concept-map-based MDS is defined as follows: Given a set of related documents, create a concept map that represents its most important content, satisfies a specified size limit and is connected 
We define a concept map as a labeled graph showing concepts as nodes and relationships between them as edges  Labels are arbitrary sequences of tokens taken from the documents, making the summarization task extractive  A concept can be an entity, abstract idea, event or activity, designated by its unique label  Good maps should be propositionally coherent, meaning that every relation together with the two connected concepts form a meaningful proposition 
The task is complex, consisting of several interdependent subtasks  One has to extract appropriate labels for concepts and relations and recognize different expressions that refer to the same concept across multiple documents  Further, one has to select the most important concepts and relations for the summary and finally organize them in a graph satisfying the connectedness and size constraints 
Related Work
Some attempts have been made to automatically construct concept maps from text, working with either single documents BIBREF14 , BIBREF9 , BIBREF15 , BIBREF16 or document clusters BIBREF17 , BIBREF18 , BIBREF19   These approaches extract concept and relation labels from syntactic structures and connect them to build a concept map  However, common task definitions and comparable evaluations are missing  In addition, only a few of them, namely Villalon 2012 and Valerio 2006, define summarization as their goal and try to compress the input to a substantially smaller size  Our newly proposed task and the created large-cluster dataset fill these gaps as they emphasize the summarization aspect of the task 
For the subtask of selecting summary-worthy concepts and relations, techniques developed for traditional summarization BIBREF20 and keyphrase extraction BIBREF21 are related and applicable  Approaches that build graphs of propositions to create a summary BIBREF22 , BIBREF23 , BIBREF24 , BIBREF25 seem to be particularly related, however, there is one important difference: While they use graphs as an intermediate representation from which a textual summary is then generated, the goal of the proposed task is to create a graph that is directly interpretable and useful for a user  In contrast, these intermediate graphs, e g  AMR, are hardly useful for a typical, non-linguist user 
For traditional summarization, the most well-known datasets emerged out of the DUC and TAC competitions  They provide clusters of news articles with gold-standard summaries  Extending these efforts, several more specialized corpora have been created: With regard to size, Nakano 2010 present a corpus of summaries for large-scale collections of web pages  Recently, corpora with more heterogeneous documents have been suggested, e g  BIBREF26 and BIBREF27   The corpus we present combines these aspects, as it has large clusters of heterogeneous documents, and provides a necessary benchmark to evaluate the proposed task 
For concept map generation, one corpus with human-created summary concept maps for student essays has been created BIBREF28   In contrast to our corpus, it only deals with single documents, requires a two orders of magnitude smaller amount of compression of the input and is not publicly available  
Other types of information representation that also model concepts and their relationships are knowledge bases, such as Freebase BIBREF29 , and ontologies  However, they both differ in important aspects: Whereas concept maps follow an open label paradigm and are meant to be interpretable by humans, knowledge bases and ontologies are usually more strictly typed and made to be machine-readable  Moreover, approaches to automatically construct them from text typically try to extract as much information as possible, while we want to summarize a document 
Low-Context Importance Annotation
Lloret 2013 describe several experiments to crowdsource reference summaries  Workers are asked to read 10 documents and then select 10 summary sentences from them for a reward of $0 05  They discovered several challenges, including poor work quality and the subjectiveness of the annotation task, indicating that crowdsourcing is not useful for this purpose 
To overcome these issues, we introduce a new task design, low-context importance annotation, to determine summary-worthy parts of documents  Compared to Lloret et al 's approach, it is more in line with crowdsourcing best practices, as the tasks are simple, intuitive and small BIBREF30 and workers receive reasonable payment BIBREF31   Most importantly, it is also much more efficient and scalable, as it does not require workers to read all documents in a cluster 
Task Design
We break down the task of importance annotation to the level of single propositions  The goal of our crowdsourcing scheme is to obtain a score for each proposition indicating its importance in a document cluster, such that a ranking according to the score would reveal what is most important and should be included in a summary  In contrast to other work, we do not show the documents to the workers at all, but provide only a description of the document cluster's topic along with the propositions  This ensures that tasks are small, simple and can be done quickly (see Figure FIGREF4 ) 
In preliminary tests, we found that this design, despite the minimal context, works reasonably on our focused clusters on common educational topics  For instance, consider Figure FIGREF4 : One can easily say that P1 is more important than P2 without reading the documents 
We distinguish two task variants:
Instead of enforcing binary importance decisions, we use a 5-point Likert-scale to allow more fine-grained annotations  The obtained labels are translated into scores (5  1) and the average of all scores for a proposition is used as an estimate for its importance  This follows the idea that while single workers might find the task subjective, the consensus of multiple workers, represented in the average score, tends to be less subjective due to the “wisdom of the crowd”  We randomly group five propositions into a task 
As an alternative, we use a second task design based on pairwise comparisons  Comparisons are known to be easier to make and more consistent BIBREF32 , but also more expensive, as the number of pairs grows quadratically with the number of objects  To reduce the cost, we group five propositions into a task and ask workers to order them by importance per drag-and-drop  From the results, we derive pairwise comparisons and use TrueSkill BIBREF35 , a powerful Bayesian rank induction model BIBREF34 , to obtain importance estimates for each proposition 
Pilot Study
To verify the proposed approach, we conducted a pilot study on Amazon Mechanical Turk using data from TAC2008 BIBREF36   We collected importance estimates for 474 propositions extracted from the first three clusters using both task designs  Each Likert-scale task was assigned to 5 different workers and awarded $0 06  For comparison tasks, we also collected 5 labels each, paid $0 05 and sampled around 7% of all possible pairs  We submitted them in batches of 100 pairs and selected pairs for subsequent batches based on the confidence of the TrueSkill model 
Following the observations of Lloret 2013, we established several measures for quality control  First, we restricted our tasks to workers from the US with an approval rate of at least 95%  Second, we identified low quality workers by measuring the correlation of each worker's Likert-scores with the average of the other four scores  The worst workers (at most 5% of all labels) were removed 
In addition, we included trap sentences, similar as in BIBREF13 , in around 80 of the tasks  In contrast to Lloret et al 's findings, both an obvious trap sentence (This sentence is not important) and a less obvious but unimportant one (Barack Obama graduated from Harvard Law) were consistently labeled as unimportant (1 08 and 1 14), indicating that the workers did the task properly 
For Likert-scale tasks, we follow Snow 2008 and calculate agreement as the average Pearson correlation of a worker's Likert-score with the average score of the remaining workers  This measure is less strict than exact label agreement and can account for close labels and high- or low-scoring workers  We observe a correlation of 0 81, indicating substantial agreement  For comparisons, the majority agreement is 0 73  To further examine the reliability of the collected data, we followed the approach of Kiritchenko 2016 and simply repeated the crowdsourcing for one of the three topics  Between the importance estimates calculated from the first and second run, we found a Pearson correlation of 0 82 (Spearman 0 78) for Likert-scale tasks and 0 69 (Spearman 0 66) for comparison tasks  This shows that the approach, despite the subjectiveness of the task, allows us to collect reliable annotations 
In addition to the reliability studies, we extrinsically evaluated the annotations in the task of summary evaluation  For each of the 58 peer summaries in TAC2008, we calculated a score as the sum of the importance estimates of the propositions it contains  Table TABREF13 shows how these peer scores, averaged over the three topics, correlate with the manual responsiveness scores assigned during TAC in comparison to ROUGE-2 and Pyramid scores  The results demonstrate that with both task designs, we obtain importance annotations that are similarly useful for summary evaluation as pyramid annotations or gold-standard summaries (used for ROUGE) 
Based on the pilot study, we conclude that the proposed crowdsourcing scheme allows us to obtain proper importance annotations for propositions  As workers are not required to read all documents, the annotation is much more efficient and scalable as with traditional methods 
Corpus Creation
This section presents the corpus construction process, as outlined in Figure FIGREF16 , combining automatic preprocessing, scalable crowdsourcing and high-quality expert annotations to be able to scale to the size of our document clusters  For every topic, we spent about $150 on crowdsourcing and 1 5h of expert annotations, while just a single annotator would already need over 8 hours (at 200 words per minute) to read all documents of a topic

ground_truth: Unanswerable

Unanswerable

prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. What was the baseline used?
Context 3: 
Source Data
As a starting point, we used the DIP corpus BIBREF37 , a collection of 49 clusters of 100 web pages on educational topics (e g  bullying, homeschooling, drugs) with a short description of each topic  It was created from a large web crawl using state-of-the-art information retrieval  We selected 30 of the topics for which we created the necessary concept map annotations 
Proposition Extraction
As concept maps consist of propositions expressing the relation between concepts (see Figure FIGREF2 ), we need to impose such a structure upon the plain text in the document clusters  This could be done by manually annotating spans representing concepts and relations, however, the size of our clusters makes this a huge effort: 2288 sentences per topic (69k in total) need to be processed  Therefore, we resort to an automatic approach 
The Open Information Extraction paradigm BIBREF38 offers a representation very similar to the desired one  For instance, from
Students with bad credit history should not lose hope and apply for federal loans with the FAFSA 
Open IE systems extract tuples of two arguments and a relation phrase representing propositions:
(s  with bad credit history, should not lose, hope)
(s  with bad credit history, apply for, federal loans with the FAFSA)
While the relation phrase is similar to a relation in a concept map, many arguments in these tuples represent useful concepts  We used Open IE 4, a state-of-the-art system BIBREF39 to process all sentences  After removing duplicates, we obtained 4137 tuples per topic 
Since we want to create a gold-standard corpus, we have to ensure that we produce high-quality data  We therefore made use of the confidence assigned to every extracted tuple to filter out low quality ones  To ensure that we do not filter too aggressively (and miss important aspects in the final summary), we manually annotated 500 tuples sampled from all topics for correctness  On the first 250 of them, we tuned the filter threshold to 0 5, which keeps 98 7% of the correct extractions in the unseen second half  After filtering, a topic had on average 2850 propositions (85k in total) 
Proposition Filtering
Despite the similarity of the Open IE paradigm, not every extracted tuple is a suitable proposition for a concept map  To reduce the effort in the subsequent steps, we therefore want to filter out unsuitable ones  A tuple is suitable if it (1) is a correct extraction, (2) is meaningful without any context and (3) has arguments that represent proper concepts  We created a guideline explaining when to label a tuple as suitable for a concept map and performed a small annotation study  Three annotators independently labeled 500 randomly sampled tuples  The agreement was 82% ( INLINEFORM0 )  We found tuples to be unsuitable mostly because they had unresolvable pronouns, conflicting with (2), or arguments that were full clauses or propositions, conflicting with (3), while (1) was mostly taken care of by the confidence filtering in § SECREF21  
Due to the high number of tuples we decided to automate the filtering step  We trained a linear SVM on the majority voted annotations  As features, we used the extraction confidence, length of arguments and relations as well as part-of-speech tags, among others  To ensure that the automatic classification does not remove suitable propositions, we tuned the classifier to avoid false negatives  In particular, we introduced class weights, improving precision on the negative class at the cost of a higher fraction of positive classifications  Additionally, we manually verified a certain number of the most uncertain negative classifications to further improve performance  When 20% of the classifications are manually verified and corrected, we found that our model trained on 350 labeled instances achieves 93% precision on negative classifications on the unseen 150 instances  We found this to be a reasonable trade-off of automation and data quality and applied the model to the full dataset 
The classifier filtered out 43% of the propositions, leaving 1622 per topic  We manually examined the 17k least confident negative classifications and corrected 955 of them  We also corrected positive classifications for certain types of tuples for which we knew the classifier to be imprecise  Finally, each topic was left with an average of 1554 propositions (47k in total) 
Importance Annotation
Given the propositions identified in the previous step, we now applied our crowdsourcing scheme as described in § SECREF4 to determine their importance  To cope with the large number of propositions, we combine the two task designs: First, we collect Likert-scores from 5 workers for each proposition, clean the data and calculate average scores  Then, using only the top 100 propositions according to these scores, we crowdsource 10% of all possible pairwise comparisons among them  Using TrueSkill, we obtain a fine-grained ranking of the 100 most important propositions 
For Likert-scores, the average agreement over all topics is 0 80, while the majority agreement for comparisons is 0 78  We repeated the data collection for three randomly selected topics and found the Pearson correlation between both runs to be 0 73 (Spearman 0 73) for Likert-scores and 0 72 (Spearman 0 71) for comparisons  These figures show that the crowdsourcing approach works on this dataset as reliably as on the TAC documents 
In total, we uploaded 53k scoring and 12k comparison tasks to Mechanical Turk, spending $4425 45 including fees  From the fine-grained ranking of the 100 most important propositions, we select the top 50 per topic to construct a summary concept map in the subsequent steps 
Proposition Revision
Having a manageable number of propositions, an annotator then applied a few straightforward transformations that correct common errors of the Open IE system  First, we break down propositions with conjunctions in either of the arguments into separate propositions per conjunct, which the Open IE system sometimes fails to do  And second, we correct span errors that might occur in the argument or relation phrases, especially when sentences were not properly segmented  As a result, we have a set of high quality propositions for our concept map, consisting of, due to the first transformation, 56 1 propositions per topic on average 
Concept Map Construction
In this final step, we connect the set of important propositions to form a graph  For instance, given the following two propositions
(student, may borrow, Stafford Loan)
(the student, does not have, a credit history)
one can easily see, although the first arguments differ slightly, that both labels describe the concept student, allowing us to build a concept map with the concepts student, Stafford Loan and credit history  The annotation task thus involves deciding which of the available propositions to include in the map, which of their concepts to merge and, when merging, which of the available labels to use  As these decisions highly depend upon each other and require context, we decided to use expert annotators rather than crowdsource the subtasks 
Annotators were given the topic description and the most important, ranked propositions  Using a simple annotation tool providing a visualization of the graph, they could connect the propositions step by step  They were instructed to reach a size of 25 concepts, the recommended maximum size for a concept map BIBREF6   Further, they should prefer more important propositions and ensure connectedness  When connecting two propositions, they were asked to keep the concept label that was appropriate for both propositions  To support the annotators, the tool used ADW BIBREF40 , a state-of-the-art approach for semantic similarity, to suggest possible connections  The annotation was carried out by graduate students with a background in NLP after receiving an introduction into the guidelines and tool and annotating a first example 
If an annotator was not able to connect 25 concepts, she was allowed to create up to three synthetic relations with freely defined labels, making the maps slightly abstractive  On average, the constructed maps have 0 77 synthetic relations, mostly connecting concepts whose relation is too obvious to be explicitly stated in text (e g  between Montessori teacher and Montessori education) 
To assess the reliability of this annotation step, we had the first three maps created by two annotators  We casted the task of selecting propositions to be included in the map as a binary decision task and observed an agreement of 84% ( INLINEFORM0 )  Second, we modeled the decision which concepts to join as a binary decision on all pairs of common concepts, observing an agreement of 95% ( INLINEFORM1 )  And finally, we compared which concept labels the annotators decided to include in the final map, observing 85% ( INLINEFORM2 ) agreement  Hence, the annotation shows substantial agreement BIBREF41  
Corpus Analysis
In this section, we describe our newly created corpus, which, in addition to having summaries in the form of concept maps, differs from traditional summarization corpora in several aspects 
Document Clusters
The corpus consists of document clusters for 30 different topics  Each of them contains around 40 documents with on average 2413 tokens, which leads to an average cluster size of 97,880 token  With these characteristics, the document clusters are 15 times larger than typical DUC clusters of ten documents and five times larger than the 25-document-clusters (Table TABREF26 )  In addition, the documents are also more variable in terms of length, as the (length-adjusted) standard deviation is twice as high as in the other corpora  With these properties, the corpus represents an interesting challenge towards real-world application scenarios, in which users typically have to deal with much more than ten documents 
Because we used a large web crawl as the source for our corpus, it contains documents from a variety of genres  To further analyze this property, we categorized a sample of 50 documents from the corpus  Among them, we found professionally written articles and blog posts (28%), educational material for parents and kids (26%), personal blog posts (16%), forum discussions and comments (12%), commented link collections (12%) and scientific articles (6%) 
In addition to the variety of genres, the documents also differ in terms of language use  To capture this property, we follow Zopf 2016 and compute, for every topic, the average Jensen-Shannon divergence between the word distribution of one document and the word distribution in the remaining documents  The higher this value is, the more the language differs between documents  We found the average divergence over all topics to be 0 3490, whereas it is 0 3019 in DUC 2004 and 0 3188 in TAC 2008A 
Concept Maps
As Table TABREF33 shows, each of the 30 reference concept maps has exactly 25 concepts and between 24 and 28 relations  Labels for both concepts and relations consist on average of 3 2 tokens, whereas the latter are a bit shorter in characters 
To obtain a better picture of what kind of text spans have been used as labels, we automatically tagged them with their part-of-speech and determined their head with a dependency parser  Concept labels tend to be headed by nouns (82%) or verbs (15%), while they also contain adjectives, prepositions and determiners  Relation labels, on the other hand, are almost always headed by a verb (94%) and contain prepositions, nouns and particles in addition  These distributions are very similar to those reported by Villalon 2010 for their (single-document) concept map corpus 
Analyzing the graph structure of the maps, we found that all of them are connected  They have on average 7 2 central concepts with more than one relation, while the remaining ones occur in only one proposition  We found that achieving a higher number of connections would mean compromising importance, i e  including less important propositions, and decided against it 
Baseline Experiments
In this section, we briefly describe a baseline and evaluation scripts that we release, with a detailed documentation, along with the corpus 
Conclusion
In this work, we presented low-context importance annotation, a novel crowdsourcing scheme that we used to create a new benchmark corpus for concept-map-based MDS  The corpus has large-scale document clusters of heterogeneous web documents, posing a challenging summarization task  Together with the corpus, we provide implementations of a baseline method and evaluation scripts and hope that our efforts facilitate future research on this variant of summarization 
Acknowledgments
We would like to thank Teresa Botschen, Andreas Hanselowski and Markus Zopf for their help with the annotation work and Christian Meyer for his valuable feedback  This work has been supported by the German Research Foundation as part of the Research Training Group “Adaptive Preparation of Information from Heterogeneous Sources” (AIPHES) under grant No  GRK 1994/1 

ground_truth: error detection system by Rei2016

error detection system by Rei2016

prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. What type of evaluation is proposed for this task?
Context 4:  We can observe that for example pages about news tend to have more sadness and anger posts, while pages about cooking and tv-shows have a high percentage of joy posts  We will use this information to find the best set of pages for a given target domain (see Section SECREF5 ) 
Model
There are two main decisions to be taken in developing our model: (i) which Facebook pages to select as training data, and (ii) which features to use to train the model, which we discuss below  Specifically, we first set on a subset of pages and then experiment with features  Further exploration of the interaction between choice of pages and choice of features is left to future work, and partly discussed in Section SECREF6   For development, we use a small portion of the Affective data set described in Section SECREF4 , that is the portion that had been released as development set for SemEval's 2007 Task 14 BIBREF7 , which contains 250 annotated sentences (Affective development, Section SECREF4 )  All results reported in this section are on this dataset  The test set of Task 14 as well as the other two datasets described in Section SECREF3 will be used to evaluate the final models (Section SECREF4 ) 
Selecting Facebook pages
Although page selection is a crucial ingredient of this approach, which we believe calls for further and deeper, dedicated investigation, for the experiments described here we took a rather simple approach  First, we selected the pages that would provide training data based on intuition and availability, then chose different combinations according to results of a basic model run on development data, and eventually tested feature combinations, still on the development set 
For the sake of simplicity and transparency, we first trained an SVM with a simple bag-of-words model and default parameters as per the Scikit-learn implementation BIBREF13 on different combinations of pages  Based on results of the attempted combinations as well as on the distribution of emotions in the development dataset (Figure FIGREF9 ), we selected a best model (B-M), namely the combined set of Time, The Guardian and Disney, which yields the highest results on development data  Time and The Guardian perform well on most emotions but Disney helps to boost the performance for the Joy class 
Features
In selecting appropriate features, we mainly relied on previous work and intuition  We experimented with different combinations, and all tests were still done on Affective development, using the pages for the best model (B-M) described above as training data  Results are in Table TABREF20   Future work will further explore the simultaneous selection of features and page combinations 
We use a set of basic text-based features to capture the emotion class  These include a tf-idf bag-of-words feature, word (2-3) and character (2-5) ngrams, and features related to the presence of negation words, and to the usage of punctuation 
This feature is used in all unsupervised models as a source of information, and we mainly include it to assess its contribution, but eventually do not use it in our final model 
We used the NRC10 Lexicon because it performed best in the experiments by BIBREF10 , which is built around the emotions anger, anticipation, disgust, fear, joy, sadness, and surprise, and the valence values positive and negative  For each word in the lexicon, a boolean value indicating presence or absence is associated to each emotion  For a whole sentence, a global score per emotion can be obtained by summing the vectors for all content words of that sentence included in the lexicon, and used as feature 
As additional feature, we also included Word Embeddings, namely distributed representations of words in a vector space, which have been exceptionally successful in boosting performance in a plethora of NLP tasks  We use three different embeddings:
Google embeddings: pre-trained embeddings trained on Google News and obtained with the skip-gram architecture described in BIBREF14   This model contains 300-dimensional vectors for 3 million words and phrases 
Facebook embeddings: embeddings that we trained on our scraped Facebook pages for a total of 20,000 sentences  Using the gensim library BIBREF15 , we trained the embeddings with the following parameters: window size of 5, learning rate of 0 01 and dimensionality of 100  We filtered out words with frequency lower than 2 occurrences 
Retrofitted embeddings: Retrofitting BIBREF16 has been shown as a simple but efficient way of informing trained embeddings with additional information derived from some lexical resource, rather than including it directly at the training stage, as it's done for example to create sense-aware BIBREF17 or sentiment-aware BIBREF18 embeddings  In this work, we retrofit general embeddings to include information about emotions, so that emotion-similar words can get closer in space  Both the Google as well as our Facebook embeddings were retrofitted with lexical information obtained from the NRC10 Lexicon mentioned above, which provides emotion-similarity for each token  Note that differently from the previous two types of embeddings, the retrofitted ones do rely on handcrafted information in the form of a lexical resource 
Results on development set
We report precision, recall, and f-score on the development set  The average f-score is reported as micro-average, to better account for the skewed distribution of the classes as well as in accordance to what is usually reported for this task BIBREF19  
From Table TABREF20 we draw three main observations  First, a simple tf-idf bag-of-word mode works already very well, to the point that the other textual and lexicon-based features don't seem to contribute to the overall f-score (0 368), although there is a rather substantial variation of scores per class  Second, Google embeddings perform a lot better than Facebook embeddings, and this is likely due to the size of the corpus used for training  Retrofitting doesn't seem to help at all for the Google embeddings, but it does boost the Facebook embeddings, leading to think that with little data, more accurate task-related information is helping, but corpus size matters most  Third, in combination with embeddings, all features work better than just using tf-idf, but removing the Lexicon feature, which is the only one based on hand-crafted resources, yields even better results  Then our best model (B-M) on development data relies entirely on automatically obtained information, both in terms of training data as well as features 
Results
In Table TABREF26 we report the results of our model on the three datasets standardly used for the evaluation of emotion classification, which we have described in Section SECREF3  
Our B-M model relies on subsets of Facebook pages for training, which were chosen according to their performance on the development set as well as on the observation of emotions distribution on different pages and in the different datasets, as described in Section SECREF4   The feature set we use is our best on the development set, namely all the features plus Google-based embeddings, but excluding the lexicon  This makes our approach completely independent of any manual annotation or handcrafted resource  Our model's performance is compared to the following systems, for which results are reported in the referred literature  Please note that no other existing model was re-implemented, and results are those reported in the respective papers 
Discussion, conclusions and future work
We have explored the potential of using Facebook reactions in a distant supervised setting to perform emotion classification  The evaluation on standard benchmarks shows that models trained as such, especially when enhanced with continuous vector representations, can achieve competitive results without relying on any handcrafted resource  An interesting aspect of our approach is the view to domain adaptation via the selection of Facebook pages to be used as training data 
We believe that this approach has a lot of potential, and we see the following directions for improvement  Feature-wise, we want to train emotion-aware embeddings, in the vein of work by tang:14, and iacobacci2015sensembed  Retrofitting FB-embeddings trained on a larger corpus might also be successful, but would rely on an external lexicon 
The largest room for yielding not only better results but also interesting insights on extensions of this approach lies in the choice of training instances, both in terms of Facebook pages to get posts from, as well as in which posts to select from the given pages  For the latter, one could for example only select posts that have a certain length, ignore posts that are only quotes or captions to images, or expand posts by including content from linked html pages, which might provide larger and better contexts BIBREF23   Additionally, and most importantly, one could use an entropy-based measure to select only posts that have a strong emotion rather than just considering the majority emotion as training label  For the former, namely the choice of Facebook pages, which we believe deserves the most investigation, one could explore several avenues, especially in relation to stance-based issues BIBREF24   In our dataset, for example, a post about Chile beating Colombia in a football match during the Copa America had very contradictory reactions, depending on which side readers would cheer for  Similarly, the very same political event, for example, would get very different reactions from readers if it was posted on Fox News or The Late Night Show, as the target audience is likely to feel very differently about the same issue  This also brings up theoretical issues related more generally to the definition of the emotion detection task, as it's strongly dependent on personal traits of the audience  Also, in this work, pages initially selected on availability and intuition were further grouped into sets to make training data according to performance on development data, and label distribution  Another criterion to be exploited would be vocabulary overlap between the pages and the datasets 
Lastly, we could develop single models for each emotion, treating the problem as a multi-label task  This would even better reflect the ambiguity and subjectivity intrinsic to assigning emotions to text, where content could be at same time joyful or sad, depending on the reader 
Acknowledgements
In addition to the anonymous reviewers, we want to thank Lucia Passaro and Barbara Plank for insightful discussions, and for providing comments on draft versions of this paper 

ground_truth: Answer with content missing: (Evaluation Metrics section) Precision, Recall, F1-scores, Strict match, METEOR, ROUGE-2