prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. What was the baseline for this task?
Context 1: Introduction
Propaganda aims at influencing people's mindset with the purpose of advancing a specific agenda  In the Internet era, thanks to the mechanism of sharing in social networks, propaganda campaigns have the potential of reaching very large audiences BIBREF0, BIBREF1, BIBREF2 
Propagandist news articles use specific techniques to convey their message, such as whataboutism, red Herring, and name calling, among many others (cf  Section SECREF3)  Whereas proving intent is not easy, we can analyse the language of a claim/article and look for the use of specific propaganda techniques  Going at this fine-grained level can yield more reliable systems and it also makes it possible to explain to the user why an article was judged as propagandist by an automatic system 
With this in mind, we organised the shared task on fine-grained propaganda detection at the NLP4IF@EMNLP-IJCNLP 2019 workshop  The task is based on a corpus of news articles annotated with an inventory of 18 propagandist techniques at the fragment level  We hope that the corpus would raise interest outside of the community of researchers studying propaganda  For example, the techniques related to fallacies and the ones relying on emotions might provide a novel setting for researchers interested in Argumentation and Sentiment Analysis 
Related Work
Propaganda has been tackled mostly at the article level  BIBREF3 created a corpus of news articles labelled as propaganda, trusted, hoax, or satire  BIBREF4 experimented with a binarized version of that corpus: propaganda vs  the other three categories  BIBREF5 annotated a large binary corpus of propagandist vs  non-propagandist articles and proposed a feature-based system for discriminating between them  In all these cases, the labels were obtained using distant supervision, assuming that all articles from a given news outlet share the label of that outlet, which inevitably introduces noise BIBREF6 
A related field is that of computational argumentation which, among others, deals with some logical fallacies related to propaganda  BIBREF7 presented a corpus of Web forum discussions with instances of ad hominem fallacy  BIBREF8, BIBREF9 introduced Argotario, a game to educate people to recognize and create fallacies, a by-product of which is a corpus with $1 3k$ arguments annotated with five fallacies such as ad hominem, red herring and irrelevant authority, which directly relate to propaganda 
Unlike BIBREF8, BIBREF9, BIBREF7, our corpus uses 18 techniques annotated on the same set of news articles  Moreover, our annotations aim at identifying the minimal fragments related to a technique instead of flagging entire arguments 
The most relevant related work is our own, which is published in parallel to this paper at EMNLP-IJCNLP 2019 BIBREF10 and describes a corpus that is a subset of the one used for this shared task 
Propaganda Techniques
Propaganda uses psychological and rhetorical techniques to achieve its objective  Such techniques include the use of logical fallacies and appeal to emotions  For the shared task, we use 18 techniques that can be found in news articles and can be judged intrinsically, without the need to retrieve supporting information from external resources  We refer the reader to BIBREF10 for more details on the propaganda techniques; below we report the list of techniques:
Propaganda Techniques ::: 1  Loaded language 
Using words/phrases with strong emotional implications (positive or negative) to influence an audience BIBREF11 
Propaganda Techniques ::: 2  Name calling or labeling 
Labeling the object of the propaganda as something the target audience fears, hates, finds undesirable or otherwise loves or praises BIBREF12 
Propaganda Techniques ::: 3  Repetition 
Repeating the same message over and over again, so that the audience will eventually accept it BIBREF13, BIBREF12 
Propaganda Techniques ::: 4  Exaggeration or minimization 
Either representing something in an excessive manner: making things larger, better, worse, or making something seem less important or smaller than it actually is BIBREF14, e g , saying that an insult was just a joke 
Propaganda Techniques ::: 5  Doubt 
Questioning the credibility of someone or something 
Propaganda Techniques ::: 6  Appeal to fear/prejudice 
Seeking to build support for an idea by instilling anxiety and/or panic in the population towards an alternative, possibly based on preconceived judgments 
Propaganda Techniques ::: 7  Flag-waving 
Playing on strong national feeling (or with respect to a group, e g , race, gender, political preference) to justify or promote an action or idea BIBREF15 
Propaganda Techniques ::: 8  Causal oversimplification 
Assuming one cause when there are multiple causes behind an issue  We include scapegoating as well: the transfer of the blame to one person or group of people without investigating the complexities of an issue 
Propaganda Techniques ::: 9  Slogans 
A brief and striking phrase that may include labeling and stereotyping  Slogans tend to act as emotional appeals BIBREF16 
Propaganda Techniques ::: 10  Appeal to authority 
Stating that a claim is true simply because a valid authority/expert on the issue supports it, without any other supporting evidence BIBREF17  We include the special case where the reference is not an authority/expert, although it is referred to as testimonial in the literature BIBREF14 
Propaganda Techniques ::: 11  Black-and-white fallacy, dictatorship 
Presenting two alternative options as the only possibilities, when in fact more possibilities exist BIBREF13  As an extreme case, telling the audience exactly what actions to take, eliminating any other possible choice (dictatorship) 
Propaganda Techniques ::: 12  Thought-terminating cliché 
Words or phrases that discourage critical thought and meaningful discussion about a given topic  They are typically short and generic sentences that offer seemingly simple answers to complex questions or that distract attention away from other lines of thought BIBREF18 
Propaganda Techniques ::: 13  Whataboutism 
Discredit an opponent's position by charging them with hypocrisy without directly disproving their argument BIBREF19 
Propaganda Techniques ::: 14  Reductio ad Hitlerum 
Persuading an audience to disapprove an action or idea by suggesting that the idea is popular with groups hated in contempt by the target audience  It can refer to any person or concept with a negative connotation BIBREF20 
Propaganda Techniques ::: 15  Red herring 
Introducing irrelevant material to the issue being discussed, so that everyone's attention is diverted away from the points made BIBREF11  Those subjected to a red herring argument are led away from the issue that had been the focus of the discussion and urged to follow an observation or claim that may be associated with the original claim, but is not highly relevant to the issue in dispute BIBREF20 
Propaganda Techniques ::: 16  Bandwagon 
Attempting to persuade the target audience to join in and take the course of action because “everyone else is taking the same action” BIBREF15 
Propaganda Techniques ::: 17  Obfuscation, intentional vagueness, confusion 
Using deliberately unclear words, to let the audience have its own interpretation BIBREF21, BIBREF11  For instance, when an unclear phrase with multiple possible meanings is used within the argument and, therefore, it does not really support the conclusion 
Propaganda Techniques ::: 18  Straw man 
When an opponent's proposition is substituted with a similar one which is then refuted in place of the original BIBREF22 
Tasks
The shared task features two subtasks:
Tasks ::: Fragment-Level Classification task (FLC) 
Given a news article, detect all spans of the text in which a propaganda technique is used  In addition, for each span the propaganda technique applied must be identified 
Tasks ::: Sentence-Level Classification task (SLC) 
A sentence is considered propagandist if it contains at least one propagandist fragment  We then define a binary classification task in which, given a sentence, the correct label, either propaganda or non-propaganda, is to be predicted 
Data
The input for both tasks consists of news articles in free-text format, collected from 36 propagandist and 12 non-propagandist news outlets and then annotated by professional annotators  More details about the data collection and the annotation, as well as statistics about the corpus can be found in BIBREF10, where an earlier version of the corpus is described, which includes 450 news articles  We further annotated 47 additional articles for the purpose of the shared task using the same protocol and the same annotators 
The training, the development, and the test partitions of the corpus used for the shared task consist of 350, 61, and 86 articles and of 16,965, 2,235, and 3,526 sentences, respectively  Figure FIGREF15 shows an annotated example, which contains several propaganda techniques  For example, the fragment babies on line 1 is an instance of both Name_Calling and Labeling  Note that the fragment not looking as though Trump killed his grandma on line 4 is an instance of Exaggeration_or_Minimisation and it overlaps with the fragment killed his grandma, which is an instance of Loaded_Language 
Table TABREF23 reports the total number of instances per technique and the percentage with respect to the total number of annotations, for the training and for the development sets 
Setup
The shared task had two phases: In the development phase, the participants were provided labeled training and development datasets; in the testing phase, testing input was further provided 
The participants tried to achieve the best performance on the development set  A live leaderboard kept track of the submissions 
The test set was released and the participants had few days to make final predictions 
In phase 2, no immediate feedback on the submissions was provided  The winner was determined based on the performance on the test set 
Evaluation ::: FLC task 
FLC is a composition of two subtasks: the identification of the propagandist text fragments and the identification of the techniques used (18-way classification task)  While F$_1$ measure is appropriate for a multi-class classification task, we modified it to account for partial matching between the spans; see BIBREF10 for more details  We further computed an F$_1$ value for each propaganda technique (not shown below for the sake of saving space, but available on the leaderboard) 
Evaluation ::: SLC task 
SLC is a binary classification task with imbalanced data  Therefore, the official evaluation measure for the task is the standard F$_1$ measure  We further report Precision and Recall 
Baselines
The baseline system for the SLC task is a very simple logistic regression classifier with default parameters, where we represent the input instances with a single feature: the length of the sentence  The performance of this baseline on the SLC task is shown in Tables TABREF33 and TABREF34 
The baseline for the FLC task generates spans and selects one of the 18 techniques randomly  The inefficacy of such a simple random baseline is illustrated in Tables TABREF36 and TABREF41 
Participants and Approaches
A total of 90 teams registered for the shared task, and 39 of them submitted predictions for a total of 3,065 submissions  For the FLC task, 21 teams made a total of 527 submissions, and for the SLC task, 35 teams made a total of 2,538 submissions 
Below, we give an overview of the approaches as described in the participants' papers  Tables TABREF28 and TABREF29 offer a high-level summary 
Participants and Approaches ::: Teams Participating in the Fragment-Level Classification Only
Team newspeak BIBREF23 achieved the best results on the test set for the FLC task using 20-way word-level classification based on BERT BIBREF24: a word could belong to one of the 18 propaganda techniques, to none of them, or to an auxiliary (token-derived) class  The team fed one sentence at a time in order to reduce the workload  In addition to experimenting with an out-of-the-box BERT, they also tried unsupervised fine-tuning both on the 1M news dataset and on Wikipedia  Their best model was based on the uncased base model of BERT, with 12 Transformer layers BIBREF25, and 110 million parameters  Moreover, oversampling of the least represented classes proved to be crucial for the final performance  Finally, careful analysis has shown that the model pays special attention to adjectives and adverbs 
Team Stalin BIBREF26 focused on data augmentation to address the relatively small size of the data for fine-tuning contextual embedding representations based on ELMo BIBREF27, BERT, and Grover BIBREF28  The balancing of the embedding space was carried out by means of synthetic minority class over-sampling  Then, the learned representations were fed into an LSTM 
Participants and Approaches ::: Teams Participating in the Sentence-Level Classification Only
Team CAUnLP BIBREF29 used two context-aware representations based on BERT  In the first representation, the target sentence is followed by the title of the article  In the second representation, the previous sentence is also added  They performed subsampling in order to deal with class imbalance, and experimented with BERT$_{BASE}$ and BERT$_{LARGE}$
Team LIACC BIBREF30 used hand-crafted features and pre-trained ELMo embeddings  They also observed a boost in performance when balancing the dataset by dropping some negative examples 
Team JUSTDeep BIBREF31 used a combination of models and features, including word embeddings based on GloVe BIBREF32 concatenated with vectors representing affection and lexical features  These were combined in an ensemble of supervised models: bi-LSTM, XGBoost, and variations of BERT 
Team YMJA BIBREF33 also based their approach on fine-tuned BERT  Inspired by kaggle competitions on sentiment analysis, they created an ensemble of models via cross-validation 
Team jinfen BIBREF34 used a logistic regression model fed with a manifold of representations, including TF IDF and BERT vectors, as well as vocabularies and readability measures 
Team Tha3aroon BIBREF35 implemented an ensemble of three classifiers: two based on BERT and one based on a universal sentence encoder BIBREF36 
Team NSIT BIBREF37 explored three of the most popular transfer learning models: various versions of ELMo, BERT, and RoBERTa BIBREF38 
Team Mindcoders BIBREF39 combined BERT, Bi-LSTM and Capsule networks BIBREF40 into a single deep neural network and pre-trained the resulting network on corpora used for related tasks, e g , emotion classification 
Finally, team ltuorp BIBREF41 used an attention transformer using BERT trained on Wikipedia and BookCorpus 
Participants and Approaches ::: Teams Participating in Both Tasks
Team MIC-CIS BIBREF42 participated in both tasks  For the sentence-level classification, they used a voting ensemble including logistic regression, convolutional neural networks, and BERT, in all cases using FastText embeddings BIBREF43 and pre-trained BERT models  Beside these representations, multiple features of readability, sentiment and emotions were considered  For the fragment-level task, they used a multi-task neural sequence tagger, based on LSTM-CRF BIBREF44, in conjunction with linguistic features  Finally, they applied sentence- and fragment-level models jointly 
Team CUNLP BIBREF45 considered two approaches for the sentence-level task  The first approach was based on fine-tuning BERT  The second approach complemented the fine-tuned BERT approach by feeding its decision into a logistic regressor, together with features from the Linguistic Inquiry and Word Count (LIWC) lexicon and punctuation-derived features  Similarly to BIBREF42, for the fragment-level problem they used a Bi-LSTM-CRF architecture, combining both character- and word-level embeddings 
Team ProperGander BIBREF46 also used BERT, but they paid special attention to the imbalance of the data, as well as to the differences between training and testing  They showed that augmenting the training data by oversampling yielded improvements when testing on data that is temporally far from the training (by increasing recall)  In order to deal with the imbalance, they performed cost-sensitive classification, i e , the errors on the smaller positive class were more costly  For the fragment-level classification, inspired by named entity recognition, they used a model based on BERT using Continuous Random Field stacked on top of an LSTM 
Evaluation Results
The results on the test set for the SLC task are shown in Table TABREF33, while Table TABREF34 presents the results on the development set at the end of phase 1 (cf  Section SECREF6)  The general decrease of the F$_1$ values between the development and the test set could indicate that systems tend to overfit on the development set  Indeed, the winning team ltuorp chose the parameters of their system both on the development set and on a subset of the training set in order to improve the robustness of their system 
Tables TABREF36 and TABREF41 report the results on the test and on the development sets for the FLC task  For this task, the results tend to be more stable across the two sets  Indeed, team newspeak managed to almost keep the same difference in performance with respect to team Antiganda  Note that team MIC-CIS managed to reach the third position despite never having submitted a run on the development set 
Conclusion and Further Work
We have described the NLP4IF@EMNLP-IJCNLP 2019 shared task on fine-grained propaganda identification  We received 25 and 12 submissions on the test set for the sentence-level classification and the fragment-level classification tasks, respectively  Overall, the sentence-level task was easier and most submitted systems managed to outperform the baseline  The fragment-level task proved to be much more challenging, with lower absolute scores, but most teams still managed to outperform the baseline 
We plan to make the schema and the dataset publicly available to be used beyond NLP4IF  We hope that the corpus would raise interest outside of the community of researchers studying propaganda: the techniques related to fallacies and the ones relying on emotions might provide a novel setting for researchers interested in Argumentation and Sentiment Analysis 
As a kind of advertisement, Task 11 at SemEval 2020 is a follow up of this shared task  It features two complimentary tasks:
Given a free-text article, identify the propagandist text spans 
Given a text span already flagged as propagandist and its context, identify the specific propaganda technique it contains 
This setting would allow participants to focus their efforts on binary sequence labeling for Task 1 and on multi-class classification for Task 2 
Acknowledgments
This research is part of the Propaganda Analysis Project, which is framed within the Tanbih project  The Tanbih project aims to limit the effect of “fake news”, propaganda, and media bias by making users aware of what they are reading, thus promoting media literacy and critical thinking, which is arguably the best way to address disinformation and “fake news ” The project is developed in collaboration between the Qatar Computing Research Institute (QCRI), HBKU and the MIT Computer Science and Artificial Intelligence Laboratory (CSAIL) 
The corpus for the task was annotated by A Data Pro, a company that performs high-quality manual annotations 
Context 2: Introduction
The task of generating natural language descriptions of structured data (such as tables) BIBREF2 , BIBREF3 , BIBREF4 has seen a growth in interest with the rise of sequence to sequence models that provide an easy way of encoding tables and generating text from them BIBREF0 , BIBREF1 , BIBREF5 , BIBREF6  
For text generation tasks, the only gold standard metric is to show the output to humans for judging its quality, but this is too expensive to apply repeatedly anytime small modifications are made to a system  Hence, automatic metrics that compare the generated text to one or more reference texts are routinely used to compare models BIBREF7   For table-to-text generation, automatic evaluation has largely relied on BLEU BIBREF8 and ROUGE BIBREF9   The underlying assumption behind these metrics is that the reference text is gold-standard, i e , it is the ideal target text that a system should generate  In practice, however, when datasets are collected automatically and heuristically, the reference texts are often not ideal  Figure FIGREF2 shows an example from the WikiBio dataset BIBREF0   Here the reference contains extra information which no system can be expected to produce given only the associated table  We call such reference texts divergent from the table 
We show that existing automatic metrics, including BLEU, correlate poorly with human judgments when the evaluation sets contain divergent references (§ SECREF36 )  For many table-to-text generation tasks, the tables themselves are in a pseudo-natural language format (e g , WikiBio, WebNLG BIBREF6 , and E2E-NLG BIBREF10 )  In such cases we propose to compare the generated text to the underlying table as well to improve evaluation  We develop a new metric, PARENT (Precision And Recall of Entailed N-grams from the Table) (§ SECREF3 )  When computing precision, PARENT effectively uses a union of the reference and the table, to reward correct information missing from the reference  When computing recall, it uses an intersection of the reference and the table, to ignore extra incorrect information in the reference  The union and intersection are computed with the help of an entailment model to decide if a text n-gram is entailed by the table  We show that this method is more effective than using the table as an additional reference  Our main contributions are:
Table-to-Text Generation
We briefly review the task of generating natural language descriptions of semi-structured data, which we refer to as tables henceforth BIBREF11 , BIBREF12   Tables can be expressed as set of records INLINEFORM0 , where each record is a tuple (entity, attribute, value)  When all the records are about the same entity, we can truncate the records to (attribute, value) pairs  For example, for the table in Figure FIGREF2 , the records are {(Birth Name, Michael Dahlquist), (Born, December 22 1965),    }  The task is to generate a text INLINEFORM1 which summarizes the records in a fluent and grammatical manner  For training and evaluation we further assume that we have a reference description INLINEFORM2 available for each table  We let INLINEFORM3 denote an evaluation set of tables, references and texts generated from a model INLINEFORM4 , and INLINEFORM5 , INLINEFORM6 denote the collection of n-grams of order INLINEFORM7 in INLINEFORM8 and INLINEFORM9 , respectively  We use INLINEFORM10 to denote the count of n-gram INLINEFORM11 in INLINEFORM12 , and INLINEFORM13 to denote the minimum of its counts in INLINEFORM14 and INLINEFORM15   Our goal is to assign a score to the model, which correlates highly with human judgments of the quality of that model 
PARENT
PARENT evaluates each instance INLINEFORM0 separately, by computing the precision and recall of INLINEFORM1 against both INLINEFORM2 and INLINEFORM3  
Evaluation via Information Extraction
BIBREF1 proposed to use an auxiliary model, trained to extract structured records from text, for evaluation  However, the extraction model presented in that work is limited to the closed-domain setting of basketball game tables and summaries  In particular, they assume that each table has exactly the same set of attributes for each entity, and that the entities can be identified in the text via string matching  These assumptions are not valid for the open-domain WikiBio dataset, and hence we train our own extraction model to replicate their evaluation scheme 
Our extraction system is a pointer-generator network BIBREF19 , which learns to produce a linearized version of the table from the text  The network learns which attributes need to be populated in the output table, along with their values  It is trained on the training set of WikiBio  At test time we parsed the output strings into a set of (attribute, value) tuples and compare it to the ground truth table  The F-score of this text-to-table system was INLINEFORM0 , which is comparable to other challenging open-domain settings BIBREF20   More details are included in the Appendix SECREF52  
Given this information extraction system, we consider the following metrics for evaluation, along the lines of BIBREF1   Content Selection (CS): F-score for the (attribute, value) pairs extracted from the generated text compared to those extracted from the reference  Relation Generation (RG): Precision for the (attribute, value) pairs extracted from the generated text compared to those in the ground truth table  RG-F: Since our task emphasizes the recall of information from the table as well, we consider another variant which computes the F-score of the extracted pairs to those in the table  We omit the content ordering metric, since our extraction system does not align records to the input text 
Experiments & Results
In this section we compare several automatic evaluation metrics by checking their correlation with the scores assigned by humans to table-to-text models  Specifically, given INLINEFORM0 models INLINEFORM1 , and their outputs on an evaluation set, we show these generated texts to humans to judge their quality, and obtain aggregated human evaluation scores for all the models, INLINEFORM2 (§ SECREF33 )  Next, to evaluate an automatic metric, we compute the scores it assigns to each model, INLINEFORM3 , and check the Pearson correlation between INLINEFORM4 and INLINEFORM5 BIBREF21  
Data & Models
Our main experiments are on the WikiBio dataset BIBREF0 , which is automatically constructed and contains many divergent references  In § SECREF47 we also present results on the data released as part of the WebNLG challenge 
We developed several models of varying quality for generating text from the tables in WikiBio  This gives us a diverse set of outputs to evaluate the automatic metrics on  Table TABREF32 lists the models along with their hyperparameter settings and their scores from the human evaluation (§ SECREF33 )  Our focus is primarily on neural sequence-to-sequence methods since these are most widely used, but we also include a template-based baseline  All neural models were trained on the WikiBio training set  Training details and sample outputs are included in Appendices SECREF56 & SECREF57  
We divide these models into two categories and measure correlation separately for both the categories  The first category, WikiBio-Systems, includes one model each from the four families listed in Table TABREF32   This category tests whether a metric can be used to compare different model families with a large variation in the quality of their outputs  The second category, WikiBio-Hyperparams, includes 13 different hyperparameter settings of PG-Net BIBREF19 , which was the best performing system overall  9 of these were obtained by varying the beam size and length normalization penalty of the decoder network BIBREF23 , and the remaining 4 were obtained by re-scoring beams of size 8 with the information extraction model described in § SECREF4   All the models in this category produce high quality fluent texts, and differ primarily on the quantity and accuracy of the information they express  Here we are testing whether a metric can be used to compare similar systems with a small variation in performance  This is an important use-case as metrics are often used to tune hyperparameters of a model 
Human Evaluation
We collected human judgments on the quality of the 16 models trained for WikiBio, plus the reference texts  Workers on a crowd-sourcing platform, proficient in English, were shown a table with pairs of generated texts, or a generated text and the reference, and asked to select the one they prefer  Figure FIGREF34 shows the instructions they were given  Paired comparisons have been shown to be superior to rating scales for comparing generated texts BIBREF24   However, for measuring correlation the comparisons need to be aggregated into real-valued scores, INLINEFORM0 , for each of the INLINEFORM1 models  For this, we use Thurstone's method BIBREF22 , which assigns a score to each model based on how many times it was preferred over an alternative 
The data collection was performed separately for models in the WikiBio-Systems and WikiBio-Hyperparams categories  1100 tables were sampled from the development set, and for each table we got 8 different sentence pairs annotated across the two categories, resulting in a total of 8800 pairwise comparisons  Each pair was judged by one worker only which means there may be noise at the instance-level, but the aggregated system-level scores had low variance (cf  Table TABREF32 )  In total around 500 different workers were involved in the annotation  References were also included in the evaluation, and they received a lower score than PG-Net, highlighting the divergence in WikiBio 
Compared Metrics
Text only: We compare BLEU BIBREF8 , ROUGE BIBREF9 , METEOR BIBREF18 , CIDEr and CIDEr-D BIBREF25 using their publicly available implementations 
Information Extraction based: We compare the CS, RG and RG-F metrics discussed in § SECREF4  
Text & Table: We compare a variant of BLEU, denoted as BLEU-T, where the values from the table are used as additional references  BLEU-T draws inspiration from iBLEU BIBREF26 but instead rewards n-grams which match the table rather than penalizing them  For PARENT, we compare both the word-overlap model (PARENT-W) and the co-occurrence model (PARENT-C) for determining entailment  We also compare versions where a single INLINEFORM0 is tuned on the entire dataset to maximize correlation with human judgments, denoted as PARENT*-W/C 
Correlation Comparison
We use bootstrap sampling (500 iterations) over the 1100 tables for which we collected human annotations to get an idea of how the correlation of each metric varies with the underlying data  In each iteration, we sample with replacement, tables along with their references and all the generated texts for that table  Then we compute aggregated human evaluation and metric scores for each of the models and compute the correlation between the two  We report the average correlation across all bootstrap samples for each metric in Table TABREF37   The distribution of correlations for the best performing metrics are shown in Figure FIGREF38  
Table TABREF37 also indicates whether PARENT is significantly better than a baseline metric  BIBREF21 suggest using the William's test for this purpose, but since we are computing correlations between only 4/13 systems at a time, this test has very weak power in our case  Hence, we use the bootstrap samples to obtain a INLINEFORM0 confidence interval of the difference in correlation between PARENT and any other metric and check whether this is above 0 BIBREF27  
Correlations are higher for the systems category than the hyperparams category  The latter is a more difficult setting since very similar models are compared, and hence the variance of the correlations is also high  Commonly used metrics which only rely on the reference (BLEU, ROUGE, METEOR, CIDEr) have only weak correlations with human judgments  In the hyperparams category, these are often negative, implying that tuning models based on these may lead to selecting worse models  BLEU performs the best among these, and adding n-grams from the table as references improves this further (BLEU-T) 
Among the extractive evaluation metrics, CS, which also only relies on the reference, has poor correlation in the hyperparams category  RG-F, and both variants of the PARENT metric achieve the highest correlation for both settings  There is no significant difference among these for the hyperparams category, but for systems, PARENT-W is significantly better than the other two  While RG-F needs a full information extraction pipeline in its implementation, PARENT-C only relies on co-occurrence counts, and PARENT-W can be used out-of-the-box for any dataset  To our knowledge, this is the first rigorous evaluation of using information extraction for generation evaluation 
On this dataset, the word-overlap model showed higher correlation than the co-occurrence model for entailment  In § SECREF47 we will show that for the WebNLG dataset, where more paraphrasing is involved between the table and text, the opposite is true  Lastly, we note that the heuristic for selecting INLINEFORM0 is sufficient to produce high correlations for PARENT, however, if human annotations are available, this can be tuned to produce significantly higher correlations (PARENT*-W/C) 
Analysis
In this section we further analyze the performance of PARENT-W under different conditions, and compare to the other best metrics from Table TABREF37  
To study the correlation as we vary the number of divergent references, we also collected binary labels from workers for whether a reference is entailed by the corresponding table  We define a reference as entailed when it mentions only information which can be inferred from the table  Each table and reference pair was judged by 3 independent workers, and we used the majority vote as the label for that pair  Overall, only INLINEFORM0 of the references were labeled as entailed by the table  Fleiss' INLINEFORM1 was INLINEFORM2 , which indicates a fair agreement  We found the workers sometimes disagreed on what information can be reasonably entailed by the table 
Figure FIGREF40 shows the correlations as we vary the percent of entailed examples in the evaluation set of WikiBio  Each point is obtained by fixing the desired proportion of entailed examples, and sampling subsets from the full set which satisfy this proportion  PARENT and RG-F remain stable and show a high correlation across the entire range, whereas BLEU and BLEU-T vary a lot  In the hyperparams category, the latter two have the worst correlation when the evaluation set contains only entailed examples, which may seem surprising  However, on closer examination we found that this subset tends to omit a lot of information from the tables  Systems which produce more information than these references are penalized by BLEU, but not in the human evaluation  PARENT overcomes this issue by measuring recall against the table in addition to the reference 
We check how different components in the computation of PARENT contribute to its correlation to human judgments  Specifically, we remove the probability INLINEFORM0 of an n-gram INLINEFORM1 being entailed by the table from Eqs  EQREF19 and EQREF23   The average correlation for PARENT-W drops to INLINEFORM5 in this case  We also try a variant of PARENT with INLINEFORM6 , which removes the contribution of Table Recall (Eq  EQREF22 )  The average correlation is INLINEFORM7 in this case  With these components, the correlation is INLINEFORM8 , showing that they are crucial to the performance of PARENT 
BIBREF28 point out that hill-climbing on an automatic metric is meaningless if that metric has a low instance-level correlation to human judgments  In Table TABREF46 we show the average accuracy of the metrics in making the same judgments as humans between pairs of generated texts  Both variants of PARENT are significantly better than the other metrics, however the best accuracy is only INLINEFORM0 for the binary task  This is a challenging task, since there are typically only subtle differences between the texts  Achieving higher instance-level accuracies will require more sophisticated language understanding models for evaluation 
WebNLG Dataset
To check how PARENT correlates with human judgments when the references are elicited from humans (and less likely to be divergent), we check its correlation with the human ratings provided for the systems competing in the WebNLG challenge BIBREF6   The task is to generate text describing 1-5 RDF triples (e g  John E Blaha, birthPlace, San Antonio), and human ratings were collected for the outputs of 9 participating systems on 223 instances  These systems include a mix of pipelined, statistical and neural methods  Each instance has upto 3 reference texts associated with the RDF triples, which we use for evaluation 
The human ratings were collected on 3 distinct aspects – grammaticality, fluency and semantics, where semantics corresponds to the degree to which a generated text agrees with the meaning of the underlying RDF triples  We report the correlation of several metrics with these ratings in Table TABREF48   Both variants of PARENT are either competitive or better than the other metrics in terms of the average correlation to all three aspects  This shows that PARENT is applicable for high quality references as well 
While BLEU has the highest correlation for the grammar and fluency aspects, PARENT does best for semantics  This suggests that the inclusion of source tables into the evaluation orients the metric more towards measuring the fidelity of the content of the generation  A similar trend is seen comparing BLEU and BLEU-T  As modern neural text generation systems are typically very fluent, measuring their fidelity is of increasing importance  Between the two entailment models, PARENT-C is better due to its higher correlation with the grammaticality and fluency aspects 
The INLINEFORM0 parameter in the calculation of PARENT decides whether to compute recall against the table or the reference (Eq  EQREF22 )  Figure FIGREF50 shows the distribution of the values taken by INLINEFORM1 using the heuristic described in § SECREF3 for instances in both WikiBio and WebNLG  For WikiBio, the recall of the references against the table is generally low, and hence the recall of the generated text relies more on the table  For WebNLG, where the references are elicited from humans, this recall is much higher (often INLINEFORM2 ), and hence the recall of the generated text relies more on the reference 
Related Work
Over the years several studies have evaluated automatic metrics for measuring text generation performance BIBREF29 , BIBREF30 , BIBREF31 , BIBREF32 , BIBREF33 , BIBREF34 , BIBREF35   The only consensus from these studies seems to be that no single metric is suitable across all tasks  A recurring theme is that metrics like BLEU and NIST BIBREF36 are not suitable for judging content quality in NLG  Recently, BIBREF37 did a comprehensive study of several metrics on the outputs of state-of-the-art NLG systems, and found that while they showed acceptable correlation with human judgments at the system level, they failed to show any correlation at the sentence level  Ours is the first study which checks the quality of metrics when table-to-text references are divergent  We show that in this case even system level correlations can be unreliable 
Hallucination BIBREF38 , BIBREF39 refers to when an NLG system generates text which mentions extra information than what is present in the source from which it is generated  Divergence can be viewed as hallucination in the reference text itself  PARENT deals with hallucination by discounting n-grams which do not overlap with either the reference or the table 
PARENT draws inspiration from iBLEU BIBREF26 , a metric for evaluating paraphrase generation, which compares the generated text to both the source text and the reference  While iBLEU penalizes texts which match the source, here we reward such texts since our task values accuracy of generated text more than the need for paraphrasing the tabular content BIBREF40   Similar to SARI for text simplification BIBREF41 and Q-BLEU for question generation BIBREF42 , PARENT falls under the category of task-specific metrics 
Conclusions
We study the automatic evaluation of table-to-text systems when the references diverge from the table  We propose a new metric, PARENT, which shows the highest correlation with humans across a range of settings with divergent references in WikiBio  We also perform the first empirical evaluation of information extraction based metrics BIBREF1 , and find RG-F to be effective  Lastly, we show that PARENT is comparable to the best existing metrics when references are elicited by humans on the WebNLG data 
Acknowledgements
Bhuwan Dhingra is supported by a fellowship from Siemens, and by grants from Google  We thank Maruan Al-Shedivat, Ian Tenney, Tom Kwiatkowski, Michael Collins, Slav Petrov, Jason Baldridge, David Reitter and other members of the Google AI Language team for helpful discussions and suggestions  We thank Sam Wiseman for sharing data for an earlier version of this paper  We also thank the anonymous reviewers for their feedback 
Information Extraction System
For evaluation via information extraction BIBREF1 we train a model for WikiBio which accepts text as input and generates a table as the output  Tables in WikiBio are open-domain, without any fixed schema for which attributes may be present or absent in an instance  Hence we employ the Pointer-Generator Network (PG-Net) BIBREF19 for this purpose  Specifically, we use a sequence-to-sequence model, whose encoder and decoder are both single-layer bi-directional LSTMs  The decoder is augmented with an attention mechanism over the states of the encoder  Further, it also uses a copy mechanism to optionally copy tokens directly from the source text  We do not use the coverage mechanism of BIBREF19 since that is specific to the task of summarization they study  The decoder is trained to produce a linearized version of the table where the rows and columns are flattened into a sequence, and separate by special tokens  Figure FIGREF53 shows an example 
Clearly, since the references are divergent, the model cannot be expected to produce the entire table, and we see some false information being hallucinated after training  Nevertheless, as we show in § SECREF36 , this system can be used for evaluating generated texts  After training, we can parse the output sequence along the special tokens INLINEFORM0 R INLINEFORM1 and INLINEFORM2 C INLINEFORM3 to get a set of (attribute, value) pairs  Table TABREF54 shows the precision, recall and F-score of these extracted pairs against the ground truth tables, where the attributes and values are compared using an exact string match 
Hyperparameters
After tuning we found the same set of hyperparameters to work well for both the table-to-text PG-Net, and the inverse information extraction PG-Net  The hidden state size of the biLSTMs was set to 200  The input and output vocabularies were set to 50000 most common words in the corpus, with additional special symbols for table attribute names (such as “birth-date”)  The embeddings of the tokens in the vocabulary were initialized with Glove BIBREF43   Learning rate of INLINEFORM0 was used during training, with the Adam optimizer, and a dropout of INLINEFORM1 was also applied to the outputs of the biLSTM  Models were trained till the loss on the dev set stopped dropping  Maximum length of a decoded text was set to 40 tokens, and that of the tables was set to 120 tokens  Various beam sizes and length normalization penalties were applied for the table-to-text system, which are listed in the main paper  For the information extraction system, we found a beam size of 8 and no length penalty to produce the highest F-score on the dev set 
Sample Outputs
Table TABREF55 shows some sample references and the corresponding predictions from the best performing model, PG-Net for WikiBio 
Context 3: Introduction
Semantic Role Labeling (SRL) has emerged as an important task in Natural Language Processing (NLP) due to its applicability in information extraction, question answering, and other NLP tasks  SRL is the problem of finding predicate-argument structure in a sentence, as illustrated below:
INLINEFORM0
Here, the predicate WRITE has two arguments: `Mike' as A0 or the writer, and `a book' as A1 or the thing written  The labels A0 and A1 correspond to the PropBank annotations BIBREF0  
As the need for SRL arises in different domains and languages, the existing manually annotated corpora become insufficient to build supervised systems  This has motivated work on unsupervised SRL BIBREF1 , BIBREF2 , BIBREF3   Previous work has indicated that unsupervised systems could benefit from the word alignment information in parallel text in two or more languages BIBREF4 , BIBREF5 , BIBREF6   For example, consider the German translation of sentence INLINEFORM0 :
INLINEFORM0
If sentences INLINEFORM0 and INLINEFORM1 have the word alignments: Mike-Mike, written-geschrieben, and book-Buch, the system might be able to predict A1 for Buch, even if there is insufficient information in the monolingual German data to learn this assignment  Thus, in languages where the resources are sparse or not good enough, or the distributions are not informative, SRL systems could be made more accurate by using parallel data with resource rich or more amenable languages 
In this paper, we propose a joint Bayesian model for unsupervised semantic role induction in multiple languages  The model consists of individual Bayesian models for each language BIBREF3 , and crosslingual latent variables to incorporate soft role agreement between aligned constituents  This latent variable approach has been demonstrated to increase the performance in a multilingual unsupervised part-of-speech tagging model based on HMMs BIBREF4   We investigate the application of this approach to unsupervised SRL, presenting the performance improvements obtained in different settings involving labeled and unlabeled data, and analyzing the annotation effort required to obtain similar gains using labeled data 
We begin by briefly describing the unsupervised SRL pipeline and the monolingual semantic role induction model we use, and then describe our multilingual model 
Unsupervised SRL Pipeline
As established in previous work BIBREF7 , BIBREF8 , we use a standard unsupervised SRL setup, consisting of the following steps:
The task we model, unsupervised semantic role induction, is the step 4 of this pipeline 
Monolingual Model
We use the Bayesian model of garg2012unsupervised as our base monolingual model  The semantic roles are predicate-specific  To model the role ordering and repetition preferences, the role inventory for each predicate is divided into Primary and Secondary roles as follows:
For example, the complete role sequence in a frame could be: INLINEFORM0 INLINEFORM1 , INLINEFORM2 , INLINEFORM3 , INLINEFORM4 , INLINEFORM5 , INLINEFORM6 , INLINEFORM7 , INLINEFORM8 INLINEFORM9   The ordering is defined as the sequence of PRs, INLINEFORM10 INLINEFORM11 , INLINEFORM12 , INLINEFORM13 , INLINEFORM14 , INLINEFORM15 INLINEFORM16   Each pair of consecutive PRs in an ordering is called an interval  Thus, INLINEFORM17 is an interval that contains two SRs, INLINEFORM18 and INLINEFORM19   An interval could also be empty, for instance INLINEFORM20 contains no SRs  When we evaluate, these roles get mapped to gold roles  For instance, the PR INLINEFORM21 could get mapped to a core role like INLINEFORM22 , INLINEFORM23 , etc  or to a modifier role like INLINEFORM24 , INLINEFORM25 , etc  garg2012unsupervised reported that, in practice, PRs mostly get mapped to core roles and SRs to modifier roles, which conforms to the linguistic motivations for this distinction 
Figure FIGREF16 illustrates two copies of the monolingual model, on either side of the crosslingual latent variables  The generative process is as follows:
All the multinomial and binomial distributions have symmetric Dirichlet and beta priors respectively  Figure FIGREF7 gives the probability equations for the monolingual model  This formulation models the global role ordering and repetition preferences using PRs, and limited context for SRs using intervals  Ordering and repetition information was found to be helpful in supervised SRL as well BIBREF9 , BIBREF8 , BIBREF10   More details, including the motivations behind this model, are in BIBREF3  
Multilingual Model
The multilingual model uses word alignments between sentences in a parallel corpus to exploit role correspondences across languages  We make copies of the monolingual model for each language and add additional crosslingual latent variables (CLVs) to couple the monolingual models, capturing crosslingual semantic role patterns  Concretely, when training on parallel sentences, whenever the head words of the arguments are aligned, we add a CLV as a parent of the two corresponding role variables  Figure FIGREF16 illustrates this model  The generative process, as explained below, remains the same as the monolingual model for the most part, with the exception of aligned roles which are now generated by both the monolingual process as well as the CLV 
Every predicate-tuple has its own inventory of CLVs specific to that tuple  Each CLV INLINEFORM0 is a multi-valued variable where each value defines a distribution over role labels for each language (denoted by INLINEFORM1 above)  These distributions over labels are trained to be peaky, so that each value INLINEFORM2 for a CLV represents a correlation between the labels that INLINEFORM3 predicts in the two languages  For example, a value INLINEFORM4 for the CLV INLINEFORM5 might give high probabilities to INLINEFORM6 and INLINEFORM7 in language 1, and to INLINEFORM8 in language 2  If INLINEFORM9 is the only value for INLINEFORM10 that gives high probability to INLINEFORM11 in language 1, and the monolingual model in language 1 decides to assign INLINEFORM12 to the role for INLINEFORM13 , then INLINEFORM14 will predict INLINEFORM15 in language 2, with high probability  We generate the CLVs via a Chinese Restaurant Process BIBREF11 , a non-parametric Bayesian model, which allows us to induce the number of CLVs for every predicate-tuple from the data  We continue to train on the non-parallel sentences using the respective monolingual models 
The multilingual model is deficient, since the aligned roles are being generated twice  Ideally, we would like to add the CLV as additional conditioning variables in the monolingual models  The new joint probability can be written as equation UID11 (Figure FIGREF7 ), which can be further decomposed following the decomposition of the monolingual model in Figure FIGREF7   However, having this additional conditioning variable breaks the Dirichlet-multinomial conjugacy, which makes it intractable to marginalize out the parameters during inference  Hence, we use an approximation where we treat each of the aligned roles as being generated twice, once by the monolingual model and once by the corresponding CLV (equation ) 
This is the first work to incorporate the coupling of aligned arguments directly in a Bayesian SRL model  This makes it easier to see how to extend this model in a principled way to incorporate additional sources of information  First, the model scales gracefully to more than two languages  If there are a total of INLINEFORM0 languages, and there is an aligned argument in INLINEFORM1 of them, the multilingual latent variable is connected to only those INLINEFORM2 aligned arguments 
Second, having one joint Bayesian model allows us to use the same model in various semi-supervised learning settings, just by fixing the annotated variables during training  Section SECREF29 evaluates a setting where we have some labeled data in one language (called source), while no labeled data in the second language (called target)  Note that this is different from a classic annotation projection setting (e g  BIBREF12 ), where the role labels are mapped from source constituents to aligned target constituents 
Inference and Training
The inference problem consists of predicting the role labels and CLVs (the hidden variables) given the predicate, its voice, and syntactic features of all the identified arguments (the visible variables)  We use a collapsed Gibbs-sampling based approach to generate samples for the hidden variables (model parameters are integrated out)  The sample counts and the priors are then used to calculate the MAP estimate of the model parameters 
For the monolingual model, the role at a given position is sampled as:
DISPLAYFORM0
where the subscript INLINEFORM0 refers to all the variables except at position INLINEFORM1 , INLINEFORM2 refers to the variables in all the training instances except the current one, and INLINEFORM3 refers to all the model parameters  The above integral has a closed form solution due to Dirichlet-multinomial conjugacy 
For sampling roles in the multilingual model, we also need to consider the probabilities of roles being generated by the CLVs:
DISPLAYFORM0
For sampling CLVs, we need to consider three factors: two corresponding to probabilities of generating the aligned roles, and the third one corresponding to selecting the CLV according to CRP 
DISPLAYFORM0
where the aligned roles INLINEFORM0 and INLINEFORM1 are connected to INLINEFORM2 , and INLINEFORM3 refers to all the variables except INLINEFORM4 , INLINEFORM5 , and INLINEFORM6  
We use the trained parameters to parse the monolingual data using the monolingual model  The crosslingual parameters are ignored even if they were used during training  Thus, the information coming from the CLVs acts as a regularizer for the monolingual models 
Evaluation
Following the setting of titovcrosslingual, we evaluate only on the arguments that were correctly identified, as the incorrectly identified arguments do not have any gold semantic labels  Evaluation is done using the metric proposed by lang2011unsupervised, which has 3 components: (i) Purity (PU) measures how well an induced cluster corresponds to a single gold role, (ii) Collocation (CO) measures how well a gold role corresponds to a single induced cluster, and (iii) F1 is the harmonic mean of PU and CO  For each predicate, let INLINEFORM0 denote the total number of argument instances, INLINEFORM1 the instances in the induced cluster INLINEFORM2 , and INLINEFORM3 the instances having label INLINEFORM4 in gold annotations  INLINEFORM5 , INLINEFORM6 , and INLINEFORM7   The score for each predicate is weighted by the number of its argument instances, and a weighted average is computed over all the predicates 
Baseline
We use the same baseline as used by lang2011unsupervised which has been shown to be difficult to outperform  This baseline assigns a semantic role to a constituent based on its syntactic function, i e  the dependency relation to its head  If there is a total of INLINEFORM0 clusters, INLINEFORM1 most frequent syntactic functions get a cluster each, and the rest are assigned to the INLINEFORM2 th cluster 
Closest Previous Work
This work is closely related to the cross-lingual unsupervised SRL work of titovcrosslingual  Their model has separate monolingual models for each language and an extra penalty term which tries to maximize INLINEFORM0 and INLINEFORM1 i e  for all the aligned arguments with role label INLINEFORM2 in language 1, it tries to find a role label INLINEFORM3 in language 2 such that the given proportion is maximized and vice verse  However, there is no efficient way to optimize the objective with this penalty term and the authors used an inference method similar to annotation projection  Further, the method does not scale naturally to more than two languages  Their algorithm first does monolingual inference in one language ignoring the penalty and then does the inference in the second language taking into account the penalty term  In contrast, our model adds the latent variables as a part of the model itself, and not an external penalty, which enables us to use the standard Bayesian learning methods such as sampling 
The monolingual model we use BIBREF3 also has two main advantages over titovcrosslingual  First, the former incorporates a global role ordering probability that is missing in the latter  Secondly, the latter defines argument-keys as a tuple of four syntactic features and all the arguments having the same argument-keys are assigned the same role  This kind of hard clustering is avoided in the former model where two constituents having the same set of features might get assigned different roles if they appear in different contexts 
Data
Following titovcrosslingual, we run our experiments on the English (EN) and German (DE) sections of the CoNLL 2009 corpus BIBREF13 , and EN-DE section of the Europarl corpus BIBREF14   We get about 40k EN and 36k DE sentences from the CoNLL 2009 training set, and about 1 5M parallel EN-DE sentences from Europarl  For appropriate comparison, we keep the same setting as in BIBREF6 for automatic parses and argument identification, which we briefly describe here  The EN sentences are parsed syntactically using MaltParser BIBREF15 and DE using LTH parser BIBREF16   All the non-auxiliary verbs are selected as predicates  In CoNLL data, this gives us about 3k EN and 500 DE predicates  The total number of predicate instances are 3 4M in EN (89k CoNLL + 3 3M Europarl) and 2 62M in DE (17k CoNLL + 2 6M Europarl)  The arguments for EN are identified using the heuristics proposed by lang2011unsupervised  However, we get an F1 score of 85 1% for argument identification on CoNLL 2009 EN data as opposed to 80 7% reported by titovcrosslingual

ground_truth: The baseline system for the SLC task is a very simple logistic regression classifier with default parameters. The baseline for the FLC task generates spans and selects one of the 18 techniques randomly.

SLC task is a very simple logistic regression classifier, FLC task generates spans and selects one of the 18 techniques randomly

prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. Ngrams of which length are aligned using PARENT?
Context 4: Introduction
Event detection on microblogging platforms such as Twitter aims to detect events preemptively  A main task in event detection is detecting events of predetermined types BIBREF0, such as concerts or controversial events based on microposts matching specific event descriptions  This task has extensive applications ranging from cyber security BIBREF1, BIBREF2 to political elections BIBREF3 or public health BIBREF4, BIBREF5  Due to the high ambiguity and inconsistency of the terms used in microposts, event detection is generally performed though statistical machine learning models, which require a labeled dataset for model training  Data labeling is, however, a long, laborious, and usually costly process  For the case of micropost classification, though positive labels can be collected (e g , using specific hashtags, or event-related date-time information), there is no straightforward way to generate negative labels useful for model training  To tackle this lack of negative labels and the significant manual efforts in data labeling, BIBREF1 (BIBREF1, BIBREF3) introduced a weak supervision based learning approach, which uses only positively labeled data, accompanied by unlabeled examples by filtering microposts that contain a certain keyword indicative of the event type under consideration (e g , `hack' for cyber security)  Another key technique in this context is expectation regularization BIBREF6, BIBREF7, BIBREF1  Here, the estimated proportion of relevant microposts in an unlabeled dataset containing a keyword is given as a keyword-specific expectation  This expectation is used in the regularization term of the model's objective function to constrain the posterior distribution of the model predictions  By doing so, the model is trained with an expectation on its prediction for microposts that contain the keyword  Such a method, however, suffers from two key problems:
Due to the unpredictability of event occurrences and the constantly changing dynamics of users' posting frequency BIBREF8, estimating the expectation associated with a keyword is a challenging task, even for domain experts;
The performance of the event detection model is constrained by the informativeness of the keyword used for model training  As of now, we lack a principled method for discovering new keywords and improve the model performance 
To address the above issues, we advocate a human-AI loop approach for discovering informative keywords and estimating their expectations reliably  Our approach iteratively leverages 1) crowd workers for estimating keyword-specific expectations, and 2) the disagreement between the model and the crowd for discovering new informative keywords  More specifically, at each iteration after we obtain a keyword-specific expectation from the crowd, we train the model using expectation regularization and select those keyword-related microposts for which the model's prediction disagrees the most with the crowd's expectation; such microposts are then presented to the crowd to identify new keywords that best explain the disagreement  By doing so, our approach identifies new keywords which convey more relevant information with respect to existing ones, thus effectively boosting model performance  By exploiting the disagreement between the model and the crowd, our approach can make efficient use of the crowd, which is of critical importance in a human-in-the-loop context BIBREF9, BIBREF10  An additional advantage of our approach is that by obtaining new keywords that improve model performance over time, we are able to gain insight into how the model learns for specific event detection tasks  Such an advantage is particularly useful for event detection using complex models, e g , deep neural networks, which are intrinsically hard to understand BIBREF11, BIBREF12  An additional challenge in involving crowd workers is that their contributions are not fully reliable BIBREF13  In the crowdsourcing literature, this problem is usually tackled with probabilistic latent variable models BIBREF14, BIBREF15, BIBREF16, which are used to perform truth inference by aggregating a redundant set of crowd contributions  Our human-AI loop approach improves the inference of keyword expectation by aggregating contributions not only from the crowd but also from the model  This, however, comes with its own challenge as the model's predictions are further dependent on the results of expectation inference, which is used for model training  To address this problem, we introduce a unified probabilistic model that seamlessly integrates expectation inference and model training, thereby allowing the former to benefit from the latter while resolving the inter-dependency between the two 
To the best of our knowledge, we are the first to propose a human-AI loop approach that iteratively improves machine learning models for event detection  In summary, our work makes the following key contributions:
A novel human-AI loop approach for micropost event detection that jointly discovers informative keywords and estimates their expectation;
A unified probabilistic model that infers keyword expectation and simultaneously performs model training;
An extensive empirical evaluation of our approach on multiple real-world datasets demonstrating that our approach significantly improves the state of the art by an average of 24 3% AUC 
The rest of this paper is organized as follows  First, we present our human-AI loop approach in Section SECREF2  Subsequently, we introduce our proposed probabilistic model in Section SECREF3  The experimental setup and results are presented in Section SECREF4  Finally, we briefly cover related work in Section SECREF5 before concluding our work in Section SECREF6 
The Human-AI Loop Approach
Given a set of labeled and unlabeled microposts, our goal is to extract informative keywords and estimate their expectations in order to train a machine learning model  To achieve this goal, our proposed human-AI loop approach comprises two crowdsourcing tasks, i e , micropost classification followed by keyword discovery, and a unified probabilistic model for expectation inference and model training  Figure FIGREF6 presents an overview of our approach  Next, we describe our approach from a process-centric perspective 
Following previous studies BIBREF1, BIBREF17, BIBREF2, we collect a set of unlabeled microposts $\mathcal {U}$ from a microblogging platform and post-filter, using an initial (set of) keyword(s), those microposts that are potentially relevant to an event category  Then, we collect a set of event-related microposts (i e , positively labeled microposts) $\mathcal {L}$, post-filtering with a list of seed events  $\mathcal {U}$ and $\mathcal {L}$ are used together to train a discriminative model (e g , a deep neural network) for classifying the relevance of microposts to an event  We denote the target model as $p_\theta (y|x)$, where $\theta $ is the model parameter to be learned and $y$ is the label of an arbitrary micropost, represented by a bag-of-words vector $x$  Our approach iterates several times $t=\lbrace 1, 2, \ldots \rbrace $ until the performance of the target model converges  Each iteration starts from the initial keyword(s) or the new keyword(s) discovered in the previous iteration  Given such a keyword, denoted by $w^{(t)}$, the iteration starts by sampling microposts containing the keyword from $\mathcal {U}$, followed by dynamically creating micropost classification tasks and publishing them on a crowdsourcing platform 
Micropost Classification  The micropost classification task requires crowd workers to label the selected microposts into two classes: event-related and non event-related  In particular, workers are given instructions and examples to differentiate event-instance related microposts and general event-category related microposts  Consider, for example, the following microposts in the context of Cyber attack events, both containing the keyword `hack':
Credit firm Equifax says 143m Americans' social security numbers exposed in hack
This micropost describes an instance of a cyber attack event that the target model should identify  This is, therefore, an event-instance related micropost and should be considered as a positive example  Contrast this with the following example:
Companies need to step their cyber security up
This micropost, though related to cyber security in general, does not mention an instance of a cyber attack event, and is of no interest to us for event detection  This is an example of a general event-category related micropost and should be considered as a negative example 
In this task, each selected micropost is labeled by multiple crowd workers  The annotations are passed to our probabilistic model for expectation inference and model training 
Expectation Inference & Model Training  Our probabilistic model takes crowd-contributed labels and the model trained in the previous iteration as input  As output, it generates a keyword-specific expectation, denoted as $e^{(t)}$, and an improved version of the micropost classification model, denoted as $p_{\theta ^{(t)}}(y|x)$  The details of our probabilistic model are given in Section SECREF3 
Keyword Discovery  The keyword discovery task aims at discovering a new keyword (or a set of keywords) that is most informative for model training with respect to existing keywords  To this end, we first apply the current model $p_{\theta ^{(t)}}(y|x)$ on the unlabeled microposts $\mathcal {U}$  For those that contain the keyword $w^{(t)}$, we calculate the disagreement between the model predictions and the keyword-specific expectation $e^{(t)}$:
and select the ones with the highest disagreement for keyword discovery  These selected microposts are supposed to contain information that can explain the disagreement between the model prediction and keyword-specific expectation, and can thus provide information that is most different from the existing set of keywords for model training 
For instance, our study shows that the expectation for the keyword `hack' is 0 20, which means only 20% of the initial set of microposts retrieved with the keyword are event-related  A micropost selected with the highest disagreement (Eq  DISPLAY_FORM7), whose likelihood of being event-related as predicted by the model is $99 9\%$, is shown as an example below:
RT @xxx: Hong Kong securities brokers hit by cyber attacks, may face more: regulator #cyber #security #hacking https://t co/rC1s9CB
This micropost contains keywords that can better indicate the relevance to a cyber security event than the initial keyword `hack', e g , `securities', `hit', and `attack' 
Note that when the keyword-specific expectation $e^{(t)}$ in Equation DISPLAY_FORM7 is high, the selected microposts will be the ones that contain keywords indicating the irrelevance of the microposts to an event category  Such keywords are also useful for model training as they help improve the model's ability to identify irrelevant microposts 
To identify new keywords in the selected microposts, we again leverage crowdsourcing, as humans are typically better than machines at providing specific explanations BIBREF18, BIBREF19  In the crowdsourcing task, workers are first asked to find those microposts where the model predictions are deemed correct  Then, from those microposts, workers are asked to find the keyword that best indicates the class of the microposts as predicted by the model  The keyword most frequently identified by the workers is then used as the initial keyword for the following iteration  In case multiple keywords are selected, e g , the top-$N$ frequent ones, workers will be asked to perform $N$ micropost classification tasks for each keyword in the next iteration, and the model training will be performed on multiple keyword-specific expectations 
Unified Probabilistic Model
This section introduces our probabilistic model that infers keyword expectation and trains the target model simultaneously  We start by formalizing the problem and introducing our model, before describing the model learning method 
Problem Formalization  We consider the problem at iteration $t$ where the corresponding keyword is $w^{(t)}$  In the current iteration, let $\mathcal {U}^{(t)} \subset \mathcal {U}$ denote the set of all microposts containing the keyword and $\mathcal {M}^{(t)}= \lbrace x_{m}\rbrace _{m=1}^M\subset \mathcal {U}^{(t)}$ be the randomly selected subset of $M$ microposts labeled by $N$ crowd workers $\mathcal {C} = \lbrace c_n\rbrace _{n=1}^N$  The annotations form a matrix $\mathbf {A}\in \mathbb {R}^{M\times N}$ where $\mathbf {A}_{mn}$ is the label for the micropost $x_m$ contributed by crowd worker $c_n$  Our goal is to infer the keyword-specific expectation $e^{(t)}$ and train the target model by learning the model parameter $\theta ^{(t)}$  An additional parameter of our probabilistic model is the reliability of crowd workers, which is essential when involving crowdsourcing  Following Dawid and Skene BIBREF14, BIBREF16, we represent the annotation reliability of worker $c_n$ by a latent confusion matrix $\pi ^{(n)}$, where the $rs$-th element $\pi _{rs}^{(n)}$ denotes the probability of $c_n$ labeling a micropost as class $r$ given the true class $s$ 
Unified Probabilistic Model ::: Expectation as Model Posterior
First, we introduce an expectation regularization technique for the weakly supervised learning of the target model $p_{\theta ^{(t)}}(y|x)$  In this setting, the objective function of the target model is composed of two parts, corresponding to the labeled microposts $\mathcal {L}$ and the unlabeled ones $\mathcal {U}$ 
The former part aims at maximizing the likelihood of the labeled microposts:
where we assume that $\theta $ is generated from a prior distribution (e g , Laplacian or Gaussian) parameterized by $\sigma $ 
To leverage unlabeled data for model training, we make use of the expectations of existing keywords, i e , {($w^{(1)}$, $e^{(1)}$),    , ($w^{(t-1)}$, $e^{(t-1)}$), ($w^{(t)}$, $e^{(t)}$)} (Note that $e^{(t)}$ is inferred), as a regularization term to constrain model training  To do so, we first give the model's expectation for each keyword $w^{(k)}$ ($1\le k\le t$) as follows:
which denotes the empirical expectation of the model’s posterior predictions on the unlabeled microposts $\mathcal {U}^{(k)}$ containing keyword $w^{(k)}$  Expectation regularization can then be formulated as the regularization of the distance between the Bernoulli distribution parameterized by the model's expectation and the expectation of the existing keyword:
where $D_{KL}[\cdot \Vert \cdot ]$ denotes the KL-divergence between the Bernoulli distributions $Ber(e^{(k)})$ and $Ber(\mathbb {E}_{x\sim \mathcal {U}^{(k)}}(y))$, and $\lambda $ controls the strength of expectation regularization 
Unified Probabilistic Model ::: Expectation as Class Prior
To learn the keyword-specific expectation $e^{(t)}$ and the crowd worker reliability $\pi ^{(n)}$ ($1\le n\le N$), we model the likelihood of the crowd-contributed labels $\mathbf {A}$ as a function of these parameters  In this context, we view the expectation as the class prior, thus performing expectation inference as the learning of the class prior  By doing so, we connect expectation inference with model training 
Specifically, we model the likelihood of an arbitrary crowd-contributed label $\mathbf {A}_{mn}$ as a mixture of multinomials where the prior is the keyword-specific expectation $e^{(t)}$:
where $e_s^{(t)}$ is the probability of the ground truth label being $s$ given the keyword-specific expectation as the class prior; $K$ is the set of possible ground truth labels (binary in our context); and $r=\mathbf {A}_{mn}$ is the crowd-contributed label  Then, for an individual micropost $x_m$, the likelihood of crowd-contributed labels $\mathbf {A}_{m:}$ is given by:
Therefore, the objective function for maximizing the likelihood of the entire annotation matrix $\mathbf {A}$ can be described as:
Unified Probabilistic Model ::: Unified Probabilistic Model
Integrating model training with expectation inference, the overall objective function of our proposed model is given by:
Figure FIGREF18 depicts a graphical representation of our model, which combines the target model for training (on the left) with the generative model for crowd-contributed labels (on the right) through a keyword-specific expectation 
Model Learning  Due to the unknown ground truth labels of crowd-annotated microposts ($y_m$ in Figure FIGREF18), we resort to expectation maximization for model learning  The learning algorithm iteratively takes two steps: the E-step and the M-step

ground_truth: Unanswerable

Answer with content missing: (Parent subsections) combine precisions for n-gram orders 1-4

prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. What does an individual model consist of?
Context 3:  This could be due to implementation differences, which unfortunately makes our EN results incomparable  For DE, the arguments are identified using the LTH system BIBREF16 , which gives an F1 score of 86 5% on the CoNLL 2009 DE data  The word alignments for the EN-DE parallel Europarl corpus are computed using GIZA++ BIBREF17   For high-precision, only the intersecting alignments in the two directions are kept  We define two semantic arguments as aligned if their head-words are aligned  In total we get 9 3M arguments for EN (240k CoNLL + 9 1M Europarl) and 4 43M for DE (32k CoNLL + 4 4M Europarl)  Out of these, 0 76M arguments are aligned 
Main Results
Since the CoNLL annotations have 21 semantic roles in total, we use 21 roles in our model as well as the baseline  Following garg2012unsupervised, we set the number of PRs to 2 (excluding INLINEFORM0 , INLINEFORM1 and INLINEFORM2 ), and SRs to 21-2=19  Table TABREF27 shows the results 
In the first setting (Line 1), we train and test the monolingual model on the CoNLL data  We observe significant improvements in F1 score over the Baseline (Line 0) in both languages  Using the CoNLL 2009 dataset alone, titovcrosslingual report an F1 score of 80 9% (PU=86 8%, CO=75 7%) for German  Thus, our monolingual model outperforms their monolingual model in German  For English, they report an F1 score of 83 6% (PU=87 5%, CO=80 1%), but note that our English results are not directly comparable to theirs due to differences argument identification, as discussed in section SECREF25   As their argument identification score is lower, perhaps their system is discarding “difficult” arguments which leads to a higher clustering score 
In the second setting (Line 2), we use the additional monolingual Europarl (EP) data for training  We get equivalent results in English and a significant improvement in German compared to our previous setting (Line 1)  The German dataset in CoNLL is quite small and benefits from the additional EP training data  In contrast, the English model is already quite good due to a relatively big dataset from CoNLL, and good accuracy syntactic parsers  Unfortunately, titovcrosslingual do not report results with this setting 
The third setting (Line 3) gives the results of our multilingual model, which adds the word alignments in the EP data  Comparing with Line 2, we get non-significant improvements in both languages  titovcrosslingual obtain an F1 score of 82 7% (PU=85 0%, CO=80 6%) for German, and 83 7% (PU=86 8%, CO=80 7%) for English  Thus, for German, our multilingual Bayesian model is able to capture the cross-lingual patterns at least as well as the external penalty term in BIBREF6   We cannot compare the English results unfortunately due to differences in argument identification 
We also compared monolingual and bilingual training data using a setting that emulates the standard supervised setup of separate training and test data sets  We train only on the EP dataset and test on the CoNLL dataset  Lines 4 and 5 of Table TABREF27 give the results  The multilingual model obtains small improvements in both languages, which confirms the results from the standard unsupervised setup, comparing lines 2 to 3 
These results indicate that little information can be learned about semantic roles from this parallel data setup  One possible explanation for this result is that the setup itself is inadequate  Given the definition of aligned arguments, only 8% of English arguments and 17% of German arguments are aligned  This plus our experiments suggest that improving the alignment model is a necessary step to making effective use of parallel data in multilingual SRI, for example by joint modeling with SRI  We leave this exploration to future work 
Multilingual Training with Labeled Data for One Language
Another motivation for jointly modeling SRL in multiple languages is the transfer of information from a resource rich language to a resource poor language  We evaluated our model in a very general annotation transfer scenario, where we have a small labeled dataset for one language (source), and a large parallel unlabeled dataset for the source and another (target) language  We investigate whether this setting improves the parameter estimates for the target language  To this end, we clamp the role annotations of the source language in the CoNLL dataset using a predefined mapping, and do not sample them during training  This data gives us good parameters for the source language, which are used to sample the roles of the source language in the unlabeled Europarl data  The CLVs aim to capture this improvement and thereby improve sampling and parameter estimates for the target language  Table TABREF28 shows the results of this experiment  We obtain small improvements in the target languages  As in the unsupervised setting, the small percentage of aligned roles probably limits the impact of the cross-lingual information 
Labeled Data in Monolingual Model
We explored the improvement in the monolingual model in a semi-supervised setting  To this end, we randomly selected INLINEFORM0 of the sentences in the CoNLL dataset as “supervised sentences” and the rest INLINEFORM1 were kept unsupervised  Next, we clamped the role labels of the supervised sentences using the predefined mapping from Section SECREF29   Sampling was done on the unsupervised sentences as usual  We then measured the clustering performance using the trained parameters 
To access the contribution of partial supervision better, we constructed a “supervised baseline” as follows  For predicates seen in the supervised sentences, a MAP estimate of the parameters was calculated using the predefined mapping  For the unseen predicates, the standard baseline was used 
Figures FIGREF33 and FIGREF33 show the performance variation with INLINEFORM0   We make the following observations:
[leftmargin=*]
In both languages, at around INLINEFORM0 , the supervised baseline starts outperforming the semi-supervised model, which suggests that manually labeling about 10% of the sentences is a good enough alternative to our training procedure  Note that 10% amounts to about 3 6k sentences in German and 4k in English  We noticed that the proportion of seen predicates increases dramatically as we increase the proportion of supervised sentences  At 10% supervised sentences, the model has already seen 63% of predicates in German and 44% in English  This explains to some extent why only 10% labeled sentences are enough 
For German, it takes about 3 5% or 1260 supervised sentences to have the same performance increase as 1 5M unlabeled sentences (Line 1 to Line 2 in Table TABREF27 )  Adding about 180 more supervised sentences also covers the benefit obtained by alignments in the multilingual model (Line 2 to Line 3 in Table TABREF27 )  There is no noticeable performance difference in English 
We also evaluated the performance variation on a completely unseen CoNLL test set  Since the test set is very small compared to the training set, the clustering evaluation is not as reliable  Nonetheless, we broadly obtained the same pattern 
Related Work
As discussed in section SECREF24 , our work is closely related to the crosslingual unsupervised SRL work of titovcrosslingual  The idea of using superlingual latent variables to capture cross-lingual information was proposed for POS tagging by naseem2009multilingual, which we use here for SRL  In a semi-supervised setting, pado2009cross used a graph based approach to transfer semantic role annotations from English to German  furstenau2009graph used a graph alignment method to measure the semantic and syntactic similarity between dependency tree arguments of known and unknown verbs 
For monolingual unsupervised SRL, swier2004unsupervised presented the first work on a domain-general corpus, the British National Corpus, using 54 verbs taken from VerbNet  garg2012unsupervised proposed a Bayesian model for this problem that we use here  titov2012bayesian also proposed a closely related Bayesian model  grenager2006unsupervised proposed a generative model but their parameter space consisted of all possible linkings of syntactic constituents and semantic roles, which made unsupervised learning difficult and a separate language-specific rule based method had to be used to constrain this space  Other proposed models include an iterative split-merge algorithm BIBREF18 and a graph-partitioning based approach BIBREF1   marquez2008semantic provide a good overview of the supervised SRL systems 
Conclusions
We propose a Bayesian model of semantic role induction (SRI) that uses crosslingual latent variables to capture role alignments in parallel corpora  The crosslingual latent variables capture correlations between roles in different languages, and regularize the parameter estimates of the monolingual models  Because this is a joint Bayesian model of multilingual SRI, we can apply the same model to a variety of training scenarios just by changing the inference procedure appropriately  We evaluate monolingual SRI with a large unlabeled dataset, bilingual SRI with a parallel corpus, bilingual SRI with annotations available for the source language, and monolingual SRI with a small labeled dataset  Increasing the amount of monolingual unlabeled data significantly improves SRI in German but not in English  Adding word alignments in parallel sentences results in small, non significant improvements, even if there is some labeled data available in the source language  This difficulty in showing the usefulness of parallel corpora for SRI may be due to the current assumptions about role alignments, which mean that only a small percentage of roles are aligned  Further analyses reveals that annotating small amounts of data can easily outperform the performance gains obtained by adding large unlabeled dataset as well as adding parallel corpora 
Future work includes training on different language pairs, on more than two languages, and with more inclusive models of role alignment 
Acknowledgments
This work was funded by the Swiss NSF grant 200021_125137 and EC FP7 grant PARLANCE 

ground_truth: Bayesian model of garg2012unsupervised as our base monolingual model