prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. What was the baseline for this task?
Context 1: Introduction
Propaganda aims at influencing people's mindset with the purpose of advancing a specific agenda  In the Internet era, thanks to the mechanism of sharing in social networks, propaganda campaigns have the potential of reaching very large audiences BIBREF0, BIBREF1, BIBREF2 
Propagandist news articles use specific techniques to convey their message, such as whataboutism, red Herring, and name calling, among many others (cf  Section SECREF3)  Whereas proving intent is not easy, we can analyse the language of a claim/article and look for the use of specific propaganda techniques  Going at this fine-grained level can yield more reliable systems and it also makes it possible to explain to the user why an article was judged as propagandist by an automatic system 
With this in mind, we organised the shared task on fine-grained propaganda detection at the NLP4IF@EMNLP-IJCNLP 2019 workshop  The task is based on a corpus of news articles annotated with an inventory of 18 propagandist techniques at the fragment level  We hope that the corpus would raise interest outside of the community of researchers studying propaganda  For example, the techniques related to fallacies and the ones relying on emotions might provide a novel setting for researchers interested in Argumentation and Sentiment Analysis 
Related Work
Propaganda has been tackled mostly at the article level  BIBREF3 created a corpus of news articles labelled as propaganda, trusted, hoax, or satire  BIBREF4 experimented with a binarized version of that corpus: propaganda vs  the other three categories  BIBREF5 annotated a large binary corpus of propagandist vs  non-propagandist articles and proposed a feature-based system for discriminating between them  In all these cases, the labels were obtained using distant supervision, assuming that all articles from a given news outlet share the label of that outlet, which inevitably introduces noise BIBREF6 
A related field is that of computational argumentation which, among others, deals with some logical fallacies related to propaganda  BIBREF7 presented a corpus of Web forum discussions with instances of ad hominem fallacy  BIBREF8, BIBREF9 introduced Argotario, a game to educate people to recognize and create fallacies, a by-product of which is a corpus with $1 3k$ arguments annotated with five fallacies such as ad hominem, red herring and irrelevant authority, which directly relate to propaganda 
Unlike BIBREF8, BIBREF9, BIBREF7, our corpus uses 18 techniques annotated on the same set of news articles  Moreover, our annotations aim at identifying the minimal fragments related to a technique instead of flagging entire arguments 
The most relevant related work is our own, which is published in parallel to this paper at EMNLP-IJCNLP 2019 BIBREF10 and describes a corpus that is a subset of the one used for this shared task 
Propaganda Techniques
Propaganda uses psychological and rhetorical techniques to achieve its objective  Such techniques include the use of logical fallacies and appeal to emotions  For the shared task, we use 18 techniques that can be found in news articles and can be judged intrinsically, without the need to retrieve supporting information from external resources  We refer the reader to BIBREF10 for more details on the propaganda techniques; below we report the list of techniques:
Propaganda Techniques ::: 1  Loaded language 
Using words/phrases with strong emotional implications (positive or negative) to influence an audience BIBREF11 
Propaganda Techniques ::: 2  Name calling or labeling 
Labeling the object of the propaganda as something the target audience fears, hates, finds undesirable or otherwise loves or praises BIBREF12 
Propaganda Techniques ::: 3  Repetition 
Repeating the same message over and over again, so that the audience will eventually accept it BIBREF13, BIBREF12 
Propaganda Techniques ::: 4  Exaggeration or minimization 
Either representing something in an excessive manner: making things larger, better, worse, or making something seem less important or smaller than it actually is BIBREF14, e g , saying that an insult was just a joke 
Propaganda Techniques ::: 5  Doubt 
Questioning the credibility of someone or something 
Propaganda Techniques ::: 6  Appeal to fear/prejudice 
Seeking to build support for an idea by instilling anxiety and/or panic in the population towards an alternative, possibly based on preconceived judgments 
Propaganda Techniques ::: 7  Flag-waving 
Playing on strong national feeling (or with respect to a group, e g , race, gender, political preference) to justify or promote an action or idea BIBREF15 
Propaganda Techniques ::: 8  Causal oversimplification 
Assuming one cause when there are multiple causes behind an issue  We include scapegoating as well: the transfer of the blame to one person or group of people without investigating the complexities of an issue 
Propaganda Techniques ::: 9  Slogans 
A brief and striking phrase that may include labeling and stereotyping  Slogans tend to act as emotional appeals BIBREF16 
Propaganda Techniques ::: 10  Appeal to authority 
Stating that a claim is true simply because a valid authority/expert on the issue supports it, without any other supporting evidence BIBREF17  We include the special case where the reference is not an authority/expert, although it is referred to as testimonial in the literature BIBREF14 
Propaganda Techniques ::: 11  Black-and-white fallacy, dictatorship 
Presenting two alternative options as the only possibilities, when in fact more possibilities exist BIBREF13  As an extreme case, telling the audience exactly what actions to take, eliminating any other possible choice (dictatorship) 
Propaganda Techniques ::: 12  Thought-terminating cliché 
Words or phrases that discourage critical thought and meaningful discussion about a given topic  They are typically short and generic sentences that offer seemingly simple answers to complex questions or that distract attention away from other lines of thought BIBREF18 
Propaganda Techniques ::: 13  Whataboutism 
Discredit an opponent's position by charging them with hypocrisy without directly disproving their argument BIBREF19 
Propaganda Techniques ::: 14  Reductio ad Hitlerum 
Persuading an audience to disapprove an action or idea by suggesting that the idea is popular with groups hated in contempt by the target audience  It can refer to any person or concept with a negative connotation BIBREF20 
Propaganda Techniques ::: 15  Red herring 
Introducing irrelevant material to the issue being discussed, so that everyone's attention is diverted away from the points made BIBREF11  Those subjected to a red herring argument are led away from the issue that had been the focus of the discussion and urged to follow an observation or claim that may be associated with the original claim, but is not highly relevant to the issue in dispute BIBREF20 
Propaganda Techniques ::: 16  Bandwagon 
Attempting to persuade the target audience to join in and take the course of action because “everyone else is taking the same action” BIBREF15 
Propaganda Techniques ::: 17  Obfuscation, intentional vagueness, confusion 
Using deliberately unclear words, to let the audience have its own interpretation BIBREF21, BIBREF11  For instance, when an unclear phrase with multiple possible meanings is used within the argument and, therefore, it does not really support the conclusion 
Propaganda Techniques ::: 18  Straw man 
When an opponent's proposition is substituted with a similar one which is then refuted in place of the original BIBREF22 
Tasks
The shared task features two subtasks:
Tasks ::: Fragment-Level Classification task (FLC) 
Given a news article, detect all spans of the text in which a propaganda technique is used  In addition, for each span the propaganda technique applied must be identified 
Tasks ::: Sentence-Level Classification task (SLC) 
A sentence is considered propagandist if it contains at least one propagandist fragment  We then define a binary classification task in which, given a sentence, the correct label, either propaganda or non-propaganda, is to be predicted 
Data
The input for both tasks consists of news articles in free-text format, collected from 36 propagandist and 12 non-propagandist news outlets and then annotated by professional annotators  More details about the data collection and the annotation, as well as statistics about the corpus can be found in BIBREF10, where an earlier version of the corpus is described, which includes 450 news articles  We further annotated 47 additional articles for the purpose of the shared task using the same protocol and the same annotators 
The training, the development, and the test partitions of the corpus used for the shared task consist of 350, 61, and 86 articles and of 16,965, 2,235, and 3,526 sentences, respectively  Figure FIGREF15 shows an annotated example, which contains several propaganda techniques  For example, the fragment babies on line 1 is an instance of both Name_Calling and Labeling  Note that the fragment not looking as though Trump killed his grandma on line 4 is an instance of Exaggeration_or_Minimisation and it overlaps with the fragment killed his grandma, which is an instance of Loaded_Language 
Table TABREF23 reports the total number of instances per technique and the percentage with respect to the total number of annotations, for the training and for the development sets 
Setup
The shared task had two phases: In the development phase, the participants were provided labeled training and development datasets; in the testing phase, testing input was further provided 
The participants tried to achieve the best performance on the development set  A live leaderboard kept track of the submissions 
The test set was released and the participants had few days to make final predictions 
In phase 2, no immediate feedback on the submissions was provided  The winner was determined based on the performance on the test set 
Evaluation ::: FLC task 
FLC is a composition of two subtasks: the identification of the propagandist text fragments and the identification of the techniques used (18-way classification task)  While F$_1$ measure is appropriate for a multi-class classification task, we modified it to account for partial matching between the spans; see BIBREF10 for more details  We further computed an F$_1$ value for each propaganda technique (not shown below for the sake of saving space, but available on the leaderboard) 
Evaluation ::: SLC task 
SLC is a binary classification task with imbalanced data  Therefore, the official evaluation measure for the task is the standard F$_1$ measure  We further report Precision and Recall 
Baselines
The baseline system for the SLC task is a very simple logistic regression classifier with default parameters, where we represent the input instances with a single feature: the length of the sentence  The performance of this baseline on the SLC task is shown in Tables TABREF33 and TABREF34 
The baseline for the FLC task generates spans and selects one of the 18 techniques randomly  The inefficacy of such a simple random baseline is illustrated in Tables TABREF36 and TABREF41 
Participants and Approaches
A total of 90 teams registered for the shared task, and 39 of them submitted predictions for a total of 3,065 submissions  For the FLC task, 21 teams made a total of 527 submissions, and for the SLC task, 35 teams made a total of 2,538 submissions 
Below, we give an overview of the approaches as described in the participants' papers  Tables TABREF28 and TABREF29 offer a high-level summary 
Participants and Approaches ::: Teams Participating in the Fragment-Level Classification Only
Team newspeak BIBREF23 achieved the best results on the test set for the FLC task using 20-way word-level classification based on BERT BIBREF24: a word could belong to one of the 18 propaganda techniques, to none of them, or to an auxiliary (token-derived) class  The team fed one sentence at a time in order to reduce the workload  In addition to experimenting with an out-of-the-box BERT, they also tried unsupervised fine-tuning both on the 1M news dataset and on Wikipedia  Their best model was based on the uncased base model of BERT, with 12 Transformer layers BIBREF25, and 110 million parameters  Moreover, oversampling of the least represented classes proved to be crucial for the final performance  Finally, careful analysis has shown that the model pays special attention to adjectives and adverbs 
Team Stalin BIBREF26 focused on data augmentation to address the relatively small size of the data for fine-tuning contextual embedding representations based on ELMo BIBREF27, BERT, and Grover BIBREF28  The balancing of the embedding space was carried out by means of synthetic minority class over-sampling  Then, the learned representations were fed into an LSTM 
Participants and Approaches ::: Teams Participating in the Sentence-Level Classification Only
Team CAUnLP BIBREF29 used two context-aware representations based on BERT  In the first representation, the target sentence is followed by the title of the article  In the second representation, the previous sentence is also added  They performed subsampling in order to deal with class imbalance, and experimented with BERT$_{BASE}$ and BERT$_{LARGE}$
Team LIACC BIBREF30 used hand-crafted features and pre-trained ELMo embeddings  They also observed a boost in performance when balancing the dataset by dropping some negative examples 
Team JUSTDeep BIBREF31 used a combination of models and features, including word embeddings based on GloVe BIBREF32 concatenated with vectors representing affection and lexical features  These were combined in an ensemble of supervised models: bi-LSTM, XGBoost, and variations of BERT 
Team YMJA BIBREF33 also based their approach on fine-tuned BERT  Inspired by kaggle competitions on sentiment analysis, they created an ensemble of models via cross-validation 
Team jinfen BIBREF34 used a logistic regression model fed with a manifold of representations, including TF IDF and BERT vectors, as well as vocabularies and readability measures 
Team Tha3aroon BIBREF35 implemented an ensemble of three classifiers: two based on BERT and one based on a universal sentence encoder BIBREF36 
Team NSIT BIBREF37 explored three of the most popular transfer learning models: various versions of ELMo, BERT, and RoBERTa BIBREF38 
Team Mindcoders BIBREF39 combined BERT, Bi-LSTM and Capsule networks BIBREF40 into a single deep neural network and pre-trained the resulting network on corpora used for related tasks, e g , emotion classification 
Finally, team ltuorp BIBREF41 used an attention transformer using BERT trained on Wikipedia and BookCorpus 
Participants and Approaches ::: Teams Participating in Both Tasks
Team MIC-CIS BIBREF42 participated in both tasks  For the sentence-level classification, they used a voting ensemble including logistic regression, convolutional neural networks, and BERT, in all cases using FastText embeddings BIBREF43 and pre-trained BERT models  Beside these representations, multiple features of readability, sentiment and emotions were considered  For the fragment-level task, they used a multi-task neural sequence tagger, based on LSTM-CRF BIBREF44, in conjunction with linguistic features  Finally, they applied sentence- and fragment-level models jointly 
Team CUNLP BIBREF45 considered two approaches for the sentence-level task  The first approach was based on fine-tuning BERT  The second approach complemented the fine-tuned BERT approach by feeding its decision into a logistic regressor, together with features from the Linguistic Inquiry and Word Count (LIWC) lexicon and punctuation-derived features  Similarly to BIBREF42, for the fragment-level problem they used a Bi-LSTM-CRF architecture, combining both character- and word-level embeddings 
Team ProperGander BIBREF46 also used BERT, but they paid special attention to the imbalance of the data, as well as to the differences between training and testing  They showed that augmenting the training data by oversampling yielded improvements when testing on data that is temporally far from the training (by increasing recall)  In order to deal with the imbalance, they performed cost-sensitive classification, i e , the errors on the smaller positive class were more costly  For the fragment-level classification, inspired by named entity recognition, they used a model based on BERT using Continuous Random Field stacked on top of an LSTM 
Evaluation Results
The results on the test set for the SLC task are shown in Table TABREF33, while Table TABREF34 presents the results on the development set at the end of phase 1 (cf  Section SECREF6)  The general decrease of the F$_1$ values between the development and the test set could indicate that systems tend to overfit on the development set  Indeed, the winning team ltuorp chose the parameters of their system both on the development set and on a subset of the training set in order to improve the robustness of their system 
Tables TABREF36 and TABREF41 report the results on the test and on the development sets for the FLC task  For this task, the results tend to be more stable across the two sets  Indeed, team newspeak managed to almost keep the same difference in performance with respect to team Antiganda  Note that team MIC-CIS managed to reach the third position despite never having submitted a run on the development set 
Conclusion and Further Work
We have described the NLP4IF@EMNLP-IJCNLP 2019 shared task on fine-grained propaganda identification  We received 25 and 12 submissions on the test set for the sentence-level classification and the fragment-level classification tasks, respectively  Overall, the sentence-level task was easier and most submitted systems managed to outperform the baseline  The fragment-level task proved to be much more challenging, with lower absolute scores, but most teams still managed to outperform the baseline 
We plan to make the schema and the dataset publicly available to be used beyond NLP4IF  We hope that the corpus would raise interest outside of the community of researchers studying propaganda: the techniques related to fallacies and the ones relying on emotions might provide a novel setting for researchers interested in Argumentation and Sentiment Analysis 
As a kind of advertisement, Task 11 at SemEval 2020 is a follow up of this shared task  It features two complimentary tasks:
Given a free-text article, identify the propagandist text spans 
Given a text span already flagged as propagandist and its context, identify the specific propaganda technique it contains 
This setting would allow participants to focus their efforts on binary sequence labeling for Task 1 and on multi-class classification for Task 2 
Acknowledgments
This research is part of the Propaganda Analysis Project, which is framed within the Tanbih project  The Tanbih project aims to limit the effect of “fake news”, propaganda, and media bias by making users aware of what they are reading, thus promoting media literacy and critical thinking, which is arguably the best way to address disinformation and “fake news ” The project is developed in collaboration between the Qatar Computing Research Institute (QCRI), HBKU and the MIT Computer Science and Artificial Intelligence Laboratory (CSAIL) 
The corpus for the task was annotated by A Data Pro, a company that performs high-quality manual annotations 
Context 2: Introduction
Blogging gained momentum in 1999 and became especially popular after the launch of freely available, hosted platforms such as blogger com or livejournal com  Blogging has progressively been used by individuals to share news, ideas, and information, but it has also developed a mainstream role to the extent that it is being used by political consultants and news services as a tool for outreach and opinion forming as well as by businesses as a marketing tool to promote products and services BIBREF0  
For this paper, we compiled a very large geolocated collection of blogs, written by individuals located in the U S , with the purpose of creating insightful mappings of the blogging community  In particular, during May-July 2015, we gathered the profile information for all the users that have self-reported their location in the U S , along with a number of posts for all their associated blogs  We utilize this blog collection to generate maps of the U S  that reflect user demographics, language use, and distributions of psycholinguistic and semantic word classes  We believe that these maps can provide valuable insights and partial verification of previous claims in support of research in linguistic geography BIBREF1 , regional personality BIBREF2 , and language analysis BIBREF3 , BIBREF4 , as well as psychology and its relation to human geography BIBREF5  
Data Collection
Our premise is that we can generate informative maps using geolocated information available on social media; therefore, we guide the blog collection process with the constraint that we only accept blogs that have specific location information  Moreover, we aim to find blogs belonging to writers from all 50 U S  states, which will allow us to build U S  maps for various dimensions of interest 
We first started by collecting a set of profiles of bloggers that met our location specifications by searching individual states on the profile finder on http://www blogger com  Starting with this list, we can locate the profile page for a user, and subsequently extract additional information, which includes fields such as name, email, occupation, industry, and so forth  It is important to note that the profile finder only identifies users that have an exact match to the location specified in the query; we thus built and ran queries that used both state abbreviations (e g , TX, AL), as well as the states' full names (e g , Texas, Alabama) 
After completing all the processing steps, we identified 197,527 bloggers with state location information  For each of these bloggers, we found their blogs (note that a blogger can have multiple blogs), for a total of 335,698 blogs  For each of these blogs, we downloaded the 21 most recent blog postings, which were cleaned of HTML tags and tokenized, resulting in a collection of 4,600,465 blog posts 
Maps from Blogs
Our dataset provides mappings between location, profile information, and language use, which we can leverage to generate maps that reflect demographic, linguistic, and psycholinguistic properties of the population represented in the dataset 
People Maps
The first map we generate depicts the distribution of the bloggers in our dataset across the U S  Figure FIGREF1 shows the density of users in our dataset in each of the 50 states  For instance, the densest state was found to be California with 11,701 users  The second densest is Texas, with 9,252 users, followed by New York, with 9,136  The state with the fewest bloggers is Delaware with 1,217 users  Not surprisingly, this distribution correlates well with the population of these states, with a Spearman's rank correlation INLINEFORM0 of 0 91 and a p-value INLINEFORM1 0 0001, and is very similar to the one reported in Lin and Halavais Lin04 
Figure FIGREF1 also shows the cities mentioned most often in our dataset  In particular, it illustrates all 227 cities that have at least 100 bloggers  The bigger the dot on the map, the larger the number of users found in that city  The five top blogger-dense cities, in order, are: Chicago, New York, Portland, Seattle, and Atlanta

ground_truth: The baseline system for the SLC task is a very simple logistic regression classifier with default parameters. The baseline for the FLC task generates spans and selects one of the 18 techniques randomly.

SLC task is a very simple logistic regression classifier, FLC task generates spans and selects one of the 18 techniques randomly

prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. How do they obtain psychological dimensions of people?
Context 2: 
We also generate two maps that delineate the gender distribution in the dataset  Overall, the blogging world seems to be dominated by females: out of 153,209 users who self-reported their gender, only 52,725 are men and 100,484 are women  Figures FIGREF1 and FIGREF1 show the percentage of male and female bloggers in each of the 50 states  As seen in this figure, there are more than the average number of male bloggers in states such as California and New York, whereas Utah and Idaho have a higher percentage of women bloggers 
Another profile element that can lead to interesting maps is the Industry field BIBREF6   Using this field, we created different maps that plot the geographical distribution of industries across the country  As an example, Figure FIGREF2 shows the percentage of the users in each state working in the automotive and tourism industries respectively 
Linguistic Maps
Another use of the information found in our dataset is to build linguistic maps, which reflect the geographic lexical variation across the 50 states BIBREF7   We generate maps that represent the relative frequency by which a word occurs in the different states  Figure FIGREF3 shows sample maps created for two different words  The figure shows the map generated for one location specific word, Maui, which unsurprisingly is found predominantly in Hawaii, and a map for a more common word, lake, which has a high occurrence rate in Minnesota (Land of 10,000 Lakes) and Utah (home of the Great Salt Lake)  Our demo described in Section SECREF4 , can also be used to generate maps for function words, which can be very telling regarding people's personality BIBREF8  
Psycholinguistic and Semantic Maps
LIWC  In addition to individual words, we can also create maps for word categories that reflect a certain psycholinguistic or semantic property  Several lexical resources, such as Roget or Linguistic Inquiry and Word Count BIBREF9 , group words into categories  Examples of such categories are Money, which includes words such as remuneration, dollar, and payment; or Positive feelings with words such as happy, cheerful, and celebration  Using the distribution of the individual words in a category, we can compile distributions for the entire category, and therefore generate maps for these word categories  For instance, figure FIGREF8 shows the maps created for two categories: Positive Feelings and Money  The maps are not surprising, and interestingly they also reflect an inverse correlation between Money and Positive Feelings  
Values  We also measure the usage of words related to people's core values as reported by Boyd et al  boyd2015  The sets of words, or themes, were excavated using the Meaning Extraction Method (MEM) BIBREF10   MEM is a topic modeling approach applied to a corpus of texts created by hundreds of survey respondents from the U S  who were asked to freely write about their personal values  To illustrate, Figure FIGREF9 shows the geographical distributions of two of these value themes: Religion and Hard Work  Southeastern states often considered as the nation's “Bible Belt” BIBREF11 were found to have generally higher usage of Religion words such as God, bible, and church  Another broad trend was that western-central states (e g , Wyoming, Nebraska, Iowa) commonly blogged about Hard Work, using words such as hard, work, and job more often than bloggers in other regions 
Web Demonstration
A prototype, interactive charting demo is available at http://lit eecs umich edu/~geoliwc/  In addition to drawing maps of the geographical distributions on the different LIWC categories, the tool can report the three most and least correlated LIWC categories in the U S  and compare the distributions of any two categories 
Conclusions
In this paper, we showed how we can effectively leverage a prodigious blog dataset  Not only does the dataset bring out the extensive linguistic content reflected in the blog posts, but also includes location information and rich metadata  These data allow for the generation of maps that reflect the demographics of the population, variations in language use, and differences in psycholinguistic and semantic categories  These mappings can be valuable to both psychologists and linguists, as well as lexicographers  A prototype demo has been made available together with the code used to collect our dataset 
Acknowledgments
This material is based in part upon work supported by the National Science Foundation (#1344257) and by the John Templeton Foundation (#48503)  Any opinions, findings, and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of the National Science Foundation or the John Templeton Foundation  We would like to thank our colleagues Hengjing Wang, Jiatao Fan, Xinghai Zhang, and Po-Jung Huang who provided technical help with the implementation of the demo 
Context 3: Introduction
With the increasing popularity of the Internet, online texts provided by social media platform (e g  Twitter) and news media sites (e g  Google news) have become important sources of real-world events  Therefore, it is crucial to automatically extract events from online texts 
Due to the high variety of events discussed online and the difficulty in obtaining annotated data for training, traditional template-based or supervised learning approaches for event extraction are no longer applicable in dealing with online texts  Nevertheless, newsworthy events are often discussed by many tweets or online news articles  Therefore, the same event could be mentioned by a high volume of redundant tweets or news articles  This property inspires the research community to devise clustering-based models BIBREF0 , BIBREF1 , BIBREF2 to discover new or previously unidentified events without extracting structured representations 
To extract structured representations of events such as who did what, when, where and why, Bayesian approaches have made some progress  Assuming that each document is assigned to a single event, which is modeled as a joint distribution over the named entities, the date and the location of the event, and the event-related keywords, Zhou et al  zhou2014simple proposed an unsupervised Latent Event Model (LEM) for open-domain event extraction  To address the limitation that LEM requires the number of events to be pre-set, Zhou et al  zhou2017event further proposed the Dirichlet Process Event Mixture Model (DPEMM) in which the number of events can be learned automatically from data  However, both LEM and DPEMM have two limitations: (1) they assume that all words in a document are generated from a single event which can be represented by a quadruple INLINEFORM0 entity, location, keyword, date INLINEFORM1   However, long texts such news articles often describe multiple events which clearly violates this assumption; (2) During the inference process of both approaches, the Gibbs sampler needs to compute the conditional posterior distribution and assigns an event for each document  This is time consuming and takes long time to converge 
To deal with these limitations, in this paper, we propose the Adversarial-neural Event Model (AEM) based on adversarial training for open-domain event extraction  The principle idea is to use a generator network to learn the projection function between the document-event distribution and four event related word distributions (entity distribution, location distribution, keyword distribution and date distribution)  Instead of providing an analytic approximation, AEM uses a discriminator network to discriminate between the reconstructed documents from latent events and the original input documents  This essentially helps the generator to construct a more realistic document from a random noise drawn from a Dirichlet distribution  Due to the flexibility of neural networks, the generator is capable of learning complicated nonlinear distributions  And the supervision signal provided by the discriminator will help generator to capture the event-related patterns  Furthermore, the discriminator also provides low-dimensional discriminative features which can be used to visualize documents and events 
The main contributions of the paper are summarized below:
Related Work
Our work is related to two lines of research, event extraction and Generative Adversarial Nets 
Event Extraction
Recently there has been much interest in event extraction from online texts, and approaches could be categorized as domain-specific and open-domain event extraction 
Domain-specific event extraction often focuses on the specific types of events (e g  sports events or city events)  Panem et al  panem2014structured devised a novel algorithm to extract attribute-value pairs and mapped them to manually generated schemes for extracting the natural disaster events  Similarly, to extract the city-traffic related event, Anantharam et al  anantharam2015extracting viewed the task as a sequential tagging problem and proposed an approach based on the conditional random fields  Zhang zhang2018event proposed an event extraction approach based on imitation learning, especially on inverse reinforcement learning 
Open-domain event extraction aims to extract events without limiting the specific types of events  To analyze individual messages and induce a canonical value for each event, Benson et al  benson2011event proposed an approach based on a structured graphical model  By representing an event with a binary tuple which is constituted by a named entity and a date, Ritter et al  ritter2012open employed some statistic to measure the strength of associations between a named entity and a date  The proposed system relies on a supervised labeler trained on annotated data  In BIBREF1 , Abdelhaq et al  developed a real-time event extraction system called EvenTweet, and each event is represented as a triple constituted by time, location and keywords  To extract more information, Wang el al  wang2015seeft developed a system employing the links in tweets and combing tweets with linked articles to identify events  Xia el al  xia2015new combined texts with the location information to detect the events with low spatial and temporal deviations  Zhou et al  zhou2014simple,zhou2017event represented event as a quadruple and proposed two Bayesian models to extract events from tweets 
Generative Adversarial Nets
As a neural-based generative model, Generative Adversarial Nets BIBREF3 have been extensively researched in natural language processing (NLP) community 
For text generation, the sequence generative adversarial network (SeqGAN) proposed in BIBREF4 incorporated a policy gradient strategy to optimize the generation process  Based on the policy gradient, Lin et al  lin2017adversarial proposed RankGAN to capture the rich structures of language by ranking and analyzing a collection of human-written and machine-written sentences  To overcome mode collapse when dealing with discrete data, Fedus et al  fedus2018maskgan proposed MaskGAN which used an actor-critic conditional GAN to fill in missing text conditioned on the surrounding context  Along this line, Wang et al  wang2018sentigan proposed SentiGAN to generate texts of different sentiment labels  Besides, Li et al  li2018learning improved the performance of semi-supervised text classification using adversarial training, BIBREF5 , BIBREF6 designed GAN-based models for distance supervision relation extraction 
Although various GAN based approaches have been explored for many applications, none of these approaches tackles open-domain event extraction from online texts  We propose a novel GAN-based event extraction model called AEM  Compared with the previous models, AEM has the following differences: (1) Unlike most GAN-based text generation approaches, a generator network is employed in AEM to learn the projection function between an event distribution and the event-related word distributions (entity, location, keyword, date)  The learned generator captures event-related patterns rather than generating text sequence; (2) Different from LEM and DPEMM, AEM uses a generator network to capture the event-related patterns and is able to mine events from different text sources (short and long)  Moreover, unlike traditional inference procedure, such as Gibbs sampling used in LEM and DPEMM, AEM could extract the events more efficiently due to the CUDA acceleration; (3) The discriminative features learned by the discriminator of AEM provide a straightforward way to visualize the extracted events 
Methodology
We describe Adversarial-neural Event Model (AEM) in this section  An event is represented as a quadruple < INLINEFORM0 >, where INLINEFORM1 stands for non-location named entities, INLINEFORM2 for a location, INLINEFORM3 for event-related keywords, INLINEFORM4 for a date, and each component in the tuple is represented by component-specific representative words 
AEM is constituted by three components: (1) The document representation module, as shown at the top of Figure FIGREF4 , defines a document representation approach which converts an input document from the online text corpus into INLINEFORM0 which captures the key event elements; (2) The generator INLINEFORM1 , as shown in the lower-left part of Figure FIGREF4 , generates a fake document INLINEFORM2 which is constituted by four multinomial distributions using an event distribution INLINEFORM3 drawn from a Dirichlet distribution as input; (3) The discriminator INLINEFORM4 , as shown in the lower-right part of Figure FIGREF4 , distinguishes the real documents from the fake ones and its output is subsequently employed as a learning signal to update the INLINEFORM5 and INLINEFORM6   The details of each component are presented below 
Document Representation
Each document INLINEFORM0 in a given corpus INLINEFORM1 is represented as a concatenation of 4 multinomial distributions which are entity distribution ( INLINEFORM2 ), location distribution ( INLINEFORM3 ), keyword distribution ( INLINEFORM4 ) and date distribution ( INLINEFORM5 ) of the document  As four distributions are calculated in a similar way, we only describe the computation of the entity distribution below as an example 
The entity distribution INLINEFORM0 is represented by a normalized INLINEFORM1 -dimensional vector weighted by TF-IDF, and it is calculated as: INLINEFORM2
where INLINEFORM0 is the pseudo corpus constructed by removing all non-entity words from INLINEFORM1 , INLINEFORM2 is the total number of distinct entities in a corpus, INLINEFORM3 denotes the number of INLINEFORM4 -th entity appeared in document INLINEFORM5 , INLINEFORM6 represents the number of documents in the corpus, and INLINEFORM7 is the number of documents that contain INLINEFORM8 -th entity, and the obtained INLINEFORM9 denotes the relevance between INLINEFORM10 -th entity and document INLINEFORM11  
Similarly, location distribution INLINEFORM0 , keyword distribution INLINEFORM1 and date distribution INLINEFORM2 of INLINEFORM3 could be calculated in the same way, and the dimensions of these distributions are denoted as INLINEFORM4 , INLINEFORM5 and INLINEFORM6 , respectively  Finally, each document INLINEFORM7 in the corpus is represented by a INLINEFORM8 -dimensional ( INLINEFORM9 = INLINEFORM10 + INLINEFORM11 + INLINEFORM12 + INLINEFORM13 ) vector INLINEFORM14 by concatenating four computed distributions 
Network Architecture
The generator network INLINEFORM0 is designed to learn the projection function between the document-event distribution INLINEFORM1 and the four document-level word distributions (entity distribution, location distribution, keyword distribution and date distribution) 
More concretely, INLINEFORM0 consists of a INLINEFORM1 -dimensional document-event distribution layer, INLINEFORM2 -dimensional hidden layer and INLINEFORM3 -dimensional event-related word distribution layer  Here, INLINEFORM4 denotes the event number, INLINEFORM5 is the number of units in the hidden layer, INLINEFORM6 is the vocabulary size and equals to INLINEFORM7 + INLINEFORM8 + INLINEFORM9 + INLINEFORM10   As shown in Figure FIGREF4 , INLINEFORM11 firstly employs a random document-event distribution INLINEFORM12 as an input  To model the multinomial property of the document-event distribution, INLINEFORM13 is drawn from a Dirichlet distribution parameterized with INLINEFORM14 which is formulated as: DISPLAYFORM0
where INLINEFORM0 is the hyper-parameter of the dirichlet distribution, INLINEFORM1 is the number of events which should be set in AEM, INLINEFORM2 , INLINEFORM3 represents the proportion of event INLINEFORM4 in the document and INLINEFORM5  
Subsequently, INLINEFORM0 transforms INLINEFORM1 into a INLINEFORM2 -dimensional hidden space using a linear layer followed by layer normalization, and the transformation is defined as: DISPLAYFORM0
where INLINEFORM0 represents the weight matrix of hidden layer, and INLINEFORM1 denotes the bias term, INLINEFORM2 is the parameter of LeakyReLU activation and is set to 0 1, INLINEFORM3 and INLINEFORM4 denote the normalized hidden states and the outputs of the hidden layer, and INLINEFORM5 represents the layer normalization 
Then, to project INLINEFORM0 into four document-level event related word distributions ( INLINEFORM1 , INLINEFORM2 , INLINEFORM3 and INLINEFORM4 shown in Figure FIGREF4 ), four subnets (each contains a linear layer, a batch normalization layer and a softmax layer) are employed in INLINEFORM5   And the exact transformation is based on the formulas below: DISPLAYFORM0
where INLINEFORM0 means softmax layer, INLINEFORM1 , INLINEFORM2 , INLINEFORM3 and INLINEFORM4 denote the weight matrices of the linear layers in subnets, INLINEFORM5 , INLINEFORM6 , INLINEFORM7 and INLINEFORM8 represent the corresponding bias terms, INLINEFORM9 , INLINEFORM10 , INLINEFORM11 and INLINEFORM12 are state vectors  INLINEFORM13 , INLINEFORM14 , INLINEFORM15 and INLINEFORM16 denote the generated entity distribution, location distribution, keyword distribution and date distribution, respectively, that correspond to the given event distribution INLINEFORM17   And each dimension represents the relevance between corresponding entity/location/keyword/date term and the input event distribution 
Finally, four generated distributions are concatenated to represent the generated document INLINEFORM0 corresponding to the input INLINEFORM1 : DISPLAYFORM0
The discriminator network INLINEFORM0 is designed as a fully-connected network which contains an input layer, a discriminative feature layer (discriminative features are employed for event visualization) and an output layer  In AEM, INLINEFORM1 uses fake document INLINEFORM2 and real document INLINEFORM3 as input and outputs the signal INLINEFORM4 to indicate the source of the input data (lower value denotes that INLINEFORM5 is prone to predict the input data as a fake document and vice versa) 
As have previously been discussed in BIBREF7 , BIBREF8 , lipschitz continuity of INLINEFORM0 network is crucial to the training of the GAN-based approaches  To ensure the lipschitz continuity of INLINEFORM1 , we employ the spectral normalization technique BIBREF9   More concretely, for each linear layer INLINEFORM2 (bias term is omitted for simplicity) in INLINEFORM3 , the weight matrix INLINEFORM4 is normalized by INLINEFORM5   Here, INLINEFORM6 is the spectral norm of the weight matrix INLINEFORM7 with the definition below: DISPLAYFORM0
which is equivalent to the largest singular value of INLINEFORM0   The weight matrix INLINEFORM1 is then normalized using: DISPLAYFORM0
Obviously, the normalized weight matrix INLINEFORM0 satisfies that INLINEFORM1 and further ensures the lipschitz continuity of the INLINEFORM2 network BIBREF9   To reduce the high cost of computing spectral norm INLINEFORM3 using singular value decomposition at each iteration, we follow BIBREF10 and employ the power iteration method to estimate INLINEFORM4 instead  With this substitution, the spectral norm can be estimated with very small additional computational time 
Objective and Training Procedure
The real document INLINEFORM0 and fake document INLINEFORM1 shown in Figure FIGREF4 could be viewed as random samples from two distributions INLINEFORM2 and INLINEFORM3 , and each of them is a joint distribution constituted by four Dirichlet distributions (corresponding to entity distribution, location distribution, keyword distribution and date distribution)  The training objective of AEM is to let the distribution INLINEFORM4 (produced by INLINEFORM5 network) to approximate the real data distribution INLINEFORM6 as much as possible 
To compare the different GAN losses, Kurach kurach2018gan takes a sober view of the current state of GAN and suggests that the Jansen-Shannon divergence used in BIBREF3 performs more stable than variant objectives  Besides, Kurach also advocates that the gradient penalty (GP) regularization devised in BIBREF8 will further improve the stability of the model  Thus, the objective function of the proposed AEM is defined as: DISPLAYFORM0
where INLINEFORM0 denotes the discriminator loss, INLINEFORM1 represents the gradient penalty regularization loss, INLINEFORM2 is the gradient penalty coefficient which trade-off the two components of objective, INLINEFORM3 could be obtained by sampling uniformly along a straight line between INLINEFORM4 and INLINEFORM5 , INLINEFORM6 denotes the corresponding distribution 
The training procedure of AEM is presented in Algorithm SECREF15 , where INLINEFORM0 is the event number, INLINEFORM1 denotes the number of discriminator iterations per generator iteration, INLINEFORM2 is the batch size, INLINEFORM3 represents the learning rate, INLINEFORM4 and INLINEFORM5 are hyper-parameters of Adam BIBREF11 , INLINEFORM6 denotes INLINEFORM7   In this paper, we set INLINEFORM8 , INLINEFORM9 , INLINEFORM10   Moreover, INLINEFORM11 , INLINEFORM12 and INLINEFORM13 are set as 0 0002, 0 5 and 0 999 
[!h] Training procedure for AEM [1] INLINEFORM0 , INLINEFORM1 , INLINEFORM2 , INLINEFORM3 , INLINEFORM4 , INLINEFORM5 , INLINEFORM6 the trained INLINEFORM7 and INLINEFORM8   Initial INLINEFORM9 parameters INLINEFORM10 and INLINEFORM11 parameter INLINEFORM12 INLINEFORM13 has not converged INLINEFORM14 INLINEFORM15 Sample INLINEFORM16 , Sample a random INLINEFORM17 Sample a random number INLINEFORM18 INLINEFORM19 INLINEFORM20 INLINEFORM21 INLINEFORM22 INLINEFORM23 INLINEFORM24 Sample INLINEFORM25 noise INLINEFORM26 INLINEFORM27
Event Generation
After the model training, the generator INLINEFORM0 learns the mapping function between the document-event distribution and the document-level event-related word distributions (entity, location, keyword and date)  In other words, with an event distribution INLINEFORM1 as input, INLINEFORM2 could generate the corresponding entity distribution, location distribution, keyword distribution and date distribution 
In AEM, we employ event seed INLINEFORM0 , an INLINEFORM1 -dimensional vector with one-hot encoding, to generate the event related word distributions  For example, in ten event setting, INLINEFORM2 represents the event seed of the first event  With the event seed INLINEFORM3 as input, the corresponding distributions could be generated by INLINEFORM4 based on the equation below: DISPLAYFORM0
where INLINEFORM0 , INLINEFORM1 , INLINEFORM2 and INLINEFORM3 denote the entity distribution, location distribution, keyword distribution and date distribution of the first event respectively 
Experiments
In this section, we firstly describe the datasets and baseline approaches used in our experiments and then present the experimental results 
Experimental Setup
To validate the effectiveness of AEM for extracting events from social media (e g  Twitter) and news media sites (e g  Google news), three datasets (FSD BIBREF12 , Twitter, and Google datasets) are employed  Details are summarized below:
FSD dataset (social media) is the first story detection dataset containing 2,499 tweets  We filter out events mentioned in less than 15 tweets since events mentioned in very few tweets are less likely to be significant  The final dataset contains 2,453 tweets annotated with 20 events 
Twitter dataset (social media) is collected from tweets published in the month of December in 2010 using Twitter streaming API  It contains 1,000 tweets annotated with 20 events 
Google dataset (news article) is a subset of GDELT Event Database INLINEFORM0 , documents are retrieved by event related words  For example, documents which contain `malaysia', `airline', `search' and `plane' are retrieved for event MH370  By combining 30 events related documents, the dataset contains 11,909 news articles 
We choose the following three models as the baselines:
K-means is a well known data clustering algorithm, we implement the algorithm using sklearn toolbox, and represent documents using bag-of-words weighted by TF-IDF 
LEM BIBREF13 is a Bayesian modeling approach for open-domain event extraction  It treats an event as a latent variable and models the generation of an event as a joint distribution of its individual event elements  We implement the algorithm with the default configuration 
DPEMM BIBREF14 is a non-parametric mixture model for event extraction  It addresses the limitation of LEM that the number of events should be known beforehand  We implement the model with the default configuration 
For social media text corpus (FSD and Twitter), a named entity tagger specifically built for Twitter is used to extract named entities including locations from tweets  A Twitter Part-of-Speech (POS) tagger BIBREF15 is used for POS tagging and only words tagged with nouns, verbs and adjectives are retained as keywords  For the Google dataset, we use the Stanford Named Entity Recognizer to identify the named entities (organization, location and person)  Due to the `date' information not being provided in the Google dataset, we further divide the non-location named entities into two categories (`person' and `organization') and employ a quadruple <organization, location, person, keyword> to denote an event in news articles  We also remove common stopwords and only keep the recognized named entities and the tokens which are verbs, nouns or adjectives 
Experimental Results
To evaluate the performance of the proposed approach, we use the evaluation metrics such as precision, recall and F-measure  Precision is defined as the proportion of the correctly identified events out of the model generated events  Recall is defined as the proportion of correctly identified true events  For calculating the precision of the 4-tuple, we use following criteria:
(1) Do the entity/organization, location, date/person and keyword that we have extracted refer to the same event?
(2) If the extracted representation contains keywords, are they informative enough to tell us what happened?
Table TABREF35 shows the event extraction results on the three datasets  The statistics are obtained with the default parameter setting that INLINEFORM0 is set to 5, number of hidden units INLINEFORM1 is set to 200, and INLINEFORM2 contains three fully-connected layers  The event number INLINEFORM3 for three datasets are set to 25, 25 and 35, respectively  The examples of extracted events are shown in Table  TABREF36  
It can be observed that K-means performs the worst over all three datasets  On the social media datasets, AEM outpoerforms both LEM and DPEMM by 6 5% and 1 7% respectively in F-measure on the FSD dataset, and 4 4% and 3 7% in F-measure on the Twitter dataset  We can also observe that apart from K-means, all the approaches perform worse on the Twitter dataset compared to FSD, possibly due to the limited size of the Twitter dataset  Moreover, on the Google dataset, the proposed AEM performs significantly better than LEM and DPEMM  It improves upon LEM by 15 5% and upon DPEMM by more than 30% in F-measure  This is because: (1) the assumption made by LEM and DPEMM that all words in a document are generated from a single event is not suitable for long text such as news articles; (2) DPEMM generates too many irrelevant events which leads to a very low precision score  Overall, we see the superior performance of AEM across all datasets, with more significant improvement on the for Google datasets (long text) 
We next visualize the detected events based on the discriminative features learned by the trained INLINEFORM0 network in AEM  The t-SNE BIBREF16 visualization results on the datasets are shown in Figure FIGREF19   For clarity, each subplot is plotted on a subset of the dataset containing ten randomly selected events  It can be observed that documents describing the same event have been grouped into the same cluster 
To further evaluate if a variation of the parameters INLINEFORM0 (the number of discriminator iterations per generator iteration), INLINEFORM1 (the number of units in hidden layer) and the structure of generator INLINEFORM2 will impact the extraction performance, additional experiments have been conducted on the Google dataset, with INLINEFORM3 set to 5, 7 and 10, INLINEFORM4 set to 100, 150 and 200, and three INLINEFORM5 structures (3, 4 and 5 layers)  The comparison results on precision, recall and F-measure are shown in Figure FIGREF20   From the results, it could be observed that AEM with the 5-layer generator performs the best and achieves 96 7% in F-measure, and the worst F-measure obtained by AEM is 85 7%  Overall, the AEM outperforms all compared approaches acorss various parameter settings, showing relatively stable performance 
Finally, we compare in Figure FIGREF37 the training time required for each model, excluding the constant time required by each model to load the data  We could observe that K-means runs fastest among all four approaches  Both LEM and DPEMM need to sample the event allocation for each document and update the relevant counts during Gibbs sampling which are time consuming  AEM only requires a fraction of the training time compared to LEM and DPEMM  Moreover, on a larger dataset such as the Google dataset, AEM appears to be far more efficient compared to LEM and DPEMM 
Conclusions and Future Work
In this paper, we have proposed a novel approach based on adversarial training to extract the structured representation of events from online text  The experimental comparison with the state-of-the-art methods shows that AEM achieves improved extraction performance, especially on long text corpora with an improvement of 15% observed in F-measure  AEM only requires a fraction of training time compared to existing Bayesian graphical modeling approaches  In future work, we will explore incorporating external knowledge (e g  word relatedness contained in word embeddings) into the learning framework for event extraction  Besides, exploring nonparametric neural event extraction approaches and detecting the evolution of events over time from news articles are other promising future directions 
Acknowledgments
We would like to thank anonymous reviewers for their valuable comments and helpful suggestions  This work was funded by the National Key Research and Development Program of China (2016YFC1306704), the National Natural Science Foundation of China (61772132), the Natural Science Foundation of Jiangsu Province of China (BK20161430) 

ground_truth: using the Meaning Extraction Method

Unanswerable

prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. Do they report results only on English data?
Context 4:  Figure FIGREF14 presents scanpaths of three participants for the sarcastic sentence S1 and the non-sarcastic sentence S2  The x-axis of the graph represents the sequence of words a reader reads, and the y-axis represents a temporal sequence in milliseconds 
Consider a sarcastic text containing incongruous phrases A and B  Our qualitative scanpath-analysis reveals that scanpaths with respect to sarcasm processing have two typical characteristics  Often, a long regression - a saccade that goes to a previously visited segment - is observed when a reader starts reading B after skimming through A  In a few cases, the fixation duration on A and B are significantly higher than the average fixation duration per word  In sentence S1, we see long and multiple regressions from the two incongruous phrases “misconception” and “cherish”, and a few instances where phrases “always cherish” and “original misconception” are fixated longer than usual  Such eye-movement behaviors are not seen for S2 
Though sarcasm induces distinctive scanpaths like the ones depicted in Figure FIGREF14 in the observed examples, presence of such patterns is not sufficient to guarantee sarcasm; such patterns may also possibly arise from literal texts  We believe that a combination of linguistic features, readability of text and features derived from scanpaths would help discriminative machine learning models learn sarcasm better 
Features for Sarcasm Detection
We describe the features used for sarcasm detection in Table   The features enlisted under lexical,implicit incongruity and explicit incongruity are borrowed from various literature (predominantly from joshi2015harnessing)  These features are essential to separate sarcasm from other forms semantic incongruity in text (for example ambiguity arising from semantic ambiguity or from metaphors)  Two additional textual features viz  readability and word count of the text are also taken under consideration  These features are used to reduce the effect of text hardness and text length on the eye-movement patterns 
Simple Gaze Based Features
Readers' eye-movement behavior, characterized by fixations, forward saccades, skips and regressions, can be directly quantified by simple statistical aggregation (i e , either computing features for individual participants and then averaging or performing a multi-instance based learning as explained in section SECREF6 )  Since these eye-movement attributes relate to the cognitive process in reading BIBREF17 , we consider these as features in our model  Some of these features have been reported by sarcasmunderstandability for modeling sarcasm understandability of readers  However, as far as we know, these features are being introduced in NLP tasks like textual sarcasm detection for the first time  The values of these features are believed to increase with the increase in the degree of surprisal caused by incongruity in text (except skip count, which will decrease) 
Complex Gaze Based Features
For these features, we rely on a graph structure, namely “saliency graphs", derived from eye-gaze information and word sequences in the text 
For each reader and each sentence, we construct a “saliency graph”, representing the reader's attention characteristics  A saliency graph for a sentence INLINEFORM0 for a reader INLINEFORM1 , represented as INLINEFORM2 , is a graph with vertices ( INLINEFORM3 ) and edges ( INLINEFORM4 ) where each vertex INLINEFORM5 corresponds to a word in INLINEFORM6 (may not be unique) and there exists an edge INLINEFORM7 between vertices INLINEFORM8 and INLINEFORM9 if R performs at least one saccade between the words corresponding to INLINEFORM10 and INLINEFORM11  
Figure FIGREF15 shows an example of a saliency graph A saliency graph may be weighted, but not necessarily connected, for a given text (as there may be words in the given text with no fixation on them)  The “complex” gaze features derived from saliency graphs are also motivated by the theory of incongruity  For instance, Edge Density of a saliency graph increases with the number of distinct saccades, which could arise from the complexity caused by presence of sarcasm  Similarly, the highest weighted degree of a graph is expected to be higher, if the node corresponds to a phrase, incongruous to some other phrase in the text 
The Sarcasm Classifier
We interpret sarcasm detection as a binary classification problem  The training data constitutes 994 examples created using our eye-movement database for sarcasm detection  To check the effectiveness of our feature set, we observe the performance of multiple classification techniques on our dataset through a stratified 10-fold cross validation  We also compare the classification accuracy of our system and the best available systems proposed by riloff2013sarcasm and joshi2015harnessing on our dataset  Using Weka BIBREF18 and LibSVM BIBREF19 APIs, we implement the following classifiers:
Results
Table TABREF17 shows the classification results considering various feature combinations for different classifiers and other systems  These are:
Unigram (with principal components of unigram feature vectors),
Sarcasm (the feature-set reported by joshi2015harnessing subsuming unigram features and features from other reported systems)
Gaze (the simple and complex cognitive features we introduce, along with readability and word count features), and
Gaze+Sarcasm (the complete set of features) 
For all regular classifiers, the gaze features are averaged across participants and augmented with linguistic and sarcasm related features  For the MILR classifier, the gaze features derived from each participant are augmented with linguistic features and thus, a multi instance “bag” of features is formed for each sentence in the training data  This multi-instance dataset is given to an MILR classifier, which follows the standard multi instance assumption to derive class-labels for each bag 
For all the classifiers, our feature combination outperforms the baselines (considering only unigram features) as well as BIBREF3 , with the MILR classifier getting an F-score improvement of 3 7% and Kappa difference of 0 08  We also achieve an improvement of 2% over the baseline, using SVM classifier, when we employ our feature set  We also observe that the gaze features alone, also capture the differences between sarcasm and non-sarcasm classes with a high-precision but a low recall 
To see if the improvement obtained is statistically significant over the state-of-the art system with textual sarcasm features alone, we perform McNemar test  The output of the SVM classifier using only linguistic features used for sarcasm detection by joshi2015harnessing and the output of the MILR classifier with the complete set of features are compared, setting threshold INLINEFORM0   There was a significant difference in the classifier's accuracy with p(two-tailed) = 0 02 with an odds-ratio of 1 43, showing that the classification accuracy improvement is unlikely to be observed by chance in 95% confidence interval 
Considering Reading Time as a Cognitive Feature along with Sarcasm Features
One may argue that, considering simple measures of reading effort like “reading time” as cognitive feature instead of the expensive eye-tracking features for sarcasm detection may be a cost-effective solution  To examine this, we repeated our experiments with “reading time” considered as the only cognitive feature, augmented with the textual features  The F-scores of all the classifiers turn out to be close to that of the classifiers considering sarcasm feature alone and the difference in the improvement is not statistically significant ( INLINEFORM0 )  One the other hand, F-scores with gaze features are superior to the F-scores when reading time is considered as a cognitive feature 
How Effective are the Cognitive Features
We examine the effectiveness of cognitive features on the classification accuracy by varying the input training data size  To examine this, we create a stratified (keeping the class ratio constant) random train-test split of 80%:20%  We train our classifier with 100%, 90%, 80% and 70% of the training data with our whole feature set, and the feature combination from joshi2015harnessing  The goodness of our system is demonstrated by improvements in F-score and Kappa statistics, shown in Figure FIGREF22  
We further analyze the importance of features by ranking the features based on (a) Chi squared test, and (b) Information Gain test, using Weka's attribute selection module  Figure FIGREF23 shows the top 20 ranked features produced by both the tests  For both the cases, we observe 16 out of top 20 features to be gaze features  Further, in each of the cases, Average Fixation Duration per Word and Largest Regression Position are seen to be the two most significant features 
Example Cases
Table TABREF21 shows a few example cases from the experiment with stratified 80%-20% train-test split 
Example sentence 1 is sarcastic, and requires extra-linguistic knowledge (about poor living conditions at Manchester)  Hence, the sarcasm detector relying only on textual features is unable to detect the underlying incongruity  However, our system predicts the label successfully, possibly helped by the gaze features 
Similarly, for sentence 2, the false sense of presence of incongruity (due to phrases like “Helped me” and “Can't stop”) affects the system with only linguistic features  Our system, though, performs well in this case also 
Sentence 3 presents a false-negative case where it was hard for even humans to get the sarcasm  This is why our gaze features (and subsequently the complete set of features) account for erroneous prediction 
In sentence 4, gaze features alone false-indicate presence of incongruity, whereas the system predicts correctly when gaze and linguistic features are taken together 
From these examples, it can be inferred that, only gaze features would not have sufficed to rule out the possibility of detecting other forms of incongruity that do not result in sarcasm 
Error Analysis
Errors committed by our system arise from multiple factors, starting from limitations of the eye-tracker hardware to errors committed by linguistic tools and resources  Also, aggregating various eye-tracking parameters to extract the cognitive features may have caused information loss in the regular classification setting 
Conclusion
In the current work, we created a novel framework to detect sarcasm, that derives insights from human cognition, that manifests over eye movement patterns  We hypothesized that distinctive eye-movement patterns, associated with reading sarcastic text, enables improved detection of sarcasm  We augmented traditional linguistic features with cognitive features obtained from readers' eye-movement data in the form of simple gaze-based features and complex features derived from a graph structure  This extended feature-set improved the success rate of the sarcasm detector by 3 7%, over the best available system  Using cognitive features in an NLP Processing system like ours is the first proposal of its kind 
Our general approach may be useful in other NLP sub-areas like sentiment and emotion analysis, text summarization and question answering, where considering textual clues alone does not prove to be sufficient  We propose to augment this work in future by exploring deeper graph and gaze features  We also propose to develop models for the purpose of learning complex gaze feature representation, that accounts for the power of individual eye movement patterns along with the aggregated patterns of eye movements 
Acknowledgments
We thank the members of CFILT Lab, especially Jaya Jha and Meghna Singh, and the students of IIT Bombay for their help and support 

ground_truth: Unanswerable

Unanswerable