prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. How effective is their NCEL approach overall?
Context 1: Introduction
This work is licensed under a Creative Commons Attribution 4 0 International License  License details: http://creativecommons org/licenses/by/4 0/
Entity linking (EL), mapping entity mentions in texts to a given knowledge base (KB), serves as a fundamental role in many fields, such as question answering BIBREF0 , semantic search BIBREF1 , and information extraction BIBREF2 , BIBREF3   However, this task is non-trivial because entity mentions are usually ambiguous  As shown in Figure FIGREF1 , the mention England refers to three entities in KB, and an entity linking system should be capable of identifying the correct entity as England cricket team rather than England and England national football team 
Entity linking is typically broken down into two main phases: (i) candidate generation obtains a set of referent entities in KB for each mention, and (ii) named entity disambiguation selects the possible candidate entity by solving a ranking problem  The key challenge lies in the ranking model that computes the relevance between candidates and the corresponding mentions based on the information both in texts and KBs BIBREF4   In terms of the features used for ranking, we classify existing EL models into two groups: local models to resolve mentions independently relying on textual context information from the surrounding words BIBREF5 , BIBREF6 , BIBREF7 , BIBREF8 , and global (collective) models, which are the main focus of this paper, that encourage the target entities of all mentions in a document to be topically coherent BIBREF9 , BIBREF10 , BIBREF11 , BIBREF12 , BIBREF13 , BIBREF14  
Global models usually build an entity graph based on KBs to capture coherent entities for all identified mentions in a document, where the nodes are entities, and edges denote their relations  The graph provides highly discriminative semantic signals (e g , entity relatedness) that are unavailable to local model BIBREF15   For example (Figure FIGREF1 ), an EL model seemly cannot find sufficient disambiguation clues for the mention England from its surrounding words, unless it utilizes the coherence information of consistent topic “cricket" among adjacent mentions England, Hussain, and Essex  Although the global model has achieved significant improvements, its limitation is threefold:
To mitigate the first limitation, recent EL studies introduce neural network (NN) models due to its amazing feature abstraction and generalization ability  In such models, words/entities are represented by low dimensional vectors in a continuous space, and features for mention as well as candidate entities are automatically learned from data BIBREF4   However, existing NN-based methods for EL are either local models BIBREF16 , BIBREF17 or merely use word/entity embeddings for feature extraction and rely on another modules for collective disambiguation, which thus cannot fully utilize the power of NN models for collective EL BIBREF18 , BIBREF19 , BIBREF20  
The second drawback of the global approach has been alleviated through approximate optimization techniques, such as PageRank/random walks BIBREF21 , graph pruning BIBREF22 , ranking SVMs BIBREF23 , or loopy belief propagation (LBP) BIBREF18 , BIBREF24   However, these methods are not differentiable and thus difficult to be integrated into neural network models (the solution for the first limitation) 
To overcome the third issue of inadequate training data, BIBREF17 has explored a massive amount of hyperlinks in Wikipedia, but these potential annotations for EL contain much noise, which may distract a naive disambiguation model BIBREF6  
In this paper, we propose a novel Neural Collective Entity Linking model (NCEL), which performs global EL combining deep neural networks with Graph Convolutional Network (GCN) BIBREF25 , BIBREF26 that allows flexible encoding of entity graphs  It integrates both local contextual information and global interdependence of mentions in a document, and is efficiently trainable in an end-to-end fashion  Particularly, we introduce attention mechanism to robustly model local contextual information by selecting informative words and filtering out the noise  On the other hand, we apply GCNs to improve discriminative signals of candidate entities by exploiting the rich structure underlying the correct entities  To alleviate the global computations, we propose to convolute on the subgraph of adjacent mentions  Thus, the overall coherence shall be achieved in a chain-like way via a sliding window over the document  To the best of our knowledge, this is the first effort to develop a unified model for neural collective entity linking 
In experiments, we first verify the efficiency of NCEL via theoretically comparing its time complexity with other collective alternatives  Afterwards, we train our neural model using collected Wikipedia hyperlinks instead of dataset-specific annotations, and perform evaluations on five public available benchmarks  The results show that NCEL consistently outperforms various baselines with a favorable generalization ability  Finally, we further present the performance on a challenging dataset WW BIBREF19 as well as qualitative results, investigating the effectiveness of each key module 
Preliminaries and Framework
We denote INLINEFORM0 as a set of entity mentions in a document INLINEFORM1 , where INLINEFORM2 is either a word INLINEFORM3 or a mention INLINEFORM4   INLINEFORM5 is the entity graph for document INLINEFORM6 derived from the given knowledge base, where INLINEFORM7 is a set of entities, INLINEFORM8 denotes the relatedness between INLINEFORM9 and higher values indicate stronger relations  Based on INLINEFORM10 , we extract a subgraph INLINEFORM11 for INLINEFORM12 , where INLINEFORM13 denotes the set of candidate entities for INLINEFORM14   Note that we don't include the relations among candidates of the same mention in INLINEFORM15 because these candidates are mutually exclusive in disambiguation 
Formally, we define the entity linking problem as follows: Given a set of mentions INLINEFORM0 in a document INLINEFORM1 , and an entity graph INLINEFORM2 , the goal is to find an assignment INLINEFORM3  
To collectively find the best assignment, NCEL aims to improve the discriminability of candidates' local features by using entity relatedness within a document via GCN, which is capable of learning a function of features on the graph through shared parameters over all nodes  Figure FIGREF10 shows the framework of NCEL including three main components:
Example As shown in Figure FIGREF10 , for the current mention England, we utilize its surrounding words as local contexts (e g , surplus), and adjacent mentions (e g , Hussian) as global information  Collectively, we utilize the candidates of England INLINEFORM0 as well as those entities of its adjacencies INLINEFORM1 to construct feature vectors for INLINEFORM2 and the subgraph of relatedness as inputs of our neural model  Let darker blue indicate higher probability of being predicted, the correct candidate INLINEFORM3 becomes bluer due to its bluer neighbor nodes of other mentions INLINEFORM4   The dashed lines denote entity relations that have indirect impacts through the sliding adjacent window , and the overall structure shall be achieved via multiple sub-graphs by traversing all mentions 
Before introducing our model, we first describe the component of candidate generation 
Candidate Generation
Similar to previous work BIBREF24 , we use the prior probability INLINEFORM0 of entity INLINEFORM1 conditioned on mention INLINEFORM2 both as a local feature and to generate candidate entities: INLINEFORM3   We compute INLINEFORM4 based on statistics of mention-entity pairs from: (i) Wikipedia page titles, redirect titles and hyperlinks, (ii) the dictionary derived from a large Web Corpus BIBREF27 , and (iii) the YAGO dictionary with a uniform distribution BIBREF22   We pick up the maximal prior if a mention-entity pair occurs in different resources  In experiments, to optimize for memory and run time, we keep only top INLINEFORM5 entities based on INLINEFORM6   In the following two sections, we will present the key components of NECL, namely feature extraction and neural network for collective entity linking 
Feature Extraction
The main goal of NCEL is to find a solution for collective entity linking using an end-to-end neural model, rather than to improve the measurements of local textual similarity or global mention/entity relatedness  Therefore, we use joint embeddings of words and entities at sense level BIBREF28 to represent mentions and its contexts for feature extraction  In this section, we give a brief description of our embeddings followed by our features used in the neural model 
Learning Joint Embeddings of Word and Entity
Following BIBREF28 , we use Wikipedia articles, hyperlinks, and entity outlinks to jointly learn word/mention and entity embeddings in a unified vector space, so that similar words/mentions and entities have similar vectors  To address the ambiguity of words/mentions, BIBREF28 represents each word/mention with multiple vectors, and each vector denotes a sense referring to an entity in KB  The quality of the embeddings is verified on both textual similarity and entity relatedness tasks 
Formally, each word/mention has a global embedding INLINEFORM0 , and multiple sense embeddings INLINEFORM1   Each sense embedding INLINEFORM2 refers to an entity embedding INLINEFORM3 , while the difference between INLINEFORM4 and INLINEFORM5 is that INLINEFORM6 models the co-occurrence information of an entity in texts (via hyperlinks) and INLINEFORM7 encodes the structured entity relations in KBs  More details can be found in the original paper 
Local Features
Local features focus on how compatible the entity is mentioned in a piece of text (i e , the mention and the context words)  Except for the prior probability (Section SECREF9 ), we define two types of local features for each candidate entity INLINEFORM0 :
String Similarity Similar to BIBREF16 , we define string based features as follows: the edit distance between mention's surface form and entity title, and boolean features indicating whether they are equivalent, whether the mention is inside, starts with or ends with entity title and vice versa 
Compatibility We also measure the compatibility of INLINEFORM0 with the mention's context words INLINEFORM1 by computing their similarities based on joint embeddings: INLINEFORM2 and INLINEFORM3 , where INLINEFORM4 is the context embedding of INLINEFORM5 conditioned on candidate INLINEFORM6 and is defined as the average sum of word global vectors weighted by attentions: INLINEFORM7
where INLINEFORM0 is the INLINEFORM1 -th word's attention from INLINEFORM2   In this way, we automatically select informative words by assigning higher attention weights, and filter out irrelevant noise through small weights  The attention INLINEFORM3 is computed as follows: INLINEFORM4
where INLINEFORM0 is the similarity measurement, and we use cosine similarity in the presented work  We concatenate the prior probability, string based similarities, compatibility similarities and the embeddings of contexts as well as the entity as the local feature vectors 
Global Features
The key idea of collective EL is to utilize the topical coherence throughout the entire document  The consistency assumption behind it is that: all mentions in a document shall be on the same topic  However, this leads to exhaustive computations if the number of mentions is large  Based on the observation that the consistency attenuates along with the distance between two mentions, we argue that the adjacent mentions might be sufficient for supporting the assumption efficiently 
Formally, we define neighbor mentions as INLINEFORM0 adjacent mentions before and after current mention INLINEFORM1 : INLINEFORM2 , where INLINEFORM3 is the pre-defined window size  Thus, the topical coherence at document level shall be achieved in a chain-like way  As shown in Figure FIGREF10 ( INLINEFORM4 ), mentions Hussain and Essex, a cricket player and the cricket club, provide adequate disambiguation clues to induce the underlying topic “cricket" for the current mention England, which impacts positively on identifying the mention surrey as another cricket club via the common neighbor mention Essex 
A degraded case happens if INLINEFORM0 is large enough to cover the entire document, and the mentions used for global features become the same as the previous work, such as BIBREF21   In experiments, we heuristically found a suitable INLINEFORM1 which is much smaller than the total number of mentions  The benefits of efficiency are in two ways: (i) to decrease time complexity, and (ii) to trim the entity graph into a fixed size of subgraph that facilitates computation acceleration through GPUs and batch techniques, which will be discussed in Section SECREF24  
Given neighbor mentions INLINEFORM0 , we extract two types of vectorial global features and structured global features for each candidate INLINEFORM1 :
Neighbor Mention Compatibility Suppose neighbor mentions are topical coherent, a candidate entity shall also be compatible with neighbor mentions if it has a high compatibility score with the current mention, otherwise not  That is, we extract the vectorial global features by computing the similarities between INLINEFORM0 and all neighbor mentions: INLINEFORM1 , where INLINEFORM2 is the mention embedding by averaging the global vectors of words in its surface form: INLINEFORM3 , where INLINEFORM4 are tokenized words of mention INLINEFORM5  
Subgraph Structure The above features reflect the consistent semantics in texts (i e , mentions)  We now extract structured global features using the relations in KB, which facilitates the inference among candidates to find the most topical coherent subset  For each document, we obtain the entity graph INLINEFORM0 by taking candidate entities of all mentions INLINEFORM1 as nodes, and using entity embeddings to compute their similarities as edges INLINEFORM2   Then, we extract the subgraph structured features INLINEFORM3 for each entity INLINEFORM4 for efficiency 
Formally, we define the subgraph as: INLINEFORM0 , where INLINEFORM1   For example (Figure FIGREF1 ), for entity England cricket team, the subgraph contains the relation from it to all candidates of neighbor mentions: England cricket team, Nasser Hussain (rugby union), Nasser Hussain, Essex, Essex County Cricket Club and Essex, New York  To support batch-wise acceleration, we represent INLINEFORM2 in the form of adjacency table based vectors: INLINEFORM3 , where INLINEFORM4 is the number of candidates per mention 
Finally, for each candidate INLINEFORM0 , we concatenate local features and neighbor mention compatibility scores as the feature vector INLINEFORM1 , and construct the subgraph structure representation INLINEFORM2 as the inputs of NCEL 
Neural Collective Entity Linking
NCEL incorporates GCN into a deep neural network to utilize structured graph information for collectively feature abstraction, while differs from conventional GCN in the way of applying the graph  Instead of the entire graph, only a subset of nodes is “visible" to each node in our proposed method, and then the overall structured information shall be reached in a chain-like way  Fixing the size of the subset, NCEL is further speeded up by batch techniques and GPUs, and is efficient to large-scale data 
Graph Convolutional Network
GCNs are a type of neural network model that deals with structured data  It takes a graph as an input and output labels for each node  As a simplification of spectral graph convolutions, the main idea of BIBREF26 is similar to a propagation model: to enhance the features of a node according to its neighbor nodes  The formulation is as follows: INLINEFORM0
where INLINEFORM0 is a normalized adjacent matrix of the input graph with self-connection, INLINEFORM1 and INLINEFORM2 are the hidden states and weights in the INLINEFORM3 -th layer, and INLINEFORM4 is a non-linear activation, such as ReLu 
Model Architecture
As shown in Figure FIGREF10 , NCEL identifies the correct candidate INLINEFORM0 for the mention INLINEFORM1 by using vectorial features as well as structured relatedness with candidates of neighbor mentions INLINEFORM2   Given feature vector INLINEFORM3 and subgraph representation INLINEFORM4 of each candidate INLINEFORM5 , we stack them as inputs for mention INLINEFORM6 : INLINEFORM7 , and the adjacent matrix INLINEFORM8 , where INLINEFORM9 denotes the subgraph with self-connection  We normalize INLINEFORM10 such that all rows sum to one, denoted as INLINEFORM11 , avoiding the change in the scale of the feature vectors 
Given INLINEFORM0 and INLINEFORM1 , the goal of NCEL is to find the best assignment: INLINEFORM2
where INLINEFORM0 is the output variable of candidates, and INLINEFORM1 is a probability function as follows: INLINEFORM2
where INLINEFORM0 is the score function parameters by INLINEFORM1   NCEL learns the mapping INLINEFORM2 through a neural network including three main modules: encoder, sub-graph convolution network (sub-GCN) and decoder  Next, we introduce them in turn 
Encoder The function of this module is to integrate different features by a multi-layer perceptron (MLP): INLINEFORM0
where INLINEFORM0 is the hidden states of the current mention, INLINEFORM1 and INLINEFORM2 are trainable parameters and bias  We use ReLu as the non-linear activation INLINEFORM3  
Sub-Graph Convolution Network Similar to GCN, this module learns to abstract features from the hidden state of the mention itself as well as its neighbors  Suppose INLINEFORM0 is the hidden states of the neighbor INLINEFORM1 , we stack them to expand the current hidden states of INLINEFORM2 as INLINEFORM3 , such that each row corresponds to that in the subgraph adjacent matrix INLINEFORM4   We define sub-graph convolution as: INLINEFORM5
where INLINEFORM0 is a trainable parameter 
Decoder After INLINEFORM0 iterations of sub-graph convolution, the hidden states integrate both features of INLINEFORM1 and its neighbors  A fully connected decoder maps INLINEFORM2 to the number of candidates as follows: INLINEFORM3
where INLINEFORM0  
Training
The parameters of network are trained to minimize cross-entropy of the predicted and ground truth INLINEFORM0 : INLINEFORM1
Suppose there are INLINEFORM0 documents in training corpus, each document has a set of mentions INLINEFORM1 , leading to totally INLINEFORM2 mention sets  The overall objective function is as follows: INLINEFORM3
Experiments
To avoid overfitting with some dataset, we train NCEL using collected Wikipedia hyperlinks instead of specific annotated data  We then evaluate the trained model on five different benchmarks to verify the linking precision as well as the generalization ability  Furthermore, we investigate the effectiveness of key modules in NCEL and give qualitative results for comprehensive analysis 
Baselines and Datasets
We compare NCEL with the following state-of-the-art EL methods including three local models and three types of global models:
Local models: He BIBREF29 and Chisholm BIBREF6 beat many global models by using auto-encoders and web links, respectively, and NTEE BIBREF16 achieves the best performance based on joint embeddings of words and entities 
Iterative model: AIDA BIBREF22 links entities by iteratively finding a dense subgraph 
Loopy Belief Propagation: Globerson BIBREF18 and PBoH BIBREF30 introduce LBP BIBREF31 techniques for collective inference, and Ganea BIBREF24 solves the global training problem via truncated fitting LBP 
PageRank/Random Walk: Boosting BIBREF32 , AGDISTISG BIBREF33 , Babelfy BIBREF34 , WAT BIBREF35 , xLisa BIBREF36 and WNED BIBREF19 performs PageRank BIBREF37 or random walk BIBREF38 on the mention-entity graph and use the convergence score for disambiguation 
For fairly comparison, we report the original scores of the baselines in the papers  Following these methods, we evaluate NCEL on the following five datasets: (1) CoNLL-YAGO BIBREF22 : the CoNLL 2003 shared task including testa of 4791 mentions in 216 documents, and testb of 4485 mentions in 213 documents  (2) TAC2010 BIBREF39 : constructed for the Text Analysis Conference that comprises 676 mentions in 352 documents for testing  (3) ACE2004 BIBREF23 : a subset of ACE2004 co-reference documents including 248 mentions in 35 documents, which is annotated by Amazon Mechanical Turk  (4) AQUAINT BIBREF40 : 50 news articles including 699 mentions from three different news agencies  (5) WW BIBREF19 : a new benchmark with balanced prior distributions of mentions, leading to a hard case of disambiguation  It has 6374 mentions in 310 documents automatically extracted from Wikipedia 
Training Details and Running Time Analysis
Training We collect 50,000 Wikipedia articles according to the number of its hyperlinks as our training data  For efficiency, we trim the articles to the first three paragraphs leading to 1,035,665 mentions in total  Using CoNLL-Test A as the development set, we evaluate the trained NCEL on the above benchmarks  We set context window to 20, neighbor mention window to 6, and top INLINEFORM0 candidates for each mention  We use two layers with 2000 and 1 hidden units in MLP encoder, and 3 layers in sub-GCN  We use early stop and fine tune the embeddings  With a batch size of 16, nearly 3 epochs cost less than 15 minutes on the server with 20 core CPU and the GeForce GTX 1080Ti GPU with 12Gb memory  We use standard Precision, Recall and F1 at mention level (Micro) and at the document level (Macro) as measurements 
Complexity Analysis Compared with local methods, the main disadvantage of collective methods is high complexity and expensive costs  Suppose there are INLINEFORM0 mentions in documents on average, among these global models, NCEL not surprisingly has the lowest time complexity INLINEFORM1 since it only considers adjacent mentions, where INLINEFORM2 is the number of sub-GCN layers indicating the iterations until convergence  AIDA has the highest time complexity INLINEFORM3 in worst case due to exhaustive iteratively finding and sorting the graph  The LBP and PageRank/random walk based methods achieve similar high time complexity of INLINEFORM4 mainly because of the inference on the entire graph 
Results on GERBIL
GERBIL BIBREF41 is a benchmark entity annotation framework that aims to provide a unified comparison among different EL methods across datasets including ACE2004, AQUAINT and CoNLL  We compare NCEL with the global models that report the performance on GERBIL 
As shown in Table TABREF26 , NCEL achieves the best performance in most cases with an average gain of 2% on Micro F1 and 3% Macro F1  The baseline methods also achieve competitive results on some datasets but fail to adapt to the others  For example, AIDA and xLisa perform quite well on ACE2004 but poorly on other datasets, or WAT, PBoH, and WNED have a favorable performance on CoNLL but lower values on ACE2004 and AQUAINT  Our proposed method performs consistently well on all datasets that demonstrates the good generalization ability 
Results on TAC2010 and WW
In this section, we investigate the effectiveness of NCEL in the “easy" and “hard" datasets, respectively  Particularly, TAC2010, which has two mentions per document on average (Section SECREF19 ) and high prior probabilities of correct candidates (Figure FIGREF28 ), is regarded as the “easy" case for EL, and WW is the “hard" case since it has the most mentions with balanced prior probabilities BIBREF19   Besides, we further compare the impact of key modules by removing the following part from NCEL: global features (NCEL-local), attention (NCEL-noatt), embedding features (NCEL-noemb), and the impact of the prior probability (prior) 
The results are shown in Table FIGREF28 and Table FIGREF28   We can see the average linking precision (Micro) of WW is lower than that of TAC2010, and NCEL outperforms all baseline methods in both easy and hard cases  In the “easy" case, local models have similar performance with global models since only little global information is available (2 mentions per document)  Besides, NN-based models, NTEE and NCEL-local, perform significantly better than others including most global models, demonstrating that the effectiveness of neural models deals with the first limitation in the introduction 
Impact of NCEL Modules
As shown in Figure FIGREF28 , the prior probability performs quite well in TAC2010 but poorly in WW  Compared with NCEL-local, the global module in NCEL brings more improvements in the “hard" case than that for “easy" dataset, because local features are discriminative enough in most cases of TAC2010, and global information becomes quite helpful when local features cannot handle  That is, our propose collective model is robust and shows a good generalization ability to difficult EL  The improvements by each main module are relatively small in TAC2010, while the modules of attention and embedding features show non-negligible impacts in WW (even worse than local model), mainly because WW contains much noise, and these two modules are effective in improving the robustness to noise and the ability of generalization by selecting informative words and providing more accurate semantics, respectively 
Qualitative Analysis
The results of example in Figure FIGREF1 are shown in Table TABREF30 , which is from CoNLL testa dataset  For mention Essex, although both NCEL and NCEL-local correctly identify entity Essex County Cricket Club, NCEL outputs higher probability due to the enhancement of neighbor mentions  Moreover, for mention England, NCEL-local cannot find enough disambiguation clues from its context words, such as surplus and requirements, and thus assigns a higher probability of 0 42 to the country England according to the prior probability  Collectively, NCEL correctly identifies England cricket team with a probability of 0 72 as compared with 0 20 in NCEL-local with the help of its neighbor mention Essex 
Conclusion
In this paper, we propose a neural model for collective entity linking that is end-to-end trainable  It applies GCN on subgraphs instead of the entire entity graph to efficiently learn features from both local and global information  We design an attention mechanism that endows NCEL robust to noisy data  Trained on collected Wikipedia hyperlinks, NCEL outperforms the state-of-the-art collective methods across five different datasets  Besides, further analysis of the impacts of main modules as well as qualitative results demonstrates its effectiveness 
In the future, we will extend our method into cross-lingual settings to help link entities in low-resourced languages by exploiting rich knowledge from high-resourced languages, and deal with NIL entities to facilitate specific applications 
Acknowledgments
The work is supported by National Key Research and Development Program of China (2017YFB1002101), NSFC key project (U1736204, 61661146007), and THUNUS NExT Co-Lab 
Context 2: Introduction
In the age of information dissemination without quality control, it has enabled malicious users to spread misinformation via social media and aim individual users with propaganda campaigns to achieve political and financial gains as well as advance a specific agenda  Often disinformation is complied in the two major forms: fake news and propaganda, where they differ in the sense that the propaganda is possibly built upon true information (e g , biased, loaded language, repetition, etc ) 
Prior works BIBREF0, BIBREF1, BIBREF2 in detecting propaganda have focused primarily at document level, typically labeling all articles from a propagandistic news outlet as propaganda and thus, often non-propagandistic articles from the outlet are mislabeled  To this end, EMNLP19DaSanMartino focuses on analyzing the use of propaganda and detecting specific propagandistic techniques in news articles at sentence and fragment level, respectively and thus, promotes explainable AI  For instance, the following text is a propaganda of type `slogan' 
Trump tweeted: $\underbrace{\text{`}`{\texttt {BUILD THE WALL!}"}}_{\text{slogan}}$
Shared Task: This work addresses the two tasks in propaganda detection BIBREF3 of different granularities: (1) Sentence-level Classification (SLC), a binary classification that predicts whether a sentence contains at least one propaganda technique, and (2) Fragment-level Classification (FLC), a token-level (multi-label) classification that identifies both the spans and the type of propaganda technique(s) 
Contributions: (1) To address SLC, we design an ensemble of different classifiers based on Logistic Regression, CNN and BERT, and leverage transfer learning benefits using the pre-trained embeddings/models from FastText and BERT  We also employed different features such as linguistic (sentiment, readability, emotion, part-of-speech and named entity tags, etc ), layout, topics, etc  (2) To address FLC, we design a multi-task neural sequence tagger based on LSTM-CRF and linguistic features to jointly detect propagandistic fragments and its type  Moreover, we investigate performing FLC and SLC jointly in a multi-granularity network based on LSTM-CRF and BERT  (3) Our system (MIC-CIS) is ranked 3rd (out of 12 participants) and 4th (out of 25 participants) in FLC and SLC tasks, respectively 
System Description ::: Linguistic, Layout and Topical Features
Some of the propaganda techniques BIBREF3 involve word and phrases that express strong emotional implications, exaggeration, minimization, doubt, national feeling, labeling , stereotyping, etc  This inspires us in extracting different features (Table TABREF1) including the complexity of text, sentiment, emotion, lexical (POS, NER, etc ), layout, etc  To further investigate, we use topical features (e g , document-topic proportion) BIBREF4, BIBREF5, BIBREF6 at sentence and document levels in order to determine irrelevant themes, if introduced to the issue being discussed (e g , Red Herring) 
For word and sentence representations, we use pre-trained vectors from FastText BIBREF7 and BERT BIBREF8 
System Description ::: Sentence-level Propaganda Detection
Figure FIGREF2 (left) describes the three components of our system for SLC task: features, classifiers and ensemble  The arrows from features-to-classifier indicate that we investigate linguistic, layout and topical features in the two binary classifiers: LogisticRegression and CNN  For CNN, we follow the architecture of DBLP:conf/emnlp/Kim14 for sentence-level classification, initializing the word vectors by FastText or BERT  We concatenate features in the last hidden layer before classification 
One of our strong classifiers includes BERT that has achieved state-of-the-art performance on multiple NLP benchmarks  Following DBLP:conf/naacl/DevlinCLT19, we fine-tune BERT for binary classification, initializing with a pre-trained model (i e , BERT-base, Cased)  Additionally, we apply a decision function such that a sentence is tagged as propaganda if prediction probability of the classifier is greater than a threshold ($\tau $)  We relax the binary decision boundary to boost recall, similar to pankajgupta:CrossRE2019 
Ensemble of Logistic Regression, CNN and BERT: In the final component, we collect predictions (i e , propaganda label) for each sentence from the three ($\mathcal {M}=3$) classifiers and thus, obtain $\mathcal {M}$ number of predictions for each sentence  We explore two ensemble strategies (Table TABREF1): majority-voting and relax-voting to boost precision and recall, respectively 
System Description ::: Fragment-level Propaganda Detection
Figure FIGREF2 (right) describes our system for FLC task, where we design sequence taggers BIBREF9, BIBREF10 in three modes: (1) LSTM-CRF BIBREF11 with word embeddings ($w\_e$) and character embeddings $c\_e$, token-level features ($t\_f$) such as polarity, POS, NER, etc  (2) LSTM-CRF+Multi-grain that jointly performs FLC and SLC with FastTextWordEmb and BERTSentEmb, respectively  Here, we add binary sentence classification loss to sequence tagging weighted by a factor of $\alpha $  (3) LSTM-CRF+Multi-task that performs propagandistic span/fragment detection (PFD) and FLC (fragment detection + 19-way classification) 
Ensemble of Multi-grain, Multi-task LSTM-CRF with BERT: Here, we build an ensemble by considering propagandistic fragments (and its type) from each of the sequence taggers  In doing so, we first perform majority voting at the fragment level for the fragment where their spans exactly overlap  In case of non-overlapping fragments, we consider all  However, when the spans overlap (though with the same label), we consider the fragment with the largest span 
Experiments and Evaluation
Data: While the SLC task is binary, the FLC consists of 18 propaganda techniques BIBREF3  We split (80-20%) the annotated corpus into 5-folds and 3-folds for SLC and FLC tasks, respectively  The development set of each the folds is represented by dev (internal); however, the un-annotated corpus used in leaderboard comparisons by dev (external)  We remove empty and single token sentences after tokenization  Experimental Setup: We use PyTorch framework for the pre-trained BERT model (Bert-base-cased), fine-tuned for SLC task  In the multi-granularity loss, we set $\alpha = 0 1$ for sentence classification based on dev (internal, fold1) scores  We use BIO tagging scheme of NER in FLC task  For CNN, we follow DBLP:conf/emnlp/Kim14 with filter-sizes of [2, 3, 4, 5, 6], 128 filters and 16 batch-size  We compute binary-F1and macro-F1 BIBREF12 in SLC and FLC, respectively on dev (internal) 
Experiments and Evaluation ::: Results: Sentence-Level Propaganda
Table TABREF10 shows the scores on dev (internal and external) for SLC task  Observe that the pre-trained embeddings (FastText or BERT) outperform TF-IDF vector representation  In row r2, we apply logistic regression classifier with BERTSentEmb that leads to improved scores over FastTextSentEmb  Subsequently, we augment the sentence vector with additional features that improves F1 on dev (external), however not dev (internal)  Next, we initialize CNN by FastTextWordEmb or BERTWordEmb and augment the last hidden layer (before classification) with BERTSentEmb and feature vectors, leading to gains in F1 for both the dev sets  Further, we fine-tune BERT and apply different thresholds in relaxing the decision boundary, where $\tau \ge 0 35$ is found optimal 
We choose the three different models in the ensemble: Logistic Regression, CNN and BERT on fold1 and subsequently an ensemble+ of r3, r6 and r12 from each fold1-5 (i e , 15 models) to obtain predictions for dev (external)  We investigate different ensemble schemes (r17-r19), where we observe that the relax-voting improves recall and therefore, the higher F1 (i e , 0 673)  In postprocess step, we check for repetition propaganda technique by computing cosine similarity between the current sentence and its preceding $w=10$ sentence vectors (i e , BERTSentEmb) in the document  If the cosine-similarity is greater than $\lambda \in \lbrace  99,  95\rbrace $, then the current sentence is labeled as propaganda due to repetition  Comparing r19 and r21, we observe a gain in recall, however an overall decrease in F1 applying postprocess 
Finally, we use the configuration of r19 on the test set  The ensemble+ of (r4, r7 r12) was analyzed after test submission  Table TABREF9 (SLC) shows that our submission is ranked at 4th position 
Experiments and Evaluation ::: Results: Fragment-Level Propaganda
Table TABREF11 shows the scores on dev (internal and external) for FLC task  Observe that the features (i e , polarity, POS and NER in row II) when introduced in LSTM-CRF improves F1  We run multi-grained LSTM-CRF without BERTSentEmb (i e , row III) and with it (i e , row IV), where the latter improves scores on dev (internal), however not on dev (external)  Finally, we perform multi-tasking with another auxiliary task of PFD  Given the scores on dev (internal and external) using different configurations (rows I-V), it is difficult to infer the optimal configuration  Thus, we choose the two best configurations (II and IV) on dev (internal) set and build an ensemble+ of predictions (discussed in section SECREF6), leading to a boost in recall and thus an improved F1 on dev (external) 
Finally, we use the ensemble+ of (II and IV) from each of the folds 1-3, i e , $|{\mathcal {M}}|=6$ models to obtain predictions on test  Table TABREF9 (FLC) shows that our submission is ranked at 3rd position 
Conclusion and Future Work
Our system (Team: MIC-CIS) explores different neural architectures (CNN, BERT and LSTM-CRF) with linguistic, layout and topical features to address the tasks of fine-grained propaganda detection  We have demonstrated gains in performance due to the features, ensemble schemes, multi-tasking and multi-granularity architectures  Compared to the other participating systems, our submissions are ranked 3rd and 4th in FLC and SLC tasks, respectively 
In future, we would like to enrich BERT models with linguistic, layout and topical features during their fine-tuning  Further, we would also be interested in understanding and analyzing the neural network learning, i e , extracting salient fragments (or key-phrases) in the sentence that generate propaganda, similar to pankajgupta:2018LISA in order to promote explainable AI 
Context 3: Introduction
With the increasing popularity of the Internet, online texts provided by social media platform (e g  Twitter) and news media sites (e g  Google news) have become important sources of real-world events  Therefore, it is crucial to automatically extract events from online texts 
Due to the high variety of events discussed online and the difficulty in obtaining annotated data for training, traditional template-based or supervised learning approaches for event extraction are no longer applicable in dealing with online texts  Nevertheless, newsworthy events are often discussed by many tweets or online news articles  Therefore, the same event could be mentioned by a high volume of redundant tweets or news articles  This property inspires the research community to devise clustering-based models BIBREF0 , BIBREF1 , BIBREF2 to discover new or previously unidentified events without extracting structured representations 
To extract structured representations of events such as who did what, when, where and why, Bayesian approaches have made some progress  Assuming that each document is assigned to a single event, which is modeled as a joint distribution over the named entities, the date and the location of the event, and the event-related keywords, Zhou et al  zhou2014simple proposed an unsupervised Latent Event Model (LEM) for open-domain event extraction  To address the limitation that LEM requires the number of events to be pre-set, Zhou et al  zhou2017event further proposed the Dirichlet Process Event Mixture Model (DPEMM) in which the number of events can be learned automatically from data  However, both LEM and DPEMM have two limitations: (1) they assume that all words in a document are generated from a single event which can be represented by a quadruple INLINEFORM0 entity, location, keyword, date INLINEFORM1   However, long texts such news articles often describe multiple events which clearly violates this assumption; (2) During the inference process of both approaches, the Gibbs sampler needs to compute the conditional posterior distribution and assigns an event for each document  This is time consuming and takes long time to converge 
To deal with these limitations, in this paper, we propose the Adversarial-neural Event Model (AEM) based on adversarial training for open-domain event extraction  The principle idea is to use a generator network to learn the projection function between the document-event distribution and four event related word distributions (entity distribution, location distribution, keyword distribution and date distribution)  Instead of providing an analytic approximation, AEM uses a discriminator network to discriminate between the reconstructed documents from latent events and the original input documents  This essentially helps the generator to construct a more realistic document from a random noise drawn from a Dirichlet distribution  Due to the flexibility of neural networks, the generator is capable of learning complicated nonlinear distributions  And the supervision signal provided by the discriminator will help generator to capture the event-related patterns  Furthermore, the discriminator also provides low-dimensional discriminative features which can be used to visualize documents and events 
The main contributions of the paper are summarized below:
Related Work
Our work is related to two lines of research, event extraction and Generative Adversarial Nets 
Event Extraction
Recently there has been much interest in event extraction from online texts, and approaches could be categorized as domain-specific and open-domain event extraction 
Domain-specific event extraction often focuses on the specific types of events (e g  sports events or city events)  Panem et al  panem2014structured devised a novel algorithm to extract attribute-value pairs and mapped them to manually generated schemes for extracting the natural disaster events  Similarly, to extract the city-traffic related event, Anantharam et al  anantharam2015extracting viewed the task as a sequential tagging problem and proposed an approach based on the conditional random fields  Zhang zhang2018event proposed an event extraction approach based on imitation learning, especially on inverse reinforcement learning 
Open-domain event extraction aims to extract events without limiting the specific types of events  To analyze individual messages and induce a canonical value for each event, Benson et al  benson2011event proposed an approach based on a structured graphical model  By representing an event with a binary tuple which is constituted by a named entity and a date, Ritter et al  ritter2012open employed some statistic to measure the strength of associations between a named entity and a date  The proposed system relies on a supervised labeler trained on annotated data  In BIBREF1 , Abdelhaq et al  developed a real-time event extraction system called EvenTweet, and each event is represented as a triple constituted by time, location and keywords  To extract more information, Wang el al  wang2015seeft developed a system employing the links in tweets and combing tweets with linked articles to identify events  Xia el al  xia2015new combined texts with the location information to detect the events with low spatial and temporal deviations  Zhou et al  zhou2014simple,zhou2017event represented event as a quadruple and proposed two Bayesian models to extract events from tweets 
Generative Adversarial Nets
As a neural-based generative model, Generative Adversarial Nets BIBREF3 have been extensively researched in natural language processing (NLP) community 
For text generation, the sequence generative adversarial network (SeqGAN) proposed in BIBREF4 incorporated a policy gradient strategy to optimize the generation process  Based on the policy gradient, Lin et al  lin2017adversarial proposed RankGAN to capture the rich structures of language by ranking and analyzing a collection of human-written and machine-written sentences  To overcome mode collapse when dealing with discrete data, Fedus et al  fedus2018maskgan proposed MaskGAN which used an actor-critic conditional GAN to fill in missing text conditioned on the surrounding context  Along this line, Wang et al  wang2018sentigan proposed SentiGAN to generate texts of different sentiment labels  Besides, Li et al  li2018learning improved the performance of semi-supervised text classification using adversarial training, BIBREF5 , BIBREF6 designed GAN-based models for distance supervision relation extraction 
Although various GAN based approaches have been explored for many applications, none of these approaches tackles open-domain event extraction from online texts  We propose a novel GAN-based event extraction model called AEM  Compared with the previous models, AEM has the following differences: (1) Unlike most GAN-based text generation approaches, a generator network is employed in AEM to learn the projection function between an event distribution and the event-related word distributions (entity, location, keyword, date)  The learned generator captures event-related patterns rather than generating text sequence; (2) Different from LEM and DPEMM, AEM uses a generator network to capture the event-related patterns and is able to mine events from different text sources (short and long)  Moreover, unlike traditional inference procedure, such as Gibbs sampling used in LEM and DPEMM, AEM could extract the events more efficiently due to the CUDA acceleration; (3) The discriminative features learned by the discriminator of AEM provide a straightforward way to visualize the extracted events 
Methodology
We describe Adversarial-neural Event Model (AEM) in this section  An event is represented as a quadruple < INLINEFORM0 >, where INLINEFORM1 stands for non-location named entities, INLINEFORM2 for a location, INLINEFORM3 for event-related keywords, INLINEFORM4 for a date, and each component in the tuple is represented by component-specific representative words 
AEM is constituted by three components: (1) The document representation module, as shown at the top of Figure FIGREF4 , defines a document representation approach which converts an input document from the online text corpus into INLINEFORM0 which captures the key event elements; (2) The generator INLINEFORM1 , as shown in the lower-left part of Figure FIGREF4 , generates a fake document INLINEFORM2 which is constituted by four multinomial distributions using an event distribution INLINEFORM3 drawn from a Dirichlet distribution as input; (3) The discriminator INLINEFORM4 , as shown in the lower-right part of Figure FIGREF4 , distinguishes the real documents from the fake ones and its output is subsequently employed as a learning signal to update the INLINEFORM5 and INLINEFORM6   The details of each component are presented below 
Document Representation
Each document INLINEFORM0 in a given corpus INLINEFORM1 is represented as a concatenation of 4 multinomial distributions which are entity distribution ( INLINEFORM2 ), location distribution ( INLINEFORM3 ), keyword distribution ( INLINEFORM4 ) and date distribution ( INLINEFORM5 ) of the document  As four distributions are calculated in a similar way, we only describe the computation of the entity distribution below as an example 
The entity distribution INLINEFORM0 is represented by a normalized INLINEFORM1 -dimensional vector weighted by TF-IDF, and it is calculated as: INLINEFORM2
where INLINEFORM0 is the pseudo corpus constructed by removing all non-entity words from INLINEFORM1 , INLINEFORM2 is the total number of distinct entities in a corpus, INLINEFORM3 denotes the number of INLINEFORM4 -th entity appeared in document INLINEFORM5 , INLINEFORM6 represents the number of documents in the corpus, and INLINEFORM7 is the number of documents that contain INLINEFORM8 -th entity, and the obtained INLINEFORM9 denotes the relevance between INLINEFORM10 -th entity and document INLINEFORM11  
Similarly, location distribution INLINEFORM0 , keyword distribution INLINEFORM1 and date distribution INLINEFORM2 of INLINEFORM3 could be calculated in the same way, and the dimensions of these distributions are denoted as INLINEFORM4 , INLINEFORM5 and INLINEFORM6 , respectively  Finally, each document INLINEFORM7 in the corpus is represented by a INLINEFORM8 -dimensional ( INLINEFORM9 = INLINEFORM10 + INLINEFORM11 + INLINEFORM12 + INLINEFORM13 ) vector INLINEFORM14 by concatenating four computed distributions 
Network Architecture
The generator network INLINEFORM0 is designed to learn the projection function between the document-event distribution INLINEFORM1 and the four document-level word distributions (entity distribution, location distribution, keyword distribution and date distribution) 
More concretely, INLINEFORM0 consists of a INLINEFORM1 -dimensional document-event distribution layer, INLINEFORM2 -dimensional hidden layer and INLINEFORM3 -dimensional event-related word distribution layer  Here, INLINEFORM4 denotes the event number, INLINEFORM5 is the number of units in the hidden layer, INLINEFORM6 is the vocabulary size and equals to INLINEFORM7 + INLINEFORM8 + INLINEFORM9 + INLINEFORM10   As shown in Figure FIGREF4 , INLINEFORM11 firstly employs a random document-event distribution INLINEFORM12 as an input  To model the multinomial property of the document-event distribution, INLINEFORM13 is drawn from a Dirichlet distribution parameterized with INLINEFORM14 which is formulated as: DISPLAYFORM0
where INLINEFORM0 is the hyper-parameter of the dirichlet distribution, INLINEFORM1 is the number of events which should be set in AEM, INLINEFORM2 , INLINEFORM3 represents the proportion of event INLINEFORM4 in the document and INLINEFORM5  
Subsequently, INLINEFORM0 transforms INLINEFORM1 into a INLINEFORM2 -dimensional hidden space using a linear layer followed by layer normalization, and the transformation is defined as: DISPLAYFORM0
where INLINEFORM0 represents the weight matrix of hidden layer, and INLINEFORM1 denotes the bias term, INLINEFORM2 is the parameter of LeakyReLU activation and is set to 0 1, INLINEFORM3 and INLINEFORM4 denote the normalized hidden states and the outputs of the hidden layer, and INLINEFORM5 represents the layer normalization 
Then, to project INLINEFORM0 into four document-level event related word distributions ( INLINEFORM1 , INLINEFORM2 , INLINEFORM3 and INLINEFORM4 shown in Figure FIGREF4 ), four subnets (each contains a linear layer, a batch normalization layer and a softmax layer) are employed in INLINEFORM5   And the exact transformation is based on the formulas below: DISPLAYFORM0
where INLINEFORM0 means softmax layer, INLINEFORM1 , INLINEFORM2 , INLINEFORM3 and INLINEFORM4 denote the weight matrices of the linear layers in subnets, INLINEFORM5 , INLINEFORM6 , INLINEFORM7 and INLINEFORM8 represent the corresponding bias terms, INLINEFORM9 , INLINEFORM10 , INLINEFORM11 and INLINEFORM12 are state vectors  INLINEFORM13 , INLINEFORM14 , INLINEFORM15 and INLINEFORM16 denote the generated entity distribution, location distribution, keyword distribution and date distribution, respectively, that correspond to the given event distribution INLINEFORM17   And each dimension represents the relevance between corresponding entity/location/keyword/date term and the input event distribution 
Finally, four generated distributions are concatenated to represent the generated document INLINEFORM0 corresponding to the input INLINEFORM1 : DISPLAYFORM0
The discriminator network INLINEFORM0 is designed as a fully-connected network which contains an input layer, a discriminative feature layer (discriminative features are employed for event visualization) and an output layer  In AEM, INLINEFORM1 uses fake document INLINEFORM2 and real document INLINEFORM3 as input and outputs the signal INLINEFORM4 to indicate the source of the input data (lower value denotes that INLINEFORM5 is prone to predict the input data as a fake document and vice versa) 
As have previously been discussed in BIBREF7 , BIBREF8 , lipschitz continuity of INLINEFORM0 network is crucial to the training of the GAN-based approaches  To ensure the lipschitz continuity of INLINEFORM1 , we employ the spectral normalization technique BIBREF9   More concretely, for each linear layer INLINEFORM2 (bias term is omitted for simplicity) in INLINEFORM3 , the weight matrix INLINEFORM4 is normalized by INLINEFORM5   Here, INLINEFORM6 is the spectral norm of the weight matrix INLINEFORM7 with the definition below: DISPLAYFORM0
which is equivalent to the largest singular value of INLINEFORM0 

ground_truth: NCEL consistently outperforms various baselines with a favorable generalization ability

prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. What is best performing model among author's submissions, what performance it had?
Context 3:  The weight matrix INLINEFORM1 is then normalized using: DISPLAYFORM0
Obviously, the normalized weight matrix INLINEFORM0 satisfies that INLINEFORM1 and further ensures the lipschitz continuity of the INLINEFORM2 network BIBREF9   To reduce the high cost of computing spectral norm INLINEFORM3 using singular value decomposition at each iteration, we follow BIBREF10 and employ the power iteration method to estimate INLINEFORM4 instead  With this substitution, the spectral norm can be estimated with very small additional computational time 
Objective and Training Procedure
The real document INLINEFORM0 and fake document INLINEFORM1 shown in Figure FIGREF4 could be viewed as random samples from two distributions INLINEFORM2 and INLINEFORM3 , and each of them is a joint distribution constituted by four Dirichlet distributions (corresponding to entity distribution, location distribution, keyword distribution and date distribution)  The training objective of AEM is to let the distribution INLINEFORM4 (produced by INLINEFORM5 network) to approximate the real data distribution INLINEFORM6 as much as possible 
To compare the different GAN losses, Kurach kurach2018gan takes a sober view of the current state of GAN and suggests that the Jansen-Shannon divergence used in BIBREF3 performs more stable than variant objectives  Besides, Kurach also advocates that the gradient penalty (GP) regularization devised in BIBREF8 will further improve the stability of the model  Thus, the objective function of the proposed AEM is defined as: DISPLAYFORM0
where INLINEFORM0 denotes the discriminator loss, INLINEFORM1 represents the gradient penalty regularization loss, INLINEFORM2 is the gradient penalty coefficient which trade-off the two components of objective, INLINEFORM3 could be obtained by sampling uniformly along a straight line between INLINEFORM4 and INLINEFORM5 , INLINEFORM6 denotes the corresponding distribution 
The training procedure of AEM is presented in Algorithm SECREF15 , where INLINEFORM0 is the event number, INLINEFORM1 denotes the number of discriminator iterations per generator iteration, INLINEFORM2 is the batch size, INLINEFORM3 represents the learning rate, INLINEFORM4 and INLINEFORM5 are hyper-parameters of Adam BIBREF11 , INLINEFORM6 denotes INLINEFORM7   In this paper, we set INLINEFORM8 , INLINEFORM9 , INLINEFORM10   Moreover, INLINEFORM11 , INLINEFORM12 and INLINEFORM13 are set as 0 0002, 0 5 and 0 999 
[!h] Training procedure for AEM [1] INLINEFORM0 , INLINEFORM1 , INLINEFORM2 , INLINEFORM3 , INLINEFORM4 , INLINEFORM5 , INLINEFORM6 the trained INLINEFORM7 and INLINEFORM8   Initial INLINEFORM9 parameters INLINEFORM10 and INLINEFORM11 parameter INLINEFORM12 INLINEFORM13 has not converged INLINEFORM14 INLINEFORM15 Sample INLINEFORM16 , Sample a random INLINEFORM17 Sample a random number INLINEFORM18 INLINEFORM19 INLINEFORM20 INLINEFORM21 INLINEFORM22 INLINEFORM23 INLINEFORM24 Sample INLINEFORM25 noise INLINEFORM26 INLINEFORM27
Event Generation
After the model training, the generator INLINEFORM0 learns the mapping function between the document-event distribution and the document-level event-related word distributions (entity, location, keyword and date)  In other words, with an event distribution INLINEFORM1 as input, INLINEFORM2 could generate the corresponding entity distribution, location distribution, keyword distribution and date distribution 
In AEM, we employ event seed INLINEFORM0 , an INLINEFORM1 -dimensional vector with one-hot encoding, to generate the event related word distributions  For example, in ten event setting, INLINEFORM2 represents the event seed of the first event  With the event seed INLINEFORM3 as input, the corresponding distributions could be generated by INLINEFORM4 based on the equation below: DISPLAYFORM0
where INLINEFORM0 , INLINEFORM1 , INLINEFORM2 and INLINEFORM3 denote the entity distribution, location distribution, keyword distribution and date distribution of the first event respectively 
Experiments
In this section, we firstly describe the datasets and baseline approaches used in our experiments and then present the experimental results 
Experimental Setup
To validate the effectiveness of AEM for extracting events from social media (e g  Twitter) and news media sites (e g  Google news), three datasets (FSD BIBREF12 , Twitter, and Google datasets) are employed  Details are summarized below:
FSD dataset (social media) is the first story detection dataset containing 2,499 tweets  We filter out events mentioned in less than 15 tweets since events mentioned in very few tweets are less likely to be significant  The final dataset contains 2,453 tweets annotated with 20 events 
Twitter dataset (social media) is collected from tweets published in the month of December in 2010 using Twitter streaming API  It contains 1,000 tweets annotated with 20 events 
Google dataset (news article) is a subset of GDELT Event Database INLINEFORM0 , documents are retrieved by event related words  For example, documents which contain `malaysia', `airline', `search' and `plane' are retrieved for event MH370  By combining 30 events related documents, the dataset contains 11,909 news articles 
We choose the following three models as the baselines:
K-means is a well known data clustering algorithm, we implement the algorithm using sklearn toolbox, and represent documents using bag-of-words weighted by TF-IDF 
LEM BIBREF13 is a Bayesian modeling approach for open-domain event extraction  It treats an event as a latent variable and models the generation of an event as a joint distribution of its individual event elements  We implement the algorithm with the default configuration 
DPEMM BIBREF14 is a non-parametric mixture model for event extraction  It addresses the limitation of LEM that the number of events should be known beforehand  We implement the model with the default configuration 
For social media text corpus (FSD and Twitter), a named entity tagger specifically built for Twitter is used to extract named entities including locations from tweets  A Twitter Part-of-Speech (POS) tagger BIBREF15 is used for POS tagging and only words tagged with nouns, verbs and adjectives are retained as keywords  For the Google dataset, we use the Stanford Named Entity Recognizer to identify the named entities (organization, location and person)  Due to the `date' information not being provided in the Google dataset, we further divide the non-location named entities into two categories (`person' and `organization') and employ a quadruple <organization, location, person, keyword> to denote an event in news articles  We also remove common stopwords and only keep the recognized named entities and the tokens which are verbs, nouns or adjectives 
Experimental Results
To evaluate the performance of the proposed approach, we use the evaluation metrics such as precision, recall and F-measure  Precision is defined as the proportion of the correctly identified events out of the model generated events  Recall is defined as the proportion of correctly identified true events  For calculating the precision of the 4-tuple, we use following criteria:
(1) Do the entity/organization, location, date/person and keyword that we have extracted refer to the same event?
(2) If the extracted representation contains keywords, are they informative enough to tell us what happened?
Table TABREF35 shows the event extraction results on the three datasets  The statistics are obtained with the default parameter setting that INLINEFORM0 is set to 5, number of hidden units INLINEFORM1 is set to 200, and INLINEFORM2 contains three fully-connected layers  The event number INLINEFORM3 for three datasets are set to 25, 25 and 35, respectively  The examples of extracted events are shown in Table  TABREF36  
It can be observed that K-means performs the worst over all three datasets  On the social media datasets, AEM outpoerforms both LEM and DPEMM by 6 5% and 1 7% respectively in F-measure on the FSD dataset, and 4 4% and 3 7% in F-measure on the Twitter dataset  We can also observe that apart from K-means, all the approaches perform worse on the Twitter dataset compared to FSD, possibly due to the limited size of the Twitter dataset  Moreover, on the Google dataset, the proposed AEM performs significantly better than LEM and DPEMM  It improves upon LEM by 15 5% and upon DPEMM by more than 30% in F-measure  This is because: (1) the assumption made by LEM and DPEMM that all words in a document are generated from a single event is not suitable for long text such as news articles; (2) DPEMM generates too many irrelevant events which leads to a very low precision score  Overall, we see the superior performance of AEM across all datasets, with more significant improvement on the for Google datasets (long text) 
We next visualize the detected events based on the discriminative features learned by the trained INLINEFORM0 network in AEM  The t-SNE BIBREF16 visualization results on the datasets are shown in Figure FIGREF19   For clarity, each subplot is plotted on a subset of the dataset containing ten randomly selected events  It can be observed that documents describing the same event have been grouped into the same cluster 
To further evaluate if a variation of the parameters INLINEFORM0 (the number of discriminator iterations per generator iteration), INLINEFORM1 (the number of units in hidden layer) and the structure of generator INLINEFORM2 will impact the extraction performance, additional experiments have been conducted on the Google dataset, with INLINEFORM3 set to 5, 7 and 10, INLINEFORM4 set to 100, 150 and 200, and three INLINEFORM5 structures (3, 4 and 5 layers)  The comparison results on precision, recall and F-measure are shown in Figure FIGREF20   From the results, it could be observed that AEM with the 5-layer generator performs the best and achieves 96 7% in F-measure, and the worst F-measure obtained by AEM is 85 7%  Overall, the AEM outperforms all compared approaches acorss various parameter settings, showing relatively stable performance 
Finally, we compare in Figure FIGREF37 the training time required for each model, excluding the constant time required by each model to load the data  We could observe that K-means runs fastest among all four approaches  Both LEM and DPEMM need to sample the event allocation for each document and update the relevant counts during Gibbs sampling which are time consuming  AEM only requires a fraction of the training time compared to LEM and DPEMM  Moreover, on a larger dataset such as the Google dataset, AEM appears to be far more efficient compared to LEM and DPEMM 
Conclusions and Future Work
In this paper, we have proposed a novel approach based on adversarial training to extract the structured representation of events from online text  The experimental comparison with the state-of-the-art methods shows that AEM achieves improved extraction performance, especially on long text corpora with an improvement of 15% observed in F-measure  AEM only requires a fraction of training time compared to existing Bayesian graphical modeling approaches  In future work, we will explore incorporating external knowledge (e g  word relatedness contained in word embeddings) into the learning framework for event extraction  Besides, exploring nonparametric neural event extraction approaches and detecting the evolution of events over time from news articles are other promising future directions 
Acknowledgments
We would like to thank anonymous reviewers for their valuable comments and helpful suggestions  This work was funded by the National Key Research and Development Program of China (2016YFC1306704), the National Natural Science Foundation of China (61772132), the Natural Science Foundation of Jiangsu Province of China (BK20161430) 

ground_truth: For SLC task, the "ltuorp" team  has the best performing  model (0.6323/0.6028/0.6649 for F1/P/R  respectively) and for FLC task the "newspeak" team  has the best performing  model (0.2488/0.2863/0.2201 for F1/P/R respectively).

prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. Do they report results only on English data?
Context 4:  We observed a speedup of 3 2x on IMDb train time per epoch compared to the optimized LSTM implementation provided in NVIDIA's cuDNN library  For specific batch sizes and sequence lengths, a 16x speed gain is possible  Figure FIGREF15 provides extensive speed comparisons 
In Figure FIGREF12 , we visualize the hidden state vectors INLINEFORM0 of the final QRNN layer on part of an example from the IMDb dataset  Even without any post-processing, changes in the hidden state are visible and interpretable in regards to the input  This is a consequence of the elementwise nature of the recurrent pooling function, which delays direct interaction between different channels of the hidden state until the computation of the next QRNN layer 
Language Modeling
We replicate the language modeling experiment of BIBREF2 and BIBREF13 to benchmark the QRNN architecture for natural language sequence prediction  The experiment uses a standard preprocessed version of the Penn Treebank (PTB) by BIBREF25  
We implemented a gated QRNN model with medium hidden size: 2 layers with 640 units in each layer  Both QRNN layers use a convolutional filter width INLINEFORM0 of two timesteps  While the “medium” models used in other work BIBREF2 , BIBREF13 consist of 650 units in each layer, it was more computationally convenient to use a multiple of 32  As the Penn Treebank is a relatively small dataset, preventing overfitting is of considerable importance and a major focus of recent research  It is not obvious in advance which of the many RNN regularization schemes would perform well when applied to the QRNN  Our tests showed encouraging results from zoneout applied to the QRNN's recurrent pooling layer, implemented as described in Section SECREF5  
The experimental settings largely followed the “medium” setup of BIBREF2   Optimization was performed by stochastic gradient descent (SGD) without momentum  The learning rate was set at 1 for six epochs, then decayed by 0 95 for each subsequent epoch, for a total of 72 epochs  We additionally used INLINEFORM0 regularization of INLINEFORM1 and rescaled gradients with norm above 10  Zoneout was applied by performing dropout with ratio 0 1 on the forget gates of the QRNN, without rescaling the output of the dropout function  Batches consist of 20 examples, each 105 timesteps 
Comparing our results on the gated QRNN with zoneout to the results of LSTMs with both ordinary and variational dropout in Table TABREF14 , we see that the QRNN is highly competitive  The QRNN without zoneout strongly outperforms both our medium LSTM and the medium LSTM of BIBREF2 which do not use recurrent dropout and is even competitive with variational LSTMs  This may be due to the limited computational capacity that the QRNN's pooling layer has relative to the LSTM's recurrent weights, providing structural regularization over the recurrence 
Without zoneout, early stopping based upon validation loss was required as the QRNN would begin overfitting  By applying a small amount of zoneout ( INLINEFORM0 ), no early stopping is required and the QRNN achieves competitive levels of perplexity to the variational LSTM of BIBREF13 , which had variational inference based dropout of 0 2 applied recurrently  Their best performing variation also used Monte Carlo (MC) dropout averaging at test time of 1000 different masks, making it computationally more expensive to run 
When training on the PTB dataset with an NVIDIA K40 GPU, we found that the QRNN is substantially faster than a standard LSTM, even when comparing against the optimized cuDNN LSTM  In Figure FIGREF15 we provide a breakdown of the time taken for Chainer's default LSTM, the cuDNN LSTM, and QRNN to perform a full forward and backward pass on a single batch during training of the RNN LM on PTB  For both LSTM implementations, running time was dominated by the RNN computations, even with the highly optimized cuDNN implementation  For the QRNN implementation, however, the “RNN” layers are no longer the bottleneck  Indeed, there are diminishing returns from further optimization of the QRNN itself as the softmax and optimization overhead take equal or greater time  Note that the softmax, over a vocabulary size of only 10,000 words, is relatively small; for tasks with larger vocabularies, the softmax would likely dominate computation time 
It is also important to note that the cuDNN library's RNN primitives do not natively support any form of recurrent dropout  That is, running an LSTM that uses a state-of-the-art regularization scheme at cuDNN-like speeds would likely require an entirely custom kernel 
Character-level Neural Machine Translation
We evaluate the sequence-to-sequence QRNN architecture described in SECREF5 on a challenging neural machine translation task, IWSLT German–English spoken-domain translation, applying fully character-level segmentation  This dataset consists of 209,772 sentence pairs of parallel training data from transcribed TED and TEDx presentations, with a mean sentence length of 103 characters for German and 93 for English  We remove training sentences with more than 300 characters in English or German, and use a unified vocabulary of 187 Unicode code points 
Our best performance on a development set (TED tst2013) was achieved using a four-layer encoder–decoder QRNN with 320 units per layer, no dropout or INLINEFORM0 regularization, and gradient rescaling to a maximum magnitude of 5  Inputs were supplied to the encoder reversed, while the encoder convolutions were not masked  The first encoder layer used convolutional filter width INLINEFORM1 , while the other encoder layers used INLINEFORM2   Optimization was performed for 10 epochs on minibatches of 16 examples using Adam BIBREF28 with INLINEFORM3 , INLINEFORM4 , INLINEFORM5 , and INLINEFORM6   Decoding was performed using beam search with beam width 8 and length normalization INLINEFORM7   The modified log-probability ranking criterion is provided in the appendix 
Results using this architecture were compared to an equal-sized four-layer encoder–decoder LSTM with attention, applying dropout of 0 2  We again optimized using Adam; other hyperparameters were equal to their values for the QRNN and the same beam search procedure was applied  Table TABREF17 shows that the QRNN outperformed the character-level LSTM, almost matching the performance of a word-level attentional baseline 
Related Work
Exploring alternatives to traditional RNNs for sequence tasks is a major area of current research  Quasi-recurrent neural networks are related to several such recently described models, especially the strongly-typed recurrent neural networks (T-RNN) introduced by BIBREF12   While the motivation and constraints described in that work are different, BIBREF12 's concepts of “learnware” and “firmware” parallel our discussion of convolution-like and pooling-like subcomponents  As the use of a fully connected layer for recurrent connections violates the constraint of “strong typing”, all strongly-typed RNN architectures (including the T-RNN, T-GRU, and T-LSTM) are also quasi-recurrent  However, some QRNN models (including those with attention or skip-connections) are not “strongly typed”  In particular, a T-RNN differs from a QRNN as described in this paper with filter size 1 and f-pooling only in the absence of an activation function on INLINEFORM0   Similarly, T-GRUs and T-LSTMs differ from QRNNs with filter size 2 and fo- or ifo-pooling respectively in that they lack INLINEFORM1 on INLINEFORM2 and use INLINEFORM3 rather than sigmoid on INLINEFORM4  
The QRNN is also related to work in hybrid convolutional–recurrent models  BIBREF31 apply CNNs at the word level to generate INLINEFORM0 -gram features used by an LSTM for text classification  BIBREF32 also tackle text classification by applying convolutions at the character level, with a stride to reduce sequence length, then feeding these features into a bidirectional LSTM  A similar approach was taken by BIBREF10 for character-level machine translation  Their model's encoder uses a convolutional layer followed by max-pooling to reduce sequence length, a four-layer highway network, and a bidirectional GRU  The parallelism of the convolutional, pooling, and highway layers allows training speed comparable to subword-level models without hard-coded text segmentation 
The QRNN encoder–decoder model shares the favorable parallelism and path-length properties exhibited by the ByteNet BIBREF33 , an architecture for character-level machine translation based on residual convolutions over binary trees  Their model was constructed to achieve three desired properties: parallelism, linear-time computational complexity, and short paths between any pair of words in order to better propagate gradient signals 
Conclusion
Intuitively, many aspects of the semantics of long sequences are context-invariant and can be computed in parallel (e g , convolutionally), but some aspects require long-distance context and must be computed recurrently  Many existing neural network architectures either fail to take advantage of the contextual information or fail to take advantage of the parallelism  QRNNs exploit both parallelism and context, exhibiting advantages from both convolutional and recurrent neural networks  QRNNs have better predictive accuracy than LSTM-based models of equal hidden size, even though they use fewer parameters and run substantially faster  Our experiments show that the speed and accuracy advantages remain consistent across tasks and at both word and character levels 
Extensions to both CNNs and RNNs are often directly applicable to the QRNN, while the model's hidden states are more interpretable than those of other recurrent architectures as its channels maintain their independence across timesteps  We believe that QRNNs can serve as a building block for long-sequence tasks that were previously impractical with traditional RNNs 
Beam search ranking criterion
The modified log-probability ranking criterion we used in beam search for translation experiments is: DISPLAYFORM0
where INLINEFORM0 is a length normalization parameter BIBREF34 , INLINEFORM1 is the INLINEFORM2 th output character, and INLINEFORM3 is a “target length” equal to the source sentence length plus five characters  This reduces at INLINEFORM4 to ordinary beam search with probabilities: DISPLAYFORM0
and at INLINEFORM0 to beam search with probabilities normalized by length (up to the target length): DISPLAYFORM0
Conveniently, this ranking criterion can be computed at intermediate beam-search timesteps, obviating the need to apply a separate reranking on complete hypotheses 

ground_truth: Unanswerable

Unanswerable