prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. Do they evaluate only on English datasets?
Context 1: Introduction
Data annotation is a major bottleneck for the application of supervised learning approaches to many problems  As a result, unsupervised methods that learn directly from unlabeled data are increasingly important  For tasks related to unsupervised syntactic analysis, discrete generative models have dominated in recent years – for example, for both part-of-speech (POS) induction BIBREF0 , BIBREF1 and unsupervised dependency parsing BIBREF2 , BIBREF3 , BIBREF4   While similar models have had success on a range of unsupervised tasks, they have mostly ignored the apparent utility of continuous word representations evident from supervised NLP applications BIBREF5 , BIBREF6   In this work, we focus on leveraging and explicitly representing continuous word embeddings within unsupervised models of syntactic structure 
Pre-trained word embeddings from massive unlabeled corpora offer a compact way of injecting a prior notion of word similarity into models that would otherwise treat words as discrete, isolated categories  However, the specific properties of language captured by any particular embedding scheme can be difficult to control, and, further, may not be ideally suited to the task at hand  For example, pre-trained skip-gram embeddings BIBREF7 with small context window size are found to capture the syntactic properties of language well BIBREF8 , BIBREF9   However, if our goal is to separate syntactic categories, this embedding space is not ideal – POS categories correspond to overlapping interspersed regions in the embedding space, evident in Figure SECREF4  
In our approach, we propose to learn a new latent embedding space as a projection of pre-trained embeddings (depicted in Figure SECREF5 ), while jointly learning latent syntactic structure – for example, POS categories or syntactic dependencies  To this end, we introduce a new generative model (shown in Figure FIGREF6 ) that first generates a latent syntactic representation (e g  a dependency parse) from a discrete structured prior (which we also call the “syntax model”), then, conditioned on this representation, generates a sequence of latent embedding random variables corresponding to each word, and finally produces the observed (pre-trained) word embeddings by projecting these latent vectors through a parameterized non-linear function  The latent embeddings can be jointly learned with the structured syntax model in a completely unsupervised fashion 
By choosing an invertible neural network as our non-linear projector, and then parameterizing our model in terms of the projection's inverse, we are able to derive tractable exact inference and marginal likelihood computation procedures so long as inference is tractable in the underlying syntax model  In sec:learn-with-inv we show that this derivation corresponds to an alternate view of our approach whereby we jointly learn a mapping of observed word embeddings to a new embedding space that is more suitable for the syntax model, but include an additional Jacobian regularization term to prevent information loss 
Recent work has sought to take advantage of word embeddings in unsupervised generative models with alternate approaches BIBREF9 , BIBREF10 , BIBREF11 , BIBREF12   BIBREF9 build an HMM with Gaussian emissions on observed word embeddings, but they do not attempt to learn new embeddings  BIBREF10 , BIBREF11 , and BIBREF12 extend HMM or dependency model with valence (DMV) BIBREF2 with multinomials that use word (or tag) embeddings in their parameterization  However, they do not represent the embeddings as latent variables 
In experiments, we instantiate our approach using both a Markov-structured syntax model and a tree-structured syntax model – specifically, the DMV  We evaluate on two tasks: part-of-speech (POS) induction and unsupervised dependency parsing without gold POS tags  Experimental results on the Penn Treebank BIBREF13 demonstrate that our approach improves the basic HMM and DMV by a large margin, leading to the state-of-the-art results on POS induction, and state-of-the-art results on unsupervised dependency parsing in the difficult training scenario where neither gold POS annotation nor punctuation-based constraints are available 
Model
As an illustrative example, we first present a baseline model for Markov syntactic structure (POS induction) that treats a sequence of pre-trained word embeddings as observations  Then, we propose our novel approach, again using Markov structure, that introduces latent word embedding variables and a neural projector  Lastly, we extend our approach to more general syntactic structures 
Example: Gaussian HMM
We start by describing the Gaussian hidden Markov model introduced by BIBREF9 , which is a locally normalized model with multinomial transitions and Gaussian emissions  Given a sentence of length INLINEFORM0 , we denote the latent POS tags as INLINEFORM1 , observed (pre-trained) word embeddings as INLINEFORM2 , transition parameters as INLINEFORM3 , and Gaussian emission parameters as INLINEFORM4   The joint distribution of data and latent variables factors as:
DISPLAYFORM0
where INLINEFORM0 is the multinomial transition probability and INLINEFORM1 is the multivariate Gaussian emission probability 
While the observed word embeddings do inform this model with a notion of word similarity – lacking in the basic multinomial HMM – the Gaussian emissions may not be sufficiently flexible to separate some syntactic categories in the complex pre-trained embedding space – for example the skip-gram embedding space as visualized in Figure SECREF4 where different POS categories overlap  Next we introduce a new approach that adds flexibility to the emission distribution by incorporating new latent embedding variables 
Markov Structure with Neural Projector
To flexibly model observed embeddings and yield a new representation space that is more suitable for the syntax model, we propose to cascade a neural network as a projection function, deterministically transforming the simple space defined by the Gaussian HMM to the observed embedding space  We denote the latent embedding of the INLINEFORM0 word in a sentence as INLINEFORM1 , and the neural projection function as INLINEFORM2 , parameterized by INLINEFORM3   In the case of sequential Markov structure, our new model corresponds to the following generative process:
For each time step INLINEFORM0 ,
[noitemsep, leftmargin=*]
Draw the latent state INLINEFORM0
Draw the latent embedding INLINEFORM0
Deterministically produce embedding
INLINEFORM0
The graphical model is depicted in Figure FIGREF6   The deterministic projection can also be viewed as sampling each observation from a point mass at INLINEFORM0   The joint distribution of our model is: DISPLAYFORM0
where INLINEFORM0 is a conditional Gaussian distribution, and INLINEFORM1 is the Dirac delta function centered at INLINEFORM2 : DISPLAYFORM0
General Structure with Neural Projector
Our approach can be applied to a broad family of structured syntax models  We denote latent embedding variables as INLINEFORM0 , discrete latent variables in the syntax model as INLINEFORM1 ( INLINEFORM2 ), where INLINEFORM3 are conditioned to generate INLINEFORM4   The joint probability of our model factors as:
DISPLAYFORM0
where INLINEFORM0 represents the probability of the syntax model, and can encode any syntactic structure – though, its factorization structure will determine whether inference is tractable in our full model  As shown in Figure FIGREF6 , we focus on two syntax models for syntactic analysis in this paper  The first is Markov-structured, which we use for POS induction, and the second is DMV-structured, which we use to learn dependency parses without supervision 
The marginal data likelihood of our model is: DISPLAYFORM0
While the discrete variables INLINEFORM0 can be marginalized out with dynamic program in many cases, it is generally intractable to marginalize out the latent continuous variables, INLINEFORM1 , for an arbitrary projection INLINEFORM2 in Eq  ( EQREF17 ), which means inference and learning may be difficult  In sec:opt, we address this issue by constraining INLINEFORM3 to be invertible, and show that this constraint enables tractable exact inference and marginal likelihood computation 
Learning & Inference
In this section, we introduce an invertibility condition for our neural projector to tackle the optimization challenge  Specifically, we constrain our neural projector with two requirements: (1) INLINEFORM0 and (2) INLINEFORM1 exists  Invertible transformations have been explored before in independent components analysis BIBREF14 , gaussianization BIBREF15 , and deep density models BIBREF16 , BIBREF17 , BIBREF18 , for unstructured data  Here, we generalize this style of approach to structured learning, and augment it with discrete latent variables ( INLINEFORM2 )  Under the invertibility condition, we derive a learning algorithm and give another view of our approach revealed by the objective function  Then, we present the architecture of a neural projector we use in experiments: a volume-preserving invertible neural network proposed by BIBREF16 for independent components estimation 
Learning with Invertibility
For ease of exposition, we explain the learning algorithm in terms of Markov structure without loss of generality  As shown in Eq  ( EQREF17 ), the optimization challenge in our approach comes from the intractability of the marginalized emission factor INLINEFORM0   If we can marginalize out INLINEFORM1 and compute INLINEFORM2 , then the posterior and marginal likelihood of our Markov-structured model can be computed with the forward-backward algorithm  We can apply Eq  ( EQREF14 ) and obtain : INLINEFORM3
By using the change of variable rule to the integration, which allows the integration variable INLINEFORM0 to be replaced by INLINEFORM1 , the marginal emission factor can be computed in closed-form when the invertibility condition is satisfied: DISPLAYFORM0
where INLINEFORM0 is a conditional Gaussian distribution, INLINEFORM1 is the Jacobian matrix of function INLINEFORM2 at INLINEFORM3 , and INLINEFORM4 represents the absolute value of its determinant  This Jacobian term is nonzero and differentiable if and only if INLINEFORM5 exists 
Eq  ( EQREF19 ) shows that we can directly calculate the marginal emission distribution INLINEFORM0   Denote the marginal data likelihood of Gaussian HMM as INLINEFORM1 , then the log marginal data likelihood of our model can be directly written as: DISPLAYFORM0
where INLINEFORM0 represents the new sequence of embeddings after applying INLINEFORM1 to each INLINEFORM2   Eq  ( EQREF20 ) shows that the training objective of our model is simply the Gaussian HMM log likelihood with an additional Jacobian regularization term  From this view, our approach can be seen as equivalent to reversely projecting the data through INLINEFORM3 to another manifold INLINEFORM4 that is directly modeled by the Gaussian HMM, with a regularization term  Intuitively, we optimize the reverse projection INLINEFORM5 to modify the INLINEFORM6 space, making it more appropriate for the syntax model  The Jacobian regularization term accounts for the volume expansion or contraction behavior of the projection  Maximizing it can be thought of as preventing information loss  In the extreme case, the Jacobian determinant is equal to zero, which means the projection is non-invertible and thus information is being lost through the projection  Such “information preserving” regularization is crucial during optimization, otherwise the trivial solution of always projecting data to the same single point to maximize likelihood is viable 
More generally, for an arbitrary syntax model the data likelihood of our approach is: DISPLAYFORM0
If the syntax model itself allows for tractable inference and marginal likelihood computation, the same dynamic program can be used to marginalize out INLINEFORM0   Therefore, our joint model inherits the tractability of the underlying syntax model 
Invertible Volume-Preserving Neural Net
For the projection we can use an arbitrary invertible function, and given the representational power of neural networks they seem a natural choice  However, calculating the inverse and Jacobian of an arbitrary neural network can be difficult, as it requires that all component functions be invertible and also requires storage of large Jacobian matrices, which is memory intensive  To address this issue, several recent papers propose specially designed invertible networks that are easily trainable yet still powerful BIBREF16 , BIBREF17 , BIBREF19   Inspired by these works, we use the invertible transformation proposed by BIBREF16 , which consists of a series of “coupling layers”  This architecture is specially designed to guarantee a unit Jacobian determinant (and thus the invertibility property) 
From Eq  ( EQREF22 ) we know that only INLINEFORM0 is required for accomplishing learning and inference; we never need to explicitly construct INLINEFORM1   Thus, we directly define the architecture of INLINEFORM2   As shown in Figure FIGREF24 , the nonlinear transformation from the observed embedding INLINEFORM3 to INLINEFORM4 represents the first coupling layer  The input in this layer is partitioned into left and right halves of dimensions, INLINEFORM5 and INLINEFORM6 , respectively  A single coupling layer is defined as: DISPLAYFORM0
where INLINEFORM0 is the coupling function and can be any nonlinear form  This transformation satisfies INLINEFORM1 , and BIBREF16 show that its Jacobian matrix is triangular with all ones on the main diagonal  Thus the Jacobian determinant is always equal to one (i e  volume-preserving) and the invertibility condition is naturally satisfied 
To be sufficiently expressive, we compose multiple coupling layers as suggested in BIBREF16   Specifically, we exchange the role of left and right half vectors at each layer as shown in Figure FIGREF24   For instance, from INLINEFORM0 to INLINEFORM1 the left subset INLINEFORM2 is unchanged, while from INLINEFORM3 to INLINEFORM4 the right subset INLINEFORM5 remains the same  Also note that composing multiple coupling layers does not change the volume-preserving and invertibility properties  Such a sequence of invertible transformations from the data space INLINEFORM6 to INLINEFORM7 is also called normalizing flow BIBREF20  
Experiments
In this section, we first describe our datasets and experimental setup  We then instantiate our approach with Markov and DMV-structured syntax models, and report results on POS tagging and dependency grammar induction respectively  Lastly, we analyze the learned latent embeddings 
Data
For both POS tagging and dependency parsing, we run experiments on the Wall Street Journal (WSJ) portion of the Penn Treebank  To create the observed data embeddings, we train skip-gram word embeddings BIBREF7 that are found to capture syntactic properties well when trained with small context window BIBREF8 , BIBREF9   Following BIBREF9 , the dimensionality INLINEFORM0 is set to 100, and the training context window size is set to 1 to encode more syntactic information  The skip-gram embeddings are trained on the one billion word language modeling benchmark dataset BIBREF21 in addition to the WSJ corpus 
General Experimental Setup
For the neural projector, we employ rectified networks as coupling function INLINEFORM0 following BIBREF16   We use a rectified network with an input layer, one hidden layer, and linear output units, the number of hidden units is set to the same as the number of input units  The number of coupling layers are varied as 4, 8, 16 for both tasks  We optimize marginal data likelihood directly using Adam BIBREF22   For both tasks in the fully unsupervised setting, we do not tune the hyper-parameters using supervised data 
Unsupervised POS tagging
For unsupervised POS tagging, we use a Markov-structured syntax model in our approach, which is a popular structure for unsupervised tagging tasks BIBREF9 , BIBREF10  
Following existing literature, we train and test on the entire WSJ corpus (49208 sentences, 1M tokens)  We use 45 tag clusters, the number of POS tags that appear in WSJ corpus  We train the discrete HMM and the Gaussian HMM BIBREF9 as baselines  For the Gaussian HMM, mean vectors of Gaussian emissions are initialized with the empirical mean of all word vectors with an additive noise  We assume diagonal covariance matrix for INLINEFORM0 and initialize it with the empirical variance of the word vectors  Following BIBREF9 , the covariance matrix is fixed during training  The multinomial probabilities are initialized as INLINEFORM1 , where INLINEFORM2   For our approach, we initialize the syntax model and Gaussian parameters with the pre-trained Gaussian HMM  The weights of layers in the rectified network are initialized from a uniform distribution with mean zero and a standard deviation of INLINEFORM3 , where INLINEFORM4 is the input dimension  We evaluate the performance of POS tagging with both Many-to-One (M-1) accuracy BIBREF23 and V-Measure (VM) BIBREF24   Given a model we found that the tagging performance is well-correlated with the training data likelihood, thus we use training data likelihood as a unsupervised criterion to select the trained model over 10 random restarts after training 50 epochs  We repeat this process 5 times and report the mean and standard deviation of performance 
We compare our approach with basic HMM, Gaussian HMM, and several state-of-the-art systems, including sophisticated HMM variants and clustering techniques with hand-engineered features  The results are presented in Table TABREF32   Through the introduced latent embeddings and additional neural projection, our approach improves over the Gaussian HMM by 5 4 points in M-1 and 5 6 points in VM  Neural HMM (NHMM) BIBREF10 is a baseline that also learns word representation jointly  Both their basic model and extended Conv version does not outperform the Gaussian HMM  Their best model incorporates another LSTM to model long distance dependency and breaks the Markov assumption, yet our approach still achieves substantial improvement over it without considering more context information  Moreover, our method outperforms the best published result that benefits from hand-engineered features BIBREF27 by 2 0 points on VM 
We found that most tagging errors happen in noun subcategories  Therefore, we do the one-to-one mapping between gold POS tags and induced clusters and plot the normalized confusion matrix of noun subcategories in Figure FIGREF35   The Gaussian HMM fails to identify “NN” and “NNS” correctly for most cases, and it often recognizes “NNPS” as “NNP”  In contrast, our approach corrects these errors well 
Unsupervised Dependency Parsing without gold POS tags
For the task of unsupervised dependency parse induction, we employ the Dependency Model with Valence (DMV) BIBREF2 as the syntax model in our approach  DMV is a generative model that defines a probability distribution over dependency parse trees and syntactic categories, generating tokens and dependencies in a head-outward fashion  While, traditionally, DMV is trained using gold POS tags as observed syntactic categories, in our approach, we treat each tag as a latent variable, as described in sec:general-neural 
Most existing approaches to this task are not fully unsupervised since they rely on gold POS tags following the original experimental setup for DMV  This is partially because automatically parsing from words is difficult even when using unsupervised syntactic categories BIBREF29   However, inducing dependencies from words alone represents a more realistic experimental condition since gold POS tags are often unavailable in practice  Previous work that has trained from words alone often requires additional linguistic constraints (like sentence internal boundaries) BIBREF29 , BIBREF30 , BIBREF31 , BIBREF32 , acoustic cues BIBREF33 , additional training data BIBREF4 , or annotated data from related languages BIBREF34   Our approach is naturally designed to train on word embeddings directly, thus we attempt to induce dependencies without using gold POS tags or other extra linguistic information 
Like previous work we use sections 02-21 of WSJ corpus as training data and evaluate on section 23, we remove punctuations and train the models on sentences of length INLINEFORM0 , “head-percolation” rules BIBREF39 are applied to obtain gold dependencies for evaluation  We train basic DMV, extended DMV (E-DMV) BIBREF35 and Gaussian DMV (which treats POS tag as unknown latent variables and generates observed word embeddings directly conditioned on them following Gaussian distribution) as baselines  Basic DMV and E-DMV are trained with Viterbi EM BIBREF40 on unsupervised POS tags induced from our Markov-structured model described in sec:pos  Multinomial parameters of the syntax model in both Gaussian DMV and our model are initialized with the pre-trained DMV baseline  Other parameters are initialized in the same way as in the POS tagging experiment  The directed dependency accuracy (DDA) is used for evaluation and we report accuracy on sentences of length INLINEFORM1 and all lengths  We train the parser until training data likelihood converges, and report the mean and standard deviation over 20 random restarts 
Our model directly observes word embeddings and does not require gold POS tags during training  Thus, results from related work trained on gold tags are not directly comparable  However, to measure how these systems might perform without gold tags, we run three recent state-of-the-art systems in our experimental setting: UR-A E-DMV BIBREF36 , Neural E-DMV BIBREF11 , and CRF Autoencoder (CRFAE) BIBREF37   We use unsupervised POS tags (induced from our Markov-structured model) in place of gold tags  We also train basic DMV on gold tags and include several state-of-the-art results on gold tags as reference points 
As shown in Table TABREF39 , our approach is able to improve over the Gaussian DMV by 4 8 points on length INLINEFORM0 and 4 8 points on all lengths, which suggests the additional latent embedding layer and neural projector are helpful  The proposed approach yields, to the best of our knowledge, state-of-the-art performance without gold POS annotation and without sentence-internal boundary information  DMV, UR-A E-DMV, Neural E-DMV, and CRFAE suffer a large decrease in performance when trained on unsupervised tags – an effect also seen in previous work BIBREF29 , BIBREF34   Since our approach induces latent POS tags jointly with dependency trees, it may be able to learn POS clusters that are more amenable to grammar induction than the unsupervised tags  We observe that CRFAE underperforms its gold-tag counterpart substantially  This may largely be a result of the model's reliance on prior linguistic rules that become unavailable when gold POS tag types are unknown  Many extensions to DMV can be considered orthogonal to our approach – they essentially focus on improving the syntax model  It is possible that incorporating these more sophisticated syntax models into our approach may lead to further improvements 
Sensitivity Analysis
In the above experiments we initialize the structured syntax components with the pre-trained Gaussian or discrete baseline, which is shown as a useful technique to help train our deep models  We further study the results with fully random initialization  In the POS tagging experiment, we report the results in Table TABREF48   While the performance with 4 layers is comparable to the pre-trained Gaussian initialization, deeper projections (8 or 16 layers) result in a dramatic drop in performance  This suggests that the structured syntax model with very deep projections is difficult to train from scratch, and a simpler projection might be a good compromise in the random initialization setting 
Different from the Markov prior in POS tagging experiments, our parsing model seems to be quite sensitive to the initialization  For example, directed accuracy of our approach on sentences of length INLINEFORM0 is below 40 0 with random initialization  This is consistent with previous work that has noted the importance of careful initialization for DMV-based models such as the commonly used harmonic initializer BIBREF2   However, it is not straightforward to apply the harmonic initializer for DMV directly in our model without using some kind of pre-training since we do not observe gold POS 
We investigate the effect of the choice of pre-trained embedding on performance while using our approach  To this end, we additionally include results using fastText embeddings BIBREF41 – which, in contrast with skip-gram embeddings, include character-level information  We set the context windows size to 1 and the dimension size to 100 as in the skip-gram training, while keeping other parameters set to their defaults  These results are summarized in Table TABREF50 and Table TABREF51   While fastText embeddings lead to reduced performance with our model, our approach still yields an improvement over the Gaussian baseline with the new observed embeddings space 
Qualitative Analysis of Embeddings
We perform qualitative analysis to understand how the latent embeddings help induce syntactic structures  First we filter out low-frequency words and punctuations in WSJ, and visualize the rest words (10k) with t-SNE BIBREF42 under different embeddings  We assign each word with its most likely gold POS tags in WSJ and color them according to the gold POS tags 
For our Markov-structured model, we have displayed the embedding space in Figure SECREF5 , where the gold POS clusters are well-formed  Further, we present five example target words and their five nearest neighbors in terms of cosine similarity  As shown in Table TABREF53 , the skip-gram embedding captures both semantic and syntactic aspects to some degree, yet our embeddings are able to focus especially on the syntactic aspects of words, in an unsupervised fashion without using any extra morphological information 
In Figure FIGREF54 we depict the learned latent embeddings with the DMV-structured syntax model  Unlike the Markov structure, the DMV structure maps a large subset of singular and plural nouns to the same overlapping region  However, two clusters of singular and plural nouns are actually separated  We inspect the two clusters and the overlapping region in Figure FIGREF54 , it turns out that the nouns in the separated clusters are words that can appear as subjects and, therefore, for which verb agreement is important to model  In contrast, the nouns in the overlapping region are typically objects  This demonstrates that the latent embeddings are focusing on aspects of language that are specifically important for modeling dependency without ever having seen examples of dependency parses  Some previous work has deliberately created embeddings to capture different notions of similarity BIBREF43 , BIBREF44 , while they use extra morphology or dependency annotations to guide the embedding learning, our approach provides a potential alternative to create new embeddings that are guided by structured syntax model, only using unlabeled text corpora 
Related Work
Our approach is related to flow-based generative models, which are first described in NICE BIBREF16 and have recently received more attention BIBREF17 , BIBREF19 , BIBREF18   This relevant work mostly adopts simple (e g  Gaussian) and fixed priors and does not attempt to learn interpretable latent structures  Another related generative model class is variational auto-encoders (VAEs) BIBREF45 that optimize a lower bound on the marginal data likelihood, and can be extended to learn latent structures BIBREF46 , BIBREF47   Against the flow-based models, VAEs remove the invertibility constraint but sacrifice the merits of exact inference and exact log likelihood computation, which potentially results in optimization challenges BIBREF48   Our approach can also be viewed in connection with generative adversarial networks (GANs) BIBREF49 that is a likelihood-free framework to learn implicit generative models  However, it is non-trivial for a gradient-based method like GANs to propagate gradients through discrete structures 
Conclusion
In this work, we define a novel generative approach to leverage continuous word representations for unsupervised learning of syntactic structure  Experiments on both POS induction and unsupervised dependency parsing tasks demonstrate the effectiveness of our proposed approach  Future work might explore more sophisticated invertible projections, or recurrent projections that jointly transform the entire input sequence 
Context 3: Introduction
Modelling the relationship between sequences is extremely significant in most retrieval or classification problems involving two sequences  Traditionally, in Siamese networks, Hadamard product or concatenation have been used to fuse two vector representations of two input sequences to form a final representation for tasks like semantic similarity, passage retrieval  This representation, subsequently, has been used to compute similarity scores which has been used in a variety of training objectives like margin loss for ranking or cross-entropy error in classification 
We have also witnessed word or phrase level similarity to create alignment matrices between two sequences BIBREF0 , BIBREF1   These alignment matrices has proved to be very useful to model the relationship between two word representations as well fuse the relevant information of one sequence into another  Empirical evidences have shown this alignment procedures have significantly performed better then simple concatenation or element-wise multiplication, especially for long sentences or paragraphs 
Attention works on creating neural alignment matrix using learnt weights without pre-computing alignment matrix and using them as features  The main objective of any attentive or alignment process is to look for matching words or phrases between two sequences and assign a high weight to the most similar pairs and vice-versa  The notion of matching or similarity maybe not semantic similarity but based on whatever task we have at hand  For example, for a task that requires capturing semantic similarity between two sequences like "how rich is tom cruise" and "how much wealth does tom cruise have", an attentive model shall discover the high similarity between "rich" and "wealthy" and assign a high weight value to the pair  Likewise, for a different task like question answering, a word "long" in a question like "how long does it take to recover from a mild fever" might be aligned with the phrase "a week" from the candidate answer "it takes almost a week to recover fully from a fever"  Thus, attention significantly aids in better understanding the relevance of a similar user query in a similar measurement task or a candidate answer in a question answering task  The final prediction score is dependent on how well the relationship between two sequences are modeled and established 
The general process of matching one sequence with another through attention includes computing the alignment matrix containing weight value between every pair of word representations belonging to both of the sequences  Subsequently, softmax function is applied on all the elements of one of the two dimensions of the matrix to represent the matching probabilities of all the word of a sequence with respect to one particular word in the other sequence 
Since attention always looks for matching word representations, it operates under the assumption that there is always a match to be found inside the sequences  We provide a theoretical limitation to it and propose another technique called conflict that looks for contrasting relationship between words in two sequences  We empirically verify that our proposed conflict mechanism combined with attention can outperform the performance of attention working solely 
Related Work
Bahdanau et al  BIBREF2 introduced attention first in neural machine translation  It used a feed-forward network over addition of encoder and decoder states to compute alignment score  Our work is very similar to this except we use element wise difference instead of addition to build our conflict function  BIBREF3 came up with a scaled dot-product attention in their Transformer model which is fast and memory-efficient  Due to the scaling factor, it didn't have the issue of gradients zeroing out  On the other hand, BIBREF4 has experimented with global and local attention based on the how many hidden states the attention function takes into account  Their experiments have revolved around three attention functions - dot, concat and general  Their findings include that dot product works best for global attention  Our work also belongs to the global attention family as we consider all the hidden states of the sequence 
Attention has been widely used in pair-classification problems like natural language inference  Wang et al  BIBREF5 introduced BIMPM which matched one sequence with another in four different fashion but one single matching function which they used as cosine  Liu et al  BIBREF6 proposed SAN for language inference which also used dot-product attention between the sequences 
Summarizing, attention has helped in achieving state-of-the-art results in NLI and QA  Prior work in attention has been mostly in similarity based approaches while our work focuses on non-matching sequences 
How attention works
Let us consider that we have two sequences INLINEFORM0 and INLINEFORM1 each with M and N words respectively  The objective of attention is two-fold: compute alignment scores (or weight) between every word representation pairs from INLINEFORM2 and INLINEFORM3 and fuse the matching information of INLINEFORM4 with INLINEFORM5 thus computing a new representation of INLINEFORM6 conditioned on INLINEFORM7  
The word representations that attention operates on can be either embeddings like GloVe or hidden states from any recurrent neural network  We denote these representations as u = INLINEFORM0 and v = INLINEFORM1   We provide a mathematical working of how a general attention mechanism works between two sequences, followed by a explanation in words: DISPLAYFORM0
Explanation: Both are sequences are non-linearly projected into two different spaces (eqn 1) and each word representation in INLINEFORM0 is matched with that in INLINEFORM1 by computing a dot-product (eqn 2)  INLINEFORM2 is a M X N matrix that stores the alignment scores between word INLINEFORM3 and INLINEFORM4 (eqn 2)  Since, the scores are not normalized, a softmax function is applied on each row to convert them to probabilities (eqn  3)

ground_truth: Yes

Unanswerable

prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. Do they report results only on English data?
Context 2: Introduction
With the increasing popularity of the Internet, online texts provided by social media platform (e g  Twitter) and news media sites (e g  Google news) have become important sources of real-world events  Therefore, it is crucial to automatically extract events from online texts 
Due to the high variety of events discussed online and the difficulty in obtaining annotated data for training, traditional template-based or supervised learning approaches for event extraction are no longer applicable in dealing with online texts  Nevertheless, newsworthy events are often discussed by many tweets or online news articles  Therefore, the same event could be mentioned by a high volume of redundant tweets or news articles  This property inspires the research community to devise clustering-based models BIBREF0 , BIBREF1 , BIBREF2 to discover new or previously unidentified events without extracting structured representations 
To extract structured representations of events such as who did what, when, where and why, Bayesian approaches have made some progress  Assuming that each document is assigned to a single event, which is modeled as a joint distribution over the named entities, the date and the location of the event, and the event-related keywords, Zhou et al  zhou2014simple proposed an unsupervised Latent Event Model (LEM) for open-domain event extraction  To address the limitation that LEM requires the number of events to be pre-set, Zhou et al  zhou2017event further proposed the Dirichlet Process Event Mixture Model (DPEMM) in which the number of events can be learned automatically from data  However, both LEM and DPEMM have two limitations: (1) they assume that all words in a document are generated from a single event which can be represented by a quadruple INLINEFORM0 entity, location, keyword, date INLINEFORM1   However, long texts such news articles often describe multiple events which clearly violates this assumption; (2) During the inference process of both approaches, the Gibbs sampler needs to compute the conditional posterior distribution and assigns an event for each document  This is time consuming and takes long time to converge 
To deal with these limitations, in this paper, we propose the Adversarial-neural Event Model (AEM) based on adversarial training for open-domain event extraction  The principle idea is to use a generator network to learn the projection function between the document-event distribution and four event related word distributions (entity distribution, location distribution, keyword distribution and date distribution)  Instead of providing an analytic approximation, AEM uses a discriminator network to discriminate between the reconstructed documents from latent events and the original input documents  This essentially helps the generator to construct a more realistic document from a random noise drawn from a Dirichlet distribution  Due to the flexibility of neural networks, the generator is capable of learning complicated nonlinear distributions  And the supervision signal provided by the discriminator will help generator to capture the event-related patterns  Furthermore, the discriminator also provides low-dimensional discriminative features which can be used to visualize documents and events 
The main contributions of the paper are summarized below:
Related Work
Our work is related to two lines of research, event extraction and Generative Adversarial Nets 
Event Extraction
Recently there has been much interest in event extraction from online texts, and approaches could be categorized as domain-specific and open-domain event extraction 
Domain-specific event extraction often focuses on the specific types of events (e g  sports events or city events)  Panem et al  panem2014structured devised a novel algorithm to extract attribute-value pairs and mapped them to manually generated schemes for extracting the natural disaster events  Similarly, to extract the city-traffic related event, Anantharam et al  anantharam2015extracting viewed the task as a sequential tagging problem and proposed an approach based on the conditional random fields  Zhang zhang2018event proposed an event extraction approach based on imitation learning, especially on inverse reinforcement learning 
Open-domain event extraction aims to extract events without limiting the specific types of events  To analyze individual messages and induce a canonical value for each event, Benson et al  benson2011event proposed an approach based on a structured graphical model  By representing an event with a binary tuple which is constituted by a named entity and a date, Ritter et al  ritter2012open employed some statistic to measure the strength of associations between a named entity and a date  The proposed system relies on a supervised labeler trained on annotated data  In BIBREF1 , Abdelhaq et al  developed a real-time event extraction system called EvenTweet, and each event is represented as a triple constituted by time, location and keywords  To extract more information, Wang el al  wang2015seeft developed a system employing the links in tweets and combing tweets with linked articles to identify events  Xia el al  xia2015new combined texts with the location information to detect the events with low spatial and temporal deviations  Zhou et al  zhou2014simple,zhou2017event represented event as a quadruple and proposed two Bayesian models to extract events from tweets 
Generative Adversarial Nets
As a neural-based generative model, Generative Adversarial Nets BIBREF3 have been extensively researched in natural language processing (NLP) community 
For text generation, the sequence generative adversarial network (SeqGAN) proposed in BIBREF4 incorporated a policy gradient strategy to optimize the generation process  Based on the policy gradient, Lin et al  lin2017adversarial proposed RankGAN to capture the rich structures of language by ranking and analyzing a collection of human-written and machine-written sentences  To overcome mode collapse when dealing with discrete data, Fedus et al  fedus2018maskgan proposed MaskGAN which used an actor-critic conditional GAN to fill in missing text conditioned on the surrounding context  Along this line, Wang et al  wang2018sentigan proposed SentiGAN to generate texts of different sentiment labels  Besides, Li et al  li2018learning improved the performance of semi-supervised text classification using adversarial training, BIBREF5 , BIBREF6 designed GAN-based models for distance supervision relation extraction 
Although various GAN based approaches have been explored for many applications, none of these approaches tackles open-domain event extraction from online texts  We propose a novel GAN-based event extraction model called AEM  Compared with the previous models, AEM has the following differences: (1) Unlike most GAN-based text generation approaches, a generator network is employed in AEM to learn the projection function between an event distribution and the event-related word distributions (entity, location, keyword, date)  The learned generator captures event-related patterns rather than generating text sequence; (2) Different from LEM and DPEMM, AEM uses a generator network to capture the event-related patterns and is able to mine events from different text sources (short and long)  Moreover, unlike traditional inference procedure, such as Gibbs sampling used in LEM and DPEMM, AEM could extract the events more efficiently due to the CUDA acceleration; (3) The discriminative features learned by the discriminator of AEM provide a straightforward way to visualize the extracted events 
Methodology
We describe Adversarial-neural Event Model (AEM) in this section  An event is represented as a quadruple < INLINEFORM0 >, where INLINEFORM1 stands for non-location named entities, INLINEFORM2 for a location, INLINEFORM3 for event-related keywords, INLINEFORM4 for a date, and each component in the tuple is represented by component-specific representative words 
AEM is constituted by three components: (1) The document representation module, as shown at the top of Figure FIGREF4 , defines a document representation approach which converts an input document from the online text corpus into INLINEFORM0 which captures the key event elements; (2) The generator INLINEFORM1 , as shown in the lower-left part of Figure FIGREF4 , generates a fake document INLINEFORM2 which is constituted by four multinomial distributions using an event distribution INLINEFORM3 drawn from a Dirichlet distribution as input; (3) The discriminator INLINEFORM4 , as shown in the lower-right part of Figure FIGREF4 , distinguishes the real documents from the fake ones and its output is subsequently employed as a learning signal to update the INLINEFORM5 and INLINEFORM6   The details of each component are presented below 
Document Representation
Each document INLINEFORM0 in a given corpus INLINEFORM1 is represented as a concatenation of 4 multinomial distributions which are entity distribution ( INLINEFORM2 ), location distribution ( INLINEFORM3 ), keyword distribution ( INLINEFORM4 ) and date distribution ( INLINEFORM5 ) of the document  As four distributions are calculated in a similar way, we only describe the computation of the entity distribution below as an example 
The entity distribution INLINEFORM0 is represented by a normalized INLINEFORM1 -dimensional vector weighted by TF-IDF, and it is calculated as: INLINEFORM2
where INLINEFORM0 is the pseudo corpus constructed by removing all non-entity words from INLINEFORM1 , INLINEFORM2 is the total number of distinct entities in a corpus, INLINEFORM3 denotes the number of INLINEFORM4 -th entity appeared in document INLINEFORM5 , INLINEFORM6 represents the number of documents in the corpus, and INLINEFORM7 is the number of documents that contain INLINEFORM8 -th entity, and the obtained INLINEFORM9 denotes the relevance between INLINEFORM10 -th entity and document INLINEFORM11  
Similarly, location distribution INLINEFORM0 , keyword distribution INLINEFORM1 and date distribution INLINEFORM2 of INLINEFORM3 could be calculated in the same way, and the dimensions of these distributions are denoted as INLINEFORM4 , INLINEFORM5 and INLINEFORM6 , respectively  Finally, each document INLINEFORM7 in the corpus is represented by a INLINEFORM8 -dimensional ( INLINEFORM9 = INLINEFORM10 + INLINEFORM11 + INLINEFORM12 + INLINEFORM13 ) vector INLINEFORM14 by concatenating four computed distributions 
Network Architecture
The generator network INLINEFORM0 is designed to learn the projection function between the document-event distribution INLINEFORM1 and the four document-level word distributions (entity distribution, location distribution, keyword distribution and date distribution) 
More concretely, INLINEFORM0 consists of a INLINEFORM1 -dimensional document-event distribution layer, INLINEFORM2 -dimensional hidden layer and INLINEFORM3 -dimensional event-related word distribution layer  Here, INLINEFORM4 denotes the event number, INLINEFORM5 is the number of units in the hidden layer, INLINEFORM6 is the vocabulary size and equals to INLINEFORM7 + INLINEFORM8 + INLINEFORM9 + INLINEFORM10   As shown in Figure FIGREF4 , INLINEFORM11 firstly employs a random document-event distribution INLINEFORM12 as an input  To model the multinomial property of the document-event distribution, INLINEFORM13 is drawn from a Dirichlet distribution parameterized with INLINEFORM14 which is formulated as: DISPLAYFORM0
where INLINEFORM0 is the hyper-parameter of the dirichlet distribution, INLINEFORM1 is the number of events which should be set in AEM, INLINEFORM2 , INLINEFORM3 represents the proportion of event INLINEFORM4 in the document and INLINEFORM5  
Subsequently, INLINEFORM0 transforms INLINEFORM1 into a INLINEFORM2 -dimensional hidden space using a linear layer followed by layer normalization, and the transformation is defined as: DISPLAYFORM0
where INLINEFORM0 represents the weight matrix of hidden layer, and INLINEFORM1 denotes the bias term, INLINEFORM2 is the parameter of LeakyReLU activation and is set to 0 1, INLINEFORM3 and INLINEFORM4 denote the normalized hidden states and the outputs of the hidden layer, and INLINEFORM5 represents the layer normalization 
Then, to project INLINEFORM0 into four document-level event related word distributions ( INLINEFORM1 , INLINEFORM2 , INLINEFORM3 and INLINEFORM4 shown in Figure FIGREF4 ), four subnets (each contains a linear layer, a batch normalization layer and a softmax layer) are employed in INLINEFORM5   And the exact transformation is based on the formulas below: DISPLAYFORM0
where INLINEFORM0 means softmax layer, INLINEFORM1 , INLINEFORM2 , INLINEFORM3 and INLINEFORM4 denote the weight matrices of the linear layers in subnets, INLINEFORM5 , INLINEFORM6 , INLINEFORM7 and INLINEFORM8 represent the corresponding bias terms, INLINEFORM9 , INLINEFORM10 , INLINEFORM11 and INLINEFORM12 are state vectors  INLINEFORM13 , INLINEFORM14 , INLINEFORM15 and INLINEFORM16 denote the generated entity distribution, location distribution, keyword distribution and date distribution, respectively, that correspond to the given event distribution INLINEFORM17   And each dimension represents the relevance between corresponding entity/location/keyword/date term and the input event distribution 
Finally, four generated distributions are concatenated to represent the generated document INLINEFORM0 corresponding to the input INLINEFORM1 : DISPLAYFORM0
The discriminator network INLINEFORM0 is designed as a fully-connected network which contains an input layer, a discriminative feature layer (discriminative features are employed for event visualization) and an output layer  In AEM, INLINEFORM1 uses fake document INLINEFORM2 and real document INLINEFORM3 as input and outputs the signal INLINEFORM4 to indicate the source of the input data (lower value denotes that INLINEFORM5 is prone to predict the input data as a fake document and vice versa) 
As have previously been discussed in BIBREF7 , BIBREF8 , lipschitz continuity of INLINEFORM0 network is crucial to the training of the GAN-based approaches  To ensure the lipschitz continuity of INLINEFORM1 , we employ the spectral normalization technique BIBREF9   More concretely, for each linear layer INLINEFORM2 (bias term is omitted for simplicity) in INLINEFORM3 , the weight matrix INLINEFORM4 is normalized by INLINEFORM5   Here, INLINEFORM6 is the spectral norm of the weight matrix INLINEFORM7 with the definition below: DISPLAYFORM0
which is equivalent to the largest singular value of INLINEFORM0   The weight matrix INLINEFORM1 is then normalized using: DISPLAYFORM0
Obviously, the normalized weight matrix INLINEFORM0 satisfies that INLINEFORM1 and further ensures the lipschitz continuity of the INLINEFORM2 network BIBREF9   To reduce the high cost of computing spectral norm INLINEFORM3 using singular value decomposition at each iteration, we follow BIBREF10 and employ the power iteration method to estimate INLINEFORM4 instead  With this substitution, the spectral norm can be estimated with very small additional computational time 
Objective and Training Procedure
The real document INLINEFORM0 and fake document INLINEFORM1 shown in Figure FIGREF4 could be viewed as random samples from two distributions INLINEFORM2 and INLINEFORM3 , and each of them is a joint distribution constituted by four Dirichlet distributions (corresponding to entity distribution, location distribution, keyword distribution and date distribution)  The training objective of AEM is to let the distribution INLINEFORM4 (produced by INLINEFORM5 network) to approximate the real data distribution INLINEFORM6 as much as possible 
To compare the different GAN losses, Kurach kurach2018gan takes a sober view of the current state of GAN and suggests that the Jansen-Shannon divergence used in BIBREF3 performs more stable than variant objectives  Besides, Kurach also advocates that the gradient penalty (GP) regularization devised in BIBREF8 will further improve the stability of the model  Thus, the objective function of the proposed AEM is defined as: DISPLAYFORM0
where INLINEFORM0 denotes the discriminator loss, INLINEFORM1 represents the gradient penalty regularization loss, INLINEFORM2 is the gradient penalty coefficient which trade-off the two components of objective, INLINEFORM3 could be obtained by sampling uniformly along a straight line between INLINEFORM4 and INLINEFORM5 , INLINEFORM6 denotes the corresponding distribution 
The training procedure of AEM is presented in Algorithm SECREF15 , where INLINEFORM0 is the event number, INLINEFORM1 denotes the number of discriminator iterations per generator iteration, INLINEFORM2 is the batch size, INLINEFORM3 represents the learning rate, INLINEFORM4 and INLINEFORM5 are hyper-parameters of Adam BIBREF11 , INLINEFORM6 denotes INLINEFORM7   In this paper, we set INLINEFORM8 , INLINEFORM9 , INLINEFORM10   Moreover, INLINEFORM11 , INLINEFORM12 and INLINEFORM13 are set as 0 0002, 0 5 and 0 999 
[!h] Training procedure for AEM [1] INLINEFORM0 , INLINEFORM1 , INLINEFORM2 , INLINEFORM3 , INLINEFORM4 , INLINEFORM5 , INLINEFORM6 the trained INLINEFORM7 and INLINEFORM8   Initial INLINEFORM9 parameters INLINEFORM10 and INLINEFORM11 parameter INLINEFORM12 INLINEFORM13 has not converged INLINEFORM14 INLINEFORM15 Sample INLINEFORM16 , Sample a random INLINEFORM17 Sample a random number INLINEFORM18 INLINEFORM19 INLINEFORM20 INLINEFORM21 INLINEFORM22 INLINEFORM23 INLINEFORM24 Sample INLINEFORM25 noise INLINEFORM26 INLINEFORM27
Event Generation
After the model training, the generator INLINEFORM0 learns the mapping function between the document-event distribution and the document-level event-related word distributions (entity, location, keyword and date)  In other words, with an event distribution INLINEFORM1 as input, INLINEFORM2 could generate the corresponding entity distribution, location distribution, keyword distribution and date distribution 
In AEM, we employ event seed INLINEFORM0 , an INLINEFORM1 -dimensional vector with one-hot encoding, to generate the event related word distributions  For example, in ten event setting, INLINEFORM2 represents the event seed of the first event  With the event seed INLINEFORM3 as input, the corresponding distributions could be generated by INLINEFORM4 based on the equation below: DISPLAYFORM0
where INLINEFORM0 , INLINEFORM1 , INLINEFORM2 and INLINEFORM3 denote the entity distribution, location distribution, keyword distribution and date distribution of the first event respectively 
Experiments
In this section, we firstly describe the datasets and baseline approaches used in our experiments and then present the experimental results 
Experimental Setup
To validate the effectiveness of AEM for extracting events from social media (e g  Twitter) and news media sites (e g  Google news), three datasets (FSD BIBREF12 , Twitter, and Google datasets) are employed  Details are summarized below:
FSD dataset (social media) is the first story detection dataset containing 2,499 tweets  We filter out events mentioned in less than 15 tweets since events mentioned in very few tweets are less likely to be significant  The final dataset contains 2,453 tweets annotated with 20 events 
Twitter dataset (social media) is collected from tweets published in the month of December in 2010 using Twitter streaming API  It contains 1,000 tweets annotated with 20 events 
Google dataset (news article) is a subset of GDELT Event Database INLINEFORM0 , documents are retrieved by event related words  For example, documents which contain `malaysia', `airline', `search' and `plane' are retrieved for event MH370  By combining 30 events related documents, the dataset contains 11,909 news articles 
We choose the following three models as the baselines:
K-means is a well known data clustering algorithm, we implement the algorithm using sklearn toolbox, and represent documents using bag-of-words weighted by TF-IDF 
LEM BIBREF13 is a Bayesian modeling approach for open-domain event extraction  It treats an event as a latent variable and models the generation of an event as a joint distribution of its individual event elements  We implement the algorithm with the default configuration 
DPEMM BIBREF14 is a non-parametric mixture model for event extraction  It addresses the limitation of LEM that the number of events should be known beforehand  We implement the model with the default configuration 
For social media text corpus (FSD and Twitter), a named entity tagger specifically built for Twitter is used to extract named entities including locations from tweets  A Twitter Part-of-Speech (POS) tagger BIBREF15 is used for POS tagging and only words tagged with nouns, verbs and adjectives are retained as keywords  For the Google dataset, we use the Stanford Named Entity Recognizer to identify the named entities (organization, location and person)  Due to the `date' information not being provided in the Google dataset, we further divide the non-location named entities into two categories (`person' and `organization') and employ a quadruple <organization, location, person, keyword> to denote an event in news articles  We also remove common stopwords and only keep the recognized named entities and the tokens which are verbs, nouns or adjectives 
Experimental Results
To evaluate the performance of the proposed approach, we use the evaluation metrics such as precision, recall and F-measure  Precision is defined as the proportion of the correctly identified events out of the model generated events  Recall is defined as the proportion of correctly identified true events  For calculating the precision of the 4-tuple, we use following criteria:
(1) Do the entity/organization, location, date/person and keyword that we have extracted refer to the same event?
(2) If the extracted representation contains keywords, are they informative enough to tell us what happened?
Table TABREF35 shows the event extraction results on the three datasets  The statistics are obtained with the default parameter setting that INLINEFORM0 is set to 5, number of hidden units INLINEFORM1 is set to 200, and INLINEFORM2 contains three fully-connected layers  The event number INLINEFORM3 for three datasets are set to 25, 25 and 35, respectively  The examples of extracted events are shown in Table  TABREF36  
It can be observed that K-means performs the worst over all three datasets  On the social media datasets, AEM outpoerforms both LEM and DPEMM by 6 5% and 1 7% respectively in F-measure on the FSD dataset, and 4 4% and 3 7% in F-measure on the Twitter dataset  We can also observe that apart from K-means, all the approaches perform worse on the Twitter dataset compared to FSD, possibly due to the limited size of the Twitter dataset  Moreover, on the Google dataset, the proposed AEM performs significantly better than LEM and DPEMM  It improves upon LEM by 15 5% and upon DPEMM by more than 30% in F-measure  This is because: (1) the assumption made by LEM and DPEMM that all words in a document are generated from a single event is not suitable for long text such as news articles; (2) DPEMM generates too many irrelevant events which leads to a very low precision score  Overall, we see the superior performance of AEM across all datasets, with more significant improvement on the for Google datasets (long text) 
We next visualize the detected events based on the discriminative features learned by the trained INLINEFORM0 network in AEM  The t-SNE BIBREF16 visualization results on the datasets are shown in Figure FIGREF19   For clarity, each subplot is plotted on a subset of the dataset containing ten randomly selected events  It can be observed that documents describing the same event have been grouped into the same cluster 
To further evaluate if a variation of the parameters INLINEFORM0 (the number of discriminator iterations per generator iteration), INLINEFORM1 (the number of units in hidden layer) and the structure of generator INLINEFORM2 will impact the extraction performance, additional experiments have been conducted on the Google dataset, with INLINEFORM3 set to 5, 7 and 10, INLINEFORM4 set to 100, 150 and 200, and three INLINEFORM5 structures (3, 4 and 5 layers)  The comparison results on precision, recall and F-measure are shown in Figure FIGREF20   From the results, it could be observed that AEM with the 5-layer generator performs the best and achieves 96 7% in F-measure, and the worst F-measure obtained by AEM is 85 7%  Overall, the AEM outperforms all compared approaches acorss various parameter settings, showing relatively stable performance 
Finally, we compare in Figure FIGREF37 the training time required for each model, excluding the constant time required by each model to load the data  We could observe that K-means runs fastest among all four approaches  Both LEM and DPEMM need to sample the event allocation for each document and update the relevant counts during Gibbs sampling which are time consuming  AEM only requires a fraction of the training time compared to LEM and DPEMM  Moreover, on a larger dataset such as the Google dataset, AEM appears to be far more efficient compared to LEM and DPEMM 
Conclusions and Future Work
In this paper, we have proposed a novel approach based on adversarial training to extract the structured representation of events from online text  The experimental comparison with the state-of-the-art methods shows that AEM achieves improved extraction performance, especially on long text corpora with an improvement of 15% observed in F-measure  AEM only requires a fraction of training time compared to existing Bayesian graphical modeling approaches  In future work, we will explore incorporating external knowledge (e g  word relatedness contained in word embeddings) into the learning framework for event extraction  Besides, exploring nonparametric neural event extraction approaches and detecting the evolution of events over time from news articles are other promising future directions 
Acknowledgments
We would like to thank anonymous reviewers for their valuable comments and helpful suggestions  This work was funded by the National Key Research and Development Program of China (2016YFC1306704), the National Natural Science Foundation of China (61772132), the Natural Science Foundation of Jiangsu Province of China (BK20161430) 
Context 3:  Thus, each row contains relative importance of words in INLINEFORM5 to a particular word INLINEFORM6   Weighted sum of INLINEFORM7 is taken (eqn  4) and fused with the word representation INLINEFORM8 using concatenation (eqn 5) 
Limits of using only Attention
Attention operates by using dot product or sometimes addition followed by linear projection to a scalar which models the similarity between two vectors  Subsequently, softmax is applied which gives high probabilities to most matching word representations  This assumes that there is some highly matched word pairs already existing and high scores will be assigned to them  Given a vector INLINEFORM0 =( INLINEFORM1 ,   , INLINEFORM2 ) on which softmax function is applied, each INLINEFORM3 INLINEFORM4 (0, 1)  It is observable that the average value of INLINEFORM5 is always INLINEFORM6   In other words, it is impossible to produce a vector having all INLINEFORM7 < INLINEFORM8 when two sequences have no matching at all 
In cases, where one or more word pairs from two different sequences are highly dissimilar, it is impossible to assign a very low probability to it without increasing the probability of some other pair somewhere else since INLINEFORM0 = 1 
For example, when we consider two sequences "height of tom cruise" and "age of sun", while computing the attention weights between the word "height" and all the words in the second sequence it can be observed that their no matching word in the latter  In this case, a standard dot-product based attention with softmax won't be able to produce weights which is below 0 33 (=1/3) for all the words in the second sequence with respect to the word "height" in the first sequence 
Conflict model
We propose a different mechanism that does the opposite of what attention does that is computing how much two sequences repel each other  This works very similar to how attention works but inversely 
We demonstrate a general model but we also realize that there can be other variants of it which may be worked out to perform better  Our approach consists of using element wise difference between two vectors followed by a linear transformation to produce a scalar weight  The remaining of the process acts similar to how attention works  Mathematically, we can express it as: DISPLAYFORM0
where INLINEFORM0 INLINEFORM1 INLINEFORM2 is a parameter that we introduce to provide a weight for the pair  The two word representations INLINEFORM3 and INLINEFORM4 are projected to a space where their element wise difference can be used to model their dissimilarity and softmax applied on them can produce high probability to more dissimilar word pairs 
It is good to note that conflict suffers from the same limitation that attention suffers from  This is when a pair of sentences are highly matching especially with multiple associations  But when the two methods work together, each compensates for the other's shortcomings 
Combination of attention and conflict
We used two weighted representations of INLINEFORM0 using weights of attention and conflict as computed in Eqn  (4) and (8) respectively  Our final representation of a word representation INLINEFORM1 conditioned on INLINEFORM2 can be expressed as: DISPLAYFORM0
where A and C denote that they are from attention and conflict models respectively 
Relation to Multi-Head attention
Multi-head attention, as introduced in BIBREF3 , computes multiple identical attention mechanism parallelly on multiple linear projections of same inputs  The parameters of each attention and projections are different in each head  Finally, they concatenate all the attentions which is similar to how we concatenate conflict and attention  However, they use dot-product to compute each of the attention 
Our combined model that contains both attention and conflict can be thought of as a 2-head attention model but both heads are different  Our conflict head explicitly captures difference between the inputs 
Visualizing attention and conflict
We observe how our conflict model learns the dissimilarities between word representations  We achieve that by visualizing the heatmap of the weight matrix INLINEFORM0 for both attention and conflict from eqns  (3) and (8)  While attention successfully learns the alignments, conflict matrix also shows that our approach models the contradicting associations like "animal" and "lake" or "australia" and "world"  These two associations are the unique pairs which are instrumental in determining that the two queries are not similar 
The model
We create two models both of which constitutes of three main parts: encoder, interaction and classifier and take two sequences as input  Except interaction, all the other parts are exactly identical between the two models  The encoder is shared among the sequences simply uses two stacked GRU layers  The interaction part consists of only attention for one model while for the another one it consists of attention and conflict combined as shown in (eqn 11) 

ground_truth: Unanswerable

Unanswerable

prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. On which tasks do they test their conflict method?
Context 3:  The classifier part is simply stacked fully-connected layers  Figure 3 shows a block diagram of how our model looks like 
Task 1: Quora Duplicate Question Pair Detection
The dataset includes pairs of questions labelled as 1 or 0 depending on whether a pair is duplicate or not respectively  This is a popular pair-level classification task on which extensive work has already been done before like BIBREF7 , BIBREF8   For this task, we make the output layer of our model to predict two probabilities for non-duplicate and duplicate  We sample the data from the original dataset so that it contains equal positive and negative classes  Original dataset has some class imbalance but for sake simplicity we don't consider it  The final data that we use has roughly 400,000 question pairs and we split this data into train and test using 8:2 ratio 
We train all our models for roughly 2 epochs with a batch size of 64  We use a hidden dimension of 150 throughout the model  The embedding layer uses ELMO BIBREF9 which has proven to be very useful in various downstream language understanding tasks  Our FC layers consists of four dense layers with INLINEFORM0 activation after each layer  The dropout rate is kept as 0 2 for every recurrent and FC linear layers  We use Adam optimizer in our experiment with epsilon=1e-8, beta=0 9 and learning rate=1e-3 
Task 2: Ranking questions in Bing's People Also Ask
People Also Ask is a feature in Bing search result page where related questions are recommended to the user  User may click on a question to view the answer  Clicking is a positive feedback that shows user's interest in the question  We use this click logs to build a question classifier using the same model in Figure 3  The problem statement is very similar to BIBREF10 where they use logistic regression to predict whether an user would click on ad  Our goal is to classify if a question is potential high-click question or not for a given query  For this, we first create a labelled data set using the click logs where any question having CTR lower than 0 3 is labelled as 0 and a question having CTR more than 0 7 as 1 
Our final data resembles that of a pair-level classifier, as in Task 1, where user query and candidate questions are input  With these data set, we train a binary classifier to detect high-click and low-click questions 
Quantitative Analysis
For both tasks, we compute classification accuracy using three model variants and report the results in Table 1 and Table 2  We observe that model with both attention and conflict combined gives the best results 
We also show the training loss curve for both the models having attention and attention combined with conflict respectively  Figure 4 and 5 shows these curves for Task 1 and Task 2 respectively  The curves are smoothed using moving average having an window size of 8  We notice that the conflict model has much steeper slope and converges to a much better minima in both the tasks  It can also be noticed that in the training procedure for the model which has both attention and conflict, the updates are much smoother 
Qualitative Comparison
We also show qualitative results where we can observe that our model with attention and conflict combined does better on cases where pairs are non-duplicate and has very small difference  We have observed that the conflict model is very sensitive to even minor differences and compensates in such cases where attention poses high bias towards similarities already there in the sequences 
Sequence 1: What are the best ways to learn French ?
Sequence 2: How do I learn french genders ?
Attention only: 1
Attention+Conflict: 0
Ground Truth: 0
Sequence 1: How do I prevent breast cancer ?
Sequence 2: Is breast cancer preventable ?
Attention only: 1
Attention+Conflict: 0
Ground Truth: 0
We provide two examples with predictions from the models with only attention and combination of attention and conflict  Each example is accompanied by the ground truth in our data 
Analyzing the gains
We analyzed the gains in Task 1 which we get from the attention-conflict model in order to ensure that they are not due to randomness in weight initialization or simply additional parameters  We particularly focused on the examples which were incorrectly marked in attention model but correctly in attention-conflict model  We saw that 70% of those cases are the ones where the pair was incorrectly marked as duplicate in the previous model but our combined model correctly marked them as non-duplicate 
Conclusion
In this work, we highlighted the limits of attention especially in cases where two sequences have a contradicting relationship based on the task it performs  To alleviate this problem and further improve the performance, we propose a conflict mechanism that tries to capture how two sequences repel each other  This acts like the inverse of attention and, empirically, we show that how conflict and attention together can improve the performance 
Future research work should be based on alternative design of conflict mechanism using other difference operators other than element wise difference which we use 

ground_truth: Task 1: Quora Duplicate Question Pair Detection, Task 2: Ranking questions

Quora Duplicate Question Pair Detection, Ranking questions in Bing's People Also Ask