prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. How does this approach compare to other WSD approaches employing word embeddings?
Context 1: Introduction
Language modelling in its inception had one-hot vector encoding of words  However, it captures only alphabetic ordering but not the word semantic similarity  Vector space models helps to learn word representations in a lower dimensional space and also captures semantic similarity  Learning word embedding aids in natural language processing tasks such as question answering and reasoning BIBREF0, stance detection BIBREF1, claim verification BIBREF2 
Recent models BIBREF3, BIBREF4 work on the basis that words with similar context share semantic similarity  BIBREF4 proposes a neural probabilistic model which models the target word probability conditioned on the previous words using a recurrent neural network  Word2Vec models BIBREF3 such as continuous bag-of-words (CBOW) predict the target word given the context, and skip-gram model works in reverse of predicting the context given the target word  While, GloVe embeddings were based on a Global matrix factorization on local contexts BIBREF5  However, the aforementioned models do not handle words with multiple meanings (polysemies) 
BIBREF6 proposes a neural network approach considering both local and global contexts in learning word embeddings (point estimates)  Their multiple prototype model handles polysemous words by providing apriori heuristics about word senses in the dataset  BIBREF7 proposes an alternative to handle polysemous words by a modified skip-gram model and EM algorithm  BIBREF8 presents a non-parametric based alternative to handle polysemies  However, these approaches fail to consider entailment relations among the words  BIBREF9 learn a Gaussian distribution per word using the expected likelihood kernel  However, for polysemous words, this may lead to word distributions with larger variances as it may have to cover various senses 
BIBREF10 proposes multimodal word distribution approach  It captures polysemy  However, the energy based objective function fails to consider asymmetry and hence entailment  Textual entailment recognition is necessary to capture lexical inference relations such as causality (for example, mosquito $\rightarrow $ malaria), hypernymy (for example, dog $\models $ animal) etc 
In this paper, we propose to obtain multi-sense word embedding distributions by using a variant of max margin objective based on the asymmetric KL divergence energy function to capture textual entailment  Multi-sense distributions are advantageous in capturing polysemous nature of words and in reducing the uncertainty per word by distributing it across senses  However, computing KL divergence between mixtures of Gaussians is intractable, and we use a KL divergence approximation based on stricter upper and lower bounds  While capturing textual entailment (asymmetry), we have also not compromised on capturing symmetrical similarity between words (for example, funny and hilarious) which will be elucidated in Section $3 1$  We also show the effectiveness of the proposed approach on the benchmark word similarity and entailment datasets in the experimental section 
Methodology ::: Word Representation
Probabilistic representation of words helps one model uncertainty in word representation, and polysemy  Given a corpus $V$, containing a list of words each represented as $w$, the probability density for a word $w$ can be represented as a mixture of Gaussians with $C$ components BIBREF10 
Here, $p_{w,j}$ represents the probability of word $w$ belonging to the component $j$, $\operatorname{\mathbf {\mu }}_{w,j}$ represents $D$ dimensional word representation corresponding to the $j^{th}$ component sense of the word $w$, and $\Sigma _{w,j}$ represents the uncertainty in representation for word $w$ belonging to component $j$ 
Objective function
The model parameters (means, covariances and mixture weights) $\theta $ can be learnt using a variant of max-margin objective BIBREF11 
Here $E_\theta (\cdot , \cdot )$ represents an energy function which assigns a score to the pair of words, $w$ is a particular word under consideration, $c$ its positive context (same context), and $c^{\prime }$ the negative context  The objective aims to push the margin of the difference between the energy function of a word $w$ to its positive context $c$ higher than its negative context $c$ by a threshold of $m$  Thus, word pairs in the same context gets a higher energy than the word pairs in the dissimilar context  BIBREF10 consider the energy function to be an expected likelihood kernel which is defined as follows 
This is similar to the cosine similarity metric over vectors and the energy between two words is maximum when they have similar distributions  But, the expected likelihood kernel is a symmetric metric which will not be suitable for capturing ordering among words and hence entailment 
Objective function ::: Proposed Energy function
As each word is represented by a mixture of Gaussian distributions, KL divergence is a better choice of energy function to capture distance between distributions  Since, KL divergence is minimum when the distributions are similar and maximum when they are dissimilar, energy function is taken as exponentiated negative KL divergence 
However, computing KL divergence between Gaussian mixtures is intractable and obtaining exact KL value is not possible  One way of approximating the KL is by Monte-Carlo approximation but it requires large number of samples to get a good approximation and is computationally expensive on high dimensional embedding space 
Alternatively, BIBREF12 presents a KL approximation between Gaussian mixtures where they obtain an upper bound through product of Gaussian approximation method and a lower bound through variational approximation method  In BIBREF13, the authors combine the lower and upper bounds from approximation methods of BIBREF12 to provide a stricter bound on KL between Gaussian mixtures  Lets consider Gaussian mixtures for the words $w$ and $v$ as follows 
The approximate KL divergence between the Gaussian mixture representations over the words $w$ and $v$ is shown in equation DISPLAY_FORM8  More details on approximation is included in the Supplementary Material 
where $EL_{ik}(w,w) = \int f_{w,i} (\operatorname{\mathbf {x}}) f_{w,k} (\operatorname{\mathbf {x}}) d\operatorname{\mathbf {x}}$ and $EL_{ij}(w,v) = \int f_{w,i} (\operatorname{\mathbf {x}}) f_{v,k} (\operatorname{\mathbf {x}}) d\operatorname{\mathbf {x}}$  Note that the expected likelihood kernel appears component wise inside the approximate KL divergence derivation 
One advantage of using KL as energy function is that it enables to capture asymmetry in entailment datasets  For eg , let us consider the words 'chair' with two senses as 'bench' and 'sling', and 'wood' with two senses as 'trees' and 'furniture'  The word chair ($w$) is entailed within wood ($v$), i e  chair $\models $ wood  Now, minimizing the KL divergence necessitates maximizing $\log {\sum _j p_{v,j} \exp ({-KL(f_{w,i} (\operatorname{\mathbf {x}})||f_{v,j}(\operatorname{\mathbf {x}}))})}$ which in turn minimizes $KL(f_{w,i}(\operatorname{\mathbf {x}})||f_{v,j}(\operatorname{\mathbf {x}}))$  This will result in the support of the $i^{th}$ component of $w$ to be within the $j^{th}$ component of $v$, and holds for all component pairs leading to the entailment of $w$ within $v$  Consequently, we can see that bench $\models $ trees, bench $\models $ furniture, sling $\models $ trees, and sling $\models $ furniture  Thus, it introduces lexical relationship between the senses of child word and that of the parent word  Minimizing the KL also necessitates maximizing $\log {\sum _j {p_{v,j}} EL_{ij}(w,v)}$ term for all component pairs among $w$ and $v$  This is similar to maximizing expected likelihood kernel, which brings the means of $f_{w,i}(\operatorname{\mathbf {x}})$ and $f_{v,j}(\operatorname{\mathbf {x}})$ closer (weighted by their co-variances) as discussed in BIBREF10  Hence, the proposed approach captures the best of both worlds, thereby catering to both word similarity and entailment 
We also note that minimizing the KL divergence necessitates minimizing $\log {\sum _k p_{w,k} \exp ({-KL(f_{w,i}||f_{w,k})})}$ which in turn maximizes $KL(f_{w,i}||f_{w,k})$  This prevents the different mixture components of a word converging to single Gaussian and encourages capturing different possible senses of the word  The same is also achieved by minimizing $\sum _k {p_{w,k}} EL_{ik}(w,w)$ term and act as a regularization term which promotes diversity in learning senses of a word 
Experimentation and Results
We train our proposed model GM$\_$KL (Gaussian Mixture using KL Divergence) on the Text8 dataset BIBREF14 which is a pre-processed data of $17M$ words from wikipedia  Of which, 71290 unique and frequent words are chosen using the subsampling trick in BIBREF15  We compare GM$\_$KL with the previous approaches w2g BIBREF9 ( single Gaussian model) and w2gm BIBREF10 (mixture of Gaussian model with expected likelihood kernel)  For all the models used for experimentation, the embedding size ($D$) was set to 50, number of mixtures to 2, context window length to 10, batch size to 128  The word embeddings were initialized using a uniform distribution in the range of $[-\sqrt{\frac{3}{D}}$, $\sqrt{\frac{3}{D}}]$ such that the expectation of variance is 1 and mean 0 BIBREF16  One could also consider initializing the word embeddings using other contextual representations such as BERT BIBREF17 and ELMo BIBREF18 in the proposed approach  In order to purely analyze the performance of $\emph {GM\_KL}$ over the other models, we have chosen initialization using uniform distribution for experiments  For computational benefits, diagonal covariance is used similar to BIBREF10  Each mixture probability is constrained in the range $[0,1]$, summing to 1 by optimizing over unconstrained scores in the range $(-\infty ,\infty )$ and converting scores to probability using softmax function  The mixture scores are initialized to 0 to ensure fairness among all the components  The threshold for negative sampling was set to $10^{-5}$, as recommended in BIBREF3  Mini-batch gradient descent with Adagrad optimizer BIBREF19 was used with initial learning rate set to $0 05$ 
Table TABREF9 shows the qualitative results of GM$\_$KL  Given a query word and component id, the set of nearest neighbours along with their respective component ids are listed  For eg , the word `plane' in its 0th component captures the `geometry' sense and so are its neighbours, and its 1st component captures `vehicle' sense and so are its corresponding neighbours  Other words such as `rock' captures both the `metal' and `music' senses, `star' captures `celebrity' and `astronomical' senses, and `phone' captures `telephony' and `internet' senses 
We quantitatively compare the performance of the GM$\_$KL, w2g, and w2gm approaches on the SCWS dataset BIBREF6  The dataset consists of 2003 word pairs of polysemous and homonymous words with labels obtained by an average of 10 human scores  The Spearman correlation between the human scores and the model scores are computed  To obtain the model score, the following metrics are used:
MaxCos: Maximum cosine similarity among all component pairs of words $w$ and $v$:
AvgCos: Average component-wise cosine similarity between the words $w$ and $v$ 
KL$\_$approx: Formulated as shown in (DISPLAY_FORM8) between the words $w$ and $v$ 
KL$\_$comp: Maximum component-wise negative KL between words $w$ and $v$:
Table TABREF17 compares the performance of the approaches on the SCWS dataset  It is evident from Table TABREF17 that GM$\_$KL achieves better correlation than existing approaches for various metrics on SCWS dataset 
Table TABREF18 shows the Spearman correlation values of GM$\_$KL model evaluated on the benchmark word similarity datasets: SL BIBREF20, WS, WS-R, WS-S BIBREF21, MEN BIBREF22, MC BIBREF23, RG BIBREF24, YP BIBREF25, MTurk-287 and MTurk-771 BIBREF26, BIBREF27, and RW BIBREF28  The metric used for comparison is 'AvgCos'  It can be seen that for most of the datasets, GM$\_$KL achieves significantly better correlation score than w2g and w2gm approaches  Other datasets such as MC and RW consist of only a single sense, and hence w2g model performs better and GM$\_$KL achieves next better performance  The YP dataset have multiple senses but does not contain entailed data and hence could not make use of entailment benefits of GM$\_$KL 
Table TABREF19 shows the evaluation results of GM$\_$KL model on the entailment datasets such as entailment pairs dataset BIBREF29 created from WordNet with both positive and negative labels, a crowdsourced dataset BIBREF30 of 79 semantic relations labelled as entailed or not and annotated distributionally similar nouns dataset BIBREF31  The 'MaxCos' similarity metric is used for evaluation and the best precision and best F1-score is shown, by picking the optimal threshold  Overall, GM$\_$KL performs better than both w2g and w2gm approaches 
Conclusion
We proposed a KL divergence based energy function for learning multi-sense word embedding distributions modelled as Gaussian mixtures  Due to the intractability of the Gaussian mixtures for the KL divergence measure, we use an approximate KL divergence function  We also demonstrated that the proposed GM$\_$KL approaches performed better than other approaches on the benchmark word similarity and entailment datasets 
tocsectionAppendices
Approximation for KL divergence between mixtures of gaussians
KL between gaussian mixtures $f_{w}(\operatorname{\mathbf {x}})$ and $f_{v}(\operatorname{\mathbf {x}})$ can be decomposed as:
BIBREF12 presents KL approximation between gaussian mixtures using
product of gaussian approximation method where KL is approximated using product of component gaussians and
variational approximation method where KL is approximated by introducing some variational parameters 
The product of component gaussian approximation method using Jensen's inequality provides upper bounds as shown in equations DISPLAY_FORM23 and  
The variational approximation method provides lower bounds as shown in equations DISPLAY_FORM24 and DISPLAY_FORM25 
where $H$ represents the entropy term and the entropy of $i^{th}$ component of word $w$ with dimension $D$ is given as
In BIBREF13, the authors combine the lower and upper bounds from approximation methods of BIBREF12 to formulate a stricter bound on KL between gaussian mixtures 
From equations DISPLAY_FORM23 and DISPLAY_FORM25, a stricter lower bound for KL between gaussian mixtures is obtained as shown in equation DISPLAY_FORM26
From equations and DISPLAY_FORM24, a stricter upper bound for KL between gaussian mixtures is obtained as shown in equation DISPLAY_FORM27
Finally, the KL between gaussian mixtures is taken as the mean of KL upper and lower bounds as shown in equation DISPLAY_FORM28 
Context 2: Introduction
We posses a wealth of prior knowledge about many natural language processing tasks  For example, in text categorization, we know that words such as NBA, player, and basketball are strong indicators of the sports category BIBREF0 , and words like terrible, boring, and messing indicate a negative polarity while words like perfect, exciting, and moving suggest a positive polarity in sentiment classification 
A key problem arisen here, is how to leverage such knowledge to guide the learning process, an interesting problem for both NLP and machine learning communities  Previous studies addressing the problem fall into several lines  First, to leverage prior knowledge to label data BIBREF1 , BIBREF2   Second, to encode prior knowledge with a prior on parameters, which can be commonly seen in many Bayesian approaches BIBREF3 , BIBREF4   Third, to formalise prior knowledge with additional variables and dependencies BIBREF5   Last, to use prior knowledge to control the distributions over latent output variables BIBREF6 , BIBREF7 , BIBREF8 , which makes the output variables easily interpretable 
However, a crucial problem, which has rarely been addressed, is the bias in the prior knowledge that we supply to the learning model  Would the model be robust or sensitive to the prior knowledge? Or, which kind of knowledge is appropriate for the task? Let's see an example: we may be a baseball fan but unfamiliar with hockey so that we can provide a few number of feature words of baseball, but much less of hockey for a baseball-hockey classification task  Such prior knowledge may mislead the model with heavy bias to baseball  If the model cannot handle this situation appropriately, the performance may be undesirable 
In this paper, we investigate into the problem in the framework of Generalized Expectation Criteria BIBREF7   The study aims to reveal the factors of reducing the sensibility of the prior knowledge and therefore to make the model more robust and practical  To this end, we introduce auxiliary regularization terms in which our prior knowledge is formalized as distribution over output variables  Recall the example just mentioned, though we do not have enough knowledge to provide features for class hockey, it is easy for us to provide some neutral words, namely words that are not strong indicators of any class, like player here  As one of the factors revealed in this paper, supplying neutral feature words can boost the performance remarkably, making the model more robust 
More attractively, we do not need manual annotation to label these neutral feature words in our proposed approach 
More specifically, we explore three regularization terms to address the problem: (1) a regularization term associated with neutral features; (2) the maximum entropy of class distribution regularization term; and (3) the KL divergence between reference and predicted class distribution  For the first manner, we simply use the most common features as neutral features and assume the neutral features are distributed uniformly over class labels  For the second and third one, we assume we have some knowledge about the class distribution which will be detailed soon later 
To summarize, the main contributions of this work are as follows:
The rest of the paper is structured as follows: In Section 2, we briefly describe the generalized expectation criteria and present the proposed regularization terms  In Section 3, we conduct extensive experiments to justify the proposed methods  We survey related work in Section 4, and summarize our work in Section 5 
Method
We address the robustness problem on top of GE-FL BIBREF0 , a GE method which leverages labeled features as prior knowledge  A labeled feature is a strong indicator of a specific class and is manually provided to the classifier  For example, words like amazing, exciting can be labeled features for class positive in sentiment classification 
Generalized Expectation Criteria
Generalized expectation (GE) criteria BIBREF7 provides us a natural way to directly constrain the model in the preferred direction  For example, when we know the proportion of each class of the dataset in a classification task, we can guide the model to predict out a pre-specified class distribution 
Formally, in a parameter estimation objective function, a GE term expresses preferences on the value of some constraint functions about the model's expectation  Given a constraint function $G({\rm x}, y)$ , a conditional model distribution $p_\theta (y|\rm x)$ , an empirical distribution $\tilde{p}({\rm x})$ over input samples and a score function $S$ , a GE term can be expressed as follows:
$$S(E_{\tilde{p}({\rm x})}[E_{p_\theta (y|{\rm x})}[G({\rm x}, y)]])$$   (Eq  4)
Learning from Labeled Features
Druck et al  ge-fl proposed GE-FL to learn from labeled features using generalized expectation criteria  When given a set of labeled features $K$ , the reference distribution over classes of these features is denoted by $\hat{p}(y| x_k), k \in K$   GE-FL introduces the divergence between this reference distribution and the model predicted distribution $p_\theta (y | x_k)$ , as a term of the objective function:
$$\mathcal {O} = \sum _{k \in K} KL(\hat{p}(y|x_k) || p_\theta (y | x_k)) + \sum _{y,i} \frac{\theta _{yi}^2}{2 \sigma ^2}$$   (Eq  6)
where $\theta _{yi}$ is the model parameter which indicates the importance of word $i$ to class $y$   The predicted distribution $p_\theta (y | x_k)$ can be expressed as follows: $ p_\theta (y | x_k) = \frac{1}{C_k} \sum _{\rm x} p_\theta (y|{\rm x})I(x_k) $
in which $I(x_k)$ is 1 if feature $k$ occurs in instance ${\rm x}$ and 0 otherwise, $C_k = \sum _{\rm x} I(x_k)$ is the number of instances with a non-zero value of feature $k$ , and $p_\theta (y|{\rm x})$ takes a softmax form as follows: $ p_\theta (y|{\rm x}) = \frac{1}{Z(\rm x)}\exp (\sum _i \theta _{yi}x_i)  $
To solve the optimization problem, L-BFGS can be used for parameter estimation 
In the framework of GE, this term can be obtained by setting the constraint function $G({\rm x}, y) = \frac{1}{C_k} \vec{I} (y)I(x_k)$ , where $\vec{I}(y)$ is an indicator vector with 1 at the index corresponding to label $y$ and 0 elsewhere 
Regularization Terms
GE-FL reduces the heavy load of instance annotation and performs well when we provide prior knowledge with no bias  In our experiments, we observe that comparable numbers of labeled features for each class have to be supplied  But as mentioned before, it is often the case that we are not able to provide enough knowledge for some of the classes  For the baseball-hockey classification task, as shown before, GE-FL will predict most of the instances as baseball  In this section, we will show three terms to make the model more robust 
Neutral features are features that are not informative indicator of any classes, for instance, word player to the baseball-hockey classification task  Such features are usually frequent words across all categories  When we set the preference distribution of the neutral features to be uniform distributed, these neutral features will prevent the model from biasing to the class that has a dominate number of labeled features 
Formally, given a set of neutral features $K^{^{\prime }}$ , the uniform distribution is $\hat{p}_u(y|x_k) = \frac{1}{|C|}, k \in K^{^{\prime }}$ , where $|C|$ is the number of classes  The objective function with the new term becomes
$$\mathcal {O}_{NE} = \mathcal {O} + \sum _{k \in K^{^{\prime }}} KL(\hat{p}_u(y|x_k) || p_\theta (y | x_k)) $$   (Eq  9)
Note that we do not need manual annotation to provide neutral features  One simple way is to take the most common features as neutral features  Experimental results show that this strategy works successfully 
Another way to prevent the model from drifting from the desired direction is to constrain the predicted class distribution on unlabeled data  When lacking knowledge about the class distribution of the data, one feasible way is to take maximum entropy principle, as below:
$$\mathcal {O}_{ME} = \mathcal {O} + \lambda \sum _{y} p(y) \log p(y)$$   (Eq  11)
where $p(y)$ is the predicted class distribution, given by $ p(y) = \frac{1}{|X|} \sum _{\rm x} p_\theta (y | \rm x)  $ To control the influence of this term on the overall objective function, we can tune $\lambda $ according to the difference in the number of labeled features of each class  In this paper, we simply set $\lambda $ to be proportional to the total number of labeled features, say $\lambda = \beta |K|$  
This maximum entropy term can be derived by setting the constraint function to $G({\rm x}, y) = \vec{I}(y)$   Therefore, $E_{p_\theta (y|{\rm x})}[G({\rm x}, y)]$ is just the model distribution $p_\theta (y|{\rm x})$ and its expectation with the empirical distribution $\tilde{p}(\rm x)$ is simply the average over input samples, namely $p(y)$   When $S$ takes the maximum entropy form, we can derive the objective function as above 
Sometimes, we have already had much knowledge about the corpus, and can estimate the class distribution roughly without labeling instances  Therefore, we introduce the KL divergence between the predicted and reference class distributions into the objective function 
Given the preference class distribution $\hat{p}(y)$ , we modify the objective function as follows:
$$\mathcal {O}_{KL} &= \mathcal {O} + \lambda KL(\hat{p}(y) || p(y))$$   (Eq  13)
Similarly, we set $\lambda = \beta |K|$  
This divergence term can be derived by setting the constraint function to $G({\rm x}, y) = \vec{I}(y)$ and setting the score function to $S(\hat{p}, p) = \sum _i \hat{p}_i \log \frac{\hat{p}_i}{p_i}$ , where $p$ and $\hat{p}$ are distributions  Note that this regularization term involves the reference class distribution which will be discussed later 
Experiments
In this section, we first justify the approach when there exists unbalance in the number of labeled features or in class distribution  Then, to test the influence of $\lambda $ , we conduct some experiments with the method which incorporates the KL divergence of class distribution  Last, we evaluate our approaches in 9 commonly used text classification datasets  We set $\lambda = 5|K|$ by default in all experiments unless there is explicit declaration  The baseline we choose here is GE-FL BIBREF0 , a method based on generalization expectation criteria 
Data Preparation
We evaluate our methods on several commonly used datasets whose themes range from sentiment, web-page, science to medical and healthcare  We use bag-of-words feature and remove stopwords in the preprocess stage  Though we have labels of all documents, we do not use them during the learning process, instead, we use the label of features 
The movie dataset, in which the task is to classify the movie reviews as positive or negtive, is used for testing the proposed approaches with unbalanced labeled features, unbalanced datasets or different $\lambda $ parameters  All unbalanced datasets are constructed based on the movie dataset by randomly removing documents of the positive class  For each experiment, we conduct 10-fold cross validation 
As described in BIBREF0 , there are two ways to obtain labeled features  The first way is to use information gain  We first calculate the mutual information of all features according to the labels of the documents and select the top 20 as labeled features for each class as a feature pool  Note that using information gain requires the document label, but this is only to simulate how we human provide prior knowledge to the model  The second way is to use LDA BIBREF9 to select features  We use the same selection process as BIBREF0 , where they first train a LDA on the dataset, and then select the most probable features of each topic (sorted by $P(w_i|t_j)$ , the probability of word $w_i$ given topic $t_j$ ) 
Similar to BIBREF10 , BIBREF0 , we estimate the reference distribution of the labeled features using a heuristic strategy  If there are $|C|$ classes in total, and $n$ classes are associated with a feature $k$ , the probability that feature $k$ is related with any one of the $n$ classes is $\frac{0 9}{n}$ and with any other class is $\frac{0 1}{|C| - n}$  
Neutral features are the most frequent words after removing stop words, and their reference distributions are uniformly distributed  We use the top 10 frequent words as neutral features in all experiments 
With Unbalanced Labeled Features
In this section, we evaluate our approach when there is unbalanced knowledge on the categories to be classified  The labeled features are obtained through information gain  Two settings are chosen:
(a) We randomly select $t \in [1, 20]$ features from the feature pool for one class, and only one feature for the other  The original balanced movie dataset is used (positive:negative=1:1) 
(b) Similar to (a), but the dataset is unbalanced, obtained by randomly removing 75% positive documents (positive:negative=1:4) 
As shown in Figure 1 , Maximum entropy principle shows improvement only on the balanced case
Context 3: Introduction
The cognitive processes involved in human language comprehension are complex and only partially identified  According to the dual-stream model of speech comprehension BIBREF1 , sound waves are first converted to phoneme-like features and further processed by a ventral stream that maps those features onto words and semantic structures, and a dorsal stream that (among other things) supports audio-short term memory  The mapping of words onto meaning is thought to be subserved by widely distributed regions of the brain that specialize in particular modalities — for example visual aspects of the word banana reside in the occipital lobe of the brain and are activated when the word banana is heard BIBREF2 — and the different representation modalities are thought to be integrated into a single coherent latent representation in the anterior temporal lobe BIBREF3   While this part of meaning representation in human language comprehension is somewhat understood, much less is known about how the meanings of words are integrated together to form the meaning of sentences and discourses  One tool researchers use to study the integration of meaning across words is electroencephelography (EEG), which measures the electrical activity of large numbers of neurons acting in concert  EEG has the temporal resolution necessary to study the processes involved in meaning integration, and certain stereotyped electrical responses to word presentations, known as event-related potentials (ERPs), have been identified with some of the processes thought to contribute to comprehension 
In this work, we consider six ERP components that have been associated in the cognitive neuroscience and psycholinguistics literature with language processing and which we analyze in the data from BIBREF0 (see Figure FIGREF1 for spatial and temporal definitions of these ERP components)  Three of these — the N400, EPNP, and PNP responses — are primarily considered markers for semantic processing, while the other three — the P600, ELAN, and LAN responses — are primarily considered markers for syntactic processing  However, the neat division of the ERP responses into either semantic or syntactic categories is controversial  The N400 response has been very well studied (for an overview see BIBREF4 ) and it is well established that it is associated with semantic complexity, but the features of language that trigger the other ERP responses we consider here are poorly understood  We propose to use a neural network pretrained as a language model to probe what features of language drive these ERP responses, and in turn to probe what features of language mediate the cognitive processes that underlie human language comprehension, and especially the integration of meaning across words 
Background
While a full discussion of each ERP component and the features of language thought to trigger each are beyond the scope of this document (for reviews see e g  BIBREF0 , BIBREF2 , BIBREF4 , BIBREF5 , and BIBREF6 ), we introduce some basic features of ERP components to help in the discussion later  ERP components are electrical potential responses measured with respect to a baseline that are triggered by an event (in our case the presentation of a new word to a participant in an experiment)  The name of each ERP component reflects whether the potential is positive or negative relative to the baseline  The N400 is so-named because it is Negative relative to a baseline (the baseline is typically recorded just before a word is presented at an electrode that is not affected by the ERP response) and because it peaks in magnitude at about 400ms after a word is presented to a participant in an experiment  The P600 is Positive relative to a baseline and peaks around 600ms after a word is presented to a participant (though its overall duration is much longer and less specific in time than the N400)  The post-N400 positivity is so-named because it is part of a biphasic response; it is a positivity that occurs after the negativity associated with the N400  The early post-N400 positivity (EPNP) is also part of a biphasic response, but the positivity has an eariler onset than the standard PNP  Finally, the LAN and ELAN are the left-anterior negativity and early left-anterior negativity respectively  These are named for their timing, spatial distribution on the scalp, and direction of difference from the baseline  It is important to note that ERP components can potentially cancel and mask each other, and that it is difficult to precisely localize the neural activity that causes the changes in electrical potential at the electrodes where those changes are measured 
Related Work
This work is most closely related to the paper from which we get the ERP data: BIBREF0   In that work, the authors relate the surprisal of a word, i e  the (negative log) probability of the word appearing in its context, to each of the ERP signals we consider here  The authors do not directly train a model to predict ERPs  Instead, models of the probability distribution of each word in context are used to compute a surprisal for each word, which is input into a mixed effects regression along with word frequency, word length, word position in the sentence, and sentence position in the experiment  The effect of the surprisal is assessed using a likelihood-ratio test  In BIBREF7 , the authors take an approach similar to BIBREF0   The authors compare the explanatory power of surprisal (as computed by an LSTM or a Recurrent Neural Network Grammar (RNNG) language model) to a measure of syntactic complexity they call “distance" that counts the number of parser actions in the RNNG language model  The authors find that surprisal (as predicted by the RNNG) and distance are both significant factors in a mixed effects regression which predicts the P600, while the surprisal as computed by an LSTM is not  Unlike BIBREF0 and BIBREF7 , we do not use a linking function (e g  surprisal) to relate a language model to ERPs

ground_truth: GM$\_$KL achieves better correlation than existing approaches for various metrics on SCWS dataset.

prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. What are the three regularization terms?
Context 2:  An obvious reason is that maximum entropy only favors uniform distribution 
Incorporating Neutral features performs similarly to maximum entropy since we assume that neutral words are uniformly distributed  Its accuracy decreases slowly when the number of labeled features becomes larger ( $t>4$ ) (Figure 1 (a)), suggesting that the model gradually biases to the class with more labeled features, just like GE-FL 
Incorporating the KL divergence of class distribution performs much better than GE-FL on both balanced and unbalanced datasets  This shows that it is effective to control the unbalance in labeled features and in the dataset 
With Balanced Labeled Features
We also compare with the baseline when the labeled features are balanced  Similar to the experiment above, the labeled features are obtained by information gain  Two settings are experimented with:
(a) We randomly select $t \in [1, 20]$ features from the feature pool for each class, and conduct comparisons on the original balanced movie dataset (positive:negtive=1:1) 
(b) Similar to (a), but the class distribution is unbalanced, by randomly removing 75% positive documents (positive:negative=1:4) 
Results are shown in Figure 2   When the dataset is balanced (Figure 2 (a)), there is little difference between GE-FL and our methods  The reason is that the proposed regularization terms provide no additional knowledge to the model and there is no bias in the labeled features  On the unbalanced dataset (Figure 2 (b)), incorporating KL divergence is much better than GE-FL since we provide additional knowledge(the true class distribution), but maximum entropy and neutral features are much worse because forcing the model to approach the uniform distribution misleads it 
With Unbalanced Class Distributions
Our methods are also evaluated on datasets with different unbalanced class distributions  We manually construct several movie datasets with class distributions of 1:2, 1:3, 1:4 by randomly removing 50%, 67%, 75% positive documents  The original balanced movie dataset is used as a control group  We test with both balanced and unbalanced labeled features  For the balanced case, we randomly select 10 features from the feature pool for each class, and for the unbalanced case, we select 10 features for one class, and 1 feature for the other  Results are shown in Figure 3  
Figure 3 (a) shows that when the dataset and the labeled features are both balanced, there is little difference between our methods and GE-FL(also see Figure 2 (a))  But when the class distribution becomes more unbalanced, the difference becomes more remarkable  Performance of neutral features and maximum entropy decrease significantly but incorporating KL divergence increases remarkably  This suggests if we have more accurate knowledge about class distribution, KL divergence can guide the model to the right direction 
Figure 3 (b) shows that when the labeled features are unbalanced, our methods significantly outperforms GE-FL  Incorporating KL divergence is robust enough to control unbalance both in the dataset and in labeled features while the other three methods are not so competitive 
The Influence of λ\lambda 
We present the influence of $\lambda $ on the method that incorporates KL divergence in this section  Since we simply set $\lambda = \beta |K|$ , we just tune $\beta $ here  Note that when $\beta = 0$ , the newly introduced regularization term is disappeared, and thus the model is actually GE-FL  Again, we test the method with different $\lambda $ in two settings:
(a) We randomly select $t \in [1, 20]$ features from the feature pool for one class, and only one feature for the other class  The original balanced movie dataset is used (positive:negative=1:1) 
(b) Similar to (a), but the dataset is unbalanced, obtained by randomly removing 75% positive documents (positive:negative=1:4) 
Results are shown in Figure 4   As expected, $\lambda $ reflects how strong the regularization is  The model tends to be closer to our preferences with the increasing of $\lambda $ on both cases 
Using LDA Selected Features
We compare our methods with GE-FL on all the 9 datasets in this section  Instead of using features obtained by information gain, we use LDA to select labeled features  Unlike information gain, LDA does not employ any instance labels to find labeled features  In this setting, we can build classification models without any instance annotation, but just with labeled features 
Table 1 shows that our three methods significantly outperform GE-FL  Incorporating neutral features performs better than GE-FL on 7 of the 9 datasets, maximum entropy is better on 8 datasets, and KL divergence better on 7 datasets 
LDA selects out the most predictive features as labeled features without considering the balance among classes  GE-FL does not exert any control on such an issue, so the performance is severely suffered  Our methods introduce auxiliary regularization terms to control such a bias problem and thus promote the model significantly 
Related Work
There have been much work that incorporate prior knowledge into learning, and two related lines are surveyed here  One is to use prior knowledge to label unlabeled instances and then apply a standard learning algorithm  The other is to constrain the model directly with prior knowledge 
Liu et al text manually labeled features which are highly predictive to unsupervised clustering assignments and use them to label unlabeled data  Chang et al guiding proposed constraint driven learning  They first used constraints and the learned model to annotate unlabeled instances, and then updated the model with the newly labeled data  Daumé daume2008cross proposed a self training method in which several models are trained on the same dataset, and only unlabeled instances that satisfy the cross task knowledge constraints are used in the self training process 
MaCallum et al gec proposed generalized expectation(GE) criteria which formalised the knowledge as constraint terms about the expectation of the model into the objective function Graça et al pr proposed posterior regularization(PR) framework which projects the model's posterior onto a set of distributions that satisfy the auxiliary constraints  Druck et al ge-fl explored constraints of labeled features in the framework of GE by forcing the model's predicted feature distribution to approach the reference distribution  Andrzejewski et al andrzejewski2011framework proposed a framework in which general domain knowledge can be easily incorporated into LDA  Altendorf et al altendorf2012learning explored monotonicity constraints to improve the accuracy while learning from sparse data  Chen et al chen2013leveraging tried to learn comprehensible topic models by leveraging multi-domain knowledge 
Mann and McCallum simple,generalized incorporated not only labeled features but also other knowledge like class distribution into the objective function of GE-FL  But they discussed only from the semi-supervised perspective and did not investigate into the robustness problem, unlike what we addressed in this paper 
There are also some active learning methods trying to use prior knowledge  Raghavan et al feedback proposed to use feedback on instances and features interlacedly, and demonstrated that feedback on features boosts the model much  Druck et al active proposed an active learning method which solicits labels on features rather than on instances and then used GE-FL to train the model 
Conclusion and Discussions
This paper investigates into the problem of how to leverage prior knowledge robustly in learning models  We propose three regularization terms on top of generalized expectation criteria  As demonstrated by the experimental results, the performance can be considerably improved when taking into account these factors  Comparative results show that our proposed methods is more effective and works more robustly against baselines  To the best of our knowledge, this is the first work to address the robustness problem of leveraging knowledge, and may inspire other research 
We then present more detailed discussions about the three regularization methods  Incorporating neutral features is the simplest way of regularization, which doesn't require any modification of GE-FL but just finding out some common features  But as Figure 1 (a) shows, only using neutral features are not strong enough to handle extremely unbalanced labeled features 
The maximum entropy regularization term shows the strong ability of controlling unbalance 
This method doesn't need any extra knowledge, and is thus suitable when we know nothing about the corpus  But this method assumes that the categories are uniformly distributed, which may not be the case in practice, and it will have a degraded performance if the assumption is violated (see Figure 1 (b), Figure 2 (b), Figure 3 (a)) 
The KL divergence performs much better on unbalanced corpora than other methods  The reason is that KL divergence utilizes the reference class distribution and doesn't make any assumptions  The fact suggests that additional knowledge does benefit the model 
However, the KL divergence term requires providing the true class distribution  Sometimes, we may have the exact knowledge about the true distribution, but sometimes we may not  Fortunately, the model is insensitive to the true distribution and therefore a rough estimation of the true distribution is sufficient  In our experiments, when the true class distribution is 1:2, where the reference class distribution is set to 1:1 5/1:2/1:2 5, the accuracy is 0 755/0 756/0 760 respectively  This provides us the possibility to perform simple computing on the corpus to obtain the distribution in reality  Or, we can set the distribution roughly with domain expertise 

ground_truth: a regularization term associated with neutral features, the maximum entropy of class distribution regularization term, the KL divergence between reference and predicted class distribution

a regularization term associated with neutral features,  the maximum entropy of class distribution, KL divergence between reference and predicted class distribution

prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. What datasets are used?
Context 3:  We thus lose the interpretability provided by the linking function, but we are able to predict a significant proportion of the variance for all of the ERP components, where prior work could not  We interpret our results through characterization of the ERPs in terms of how they relate to each other and to eye-tracking data rather than through a linking function  The authors in BIBREF8 also use a recurrent neural network to predict neural activity directly  In that work the authors predict magnetoencephalography (MEG) activity, a close cousin to EEG, recorded while participants read a chapter of Harry Potter and the Sorcerer’s Stone BIBREF9   Their approach to characterization of processing at each MEG sensor location is to determine whether it is best predicted by the context vector of the recurrent network (prior to the current word being processed), the embedding of the current word, or the probability of the current word given the context  In future work we also intend to add these types of studies to the ERP predictions 
Discussion
In this work we find that all six of the ERP components from BIBREF0 can be predicted above chance by a model which has been pretrained using a language modeling objective and then directly trained to predict the components  This is in contrast to prior work which has successfully linked language models to the N400 BIBREF0 and P600 BIBREF7 but not the other ERP components  We also note that contrary to BIBREF7 , we find that an LSTM does contain information that can be used to predict EEG data, and in particular that it can predict the P600  We speculate that the analysis used in BIBREF7 did not find reliable effects because the language models were related to the EEG data through functions chosen a priori (the surprisal, and the `distance' metric)  These functions, though interpretable, might be interpretable at the cost of losing much of the information in the representations learned by the network 
In addition, we show through our multitask learning analysis that information is shared between ERP components, and between ERP components and behavioral data  Although these relationships must be viewed with caution until they can be verified across multiple datasets and with more variation in neural network architectures, here we consider some potential reasons for our findings  The broad point we wish to make is that by better understanding which ERP components share information with each other and with behavioral data through the type of analysis we present here (multitask learning) or other means, we can better understand what drives each ERP component and in turn the processes involved in human language comprehension 
Conclusion
We have shown that ERP components can be predicted from neural networks pretrained as language models and fine-tuned to directly predict those components  To the best of our knowledge, prior work has not successfully used statistical models to predict all of these components  Furthermore, we have shown that multitask learning benefits the prediction of ERP components and can suggest how components relate to each other  At present, these joint-training benefit relationships are only suggestive, but if these relationships ultimately lead to insights about what drives each ERP component, then the components become more useful tools for studying human language comprehension  By using multitask learning as a method of characterization, we have found some expected relationships (LAN+P600 and ELAN+P600) and several more surprising relationships  We believe that this is exactly the kind of finding that makes multitask learning an interesting exploratory technique in this area  Additionally, we have shown that information can be shared between heterogeneous types of data (eye-tracking, self-paced reading, and ERP components) in the domain of human language processing prediction, and in particular between behavioral and neural data  Given the small datasets associated with human language processing, using heterogeneous data is a potentially major advantage of a multitask approach  In future work, we will further explore what information is encoded into the model representations when neural and behavioral data are used to train neural networks, and how these representations differ from the representations in a model trained on language alone 
Acknowledgments
We thank our reviewers for their valuable feedback  This work is supported in part by National Institutes of Health grant number U01NS098969 
Appendix
Here we present a visualization (Figure FIGREF21 ) of the results presented in Table TABREF9 of the main paper, and a visualization (Figure FIGREF22 ) of a more complete set of results from which the information in Table TABREF16 of the main paper is drawn  We also show supplemental results for variants of our primary analysis on multitask learning with eye-tracking, self-paced reading time and ERP data  In the variants we modify the input representation to our decoder network to see whether the relationships between the behavioral data and neural activity appear to be consistent with different choices of encoder architectures  Additional (and more varied) choices or architectures are left to future work  The results in Table TABREF23 reflect using only the forward-encoder (rather than the bi-LSTM) in the encoder network, while the results in Table TABREF24 reflect using only the word embeddings (i e  bypassing the LSTM entirely)  While the results are clearly worse for each of these choices of architecture than for using a bi-LSTM encoder, the relationships between the behavioral data and the ERP signals is qualitatively similar  Finally, TABREF25 shows the Pearson correlation coefficient between different measures  We note that the patterns of correlation are different than the patterns of which measures benefit from joint training with each other 

ground_truth: Answer with content missing: (Whole Method and Results sections) The primary dataset we use is the ERP data collected and computed by Frank et al. (2015), and we also use behavioral data (eye-tracking data and self-paced reading times) from Frank et al. (2013) which were collected on the same set of 205 sentences.
Select:
- ERP data collected and computed by Frank et al. (2015)
- behavioral data (eye-tracking data and self-paced reading times) from Frank et al. (2013)

the ERP data: BIBREF0