prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. What was the previous state of the art for this task?
Context 1: Introduction
Massive Open Online Courses (MOOCs) have strived to bridge the social gap in higher education by bringing quality education from reputed universities to students at large  Such massive scaling through online classrooms, however, disrupt co-located, synchronous two-way communication between the students and the instructor 
MOOC platforms provide discussion forums for students to talk to their classmates about the lectures, homeworks, quizzes and provide a venue to socialise  Instructors (defined here as the course instructors, their teaching assistants and the MOOC platform's technical staff) monitor the discussion forum to post (reply to their message) in discussion threads among students  We refer to this posting as intervention, following prior work BIBREF0   However, due to large student enrolment, the student–instructor ratio in MOOCs is very high Therefore, instructors are not able to monitor and participate in all student discussions  To address this problem, a number of works have proposed systems e g , BIBREF0 , BIBREF1 to aid instructors to selectively intervene on student discussions where they are needed the most 
In this paper, we improve the state-of-the-art for instructor intervention in MOOC forums  We propose the first neural models for this prediction problem  We show that modelling the thread structure and the sequence of posts explicitly improves performance  Instructors in different MOOCs from different subject areas intervene differently  For example, on a Science, Technology, Engineering and Mathematics (STEM) MOOC, instructors may often intervene early as possible to resolve misunderstanding of the subject material and prevent confusion  However, in a Humanities MOOC, instructors allow for the students to explore open-ended discussions and debate among themselves  Such instructors may prefer to intervene later in the discussion to encourage further discussion or resolve conflicts among students  We therefore propose attention models to infer the latent context, i e , the series of posts that trigger an intervention  Earlier studies on MOOC forum intervention either model the entire context or require the context size to be specified explicitly 
Problem Statement
A thread INLINEFORM0 consists of a series of posts INLINEFORM1 through INLINEFORM2 where INLINEFORM3 is an instructor's post when INLINEFORM4 is intervened, if applicable  INLINEFORM5 is considered intervened if an instructor had posted at least once  The problem of predicting instructor intervention is cast as a binary classification problem  Intervened threads are predicted as 1 given while non-intervened threads are predicted as 0 given posts INLINEFORM6 through INLINEFORM7  
The primary problem leads to a secondary problem of inferring the appropriate amount of context to intervene  We define a context INLINEFORM0 of a post INLINEFORM1 as a series of linear contiguous posts INLINEFORM2 through INLINEFORM3 where INLINEFORM4   The problem of inferring context is to identify context INLINEFORM5 from a set of candidate contexts INLINEFORM6  
Modelling Context in Forums
Context has been used and modelled in various ways for different problems in discussion forums  In a work on a closely related problem of forum thread retrieval BIBREF2 models context using inter-post discourse e g , Question-Answer  BIBREF3 models the structural dependencies and relationships between forum posts using a conditional random field in their problem to infer the reply structure  Unlike BIBREF2 , BIBREF3 can be used to model any structural dependency and is, therefore, more general  In this paper, we seek to infer general dependencies between a reply and its previous context whereas BIBREF3 inference is limited to pairs of posts  More recently BIBREF4 proposed a context based model which factorises attention over threads of different lengths  Differently, we do not model length but the context before a post  However, our attention models cater to threads of all lengths 
BIBREF5 proposed graph structured LSTM to model the explicit reply structure in Reddit forums  Our work does not assume access to such a reply structure because 1) Coursera forums do not provide one and 2) forum participants often err by posting their reply to a different post than that they intended  At the other end of the spectrum are document classification models that do not assume structure in the document layout but try to infer inherent structure in the natural language, viz, words, sentences, paragraphs and documents  Hierarchical attention BIBREF6 is a well know recent work that classifies documents using a multi-level LSTMs with attention mechanism to select important units at each hierarchical level  Differently, we propose a hierarchical model that encodes layout hierarchy between a post and a thread but also infers reply structure using a attention mechanism since the layout does not reliably encode it 
Instructor Intervention in MOOC forums
The problem of predicting instructor intervention in MOOCs was proposed by BIBREF0   Later BIBREF7 evaluated baseline models by BIBREF0 over a larger corpus and found the results to vary widely across MOOCs  Since then subsequent works have used similar diverse evaluations on the same prediction problem BIBREF1 , BIBREF8   BIBREF1 proposed models with discourse features to enable better prediction over unseen MOOCs  BIBREF8 recently showed interventions on Coursera forums to be biased by the position at which a thread appears to an instructor viewing the forum interface and proposed methods for debiased prediction 
While all works since BIBREF0 address key limitations in this line of research, they have not investigated the role of structure and sequence in the threaded discussion in predicting instructor interventions  BIBREF0 proposed probabilistic graphical models to model structure and sequence  They inferred vocabulary dependent latent post categories to model the thread sequence and infer states that triggered intervention  Their model, however, requires a hyperparameter for the number of latent states  It is likely that their empirically reported setting will not generalise due to their weak evaluation BIBREF7   In this paper, we propose models to infer the context that triggers instructor intervention that does not require context lengths to be set apriori  All our proposed models generalise over modelling assumptions made by BIBREF0  
For the purpose of comparison against a state-of-the-art and competing baselines we choose BIBREF7 since BIBREF0 's system and data are not available for replication 
Data and Preprocessing
We evaluate our proposed models over a corpus of 12 MOOC iterations (offerings) on Coursera org In partnership with Coursera and in line with its Terms of Service, we obtained the data for use in our academic research  Following prior work BIBREF7 we evaluate over a diverse dataset to represent MOOCs of varying sizes, instructor styles, instructor team sizes and number of threads intervened  We only include threads from sub-forums on Lecture, Homework, Quiz and Exam  We also normalise and label sub-forums with other non-standard names (e g , Assignments instead of Homework) into of the four said sub-forums  Threads on general discussion, meet and greet and other custom sub-forums for social chitchat are omitted as our focus is to aid instructors on intervening on discussion on the subject matter  We also exclude announcement threads and other threads started by instructors since they are not interventions  We preprocess each thread by replacing URLs, equations and other mathematical formulae and references to timestamps in lecture videos by tokens INLINEFORM0 URL INLINEFORM1 , INLINEFORM2 MATH INLINEFORM3 , INLINEFORM4 TIMEREF INLINEFORM5 respectively  We also truncate intervened threads to only include posts before the first instructor post since the instructor's and subsequent posts will bias the prediction due to the instructor's post 
Model
The key innovation of our work is to decompose the intervention prediction problem into a two-stage model that first explicitly tries to discover the proper context to which a potential intervention could be replying to, and then, predict the intervention status  This model implicitly assesses the importance (or urgency) of the existing thread's context to decide whether an intervention is necessary  For example in Figure SECREF1 , prior to the instructor's intervention, the ultimate post (Post #6) by Student 2 already acknowledged the OP's gratitude for his answer  In this regard, the instructor may have decided to use this point to summarize the entire thread to consolidate all the pertinent positions  Here, we might assume that the instructor's reply takes the entire thread (Posts #1–6) as the context for her reply 
This subproblem of inferring the context scope is where our innovation centers on  To be clear, in order to make the prediction that a instruction intervention is now necessary on a thread, the instructor's reply is not yet available — the model predicts whether a reply is necessary — so in the example, only Posts #1–6 are available in the problem setting  To infer the context, we have to decide which subsequence of posts are the most plausible motivation for an intervention 
Recent work in deep neural modeling has used an attention mechanism as a focusing query to highlight specific items within the input history that significantly influence the current decision point  Our work employs this mechanism – but with a twist: due to the fact that the actual instructor intervention is not (yet) available at the decision timing, we cannot use any actual intervention to decide the context  To employ attention, we must then employ a surrogate text as the query to train our prediction model  Our model variants model assess the suitability of such surrogate texts for the attention mechanism basis 
Congruent with the representation of the input forums, in all our proposed models, we encode the discussion thread hierarchically  We first build representations for each post by passing pre-trained word vector representations from GloVe BIBREF9 for each word through an LSTM BIBREF10 , INLINEFORM0   We use the last layer output of the LSTM as a representation of the post  We refer this as the post vector INLINEFORM1  
Then each post INLINEFORM0 is passed through another LSTM, INLINEFORM1 , whose last layer output forms the encoding of the entire thread  Hidden unit outputs of INLINEFORM2 represent the contexts INLINEFORM3 ; that is, snapshots of the threads after each post, as shown in Figure FIGREF1  
The INLINEFORM0 and INLINEFORM1 together constitute the hierarchical LSTM (hLSTM) model  This general hLSTM model serves as the basis for our model exploration in the rest of this section 
Contextual Attention Models
When they intervene, instructors either pay attention to a specific post or a series of posts, which trigger their reply  However, instructors rarely explicitly indicate to which post(s) their intervention is in relation to  This is the case in our corpus, party due to Coursera's user interface which only allows for single level comments (see Figure FIGREF2 )  Based solely on the binary, thread-level intervention signal, our secondary objective seeks to infer the appropriate context – represented by a sequence of posts – as the basis for the intervention 
We only consider linear contiguous series of posts starting with the thread's original post to constitute to a context; e g , INLINEFORM0   This is a reasonable as MOOC forum posts always reply to the original post or to a subsequent post, which in turn replies to the original post  This is in contrast to forums such as Reddit that have a tree or graph-like structure that require forum structure to be modelled explicitly, such as in BIBREF5  
We propose three neural attention BIBREF11 variants based on how an instructor might attend and reply to a context in a thread: the ultimate, penultimate and any post attention models  We review each of these in turn 
Ultimate Post Attention (UPA) Model  In this model we attend to the context represented by hidden state of the INLINEFORM0   We use the post prior to the instructor's reply as a query over the contexts INLINEFORM1 to compute attention weights INLINEFORM2 , which are then used to compute the attended context representation INLINEFORM3 (recall again that the intervention text itself is not available for this purpose)  This attention formulation makes an equivalence between the final INLINEFORM4 post and the prospective intervention, using Post INLINEFORM5 as the query for finding the appropriate context INLINEFORM6 , inclusive of itself INLINEFORM7   Said in another way, UPA uses the most recent content in the thread as the attentional query for context 
For example, if post INLINEFORM0 is the instructor's reply, post INLINEFORM1 will query over the contexts INLINEFORM2 and INLINEFORM3   The model schematic is shown in Figure FIGREF12  
The attended context representations are computed as: DISPLAYFORM0
The INLINEFORM0 representation is then passed through a fully connected softmax layer to yield the binary prediction 
Penultimate Post Attention (PPA) Model  While the UPA model uses the most recent text and makes the ultimate post itself available as potential context, our the ultimate post may be better modeled as having any of its prior posts as potential context  Penultimate Post Attention (PPA) variant does this  The schematic and the equations for the PPA model are obtained by summing over contexts INLINEFORM0 in Equation EQREF10 and Figure FIGREF12   While we could properly model such a context inference decision with any post INLINEFORM1 and prospective contexts INLINEFORM2 (where INLINEFORM3 is a random post), it makes sense to use the penultimate post, as we can make the most information available to the model for the context inference 
The attended context representations are computed as: DISPLAYFORM0
Any Post Attention (APA) Model  APA further relaxes both UPA and PPA, allowing APA to generalize and hypothesize that the prospective instructor intervention is based on the context that any previous post INLINEFORM0 replied to  In this model, each post INLINEFORM1 is set as a query to attend to its previous context INLINEFORM2   For example, INLINEFORM3 will attend to INLINEFORM4   Different from standard attention mechanisms, APA attention weights INLINEFORM5 are obtained by normalising interaction matrix over the different queries 
In APA, the attention context INLINEFORM0 is computed via: DISPLAYFORM0
Evaluation
The baseline and the models are evaluated on a corpus of 12 MOOC discussion forums  We train on 80% of the training data and report evaluation results on the held-out 20% of test data  We report INLINEFORM0 scores on the positive class (interventions), in line with prior work  We also argue that recall of the positive class is more important than precision, since it is costlier for instructors to miss intervening on a thread than spending irrelevant time intervening on a less critical threads due to false positives 
Model hyperpameter settings  All proposed and baseline neural models are trained using Adam optimizer with a learning rate of 0 001  We used cross-entropy as loss function  Importantly we updated the model parameters during training after each instance as in vanilla stochastic gradient descent; this setting was practical since data on most courses had only a few hundred instances enabling convergence within a reasonable training time of a few hours (see Table TABREF15 , column 2)  Models were trained for a single epoch as most of our courses with a few hundred thread converged after a single epoch  We used 300-dimensional GloVe vectors and permitted the embeddings to be updated during the model's end-to-end training  The hidden dimension size of both INLINEFORM0 and INLINEFORM1 are set to 128 for all the models 
Baselines  We compare our models against a neural baseline models, hierarchical LSTM (hLSTM), with the attention ablated but with access to the complete context, and a strong, open-sourced feature-rich baseline BIBREF7   We choose BIBREF7 over other prior works such as BIBREF0 since we do not have access to the dataset or the system used in their papers for replication  BIBREF7 is a logistic regression classifier with features inclusive of bag-of-words representation of the unigrams and thread length, normalised counts of agreements to previous posts, counts of non-lexical reference items such as URLs, and the Coursera forum type in which a thread appeared  We also report aggregated results from a hLSTM model with access only to the last post as context for comparison  Table TABREF17 compares the performance of these baselines against our proposed methods 
Results
Table TABREF15 shows performance of all our proposed models and the neural baseline over our 12 MOOC dataset  Our models of UPA, PPA individually better the baseline by 5 and 2% on INLINEFORM0 and 3 and 6% on recall respectively  UPA performs the best in terms of INLINEFORM1 on average while PPA performs the best in terms of recall on average  At the individual course level, however, the results are mixed  UPA performs the best on INLINEFORM2 on 5 out of 12 courses, PPA on 3 out 12 courses, APA 1 out of 12 courses and the baseline hLSTM on 1  PPA performs the best on recall on 7 out of the 12 courses  We also note that course level performance differences correlate with the course size and intervention ratio (hereafter, i ratio), which is the ratio of intervened to non-intervened threads  UPA performs better than PPA and APA on low intervention courses (i ratio INLINEFORM3 0 25) mainly because PPA and APA's performance drops steeply when i ratio drops (see col 2 parenthesis and INLINEFORM4 of PPA and APA)  While all the proposed models beat the baseline on every course except casebased-2  On medicalneuro-2 and compilers-4 which have the lowest i ratio among the 12 courses none of the neural models better the reported baseline BIBREF7 (course level not scores not shown in this paper)  The effect is pronounced in compilers-4 course where none of the neural models were able to predict any intervened threads  This is due to the inherent weakness of standard neural models, which are unable to learn features well enough when faced with sparse data 
The best performance of UPA indicates that the reply context of the instructor's post INLINEFORM0 correlates strongly with that of the previous post INLINEFORM1   This is not surprising since normal conversations are typically structured that way 
Discussion
In order to further understand the models' ability to infer the context and its effect on intervention prediction, we further investigate the following research questions 
RQ1  Does context inference help intervention prediction?
In order to understand if context inference is useful to intervention prediction, we ablate the attention components and experiment with the vanilla hierarchical LSTM model  Row 3 of Table TABREF17 shows the macro averaged result from this experiment  The UPA and PPA attention models better the vanilla hLSTM by 5% and 2% on average in INLINEFORM0 respectively  Recall that the vanilla hLSTM already has access to a context consisting of all posts (from INLINEFORM1 through INLINEFORM2 )  In contrast, the UPA and PPA models selectively infers a context for INLINEFORM3 and INLINEFORM4 posts, respectively, and use it to predict intervention  The improved performance of our attention models that actively select their optimal context, over a model with the complete thread as context, hLSTM, shows that the context inference improves intervention prediction over using the default full context 
RQ2  How well do the models perform across threads of different lengths? To understand the models' prediction performance across threads of different lengths, we bin threads by length and study the models' recall  We choose three courses, ml-5, rprog-3 and calc-1, from our corpus of 12 with the highest number of positive instances ( INLINEFORM0 100 threads)  We limit our analysis to these since binning renders courses with fewer positive instances sparse  Figure FIGREF18 shows performance across thread lengths from 1 through 7 posts and INLINEFORM1 posts  Clearly, the UPA model performs much better on shorter threads than on longer threads while PPA and APA works better on longer threads  Although, UPA is the best performing model in terms of overall INLINEFORM2 its performance drops steeply on threads of length INLINEFORM3   UPA's overall best performance is because most of the interventions in the corpus happen after one post  To highlight the performance of APA we show an example from smac-1 in Figure FIGREF22 with nine posts which was predicted correctly as intervened by APA but not by other models  Threads shows students confused over a missing figure in a homework  The instructor finally shows up, though late, to resolve the confusion 
RQ3  Do models trained with different context lengths perform better than when trained on a single context length?
We find that context length has a regularising effect on the model's performance at test time  This is not surprising since models trained with threads of single context length will not generalise to infer different context lengths  Row 4 of Table TABREF17 shows a steep performance drop in training by classifier with all threads truncated to a context of just one post, INLINEFORM0 , the post immediately preceding the intervened post  We also conducted an experiment with a multi-objective loss function with an additive cross-entropy term where each term computes loss from a model with context limited to a length of 3  We chose 3 since intervened threads in all the courses had a median length between 3 and 4  We achieved an INLINEFORM1 of 0 45 with a precision of 0 47 and recall of 0 43  This achieves a performance comparable to that of the BIBREF7 with context length set to only to 3  This approach of using infinitely many loss terms for each context length from 1 through the maximum thread length in a course is naive and not practical  We only use this model to show the importance of training the model with loss from threads of different lengths to prevent models overfitting to threads of specific context lengths 
Conclusion
We predict instructor intervention on student discussions by first inferring the optimal size of the context needed to decide on the intervention decision for the intervened post  We first show that a structured representation of the complete thread as the context is better than a bag-of-words, feature-rich representation  We then propose attention-based models to infer and select a context – defined as a contiguous subsequence of student posts – to improve over a model that always takes the complete thread as a context to prediction intervention  Our Any Post Attention (APA) model enables instructors to tune the model to predict intervention early or late  We posit our APA model will enable MOOC instructors employing varying pedagogical styles to use the model equally well  We introspect the attention models' performance across threads of varying lengths and show that APA predicts intervention on longer threads, which possesses more candidate contexts, better 
We note that the recall of the predictive models for longer threads (that is, threads of length greater 2) can still be improved  Models perform differently between shorter and longer length  An ensemble model or a multi-objective loss function is thus planned in our future work to better prediction on such longer threads 
Context 2: Introduction
Language modelling in its inception had one-hot vector encoding of words  However, it captures only alphabetic ordering but not the word semantic similarity  Vector space models helps to learn word representations in a lower dimensional space and also captures semantic similarity  Learning word embedding aids in natural language processing tasks such as question answering and reasoning BIBREF0, stance detection BIBREF1, claim verification BIBREF2 
Recent models BIBREF3, BIBREF4 work on the basis that words with similar context share semantic similarity  BIBREF4 proposes a neural probabilistic model which models the target word probability conditioned on the previous words using a recurrent neural network  Word2Vec models BIBREF3 such as continuous bag-of-words (CBOW) predict the target word given the context, and skip-gram model works in reverse of predicting the context given the target word  While, GloVe embeddings were based on a Global matrix factorization on local contexts BIBREF5  However, the aforementioned models do not handle words with multiple meanings (polysemies) 
BIBREF6 proposes a neural network approach considering both local and global contexts in learning word embeddings (point estimates)  Their multiple prototype model handles polysemous words by providing apriori heuristics about word senses in the dataset  BIBREF7 proposes an alternative to handle polysemous words by a modified skip-gram model and EM algorithm  BIBREF8 presents a non-parametric based alternative to handle polysemies  However, these approaches fail to consider entailment relations among the words  BIBREF9 learn a Gaussian distribution per word using the expected likelihood kernel  However, for polysemous words, this may lead to word distributions with larger variances as it may have to cover various senses 
BIBREF10 proposes multimodal word distribution approach  It captures polysemy  However, the energy based objective function fails to consider asymmetry and hence entailment  Textual entailment recognition is necessary to capture lexical inference relations such as causality (for example, mosquito $\rightarrow $ malaria), hypernymy (for example, dog $\models $ animal) etc 
In this paper, we propose to obtain multi-sense word embedding distributions by using a variant of max margin objective based on the asymmetric KL divergence energy function to capture textual entailment  Multi-sense distributions are advantageous in capturing polysemous nature of words and in reducing the uncertainty per word by distributing it across senses  However, computing KL divergence between mixtures of Gaussians is intractable, and we use a KL divergence approximation based on stricter upper and lower bounds  While capturing textual entailment (asymmetry), we have also not compromised on capturing symmetrical similarity between words (for example, funny and hilarious) which will be elucidated in Section $3 1$  We also show the effectiveness of the proposed approach on the benchmark word similarity and entailment datasets in the experimental section 
Methodology ::: Word Representation
Probabilistic representation of words helps one model uncertainty in word representation, and polysemy  Given a corpus $V$, containing a list of words each represented as $w$, the probability density for a word $w$ can be represented as a mixture of Gaussians with $C$ components BIBREF10 
Here, $p_{w,j}$ represents the probability of word $w$ belonging to the component $j$, $\operatorname{\mathbf {\mu }}_{w,j}$ represents $D$ dimensional word representation corresponding to the $j^{th}$ component sense of the word $w$, and $\Sigma _{w,j}$ represents the uncertainty in representation for word $w$ belonging to component $j$ 
Objective function
The model parameters (means, covariances and mixture weights) $\theta $ can be learnt using a variant of max-margin objective BIBREF11 
Here $E_\theta (\cdot , \cdot )$ represents an energy function which assigns a score to the pair of words, $w$ is a particular word under consideration, $c$ its positive context (same context), and $c^{\prime }$ the negative context  The objective aims to push the margin of the difference between the energy function of a word $w$ to its positive context $c$ higher than its negative context $c$ by a threshold of $m$  Thus, word pairs in the same context gets a higher energy than the word pairs in the dissimilar context  BIBREF10 consider the energy function to be an expected likelihood kernel which is defined as follows 
This is similar to the cosine similarity metric over vectors and the energy between two words is maximum when they have similar distributions  But, the expected likelihood kernel is a symmetric metric which will not be suitable for capturing ordering among words and hence entailment 
Objective function ::: Proposed Energy function
As each word is represented by a mixture of Gaussian distributions, KL divergence is a better choice of energy function to capture distance between distributions  Since, KL divergence is minimum when the distributions are similar and maximum when they are dissimilar, energy function is taken as exponentiated negative KL divergence 
However, computing KL divergence between Gaussian mixtures is intractable and obtaining exact KL value is not possible  One way of approximating the KL is by Monte-Carlo approximation but it requires large number of samples to get a good approximation and is computationally expensive on high dimensional embedding space 
Alternatively, BIBREF12 presents a KL approximation between Gaussian mixtures where they obtain an upper bound through product of Gaussian approximation method and a lower bound through variational approximation method  In BIBREF13, the authors combine the lower and upper bounds from approximation methods of BIBREF12 to provide a stricter bound on KL between Gaussian mixtures  Lets consider Gaussian mixtures for the words $w$ and $v$ as follows 
The approximate KL divergence between the Gaussian mixture representations over the words $w$ and $v$ is shown in equation DISPLAY_FORM8  More details on approximation is included in the Supplementary Material 
where $EL_{ik}(w,w) = \int f_{w,i} (\operatorname{\mathbf {x}}) f_{w,k} (\operatorname{\mathbf {x}}) d\operatorname{\mathbf {x}}$ and $EL_{ij}(w,v) = \int f_{w,i} (\operatorname{\mathbf {x}}) f_{v,k} (\operatorname{\mathbf {x}}) d\operatorname{\mathbf {x}}$  Note that the expected likelihood kernel appears component wise inside the approximate KL divergence derivation 
One advantage of using KL as energy function is that it enables to capture asymmetry in entailment datasets  For eg , let us consider the words 'chair' with two senses as 'bench' and 'sling', and 'wood' with two senses as 'trees' and 'furniture'  The word chair ($w$) is entailed within wood ($v$), i e  chair $\models $ wood  Now, minimizing the KL divergence necessitates maximizing $\log {\sum _j p_{v,j} \exp ({-KL(f_{w,i} (\operatorname{\mathbf {x}})||f_{v,j}(\operatorname{\mathbf {x}}))})}$ which in turn minimizes $KL(f_{w,i}(\operatorname{\mathbf {x}})||f_{v,j}(\operatorname{\mathbf {x}}))$  This will result in the support of the $i^{th}$ component of $w$ to be within the $j^{th}$ component of $v$, and holds for all component pairs leading to the entailment of $w$ within $v$  Consequently, we can see that bench $\models $ trees, bench $\models $ furniture, sling $\models $ trees, and sling $\models $ furniture  Thus, it introduces lexical relationship between the senses of child word and that of the parent word  Minimizing the KL also necessitates maximizing $\log {\sum _j {p_{v,j}} EL_{ij}(w,v)}$ term for all component pairs among $w$ and $v$  This is similar to maximizing expected likelihood kernel, which brings the means of $f_{w,i}(\operatorname{\mathbf {x}})$ and $f_{v,j}(\operatorname{\mathbf {x}})$ closer (weighted by their co-variances) as discussed in BIBREF10  Hence, the proposed approach captures the best of both worlds, thereby catering to both word similarity and entailment 
We also note that minimizing the KL divergence necessitates minimizing $\log {\sum _k p_{w,k} \exp ({-KL(f_{w,i}||f_{w,k})})}$ which in turn maximizes $KL(f_{w,i}||f_{w,k})$  This prevents the different mixture components of a word converging to single Gaussian and encourages capturing different possible senses of the word  The same is also achieved by minimizing $\sum _k {p_{w,k}} EL_{ik}(w,w)$ term and act as a regularization term which promotes diversity in learning senses of a word 
Experimentation and Results
We train our proposed model GM$\_$KL (Gaussian Mixture using KL Divergence) on the Text8 dataset BIBREF14 which is a pre-processed data of $17M$ words from wikipedia  Of which, 71290 unique and frequent words are chosen using the subsampling trick in BIBREF15  We compare GM$\_$KL with the previous approaches w2g BIBREF9 ( single Gaussian model) and w2gm BIBREF10 (mixture of Gaussian model with expected likelihood kernel)  For all the models used for experimentation, the embedding size ($D$) was set to 50, number of mixtures to 2, context window length to 10, batch size to 128  The word embeddings were initialized using a uniform distribution in the range of $[-\sqrt{\frac{3}{D}}$, $\sqrt{\frac{3}{D}}]$ such that the expectation of variance is 1 and mean 0 BIBREF16  One could also consider initializing the word embeddings using other contextual representations such as BERT BIBREF17 and ELMo BIBREF18 in the proposed approach  In order to purely analyze the performance of $\emph {GM\_KL}$ over the other models, we have chosen initialization using uniform distribution for experiments  For computational benefits, diagonal covariance is used similar to BIBREF10  Each mixture probability is constrained in the range $[0,1]$, summing to 1 by optimizing over unconstrained scores in the range $(-\infty ,\infty )$ and converting scores to probability using softmax function  The mixture scores are initialized to 0 to ensure fairness among all the components  The threshold for negative sampling was set to $10^{-5}$, as recommended in BIBREF3  Mini-batch gradient descent with Adagrad optimizer BIBREF19 was used with initial learning rate set to $0 05$ 
Table TABREF9 shows the qualitative results of GM$\_$KL  Given a query word and component id, the set of nearest neighbours along with their respective component ids are listed  For eg , the word `plane' in its 0th component captures the `geometry' sense and so are its neighbours, and its 1st component captures `vehicle' sense and so are its corresponding neighbours  Other words such as `rock' captures both the `metal' and `music' senses, `star' captures `celebrity' and `astronomical' senses, and `phone' captures `telephony' and `internet' senses 
We quantitatively compare the performance of the GM$\_$KL, w2g, and w2gm approaches on the SCWS dataset BIBREF6  The dataset consists of 2003 word pairs of polysemous and homonymous words with labels obtained by an average of 10 human scores  The Spearman correlation between the human scores and the model scores are computed  To obtain the model score, the following metrics are used:
MaxCos: Maximum cosine similarity among all component pairs of words $w$ and $v$:
AvgCos: Average component-wise cosine similarity between the words $w$ and $v$ 
KL$\_$approx: Formulated as shown in (DISPLAY_FORM8) between the words $w$ and $v$ 
KL$\_$comp: Maximum component-wise negative KL between words $w$ and $v$:
Table TABREF17 compares the performance of the approaches on the SCWS dataset  It is evident from Table TABREF17 that GM$\_$KL achieves better correlation than existing approaches for various metrics on SCWS dataset 
Table TABREF18 shows the Spearman correlation values of GM$\_$KL model evaluated on the benchmark word similarity datasets: SL BIBREF20, WS, WS-R, WS-S BIBREF21, MEN BIBREF22, MC BIBREF23, RG BIBREF24, YP BIBREF25, MTurk-287 and MTurk-771 BIBREF26, BIBREF27, and RW BIBREF28  The metric used for comparison is 'AvgCos'  It can be seen that for most of the datasets, GM$\_$KL achieves significantly better correlation score than w2g and w2gm approaches  Other datasets such as MC and RW consist of only a single sense, and hence w2g model performs better and GM$\_$KL achieves next better performance  The YP dataset have multiple senses but does not contain entailed data and hence could not make use of entailment benefits of GM$\_$KL 
Table TABREF19 shows the evaluation results of GM$\_$KL model on the entailment datasets such as entailment pairs dataset BIBREF29 created from WordNet with both positive and negative labels, a crowdsourced dataset BIBREF30 of 79 semantic relations labelled as entailed or not and annotated distributionally similar nouns dataset BIBREF31  The 'MaxCos' similarity metric is used for evaluation and the best precision and best F1-score is shown, by picking the optimal threshold  Overall, GM$\_$KL performs better than both w2g and w2gm approaches 
Conclusion
We proposed a KL divergence based energy function for learning multi-sense word embedding distributions modelled as Gaussian mixtures  Due to the intractability of the Gaussian mixtures for the KL divergence measure, we use an approximate KL divergence function  We also demonstrated that the proposed GM$\_$KL approaches performed better than other approaches on the benchmark word similarity and entailment datasets 
tocsectionAppendices
Approximation for KL divergence between mixtures of gaussians
KL between gaussian mixtures $f_{w}(\operatorname{\mathbf {x}})$ and $f_{v}(\operatorname{\mathbf {x}})$ can be decomposed as:
BIBREF12 presents KL approximation between gaussian mixtures using
product of gaussian approximation method where KL is approximated using product of component gaussians and
variational approximation method where KL is approximated by introducing some variational parameters 
The product of component gaussian approximation method using Jensen's inequality provides upper bounds as shown in equations DISPLAY_FORM23 and  
The variational approximation method provides lower bounds as shown in equations DISPLAY_FORM24 and DISPLAY_FORM25 
where $H$ represents the entropy term and the entropy of $i^{th}$ component of word $w$ with dimension $D$ is given as
In BIBREF13, the authors combine the lower and upper bounds from approximation methods of BIBREF12 to formulate a stricter bound on KL between gaussian mixtures 
From equations DISPLAY_FORM23 and DISPLAY_FORM25, a stricter lower bound for KL between gaussian mixtures is obtained as shown in equation DISPLAY_FORM26
From equations and DISPLAY_FORM24, a stricter upper bound for KL between gaussian mixtures is obtained as shown in equation DISPLAY_FORM27
Finally, the KL between gaussian mixtures is taken as the mean of KL upper and lower bounds as shown in equation DISPLAY_FORM28 

ground_truth: hLSTM

hLSTM

prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. How does this approach compare to other WSD approaches employing word embeddings?
Context 3: Introduction
Deep Learning approaches have achieved impressive results on various NLP tasks BIBREF0 , BIBREF1 , BIBREF2 and have become the de facto approach for any NLP task  However, these deep learning techniques have found to be less effective for low-resource languages when the available training data is very less BIBREF3   Recently, several approaches like Multi-task learning BIBREF4 , multilingual learning BIBREF5 , semi-supervised learning BIBREF2 , BIBREF6 and transfer learning BIBREF7 , BIBREF3 have been explored by the deep learning community to overcome data sparsity in low-resource languages  Transfer learning trains a model for a parent task and fine-tunes the learned parent model weights (features) for a related child task BIBREF7 , BIBREF8   This effectively reduces the requirement on training data for the child task as the model would have learned relevant features from the parent task data thereby, improving the performance on the child task 
Transfer learning has also been explored in the multilingual Neural Machine Translation BIBREF3 , BIBREF9 , BIBREF10   The goal is to improve the NMT performance on the source to target language pair (child task) using an assisting source language (assisting to target translation is the parent task)  Here, the parent model is trained on the assisting and target language parallel corpus and the trained weights are used to initialize the child model  The child model can now be fine-tuned on the source-target language pairs, if parallel corpus is available  The divergence between the source and the assisting language can adversely impact the benefits obtained from transfer learning  Multiple studies have shown that transfer learning works best when the languages are related BIBREF3 , BIBREF10 , BIBREF9   Several studies have tried to address lexical divergence between the source and the target languages BIBREF10 , BIBREF11 , BIBREF12   However, the effect of word order divergence and its mitigation has not been explored  In a practical setting, it is not uncommon to have source and assisting languages with different word order  For instance, it is possible to find parallel corpora between English and some Indian languages, but very little parallel corpora between Indian languages  Hence, it is natural to use English as an assisting language for inter-Indian language translation 
To see how word order divergence can be detrimental, let us consider the case of the standard RNN (Bi-LSTM) encoder-attention-decoder architecture BIBREF13   The encoder generates contextual representations (annotation vectors) for each source word, which are used by the attention network to match the source words to the current decoder state  The contextual representation is word-order dependent  Hence, if the assisting and the source languages do not have similar word order the generated contextual representations will not be consistent  The attention network (and hence the decoder) sees different contextual representations for similar words in parallel sentences across different languages  This makes it difficult to transfer knowledge learned from the assisting language to the source language 
We illustrate this by visualizing the contextual representations generated by the encoder of an English to Hindi NMT system for two versions of the English input: (a) original word order (SVO) (b) word order of the source language (SOV, for Bengali)  Figure FIGREF1 shows that the encoder representations obtained are very different  The attention network and the decoder now have to work with very different representations  Note that the plot below does not take into account further lexical and other divergences between source and assisting languages, since we demonstrated word order divergence with the same language on the source side 
To address this word order divergence, we propose to pre-order the assisting language sentences to match the word order of the source language  We consider an extremely resource constrained scenario, where we do not have any parallel corpus for the child task  We are limited to a bilingual dictionary for transfer information from the assisting to the source language  From our experiments, we show that there is a significant increase in the translation accuracy for the unseen source-target language pair 
Addressing Lexical Divergence
BIBREF3 explored transfer learning for NMT on low-resource languages  They studied the influence of language divergence between languages chosen for training the parent and child model, and showed that choosing similar languages for training the parent and child model leads to better improvements from transfer learning  A limitation of BIBREF3 approach is that they ignore the lexical similarity between languages and also the source language embeddings are randomly initialized  BIBREF10 , BIBREF11 , BIBREF12 take advantage of lexical similarity between languages in their work  BIBREF10 proposed to use Byte-Pair Encoding (BPE) to represent the sentences in both the parent and the child language to overcome the above limitation  They show using BPE benefits transfer learning especially when the involved languages are closely-related agglutinative languages  Similarly, BIBREF11 utilize lexical similarity between the source and assisting languages by training a character-level NMT system  BIBREF12 address lexical divergence by using bilingual embeddings and mixture of universal token embeddings  One of the languages' vocabulary, usually English vocabulary is considered as universal tokens and every word in the other languages is represented as a mixture of universal tokens  They show results on extremely low-resource languages 
Addressing Word Order Divergence
To the best of our knowledge, no work has addressed word order divergence in transfer learning for multilingual NMT  However, some work exists for other NLP tasks that could potentially address word order  For Named Entity Recognition (NER), BIBREF14 use a self-attention layer after the Bi-LSTM layer to address word-order divergence for Named Entity Recognition (NER) task  The approach does not show any significant improvements over multiple languages  A possible reason is that the divergence has to be addressed before/during construction of the contextual embeddings in the Bi-LSTM layer, and the subsequent self-attention layer does not address word-order divergence  BIBREF15 use adversarial training for cross-lingual question-question similarity ranking in community question answering  The adversarial training tries to force the encoder representations of similar sentences from different input languages to have similar representations 
Use of Pre-ordering
Pre-ordering the source language sentences to match the target language word order has been useful in addressing word-order divergence for Phrase-Based SMT BIBREF16 , BIBREF17 , BIBREF18 , BIBREF19   Recently, BIBREF20 proposed a way to measure and reduce the divergence between the source and target languages based on morphological and syntactic properties, also termed as anisomorphism  They demonstrated that by reducing the anisomorphism between the source and target languages, consistent improvements in NMT performance were obtained  The NMT system used additional features like word forms, POS tags and dependency relations in addition to parallel corpora  On the other hand, BIBREF21 observed a drop in performance due to pre-ordering for NMT  Unlike BIBREF20 , the NMT system was trained on pre-ordered sentences and no additional features were provided to the system  Note that all these works address source-target divergence, not divergence between source languages in multilingual NMT 
Proposed Solution
Consider the task of translating from an extremely low-resource language (source) to a target language  The parallel corpus between the two languages if available may be too small to train a NMT model  Similar to existing works BIBREF3 , BIBREF10 , BIBREF12 , we use transfer learning to overcome data sparsity and train a NMT model between the source and the target languages

ground_truth: GM$\_$KL achieves better correlation than existing approaches for various metrics on SCWS dataset.

prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. How do they match words before reordering them?
Context 3:  Specifically, the NMT model (parent model) is trained on the assisting language and target language pairs  We choose English as the assisting language in all our experiments  In our resource-scarce scenario, we have no parallel corpus for the child task  Hence, at test time, the source language sentence is translated using the parent model after performing a word-by-word translation into the assisting language 
Since the source language and the assisting language (English) have different word order, we hypothesize that it leads to inconsistencies in the contextual representations generated by the encoder for the two languages  In this paper, we propose to pre-order English sentences (assisting language sentences) to match the word-order of the source language and train the parent model on this pre-ordered corpus  In our experiments, we look at scenarios where the assisting language has SVO word order and the source language has SOV word order 
For instance, consider the English sentence Anurag will meet Thakur  One of the pre-ordering rule swaps the position of the noun phrase followed by a transitive verb with the transitive verb  The original and the resulting re-ordered parse tree will be as shown in the Table TABREF5   Applying this reordering rule to the above sentence Anurag will meet Thakur will yield the reordered sentence Anurag Thakur will meet  Additionally, the Table TABREF5 shows the parse trees for the above sentence with and without pre-ordering 
Pre-ordering should also be beneficial for other word order divergence scenarios (e g , SOV to SVO), but we leave verification of these additional scenarios for future work 
Experimental Setup
In this section, we describe the languages experimented with, datasets used, the network hyper-parameters used in our experiments 
Languages
We experimented with English INLINEFORM0 Hindi translation as the parent task  English is the assisting source language  Bengali, Gujarati, Marathi, Malayalam and Tamil are the primary source languages, and translation from these to Hindi constitute the child tasks  Hindi, Bengali, Gujarati and Marathi are Indo-Aryan languages, while Malayalam and Tamil are Dravidian languages  All these languages have a canonical SOV word order 
Datasets
For training English-Hindi NMT systems, we use the IITB English-Hindi parallel corpus BIBREF22 ( INLINEFORM0 sentences from the training set) and the ILCI English-Hindi parallel corpus ( INLINEFORM1 sentences)  The ILCI (Indian Language Corpora Initiative) multilingual parallel corpus BIBREF23 spans multiple Indian languages from the health and tourism domains  We use the 520-sentence dev-set of the IITB parallel corpus for validation  For each child task, we use INLINEFORM2 sentences from ILCI corpus as the test set 
Network
We use OpenNMT-Torch BIBREF24 to train the NMT system  We use the standard sequence-to-sequence architecture with attention BIBREF13   We use an encoder which contains two layers of bidirectional LSTMs with 500 neurons each  The decoder contains two LSTM layers with 500 neurons each  Input feeding approach BIBREF1 is used where the previous attention hidden state is fed as input to the decoder LSTM  We use a mini-batch of size 50 and use a dropout layer  We begin with an initial learning rate of INLINEFORM0 and decay the learning rate by a factor of INLINEFORM1 when the perplexity on validation set increases  The training is stopped when the learning rate falls below INLINEFORM2 or number of epochs=22  The English input is initialized with pre-trained embeddings trained using fastText BIBREF25  
English vocabulary consists of INLINEFORM0 tokens appearing at least 2 times in the English training corpus  For constructing the Hindi vocabulary we considered only those tokens appearing at least 5 times in the training split resulting in a vocabulary size of INLINEFORM1 tokens  For representing English and other source languages into a common space, we translate each word in the source language into English using a bilingual dictionary (Google Translate word translation in our case)  In an end-to-end solution, it would have been ideal to use bilingual embeddings or obtain word-by-word translations via bilingual embeddings BIBREF14   But, the quality of publicly available bilingual embeddings for English-Indian languages is very low for obtaining good-quality, bilingual representations BIBREF26 , BIBREF27   We also found that these embeddings were not useful for transfer learning 
We use the CFILT-preorder system for reordering English sentences to match the Indian language word order  It contains two re-ordering systems: (1) generic rules that apply to all Indian languages BIBREF17 , and (2) hindi-tuned rules which improve the generic rules by incorporating improvements found through an error analysis of English-Hindi reordering BIBREF28   These Hindi-tuned rules have been found to improve reordering for many English to Indian language pairs BIBREF29  
Results
In this section, we describe the results from our experiments on NMT task  We report the results on X-Hindi pair, where X is one of Bengali, Gujarati, Marathi, Tamil, and Malayalam  The results are presented in the Table TABREF6   We report BLEU scores and LeBLEU scores BIBREF30   We observe that both the pre-ordering configurations significantly improve the BLEU scores over the baseline scores  We observe larger gains when generic pre-ordering rules are used compared to the Hindi-tuned pre-ordering rules 
These results support our hypothesis that word-order divergence can limit the benefits of multilingual translation  Reducing the word order divergence can improve translation in extremely low-resource scenarios 
An analysis of the outputs revealed that pre-ordering significantly reducing the number of UNK tokens (placeholder for unknown words) in the test output (Table TABREF14 )  We hypothesize that due to word order divergence between English and Indian languages, the encoder representation generated is not consistent leading to decoder generating unknown words  However, the pre-ordered models generate better contextual representations leading to less number of unknown tokens and better translation which is also reflected in the BLEU scores 
Conclusion
In this paper, we show that handling word-order divergence between source and assisting languages is crucial for the success of multilingual NMT in an extremely low-resource setting  We show that pre-ordering the assisting language to match the word order of the source language significantly improves translation quality in an extremely low-resource setting  While the current work focused on Indian languages, we would like to validate the hypothesis on a more diverse set of languages 

ground_truth: Unanswerable

CFILT-preorder system