prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. On what datasets are experiments performed?
Context 1: Introduction
Question Generation (QG) is the task of automatically creating questions from a range of inputs, such as natural language text BIBREF0, knowledge base BIBREF1 and image BIBREF2  QG is an increasingly important area in NLP with various application scenarios such as intelligence tutor systems, open-domain chatbots and question answering dataset construction  In this paper, we focus on question generation from reading comprehension materials like SQuAD BIBREF3  As shown in Figure FIGREF1, given a sentence in the reading comprehension paragraph and the text fragment (i e , the answer) that we want to ask about, we aim to generate a question that is asked about the specified answer 
Question generation for reading comprehension is firstly formalized as a declarative-to-interrogative sentence transformation problem with predefined rules or templates BIBREF4, BIBREF0  With the rise of neural models, Du2017LearningTA propose to model this task under the sequence-to-sequence (Seq2Seq) learning framework BIBREF5 with attention mechanism BIBREF6  However, question generation is a one-to-many sequence generation problem, i e , several aspects can be asked given a sentence  Zhou2017NeuralQG propose the answer-aware question generation setting which assumes the answer, a contiguous span inside the input sentence, is already known before question generation  To capture answer-relevant words in the sentence, they adopt a BIO tagging scheme to incorporate the answer position embedding in Seq2Seq learning  Furthermore, Sun2018AnswerfocusedAP propose that tokens close to the answer fragments are more likely to be answer-relevant  Therefore, they explicitly encode the relative distance between sentence words and the answer via position embedding and position-aware attention 
Although existing proximity-based answer-aware approaches achieve reasonable performance, we argue that such intuition may not apply to all cases especially for sentences with complex structure  For example, Figure FIGREF1 shows such an example where those approaches fail  This sentence contains a few facts and due to the parenthesis (i e  “the area's coldest month”), some facts intertwine: “The daily mean temperature in January is 0 3$^\circ $C” and “January is the area's coldest month”  From the question generated by a proximity-based answer-aware baseline, we find that it wrongly uses the word “coldest” but misses the correct word “mean” because “coldest” has a shorter distance to the answer “0 3$^\circ $C” 
In summary, their intuition that “the neighboring words of the answer are more likely to be answer-relevant and have a higher chance to be used in the question” is not reliable  To quantitatively show this drawback of these models, we implement the approach proposed by Sun2018AnswerfocusedAP and analyze its performance under different relative distances between the answer and other non-stop sentence words that also appear in the ground truth question  The results are shown in Table TABREF2  We find that the performance drops at most 36% when the relative distance increases from “$0\sim 10$” to “$>10$”  In other words, when the useful context is located far away from the answer, current proximity-based answer-aware approaches will become less effective, since they overly emphasize neighboring words of the answer 
To address this issue, we extract the structured answer-relevant relations from sentences and propose a method to jointly model such structured relation and the unstructured sentence for question generation  The structured answer-relevant relation is likely to be to the point context and thus can help keep the generated question to the point  For example, Figure FIGREF1 shows our framework can extract the right answer-relevant relation (“The daily mean temperature in January”, “is”, “32 6$^\circ $F (0 3$^\circ $C)”) among multiple facts  With the help of such structured information, our model is less likely to be confused by sentences with a complex structure  Specifically, we firstly extract multiple relations with an off-the-shelf Open Information Extraction (OpenIE) toolbox BIBREF7, then we select the relation that is most relevant to the answer with carefully designed heuristic rules 
Nevertheless, it is challenging to train a model to effectively utilize both the unstructured sentence and the structured answer-relevant relation because both of them could be noisy: the unstructured sentence may contain multiple facts which are irrelevant to the target question, while the limitation of the OpenIE tool may produce less accurate extracted relations  To explore their advantages simultaneously and avoid the drawbacks, we design a gated attention mechanism and a dual copy mechanism based on the encoder-decoder framework, where the former learns to control the information flow between the unstructured and structured inputs, while the latter learns to copy words from two sources to maintain the informativeness and faithfulness of generated questions 
In the evaluations on the SQuAD dataset, our system achieves significant and consistent improvement as compared to all baseline methods  In particular, we demonstrate that the improvement is more significant with a larger relative distance between the answer and other non-stop sentence words that also appear in the ground truth question  Furthermore, our model is capable of generating diverse questions for a single sentence-answer pair where the sentence conveys multiple relations of its answer fragment 
Framework Description
In this section, we first introduce the task definition and our protocol to extract structured answer-relevant relations  Then we formalize the task under the encoder-decoder framework with gated attention and dual copy mechanism 
Framework Description ::: Problem Definition
We formalize our task as an answer-aware Question Generation (QG) problem BIBREF8, which assumes answer phrases are given before generating questions  Moreover, answer phrases are shown as text fragments in passages  Formally, given the sentence $S$, the answer $A$, and the answer-relevant relation $M$, the task of QG aims to find the best question $\overline{Q}$ such that,
where $A$ is a contiguous span inside $S$ 
Framework Description ::: Answer-relevant Relation Extraction
We utilize an off-the-shelf toolbox of OpenIE to the derive structured answer-relevant relations from sentences as to the point contexts  Relations extracted by OpenIE can be represented either in a triple format or in an n-ary format with several secondary arguments, and we employ the latter to keep the extractions as informative as possible and avoid extracting too many similar relations in different granularities from one sentence  We join all arguments in the extracted n-ary relation into a sequence as our to the point context  Figure FIGREF5 shows n-ary relations extracted from OpenIE  As we can see, OpenIE extracts multiple relations for complex sentences  Here we select the most informative relation according to three criteria in the order of descending importance: (1) having the maximal number of overlapped tokens between the answer and the relation; (2) being assigned the highest confidence score by OpenIE; (3) containing maximum non-stop words  As shown in Figure FIGREF5, our criteria can select answer-relevant relations (waved in Figure FIGREF5), which is especially useful for sentences with extraneous information  In rare cases, OpenIE cannot extract any relation, we treat the sentence itself as the to the point context 
Table TABREF8 shows some statistics to verify the intuition that the extracted relations can serve as more to the point context  We find that the tokens in relations are 61% more likely to be used in the target question than the tokens in sentences, and thus they are more to the point  On the other hand, on average the sentences contain one more question token than the relations (1 86 v s  2 87)  Therefore, it is still necessary to take the original sentence into account to generate a more accurate question 
Framework Description ::: Our Proposed Model ::: Overview 
As shown in Figure FIGREF10, our framework consists offour components (1) Sentence Encoder and Relation Encoder, (2) Decoder, (3) Gated Attention Mechanism and (4) Dual Copy Mechanism  The sentence encoder and relation encoder encode the unstructured sentence and the structured answer-relevant relation, respectively  To select and combine the source information from the two encoders, a gated attention mechanism is employed to jointly attend both contextualized information sources, and a dual copy mechanism copies words from either the sentence or the relation 
Framework Description ::: Our Proposed Model ::: Answer-aware Encoder 
We employ two encoders to integrate information from the unstructured sentence $S$ and the answer-relevant relation $M$ separately  Sentence encoder takes in feature-enriched embeddings including word embeddings $\mathbf {w}$, linguistic embeddings $\mathbf {l}$ and answer position embeddings $\mathbf {a}$  We follow BIBREF9 to transform POS and NER tags into continuous representation ($\mathbf {l}^p$ and $\mathbf {l}^n$) and adopt a BIO labelling scheme to derive the answer position embedding (B: the first token of the answer, I: tokens within the answer fragment except the first one, O: tokens outside of the answer fragment)  For each word $w_i$ in the sentence $S$, we simply concatenate all features as input: $\mathbf {x}_i^s= [\mathbf {w}_i; \mathbf {l}^p_i; \mathbf {l}^n_i; \mathbf {a}_i]$  Here $[\mathbf {a};\mathbf {b}]$ denotes the concatenation of vectors $\mathbf {a}$ and $\mathbf {b}$ 
We use bidirectional LSTMs to encode the sentence $(\mathbf {x}_1^s, \mathbf {x}_2^s,    , \mathbf {x}_n^s)$ to get a contextualized representation for each token:
where $\overrightarrow{\mathbf {h}}^{s}_i$ and $\overleftarrow{\mathbf {h}}^{s}_i$ are the hidden states at the $i$-th time step of the forward and the backward LSTMs  The output state of the sentence encoder is the concatenation of forward and backward hidden states: $\mathbf {h}^{s}_i=[\overrightarrow{\mathbf {h}}^{s}_i;\overleftarrow{\mathbf {h}}^{s}_i]$  The contextualized representation of the sentence is $(\mathbf {h}^{s}_1, \mathbf {h}^{s}_2,    , \mathbf {h}^{s}_n)$ 
For the relation encoder, we firstly join all items in the n-ary relation $M$ into a sequence  Then we only take answer position embedding as an extra feature for the sequence: $\mathbf {x}_i^m= [\mathbf {w}_i; \mathbf {a}_i]$  Similarly, we take another bidirectional LSTMs to encode the relation sequence and derive the corresponding contextualized representation $(\mathbf {h}^{m}_1, \mathbf {h}^{m}_2,    , \mathbf {h}^{m}_n)$ 
Framework Description ::: Our Proposed Model ::: Decoder 
We use an LSTM as the decoder to generate the question  The decoder predicts the word probability distribution at each decoding timestep to generate the question  At the t-th timestep, it reads the word embedding $\mathbf {w}_{t}$ and the hidden state $\mathbf {u}_{t-1}$ of the previous timestep to generate the current hidden state:
Framework Description ::: Our Proposed Model ::: Gated Attention Mechanism 
We design a gated attention mechanism to jointly attend the sentence representation and the relation representation  For sentence representation $(\mathbf {h}^{s}_1, \mathbf {h}^{s}_2,    , \mathbf {h}^{s}_n)$, we employ the Luong2015EffectiveAT's attention mechanism to obtain the sentence context vector $\mathbf {c}^s_t$,
where $\mathbf {W}_a$ is a trainable weight  Similarly, we obtain the vector $\mathbf {c}^m_t$ from the relation representation $(\mathbf {h}^{m}_1, \mathbf {h}^{m}_2,    , \mathbf {h}^{m}_n)$  To jointly model the sentence and the relation, a gating mechanism is designed to control the information flow from two sources:
where $\odot $ represents element-wise dot production and $\mathbf {W}_g, \mathbf {W}_h$ are trainable weights  Finally, the predicted probability distribution over the vocabulary $V$ is computed as:
where $\mathbf {W}_V$ and $\mathbf {b}_V$ are parameters 
Framework Description ::: Our Proposed Model ::: Dual Copy Mechanism 
To deal with the rare and unknown words, the decoder applies the pointing method BIBREF10, BIBREF11, BIBREF12 to allow copying a token from the input sentence at the $t$-th decoding step  We reuse the attention score $\mathbf {\alpha }_{t}^s$ and $\mathbf {\alpha }_{t}^m$ to derive the copy probability over two source inputs:
Different from the standard pointing method, we design a dual copy mechanism to copy from two sources with two gates  The first gate is designed for determining copy tokens from two sources of inputs or generate next word from $P_V$, which is computed as $g^v_t = \text{sigmoid}(\mathbf {w}^v_g \tilde{\mathbf {h}}_t + b^v_g)$  The second gate takes charge of selecting the source (sentence or relation) to copy from, which is computed as $g^c_t = \text{sigmoid}(\mathbf {w}^c_g [\mathbf {c}_t^s;\mathbf {c}_t^m] + b^c_g)$  Finally, we combine all probabilities $P_V$, $P_S$ and $P_M$ through two soft gates $g^v_t$ and $g^c_t$  The probability of predicting $w$ as the $t$-th token of the question is:
Framework Description ::: Our Proposed Model ::: Training and Inference 
Given the answer $A$, sentence $S$ and relation $M$, the training objective is to minimize the negative log-likelihood with regard to all parameters:
where $\mathcal {\lbrace }Q\rbrace $ is the set of all training instances, $\theta $ denotes model parameters and $\text{log} P(Q|A,S,M;\theta )$ is the conditional log-likelihood of $Q$ 
In testing, our model targets to generate a question $Q$ by maximizing:
Experimental Setting ::: Dataset & Metrics
We conduct experiments on the SQuAD dataset BIBREF3  It contains 536 Wikipedia articles and 100k crowd-sourced question-answer pairs  The questions are written by crowd-workers and the answers are spans of tokens in the articles  We employ two different data splits by following Zhou2017NeuralQG and Du2017LearningTA   In Zhou2017NeuralQG, the original SQuAD development set is evenly divided into dev and test sets, while Du2017LearningTA treats SQuAD development set as its development set and splits original SQuAD training set into a training set and a test set  We also filter out questions which do not have any overlapped non-stop words with the corresponding sentences and perform some preprocessing steps, such as tokenization and sentence splitting  The data statistics are given in Table TABREF27 
We evaluate with all commonly-used metrics in question generation BIBREF13: BLEU-1 (B1), BLEU-2 (B2), BLEU-3 (B3), BLEU-4 (B4) BIBREF17, METEOR (MET) BIBREF18 and ROUGE-L (R-L) BIBREF19  We use the evaluation script released by Chen2015MicrosoftCC 
Experimental Setting ::: Baseline Models
We compare with the following models 
[leftmargin=*]
s2s BIBREF13 proposes an attention-based sequence-to-sequence neural network for question generation 
NQG++ BIBREF9 takes the answer position feature and linguistic features into consideration and equips the Seq2Seq model with copy mechanism 
M2S+cp BIBREF14 conducts multi-perspective matching between the answer and the sentence to derive an answer-aware sentence representation for question generation 
s2s+MP+GSA BIBREF8 introduces a gated self-attention into the encoder and a maxout pointer mechanism into the decoder  We report their sentence-level results for a fair comparison 
Hybrid BIBREF15 is a hybrid model which considers the answer embedding for the question word generation and the position of context words for modeling the relative distance between the context words and the answer 
ASs2s BIBREF16 replaces the answer in the sentence with a special token to avoid its appearance in the generated questions 
Experimental Setting ::: Implementation Details
We take the most frequent 20k words as our vocabulary and use the GloVe word embeddings BIBREF20 for initialization  The embedding dimensions for POS, NER, answer position are set to 20  We use two-layer LSTMs in both encoder and decoder, and the LSTMs hidden unit size is set to 600 
We use dropout BIBREF21 with the probability $p=0 3$  All trainable parameters, except word embeddings, are randomly initialized with the Xavier uniform in $(-0 1, 0 1)$ BIBREF22  For optimization in the training, we use SGD as the optimizer with a minibatch size of 64 and an initial learning rate of 1 0  We train the model for 15 epochs and start halving the learning rate after the 8th epoch  We set the gradient norm upper bound to 3 during the training 
We adopt the teacher-forcing for the training  In the testing, we select the model with the lowest perplexity and beam search with size 3 is employed for generating questions  All hyper-parameters and models are selected on the validation dataset 
Results and Analysis ::: Main Results
Table TABREF30 shows automatic evaluation results for our model and baselines (copied from their papers)  Our proposed model which combines structured answer-relevant relations and unstructured sentences achieves significant improvements over proximity-based answer-aware models BIBREF9, BIBREF15 on both dataset splits  Presumably, our structured answer-relevant relation is a generalization of the context explored by the proximity-based methods because they can only capture short dependencies around answer fragments while our extractions can capture both short and long dependencies given the answer fragments  Moreover, our proposed framework is a general one to jointly leverage structured relations and unstructured sentences  All compared baseline models which only consider unstructured sentences can be further enhanced under our framework 
Recall that existing proximity-based answer-aware models perform poorly when the distance between the answer fragment and other non-stop sentence words that also appear in the ground truth question is large (Table TABREF2)  Here we investigate whether our proposed model using the structured answer-relevant relations can alleviate this issue or not, by conducting experiments for our model under the same setting as in Table TABREF2  The broken-down performances by different relative distances are shown in Table TABREF40  We find that our proposed model outperforms Hybrid (our re-implemented version for this experiment) on all ranges of relative distances, which shows that the structured answer-relevant relations can capture both short and long term answer-relevant dependencies of the answer in sentences  Furthermore, comparing the performance difference between Hybrid and our model, we find the improvements become more significant when the distance increases from “$0\sim 10$” to “$>10$”  One reason is that our model can extract relations with distant dependencies to the answer, which greatly helps our model ignore the extraneous information  Proximity-based answer-aware models may overly emphasize the neighboring words of answers and become less effective as the useful context becomes further away from the answer in the complex sentences  In fact, the breakdown intervals in Table TABREF40 naturally bound its sentence length, say for “$>10$”, the sentences in this group must be longer than 10  Thus, the length variances in these two intervals could be significant  To further validate whether our model can extract long term dependency words  We rerun the analysis of Table TABREF40 only for long sentences (length $>$ 20) of each interval  The improvement percentages over Hybrid are shown in Table TABREF40, which become more significant when the distance increases from “$0\sim 10$” to “$>10$” 
Results and Analysis ::: Case Study
Figure FIGREF42 provides example questions generated by crowd-workers (ground truth questions), the baseline Hybrid BIBREF15, and our model  In the first case, there are two subsequences in the input and the answer has no relation with the second subsequence  However, we see that the baseline model prediction copies irrelevant words “The New York Times” while our model can avoid using the extraneous subsequence “The New York Times noted    ” with the help of the structured answer-relevant relation  Compared with the ground truth question, our model cannot capture the cross-sentence information like “her fifth album”, where the techniques in paragraph-level QG models BIBREF8 may help  In the second case, as discussed in Section SECREF1, this sentence contains a few facts and some facts intertwine  We find that our model can capture distant answer-relevant dependencies such as “mean temperature” while the proximity-based baseline model wrongly takes neighboring words of the answer like “coldest” in the generated question 
Results and Analysis ::: Diverse Question Generation
Another interesting observation is that for the same answer-sentence pair, our model can generate diverse questions by taking different answer-relevant relations as input  Such capability improves the interpretability of our model because the model is given not only what to be asked (i e , the answer) but also the related fact (i e , the answer-relevant relation) to be covered in the question  In contrast, proximity-based answer-aware models can only generate one question given the sentence-answer pair regardless of how many answer-relevant relations in the sentence  We think such capability can also validate our motivation: questions should be generated according to the answer-aware relations instead of neighboring words of answer fragments  Figure FIGREF45 show two examples of diverse question generation  In the first case, the answer fragment `Hugh L  Dryden' is the appositive to `NASA Deputy Administrator' but the subject to the following tokens `announced the Apollo program    '  Our framework can extract these two answer-relevant relations, and by feeding them to our model separately, we can receive two questions asking different relations with regard to the answer 
Related Work
The topic of question generation, initially motivated for educational purposes, is tackled by designing many complex rules for specific question types BIBREF4, BIBREF23  Heilman2010GoodQS improve rule-based question generation by introducing a statistical ranking model  First, they remove extraneous information in the sentence to transform it into a simpler one, which can be transformed easily into a succinct question with predefined sets of general rules  Then they adopt an overgenerate-and-rank approach to select the best candidate considering several features 
With the rise of dominant neural sequence-to-sequence learning models BIBREF5, Du2017LearningTA frame question generation as a sequence-to-sequence learning problem  Compared with rule-based approaches, neural models BIBREF24 can generate more fluent and grammatical questions  However, question generation is a one-to-many sequence generation problem, i e , several aspects can be asked given a sentence, which confuses the model during train and prevents concrete automatic evaluation  To tackle this issue, Zhou2017NeuralQG propose the answer-aware question generation setting which assumes the answer is already known and acts as a contiguous span inside the input sentence  They adopt a BIO tagging scheme to incorporate the answer position information as learned embedding features in Seq2Seq learning  Song2018LeveragingCI explicitly model the information between answer and sentence with a multi-perspective matching model  Kim2019ImprovingNQ also focus on the answer information and proposed an answer-separated Seq2Seq model by masking the answer with special tokens  All answer-aware neural models treat question generation as a one-to-one mapping problem, but existing models perform poorly for sentences with a complex structure (as shown in Table TABREF2) 
Our work is inspired by the process of extraneous information removing in BIBREF0, BIBREF25  Different from Heilman2010GoodQS which directly use the simplified sentence for generation and cao2018faithful which only consider aggregate two sources of information via gated attention in summarization, we propose to combine the structured answer-relevant relation and the original sentence  Factoid question generation from structured text is initially investigated by Serban2016GeneratingFQ, but our focus here is leveraging structured inputs to help question generation over unstructured sentences  Our proposed model can take advantage of unstructured sentences and structured answer-relevant relations to maintain informativeness and faithfulness of generated questions  The proposed model can also be generalized in other conditional sequence generation tasks which require multiple sources of inputs, e g , distractor generation for multiple choice questions BIBREF26 
Conclusions and Future Work
In this paper, we propose a question generation system which combines unstructured sentences and structured answer-relevant relations for generation  The unstructured sentences maintain the informativeness of generated questions while structured answer-relevant relations keep the faithfulness of questions  Extensive experiments demonstrate that our proposed model achieves state-of-the-art performance across several metrics  Furthermore, our model can generate diverse questions with different structured answer-relevant relations  For future work, there are some interesting dimensions to explore, such as difficulty levels BIBREF27, paragraph-level information BIBREF8 and conversational question generation BIBREF28 
Acknowledgments
This work is supported by the Research Grants Council of the Hong Kong Special Administrative Region, China (No  CUHK 14208815 and No  CUHK 14210717 of the General Research Fund)  We would like to thank the anonymous reviewers for their comments  We would also like to thank Department of Computer Science and Engineering, The Chinese University of Hong Kong for the conference grant support 
Context 2: Introduction
Deep Learning approaches have achieved impressive results on various NLP tasks BIBREF0 , BIBREF1 , BIBREF2 and have become the de facto approach for any NLP task  However, these deep learning techniques have found to be less effective for low-resource languages when the available training data is very less BIBREF3   Recently, several approaches like Multi-task learning BIBREF4 , multilingual learning BIBREF5 , semi-supervised learning BIBREF2 , BIBREF6 and transfer learning BIBREF7 , BIBREF3 have been explored by the deep learning community to overcome data sparsity in low-resource languages  Transfer learning trains a model for a parent task and fine-tunes the learned parent model weights (features) for a related child task BIBREF7 , BIBREF8   This effectively reduces the requirement on training data for the child task as the model would have learned relevant features from the parent task data thereby, improving the performance on the child task 
Transfer learning has also been explored in the multilingual Neural Machine Translation BIBREF3 , BIBREF9 , BIBREF10   The goal is to improve the NMT performance on the source to target language pair (child task) using an assisting source language (assisting to target translation is the parent task)  Here, the parent model is trained on the assisting and target language parallel corpus and the trained weights are used to initialize the child model  The child model can now be fine-tuned on the source-target language pairs, if parallel corpus is available  The divergence between the source and the assisting language can adversely impact the benefits obtained from transfer learning  Multiple studies have shown that transfer learning works best when the languages are related BIBREF3 , BIBREF10 , BIBREF9   Several studies have tried to address lexical divergence between the source and the target languages BIBREF10 , BIBREF11 , BIBREF12   However, the effect of word order divergence and its mitigation has not been explored  In a practical setting, it is not uncommon to have source and assisting languages with different word order  For instance, it is possible to find parallel corpora between English and some Indian languages, but very little parallel corpora between Indian languages  Hence, it is natural to use English as an assisting language for inter-Indian language translation 
To see how word order divergence can be detrimental, let us consider the case of the standard RNN (Bi-LSTM) encoder-attention-decoder architecture BIBREF13   The encoder generates contextual representations (annotation vectors) for each source word, which are used by the attention network to match the source words to the current decoder state  The contextual representation is word-order dependent  Hence, if the assisting and the source languages do not have similar word order the generated contextual representations will not be consistent  The attention network (and hence the decoder) sees different contextual representations for similar words in parallel sentences across different languages  This makes it difficult to transfer knowledge learned from the assisting language to the source language 
We illustrate this by visualizing the contextual representations generated by the encoder of an English to Hindi NMT system for two versions of the English input: (a) original word order (SVO) (b) word order of the source language (SOV, for Bengali)  Figure FIGREF1 shows that the encoder representations obtained are very different  The attention network and the decoder now have to work with very different representations  Note that the plot below does not take into account further lexical and other divergences between source and assisting languages, since we demonstrated word order divergence with the same language on the source side 
To address this word order divergence, we propose to pre-order the assisting language sentences to match the word order of the source language  We consider an extremely resource constrained scenario, where we do not have any parallel corpus for the child task  We are limited to a bilingual dictionary for transfer information from the assisting to the source language  From our experiments, we show that there is a significant increase in the translation accuracy for the unseen source-target language pair 
Addressing Lexical Divergence
BIBREF3 explored transfer learning for NMT on low-resource languages  They studied the influence of language divergence between languages chosen for training the parent and child model, and showed that choosing similar languages for training the parent and child model leads to better improvements from transfer learning  A limitation of BIBREF3 approach is that they ignore the lexical similarity between languages and also the source language embeddings are randomly initialized  BIBREF10 , BIBREF11 , BIBREF12 take advantage of lexical similarity between languages in their work  BIBREF10 proposed to use Byte-Pair Encoding (BPE) to represent the sentences in both the parent and the child language to overcome the above limitation  They show using BPE benefits transfer learning especially when the involved languages are closely-related agglutinative languages  Similarly, BIBREF11 utilize lexical similarity between the source and assisting languages by training a character-level NMT system  BIBREF12 address lexical divergence by using bilingual embeddings and mixture of universal token embeddings  One of the languages' vocabulary, usually English vocabulary is considered as universal tokens and every word in the other languages is represented as a mixture of universal tokens  They show results on extremely low-resource languages 
Addressing Word Order Divergence
To the best of our knowledge, no work has addressed word order divergence in transfer learning for multilingual NMT  However, some work exists for other NLP tasks that could potentially address word order  For Named Entity Recognition (NER), BIBREF14 use a self-attention layer after the Bi-LSTM layer to address word-order divergence for Named Entity Recognition (NER) task  The approach does not show any significant improvements over multiple languages  A possible reason is that the divergence has to be addressed before/during construction of the contextual embeddings in the Bi-LSTM layer, and the subsequent self-attention layer does not address word-order divergence  BIBREF15 use adversarial training for cross-lingual question-question similarity ranking in community question answering  The adversarial training tries to force the encoder representations of similar sentences from different input languages to have similar representations 
Use of Pre-ordering
Pre-ordering the source language sentences to match the target language word order has been useful in addressing word-order divergence for Phrase-Based SMT BIBREF16 , BIBREF17 , BIBREF18 , BIBREF19   Recently, BIBREF20 proposed a way to measure and reduce the divergence between the source and target languages based on morphological and syntactic properties, also termed as anisomorphism  They demonstrated that by reducing the anisomorphism between the source and target languages, consistent improvements in NMT performance were obtained  The NMT system used additional features like word forms, POS tags and dependency relations in addition to parallel corpora  On the other hand, BIBREF21 observed a drop in performance due to pre-ordering for NMT  Unlike BIBREF20 , the NMT system was trained on pre-ordered sentences and no additional features were provided to the system  Note that all these works address source-target divergence, not divergence between source languages in multilingual NMT 
Proposed Solution
Consider the task of translating from an extremely low-resource language (source) to a target language  The parallel corpus between the two languages if available may be too small to train a NMT model  Similar to existing works BIBREF3 , BIBREF10 , BIBREF12 , we use transfer learning to overcome data sparsity and train a NMT model between the source and the target languages
Context 3: Introduction
Measures of semantic similarity and relatedness quantify the degree to which two concepts are similar (e g , INLINEFORM0 – INLINEFORM1 ) or related (e g , INLINEFORM2 – INLINEFORM3 )  Semantic similarity can be viewed as a special case of semantic relatedness – to be similar is one of many ways that a pair of concepts may be related  The automated discovery of groups of semantically similar or related terms is critical to improving the retrieval BIBREF0 and clustering BIBREF1 of biomedical and clinical documents, and the development of biomedical terminologies and ontologies BIBREF2  
There is a long history in using distributional methods to discover semantic similarity and relatedness (e g , BIBREF3 , BIBREF4 , BIBREF5 , BIBREF6 )  These methods are all based on the distributional hypothesis, which holds that two terms that are distributionally similar (i e , used in the same context) will also be semantically similar BIBREF7 , BIBREF8   Recently word embedding techniques such as word2vec BIBREF9 have become very popular  Despite the prominent role that neural networks play in many of these approaches, at their core they remain distributional techniques that typically start with a word by word co–occurrence matrix, much like many of the more traditional approaches 
However, despite these successes distributional methods do not perform well when data is very sparse (which is common)  One possible solution is to use second–order co–occurrence vectors BIBREF10 , BIBREF11   In this approach the similarity between two words is not strictly based on their co–occurrence frequencies, but rather on the frequencies of the other words which occur with both of them (i e , second order co–occurrences)  This approach has been shown to be successful in quantifying semantic relatedness BIBREF12 , BIBREF13   However, while more robust in the face of sparsity, second–order methods can result in significant amounts of noise, where contextual information that is overly general is included and does not contribute to quantifying the semantic relatedness between the two concepts 
Our goal then is to discover methods that automatically reduce the amount of noise in a second–order co–occurrence vector  We achieve this by incorporating pairwise semantic similarity scores derived from a taxonomy into our second–order vectors, and then using these scores to select only the most semantically similar co–occurrences (thereby reducing noise) 
We evaluate our method on two datasets that have been annotated in multiple ways  One has been annotated for both similarity and relatedness, and the other has been annotated for relatedness by two different types of experts (medical doctors and medical coders)  Our results show that integrating second order co–occurrences with measures of semantic similarity increases correlation with our human reference standards  We also compare our result to a number of other studies which have applied various word embedding methods to the same reference standards we have used  We find that our method often performs at a comparable or higher level than these approaches  These results suggest that our methods of integrating semantic similarity and relatedness values have the potential to improve performance of purely distributional methods 
Similarity and Relatedness Measures
This section describes the similarity and relatedness measures we integrate in our second–order co–occurrence vectors  We use two taxonomies in this study, SNOMED–CT and MeSH  SNOMED–CT (Systematized Nomenclature of Medicine Clinical Terms) is a comprehensive clinical terminology created for the electronic representation of clinical health information  MeSH (Medical Subject Headings) is a taxonomy of biomedical terms developed for indexing biomedical journal articles 
We obtain SNOMED–CT and MeSH via the Unified Medical Language System (UMLS) Metathesaurus (version 2016AA)  The Metathesaurus contains approximately 2 million biomedical and clinical concepts from over 150 different terminologies that have been semi–automatically integrated into a single source  Concepts in the Metathesaurus are connected largely by two types of hierarchical relations: INLINEFORM0 / INLINEFORM1 (PAR/CHD) and INLINEFORM2 / INLINEFORM3 (RB/RN) 
Similarity Measures
Measures of semantic similarity can be classified into three broad categories : path–based, feature–based and information content (IC)  Path–based similarity measures use the structure of a taxonomy to measure similarity – concepts positioned close to each other are more similar than those further apart  Feature–based methods rely on set theoretic measures of overlap between features (union and intersection)  The information content measures quantify the amount of information that a concept provides – more specific concepts have a higher amount of information content 
RadaMBB89 introduce the Conceptual Distance measure  This measure is simply the length of the shortest path between two concepts ( INLINEFORM0 and INLINEFORM1 ) in the MeSH hierarchy  Paths are based on broader than (RB) and narrower than (RN) relations  CaviedesC04 extends this measure to use parent (PAR) and child (CHD) relations  Our INLINEFORM2 measure is simply the reciprocal of this shortest path value (Equation EQREF3 ), so that larger values (approaching 1) indicate a high degree of similarity  DISPLAYFORM0
While the simplicity of INLINEFORM0 is appealing, it can be misleading when concepts are at different levels of specificity  Two very general concepts may have the same path length as two very specific concepts  WuP94 introduce a correction to INLINEFORM1 that incorporates the depth of the concepts, and the depth of their Least Common Subsumer (LCS)  This is the most specific ancestor two concepts share  In this measure, similarity is twice the depth of the two concept's LCS divided by the product of the depths of the individual concepts (Equation EQREF4 )  Note that if there are multiple LCSs for a pair of concepts, the deepest of them is used in this measure  DISPLAYFORM0
ZhongZLY02 take a very similar approach and again scale the depth of the LCS by the sum of the depths of the two concepts (Equation EQREF5 ), where INLINEFORM0   The value of INLINEFORM1 was set to 2 based on their recommendations  DISPLAYFORM0
PekarS02 offer another variation on INLINEFORM0 , where the shortest path of the two concepts to the LCS is used, in addition to the shortest bath between the LCS and the root of the taxonomy (Equation EQREF6 )  DISPLAYFORM0
Feature–based methods represent each concept as a set of features and then measure the overlap or sharing of features to measure similarity  In particular, each concept is represented as the set of their ancestors, and similarity is a ratio of the intersection and union of these features 
MaedcheS01 quantify the similarity between two concepts as the ratio of the intersection over their union as shown in Equation EQREF8   DISPLAYFORM0
BatetSV11 extend this by excluding any shared features (in the numerator) as shown in Equation EQREF9   DISPLAYFORM0
Information content is formally defined as the negative log of the probability of a concept  The effect of this is to assign rare (low probability) concepts a high measure of information content, since the underlying assumption is that more specific concepts are less frequently used than more common ones 
Resnik95 modified this notion of information content in order to use it as a similarity measure  He defines the similarity of two concepts to be the information content of their LCS (Equation EQREF11 )  DISPLAYFORM0
JiangC97, Lin98, and PirroE10 extend INLINEFORM0 by incorporating the information content of the individual concepts in various different ways  Lin98 defines the similarity between two concepts as the ratio of information content of the LCS with the sum of the individual concept's information content (Equation EQREF12 )  Note that INLINEFORM1 has the same form as INLINEFORM2 and INLINEFORM3 , and is in effect using information content as a measure of specificity (rather than depth)  If there is more than one possible LCS, the LCS with the greatest IC is chosen  DISPLAYFORM0
JiangC97 define the distance between two concepts to be the sum of the information content of the two concepts minus twice the information content of the concepts' LCS  We modify this from a distance to a similarity measure by taking the reciprocal of the distance (Equation EQREF13 )  Note that the denominator of INLINEFORM0 is very similar to the numerator of INLINEFORM1   DISPLAYFORM0
PirroE10 define the similarity between two concepts as the information content of the two concept's LCS divided by the sum of their individual information content values minus the information content of their LCS (Equation EQREF14 )  Note that INLINEFORM0 can be viewed as a set–theoretic version of INLINEFORM1   DISPLAYFORM0
Information Content
The information content of a concept may be derived from a corpus (corpus–based) or directly from a taxonomy (intrinsic–based)  In this work we focus on corpus–based techniques 
For corpus–based information content, we estimate the probability of a concept INLINEFORM0 by taking the sum of the probability of the concept INLINEFORM1 and the probability its descendants INLINEFORM2 (Equation EQREF16 )  DISPLAYFORM0
The initial probabilities of a concept ( INLINEFORM0 ) and its descendants ( INLINEFORM1 ) are obtained by dividing the number of times each concept and descendant occurs in the corpus, and dividing that by the total numbers of concepts ( INLINEFORM2 ) 
Ideally the corpus from which we are estimating the probabilities of concepts will be sense–tagged  However, sense–tagging is a challenging problem in its own right, and it is not always possible to carry out reliably on larger amounts of text  In fact in this paper we did not use any sense–tagging of the corpus we derived information content from 
Instead, we estimated the probability of a concept by using the UMLSonMedline dataset  This was created by the National Library of Medicine and consists of concepts from the 2009AB UMLS and the counts of the number of times they occurred in a snapshot of Medline taken on 12 January, 2009  These counts were obtained by using the Essie Search Engine BIBREF14 which queried Medline with normalized strings from the 2009AB MRCONSO table in the UMLS  The frequency of a CUI was obtained by aggregating the frequency counts of the terms associated with the CUI to provide a rough estimate of its frequency  The information content measures then use this information to calculate the probability of a concept 
Another alternative is the use of Intrinsic Information Content  It assess the informativeness of concept based on its placement within a taxonomy by considering the number of incoming (ancestors) relative to outgoing (descendant) links BIBREF15 (Equation EQREF17 )  DISPLAYFORM0
where INLINEFORM0 are the number of descendants of concept INLINEFORM1 that are leaf nodes, INLINEFORM2 are the number of concept INLINEFORM3 's ancestors and INLINEFORM4 are the total number of leaf nodes in the taxonomy 
Relatedness Measures
Lesk86 observed that concepts that are related should share more words in their respective definitions than concepts that are less connected  He was able to perform word sense disambiguation by identifying the senses of words in a sentence with the largest number of overlaps between their definitions  An overlap is the longest sequence of one or more consecutive words that occur in both definitions  BanerjeeP03 extended this idea to WordNet, but observed that WordNet glosses are often very short, and did not contain enough information to distinguish between multiple concepts  Therefore, they created a super–gloss for each concept by adding the glosses of related concepts to the gloss of the concept itself (and then finding overlaps) 
PatwardhanP06 adapted this measure to second–order co–occurrence vectors  In this approach, a vector is created for each word in a concept's definition that shows which words co–occur with it in a corpus  These word vectors are averaged to create a single co-occurrence vector for the concept  The similarity between the concepts is calculated by taking the cosine between the concepts second–order vectors  LiuMPMP12 modified and extended this measure to be used to quantify the relatedness between biomedical and clinical terms in the UMLS  The work in this paper can be seen as a further extension of PatwardhanP06 and LiuMPMP12 
Method
In this section, we describe our second–order similarity vector measure  This incorporates both contextual information using the term pair's definition and their pairwise semantic similarity scores derived from a taxonomy  There are two stages to our approach  First, a co–occurrence matrix must be constructed  Second, this matrix is used to construct a second–order co–occurrence vector for each concept in a pair of concepts to be measured for relatedness 
Co–occurrence Matrix Construction
We build an INLINEFORM0 similarity matrix using an external corpus where the rows and columns represent words within the corpus and the element contains the similarity score between the row word and column word using the similarity measures discussed above  If a word maps to more than one possible sense, we use the sense that returns the highest similarity score

ground_truth: SQuAD

SQuAD

prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. How do they match words before reordering them?
Context 2:  Specifically, the NMT model (parent model) is trained on the assisting language and target language pairs  We choose English as the assisting language in all our experiments  In our resource-scarce scenario, we have no parallel corpus for the child task  Hence, at test time, the source language sentence is translated using the parent model after performing a word-by-word translation into the assisting language 
Since the source language and the assisting language (English) have different word order, we hypothesize that it leads to inconsistencies in the contextual representations generated by the encoder for the two languages  In this paper, we propose to pre-order English sentences (assisting language sentences) to match the word-order of the source language and train the parent model on this pre-ordered corpus  In our experiments, we look at scenarios where the assisting language has SVO word order and the source language has SOV word order 
For instance, consider the English sentence Anurag will meet Thakur  One of the pre-ordering rule swaps the position of the noun phrase followed by a transitive verb with the transitive verb  The original and the resulting re-ordered parse tree will be as shown in the Table TABREF5   Applying this reordering rule to the above sentence Anurag will meet Thakur will yield the reordered sentence Anurag Thakur will meet  Additionally, the Table TABREF5 shows the parse trees for the above sentence with and without pre-ordering 
Pre-ordering should also be beneficial for other word order divergence scenarios (e g , SOV to SVO), but we leave verification of these additional scenarios for future work 
Experimental Setup
In this section, we describe the languages experimented with, datasets used, the network hyper-parameters used in our experiments 
Languages
We experimented with English INLINEFORM0 Hindi translation as the parent task  English is the assisting source language  Bengali, Gujarati, Marathi, Malayalam and Tamil are the primary source languages, and translation from these to Hindi constitute the child tasks  Hindi, Bengali, Gujarati and Marathi are Indo-Aryan languages, while Malayalam and Tamil are Dravidian languages  All these languages have a canonical SOV word order 
Datasets
For training English-Hindi NMT systems, we use the IITB English-Hindi parallel corpus BIBREF22 ( INLINEFORM0 sentences from the training set) and the ILCI English-Hindi parallel corpus ( INLINEFORM1 sentences)  The ILCI (Indian Language Corpora Initiative) multilingual parallel corpus BIBREF23 spans multiple Indian languages from the health and tourism domains  We use the 520-sentence dev-set of the IITB parallel corpus for validation  For each child task, we use INLINEFORM2 sentences from ILCI corpus as the test set 
Network
We use OpenNMT-Torch BIBREF24 to train the NMT system  We use the standard sequence-to-sequence architecture with attention BIBREF13   We use an encoder which contains two layers of bidirectional LSTMs with 500 neurons each  The decoder contains two LSTM layers with 500 neurons each  Input feeding approach BIBREF1 is used where the previous attention hidden state is fed as input to the decoder LSTM  We use a mini-batch of size 50 and use a dropout layer  We begin with an initial learning rate of INLINEFORM0 and decay the learning rate by a factor of INLINEFORM1 when the perplexity on validation set increases  The training is stopped when the learning rate falls below INLINEFORM2 or number of epochs=22  The English input is initialized with pre-trained embeddings trained using fastText BIBREF25  
English vocabulary consists of INLINEFORM0 tokens appearing at least 2 times in the English training corpus  For constructing the Hindi vocabulary we considered only those tokens appearing at least 5 times in the training split resulting in a vocabulary size of INLINEFORM1 tokens  For representing English and other source languages into a common space, we translate each word in the source language into English using a bilingual dictionary (Google Translate word translation in our case)  In an end-to-end solution, it would have been ideal to use bilingual embeddings or obtain word-by-word translations via bilingual embeddings BIBREF14   But, the quality of publicly available bilingual embeddings for English-Indian languages is very low for obtaining good-quality, bilingual representations BIBREF26 , BIBREF27   We also found that these embeddings were not useful for transfer learning 
We use the CFILT-preorder system for reordering English sentences to match the Indian language word order  It contains two re-ordering systems: (1) generic rules that apply to all Indian languages BIBREF17 , and (2) hindi-tuned rules which improve the generic rules by incorporating improvements found through an error analysis of English-Hindi reordering BIBREF28   These Hindi-tuned rules have been found to improve reordering for many English to Indian language pairs BIBREF29  
Results
In this section, we describe the results from our experiments on NMT task  We report the results on X-Hindi pair, where X is one of Bengali, Gujarati, Marathi, Tamil, and Malayalam  The results are presented in the Table TABREF6   We report BLEU scores and LeBLEU scores BIBREF30   We observe that both the pre-ordering configurations significantly improve the BLEU scores over the baseline scores  We observe larger gains when generic pre-ordering rules are used compared to the Hindi-tuned pre-ordering rules 
These results support our hypothesis that word-order divergence can limit the benefits of multilingual translation  Reducing the word order divergence can improve translation in extremely low-resource scenarios 
An analysis of the outputs revealed that pre-ordering significantly reducing the number of UNK tokens (placeholder for unknown words) in the test output (Table TABREF14 )  We hypothesize that due to word order divergence between English and Indian languages, the encoder representation generated is not consistent leading to decoder generating unknown words  However, the pre-ordered models generate better contextual representations leading to less number of unknown tokens and better translation which is also reflected in the BLEU scores 
Conclusion
In this paper, we show that handling word-order divergence between source and assisting languages is crucial for the success of multilingual NMT in an extremely low-resource setting  We show that pre-ordering the assisting language to match the word order of the source language significantly improves translation quality in an extremely low-resource setting  While the current work focused on Indian languages, we would like to validate the hypothesis on a more diverse set of languages 

ground_truth: Unanswerable

CFILT-preorder system

prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. What embedding techniques are explored in the paper?
Context 3: 
For this paper our external corpus was the NLM 2015 Medline baseline  Medline is a bibliographic database containing over 23 million citations to journal articles in the biomedical domain and is maintained by National Library of Medicine  The 2015 Medline Baseline encompasses approximately 5,600 journals starting from 1948 and contains 23,343,329 citations, of which 2,579,239 contain abstracts  In this work, we use Medline titles and abstracts from 1975 to present day  Prior to 1975, only 2% of the citations contained an abstract  We then calculate the similarity for each bigram in this dataset and include those that have a similarity score greater than a specified threshold on these experiments 
Measure Term Pairs for Relatedness
We obtain definitions for each of the two terms we wish to measure  Due to the sparsity and inconsistencies of the definitions in the UMLS, we not only use the definition of the term (CUI) but also include the definition of its related concepts  This follows the method proposed by PatwardhanP06 for general English and WordNet, and which was adapted for the UMLS and the medical domain by LiuMPMP12  In particular we add the definitions of any concepts connected via a parent (PAR), child (CHD), RB (broader than), RN (narrower than) or TERM (terms associated with CUI) relation  All of the definitions for a term are combined into a single super–gloss  At the end of this process we should have two super–glosses, one for each term to be measured for relatedness 
Next, we process each super–gloss as follows:
We extract a first–order co–occurrence vector for each term in the super–gloss from the co–occurrence matrix created previously 
We take the average of the first order co–occurrence vectors associated with the terms in a super–gloss and use that to represent the meaning of the term  This is a second–order co–occurrence vector 
After a second–order co–occurrence vector has been constructed for each term, then we calculate the cosine between these two vectors to measure the relatedness of the terms 
Data
We use two reference standards to evaluate the semantic similarity and relatedness measures   UMNSRS was annotated for both similarity and relatedness by medical residents  MiniMayoSRS was annotated for relatedness by medical doctors (MD) and medical coders (coder)  In this section, we describe these data sets and describe a few of their differences 
MiniMayoSRS: The MayoSRS, developed by PakhomovPMMRC10, consists of 101 clinical term pairs whose relatedness was determined by nine medical coders and three physicians from the Mayo Clinic  The relatedness of each term pair was assessed based on a four point scale: (4 0) practically synonymous, (3 0) related, (2 0) marginally related and (1 0) unrelated  MiniMayoSRS is a subset of the MayoSRS and consists of 30 term pairs on which a higher inter–annotator agreement was achieved  The average correlation between physicians is 0 68  The average correlation between medical coders is 0 78  We evaluate our method on the mean of the physician scores, and the mean of the coders scores in this subset in the same manner as reported by PedersenPPC07 
UMNSRS: The University of Minnesota Semantic Relatedness Set (UMNSRS) was developed by PakhomovMALPM10, and consists of 725 clinical term pairs whose semantic similarity and relatedness was determined independently by four medical residents from the University of Minnesota Medical School  The similarity and relatedness of each term pair was annotated based on a continuous scale by having the resident touch a bar on a touch sensitive computer screen to indicate the degree of similarity or relatedness  The Intraclass Correlation Coefficient (ICC) for the reference standard tagged for similarity was 0 47, and 0 50 for relatedness  Therefore, as suggested by Pakhomov and colleagues,we use a subset of the ratings consisting of 401 pairs for the similarity set and 430 pairs for the relatedness set which each have an ICC of 0 73 
Experimental Framework
We conducted our experiments using the freely available open source software package UMLS::Similarity BIBREF16 version 1 47  This package takes as input two terms (or UMLS concepts) and returns their similarity or relatedness using the measures discussed in Section SECREF2  
Correlation between the similarity measures and human judgments were estimated using Spearman's Rank Correlation ( INLINEFORM0 )  Spearman's measures the statistical dependence between two variables to assess how well the relationship between the rankings of the variables can be described using a monotonic function  We used Fisher's r-to-z transformation BIBREF17 to calculate the significance between the correlation results 
Results and Discussion
Table TABREF26 shows the Spearman's Rank Correlation between the human scores from the four reference standards and the scores from the various measures of similarity introduced in Section SECREF2   Each class of measure is followed by the scores obtained when integrating our second order vector approach with these measures of semantic similarity 
Results Comparison
The results for UMNSRS tagged for similarity ( INLINEFORM0 ) and MiniMayoSRS tagged by coders show that all of the second-order similarity vector measures ( INLINEFORM1 ) except for INLINEFORM2 - INLINEFORM3 obtain a higher correlation than the original measures  We found that INLINEFORM4 - INLINEFORM5 and INLINEFORM6 - INLINEFORM7 obtain the highest correlations of all these results with human judgments 
For the UMNSRS dataset tagged for relatedness and MiniMayoSRS tagged by physicians (MD), the original INLINEFORM0 measure obtains a higher correlation than our measure ( INLINEFORM1 ) although the difference is not statistically significant ( INLINEFORM2 ) 
In order to analyze and better understand these results, we filtered the bigram pairs used to create the initial similarity matrix based on the strength of their similarity using the INLINEFORM0 and the INLINEFORM1 measures  Note that the INLINEFORM2 measure holds to a 0 to 1 scale, while INLINEFORM3 ranges from 0 to an unspecified upper bound that is dependent on the size of the corpus from which information content is estimated  As such we use a different range of threshold values for each measure  We discuss the results of this filtering below 
Thresholding Experiments
Table TABREF29 shows the results of applying the threshold parameter on each of the reference standards using the INLINEFORM0 measure  For example, a threshold of 0 indicates that all of the bigrams were included in the similarity matrix; and a threshold of 1 indicates that only the bigram pairs with a similarity score greater than one were included 
These results show that using a threshold cutoff of 2 obtains the highest correlation for the UMNSRS dataset, and that a threshold cutoff of 4 obtains the highest correlation for the MiniMayoSRS dataset  All of the results show an increase in correlation with human judgments when incorporating a threshold cutoff over all of the original measures  The increase in the correlation for the UMNSRS tagged for similarity is statistically significant ( INLINEFORM0 ), however this is not the case for the UMNSRS tagged for relatedness nor for the MiniMayoSRS data 
Similarly, Table TABREF30 shows the results of applying the threshold parameter (T) on each of the reference standards using the INLINEFORM0 measure  Although, unlike INLINEFORM1 whose scores are greater than or equal to 0 without an upper limit, the INLINEFORM2 measure returns scores between 0 and 1 (inclusive)  Therefore, here a threshold of 0 indicates that all of the bigrams were included in the similarity matrix; and a threshold of INLINEFORM3 indicates that only the bigram pairs with a similarity score greater than INLINEFORM4 were included  The results show an increase in accuracy for all of the datasets except for the MiniMayoSRS tagged for physicians  The increase in the results for the UMNSRS tagged for similarity and the MayoSRS is statistically significant ( INLINEFORM5 )  This is not the case for the UMNSRS tagged for relatedness nor the MiniMayoSRS 
Overall, these results indicate that including only those bigrams that have a sufficiently high similarity score increases the correlation results with human judgments, but what quantifies as sufficiently high varies depending on the dataset and measure 
Comparison with Previous Work
Recently, word embeddings BIBREF9 have become a popular method for measuring semantic relatedness in the biomedical domain  This is a neural network based approach that learns a representation of a word by word co–occurrence matrix  The basic idea is that the neural network learns a series of weights (the hidden layer within the neural network) that either maximizes the probability of a word given its context, referred to as the continuous bag of words (CBOW) approach, or that maximizes the probability of the context given a word, referred to as the Skip–gram approach  These approaches have been used in numerous recent papers 
muneeb2015evalutating trained both the Skip–gram and CBOW models over the PubMed Central Open Access (PMC) corpus of approximately 1 25 million articles  They evaluated the models on a subset of the UMNSRS data, removing word pairs that did not occur in their training corpus more than ten times  chiu2016how evaluated both the the Skip–gram and CBOW models over the PMC corpus and PubMed  They also evaluated the models on a subset of the UMNSRS ignoring those words that did not appear in their training corpus  Pakhomov2016corpus trained CBOW model over three different types of corpora: clinical (clinical notes from the Fairview Health System), biomedical (PMC corpus), and general English (Wikipedia)  They evaluated their method using a subset of the UMNSRS restricting to single word term pairs and removing those not found within their training corpus  sajad2015domain trained the Skip–gram model over CUIs identified by MetaMap on the OHSUMED corpus, a collection of 348,566 biomedical research articles  They evaluated the method on the complete UMNSRS, MiniMayoSRS and the MayoSRS datasets; any subset information about the dataset was not explicitly stated therefore we believe a direct comparison may be possible 
In addition, a previous work very closely related to ours is a retrofitting vector method proposed by YuCBJW16 that incorporates ontological information into a vector representation by including semantically related words  In their measure, they first map a biomedical term to MeSH terms, and second build a word vector based on the documents assigned to the respective MeSH term  They then retrofit the vector by including semantically related words found in the Unified Medical Language System  They evaluate their method on the MiniMayoSRS dataset 
Table TABREF31 shows a comparison to the top correlation scores reported by each of these works on the respective datasets (or subsets) they evaluated their methods on  N refers to the number of term pairs in the dataset the authors report they evaluated their method  The table also includes our top scoring results: the integrated vector-res and vector-faith  The results show that integrating semantic similarity measures into second–order co–occurrence vectors obtains a higher or on–par correlation with human judgments as the previous works reported results with the exception of the UMNSRS rel dataset  The results reported by Pakhomov2016corpus and chiu2016how obtain a higher correlation although the results can not be directly compared because both works used different subsets of the term pairs from the UMNSRS dataset 
Conclusion and Future Work
We have presented a method for quantifying the similarity and relatedness between two terms that integrates pair–wise similarity scores into second–order vectors  The goal of this approach is two–fold  First, we restrict the context used by the vector measure to words that exist in the biomedical domain, and second, we apply larger weights to those word pairs that are more similar to each other  Our hypothesis was that this combination would reduce the amount of noise in the vectors and therefore increase their correlation with human judgments  We evaluated our method on datasets that have been manually annotated for relatedness and similarity and found evidence to support this hypothesis  In particular we discovered that guiding the creation of a second–order context vector by selecting term pairs from biomedical text based on their semantic similarity led to improved levels of correlation with human judgment 
We also explored using a threshold cutoff to include only those term pairs that obtained a sufficiently large level of similarity  We found that eliminating less similar pairs improved the overall results (to a point)  In the future, we plan to explore metrics to automatically determine the threshold cutoff appropriate for a given dataset and measure  We also plan to explore additional features that can be integrated with a second–order vector measure that will reduce the noise but still provide sufficient information to quantify relatedness  We are particularly interested in approaches that learn word, phrase, and sentence embeddings from structured corpora such as literature BIBREF23 and dictionary entries BIBREF24   Such embeddings could be integrated into a second–order vector or be used on their own 
Finally, we compared our proposed method to other distributional approaches, focusing on those that used word embeddings  Our results showed that integrating semantic similarity measures into second–order co–occurrence vectors obtains the same or higher correlation with human judgments as do various different word embedding approaches  However, a direct comparison was not possible due to variations in the subsets of the UMNSRS evaluation dataset used  In the future, we would not only like to conduct a direct comparison but also explore integrating semantic similarity into various kinds of word embeddings by training on pair–wise values of semantic similarity as well as co–occurrence statistics 

ground_truth: Skip–gram, CBOW

integrated vector-res, vector-faith, Skip–gram, CBOW