prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. Which toolkits do they use?
Context 1: Introduction
As social media, specially Twitter, takes on an influential role in presidential elections in the U S , natural language processing of political tweets BIBREF0 has the potential to help with nowcasting and forecasting of election results as well as identifying the main issues with a candidate – tasks of much interest to journalists, political scientists, and campaign organizers BIBREF1  As a methodology to obtain training data for a machine learning system that analyzes political tweets, BIBREF2 devised a crowdsourcing scheme with variable crowdworker numbers based on the difficulty of the annotation task  They provided a dataset of tweets where the sentiments towards political candidates were labeled both by experts in political communication and by crowdworkers who were likely not domain experts  BIBREF2 revealed that crowdworkers can match expert performance relatively accurately and in a budget-efficient manner  Given this result, the authors envisioned future work in which groundtruth labels would be crowdsourced for a large number of tweets and then used to design an automated NLP tool for political tweet analysis 
The question we address here is: How accurate are existing NLP tools for political tweet analysis? These tools would provide a baseline performance that any new machine learning system for political tweet analysis would compete against  We here explore whether existing NLP systems can answer the questions "What sentiment?" and "Towards whom?" accurately for the dataset of political tweets provided by BIBREF2  In our analysis, we include NLP tools with publicly-available APIs, even if the tools were not specifically designed for short texts like tweets, and, in particular, political tweets 
Our experiments reveal that the task of entity-level sentiment analysis is difficult for existing tools to answer accurately while the recognition of the entity, here, which politician, was easier 
NLP Toolkits
NLP toolkits typically have the following capabilities: tokenization, part-of-speech (PoS) tagging, chunking, named entity recognition and sentiment analysis  In a study by BIBREF3, it is shown that the well-known NLP toolkits NLTK BIBREF4, Stanford CoreNLP BIBREF5, and TwitterNLP BIBREF6 have tokenization, PoS tagging and NER modules in their pipelines  There are two main approaches for NER: (1) rule-based and (2) statistical or machine learning based  The most ubiquitous algorithms for sequence tagging use Hidden Markov Models BIBREF7, Maximum Entropy Markov Models BIBREF7, BIBREF8, or Conditional Random Fields BIBREF9  Recent works BIBREF10, BIBREF11 have used recurrent neural networks with attention modules for NER 
Sentiment detection tools like SentiStrength BIBREF12 and TensiStrength BIBREF13 are rule-based tools, relying on various dictionaries of emoticons, slangs, idioms, and ironic phrases, and set of rules that can detect the sentiment of a sentence overall or a targeted sentiment  Given a list of keywords, TensiStrength (similar to SentiStrength) reports the sentiment towards selected entities in a sentence, based on five levels of relaxation and five levels of stress 
Among commercial NLP toolkits (e g , BIBREF14, BIBREF15, BIBREF16), we selected BIBREF17 and BIBREF18 for our experiments, which, to the best of our knowledge, are the only publicly accessible commercial APIs for the task of entity-level sentiment analysis that is agnostic to the text domain  We also report results of TensiStrength BIBREF13, TwitterNLP BIBREF6, BIBREF19, CogComp-NLP BIBREF20, and Stanford NLP NER BIBREF21 
Dataset and Analysis Methodology
We used the 1,000-tweet dataset by BIBREF2 that contains the named-entities labels and entity-level sentiments for each of the four 2016 presidential primary candidates Bernie Sanders, Donald Trump, Hillary Clinton, and Ted Cruz, provided by crowdworkers, and by experts in political communication, whose labels are considered groundtruth  The crowdworkers were located in the US and hired on the BIBREF22 platform  For the task of entity-level sentiment analysis, a 3-scale rating of "negative," "neutral," and "positive" was used by the annotators 
BIBREF2 proposed a decision tree approach for computing the number of crowdworkers who should analyze a tweet based on the difficulty of the task  Tweets are labeled by 2, 3, 5, or 7 workers based on the difficulty of the task and the level of disagreement between the crowdworkers  The model computes the number of workers based on how long a tweet is, the presence of a link in a tweet, and the number of present sarcasm signals  Sarcasm is often used in political tweets and causes disagreement between the crowdworkers  The tweets that are deemed to be sarcastic by the decision tree model, are expected to be more difficult to annotate, and hence are allocated more crowdworkers to work on 
We conducted two sets of experiments  In the first set, we used BIBREF23, BIBREF17, and BIBREF18, for entity-level sentiment analysis; in the second set, BIBREF17, BIBREF19, BIBREF24, BIBREF25, and BIBREF26, BIBREF18 for named-entity recognition 
In the experiments that we conducted with TwitterNLP for named-entity recognition, we worked with the default values of the model  Furthermore, we selected the 3-class Stanford NER model, which uses the classes “person,” “organization,” and “location” because it resulted in higher accuracy compared to the 7-class model  For CogComp-NLP NER we used Ontonotes 5 0 NER model BIBREF27  For spaCy NER we used the `en_core_web_lg' model 
We report the experimental results for our two tasks in terms of the correct classification rate (CCR)  For sentiment analysis, we have a three-class problem (positive, negative, and neutral), where the classes are mutually exclusive  The CCR, averaged for a set of tweets, is defined to be the number of correctly-predicted sentiments over the number of groundtruth sentiments in these tweets  For NER, we consider that each tweet may reference up to four candidates, i e , targeted entities  The CCR, averaged for a set of tweets, is the number of correctly predicted entities (candidates) over the number of groundtruth entities (candidates) in this set 
Results and Discussion
The dataset of 1,000 randomly selected tweets contains more than twice as many tweets about Trump than about the other candidates  In the named-entity recognition experiment, the average CCR of crowdworkers was 98 6%, while the CCR of the automated systems ranged from 77 2% to 96 7%  For four of the automated systems, detecting the entity Trump was more difficult than the other entities (e g , spaCy 72 7% for the entity Trump vs  above 91% for the other entities)  An example of incorrect NER is shown in Figure FIGREF1 top  The difficulties the automated tools had in NER may be explained by the fact that the tools were not trained on tweets, except for TwitterNLP, which was not in active development when the data was created BIBREF1 
In the sentiment analysis experiments, we found that a tweet may contain multiple sentiments  The groundtruth labels contain 210 positive sentiments, 521 neutral sentiments, and 305 negative sentiments to the candidates  We measured the CCR, across all tweets, to be 31 7% for Rosette Text Analytics, 43 2% for Google Cloud, 44 2% for TensiStrength, and 74 7% for the crowdworkers  This means the difference between the performance of the tools and the crowdworkers is significant – more than 30 percent points 
Crowdworkers correctly identified 62% of the neutral, 85% of the positive, and 92% of the negative sentiments  Google Cloud correctly identified 88% of the neutral sentiments, but only 3% of the positive, and 19% of the negative sentiments  TensiStrength correctly identified 87 2% of the neutral sentiments, but 10 5% of the positive, and 8 1% of the negative sentiments  Rosette Text Analytics correctly identified 22 7% of neutral sentiments, 38 1% of negative sentiments and 40 9% of positive sentiments  The lowest and highest CCR pertains to tweets about Trump and Sanders for both Google Cloud and TensiStrength, Trump and Clinton for Rosette Text Analytics, and Clinton and Cruz for crowdworkers  An example of incorrect ELS analysis is shown in Figure FIGREF1 bottom 
Conclusions and Future Work
Our results show that existing NLP systems cannot accurately perform sentiment analysis of political tweets in the dataset we experimented with  Labeling by humans, even non-expert crowdworkers, yields accuracy results that are well above the results of existing automated NLP systems  In future work we will therefore use a crowdworker-labeled dataset to train a new machine-learning based NLP system for tweet analysis  We will ensure that the training data is balanced among classes  Our plan is to use state-of-the-art deep neural networks and compare their performance for entity-level sentiment analysis of political tweets 
Acknowledgments
Partial support of this work by the Hariri Institute for Computing and Computational Science & Engineering at Boston University (to L G ) and a Google Faculty Research Award (to M B  and L G ) is gratefully acknowledged  Additionally, we would like to thank Daniel Khashabi for his help in running the CogComp-NLP Python API and Mike Thelwal for his help with TensiStrength  We are also grateful to the Stanford NLP group for clarifying some of the questions we had with regards to the Stanford NER tool 
Context 2: Introduction
Question Generation (QG) is the task of automatically creating questions from a range of inputs, such as natural language text BIBREF0, knowledge base BIBREF1 and image BIBREF2  QG is an increasingly important area in NLP with various application scenarios such as intelligence tutor systems, open-domain chatbots and question answering dataset construction  In this paper, we focus on question generation from reading comprehension materials like SQuAD BIBREF3  As shown in Figure FIGREF1, given a sentence in the reading comprehension paragraph and the text fragment (i e , the answer) that we want to ask about, we aim to generate a question that is asked about the specified answer 
Question generation for reading comprehension is firstly formalized as a declarative-to-interrogative sentence transformation problem with predefined rules or templates BIBREF4, BIBREF0  With the rise of neural models, Du2017LearningTA propose to model this task under the sequence-to-sequence (Seq2Seq) learning framework BIBREF5 with attention mechanism BIBREF6  However, question generation is a one-to-many sequence generation problem, i e , several aspects can be asked given a sentence  Zhou2017NeuralQG propose the answer-aware question generation setting which assumes the answer, a contiguous span inside the input sentence, is already known before question generation  To capture answer-relevant words in the sentence, they adopt a BIO tagging scheme to incorporate the answer position embedding in Seq2Seq learning  Furthermore, Sun2018AnswerfocusedAP propose that tokens close to the answer fragments are more likely to be answer-relevant  Therefore, they explicitly encode the relative distance between sentence words and the answer via position embedding and position-aware attention 
Although existing proximity-based answer-aware approaches achieve reasonable performance, we argue that such intuition may not apply to all cases especially for sentences with complex structure  For example, Figure FIGREF1 shows such an example where those approaches fail  This sentence contains a few facts and due to the parenthesis (i e  “the area's coldest month”), some facts intertwine: “The daily mean temperature in January is 0 3$^\circ $C” and “January is the area's coldest month”  From the question generated by a proximity-based answer-aware baseline, we find that it wrongly uses the word “coldest” but misses the correct word “mean” because “coldest” has a shorter distance to the answer “0 3$^\circ $C” 
In summary, their intuition that “the neighboring words of the answer are more likely to be answer-relevant and have a higher chance to be used in the question” is not reliable  To quantitatively show this drawback of these models, we implement the approach proposed by Sun2018AnswerfocusedAP and analyze its performance under different relative distances between the answer and other non-stop sentence words that also appear in the ground truth question  The results are shown in Table TABREF2  We find that the performance drops at most 36% when the relative distance increases from “$0\sim 10$” to “$>10$”  In other words, when the useful context is located far away from the answer, current proximity-based answer-aware approaches will become less effective, since they overly emphasize neighboring words of the answer 
To address this issue, we extract the structured answer-relevant relations from sentences and propose a method to jointly model such structured relation and the unstructured sentence for question generation  The structured answer-relevant relation is likely to be to the point context and thus can help keep the generated question to the point  For example, Figure FIGREF1 shows our framework can extract the right answer-relevant relation (“The daily mean temperature in January”, “is”, “32 6$^\circ $F (0 3$^\circ $C)”) among multiple facts  With the help of such structured information, our model is less likely to be confused by sentences with a complex structure  Specifically, we firstly extract multiple relations with an off-the-shelf Open Information Extraction (OpenIE) toolbox BIBREF7, then we select the relation that is most relevant to the answer with carefully designed heuristic rules 
Nevertheless, it is challenging to train a model to effectively utilize both the unstructured sentence and the structured answer-relevant relation because both of them could be noisy: the unstructured sentence may contain multiple facts which are irrelevant to the target question, while the limitation of the OpenIE tool may produce less accurate extracted relations  To explore their advantages simultaneously and avoid the drawbacks, we design a gated attention mechanism and a dual copy mechanism based on the encoder-decoder framework, where the former learns to control the information flow between the unstructured and structured inputs, while the latter learns to copy words from two sources to maintain the informativeness and faithfulness of generated questions 
In the evaluations on the SQuAD dataset, our system achieves significant and consistent improvement as compared to all baseline methods  In particular, we demonstrate that the improvement is more significant with a larger relative distance between the answer and other non-stop sentence words that also appear in the ground truth question  Furthermore, our model is capable of generating diverse questions for a single sentence-answer pair where the sentence conveys multiple relations of its answer fragment 
Framework Description
In this section, we first introduce the task definition and our protocol to extract structured answer-relevant relations  Then we formalize the task under the encoder-decoder framework with gated attention and dual copy mechanism 
Framework Description ::: Problem Definition
We formalize our task as an answer-aware Question Generation (QG) problem BIBREF8, which assumes answer phrases are given before generating questions  Moreover, answer phrases are shown as text fragments in passages  Formally, given the sentence $S$, the answer $A$, and the answer-relevant relation $M$, the task of QG aims to find the best question $\overline{Q}$ such that,
where $A$ is a contiguous span inside $S$ 
Framework Description ::: Answer-relevant Relation Extraction
We utilize an off-the-shelf toolbox of OpenIE to the derive structured answer-relevant relations from sentences as to the point contexts  Relations extracted by OpenIE can be represented either in a triple format or in an n-ary format with several secondary arguments, and we employ the latter to keep the extractions as informative as possible and avoid extracting too many similar relations in different granularities from one sentence  We join all arguments in the extracted n-ary relation into a sequence as our to the point context  Figure FIGREF5 shows n-ary relations extracted from OpenIE  As we can see, OpenIE extracts multiple relations for complex sentences  Here we select the most informative relation according to three criteria in the order of descending importance: (1) having the maximal number of overlapped tokens between the answer and the relation; (2) being assigned the highest confidence score by OpenIE; (3) containing maximum non-stop words  As shown in Figure FIGREF5, our criteria can select answer-relevant relations (waved in Figure FIGREF5), which is especially useful for sentences with extraneous information  In rare cases, OpenIE cannot extract any relation, we treat the sentence itself as the to the point context 
Table TABREF8 shows some statistics to verify the intuition that the extracted relations can serve as more to the point context  We find that the tokens in relations are 61% more likely to be used in the target question than the tokens in sentences, and thus they are more to the point  On the other hand, on average the sentences contain one more question token than the relations (1 86 v s  2 87)  Therefore, it is still necessary to take the original sentence into account to generate a more accurate question 
Framework Description ::: Our Proposed Model ::: Overview 
As shown in Figure FIGREF10, our framework consists offour components (1) Sentence Encoder and Relation Encoder, (2) Decoder, (3) Gated Attention Mechanism and (4) Dual Copy Mechanism  The sentence encoder and relation encoder encode the unstructured sentence and the structured answer-relevant relation, respectively  To select and combine the source information from the two encoders, a gated attention mechanism is employed to jointly attend both contextualized information sources, and a dual copy mechanism copies words from either the sentence or the relation 
Framework Description ::: Our Proposed Model ::: Answer-aware Encoder 
We employ two encoders to integrate information from the unstructured sentence $S$ and the answer-relevant relation $M$ separately  Sentence encoder takes in feature-enriched embeddings including word embeddings $\mathbf {w}$, linguistic embeddings $\mathbf {l}$ and answer position embeddings $\mathbf {a}$  We follow BIBREF9 to transform POS and NER tags into continuous representation ($\mathbf {l}^p$ and $\mathbf {l}^n$) and adopt a BIO labelling scheme to derive the answer position embedding (B: the first token of the answer, I: tokens within the answer fragment except the first one, O: tokens outside of the answer fragment)  For each word $w_i$ in the sentence $S$, we simply concatenate all features as input: $\mathbf {x}_i^s= [\mathbf {w}_i; \mathbf {l}^p_i; \mathbf {l}^n_i; \mathbf {a}_i]$  Here $[\mathbf {a};\mathbf {b}]$ denotes the concatenation of vectors $\mathbf {a}$ and $\mathbf {b}$ 
We use bidirectional LSTMs to encode the sentence $(\mathbf {x}_1^s, \mathbf {x}_2^s,    , \mathbf {x}_n^s)$ to get a contextualized representation for each token:
where $\overrightarrow{\mathbf {h}}^{s}_i$ and $\overleftarrow{\mathbf {h}}^{s}_i$ are the hidden states at the $i$-th time step of the forward and the backward LSTMs  The output state of the sentence encoder is the concatenation of forward and backward hidden states: $\mathbf {h}^{s}_i=[\overrightarrow{\mathbf {h}}^{s}_i;\overleftarrow{\mathbf {h}}^{s}_i]$  The contextualized representation of the sentence is $(\mathbf {h}^{s}_1, \mathbf {h}^{s}_2,    , \mathbf {h}^{s}_n)$ 
For the relation encoder, we firstly join all items in the n-ary relation $M$ into a sequence  Then we only take answer position embedding as an extra feature for the sequence: $\mathbf {x}_i^m= [\mathbf {w}_i; \mathbf {a}_i]$  Similarly, we take another bidirectional LSTMs to encode the relation sequence and derive the corresponding contextualized representation $(\mathbf {h}^{m}_1, \mathbf {h}^{m}_2,    , \mathbf {h}^{m}_n)$ 
Framework Description ::: Our Proposed Model ::: Decoder 
We use an LSTM as the decoder to generate the question  The decoder predicts the word probability distribution at each decoding timestep to generate the question  At the t-th timestep, it reads the word embedding $\mathbf {w}_{t}$ and the hidden state $\mathbf {u}_{t-1}$ of the previous timestep to generate the current hidden state:
Framework Description ::: Our Proposed Model ::: Gated Attention Mechanism 
We design a gated attention mechanism to jointly attend the sentence representation and the relation representation  For sentence representation $(\mathbf {h}^{s}_1, \mathbf {h}^{s}_2,    , \mathbf {h}^{s}_n)$, we employ the Luong2015EffectiveAT's attention mechanism to obtain the sentence context vector $\mathbf {c}^s_t$,
where $\mathbf {W}_a$ is a trainable weight  Similarly, we obtain the vector $\mathbf {c}^m_t$ from the relation representation $(\mathbf {h}^{m}_1, \mathbf {h}^{m}_2,    , \mathbf {h}^{m}_n)$  To jointly model the sentence and the relation, a gating mechanism is designed to control the information flow from two sources:
where $\odot $ represents element-wise dot production and $\mathbf {W}_g, \mathbf {W}_h$ are trainable weights  Finally, the predicted probability distribution over the vocabulary $V$ is computed as:
where $\mathbf {W}_V$ and $\mathbf {b}_V$ are parameters 
Framework Description ::: Our Proposed Model ::: Dual Copy Mechanism 
To deal with the rare and unknown words, the decoder applies the pointing method BIBREF10, BIBREF11, BIBREF12 to allow copying a token from the input sentence at the $t$-th decoding step  We reuse the attention score $\mathbf {\alpha }_{t}^s$ and $\mathbf {\alpha }_{t}^m$ to derive the copy probability over two source inputs:
Different from the standard pointing method, we design a dual copy mechanism to copy from two sources with two gates  The first gate is designed for determining copy tokens from two sources of inputs or generate next word from $P_V$, which is computed as $g^v_t = \text{sigmoid}(\mathbf {w}^v_g \tilde{\mathbf {h}}_t + b^v_g)$  The second gate takes charge of selecting the source (sentence or relation) to copy from, which is computed as $g^c_t = \text{sigmoid}(\mathbf {w}^c_g [\mathbf {c}_t^s;\mathbf {c}_t^m] + b^c_g)$  Finally, we combine all probabilities $P_V$, $P_S$ and $P_M$ through two soft gates $g^v_t$ and $g^c_t$
Context 3: Introduction
Neural Machine Translation (NMT) has shown its effectiveness in translation tasks when NMT systems perform best in recent machine translation campaigns BIBREF0 , BIBREF1   Compared to phrase-based Statistical Machine Translation (SMT) which is basically an ensemble of different features trained and tuned separately, NMT directly modeling the translation relationship between source and target sentences  Unlike SMT, NMT does not require much linguistic information and large monolingual data to achieve good performances 
An NMT consists of an encoder which recursively reads and represents the whole source sentence into a context vector and a recurrent decoder which takes the context vector and its previous state to predict the next target word  It is then trained in an end-to-end fashion to learn parameters which maximizes the likelihood between the outputs and the references  Recently, attention-based NMT has been featured in most state-of-the-art systems  First introduced by BIBREF2 , attention mechanism is integrated in decoder side as feedforward layers  It allows the NMT to decide which source words should take part in the predicting process of the next target words  It helps to improve NMTs significantly  Nevertheless, since the attention mechanism is specific to a particular source sentence and the considering target word, it is also specific to particular language pairs 
Some recent work has focused on extending the NMT framework to multilingual scenarios  By training such network using parallel corpora in number of different languages, NMT could benefit from additional information embedded in a common semantic space across languages  Basically, the proposed NMT are required to employ multiple encoders or multiple decoders to deal with multilinguality  Furthermore, in order to avoid the tight dependency of the attention mechanism to specific language pairs, they also need to modify their architecture to combine either the encoders or the attention layers  These modifications are specific to the purpose of the tasks as well  Thus, those multilingual NMTs are more complicated, much more free parameters to learn and more difficult to perform standard trainings compared to the original NMT 
In this paper, we introduce a unified approach to seamlessly extend the original NMT to multilingual settings  Our approach allows us to integrate any language in any side of the encoder-decoder architecture with only one encoder and one decoder for all the languages involved  Moreover, it is not necessary to do any network modification to enable attention mechanism in our NMT systems  We then apply our proprosed framework in two demanding scenarios: under-resourced translation and zero-resourced translation  The results show that bringing multilinguality to NMT helps to improve individual translations  With some insightful analyses of the results, we set our goal toward a fully multilingual NMT framework 
The paper starts with a detailed introduction to attention-based NMT  In Section SECREF3 , related work about multi-task NMT is reviewed  Section SECREF5 describes our proposed approach and thorough comparisons to the related work  It is followed by a section of evaluating our systems in two aforementioned scenarios, in which different strategies have been employed under a unified approach (Section SECREF4 )  Finally, the paper ends with conclusion and future work 
This work is licenced under a Creative Commons Attribution 4 0 International License  License details: http://creativecommons org/licenses/by/4 0/
Neural Machine Translation: Background
An NMT system consists of an encoder which automatically learns the characteristics of a source sentence into fix-length context vectors and a decoder that recursively combines the produced context vectors with the previous target word to generate the most probable word from a target vocabulary 
More specifically, a bidirectional recurrent encoder reads every words INLINEFORM0 of a source sentence INLINEFORM1 and encodes a representation INLINEFORM2 of the sentence into a fixed-length vector INLINEFORM3 concatinated from those of the forward and backward directions: INLINEFORM4
Here INLINEFORM0 is the one-hot vector of the word INLINEFORM1 and INLINEFORM2 is the word embedding matrix which is shared across the source words  INLINEFORM3 is the recurrent unit computing the current hidden state of the encoder based on the previous hidden state  INLINEFORM4 is then called an annotation vector, which encodes the source sentence up to the time INLINEFORM5 from both forward and backward directions  Recurrent units in NMT can be a simple recurrent neural network unit (RNN), a Long Short-Term Memory unit (LSTM) BIBREF3 or a Gated Recurrent Unit (GRU) BIBREF4
Similar to the encoder, the recurrent decoder generates one target word INLINEFORM0 to form a translated target sentence INLINEFORM1 in the end  At the time INLINEFORM2 , it takes the previous hidden state of the decoder INLINEFORM3 , the previous embedded word representation INLINEFORM4 and a time-specific context vector INLINEFORM5 as inputs to calculate the current hidden state INLINEFORM6 : INLINEFORM7
Again, INLINEFORM0 is the recurrent activation function of the decoder and INLINEFORM1 is the shared word embedding matrix of the target sentences  The context vector INLINEFORM2 is calculated based on the annotation vectors from the encoder  Before feeding the annotation vectors into the decoder, an attention mechanism is set up in between, in order to choose which annotation vectors should contribute to the predicting decision of the next target word  Intuitively, a relevance between the previous target word and the annotation vectors can be used to form some attention scenario  There exists several ways to calculate the relevance as shown in BIBREF5 , but what we describe here follows the proposed method of BIBREF2 DISPLAYFORM0
In BIBREF2 , this attention mechanism, originally called alignment model, has been employed as a simple feedforward network with the first layer is a learnable layer via INLINEFORM0 , INLINEFORM1 and INLINEFORM2   The relevance scores INLINEFORM3 are then normalized into attention weights INLINEFORM4 and the context vector INLINEFORM5 is calculated as the weighted sum of all annotation vectors INLINEFORM6   Depending on how much attention the target word at time INLINEFORM7 put on the source states INLINEFORM8 , a soft alignment is learned  By being employed this way, word alignment is not a latent variable but a parametrized function, making the alignment model differentiable  Thus, it could be trained together with the whole architecture using backpropagation 
One of the most severe problems of NMT is handling of the rare words, which are not in the short lists of the vocabularies, i e  out-of-vocabulary (OOV) words, or do not appear in the training set at all  In BIBREF6 , the rare target words are copied from their aligned source words after the translation  This heuristic works well with OOV words and named entities but unable to translate unseen words  In BIBREF7 , their proposed NMT models have been shown to not only be effective on reducing vocabulary sizes but also have the ability to generate unseen words  This is achieved by segmenting the rare words into subword units and translating them  The state-of-the-art translation systems essentially employ subword NMT BIBREF7  
Universal Encoder and Decoder for Multilingual Neural Machine Translation
While the majority of previous research has focused on improving the performance of NMT on individual language pairs with individual NMT systems, recent work has started investigating potential ways to conduct the translation involved in multiple languages using a single NMT system  The possible reason explaining these efforts lies on the unique architecture of NMT  Unlike SMT, NMT consists of separated neural networks for the source and target sides, or the encoder and decoder, respectively  This allows these components to map a sentence in any language to a representation in an embedding space which is believed to share common semantics among the source languages involved  From that shared space, the decoder, with some implicit or explicit relevant constraints, could transform the representation into a concrete sentence in any desired language  In this section, we review some related work on this matter  We then describe a unified approach toward an universal attention-based NMT scheme  Our approach does not require any architecture modification and it can be trained to learn a minimal number of parameters compared to the other work 
Related Work
By extending the solution of sequence-to-sequence modeling using encoder-decoder architectures to multi-task learning, Luong2016 managed to achieve better performance on some INLINEFORM0 tasks such as translation, parsing and image captioning compared to individual tasks  Specifically in translation, the work utilizes multiple encoders to translate from multiple languages, and multiple decoders to translate to multiple languages  In this view of multilingual translation, each language in source or target side is modeled by one encoder or decoder, depending on the side of the translation  Due to the natural diversity between two tasks in that multi-task learning scenario, e g  translation and parsing, it could not feature the attention mechanism although it has proven its effectiveness in NMT  There exists two directions which proposed for multilingual translation scenarios where they leverage the attention mechanism  The first one is indicated in the work from BIBREF8 , where it introduce an one-to-many multilingual NMT system to translates from one source language into multiple target languages  Having one source language, the attention mechanism is then handed over to the corresponding decoder  The objective function is changed to adapt to multilingual settings  In testing time, the parameters specific to a desired language pair are used to perform the translation

ground_truth: BIBREF17, BIBREF18, TensiStrength BIBREF13, TwitterNLP BIBREF6, BIBREF19, CogComp-NLP BIBREF20, Stanford NLP NER BIBREF21

BIBREF23, BIBREF17, BIBREF18, BIBREF19, BIBREF24, BIBREF25, BIBREF26

prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. On what datasets are experiments performed?
Context 2:  The probability of predicting $w$ as the $t$-th token of the question is:
Framework Description ::: Our Proposed Model ::: Training and Inference 
Given the answer $A$, sentence $S$ and relation $M$, the training objective is to minimize the negative log-likelihood with regard to all parameters:
where $\mathcal {\lbrace }Q\rbrace $ is the set of all training instances, $\theta $ denotes model parameters and $\text{log} P(Q|A,S,M;\theta )$ is the conditional log-likelihood of $Q$ 
In testing, our model targets to generate a question $Q$ by maximizing:
Experimental Setting ::: Dataset & Metrics
We conduct experiments on the SQuAD dataset BIBREF3  It contains 536 Wikipedia articles and 100k crowd-sourced question-answer pairs  The questions are written by crowd-workers and the answers are spans of tokens in the articles  We employ two different data splits by following Zhou2017NeuralQG and Du2017LearningTA   In Zhou2017NeuralQG, the original SQuAD development set is evenly divided into dev and test sets, while Du2017LearningTA treats SQuAD development set as its development set and splits original SQuAD training set into a training set and a test set  We also filter out questions which do not have any overlapped non-stop words with the corresponding sentences and perform some preprocessing steps, such as tokenization and sentence splitting  The data statistics are given in Table TABREF27 
We evaluate with all commonly-used metrics in question generation BIBREF13: BLEU-1 (B1), BLEU-2 (B2), BLEU-3 (B3), BLEU-4 (B4) BIBREF17, METEOR (MET) BIBREF18 and ROUGE-L (R-L) BIBREF19  We use the evaluation script released by Chen2015MicrosoftCC 
Experimental Setting ::: Baseline Models
We compare with the following models 
[leftmargin=*]
s2s BIBREF13 proposes an attention-based sequence-to-sequence neural network for question generation 
NQG++ BIBREF9 takes the answer position feature and linguistic features into consideration and equips the Seq2Seq model with copy mechanism 
M2S+cp BIBREF14 conducts multi-perspective matching between the answer and the sentence to derive an answer-aware sentence representation for question generation 
s2s+MP+GSA BIBREF8 introduces a gated self-attention into the encoder and a maxout pointer mechanism into the decoder  We report their sentence-level results for a fair comparison 
Hybrid BIBREF15 is a hybrid model which considers the answer embedding for the question word generation and the position of context words for modeling the relative distance between the context words and the answer 
ASs2s BIBREF16 replaces the answer in the sentence with a special token to avoid its appearance in the generated questions 
Experimental Setting ::: Implementation Details
We take the most frequent 20k words as our vocabulary and use the GloVe word embeddings BIBREF20 for initialization  The embedding dimensions for POS, NER, answer position are set to 20  We use two-layer LSTMs in both encoder and decoder, and the LSTMs hidden unit size is set to 600 
We use dropout BIBREF21 with the probability $p=0 3$  All trainable parameters, except word embeddings, are randomly initialized with the Xavier uniform in $(-0 1, 0 1)$ BIBREF22  For optimization in the training, we use SGD as the optimizer with a minibatch size of 64 and an initial learning rate of 1 0  We train the model for 15 epochs and start halving the learning rate after the 8th epoch  We set the gradient norm upper bound to 3 during the training 
We adopt the teacher-forcing for the training  In the testing, we select the model with the lowest perplexity and beam search with size 3 is employed for generating questions  All hyper-parameters and models are selected on the validation dataset 
Results and Analysis ::: Main Results
Table TABREF30 shows automatic evaluation results for our model and baselines (copied from their papers)  Our proposed model which combines structured answer-relevant relations and unstructured sentences achieves significant improvements over proximity-based answer-aware models BIBREF9, BIBREF15 on both dataset splits  Presumably, our structured answer-relevant relation is a generalization of the context explored by the proximity-based methods because they can only capture short dependencies around answer fragments while our extractions can capture both short and long dependencies given the answer fragments  Moreover, our proposed framework is a general one to jointly leverage structured relations and unstructured sentences  All compared baseline models which only consider unstructured sentences can be further enhanced under our framework 
Recall that existing proximity-based answer-aware models perform poorly when the distance between the answer fragment and other non-stop sentence words that also appear in the ground truth question is large (Table TABREF2)  Here we investigate whether our proposed model using the structured answer-relevant relations can alleviate this issue or not, by conducting experiments for our model under the same setting as in Table TABREF2  The broken-down performances by different relative distances are shown in Table TABREF40  We find that our proposed model outperforms Hybrid (our re-implemented version for this experiment) on all ranges of relative distances, which shows that the structured answer-relevant relations can capture both short and long term answer-relevant dependencies of the answer in sentences  Furthermore, comparing the performance difference between Hybrid and our model, we find the improvements become more significant when the distance increases from “$0\sim 10$” to “$>10$”  One reason is that our model can extract relations with distant dependencies to the answer, which greatly helps our model ignore the extraneous information  Proximity-based answer-aware models may overly emphasize the neighboring words of answers and become less effective as the useful context becomes further away from the answer in the complex sentences  In fact, the breakdown intervals in Table TABREF40 naturally bound its sentence length, say for “$>10$”, the sentences in this group must be longer than 10  Thus, the length variances in these two intervals could be significant  To further validate whether our model can extract long term dependency words  We rerun the analysis of Table TABREF40 only for long sentences (length $>$ 20) of each interval  The improvement percentages over Hybrid are shown in Table TABREF40, which become more significant when the distance increases from “$0\sim 10$” to “$>10$” 
Results and Analysis ::: Case Study
Figure FIGREF42 provides example questions generated by crowd-workers (ground truth questions), the baseline Hybrid BIBREF15, and our model  In the first case, there are two subsequences in the input and the answer has no relation with the second subsequence  However, we see that the baseline model prediction copies irrelevant words “The New York Times” while our model can avoid using the extraneous subsequence “The New York Times noted    ” with the help of the structured answer-relevant relation  Compared with the ground truth question, our model cannot capture the cross-sentence information like “her fifth album”, where the techniques in paragraph-level QG models BIBREF8 may help  In the second case, as discussed in Section SECREF1, this sentence contains a few facts and some facts intertwine  We find that our model can capture distant answer-relevant dependencies such as “mean temperature” while the proximity-based baseline model wrongly takes neighboring words of the answer like “coldest” in the generated question 
Results and Analysis ::: Diverse Question Generation
Another interesting observation is that for the same answer-sentence pair, our model can generate diverse questions by taking different answer-relevant relations as input  Such capability improves the interpretability of our model because the model is given not only what to be asked (i e , the answer) but also the related fact (i e , the answer-relevant relation) to be covered in the question  In contrast, proximity-based answer-aware models can only generate one question given the sentence-answer pair regardless of how many answer-relevant relations in the sentence  We think such capability can also validate our motivation: questions should be generated according to the answer-aware relations instead of neighboring words of answer fragments  Figure FIGREF45 show two examples of diverse question generation  In the first case, the answer fragment `Hugh L  Dryden' is the appositive to `NASA Deputy Administrator' but the subject to the following tokens `announced the Apollo program    '  Our framework can extract these two answer-relevant relations, and by feeding them to our model separately, we can receive two questions asking different relations with regard to the answer 
Related Work
The topic of question generation, initially motivated for educational purposes, is tackled by designing many complex rules for specific question types BIBREF4, BIBREF23  Heilman2010GoodQS improve rule-based question generation by introducing a statistical ranking model  First, they remove extraneous information in the sentence to transform it into a simpler one, which can be transformed easily into a succinct question with predefined sets of general rules  Then they adopt an overgenerate-and-rank approach to select the best candidate considering several features 
With the rise of dominant neural sequence-to-sequence learning models BIBREF5, Du2017LearningTA frame question generation as a sequence-to-sequence learning problem  Compared with rule-based approaches, neural models BIBREF24 can generate more fluent and grammatical questions  However, question generation is a one-to-many sequence generation problem, i e , several aspects can be asked given a sentence, which confuses the model during train and prevents concrete automatic evaluation  To tackle this issue, Zhou2017NeuralQG propose the answer-aware question generation setting which assumes the answer is already known and acts as a contiguous span inside the input sentence  They adopt a BIO tagging scheme to incorporate the answer position information as learned embedding features in Seq2Seq learning  Song2018LeveragingCI explicitly model the information between answer and sentence with a multi-perspective matching model  Kim2019ImprovingNQ also focus on the answer information and proposed an answer-separated Seq2Seq model by masking the answer with special tokens  All answer-aware neural models treat question generation as a one-to-one mapping problem, but existing models perform poorly for sentences with a complex structure (as shown in Table TABREF2) 
Our work is inspired by the process of extraneous information removing in BIBREF0, BIBREF25  Different from Heilman2010GoodQS which directly use the simplified sentence for generation and cao2018faithful which only consider aggregate two sources of information via gated attention in summarization, we propose to combine the structured answer-relevant relation and the original sentence  Factoid question generation from structured text is initially investigated by Serban2016GeneratingFQ, but our focus here is leveraging structured inputs to help question generation over unstructured sentences  Our proposed model can take advantage of unstructured sentences and structured answer-relevant relations to maintain informativeness and faithfulness of generated questions  The proposed model can also be generalized in other conditional sequence generation tasks which require multiple sources of inputs, e g , distractor generation for multiple choice questions BIBREF26 
Conclusions and Future Work
In this paper, we propose a question generation system which combines unstructured sentences and structured answer-relevant relations for generation  The unstructured sentences maintain the informativeness of generated questions while structured answer-relevant relations keep the faithfulness of questions  Extensive experiments demonstrate that our proposed model achieves state-of-the-art performance across several metrics  Furthermore, our model can generate diverse questions with different structured answer-relevant relations  For future work, there are some interesting dimensions to explore, such as difficulty levels BIBREF27, paragraph-level information BIBREF8 and conversational question generation BIBREF28 
Acknowledgments
This work is supported by the Research Grants Council of the Hong Kong Special Administrative Region, China (No  CUHK 14208815 and No  CUHK 14210717 of the General Research Fund)  We would like to thank the anonymous reviewers for their comments  We would also like to thank Department of Computer Science and Engineering, The Chinese University of Hong Kong for the conference grant support 
Context 3: 
Firat2016 proposed another approach which genuinely delivers attention-based NMT to multilingual translation  As in BIBREF9 , their approach utilizes one encoder per source language and one decoder per target language for many-to-many translation tasks  Instead of a quadratic number of independent attention layers, however, one single attention mechanism is integrated into their NMT, performing an affine transformation between the hidden layer of INLINEFORM0 source languages and that one of INLINEFORM1 target languages  It is required to change their architecture to accomodate such a complicated shared attention mechanism 
In a separate effort to achieve multilingual NMT, the work of Zoph2016 leverages available parallel data from other language pairs to help reducing possible ambiguities in the translation process into a single target language  They employed the multi-source attention-based NMT in a way that only one attention mechanism is required despite having multiple encoders  To achieve this, the outputs of the encoders were combined before feeding to the attention layer  They implemented two types of encoder combination; One is adding a non-linear layer on the concatenation of the encoders' hidden states  The other is using a variant of LSTM taking the respective gate values from the individual LSTM units of the encoders  As a result, the combined hidden states contain information from both encoders , thus encode the common semantic of the two source languages 
Universal Encoder and Decoder
Inspired by the multi-source NMT as additional parallel data in several languages are expected to benefit single translations, we aim to develop a NMT-based approach toward an universal framework to perform multilingual translation  Our solution features two treatments: 1) Coding the words in different languages as different words in the language-mixed vocabularies and 2) Forcing the NMT to translating a representation of source sentences into the sentences in a desired target language 
Language-specific Coding  When the encoder of a NMT system considers words across languages as different words, with a well-chosen architecture, it is expected to be able to learn a good representation of the source words in an embedding space in which words carrying similar meaning would have a closer distance to each others than those are semantically different  This should hold true when the words have the same or similar surface form, such as (@de@Obama; @en@Obama) or (@de@Projektion; @en@projection)  This should also hold true when the words have the same or similar meaning across languages, such as (@en@car; @en@automobile) or (@de@Flussufer; @en@bank)  Our encoder then acts similarly to the one of multi-source approach BIBREF10 , collecting additional information from other sources for better translations, but with a much simpler embedding function  Unlike them, we need only one encoder, so we could reduce the number of parameters to learn  Furthermore, we neither need to change the network architecture nor depend on which recurrent unit (GRU, LSTM or simple RNN) is currently using in the encoder 
We could apply the same trick to the target sentences and thus enable many-to-many translation capability of our NMT system  Similar to the multi-target translation BIBREF8 , we exploit further the correlation in semantics of those target sentences across different languages  The main difference between our approach and the work of BIBREF8 is that we need only one decoder for all target languages  Given one encoder for multiple source languages and one decoder for multiple target languages, it is trivial to incorporate the attention mechanism as in the case of a regular NMT for single language translation  In training, the attention layers were directed to learn relevant alignments between words in specific language pair and forward the produced context vector to the decoder  Now we rely totally on the network to learn good alignments between source and target sides  In fact, giving more information, our system are able to form nice alignments 
In comparison to other research that could perform complete multi-task learning, e g  the work from BIBREF9 or the approach proposed by BIBREF11 , our method is able to accommodate the attention layers seemlessly and easily  It also draws a clear distinction from those works in term of the complexity of the whole network: considerably less parameters to learn, thus reduces overfitting, with a conventional attention mechanism and a standard training procedure 
Target Forcing  While language-specific coding allows us to implement a multilingual attention-based NMT, there are two issues we have to consider before training the network  The first is that the number of rare words would increase in proportion with the number of languages involved  This might be solved by applying a rare word treatment method with appropriate awareness of the vocabularies' size  The second one is more problematic: Ambiguity level in the translation process definitely increases due to the additional introduction of words having the same or similar meaning across languages at both source and target sides  We deal with the problem by explicitly forcing the attention and translation to the direction that we prefer, expecting the information would limit the ambiguity to the scope of one language instead of all target languages  We realize this idea by adding at the beginning and at the end of every source sentences a special symbol indicating the language they would be translated into  For example, in a multilingual NMT, when a source sentence is German and the target language is English, the original sentence (already language-specific coded) is:
@de@darum @de@geht @de@es @de@in @de@meinem @de@Vortrag
Now when we force it to be translated into English, the target-forced sentence becomes:
<E> @de@darum @de@geht @de@es @de@in @de@meinem @de@Vortrag <E>
Due to the nature of recurrent units used in the encoder and decoder, in training, those starting symbols encourage the network learning the translation of following target words in a particular language pair  In testing time, information of the target language we provided help to limit the translated candidates, hence forming the translation in the desired language 
Figure FIGREF8 illustrates the essence of our approach  With two steps in the preprocessing phase, namely language-specific coding and target forcing, we are able to employ multilingual attention-based NMT without any special treatment in training such a standard architecture  Our encoder and attention-enable decoder can be seen as a shared encoder and decoder across languages, or an universal encoder and decoder  The flexibitily of our approach allow us to integrate any language into source or target side  As we will see in Section SECREF4 , it has proven to be extremely helpful not only in low-resourced scenarios but also in translation of well-resourced language pairs as it provides a novel way to make use of large monolingual corpora in NMT 
Evaluation
In this section, we describe the evaluation of our proposed approach in comparisons with the strong baselines using NMT in two scenarios: the translation of an under-resource language pair and the translation of a language pair that does not exist any paralled data at all 
Experimental Settings
Training Data  We choose WIT3's TED corpus BIBREF12 as the basis of our experiments since it might be the only high-quality parallel data of many low-resourced language pairs  TED is also multilingual in a sense that it includes numbers of talks which are commonly translated into many languages  In addition, we use a much larger corpus provided freely by WMT organizers when we evaluate the impact of our approach in a real machine translation campaign  It includes the paralled corpus extracted from the digital corpus of European Parliament (EPPS), the News Commentary (NC) and the web-crawled parallel data (CommonCrawl)  While the number of sentences in popular TED corpora varies from 13 thousands to 17 thousands, the total number of sentences in those larger corpus is approximately 3 million sentences 
Neural Machine Translation Setup  All experiments have been conducted using NMT framework Nematus, Following the work of Sennrich2016a, subword segmentation is handled in the prepocessing phase using Byte-Pair Encoding (BPE)  Excepts stated clearly in some experiments, we set the number of BPE merging operations at 39500 on the joint of source and target data  When training all NMT systems, we take out the sentence pairs exceeding 50-word length and shuffle them inside every minibatch  Our short-list vocabularies contain 40,000 most frequent words while the others are considered as rare words and applied the subword translation  We use an 1024-cell GRU layer and 1000-dimensional embeddings with dropout at every layer with the probability of 0 2 in the embedding and hidden layers and 0 1 in the input and ourput layers  We trained our systems using gradient descent optimization with Adadelta BIBREF13 on minibatches of size 80 and the gradient is rescaled whenever its norm exceed 1 0  All the trainings last approximately seven days if the early-stopping condition could not be reached  At a certain time, an external evaluation script on BLEU BIBREF14 is conducted on a development set to decide the early-stopping condition  This evaluation script has also being used to choose the model archiving the best BLEU on the development set instead of the maximal loglikelihood between the translations and target sentences while training  In translation, the framework produces INLINEFORM0 -best candidates and we then use a beam search with the beam size of 12 to get the best translation 
Under-resourced Translation
First, we consider the translation for an under-resourced pair of languages  Here a small portion of the large parallel corpus for English-German is used as a simulation for the scenario where we do not have much parallel data: Translating texts in English to German  We perform language-specific coding in both source and target sides  By accommodating the German monolingual data as an additional input (German INLINEFORM0 German), which we called the mix-source approach, we could enrich the training data in a simple, natural way  Given this under-resourced situation, it could help our NMT obtain a better representation of the source side, hence, able to learn the translation relationship better  Including monolingual data in this way might also improve the translation of some rare word types such as named entities  Furthermore, as the ultimate goal of our work, we would like to investigate the advantages of multilinguality in NMT

ground_truth: SQuAD

SQuAD

prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. Do they test their framework performance on commonly used language pairs, such as English-to-German?
Context 3:  We incorporate a similar portion of French-German parallel corpus into the English-German one  As discussed in Section SECREF5 , it is expected to help reducing the ambiguity in translation between one language pair since it utilizes the semantic context provided by the other source language  We name this mix-multi-source 
Table TABREF16 summarizes the performance of our systems measured in BLEU on two test sets, tst2013 and tst2014  Compared to the baseline NMT system which is solely trained on TED English-German data, our mix-source system achieves a considerable improvement of 2 6 BLEU points on tst2013 and 2 1 BLEU points on and tst2014   Adding French data to the source side and their corresponding German data to the target side in our mix-multi-source system also help to gain 2 2 and 1 6 BLEU points more on tst2013 tst2014, respectively  We observe a better improvement from our mix-source system compared to our mix-multi-source system  We speculate the reason that the mix-source encoder utilize the same information shared in two languages while the mix-multi-source receives and processes similar information in the other language but not necessarily the same  We might validate this hypothesis by comparing two systems trained on a common English-German-French corpus of TED  We put it in our future work's plan 
As we expected Figure FIGREF19 shows how different words in different languages can be close in the shared space after being learned to translate into a common language  We extract the word embeddings from the encoder of the mix-multi-source (En,Fr INLINEFORM0 De,De) after training, remove the language-specific codes (@en@ and @fr@)and project the word vectors to the 2D space using t-SNE BIBREF15  
Using large monolingual data in NMT 
A standard NMT system employs parallel data only  While good parallel corpora are limited in number, getting monolingual data of an arbitrary language is trivial  To make use of German monolingual corpus in an English INLINEFORM0 German NMT system, sennrich2016b built a separate German INLINEFORM1 English NMT using the same parallel corpus, then they used that system to translate the German monolingual corpus back to English, forming a synthesis parallel data  gulcehre2015 trained another RNN-based language model to score the monolingual corpus and integrate it to the NMT system through shallow or deep fusion  Both methods requires to train separate systems with possibly different hyperparameters for each  Conversely, by applying mix-source method to the big monolingual data, we need to train only one network  We mix the TED parallel corpus and the substantial monolingual corpus (EPPS+NC+CommonCrawl) and train a mix-source NMT system from those data 
The first result is not encouraging when its performance is even worse than the baseline NMT which is trained on the small parallel data only  Not using the same information in the source side, as we discussed in case of mix-multi-source strategy, could explain the degrading in performance of such a system  But we believe that the magnitude and unbalancing of the corpus are the main reasons  The data contains nearly four millions sentences but only around twenty thousands of them (0 5%) are the genuine parallel data  As a quick attempt, after we get the model with that big data, we continue training on the real parallel corpus for some more epochs  When this adaptation is applied, our system brings an improvement of +1 52 BLEU on tst2013 and +1 06 BLEU on tst2014 (Table TABREF21 ) 
Zero-resourced Translation
Among low-resourced scenarios, zero-resourced translation task stands in an extreme level  A zero-resourced translation task is one of the most difficult situation when there is no parallel data between the translating language pair  To the best of our knowledge, there have been yet existed a published work about using NMT for zero-resourced translation tasks up to now  In this section, we extend our strategies using the proposed multilingual NMT approach as first attempts to this extreme situation 
We employ language-specific coding and target forcing in a strategy called bridge  Unlike the strategies used in under-resourced translation task, bridge is an entire many-to-many multilingual NMT  Simulating a zero-resourced German INLINEFORM0 French translation task given the available German-English and English-French parallel corpora, after applying language-specific coding and target forcing for each corpus, we mix those data with an English-English data as a “bridge” creating some connection between German and French  We also propose a variant of this strategy that we incorporate French-French data  And we call it universal 
We evaluate bridge and universal systems on two German INLINEFORM0 French test sets  They are compared to a direct system, which is an NMT trained on German INLINEFORM1 French data, and to a pivot system, which essentially consists of two separate NMTs trained to translate from German to English and English to French  The direct system should not exist in a real zero-resourced situation  We refer it as the perfect system for comparison purpose only  In case of the pivot system, to generate a translated text in French from a German sentence, we first translate it to English, then the output sentence is fed to the English INLINEFORM2 German NMT system to obtain the French translation  Since there are more than two languages involved in those systems, we increase the number of BPE merging operations proportionally in order to reduce the number of rare words in such systems  We do not expect our proposed systems to perform well with this primitive way of building direct translating connections since this is essentially a difficult task  We report the performance of those systems in Table TABREF23  
Unsupprisingly, both bridge and universal systems perform worse than the pivot one  We consider two possible reasons:
Our target forcing mechanism is moderately primitive  Since the process is applied after language-specific coding, the target forcing symbol is the same for all source sentences in every languages  Thus, the forcing strength might not be enough to guide the decision of the next words  Once the very first word is translated into a word in wrong language, the following words tend to be translated into that wrong language again  Table TABREF24 shows some statistics of the translated words and sentences in wrong language 
Balancing of the training corpus  Although it is not severe as in the case of mix-source system for large monolingual data, the limited number of sentences in target language can affect the training  The difference of 1 07 BLEU points between bridge and universal might explain this assumption as we added more target data (French) in universal strategy, thus reducing the unbalance in training 
Those issues would be addressed in our following future work toward the multilingual attention-based NMT 
Conclusion and Future Work
In this paper, we present our first attempts in building a multilingual Neural Machine Translation framework  By treating words in different languages as different words and force the attention and translation to the direction of desired target language, we are able to employ attention-enable NMT toward a multilingual translation system  Our proposed approach alleviates the need of complicated architecture re-designing when accommodating attention mechanism  In addition, the number of free parameters to learn in our network does not go beyond that magnitute of a single NMT system  With its universality, our approach has shown its effectiveness in an under-resourced translation task with considerable improvements  In addition, the approach has achieved interesting and promising results when applied in the translation task that there is no direct parallel corpus between source and target languages 
Nevertheless, there are issues that we can continue working on to address in future work  A more balancing data would be helpful for this framework  The mechanism of forcing the NMT system to the right target language could be improved  We could conduct more detailed analyses of the various strategies under the framework to show its universarity 

ground_truth: Yes

Yes