prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. Do they test their framework performance on commonly used language pairs, such as English-to-German?
Context 1: Introduction
Neural Machine Translation (NMT) has shown its effectiveness in translation tasks when NMT systems perform best in recent machine translation campaigns BIBREF0 , BIBREF1   Compared to phrase-based Statistical Machine Translation (SMT) which is basically an ensemble of different features trained and tuned separately, NMT directly modeling the translation relationship between source and target sentences  Unlike SMT, NMT does not require much linguistic information and large monolingual data to achieve good performances 
An NMT consists of an encoder which recursively reads and represents the whole source sentence into a context vector and a recurrent decoder which takes the context vector and its previous state to predict the next target word  It is then trained in an end-to-end fashion to learn parameters which maximizes the likelihood between the outputs and the references  Recently, attention-based NMT has been featured in most state-of-the-art systems  First introduced by BIBREF2 , attention mechanism is integrated in decoder side as feedforward layers  It allows the NMT to decide which source words should take part in the predicting process of the next target words  It helps to improve NMTs significantly  Nevertheless, since the attention mechanism is specific to a particular source sentence and the considering target word, it is also specific to particular language pairs 
Some recent work has focused on extending the NMT framework to multilingual scenarios  By training such network using parallel corpora in number of different languages, NMT could benefit from additional information embedded in a common semantic space across languages  Basically, the proposed NMT are required to employ multiple encoders or multiple decoders to deal with multilinguality  Furthermore, in order to avoid the tight dependency of the attention mechanism to specific language pairs, they also need to modify their architecture to combine either the encoders or the attention layers  These modifications are specific to the purpose of the tasks as well  Thus, those multilingual NMTs are more complicated, much more free parameters to learn and more difficult to perform standard trainings compared to the original NMT 
In this paper, we introduce a unified approach to seamlessly extend the original NMT to multilingual settings  Our approach allows us to integrate any language in any side of the encoder-decoder architecture with only one encoder and one decoder for all the languages involved  Moreover, it is not necessary to do any network modification to enable attention mechanism in our NMT systems  We then apply our proprosed framework in two demanding scenarios: under-resourced translation and zero-resourced translation  The results show that bringing multilinguality to NMT helps to improve individual translations  With some insightful analyses of the results, we set our goal toward a fully multilingual NMT framework 
The paper starts with a detailed introduction to attention-based NMT  In Section SECREF3 , related work about multi-task NMT is reviewed  Section SECREF5 describes our proposed approach and thorough comparisons to the related work  It is followed by a section of evaluating our systems in two aforementioned scenarios, in which different strategies have been employed under a unified approach (Section SECREF4 )  Finally, the paper ends with conclusion and future work 
This work is licenced under a Creative Commons Attribution 4 0 International License  License details: http://creativecommons org/licenses/by/4 0/
Neural Machine Translation: Background
An NMT system consists of an encoder which automatically learns the characteristics of a source sentence into fix-length context vectors and a decoder that recursively combines the produced context vectors with the previous target word to generate the most probable word from a target vocabulary 
More specifically, a bidirectional recurrent encoder reads every words INLINEFORM0 of a source sentence INLINEFORM1 and encodes a representation INLINEFORM2 of the sentence into a fixed-length vector INLINEFORM3 concatinated from those of the forward and backward directions: INLINEFORM4
Here INLINEFORM0 is the one-hot vector of the word INLINEFORM1 and INLINEFORM2 is the word embedding matrix which is shared across the source words  INLINEFORM3 is the recurrent unit computing the current hidden state of the encoder based on the previous hidden state  INLINEFORM4 is then called an annotation vector, which encodes the source sentence up to the time INLINEFORM5 from both forward and backward directions  Recurrent units in NMT can be a simple recurrent neural network unit (RNN), a Long Short-Term Memory unit (LSTM) BIBREF3 or a Gated Recurrent Unit (GRU) BIBREF4
Similar to the encoder, the recurrent decoder generates one target word INLINEFORM0 to form a translated target sentence INLINEFORM1 in the end  At the time INLINEFORM2 , it takes the previous hidden state of the decoder INLINEFORM3 , the previous embedded word representation INLINEFORM4 and a time-specific context vector INLINEFORM5 as inputs to calculate the current hidden state INLINEFORM6 : INLINEFORM7
Again, INLINEFORM0 is the recurrent activation function of the decoder and INLINEFORM1 is the shared word embedding matrix of the target sentences  The context vector INLINEFORM2 is calculated based on the annotation vectors from the encoder  Before feeding the annotation vectors into the decoder, an attention mechanism is set up in between, in order to choose which annotation vectors should contribute to the predicting decision of the next target word  Intuitively, a relevance between the previous target word and the annotation vectors can be used to form some attention scenario  There exists several ways to calculate the relevance as shown in BIBREF5 , but what we describe here follows the proposed method of BIBREF2 DISPLAYFORM0
In BIBREF2 , this attention mechanism, originally called alignment model, has been employed as a simple feedforward network with the first layer is a learnable layer via INLINEFORM0 , INLINEFORM1 and INLINEFORM2   The relevance scores INLINEFORM3 are then normalized into attention weights INLINEFORM4 and the context vector INLINEFORM5 is calculated as the weighted sum of all annotation vectors INLINEFORM6   Depending on how much attention the target word at time INLINEFORM7 put on the source states INLINEFORM8 , a soft alignment is learned  By being employed this way, word alignment is not a latent variable but a parametrized function, making the alignment model differentiable  Thus, it could be trained together with the whole architecture using backpropagation 
One of the most severe problems of NMT is handling of the rare words, which are not in the short lists of the vocabularies, i e  out-of-vocabulary (OOV) words, or do not appear in the training set at all  In BIBREF6 , the rare target words are copied from their aligned source words after the translation  This heuristic works well with OOV words and named entities but unable to translate unseen words  In BIBREF7 , their proposed NMT models have been shown to not only be effective on reducing vocabulary sizes but also have the ability to generate unseen words  This is achieved by segmenting the rare words into subword units and translating them  The state-of-the-art translation systems essentially employ subword NMT BIBREF7  
Universal Encoder and Decoder for Multilingual Neural Machine Translation
While the majority of previous research has focused on improving the performance of NMT on individual language pairs with individual NMT systems, recent work has started investigating potential ways to conduct the translation involved in multiple languages using a single NMT system  The possible reason explaining these efforts lies on the unique architecture of NMT  Unlike SMT, NMT consists of separated neural networks for the source and target sides, or the encoder and decoder, respectively  This allows these components to map a sentence in any language to a representation in an embedding space which is believed to share common semantics among the source languages involved  From that shared space, the decoder, with some implicit or explicit relevant constraints, could transform the representation into a concrete sentence in any desired language  In this section, we review some related work on this matter  We then describe a unified approach toward an universal attention-based NMT scheme  Our approach does not require any architecture modification and it can be trained to learn a minimal number of parameters compared to the other work 
Related Work
By extending the solution of sequence-to-sequence modeling using encoder-decoder architectures to multi-task learning, Luong2016 managed to achieve better performance on some INLINEFORM0 tasks such as translation, parsing and image captioning compared to individual tasks  Specifically in translation, the work utilizes multiple encoders to translate from multiple languages, and multiple decoders to translate to multiple languages  In this view of multilingual translation, each language in source or target side is modeled by one encoder or decoder, depending on the side of the translation  Due to the natural diversity between two tasks in that multi-task learning scenario, e g  translation and parsing, it could not feature the attention mechanism although it has proven its effectiveness in NMT  There exists two directions which proposed for multilingual translation scenarios where they leverage the attention mechanism  The first one is indicated in the work from BIBREF8 , where it introduce an one-to-many multilingual NMT system to translates from one source language into multiple target languages  Having one source language, the attention mechanism is then handed over to the corresponding decoder  The objective function is changed to adapt to multilingual settings  In testing time, the parameters specific to a desired language pair are used to perform the translation 
Firat2016 proposed another approach which genuinely delivers attention-based NMT to multilingual translation  As in BIBREF9 , their approach utilizes one encoder per source language and one decoder per target language for many-to-many translation tasks  Instead of a quadratic number of independent attention layers, however, one single attention mechanism is integrated into their NMT, performing an affine transformation between the hidden layer of INLINEFORM0 source languages and that one of INLINEFORM1 target languages  It is required to change their architecture to accomodate such a complicated shared attention mechanism 
In a separate effort to achieve multilingual NMT, the work of Zoph2016 leverages available parallel data from other language pairs to help reducing possible ambiguities in the translation process into a single target language  They employed the multi-source attention-based NMT in a way that only one attention mechanism is required despite having multiple encoders  To achieve this, the outputs of the encoders were combined before feeding to the attention layer  They implemented two types of encoder combination; One is adding a non-linear layer on the concatenation of the encoders' hidden states  The other is using a variant of LSTM taking the respective gate values from the individual LSTM units of the encoders  As a result, the combined hidden states contain information from both encoders , thus encode the common semantic of the two source languages 
Universal Encoder and Decoder
Inspired by the multi-source NMT as additional parallel data in several languages are expected to benefit single translations, we aim to develop a NMT-based approach toward an universal framework to perform multilingual translation  Our solution features two treatments: 1) Coding the words in different languages as different words in the language-mixed vocabularies and 2) Forcing the NMT to translating a representation of source sentences into the sentences in a desired target language 
Language-specific Coding  When the encoder of a NMT system considers words across languages as different words, with a well-chosen architecture, it is expected to be able to learn a good representation of the source words in an embedding space in which words carrying similar meaning would have a closer distance to each others than those are semantically different  This should hold true when the words have the same or similar surface form, such as (@de@Obama; @en@Obama) or (@de@Projektion; @en@projection)  This should also hold true when the words have the same or similar meaning across languages, such as (@en@car; @en@automobile) or (@de@Flussufer; @en@bank)  Our encoder then acts similarly to the one of multi-source approach BIBREF10 , collecting additional information from other sources for better translations, but with a much simpler embedding function  Unlike them, we need only one encoder, so we could reduce the number of parameters to learn  Furthermore, we neither need to change the network architecture nor depend on which recurrent unit (GRU, LSTM or simple RNN) is currently using in the encoder 
We could apply the same trick to the target sentences and thus enable many-to-many translation capability of our NMT system  Similar to the multi-target translation BIBREF8 , we exploit further the correlation in semantics of those target sentences across different languages  The main difference between our approach and the work of BIBREF8 is that we need only one decoder for all target languages  Given one encoder for multiple source languages and one decoder for multiple target languages, it is trivial to incorporate the attention mechanism as in the case of a regular NMT for single language translation  In training, the attention layers were directed to learn relevant alignments between words in specific language pair and forward the produced context vector to the decoder  Now we rely totally on the network to learn good alignments between source and target sides  In fact, giving more information, our system are able to form nice alignments 
In comparison to other research that could perform complete multi-task learning, e g  the work from BIBREF9 or the approach proposed by BIBREF11 , our method is able to accommodate the attention layers seemlessly and easily  It also draws a clear distinction from those works in term of the complexity of the whole network: considerably less parameters to learn, thus reduces overfitting, with a conventional attention mechanism and a standard training procedure 
Target Forcing  While language-specific coding allows us to implement a multilingual attention-based NMT, there are two issues we have to consider before training the network  The first is that the number of rare words would increase in proportion with the number of languages involved  This might be solved by applying a rare word treatment method with appropriate awareness of the vocabularies' size  The second one is more problematic: Ambiguity level in the translation process definitely increases due to the additional introduction of words having the same or similar meaning across languages at both source and target sides  We deal with the problem by explicitly forcing the attention and translation to the direction that we prefer, expecting the information would limit the ambiguity to the scope of one language instead of all target languages  We realize this idea by adding at the beginning and at the end of every source sentences a special symbol indicating the language they would be translated into  For example, in a multilingual NMT, when a source sentence is German and the target language is English, the original sentence (already language-specific coded) is:
@de@darum @de@geht @de@es @de@in @de@meinem @de@Vortrag
Now when we force it to be translated into English, the target-forced sentence becomes:
<E> @de@darum @de@geht @de@es @de@in @de@meinem @de@Vortrag <E>
Due to the nature of recurrent units used in the encoder and decoder, in training, those starting symbols encourage the network learning the translation of following target words in a particular language pair  In testing time, information of the target language we provided help to limit the translated candidates, hence forming the translation in the desired language 
Figure FIGREF8 illustrates the essence of our approach  With two steps in the preprocessing phase, namely language-specific coding and target forcing, we are able to employ multilingual attention-based NMT without any special treatment in training such a standard architecture  Our encoder and attention-enable decoder can be seen as a shared encoder and decoder across languages, or an universal encoder and decoder  The flexibitily of our approach allow us to integrate any language into source or target side  As we will see in Section SECREF4 , it has proven to be extremely helpful not only in low-resourced scenarios but also in translation of well-resourced language pairs as it provides a novel way to make use of large monolingual corpora in NMT 
Evaluation
In this section, we describe the evaluation of our proposed approach in comparisons with the strong baselines using NMT in two scenarios: the translation of an under-resource language pair and the translation of a language pair that does not exist any paralled data at all 
Experimental Settings
Training Data  We choose WIT3's TED corpus BIBREF12 as the basis of our experiments since it might be the only high-quality parallel data of many low-resourced language pairs  TED is also multilingual in a sense that it includes numbers of talks which are commonly translated into many languages  In addition, we use a much larger corpus provided freely by WMT organizers when we evaluate the impact of our approach in a real machine translation campaign  It includes the paralled corpus extracted from the digital corpus of European Parliament (EPPS), the News Commentary (NC) and the web-crawled parallel data (CommonCrawl)  While the number of sentences in popular TED corpora varies from 13 thousands to 17 thousands, the total number of sentences in those larger corpus is approximately 3 million sentences 
Neural Machine Translation Setup  All experiments have been conducted using NMT framework Nematus, Following the work of Sennrich2016a, subword segmentation is handled in the prepocessing phase using Byte-Pair Encoding (BPE)  Excepts stated clearly in some experiments, we set the number of BPE merging operations at 39500 on the joint of source and target data  When training all NMT systems, we take out the sentence pairs exceeding 50-word length and shuffle them inside every minibatch  Our short-list vocabularies contain 40,000 most frequent words while the others are considered as rare words and applied the subword translation  We use an 1024-cell GRU layer and 1000-dimensional embeddings with dropout at every layer with the probability of 0 2 in the embedding and hidden layers and 0 1 in the input and ourput layers  We trained our systems using gradient descent optimization with Adadelta BIBREF13 on minibatches of size 80 and the gradient is rescaled whenever its norm exceed 1 0  All the trainings last approximately seven days if the early-stopping condition could not be reached  At a certain time, an external evaluation script on BLEU BIBREF14 is conducted on a development set to decide the early-stopping condition  This evaluation script has also being used to choose the model archiving the best BLEU on the development set instead of the maximal loglikelihood between the translations and target sentences while training  In translation, the framework produces INLINEFORM0 -best candidates and we then use a beam search with the beam size of 12 to get the best translation 
Under-resourced Translation
First, we consider the translation for an under-resourced pair of languages  Here a small portion of the large parallel corpus for English-German is used as a simulation for the scenario where we do not have much parallel data: Translating texts in English to German  We perform language-specific coding in both source and target sides  By accommodating the German monolingual data as an additional input (German INLINEFORM0 German), which we called the mix-source approach, we could enrich the training data in a simple, natural way  Given this under-resourced situation, it could help our NMT obtain a better representation of the source side, hence, able to learn the translation relationship better  Including monolingual data in this way might also improve the translation of some rare word types such as named entities  Furthermore, as the ultimate goal of our work, we would like to investigate the advantages of multilinguality in NMT  We incorporate a similar portion of French-German parallel corpus into the English-German one  As discussed in Section SECREF5 , it is expected to help reducing the ambiguity in translation between one language pair since it utilizes the semantic context provided by the other source language  We name this mix-multi-source 
Table TABREF16 summarizes the performance of our systems measured in BLEU on two test sets, tst2013 and tst2014  Compared to the baseline NMT system which is solely trained on TED English-German data, our mix-source system achieves a considerable improvement of 2 6 BLEU points on tst2013 and 2 1 BLEU points on and tst2014   Adding French data to the source side and their corresponding German data to the target side in our mix-multi-source system also help to gain 2 2 and 1 6 BLEU points more on tst2013 tst2014, respectively  We observe a better improvement from our mix-source system compared to our mix-multi-source system  We speculate the reason that the mix-source encoder utilize the same information shared in two languages while the mix-multi-source receives and processes similar information in the other language but not necessarily the same  We might validate this hypothesis by comparing two systems trained on a common English-German-French corpus of TED  We put it in our future work's plan 
As we expected Figure FIGREF19 shows how different words in different languages can be close in the shared space after being learned to translate into a common language  We extract the word embeddings from the encoder of the mix-multi-source (En,Fr INLINEFORM0 De,De) after training, remove the language-specific codes (@en@ and @fr@)and project the word vectors to the 2D space using t-SNE BIBREF15  
Using large monolingual data in NMT 
A standard NMT system employs parallel data only  While good parallel corpora are limited in number, getting monolingual data of an arbitrary language is trivial  To make use of German monolingual corpus in an English INLINEFORM0 German NMT system, sennrich2016b built a separate German INLINEFORM1 English NMT using the same parallel corpus, then they used that system to translate the German monolingual corpus back to English, forming a synthesis parallel data  gulcehre2015 trained another RNN-based language model to score the monolingual corpus and integrate it to the NMT system through shallow or deep fusion  Both methods requires to train separate systems with possibly different hyperparameters for each  Conversely, by applying mix-source method to the big monolingual data, we need to train only one network  We mix the TED parallel corpus and the substantial monolingual corpus (EPPS+NC+CommonCrawl) and train a mix-source NMT system from those data 
The first result is not encouraging when its performance is even worse than the baseline NMT which is trained on the small parallel data only  Not using the same information in the source side, as we discussed in case of mix-multi-source strategy, could explain the degrading in performance of such a system  But we believe that the magnitude and unbalancing of the corpus are the main reasons  The data contains nearly four millions sentences but only around twenty thousands of them (0 5%) are the genuine parallel data  As a quick attempt, after we get the model with that big data, we continue training on the real parallel corpus for some more epochs  When this adaptation is applied, our system brings an improvement of +1 52 BLEU on tst2013 and +1 06 BLEU on tst2014 (Table TABREF21 ) 
Zero-resourced Translation
Among low-resourced scenarios, zero-resourced translation task stands in an extreme level  A zero-resourced translation task is one of the most difficult situation when there is no parallel data between the translating language pair  To the best of our knowledge, there have been yet existed a published work about using NMT for zero-resourced translation tasks up to now  In this section, we extend our strategies using the proposed multilingual NMT approach as first attempts to this extreme situation 
We employ language-specific coding and target forcing in a strategy called bridge  Unlike the strategies used in under-resourced translation task, bridge is an entire many-to-many multilingual NMT  Simulating a zero-resourced German INLINEFORM0 French translation task given the available German-English and English-French parallel corpora, after applying language-specific coding and target forcing for each corpus, we mix those data with an English-English data as a “bridge” creating some connection between German and French  We also propose a variant of this strategy that we incorporate French-French data  And we call it universal 
We evaluate bridge and universal systems on two German INLINEFORM0 French test sets  They are compared to a direct system, which is an NMT trained on German INLINEFORM1 French data, and to a pivot system, which essentially consists of two separate NMTs trained to translate from German to English and English to French  The direct system should not exist in a real zero-resourced situation  We refer it as the perfect system for comparison purpose only  In case of the pivot system, to generate a translated text in French from a German sentence, we first translate it to English, then the output sentence is fed to the English INLINEFORM2 German NMT system to obtain the French translation  Since there are more than two languages involved in those systems, we increase the number of BPE merging operations proportionally in order to reduce the number of rare words in such systems  We do not expect our proposed systems to perform well with this primitive way of building direct translating connections since this is essentially a difficult task  We report the performance of those systems in Table TABREF23  
Unsupprisingly, both bridge and universal systems perform worse than the pivot one  We consider two possible reasons:
Our target forcing mechanism is moderately primitive  Since the process is applied after language-specific coding, the target forcing symbol is the same for all source sentences in every languages  Thus, the forcing strength might not be enough to guide the decision of the next words  Once the very first word is translated into a word in wrong language, the following words tend to be translated into that wrong language again  Table TABREF24 shows some statistics of the translated words and sentences in wrong language 
Balancing of the training corpus  Although it is not severe as in the case of mix-source system for large monolingual data, the limited number of sentences in target language can affect the training  The difference of 1 07 BLEU points between bridge and universal might explain this assumption as we added more target data (French) in universal strategy, thus reducing the unbalance in training 
Those issues would be addressed in our following future work toward the multilingual attention-based NMT 
Conclusion and Future Work
In this paper, we present our first attempts in building a multilingual Neural Machine Translation framework  By treating words in different languages as different words and force the attention and translation to the direction of desired target language, we are able to employ attention-enable NMT toward a multilingual translation system  Our proposed approach alleviates the need of complicated architecture re-designing when accommodating attention mechanism  In addition, the number of free parameters to learn in our network does not go beyond that magnitute of a single NMT system  With its universality, our approach has shown its effectiveness in an under-resourced translation task with considerable improvements  In addition, the approach has achieved interesting and promising results when applied in the translation task that there is no direct parallel corpus between source and target languages 
Nevertheless, there are issues that we can continue working on to address in future work  A more balancing data would be helpful for this framework  The mechanism of forcing the NMT system to the right target language could be improved  We could conduct more detailed analyses of the various strategies under the framework to show its universarity 
Context 2: Introduction
This paper describes our approach and results for Task 2 of the CoNLL–SIGMORPHON 2018 shared task on universal morphological reinflection BIBREF0   The task is to generate an inflected word form given its lemma and the context in which it occurs 
Morphological (re)inflection from context is of particular relevance to the field of computational linguistics: it is compelling to estimate how well a machine-learned system can capture the morphosyntactic properties of a word given its context, and map those properties to the correct surface form for a given lemma 
There are two tracks of Task 2 of CoNLL–SIGMORPHON 2018: in Track 1 the context is given in terms of word forms, lemmas and morphosyntactic descriptions (MSD); in Track 2 only word forms are available  See Table TABREF1 for an example  Task 2 is additionally split in three settings based on data size: high, medium and low, with high-resource datasets consisting of up to 70K instances per language, and low-resource datasets consisting of only about 1K instances 
The baseline provided by the shared task organisers is a seq2seq model with attention (similar to the winning system for reinflection in CoNLL–SIGMORPHON 2016, BIBREF1 ), which receives information about context through an embedding of the two words immediately adjacent to the target form  We use this baseline implementation as a starting point and achieve the best overall accuracy of 49 87 on Task 2 by introducing three augmentations to the provided baseline system: (1) We use an LSTM to encode the entire available context; (2) We employ a multi-task learning approach with the auxiliary objective of MSD prediction; and (3) We train the auxiliary component in a multilingual fashion, over sets of two to three languages 
In analysing the performance of our system, we found that encoding the full context improves performance considerably for all languages: 11 15 percentage points on average, although it also highly increases the variance in results  Multi-task learning, paired with multilingual training and subsequent monolingual finetuning, scored highest for five out of seven languages, improving accuracy by another 9 86% on average 
System Description
Our system is a modification of the provided CoNLL–SIGMORPHON 2018 baseline system, so we begin this section with a reiteration of the baseline system architecture, followed by a description of the three augmentations we introduce 
Baseline
The CoNLL–SIGMORPHON 2018 baseline is described as follows:
The system is an encoder-decoder on character sequences  It takes a lemma as input and generates a word form  The process is conditioned on the context of the lemma [   ] The baseline treats the lemma, word form and MSD of the previous and following word as context in track 1  In track 2, the baseline only considers the word forms of the previous and next word  [   ] The baseline system concatenates embeddings for context word forms, lemmas and MSDs into a context vector  The baseline then computes character embeddings for each character in the input lemma  Each of these is concatenated with a copy of the context vector  The resulting sequence of vectors is encoded using an LSTM encoder  Subsequently, an LSTM decoder generates the characters in the output word form using encoder states and an attention mechanism 
To that we add a few details regarding model size and training schedule:
the number of LSTM layers is one;
embedding size, LSTM layer size and attention layer size is 100;
models are trained for 20 epochs;
on every epoch, training data is subsampled at a rate of 0 3;
LSTM dropout is applied at a rate 0 3;
context word forms are randomly dropped at a rate of 0 1;
the Adam optimiser is used, with a default learning rate of 0 001; and
trained models are evaluated on the development data (the data for the shared task comes already split in train and dev sets) 
Our system
Here we compare and contrast our system to the baseline system  A diagram of our system is shown in Figure FIGREF4  
The idea behind this modification is to provide the encoder with access to all morpho-syntactic cues present in the sentence  In contrast to the baseline, which only encodes the immediately adjacent context of a target word, we encode the entire context  All context word forms, lemmas, and MSD tags (in Track 1) are embedded in their respective high-dimensional spaces as before, and their embeddings are concatenated  However, we now reduce the entire past context to a fixed-size vector by encoding it with a forward LSTM, and we similarly represent the future context by encoding it with a backwards LSTM 
We introduce an auxiliary objective that is meant to increase the morpho-syntactic awareness of the encoder and to regularise the learning process—the task is to predict the MSD tag of the target form  MSD tag predictions are conditioned on the context encoding, as described in UID15   Tags are generated with an LSTM one component at a time, e g  the tag PRO;NOM;SG;1 is predicted as a sequence of four components, INLINEFORM0 PRO, NOM, SG, 1 INLINEFORM1  
For every training instance, we backpropagate the sum of the main loss and the auxiliary loss without any weighting 
As MSD tags are only available in Track 1, this augmentation only applies to this track 
The parameters of the entire MSD (auxiliary-task) decoder are shared across languages 
Since a grouping of the languages based on language family would have left several languages in single-member groups (e g  Russian is the sole representative of the Slavic family), we experiment with random groupings of two to three languages  Multilingual training is performed by randomly alternating between languages for every new minibatch  We do not pass any information to the auxiliary decoder as to the source language of the signal it is receiving, as we assume abstract morpho-syntactic features are shared across languages 
After 20 epochs of multilingual training, we perform 5 epochs of monolingual finetuning for each language  For this phase, we reduce the learning rate to a tenth of the original learning rate, i e  0 0001, to ensure that the models are indeed being finetuned rather than retrained 
We keep all hyperparameters the same as in the baseline
Context 3: Introduction
The task of generating natural language descriptions of structured data (such as tables) BIBREF2 , BIBREF3 , BIBREF4 has seen a growth in interest with the rise of sequence to sequence models that provide an easy way of encoding tables and generating text from them BIBREF0 , BIBREF1 , BIBREF5 , BIBREF6  
For text generation tasks, the only gold standard metric is to show the output to humans for judging its quality, but this is too expensive to apply repeatedly anytime small modifications are made to a system  Hence, automatic metrics that compare the generated text to one or more reference texts are routinely used to compare models BIBREF7   For table-to-text generation, automatic evaluation has largely relied on BLEU BIBREF8 and ROUGE BIBREF9   The underlying assumption behind these metrics is that the reference text is gold-standard, i e , it is the ideal target text that a system should generate  In practice, however, when datasets are collected automatically and heuristically, the reference texts are often not ideal  Figure FIGREF2 shows an example from the WikiBio dataset BIBREF0   Here the reference contains extra information which no system can be expected to produce given only the associated table  We call such reference texts divergent from the table 
We show that existing automatic metrics, including BLEU, correlate poorly with human judgments when the evaluation sets contain divergent references (§ SECREF36 )  For many table-to-text generation tasks, the tables themselves are in a pseudo-natural language format (e g , WikiBio, WebNLG BIBREF6 , and E2E-NLG BIBREF10 )  In such cases we propose to compare the generated text to the underlying table as well to improve evaluation  We develop a new metric, PARENT (Precision And Recall of Entailed N-grams from the Table) (§ SECREF3 )  When computing precision, PARENT effectively uses a union of the reference and the table, to reward correct information missing from the reference  When computing recall, it uses an intersection of the reference and the table, to ignore extra incorrect information in the reference  The union and intersection are computed with the help of an entailment model to decide if a text n-gram is entailed by the table  We show that this method is more effective than using the table as an additional reference  Our main contributions are:
Table-to-Text Generation
We briefly review the task of generating natural language descriptions of semi-structured data, which we refer to as tables henceforth BIBREF11 , BIBREF12   Tables can be expressed as set of records INLINEFORM0 , where each record is a tuple (entity, attribute, value)  When all the records are about the same entity, we can truncate the records to (attribute, value) pairs  For example, for the table in Figure FIGREF2 , the records are {(Birth Name, Michael Dahlquist), (Born, December 22 1965),    }  The task is to generate a text INLINEFORM1 which summarizes the records in a fluent and grammatical manner  For training and evaluation we further assume that we have a reference description INLINEFORM2 available for each table  We let INLINEFORM3 denote an evaluation set of tables, references and texts generated from a model INLINEFORM4 , and INLINEFORM5 , INLINEFORM6 denote the collection of n-grams of order INLINEFORM7 in INLINEFORM8 and INLINEFORM9 , respectively  We use INLINEFORM10 to denote the count of n-gram INLINEFORM11 in INLINEFORM12 , and INLINEFORM13 to denote the minimum of its counts in INLINEFORM14 and INLINEFORM15   Our goal is to assign a score to the model, which correlates highly with human judgments of the quality of that model 
PARENT
PARENT evaluates each instance INLINEFORM0 separately, by computing the precision and recall of INLINEFORM1 against both INLINEFORM2 and INLINEFORM3  
Evaluation via Information Extraction
BIBREF1 proposed to use an auxiliary model, trained to extract structured records from text, for evaluation  However, the extraction model presented in that work is limited to the closed-domain setting of basketball game tables and summaries  In particular, they assume that each table has exactly the same set of attributes for each entity, and that the entities can be identified in the text via string matching  These assumptions are not valid for the open-domain WikiBio dataset, and hence we train our own extraction model to replicate their evaluation scheme 
Our extraction system is a pointer-generator network BIBREF19 , which learns to produce a linearized version of the table from the text  The network learns which attributes need to be populated in the output table, along with their values  It is trained on the training set of WikiBio  At test time we parsed the output strings into a set of (attribute, value) tuples and compare it to the ground truth table  The F-score of this text-to-table system was INLINEFORM0 , which is comparable to other challenging open-domain settings BIBREF20   More details are included in the Appendix SECREF52  
Given this information extraction system, we consider the following metrics for evaluation, along the lines of BIBREF1   Content Selection (CS): F-score for the (attribute, value) pairs extracted from the generated text compared to those extracted from the reference  Relation Generation (RG): Precision for the (attribute, value) pairs extracted from the generated text compared to those in the ground truth table  RG-F: Since our task emphasizes the recall of information from the table as well, we consider another variant which computes the F-score of the extracted pairs to those in the table  We omit the content ordering metric, since our extraction system does not align records to the input text 
Experiments & Results
In this section we compare several automatic evaluation metrics by checking their correlation with the scores assigned by humans to table-to-text models  Specifically, given INLINEFORM0 models INLINEFORM1 , and their outputs on an evaluation set, we show these generated texts to humans to judge their quality, and obtain aggregated human evaluation scores for all the models, INLINEFORM2 (§ SECREF33 )  Next, to evaluate an automatic metric, we compute the scores it assigns to each model, INLINEFORM3 , and check the Pearson correlation between INLINEFORM4 and INLINEFORM5 BIBREF21  
Data & Models
Our main experiments are on the WikiBio dataset BIBREF0 , which is automatically constructed and contains many divergent references  In § SECREF47 we also present results on the data released as part of the WebNLG challenge 
We developed several models of varying quality for generating text from the tables in WikiBio  This gives us a diverse set of outputs to evaluate the automatic metrics on  Table TABREF32 lists the models along with their hyperparameter settings and their scores from the human evaluation (§ SECREF33 )  Our focus is primarily on neural sequence-to-sequence methods since these are most widely used, but we also include a template-based baseline  All neural models were trained on the WikiBio training set  Training details and sample outputs are included in Appendices SECREF56 & SECREF57  
We divide these models into two categories and measure correlation separately for both the categories  The first category, WikiBio-Systems, includes one model each from the four families listed in Table TABREF32   This category tests whether a metric can be used to compare different model families with a large variation in the quality of their outputs  The second category, WikiBio-Hyperparams, includes 13 different hyperparameter settings of PG-Net BIBREF19 , which was the best performing system overall  9 of these were obtained by varying the beam size and length normalization penalty of the decoder network BIBREF23 , and the remaining 4 were obtained by re-scoring beams of size 8 with the information extraction model described in § SECREF4   All the models in this category produce high quality fluent texts, and differ primarily on the quantity and accuracy of the information they express  Here we are testing whether a metric can be used to compare similar systems with a small variation in performance  This is an important use-case as metrics are often used to tune hyperparameters of a model 
Human Evaluation
We collected human judgments on the quality of the 16 models trained for WikiBio, plus the reference texts  Workers on a crowd-sourcing platform, proficient in English, were shown a table with pairs of generated texts, or a generated text and the reference, and asked to select the one they prefer  Figure FIGREF34 shows the instructions they were given  Paired comparisons have been shown to be superior to rating scales for comparing generated texts BIBREF24   However, for measuring correlation the comparisons need to be aggregated into real-valued scores, INLINEFORM0 , for each of the INLINEFORM1 models  For this, we use Thurstone's method BIBREF22 , which assigns a score to each model based on how many times it was preferred over an alternative 
The data collection was performed separately for models in the WikiBio-Systems and WikiBio-Hyperparams categories  1100 tables were sampled from the development set, and for each table we got 8 different sentence pairs annotated across the two categories, resulting in a total of 8800 pairwise comparisons  Each pair was judged by one worker only which means there may be noise at the instance-level, but the aggregated system-level scores had low variance (cf  Table TABREF32 )  In total around 500 different workers were involved in the annotation  References were also included in the evaluation, and they received a lower score than PG-Net, highlighting the divergence in WikiBio 
Compared Metrics
Text only: We compare BLEU BIBREF8 , ROUGE BIBREF9 , METEOR BIBREF18 , CIDEr and CIDEr-D BIBREF25 using their publicly available implementations 
Information Extraction based: We compare the CS, RG and RG-F metrics discussed in § SECREF4  
Text & Table: We compare a variant of BLEU, denoted as BLEU-T, where the values from the table are used as additional references  BLEU-T draws inspiration from iBLEU BIBREF26 but instead rewards n-grams which match the table rather than penalizing them  For PARENT, we compare both the word-overlap model (PARENT-W) and the co-occurrence model (PARENT-C) for determining entailment  We also compare versions where a single INLINEFORM0 is tuned on the entire dataset to maximize correlation with human judgments, denoted as PARENT*-W/C 
Correlation Comparison
We use bootstrap sampling (500 iterations) over the 1100 tables for which we collected human annotations to get an idea of how the correlation of each metric varies with the underlying data  In each iteration, we sample with replacement, tables along with their references and all the generated texts for that table  Then we compute aggregated human evaluation and metric scores for each of the models and compute the correlation between the two  We report the average correlation across all bootstrap samples for each metric in Table TABREF37   The distribution of correlations for the best performing metrics are shown in Figure FIGREF38  
Table TABREF37 also indicates whether PARENT is significantly better than a baseline metric  BIBREF21 suggest using the William's test for this purpose, but since we are computing correlations between only 4/13 systems at a time, this test has very weak power in our case  Hence, we use the bootstrap samples to obtain a INLINEFORM0 confidence interval of the difference in correlation between PARENT and any other metric and check whether this is above 0 BIBREF27  
Correlations are higher for the systems category than the hyperparams category  The latter is a more difficult setting since very similar models are compared, and hence the variance of the correlations is also high  Commonly used metrics which only rely on the reference (BLEU, ROUGE, METEOR, CIDEr) have only weak correlations with human judgments  In the hyperparams category, these are often negative, implying that tuning models based on these may lead to selecting worse models  BLEU performs the best among these, and adding n-grams from the table as references improves this further (BLEU-T) 
Among the extractive evaluation metrics, CS, which also only relies on the reference, has poor correlation in the hyperparams category  RG-F, and both variants of the PARENT metric achieve the highest correlation for both settings  There is no significant difference among these for the hyperparams category, but for systems, PARENT-W is significantly better than the other two  While RG-F needs a full information extraction pipeline in its implementation, PARENT-C only relies on co-occurrence counts, and PARENT-W can be used out-of-the-box for any dataset  To our knowledge, this is the first rigorous evaluation of using information extraction for generation evaluation 
On this dataset, the word-overlap model showed higher correlation than the co-occurrence model for entailment  In § SECREF47 we will show that for the WebNLG dataset, where more paraphrasing is involved between the table and text, the opposite is true  Lastly, we note that the heuristic for selecting INLINEFORM0 is sufficient to produce high correlations for PARENT, however, if human annotations are available, this can be tuned to produce significantly higher correlations (PARENT*-W/C) 
Analysis
In this section we further analyze the performance of PARENT-W under different conditions, and compare to the other best metrics from Table TABREF37  
To study the correlation as we vary the number of divergent references, we also collected binary labels from workers for whether a reference is entailed by the corresponding table  We define a reference as entailed when it mentions only information which can be inferred from the table  Each table and reference pair was judged by 3 independent workers, and we used the majority vote as the label for that pair  Overall, only INLINEFORM0 of the references were labeled as entailed by the table  Fleiss' INLINEFORM1 was INLINEFORM2 , which indicates a fair agreement  We found the workers sometimes disagreed on what information can be reasonably entailed by the table 
Figure FIGREF40 shows the correlations as we vary the percent of entailed examples in the evaluation set of WikiBio  Each point is obtained by fixing the desired proportion of entailed examples, and sampling subsets from the full set which satisfy this proportion  PARENT and RG-F remain stable and show a high correlation across the entire range, whereas BLEU and BLEU-T vary a lot  In the hyperparams category, the latter two have the worst correlation when the evaluation set contains only entailed examples, which may seem surprising  However, on closer examination we found that this subset tends to omit a lot of information from the tables  Systems which produce more information than these references are penalized by BLEU, but not in the human evaluation  PARENT overcomes this issue by measuring recall against the table in addition to the reference 
We check how different components in the computation of PARENT contribute to its correlation to human judgments  Specifically, we remove the probability INLINEFORM0 of an n-gram INLINEFORM1 being entailed by the table from Eqs  EQREF19 and EQREF23   The average correlation for PARENT-W drops to INLINEFORM5 in this case  We also try a variant of PARENT with INLINEFORM6 , which removes the contribution of Table Recall (Eq  EQREF22 )  The average correlation is INLINEFORM7 in this case  With these components, the correlation is INLINEFORM8 , showing that they are crucial to the performance of PARENT 
BIBREF28 point out that hill-climbing on an automatic metric is meaningless if that metric has a low instance-level correlation to human judgments  In Table TABREF46 we show the average accuracy of the metrics in making the same judgments as humans between pairs of generated texts  Both variants of PARENT are significantly better than the other metrics, however the best accuracy is only INLINEFORM0 for the binary task  This is a challenging task, since there are typically only subtle differences between the texts  Achieving higher instance-level accuracies will require more sophisticated language understanding models for evaluation 
WebNLG Dataset
To check how PARENT correlates with human judgments when the references are elicited from humans (and less likely to be divergent), we check its correlation with the human ratings provided for the systems competing in the WebNLG challenge BIBREF6   The task is to generate text describing 1-5 RDF triples (e g  John E Blaha, birthPlace, San Antonio), and human ratings were collected for the outputs of 9 participating systems on 223 instances  These systems include a mix of pipelined, statistical and neural methods  Each instance has upto 3 reference texts associated with the RDF triples, which we use for evaluation 
The human ratings were collected on 3 distinct aspects – grammaticality, fluency and semantics, where semantics corresponds to the degree to which a generated text agrees with the meaning of the underlying RDF triples  We report the correlation of several metrics with these ratings in Table TABREF48   Both variants of PARENT are either competitive or better than the other metrics in terms of the average correlation to all three aspects  This shows that PARENT is applicable for high quality references as well 
While BLEU has the highest correlation for the grammar and fluency aspects, PARENT does best for semantics  This suggests that the inclusion of source tables into the evaluation orients the metric more towards measuring the fidelity of the content of the generation  A similar trend is seen comparing BLEU and BLEU-T  As modern neural text generation systems are typically very fluent, measuring their fidelity is of increasing importance  Between the two entailment models, PARENT-C is better due to its higher correlation with the grammaticality and fluency aspects 
The INLINEFORM0 parameter in the calculation of PARENT decides whether to compute recall against the table or the reference (Eq  EQREF22 )  Figure FIGREF50 shows the distribution of the values taken by INLINEFORM1 using the heuristic described in § SECREF3 for instances in both WikiBio and WebNLG  For WikiBio, the recall of the references against the table is generally low, and hence the recall of the generated text relies more on the table  For WebNLG, where the references are elicited from humans, this recall is much higher (often INLINEFORM2 ), and hence the recall of the generated text relies more on the reference 
Related Work
Over the years several studies have evaluated automatic metrics for measuring text generation performance BIBREF29 , BIBREF30 , BIBREF31 , BIBREF32 , BIBREF33 , BIBREF34 , BIBREF35   The only consensus from these studies seems to be that no single metric is suitable across all tasks  A recurring theme is that metrics like BLEU and NIST BIBREF36 are not suitable for judging content quality in NLG  Recently, BIBREF37 did a comprehensive study of several metrics on the outputs of state-of-the-art NLG systems, and found that while they showed acceptable correlation with human judgments at the system level, they failed to show any correlation at the sentence level  Ours is the first study which checks the quality of metrics when table-to-text references are divergent  We show that in this case even system level correlations can be unreliable 
Hallucination BIBREF38 , BIBREF39 refers to when an NLG system generates text which mentions extra information than what is present in the source from which it is generated  Divergence can be viewed as hallucination in the reference text itself  PARENT deals with hallucination by discounting n-grams which do not overlap with either the reference or the table 
PARENT draws inspiration from iBLEU BIBREF26 , a metric for evaluating paraphrase generation, which compares the generated text to both the source text and the reference  While iBLEU penalizes texts which match the source, here we reward such texts since our task values accuracy of generated text more than the need for paraphrasing the tabular content BIBREF40   Similar to SARI for text simplification BIBREF41 and Q-BLEU for question generation BIBREF42 , PARENT falls under the category of task-specific metrics 
Conclusions
We study the automatic evaluation of table-to-text systems when the references diverge from the table  We propose a new metric, PARENT, which shows the highest correlation with humans across a range of settings with divergent references in WikiBio  We also perform the first empirical evaluation of information extraction based metrics BIBREF1 , and find RG-F to be effective  Lastly, we show that PARENT is comparable to the best existing metrics when references are elicited by humans on the WebNLG data 
Acknowledgements
Bhuwan Dhingra is supported by a fellowship from Siemens, and by grants from Google  We thank Maruan Al-Shedivat, Ian Tenney, Tom Kwiatkowski, Michael Collins, Slav Petrov, Jason Baldridge, David Reitter and other members of the Google AI Language team for helpful discussions and suggestions  We thank Sam Wiseman for sharing data for an earlier version of this paper  We also thank the anonymous reviewers for their feedback 
Information Extraction System
For evaluation via information extraction BIBREF1 we train a model for WikiBio which accepts text as input and generates a table as the output  Tables in WikiBio are open-domain, without any fixed schema for which attributes may be present or absent in an instance  Hence we employ the Pointer-Generator Network (PG-Net) BIBREF19 for this purpose  Specifically, we use a sequence-to-sequence model, whose encoder and decoder are both single-layer bi-directional LSTMs  The decoder is augmented with an attention mechanism over the states of the encoder  Further, it also uses a copy mechanism to optionally copy tokens directly from the source text  We do not use the coverage mechanism of BIBREF19 since that is specific to the task of summarization they study  The decoder is trained to produce a linearized version of the table where the rows and columns are flattened into a sequence, and separate by special tokens  Figure FIGREF53 shows an example 
Clearly, since the references are divergent, the model cannot be expected to produce the entire table, and we see some false information being hallucinated after training  Nevertheless, as we show in § SECREF36 , this system can be used for evaluating generated texts  After training, we can parse the output sequence along the special tokens INLINEFORM0 R INLINEFORM1 and INLINEFORM2 C INLINEFORM3 to get a set of (attribute, value) pairs  Table TABREF54 shows the precision, recall and F-score of these extracted pairs against the ground truth tables, where the attributes and values are compared using an exact string match 
Hyperparameters
After tuning we found the same set of hyperparameters to work well for both the table-to-text PG-Net, and the inverse information extraction PG-Net  The hidden state size of the biLSTMs was set to 200  The input and output vocabularies were set to 50000 most common words in the corpus, with additional special symbols for table attribute names (such as “birth-date”)  The embeddings of the tokens in the vocabulary were initialized with Glove BIBREF43   Learning rate of INLINEFORM0 was used during training, with the Adam optimizer, and a dropout of INLINEFORM1 was also applied to the outputs of the biLSTM  Models were trained till the loss on the dev set stopped dropping  Maximum length of a decoded text was set to 40 tokens, and that of the tables was set to 120 tokens  Various beam sizes and length normalization penalties were applied for the table-to-text system, which are listed in the main paper  For the information extraction system, we found a beam size of 8 and no length penalty to produce the highest F-score on the dev set 
Sample Outputs
Table TABREF55 shows some sample references and the corresponding predictions from the best performing model, PG-Net for WikiBio 

ground_truth: Yes

Yes

prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. What architecture does the encoder have?
Context 2:  Training data is split 90:10 for training and validation  We train our models for 50 epochs, adding early stopping with a tolerance of five epochs of no improvement in the validation loss  We do not subsample from the training data 
We train models for 50 different random combinations of two to three languages in Track 1, and 50 monolingual models for each language in Track 2  Instead of picking the single model that performs best on the development set and thus risking to select a model that highly overfits that data, we use an ensemble of the five best models, and make the final prediction for a given target form with a majority vote over the five predictions 
Results and Discussion
Test results are listed in Table TABREF17   Our system outperforms the baseline for all settings and languages in Track 1 and for almost all in Track 2—only in the high resource setting is our system not definitively superior to the baseline 
Interestingly, our results in the low resource setting are often higher for Track 2 than for Track 1, even though contextual information is less explicit in the Track 2 data and the multilingual multi-tasking approach does not apply to this track  We interpret this finding as an indicator that a simpler model with fewer parameters works better in a setting of limited training data  Nevertheless, we focus on the low resource setting in the analysis below due to time limitations  As our Track 1 results are still substantially higher than the baseline results, we consider this analysis valid and insightful 
Ablation Study
We analyse the incremental effect of the different features in our system, focusing on the low-resource setting in Track 1 and using development data 
Encoding the entire context with an LSTM highly increases the variance of the observed results  So we trained fifty models for each language and each architecture  Figure FIGREF23 visualises the means and standard deviations over the trained models  In addition, we visualise the average accuracy for the five best models for each language and architecture, as these are the models we use in the final ensemble prediction  Below we refer to these numbers only 
The results indicate that encoding the full context with an LSTM highly enhances the performance of the model, by 11 15% on average  This observation explains the high results we obtain also for Track 2 
Adding the auxiliary objective of MSD prediction has a variable effect: for four languages (de, en, es, and sv) the effect is positive, while for the rest it is negative  We consider this to be an issue of insufficient data for the training of the auxiliary component in the low resource setting we are working with 
We indeed see results improving drastically with the introduction of multilingual training, with multilingual results being 7 96% higher than monolingual ones on average 
We studied the five best models for each language as emerging from the multilingual training (listed in Table TABREF27 ) and found no strong linguistic patterns  The en–sv pairing seems to yield good models for these languages, which could be explained in terms of their common language family and similar morphology  The other natural pairings, however, fr–es, and de–sv, are not so frequent among the best models for these pairs of languages 
Finally, monolingual finetuning improves accuracy across the board, as one would expect, by 2 72% on average 
The final observation to be made based on this breakdown of results is that the multi-tasking approach paired with multilingual training and subsequent monolingual finetuning outperforms the other architectures for five out of seven languages: de, en, fr, ru and sv  For the other two languages in the dataset, es and fi, the difference between this approach and the approach that emerged as best for them is less than 1%  The overall improvement of the multilingual multi-tasking approach over the baseline is 18 30% 
Error analysis
Here we study the errors produced by our system on the English test set to better understand the remaining shortcomings of the approach  A small portion of the wrong predictions point to an incorrect interpretation of the morpho-syntactic conditioning of the context, e g  the system predicted plan instead of plans in the context Our _ include raising private capital  The majority of wrong predictions, however, are nonsensical, like bomb for job, fify for fixing, and gnderrate for understand  This observation suggests that generally the system did not learn to copy the characters of lemma into inflected form, which is all it needs to do in a large number of cases  This issue could be alleviated with simple data augmentation techniques that encourage autoencoding BIBREF2  
MSD prediction
Figure FIGREF32 summarises the average MSD-prediction accuracy for the multi-tasking experiments discussed above  Accuracy here is generally higher than on the main task, with the multilingual finetuned setup for Spanish and the monolingual setup for French scoring best: 66 59% and 65 35%, respectively  This observation illustrates the added difficulty of generating the correct surface form even when the morphosyntactic description has been identified correctly 
We observe some correlation between these numbers and accuracy on the main task: for de, en, ru and sv, the brown, pink and blue bars here pattern in the same way as the corresponding INLINEFORM0 's in Figure FIGREF23   One notable exception to this pattern is fr where inflection gains a lot from multilingual training, while MSD prediction suffers greatly  Notice that the magnitude of change is not always the same, however, even when the general direction matches: for ru, for example, multilingual training benefits inflection much more than in benefits MSD prediction, even though the MSD decoder is the only component that is actually shared between languages  This observation illustrates the two-fold effect of multi-task training: an auxiliary task can either inform the main task through the parameters the two tasks share, or it can help the main task learning through its regularising effect 
Related Work
Our system is inspired by previous work on multi-task learning and multi-lingual learning, mainly building on two intuitions: (1) jointly learning related tasks tends to be beneficial BIBREF3 , BIBREF4 , BIBREF5 , BIBREF6 , BIBREF7 ; and (2) jointly learning related languages in an MTL-inspired framework tends to be beneficial BIBREF8 , BIBREF9 , BIBREF10   In the context of computational morphology, multi-lingual approaches have previously been employed for morphological reinflection BIBREF2 and for paradigm completion BIBREF11   In both of these cases, however, the available datasets covered more languages, 40 and 21, respectively, which allowed for linguistically-motivated language groupings and for parameter sharing directly on the level of characters  BIBREF10 explore parameter sharing between related languages for dependency parsing, and find that sharing is more beneficial in the case of closely related languages 
Conclusions
In this paper we described our system for the CoNLL–SIGMORPHON 2018 shared task on Universal Morphological Reinflection, Task 2, which achieved the best performance out of all systems submitted, an overall accuracy of 49 87  We showed in an ablation study that this is due to three core innovations, which extend a character-based encoder-decoder model: (1) a wide context window, encoding the entire available context; (2) multi-task learning with the auxiliary task of MSD prediction, which acts as a regulariser; (3) a multilingual approach, exploiting information across languages  In future work we aim to gain better understanding of the increase in variance of the results introduced by each of our modifications and the reasons for the varying effect of multi-task learning for different languages 
Acknowledgements
We gratefully acknowledge the support of the NVIDIA Corporation with the donation of the Titan Xp GPU used for this research 

ground_truth: LSTM

LSTM

prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. Ngrams of which length are aligned using PARENT?
Context 4: 
Sequence Labeling  Predict each token in a sequence into predefined types  Common tasks include NER, POS tagging, Slot tagging, etc 
Knowledge Distillation BIBREF7   Teacher-Student based knowledge distillation is one common approach for model compression  NeuronBlocks provides knowledge distillation template to improve the inference speed of heavy DNN models like BERT/GPT 
Extractive Machine Reading Comprehension  Given a pair of question and passage, predict the start and end positions of the answer spans in the passage 
User Interface
NeuronBlocks provides convenient user interface for users to build, train, and test DNN models  The details are described in the following 
[itemsep= -0 4em,topsep = 0 3em, align=left, labelsep=-0 6em, leftmargin=1 2em]
I/O interface  This part defines model input/output, such as training data, pre-trained models/embeddings, model saving path, etc 
Model Architecture interface  This is the key part of the configuration file, which defines the whole model architecture  Figure FIGREF19 shows an example of how to specify a model architecture using the blocks in NeuronBlocks  To be more specific, it consists of a list of layers/blocks to construct the architecture, where the blocks are supplied in the gallery of Block Zoo 
Training Parameters interface  In this part, the model optimizer as well as all other training hyper parameters are indicated 
Workflow
Figure FIGREF34 shows the workflow of building DNN models in NeuronBlocks  Users only need to write a JSON configuration file  They can either instantiate an existing template from Model Zoo, or construct a new architecture based on the blocks from Block Zoo  This configuration file is shared across training, test, and prediction  For model hyper-parameter tuning or architecture modification, users just need to change the JSON configuration file  Advanced users can also contribute novel customized blocks into Block Zoo, as long as they follow the same interface guidelines with the existing blocks  These new blocks can be further shared across all users for model architecture design  Moreover, NeuronBlocks has flexible platform support, such as GPU/CPU, GPU management platforms like PAI 
Experiments
To verify the performance of NeuronBlocks, we conducted extensive experiments for common NLP tasks on public data sets including CoNLL-2003 BIBREF14 , GLUE benchmark BIBREF13 , and WikiQA corpus BIBREF15   The experimental results showed that the models built with NeuronBlocks can achieve reliable and competitive results on various tasks, with productivity greatly improved 
Sequence Labeling
For sequence labeling task, we evaluated NeuronBlocks on CoNLL-2003 BIBREF14 English NER dataset, following most works on the same task  This dataset includes four types of named entities, namely, PERSON, LOCATION, ORGANIZATION, and MISC  We adopted the BIOES tagging scheme instead of IOB, as many previous works indicated meaningful improvement with BIOES scheme BIBREF16 , BIBREF17   Table TABREF28 shows the results on CoNLL-2003 Englist testb dataset, with 12 different combinations of network layers/blocks, such as word/character embedding, CNN/LSTM and CRF  The results suggest that the flexible combination of layers/blocks in NeuronBlocks can easily reproduce the performance of original models, with comparative or slightly better performance 
GLUE Benchmark
The General Language Understanding Evaluation (GLUE) benchmark BIBREF13 is a collection of natural language understanding tasks  We experimented on the GLUE benchmark tasks using BiLSTM and Attention based models  As shown in Table TABREF29 , the models built by NeuronBlocks can achieve competitive or even better results on GLUE tasks with minimal coding efforts 
Knowledge Distillation
We evaluated Knowledge Distillation task in NeuronBlocks on a dataset collected from one commercial search engine  We refer to this dataset as Domain Classification Dataset  Each sample in this dataset consists of two parts, i e , a question and a binary label indicating whether the question belongs to a specific domain  Table TABREF36 shows the results, where Area Under Curve (AUC) metric is used as the performance evaluation criteria and Queries per Second (QPS) is used to measure inference speed  By knowledge distillation training approach, the student model by NeuronBlocks managed to get 23-27 times inference speedup with only small performance regression compared with BERTbase fine-tuned classifier 
WikiQA
The WikiQA corpus BIBREF15 is a publicly available dataset for open-domain question answering  This dataset contains 3,047 questions from Bing query logs, each associated with some candidate answer sentences from Wikipedia  We conducted experiments on WikiQA dataset using CNN, BiLSTM, and Attention based models  The results are shown in Table TABREF41   The models built in NeuronBlocks achieved competitive or even better results with simple model configurations 
Conclusion and Future Work
In this paper, we introduce NeuronBlocks, a DNN toolkit for NLP tasks built on PyTorch  NeuronBlocks targets three types of engineers, and provides a two-layer solution to satisfy the requirements from all three types of users  To be more specific, the Model Zoo consists of various templates for the most common NLP tasks, while the Block Zoo supplies a gallery of alternative layers/modules for the networks  Such design achieves a balance between generality and flexibility  Extensive experiments have verified the effectiveness of this approach  NeuronBlocks has been widely used in a product team of a commercial search engine, and significantly improved the productivity for developing NLP DNN approaches 
As an open-source toolkit, we will further extend it in various directions  The following names a few examples 

ground_truth: Unanswerable

Answer with content missing: (Parent subsections) combine precisions for n-gram orders 1-4