prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. Where does the ancient Chinese dataset come from?
Context 1: Introduction
Ancient Chinese is the writing language in ancient China  It is a treasure of Chinese culture which brings together the wisdom and ideas of the Chinese nation and chronicles the ancient cultural heritage of China  Learning ancient Chinese not only helps people to understand and inherit the wisdom of the ancients, but also promotes people to absorb and develop Chinese culture 
However, it is difficult for modern people to read ancient Chinese  Firstly, compared with modern Chinese, ancient Chinese is more concise and shorter  The grammatical order of modern Chinese is also quite different from that of ancient Chinese  Secondly, most modern Chinese words are double syllables, while the most of the ancient Chinese words are monosyllabic  Thirdly, there is more than one polysemous phenomenon in ancient Chinese  In addition, manual translation has a high cost  Therefore, it is meaningful and useful to study the automatic translation from ancient Chinese to modern Chinese  Through ancient-modern Chinese translation, the wisdom, talent and accumulated experience of the predecessors can be passed on to more people 
Neural machine translation (NMT) BIBREF0 , BIBREF1 , BIBREF2 , BIBREF3 , BIBREF4 has achieved remarkable performance on many bilingual translation tasks  It is an end-to-end learning approach for machine translation, with the potential to show great advantages over the statistic machine translation (SMT) systems  However, NMT approach has not been widely applied to the ancient-modern Chinese translation task  One of the main reasons is the limited high-quality parallel data resource 
The most popular method of acquiring translation examples is bilingual text alignment BIBREF5   This kind of method can be classified into two types: lexical-based and statistical-based  The lexical-based approaches BIBREF6 , BIBREF7 focus on lexical information, which utilize the bilingual dictionary BIBREF8 , BIBREF9 or lexical features  Meanwhile, the statistical-based approaches BIBREF10 , BIBREF11 rely on statistical information, such as sentence length ratio in two languages and align mode probability 
However, these methods are designed for other bilingual language pairs that are written in different language characters (e g  English-French, Chinese-Japanese)  The ancient-modern Chinese has some characteristics that are quite different from other language pairs  For example, ancient and modern Chinese are both written in Chinese characters, but ancient Chinese is highly concise and its syntactical structure is different from modern Chinese  The traditional methods do not take these characteristics into account  In this paper, we propose an effective ancient-modern Chinese text alignment method at the level of clause based on the characteristics of these two languages  The proposed method combines both lexical-based information and statistical-based information, which achieves 94 2 F1-score on Test set  Recently, a simple longest common subsequence based approach for ancient-modern Chinese sentence alignment is proposed in BIBREF12   Our experiments showed that our proposed alignment approach performs much better than their method 
We apply the proposed method to create a large translation parallel corpus which contains INLINEFORM0 1 24M bilingual sentence pairs  To our best knowledge, this is the first large high-quality ancient-modern Chinese dataset  Furthermore, we test SMT models and various NMT models on the created dataset and provide a strong baseline for this task 
Overview
There are four steps to build the ancient-modern Chinese translation dataset: (i) The parallel corpus crawling and cleaning  (ii) The paragraph alignment  (iii) The clause alignment based on aligned paragraphs  (iv) Augmenting data by merging aligned adjacent clauses  The most critical step is the third step 
Clause Alignment
In the clause alignment step, we combine both statistical-based and lexical-based information to measure the score for each possible clause alignment between ancient and modern Chinese strings  The dynamic programming is employed to further find overall optimal alignment paragraph by paragraph  According to the characteristics of the ancient and modern Chinese languages, we consider the following factors to measure the alignment score INLINEFORM0 between a bilingual clause pair:
Lexical Matching  The lexical matching score is used to calculate the matching coverage of the ancient clause INLINEFORM0   It contains two parts: exact matching and dictionary matching  An ancient Chinese character usually corresponds to one or more modern Chinese words  In the first part, we carry out Chinese Word segmentation to the modern Chinese clause INLINEFORM1   Then we match the ancient characters and modern words in the order from left to right  In further matching, the words that have been matched will be deleted from the original clauses 
However, some ancient characters do not appear in its corresponding modern Chinese words  An ancient Chinese dictionary is employed to address this issue  We preprocess the ancient Chinese dictionary and remove the stop words  In this dictionary matching step, we retrieve the dictionary definition of each unmatched ancient character and use it to match the remaining modern Chinese words  To reduce the impact of universal word matching, we use Inverse Document Frequency (IDF) to weight the matching words  The lexical matching score is calculated as: DISPLAYFORM0
The above equation is used to calculate the matching coverage of the ancient clause INLINEFORM0   The first term of equation ( EQREF8 ) represents exact matching score  INLINEFORM1 denotes the length of INLINEFORM2 , INLINEFORM3 denotes each ancient character in INLINEFORM4 , and the indicator function INLINEFORM5 indicates whether the character INLINEFORM6 can match the words in the clause INLINEFORM7   The second term is dictionary matching score  Here INLINEFORM8 and INLINEFORM9 represent the remaining unmatched strings of INLINEFORM10 and INLINEFORM11 , respectively  INLINEFORM12 denotes the INLINEFORM13 -th character in the dictionary definition of the INLINEFORM14 and its IDF score is denoted as INLINEFORM15   The INLINEFORM16 is a predefined parameter which is used to normalize the IDF score  We tuned the value of this parameter on the Dev set 
Statistical Information  Similar to BIBREF11 and BIBREF6 , the statistical information contains alignment mode and length information  There are many alignment modes between ancient and modern Chinese languages  If one ancient Chinese clause aligns two adjacent modern Chinese clauses, we call this alignment as 1-2 alignment mode  We show some examples of different alignment modes in Figure FIGREF9   In this paper, we only consider 1-0, 0-1, 1-1, 1-2, 2-1 and 2-2 alignment modes which account for INLINEFORM0 of the Dev set  We estimate the probability Pr INLINEFORM1 n-m INLINEFORM2 of each alignment mode n-m on the Dev set  To utilize length information, we make an investigation on length correlation between these two languages  Based on the assumption of BIBREF11 that each character in one language gives rise to a random number of characters in the other language and those random variables INLINEFORM3 are independent and identically distributed with a normal distribution, we estimate the mean INLINEFORM4 and standard deviation INLINEFORM5 from the paragraph aligned parallel corpus  Given a clause pair INLINEFORM6 , the statistical information score can be calculated by: DISPLAYFORM0
where INLINEFORM0 denotes the normal distribution probability density function 
Edit Distance  Because ancient and modern Chinese are both written in Chinese characters, we also consider using the edit distance  It is a way of quantifying the dissimilarity between two strings by counting the minimum number of operations (insertion, deletion, and substitution) required to transform one string into the other  Here we define the edit distance score as: DISPLAYFORM0
Dynamic Programming  The overall alignment score for each possible clause alignment is as follows: DISPLAYFORM0
Here INLINEFORM0 and INLINEFORM1 are pre-defined interpolation factors  We use dynamic programming to find the overall optimal alignment paragraph by paragraph  Let INLINEFORM2 be total alignment scores of aligning the first to INLINEFORM3 -th ancient Chinese clauses with the first to to INLINEFORM4 -th modern Chinese clauses, and the recurrence then can be described as follows: DISPLAYFORM0
Where INLINEFORM0 denotes concatenate clause INLINEFORM1 to clause INLINEFORM2   As we discussed above, here we only consider 1-0, 0-1, 1-1, 1-2, 2-1 and 2-2 alignment modes 
Ancient-Modern Chinese Dataset
Data Collection  To build the large ancient-modern Chinese dataset, we collected 1 7K bilingual ancient-modern Chinese articles from the internet  More specifically, a large part of the ancient Chinese data we used come from ancient Chinese history records in several dynasties (about 1000BC-200BC) and articles written by celebrities of that era  They used plain and accurate words to express what happened at that time, and thus ensure the generality of the translated materials 
Paragraph Alignment  To further ensure the quality of the new dataset, the work of paragraph alignment is manually completed  After data cleaning and manual paragraph alignment, we obtained 35K aligned bilingual paragraphs 
Clause Alignment  We applied our clause alignment algorithm on the 35K aligned bilingual paragraphs and obtained 517K aligned bilingual clauses  The reason we use clause alignment algorithm instead of sentence alignment is because we can construct more aligned sentences more flexibly and conveniently  To be specific, we can get multiple additional sentence level bilingual pairs by “data augmentation” 
Data Augmentation  We augmented the data in the following way: Given an aligned clause pair, we merged its adjacent clause pairs as a new sample pair  For example, suppose we have three adjacent clause level bilingual pairs: ( INLINEFORM0 , INLINEFORM1 ), ( INLINEFORM2 , INLINEFORM3 ), and ( INLINEFORM4 , INLINEFORM5 )  We can get some additional sentence level bilingual pairs, such as: ( INLINEFORM6 , INLINEFORM7 ) and ( INLINEFORM8 , INLINEFORM9 )  Here INLINEFORM10 , INLINEFORM11 , and INLINEFORM12 are adjacent clauses in the original paragraph, and INLINEFORM13 denotes concatenate clause INLINEFORM14 to clause INLINEFORM15   The advantage of using this data augmentation method is that compared with only using ( INLINEFORM16 , INLINEFORM17 ) as the training data, we can also use ( INLINEFORM18 , INLINEFORM19 ) and ( INLINEFORM20 , INLINEFORM21 ) as the training data, which can provide richer supervision information for the model and make the model learn the align information between the source language and the target language better  After the data augmentation, we filtered the sentences which are longer than 50 or contain more than four clause pairs 
Dataset Creation  Finally, we split the dataset into three sets: training (Train), development (Dev) and testing (Test)  Note that the unaugmented dataset contains 517K aligned bilingual clause pairs from 35K aligned bilingual paragraphs  To keep all the sentences in different sets come from different articles, we split the 35K aligned bilingual paragraphs into Train, Dev and Test sets following these ratios respectively: 80%, 10%, 10%  Before data augmentation, the unaugmented Train set contains INLINEFORM0 aligned bilingual clause pairs from 28K aligned bilingual paragraphs  Then we augmented the Train, Dev and Test sets respectively  Note that the augmented Train, Dev and Test sets also contain the unaugmented data  The statistical information of the three data sets is shown in Table TABREF17   We show some examples of data in Figure FIGREF14  
RNN-based NMT model
We first briefly introduce the RNN based Neural Machine Translation (RNN-based NMT) model  The RNN-based NMT with attention mechanism BIBREF0 has achieved remarkable performance on many translation tasks  It consists of encoder and decoder part 
We firstly introduce the encoder part  The input word sequence of source language are individually mapped into a INLINEFORM0 -dimensional vector space INLINEFORM1   Then a bi-directional RNN BIBREF15 with GRU BIBREF16 or LSTM BIBREF17 cell converts these vectors into a sequences of hidden states INLINEFORM2  
For the decoder part, another RNN is used to generate target sequence INLINEFORM0   The attention mechanism BIBREF0 , BIBREF18 is employed to allow the decoder to refer back to the hidden state sequence and focus on a particular segment  The INLINEFORM1 -th hidden state INLINEFORM2 of decoder part is calculated as: DISPLAYFORM0
Here g INLINEFORM0 is a linear combination of attended context vector c INLINEFORM1 and INLINEFORM2 is the word embedding of (i-1)-th target word: DISPLAYFORM0
The attended context vector c INLINEFORM0 is computed as a weighted sum of the hidden states of the encoder: DISPLAYFORM0
The probability distribution vector of the next word INLINEFORM0 is generated according to the following: DISPLAYFORM0
We take this model as the basic RNN-based NMT model in the following experiments 
Transformer-NMT
Recently, the Transformer model BIBREF4 has made remarkable progress in machine translation  This model contains a multi-head self-attention encoder and a multi-head self-attention decoder 
As proposed by BIBREF4 , an attention function maps a query and a set of key-value pairs to an output, where the queries INLINEFORM0 , keys INLINEFORM1 , and values INLINEFORM2 are all vectors  The input consists of queries and keys of dimension INLINEFORM3 , and values of dimension INLINEFORM4   The attention function is given by: DISPLAYFORM0
Multi-head attention mechanism projects queries, keys and values to INLINEFORM0 different representation subspaces and calculates corresponding attention  The attention function outputs are concatenated and projected again before giving the final output  Multi-head attention allows the model to attend to multiple features at different positions 
The encoder is composed of a stack of INLINEFORM0 identical layers  Each layer has two sub-layers: multi-head self-attention mechanism and position-wise fully connected feed-forward network  Similarly, the decoder is also composed of a stack of INLINEFORM1 identical layers  In addition to the two sub-layers in each encoder layer, the decoder contains a third sub-layer which performs multi-head attention over the output of the encoder stack (see more details in BIBREF4 ) 
Experiments
Our experiments revolve around the following questions: Q1: As we consider three factors for clause alignment, do all these factors help? How does our method compare with previous methods? Q2: How does the NMT and SMT models perform on this new dataset we build?
Clause Alignment Results (Q1)
In order to evaluate our clause alignment algorithm, we manually aligned bilingual clauses from 37 bilingual ancient-modern Chinese articles, and finally got 4K aligned bilingual clauses as the Test set and 2K clauses as the Dev set 
Metrics  We used F1-score and precision score as the evaluation metrics  Suppose that we get INLINEFORM0 bilingual clause pairs after running the algorithm on the Test set, and there are INLINEFORM1 bilingual clause pairs of these INLINEFORM2 pairs are in the ground truth of the Test set, the precision score is defined as INLINEFORM3 (the algorithm gives INLINEFORM4 outputs, INLINEFORM5 of which are correct)  And suppose that the ground truth of the Test set contains INLINEFORM6 bilingual clause pairs, the recall score is INLINEFORM7 (there are INLINEFORM8 ground truth samples, INLINEFORM9 of which are output by the algorithm), then the F1-score is INLINEFORM10  
Baselines  Since the related work BIBREF10 , BIBREF11 can be seen as the ablation cases of our method (only statistical score INLINEFORM0 with dynamic programming), we compared the full proposed method with its variants on the Test set for ablation study  In addition, we also compared our method with the longest common subsequence (LCS) based approach proposed by BIBREF12   To the best of our knowledge, BIBREF12 is the latest related work which are designed for Ancient-Modern Chinese alignment 
Hyper-parameters  For the proposed method, we estimated INLINEFORM0 and INLINEFORM1 on all aligned paragraphs  The probability Pr INLINEFORM2 n-m INLINEFORM3 of each alignment mode n-m was estimated on the Dev set  For the hyper-parameters INLINEFORM4 , INLINEFORM5 and INLINEFORM6 , the grid search was applied to tune them on the Dev set  In order to show the effect of hyper-parameters INLINEFORM7 , INLINEFORM8 , and INLINEFORM9 , we reported the results of various hyper-parameters on the Dev set in Table TABREF26   Based on the results of grid search on the Dev set, we set INLINEFORM10 , INLINEFORM11 , and INLINEFORM12 in the following experiment  The Jieba Chinese text segmentation is employed for modern Chinese word segmentation 
Results  The results on the Test set are shown in Table TABREF28 , the abbreviation w/o means removing a particular part from the setting  From the results, we can see that the lexical matching score is the most important among these three factors, and statistical information score is more important than edit distance score  Moreover, the dictionary term in lexical matching score significantly improves the performance  From these results, we obtain the best setting that involves all these three factors  We used this setting for dataset creation  Furthermore, the proposed method performs much better than LCS BIBREF12  
Translation Results (Q2)
In this experiment, we analyzed and compared the performance of the SMT and various NMT models on our built dataset  To verify the effectiveness of our data augmented method  We trained the NMT and SMT models on both unaugmented dataset (including 0 46M training pairs) and augmented dataset, and test all the models on the same Test set which is augmented  The models to be tested and their configurations are as follows:
SMT  The state-of-art Moses toolkit BIBREF19 was used to train SMT model  We used KenLM BIBREF20 to train a 5-gram language model, and the GIZA++ toolkit to align the data 
RNN-based NMT  The basic RNN-based NMT model is based on BIBREF0 which is introduced above  Both the encoder and decoder used 2-layer RNN with 1024 LSTM cells, and the encoder is a bi-directional RNN  The batch size, threshold of element-wise gradient clipping and initial learning rate of Adam optimizer BIBREF21 were set to 128, 5 0 and 0 001  When trained the model on augmented dataset, we used 4-layer RNN  Several techniques were investigated to train the model, including layer-normalization BIBREF22 , RNN-dropout BIBREF23 , and learning rate decay BIBREF1   The hyper-parameters were chosen empirically and adjusted in the Dev set  Furthermore, we tested the basic NMT model with several techniques, such as target language reversal BIBREF24 (reversing the order of the words in all target sentences, but not source sentences), residual connection BIBREF25 and pre-trained word2vec BIBREF26   For word embedding pre-training, we collected an external ancient corpus which contains INLINEFORM0 134M tokens 
Transformer-NMT  We also trained the Transformer model BIBREF4 which is a strong baseline of NMT on both augmented and unaugmented parallel corpus  The training configuration of the Transformer model is shown in Table TABREF32   The hyper-parameters are set based on the settings in the paper BIBREF4 and the sizes of our training sets 
For the evaluation, we used the average of 1 to 4 gram BLEUs multiplied by a brevity penalty BIBREF27 which computed by multi-bleu perl in Moses as metrics  The results are reported in Table TABREF34   For RNN-based NMT, we can see that target language reversal, residual connection, and word2vec can further improve the performance of the basic RNN-based NMT model  However, we find that word2vec and reversal tricks seem no obvious improvement when trained the RNN-based NMT and Transformer models on augmented parallel corpus  For SMT, it performs better than NMT models when they were trained on the unaugmented dataset  Nevertheless, when trained on the augmented dataset, both the RNN-based NMT model and Transformer based NMT model outperform the SMT model  In addition, as with other translation tasks BIBREF4 , the Transformer also performs better than RNN-based NMT 
Because the Test set contains both augmented and unaugmented data, it is not surprising that the RNN-based NMT model and Transformer based NMT model trained on unaugmented data would perform poorly  In order to further verify the effect of data augmentation, we report the test results of the models on only unaugmented test data (including 48K test pairs) in Table TABREF35   From the results, it can be seen that the data augmentation can still improve the models 
Analysis
The generated samples of various models are shown in Figure FIGREF36   Besides BLEU scores, we analyze these examples from a human perspective and draw some conclusions  At the same time, we design different metrics and evaluate on the whole Test set to support our conclusions as follows:
On the one hand, we further compare the translation results from the perspective of people  We find that although the original meaning can be basically translated by SMT, its translation results are less smooth when compared with the other two NMT models (RNN-based NMT and Transformer)  For example, the translations of SMT are usually lack of auxiliary words, conjunctions and function words, which is not consistent with human translation habits  To further confirm this conclusion, the average length of the translation results of the three models are measured (RNN-based NMT:17 12, SMT:15 50, Transformer:16 78, Reference:16 47)  We can see that the average length of the SMT outputs is shortest, and the length gaps between the SMT outputs and the references are largest  Meanwhile, the average length of the sentences translated by Transformer is closest to the average length of references  These results indirectly verify our point of view, and show that the NMT models perform better than SMT in this task 
On the other hand, there still exists some problems to be solved  We observe that translating proper nouns and personal pronouns (such as names, place names and ancient-specific appellations) is very difficult for all of these models  For instance, the ancient Chinese appellation `Zhen' should be translated into `Wo' in modern Chinese  Unfortunately, we calculate the accurate rate of some special words (such as `Zhen',`Chen' and `Gua'), and find that this rate is very low (the accurate rate of translating `Zhen' are: RNN-based NMT:0 14, SMT:0 16, Transformer:0 05)  We will focus on this issue in the future 
Conclusion and Future Work
We propose an effective ancient-modern Chinese clause alignment method which achieves 94 2 F1-score on Test set  Based on it, we build a large scale parallel corpus which contains INLINEFORM0 1 24M bilingual sentence pairs  To our best knowledge, this is the first large high-quality ancient-modern Chinese dataset  In addition, we test the performance of the SMT and various NMT models on our built dataset and provide a strong NMT baseline for this task which achieves 27 16 BLEU score (4-gram)  We further analyze the performance of the SMT and various NMT models and summarize some specific problems that machine translation models will encounter when translating ancient Chinese 
For the future work, firstly, we are going to expand the dataset using the proposed method continually  Secondly, we will focus on solving the problem of proper noun translation and improve the translation system according to the features of ancient Chinese translation  Finally, we plan to introduce some techniques of statistical translation into neural machine translation to improve the performance 
This work is supported by National Natural Science Fund for Distinguished Young Scholar (Grant No  61625204) and partially supported by the State Key Program of National Science Foundation of China (Grant Nos  61836006 and 61432014) 
Context 2: Introduction
Neural networks have been successfully used to describe images with text using sequence-to-sequence models BIBREF0  However, the results are simple and dry captions which are one or two phrases long  Humans looking at a painting see more than just objects  Paintings stimulate sentiments, metaphors and stories as well  Therefore, our goal is to have a neural network describe the painting artistically in a style of choice  As a proof of concept, we present a model which generates Shakespearean prose for a given painting as shown in Figure FIGREF1  Accomplishing this task is difficult with traditional sequence to sequence models since there does not exist a large collection of Shakespearean prose which describes paintings: Shakespeare's works describes a single painting shown in Figure FIGREF3  Fortunately we have a dataset of modern English poems which describe images BIBREF1 and line-by-line modern paraphrases of Shakespeare's plays BIBREF2  Our solution is therefore to combine two separately trained models to synthesize Shakespearean prose for a given painting 
Introduction ::: Related work
A general end-to-end approach to sequence learning BIBREF3 makes minimal assumptions on the sequence structure  This model is widely used in tasks such as machine translation, text summarization, conversational modeling, and image captioning  A generative model using a deep recurrent architecture BIBREF0 has also beeen used for generating phrases describing an image  The task of synthesizing multiple lines of poetry for a given image BIBREF1 is accomplished by extracting poetic clues from images  Given the context image, the network associates image attributes with poetic descriptions using a convolutional neural net  The poem is generated using a recurrent neural net which is trained using multi-adversarial training via policy gradient 
Transforming text from modern English to Shakespearean English using text "style transfer" is challenging  An end to end approach using a sequence-to-sequence model over a parallel text corpus BIBREF2 has been proposed based on machine translation  In the absence of a parallel text corpus, generative adversarial networks (GANs) have been used, which simultaneously train two models: a generative model which captures the data distribution, and a discriminative model which evaluates the performance of the generator  Using a target domain language model as a discriminator has also been employed BIBREF4, providing richer and more stable token-level feedback during the learning process  A key challenge in both image and text style transfer is separating content from style BIBREF5, BIBREF6, BIBREF7  Cross-aligned auto-encoder models have focused on style transfer using non-parallel text BIBREF7  Recently, a fine grained model for text style transfer has been proposed BIBREF8 which controls several factors of variation in textual data by using back-translation  This allows control over multiple attributes, such as gender and sentiment, and fine-grained control over the trade-off between content and style 
Methods
We use a total three datasets: two datasets for generating an English poem from an image, and Shakespeare plays and their English translations for text style transfer 
We train a model for generating poems from images based on two datasets BIBREF1  The first dataset consists of image and poem pairs, namely a multi-modal poem dataset (MultiM-Poem), and the second dataset is a large poem corpus, namely a uni-modal poem dataset (UniM-Poem)  The image and poem pairs are extended by adding the nearest three neighbor poems from the poem corpus without redundancy, and an extended image and poem pair dataset is constructed and denoted as MultiM-Poem(Ex)BIBREF1 
We use a collection of line-by-line modern paraphrases for 16 of Shakespeare’s plays BIBREF2, for training a style transfer network from English poems to Shakespearean prose  We use 18,395 sentences from the training data split  We keep 1,218 sentences in the validation data set and 1,462 sentences in our test set 
Methods ::: Image To Poem Actor-Critic Model
For generating a poem from images we use an existing actor-critic architecture BIBREF1  This involves 3 parallel CNNs: an object CNN, sentiment CNN, and scene CNN, for feature extraction  These features are combined with a skip-thought model which provides poetic clues, which are then fed into a sequence-to-sequence model trained by policy gradient with 2 discriminator networks for rewards  This as a whole forms a pipeline that takes in an image and outputs a poem as shown on the top left of Figure FIGREF4  A CNN-RNN generative model acts as an agent  The parameters of this agent define a policy whose execution determines which word is selected as an action  When the agent selects all words in a poem, it receives a reward  Two discriminative networks, shown on the top right of Figure FIGREF4, are defined to serve as rewards concerning whether the generated poem properly describes the input image and whether the generated poem is poetic  The goal of the poem generation model is to generate a sequence of words as a poem for an image to maximize the expected return 
Methods ::: Shakespearizing Poetic Captions
For Shakespearizing modern English texts, we experimented with various types of sequence to sequence models  Since the size of the parallel translation data available is small, we leverage a dictionary providing a mapping between Shakespearean words and modern English words to retrofit pre-trained word embeddings
Context 3: Introduction
Neural Machine Translation (NMT) has shown its effectiveness in translation tasks when NMT systems perform best in recent machine translation campaigns BIBREF0 , BIBREF1   Compared to phrase-based Statistical Machine Translation (SMT) which is basically an ensemble of different features trained and tuned separately, NMT directly modeling the translation relationship between source and target sentences  Unlike SMT, NMT does not require much linguistic information and large monolingual data to achieve good performances 
An NMT consists of an encoder which recursively reads and represents the whole source sentence into a context vector and a recurrent decoder which takes the context vector and its previous state to predict the next target word  It is then trained in an end-to-end fashion to learn parameters which maximizes the likelihood between the outputs and the references  Recently, attention-based NMT has been featured in most state-of-the-art systems  First introduced by BIBREF2 , attention mechanism is integrated in decoder side as feedforward layers  It allows the NMT to decide which source words should take part in the predicting process of the next target words  It helps to improve NMTs significantly  Nevertheless, since the attention mechanism is specific to a particular source sentence and the considering target word, it is also specific to particular language pairs 
Some recent work has focused on extending the NMT framework to multilingual scenarios  By training such network using parallel corpora in number of different languages, NMT could benefit from additional information embedded in a common semantic space across languages  Basically, the proposed NMT are required to employ multiple encoders or multiple decoders to deal with multilinguality  Furthermore, in order to avoid the tight dependency of the attention mechanism to specific language pairs, they also need to modify their architecture to combine either the encoders or the attention layers  These modifications are specific to the purpose of the tasks as well  Thus, those multilingual NMTs are more complicated, much more free parameters to learn and more difficult to perform standard trainings compared to the original NMT 
In this paper, we introduce a unified approach to seamlessly extend the original NMT to multilingual settings  Our approach allows us to integrate any language in any side of the encoder-decoder architecture with only one encoder and one decoder for all the languages involved  Moreover, it is not necessary to do any network modification to enable attention mechanism in our NMT systems  We then apply our proprosed framework in two demanding scenarios: under-resourced translation and zero-resourced translation  The results show that bringing multilinguality to NMT helps to improve individual translations  With some insightful analyses of the results, we set our goal toward a fully multilingual NMT framework 
The paper starts with a detailed introduction to attention-based NMT  In Section SECREF3 , related work about multi-task NMT is reviewed  Section SECREF5 describes our proposed approach and thorough comparisons to the related work  It is followed by a section of evaluating our systems in two aforementioned scenarios, in which different strategies have been employed under a unified approach (Section SECREF4 )  Finally, the paper ends with conclusion and future work 
This work is licenced under a Creative Commons Attribution 4 0 International License  License details: http://creativecommons org/licenses/by/4 0/
Neural Machine Translation: Background
An NMT system consists of an encoder which automatically learns the characteristics of a source sentence into fix-length context vectors and a decoder that recursively combines the produced context vectors with the previous target word to generate the most probable word from a target vocabulary 
More specifically, a bidirectional recurrent encoder reads every words INLINEFORM0 of a source sentence INLINEFORM1 and encodes a representation INLINEFORM2 of the sentence into a fixed-length vector INLINEFORM3 concatinated from those of the forward and backward directions: INLINEFORM4
Here INLINEFORM0 is the one-hot vector of the word INLINEFORM1 and INLINEFORM2 is the word embedding matrix which is shared across the source words  INLINEFORM3 is the recurrent unit computing the current hidden state of the encoder based on the previous hidden state  INLINEFORM4 is then called an annotation vector, which encodes the source sentence up to the time INLINEFORM5 from both forward and backward directions  Recurrent units in NMT can be a simple recurrent neural network unit (RNN), a Long Short-Term Memory unit (LSTM) BIBREF3 or a Gated Recurrent Unit (GRU) BIBREF4
Similar to the encoder, the recurrent decoder generates one target word INLINEFORM0 to form a translated target sentence INLINEFORM1 in the end  At the time INLINEFORM2 , it takes the previous hidden state of the decoder INLINEFORM3 , the previous embedded word representation INLINEFORM4 and a time-specific context vector INLINEFORM5 as inputs to calculate the current hidden state INLINEFORM6 : INLINEFORM7
Again, INLINEFORM0 is the recurrent activation function of the decoder and INLINEFORM1 is the shared word embedding matrix of the target sentences  The context vector INLINEFORM2 is calculated based on the annotation vectors from the encoder  Before feeding the annotation vectors into the decoder, an attention mechanism is set up in between, in order to choose which annotation vectors should contribute to the predicting decision of the next target word  Intuitively, a relevance between the previous target word and the annotation vectors can be used to form some attention scenario  There exists several ways to calculate the relevance as shown in BIBREF5 , but what we describe here follows the proposed method of BIBREF2 DISPLAYFORM0
In BIBREF2 , this attention mechanism, originally called alignment model, has been employed as a simple feedforward network with the first layer is a learnable layer via INLINEFORM0 , INLINEFORM1 and INLINEFORM2   The relevance scores INLINEFORM3 are then normalized into attention weights INLINEFORM4 and the context vector INLINEFORM5 is calculated as the weighted sum of all annotation vectors INLINEFORM6   Depending on how much attention the target word at time INLINEFORM7 put on the source states INLINEFORM8 , a soft alignment is learned  By being employed this way, word alignment is not a latent variable but a parametrized function, making the alignment model differentiable  Thus, it could be trained together with the whole architecture using backpropagation 
One of the most severe problems of NMT is handling of the rare words, which are not in the short lists of the vocabularies, i e  out-of-vocabulary (OOV) words, or do not appear in the training set at all  In BIBREF6 , the rare target words are copied from their aligned source words after the translation  This heuristic works well with OOV words and named entities but unable to translate unseen words  In BIBREF7 , their proposed NMT models have been shown to not only be effective on reducing vocabulary sizes but also have the ability to generate unseen words  This is achieved by segmenting the rare words into subword units and translating them  The state-of-the-art translation systems essentially employ subword NMT BIBREF7  
Universal Encoder and Decoder for Multilingual Neural Machine Translation
While the majority of previous research has focused on improving the performance of NMT on individual language pairs with individual NMT systems, recent work has started investigating potential ways to conduct the translation involved in multiple languages using a single NMT system  The possible reason explaining these efforts lies on the unique architecture of NMT  Unlike SMT, NMT consists of separated neural networks for the source and target sides, or the encoder and decoder, respectively  This allows these components to map a sentence in any language to a representation in an embedding space which is believed to share common semantics among the source languages involved  From that shared space, the decoder, with some implicit or explicit relevant constraints, could transform the representation into a concrete sentence in any desired language  In this section, we review some related work on this matter  We then describe a unified approach toward an universal attention-based NMT scheme  Our approach does not require any architecture modification and it can be trained to learn a minimal number of parameters compared to the other work 
Related Work
By extending the solution of sequence-to-sequence modeling using encoder-decoder architectures to multi-task learning, Luong2016 managed to achieve better performance on some INLINEFORM0 tasks such as translation, parsing and image captioning compared to individual tasks  Specifically in translation, the work utilizes multiple encoders to translate from multiple languages, and multiple decoders to translate to multiple languages  In this view of multilingual translation, each language in source or target side is modeled by one encoder or decoder, depending on the side of the translation  Due to the natural diversity between two tasks in that multi-task learning scenario, e g  translation and parsing, it could not feature the attention mechanism although it has proven its effectiveness in NMT  There exists two directions which proposed for multilingual translation scenarios where they leverage the attention mechanism  The first one is indicated in the work from BIBREF8 , where it introduce an one-to-many multilingual NMT system to translates from one source language into multiple target languages  Having one source language, the attention mechanism is then handed over to the corresponding decoder  The objective function is changed to adapt to multilingual settings  In testing time, the parameters specific to a desired language pair are used to perform the translation

ground_truth: ancient Chinese history records in several dynasties (about 1000BC-200BC) and articles written by celebrities of that era

Ancient Chinese history records in several dynasties and articles written by celebrities during 1000BC-200BC collected from the internet 

prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. What models are used for painting embedding and what for language style transfer?
Context 2:  Incorporating this extra information improves the translation task  The large number of shared word types between the source and target sentences indicates that sharing the representation between them is beneficial 
Methods ::: Shakespearizing Poetic Captions ::: Seq2Seq with Attention
We use a sequence-to-sequence model which consists of a single layer unidrectional LSTM encoder and a single layer LSTM decoder and pre-trained retrofitted word embeddings shared between source and target sentences  We experimented with two different types of attention: global attention BIBREF9, in which the model makes use of the output from the encoder and decoder for the current time step only, and Bahdanau attention BIBREF10, where computing attention requires the output of the decoder from the prior time step  We found that global attention performs better in practice for our task of text style transfer 
Methods ::: Shakespearizing Poetic Captions ::: Seq2Seq with a Pointer Network
Since a pair of corresponding Shakespeare and modern English sentences have significant vocabulary overlap we extend the sequence-to-sequence model mentioned above using pointer networks BIBREF11 that provide location based attention and have been used to enable copying of tokens directly from the input  Moreover, there are lot of proper nouns and rare words which might not be predicted by a vanilla sequence to sequence model 
Methods ::: Shakespearizing Poetic Captions ::: Prediction
For both seq2seq models, we use the attention matrices returned at each decoder time step during inference, to compute the next word in the translated sequence if the decoder output at the current time step is the UNK token  We replace the UNKs in the target output with the highest aligned, maximum attention, source word  The seq2seq model with global attention gives the best results with an average target BLEU score of 29 65 on the style transfer dataset, compared with an average target BLEU score of 26 97 using the seq2seq model with pointer networks 
Results
We perform a qualitative analysis of the Shakespearean prose generated for the input paintings  We conducted a survey, in which we presented famous paintings including those shown in Figures FIGREF1 and FIGREF10 and the corresponding Shakespearean prose generated by the model, and asked 32 students to rate them on the basis of content, creativity and similarity to Shakespearean style on a Likert scale of 1-5  Figure FIGREF12 shows the result of our human evaluation 
The average content score across the paintings is 3 7 which demonstrates that the prose generated is relevant to the painting  The average creativity score is 3 9 which demonstrates that the model captures more than basic objects in the painting successfully using poetic clues in the scene  The average style score is 3 9 which demonstrates that the prose generated is perceived to be in the style of Shakespeare 
We also perform a quantitative analysis of style transfer by generating BLEU scores for the model output using the style transfer dataset  The variation of the BLEU scores with the source sentence lengths is shown in Figure FIGREF11  As expected, the BLEU scores decrease with increase in source sentence lengths 
Results ::: Implementation
All models were trained on Google Colab with a single GPU using Python 3 6 and Tensorflow 2 0  The number of hidden units for the encoder and decoder is 1,576 and 256 for seq2seq with global attention and seq2seq with pointer networks respectively  Adam optimizer was used with the default learning rate of 0 001  The model was trained for 25 epochs  We use pre-trained retrofitted word embeddings of dimension 192 
Results ::: Limitations
Since we do not have an end-to-end dataset, the generated English poem may not work well with Shakespeare style transfer as shown in Figure FIGREF12 for "Starry Night" with a low average content score  This happens when the style transfer dataset does not have similar words in the training set of sentences  A solution would be to expand the style transfer dataset, for a better representation of the poem data 
Results ::: Conclusions and Future Work
In conclusion, combining two pipelines with an intermediate representation works well in practice  We observe that a CNN-RNN based image-to-poem net combined with a seq2seq model with parallel text corpus for text style transfer synthesizes Shakespeare-style prose for a given painting  For the seq2seq model used, we observe that it performs better in practice using global attention as compared with local attention  We make our models and code publicly available BIBREF12  In future work we would like to experiment with GANs in the absence of non-parallel datasets, so that we can use varied styles for text style transfer  We would also like to experiment with cross aligned auto-encoders, which form a latent content representation, to efficiently separate style and content 
Context 3: 
Firat2016 proposed another approach which genuinely delivers attention-based NMT to multilingual translation  As in BIBREF9 , their approach utilizes one encoder per source language and one decoder per target language for many-to-many translation tasks  Instead of a quadratic number of independent attention layers, however, one single attention mechanism is integrated into their NMT, performing an affine transformation between the hidden layer of INLINEFORM0 source languages and that one of INLINEFORM1 target languages  It is required to change their architecture to accomodate such a complicated shared attention mechanism 
In a separate effort to achieve multilingual NMT, the work of Zoph2016 leverages available parallel data from other language pairs to help reducing possible ambiguities in the translation process into a single target language  They employed the multi-source attention-based NMT in a way that only one attention mechanism is required despite having multiple encoders  To achieve this, the outputs of the encoders were combined before feeding to the attention layer  They implemented two types of encoder combination; One is adding a non-linear layer on the concatenation of the encoders' hidden states  The other is using a variant of LSTM taking the respective gate values from the individual LSTM units of the encoders  As a result, the combined hidden states contain information from both encoders , thus encode the common semantic of the two source languages 
Universal Encoder and Decoder
Inspired by the multi-source NMT as additional parallel data in several languages are expected to benefit single translations, we aim to develop a NMT-based approach toward an universal framework to perform multilingual translation  Our solution features two treatments: 1) Coding the words in different languages as different words in the language-mixed vocabularies and 2) Forcing the NMT to translating a representation of source sentences into the sentences in a desired target language 
Language-specific Coding  When the encoder of a NMT system considers words across languages as different words, with a well-chosen architecture, it is expected to be able to learn a good representation of the source words in an embedding space in which words carrying similar meaning would have a closer distance to each others than those are semantically different  This should hold true when the words have the same or similar surface form, such as (@de@Obama; @en@Obama) or (@de@Projektion; @en@projection)  This should also hold true when the words have the same or similar meaning across languages, such as (@en@car; @en@automobile) or (@de@Flussufer; @en@bank)  Our encoder then acts similarly to the one of multi-source approach BIBREF10 , collecting additional information from other sources for better translations, but with a much simpler embedding function  Unlike them, we need only one encoder, so we could reduce the number of parameters to learn  Furthermore, we neither need to change the network architecture nor depend on which recurrent unit (GRU, LSTM or simple RNN) is currently using in the encoder 
We could apply the same trick to the target sentences and thus enable many-to-many translation capability of our NMT system  Similar to the multi-target translation BIBREF8 , we exploit further the correlation in semantics of those target sentences across different languages  The main difference between our approach and the work of BIBREF8 is that we need only one decoder for all target languages  Given one encoder for multiple source languages and one decoder for multiple target languages, it is trivial to incorporate the attention mechanism as in the case of a regular NMT for single language translation  In training, the attention layers were directed to learn relevant alignments between words in specific language pair and forward the produced context vector to the decoder  Now we rely totally on the network to learn good alignments between source and target sides  In fact, giving more information, our system are able to form nice alignments 
In comparison to other research that could perform complete multi-task learning, e g  the work from BIBREF9 or the approach proposed by BIBREF11 , our method is able to accommodate the attention layers seemlessly and easily  It also draws a clear distinction from those works in term of the complexity of the whole network: considerably less parameters to learn, thus reduces overfitting, with a conventional attention mechanism and a standard training procedure 
Target Forcing  While language-specific coding allows us to implement a multilingual attention-based NMT, there are two issues we have to consider before training the network  The first is that the number of rare words would increase in proportion with the number of languages involved  This might be solved by applying a rare word treatment method with appropriate awareness of the vocabularies' size  The second one is more problematic: Ambiguity level in the translation process definitely increases due to the additional introduction of words having the same or similar meaning across languages at both source and target sides  We deal with the problem by explicitly forcing the attention and translation to the direction that we prefer, expecting the information would limit the ambiguity to the scope of one language instead of all target languages  We realize this idea by adding at the beginning and at the end of every source sentences a special symbol indicating the language they would be translated into  For example, in a multilingual NMT, when a source sentence is German and the target language is English, the original sentence (already language-specific coded) is:
@de@darum @de@geht @de@es @de@in @de@meinem @de@Vortrag
Now when we force it to be translated into English, the target-forced sentence becomes:
<E> @de@darum @de@geht @de@es @de@in @de@meinem @de@Vortrag <E>
Due to the nature of recurrent units used in the encoder and decoder, in training, those starting symbols encourage the network learning the translation of following target words in a particular language pair  In testing time, information of the target language we provided help to limit the translated candidates, hence forming the translation in the desired language 
Figure FIGREF8 illustrates the essence of our approach  With two steps in the preprocessing phase, namely language-specific coding and target forcing, we are able to employ multilingual attention-based NMT without any special treatment in training such a standard architecture  Our encoder and attention-enable decoder can be seen as a shared encoder and decoder across languages, or an universal encoder and decoder  The flexibitily of our approach allow us to integrate any language into source or target side  As we will see in Section SECREF4 , it has proven to be extremely helpful not only in low-resourced scenarios but also in translation of well-resourced language pairs as it provides a novel way to make use of large monolingual corpora in NMT 
Evaluation
In this section, we describe the evaluation of our proposed approach in comparisons with the strong baselines using NMT in two scenarios: the translation of an under-resource language pair and the translation of a language pair that does not exist any paralled data at all 
Experimental Settings
Training Data  We choose WIT3's TED corpus BIBREF12 as the basis of our experiments since it might be the only high-quality parallel data of many low-resourced language pairs  TED is also multilingual in a sense that it includes numbers of talks which are commonly translated into many languages  In addition, we use a much larger corpus provided freely by WMT organizers when we evaluate the impact of our approach in a real machine translation campaign  It includes the paralled corpus extracted from the digital corpus of European Parliament (EPPS), the News Commentary (NC) and the web-crawled parallel data (CommonCrawl)  While the number of sentences in popular TED corpora varies from 13 thousands to 17 thousands, the total number of sentences in those larger corpus is approximately 3 million sentences 
Neural Machine Translation Setup  All experiments have been conducted using NMT framework Nematus, Following the work of Sennrich2016a, subword segmentation is handled in the prepocessing phase using Byte-Pair Encoding (BPE)  Excepts stated clearly in some experiments, we set the number of BPE merging operations at 39500 on the joint of source and target data  When training all NMT systems, we take out the sentence pairs exceeding 50-word length and shuffle them inside every minibatch  Our short-list vocabularies contain 40,000 most frequent words while the others are considered as rare words and applied the subword translation  We use an 1024-cell GRU layer and 1000-dimensional embeddings with dropout at every layer with the probability of 0 2 in the embedding and hidden layers and 0 1 in the input and ourput layers  We trained our systems using gradient descent optimization with Adadelta BIBREF13 on minibatches of size 80 and the gradient is rescaled whenever its norm exceed 1 0  All the trainings last approximately seven days if the early-stopping condition could not be reached  At a certain time, an external evaluation script on BLEU BIBREF14 is conducted on a development set to decide the early-stopping condition  This evaluation script has also being used to choose the model archiving the best BLEU on the development set instead of the maximal loglikelihood between the translations and target sentences while training  In translation, the framework produces INLINEFORM0 -best candidates and we then use a beam search with the beam size of 12 to get the best translation 
Under-resourced Translation
First, we consider the translation for an under-resourced pair of languages  Here a small portion of the large parallel corpus for English-German is used as a simulation for the scenario where we do not have much parallel data: Translating texts in English to German  We perform language-specific coding in both source and target sides  By accommodating the German monolingual data as an additional input (German INLINEFORM0 German), which we called the mix-source approach, we could enrich the training data in a simple, natural way  Given this under-resourced situation, it could help our NMT obtain a better representation of the source side, hence, able to learn the translation relationship better  Including monolingual data in this way might also improve the translation of some rare word types such as named entities  Furthermore, as the ultimate goal of our work, we would like to investigate the advantages of multilinguality in NMT

ground_truth: generating a poem from images we use an existing actor-critic architecture, various types of sequence to sequence models

prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. Do they test their framework performance on commonly used language pairs, such as English-to-German?
Context 3:  We incorporate a similar portion of French-German parallel corpus into the English-German one  As discussed in Section SECREF5 , it is expected to help reducing the ambiguity in translation between one language pair since it utilizes the semantic context provided by the other source language  We name this mix-multi-source 
Table TABREF16 summarizes the performance of our systems measured in BLEU on two test sets, tst2013 and tst2014  Compared to the baseline NMT system which is solely trained on TED English-German data, our mix-source system achieves a considerable improvement of 2 6 BLEU points on tst2013 and 2 1 BLEU points on and tst2014   Adding French data to the source side and their corresponding German data to the target side in our mix-multi-source system also help to gain 2 2 and 1 6 BLEU points more on tst2013 tst2014, respectively  We observe a better improvement from our mix-source system compared to our mix-multi-source system  We speculate the reason that the mix-source encoder utilize the same information shared in two languages while the mix-multi-source receives and processes similar information in the other language but not necessarily the same  We might validate this hypothesis by comparing two systems trained on a common English-German-French corpus of TED  We put it in our future work's plan 
As we expected Figure FIGREF19 shows how different words in different languages can be close in the shared space after being learned to translate into a common language  We extract the word embeddings from the encoder of the mix-multi-source (En,Fr INLINEFORM0 De,De) after training, remove the language-specific codes (@en@ and @fr@)and project the word vectors to the 2D space using t-SNE BIBREF15  
Using large monolingual data in NMT 
A standard NMT system employs parallel data only  While good parallel corpora are limited in number, getting monolingual data of an arbitrary language is trivial  To make use of German monolingual corpus in an English INLINEFORM0 German NMT system, sennrich2016b built a separate German INLINEFORM1 English NMT using the same parallel corpus, then they used that system to translate the German monolingual corpus back to English, forming a synthesis parallel data  gulcehre2015 trained another RNN-based language model to score the monolingual corpus and integrate it to the NMT system through shallow or deep fusion  Both methods requires to train separate systems with possibly different hyperparameters for each  Conversely, by applying mix-source method to the big monolingual data, we need to train only one network  We mix the TED parallel corpus and the substantial monolingual corpus (EPPS+NC+CommonCrawl) and train a mix-source NMT system from those data 
The first result is not encouraging when its performance is even worse than the baseline NMT which is trained on the small parallel data only  Not using the same information in the source side, as we discussed in case of mix-multi-source strategy, could explain the degrading in performance of such a system  But we believe that the magnitude and unbalancing of the corpus are the main reasons  The data contains nearly four millions sentences but only around twenty thousands of them (0 5%) are the genuine parallel data  As a quick attempt, after we get the model with that big data, we continue training on the real parallel corpus for some more epochs  When this adaptation is applied, our system brings an improvement of +1 52 BLEU on tst2013 and +1 06 BLEU on tst2014 (Table TABREF21 ) 
Zero-resourced Translation
Among low-resourced scenarios, zero-resourced translation task stands in an extreme level  A zero-resourced translation task is one of the most difficult situation when there is no parallel data between the translating language pair  To the best of our knowledge, there have been yet existed a published work about using NMT for zero-resourced translation tasks up to now  In this section, we extend our strategies using the proposed multilingual NMT approach as first attempts to this extreme situation 
We employ language-specific coding and target forcing in a strategy called bridge  Unlike the strategies used in under-resourced translation task, bridge is an entire many-to-many multilingual NMT  Simulating a zero-resourced German INLINEFORM0 French translation task given the available German-English and English-French parallel corpora, after applying language-specific coding and target forcing for each corpus, we mix those data with an English-English data as a “bridge” creating some connection between German and French  We also propose a variant of this strategy that we incorporate French-French data  And we call it universal 
We evaluate bridge and universal systems on two German INLINEFORM0 French test sets  They are compared to a direct system, which is an NMT trained on German INLINEFORM1 French data, and to a pivot system, which essentially consists of two separate NMTs trained to translate from German to English and English to French  The direct system should not exist in a real zero-resourced situation  We refer it as the perfect system for comparison purpose only  In case of the pivot system, to generate a translated text in French from a German sentence, we first translate it to English, then the output sentence is fed to the English INLINEFORM2 German NMT system to obtain the French translation  Since there are more than two languages involved in those systems, we increase the number of BPE merging operations proportionally in order to reduce the number of rare words in such systems  We do not expect our proposed systems to perform well with this primitive way of building direct translating connections since this is essentially a difficult task  We report the performance of those systems in Table TABREF23  
Unsupprisingly, both bridge and universal systems perform worse than the pivot one  We consider two possible reasons:
Our target forcing mechanism is moderately primitive  Since the process is applied after language-specific coding, the target forcing symbol is the same for all source sentences in every languages  Thus, the forcing strength might not be enough to guide the decision of the next words  Once the very first word is translated into a word in wrong language, the following words tend to be translated into that wrong language again  Table TABREF24 shows some statistics of the translated words and sentences in wrong language 
Balancing of the training corpus  Although it is not severe as in the case of mix-source system for large monolingual data, the limited number of sentences in target language can affect the training  The difference of 1 07 BLEU points between bridge and universal might explain this assumption as we added more target data (French) in universal strategy, thus reducing the unbalance in training 
Those issues would be addressed in our following future work toward the multilingual attention-based NMT 
Conclusion and Future Work
In this paper, we present our first attempts in building a multilingual Neural Machine Translation framework  By treating words in different languages as different words and force the attention and translation to the direction of desired target language, we are able to employ attention-enable NMT toward a multilingual translation system  Our proposed approach alleviates the need of complicated architecture re-designing when accommodating attention mechanism  In addition, the number of free parameters to learn in our network does not go beyond that magnitute of a single NMT system  With its universality, our approach has shown its effectiveness in an under-resourced translation task with considerable improvements  In addition, the approach has achieved interesting and promising results when applied in the translation task that there is no direct parallel corpus between source and target languages 
Nevertheless, there are issues that we can continue working on to address in future work  A more balancing data would be helpful for this framework  The mechanism of forcing the NMT system to the right target language could be improved  We could conduct more detailed analyses of the various strategies under the framework to show its universarity 

ground_truth: Yes

Yes