prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. what was the baseline?
Context 1: Introduction
Neural machine translation (NMT) BIBREF0 , BIBREF1 , BIBREF2 has enabled end-to-end training of a translation system without needing to deal with word alignments, translation rules, and complicated decoding algorithms, which are the characteristics of phrase-based statistical machine translation (PBSMT) BIBREF3   Although NMT can be significantly better than PBSMT in resource-rich scenarios, PBSMT performs better in low-resource scenarios BIBREF4   Only by exploiting cross-lingual transfer learning techniques BIBREF5 , BIBREF6 , BIBREF7 , can the NMT performance approach PBSMT performance in low-resource scenarios 
However, such methods usually require an NMT model trained on a resource-rich language pair like French INLINEFORM0 English (parent), which is to be fine-tuned for a low-resource language pair like Uzbek INLINEFORM1 English (child)  On the other hand, multilingual approaches BIBREF8 propose to train a single model to translate multiple language pairs  However, these approaches are effective only when the parent target or source language is relatively resource-rich like English (En)  Furthermore, the parents and children models should be trained on similar domains; otherwise, one has to take into account an additional problem of domain adaptation BIBREF9  
In this paper, we work on a linguistically distant and thus challenging language pair Japanese INLINEFORM0 Russian (Ja INLINEFORM1 Ru) which has only 12k lines of news domain parallel corpus and hence is extremely resource-poor  Furthermore, the amount of indirect in-domain parallel corpora, i e , Ja INLINEFORM2 En and Ru INLINEFORM3 En, are also small  As we demonstrate in Section SECREF4 , this severely limits the performance of prominent low-resource techniques, such as multilingual modeling, back-translation, and pivot-based PBSMT  To remedy this, we propose a novel multistage fine-tuning method for NMT that combines multilingual modeling BIBREF8 and domain adaptation BIBREF9  
We have addressed two important research questions (RQs) in the context of extremely low-resource machine translation (MT) and our explorations have derived rational contributions (CTs) as follows:
To the best of our knowledge, we are the first to perform such an extensive evaluation of extremely low-resource MT problem and propose a novel multilingual multistage fine-tuning approach involving multilingual modeling and domain adaptation to address it 
Our Japanese–Russian Setting
In this paper, we deal with Ja INLINEFORM0 Ru news translation  This language pair is very challenging because the languages involved have completely different writing system, phonology, morphology, grammar, and syntax  Among various domains, we experimented with translations in the news domain, considering the importance of sharing news between different language speakers  Moreover, news domain is one of the most challenging tasks, due to large presence of out-of-vocabulary (OOV) tokens and long sentences  To establish and evaluate existing methods, we also involved English as the third language  As direct parallel corpora are scarce, involving a language such as English for pivoting is quite common BIBREF10  
There has been no clean held-out parallel data for Ja INLINEFORM0 Ru and Ja INLINEFORM1 En news translation  Therefore, we manually compiled development and test sets using News Commentary data as a source  Since the given Ja INLINEFORM2 Ru and Ja INLINEFORM3 En data share many lines in the Japanese side, we first compiled tri-text data  Then, from each line, corresponding parts across languages were manually identified, and unaligned parts were split off into a new line  Note that we have never merged two or more lines  As a result, we obtained 1,654 lines of data comprising trilingual, bilingual, and monolingual segments (mainly sentences) as summarized in Table TABREF8   Finally, for the sake of comparability, we randomly chose 600 trilingual sentences to create a test set, and concatenated the rest of them and bilingual sentences to form development sets 
Our manually aligned development and test sets are publicly available 
Related Work
koehn-knowles:2017:NMT showed that NMT is unable to handle low-resource language pairs as opposed to PBSMT  Transfer learning approaches BIBREF5 , BIBREF6 , BIBREF7 work well when a large helping parallel corpus is available  This restricts one of the source or the target languages to be English which, in our case, is not possible  Approaches involving bi-directional NMT modeling is shown to drastically improve low-resource translation BIBREF11   However, like most other, this work focuses on translation from and into English 
Remaining options include (a) unsupervised MT BIBREF12 , BIBREF13 , BIBREF14 , (b) parallel sentence mining from non-parallel or comparable corpora BIBREF15 , BIBREF16 , (c) generating pseudo-parallel data BIBREF17 , and (d) MT based on pivot languages BIBREF10   The linguistic distance between Japanese and Russian makes it extremely difficult to learn bilingual knowledge, such as bilingual lexicons and bilingual word embeddings  Unsupervised MT is thus not promising yet, due to its heavy reliance on accurate bilingual word embeddings  Neither does parallel sentence mining, due to the difficulty of obtaining accurate bilingual lexicons  Pseudo-parallel data can be used to augment existing parallel corpora for training, and previous work has reported that such data generated by so-called back-translation can substantially improve the quality of NMT  However, this approach requires base MT systems that can generate somewhat accurate translations  It is thus infeasible in our scenario, because we can obtain only a weak system which is the consequence of an extremely low-resource situation  MT based on pivot languages requires large in-domain parallel corpora involving the pivot languages  This technique is thus infeasible, because the in-domain parallel corpora for Ja INLINEFORM0 En and Ru INLINEFORM1 En pairs are also extremely limited, whereas there are large parallel corpora in other domains  Section SECREF4 empirically confirms the limit of these existing approaches 
Fortunately, there are two useful transfer learning solutions using NMT: (e) multilingual modeling to incorporate multiple language pairs into a single model BIBREF8 and (f) domain adaptation to incorporate out-of-domain data BIBREF9   In this paper, we explore a novel method involving step-wise fine-tuning to combine these two methods  By improving the translation quality in this way, we can also increase the likelihood of pseudo-parallel data being useful to further improve translation quality 
Limit of Using only In-domain Data
This section answers our first research question, [RQ1], about the translation quality that we can achieve using existing methods and in-domain parallel and monolingual data  We then use the strongest model to conduct experiments on generating and utilizing back-translated pseudo-parallel data for augmenting NMT  Our intention is to empirically identify the most effective practices as well as recognize the limitations of relying only on in-domain parallel corpora 
Data
To train MT systems among the three languages, i e , Japanese, Russian, and English, we used all the parallel data provided by Global Voices, more specifically those available at OPUS  Table TABREF9 summarizes the size of train/development/test splits used in our experiments  The number of parallel sentences for Ja INLINEFORM0 Ru is 12k, for Ja INLINEFORM1 En is 47k, and for Ru INLINEFORM2 En is 82k  Note that the three corpora are not mutually exclusive: 9k out of 12k sentences in the Ja INLINEFORM3 Ru corpus were also included in the other two parallel corpora, associated with identical English translations  This puts a limit on the positive impact that the helping corpora can have on the translation quality 
Even when one focuses on low-resource language pairs, we often have access to larger quantities of in-domain monolingual data of each language  Such monolingual data are useful to improve quality of MT, for example, as the source of pseudo-parallel data for augmenting training data for NMT BIBREF17 and as the training data for large and smoothed language models for PBSMT BIBREF4   Table TABREF13 summarizes the statistics on our monolingual corpora for several domains including the news domain  Note that we removed from the Global Voices monolingual corpora those sentences that are already present in the parallel corpus 
https://dumps wikimedia org/backup-index html (20180501) http://www statmt org/wmt18/translation-task html https://www yomiuri co jp/database/glossary/ http://www cs jhu edu/~kevinduh/a/multitarget-tedtalks/ http://opus nlpl eu/Tatoeba-v2 php
We tokenized English and Russian sentences using tokenizer perl of Moses BIBREF3   To tokenize Japanese sentences, we used MeCab with the IPA dictionary  After tokenization, we eliminated duplicated sentence pairs and sentences with more than 100 tokens for all the languages 
MT Methods Examined
We began with evaluating standard MT paradigms, i e , PBSMT BIBREF3 and NMT BIBREF1   As for PBSMT, we also examined two advanced methods: pivot-based translation relying on a helping language BIBREF10 and induction of phrase tables from monolingual data BIBREF14  
As for NMT, we compared two types of encoder-decoder architectures: attentional RNN-based model (RNMT) BIBREF2 and the Transformer model BIBREF18   In addition to standard uni-directional modeling, to cope with the low-resource problem, we examined two multi-directional models: bi-directional model BIBREF11 and multi-to-multi (M2M) model BIBREF8  
After identifying the best model, we also examined the usefulness of a data augmentation method based on back-translation BIBREF17  
First, we built a PBSMT system for each of the six translation directions  We obtained phrase tables from parallel corpus using SyMGIZA++ with the grow-diag-final heuristics for word alignment, and Moses for phrase pair extraction  Then, we trained a bi-directional MSD (monotone, swap, and discontinuous) lexicalized reordering model  We also trained three 5-gram language models, using KenLM on the following monolingual data: (1) the target side of the parallel data, (2) the concatenation of (1) and the monolingual data from Global Voices, and (3) the concatenation of (1) and all monolingual data in the news domain in Table TABREF13  
Subsequently, using English as the pivot language, we examined the following three types of pivot-based PBSMT systems BIBREF10 , BIBREF19 for each of Ja INLINEFORM0 Ru and Ru INLINEFORM1 Ja 
2-step decoding using the source-to-English and English-to-target systems 
Obtain a new phrase table from synthetic parallel data generated by translating English side of the target–English training parallel data to the source language with the English-to-source system 
Compile a new phrase table combining those for the source-to-English and English-to-target systems 
Among these three, triangulation is the most computationally expensive method  Although we had filtered the component phrase tables using the statistical significance pruning method BIBREF20 , triangulation can generate an enormous number of phrase pairs  To reduce the computational cost during decoding and the negative effects of potentially noisy phrase pairs, we retained for each source phrase INLINEFORM0 only the INLINEFORM1 -best translations INLINEFORM2 according to the forward translation probability INLINEFORM3 calculated from the conditional probabilities in the component models as defined in utiyama:07  For each of the retained phrase pairs, we also calculated the backward translation probability, INLINEFORM4 , and lexical translation probabilities, INLINEFORM5 and INLINEFORM6 , in the same manner as INLINEFORM7  
We also investigated the utility of recent advances in unsupervised MT  Even though we began with a publicly available implementation of unsupervised PBSMT BIBREF13 , it crashed due to unknown reasons  We therefore followed another method described in marie:usmt-unmt  Instead of short INLINEFORM0 -grams BIBREF12 , BIBREF13 , we collected a set of phrases in Japanese and Russian from respective monolingual data using the word2phrase algorithm BIBREF21 , as in marie:usmt-unmt  To reduce the complexity, we used randomly selected 10M monolingual sentences, and 300k most frequent phrases made of words among the 300k most frequent words  For each source phrase INLINEFORM1 , we selected 300-best target phrases INLINEFORM2 according to the translation probability as in D18-1549: INLINEFORM3 where INLINEFORM4 stands for a bilingual embedding of a given phrase, obtained through averaging bilingual embeddings of constituent words learned from the two monolingual data using fastText and vecmap  For each of the retained phrase pair, INLINEFORM5 was computed analogously  We also computed lexical translation probabilities relying on those learned from the given small parallel corpus 
Up to four phrase tables were jointly exploited by the multiple decoding path ability of Moses  Weights for the features were tuned using KB-MIRA BIBREF22 on the development set; we took the best weights after 15 iterations  Two hyper-parameters, namely, INLINEFORM0 for the number of pivot-based phrase pairs per source phrase and INLINEFORM1 for distortion limit, were determined by a grid search on INLINEFORM2 and INLINEFORM3   In contrast, we used predetermined hyper-parameters for phrase table induction from monolingual data, following the convention: 200 for the dimension of word and phrase embeddings and INLINEFORM4  
We used the open-source implementation of the RNMT and the Transformer models in tensor2tensor  A uni-directional model for each of the six translation directions was trained on the corresponding parallel corpus  Bi-directional and M2M models were realized by adding an artificial token that specifies the target language to the beginning of each source sentence and shuffling the entire training data BIBREF8  
Table TABREF22 contains some specific hyper-parameters for our baseline NMT models  The hyper-parameters not mentioned in this table used the default values in tensor2tensor  For M2M systems, we over-sampled Ja INLINEFORM0 Ru and Ja INLINEFORM1 En training data so that their sizes match the largest Ru INLINEFORM2 En data  To reduce the number of unknown words, we used tensor2tensor's internal sub-word segmentation mechanism  Since we work in a low-resource setting, we used shared sub-word vocabularies of size 16k for the uni- and bi-directional models and 32k for the M2M models  The number of training iterations was determined by early-stopping: we evaluated our models on the development set every 1,000 updates, and stopped training if BLEU score for the development set was not improved for 10,000 updates (10 check-points)  Note that the development set was created by concatenating those for the individual translation directions without any over-sampling 
Having trained the models, we averaged the last 10 check-points and decoded the test sets with a beam size of 4 and a length penalty which was tuned by a linear search on the BLEU score for the development set 
Similarly to PBSMT, we also evaluated “Cascade” and “Synthesize” methods with uni-directional NMT models 
Results
We evaluated MT models using case-sensitive and tokenized BLEU BIBREF23 on test sets, using Moses's multi-bleu perl  Statistical significance ( INLINEFORM0 ) on the difference of BLEU scores was tested by Moses's bootstrap-hypothesis-difference-significance pl 
Tables TABREF27 and TABREF31 show BLEU scores of all the models, except the NMT systems augmented with back-translations  Whereas some models achieved reasonable BLEU scores for Ja INLINEFORM0 En and Ru INLINEFORM1 En translation, all the results for Ja INLINEFORM2 Ru, which is our main concern, were abysmal 
Among the NMT models, Transformer models (b INLINEFORM0 ) were proven to be better than RNMT models (a INLINEFORM1 )  RNMT models could not even outperform the uni-directional PBSMT models (c1)  M2M models (a3) and (b3) outperformed their corresponding uni- and bi-directional models in most cases  It is worth noting that in this extremely low-resource scenario, BLEU scores of the M2M RNMT model for the largest language pair, i e , Ru INLINEFORM2 En, were lower than those of the uni- and bi-directional RNMT models as in TACL1081  In contrast, with the M2M Transformer model, Ru INLINEFORM3 En also benefited from multilingualism 
Standard PBSMT models (c1) achieved higher BLEU scores than uni-directional NMT models (a1) and (b1), as reported by koehn-knowles:2017:NMT, whereas they underperform the M2M Transformer NMT model (b3)  As shown in Table TABREF31 , pivot-based PBSMT systems always achieved higher BLEU scores than (c1)  The best model with three phrase tables, labeled “Synthesize / Triangulate / Gold,” brought visible BLEU gains with substantial reduction of OOV tokens (3047 INLINEFORM0 1180 for Ja INLINEFORM1 Ru, 4463 INLINEFORM2 1812 for Ru INLINEFORM3 Ja)  However, further extension with phrase tables induced from monolingual data did not push the limit, despite their high coverage; only 336 and 677 OOV tokens were left for the two translation directions, respectively  This is due to the poor quality of the bilingual word embeddings used to extract the phrase table, as envisaged in Section SECREF3  
None of pivot-based approaches with uni-directional NMT models could even remotely rival the M2M Transformer NMT model (b3) 
Table TABREF46 shows the results of our multistage fine-tuning, where the IDs of each row refer to those described in Section SECREF41   First of all, the final models of our multistage fine-tuning, i e , V and VII, achieved significantly higher BLEU scores than (b3) in Table TABREF27 , a weak baseline without using any monolingual data, and #10 in Table TABREF33 , a strong baseline established with monolingual data 
The performance of the initial model (I) depends on the language pair  For Ja INLINEFORM0 Ru pair, it cannot achieve minimum level of quality since the model has never seen parallel data for this pair  The performance on Ja INLINEFORM1 En pair was much lower than the two baseline models, reflecting the crucial mismatch between training and testing domains  In contrast, Ru INLINEFORM2 En pair benefited the most and achieved surprisingly high BLEU scores  The reason might be due to the proximity of out-of-domain training data and in-domain test data 
The first fine-tuning stage significantly pushed up the translation quality for Ja INLINEFORM0 En and Ru INLINEFORM1 En pairs, in both cases with fine-tuning (II) and mixed fine-tuning (III)  At this stage, both models performed only poorly for Ja INLINEFORM2 Ru pair as they have not yet seen Ja INLINEFORM3 Ru parallel data  Either model had a consistent advantage to the other 
When these models were further fine-tuned only on the in-domain Ja INLINEFORM0 Ru parallel data (IV and VI), we obtained translations of better quality than the two baselines for Ja INLINEFORM1 Ru pair  However, as a result of complete ignorance of Ja INLINEFORM2 En and Ru INLINEFORM3 En pairs, the models only produced translations of poor quality for these language pairs  In contrast, mixed fine-tuning for the second fine-tuning stage (V and VII) resulted in consistently better models than conventional fine-tuning (IV and VI), irrespective of the choice at the first stage, thanks to the gradual shift of parameters realized by in-domain Ja INLINEFORM4 En and Ru INLINEFORM5 En parallel data  Unfortunately, the translation quality for Ja INLINEFORM6 En and Ru INLINEFORM7 En pairs sometimes degraded from II and III  Nevertheless, the BLEU scores still retain the large margin against two baselines 
The last three rows in Table TABREF46 present BLEU scores obtained by the methods with fewer fine-tuning steps  The most naive model I', trained on the balanced mixture of whole five types of corpora from scratch, and the model II', obtained through a single-step conventional fine-tuning of I on all the in-domain data, achieved only BLEU scores consistently worse than VII  In contrast, when we merged our two fine-tuning steps into a single mixed fine-tuning on I, we obtained a model III' which is better for the Ja INLINEFORM0 Ru pair than VII  Nevertheless, they are still comparable to those of VII and the BLEU scores for the other two language pairs are much lower than VII  As such, we conclude that our multistage fine-tuning leads to a more robust in-domain multilingual model 
Augmentation with Back-translation
Given that the M2M Transformer NMT model (b3) achieved best results for most of the translation directions and competitive results for the rest, we further explored it through back-translation 
We examined the utility of pseudo-parallel data for all the six translation directions, unlike the work of lakew2017improving and lakew2018comparison, which concentrate only on the zero-shot language pair, and the work of W18-2710, which compares only uni- or bi-directional models  We investigated whether each translation direction in M2M models will benefit from pseudo-parallel data and if so, what kind of improvement takes place 
First, we selected sentences to be back-translated from in-domain monolingual data (Table TABREF13 ), relying on the score proposed by moore:intelligent via the following procedure 
For each language, train two 4-gram language models, using KenLM: an in-domain one on all the Global Voices data, i e , both parallel and monolingual data, and a general-domain one on the concatenation of Global Voices, IWSLT, and Tatoeba data 
For each language, discard sentences containing OOVs according to the in-domain language model 
For each translation direction, select the INLINEFORM0 -best monolingual sentences in the news domain, according to the difference between cross-entropy scores given by the in-domain and general-domain language models 
Whereas W18-2710 exploited monolingual data much larger than parallel data, we maintained a 1:1 ratio between them BIBREF8 , setting INLINEFORM0 to the number of lines of parallel data of given language pair 
Selected monolingual sentences were then translated using the M2M Transformer NMT model (b3) to compose pseudo-parallel data  Then, the pseudo-parallel data were enlarged by over-sampling as summarized in Table TABREF32   Finally, new NMT models were trained on the concatenation of the original parallel and pseudo-parallel data from scratch in the same manner as the previous NMT models with the same hyper-parameters 
Table TABREF33 shows the BLEU scores achieved by several reasonable combinations of six-way pseudo-parallel data  We observed that the use of all six-way pseudo-parallel data (#10) significantly improved the base model for all the translation directions, except En INLINEFORM0 Ru  A translation direction often benefited when the pseudo-parallel data for that specific direction was used 
Summary
We have evaluated an extensive variation of MT models that rely only on in-domain parallel and monolingual data  However, the resulting BLEU scores for Ja INLINEFORM2 Ru and Ru INLINEFORM3 Ja tasks do not exceed 10 BLEU points, implying the inherent limitation of the in-domain data as well as the difficulty of these translation directions 
Exploiting Large Out-of-Domain Data Involving a Helping Language
The limitation of relying only on in-domain data demonstrated in Section SECREF4 motivates us to explore other types of parallel data  As raised in our second research question, [RQ2], we considered the effective ways to exploit out-of-domain data 
According to language pair and domain, parallel data can be classified into four categories in Table TABREF40   Among all the categories, out-of-domain data for the language pair of interest have been exploited in the domain adaptation scenarios (C INLINEFORM0 A) BIBREF9   However, for Ja INLINEFORM1 Ru, no out-of-domain data is available  To exploit out-of-domain parallel data for Ja INLINEFORM2 En and Ru INLINEFORM3 En pairs instead, we propose a multistage fine-tuning method, which combines two types of transfer learning, i e , domain adaptation for Ja INLINEFORM4 En and Ru INLINEFORM5 En (D INLINEFORM6 B) and multilingual transfer (B INLINEFORM7 A), relying on the M2M model examined in Section SECREF4   We also examined the utility of fine-tuning for iteratively generating and using pseudo-parallel data 
Multistage Fine-tuning
Simply using NMT systems trained on out-of-domain data for in-domain translation is known to perform badly  In order to effectively use large-scale out-of-domain data for our extremely low-resource task, we propose to perform domain adaptation through either (a) conventional fine-tuning, where an NMT system trained on out-of-domain data is fine-tuned only on in-domain data, or (b) mixed fine-tuning BIBREF9 , where pre-trained out-of-domain NMT system is fine-tuned using a mixture of in-domain and out-of-domain data  The same options are available for transferring from Ja INLINEFORM0 En and Ru INLINEFORM1 En to Ja INLINEFORM2 Ru 
We inevitably involve two types of transfer learning, i e , domain adaptation for Ja INLINEFORM0 En and Ru INLINEFORM1 En and multilingual transfer for Ja INLINEFORM2 Ru pair  Among several conceivable options for managing these two problems, we examined the following multistage fine-tuning 
Pre-train a multilingual model only on the Ja INLINEFORM0 En and Ru INLINEFORM1 En out-of-domain parallel data (I), where the vocabulary of the model is determined on the basis of the in-domain parallel data in the same manner as the M2M NMT models examined in Section SECREF4  
Fine-tune the pre-trained model (I) on the in-domain Ja INLINEFORM0 En and Ru INLINEFORM1 En parallel data (fine-tuning, II) or on the mixture of in-domain and out-of-domain Ja INLINEFORM2 En and Ru INLINEFORM3 En parallel data (mixed fine-tuning, III) 
Further fine-tune the models (each of II and III) for Ja INLINEFORM0 Ru on in-domain parallel data for this language pair only (fine-tuning, IV and VI) or on all the in-domain parallel data (mixed fine-tuning, V and VII) 
We chose this way due to the following two reasons  First, we need to take a balance between several different parallel corpora sizes  The other reason is division of labor; we assume that solving each sub-problem one by one should enable gradual shift of parameters 
Data Selection
As an additional large-scale out-of-domain parallel data for Ja INLINEFORM0 En, we used the cleanest 1 5M sentences from the Asian Scientific Paper Excerpt Corpus (ASPEC) BIBREF24   As for Ru INLINEFORM1 En, we used the UN and Yandex corpora released for the WMT 2018 News Translation Task  We retained Ru INLINEFORM2 En sentence pairs that contain at least one OOV token in both sides, according to the in-domain language model trained in Section SECREF34   Table TABREF45 summarizes the statistics on the remaining out-of-domain parallel data 
Further Augmentation with Back-translation
Having obtained a better model, we examined again the utility of back-translation  More precisely, we investigated (a) whether the pseudo-parallel data generated by an improved NMT model leads to a further improvement, and (b) whether one more stage of fine-tuning on the mixture of original parallel and pseudo-parallel data will result in a model better than training a new model from scratch as examined in Section SECREF34  
Given an NMT model, we first generated six-way pseudo-parallel data by translating monolingual data  For the sake of comparability, we used the identical monolingual sentences sampled in Section SECREF34   Then, we further fine-tuned the given model on the mixture of the generated pseudo-parallel data and the original parallel data, following the same over-sampling procedure in Section SECREF34   We repeated these steps five times 
Table TABREF51 shows the results  “new #10” in the second row indicates an M2M Transformer model trained from scratch on the mixture of six-way pseudo-parallel data generated by VII and the original parallel data  It achieved higher BLEU scores than #10 in Table TABREF33 thanks to the pseudo-parallel data of better quality, but underperformed the base NMT model VII  In contrast, our fine-tuned model VIII successfully surpassed VII, and one more iteration (IX) further improved BLEU scores for all translation directions, except Ru INLINEFORM0 En  Although further iterations did not necessarily gain BLEU scores, we came to a much higher plateau compared to the results in Section SECREF4  
Conclusion
In this paper, we challenged the difficult task of Ja INLINEFORM0 Ru news domain translation in an extremely low-resource setting  We empirically confirmed the limited success of well-established solutions when restricted to in-domain data  Then, to incorporate out-of-domain data, we proposed a multilingual multistage fine-tuning approach and observed that it substantially improves Ja INLINEFORM1 Ru translation by over 3 7 BLEU points compared to a strong baseline, as summarized in Table TABREF53   This paper contains an empirical comparison of several existing approaches and hence we hope that our paper can act as a guideline to researchers attempting to tackle extremely low-resource translation 
In the future, we plan to confirm further fine-tuning for each of specific translation directions  We will also explore the way to exploit out-of-domain pseudo-parallel data, better domain-adaptation approaches, and additional challenging language pairs 
Acknowledgments
This work was carried out when Aizhan Imankulova was taking up an internship at NICT, Japan  We would like to thank the reviewers for their insightful comments  A part of this work was conducted under the program “Promotion of Global Communications Plan: Research, Development, and Social Demonstration of Multilingual Speech Translation Technology” of the Ministry of Internal Affairs and Communications (MIC), Japan 
Context 2: Introduction
The recently introduced BERT model BIBREF0 exhibits strong performance on several language understanding benchmarks  To what extent does it capture syntax-sensitive structures?
Recent work examines the extent to which RNN-based models capture syntax-sensitive phenomena that are traditionally taken as evidence for the existence in hierarchical structure  In particular, in BIBREF1 we assess the ability of LSTMs to learn subject-verb agreement patterns in English, and evaluate on naturally occurring wikipedia sentences  BIBREF2 also consider subject-verb agreement, but in a “colorless green ideas” setting in which content words in naturally occurring sentences are replaced with random words with the same part-of-speech and inflection, thus ensuring a focus on syntax rather than on selectional-preferences based cues  BIBREF3 consider a wider range of syntactic phenomena (subject-verb agreement, reflexive anaphora, negative polarity items) using manually constructed stimuli, allowing for greater coverage and control than in the naturally occurring setting 
The BERT model is based on the “Transformer” architecture BIBREF4 , which—in contrast to RNNs—relies purely on attention mechanisms, and does not have an explicit notion of word order beyond marking each word with its absolute-position embedding  This reliance on attention may lead one to expect decreased performance on syntax-sensitive tasks compared to RNN (LSTM) models that do model word order directly, and explicitly track states across the sentence  Indeed, BIBREF5 finds that transformer-based models perform worse than LSTM models on the BIBREF1 agreement prediction dataset  In contrast, BIBREF6 find that self-attention performs on par with LSTM for syntax sensitive dependencies in the context of machine-translation, and performance on syntactic tasks is correlated with the number of attention heads in multi-head attention 
I adapt the evaluation protocol and stimuli of BIBREF1 , BIBREF2 and BIBREF3 to the bidirectional setting required by BERT, and evaluate the pre-trained BERT models (both the Large and the Base models)  Surprisingly (at least to me), the out-of-the-box models (without any task-specific fine-tuning) perform very well on all the syntactic tasks 
Methodology
I use the stimuli provided by BIBREF1 , BIBREF2 , BIBREF3 , but change the experimental protocol to adapt it to the bidirectional nature of the BERT model  This requires discarding some of the stimuli, as described below  Thus, the numbers are not strictly comparable to those reported in previous work 
Previous setups
All three previous work use uni-directional language-model-like models 
BIBREF1 start with existing sentences from wikipedia that contain a present-tense verb  They feed each sentence word by word into an LSTM, stop right before the focus verb, and ask the model to predict a binary plural/singular decision (supervised setup) or compare the probability assigned by a pre-trained language model (LM) to the plural vs singular forms of the verb (LM setup)  The evaluation is then performed on sentences with “agreement attractors” in which at there is at least one noun between the verb and its subject, and all of the nouns between the verb and subject are of the opposite number from the subject 
BIBREF2 also start with existing sentences  However, in order to control for the possibillity of the model learning to rely on “semantic” selectional-preferences cues rather than syntactic ones, they replace each content word with random words from the same part-of-speech and inflection  This results in “coloreless green ideas” nonce sentences  The evaluation is then performed similarly to the LM setup of BIBREF1 : the sentence is fed into a pre-traiend LSTM LM up to the focus verb, and the model is considered correct if the probability assigned to the correct inflection of the original verb form given the prefix is larger than that assigned to the incorrect inflection 
BIBREF3 focus on manually constructed and controlled stimuli, that also emphasizes linguistic structure over selectional preferences  They construct minimal pairs of grammatical and ungrammatical sentences, feed each one in its entirety into a pre-trained LSTM-LM, and compare the perplexity assigned by the model to the grammatical and ungrammatical sentences  The model is “correct” if it assigns the grammatical sentence a higher probability than to the ungrammatical one  Since the minimal pairs for most phenomena differ only in a single word (the focus verb), this scoring is very similar to the one used in the two previous works  However, it does consider the continuation of the sentence after the focus verb, and also allows for assessing phenomena that require change into two or more words (like negative polarity items) 
Adaptation to the BERT model
In contrast to these works, the BERT model is bi-directional: it is trained to predict the identity of masked words based on both the prefix and suffix surrounding these words  I adapt the uni-directional setup by feeding into BERT the complete sentence, while masking out the single focus verb  I then ask BERT for its word predictions for the masked position, and compare the score assigned to the original correct verb to the score assigned to the incorrect one 
For example, for the sentence:
a 2002 systemic review of herbal products found that several herbs , including peppermint and caraway , have anti-dyspeptic effects for non-ulcer dyspepsia with “ encouraging safety profiles ”   (from BIBREF1 )
I feed into BERT:
[CLS] a 2002 systemic review of herbal products found that several herbs , including peppermint and caraway , [MASK] anti-dyspeptic effects for non-ulcer dyspepsia with “ encouraging safety profiles ”   and look for the score assigned to the words have and has at the masked position 
Similarly, for the pair
the game that the guard hates is bad  
the game that the guard hates are bad  
(from BIBREF3 ), I feed into BERT:
[CLS] the game that the guard hates [MASK] bad  
and compare the scores predicted for is and are 
This differs from BIBREF1 and BIBREF2 by considering the entire sentence (excluding the verb) and not just its prefix leading to the verb, and differs from BIBREF3 by conditioning the focus verb on bidirectional context 
I use the PyTorch implementation of BERT, with the pre-trained models supplied by Google  I experiment with the bert-large-uncased and bert-base-uncased models 
The bi-directional setup precludes using using the NPI stimuli of BIBREF3 , in which the minimal pair differs in two words position, which I discard from the evaluation  I also discard the agreement cases involving the verbs is or are in BIBREF1 and in BIBREF2 , because some of them are copular construction, in which strong agreement hints can be found also on the object following the verb  This is not an issue in the manually constructed BIBREF3 stimuli due to the patterns they chose 
Finally, I discard stimuli in which the focus verb or its plural/singular inflection does not appear as a single word in the BERT word-piece-based vocabulary (and hence cannot be predicted by the model)  This include discarding BIBREF3 stimuli involving the words swims or admires, resulting in 23,368 discarded pairs (out of 152,300)  I similarly discard 680 sentences from BIBREF1 where the focus verb or its inflection were one of 108 out-of-vocabulary tokens, and 28 sentence-pairs (8 tokens) from BIBREF2  
The BERT results are not directly comparable to the numbers reported in previous work  Beyond the differences due to bidirectionality and the discarded stimuli, the BERT models are also trained on a different and larger corpus (covering both wikipedia and books) 
Code is available at https://github com/yoavg/bert-syntax 
Results
Tables 1 , 2 and 3 show the results  All cases exhibit high scores—in the vast majority of the cases substantially higher than reported in previous work  As discussed above, the results are not directly comparable to previous work: the BERT models are trained on different (and larger) data, are allowed to access the suffix of the sentence in addition to its prefix, and are evaluated on somewhat different data due to discarding OOV items  Still, taken together, the high performance numbers indicate that the purely attention-based BERT models are likely capable of capturing the same kind of syntactic regularities that LSTM-based models are capable of capturing, at least as well as the LSTM models and probably better 
Another noticeable and interesting trend is that larger is not necessarily better: the BERT-Base model outperforms the BERT-Large model on many of the syntactic conditions 
Discussion
The BERT models perform remarkably well on all the syntactic test cases  I expected the attention-based mechanism to fail on these (compared to the LSTM-based models), and am surprised by these results  The BIBREF2 and BIBREF3 conditions rule out the possibility of overly relying on selectional preference cues or memorizing the wikipedia training data, and suggest real syntactic generalization is taking place  Exploring the extent to which deep purely-attention-based architectures such as BERT are capable of capturing hierarchy-sensitive and syntactic dependencies—as well as the mechanisms by which this is achieved—is a fascinating area for future research 

ground_truth: pivot-based translation relying on a helping language BIBREF10, nduction of phrase tables from monolingual data BIBREF14 , attentional RNN-based model (RNMT) BIBREF2, Transformer model BIBREF18, bi-directional model BIBREF11, multi-to-multi (M2M) model BIBREF8, back-translation BIBREF17

M2M Transformer

prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. Were any of these tasks evaluated in any previous work?
Context 3: Introduction
Knowledge Base Question Answering (KBQA) systems answer questions by obtaining information from KB tuples BIBREF0 , BIBREF1 , BIBREF2 , BIBREF3 , BIBREF4 , BIBREF5   For an input question, these systems typically generate a KB query, which can be executed to retrieve the answers from a KB  Figure 1 illustrates the process used to parse two sample questions in a KBQA system: (a) a single-relation question, which can be answered with a single $<$ head-entity, relation, tail-entity $>$ KB tuple BIBREF6 , BIBREF7 , BIBREF2 ; and (b) a more complex case, where some constraints need to be handled for multiple entities in the question  The KBQA system in the figure performs two key tasks: (1) entity linking, which links $n$ -grams in questions to KB entities, and (2) relation detection, which identifies the KB relation(s) a question refers to 
The main focus of this work is to improve the relation detection subtask and further explore how it can contribute to the KBQA system  Although general relation detection methods are well studied in the NLP community, such studies usually do not take the end task of KBQA into consideration  As a result, there is a significant gap between general relation detection studies and KB-specific relation detection  First, in most general relation detection tasks, the number of target relations is limited, normally smaller than 100  In contrast, in KBQA even a small KB, like Freebase2M BIBREF2 , contains more than 6,000 relation types  Second, relation detection for KBQA often becomes a zero-shot learning task, since some test instances may have unseen relations in the training data  For example, the SimpleQuestions BIBREF2 data set has 14% of the golden test relations not observed in golden training tuples  Third, as shown in Figure 1 (b), for some KBQA tasks like WebQuestions BIBREF0 , we need to predict a chain of relations instead of a single relation  This increases the number of target relation types and the sizes of candidate relation pools, further increasing the difficulty of KB relation detection  Owing to these reasons, KB relation detection is significantly more challenging compared to general relation detection tasks 
This paper improves KB relation detection to cope with the problems mentioned above  First, in order to deal with the unseen relations, we propose to break the relation names into word sequences for question-relation matching  Second, noticing that original relation names can sometimes help to match longer question contexts, we propose to build both relation-level and word-level relation representations  Third, we use deep bidirectional LSTMs (BiLSTMs) to learn different levels of question representations in order to match the different levels of relation information  Finally, we propose a residual learning method for sequence matching, which makes the model training easier and results in more abstract (deeper) question representations, thus improves hierarchical matching 
In order to assess how the proposed improved relation detection could benefit the KBQA end task, we also propose a simple KBQA implementation composed of two-step relation detection  Given an input question and a set of candidate entities retrieved by an entity linker based on the question, our proposed relation detection model plays a key role in the KBQA process: (1) Re-ranking the entity candidates according to whether they connect to high confident relations detected from the raw question text by the relation detection model  This step is important to deal with the ambiguities normally present in entity linking results  (2) Finding the core relation (chains) for each topic entity selection from a much smaller candidate entity set after re-ranking  The above steps are followed by an optional constraint detection step, when the question cannot be answered by single relations (e g , multiple entities in the question)  Finally the highest scored query from the above steps is used to query the KB for answers 
Our main contributions include: (i) An improved relation detection model by hierarchical matching between questions and relations with residual learning; (ii) We demonstrate that the improved relation detector enables our simple KBQA system to achieve state-of-the-art results on both single-relation and multi-relation KBQA tasks 
Background: Different Granularity in KB Relations
Previous research BIBREF4 , BIBREF20 formulates KB relation detection as a sequence matching problem  However, while the questions are natural word sequences, how to represent relations as sequences remains a challenging problem  Here we give an overview of two types of relation sequence representations commonly used in previous work 
(1) Relation Name as a Single Token (relation-level)  In this case, each relation name is treated as a unique token  The problem with this approach is that it suffers from the low relation coverage due to limited amount of training data, thus cannot generalize well to large number of open-domain relations  For example, in Figure 1 , when treating relation names as single tokens, it will be difficult to match the questions to relation names “episodes_written” and “starring_roles” if these names do not appear in training data – their relation embeddings $\mathbf {h}^r$ s will be random vectors thus are not comparable to question embeddings $\mathbf {h}^q$ s 
(2) Relation as Word Sequence (word-level)  In this case, the relation is treated as a sequence of words from the tokenized relation name  It has better generalization, but suffers from the lack of global information from the original relation names  For example in Figure 1 (b), when doing only word-level matching, it is difficult to rank the target relation “starring_roles” higher compared to the incorrect relation “plays_produced”  This is because the incorrect relation contains word “plays”, which is more similar to the question (containing word “play”) in the embedding space  On the other hand, if the target relation co-occurs with questions related to “tv appearance” in training, by treating the whole relation as a token (i e  relation id), we could better learn the correspondence between this token and phrases like “tv show” and “play on” 
The two types of relation representation contain different levels of abstraction  As shown in Table 1 , the word-level focuses more on local information (words and short phrases), and the relation-level focus more on global information (long phrases and skip-grams) but suffer from data sparsity  Since both these levels of granularity have their own pros and cons, we propose a hierarchical matching approach for KB relation detection: for a candidate relation, our approach matches the input question to both word-level and relation-level representations to get the final ranking score  Section "Improved KB Relation Detection" gives the details of our proposed approach 
Improved KB Relation Detection
This section describes our hierarchical sequence matching with residual learning approach for relation detection  In order to match the question to different aspects of a relation (with different abstraction levels), we deal with three problems as follows on learning question/relation representations 
Relation Representations from Different Granularity
We provide our model with both types of relation representation: word-level and relation-level  Therefore, the input relation becomes $\mathbf {r}=\lbrace r^{word}_1,\cdots ,r^{word}_{M_1}\rbrace  \cup \lbrace r^{rel}_1,\cdots ,r^{rel}_{M_2}\rbrace $ , where the first $M_1$ tokens are words (e g  {episode, written}), and the last $M_2$ tokens are relation names, e g , {episode_written} or {starring_roles, series} (when the target is a chain like in Figure 1 (b))  We transform each token above to its word embedding then use two BiLSTMs (with shared parameters) to get their hidden representations $[\mathbf {B}^{word}_{1:M_1}:\mathbf {B}^{rel}_{1:M_2}]$ (each row vector $\mathbf {\beta }_i$ is the concatenation between forward/backward representations at $i$ )  We initialize the relation sequence LSTMs with the final state representations of the word sequence, as a back-off for unseen relations  We apply one max-pooling on these two sets of vectors and get the final relation representation $\mathbf {h}^r$  
Different Abstractions of Questions Representations
From Table 1 , we can see that different parts of a relation could match different contexts of question texts  Usually relation names could match longer phrases in the question and relation words could match short phrases  Yet different words might match phrases of different lengths 
As a result, we hope the question representations could also comprise vectors that summarize various lengths of phrase information (different levels of abstraction), in order to match relation representations of different granularity  We deal with this problem by applying deep BiLSTMs on questions  The first-layer of BiLSTM works on the word embeddings of question words $\mathbf {q}=\lbrace q_1,\cdots ,q_N\rbrace $ and gets hidden representations $\mathbf {\Gamma }^{(1)}_{1:N}=[\mathbf {\gamma }^{(1)}_1;\cdots ;\mathbf {\gamma }^{(1)}_N]$   The second-layer BiLSTM works on $\mathbf {\Gamma }^{(1)}_{1:N}$ to get the second set of hidden representations $\mathbf {\Gamma }^{(2)}_{1:N}$   Since the second BiLSTM starts with the hidden vectors from the first layer, intuitively it could learn more general and abstract information compared to the first layer 
Note that the first(second)-layer of question representations does not necessarily correspond to the word(relation)-level relation representations, instead either layer of question representations could potentially match to either level of relation representations  This raises the difficulty of matching between different levels of relation/question representations; the following section gives our proposal to deal with such problem 
Hierarchical Matching between Relation and Question
Now we have question contexts of different lengths encoded in $\mathbf {\Gamma }^{(1)}_{1:N}$ and $\mathbf {\Gamma }^{(2)}_{1:N}$   Unlike the standard usage of deep BiLSTMs that employs the representations in the final layer for prediction, here we expect that two layers of question representations can be complementary to each other and both should be compared to the relation representation space (Hierarchical Matching)  This is important for our task since each relation token can correspond to phrases of different lengths, mainly because of syntactic variations  For example in Table 1 , the relation word written could be matched to either the same single word in the question or a much longer phrase be the writer of 
We could perform the above hierarchical matching by computing the similarity between each layer of $\mathbf {\Gamma }$ and $\mathbf {h}^r$ separately and doing the (weighted) sum between the two scores  However this does not give significant improvement (see Table 2 )  Our analysis in Section "Relation Detection Results" shows that this naive method suffers from the training difficulty, evidenced by that the converged training loss of this model is much higher than that of a single-layer baseline model  This is mainly because (1) Deep BiLSTMs do not guarantee that the two-levels of question hidden representations are comparable, the training usually falls to local optima where one layer has good matching scores and the other always has weight close to 0  (2) The training of deeper architectures itself is more difficult 
To overcome the above difficulties, we adopt the idea from Residual Networks BIBREF23 for hierarchical matching by adding shortcut connections between two BiLSTM layers  We proposed two ways of such Hierarchical Residual Matching: (1) Connecting each $\mathbf {\gamma }^{(1)}_i$ and $\mathbf {\gamma }^{(2)}_i$ , resulting in a $\mathbf {\gamma }^{^{\prime }}_i=\mathbf {\gamma }^{(1)}_i + \mathbf {\gamma }^{(2)}_i$ for each position $i$   Then the final question representation $\mathbf {h}^q$ becomes a max-pooling over all $\mathbf {\gamma }^{^{\prime }}_i$ s, 1 $\le $ i $\le $ $N$   (2) Applying max-pooling on $\mathbf {\Gamma }^{(1)}_{1:N}$ and $\mathbf {\gamma }^{(2)}_i$0 to get $\mathbf {\gamma }^{(2)}_i$1 and $\mathbf {\gamma }^{(2)}_i$2 , respectively, then setting $\mathbf {\gamma }^{(2)}_i$3   Finally we compute the matching score of $\mathbf {\gamma }^{(2)}_i$4 given $\mathbf {\gamma }^{(2)}_i$5 as $\mathbf {\gamma }^{(2)}_i$6  
Intuitively, the proposed method should benefit from hierarchical training since the second layer is fitting the residues from the first layer of matching, so the two layers of representations are more likely to be complementary to each other  This also ensures the vector spaces of two layers are comparable and makes the second-layer training easier 
During training we adopt a ranking loss to maximizing the margin between the gold relation $\mathbf {r}^+$ and other relations $\mathbf {r}^-$ in the candidate pool $R$  
$$l_{\mathrm {rel}} = \max \lbrace 0, \gamma - s_{\mathrm {rel}}(\mathbf {r}^+; \mathbf {q}) + s_{\mathrm {rel}}(\mathbf {r}^-; \mathbf {q})\rbrace  \nonumber $$   (Eq  12)
where $\gamma $ is a constant parameter  Fig 2 summarizes the above Hierarchical Residual BiLSTM (HR-BiLSTM) model 
Another way of hierarchical matching consists in relying on attention mechanism, e g  BIBREF24 , to find the correspondence between different levels of representations  This performs below the HR-BiLSTM (see Table 2 ) 
KBQA Enhanced by Relation Detection
This section describes our KBQA pipeline system  We make minimal efforts beyond the training of the relation detection model, making the whole system easy to build 
Following previous work BIBREF4 , BIBREF5 , our KBQA system takes an existing entity linker to produce the top- $K$ linked entities, $EL_K(q)$ , for a question $q$ (“initial entity linking”)  Then we generate the KB queries for $q$ following the four steps illustrated in Algorithm "KBQA Enhanced by Relation Detection"  
[htbp] InputInput OutputOutput Top query tuple $(\hat{e},\hat{r}, \lbrace (c, r_c)\rbrace )$ Entity Re-Ranking (first-step relation detection): Use the raw question text as input for a relation detector to score all relations in the KB that are associated to the entities in $EL_K(q)$ ; use the relation scores to re-rank $EL_K(q)$ and generate a shorter list $EL^{\prime }_{K^{\prime }}(q)$ containing the top- $K^{\prime }$ entity candidates (Section "Entity Re-Ranking" ) Relation Detection: Detect relation(s) using the reformatted question text in which the topic entity is replaced by a special token $<$ e $>$ (Section "Relation Detection" ) Query Generation: Combine the scores from step 1 and 2, and select the top pair $(\hat{e},\hat{r})$ (Section "Query Generation" ) Constraint Detection (optional): Compute similarity between $q$ and any neighbor entity $c$ of the entities along $EL_K(q)$0 (connecting by a relation $EL_K(q)$1 ) , add the high scoring $EL_K(q)$2 and $EL_K(q)$3 to the query (Section "Constraint Detection" )  KBQA with two-step relation detection
Compared to previous approaches, the main difference is that we have an additional entity re-ranking step after the initial entity linking  We have this step because we have observed that entity linking sometimes becomes a bottleneck in KBQA systems  For example, on SimpleQuestions the best reported linker could only get 72 7% top-1 accuracy on identifying topic entities  This is usually due to the ambiguities of entity names, e g  in Fig 1 (a), there are TV writer and baseball player “Mike Kelley”, which is impossible to distinguish with only entity name matching 
Having observed that different entity candidates usually connect to different relations, here we propose to help entity disambiguation in the initial entity linking with relations detected in questions 
Sections "Entity Re-Ranking" and "Relation Detection" elaborate how our relation detection help to re-rank entities in the initial entity linking, and then those re-ranked entities enable more accurate relation detection  The KBQA end task, as a result, benefits from this process 
Entity Re-Ranking
In this step, we use the raw question text as input for a relation detector to score all relations in the KB with connections to at least one of the entity candidates in $EL_K(q)$   We call this step relation detection on entity set since it does not work on a single topic entity as the usual settings  We use the HR-BiLSTM as described in Sec  "Improved KB Relation Detection"   For each question $q$ , after generating a score $s_{rel}(r;q)$ for each relation using HR-BiLSTM, we use the top $l$ best scoring relations ( $R^{l}_q$ ) to re-rank the original entity candidates  Concretely, for each entity $e$ and its associated relations $R_e$ , given the original entity linker score $s_{linker}$ , and the score of the most confident relation $r\in R_q^{l} \cap R_e$ , we sum these two scores to re-rank the entities:
$$s_{\mathrm {rerank}}(e;q) =& \alpha \cdot s_{\mathrm {linker}}(e;q) \nonumber \\ + & (1-\alpha ) \cdot \max _{r \in R_q^{l} \cap R_e} s_{\mathrm {rel}}(r;q) \nonumber $$   (Eq

ground_truth: Yes

Yes

prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. On which benchmarks they achieve the state of the art?
Context 3:  15)
Finally, we select top $K^{\prime }$ $<$ $K$ entities according to score $s_{rerank}$ to form the re-ranked list $EL_{K^{\prime }}^{^{\prime }}(q)$  
We use the same example in Fig 1 (a) to illustrate the idea  Given the input question in the example, a relation detector is very likely to assign high scores to relations such as “episodes_written”, “author_of” and “profession”  Then, according to the connections of entity candidates in KB, we find that the TV writer “Mike Kelley” will be scored higher than the baseball player “Mike Kelley”, because the former has the relations “episodes_written” and “profession”  This method can be viewed as exploiting entity-relation collocation for entity linking 
Relation Detection
In this step, for each candidate entity $e \in EL_K^{\prime }(q)$ , we use the question text as the input to a relation detector to score all the relations $r \in R_e$ that are associated to the entity $e$ in the KB  Because we have a single topic entity input in this step, we do the following question reformatting: we replace the the candidate $e$ 's entity mention in $q$ with a token “ $<$ e $>$ ”  This helps the model better distinguish the relative position of each word compared to the entity  We use the HR-BiLSTM model to predict the score of each relation $r \in R_e$ : $s_{rel} (r;e,q)$  
Query Generation
Finally, the system outputs the $<$ entity, relation (or core-chain) $>$ pair $(\hat{e}, \hat{r})$ according to:
$$s(\hat{e}, \hat{r}; q) =& \max _{e \in EL_{K^{\prime }}^{^{\prime }}(q), r \in R_e} \left( \beta \cdot s_{\mathrm {rerank}}(e;q) \right  \nonumber \\ &\left + (1-\beta ) \cdot s_{\mathrm {rel}} (r;e,q) \right), \nonumber $$   (Eq  19)
where $\beta $ is a hyperparameter to be tuned 
Constraint Detection
Similar to BIBREF4 , we adopt an additional constraint detection step based on text matching  Our method can be viewed as entity-linking on a KB sub-graph  It contains two steps: (1) Sub-graph generation: given the top scored query generated by the previous 3 steps, for each node $v$ (answer node or the CVT node like in Figure 1 (b)), we collect all the nodes $c$ connecting to $v$ (with relation $r_c$ ) with any relation, and generate a sub-graph associated to the original query  (2) Entity-linking on sub-graph nodes: we compute a matching score between each $n$ -gram in the input question (without overlapping the topic entity) and entity name of $c$ (except for the node in the original query) by taking into account the maximum overlapping sequence of characters between them (see Appendix A for details and B for special rules dealing with date/answer type constraints)  If the matching score is larger than a threshold $\theta $ (tuned on training set), we will add the constraint entity $c$ (and $r_c$ ) to the query by attaching it to the corresponding node $v$ on the core-chain 
Experiments
Task Introduction & Settings
We use the SimpleQuestions BIBREF2 and WebQSP BIBREF25 datasets  Each question in these datasets is labeled with the gold semantic parse  Hence we can directly evaluate relation detection performance independently as well as evaluate on the KBQA end task 
SimpleQuestions (SQ): It is a single-relation KBQA task  The KB we use consists of a Freebase subset with 2M entities (FB2M) BIBREF2 , in order to compare with previous research  yin2016simple also evaluated their relation extractor on this data set and released their proposed question-relation pairs, so we run our relation detection model on their data set  For the KBQA evaluation, we also start with their entity linking results  Therefore, our results can be compared with their reported results on both tasks 
WebQSP (WQ): A multi-relation KBQA task  We use the entire Freebase KB for evaluation purposes  Following yih-EtAl:2016:P16-2, we use S-MART BIBREF26 entity-linking outputs  In order to evaluate the relation detection models, we create a new relation detection task from the WebQSP data set  For each question and its labeled semantic parse: (1) we first select the topic entity from the parse; and then (2) select all the relations and relation chains (length $\le $ 2) connected to the topic entity, and set the core-chain labeled in the parse as the positive label and all the others as the negative examples 
We tune the following hyper-parameters on development sets: (1) the size of hidden states for LSTMs ({50, 100, 200, 400}); (2) learning rate ({0 1, 0 5, 1 0, 2 0}); (3) whether the shortcut connections are between hidden states or between max-pooling results (see Section "Hierarchical Matching between Relation and Question" ); and (4) the number of training epochs 
For both the relation detection experiments and the second-step relation detection in KBQA, we have entity replacement first (see Section "Relation Detection" and Figure 1 )  All word vectors are initialized with 300- $d$ pretrained word embeddings BIBREF27   The embeddings of relation names are randomly initialized, since existing pre-trained relation embeddings (e g  TransE) usually support limited sets of relation names  We leave the usage of pre-trained relation embeddings to future work 
Relation Detection Results
Table 2 shows the results on two relation detection tasks  The AMPCNN result is from BIBREF20 , which yielded state-of-the-art scores by outperforming several attention-based methods  We re-implemented the BiCNN model from BIBREF4 , where both questions and relations are represented with the word hash trick on character tri-grams  The baseline BiLSTM with relation word sequence appears to be the best baseline on WebQSP and is close to the previous best result of AMPCNN on SimpleQuestions  Our proposed HR-BiLSTM outperformed the best baselines on both tasks by margins of 2-3% (p $<$ 0 001 and 0 01 compared to the best baseline BiLSTM w/ words on SQ and WQ respectively) 
Note that using only relation names instead of words results in a weaker baseline BiLSTM model  The model yields a significant performance drop on SimpleQuestions (91 2% to 88 9%)  However, the drop is much smaller on WebQSP, and it suggests that unseen relations have a much bigger impact on SimpleQuestions 
The bottom of Table 2 shows ablation results of the proposed HR-BiLSTM  First, hierarchical matching between questions and both relation names and relation words yields improvement on both datasets, especially for SimpleQuestions (93 3% vs  91 2/88 8%)  Second, residual learning helps hierarchical matching compared to weighted-sum and attention-based baselines (see Section "Hierarchical Matching between Relation and Question" )  For the attention-based baseline, we tried the model from BIBREF24 and its one-way variations, where the one-way model gives better results  Note that residual learning significantly helps on WebQSP (80 65% to 82 53%), while it does not help as much on SimpleQuestions  On SimpleQuestions, even removing the deep layers only causes a small drop in performance  WebQSP benefits more from residual and deeper architecture, possibly because in this dataset it is more important to handle larger scope of context matching 
Finally, on WebQSP, replacing BiLSTM with CNN in our hierarchical matching framework results in a large performance drop  Yet on SimpleQuestions the gap is much smaller  We believe this is because the LSTM relation encoder can better learn the composition of chains of relations in WebQSP, as it is better at dealing with longer dependencies 
Next, we present empirical evidences, which show why our HR-BiLSTM model achieves the best scores  We use WebQSP for the analysis purposes  First, we have the hypothesis that training of the weighted-sum model usually falls to local optima, since deep BiLSTMs do not guarantee that the two-levels of question hidden representations are comparable  This is evidenced by that during training one layer usually gets a weight close to 0 thus is ignored  For example, one run gives us weights of -75 39/0 14 for the two layers (we take exponential for the final weighted sum)  It also gives much lower training accuracy (91 94%) compared to HR-BiLSTM (95 67%), suffering from training difficulty 
Second, compared to our deep BiLSTM with shortcut connections, we have the hypothesis that for KB relation detection, training deep BiLSTMs is more difficult without shortcut connections  Our experiments suggest that deeper BiLSTM does not always result in lower training accuracy  In the experiments a two-layer BiLSTM converges to 94 99%, even lower than the 95 25% achieved by a single-layer BiLSTM  Under our setting the two-layer model captures the single-layer model as a special case (so it could potentially better fit the training data), this result suggests that the deep BiLSTM without shortcut connections might suffers more from training difficulty 
Finally, we hypothesize that HR-BiLSTM is more than combination of two BiLSTMs with residual connections, because it encourages the hierarchical architecture to learn different levels of abstraction  To verify this, we replace the deep BiLSTM question encoder with two single-layer BiLSTMs (both on words) with shortcut connections between their hidden states  This decreases test accuracy to 76 11%  It gives similar training accuracy compared to HR-BiLSTM, indicating a more serious over-fitting problem  This proves that the residual and deep structures both contribute to the good performance of HR-BiLSTM 
KBQA End-Task Results
Table 3 compares our system with two published baselines (1) STAGG BIBREF4 , the state-of-the-art on WebQSP and (2) AMPCNN BIBREF20 , the state-of-the-art on SimpleQuestions  Since these two baselines are specially designed/tuned for one particular dataset, they do not generalize well when applied to the other dataset  In order to highlight the effect of different relation detection models on the KBQA end-task, we also implemented another baseline that uses our KBQA system but replaces HR-BiLSTM with our implementation of AMPCNN (for SimpleQuestions) or the char-3-gram BiCNN (for WebQSP) relation detectors (second block in Table 3 ) 
Compared to the baseline relation detector (3rd row of results), our method, which includes an improved relation detector (HR-BiLSTM), improves the KBQA end task by 2-3% (4th row)  Note that in contrast to previous KBQA systems, our system does not use joint-inference or feature-based re-ranking step, nevertheless it still achieves better or comparable results to the state-of-the-art 
The third block of the table details two ablation tests for the proposed components in our KBQA systems: (1) Removing the entity re-ranking step significantly decreases the scores  Since the re-ranking step relies on the relation detection models, this shows that our HR-BiLSTM model contributes to the good performance in multiple ways  Appendix C gives the detailed performance of the re-ranking step  (2) In contrast to the conclusion in BIBREF4 , constraint detection is crucial for our system  This is probably because our joint performance on topic entity and core-chain detection is more accurate (77 5% top-1 accuracy), leaving a huge potential (77 5% vs  58 0%) for the constraint detection module to improve 
Finally, like STAGG, which uses multiple relation detectors (see yih2015semantic for the three models used), we also try to use the top-3 relation detectors from Section "Relation Detection Results"   As shown on the last row of Table 3 , this gives a significant performance boost, resulting in a new state-of-the-art result on SimpleQuestions and a result comparable to the state-of-the-art on WebQSP 
Conclusion
KB relation detection is a key step in KBQA and is significantly different from general relation extraction tasks  We propose a novel KB relation detection model, HR-BiLSTM, that performs hierarchical matching between questions and KB relations  Our model outperforms the previous methods on KB relation detection tasks and allows our KBQA system to achieve state-of-the-arts  For future work, we will investigate the integration of our HR-BiLSTM into end-to-end systems  For example, our model could be integrated into the decoder in BIBREF31 , to provide better sequence prediction  We will also investigate new emerging datasets like GraphQuestions BIBREF32 and ComplexQuestions BIBREF30 to handle more characteristics of general QA 

ground_truth: SimpleQuestions, WebQSP

WebQSP, SimpleQuestions