prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. what was the baseline?
Context 1: Introduction
Neural machine translation (NMT) BIBREF0 , BIBREF1 , BIBREF2 has enabled end-to-end training of a translation system without needing to deal with word alignments, translation rules, and complicated decoding algorithms, which are the characteristics of phrase-based statistical machine translation (PBSMT) BIBREF3   Although NMT can be significantly better than PBSMT in resource-rich scenarios, PBSMT performs better in low-resource scenarios BIBREF4   Only by exploiting cross-lingual transfer learning techniques BIBREF5 , BIBREF6 , BIBREF7 , can the NMT performance approach PBSMT performance in low-resource scenarios 
However, such methods usually require an NMT model trained on a resource-rich language pair like French INLINEFORM0 English (parent), which is to be fine-tuned for a low-resource language pair like Uzbek INLINEFORM1 English (child)  On the other hand, multilingual approaches BIBREF8 propose to train a single model to translate multiple language pairs  However, these approaches are effective only when the parent target or source language is relatively resource-rich like English (En)  Furthermore, the parents and children models should be trained on similar domains; otherwise, one has to take into account an additional problem of domain adaptation BIBREF9  
In this paper, we work on a linguistically distant and thus challenging language pair Japanese INLINEFORM0 Russian (Ja INLINEFORM1 Ru) which has only 12k lines of news domain parallel corpus and hence is extremely resource-poor  Furthermore, the amount of indirect in-domain parallel corpora, i e , Ja INLINEFORM2 En and Ru INLINEFORM3 En, are also small  As we demonstrate in Section SECREF4 , this severely limits the performance of prominent low-resource techniques, such as multilingual modeling, back-translation, and pivot-based PBSMT  To remedy this, we propose a novel multistage fine-tuning method for NMT that combines multilingual modeling BIBREF8 and domain adaptation BIBREF9  
We have addressed two important research questions (RQs) in the context of extremely low-resource machine translation (MT) and our explorations have derived rational contributions (CTs) as follows:
To the best of our knowledge, we are the first to perform such an extensive evaluation of extremely low-resource MT problem and propose a novel multilingual multistage fine-tuning approach involving multilingual modeling and domain adaptation to address it 
Our Japanese–Russian Setting
In this paper, we deal with Ja INLINEFORM0 Ru news translation  This language pair is very challenging because the languages involved have completely different writing system, phonology, morphology, grammar, and syntax  Among various domains, we experimented with translations in the news domain, considering the importance of sharing news between different language speakers  Moreover, news domain is one of the most challenging tasks, due to large presence of out-of-vocabulary (OOV) tokens and long sentences  To establish and evaluate existing methods, we also involved English as the third language  As direct parallel corpora are scarce, involving a language such as English for pivoting is quite common BIBREF10  
There has been no clean held-out parallel data for Ja INLINEFORM0 Ru and Ja INLINEFORM1 En news translation  Therefore, we manually compiled development and test sets using News Commentary data as a source  Since the given Ja INLINEFORM2 Ru and Ja INLINEFORM3 En data share many lines in the Japanese side, we first compiled tri-text data  Then, from each line, corresponding parts across languages were manually identified, and unaligned parts were split off into a new line  Note that we have never merged two or more lines  As a result, we obtained 1,654 lines of data comprising trilingual, bilingual, and monolingual segments (mainly sentences) as summarized in Table TABREF8   Finally, for the sake of comparability, we randomly chose 600 trilingual sentences to create a test set, and concatenated the rest of them and bilingual sentences to form development sets 
Our manually aligned development and test sets are publicly available 
Related Work
koehn-knowles:2017:NMT showed that NMT is unable to handle low-resource language pairs as opposed to PBSMT  Transfer learning approaches BIBREF5 , BIBREF6 , BIBREF7 work well when a large helping parallel corpus is available  This restricts one of the source or the target languages to be English which, in our case, is not possible  Approaches involving bi-directional NMT modeling is shown to drastically improve low-resource translation BIBREF11   However, like most other, this work focuses on translation from and into English 
Remaining options include (a) unsupervised MT BIBREF12 , BIBREF13 , BIBREF14 , (b) parallel sentence mining from non-parallel or comparable corpora BIBREF15 , BIBREF16 , (c) generating pseudo-parallel data BIBREF17 , and (d) MT based on pivot languages BIBREF10   The linguistic distance between Japanese and Russian makes it extremely difficult to learn bilingual knowledge, such as bilingual lexicons and bilingual word embeddings  Unsupervised MT is thus not promising yet, due to its heavy reliance on accurate bilingual word embeddings  Neither does parallel sentence mining, due to the difficulty of obtaining accurate bilingual lexicons  Pseudo-parallel data can be used to augment existing parallel corpora for training, and previous work has reported that such data generated by so-called back-translation can substantially improve the quality of NMT  However, this approach requires base MT systems that can generate somewhat accurate translations  It is thus infeasible in our scenario, because we can obtain only a weak system which is the consequence of an extremely low-resource situation  MT based on pivot languages requires large in-domain parallel corpora involving the pivot languages  This technique is thus infeasible, because the in-domain parallel corpora for Ja INLINEFORM0 En and Ru INLINEFORM1 En pairs are also extremely limited, whereas there are large parallel corpora in other domains  Section SECREF4 empirically confirms the limit of these existing approaches 
Fortunately, there are two useful transfer learning solutions using NMT: (e) multilingual modeling to incorporate multiple language pairs into a single model BIBREF8 and (f) domain adaptation to incorporate out-of-domain data BIBREF9   In this paper, we explore a novel method involving step-wise fine-tuning to combine these two methods  By improving the translation quality in this way, we can also increase the likelihood of pseudo-parallel data being useful to further improve translation quality 
Limit of Using only In-domain Data
This section answers our first research question, [RQ1], about the translation quality that we can achieve using existing methods and in-domain parallel and monolingual data  We then use the strongest model to conduct experiments on generating and utilizing back-translated pseudo-parallel data for augmenting NMT  Our intention is to empirically identify the most effective practices as well as recognize the limitations of relying only on in-domain parallel corpora 
Data
To train MT systems among the three languages, i e , Japanese, Russian, and English, we used all the parallel data provided by Global Voices, more specifically those available at OPUS  Table TABREF9 summarizes the size of train/development/test splits used in our experiments  The number of parallel sentences for Ja INLINEFORM0 Ru is 12k, for Ja INLINEFORM1 En is 47k, and for Ru INLINEFORM2 En is 82k  Note that the three corpora are not mutually exclusive: 9k out of 12k sentences in the Ja INLINEFORM3 Ru corpus were also included in the other two parallel corpora, associated with identical English translations  This puts a limit on the positive impact that the helping corpora can have on the translation quality 
Even when one focuses on low-resource language pairs, we often have access to larger quantities of in-domain monolingual data of each language  Such monolingual data are useful to improve quality of MT, for example, as the source of pseudo-parallel data for augmenting training data for NMT BIBREF17 and as the training data for large and smoothed language models for PBSMT BIBREF4   Table TABREF13 summarizes the statistics on our monolingual corpora for several domains including the news domain  Note that we removed from the Global Voices monolingual corpora those sentences that are already present in the parallel corpus 
https://dumps wikimedia org/backup-index html (20180501) http://www statmt org/wmt18/translation-task html https://www yomiuri co jp/database/glossary/ http://www cs jhu edu/~kevinduh/a/multitarget-tedtalks/ http://opus nlpl eu/Tatoeba-v2 php
We tokenized English and Russian sentences using tokenizer perl of Moses BIBREF3   To tokenize Japanese sentences, we used MeCab with the IPA dictionary  After tokenization, we eliminated duplicated sentence pairs and sentences with more than 100 tokens for all the languages 
MT Methods Examined
We began with evaluating standard MT paradigms, i e , PBSMT BIBREF3 and NMT BIBREF1   As for PBSMT, we also examined two advanced methods: pivot-based translation relying on a helping language BIBREF10 and induction of phrase tables from monolingual data BIBREF14  
As for NMT, we compared two types of encoder-decoder architectures: attentional RNN-based model (RNMT) BIBREF2 and the Transformer model BIBREF18   In addition to standard uni-directional modeling, to cope with the low-resource problem, we examined two multi-directional models: bi-directional model BIBREF11 and multi-to-multi (M2M) model BIBREF8  
After identifying the best model, we also examined the usefulness of a data augmentation method based on back-translation BIBREF17  
First, we built a PBSMT system for each of the six translation directions  We obtained phrase tables from parallel corpus using SyMGIZA++ with the grow-diag-final heuristics for word alignment, and Moses for phrase pair extraction  Then, we trained a bi-directional MSD (monotone, swap, and discontinuous) lexicalized reordering model  We also trained three 5-gram language models, using KenLM on the following monolingual data: (1) the target side of the parallel data, (2) the concatenation of (1) and the monolingual data from Global Voices, and (3) the concatenation of (1) and all monolingual data in the news domain in Table TABREF13  
Subsequently, using English as the pivot language, we examined the following three types of pivot-based PBSMT systems BIBREF10 , BIBREF19 for each of Ja INLINEFORM0 Ru and Ru INLINEFORM1 Ja 
2-step decoding using the source-to-English and English-to-target systems 
Obtain a new phrase table from synthetic parallel data generated by translating English side of the target–English training parallel data to the source language with the English-to-source system 
Compile a new phrase table combining those for the source-to-English and English-to-target systems 
Among these three, triangulation is the most computationally expensive method  Although we had filtered the component phrase tables using the statistical significance pruning method BIBREF20 , triangulation can generate an enormous number of phrase pairs  To reduce the computational cost during decoding and the negative effects of potentially noisy phrase pairs, we retained for each source phrase INLINEFORM0 only the INLINEFORM1 -best translations INLINEFORM2 according to the forward translation probability INLINEFORM3 calculated from the conditional probabilities in the component models as defined in utiyama:07  For each of the retained phrase pairs, we also calculated the backward translation probability, INLINEFORM4 , and lexical translation probabilities, INLINEFORM5 and INLINEFORM6 , in the same manner as INLINEFORM7  
We also investigated the utility of recent advances in unsupervised MT  Even though we began with a publicly available implementation of unsupervised PBSMT BIBREF13 , it crashed due to unknown reasons  We therefore followed another method described in marie:usmt-unmt  Instead of short INLINEFORM0 -grams BIBREF12 , BIBREF13 , we collected a set of phrases in Japanese and Russian from respective monolingual data using the word2phrase algorithm BIBREF21 , as in marie:usmt-unmt  To reduce the complexity, we used randomly selected 10M monolingual sentences, and 300k most frequent phrases made of words among the 300k most frequent words  For each source phrase INLINEFORM1 , we selected 300-best target phrases INLINEFORM2 according to the translation probability as in D18-1549: INLINEFORM3 where INLINEFORM4 stands for a bilingual embedding of a given phrase, obtained through averaging bilingual embeddings of constituent words learned from the two monolingual data using fastText and vecmap  For each of the retained phrase pair, INLINEFORM5 was computed analogously  We also computed lexical translation probabilities relying on those learned from the given small parallel corpus 
Up to four phrase tables were jointly exploited by the multiple decoding path ability of Moses  Weights for the features were tuned using KB-MIRA BIBREF22 on the development set; we took the best weights after 15 iterations  Two hyper-parameters, namely, INLINEFORM0 for the number of pivot-based phrase pairs per source phrase and INLINEFORM1 for distortion limit, were determined by a grid search on INLINEFORM2 and INLINEFORM3   In contrast, we used predetermined hyper-parameters for phrase table induction from monolingual data, following the convention: 200 for the dimension of word and phrase embeddings and INLINEFORM4  
We used the open-source implementation of the RNMT and the Transformer models in tensor2tensor  A uni-directional model for each of the six translation directions was trained on the corresponding parallel corpus  Bi-directional and M2M models were realized by adding an artificial token that specifies the target language to the beginning of each source sentence and shuffling the entire training data BIBREF8  
Table TABREF22 contains some specific hyper-parameters for our baseline NMT models  The hyper-parameters not mentioned in this table used the default values in tensor2tensor  For M2M systems, we over-sampled Ja INLINEFORM0 Ru and Ja INLINEFORM1 En training data so that their sizes match the largest Ru INLINEFORM2 En data  To reduce the number of unknown words, we used tensor2tensor's internal sub-word segmentation mechanism  Since we work in a low-resource setting, we used shared sub-word vocabularies of size 16k for the uni- and bi-directional models and 32k for the M2M models  The number of training iterations was determined by early-stopping: we evaluated our models on the development set every 1,000 updates, and stopped training if BLEU score for the development set was not improved for 10,000 updates (10 check-points)  Note that the development set was created by concatenating those for the individual translation directions without any over-sampling 
Having trained the models, we averaged the last 10 check-points and decoded the test sets with a beam size of 4 and a length penalty which was tuned by a linear search on the BLEU score for the development set 
Similarly to PBSMT, we also evaluated “Cascade” and “Synthesize” methods with uni-directional NMT models 
Results
We evaluated MT models using case-sensitive and tokenized BLEU BIBREF23 on test sets, using Moses's multi-bleu perl  Statistical significance ( INLINEFORM0 ) on the difference of BLEU scores was tested by Moses's bootstrap-hypothesis-difference-significance pl 
Tables TABREF27 and TABREF31 show BLEU scores of all the models, except the NMT systems augmented with back-translations  Whereas some models achieved reasonable BLEU scores for Ja INLINEFORM0 En and Ru INLINEFORM1 En translation, all the results for Ja INLINEFORM2 Ru, which is our main concern, were abysmal 
Among the NMT models, Transformer models (b INLINEFORM0 ) were proven to be better than RNMT models (a INLINEFORM1 )  RNMT models could not even outperform the uni-directional PBSMT models (c1)  M2M models (a3) and (b3) outperformed their corresponding uni- and bi-directional models in most cases  It is worth noting that in this extremely low-resource scenario, BLEU scores of the M2M RNMT model for the largest language pair, i e , Ru INLINEFORM2 En, were lower than those of the uni- and bi-directional RNMT models as in TACL1081  In contrast, with the M2M Transformer model, Ru INLINEFORM3 En also benefited from multilingualism 
Standard PBSMT models (c1) achieved higher BLEU scores than uni-directional NMT models (a1) and (b1), as reported by koehn-knowles:2017:NMT, whereas they underperform the M2M Transformer NMT model (b3)  As shown in Table TABREF31 , pivot-based PBSMT systems always achieved higher BLEU scores than (c1)  The best model with three phrase tables, labeled “Synthesize / Triangulate / Gold,” brought visible BLEU gains with substantial reduction of OOV tokens (3047 INLINEFORM0 1180 for Ja INLINEFORM1 Ru, 4463 INLINEFORM2 1812 for Ru INLINEFORM3 Ja)  However, further extension with phrase tables induced from monolingual data did not push the limit, despite their high coverage; only 336 and 677 OOV tokens were left for the two translation directions, respectively  This is due to the poor quality of the bilingual word embeddings used to extract the phrase table, as envisaged in Section SECREF3  
None of pivot-based approaches with uni-directional NMT models could even remotely rival the M2M Transformer NMT model (b3) 
Table TABREF46 shows the results of our multistage fine-tuning, where the IDs of each row refer to those described in Section SECREF41   First of all, the final models of our multistage fine-tuning, i e , V and VII, achieved significantly higher BLEU scores than (b3) in Table TABREF27 , a weak baseline without using any monolingual data, and #10 in Table TABREF33 , a strong baseline established with monolingual data 
The performance of the initial model (I) depends on the language pair  For Ja INLINEFORM0 Ru pair, it cannot achieve minimum level of quality since the model has never seen parallel data for this pair  The performance on Ja INLINEFORM1 En pair was much lower than the two baseline models, reflecting the crucial mismatch between training and testing domains  In contrast, Ru INLINEFORM2 En pair benefited the most and achieved surprisingly high BLEU scores  The reason might be due to the proximity of out-of-domain training data and in-domain test data 
The first fine-tuning stage significantly pushed up the translation quality for Ja INLINEFORM0 En and Ru INLINEFORM1 En pairs, in both cases with fine-tuning (II) and mixed fine-tuning (III)  At this stage, both models performed only poorly for Ja INLINEFORM2 Ru pair as they have not yet seen Ja INLINEFORM3 Ru parallel data  Either model had a consistent advantage to the other 
When these models were further fine-tuned only on the in-domain Ja INLINEFORM0 Ru parallel data (IV and VI), we obtained translations of better quality than the two baselines for Ja INLINEFORM1 Ru pair  However, as a result of complete ignorance of Ja INLINEFORM2 En and Ru INLINEFORM3 En pairs, the models only produced translations of poor quality for these language pairs  In contrast, mixed fine-tuning for the second fine-tuning stage (V and VII) resulted in consistently better models than conventional fine-tuning (IV and VI), irrespective of the choice at the first stage, thanks to the gradual shift of parameters realized by in-domain Ja INLINEFORM4 En and Ru INLINEFORM5 En parallel data  Unfortunately, the translation quality for Ja INLINEFORM6 En and Ru INLINEFORM7 En pairs sometimes degraded from II and III  Nevertheless, the BLEU scores still retain the large margin against two baselines 
The last three rows in Table TABREF46 present BLEU scores obtained by the methods with fewer fine-tuning steps  The most naive model I', trained on the balanced mixture of whole five types of corpora from scratch, and the model II', obtained through a single-step conventional fine-tuning of I on all the in-domain data, achieved only BLEU scores consistently worse than VII  In contrast, when we merged our two fine-tuning steps into a single mixed fine-tuning on I, we obtained a model III' which is better for the Ja INLINEFORM0 Ru pair than VII  Nevertheless, they are still comparable to those of VII and the BLEU scores for the other two language pairs are much lower than VII  As such, we conclude that our multistage fine-tuning leads to a more robust in-domain multilingual model 
Augmentation with Back-translation
Given that the M2M Transformer NMT model (b3) achieved best results for most of the translation directions and competitive results for the rest, we further explored it through back-translation 
We examined the utility of pseudo-parallel data for all the six translation directions, unlike the work of lakew2017improving and lakew2018comparison, which concentrate only on the zero-shot language pair, and the work of W18-2710, which compares only uni- or bi-directional models  We investigated whether each translation direction in M2M models will benefit from pseudo-parallel data and if so, what kind of improvement takes place 
First, we selected sentences to be back-translated from in-domain monolingual data (Table TABREF13 ), relying on the score proposed by moore:intelligent via the following procedure 
For each language, train two 4-gram language models, using KenLM: an in-domain one on all the Global Voices data, i e , both parallel and monolingual data, and a general-domain one on the concatenation of Global Voices, IWSLT, and Tatoeba data 
For each language, discard sentences containing OOVs according to the in-domain language model 
For each translation direction, select the INLINEFORM0 -best monolingual sentences in the news domain, according to the difference between cross-entropy scores given by the in-domain and general-domain language models 
Whereas W18-2710 exploited monolingual data much larger than parallel data, we maintained a 1:1 ratio between them BIBREF8 , setting INLINEFORM0 to the number of lines of parallel data of given language pair 
Selected monolingual sentences were then translated using the M2M Transformer NMT model (b3) to compose pseudo-parallel data  Then, the pseudo-parallel data were enlarged by over-sampling as summarized in Table TABREF32   Finally, new NMT models were trained on the concatenation of the original parallel and pseudo-parallel data from scratch in the same manner as the previous NMT models with the same hyper-parameters 
Table TABREF33 shows the BLEU scores achieved by several reasonable combinations of six-way pseudo-parallel data  We observed that the use of all six-way pseudo-parallel data (#10) significantly improved the base model for all the translation directions, except En INLINEFORM0 Ru  A translation direction often benefited when the pseudo-parallel data for that specific direction was used 
Summary
We have evaluated an extensive variation of MT models that rely only on in-domain parallel and monolingual data  However, the resulting BLEU scores for Ja INLINEFORM2 Ru and Ru INLINEFORM3 Ja tasks do not exceed 10 BLEU points, implying the inherent limitation of the in-domain data as well as the difficulty of these translation directions 
Exploiting Large Out-of-Domain Data Involving a Helping Language
The limitation of relying only on in-domain data demonstrated in Section SECREF4 motivates us to explore other types of parallel data  As raised in our second research question, [RQ2], we considered the effective ways to exploit out-of-domain data 
According to language pair and domain, parallel data can be classified into four categories in Table TABREF40   Among all the categories, out-of-domain data for the language pair of interest have been exploited in the domain adaptation scenarios (C INLINEFORM0 A) BIBREF9   However, for Ja INLINEFORM1 Ru, no out-of-domain data is available  To exploit out-of-domain parallel data for Ja INLINEFORM2 En and Ru INLINEFORM3 En pairs instead, we propose a multistage fine-tuning method, which combines two types of transfer learning, i e , domain adaptation for Ja INLINEFORM4 En and Ru INLINEFORM5 En (D INLINEFORM6 B) and multilingual transfer (B INLINEFORM7 A), relying on the M2M model examined in Section SECREF4   We also examined the utility of fine-tuning for iteratively generating and using pseudo-parallel data 
Multistage Fine-tuning
Simply using NMT systems trained on out-of-domain data for in-domain translation is known to perform badly  In order to effectively use large-scale out-of-domain data for our extremely low-resource task, we propose to perform domain adaptation through either (a) conventional fine-tuning, where an NMT system trained on out-of-domain data is fine-tuned only on in-domain data, or (b) mixed fine-tuning BIBREF9 , where pre-trained out-of-domain NMT system is fine-tuned using a mixture of in-domain and out-of-domain data  The same options are available for transferring from Ja INLINEFORM0 En and Ru INLINEFORM1 En to Ja INLINEFORM2 Ru 
We inevitably involve two types of transfer learning, i e , domain adaptation for Ja INLINEFORM0 En and Ru INLINEFORM1 En and multilingual transfer for Ja INLINEFORM2 Ru pair  Among several conceivable options for managing these two problems, we examined the following multistage fine-tuning 
Pre-train a multilingual model only on the Ja INLINEFORM0 En and Ru INLINEFORM1 En out-of-domain parallel data (I), where the vocabulary of the model is determined on the basis of the in-domain parallel data in the same manner as the M2M NMT models examined in Section SECREF4  
Fine-tune the pre-trained model (I) on the in-domain Ja INLINEFORM0 En and Ru INLINEFORM1 En parallel data (fine-tuning, II) or on the mixture of in-domain and out-of-domain Ja INLINEFORM2 En and Ru INLINEFORM3 En parallel data (mixed fine-tuning, III) 
Further fine-tune the models (each of II and III) for Ja INLINEFORM0 Ru on in-domain parallel data for this language pair only (fine-tuning, IV and VI) or on all the in-domain parallel data (mixed fine-tuning, V and VII) 
We chose this way due to the following two reasons  First, we need to take a balance between several different parallel corpora sizes  The other reason is division of labor; we assume that solving each sub-problem one by one should enable gradual shift of parameters 
Data Selection
As an additional large-scale out-of-domain parallel data for Ja INLINEFORM0 En, we used the cleanest 1 5M sentences from the Asian Scientific Paper Excerpt Corpus (ASPEC) BIBREF24   As for Ru INLINEFORM1 En, we used the UN and Yandex corpora released for the WMT 2018 News Translation Task  We retained Ru INLINEFORM2 En sentence pairs that contain at least one OOV token in both sides, according to the in-domain language model trained in Section SECREF34   Table TABREF45 summarizes the statistics on the remaining out-of-domain parallel data 
Further Augmentation with Back-translation
Having obtained a better model, we examined again the utility of back-translation  More precisely, we investigated (a) whether the pseudo-parallel data generated by an improved NMT model leads to a further improvement, and (b) whether one more stage of fine-tuning on the mixture of original parallel and pseudo-parallel data will result in a model better than training a new model from scratch as examined in Section SECREF34  
Given an NMT model, we first generated six-way pseudo-parallel data by translating monolingual data  For the sake of comparability, we used the identical monolingual sentences sampled in Section SECREF34   Then, we further fine-tuned the given model on the mixture of the generated pseudo-parallel data and the original parallel data, following the same over-sampling procedure in Section SECREF34   We repeated these steps five times 
Table TABREF51 shows the results  “new #10” in the second row indicates an M2M Transformer model trained from scratch on the mixture of six-way pseudo-parallel data generated by VII and the original parallel data  It achieved higher BLEU scores than #10 in Table TABREF33 thanks to the pseudo-parallel data of better quality, but underperformed the base NMT model VII  In contrast, our fine-tuned model VIII successfully surpassed VII, and one more iteration (IX) further improved BLEU scores for all translation directions, except Ru INLINEFORM0 En  Although further iterations did not necessarily gain BLEU scores, we came to a much higher plateau compared to the results in Section SECREF4  
Conclusion
In this paper, we challenged the difficult task of Ja INLINEFORM0 Ru news domain translation in an extremely low-resource setting  We empirically confirmed the limited success of well-established solutions when restricted to in-domain data  Then, to incorporate out-of-domain data, we proposed a multilingual multistage fine-tuning approach and observed that it substantially improves Ja INLINEFORM1 Ru translation by over 3 7 BLEU points compared to a strong baseline, as summarized in Table TABREF53   This paper contains an empirical comparison of several existing approaches and hence we hope that our paper can act as a guideline to researchers attempting to tackle extremely low-resource translation 
In the future, we plan to confirm further fine-tuning for each of specific translation directions  We will also explore the way to exploit out-of-domain pseudo-parallel data, better domain-adaptation approaches, and additional challenging language pairs 
Acknowledgments
This work was carried out when Aizhan Imankulova was taking up an internship at NICT, Japan  We would like to thank the reviewers for their insightful comments  A part of this work was conducted under the program “Promotion of Global Communications Plan: Research, Development, and Social Demonstration of Multilingual Speech Translation Technology” of the Ministry of Internal Affairs and Communications (MIC), Japan 
Context 2: Introduction
BioASQ is a biomedical document classification, document retrieval, and question answering competition, currently in its seventh year  We provide an overview of our submissions to semantic question answering task (7b, Phase B) of BioASQ 7 (except for 'ideal answer' test, in which we did not participate this year)  In this task systems are provided with biomedical questions and are required to submit ideal and exact answers to those questions  We have used BioBERT BIBREF0 based system , see also Bidirectional Encoder Representations from Transformers(BERT) BIBREF1, and we fine tuned it for the biomedical question answering task  Our system scored near the top for factoid questions for all the batches of the challenge  More specifially, in the third test batch set, our system achieved highest ‘MRR’ score for Factoid Question Answering task  Also, for List-type question answering task our system achieved highest recall score in the fourth test batch set  Along with our detailed approach, we present the results for our submissions and also highlight identified downsides for our current approach and ways to improve them in our future experiments  In last test batch results we placed 4th for List-type questions and 3rd for Factoid-type questions )
The QA task is organized in two phases  Phase A deals with retrieval of the relevant document, snippets, concepts, and RDF triples, and phase B deals with exact and ideal answer generations (which is a paragraph size summary of snippets)  Exact answer generation is required for factoid, list, and yes/no type question 
BioASQ organizers provide the training and testing data  The training data consists of questions, gold standard documents, snippets, concepts, and ideal answers (which we did not use in this paper, but we used last year BIBREF2)  The test data is split between phases A and B  The Phase A dataset consists of the questions, unique ids, question types  The Phase B dataset consists of the questions, golden standard documents, snippets, unique ids and question types  Exact answers for factoid type questions are evaluated using strict accuracy (the top answer), lenient accuracy (the top 5 answers), and MRR (Mean Reciprocal Rank) which takes into account the ranks of returned answers  Answers for the list type question are evaluated using precision, recall, and F-measure 
Related Work ::: BioAsq
Sharma et al  BIBREF3 describe a system with two stage process for factoid and list type question answering  Their system extracts relevant entities and then runs supervised classifier to rank the entities  Wiese et al  BIBREF4 propose neural network based model for Factoid and List-type question answering task  The model is based on Fast QA and predicts the answer span in the passage for a given question  The model is trained on SQuAD data set and fine tuned on the BioASQ data  Dimitriadis et al  BIBREF5 proposed two stage process for Factoid question answering task  Their system uses general purpose tools such as Metamap, BeCas to identify candidate sentences  These candidate sentences are represented in the form of features, and are then ranked by the binary classifier  Classifier is trained on candidate sentences extracted from relevant questions, snippets and correct answers from BioASQ challenge  For factoid question answering task highest ‘MRR’ achieved in the 6th edition of BioASQ competition is ‘0 4325’  Our system is a neural network model based on contextual word embeddings BIBREF1 and achieved a ‘MRR’ score ‘0 6103’ in one of the test batches for Factoid Question Answering task 
Related Work ::: A minimum background on BERT
BERT stands for "Bidirectional Encoder Representations from Transformers" BIBREF1 is a contextual word embedding model  Given a sentence as an input, contextual embedding for the words are returned  The BERT model was designed so it can be fine tuned for 11 different tasks BIBREF1, including question answering tasks  For a question answering task, question and paragraph (context) are given as an input  A BERT standard is that question text and paragraph text are separated by a separator [Sep]  BERT question-answering fine tuning involves adding softmax layer  Softmax layer takes contextual word embeddings from BERT as input and learns to identity answer span present in the paragraph (context)  This process is represented in Figure FIGREF4 
BERT was originally trained to perform tasks such as language model creation using masked words and next-sentence-prediction  In other words BERT weights are learned such that context is used in building the representation of the word, not just as a loss function to help learn a context-independent representation  For detailed understanding of BERT Architecture, please refer to the original BERT paper BIBREF1 
Related Work ::: A minimum background on BERT ::: Comparison of Word Embeddings and Contextual Word Embeddings
A ‘word embedding’ is a learned representation  It is represented in the form of vector where words that have the same meaning have a similar vector representation  Consider a word embedding model 'word2vec' BIBREF6 trained on a corpus  Word embeddings generated from the model are context independent that is, word embeddings are returned regardless of where the words appear in a sentence and regardless of e g  the sentiment of the sentence  However, contextual word embedding models like BERT also takes context of the word into consideration 
Related Work ::: Comparison of BERT and Bio-BERT
‘BERT’ and BioBERT are very similar in terms of architecture  Difference is that ‘BERT’ is pretrained on Wikipedia articles, whereas BioBERT version used in our experiments is pretrained on Wikipedia, PMC and PubMed articles  Therefore BioBERT model is expected to perform well with biomedical text, in terms of generating contextual word embeddings 
BioBERT model used in our experiments is based on BERT-Base Architecture; BERT-Base has 12 transformer Layers where as BERT-Large has 24 transformer layers  Moreover contextual word embedding vector size is 768 for BERT-Base and more for BERT-large  According to BIBREF1 Bert-Large, fine-tuned on SQuAD 1 1 question answering data BIBREF7 can achieve F1 Score of 90 9 for Question Answering task where as BERT-Base Fine-tuned on the same SQuAD question answering BIBREF7 data could achieve F1 score of 88 5  One downside of the current version BioBERT is that word-piece vocabulary is the same as that of original BERT Model, as a result word-piece vocabulary does not include biomedical jargon  Lee et al  BIBREF0 created BioBERT, using the same pre-trained BERT released by Google, and hence in the word-piece vocabulary (vocab txt), as a result biomedical jargon is not included in word-piece vocabulary  Modifying word-piece vocabulary (vocab txt) at this stage would loose original compatibility with ‘BERT’, hence it is left unmodified 
In our future work we would like to build pre-trained ‘BERT’ model from scratch  We would pretrain the model with biomedical corpus (PubMed, ‘PMC’) and Wikipedia  Doing so would give us scope to create word piece vocabulary to include biomedical jargon and there are chances of model performing better with biomedical jargon being included in the word piece vocabulary  We will consider this scenario in the future, or wait for the next version of BioBERT 
Experiments: Factoid Question Answering Task
For Factoid Question Answering task, we fine tuned BioBERT BIBREF0 with question answering data and added new features  Fig  FIGREF4 shows the architecture of BioBERT fine tuned for question answering tasks: Input to BioBERT is word tokenized embeddings for question and the paragraph (Context)  As per the ‘BERT’ BIBREF1 standards, tokens ‘[CLS]’ and ‘[SEP]’ are appended to the tokenized input as illustrated in the figure  The resulting model has a softmax layer formed for predicting answer span indices in the given paragraph (Context)  On test data, the fine tuned model generates $n$-best predictions for each question  For a question, $n$-best corresponds that $n$ answers are returned as possible answers in the decreasing order of confidence  Variable $n$ is configurable  In our paper, any further mentions of ‘answer returned by the model’ correspond to the top answer returned by the model 
Experiments: Factoid Question Answering Task ::: Setup
BioASQ provides the training data  This data is based on previous BioASQ competitions  Train data we have considered is aggregate of all train data sets till the 5th version of BioASQ competition  We cleaned the data, that is, question-answering data without answers are removed and left with a total count of ‘530’ question answers  The data is split into train and test data in the ratio of 94 to 6; that is, count of '495' for training and '35' for testing 
The original data format is converted to the BERT/BioBERT format, where BioBERT expects ‘start_index’ of the actual answer  The ‘start_index corresponds to the index of the answer text present in the paragraph/ Context  For finding ‘start_index’ we used built-in python function find()  The function returns the lowest index of the actual answer present in the context(paragraph)  If the answer is not found ‘-1’ is returned as the index  The efficient way of finding start_index is, if the paragraph (Context) has multiple instances of answer text, then ‘start_index’ of the answer should be that instance of answer text whose context actually matches with what’s been asked in the question 
Example (Question, Answer and Paragraph from BIBREF8):
Question: Which drug should be used as an antidote in benzodiazepine overdose?
Answer: 'Flumazenil'
Paragraph(context):
"Flumazenil use in benzodiazepine overdose in the UK: a retrospective survey of NPIS data  OBJECTIVE: Benzodiazepine (BZD) overdose (OD) continues to cause significant morbidity and mortality in the UK  Flumazenil is an effective antidote but there is a risk of seizures, particularly in those who have co-ingested tricyclic antidepressants  A study was undertaken to examine the frequency of use, safety and efficacy of flumazenil in the management of BZD OD in the UK  METHODS: A 2-year retrospective cohort study was performed of all enquiries to the UK National Poisons Information Service involving BZD OD  RESULTS: Flumazenil was administered to 80 patients in 4504 BZD-related enquiries, 68 of whom did not have ventilatory failure or had recognised contraindications to flumazenil  Factors associated with flumazenil use were increased age, severe poisoning and ventilatory failure  Co-ingestion of tricyclic antidepressants and chronic obstructive pulmonary disease did not influence flumazenil administration  Seizure frequency in patients not treated with flumazenil was 0 3%" 
Actual answer is 'Flumazenil', but there are multiple instances of word 'Flu-mazenil'  Efficient way to identify the start-index for 'Flumazenil'(answer) is to find that particular instance of the word 'Flumazenil' which matches the context for the question  In the above example 'Flumazenil' highlighted in bold is the actual instance that matches question's context  Unfortunately, we could not identify readily available tools that can achieve this goal  In our future work, we look forward to handling these scenarios effectively 
Note: The creators of 'SQuAD' BIBREF7 have handled the task of identifying answer's start_index effectively  But 'SQuAD' data set is much more general and does not include biomedical question answering data 
Experiments: Factoid Question Answering Task ::: Training and error analysis
During our training with the BioASQ data, learning rate is set to 3e-5, as mentioned in the BioBERT paper BIBREF0  We started training the model with 495 available train data and 35 test data by setting number of epochs to 50  After training with these hyper-parameters training accuracy(exact match) was 99 3%(overfitting) and testing accuracy is only 4%  In the next iteration we reduced the number of epochs to 25 then training accuracy is reduced to 98 5% and test accuracy moved to 5%  We further reduced number of epochs to 15, and the resulting training accuracy was 70% and test accuracy 15%  In the next iteration set number of epochs to 12 and achieved train accuracy of 57 7% and test accuracy 23 3%  Repeated the experiment with 11 epochs and found training accuracy to be 57 7% and test accuracy to be same 22%  In the next iteration we set number of epochs to '9' and found training accuracy of 48% and test accuracy of 15%  Hence optimum number of epochs is taken as 12 epochs 
During our error analysis we found that on test data, model tends to return text in the beginning of the context(paragraph) as the answer  On analysing train data, we found that there are '120'(out of '495') question answering data instances having start_index:0, meaning 120( 25%) question answering data has first word(s) in the context(paragraph) as the answer  We removed 70% of those instances in order to make train data more balanced  In the new train data set we are left with '411' question answering data instances  This time we got the highest test accuracy of 26% at 11 epochs  We have submitted our results for BioASQ test batch-2, got strict accuracy of 32% and our system stood in 2nd place  Initially, hyper-parameter- 'batch size' is set to ‘400’  Later it is tuned to '32'  Although accuracy(exact answer match) remained at 26%, model generated concise and better answers at batch size ‘32’, that is wrong answers are close to the expected answer in good number of cases 
Example (from BIBREF8)
Question: Which mutated gene causes Chediak Higashi Syndrome?
Exact Answer: ‘lysosomal trafficking regulator gene’ 
The answer returned by a model trained at ‘400’ batch size is ‘Autosomal-recessive complicated spastic paraplegia with a novel lysosomal trafficking regulator’, and from the one trained at ‘32’ batch size is ‘lysosomal trafficking regulator’ 
In further experiments, we have fine tuned the BioBERT model with both ‘SQuAD’ dataset (version 2 0) and BioAsq train data  For training on ‘SQuAD’, hyper parameters- Learning rate and number of epochs are set to ‘3e-3’ and ‘3’ respectively as mentioned in the paper BIBREF1  Test accuracy of the model boosted to 44%  In one more experiment we trained model only on ‘SQuAD’ dataset, this time test accuracy of the model moved to 47%  The reason model did not perform up to the mark when trained with ‘SQuAD’ alongside BioASQ data could be that in formatted BioASQ data, start_index for the answer is not accurate, and affected the overall accuracy 
Our Systems and Their Performance on Factoid Questions
We have experimented with several systems and their variations, e g  created by training with specific additional features (see next subsection)  Here is their list and short descriptions  Unfortunately we did not pay attention to naming, and the systems evolved between test batches, so the overall picture can only be understood by looking at the details 
When we started the experiments our objective was to see whether BioBERT and entailment-based techniques can provide value for in the context of biomedical question answering  The answer to both questions was a yes, qualified by many examples clearly showing the limitations of both methods  Therefore we tried to address some of these limitations using feature engineering with mixed results: some clear errors got corrected and new errors got introduced, without overall improvement but convincing us that in future experiments it might be worth trying feature engineering again especially if more training data were available 
Overall we experimented with several approaches with the following aspects of the systems changing between batches, that is being absent or present:
training on BioAsq data vs  training on SQuAD
using the BioAsq snippets for context vs  using the documents from the provided URLs for context
adding or not the LAT, i e  lexical answer type, feature (see BIBREF9, BIBREF10 and an explanation in the subsection just below) 
For Yes/No questions (only) we experimented with the entailment methods 
We will discuss the performance of these models below and in Section 6  But before we do that, let us discuss a feature engineering experiment which eventually produced mixed results, but where we feel it is potentially useful in future experiments 
Our Systems and Their Performance on Factoid Questions ::: LAT Feature considered and its impact (slightly negative)
During error analysis we found that for some cases, answer being returned by the model is far away from what it is being asked in the Question 
Example: (from BIBREF8)
Question: Hy's law measures failure of which organ?
Actual Answer: ‘Liver’ 
The answer returned by one of our models was ‘alanine aminotransferase’, which is an enzyme  The model returns an enzyme, when the question asked for the organ name  To address this type of errors, we decided to try the concepts of ‘Lexical Answer Type’ (LAT) and Focus Word, which was used in IBM Watson, see BIBREF11 for overview; BIBREF10 for technical details, and BIBREF9 for details on question analysis  In an example given in the last source we read:
POETS & POETRY: He was a bank clerk in the Yukon before he published "Songs of a Sourdough" in 1907 
The focus is the part of the question that is a reference to the answer  In the example above, the focus is "he" 
LATs are terms in the question that indicate what type of entity is being asked for 
The headword of the focus is generally a LAT, but questions often contain additional LATs, and in the Jeopardy! domain, categories are an additional source of LATs 
(   ) In the example, LATs are "he", "clerk", and "poet" 
For example in the question "Which plant does oleuropein originate from?" (BIBREF8)  The LAT here is ‘plant’  For the BioAsq task we did not need to explicitly distinguish between the focus and the LAT concepts  In this example, the expectation is that answer returned by the model is a plant  Thus it is conceivable that the cosine distance between contextual embedding of word 'plant' in the question and contextual embedding for the answer present in the paragraph(context) is comparatively low  As a result model learns to adjust its weights during training phase and returns answers with low cosine distance with the LAT 
We used Stanford CoreNLP BIBREF12 library to write rules for extracting lexical answer type present in the question, both 'parts of speech'(POS) and dependency parsing functionality was used  We incorporated the Lexical Answer Type into one of our systems, UNCC_QA1 in Batch 4  This system underperformed our system FACTOIDS by about 3% in the MRR measure, but corrected errors such as in the example above 
Our Systems and Their Performance on Factoid Questions ::: LAT Feature considered and its impact (slightly negative) ::: Assumptions and rules for deriving lexical answer type 
There are different question types: ‘Which’, ‘What’, ‘When’, ‘How’ etc  Each type of question is being handled differently and there are commonalities among the rules written for different question types  Question words are identified through parts of speech tags: 'WDT', 'WRB' ,'WP'  We assumed that LAT is a ‘Noun’ and follows the question word  Often it was also a subject (nsubj)  This process is illustrated in Fig FIGREF15 
LAT computation was governed by a few simple rules, e g  when a question has multiple words that are 'Subjects’ (and ‘Noun’), a word that is in proximity to the question word is considered as ‘LAT’  These rules are different for each "Wh" word 
Namely, when the word immediately following the question word is a Noun, window size is set to ‘3’  The window size ‘3’ means we iterate through the next ‘3’ words to check if any of the word is both Noun and Subject, If so, such word is considered the ‘LAT’; else the word that is present very next to the question word is considered as the ‘LAT’ 
For questions with words ‘Which’ , ‘What’, ‘When’; a Noun immediately following the question word is very often the LAT, e g  'enzyme' in Which enzyme is targeted by Evolocumab?  When the word immediately following the question word is not a Noun, e g  in What is the function of the protein Magt1? the window size is set to ‘5’, and we iterate through the next ‘5’ words (if present) and search for the word that is both Noun and Subject
Context 3: Introduction
As social media, specially Twitter, takes on an influential role in presidential elections in the U S , natural language processing of political tweets BIBREF0 has the potential to help with nowcasting and forecasting of election results as well as identifying the main issues with a candidate – tasks of much interest to journalists, political scientists, and campaign organizers BIBREF1  As a methodology to obtain training data for a machine learning system that analyzes political tweets, BIBREF2 devised a crowdsourcing scheme with variable crowdworker numbers based on the difficulty of the annotation task  They provided a dataset of tweets where the sentiments towards political candidates were labeled both by experts in political communication and by crowdworkers who were likely not domain experts  BIBREF2 revealed that crowdworkers can match expert performance relatively accurately and in a budget-efficient manner  Given this result, the authors envisioned future work in which groundtruth labels would be crowdsourced for a large number of tweets and then used to design an automated NLP tool for political tweet analysis 
The question we address here is: How accurate are existing NLP tools for political tweet analysis? These tools would provide a baseline performance that any new machine learning system for political tweet analysis would compete against  We here explore whether existing NLP systems can answer the questions "What sentiment?" and "Towards whom?" accurately for the dataset of political tweets provided by BIBREF2  In our analysis, we include NLP tools with publicly-available APIs, even if the tools were not specifically designed for short texts like tweets, and, in particular, political tweets 
Our experiments reveal that the task of entity-level sentiment analysis is difficult for existing tools to answer accurately while the recognition of the entity, here, which politician, was easier 
NLP Toolkits
NLP toolkits typically have the following capabilities: tokenization, part-of-speech (PoS) tagging, chunking, named entity recognition and sentiment analysis  In a study by BIBREF3, it is shown that the well-known NLP toolkits NLTK BIBREF4, Stanford CoreNLP BIBREF5, and TwitterNLP BIBREF6 have tokenization, PoS tagging and NER modules in their pipelines  There are two main approaches for NER: (1) rule-based and (2) statistical or machine learning based  The most ubiquitous algorithms for sequence tagging use Hidden Markov Models BIBREF7, Maximum Entropy Markov Models BIBREF7, BIBREF8, or Conditional Random Fields BIBREF9  Recent works BIBREF10, BIBREF11 have used recurrent neural networks with attention modules for NER 
Sentiment detection tools like SentiStrength BIBREF12 and TensiStrength BIBREF13 are rule-based tools, relying on various dictionaries of emoticons, slangs, idioms, and ironic phrases, and set of rules that can detect the sentiment of a sentence overall or a targeted sentiment  Given a list of keywords, TensiStrength (similar to SentiStrength) reports the sentiment towards selected entities in a sentence, based on five levels of relaxation and five levels of stress 
Among commercial NLP toolkits (e g , BIBREF14, BIBREF15, BIBREF16), we selected BIBREF17 and BIBREF18 for our experiments, which, to the best of our knowledge, are the only publicly accessible commercial APIs for the task of entity-level sentiment analysis that is agnostic to the text domain  We also report results of TensiStrength BIBREF13, TwitterNLP BIBREF6, BIBREF19, CogComp-NLP BIBREF20, and Stanford NLP NER BIBREF21 
Dataset and Analysis Methodology
We used the 1,000-tweet dataset by BIBREF2 that contains the named-entities labels and entity-level sentiments for each of the four 2016 presidential primary candidates Bernie Sanders, Donald Trump, Hillary Clinton, and Ted Cruz, provided by crowdworkers, and by experts in political communication, whose labels are considered groundtruth  The crowdworkers were located in the US and hired on the BIBREF22 platform  For the task of entity-level sentiment analysis, a 3-scale rating of "negative," "neutral," and "positive" was used by the annotators 
BIBREF2 proposed a decision tree approach for computing the number of crowdworkers who should analyze a tweet based on the difficulty of the task  Tweets are labeled by 2, 3, 5, or 7 workers based on the difficulty of the task and the level of disagreement between the crowdworkers  The model computes the number of workers based on how long a tweet is, the presence of a link in a tweet, and the number of present sarcasm signals  Sarcasm is often used in political tweets and causes disagreement between the crowdworkers  The tweets that are deemed to be sarcastic by the decision tree model, are expected to be more difficult to annotate, and hence are allocated more crowdworkers to work on

ground_truth: pivot-based translation relying on a helping language BIBREF10, nduction of phrase tables from monolingual data BIBREF14 , attentional RNN-based model (RNMT) BIBREF2, Transformer model BIBREF18, bi-directional model BIBREF11, multi-to-multi (M2M) model BIBREF8, back-translation BIBREF17

M2M Transformer

prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. What was their highest MRR score?
Context 2:  If present, the word is considered as the ‘LAT’; else, the Noun in close proximity to the question word and following it is returned as the ‘LAT’ 
For questions with question words: ‘When’, ‘Who’, ‘Why’, the ’LAT’ is a question word itself  For the word ‘How', e g  in How many selenoproteins are encoded in the human genome?, we look at the adjective and if we find one, we take it to be the LAT, otherwise the word 'How' is considered as the ‘LAT’ 
Perhaps because of using only very simple rules, the accuracy for ‘LAT’ derivation is 75%; that is, in the remaining 25% of the cases the LAT word is identified incorrectly  Worth noting is that the overall performance the system that used LATs was slightly inferior to the system without LATs, but the types of errors changed  The training used BioBERT with the LAT feature as part of the input string  The errors it introduces usually involve finding the wrong element of the correct type e g  wrong enzyme when two similar enzymes are described in the text, or 'neuron' when asked about a type of cell with a certain function, when the answer calls for a different cell category, adipocytes, and both are mentioned in the text  We feel with more data and additional tuning or perhaps using an ensemble model, we might be able to keep the correct answers, and improve the results on the confusing examples like the one mentioned above  Therefore if we improve our ‘LAT’ derivation logic, or have larger datasets, then perhaps the neural network techniques they will yield better results 
Our Systems and Their Performance on Factoid Questions ::: Impact of Training using BioAsq data (slightly negative)
Training on BioAsq data in our entry in Batch 1 and Batch 2 under the name QA1 showed it might lead to overfitting  This happened both with (Batch 2) and without (Batch 1) hyperparameters tuning: abysmal 18% MRR in Batch 1, and slighly better one, 40% in Batch 2 (although in Batch 2 it was overall the second best result in MRR but 16% lower than the highest score) 
In Batch 3 (only), our UNCC_QA3 system was fine tuned on BioAsq and SQuAD 2 0 BIBREF7, and for data preprocessing Context paragraph is generated from relevant snippets provided in the test data  This system underperformed, by about 2% in MRR, our other entry UNCC_QA1, which was also an overall category winner for this batch  The latter was also trained on SQuAD, but not on BioAsq  We suspect that the reason could be the simplistic nature of the find() function described in Section 3 1  So, this could be an area where a better algorithm for finding the best occurrence of an entity could improve performance 
Our Systems and Their Performance on Factoid Questions ::: Impact of Using Context from URLs (negative)
In some experiments, for context in testing, we used documents for which URL pointers are provided in BioAsq  However, our system UNCC_QA3 underperformed our other system tested only on the provided snippets 
In Batch 5 the underperformance was about 6% of MRR, compared to our best system UNCC_QA1, and by 9% to the top performer 
Performance on Yes/No and List questions
Our work focused on Factoid questions  But we also have done experiments on List-type and Yes/No questions 
Performance on Yes/No and List questions ::: Entailment improves Yes/No accuracy
We started by answering always YES (in batch 2 and 3) to get the baseline performance  For batch 4 we used entailment  Our algorithm was very simple: Given a question we iterate through the candidate sentences and try to find any candidate sentence is contradicting the question (with confidence over 50%), if so 'No' is returned as answer, else 'Yes' is returned  In batch 4 this strategy produced better than the BioAsq baseline performance, and compared to our other systems, the use of entailment increased the performance by about 13% (macro F1 score)  We used 'AllenNlp' BIBREF13 entailment library to find entailment of the candidate sentences with question 
Performance on Yes/No and List questions ::: For List-type the URLs have negative impact
Overall, we followed the similar strategy that's been followed for Factoid Question Answering task  We started our experiment with batch 2, where we submitted 20 best answers (with context from snippets)  Starting with batch 3, we performed post processing: once models generate answer predictions (n-best predictions), we do post-processing on the predicted answers  In test batch 4, our system (called FACTOIDS) achieved highest recall score of ‘0 7033’ but low precision of 0 1119, leaving open the question of how could we have better balanced the two measures 
In the post-processing phase, we take the top ‘20’ (batch 3) and top 5 (batch 4 and 5), predicted answers, tokenize them using common separators: 'comma' , 'and', 'also', 'as well as'  Tokens with characters count more than ‘100’ are eliminated and rest of the tokens are added to the list of possible answers  BioASQ evaluation mechanism does not consider snippets with more than ‘100’ characters as a valid answer  Considering lengthy snippets in to the list of answers would reduce the mean precision score  As a final step, duplicate snippets in the answer pool are removed  For example, consider these top 3 answers predicted by the system (before post-processing):
{
"text": "dendritic cells",
"probability": 0 7554540733426441,
"start_logit": 8 466046333312988,
"end_logit": 9 536355018615723
},
{
"text": "neutrophils, macrophages and
distinct subtypes of dendritic cells",
"probability": 0 13806867348304214,
"start_logit": 6 766478538513184,
"end_logit": 9 536355018615723
},
{
"text": "macrophages and distinct subtypes of dendritic",
"probability": 0 013973475271178242,
"start_logit": 6 766478538513184,
"end_logit": 7 24576473236084
},
After execution of post-processing heuristics, the list of answers returned is as follows:
["dendritic cells"],
["neutrophils"],
["macrophages"],
["distinct subtypes of dendritic cells"]
Summary of our results
The tables below summarize all our results  They show that the performance of our systems was mixed  The simple architectures and algorithm we used worked very well only in Batch 3  However, we feel we can built a better system based on this experience  In particular we observed both the value of contextual embeddings and of feature engineering (LAT), however we failed to combine them properly 
Summary of our results ::: Factoid questions ::: Systems used in Batch 5 experiments
System description for ‘UNCC_QA1’: The system was finetuned on the SQuAD 2 0  For data preprocessing Context / paragraph was generated from relevant snippets provided in the test data 
System description for ‘QA1’ : ‘LAT’ feature was added and finetuned with SQuAD 2 0  For data preprocessing Context / paragraph was generated from relevant snippets provided in the test data 
System description for ‘UNCC_QA3’ : Fine tuning process is same as it is done for the system ‘UNCC_QA1’ in test batch-5  Difference is during data preprocessing, Context/paragraph is generated from the relevant documents for which URLS are included in the test data 
Summary of our results ::: List Questions
For List-type questions, although post processing helped in the later batches, we never managed to obtain competitive precision, although our recall was good 
Summary of our results ::: Yes/No questions
The only thing worth remembering from our performance is that using entailment can have a measurable impact (at least with respect to a weak baseline)  The results (weak) are in Table 3 
Discussion, Future Experiments, and Conclusions ::: Summary:
In contrast to 2018, when we submitted BIBREF2 to BioASQ a system based on extractive summarization (and scored very high in the ideal answer category), this year we mainly targeted factoid question answering task and focused on experimenting with BioBERT  After these experiments we see the promise of BioBERT in QA tasks, but we also see its limitations  The latter we tried to address with mixed results using feature engineering  Overall these experiments allowed us to secure a best and a second best score in different test batches  Along with Factoid-type question, we also tried ‘Yes/No’ and ‘List’-type questions, and did reasonably well with our very simple approach 
For Yes/No the moral worth remembering is that reasoning has a potential to influence results, as evidenced by our adding the AllenNLP entailment BIBREF13 system increased its performance 
All our data and software is available at Github, in the previously referenced URL (end of Section 2) 
Discussion, Future Experiments, and Conclusions ::: Future experiments
In the current model, we have a shallow neural network with a softmax layer for predicting answer span  Shallow networks however are not good at generalizations  In our future experiments we would like to create dense question answering neural network with a softmax layer for predicting answer span  The main idea is to get contextual word embedding for the words present in the question and paragraph (Context) and feed the contextual word embeddings retrieved from the last layer of BioBERT to the dense question answering network  The mentioned dense layered question answering neural network need to be tuned for finding right hyper parameters  An example of such architecture is shown in Fig FIGREF30 
In one more experiment, we would like to add a better version of ‘LAT’ contextual word embedding as a feature, along with the actual contextual word embeddings for question text, and Context and feed them as input to the dense question answering neural network  By this experiment, we would like to find if ‘LAT’ feature is improving overall answer prediction accuracy  Adding ‘LAT’ feature this way instead of feeding this word piece embedding directly to the BioBERT (as we did in our above experiments) would not downgrade the quality of contextual word embeddings generated form ‘BioBERT'  Quality contextual word embeddings would lead to efficient transfer learning and chances are that it would improve the model's answer prediction accuracy 
We also see potential for incorporating domain specific inference into the task e g  using the MedNLI dataset BIBREF14  For all types of experiments it might be worth exploring clinical BERT embeddings BIBREF15, explicitly incorporating domain knowledge (e g  BIBREF16) and possibly deeper discourse representations (e g  BIBREF17) 
APPENDIX
In this appendix we provide additional details about the implementations 
APPENDIX ::: Systems and their descriptions:
We used several variants of our systems when experimenting with the BioASQ problems  In retrospect, it would be much easier to understand the changes if we adopted some mnemonic conventions in naming the systems  So, we apologize for the names that do not reflect the modifications, and necessitate this list 
APPENDIX ::: Systems and their descriptions: ::: Factoid Type Question Answering:
We preprocessed the test data to convert test data to BioBERT format, We generated Context/paragraph by either aggregating relevant snippets provided or by aggregating documents for which URLS are provided in the BioASQ test data 
APPENDIX ::: Systems and their descriptions: ::: System description for QA1:
We generated Context/paragraph by aggregating relevant snippets available in the test data and mapped it against the question text and question id  We ignored the content present in the documents (document URLS were provided in the original test data)  The model is finetuned with BioASQ data 
data preprocessing is done in the same way as it is done for test batch-1  Model fine tuned on BioASQ data 
‘LAT’/ Focus word feature added and fine tuned with SQuAD 2 0 [reference]  For data preprocessing Context / paragraph is generated from relevant snippets provided in the test data 
APPENDIX ::: Systems and their descriptions: ::: System description for UNCC_QA_1:
System is finetuned on the SQuAD 2 0 [reference]  For data preprocessing Context / paragraph is generated from relevant snippets provided in the test data 
‘LAT’/ Focus word feature added and fine tuned with SQuAD 2 0 [reference]  For data preprocessing Context / paragraph is generated from relevant snippets provided in the test data 
The System is finetuned on the SQuAD 2 0  For data preprocessing Context / paragraph is generated from relevant snippets provided in the test data 
APPENDIX ::: Systems and their descriptions: ::: System description for UNCC_QA3:
System is finetuned on the SQuAD 2 0 [reference] and BioASQ dataset[] For data preprocessing Context / paragraph is generated from relevant snippets provided in the test data 
Fine tuning process is same as it is done for the system ‘UNCC_QA_1’ in test batch-5  Difference is during data preprocessing, Context/paragraph is generated form from the relevant documents for which URLS are included in the test data 
APPENDIX ::: Systems and their descriptions: ::: System description for UNCC_QA2:
Fine tuning process is same as for ‘UNCC_QA_1 ’  Difference is Context/paragraph is generated form from the relevant documents for which URLS are included in the test data  System ‘UNCC_QA_1’ got the highest ‘MRR’ score in the 3rd test batch set 
APPENDIX ::: Systems and their descriptions: ::: System description for FACTOIDS:
The System is finetuned on the SQuAD 2 0  For data preprocessing Context / paragraph is generated from relevant snippets provided in the test data 
APPENDIX ::: Systems and their descriptions: ::: List Type Questions:
We attempted List type questions starting from test batch ‘2’  Used similar approach that's been followed for Factoid Question answering task  For all the test batch sets, in the data pre processing phase Context/ paragraph is generated either by aggregating relevant snippets or by aggregating documents(URLS) provided in the BioASQ test data 
For test batch-2, model (System: QA1) is finetuned on BioASQ data and submitted top ‘20’ answers predicted by the model as the list of answers  system ‘QA1’ achieved low F-Measure score:‘0 0786’ in the second test batch  In the further test batches for List type questions, we finetuned the model on Squad data set [reference], implemented post processing techniques (refer section 5 2) and achieved a better F-measure score: ‘0 2862’ in the final test batch set 
In test batch-3 (Systems : ‘QA1’/’’UNCC_QA_1’/’UNCC_QA3’/’UNCC_QA2’) top 20 answers returned by the model is sent for post processing and in test batch 4 and 5 only top 5 answers are sent for post processing  System UNCC_QA2(in batch 3) for List type question answering, Context is generated from documents for which URLS are provided in the BioASQ test data  for the rest of the systems (in test batch-3) for List Type question answering task snippets present in the BioaSQ test data are used to generate context 
In test batch-4 (System : ‘FACTOIDS’/’UNCC_QA_1’/’UNCC_QA3’’) top 5 answers returned by the model is sent for post processing  In case of system ‘FACTOIDS’ snippets in the test data were used to generate context  for systems ’UNCC_QA_1’ and ’UNCC_QA3’ context is generated from the documents for which URLS are provided in the BioASQ test data 
In test batch-5 ( Systems: ‘QA1’/’UNCC_QA_1’/’UNCC_QA3’/’UNCC_QA2’ ) our approach is the same as that of test batch-4 where top 5 answers returned by the model is sent for post processing  for all the systems (in test batch-5) context is generated from the snippets provided in the BioASQ test data 
APPENDIX ::: Systems and their descriptions: ::: Yes/No Type Questions:
For the first 3 test batches, We have submitted answer ‘Yes’ to all the questions  Later, we employed ‘Sentence Entailment’ techniques(refer section 6 0) for the fourth and fifth test batch sets  Our Systems with ‘Sentence Entailment’ approach (for ‘Yes’/ ‘No’ question answering): ‘UNCC_QA_1’(test batch-4), UNCC_QA3(test batch-5) 
APPENDIX ::: Additional details for Yes/No Type Questions
We used Textual Entailment in Batch 4 and 5 for ‘Yes’/‘No’ question type  The algorithm was very simple: Given a question we iterate through the candidate sentences, and look for any candidate sentences contradicting the question  If we find one 'No' is returned as answer, else 'Yes' is returned  (The confidence for contradiction was set at 50%) We used AllenNLP BIBREF13 entailment library to find entailment of the candidate sentences with question 
Flow Chart for Yes/No Question answer processing is shown in Fig FIGREF51
APPENDIX ::: Assumptions, rules and logic flow for deriving Lexical Answer Types from questions
There are different question types, and we distinguished them based on the question words: ‘Which’, ‘What’, ‘When’, ‘How’ etc  Each type of question is being handled differently and there are commonalities among the rules written for different question types  How are question words identified? question words have parts of speech(POS): 'WDT', 'WRB', 'WP' 
Assumptions:
1) Lexical answer type (‘LAT’) or focus word is of type Noun and follows the question word 
2) The LAT word is a Subject  (This clearly not always true, but we used a very simple method)  Note: ‘StanfordNLP’ dependency parsing tag for identifying subject is 'nsubj' or 'nsubjpass' 
3) When a question has multiple words that are of type Subject (and Noun), a word that is in proximity to the question word is considered as ‘LAT’ 
4) For questions with question words: ‘When’, ‘Who’, ‘Why’, the ’LAT’ is a question word itself that is, ‘When’, ‘Who’, ‘Why’ respectively 
Rules and logic flow to traverse a question: The three cases below describe the logic flow of finding LATs  The figures show the grammatical structures used for this purpose 
APPENDIX ::: Assumptions, rules and logic flow for deriving Lexical Answer Types from questions ::: Case-1:
Question with question word ‘How’ 
For questions with question word 'How', the adjective that follows the question word is considered as ‘LAT’ (need not follow immediately)  If an adjective is absent, word 'How' is considered as ‘LAT’  When there are multiple words that are adjectives, a word in close proximity to the question word and follows it is returned as ‘LAT’  Note: The part of speech tag to identify adjectives is 'JJ'  For Other possible question words like ‘whose’  ‘LAT’/Focus word is question words itself 
Example Question: How many selenoproteins are encoded in the human genome?
APPENDIX ::: Assumptions, rules and logic flow for deriving Lexical Answer Types from questions ::: Case-2:
Questions with question words ‘Which’ , ‘What’ and all other possible question words; a 'Noun' immediately following the question word 
Example Question: Which enzyme is targeted by Evolocumab?
Here, Focus word/LAT is ‘enzyme’ which is both Noun and Subject and immediately follows the question word 
When the word immediately following the question word is a noun, the window size is set to ‘3’  This size ‘3’ means that we iterate through the next ‘3’ words (if present) to check if any of the word is both 'Noun' and 'Subject', If so, the word is considered as ‘LAT’/Focus Word  Else the word that is present very next to the question word is considered as ‘LAT’ 
APPENDIX ::: Assumptions, rules and logic flow for deriving Lexical Answer Types from questions ::: Case-3:
Questions with question words ‘Which’ , ‘What’ and all other possible question words; word immediately following the question word is not a 'Noun' 
Example Question: What is the function of the protein Magt1?
Here, Focus word/LAT is ‘function ’ which is both Noun and Subject and does not immediately follow the question word 
When the very next word following the question word is not a Noun, window size is set to ‘5’  Window size ‘5’ corresponds that we iterate through the next ‘5’ words (if present) and search for the word that is both Noun and Subject  If present, the word is considered as ‘LAT’  Else, the 'Noun' close proximity to the question word and follows it is returned as ‘LAT’ 
Ad we mentioned earlier, the accuracy for ‘LAT’ derivation is 75 percent  But clearly the simple logic described above can be improved, as shown in BIBREF9, BIBREF10  Whether this in turn produces improvements in this particular task is an open question 
APPENDIX ::: Proposing Future Experiments
In the current model, we have a shallow neural network with a softmax layer for predicting answer span  Shallow networks however are not good at generalizations  In our future experiments we would like to create dense question answering neural network with a softmax layer for predicting answer span  The main idea is to get contextual word embedding for the words present in the question and paragraph (Context) and feed the contextual word embeddings retrieved from the last layer of BioBERT to the dense question answering network  The mentioned dense layered question answering Neural network need to be tuned for finding right hyper parameters  An example of such architecture is shown in Fig FIGREF30 
In another experiment we would like to only feed contextual word embeddings for Focus word/ ‘LAT’, paragraph/ Context as input to the question answering neural network  In this experiment we would neglect all embeddings for the question text except that of Focus word/ ‘LAT’  Our assumption and idea for considering focus word and neglecting remaining words in the question is that during training phase it would make more precise for the model to identify the focus of the question and map answers against the question’s focus  To validate our assumption, we would like to take sample question answering data and find the cosine distance between contextual embedding of Focus word and that of the actual answer and verify if the cosine distance is comparatively low in most of the cases 
In one more experiment, we would like to add a better version of ‘LAT’ contextual word embedding as a feature, along with the actual contextual word embeddings for question text, and Context and feed them as input to the dense question answering neural network  By this experiment, we would like to find if ‘LAT’ feature is improving overall answer prediction accuracy  Adding ‘LAT’ feature this way instead of feeding Focus word’s word piece embedding directly (as we did in our above experiments) to the BioBERT would not downgrade the quality of contextual word embeddings generated form ‘BioBERT'  Quality contextual word embeddings would lead to efficient transfer learning and chances are that it would improve the model's answer prediction accuracy 
Context 3: 
We conducted two sets of experiments  In the first set, we used BIBREF23, BIBREF17, and BIBREF18, for entity-level sentiment analysis; in the second set, BIBREF17, BIBREF19, BIBREF24, BIBREF25, and BIBREF26, BIBREF18 for named-entity recognition 
In the experiments that we conducted with TwitterNLP for named-entity recognition, we worked with the default values of the model  Furthermore, we selected the 3-class Stanford NER model, which uses the classes “person,” “organization,” and “location” because it resulted in higher accuracy compared to the 7-class model  For CogComp-NLP NER we used Ontonotes 5 0 NER model BIBREF27  For spaCy NER we used the `en_core_web_lg' model 
We report the experimental results for our two tasks in terms of the correct classification rate (CCR)  For sentiment analysis, we have a three-class problem (positive, negative, and neutral), where the classes are mutually exclusive  The CCR, averaged for a set of tweets, is defined to be the number of correctly-predicted sentiments over the number of groundtruth sentiments in these tweets  For NER, we consider that each tweet may reference up to four candidates, i e , targeted entities  The CCR, averaged for a set of tweets, is the number of correctly predicted entities (candidates) over the number of groundtruth entities (candidates) in this set 
Results and Discussion
The dataset of 1,000 randomly selected tweets contains more than twice as many tweets about Trump than about the other candidates  In the named-entity recognition experiment, the average CCR of crowdworkers was 98 6%, while the CCR of the automated systems ranged from 77 2% to 96 7%  For four of the automated systems, detecting the entity Trump was more difficult than the other entities (e g , spaCy 72 7% for the entity Trump vs  above 91% for the other entities)  An example of incorrect NER is shown in Figure FIGREF1 top  The difficulties the automated tools had in NER may be explained by the fact that the tools were not trained on tweets, except for TwitterNLP, which was not in active development when the data was created BIBREF1 
In the sentiment analysis experiments, we found that a tweet may contain multiple sentiments  The groundtruth labels contain 210 positive sentiments, 521 neutral sentiments, and 305 negative sentiments to the candidates  We measured the CCR, across all tweets, to be 31 7% for Rosette Text Analytics, 43

ground_truth: 0.5115

0.6103

prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. Which toolkits do they use?
Context 3: 2% for Google Cloud, 44 2% for TensiStrength, and 74 7% for the crowdworkers  This means the difference between the performance of the tools and the crowdworkers is significant – more than 30 percent points 
Crowdworkers correctly identified 62% of the neutral, 85% of the positive, and 92% of the negative sentiments  Google Cloud correctly identified 88% of the neutral sentiments, but only 3% of the positive, and 19% of the negative sentiments  TensiStrength correctly identified 87 2% of the neutral sentiments, but 10 5% of the positive, and 8 1% of the negative sentiments  Rosette Text Analytics correctly identified 22 7% of neutral sentiments, 38 1% of negative sentiments and 40 9% of positive sentiments  The lowest and highest CCR pertains to tweets about Trump and Sanders for both Google Cloud and TensiStrength, Trump and Clinton for Rosette Text Analytics, and Clinton and Cruz for crowdworkers  An example of incorrect ELS analysis is shown in Figure FIGREF1 bottom 
Conclusions and Future Work
Our results show that existing NLP systems cannot accurately perform sentiment analysis of political tweets in the dataset we experimented with  Labeling by humans, even non-expert crowdworkers, yields accuracy results that are well above the results of existing automated NLP systems  In future work we will therefore use a crowdworker-labeled dataset to train a new machine-learning based NLP system for tweet analysis  We will ensure that the training data is balanced among classes  Our plan is to use state-of-the-art deep neural networks and compare their performance for entity-level sentiment analysis of political tweets 
Acknowledgments
Partial support of this work by the Hariri Institute for Computing and Computational Science & Engineering at Boston University (to L G ) and a Google Faculty Research Award (to M B  and L G ) is gratefully acknowledged  Additionally, we would like to thank Daniel Khashabi for his help in running the CogComp-NLP Python API and Mike Thelwal for his help with TensiStrength  We are also grateful to the Stanford NLP group for clarifying some of the questions we had with regards to the Stanford NER tool 

ground_truth: BIBREF17, BIBREF18, TensiStrength BIBREF13, TwitterNLP BIBREF6, BIBREF19, CogComp-NLP BIBREF20, Stanford NLP NER BIBREF21

BIBREF23, BIBREF17, BIBREF18, BIBREF19, BIBREF24, BIBREF25, BIBREF26