prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. How was the training data translated?
Context 1: Introduction
Understanding the emotions expressed in a text or message is of high relevance nowadays  Companies are interested in this to get an understanding of the sentiment of their current customers regarding their products and the sentiment of their potential customers to attract new ones  Moreover, changes in a product or a company may also affect the sentiment of a customer  However, the intensity of an emotion is crucial in determining the urgency and importance of that sentiment  If someone is only slightly happy about a product, is a customer willing to buy it again? Conversely, if someone is very angry about customer service, his or her complaint might be given priority over somewhat milder complaints 
BIBREF0 present four tasks in which systems have to automatically determine the intensity of emotions (EI) or the intensity of the sentiment (Valence) of tweets in the languages English, Arabic, and Spanish  The goal is to either predict a continuous regression (reg) value or to do ordinal classification (oc) based on a number of predefined categories  The EI tasks have separate training sets for four different emotions: anger, fear, joy and sadness  Due to the large number of subtasks and the fact that this language does not have many resources readily available, we only focus on the Spanish subtasks  Our work makes the following contributions:
Our submissions ranked second (EI-Reg), second (EI-Oc), fourth (V-Reg) and fifth (V-Oc), demonstrating that the proposed method is accurate in automatically determining the intensity of emotions and sentiment of Spanish tweets  This paper will first focus on the datasets, the data generation procedure, and the techniques and tools used  Then we present the results in detail, after which we perform a small error analysis on the largest mistakes our model made  We conclude with some possible ideas for future work 
Data
For each task, the training data that was made available by the organizers is used, which is a selection of tweets with for each tweet a label describing the intensity of the emotion or sentiment BIBREF1   Links and usernames were replaced by the general tokens URL and @username, after which the tweets were tokenized by using TweetTokenizer  All text was lowercased  In a post-processing step, it was ensured that each emoji is tokenized as a single token 
Word Embeddings
To be able to train word embeddings, Spanish tweets were scraped between November 8, 2017 and January 12, 2018  We chose to create our own embeddings instead of using pre-trained embeddings, because this way the embeddings would resemble the provided data set: both are based on Twitter data  Added to this set was the Affect in Tweets Distant Supervision Corpus (DISC) made available by the organizers BIBREF0 and a set of 4 1 million tweets from 2015, obtained from BIBREF2   After removing duplicate tweets and tweets with fewer than ten tokens, this resulted in a set of 58 7 million tweets, containing 1 1 billion tokens  The tweets were preprocessed using the method described in Section SECREF6   The word embeddings were created using word2vec in the gensim library BIBREF3 , using CBOW, a window size of 40 and a minimum count of 5  The feature vectors for each tweet were then created by using the AffectiveTweets WEKA package BIBREF4  
Translating Lexicons
Most lexical resources for sentiment analysis are in English  To still be able to benefit from these sources, the lexicons in the AffectiveTweets package were translated to Spanish, using the machine translation platform Apertium BIBREF5  
All lexicons from the AffectiveTweets package were translated, except for SentiStrength  Instead of translating this lexicon, the English version was replaced by the Spanish variant made available by BIBREF6  
For each subtask, the optimal combination of lexicons was determined  This was done by first calculating the benefits of adding each lexicon individually, after which only beneficial lexicons were added until the score did not increase anymore (e g  after adding the best four lexicons the fifth one did not help anymore, so only four were added)  The tests were performed using a default SVM model, with the set of word embeddings described in the previous section  Each subtask thus uses a different set of lexicons (see Table TABREF1 for an overview of the lexicons used in our final ensemble)  For each subtask, this resulted in a (modest) increase on the development set, between 0 01 and 0 05 
Translating Data
The training set provided by BIBREF0 is not very large, so it was interesting to find a way to augment the training set  A possible method is to simply translate the datasets into other languages, leaving the labels intact  Since the present study focuses on Spanish tweets, all tweets from the English datasets were translated into Spanish  This new set of “Spanish” data was then added to our original training set  Again, the machine translation platform Apertium BIBREF5 was used for the translation of the datasets 
Algorithms Used
Three types of models were used in our system, a feed-forward neural network, an LSTM network and an SVM regressor  The neural nets were inspired by the work of Prayas BIBREF7 in the previous shared task  Different regression algorithms (e g  AdaBoost, XGBoost) were also tried due to the success of SeerNet BIBREF8 , but our study was not able to reproduce their results for Spanish 
For both the LSTM network and the feed-forward network, a parameter search was done for the number of layers, the number of nodes and dropout used  This was done for each subtask, i e  different tasks can have a different number of layers  All models were implemented using Keras BIBREF9   After the best parameter settings were found, the results of 10 system runs to produce our predictions were averaged (note that this is different from averaging our different type of models in Section SECREF16 )  For the SVM (implemented in scikit-learn BIBREF10 ), the RBF kernel was used and a parameter search was conducted for epsilon  Detailed parameter settings for each subtask are shown in Table TABREF12   Each parameter search was performed using 10-fold cross validation, as to not overfit on the development set 
Semi-supervised Learning
One of the aims of this study was to see if using semi-supervised learning is beneficial for emotion intensity tasks  For this purpose, the DISC BIBREF0 corpus was used  This corpus was created by querying certain emotion-related words, which makes it very suitable as a semi-supervised corpus  However, the specific emotion the tweet belonged to was not made public  Therefore, a method was applied to automatically assign the tweets to an emotion by comparing our scraped tweets to this new data set 
First, in an attempt to obtain the query-terms, we selected the 100 words which occurred most frequently in the DISC corpus, in comparison with their frequencies in our own scraped tweets corpus  Words that were clearly not indicators of emotion were removed  The rest was annotated per emotion or removed if it was unclear to which emotion the word belonged  This allowed us to create silver datasets per emotion, assigning tweets to an emotion if an annotated emotion-word occurred in the tweet 
Our semi-supervised approach is quite straightforward: first a model is trained on the training set and then this model is used to predict the labels of the silver data  This silver data is then simply added to our training set, after which the model is retrained  However, an extra step is applied to ensure that the silver data is of reasonable quality  Instead of training a single model initially, ten different models were trained which predict the labels of the silver instances  If the highest and lowest prediction do not differ more than a certain threshold the silver instance is maintained, otherwise it is discarded 
This results in two parameters that could be optimized: the threshold and the number of silver instances that would be added  This method can be applied to both the LSTM and feed-forward networks that were used  An overview of the characteristics of our data set with the final parameter settings is shown in Table TABREF14   Usually, only a small subset of data was added to our training set, meaning that most of the silver data is not used in the experiments  Note that since only the emotions were annotated, this method is only applicable to the EI tasks 
Ensembling
To boost performance, the SVM, LSTM, and feed-forward models were combined into an ensemble  For both the LSTM and feed-forward approach, three different models were trained  The first model was trained on the training data (regular), the second model was trained on both the training and translated training data (translated) and the third one was trained on both the training data and the semi-supervised data (silver)  Due to the nature of the SVM algorithm, semi-supervised learning does not help, so only the regular and translated model were trained in this case  This results in 8 different models per subtask  Note that for the valence tasks no silver training data was obtained, meaning that for those tasks the semi-supervised models could not be used 
Per task, the LSTM and feed-forward model's predictions were averaged over 10 prediction runs  Subsequently, the predictions of all individual models were combined into an average  Finally, models were removed from the ensemble in a stepwise manner if the removal increased the average score  This was done based on their original scores, i e  starting out by trying to remove the worst individual model and working our way up to the best model  We only consider it an increase in score if the difference is larger than 0 002 (i e  the difference between 0 716 and 0 718)  If at some point the score does not increase and we are therefore unable to remove a model, the process is stopped and our best ensemble of models has been found  This process uses the scores on the development set of different combinations of models  Note that this means that the ensembles for different subtasks can contain different sets of models  The final model selections can be found in Table TABREF17  
Results and Discussion
Table TABREF18 shows the results on the development set of all individuals models, distinguishing the three types of training: regular (r), translated (t) and semi-supervised (s)  In Tables TABREF17 and TABREF18 , the letter behind each model (e g  SVM-r, LSTM-r) corresponds to the type of training used  Comparing the regular and translated columns for the three algorithms, it shows that in 22 out of 30 cases, using translated instances as extra training data resulted in an improvement  For the semi-supervised learning approach, an improvement is found in 15 out of 16 cases  Moreover, our best individual model for each subtask (bolded scores in Table TABREF18 ) is always either a translated or semi-supervised model  Table TABREF18 also shows that, in general, our feed-forward network obtained the best results, having the highest F-score for 8 out of 10 subtasks 
However, Table TABREF19 shows that these scores can still be improved by averaging or ensembling the individual models  On the dev set, averaging our 8 individual models results in a better score for 8 out of 10 subtasks, while creating an ensemble beats all of the individual models as well as the average for each subtask  On the test set, however, only a small increase in score (if any) is found for stepwise ensembling, compared to averaging  Even though the results do not get worse, we cannot conclude that stepwise ensembling is a better method than simply averaging 
Our official scores (column Ens Test in Table TABREF19 ) have placed us second (EI-Reg, EI-Oc), fourth (V-Reg) and fifth (V-Oc) on the SemEval AIT-2018 leaderboard  However, it is evident that the results obtained on the test set are not always in line with those achieved on the development set  Especially on the anger subtask for both EI-Reg and EI-Oc, the scores are considerably lower on the test set in comparison with the results on the development set  Therefore, a small error analysis was performed on the instances where our final model made the largest errors 
Error Analysis
Due to some large differences between our results on the dev and test set of this task, we performed a small error analysis in order to see what caused these differences  For EI-Reg-anger, the gold labels were compared to our own predictions, and we manually checked 50 instances for which our system made the largest errors 
Some examples that were indicative of the shortcomings of our system are shown in Table TABREF20   First of all, our system did not take into account capitalization  The implications of this are shown in the first sentence, where capitalization intensifies the emotion used in the sentence  In the second sentence, the name Imperator Furiosa is not understood  Since our texts were lowercased, our system was unable to capture the named entity and thought the sentence was about an angry emperor instead  In the third sentence, our system fails to capture that when you are so angry that it makes you laugh, it results in a reduced intensity of the angriness  Finally, in the fourth sentence, it is the figurative language me infla la vena (it inflates my vein) that the system is not able to understand 
The first two error-categories might be solved by including smart features regarding capitalization and named entity recognition  However, the last two categories are problems of natural language understanding and will be very difficult to fix 
Conclusion
To conclude, the present study described our submission for the Semeval 2018 Shared Task on Affect in Tweets  We participated in four Spanish subtasks and our submissions ranked second, second, fourth and fifth place  Our study aimed to investigate whether the automatic generation of additional training data through translation and semi-supervised learning, as well as the creation of stepwise ensembles, increase the performance of our Spanish-language models  Strong support was found for the translation and semi-supervised learning approaches; our best models for all subtasks use either one of these approaches  These results suggest that both of these additional data resources are beneficial when determining emotion intensity (for Spanish)  However, the creation of a stepwise ensemble from the best models did not result in better performance compared to simply averaging the models  In addition, some signs of overfitting on the dev set were found  In future work, we would like to apply the methods (translation and semi-supervised learning) used on Spanish on other low-resource languages and potentially also on other tasks 
Context 2: Introduction
End-to-end speech-to-text translation (ST) has attracted much attention recently BIBREF2, BIBREF3, BIBREF4, BIBREF5, BIBREF6 given its simplicity against cascading automatic speech recognition (ASR) and machine translation (MT) systems  The lack of labeled data, however, has become a major blocker for bridging the performance gaps between end-to-end models and cascading systems  Several corpora have been developed in recent years  post2013improved introduced a 38-hour Spanish-English ST corpus by augmenting the transcripts of the Fisher and Callhome corpora with English translations  di-gangi-etal-2019-must created the largest ST corpus to date from TED talks but the language pairs involved are out of English only  beilharz2019librivoxdeen created a 110-hour German-English ST corpus from LibriVox audiobooks  godard-etal-2018-low created a Moboshi-French ST corpus as part of a rare language documentation effort  woldeyohannis provided an Amharic-English ST corpus in the tourism domain  boito2019mass created a multilingual ST corpus involving 8 languages from a multilingual speech corpus based on Bible readings BIBREF7  Previous work either involves language pairs out of English, very specific domains, very low resource languages or a limited set of language pairs  This limits the scope of study, including the latest explorations on end-to-end multilingual ST BIBREF8, BIBREF9  Our work is mostly similar and concurrent to iranzosnchez2019europarlst who created a multilingual ST corpus from the European Parliament proceedings  The corpus we introduce has larger speech durations and more translation tokens  It is diversified with multiple speakers per transcript/translation  Finally, we provide additional out-of-domain test sets 
In this paper, we introduce CoVoST, a multilingual ST corpus based on Common Voice BIBREF10 for 11 languages into English, diversified with over 11,000 speakers and over 60 accents  It includes a total 708 hours of French (Fr), German (De), Dutch (Nl), Russian (Ru), Spanish (Es), Italian (It), Turkish (Tr), Persian (Fa), Swedish (Sv), Mongolian (Mn) and Chinese (Zh) speeches, with French and German ones having the largest durations among existing public corpora  We also collect an additional evaluation corpus from Tatoeba for French, German, Dutch, Russian and Spanish, resulting in a total of 9 3 hours of speech  Both corpora are created at the sentence level and do not require additional alignments or segmentation  Using the official Common Voice train-development-test split, we also provide baseline models, including, to our knowledge, the first end-to-end many-to-one multilingual ST models  CoVoST is released under CC0 license and free to use  The Tatoeba evaluation samples are also available under friendly CC licenses  All the data can be acquired at https://github com/facebookresearch/covost 
Data Collection and Processing ::: Common Voice (CoVo)
Common Voice BIBREF10 is a crowdsourcing speech recognition corpus with an open CC0 license  Contributors record voice clips by reading from a bank of donated sentences  Each voice clip was validated by at least two other users  Most of the sentences are covered by multiple speakers, with potentially different genders, age groups or accents 
Raw CoVo data contains samples that passed validation as well as those that did not  To build CoVoST, we only use the former one and reuse the official train-development-test partition of the validated data  As of January 2020, the latest CoVo 2019-06-12 release includes 29 languages  CoVoST is currently built on that release and covers the following 11 languages: French, German, Dutch, Russian, Spanish, Italian, Turkish, Persian, Swedish, Mongolian and Chinese 
Validated transcripts were sent to professional translators  Note that the translators had access to the transcripts but not the corresponding voice clips since clips would not carry additional information  Since transcripts were duplicated due to multiple speakers, we deduplicated the transcripts before sending them to translators  As a result, different voice clips of the same content (transcript) will have identical translations in CoVoST for train, development and test splits 
In order to control the quality of the professional translations, we applied various sanity checks to the translations BIBREF11  1) For German-English, French-English and Russian-English translations, we computed sentence-level BLEU BIBREF12 with the NLTK BIBREF13 implementation between the human translations and the automatic translations produced by a state-of-the-art system BIBREF14 (the French-English system was a Transformer big BIBREF15 separately trained on WMT14)  We applied this method to these three language pairs only as we are confident about the quality of the corresponding systems  Translations with a score that was too low were manually inspected and sent back to the translators when needed  2) We manually inspected examples where the source transcript was identical to the translation  3) We measured the perplexity of the translations using a language model trained on a large amount of clean monolingual data BIBREF14  We manually inspected examples where the translation had a high perplexity and sent them back to translators accordingly  4) We computed the ratio of English characters in the translations  We manually inspected examples with a low ratio and sent them back to translators accordingly  5) Finally, we used VizSeq BIBREF16 to calculate similarity scores between transcripts and translations based on LASER cross-lingual sentence embeddings BIBREF17  Samples with low scores were manually inspected and sent back for translation when needed 
We also sanity check the overlaps of train, development and test sets in terms of transcripts and voice clips (via MD5 file hashing), and confirm they are totally disjoint 
Data Collection and Processing ::: Tatoeba (TT)
Tatoeba (TT) is a community built language learning corpus having sentences aligned across multiple languages with the corresponding speech partially available  Its sentences are on average shorter than those in CoVoST (see also Table TABREF2) given the original purpose of language learning  Sentences in TT are licensed under CC BY 2 0 FR and part of the speeches are available under various CC licenses 
We construct an evaluation set from TT (for French, German, Dutch, Russian and Spanish) as a complement to CoVoST development and test sets  We collect (speech, transcript, English translation) triplets for the 5 languages and do not include those whose speech has a broken URL or is not CC licensed  We further filter these samples by sentence lengths (minimum 4 words including punctuations) to reduce the portion of short sentences  This makes the resulting evaluation set closer to real-world scenarios and more challenging 
We run the same quality checks for TT as for CoVoST but we do not find poor quality translations according to our criteria  Finally, we report the overlap between CoVo transcripts and TT sentences in Table TABREF5  We found a minimal overlap, which makes the TT evaluation set a suitable additional test set when training on CoVoST 
Data Analysis ::: Basic Statistics
Basic statistics for CoVoST and TT are listed in Table TABREF2 including (unique) sentence counts, speech durations, speaker demographics (partially available) as well as vocabulary and token statistics (based on Moses-tokenized sentences by sacreMoses) on both transcripts and translations  We see that CoVoST has over 327 hours of German speeches and over 171 hours of French speeches, which, to our knowledge, corresponds to the largest corpus among existing public ST corpora (the second largest is 110 hours BIBREF18 for German and 38 hours BIBREF19 for French)  Moreover, CoVoST has a total of 18 hours of Dutch speeches, to our knowledge, contributing the first public Dutch ST resource  CoVoST also has around 27-hour Russian speeches, 37-hour Italian speeches and 67-hour Persian speeches, which is 1 8 times, 2 5 times and 13 3 times of the previous largest public one BIBREF7  Most of the sentences (transcripts) in CoVoST are covered by multiple speakers with potentially different accents, resulting in a rich diversity in the speeches  For example, there are over 1,000 speakers and over 10 accents in the French and German development / test sets  This enables good coverage of speech variations in both model training and evaluation 
Data Analysis ::: Speaker Diversity
As we can see from Table TABREF2, CoVoST is diversified with a rich set of speakers and accents  We further inspect the speaker demographics in terms of sample distributions with respect to speaker counts, accent counts and age groups, which is shown in Figure FIGREF6, FIGREF7 and FIGREF8  We observe that for 8 of the 11 languages, at least 60% of the sentences (transcripts) are covered by multiple speakers  Over 80% of the French sentences have at least 3 speakers  And for German sentences, even over 90% of them have at least 5 speakers  Similarly, we see that a large portion of sentences are spoken in multiple accents for French, German, Dutch and Spanish  Speakers of each language also spread widely across different age groups (below 20, 20s, 30s, 40s, 50s, 60s and 70s) 
Baseline Results
We provide baselines using the official train-development-test split on the following tasks: automatic speech recognition (ASR), machine translation (MT) and speech translation (ST) 
Baseline Results ::: Experimental Settings ::: Data Preprocessing
We convert raw MP3 audio files from CoVo and TT into mono-channel waveforms, and downsample them to 16,000 Hz  For transcripts and translations, we normalize the punctuation, we tokenize the text with sacreMoses and lowercase it  For transcripts, we further remove all punctuation markers except for apostrophes  We use character vocabularies on all the tasks, with 100% coverage of all the characters  Preliminary experimentation showed that character vocabularies provided more stable training than BPE  For MT, the vocabulary is created jointly on both transcripts and translations  We extract 80-channel log-mel filterbank features, computed with a 25ms window size and 10ms window shift using torchaudio  The features are normalized to 0 mean and 1 0 standard deviation  We remove samples having more than 3,000 frames or more than 256 characters for GPU memory efficiency (less than 25 samples are removed for all languages) 
Baseline Results ::: Experimental Settings ::: Model Training
Our ASR and ST models follow the architecture in berard2018end, but have 3 decoder layers like that in pino2019harnessing  For MT, we use a Transformer base architecture BIBREF15, but with 3 encoder layers, 3 decoder layers and 0 3 dropout  We use a batch size of 10,000 frames for ASR and ST, and a batch size of 4,000 tokens for MT  We train all models using Fairseq BIBREF20 for up to 200,000 updates  We use SpecAugment BIBREF21 for ASR and ST to alleviate overfitting 
Baseline Results ::: Experimental Settings ::: Inference and Evaluation
We use a beam size of 5 for all models  We use the best checkpoint by validation loss for MT, and average the last 5 checkpoints for ASR and ST  For MT and ST, we report case-insensitive tokenized BLEU BIBREF22 using sacreBLEU BIBREF23  For ASR, we report word error rate (WER) and character error rate (CER) using VizSeq 
Baseline Results ::: Automatic Speech Recognition (ASR)
For simplicity, we use the same model architecture for ASR and ST, although we do not leverage ASR models to pretrain ST model encoders later  Table TABREF18 shows the word error rate (WER) and character error rate (CER) for ASR models  We see that French and German perform the best given they are the two highest resource languages in CoVoST  The other languages are relatively low resource (especially Turkish and Swedish) and the ASR models are having difficulties to learn from this data 
Baseline Results ::: Machine Translation (MT)
MT models take transcripts (without punctuation) as inputs and outputs translations (with punctuation)  For simplicity, we do not change the text preprocessing methods for MT to correct this mismatch  Moreover, this mismatch also exists in cascading ST systems, where MT model inputs are the outputs of an ASR model  Table TABREF20 shows the BLEU scores of MT models  We notice that the results are consistent with what we see from ASR models  For example thanks to abundant training data, French has a decent BLEU score of 29 8/25 4  German doesn't perform well, because of less richness of content (transcripts)  The other languages are low resource in CoVoST and it is difficult to train decent models without additional data or pre-training techniques 
Baseline Results ::: Speech Translation (ST)
CoVoST is a many-to-one multilingual ST corpus  While end-to-end one-to-many and many-to-many multilingual ST models have been explored very recently BIBREF8, BIBREF9, many-to-one multilingual models, to our knowledge, have not  We hence use CoVoST to examine this setting  Table TABREF22 and TABREF23 show the BLEU scores for both bilingual and multilingual end-to-end ST models trained on CoVoST  We observe that combining speeches from multiple languages is consistently bringing gains to low-resource languages (all besides French and German)  This includes combinations of distant languages, such as Ru+Fr, Tr+Fr and Zh+Fr  Moreover, some combinations do bring gains to high-resource language (French) as well: Es+Fr, Tr+Fr and Mn+Fr  We simply provide the most basic many-to-one multilingual baselines here, and leave the full exploration of the best configurations to future work  Finally, we note that for some language pairs, absolute BLEU numbers are relatively low as we restrict model training to the supervised data  We encourage the community to improve upon those baselines, for example by leveraging semi-supervised training 
Baseline Results ::: Multi-Speaker Evaluation
In CoVoST, large portion of transcripts are covered by multiple speakers with different genders, accents and age groups  Besides the standard corpus-level BLEU scores, we also want to evaluate model output variance on the same content (transcript) but different speakers  We hence propose to group samples (and their sentence BLEU scores) by transcript, and then calculate average per-group mean and average coefficient of variation defined as follows:
and
where $G$ is the set of sentence BLEU scores grouped by transcript and $G^{\prime } = \lbrace g | g\in G, |g|>1, \textrm {Mean}(g) > 0 \rbrace $ 
$\textrm {BLEU}_{MS}$ provides a normalized quality score as oppose to corpus-level BLEU or unnormalized average of sentence BLEU  And $\textrm {CoefVar}_{MS}$ is a standardized measure of model stability against different speakers (the lower the better)  Table TABREF24 shows the $\textrm {BLEU}_{MS}$ and $\textrm {CoefVar}_{MS}$ of our ST models on CoVoST test set  We see that German and Persian have the worst $\textrm {CoefVar}_{MS}$ (least stable) given their rich speaker diversity in the test set and relatively small train set (see also Figure FIGREF6 and Table TABREF2)  Dutch also has poor $\textrm {CoefVar}_{MS}$ because of the lack of training data  Multilingual models are consistantly more stable on low-resource languages  Ru+Fr, Tr+Fr, Fa+Fr and Zh+Fr even have better $\textrm {CoefVar}_{MS}$ than all individual languages 
Conclusion
We introduce a multilingual speech-to-text translation corpus, CoVoST, for 11 languages into English, diversified with over 11,000 speakers and over 60 accents  We also provide baseline results, including, to our knowledge, the first end-to-end many-to-one multilingual model for spoken language translation  CoVoST is free to use with a CC0 license, and the additional Tatoeba evaluation samples are also CC-licensed 

ground_truth: using the machine translation platform Apertium 

machine translation platform Apertium BIBREF5

prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. Is Arabic one of the 11 languages in CoVost?
Context 3: Introduction
Deep learning systems have shown a lot of promise for extractive Question Answering (QA), with performance comparable to humans when large scale data is available  However, practitioners looking to build QA systems for specific applications may not have the resources to collect tens of thousands of questions on corpora of their choice  At the same time, state-of-the-art machine reading systems do not lend well to low-resource QA settings where the number of labeled question-answer pairs are limited (c f  Table 2 )  Semi-supervised QA methods like BIBREF0 aim to improve this performance by leveraging unlabeled data which is easier to collect 
In this work, we present a semi-supervised QA system which requires the end user to specify a set of base documents and only a small set of question-answer pairs over a subset of these documents  Our proposed system consists of three stages  First, we construct cloze-style questions (predicting missing spans of text) from the unlabeled corpus; next, we use the generated clozes to pre-train a powerful neural network model for extractive QA BIBREF1 , BIBREF2 ; and finally, we fine-tune the model on the small set of provided QA pairs 
Our cloze construction process builds on a typical writing phenomenon and document structure: an introduction precedes and summarizes the main body of the article  Many large corpora follow such a structure, including Wikipedia, academic papers, and news articles  We hypothesize that we can benefit from the un-annotated corpora to better answer various questions – at least ones that are lexically similar to the content in base documents and directly require factual information 
We apply the proposed system on three datasets from different domains – SQuAD BIBREF3 , TriviaQA-Web BIBREF4 and the BioASQ challenge BIBREF5   We observe significant improvements in a low-resource setting across all three datasets  For SQuAD and TriviaQA, we attain an F1 score of more than 50% by merely using 1% of the training data  Our system outperforms the approaches for semi-supervised QA presented in BIBREF0 , and a baseline which uses the same unlabeled data but with a language modeling objective for pretraining  In the BioASQ challenge, we outperform the best performing system from previous year's challenge, improving over a baseline which does transfer learning from the SQuAD dataset  Our analysis reveals that questions which ask for factual information and match to specific parts of the context documents benefit the most from pretraining on automatically constructed clozes 
Related Work
Semi-supervised learning augments the labeled dataset $L$ with a potentially larger unlabeled dataset $U$   BIBREF0 presented a model, GDAN, which trained an auxiliary neural network to generate questions from passages by reinforcement learning, and augment the labeled dataset with the generated questions to train the QA model  Here we use a much simpler heuristic to generate the auxiliary questions, which also turns out to be more effective as we show superior performance compared to GDAN  Several approaches have been suggested for generating natural questions BIBREF6 , BIBREF7 , BIBREF8 , however none of them show a significant improvement of using the generated questions in a semi-supervised setting  Recent papers also use unlabeled data for QA by training large language models and extracting contextual word vectors from them to input to the QA model BIBREF9 , BIBREF10 , BIBREF11   The applicability of this method in the low-resource setting is unclear as the extra inputs increase the number of parameters in the QA model, however, our pretraining can be easily applied to these models as well 
Domain adaptation (and Transfer learning) leverage existing large scale datasets from a source domain (or task) to improve performance on a target domain (or task)  For deep learning and QA, a common approach is to pretrain on the source dataset and then fine-tune on the target dataset BIBREF12 , BIBREF13   BIBREF14 used SQuAD as a source for the target BioASQ dataset, and BIBREF15 used Book Test BIBREF16 as source for the target SQuAD dataset  BIBREF17 transfer learned model layers from the tasks of sequence labeling, text classification and relation classification to show small improvements on SQuAD  All these works use manually curated source datatset, which in themselves are expensive to collect  Instead, we show that it is possible to automatically construct the source dataset from the same domain as the target, which turns out to be more beneficial in terms of performance as well (c f  Section "Experiments & Results" )  Several cloze datasets have been proposed in the literature which use heuristics for construction BIBREF18 , BIBREF19 , BIBREF20   We further see the usability of such a dataset in a semi-supervised setting 
Methodology
Our system comprises of following three steps:
Cloze generation: Most of the documents typically follow a template, they begin with an introduction that provides an overview and a brief summary for what is to follow  We assume such a structure while constructing our cloze style questions  When there is no clear demarcation, we treat the first $K\%$ (hyperparameter, in our case 20%) of the document as the introduction  While noisy, this heuristic generates a large number of clozes given any corpus, which we found to be beneficial for semi-supervised learning despite the noise 
We use a standard NLP pipeline based on Stanford CoreNLP (for SQuAD, TrivaQA and PubMed) and the BANNER Named Entity Recognizer (only for PubMed articles) to identify entities and phrases  Assume that a document comprises of introduction sentences $\lbrace q_1, q_2,     q_n\rbrace $ , and the remaining passages $\lbrace p_1, p_2,    p_m\rbrace $   Additionally, let's say that each sentence $q_i$ in introduction is composed of words $\lbrace w_1, w_2,     w_{l_{q_i}}\rbrace $ , where $l_{q_i}$ is the length of $q_i$   We consider a $\text{match} (q_i, p_j)$ , if there is an exact string match of a sequence of words $\lbrace w_k, w_{k+1},    w_{l_{q_i}}\rbrace $ between the sentence $q_i$ and passage $p_j$   If this sequence is either a noun phrase, verb phrase, adjective phrase or a named entity in $\lbrace p_1, p_2,    p_m\rbrace $0 , as recognized by CoreNLP or BANNER, we select it as an answer span $\lbrace p_1, p_2,    p_m\rbrace $1   Additionally, we use $\lbrace p_1, p_2,    p_m\rbrace $2 as the passage $\lbrace p_1, p_2,    p_m\rbrace $3 and form a cloze question $\lbrace p_1, p_2,    p_m\rbrace $4 from the answer bearing sentence $\lbrace p_1, p_2,    p_m\rbrace $5 by replacing $\lbrace p_1, p_2,    p_m\rbrace $6 with a placeholder  As a result, we obtain passage-question-answer ( $\lbrace p_1, p_2,    p_m\rbrace $7 ) triples (Table 1 shows an example)  As a post-processing step, we prune out $\lbrace p_1, p_2,    p_m\rbrace $8 triples where the word overlap between the question (Q) and passage (P) is less than 2 words (after excluding the stop words) 
The process relies on the fact that answer candidates from the introduction are likely to be discussed in detail in the remainder of the article  In effect, the cloze question from the introduction and the matching paragraph in the body forms a question and context passage pair  We create two cloze datasets, one each from Wikipedia corpus (for SQuAD and TriviaQA) and PUBMed academic papers (for the BioASQ challenge), consisting of 2 2M and 1M clozes respectively  From analyzing the cloze data manually, we were able to answer 76% times for the Wikipedia set and 80% times for the PUBMed set using the information in the passage  In most cases the cloze paraphrased the information in the passage, which we hypothesized to be a useful signal for the downstream QA task 
We also investigate the utility of forming subsets of the large cloze corpus, where we select the top passage-question-answer triples, based on the different criteria, like i) jaccard similarity of answer bearing sentence in introduction and the passage ii) the tf-idf scores of answer candidates and iii) the length of answer candidates  However, we empirically find that we were better off using the entire set rather than these subsets 
Pre-training: We make use of the generated cloze dataset to pre-train an expressive neural network designed for the task of reading comprehension  We work with two publicly available neural network models – the GA Reader BIBREF2 (to enable comparison with prior work) and BiDAF + Self-Attention (SA) model from BIBREF1 (which is among the best performing models on SQuAD and TriviaQA)  After pretraining, the performance of BiDAF+SA on a dev set of the (Wikipedia) cloze questions is 0 58 F1 score and 0 55 Exact Match (EM) score  This implies that the cloze corpus is neither too easy, nor too difficult to answer 
Fine Tuning: We fine tune the pre-trained model, from the previous step, over a small set of labelled question-answer pairs  As we shall later see, this step is crucial, and it only requires a handful of labelled questions to achieve a significant proportion of the performance typically attained by training on tens of thousands of questions 
Datasets
We apply our system to three datasets from different domains  SQuAD BIBREF3 consists of questions whose answers are free form spans of text from passages in Wikipedia articles  We follow the same setting as in BIBREF0 , and split $10\%$ of training questions as the test set, and report performance when training on subsets of the remaining data ranging from $1\%$ to $90\%$ of the full set  We also report the performance on the dev set when trained on the full training set ( $1^\ast $ in Table 2 )  We use the same hyperparameter settings as in prior work  We compare and study four different settings: 1) the Supervised Learning (SL) setting, which is only trained on the supervised data, 2) the best performing GDAN model from BIBREF0 , 3) pretraining on a Language Modeling (LM) objective and fine-tuning on the supervised data, and 4) pretraining on the Cloze dataset and fine-tuning on the supervised data  The LM and Cloze methods use exactly the same data for pretraining, but differ in the loss functions used  We report F1 and EM scores on our test set using the official evaluation scripts provided by the authors of the dataset 
TriviaQA BIBREF4 comprises of over 95K web question-answer-evidence triples  Like SQuAD, the answers are spans of text  Similar to the setting in SQuAD, we create multiple smaller subsets of the entire set  For our semi-supervised QA system, we use the BiDAF+SA model BIBREF1 – the highest performing publicly available system for TrivaQA  Here again, we compare the supervised learning (SL) settings against the pretraining on Cloze set and fine tuning on the supervised set  We report F1 and EM scores on the dev set 
We also test on the BioASQ 5b dataset, which consists of question-answer pairs from PubMed abstracts  We use the publicly available system from BIBREF14 , and follow the exact same setup as theirs, focusing only on factoid and list questions  For this setting, there are only 899 questions for training  Since this is already a low-resource problem we only report results using 5-fold cross-validation on all the available data  We report Mean Reciprocal Rank (MRR) on the factoid questions, and F1 score for the list questions 
Main Results
Table 2 shows a comparison of the discussed settings on both SQuAD and TriviaQA  Without any fine-tuning (column 0) the performance is low, probably because the model never saw a real question, but we see significant gains with Cloze pretraining even with very little labeled data  The BiDAF+SA model, exceeds an F1 score of $50\%$ with only $1\%$ of the training data (454 questions for SQuAD, and 746 questions for TriviaQA), and approaches $90\%$ of the best performance with only $10\%$ labeled data  The gains over the SL setting, however, diminish as the size of the labeled set increases and are small when the full dataset is available 
Cloze pretraining outperforms the GDAN baseline from BIBREF0 using the same SQuAD dataset splits  Additionally, we show improvements in the $90\%$ data case unlike GDAN  Our approach is also applicable in the extremely low-resource setting of $1\%$ data, which we suspect GDAN might have trouble with since it uses the labeled data to do reinforcement learning  Furthermore, we are able to use the same cloze dataset to improve performance on both SQuAD and TriviaQA datasets  When we use the same unlabeled data to pre-train with a language modeling objective, the performance is worse, showing the bias we introduce by constructing clozes is important 
On the BioASQ dataset (Table 3 ) we again see a significant improvement when pretraining with the cloze questions over the supervised baseline  The improvement is smaller than what we observe with SQuAD and TriviaQA datasets – we believe this is because questions are generally more difficult in BioASQ  BIBREF14 showed that pretraining on SQuAD dataset improves the downstream performance on BioASQ  Here, we show a much larger improvement by pretraining on cloze questions constructed in an unsupervised manner from the same domain 
Analysis
Regression Analysis: To understand which types of questions benefit from pre-training, we pre-specified certain features (see Figure 1 right) for each of the dev set questions in SQuAD, and then performed linear regression to predict the F1 score for that question from these features  We predict the F1 scores from the cloze pretrained model ( $y^{\text{cloze}}$ ), the supervised model ( $y^{\text{sl}}$ ), and the difference of the two ( $y^{\text{cloze}}-y^{\text{sl}}$ ), when using $10\%$ of labeled data  The coefficients of the fitted model are shown in Figure 1 (left) along with their std errors  Positive coefficients indicate that a high value of that feature is predictive of a high F1 score, and a negative coefficient indicates that a small value of that feature is predictive of a high F1 score (or a high difference of F1 scores from the two models in the case of $y^{\text{cloze}}-y^{\text{sl}}$ ) 
The two strongest effects we observe are that a high lexical overlap between the question and the sentence containing the answer is indicative of high boost with pretraining, and that a high lexical overlap between the question and the whole passage is indicative of the opposite  This is hardly surprising, since our cloze construction process is biased towards questions which have a similar phrasing to the answer sentences in context  Hence, test questions with a similar property are answered correctly after pretraining, whereas those with a high overlap with the whole passage tend to have lower performance  The pretraining also favors questions with short answers because the cloze construction process produces short answer spans  Also passages and questions which consist of tokens infrequent in the SQuAD training corpus receive a large boost after pretraining, since the unlabeled data covers a larger domain 
Performance on question types: Figure 2 shows the average gain in F1 score for different types of questions, when we pretrain on the clozes compared to the supervised case  This analysis is done on the $10\%$ split of the SQuAD training set  We consider two classifications of each question – one determined on the first word (usually a wh-word) of the question (Figure 2 (bottom)) and one based on the output of a separate question type classifier adapted from BIBREF21   We use the coarse grain labels namely Abbreviation (ABBR), Entity (ENTY), Description (DESC), Human (HUM), Location (LOC), Numeric (NUM) trained on a Logistic Regression classification system   While there is an improvement across the board, we find that abbreviation questions in particular receive a large boost  Also, "why" questions show the least improvement, which is in line with our expectation, since these usually require reasoning or world knowledge which cloze questions rarely require 
Conclusion
In this paper, we show that pre-training QA models with automatically constructed cloze questions improves the performance of the models significantly, especially when there are few labeled examples  The performance of the model trained only on the cloze questions is poor, validating the need for fine-tuning  Through regression analysis, we find that pretraining helps with questions which ask for factual information located in a specific part of the context  For future work, we plan to explore the active learning setup for this task – specifically, which passages and / or types of questions can we select to annotate, such that there is a maximum performance gain from fine-tuning  We also want to explore how to adapt cloze style pre-training to NLP tasks other than QA 
Acknowledgments
Bhuwan Dhingra is supported by NSF under grants CCF-1414030 and IIS-1250956 and by grants from Google  Danish Pruthi and Dheeraj Rajagopal are supported by the DARPA Big Mechanism program under ARO contract W911NF-14-1-0436 

ground_truth: No

No

prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. Is it possible to convert a cloze-style questions to a naturally-looking questions?
Context 4: , where each $N=5$) 
Probing Methodology and Modeling ::: Task Definition and Modeling ::: Question+Answer Encoder 
To model this, our investigation centers around the use of the transformer-based BIBREF25 BERT encoder and fine-tuning approach of BIBREF7 (see also BIBREF6)  For each question and individual answer pair $q^{(j)}_{a_{i}}$, we assume the following rendering of this input:
which is run through the pre-trained BERT encoder to generate a representation for $ q^{(j)}_{a_{i}}$ using the hidden state representation for CLS (i e , the classifier token) $\textbf {c}_{i}$:
The probability of a given answer $p^{(j)}_{i}$ is then computed as $p^{(j)}_{i} \propto e^{\textbf {v}\cdot \textbf {c}^{(j)}_{i}}$, which uses an additional set of classification parameters $\textbf {v} \in \mathbb {R}^{H}$ that are optimized (along with the full transformer network) by taking the final loss of the probability of each correct answer $p_{i^{*}}$ over all answer choices:
We specifically use BERT-large uncased with whole-word masking, as well as the RoBERTa-large model from BIBREF9, which is a more robustly trained version of the original BERT model  Our system uses the implementations provided in AllenNLP BIBREF39 and Huggingface BIBREF40 
Probing Methodology and Modeling ::: Task Definition and Modeling ::: Baselines and Sanity Checks 
When creating synthetic datasets, it is important to ensure that systematic biases, or annotation artifacts BIBREF41, are not introduced into the resulting probes and that the target datasets are sufficiently challenging (or good, in the sense of BIBREF42)  To test for this, we use several of the MCQA baseline models first introduced in BIBREF0, which take inspiration from the LSTM-based models used in BIBREF43 for NLI and various partial-input baselines based on these models 
Following the notation from BIBREF0, for any given sequence $s$ of tokens in $\lbrace q^{(j)}, a_{1}^{(j)},   ,a_{N}^{(j)}\rbrace $ in $D$, an encoding of $s$ is given as $h_{s}^{(j)} = \textbf {BiLSTM}(\textsc {embed}(s)) \in \mathbb {R}^{|s| \times 2h}$ (where $h$ is the dimension of the hidden state in each directional network, and embed$(\cdot )$ is an embedding function that assigns token-level embeddings to each token in $s$)  A contextual representation for each $s$ is then built by applying an element-wise max operation over $h_{s}$ as follows:
With these contextual representations, different baseline models can be constructed  For example, a Choice-Only model, which is a variant of the well-known hypothesis-only baseline used in NLI BIBREF46, scores each choice $c_{i}$ in the following way:
for $\textbf {W}^{T} \in \mathbb {R}^{2h}$ independently of the question and assigns a probability to each answer $p_{i}^{(j)} \propto e^{\alpha _{i}^{(j)}}$ 
A slight variant of this model, the Choice-to-choice model, tries to single out a given answer choice relative to other choices by scoring all choice pairs $\alpha _{i,i^{\prime }}^{(j)} = \textsc {Att}(r^{(j)}_{c_{i}},r^{(j)}_{c_{i^{\prime }}}) \in \mathbb {R}$ using a learned attention mechanism Att and finding the choice with the minimal similarity to other options (for full details, see their original paper)  In using these partial-input baselines, which we train directly on each target probe, we can check whether systematic biases related to answer choices were introduced into the data creation process 
A Question-to-choice model, in contrast, uses the contextual representations for each question and individual choice and an attention model Att model to get a score $\alpha ^{(j)}_{q,i} = \textsc {Att}(r^{(j)}_{q},r^{(j)}_{c_{i}}) \in \mathbb {R}$ as above  Here we also experiment with using ESIM BIBREF47 to generate the contextual representations $r$, as well as a simpler VecSimilarity model that measures the average vector similarity between question and answer tokens: $\alpha ^{(j)}_{q,i} = \textsc {Sim}(\textsc {embed}(q^{(j)}),\textsc {embed}(c^{(j)}_{i}))$  In contrast to the models above, these sets of baselines are used to check for artifacts between questions and answers that are not captured in the partial-input baselines (see discussion in BIBREF49) and ensure that the overall MCQA tasks are sufficiently difficult for our transformer models 
Probing Methodology and Modeling ::: Inoculation and Pre-training
Using the various models introduced above, we train these models on benchmark tasks in the science domain and look at model performance on our probes with and without additional training on samples of probe data, building on the idea of inoculation from BIBREF23  Model inoculation is the idea of continuing to train models on new challenge tasks (in our cases, separately for each probe) using only a small amount of examples  Unlike in ordinary fine-tuning, the goal is not to learn an entirely re-purposed model, but to improve on (or vaccinate against) particular phenomena (e g , our synthetic probes) that potentially deviate from a model's original training distribution (but that nonetheless might involve knowledge already contained in the model) 
In the variant proposed in BIBREF22, for each pre-trained (science) model and architecture $M_{a}$ we continue training the model on $k$ new probe examples (with a maximum of $k=$ 3k) under a set of different hyper-parameter configurations $j \in \lbrace 1,    , J\rbrace $ and identify, for each $k$, the model $M_{*}^{a,k}$ with the best aggregate performance $S$ on the original (orig) and new task:
As in BIBREF22, we found all models to be especially sensitive to different learning rates, and performed comprehensive hyper-parameters searches that also manipulate the number of iterations and random seeds used 
Using this methodology, we can see how much exposure to new data it takes for a given model to master a new task, and whether there are phenomena that stress particular models (e g , lead to catastrophic forgetting of the original task)  Given the restrictions on the number of fine-tuning examples, our assumption is that when models are able to maintain good performance on their original task during inoculation, the quickness with which they are able to learn the inoculated task provides evidence of prior competence, which is precisely what we aim to probe  To measure past performance, we define a model's inoculation cost as the difference in the performance of this model on its original task before and after inoculation 
We pre-train on an aggregated training set of the benchmark science exams detailed in Table TABREF21, and created an aggregate development set of around 4k science questions for evaluating overall science performance and inoculation costs  To handle the mismatch between number of answer choices in these sets, we made all sets 5-way by adding empty answers as needed  We also experimented with a slight variant of inoculation, called add-some inoculation, which involves balancing the inoculation training sets with naturalistic science questions  We reserve the MCQL dataset in Table TABREF21 for this purpose, and experiment with balancing each probe example with a science example (x1 matching) and adding twice as many science questions (x2 matching, up to 3k) for each new example 
Probing Methodology and Modeling ::: Evaluating Model Competence
The standard way to evaluate our MCQA models is by looking at the overall accuracy of the correct answer prediction, or what we call instance-level accuracy (as in Table TABREF25)  Given the nature of our data and the existence of semantic clusters as detailed in Section SECREF11 (i e , sets of questions and answers under different distractor choices and inference complexity), we also measure a model's cluster-level (or strict cluster) accuracy, which requires correctly answering all questions in a cluster  Example semantic clusters are shown in Table TABREF30; in the first case, there are 6 ISA$^\uparrow $ questions (including perturbations) about the concept trouser n 01 (e g , involving knowing that trousers are a type of consumer good and garment/clothing), which a model must answer in order to receive full credit 
Our cluster-based analysis is motivated by the idea that if a model truly knows the meaning of a given concept, such as the concept of trousers, then it should be able to answer arbitrary questions about this concept without sensitivity to varied distractors  While our strict cluster metric is simplistic, it takes inspiration from work on visual QA BIBREF53, and allows us to evaluate how consistent and robust models are across our different probes, and to get insight into whether errors are concentrated on a small set of concepts or widespread across clusters 
Results and Findings
In this section, we provide the results of the empirical questions first introduced in Figure FIGREF1, starting with the results of our baseline models 
Results and Findings ::: Are our Probes Sufficiently Challenging?
As shown in Table TABREF25, most of our partial-input baselines (i e , Choice-Only and Choice-to-Choice models) failed to perform well on our dataset probes across a wide range of models, showing that such probes are generally immune from biases relating to how distractors were generated  As already discussed in Section SECREF13, however, initial versions of the DictionaryQA dataset had unforeseen biases partly related to whether distractors were sampled from entries without example sentences, which resulted in high Choice-Only-GloVe scores ranging around 56% accuracy before a filtering step was applied to remove these distractors 
We had similar issues with the hypernymy probe which, even after a filtering step that used our Choice-to-Choice-GloVe model, still leads to high results on the BERT and RoBERTa choice-only models  Given that several attempts were made to entirely de-duplicate the different splits (both in terms of gold answers and distractor types), the source of these biases is not at all obvious, which shows how easy it is for unintended biases in expert knowledge to appear in the resulting datasets and the importance of having rigorous baselines  We also note the large gap in some cases between the BERT and RoBERTa versus GloVe choice-only models, which highlights the need for having partial-input baselines that use the best available models 
Using a more conventional set of Task-Specific QA models (i e , the LSTM-based Question-to-Choice models trained directly on the probes), we can see that results are not particularly strong on any of the datasets, suggesting that our probes are indeed sufficiently challenging and largely immune from overt artifacts  The poor performance of the VecSimilarity (which uses pre-trained Word2Vec embeddings without additional training) provides additional evidence that elementary lexical matching strategies are insufficient for solving any of the probing tasks 
Results and Findings ::: How well do pre-trained MCQA models do?
Science models that use non-transformer based encoders, such as the ESIM model with GloVe and ELMO, perform poorly across all probes, in many cases scoring near random chance, showing limits to how well they generalize from science to other tasks even with pre-trained GloVe and ELMO embeddings  In sharp contrast, the transformer models have mixed results, the most striking result being the RoBERTa models on the definitions and synonymy probes (achieving a test accuracy of 77% and 61%, respectively), which outperform several of the task-specific LSTM models trained directly on the probes  At first glance, this suggests that RoBERTa, which generally far outpaces even BERT across most probes, has high competence of definitions and synonyms even without explicit training on our new tasks 
Given the controlled nature of our probes, we can get a more detailed view of how well the science models are performing across different reasoning and distractor types, as shown in the first column of Figure FIGREF28 for ESIM and RoBERTa  The ESIM science model without training has uniformly poor performance across all categories, whereas the performance of RoBERTa is more varied  Across all datasets and number of hops (i e , the rows in the heat maps), model performance for RoBERTa is consistently highest among examples with random distractors (i e , the first column), and lowest in cases involving distractors that are closest in WordNet space (e g , sister and ISA, or up/down, distractors of distance $k^{\prime }=1$)  This is not surprising, given that, in the first case, random distractors are likely to be the easiest category (and the opposite for distractors close in space), but suggests that RoBERTa might only be getting the easiest cases correct 
Model performance also clearly degrades for hypernymy and hyponymy across all models as the number of hops $k$ increases (see red dashed boxes)  For example, problems that involve hyponym reasoning with sister distractors of distance $k^{\prime }=1$ (i e , the second column) degrades from 47% to 15% when the number of hops $k$ increases from 1 to 4  This general tendency persists even after additional fine-tuning, as we discuss next, and gives evidence that models are limited in their capacity for certain types of multi-hop inferences 
As discussed by BIBREF26, the choice of generation templates can have a significant effect on model performance  The results so far should therefore be regarded as a lower bound on model competence  It is possible that model performance is high for definitions, for example, because the associated templates best align with the science training distribution (which we know little about)  For this reason, the subsequent inoculation step is important—it gives the model an opportunity to learn about our target templates and couple this learned knowledge with its general knowledge acquired during pre-training and science training (which is, again, what we aim to probe) 
Results and Findings ::: Can Models Be Effectively Inoculated?
Model performance after additional fine-tuning, or inoculation, is shown in the last 3 rows of Table TABREF25, along with learning curves shown in Figure FIGREF29 for a selection of probes and models  In the former case, the performance represents the model (and inoculation amount) with the highest aggregate performance over the old task and new probe  Here we again see the transformer-based models outperform non-transformer models, and that better models correlate with lower inoculation costs  For example, when inoculating on synonymy, the cost for ESIM is around 7% reduced accuracy on its original task, as opposed to $< 1$% and around 1% for BERT and RoBERTa, respectively  This shows the high capacity for transformer models to absorb new tasks with minimal costs, as also observed in BIBREF22 for NLI 
As shown in Figure FIGREF29, transformer models tend to learn most tasks fairly quickly while keeping constant scores on their original tasks (i e , the flat dashed lines observed in plots 1-4), which gives evidence of high competence  In both cases, add-some inoculation proves to be a cheap and easy way to 1) improve scores on the probing tasks (i e , the solid black and blue lines in plot 1) and; 2) minimize loss on science (e g , the blue and black dashed lines in plots 2-4)  The opposite is the case for ESIM (plots 5-6); models are generally unable to simultaneously learn individual probes without degrading on their original task, and adding more science data during inoculation confuses models on both tasks 
As shown in Figure FIGREF28, RoBERTa is able to significantly improve performance across most categories even after inoculation with a mere 100 examples (the middle plot), which again provides strong evidence of prior competence  As an example, RoBERTa improves on 2-hop hyponymy inference with random distractors by 18% (from 59% to 77%)  After 3k examples, the model has high performance on virtually all categories (the same score increases from 59% to 87%), however results still tends to degrade as a function of hop and distractor complexity, as discussed above 
Despite the high performance of our transformer models after inoculation, model performance on most probes (with the exception of Definitions) averages around 80% for our best models  This suggests that there is still considerable room for improvement, especially for synonymy and word sense, which is a topic that we discuss more in Section SECREF6 
Results and Findings ::: Are Models Consistent across Clusters?
Table TABREF32 shows cluster-level accuracies for the different WordNetQA probes  As with performance across the different inference/distractor categories, these results are mixed  For some probes, such as definitions, our best models appear to be rather robust; e g , our RoBERTa model has a cluster accuracy of $75\%$, meaning that it can answer all questions perfectly for 75% of the target concepts and that errors are concentrated on a small minority (25%) of concepts  On synonymy and hypernymy, both BERT and RoBERTa appear robust on the majority of concepts, showing that errors are similarly concentrated  In contrast, our best model on hyponymy has an accuracy of 36%, meaning that its errors are spread across many concepts, thus suggesting less robustness 
Table TABREF30 shows a selection of semantic clusters involving ISA reasoning, as well as the model performance over different answers (shown symbolically) and perturbations  For example, in the the second case, the cluster is based around the concept/synset oppose v 06 and involves 4 inferences and a total 24 questions (i e , inferences with perturbations)  Our weakest model, ESIM, answers only 5 out of 24 questions correctly, whereas RoBERTa gets 21/24  In the other cases, RoBERTa gets all clusters correct, whereas BERT and ESIM get none of them correct 
We emphasize that these results only provide a crude look into model consistency and robustness  Recalling again the details in Table TABREF12, probes differ in terms of average size of clusters  Hyponymy, in virtue of having many more questions per cluster, might simply be a much more difficult dataset  In addition, such a strict evaluation does not take into account potential errors inside of clusters, which is an important issue that we discuss in the next section  We leave addressing such issues and coming up with more insightful cluster-based metrics for future work 
Discussion and Conclusion
We presented several new challenge datasets and a novel methodology for automatically building such datasets from knowledge graphs and taxonomies  We used these to probe state-of-the-art open-domain QA models (centering around models based on variants of BERT)  While our general methodology is amendable to any target knowledge resource or QA model/domain, we focus on probing definitions and ISA knowledge using open-source dictionaries and MCQA models trained in the science domain 
We find, consistent with recent probing studies BIBREF26, that transformer-based models have a remarkable ability to answer questions that involve complex forms of relational knowledge, both with and without explicit exposure to our new target tasks  In the latter case, a newer RoBERTa model trained only on benchmark science tasks is able to outperform several task-specific LSTM-based models trained directly on our probing data  When re-trained on small samples (e g , 100 examples) of probing data using variations of the lossless inoculation strategy from BIBREF22, RoBERTa is able to master many aspects of our probes with virtually no performance loss on its original QA task 
These positive results suggest that transformer-based models, especially models additionally fine-tuned on small samples of synthetic data, can be used in place of task-specific models used for querying relational knowledge, as has already been done for targeted tasks such as word sense disambiguation BIBREF54  Since models seem to already contain considerable amounts of relational knowledge, our simple inoculation strategy, which tries to nudge models to bring out this knowledge explicitly, could serve as a cheaper alternative to recent attempts to build architectures that explicitly incorporate structured knowledge BIBREF55; we see many areas where our inoculation strategy could be improved for such purposes, including having more complex loss functions that manage old and new information, as well as using techniques that take into account network plasticity BIBREF56 
The main appeal of using automatically generate datasets is the ability to systematically manipulate and control the complexity of target questions, which allows for more controlled experimentation and new forms of evaluation  Despite the positive results described above, results that look directly at the effect of different types of distractors and the complexity of reasoning show that our best models, even after additional fine-tuning, struggle with certain categories of hard distractors and multi-hop inferences  For some probes, our cluster-based analysis also reveals that errors are widespread across concept clusters, suggesting that models are not always consistent and robust  These results, taken together with our findings about the vulnerability of synthetic datasets to systematic biases, suggest that there is much room for improvement and that the positive results should be taken with a grain of salt  Developing better ways to evaluate semantic clusters and model robustness would be a step in this direction 
We emphasize that using synthetic versus naturalistic QA data comes with important trade-offs  While we are able to generate large amounts of systematically controlled data at virtually no cost or need for manual annotation, it is much harder to validate the quality of such data at such a scale and such varying levels of complexity  Conversely, with benchmark QA datasets, it is much harder to perform the type of careful manipulations and cluster-based analyses we report here  While we assume that the expert knowledge we employ, in virtue of being hand-curated by human experts, is generally correct, we know that such resources are fallible and error-prone  Initial crowd-sourcing experiments that look at validating samples of our data show high agreement across probes and that human scores correlate with the model trends across the probe categories  More details of these studies are left for future work 

ground_truth: Unanswerable

Unanswerable