prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. What dataset does this approach achieve state of the art results on?
Context 1: Data
We build and test our MMT models on the Multi30K dataset BIBREF21   Each image in Multi30K contains one English (EN) description taken from Flickr30K BIBREF22 and human translations into German (DE), French (FR) and Czech BIBREF23 , BIBREF24 , BIBREF25   The dataset contains 29,000 instances for training, 1,014 for development, and 1,000 for test  We only experiment with German and French, which are languages for which we have in-house expertise for the type of analysis we present  In addition to the official Multi30K test set (test 2016), we also use the test set from the latest WMT evaluation competition, test 2018 BIBREF25  
Degradation of source
In addition to using the Multi30K dataset as is (standard setup), we probe the ability of our models to address the three linguistic phenomena where additional context has been proved important (Section ): ambiguities, gender-neutral words and noisy input  In a controlled experiment where we aim to remove the influence of frequency biases, we degrade the source sentences by masking words through three strategies to replace words by a placeholder: random source words, ambiguous source words and gender unmarked source words  The procedure is applied to the train, validation and test sets  For the resulting dataset generated for each setting, we compare models having access to text-only context versus additional text and multimodal contexts  We seek to get insights into the contribution of each type of context to address each type of degradation 
In this setting (RND) we simulate erroneous source words by randomly dropping source content words  We first tag the entire source sentences using the spacy toolkit BIBREF26 and then drop nouns, verbs, adjectives and adverbs and replace these with a default BLANK token  By focusing on content words, we differ from previous work that suggests that neural machine translation is robust to non-content word noise in the source BIBREF27  
In this setting (AMB), we rely on the MLT dataset BIBREF11 which provides a list of source words with multiple translations in the Multi30k training set  We replace ambiguous words with the BLANK token in the source language, which results in two language-specific datasets 
In this setting (PERS), we use the Flickr Entities dataset BIBREF28 to identify all the words that were annotated by humans as corresponding to the category person  We then replace such source words with the BLANK token 
The statistics of the resulting datasets for the three degradation strategies are shown in Table TABREF10   We note that RND and PERS are the same for language pairs as the degradation only depends on the source side, while for AMB the words replaced depend on the target language 
Models
Based on the models described in Section we experiment with eight variants: (a) baseline transformer model (base); (b) base with AIC (base+sum); (c) base with AIF using spacial (base+att) or object based (base+obj) image features; (d) standard deliberation model (del); (e) deliberation models enriched with image information: del+sum, del+att and del+obj 
Training
In all cases, we optimise our models with cross entropy loss  For deliberation network models, we first train the standard transformer model until convergence, and use it to initialise the encoder and first-pass decoder  For each of the training samples, we follow BIBREF19 and obtain a set of 10-best samples from the first pass decoder, with a beam search of size 10  We use these as the first-pass decoder samples  We use Adam as optimiser BIBREF29 and train the model until convergence 
Results
In this section we present results of our experiments, first in the original dataset without any source degradation (Section SECREF18 ) and then in the setup with various source degradation strategies (Section SECREF25 ) 
Standard setup
Table TABREF14 shows the results of our main experiments on the 2016 and 2018 test sets for French and German  We use Meteor BIBREF31 as the main metric, as in the WMT tasks BIBREF25   We compare our transformer baseline to transformer models enriched with image information, as well as to the deliberation models, with or without image information 
We first note that our multimodal models achieve the state of the art performance for transformer networks (constrained models) on the English-German dataset, as compared to BIBREF30   Second, our deliberation models lead to significant improvements over this baseline across test sets (average INLINEFORM0 , INLINEFORM1 ) 
Transformer-based models enriched with image information (base+sum, base+att and base+obj), on the other hand, show no major improvements with respect to the base performance  This is also the case for deliberation models with image information (del+sum, del+att, del+obj), which do not show significant improvement over the vanilla deliberation performance (del) 
However, as it has been shown in the WMT shared tasks on MMT BIBREF23 , BIBREF24 , BIBREF25 , automatic metrics often fail to capture nuances in translation quality, such as, the ones we expect the visual modality to help with, which – according to human perception – lead to better translations  To test this assumption in our settings, we performed human evaluation involving professional translators and native speakers of both French and German (three annotators) 
The annotators were asked to rank randomly selected test samples according to how well they convey the meaning of the source, given the image (50 samples per language pair per annotator)  For each source segment, the annotator was shown the outputs of three systems: base+att, the current MMT state-of-the-art BIBREF30 , del and del+obj  A rank could be assigned from 1 to 3, allowing ties BIBREF32   Annotators could assign zero rank to all translations if they were judged incomprehensible  Following the common practice in WMT BIBREF32 , each system was then assigned a score which reflects the proportion of times it was judged to be better or equal other systems 
Table TABREF19 shows the human evaluation results  They are consistent with the automatic evaluation results when it comes to the preference of humans towards the deliberation-based setups, but show a more positive outlook regarding the addition of visual information (del+obj over del) for French 
Manual inspection of translations suggests that deliberation setups tend to improve both the grammaticality and adequacy of the first pass outputs  For German, the most common modifications performed by the second-pass decoder are substitutions of adjectives and verbs (for test 2016, 15% and 12% respectively, of all the edit distance operations)  Changes to adjectives are mainly grammatical, changes to verbs are contextual (e g , changing laufen to rennen, both verbs mean run, but the second refers to running very fast)  For French, 15% of all the changes are substitutions of nouns (for test 2016)  These are again very contextual  For example, the French word travailleur (worker) is replaced by ouvrier (manual worker) in the contexts where tools, machinery or buildings are mentioned  For our analysis we used again spacy 
The information on detected objects is particularly helpful for specific adequacy issues  Figure FIGREF15 demonstrates some such cases  In the first case, the base+att model misses the translation of race car: the German word Rennen translates only the word race  del introduces the word car (Auto) into the translation  Finally, del+obj correctly translates the expression race car (Rennwagen) by exploiting the object information  For French, del translates the source part in a body of water, missing from the base+att translation  del+obj additionally translated the word paddling according to the detected object Paddle 
Source degradation setup
Results of our source degradation experiments are shown in Table TABREF20   A first observation is that – as with the standard setup – the performance of our deliberation models is overall better than that of the base models  The results of the multimodal models differ for German and French  For German, del+obj is the most successful configuration and shows statistically significant improvements over base for all setups  Moreover, for RND and AMB, it shows statistically significant improvements over del  However, especially for RND and AMB, del and del+sum are either the same or slightly worse than base 
For French, all the deliberation models show statistically significant improvements over base (average INLINEFORM0 , INLINEFORM1 ), but the image information added to del only improve scores significantly for test 2018 RND 
This difference in performances for French and German is potentially related to the need of more significant restructurings while translating from English into German  This is where a more complex del+obj architecture is more helpful  This is especially true for RND and AMB setups where blanked words could also be verbs, the part-of-speech most influenced by word order differences between English and German (see the decreasing complexity of translations for del and del+obj for the example (c) in Figure FIGREF21 ) 
To get an insight into the contribution of different contexts to the resolution of blanks, we performed manual analysis of examples coming from the English-German base, del and del+obj setups (50 random examples per setup), where we count correctly translated blanks per system 
The results are shown in Table TABREF27   As expected, they show that the RND and AMB blanks are more difficult to resolve (at most 40% resolved as compared to 61% for PERS)  Translations of the majority of those blanks tend to be guessed by the textual context alone (especially for verbs)  Image information is more helpful for PERS: we observe an increase of 10% in resolved blanks for del+obj as compared to del  However, for PERS the textual context is still enough in the majority of the cases: models tend to associate men with sports or women with cooking and are usually right (see Figure FIGREF21 example (c)) 
The cases where image helps seem to be those with rather generic contexts: see Figure FIGREF21 (b) where enjoying a summer day is not associated with any particular gender and make other models choose homme (man) or femme (woman), and only base+obj chooses enfant (child) (the option closest to the reference) 
In some cases detected objects are inaccurate or not precise enough to be helpful (e g , when an object Person is detected) and can even harm correct translations 
Conclusions
We have proposed a novel approach to multimodal machine translation which makes better use of context, both textual and visual  Our results show that further exploring textual context through deliberation networks already leads to better results than the previous state of the art  Adding visual information, and in particular structural representations of this information, proved beneficial when input text contains noise and the language pair requires substantial restructuring from source to target  Our findings suggest that the combination of a deliberation approach and information from additional modalities is a promising direction for machine translation that is robust to noisy input  Our code and pre-processing scripts are available at https://github com/ImperialNLP/MMT-Delib 
Acknowledgments
The authors thank the anonymous reviewers for their useful feedback  This work was supported by the MultiMT (H2020 ERC Starting Grant No  678017) and MMVC (Newton Fund Institutional Links Grant, ID 352343575) projects  We also thank the annotators for their valuable help 
Context 2: Introduction
Deep learning systems have shown a lot of promise for extractive Question Answering (QA), with performance comparable to humans when large scale data is available  However, practitioners looking to build QA systems for specific applications may not have the resources to collect tens of thousands of questions on corpora of their choice  At the same time, state-of-the-art machine reading systems do not lend well to low-resource QA settings where the number of labeled question-answer pairs are limited (c f  Table 2 )  Semi-supervised QA methods like BIBREF0 aim to improve this performance by leveraging unlabeled data which is easier to collect 
In this work, we present a semi-supervised QA system which requires the end user to specify a set of base documents and only a small set of question-answer pairs over a subset of these documents  Our proposed system consists of three stages  First, we construct cloze-style questions (predicting missing spans of text) from the unlabeled corpus; next, we use the generated clozes to pre-train a powerful neural network model for extractive QA BIBREF1 , BIBREF2 ; and finally, we fine-tune the model on the small set of provided QA pairs 
Our cloze construction process builds on a typical writing phenomenon and document structure: an introduction precedes and summarizes the main body of the article  Many large corpora follow such a structure, including Wikipedia, academic papers, and news articles  We hypothesize that we can benefit from the un-annotated corpora to better answer various questions – at least ones that are lexically similar to the content in base documents and directly require factual information 
We apply the proposed system on three datasets from different domains – SQuAD BIBREF3 , TriviaQA-Web BIBREF4 and the BioASQ challenge BIBREF5   We observe significant improvements in a low-resource setting across all three datasets  For SQuAD and TriviaQA, we attain an F1 score of more than 50% by merely using 1% of the training data  Our system outperforms the approaches for semi-supervised QA presented in BIBREF0 , and a baseline which uses the same unlabeled data but with a language modeling objective for pretraining  In the BioASQ challenge, we outperform the best performing system from previous year's challenge, improving over a baseline which does transfer learning from the SQuAD dataset  Our analysis reveals that questions which ask for factual information and match to specific parts of the context documents benefit the most from pretraining on automatically constructed clozes 
Related Work
Semi-supervised learning augments the labeled dataset $L$ with a potentially larger unlabeled dataset $U$   BIBREF0 presented a model, GDAN, which trained an auxiliary neural network to generate questions from passages by reinforcement learning, and augment the labeled dataset with the generated questions to train the QA model  Here we use a much simpler heuristic to generate the auxiliary questions, which also turns out to be more effective as we show superior performance compared to GDAN  Several approaches have been suggested for generating natural questions BIBREF6 , BIBREF7 , BIBREF8 , however none of them show a significant improvement of using the generated questions in a semi-supervised setting  Recent papers also use unlabeled data for QA by training large language models and extracting contextual word vectors from them to input to the QA model BIBREF9 , BIBREF10 , BIBREF11   The applicability of this method in the low-resource setting is unclear as the extra inputs increase the number of parameters in the QA model, however, our pretraining can be easily applied to these models as well 
Domain adaptation (and Transfer learning) leverage existing large scale datasets from a source domain (or task) to improve performance on a target domain (or task)  For deep learning and QA, a common approach is to pretrain on the source dataset and then fine-tune on the target dataset BIBREF12 , BIBREF13   BIBREF14 used SQuAD as a source for the target BioASQ dataset, and BIBREF15 used Book Test BIBREF16 as source for the target SQuAD dataset  BIBREF17 transfer learned model layers from the tasks of sequence labeling, text classification and relation classification to show small improvements on SQuAD  All these works use manually curated source datatset, which in themselves are expensive to collect  Instead, we show that it is possible to automatically construct the source dataset from the same domain as the target, which turns out to be more beneficial in terms of performance as well (c f  Section "Experiments & Results" )  Several cloze datasets have been proposed in the literature which use heuristics for construction BIBREF18 , BIBREF19 , BIBREF20   We further see the usability of such a dataset in a semi-supervised setting 
Methodology
Our system comprises of following three steps:
Cloze generation: Most of the documents typically follow a template, they begin with an introduction that provides an overview and a brief summary for what is to follow  We assume such a structure while constructing our cloze style questions  When there is no clear demarcation, we treat the first $K\%$ (hyperparameter, in our case 20%) of the document as the introduction  While noisy, this heuristic generates a large number of clozes given any corpus, which we found to be beneficial for semi-supervised learning despite the noise 
We use a standard NLP pipeline based on Stanford CoreNLP (for SQuAD, TrivaQA and PubMed) and the BANNER Named Entity Recognizer (only for PubMed articles) to identify entities and phrases  Assume that a document comprises of introduction sentences $\lbrace q_1, q_2,     q_n\rbrace $ , and the remaining passages $\lbrace p_1, p_2,    p_m\rbrace $   Additionally, let's say that each sentence $q_i$ in introduction is composed of words $\lbrace w_1, w_2,     w_{l_{q_i}}\rbrace $ , where $l_{q_i}$ is the length of $q_i$   We consider a $\text{match} (q_i, p_j)$ , if there is an exact string match of a sequence of words $\lbrace w_k, w_{k+1},    w_{l_{q_i}}\rbrace $ between the sentence $q_i$ and passage $p_j$   If this sequence is either a noun phrase, verb phrase, adjective phrase or a named entity in $\lbrace p_1, p_2,    p_m\rbrace $0 , as recognized by CoreNLP or BANNER, we select it as an answer span $\lbrace p_1, p_2,    p_m\rbrace $1   Additionally, we use $\lbrace p_1, p_2,    p_m\rbrace $2 as the passage $\lbrace p_1, p_2,    p_m\rbrace $3 and form a cloze question $\lbrace p_1, p_2,    p_m\rbrace $4 from the answer bearing sentence $\lbrace p_1, p_2,    p_m\rbrace $5 by replacing $\lbrace p_1, p_2,    p_m\rbrace $6 with a placeholder  As a result, we obtain passage-question-answer ( $\lbrace p_1, p_2,    p_m\rbrace $7 ) triples (Table 1 shows an example)  As a post-processing step, we prune out $\lbrace p_1, p_2,    p_m\rbrace $8 triples where the word overlap between the question (Q) and passage (P) is less than 2 words (after excluding the stop words) 
The process relies on the fact that answer candidates from the introduction are likely to be discussed in detail in the remainder of the article  In effect, the cloze question from the introduction and the matching paragraph in the body forms a question and context passage pair  We create two cloze datasets, one each from Wikipedia corpus (for SQuAD and TriviaQA) and PUBMed academic papers (for the BioASQ challenge), consisting of 2 2M and 1M clozes respectively  From analyzing the cloze data manually, we were able to answer 76% times for the Wikipedia set and 80% times for the PUBMed set using the information in the passage  In most cases the cloze paraphrased the information in the passage, which we hypothesized to be a useful signal for the downstream QA task 
We also investigate the utility of forming subsets of the large cloze corpus, where we select the top passage-question-answer triples, based on the different criteria, like i) jaccard similarity of answer bearing sentence in introduction and the passage ii) the tf-idf scores of answer candidates and iii) the length of answer candidates  However, we empirically find that we were better off using the entire set rather than these subsets 
Pre-training: We make use of the generated cloze dataset to pre-train an expressive neural network designed for the task of reading comprehension  We work with two publicly available neural network models – the GA Reader BIBREF2 (to enable comparison with prior work) and BiDAF + Self-Attention (SA) model from BIBREF1 (which is among the best performing models on SQuAD and TriviaQA)  After pretraining, the performance of BiDAF+SA on a dev set of the (Wikipedia) cloze questions is 0 58 F1 score and 0 55 Exact Match (EM) score  This implies that the cloze corpus is neither too easy, nor too difficult to answer 
Fine Tuning: We fine tune the pre-trained model, from the previous step, over a small set of labelled question-answer pairs  As we shall later see, this step is crucial, and it only requires a handful of labelled questions to achieve a significant proportion of the performance typically attained by training on tens of thousands of questions 
Datasets
We apply our system to three datasets from different domains  SQuAD BIBREF3 consists of questions whose answers are free form spans of text from passages in Wikipedia articles  We follow the same setting as in BIBREF0 , and split $10\%$ of training questions as the test set, and report performance when training on subsets of the remaining data ranging from $1\%$ to $90\%$ of the full set  We also report the performance on the dev set when trained on the full training set ( $1^\ast $ in Table 2 )  We use the same hyperparameter settings as in prior work  We compare and study four different settings: 1) the Supervised Learning (SL) setting, which is only trained on the supervised data, 2) the best performing GDAN model from BIBREF0 , 3) pretraining on a Language Modeling (LM) objective and fine-tuning on the supervised data, and 4) pretraining on the Cloze dataset and fine-tuning on the supervised data  The LM and Cloze methods use exactly the same data for pretraining, but differ in the loss functions used  We report F1 and EM scores on our test set using the official evaluation scripts provided by the authors of the dataset 
TriviaQA BIBREF4 comprises of over 95K web question-answer-evidence triples  Like SQuAD, the answers are spans of text  Similar to the setting in SQuAD, we create multiple smaller subsets of the entire set  For our semi-supervised QA system, we use the BiDAF+SA model BIBREF1 – the highest performing publicly available system for TrivaQA  Here again, we compare the supervised learning (SL) settings against the pretraining on Cloze set and fine tuning on the supervised set  We report F1 and EM scores on the dev set 
We also test on the BioASQ 5b dataset, which consists of question-answer pairs from PubMed abstracts  We use the publicly available system from BIBREF14 , and follow the exact same setup as theirs, focusing only on factoid and list questions  For this setting, there are only 899 questions for training  Since this is already a low-resource problem we only report results using 5-fold cross-validation on all the available data  We report Mean Reciprocal Rank (MRR) on the factoid questions, and F1 score for the list questions 
Main Results
Table 2 shows a comparison of the discussed settings on both SQuAD and TriviaQA  Without any fine-tuning (column 0) the performance is low, probably because the model never saw a real question, but we see significant gains with Cloze pretraining even with very little labeled data  The BiDAF+SA model, exceeds an F1 score of $50\%$ with only $1\%$ of the training data (454 questions for SQuAD, and 746 questions for TriviaQA), and approaches $90\%$ of the best performance with only $10\%$ labeled data  The gains over the SL setting, however, diminish as the size of the labeled set increases and are small when the full dataset is available 
Cloze pretraining outperforms the GDAN baseline from BIBREF0 using the same SQuAD dataset splits  Additionally, we show improvements in the $90\%$ data case unlike GDAN  Our approach is also applicable in the extremely low-resource setting of $1\%$ data, which we suspect GDAN might have trouble with since it uses the labeled data to do reinforcement learning  Furthermore, we are able to use the same cloze dataset to improve performance on both SQuAD and TriviaQA datasets  When we use the same unlabeled data to pre-train with a language modeling objective, the performance is worse, showing the bias we introduce by constructing clozes is important 
On the BioASQ dataset (Table 3 ) we again see a significant improvement when pretraining with the cloze questions over the supervised baseline  The improvement is smaller than what we observe with SQuAD and TriviaQA datasets – we believe this is because questions are generally more difficult in BioASQ  BIBREF14 showed that pretraining on SQuAD dataset improves the downstream performance on BioASQ  Here, we show a much larger improvement by pretraining on cloze questions constructed in an unsupervised manner from the same domain 
Analysis
Regression Analysis: To understand which types of questions benefit from pre-training, we pre-specified certain features (see Figure 1 right) for each of the dev set questions in SQuAD, and then performed linear regression to predict the F1 score for that question from these features  We predict the F1 scores from the cloze pretrained model ( $y^{\text{cloze}}$ ), the supervised model ( $y^{\text{sl}}$ ), and the difference of the two ( $y^{\text{cloze}}-y^{\text{sl}}$ ), when using $10\%$ of labeled data  The coefficients of the fitted model are shown in Figure 1 (left) along with their std errors  Positive coefficients indicate that a high value of that feature is predictive of a high F1 score, and a negative coefficient indicates that a small value of that feature is predictive of a high F1 score (or a high difference of F1 scores from the two models in the case of $y^{\text{cloze}}-y^{\text{sl}}$ ) 
The two strongest effects we observe are that a high lexical overlap between the question and the sentence containing the answer is indicative of high boost with pretraining, and that a high lexical overlap between the question and the whole passage is indicative of the opposite  This is hardly surprising, since our cloze construction process is biased towards questions which have a similar phrasing to the answer sentences in context  Hence, test questions with a similar property are answered correctly after pretraining, whereas those with a high overlap with the whole passage tend to have lower performance  The pretraining also favors questions with short answers because the cloze construction process produces short answer spans  Also passages and questions which consist of tokens infrequent in the SQuAD training corpus receive a large boost after pretraining, since the unlabeled data covers a larger domain 
Performance on question types: Figure 2 shows the average gain in F1 score for different types of questions, when we pretrain on the clozes compared to the supervised case  This analysis is done on the $10\%$ split of the SQuAD training set  We consider two classifications of each question – one determined on the first word (usually a wh-word) of the question (Figure 2 (bottom)) and one based on the output of a separate question type classifier adapted from BIBREF21   We use the coarse grain labels namely Abbreviation (ABBR), Entity (ENTY), Description (DESC), Human (HUM), Location (LOC), Numeric (NUM) trained on a Logistic Regression classification system   While there is an improvement across the board, we find that abbreviation questions in particular receive a large boost  Also, "why" questions show the least improvement, which is in line with our expectation, since these usually require reasoning or world knowledge which cloze questions rarely require 
Conclusion
In this paper, we show that pre-training QA models with automatically constructed cloze questions improves the performance of the models significantly, especially when there are few labeled examples  The performance of the model trained only on the cloze questions is poor, validating the need for fine-tuning  Through regression analysis, we find that pretraining helps with questions which ask for factual information located in a specific part of the context  For future work, we plan to explore the active learning setup for this task – specifically, which passages and / or types of questions can we select to annotate, such that there is a maximum performance gain from fine-tuning  We also want to explore how to adapt cloze style pre-training to NLP tasks other than QA 
Acknowledgments
Bhuwan Dhingra is supported by NSF under grants CCF-1414030 and IIS-1250956 and by grants from Google  Danish Pruthi and Dheeraj Rajagopal are supported by the DARPA Big Mechanism program under ARO contract W911NF-14-1-0436 
Context 3: Introduction
Grammar induction is the task of inducing hierarchical syntactic structure from data  Statistical approaches to grammar induction require specifying a probabilistic grammar (e g  formalism, number and shape of rules), and fitting its parameters through optimization  Early work found that it was difficult to induce probabilistic context-free grammars (PCFG) from natural language data through direct methods, such as optimizing the log likelihood with the EM algorithm BIBREF0 , BIBREF1   While the reasons for the failure are manifold and not completely understood, two major potential causes are the ill-behaved optimization landscape and the overly strict independence assumptions of PCFGs  More successful approaches to grammar induction have thus resorted to carefully-crafted auxiliary objectives BIBREF2 , priors or non-parametric models BIBREF3 , BIBREF4 , BIBREF5 , BIBREF6 , and manually-engineered features BIBREF7 , BIBREF8 to encourage the desired structures to emerge 
We revisit these aforementioned issues in light of advances in model parameterization and inference  First, contrary to common wisdom, we find that parameterizing a PCFG's rule probabilities with neural networks over distributed representations makes it possible to induce linguistically meaningful grammars by simply optimizing log likelihood  While the optimization problem remains non-convex, recent work suggests that there are optimization benefits afforded by over-parameterized models BIBREF9 , BIBREF10 , BIBREF11 , and we indeed find that this neural PCFG is significantly easier to optimize than the traditional PCFG  Second, this factored parameterization makes it straightforward to incorporate side information into rule probabilities through a sentence-level continuous latent vector, which effectively allows different contexts in a derivation to coordinate  In this compound PCFG—continuous mixture of PCFGs—the context-free assumptions hold conditioned on the latent vector but not unconditionally, thereby obtaining longer-range dependencies within a tree-based generative process 
To utilize this approach, we need to efficiently optimize the log marginal likelihood of observed sentences  While compound PCFGs break efficient inference, if the latent vector is known the distribution over trees reduces to a standard PCFG  This property allows us to perform grammar induction using a collapsed approach where the latent trees are marginalized out exactly with dynamic programming  To handle the latent vector, we employ standard amortized inference using reparameterized samples from a variational posterior approximated from an inference network BIBREF12 , BIBREF13  
On standard benchmarks for English and Chinese, the proposed approach is found to perform favorably against recent neural network-based approaches to grammar induction BIBREF14 , BIBREF15 , BIBREF16 , BIBREF17  
Probabilistic Context-Free Grammars
We consider context-free grammars (CFG) consisting of a 5-tuple INLINEFORM0 where INLINEFORM1 is the distinguished start symbol, INLINEFORM2 is a finite set of nonterminals, INLINEFORM3 is a finite set of preterminals, INLINEFORM6 is a finite set of terminal symbols, and INLINEFORM7 is a finite set of rules of the form,
INLINEFORM0
A probabilistic context-free grammar (PCFG) consists of a grammar INLINEFORM0 and rule probabilities INLINEFORM1 such that INLINEFORM2 is the probability of the rule INLINEFORM3   Letting INLINEFORM4 be the set of all parse trees of INLINEFORM5 , a PCFG defines a probability distribution over INLINEFORM6 via INLINEFORM7 where INLINEFORM8 is the set of rules used in the derivation of INLINEFORM9   It also defines a distribution over string of terminals INLINEFORM10 via
INLINEFORM0
where INLINEFORM0 , i e  the set of trees INLINEFORM1 such that INLINEFORM2 's leaves are INLINEFORM3   We will slightly abuse notation and use
INLINEFORM0
to denote the posterior distribution over the unobserved latent trees given the observed sentence INLINEFORM0 , where INLINEFORM1 is the indicator function 
Compound PCFGs
A compound probability distribution BIBREF19 is a distribution whose parameters are themselves random variables  These distributions generalize mixture models to the continuous case, for example in factor analysis which assumes the following generative process,
INLINEFORM0
Compound distributions provide the ability to model rich generative processes, but marginalizing over the latent parameter can be computationally intractable unless conjugacy can be exploited 
In this work, we study compound probabilistic context-free grammars whose distribution over trees arises from the following generative process: we first obtain rule probabilities via
INLINEFORM0
where INLINEFORM0 is a prior with parameters INLINEFORM1 (spherical Gaussian in this paper), and INLINEFORM2 is a neural network that concatenates the input symbol embeddings with INLINEFORM3 and outputs the sentence-level rule probabilities INLINEFORM4 ,
INLINEFORM0
where INLINEFORM0 denotes vector concatenation  Then a tree/sentence is sampled from a PCFG with rule probabilities given by INLINEFORM1 ,
INLINEFORM0
This can be viewed as a continuous mixture of PCFGs, or alternatively, a Bayesian PCFG with a prior on sentence-level rule probabilities parameterized by INLINEFORM0   Importantly, under this generative model the context-free assumptions hold conditioned on INLINEFORM3 , but they do not hold unconditionally  This is shown in Figure FIGREF3 (right) where there is a dependence path through INLINEFORM4 if it is not conditioned upon  Compound PCFGs give rise to a marginal distribution over parse trees INLINEFORM5 via
INLINEFORM0
where INLINEFORM0   The subscript in INLINEFORM1 denotes the fact that the rule probabilities depend on INLINEFORM2   Compound PCFGs are clearly more expressive than PCFGs as each sentence has its own set of rule probabilities  However, it still assumes a tree-based generative process, making it possible to learn latent tree structures 
Our motivation for the compound PCFG is based on the observation that for grammar induction, context-free assumptions are generally made not because they represent an adequate model of natural language, but because they allow for tractable training  We can in principle model richer dependencies through vertical/horizontal Markovization BIBREF21 , BIBREF22 and lexicalization BIBREF23   However such dependencies complicate training due to the rapid increase in the number of rules  Under this view, we can interpret the compound PCFG as a restricted version of some lexicalized, higher-order PCFG where a child can depend on structural and lexical context through a shared latent vector  We hypothesize that this dependence among siblings is especially useful in grammar induction from words, where (for example) if we know that watched is used as a verb then the noun phrase is likely to be a movie 
In contrast to the usual Bayesian treatment of PCFGs which places priors on global rule probabilities BIBREF3 , BIBREF4 , BIBREF6 , the compound PCFG assumes a prior on local, sentence-level rule probabilities  It is therefore closely related to the Bayesian grammars studied by BIBREF25 and BIBREF26 , who also sample local rule probabilities from a logistic normal prior for training dependency models with valence (DMV) BIBREF27  
Experimental Setup
Results and Discussion
Table TABREF23 shows the unlabeled INLINEFORM0 scores for our models and various baselines  All models soundly outperform right branching baselines, and we find that the neural PCFG/compound PCFG are strong models for grammar induction  In particular the compound PCFG outperforms other models by an appreciable margin on both English and Chinese  We again note that we were unable to induce meaningful grammars through a traditional PCFG with the scalar parameterization despite a thorough hyperparameter search  See lab:full for the full results (including corpus-level INLINEFORM1 ) broken down by sentence length 
Table TABREF27 analyzes the learned tree structures  We compare similarity as measured by INLINEFORM0 against gold, left, right, and “self" trees (top), where self INLINEFORM1 score is calculated by averaging over all 6 pairs obtained from 4 different runs  We find that PRPN is particularly consistent across multiple runs  We also observe that different models are better at identifying different constituent labels, as measured by label recall (Table TABREF27 , bottom)  While left as future work, this naturally suggests an ensemble approach wherein the empirical probabilities of constituents (obtained by averaging the predicted binary constituent labels from the different models) are used either to supervise another model or directly as potentials in a CRF constituency parser  Finally, all models seemed to have some difficulty in identifying SBAR/VP constituents which typically span more words than NP constituents 
Related Work
Grammar induction has a long and rich history in natural language processing  Early work on grammar induction with pure unsupervised learning was mostly negative BIBREF0 , BIBREF1 , BIBREF74 , though BIBREF75 reported some success on partially bracketed data  BIBREF76 and BIBREF2 were some of the first successful statistical approaches to grammar induction  In particular, the constituent-context model (CCM) of BIBREF2 , which explicitly models both constituents and distituents, was the basis for much subsequent work BIBREF27 , BIBREF7 , BIBREF8   Other works have explored imposing inductive biases through Bayesian priors BIBREF4 , BIBREF5 , BIBREF6 , modified objectives BIBREF42 , and additional constraints on recursion depth BIBREF77 , BIBREF48 

ground_truth: the English-German dataset

prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. Is it possible to convert a cloze-style questions to a naturally-looking questions?
Context 4: Introduction
Word embeddings are representations of words in numerical form, as vectors of typically several hundred dimensions  The vectors are used as an input to machine learning models; for complex language processing tasks these are typically deep neural networks  The embedding vectors are obtained from specialized learning tasks, based on neural networks, e g , word2vec BIBREF0, GloVe BIBREF1, FastText BIBREF2, ELMo BIBREF3, and BERT BIBREF4  For training, the embeddings algorithms use large monolingual corpora that encode important information about word meaning as distances between vectors  In order to enable downstream machine learning on text understanding tasks, the embeddings shall preserve semantic relations between words, and this is true even across languages 
Probably the best known word embeddings are produced by the word2vec method BIBREF5  The problem with word2vec embeddings is their failure to express polysemous words  During training of an embedding, all senses of a given word (e g , paper as a material, as a newspaper, as a scientific work, and as an exam) contribute relevant information in proportion to their frequency in the training corpus  This causes the final vector to be placed somewhere in the weighted middle of all words' meanings  Consequently, rare meanings of words are poorly expressed with word2vec and the resulting vectors do not offer good semantic representations  For example, none of the 50 closest vectors of the word paper is related to science 
The idea of contextual embeddings is to generate a different vector for each context a word appears in and the context is typically defined sentence-wise  To a large extent, this solves the problems with word polysemy, i e  the context of a sentence is typically enough to disambiguate different meanings of a word for humans and so it is for the learning algorithms  In this work, we describe high-quality models for contextual embeddings, called ELMo BIBREF3, precomputed for seven morphologically rich, less-resourced languages: Slovenian, Croatian, Finnish, Estonian, Latvian, Lithuanian, and Swedish  ELMo is one of the most successful approaches to contextual word embeddings  At time of its creation, ELMo has been shown to outperform previous word embeddings BIBREF3 like word2vec and GloVe on many NLP tasks, e g , question answering, named entity extraction, sentiment analysis, textual entailment, semantic role labeling, and coreference resolution 
This report is split into further five sections  In section SECREF2, we describe the contextual embeddings ELMo  In Section SECREF3, we describe the datasets used and in Section SECREF4 we describe preprocessing and training of the embeddings  We describe the methodology for evaluation of created vectors and results in Section SECREF5  We present conclusion in Section SECREF6 where we also outline plans for further work 
ELMo
Typical word embeddings models or representations, such as word2vec BIBREF0, GloVe BIBREF1, or FastText BIBREF2, are fast to train and have been pre-trained for a number of different languages  They do not capture the context, though, so each word is always given the same vector, regardless of its context or meaning  This is especially problematic for polysemous words  ELMo (Embeddings from Language Models) embedding BIBREF3 is one of the state-of-the-art pretrained transfer learning models, that remedies the problem and introduces a contextual component 
ELMo model`s architecture consists of three neural network layers  The output of the model after each layer gives one set of embeddings, altogether three sets  The first layer is a CNN layer, which operates on a character level  It is context independent, so each word always gets the same embedding, regardless of its context  It is followed by two biLM layers  A biLM layer consists of two concatenated LSTMs  In the first LSTM, we try to predict the following word, based on the given past words, where each word is represented by the embeddings from the CNN layer  In the second LSTM, we try to predict the preceding word, based on the given following words  It is equivalent to the first LSTM, just reading the text in reverse 
In NLP tasks, any set of these embeddings may be used; however, a weighted average is usually used  The weights of the average are learned during the training of the model for the specific task  Additionally, an entire ELMo model can be fine-tuned on a specific end task 
Although ELMo is trained on character level and is able to handle out-of-vocabulary words, a vocabulary file containing most common tokens is used for efficiency during training and embedding generation  The original ELMo model was trained on a one billion word large English corpus, with a given vocabulary file of about 800,000 words  Later, ELMo models for other languages were trained as well, but limited to larger languages with many resources, like German and Japanese 
ELMo ::: ELMoForManyLangs
Recently, ELMoForManyLangs BIBREF6 project released pre-trained ELMo models for a number of different languages BIBREF7  These models, however, were trained on a significantly smaller datasets  They used 20-million-words data randomly sampled from the raw text released by the CoNLL 2017 Shared Task - Automatically Annotated Raw Texts and Word Embeddings BIBREF8, which is a combination of Wikipedia dump and common crawl  The quality of these models is questionable  For example, we compared the Latvian model by ELMoForManyLangs with a model we trained on a complete (wikidump + common crawl) Latvian corpus, which has about 280 million tokens  The difference of each model on the word analogy task is shown in Figure FIGREF16 in Section SECREF5  As the results of the ELMoForManyLangs embeddings are significantly worse than using the full corpus, we can conclude that these embeddings are not of sufficient quality  For that reason, we computed ELMo embeddings for seven languages on much larger corpora  As this effort requires access to large amount of textual data and considerable computational resources, we made the precomputed models publicly available by depositing them to Clarin repository 
Training Data
We trained ELMo models for seven languages: Slovenian, Croatian, Finnish, Estonian, Latvian, Lithuanian and Swedish  To obtain high-quality embeddings, we used large monolingual corpora from various sources for each language  Some corpora are available online under permissive licences, others are available only for research purposes or have limited availability  The corpora used in training datasets are a mix of news articles and general web crawl, which we preprocessed and deduplicated  Below we shortly describe the used corpora in alphabetical order of the involved languages  Their names and sizes are summarized in Table TABREF3 
Croatian dataset include hrWaC 2 1 corpus BIBREF9, Riznica BIBREF10, and articles of Croatian branch of Styria media house, made available to us through partnership in a joint project  hrWaC was built by crawling the  hr internet domain in 2011 and 2014  Riznica is composed of Croatian fiction and non-fiction prose, poetry, drama, textbooks, manuals, etc  The Styria dataset consists of 570,219 news articles published on the Croatian 24sata news portal and niche portals related to 24sata 
Estonian dataset contains texts from two sources, CoNLL 2017 Shared Task - Automatically Annotated Raw Texts and Word Embeddings BIBREF8, and news articles made available to us by Ekspress Meedia due to partnership in the project  Ekspress Meedia dataset is composed of Estonian news articles between years 2009 and 2019  The CoNLL 2017 corpus is composed of Estonian Wikipedia and webcrawl 
Finnish dataset contains articles by Finnish news agency STT, Finnish part of the CoNLL 2017 dataset, and Ylilauta downloadable version BIBREF11  STT news articles were published between years 1992 and 2018  Ylilauta is a Finnish online discussion board; the corpus contains parts of the discussions from 2012 to 2014 
Latvian dataset consists only of the Latvian portion of the ConLL 2017 corpus 
Lithuanian dataset is composed of Lithuanian Wikipedia articles from 2018, DGT-UD corpus, and LtTenTen  DGT-UD is a parallel corpus of 23 official languages of the EU, composed of JRC DGT translation memory of European law, automatically annotated with UD-Pipe 1 2  LtTenTen is Lithuanian web corpus made up of texts collected from the internet in April 2014 BIBREF12 
Slovene dataset is formed from the Gigafida 2 0 corpus BIBREF13  It is a general language corpus composed of various sources, mostly newspapers, internet pages, and magazines, but also fiction and non-fiction prose, textbooks, etc 
Swedish dataset is composed of STT Swedish articles and Swedish part of CoNLL 2017  The Finnish news agency STT publishes some of its articles in Swedish language  They were made available to us through partnership in a joint project  The corpus contains those articles from 1992 to 2017 
Preprocessing and Training
Prior to training the ELMo models, we sentence and word tokenized all the datasets  The text was formatted in such a way that each sentence was in its own line with tokens separated by white spaces  CoNLL 2017, DGT-UD and LtTenTen14 corpora were already pre-tokenized  We tokenized the others using the NLTK library and its tokenizers for each of the languages

ground_truth: Unanswerable

Unanswerable

prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. which chinese datasets were used?
Context 3: 
While the framework of specifying the structure of a grammar and learning the parameters is common, other methods exist  BIBREF43 consider a nonparametric-style approach to unsupervised parsing by using random subsets of training subtrees to parse new sentences  BIBREF46 utilize an incremental algorithm to unsupervised parsing which makes local decisions to create constituents based on a complex set of heuristics  BIBREF47 induce parse trees through cascaded applications of finite state models 
More recently, neural network-based approaches to grammar induction have shown promising results on inducing parse trees directly from words  BIBREF14 , BIBREF15 learn tree structures through soft gating layers within neural language models, while BIBREF16 combine recursive autoencoders with the inside-outside algorithm  BIBREF17 train unsupervised recurrent neural network grammars with a structured inference network to induce latent trees, and BIBREF78 utilize image captions to identify and ground constituents 
Our work is also related to latent variable PCFGs BIBREF79 , BIBREF80 , BIBREF81 , which extend PCFGs to the latent variable setting by splitting nonterminal symbols into latent subsymbols  In particular, latent vector grammars BIBREF82 and compositional vector grammars BIBREF83 also employ continuous vectors within their grammars  However these approaches have been employed for learning supervised parsers on annotated treebanks, in contrast to the unsupervised setting of the current work 
Conclusion
This work explores grammar induction with compound PCFGs, which modulate rule probabilities with per-sentence continuous latent vectors  The latent vector induces marginal dependencies beyond the traditional first-order context-free assumptions within a tree-based generative process, leading to improved performance  The collapsed amortized variational inference approach is general and can be used for generative models which admit tractable inference through partial conditioning  Learning deep generative models which exhibit such conditional Markov properties is an interesting direction for future work 
Acknowledgments
We thank Phil Blunsom for initial discussions which seeded many of the core ideas in the present work  We also thank Yonatan Belinkov and Shay Cohen for helpful feedback, and Andrew Drozdov for providing the parsed dataset from their DIORA model  YK is supported by a Google Fellowship  AMR acknowledges the support of NSF 1704834, 1845664, AWS, and Oracle 
Model Parameterization
We associate an input embedding INLINEFORM0 for each symbol INLINEFORM1 on the left side of a rule (i e  INLINEFORM2 ) and run a neural network over INLINEFORM3 to obtain the rule probabilities  Concretely, each rule type INLINEFORM4 is parameterized as follows, INLINEFORM5
where INLINEFORM0 is the product space INLINEFORM1 , and INLINEFORM2 are MLPs with two residual layers, INLINEFORM3
The bias terms for the above expressions (including for the rule probabilities) are omitted for notational brevity  In Figure FIGREF3 we use the following to refer to rule probabilities of different rule types, INLINEFORM0
where INLINEFORM0 denotes the set of rules with INLINEFORM1 on the left hand side 
The compound PCFG rule probabilities INLINEFORM0 given a latent vector INLINEFORM1 , INLINEFORM2
Again the bias terms are omitted for brevity, and INLINEFORM0 are as before where the first layer's input dimensions are appropriately changed to account for concatenation with INLINEFORM1  
Corpus/Sentence F 1 F_1 by Sentence Length
For completeness we show the corpus-level and sentence-level INLINEFORM0 broken down by sentence length in Table TABREF44 , averaged across 4 different runs of each model 
Experiments with RNNGs
For experiments on supervising RNNGs with induced trees, we use the parameterization and hyperparameters from BIBREF17 , which uses a 2-layer 650-dimensional stack LSTM (with dropout of 0 5) and a 650-dimensional tree LSTM BIBREF88 , BIBREF90 as the composition function 
Concretely, the generative story is as follows: first, the stack representation is used to predict the next action (shift or reduce) via an affine transformation followed by a sigmoid  If shift is chosen, we obtain a distribution over the vocabulary via another affine transformation over the stack representation followed by a softmax  Then we sample the next word from this distribution and shift the generated word onto the stack using the stack LSTM  If reduce is chosen, we pop the last two elements off the stack and use the tree LSTM to obtain a new representation  This new representation is shifted onto the stack via the stack LSTM  Note that this RNNG parameterization is slightly different than the original from BIBREF53 , which does not ignore constituent labels and utilizes a bidirectional LSTM as the composition function instead of a tree LSTM  As our RNNG parameterization only works with binary trees, we binarize the gold trees with right binarization for the RNNG trained on gold trees (trees from the unsupervised methods explored in this paper are already binary)  The RNNG also trains a discriminative parser alongside the generative model for evaluation with importance sampling  We use a CRF parser whose span score parameterization is similar similar to recent works BIBREF89 , BIBREF87 , BIBREF85 : position embeddings are added to word embeddings, and a bidirectional LSTM with 256 hidden dimensions is run over the input representations to obtain the forward and backward hidden states  The score INLINEFORM0 for a constituent spanning the INLINEFORM1 -th and INLINEFORM2 -th word is given by,
INLINEFORM0
where the MLP has a single hidden layer with INLINEFORM0 nonlinearity followed by layer normalization BIBREF84  
For experiments on fine-tuning the RNNG with the unsupervised RNNG, we take the discriminative parser (which is also pretrained alongside the RNNG on induced trees) to be the structured inference network for optimizing the evidence lower bound  We refer the reader to BIBREF17 and their open source implementation for additional details  We also observe that as noted by BIBREF17 , a URNNG trained from scratch on this version of PTB without punctuation failed to outperform a right-branching baseline 
The LSTM language model baseline is the same size as the stack LSTM (i e  2 layers, 650 hidden units, dropout of 0 5), and is therefore equivalent to an RNNG with completely right branching trees  The PRPN/ON baselines for perplexity/syntactic evaluation in Table TABREF30 also have 2 layers with 650 hidden units and 0 5 dropout  Therefore all models considered in Table TABREF30 have roughly the same capacity  For all models we share input/output word embeddings BIBREF86   Perplexity estimation for the RNNGs and the compound PCFG uses 1000 importance-weighted samples 
For grammaticality judgment, we modify the publicly available dataset from BIBREF56 to only keep sentence pairs that did not have any unknown words with respect to our PTB vocabulary of 10K words  This results in 33K sentence pairs for evaluation 
Nonterminal/Preterminal Alignments
Figure FIGREF50 shows the part-of-speech alignments and Table TABREF46 shows the nonterminal label alignments for the compound PCFG/neural PCFG 
Subtree Analysis
Table TABREF53 lists more examples of constituents within each subtree as the top principical component is varied  Due to data sparsity, the subtree analysis is performed on the full dataset  See section UID36 for more details 

ground_truth: Answer with content missing: (Data section) Chinese with version 5.1 of the Chinese Penn Treebank (CTB)