prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. How does their ensemble method work?
Context 1: Introduction
Since humans amass more and more generally available data in the form of unstructured text it would be very useful to teach machines to read and comprehend such data and then use this understanding to answer our questions  A significant amount of research has recently focused on answering one particular kind of questions the answer to which depends on understanding a context document  These are cloze-style questions BIBREF0 which require the reader to fill in a missing word in a sentence  An important advantage of such questions is that they can be generated automatically from a suitable text corpus which allows us to produce a practically unlimited amount of them  That opens the task to notoriously data-hungry deep-learning techniques which now seem to outperform all alternative approaches 
Two such large-scale datasets have recently been proposed by researchers from Google DeepMind and Facebook AI: the CNN/Daily Mail dataset BIBREF1 and the Children's Book Test (CBT) BIBREF2 respectively  These have attracted a lot of attention from the research community BIBREF3 , BIBREF4 , BIBREF5 , BIBREF6 , BIBREF7 , BIBREF8 , BIBREF9 , BIBREF10 , BIBREF11 , BIBREF12 with a new state-of-the-art model coming out every few weeks 
However if our goal is a production-level system actually capable of helping humans, we want the model to use all available resources as efficiently as possible  Given that
we believe that if the community is striving to bring the performance as far as possible, it should move its work to larger data 
This thinking goes in line with recent developments in the area of language modelling  For a long time models were being compared on several "standard" datasets with publications often presenting minuscule improvements in performance  Then the large-scale One Billion Word corpus dataset appeared BIBREF15 and it allowed Jozefowicz et al  to train much larger LSTM models BIBREF16 that almost halved the state-of-the-art perplexity on this dataset 
We think it is time to make a similar step in the area of text comprehension  Hence we are introducing the BookTest, a new dataset very similar to the Children's Book test but more than 60 times larger to enable training larger models even in the domain of text comprehension  Furthermore the methodology used to create our data can later be used to create even larger datasets when the need arises thanks to further technological progress 
We show that if we evaluate a model trained on the new dataset on the now standard Children's Book Test dataset, we see an improvement in accuracy much larger than other research groups achieved by enhancing the model architecture itself (while still using the original CBT training data)  By training on the new dataset, we reduce the prediction error by almost one third  On the named-entity version of CBT this brings the ensemble of our models to the level of human baseline as reported by Facebook BIBREF2   However in the final section we show in our own human study that there is still room for improvement on the CBT beyond the performance of our model 
Task Description
A natural way of testing a reader's comprehension of a text is to ask her a question the answer to which can be deduced from the text  Hence the task we are trying to solve consists of answering a cloze-style question, the answer to which depends on the understanding of a context document provided with the question  The model is also provided with a set of possible answers from which the correct one is to be selected  This can be formalized as follows:
The training data consist of tuples INLINEFORM0 , where INLINEFORM1 is a question, INLINEFORM2 is a document that contains the answer to question INLINEFORM3 , INLINEFORM4 is a set of possible answers and INLINEFORM5 is the ground-truth answer  Both INLINEFORM6 and INLINEFORM7 are sequences of words from vocabulary INLINEFORM8   We also assume that all possible answers are words from the vocabulary, that is INLINEFORM9   In the CBT and CNN/Daily Mail datasets it is also true that the ground-truth answer INLINEFORM10 appears in the document  This is exploited by many machine learning models BIBREF2 , BIBREF4 , BIBREF6 , BIBREF7 , BIBREF8 , BIBREF10 , BIBREF11 , BIBREF12 , however some do not explicitly depend on this property BIBREF1 , BIBREF3 , BIBREF5 , BIBREF9
Current Landscape
We will now briefly review what datasets for text comprehension have been published up to date and look at models which have been recently applied to solving the task we have just described 
Datasets
A crucial condition for applying deep-learning techniques is to have a huge amount of data available for training  For question answering this specifically means having a large number of document-question-answer triples available  While there is an unlimited amount of text available, coming up with relevant questions and the corresponding answers can be extremely labour-intensive if done by human annotators  There were efforts to provide such human-generated datasets, e g  Microsoft's MCTest BIBREF17 , however their scale is not suitable for deep learning without pre-training on other data BIBREF18 (such as using pre-trained word embedding vectors) 
Google DeepMind managed to avoid this scale issue with their way of generating document-question-answer triples automatically, closely followed by Facebook with a similar method  Let us now briefly introduce the two resulting datasets whose properties are summarized in Table TABREF8  
These two datasets BIBREF1 exploit a useful feature of online news articles – many articles include a short summarizing sentence near the top of the page  Since all information in the summary sentence is also presented in the article body, we get a nice cloze-style question about the article contents by removing a word from the short summary 
The dataset's authors also replaced all named entities in the dataset by anonymous tokens which are further shuffled for each new batch  This forces the model to rely solely on information from the context document, not being able to transfer any meaning of the named entities between documents 
This restricts the task to one specific aspect of context-dependent question answering which may be useful however it moves the task further from the real application scenario, where we would like the model to use all information available to answer questions  Furthermore Chen et al  BIBREF5 have suggested that this can make about 17% of the questions unanswerable even by humans  They also claim that more than a half of the question sentences are mere paraphrases or exact matches of a single sentence from the context document  This raises a question to what extent the dataset can test deeper understanding of the articles 
The Children's Book Test BIBREF2 uses a different source - books freely available thanks to Project Gutenberg  Since no summary is available, each example consists of a context document formed from 20 consecutive sentences from the story together with a question formed from the subsequent sentence 
The dataset comes in four flavours depending on what type of word is omitted from the question sentence  Based on human evaluation done in BIBREF2 it seems that NE and CN are more context dependent than the other two types – prepositions and verbs  Therefore we (and all of the recent publications) focus only on these two word types 
Several new datasets related to the (now almost standard) ones above emerged recently  We will now briefly present them and explain how the dataset we are introducing in this article differs from them 
The LAMBADA dataset BIBREF19 is designed to measure progress in understanding common-sense questions about short stories that can be easily answered by humans but cannot be answered by current standard machine-learning models (e g  plain LSTM language models)  This dataset is useful for measuring the gap between humans and machine learning algorithms  However, by contrast to our BookTest dataset, it will not allow us to track progress towards the performance of the baseline systems or on examples where machine learning may show super-human performance  Also LAMBADA is just a diagnostic dataset and does not provide ready-to-use question-answering training data, just a plain-text corpus which may moreover include copyrighted books making its use potentially problematic for some purposes  We are providing ready training data consisting of copyright-free books only 
The SQuAD dataset BIBREF20 based on Wikipedia and the Who-did-What dataset BIBREF21 based on Gigaword news articles are factoid question-answering datasets where a multi-word answer should be extracted from a context document  This is in contrast to the previous datasets, including CNN/DM, CBT, LAMBADA and our new dataset, which require only single-word answers  Both these datasets however provide less than 130,000 training questions, two orders of magnitude less than our dataset does 
The Story Cloze Test BIBREF22 provides a crowd-sourced corpus of 49,255 commonsense stories for training and 3,744 testing stories with right and wrong endings  Hence the dataset is again rather small  Similarly to LAMBADA, the Story Cloze Test was designed to be easily answerable by humans 
In the WikiReading BIBREF23 dataset the context document is formed from a Wikipedia article and the question-answer pair is taken from the corresponding WikiData page  For each entity (e g  Hillary Clinton), WikiData contain a number of property-value pairs (e g  place of birth: Chicago) which form the datasets's question-answer pairs  The dataset is certainly relevant to the community, however the questions are of very limited variety with only 20 properties (and hence unique questions) covering INLINEFORM0 of the dataset  Furthermore many of the frequent properties are mentioned at a set spot within the article (e g  the date of birth is almost always in brackets behind the name of a person) which may make the task easier for machines  We are trying to provide a more varied dataset 
Although there are several datasets related to task we are aiming to solve, they differ sufficiently for our dataset to bring new value to the community  Its biggest advantage is its size which can furthermore be easily upscaled without expensive human annotation  Finally while we are emphasizing the differences, models could certainly benefit from as diverse a collection of datasets as possible 
Machine Learning Models
A first major work applying deep-learning techniques to text comprehension was Hermann et al  BIBREF1   This work was followed by the application of Memory Networks to the same task BIBREF2   Later three models emerged around the same time BIBREF3 , BIBREF4 , BIBREF5 including our psr model BIBREF4   The AS Reader inspired several subsequent models that use it as a sub-component in a diverse ensemble BIBREF8 ; extend it with a hierarchical structure BIBREF6 , BIBREF24 , BIBREF7 ; compute attention over the context document for every word in the query BIBREF10 or use two-way context-query attention mechanism for every word in the context and the query BIBREF11 that is similar in its spirit to models recently proposed in different domains, e g  BIBREF25 in information retrieval  Other neural approaches to text comprehension are explored in BIBREF9 , BIBREF12  
Possible Directions for Improvements
Accuracy in any machine learning tasks can be enhanced either by improving a machine learning model or by using more in-domain training data  Current state of the art models BIBREF6 , BIBREF7 , BIBREF8 , BIBREF11 improve over AS Reader's accuracy on CBT NE and CN datasets by 1-2 percent absolute  This suggests that with current techniques there is only limited room for improvement on the algorithmic side 
The other possibility to improve performance is simply to use more training data  The importance of training data was highlighted by the frequently quoted Mercer's statement that “There is no data like more data ” The observation that having more data is often more important than having better algorithms has been frequently stressed since then BIBREF13 , BIBREF14  
As a step in the direction of exploiting the potential of more data in the domain of text comprehension, we created a new dataset called BookTest similar to, but much larger than the widely used CBT and CNN/DM datasets 
BookTest
Similarly to the CBT, our BookTest dataset is derived from books available through project Gutenberg  We used 3555 copyright-free books to extract CN examples and 10507 books for NE examples, for comparison the CBT dataset was extracted from just 108 books 
When creating our dataset we follow the same procedure as was used to create the CBT dataset BIBREF2   That is, we detect whether each sentence contains either a named entity or a common noun that already appeared in one of the preceding twenty sentences  This word is then replaced by a gap tag (XXXXX) in this sentence which is hence turned into a cloze-style question  The preceding 20 sentences are used as the context document  For common noun and named entity detection we use the Stanford POS tagger BIBREF27 and Stanford NER BIBREF28  
The training dataset consists of the original CBT NE and CN data extended with new NE and CN examples  The new BookTest dataset hence contains INLINEFORM0 training examples and INLINEFORM1 tokens 
The validation dataset consists of INLINEFORM0 NE and INLINEFORM1 CN questions  We have one test set for NE and one for CN, each containing INLINEFORM2 examples  The training, validation and test sets were generated from non-overlapping sets of books 
When generating the dataset we removed all editions of books used to create CBT validation and test sets from our training dataset  Therefore the models trained on the BookTest corpus can be evaluated on the original CBT data and they can be compared with recent text-comprehension models utilizing this dataset BIBREF2 , BIBREF4 , BIBREF5 , BIBREF6 , BIBREF7 , BIBREF8 , BIBREF9 , BIBREF10 , BIBREF11  
Baselines
We will now use our psr model to evaluate the performance gain from increasing the dataset size 
AS Reader
In BIBREF4 we introduced the psr , which at the time of publication significantly outperformed all other architectures on the CNN, DM and CBT datasets  This model is built to leverage the fact that the answer is a single word from the context document  Similarly to many other models it uses attention over the document – intuitively a measure of how relevant each word is to answering the question  However while most previous models used this attention as weights to calculate a blended representation of the answer word, we simply sum the attention across all occurrences of each unique words and then simply select the word with the highest sum as the final answer  While simple, this trick seems both to improve accuracy and to speed-up training  It was adopted by many subsequent models BIBREF8 , BIBREF6 , BIBREF7 , BIBREF10 , BIBREF11 , BIBREF24  
Let us now describe the model in more detail  Figure FIGREF21 may help you in understanding the following paragraphs 
The words from the document and the question are first converted into vector embeddings using a look-up matrix INLINEFORM0   The document is then read by a bidirectional GRU network BIBREF29   A concatenation of the hidden states of the forward and backward GRUs at each word is then used as a contextual embedding of this word, intuitively representing the context in which the word is appearing  We can also understand it as representing the set of questions to which this word may be an answer 
Similarly the question is read by a bidirectional GRU but in this case only the final hidden states are concatenated to form the question embedding 
The attention over each word in the context is then calculated as the dot product of its contextual embedding with the question embedding  This attention is then normalized by the softmax function and summed across all occurrences of each answer candidate  The candidate with most accumulated attention is selected as the final answer 
For a more detailed description of the model including equations check BIBREF4   More details about the training setup and model hyperparameters can be found in the Appendix 
During our past experiments on the CNN, DM and CBT datasets BIBREF4 each unique word from the training, validation and test datasets had its row in the look-up matrix INLINEFORM0   However as we radically increased the dataset size, this would result in an extremely large number of model parameters so we decided to limit the vocabulary size to INLINEFORM1 most frequent words  For each example, each unique out-of-vocabulary word is now mapped on one of 1000 anonymous tokens which are randomly initialized and untrained  Fixing the embeddings of these anonymous tags proved to significantly improve the performance 
While mostly using the original AS Reader model, we have also tried introducing a minor tweak in some instances of the model  We tried initializing the context encoder GRU's hidden state by letting the encoder read the question first before proceeding to read the context document  Intuitively this allows the encoder to know in advance what to look for when reading over the context document 
Including models of this kind in the ensemble helped to improve the performance 
Results
Table TABREF25 shows the accuracy of the psr and other architectures on the CBT validation and test data  The last two rows show the performance of the psr trained on the BookTest dataset; all the other models have been trained on the original CBT training data 
If we take the best psr ensemble trained on CBT as a baseline, improving the model architecture as in BIBREF6 , BIBREF7 , BIBREF8 , BIBREF9 , BIBREF10 , BIBREF11 , continuing to use the original CBT training data, lead to improvements of INLINEFORM0 and INLINEFORM1 absolute on named entities and common nouns respectively  By contrast, inflating the training dataset provided a boost of INLINEFORM2 while using the same model  The ensemble of our models even exceeded the human baseline provided by Facebook BIBREF2 on the Common Noun dataset 
Our model takes approximately two weeks to converge when trained on the BookTest dataset on a single Nvidia Tesla K40 GPU 
Discussion
Embracing the abundance of data may mean focusing on other aspects of system design than with smaller data  Here are some of the challenges that we need to face in this situation 
Firstly, since the amount of data is practically unlimited – we could even generate them on the fly resulting in continuous learning similar to the Never-Ending Language Learning by Carnegie Mellon University BIBREF30 – it is now the speed of training that determines how much data the model is able to see  Since more training data significantly help the model performance, focusing on speeding up the algorithm may be more important than ever before  This may for instance influence the decision whether to use regularization such as dropout which does seem to somewhat improve the model performance, however usually at a cost of slowing down training 
Thanks to its simplicity, the psr seems to be training fast - for example around seven times faster than the models proposed by Chen et al  BIBREF5   Hence the psr may be particularly suitable for training on large datasets 
The second challenge is how to generalize the performance gains from large data to a specific target domain  While there are huge amounts of natural language data in general, it may not be the case in the domain where we may want to ultimately apply our model 
Hence we are usually not facing a scenario of simply using a larger amount of the same training data, but rather extending training to a related domain of data, hoping that some of what the model learns on the new data will still help it on the original task 
This is highlighted by our observations from applying a model trained on the BookTest to Children's Book Test test data  If we move model training from joint CBT NE+CN training data to a subset of the BookTest of the same size (230k examples), we see a drop in accuracy of around 10% on the CBT test datasets 
Hence even though the Children's Book Test and BookTest datasets are almost as close as two disjoint datasets can get, the transfer is still very imperfect   Rightly choosing data to augment the in-domain training data is certainly a problem worth exploring in future work 
Our results show that given enough data the AS Reader was able to exceed the human performance on CBT CN reported by Facebook  However we hypothesized that the system is still not achieving its full potential so we decided to examine the room for improvement in our own small human study 
Human Study
After adding more data we have the performance on the CBT validation and test datasets soaring  However is there still potential for much further growth beyond the results we have observed?
We decided to explore the remaining space for improvement on the CBT by testing humans on a random subset of 50 named entity and 50 common noun validation questions that the psr ensemble could not answer correctly  These questions were answered by 10 non-native English speakers from our research laboratory, each on a disjoint subset of questions   Participants had unlimited time to answer the questions and were told that these questions were not correctly answered by a machine, providing additional motivation to prove they are better than computers  The results of the human study are summarized in Table TABREF28   They show that a majority of questions that our system could not answer so far are in fact answerable  This suggests that 1) the original human baselines might have been underestimated, however, it might also be the case that there are some examples that can be answered by machines and not by humans; 2) there is still space for improvement 
A system that would answer correctly every time when either our ensemble or human answered correctly would achieve accuracy over 92% percent on both validation and test NE datasets and over 96% on both CN datasets  Hence it still makes sense to use CBT dataset to study further improvements of text-comprehension systems 
Conclusion
Few ways of improving model performance are as solidly established as using more training data  Yet we believe this principle has been somewhat neglected by recent research in text comprehension  While there is a practically unlimited amount of data available in this field, most research was performed on unnecessarily small datasets 
As a gentle reminder to the community we have shown that simply infusing a model with more data can yield performance improvements of up to INLINEFORM0 where several attempts to improve the model architecture on the same training data have given gains of at most INLINEFORM1 compared to our best ensemble result  Yes, experiments on small datasets certainly can bring useful insights  However we believe that the community should also embrace the real-world scenario of data abundance 
The BookTest dataset we are proposing gives the reading-comprehension community an opportunity to make a step in that direction 
Training Details
The training details are similar to those in BIBREF4 however we are including them here for completeness 
To train the model we used stochastic gradient descent with the ADAM update rule BIBREF32 and learning rates of INLINEFORM0 , INLINEFORM1 and INLINEFORM2   The best learning rate in our experiments was INLINEFORM3   We minimized negative log-likelihood as the training objective 
The initial weights in the word-embedding matrix were drawn randomly uniformly from the interval INLINEFORM0   Weights in the GRU networks were initialized by random orthogonal matrices BIBREF34 and biases were initialized to zero  We also used a gradient clipping BIBREF33 threshold of 10 and batches of sizes between 32 or 256  Increasing the batch from 32 to 128 seems to significantly improve performance on the large dataset - something we did not observe on the original CBT data  Increasing the batch size much above 128 is currently difficult due to memory constraints of the GPU 
During training we randomly shuffled all examples at the beginning of each epoch  To speed up training, we always pre-fetched 10 batches worth of examples and sorted them according to document length  Hence each batch contained documents of roughly the same length 
We also did not use pre-trained word embeddings 
We did not perform any text pre-processing since the datasets were already tokenized 
During training we evaluated the model performance every 12 hours and at the end of each epoch and stopped training when the error on the 20k BookTest validation set started increasing  We explored the hyperparameter space by training 67 different models The region of the parameter space that we explored together with the parameters of the model with best validation accuracy are summarized in Table TABREF29  
Our model was implemented using Theano BIBREF31 and Blocks BIBREF35  
The ensembles were formed by simply averaging the predictions from the constituent single models  These single models were selected using the following algorithm 
We started with the best performing model according to validation performance  Then in each step we tried adding the best performing model that had not been previously tried  We kept it in the ensemble if it did improve its validation performance and discarded it otherwise  This way we gradually tried each model once  We call the resulting model a greedy ensemble  We used the INLINEFORM0 BookTest validation dataset for this procedure 
The algorithm was offered 10 models and selected 5 of them for the final ensemble 
Context 3: Introduction
Deep Neural Networks (DNN) have been widely employed in industry for solving various Natural Language Processing (NLP) tasks, such as text classification, sequence labeling, question answering, etc  However, when engineers apply DNN models to address specific NLP tasks, they often face the following challenges 
The above challenges often hinder the productivity of engineers, and result in less optimal solutions to their given tasks  This motivates us to develop an NLP toolkit for DNN models, which facilitates engineers to develop DNN approaches  Before designing this NLP toolkit, we conducted a survey among engineers and identified a spectrum of three typical personas 
To satisfy the requirements of all the above three personas, the NLP toolkit has to be generic enough to cover as many tasks as possible  At the same time, it also needs to be flexible enough to allow alternative network architectures as well as customized modules  Therefore, we analyzed the NLP jobs submitted to a commercial centralized GPU cluster  Table TABREF11 showed that about 87 5% NLP related jobs belong to a few common tasks, including sentence classification, text matching, sequence labeling, MRC, etc  It further suggested that more than 90% of the networks were composed of several common components, such as embedding, CNN/RNN, Transformer and so on 
Based on the above observations, we developed NeuronBlocks, a DNN toolkit for NLP tasks  The basic idea is to provide two layers of support to the engineers  The upper layer targets common NLP tasks  For each task, the toolkit contains several end-to-end network templates, which can be immediately instantiated with simple configuration  The bottom layer consists of a suite of reusable and standard components, which can be adopted as building blocks to construct networks with complex architecture  By following the interface guidelines, users can also contribute to this gallery of components with their own modules 
The technical contributions of NeuronBlocks are summarized into the following three aspects 
Related Work
There are several general-purpose deep learning frameworks, such as TensorFlow, PyTorch and Keras, which have gained popularity in NLP community  These frameworks offer huge flexibility in DNN model design and support various NLP tasks  However, building models under these frameworks requires a large overhead of mastering these framework details  Therefore, higher level abstraction to hide the framework details is favored by many engineers 
There are also several popular deep learning toolkits in NLP, including OpenNMT BIBREF0 , AllenNLP BIBREF1 etc  OpenNMT is an open-source toolkit mainly targeting neural machine translation or other natural language generation tasks  AllenNLP provides several pre-built models for NLP tasks, such as semantic role labeling, machine comprehension, textual entailment, etc  Although these toolkits reduce the development cost, they are limited to certain tasks, and thus not flexible enough to support new network architectures or new components 
Design
The Neuronblocks is built on PyTorch  The overall framework is illustrated in Figure FIGREF16   It consists of two layers: the Block Zoo and the Model Zoo  In Block Zoo, the most commonly used components of deep neural networks are categorized into several groups according to their functions  Within each category, several alternative components are encapsulated into standard and reusable blocks with a consistent interface  These blocks serve as basic and exchangeable units to construct complex network architectures for different NLP tasks  In Model Zoo, the most popular NLP tasks are identified  For each task, several end-to-end network templates are provided in the form of JSON configuration files  Users can simply browse these configurations and choose one or more to instantiate  The whole task can be completed without any coding efforts 
Block Zoo
We recognize the following major functional categories of neural network components  Each category covers as many commonly used modules as possible  The Block Zoo is an open framework, and more modules can be added in the future 
[itemsep= -0

ground_truth: simply averaging the predictions from the constituent single models

prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. What are the results from these proposed strategies?
Context 2: Introduction and Background
Many reinforcement learning algorithms are designed for relatively small discrete or continuous action spaces and so have trouble scaling  Text-adventure games—or interaction fictions—are simulations in which both an agents' state and action spaces are in textual natural language  An example of a one turn agent interaction in the popular text-game Zork1 can be seen in Fig  FIGREF1  Text-adventure games provide us with multiple challenges in the form of partial observability, commonsense reasoning, and a combinatorially-sized state-action space  Text-adventure games are structured as long puzzles or quests, interspersed with bottlenecks  The quests can usually be completed through multiple branching paths  However, games can also feature one or more bottlenecks  Bottlenecks are areas that an agent must pass through in order to progress to the next section of the game regardless of what path the agent has taken to complete that section of the quest BIBREF0  In this work, we focus on more effectively exploring this space and surpassing these bottlenecks—building on prior work that focuses on tackling the other problems 
Formally, we use the definition of text-adventure games as seen in BIBREF1 and BIBREF2  These games are partially observable Markov decision processes (POMDPs), represented as a 7-tuple of $\langle S,T,A,\Omega , O,R, \gamma \rangle $ representing the set of environment states, mostly deterministic conditional transition probabilities between states, the vocabulary or words used to compose text commands, observations returned by the game, observation conditional probabilities, reward function, and the discount factor respectively  For our purposes, understanding the exact state and action spaces we use in this work is critical and so we define each of these in relative depth 
Action-Space  To solve Zork1, the cannonical text-adventure games, requires the generation of actions consisting of up to five-words from a relatively modest vocabulary of 697 words recognized by the game’s parser  This results in $\mathcal {O}(697^5)={1 64e14}$ possible actions at every step  To facilitate text-adventure game playing, BIBREF2 introduce Jericho, a framework for interacting with text-games  They propose a template-based action space in which the agent first selects a template, consisting of an action verb and preposition, and then filling that in with relevant entities $($e g  $[get]$ $ [from] $ $)$  Zork1 has 237 templates, each with up to two blanks, yielding a template-action space of size $\mathcal {O}(237 \times 697^2)={1 15e8}$  This space is still far larger than most used by previous approaches applying reinforcement learning to text-based games 
State-Representation  Prior work has shown that knowledge graphs are effective in terms of dealing with the challenges of partial observability $($BIBREF3 BIBREF3; BIBREF4$)$  A knowledge graph is a set of 3-tuples of the form $\langle subject, relation, object \rangle $  These triples are extracted from the observations using Stanford's Open Information Extraction (OpenIE) BIBREF5  Human-made text-adventure games often contain relatively complex semi-structured information that OpenIE is not designed to parse and so they add additional rules to ensure that the correct information is parsed  The graph itself is more or less a map of the world, with information about objects' affordances and attributes linked to the rooms that they are place in a map  The graph also makes a distinction with respect to items that are in the agent's possession or in their immediate surrounding environment  An example of what the knowledge graph looks like and specific implementation details can be found in Appendix SECREF14 
BIBREF6 introduce the KG-A2C, which uses a knowledge graph based state-representation to aid in the section of actions in a combinatorially-sized action-space—specifically they use the knowledge graph to constrain the kinds of entities that can be filled in the blanks in the template action-space  They test their approach on Zork1, showing the combination of the knowledge graph and template action selection resulted in improvements over existing methods  They note that their approach reaches a score of 40 which corresponds to a bottleneck in Zork1 where the player is eaten by a “grue” (resulting in negative reward) if the player has not first lit a lamp  The lamp must be lit many steps after first being encountered, in a different section of the game; this action is necessary to continue exploring but doesn’t immediately produce any positive reward  That is, there is a long term dependency between actions that is not immediately rewarded, as seen in Figure FIGREF1  Others using artificially constrained action spaces also report an inability to pass through this bottleneck BIBREF7, BIBREF8  They pose a significant challenge for these methods because the agent does not see the correct action sequence to pass the bottleneck enough times  This is in part due to the fact that for that sequence to be reinforced, the agent needs to reach the next possible reward beyond the bottleneck 
More efficient exploration strategies are required to pass bottlenecks  Our contributions are two-fold  We first introduce a method that detects bottlenecks in text-games using the overall reward gained and the knowledge graph state  This method freezes the policy used to reach the bottleneck and restarts the training from there on out, additionally conducting a backtracking search to ensure that a sub-optimal policy has not been frozen  The second contribution explore how to leverage knowledge graphs to improve existing exploration algorithms for dealing with combinatorial action-spaces such as Go-Explore BIBREF9  We additionally present a comparative ablation study analyzing the performance of these methods on the popular text-game Zork1 
Exploration Methods
In this section, we describe methods to explore combinatorially sized action spaces such as text-games—focusing especially on methods that can deal with their inherent bottleneck structure  We first describe our method that explicitly attempts to detect bottlenecks and then describe how an exploration algorithm such as Go Explore BIBREF9 can leverage knowledge graphs 
KG-A2C-chained An example of a bottleneck can be seen in Figure FIGREF1  We extend the KG-A2C algorithm as follows  First, we detect bottlenecks as states where the agent is unable to progress any further  We set a patience parameter and if the agent has not seen a higher score in patience steps, the agent assumes it has been limited by a bottleneck  Second, when a bottleneck is found, we freeze the policy that gets the agent to the state with the highest score  The agent then begins training a new policy from that particular state 
Simply freezing the policy that led to the bottleneck, however, can potentially result in a policy one that is globally sub-optimal  We therefore employ a backtracking strategy that restarts exploration from each of the $n$ previous steps—searching for a more optimal policy that reaches that bottleneck  At each step, we keep track of a buffer of $n$ states and admissible actions that led up to that locally optimal state  We force the agent to explore from this state to attempt to drive it out of the local optima  If it is further unable to find itself out of this local optima, we refresh the training process again, but starting at the state immediately before the agent reaches the local optima  If this continues to fail, we continue to iterate through this buffer of seen states states up to that local optima until we either find a more optimal state or we run out of states to refresh from, in which we terminate the training algorithm 
KG-A2C-Explore Go-Explore BIBREF9 is an algorithm that is designed to keep track of sub-optimal and under-explored states in order to allow the agent to explore upon more optimal states that may be a result of sparse rewards  The Go-Explore algorithm consists of two phases, the first to continuously explore until a set of promising states and corresponding trajectories are found on the basis of total score, and the second to robustify this found policy against potential stochasticity in the game  Promising states are defined as those states when explored from will likely result in higher reward trajectories  Since the text games we are dealing with are mostly deterministic, with the exception of Zork in later stages, we only focus on using Phase 1 of the Go-Explore algorithm to find an optimal policy  BIBREF10 look at applying Go-Explore to text-games on a set of simpler games generated using the game generation framework TextWorld BIBREF1  Instead of training a policy network in parallel to generate actions used for exploration, they use a small set of “admissible actions”—actions guaranteed to change the world state at any given step during Phase 1—to explore and find high reward trajectories  This space of actions is relatively small (of the order of $10^2$ per step) and so finding high reward trajectories in larger action-spaces such as in Zork would be infeasible
Go-Explore maintains an archive of cells—defined as a set of states that map to a single representation—to keep track of promising states  BIBREF9 simply encodes each cell by keeping track of the agent's position and BIBREF10 use the textual observations encoded by recurrent neural network as a cell representation  We improve on this implementation by training the KG-A2C network in parallel, using the snapshot of the knowledge graph in conjunction with the game state to further encode the current state and use this as a cell representation  At each step, Go-Explore chooses a cell to explore at random (weighted by score to prefer more advanced cells)  The KG-A2C will run for a number of steps, starting with the knowledge graph state and the last seen state of the game from the cell  This will generate a trajectory for the agent while further training the KG-A2C at each iteration, creating a new representation for the knowledge graph as well as a new game state for the cell  After expanding a cell, Go-Explore will continue to sample cells by weight to continue expanding its known states  At the same time, KG-A2C will benefit from the heuristics of selecting preferred cells and be trained on promising states more often 
Evaluation
We compare our two exploration strategies to the following baselines and ablations:
KG-A2C This is the exact same method presented in BIBREF6 with no modifications 
A2C Represents the same approach as KG-A2C but with all the knowledge graph components removed  The state representation is text only encoded using recurrent networks 
A2C-chained Is a variation on KG-A2C-chained where we use our policy chaining approach with the A2C method to train the agent instead of KG-A2C 
A2C-Explore Uses A2C in addition to the exploration strategy seen in KG-A2C-Explore  The cell representations here are defined in terms of the recurrent network based encoding of the textual observation 
Figure FIGREF10 shows that agents utilizing knowledge-graphs in addition to either enhanced exploration method far outperform the baseline A2C and KG-A2C  KG-A2C-chained and KG-A2C-Explore both pass the bottleneck of a score of 40, whereas A2C-Explore gets to the bottleneck but cannot surpass it 
There are a couple of key insights that can be drawn from these results The first is that the knowledge graph appears to be critical; it is theorized to help with partial observability  However the knowledge graph representation isn't sufficient in that the knowledge graph representation without enhanced exploration methods cannot surpass the bottleneck  A2C-chained—which explores without a knowledge graph—fails to even outperform the baseline A2C  We hypothesize that this is due to the knowledge graph aiding implicitly in the sample efficiency of bottleneck detection and subsequent exploration  That is, exploring after backtracking from a potentially detected bottleneck is much more efficient in the knowledge graph based agent 
The Go-Explore based exploration algorithm sees less of a difference between agents  A2C-Explore converges more quickly, but to a lower reward trajectory that fails to pass the bottleneck, whereas KG-A2C-Explore takes longer to reach a similar reward but consistently makes it through the bottleneck  The knowledge graph cell representation appears to thus be a better indication of what a promising state is as opposed to just the textual observation 
Comparing the advanced exploration methods when using the knowledge graph, we see that both agents successfully pass the bottleneck corresponding to entering the cellar and lighting the lamp and reach comparable scores within a margin of error  KG-A2C-chained is significantly more sample efficient and converges faster  We can infer that chaining policies by explicitly detecting bottlenecks lets us pass it more quickly than attempting to find promising cell representations with Go-Explore  This form of chained exploration with backtracking is particularly suited to sequential decision making problems that can be represented as acyclic directed graphs as in Figure FIGREF1 
Appendix ::: Zork1
Zork1 is one of the first text-adventure games and heavily influences games released later in terms of narrative style and game structure  It is a dungeon crawler where the player must explore a vast world and collect a series of treasures  It was identified by BIBREF2 as a moonshot game and has been the subject of much work in leaning agents BIBREF12, BIBREF7, BIBREF11, BIBREF8  Rewards are given to the player when they collect treasures as well as when important intermediate milestones needed to further explore the world are passed  Figure FIGREF15 and Figure FIGREF1 show us a map of the world of Zork1 and the corresponding quest structure 
The bottleneck seen at a score of around 40 is when the player first enters the cellar on the right side of the map  The cellar is dark and you need to immediately light the lamp to see anything  Attempting to explore the cellar in the dark results in you being instantly killed by a monster known as a “grue” 
Appendix ::: Knowledge Graph Rules
We make no changes from the graph update rules used by BIBREF6  Candidate interactive objects are identified by performing part-of-speech tagging on the current observation, identifying singular and proper nouns as well as adjectives, and are then filtered by checking if they can be examined using the command $examine$ $OBJ$  Only the interactive objects not found in the inventory are linked to the node corresponding to the current room and the inventory items are linked to the “you” node  The only other rule applied uses the navigational actions performed by the agent to infer the relative positions of rooms, e g  $\langle kitchen,down,cellar \rangle $ when the agent performs $go$ $down$ when in the kitchen to move to the cellar 
Appendix ::: Hyperparameters
Hyperparameters used for our agents are given below  Patience and buffer size are used for the policy chaining method as described in Section SECREF2  Cell step size is a parameter used for Go-Explore and describes how many steps are taken when exploring in a given cell state  Base hyperparameters for KG-A2C are taken from BIBREF6 and the same parameters are used for A2C 
Context 3: 4em,topsep = 0 3em, align=left, labelsep=-0 6em, leftmargin=1 2em]
Embedding Layer: Word/character embedding and extra handcrafted feature embedding such as pos-tagging are supported 
Neural Network Layers: Block zoo provides common layers like RNN, CNN, QRNN BIBREF2 , Transformer BIBREF3 , Highway network, Encoder Decoder architecture, etc  Furthermore, attention mechanisms are widely used in neural networks  Thus we also support multiple attention layers, such as Linear/Bi-linear Attention, Full Attention BIBREF4 , Bidirectional attention flow BIBREF5 , etc  Meanwhile, regularization layers such as Dropout, Layer Norm, Batch Norm, etc are also supported for improving generalization ability 
Loss Function: Besides of the loss functions built in PyTorch, we offer more options such as Focal Loss BIBREF6  
Metrics: For classification task, AUC, Accuracy, Precision/Recall, F1 metrics are supported  For sequence labeling task, F1/Accuracy are supported  For knowledge distillation task, MSE/RMSE are supported  For MRC task, ExactMatch/F1 are supported 
Model Zoo
In NeuronBlocks, we identify four types of most popular NLP tasks  For each task, we provide various end-to-end network templates 
[itemsep= -0 4em,topsep = 0 3em, align=left, labelsep=-0 6em, leftmargin=1 2em]
Text Classification and Matching  Tasks such as domain/intent classification, question answer matching are supported 
Sequence Labeling  Predict each token in a sequence into predefined types  Common tasks include NER, POS tagging, Slot tagging, etc 
Knowledge Distillation BIBREF7   Teacher-Student based knowledge distillation is one common approach for model compression  NeuronBlocks provides knowledge distillation template to improve the inference speed of heavy DNN models like BERT/GPT 
Extractive Machine Reading Comprehension  Given a pair of question and passage, predict the start and end positions of the answer spans in the passage 
User Interface
NeuronBlocks provides convenient user interface for users to build, train, and test DNN models  The details are described in the following 
[itemsep= -0 4em,topsep = 0 3em, align=left, labelsep=-0 6em, leftmargin=1 2em]
I/O interface  This part defines model input/output, such as training data, pre-trained models/embeddings, model saving path, etc 
Model Architecture interface  This is the key part of the configuration file, which defines the whole model architecture  Figure FIGREF19 shows an example of how to specify a model architecture using the blocks in NeuronBlocks

ground_truth: Reward of 11.8 for the A2C-chained model, 41.8 for the KG-A2C-chained model, 40 for A2C-Explore and 44 for KG-A2C-Explore.

KG-A2C-chained and KG-A2C-Explore both pass the bottleneck of a score of 40

prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. What neural network modules are included in NeuronBlocks?
Context 3:  To be more specific, it consists of a list of layers/blocks to construct the architecture, where the blocks are supplied in the gallery of Block Zoo 
Training Parameters interface  In this part, the model optimizer as well as all other training hyper parameters are indicated 
Workflow
Figure FIGREF34 shows the workflow of building DNN models in NeuronBlocks  Users only need to write a JSON configuration file  They can either instantiate an existing template from Model Zoo, or construct a new architecture based on the blocks from Block Zoo  This configuration file is shared across training, test, and prediction  For model hyper-parameter tuning or architecture modification, users just need to change the JSON configuration file  Advanced users can also contribute novel customized blocks into Block Zoo, as long as they follow the same interface guidelines with the existing blocks  These new blocks can be further shared across all users for model architecture design  Moreover, NeuronBlocks has flexible platform support, such as GPU/CPU, GPU management platforms like PAI 
Experiments
To verify the performance of NeuronBlocks, we conducted extensive experiments for common NLP tasks on public data sets including CoNLL-2003 BIBREF14 , GLUE benchmark BIBREF13 , and WikiQA corpus BIBREF15   The experimental results showed that the models built with NeuronBlocks can achieve reliable and competitive results on various tasks, with productivity greatly improved 
Sequence Labeling
For sequence labeling task, we evaluated NeuronBlocks on CoNLL-2003 BIBREF14 English NER dataset, following most works on the same task  This dataset includes four types of named entities, namely, PERSON, LOCATION, ORGANIZATION, and MISC  We adopted the BIOES tagging scheme instead of IOB, as many previous works indicated meaningful improvement with BIOES scheme BIBREF16 , BIBREF17   Table TABREF28 shows the results on CoNLL-2003 Englist testb dataset, with 12 different combinations of network layers/blocks, such as word/character embedding, CNN/LSTM and CRF  The results suggest that the flexible combination of layers/blocks in NeuronBlocks can easily reproduce the performance of original models, with comparative or slightly better performance 
GLUE Benchmark
The General Language Understanding Evaluation (GLUE) benchmark BIBREF13 is a collection of natural language understanding tasks  We experimented on the GLUE benchmark tasks using BiLSTM and Attention based models  As shown in Table TABREF29 , the models built by NeuronBlocks can achieve competitive or even better results on GLUE tasks with minimal coding efforts 
Knowledge Distillation
We evaluated Knowledge Distillation task in NeuronBlocks on a dataset collected from one commercial search engine  We refer to this dataset as Domain Classification Dataset  Each sample in this dataset consists of two parts, i e , a question and a binary label indicating whether the question belongs to a specific domain  Table TABREF36 shows the results, where Area Under Curve (AUC) metric is used as the performance evaluation criteria and Queries per Second (QPS) is used to measure inference speed  By knowledge distillation training approach, the student model by NeuronBlocks managed to get 23-27 times inference speedup with only small performance regression compared with BERTbase fine-tuned classifier 
WikiQA
The WikiQA corpus BIBREF15 is a publicly available dataset for open-domain question answering  This dataset contains 3,047 questions from Bing query logs, each associated with some candidate answer sentences from Wikipedia  We conducted experiments on WikiQA dataset using CNN, BiLSTM, and Attention based models  The results are shown in Table TABREF41   The models built in NeuronBlocks achieved competitive or even better results with simple model configurations 
Conclusion and Future Work
In this paper, we introduce NeuronBlocks, a DNN toolkit for NLP tasks built on PyTorch  NeuronBlocks targets three types of engineers, and provides a two-layer solution to satisfy the requirements from all three types of users  To be more specific, the Model Zoo consists of various templates for the most common NLP tasks, while the Block Zoo supplies a gallery of alternative layers/modules for the networks  Such design achieves a balance between generality and flexibility  Extensive experiments have verified the effectiveness of this approach  NeuronBlocks has been widely used in a product team of a commercial search engine, and significantly improved the productivity for developing NLP DNN approaches 
As an open-source toolkit, we will further extend it in various directions  The following names a few examples 

ground_truth: Embedding Layer, Neural Network Layers, Loss Function, Metrics

Embedding Layer, Neural Network Layers, Loss Function, Metrics