prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. What metrics are used for evaluation?
Context 1: Introduction
Pre-trained models BIBREF0, BIBREF1 have received much of attention recently thanks to their impressive results in many down stream NLP tasks  Additionally, multilingual pre-trained models enable many NLP applications for other languages via zero-short cross-lingual transfer  Zero-shot cross-lingual transfer has shown promising results for rapidly building applications for low resource languages  BIBREF2 show the potential of multilingual-BERT BIBREF0 in zero-shot transfer for a large number of languages from different language families on five NLP tasks, namely, natural language inference, document classification, named entity recognition, part-of-speech tagging, and dependency parsing 
Although multilingual models are an important ingredient for enhancing language technology in many languages, recent research on improving pre-trained models puts much emphasis on English BIBREF3, BIBREF4, BIBREF5  The current state of affairs makes it difficult to translate advancements in pre-training from English to non-English languages  To our best knowledge, there are only three available multilingual pre-trained models to date: (1) the multilingual-BERT (mBERT) that supports 104 languages, (2) cross-lingual language model BIBREF6 that supports 100 languages, and (3) Language Agnostic SEntence Representations BIBREF7 that supports 93 languages  Among the three models, LASER is based on neural machine translation approach and strictly requires parallel data to train 
Do multilingual models always need to be trained from scratch? Can we transfer linguistic knowledge learned by English pre-trained models to other languages? In this work, we develop a technique to rapidly transfer an existing pre-trained model from English to other languages in an energy efficient way BIBREF8  As the first step, we focus on building a bilingual language model (LM) of English and a target language  Starting from a pre-trained English LM, we learn the target language specific parameters (i e , word embeddings), while keeping the encoder layers of the pre-trained English LM fixed  We then fine-tune both English and target model to obtain the bilingual LM  We apply our approach to autoencoding language models with masked language model objective and show the advantage of the proposed approach in zero-shot transfer  Our main contributions in this work are:
We propose a fast adaptation method for obtaining a bilingual BERT$_{\textsc {base}}$ of English and a target language within a day using one Tesla V100 16GB GPU 
We evaluate our bilingual LMs for six languages on two zero-shot cross-lingual transfer tasks, namely natural language inference BIBREF9 and universal dependency parsing  We show that our models offer competitive performance or even better that mBERT 
We illustrate that our bilingual LMs can serve as an excellent feature extractor in supervised dependency parsing task 
Bilingual Pre-trained LMs
We first provide some background of pre-trained language models  Let $_e$ be English word-embeddings and $\Psi ()$ be the Transformer BIBREF10 encoder with parameters $$  Let $_{w_i}$ denote the embedding of word $w_i$ (i e , $_{w_i} = _e[w_1]$)  We omit positional embeddings and bias for clarity  A pre-trained LM typically performs the following computations: (i) transform a sequence of input tokens to contextualized representations $[_{w_1},\dots ,_{w_n}] = \Psi (_{w_1}, \dots , _{w_n}; )$, and (ii) predict an output word $y_i$ at $i^{\text{th}}$ position $p(y_i | _{w_i}) \propto \exp (_{w_i}^\top _{y_i})$ 
Autoencoding LM BIBREF0 corrupts some input tokens $w_i$ by replacing them with a special token [MASK]  It then predicts the original tokens $y_i = w_i$ from the corrupted tokens  Autoregressive LM BIBREF3 predicts the next token ($y_i = w_{i+1}$) given all the previous tokens  The recently proposed XLNet model BIBREF5 is an autoregressive LM that factorizes output with all possible permutations, which shows empirical performance improvement over GPT-2 due to the ability to capture bidirectional context  Here we assume that the encoder performs necessary masking with respect to each training objective 
Given an English pre-trained LM, we wish to learn a bilingual LM for English and a given target language $f$ under a limited computational resource budget  To quickly build a bilingual LM, we directly adapt the English pre-traind model to the target model  Our approach consists of three steps  First, we initialize target language word-embeddings $_f$ in the English embedding space such that embeddings of a target word and its English equivalents are close together (§SECREF8)  Next, we create a target LM from the target embeddings and the English encoder $\Psi ()$  We then fine-tune target embeddings while keeping $\Psi ()$ fixed (§SECREF14)  Finally, we construct a bilingual LM of $_e$, $_f$, and $\Psi ()$ and fine-tune all the parameters (§SECREF15)  Figure FIGREF7 illustrates the last two steps in our approach 
Bilingual Pre-trained LMs ::: Initializing Target Embeddings
Our approach to learn the initial foreign word embeddings $_f \in ^{|V_f| \times d}$ is based on the idea of mapping the trained English word embeddings $_e \in ^{|V_e| \times d}$ to $_f$ such that if a foreign word and an English word are similar in meaning then their embeddings are similar  Borrowing the idea of universal lexical sharing from BIBREF11, we represent each foreign word embedding $_f[i] \in ^d$ as a linear combination of English word embeddings $_e[j] \in ^d$
where $_i\in ^{|V_e|}$ is a sparse vector and $\sum _j^{|V_e|} \alpha _{ij} = 1$ 
In this step of initializing foreign embeddings, having a good estimation of $$ could speed of the convergence when tuning the foreign model and enable zero-shot transfer (§SECREF5)  In the following, we discuss how to estimate $_i\;\forall i\in \lbrace 1,2, \dots , |V_f|\rbrace $ under two scenarios: (i) we have parallel data of English-foreign, and (ii) we only rely on English and foreign monolingual data 
Bilingual Pre-trained LMs ::: Initializing Target Embeddings ::: Learning from Parallel Corpus
Given an English-foreign parallel corpus, we can estimate word translation probability $p(e\,|\,f)$ for any (English-foreign) pair $(e, f)$ using popular word-alignment BIBREF12 toolkits such as fast-align BIBREF13  We then assign:
Since $_i$ is estimated from word alignment, it is a sparse vector 
Bilingual Pre-trained LMs ::: Initializing Target Embeddings ::: Learning from Monolingual Corpus
For low resource languages, parallel data may not be available  In this case, we rely only on monolingual data (e g , Wikipedias)  We estimate word translation probabilities from word embeddings of the two languages  Word vectors of these languages can be learned using fastText BIBREF14 and then are aligned into a shared space with English BIBREF15, BIBREF16  Unlike learning contextualized representations, learning word vectors is fast and computationally cheap  Given the aligned vectors $\bar{}_f$ of foreign and $\bar{}_e$ of English, we calculate the word translation matrix $\in ^{|V_f|\times |V_e|}$ as
Here, we use $\mathrm {sparsemax}$ BIBREF17 instead of softmax  Sparsemax is a sparse version of softmax and it puts zero probabilities on most of the word in the English vocabulary except few English words that are similar to a given foreign word  This property is desirable in our approach since it leads to a better initialization of the foreign embeddings 
Bilingual Pre-trained LMs ::: Fine-tuning Target Embeddings
After initializing foreign word-embeddings, we replace English word-embeddings in the English pre-trained LM with foreign word-embeddings to obtain the foreign LM  We then fine-tune only foreign word-embeddings on monolingual data  The training objective is the same as the training objective of the English pre-trained LM (i e , masked LM for BERT)  Since the trained encoder $\Psi ()$ is good at capturing association, the purpose of this step is to further optimize target embeddings such that the target LM can utilized the trained encoder for association task  For example, if the words Albert Camus presented in a French input sequence, the self-attention in the encoder more likely attends to words absurde and existentialisme once their embeddings are tuned 
Bilingual Pre-trained LMs ::: Fine-tuning Bilingual LM
We create a bilingual LM by plugging foreign language specific parameters to the pre-trained English LM (Figure FIGREF7)  The new model has two separate embedding layers and output layers, one for English and one for foreign language  The encoder layer in between is shared  We then fine-tune this model using English and foreign monolingual data  Here, we keep tuning the model on English to ensure that it does not forget what it has learned in English and that we can use the resulting model for zero-shot transfer (§SECREF3)  In this step, the encoder parameters are also updated so that in can learn syntactic aspects (i e , word order, morphological agreement) of the target languages 
Zero-shot Experiments
We build our bilingual LMs, named RAMEN, starting from BERT$_{\textsc {base}}$, BERT$_{\textsc {large}}$, RoBERTa$_{\textsc {base}}$, and RoBERTa$_{\textsc {large}}$ pre-trained models  Using BERT$_{\textsc {base}}$ allows us to compare the results with mBERT model  Using BERT$_{\textsc {large}}$ and RoBERTa allows us to investigate whether the performance of the target LM correlates with the performance of the source LM  We evaluate our models on two cross-lingual zero-shot tasks: (1) Cross-lingual Natural Language Inference (XNLI) and (2) dependency parsing 
Zero-shot Experiments ::: Data
We evaluate our approach for six target languages: French (fr), Russian (ru), Arabic (ar), Chinese (zh), Hindi (hi), and Vietnamese (vi)  These languages belong to four different language families  French, Russian, and Hindi are Indo-European languages, similar to English  Arabic, Chinese, and Vietnamese belong to Afro-Asiatic, Sino-Tibetan, and Austro-Asiatic family respectively  The choice of the six languages also reflects different training conditions depending on the amount of monolingual data  French and Russian, and Arabic can be regarded as high resource languages whereas Hindi has far less data and can be considered as low resource 
For experiments that use parallel data to initialize foreign specific parameters, we use the same datasets in the work of BIBREF6  Specifically, we use United Nations Parallel Corpus BIBREF18 for en-ru, en-ar, en-zh, and en-fr  We collect en-hi parallel data from IIT Bombay corpus BIBREF19 and en-vi data from OpenSubtitles 2018  For experiments that use only monolingual data to initialize foreign parameters, instead of training word-vectors from the scratch, we use the pre-trained word vectors from fastText BIBREF14 to estimate word translation probabilities (Eq  DISPLAY_FORM13)  We align these vectors into a common space using orthogonal Procrustes BIBREF20, BIBREF15, BIBREF16  We only use identical words between the two languages as the supervised signal  We use WikiExtractor to extract extract raw sentences from Wikipedias as monolingual data for fine-tuning target embeddings and bilingual LMs (§SECREF15)  We do not lowercase or remove accents in our data preprocessing pipeline 
We tokenize English using the provided tokenizer from pre-trained models  For target languages, we use fastBPE to learn 30,000 BPE codes and 50,000 codes when transferring from BERT and RoBERTa respectively  We truncate the BPE vocabulary of foreign languages to match the size of the English vocabulary in the source models  Precisely, the size of foreign vocabulary is set to 32,000 when transferring from BERT and 50,000 when transferring from RoBERTa 
We use XNLI dataset BIBREF9 for classification task and Universal Dependencies v2 4 BIBREF21 for parsing task  Since a language might have more than one treebank in Universal Dependencies, we use the following treebanks: en_ewt (English), fr_gsd (French), ru_syntagrus (Russian) ar_padt (Arabic), vi_vtb (Vietnamese), hi_hdtb (Hindi), and zh_gsd (Chinese) 
Zero-shot Experiments ::: Data ::: Remark on BPE
BIBREF22 show that sharing subwords between languages improves alignments between embedding spaces  BIBREF2 observe a strong correlation between the percentage of overlapping subwords and mBERT's performances for cross-lingual zero-shot transfer  However, in our current approach, subwords between source and target are not shared  A subword that is in both English and foreign vocabulary has two different embeddings 
Zero-shot Experiments ::: Estimating translation probabilities
Since pre-trained models operate on subword level, we need to estimate subword translation probabilities  Therefore, we subsample 2M sentence pairs from each parallel corpus and tokenize the data into subwords before running fast-align BIBREF13 
Estimating subword translation probabilities from aligned word vectors requires an additional processing step since the provided vectors from fastText are not at subword level  We use the following approximation to obtain subword vectors: the vector $_s$ of subword $s$ is the weighted average of all the aligned word vectors $_{w_i}$ that have $s$ as an subword
where $p(w_j)$ is the unigram probability of word $w_j$ and $n_s = \sum _{w_j:\, s\in w_j} p(w_j)$  We take the top 50,000 words in each aligned word-vectors to compute subword vectors 
In both cases, not all the words in the foreign vocabulary can be initialized from the English word-embeddings  Those words are initialized randomly from a Gaussian $\mathcal {N}(0, {1}{d^2})$ 
Zero-shot Experiments ::: Hyper-parameters
In all the experiments, we tune RAMEN$_{\textsc {base}}$ for 175,000 updates and RAMEN$_{\textsc {large}}$ for 275,000 updates where the first 25,000 updates are for language specific parameters  The sequence length is set to 256  The mini-batch size are 64 and 24 when tuning language specific parameters using RAMEN$_{\textsc {base}}$ and RAMEN$_{\textsc {large}}$ respectively  For tuning bilingual LMs, we use a mini-batch size of 64 for RAMEN$_{\textsc {base}}$ and 24 for RAMEN$_{\textsc {large}}$ where half of the batch are English sequences and the other half are foreign sequences  This strategy of balancing mini-batch has been used in multilingual neural machine translation BIBREF23, BIBREF24 
We optimize RAMEN$_{\textsc {base}}$ using Lookahead optimizer BIBREF25 wrapped around Adam with the learning rate of $10^{-4}$, the number of fast weight updates $k=5$, and interpolation parameter $\alpha =0 5$  We choose Lookahead optimizer because it has been shown to be robust to the initial parameters of the based optimizer (Adam)  For Adam optimizer, we linearly increase the learning rate from $10^{-7}$ to $10^{-4}$ in the first 4000 updates and then follow an inverse square root decay  All RAMEN$_{\textsc {large}}$ models are optimized with Adam due to memory limit 
When fine-tuning RAMEN on XNLI and UD, we use a mini-batch size of 32, Adam's learning rate of $10^{-5}$  The number of epochs are set to 4 and 50 for XNLI and UD tasks respectively  All experiments are carried out on a single Tesla V100 16GB GPU  Each RAMEN$_{\textsc {base}}$ model is trained within a day and each RAMEN$_{\textsc {large}}$ is trained within two days 
Results
In this section, we present the results of out models for two zero-shot cross lingual transfer tasks: XNLI and dependency parsing 
Results ::: Cross-lingual Natural Language Inference
Table TABREF32 shows the XNLI test accuracy  For reference, we also include the scores from the previous work, notably the state-of-the-art system XLM BIBREF6  Before discussing the results, we spell out that the fairest comparison in this experiment is the comparison between mBERT and RAMEN$_{\textsc {base}}$+BERT trained with monolingual only 
We first discuss the transfer results from BERT  Initialized from fastText vectors, RAMEN$_{\textsc {base}}$ slightly outperforms mBERT by 1 9 points on average and widen the gap of 3 3 points on Arabic  RAMEN$_{\textsc {base}}$ gains extra 0 8 points on average when initialized from parallel data  With triple number of parameters, RAMEN$_{\textsc {large}}$ offers an additional boost in term of accuracy and initialization with parallel data consistently improves the performance  It has been shown that BERT$_{\textsc {large}}$ significantly outperforms BERT$_{\textsc {base}}$ on 11 English NLP tasks BIBREF0, the strength of BERT$_{\textsc {large}}$ also shows up when adapted to foreign languages 
Transferring from RoBERTa leads to better zero-shot accuracies  With the same initializing condition, RAMEN$_{\textsc {base}}$+RoBERTa outperforms RAMEN$_{\textsc {base}}$+BERT on average by 2 9 and 2 3 points when initializing from monolingual and parallel data respectively  This result show that with similar number of parameters, our approach benefits from a better English pre-trained model  When transferring from RoBERTa$_{\textsc {large}}$, we obtain state-of-the-art results for five languages 
Currently, RAMEN only uses parallel data to initialize foreign embeddings  RAMEN can also exploit parallel data through translation objective proposed in XLM  We believe that by utilizing parallel data during the fine-tuning of RAMEN would bring additional benefits for zero-shot tasks  We leave this exploration to future work  In summary, starting from BERT$_{\textsc {base}}$, our approach obtains competitive bilingual LMs with mBERT for zero-shot XNLI  Our approach shows the accuracy gains when adapting from a better pre-trained model 
Results ::: Universal Dependency Parsing
We build on top of RAMEN a graph-based dependency parser BIBREF27  For the purpose of evaluating the contextual representations learned by our model, we do not use part-of-speech tags  Contextualized representations are directly fed into Deep-Biaffine layers to predict arc and label scores  Table TABREF34 presents the Labeled Attachment Scores (LAS) for zero-shot dependency parsing 
We first look at the fairest comparison between mBERT and monolingually initialized RAMEN$_{\textsc {base}}$+BERT  The latter outperforms the former on five languages except Arabic  We observe the largest gain of +5 2 LAS for French  Chinese enjoys +3 1 LAS from our approach  With similar architecture (12 or 24 layers) and initialization (using monolingual or parallel data), RAMEN+RoBERTa performs better than RAMEN+BERT for most of the languages  Arabic and Hindi benefit the most from bigger models  For the other four languages, RAMEN$_{\textsc {large}}$ renders a modest improvement over RAMEN$_{\textsc {base}}$ 
Analysis ::: Impact of initialization
Initializing foreign embeddings is the backbone of our approach  A good initialization leads to better zero-shot transfer results and enables fast adaptation  To verify the importance of a good initialization, we train a RAMEN$_{\textsc {base}}$+RoBERTa with foreign word-embeddings are initialized randomly from $\mathcal {N}(0, {1}{d^2})$  For a fair comparison, we use the same hyper-parameters in §SECREF27  Table TABREF36 shows the results of XNLI and UD parsing of random initialization  In comparison to the initialization using aligned fastText vectors, random initialization decreases the zero-shot performance of RAMEN$_{\textsc {base}}$ by 15 9% for XNLI and 27 8 points for UD parsing on average  We also see that zero-shot parsing of SOV languages (Arabic and Hindi) suffers random initialization 
Analysis ::: Are contextual representations from RAMEN also good for supervised parsing?
All the RAMEN models are built from English and tuned on English for zero-shot cross-lingual tasks  It is reasonable to expect RAMENs do well in those tasks as we have shown in our experiments  But are they also a good feature extractor for supervised tasks? We offer a partial answer to this question by evaluating our model for supervised dependency parsing on UD datasets 
We used train/dev/test splits provided in UD to train and evaluate our RAMEN-based parser  Table TABREF38 summarizes the results (LAS) of our supervised parser  For a fair comparison, we choose mBERT as the baseline and all the RAMEN models are initialized from aligned fastText vectors  With the same architecture of 12 Transformer layers, RAMEN$_{\textsc {base}}$+BERT performs competitive to mBERT and outshines mBERT by +1 2 points for Vietnamese  The best LAS results are obtained by RAMEN$_{\textsc {large}}$+RoBERTa with 24 Transformer layers  Overall, our results indicate the potential of using contextual representations from RAMEN for supervised tasks 
Analysis ::: How does linguistic knowledge transfer happen through each training stages?
We evaluate the performance of RAMEN+RoBERTa$_{\textsc {base}}$ (initialized from monolingual data) at each training steps: initialization of word embeddings (0K update), fine-tuning target embeddings (25K), and fine-tuning the model on both English and target language (at each 25K updates)  The results are presented in Figure FIGREF40 
Without fine-tuning, the average accuracy of XLNI is 39 7% for a three-ways classification task, and the average LAS score is 3 6 for dependency parsing  We see the biggest leap in the performance after 50K updates  While semantic similarity task profits significantly at 25K updates of the target embeddings, syntactic task benefits with further fine-tuning the encoder  This is expected since the target languages might exhibit different syntactic structures than English and fine-tuning encoder helps to capture language specific structures  We observe a substantial gain of 19-30 LAS for all languages except French after 50K updates 
Language similarities have more impact on transferring syntax than semantics  Without tuning the English encoder, French enjoys 50 3 LAS for being closely related to English, whereas Arabic and Hindi, SOV languages, modestly reach 4 2 and 6 4 points using the SVO encoder  Although Chinese has SVO order, it is often seen as head-final while English is strong head-initial  Perhaps, this explains the poor performance for Chinese 
Limitations
While we have successfully adapted autoencoding pre-trained LMs from English to other languages, the question whether our approach can also be applied for autoregressive LM such as XLNet still remains  We leave the investigation to future work 
Conclusions
In this work, we have presented a simple and effective approach for rapidly building a bilingual LM under a limited computational budget  Using BERT as the starting point, we demonstrate that our approach produces better than mBERT on two cross-lingual zero-shot sentence classification and dependency parsing  We find that the performance of our bilingual LM, RAMEN, correlates with the performance of the original pre-trained English models  We also find that RAMEN is also a powerful feature extractor in supervised dependency parsing  Finally, we hope that our work sparks of interest in developing fast and effective methods for transferring pre-trained English models to other languages 
Context 2: Introduction
How humans process language has become increasingly relevant in natural language processing since physiological data during language understanding is more accessible and recorded with less effort  In this work, we focus on eye-tracking and electroencephalography (EEG) recordings to capture the reading process  On one hand, eye movement data provides millisecond-accurate records about where humans look when they are reading, and is highly correlated with the cognitive load associated with different stages of text processing  On the other hand, EEG records electrical brain activity across the scalp and is a direct measure of physiological processes, including language processing  The combination of both measurement methods enables us to study the language understanding process in a more natural setting, where participants read full sentences at a time, in their own speed  Eye-tracking then permits us to define exact word boundaries in the timeline of a subject reading a sentence, allowing the extraction of brain activity signals for each word 
Human cognitive language processing data is immensely useful for NLP: Not only can it be leveraged to improve NLP applications (e g  barrett2016weakly for part-of-speech tagging or klerke2016improving for sentence compression), but also to evaluate state-of-the-art machine learning systems  For example, hollenstein2019cognival evaluate word embeddings, or schwartz2019inducing fine-tune language models with brain-relevant bias 
Additionally, the availability of labelled data plays a crucial role in all supervised machine learning applications  Physiological data can be used to understand and improve the labelling process (e g  tokunaga2017eye), and, for instance, to build cost models for active learning scenarios BIBREF0  Is it possible to replace this expensive manual work with models trained on physiological activity data recorded from humans while reading? That is to say, can we find and extract relevant aspects of text understanding and annotation directly from the source, i e  eye-tracking and brain activity signals during reading?
Motivated by these questions and our previously released dataset, ZuCo 1 0 BIBREF1, we developed this new corpus, where we specifically aim to collect recordings during natural reading as well as during annotation 
We provide the first dataset of simultaneous eye movement and brain activity recordings to analyze and compare normal reading to task-specific reading during annotation  The Zurich Cognitive Language Processing Corpus (ZuCo) 2 0, including raw and preprocessed eye-tracking and electroencephalography (EEG) data of 18 subjects, as well as the recording and preprocessing scripts, is publicly available at https://osf io/2urht/  It contains physiological data of each subject reading 739 English sentences from Wikipedia (see example in Figure FIGREF1)  We want to highlight the re-use potential of this data  In addition to the psycholinguistic motivation, this corpus is especially tailored for training and evaluating machine learning algorithms for NLP purposes  We conduct a detailed technical validation of the data as proof of the quality of the recordings 
Related Work
Some eye-tracking corpora of natural reading (e g  the Dundee BIBREF2, Provo BIBREF3 and GECO corpus BIBREF4), and a few EEG corpora (for example, the UCL corpus BIBREF5) are available  It has been shown that this type of cognitive processing data is useful for improving and evaluating NLP methods (e g  barrett2018sequence,hollenstein2019cognival, hale2018finding)  However, before the Zurich Cognitive Language Processing Corpus (ZuCo 1 0), there was no available data for simultaneous eye-tracking and EEG recordings of natural reading  dimigen2011coregistration studied the linguistic effects of eye movements and EEG co-registration in natural reading and showed that they accurately represent lexical processing  Moreover, the simultaneous recordings are crucial to extract word-level brain activity signals 
While the above mentioned studies analyze and leverage natural reading, some NLP work has used eye-tracking during annotation (but, as of yet, not EEG data)  mishra2016predicting and joshi2014measuring recorded eye-tracking during binary sentiment annotation (positive/negative)  This data was used to determine the annotation complexity of the text passages based on eye movement metrics and for sarcasm detection BIBREF6  Moreover, eye-tracking has been used to analyze the word sense annotation process in Hindi BIBREF7, named entity annotation in Japanese BIBREF8, and to leverage annotator gaze behaviour for coreference resolution BIBREF9  Finally, tomanek2010cognitive used eye-tracking data during entity annotation to build a cost model for active learning  However, until now there is no available data or research that analyzes the differences in the human processing of normal reading versus annotation 
Related Work ::: ZuCo1 0
In previous work, we recorded a first dataset of simultaneous eye-tracking and EEG during natural reading BIBREF1  ZuCo 1 0 consists of three reading tasks, two of which contain very similar reading material and experiments as presented in the current work  However, the main difference and reason for recording ZuCo 2 0, consists in the experiment procedure  For ZuCo 1 0 the normal reading and task-specific reading paradigms were recorded in different sessions on different days  Therefore, the recorded data is not appropriate as a means of comparison between natural reading and annotation, since the differences in the brain activity data might result mostly from the different sessions due to the sensitivity of EEG  This, and extending the dataset with more sentences and more subjects, were the main factors for recording the current corpus  We purposefully maintained an overlap of some sentences between both datasets to allow additional analyses (details are described in Section SECREF7) 
Corpus Construction
In this section we describe the contents and experimental design of the ZuCo 2 0 corpus 
Corpus Construction ::: Participants
We recorded data from 19 participants and discarded the data of one of them due to technical difficulties with the eye-tracking calibration  Hence, we share the data of 18 participants  All participants are healthy adults (mean age = 34 (SD=8 3), 10 females)  Their native language is English, originating from Australia, Canada, UK, USA or South Africa  Two participants are left-handed and three participants wear glasses for reading  Details on subject demographics can be found in Table TABREF4  All participants gave written consent for their participation and the re-use of the data prior to the start of the experiments  The study was approved by the Ethics Commission of the University of Zurich 
Corpus Construction ::: Reading materials
During the recording session, the participants read 739 sentences that were selected from the Wikipedia corpus provided by culotta2006integrating  This corpus was chosen because it provides annotations of semantic relations  We included seven of the originally defined relation types: political_affiliation, education, founder, wife/husband, job_title, nationality, and employer  The sentences were chosen in the same length range as ZuCo 1 0, and with similar Flesch reading ease scores  The dataset statistics are shown in Table TABREF2 
Of the 739 sentences, the participants read 349 sentences in a normal reading paradigm, and 390 sentences in a task-specific reading paradigm, in which they had to determine whether a certain relation type occurred in the sentence or not  Table TABREF3 shows the distribution of the different relation types in the sentences of the task-specific annotation paradigm 
Purposefully, there are 63 duplicates between the normal reading and the task-specific sentences (8% of all sentences)  The intention of these duplicate sentences is to provide a set of sentences read twice by all participants with a different task in mind  Hence, this enables the comparison of eye-tracking and brain activity data when reading normally and when annotating specific relations (see examples in Section SECREF4) 
Furthermore, there is also an overlap in the sentences between ZuCo 1 0 and ZuCo 2 0  100 normal reading and 85 task-specific sentences recorded for this dataset were already recorded in ZuCo 1 0  This allows for comparisons between the different recording procedures (i e  session-specific effects) and between more participants (subject-specific effects) 
Corpus Construction ::: Experimental design
As mentioned above, we recorded two different reading tasks for the ZuCo 2 0 dataset  During both tasks the participants were able to read in their own speed, using a control pad to move to the next sentence and to answer the control questions, which allowed for natural reading  Since each subject reads at their own personal pace, the reading speed between varies between subjects  Table TABREF4 shows the average reading speed for each task, i e  the average number of seconds a subject spends per sentence before switching to the next one 
All 739 sentences were recorded in a single session for each participant  The duration of the recording sessions was between 100 and 180 minutes, depending on the time required to set up and calibrate the devices, and the personal reading speed of the participants 
We recorded 14 blocks of approx  50 sentences, alternating between tasks: 50 sentences of normal reading, followed by 50 sentences of task-specific reading  The order of blocks and sentences within blocks was identical for all subjects  Each sentence block was preceded by a practice round of three sentences 
Corpus Construction ::: Experimental design ::: Normal reading (NR)
In the first task, participants were instructed to read the sentences naturally, without any specific task other than comprehension  Participants were told to read the sentences normally without any special instructions  Figure FIGREF8 (left) shows an example sentence as it was depicted on the screen during recording  As shown in Figure FIGREF8 (middle), the control condition for this task consisted of single-choice questions about the content of the previous sentence  12% of randomly selected sentences were followed by such a comprehension question with three answer options on a separate screen 
Corpus Construction ::: Experimental design ::: Task-specific reading (TSR)
In the second task, the participants were instructed to search for a specific relation in each sentence they read  Instead of comprehension questions, the participants had to decide for each sentence whether it contains the relation or not, i e  they were actively annotating each sentence  Figure FIGREF8 (right) shows an example screen for this task  17% of the sentences did not include the relation type and were used as control conditions  All sentences within one block involved the same relation type
Context 3: Introduction and Background
Many reinforcement learning algorithms are designed for relatively small discrete or continuous action spaces and so have trouble scaling  Text-adventure games—or interaction fictions—are simulations in which both an agents' state and action spaces are in textual natural language  An example of a one turn agent interaction in the popular text-game Zork1 can be seen in Fig  FIGREF1  Text-adventure games provide us with multiple challenges in the form of partial observability, commonsense reasoning, and a combinatorially-sized state-action space  Text-adventure games are structured as long puzzles or quests, interspersed with bottlenecks  The quests can usually be completed through multiple branching paths  However, games can also feature one or more bottlenecks  Bottlenecks are areas that an agent must pass through in order to progress to the next section of the game regardless of what path the agent has taken to complete that section of the quest BIBREF0  In this work, we focus on more effectively exploring this space and surpassing these bottlenecks—building on prior work that focuses on tackling the other problems 
Formally, we use the definition of text-adventure games as seen in BIBREF1 and BIBREF2  These games are partially observable Markov decision processes (POMDPs), represented as a 7-tuple of $\langle S,T,A,\Omega , O,R, \gamma \rangle $ representing the set of environment states, mostly deterministic conditional transition probabilities between states, the vocabulary or words used to compose text commands, observations returned by the game, observation conditional probabilities, reward function, and the discount factor respectively  For our purposes, understanding the exact state and action spaces we use in this work is critical and so we define each of these in relative depth 
Action-Space  To solve Zork1, the cannonical text-adventure games, requires the generation of actions consisting of up to five-words from a relatively modest vocabulary of 697 words recognized by the game’s parser  This results in $\mathcal {O}(697^5)={1 64e14}$ possible actions at every step  To facilitate text-adventure game playing, BIBREF2 introduce Jericho, a framework for interacting with text-games  They propose a template-based action space in which the agent first selects a template, consisting of an action verb and preposition, and then filling that in with relevant entities $($e g  $[get]$ $ [from] $ $)$  Zork1 has 237 templates, each with up to two blanks, yielding a template-action space of size $\mathcal {O}(237 \times 697^2)={1 15e8}$  This space is still far larger than most used by previous approaches applying reinforcement learning to text-based games 
State-Representation  Prior work has shown that knowledge graphs are effective in terms of dealing with the challenges of partial observability $($BIBREF3 BIBREF3; BIBREF4$)$  A knowledge graph is a set of 3-tuples of the form $\langle subject, relation, object \rangle $  These triples are extracted from the observations using Stanford's Open Information Extraction (OpenIE) BIBREF5  Human-made text-adventure games often contain relatively complex semi-structured information that OpenIE is not designed to parse and so they add additional rules to ensure that the correct information is parsed  The graph itself is more or less a map of the world, with information about objects' affordances and attributes linked to the rooms that they are place in a map  The graph also makes a distinction with respect to items that are in the agent's possession or in their immediate surrounding environment  An example of what the knowledge graph looks like and specific implementation details can be found in Appendix SECREF14 
BIBREF6 introduce the KG-A2C, which uses a knowledge graph based state-representation to aid in the section of actions in a combinatorially-sized action-space—specifically they use the knowledge graph to constrain the kinds of entities that can be filled in the blanks in the template action-space  They test their approach on Zork1, showing the combination of the knowledge graph and template action selection resulted in improvements over existing methods  They note that their approach reaches a score of 40 which corresponds to a bottleneck in Zork1 where the player is eaten by a “grue” (resulting in negative reward) if the player has not first lit a lamp  The lamp must be lit many steps after first being encountered, in a different section of the game; this action is necessary to continue exploring but doesn’t immediately produce any positive reward  That is, there is a long term dependency between actions that is not immediately rewarded, as seen in Figure FIGREF1

ground_truth: translation probabilities, Labeled Attachment Scores (LAS)

accuracy, Labeled Attachment Scores (LAS)

prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. Did they experiment with this new dataset?
Context 2:  The blocks started with a practice round, which described the relation and was followed by three sample sentences, so that the participants would be familiar with the respective relation type 
Corpus Construction ::: Linguistic assessment
As a linguistic assessment, the vocabulary and language proficiency of the participants was tested with the LexTALE test (Lexical Test for Advanced Learners of English, lemhofer2012introducing)  This is an unspeeded lexical decision task designed for intermediate to highly proficient language users  The average LexTALE score over all participants was 88 54%  Moreover, we also report the scores the participants achieved with their answers to the reading comprehension control questions and their relation annotations  The detailed scores for all participants are also presented in Table TABREF4 
Corpus Construction ::: Data acquisition
Data acquisition took place in a sound-attenuated and dark experiment room  Participants were seated at a distance of 68cm from a 24-inch monitor with a resolution of 800x600 pixels  A stable head position was ensured via a chin rest  Participants were instructed to stay as still as possible during the tasks to avoid motor EEG artifacts  Participants were also offered snacks and water during the breaks and were encouraged to rest  All sentences were presented at the same position on the screen and could span multiple lines  The sentences were presented in black on a light grey background with font size 20-point Arial, resulting in a letter height of 0 8 mm  The experiment was programmed in MATLAB 2016b BIBREF10, using PsychToolbox BIBREF11  Participants completed the tasks sitting alone in the room, while two research assistants were monitoring their progress in the adjoining room  All recording scripts including detailed participant instructions are available alongside the data 
Corpus Construction ::: Data acquisition ::: Eye-tracking acquisition
Eye position and pupil size were recorded with an infrared video-based eye tracker (EyeLink 1000 Plus, SR Research) at a sampling rate of 500 Hz  The eye tracker was calibrated with a 9-point grid at the beginning of the session and re-validated before each block of sentences 
Corpus Construction ::: Data acquisition ::: EEG acquisition
High-density EEG data were recorded at a sampling rate of 500 Hz with a bandpass of 0 1 to 100 Hz, using a 128-channel EEG Geodesic Hydrocel system (Electrical Geodesics)  The recording reference was set at electrode Cz  The head circumference of each participant was measured to select an appropriately sized EEG net  To ensure good contact, the impedance of each electrode was checked prior to recording, and was kept below 40 kOhm  Electrode impedance levels were checked after every third block of 50 sentences (approx  every 30 mins) and reduced if necessary 
Corpus Construction ::: Preprocessing and feature extraction ::: Eye-tracking
The eye-tracking data consists of (x,y) gaze location entries for all individual fixations (Figure FIGREF1b)  Coordinates were given in pixels with respect to the monitor coordinates (the upper left corner of the screen was (0,0) and down/right was positive)  We provide this raw data as well as various engineered eye-tracking features  For this feature extraction only fixations within the boundaries of each displayed word were extracted  Data points distinctly not associated with reading (minimum distance of 50 pixels to the text) were excluded  Additionally, fixations shorter than 100 ms were excluded from the analyses, because these are unlikely to reflect fixations relevant for reading BIBREF12  On the basis of the GECO and ZuCo 1 0 corpora, we extracted the following features: (i) gaze duration (GD), the sum of all fixations on the current word in the first-pass reading before the eye moves out of the word; (ii) total reading time (TRT), the sum of all fixation durations on the current word, including regressions; (iii) first fixation duration (FFD), the duration of the first fixation on the prevailing word; (iv) single fixation duration (SFD), the duration of the first and only fixation on the current word; and (v) go-past time (GPT), the sum of all fixations prior to progressing to the right of the current word, including regressions to previous words that originated from the current word  For each of these eye-tracking features we additionally computed the pupil size  Furthermore, we extracted the number of fixations and mean pupil size for each word and sentence 
Corpus Construction ::: Preprocessing and feature extraction ::: EEG
The EEG data shared in this project are available as raw data, but also preprocessed with Automagic (version 1 4 6, pedroni2019automagic), a tool for automatic EEG data cleaning and validation  105 EEG channels (i e  electrodes) were used from the scalp recordings  9 EOG channels were used for artifact removal and additional 14 channels lying mainly on the neck and face were discarded before data analysis  Bad channels were identified and interpolated  We used the Multiple Artifact Rejection Algorithm (MARA), a supervised machine learning algorithm that evaluates ICA components, for automatic artifact rejection  MARA has been trained on manual component classifications, and thus captures a wide range of artifacts  MARA is especially effective at detecting and removing eye and muscle artifact components  The effect of this preprocessing can be seen in Figure FIGREF1d 
After preprocessing, we synchronized the EEG and eye-tracking data to enable EEG analyses time-locked to the onsets of fixations  To compute oscillatory power measures, we band-pass filtered the continuous EEG signals across an entire reading task for five different frequency bands resulting in a time-series for each frequency band  The independent frequency bands were determined as follows: theta$_1$ (4–6 Hz), theta$_2$ (6 5–8 Hz), alpha$_1$ (8 5–10 Hz), alpha$_2$ (10 5–13 Hz), beta$_1$ (13 5–18 Hz), beta$_2$ (18 5–30 Hz), gamma$_1$ (30 5–40 Hz), and gamma$_2$ (40–49 5 Hz)  We then applied a Hilbert transformation to each of these time-series  We specifically chose the Hilbert transformation to maintain the temporal information of the amplitude of the frequency bands, to enable the power of the different frequencies for time segments defined through the fixations from the eye-tracking recording  Thus, for each eye-tracking feature we computed the corresponding EEG feature in each frequency band  Furthermore, we extracted sentence-level EEG features by calculating the power in each frequency band, and additionally, the difference of the power spectra between frontal left and right homologue electrodes pairs  For each eye-tracking based EEG feature, all channels were subject to an artifact rejection criterion of $90\mu V$ to exclude trials with transient noise 
Data Validation
The aim of the technical validation of the data is to guarantee good recording quality and to replicate findings of previous studies investigating co-registration of EEG and eye movement data during natural reading tasks (e g  dimigen2011coregistration)  We also compare the results to ZuCo 1 0 BIBREF1, which allows a more direct comparison due to the analogous recording procedure 
Data Validation ::: Eye-tracking
We validated the recorded eye-tracking data by analyzing the fixations made by all subjects through their reading speed and omission rate on sentence level  The omission rate is defined as the percentage of words that is not fixated in a sentence  Figure FIGREF10 (middle) shows the mean reading speed over all subjects, measured in seconds per sentence and Figure FIGREF10 (right) shows the mean omission rates aggregated over all subjects for each task  Clearly, the participants made less fixations during the task-specific reading, which lead to faster reading speed 
Moreover, we corroborated these sentence-level metrics by visualizing the skipping proportion on word level (Figure FIGREF13)  The skipping proportion is the average rate of words being skipped (i e  not being fixated) in a sentence  As expected, this also increases in the task-specific reading 
Although the reading material is from the same source and of the same length range (see Figure FIGREF10 (left)), in the first task (NR) passive reading was recorded, while in the second task (TSR) the subjects had to annotate a specific relation type in each sentence  Thus, the task-specific annotation reading lead to shorter passes because the goal was merely to recognize a relation in the text, but not necessarily to process the every word in each sentence  This distinct reading behavior is shown in Figure FIGREF15, where fixations occur until the end of the sentence during normal reading, while during task-specific reading the fixations stop after the decisive words to detect a given relation type  Finally, we also analyzed the average reading times for each of the extracted eye-tracking features  The means and distributions for both tasks are shown in Figure FIGREF21  These results are in line with the recorded data in ZuCo 1 0, as well as with the features extracted in the GECO corpus BIBREF4 
Data Validation ::: EEG
As a first validation step, we extracted fixation-related potentials (FRPs), where the EEG signal during all fixations of one task are averaged  Figure FIGREF24 shows the time-series of the resulting FRPs for two electrodes (PO8 and Cz), as well as topographies of the voltage distributions across the scalp at selected points in time  The five components (for which the scalp topographies are plotted) are highly similar in the time-course of the chosen electrodes to dimigen2011coregistration as well as to ZuCo 1 0 
Moreover, these previous studies were able to show an effect of fixation duration on the resulting FRPs  To show this dependency we followed two approaches  First, for each reading task, all single-trial FRPs were ordered by fixation duration and a vertical sliding time-window was used to smooth the data BIBREF13  Figure FIGREF25 (bottom) shows the resulting plots  In line with this previous work, a first positivation can be identified at 100 ms post-fixation onset  A second positive peak is located dependent on the duration of the fixation, which can be explained by the time-jittered succeeding fixation  The second approach is based on henderson2013co in which single trial EEG segments are clustered by the duration of the current fixation  As shown in Figure FIGREF25 (top), we chose four clusters and averaged the data within each cluster to four distinct FRPs, depending on the fixation duration  Again, the same positivation peaks become apparent  Both findings are consistent with the previous work mentioned and with our findings from ZuCo 1 0 
Conclusion
We presented a new, freely available corpus of eye movement and electrical brain activity recordings during natural reading as well as during annotation  This is the first dataset that allows for the comparison between these two reading paradigms  We described the materials and experiment design in detail and conducted an extensive validation to ensure the quality of the recorded data  Since this corpus is tailored to cognitively-inspired NLP, the applications and re-use potentials of this data are extensive  The provided word-level and sentence-level eye-tracking and EEG features can be used to improve and evaluate NLP and machine learning methods, for instance, to evaluate linguistic phenomena in neural models via psycholinguistic data  In addition, because the sentences contains semantic relation labels and the annotations of all participants, it can also be widely used for relation extraction and classification  Finally, the two carefully constructed reading paradigms allow for the comparison between normal reading and reading during annotation, which can be relevant to improve the manual labelling process as well as the quality of the annotations for supervised machine learning 
Context 3:  Others using artificially constrained action spaces also report an inability to pass through this bottleneck BIBREF7, BIBREF8  They pose a significant challenge for these methods because the agent does not see the correct action sequence to pass the bottleneck enough times  This is in part due to the fact that for that sequence to be reinforced, the agent needs to reach the next possible reward beyond the bottleneck 
More efficient exploration strategies are required to pass bottlenecks  Our contributions are two-fold  We first introduce a method that detects bottlenecks in text-games using the overall reward gained and the knowledge graph state  This method freezes the policy used to reach the bottleneck and restarts the training from there on out, additionally conducting a backtracking search to ensure that a sub-optimal policy has not been frozen  The second contribution explore how to leverage knowledge graphs to improve existing exploration algorithms for dealing with combinatorial action-spaces such as Go-Explore BIBREF9  We additionally present a comparative ablation study analyzing the performance of these methods on the popular text-game Zork1 
Exploration Methods
In this section, we describe methods to explore combinatorially sized action spaces such as text-games—focusing especially on methods that can deal with their inherent bottleneck structure  We first describe our method that explicitly attempts to detect bottlenecks and then describe how an exploration algorithm such as Go Explore BIBREF9 can leverage knowledge graphs 
KG-A2C-chained An example of a bottleneck can be seen in Figure FIGREF1  We extend the KG-A2C algorithm as follows  First, we detect bottlenecks as states where the agent is unable to progress any further  We set a patience parameter and if the agent has not seen a higher score in patience steps, the agent assumes it has been limited by a bottleneck  Second, when a bottleneck is found, we freeze the policy that gets the agent to the state with the highest score  The agent then begins training a new policy from that particular state 
Simply freezing the policy that led to the bottleneck, however, can potentially result in a policy one that is globally sub-optimal  We therefore employ a backtracking strategy that restarts exploration from each of the $n$ previous steps—searching for a more optimal policy that reaches that bottleneck  At each step, we keep track of a buffer of $n$ states and admissible actions that led up to that locally optimal state  We force the agent to explore from this state to attempt to drive it out of the local optima  If it is further unable to find itself out of this local optima, we refresh the training process again, but starting at the state immediately before the agent reaches the local optima  If this continues to fail, we continue to iterate through this buffer of seen states states up to that local optima until we either find a more optimal state or we run out of states to refresh from, in which we terminate the training algorithm 
KG-A2C-Explore Go-Explore BIBREF9 is an algorithm that is designed to keep track of sub-optimal and under-explored states in order to allow the agent to explore upon more optimal states that may be a result of sparse rewards  The Go-Explore algorithm consists of two phases, the first to continuously explore until a set of promising states and corresponding trajectories are found on the basis of total score, and the second to robustify this found policy against potential stochasticity in the game  Promising states are defined as those states when explored from will likely result in higher reward trajectories  Since the text games we are dealing with are mostly deterministic, with the exception of Zork in later stages, we only focus on using Phase 1 of the Go-Explore algorithm to find an optimal policy  BIBREF10 look at applying Go-Explore to text-games on a set of simpler games generated using the game generation framework TextWorld BIBREF1  Instead of training a policy network in parallel to generate actions used for exploration, they use a small set of “admissible actions”—actions guaranteed to change the world state at any given step during Phase 1—to explore and find high reward trajectories  This space of actions is relatively small (of the order of $10^2$ per step) and so finding high reward trajectories in larger action-spaces such as in Zork would be infeasible
Go-Explore maintains an archive of cells—defined as a set of states that map to a single representation—to keep track of promising states  BIBREF9 simply encodes each cell by keeping track of the agent's position and BIBREF10 use the textual observations encoded by recurrent neural network as a cell representation  We improve on this implementation by training the KG-A2C network in parallel, using the snapshot of the knowledge graph in conjunction with the game state to further encode the current state and use this as a cell representation  At each step, Go-Explore chooses a cell to explore at random (weighted by score to prefer more advanced cells)  The KG-A2C will run for a number of steps, starting with the knowledge graph state and the last seen state of the game from the cell  This will generate a trajectory for the agent while further training the KG-A2C at each iteration, creating a new representation for the knowledge graph as well as a new game state for the cell  After expanding a cell, Go-Explore will continue to sample cells by weight to continue expanding its known states  At the same time, KG-A2C will benefit from the heuristics of selecting preferred cells and be trained on promising states more often

ground_truth: No

prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. What are the results from these proposed strategies?
Context 3: 
Evaluation
We compare our two exploration strategies to the following baselines and ablations:
KG-A2C This is the exact same method presented in BIBREF6 with no modifications 
A2C Represents the same approach as KG-A2C but with all the knowledge graph components removed  The state representation is text only encoded using recurrent networks 
A2C-chained Is a variation on KG-A2C-chained where we use our policy chaining approach with the A2C method to train the agent instead of KG-A2C 
A2C-Explore Uses A2C in addition to the exploration strategy seen in KG-A2C-Explore  The cell representations here are defined in terms of the recurrent network based encoding of the textual observation 
Figure FIGREF10 shows that agents utilizing knowledge-graphs in addition to either enhanced exploration method far outperform the baseline A2C and KG-A2C  KG-A2C-chained and KG-A2C-Explore both pass the bottleneck of a score of 40, whereas A2C-Explore gets to the bottleneck but cannot surpass it 
There are a couple of key insights that can be drawn from these results The first is that the knowledge graph appears to be critical; it is theorized to help with partial observability  However the knowledge graph representation isn't sufficient in that the knowledge graph representation without enhanced exploration methods cannot surpass the bottleneck  A2C-chained—which explores without a knowledge graph—fails to even outperform the baseline A2C  We hypothesize that this is due to the knowledge graph aiding implicitly in the sample efficiency of bottleneck detection and subsequent exploration  That is, exploring after backtracking from a potentially detected bottleneck is much more efficient in the knowledge graph based agent 
The Go-Explore based exploration algorithm sees less of a difference between agents  A2C-Explore converges more quickly, but to a lower reward trajectory that fails to pass the bottleneck, whereas KG-A2C-Explore takes longer to reach a similar reward but consistently makes it through the bottleneck  The knowledge graph cell representation appears to thus be a better indication of what a promising state is as opposed to just the textual observation 
Comparing the advanced exploration methods when using the knowledge graph, we see that both agents successfully pass the bottleneck corresponding to entering the cellar and lighting the lamp and reach comparable scores within a margin of error  KG-A2C-chained is significantly more sample efficient and converges faster  We can infer that chaining policies by explicitly detecting bottlenecks lets us pass it more quickly than attempting to find promising cell representations with Go-Explore  This form of chained exploration with backtracking is particularly suited to sequential decision making problems that can be represented as acyclic directed graphs as in Figure FIGREF1 
Appendix ::: Zork1
Zork1 is one of the first text-adventure games and heavily influences games released later in terms of narrative style and game structure  It is a dungeon crawler where the player must explore a vast world and collect a series of treasures  It was identified by BIBREF2 as a moonshot game and has been the subject of much work in leaning agents BIBREF12, BIBREF7, BIBREF11, BIBREF8  Rewards are given to the player when they collect treasures as well as when important intermediate milestones needed to further explore the world are passed  Figure FIGREF15 and Figure FIGREF1 show us a map of the world of Zork1 and the corresponding quest structure 
The bottleneck seen at a score of around 40 is when the player first enters the cellar on the right side of the map  The cellar is dark and you need to immediately light the lamp to see anything  Attempting to explore the cellar in the dark results in you being instantly killed by a monster known as a “grue” 
Appendix ::: Knowledge Graph Rules
We make no changes from the graph update rules used by BIBREF6  Candidate interactive objects are identified by performing part-of-speech tagging on the current observation, identifying singular and proper nouns as well as adjectives, and are then filtered by checking if they can be examined using the command $examine$ $OBJ$  Only the interactive objects not found in the inventory are linked to the node corresponding to the current room and the inventory items are linked to the “you” node  The only other rule applied uses the navigational actions performed by the agent to infer the relative positions of rooms, e g  $\langle kitchen,down,cellar \rangle $ when the agent performs $go$ $down$ when in the kitchen to move to the cellar 
Appendix ::: Hyperparameters
Hyperparameters used for our agents are given below  Patience and buffer size are used for the policy chaining method as described in Section SECREF2  Cell step size is a parameter used for Go-Explore and describes how many steps are taken when exploring in a given cell state  Base hyperparameters for KG-A2C are taken from BIBREF6 and the same parameters are used for A2C 

ground_truth: Reward of 11.8 for the A2C-chained model, 41.8 for the KG-A2C-chained model, 40 for A2C-Explore and 44 for KG-A2C-Explore.

KG-A2C-chained and KG-A2C-Explore both pass the bottleneck of a score of 40