prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. Is it possible to convert a cloze-style questions to a naturally-looking questions?
Context 1: Introduction
Deep learning systems have shown a lot of promise for extractive Question Answering (QA), with performance comparable to humans when large scale data is available  However, practitioners looking to build QA systems for specific applications may not have the resources to collect tens of thousands of questions on corpora of their choice  At the same time, state-of-the-art machine reading systems do not lend well to low-resource QA settings where the number of labeled question-answer pairs are limited (c f  Table 2 )  Semi-supervised QA methods like BIBREF0 aim to improve this performance by leveraging unlabeled data which is easier to collect 
In this work, we present a semi-supervised QA system which requires the end user to specify a set of base documents and only a small set of question-answer pairs over a subset of these documents  Our proposed system consists of three stages  First, we construct cloze-style questions (predicting missing spans of text) from the unlabeled corpus; next, we use the generated clozes to pre-train a powerful neural network model for extractive QA BIBREF1 , BIBREF2 ; and finally, we fine-tune the model on the small set of provided QA pairs 
Our cloze construction process builds on a typical writing phenomenon and document structure: an introduction precedes and summarizes the main body of the article  Many large corpora follow such a structure, including Wikipedia, academic papers, and news articles  We hypothesize that we can benefit from the un-annotated corpora to better answer various questions – at least ones that are lexically similar to the content in base documents and directly require factual information 
We apply the proposed system on three datasets from different domains – SQuAD BIBREF3 , TriviaQA-Web BIBREF4 and the BioASQ challenge BIBREF5   We observe significant improvements in a low-resource setting across all three datasets  For SQuAD and TriviaQA, we attain an F1 score of more than 50% by merely using 1% of the training data  Our system outperforms the approaches for semi-supervised QA presented in BIBREF0 , and a baseline which uses the same unlabeled data but with a language modeling objective for pretraining  In the BioASQ challenge, we outperform the best performing system from previous year's challenge, improving over a baseline which does transfer learning from the SQuAD dataset  Our analysis reveals that questions which ask for factual information and match to specific parts of the context documents benefit the most from pretraining on automatically constructed clozes 
Related Work
Semi-supervised learning augments the labeled dataset $L$ with a potentially larger unlabeled dataset $U$   BIBREF0 presented a model, GDAN, which trained an auxiliary neural network to generate questions from passages by reinforcement learning, and augment the labeled dataset with the generated questions to train the QA model  Here we use a much simpler heuristic to generate the auxiliary questions, which also turns out to be more effective as we show superior performance compared to GDAN  Several approaches have been suggested for generating natural questions BIBREF6 , BIBREF7 , BIBREF8 , however none of them show a significant improvement of using the generated questions in a semi-supervised setting  Recent papers also use unlabeled data for QA by training large language models and extracting contextual word vectors from them to input to the QA model BIBREF9 , BIBREF10 , BIBREF11   The applicability of this method in the low-resource setting is unclear as the extra inputs increase the number of parameters in the QA model, however, our pretraining can be easily applied to these models as well 
Domain adaptation (and Transfer learning) leverage existing large scale datasets from a source domain (or task) to improve performance on a target domain (or task)  For deep learning and QA, a common approach is to pretrain on the source dataset and then fine-tune on the target dataset BIBREF12 , BIBREF13   BIBREF14 used SQuAD as a source for the target BioASQ dataset, and BIBREF15 used Book Test BIBREF16 as source for the target SQuAD dataset  BIBREF17 transfer learned model layers from the tasks of sequence labeling, text classification and relation classification to show small improvements on SQuAD  All these works use manually curated source datatset, which in themselves are expensive to collect  Instead, we show that it is possible to automatically construct the source dataset from the same domain as the target, which turns out to be more beneficial in terms of performance as well (c f  Section "Experiments & Results" )  Several cloze datasets have been proposed in the literature which use heuristics for construction BIBREF18 , BIBREF19 , BIBREF20   We further see the usability of such a dataset in a semi-supervised setting 
Methodology
Our system comprises of following three steps:
Cloze generation: Most of the documents typically follow a template, they begin with an introduction that provides an overview and a brief summary for what is to follow  We assume such a structure while constructing our cloze style questions  When there is no clear demarcation, we treat the first $K\%$ (hyperparameter, in our case 20%) of the document as the introduction  While noisy, this heuristic generates a large number of clozes given any corpus, which we found to be beneficial for semi-supervised learning despite the noise 
We use a standard NLP pipeline based on Stanford CoreNLP (for SQuAD, TrivaQA and PubMed) and the BANNER Named Entity Recognizer (only for PubMed articles) to identify entities and phrases  Assume that a document comprises of introduction sentences $\lbrace q_1, q_2,     q_n\rbrace $ , and the remaining passages $\lbrace p_1, p_2,    p_m\rbrace $   Additionally, let's say that each sentence $q_i$ in introduction is composed of words $\lbrace w_1, w_2,     w_{l_{q_i}}\rbrace $ , where $l_{q_i}$ is the length of $q_i$   We consider a $\text{match} (q_i, p_j)$ , if there is an exact string match of a sequence of words $\lbrace w_k, w_{k+1},    w_{l_{q_i}}\rbrace $ between the sentence $q_i$ and passage $p_j$   If this sequence is either a noun phrase, verb phrase, adjective phrase or a named entity in $\lbrace p_1, p_2,    p_m\rbrace $0 , as recognized by CoreNLP or BANNER, we select it as an answer span $\lbrace p_1, p_2,    p_m\rbrace $1   Additionally, we use $\lbrace p_1, p_2,    p_m\rbrace $2 as the passage $\lbrace p_1, p_2,    p_m\rbrace $3 and form a cloze question $\lbrace p_1, p_2,    p_m\rbrace $4 from the answer bearing sentence $\lbrace p_1, p_2,    p_m\rbrace $5 by replacing $\lbrace p_1, p_2,    p_m\rbrace $6 with a placeholder  As a result, we obtain passage-question-answer ( $\lbrace p_1, p_2,    p_m\rbrace $7 ) triples (Table 1 shows an example)  As a post-processing step, we prune out $\lbrace p_1, p_2,    p_m\rbrace $8 triples where the word overlap between the question (Q) and passage (P) is less than 2 words (after excluding the stop words) 
The process relies on the fact that answer candidates from the introduction are likely to be discussed in detail in the remainder of the article  In effect, the cloze question from the introduction and the matching paragraph in the body forms a question and context passage pair  We create two cloze datasets, one each from Wikipedia corpus (for SQuAD and TriviaQA) and PUBMed academic papers (for the BioASQ challenge), consisting of 2 2M and 1M clozes respectively  From analyzing the cloze data manually, we were able to answer 76% times for the Wikipedia set and 80% times for the PUBMed set using the information in the passage  In most cases the cloze paraphrased the information in the passage, which we hypothesized to be a useful signal for the downstream QA task 
We also investigate the utility of forming subsets of the large cloze corpus, where we select the top passage-question-answer triples, based on the different criteria, like i) jaccard similarity of answer bearing sentence in introduction and the passage ii) the tf-idf scores of answer candidates and iii) the length of answer candidates  However, we empirically find that we were better off using the entire set rather than these subsets 
Pre-training: We make use of the generated cloze dataset to pre-train an expressive neural network designed for the task of reading comprehension  We work with two publicly available neural network models – the GA Reader BIBREF2 (to enable comparison with prior work) and BiDAF + Self-Attention (SA) model from BIBREF1 (which is among the best performing models on SQuAD and TriviaQA)  After pretraining, the performance of BiDAF+SA on a dev set of the (Wikipedia) cloze questions is 0 58 F1 score and 0 55 Exact Match (EM) score  This implies that the cloze corpus is neither too easy, nor too difficult to answer 
Fine Tuning: We fine tune the pre-trained model, from the previous step, over a small set of labelled question-answer pairs  As we shall later see, this step is crucial, and it only requires a handful of labelled questions to achieve a significant proportion of the performance typically attained by training on tens of thousands of questions 
Datasets
We apply our system to three datasets from different domains  SQuAD BIBREF3 consists of questions whose answers are free form spans of text from passages in Wikipedia articles  We follow the same setting as in BIBREF0 , and split $10\%$ of training questions as the test set, and report performance when training on subsets of the remaining data ranging from $1\%$ to $90\%$ of the full set  We also report the performance on the dev set when trained on the full training set ( $1^\ast $ in Table 2 )  We use the same hyperparameter settings as in prior work  We compare and study four different settings: 1) the Supervised Learning (SL) setting, which is only trained on the supervised data, 2) the best performing GDAN model from BIBREF0 , 3) pretraining on a Language Modeling (LM) objective and fine-tuning on the supervised data, and 4) pretraining on the Cloze dataset and fine-tuning on the supervised data  The LM and Cloze methods use exactly the same data for pretraining, but differ in the loss functions used  We report F1 and EM scores on our test set using the official evaluation scripts provided by the authors of the dataset 
TriviaQA BIBREF4 comprises of over 95K web question-answer-evidence triples  Like SQuAD, the answers are spans of text  Similar to the setting in SQuAD, we create multiple smaller subsets of the entire set  For our semi-supervised QA system, we use the BiDAF+SA model BIBREF1 – the highest performing publicly available system for TrivaQA  Here again, we compare the supervised learning (SL) settings against the pretraining on Cloze set and fine tuning on the supervised set  We report F1 and EM scores on the dev set 
We also test on the BioASQ 5b dataset, which consists of question-answer pairs from PubMed abstracts  We use the publicly available system from BIBREF14 , and follow the exact same setup as theirs, focusing only on factoid and list questions  For this setting, there are only 899 questions for training  Since this is already a low-resource problem we only report results using 5-fold cross-validation on all the available data  We report Mean Reciprocal Rank (MRR) on the factoid questions, and F1 score for the list questions 
Main Results
Table 2 shows a comparison of the discussed settings on both SQuAD and TriviaQA  Without any fine-tuning (column 0) the performance is low, probably because the model never saw a real question, but we see significant gains with Cloze pretraining even with very little labeled data  The BiDAF+SA model, exceeds an F1 score of $50\%$ with only $1\%$ of the training data (454 questions for SQuAD, and 746 questions for TriviaQA), and approaches $90\%$ of the best performance with only $10\%$ labeled data  The gains over the SL setting, however, diminish as the size of the labeled set increases and are small when the full dataset is available 
Cloze pretraining outperforms the GDAN baseline from BIBREF0 using the same SQuAD dataset splits  Additionally, we show improvements in the $90\%$ data case unlike GDAN  Our approach is also applicable in the extremely low-resource setting of $1\%$ data, which we suspect GDAN might have trouble with since it uses the labeled data to do reinforcement learning  Furthermore, we are able to use the same cloze dataset to improve performance on both SQuAD and TriviaQA datasets  When we use the same unlabeled data to pre-train with a language modeling objective, the performance is worse, showing the bias we introduce by constructing clozes is important 
On the BioASQ dataset (Table 3 ) we again see a significant improvement when pretraining with the cloze questions over the supervised baseline  The improvement is smaller than what we observe with SQuAD and TriviaQA datasets – we believe this is because questions are generally more difficult in BioASQ  BIBREF14 showed that pretraining on SQuAD dataset improves the downstream performance on BioASQ  Here, we show a much larger improvement by pretraining on cloze questions constructed in an unsupervised manner from the same domain 
Analysis
Regression Analysis: To understand which types of questions benefit from pre-training, we pre-specified certain features (see Figure 1 right) for each of the dev set questions in SQuAD, and then performed linear regression to predict the F1 score for that question from these features  We predict the F1 scores from the cloze pretrained model ( $y^{\text{cloze}}$ ), the supervised model ( $y^{\text{sl}}$ ), and the difference of the two ( $y^{\text{cloze}}-y^{\text{sl}}$ ), when using $10\%$ of labeled data  The coefficients of the fitted model are shown in Figure 1 (left) along with their std errors  Positive coefficients indicate that a high value of that feature is predictive of a high F1 score, and a negative coefficient indicates that a small value of that feature is predictive of a high F1 score (or a high difference of F1 scores from the two models in the case of $y^{\text{cloze}}-y^{\text{sl}}$ ) 
The two strongest effects we observe are that a high lexical overlap between the question and the sentence containing the answer is indicative of high boost with pretraining, and that a high lexical overlap between the question and the whole passage is indicative of the opposite  This is hardly surprising, since our cloze construction process is biased towards questions which have a similar phrasing to the answer sentences in context  Hence, test questions with a similar property are answered correctly after pretraining, whereas those with a high overlap with the whole passage tend to have lower performance  The pretraining also favors questions with short answers because the cloze construction process produces short answer spans  Also passages and questions which consist of tokens infrequent in the SQuAD training corpus receive a large boost after pretraining, since the unlabeled data covers a larger domain 
Performance on question types: Figure 2 shows the average gain in F1 score for different types of questions, when we pretrain on the clozes compared to the supervised case  This analysis is done on the $10\%$ split of the SQuAD training set  We consider two classifications of each question – one determined on the first word (usually a wh-word) of the question (Figure 2 (bottom)) and one based on the output of a separate question type classifier adapted from BIBREF21   We use the coarse grain labels namely Abbreviation (ABBR), Entity (ENTY), Description (DESC), Human (HUM), Location (LOC), Numeric (NUM) trained on a Logistic Regression classification system   While there is an improvement across the board, we find that abbreviation questions in particular receive a large boost  Also, "why" questions show the least improvement, which is in line with our expectation, since these usually require reasoning or world knowledge which cloze questions rarely require 
Conclusion
In this paper, we show that pre-training QA models with automatically constructed cloze questions improves the performance of the models significantly, especially when there are few labeled examples  The performance of the model trained only on the cloze questions is poor, validating the need for fine-tuning  Through regression analysis, we find that pretraining helps with questions which ask for factual information located in a specific part of the context  For future work, we plan to explore the active learning setup for this task – specifically, which passages and / or types of questions can we select to annotate, such that there is a maximum performance gain from fine-tuning  We also want to explore how to adapt cloze style pre-training to NLP tasks other than QA 
Acknowledgments
Bhuwan Dhingra is supported by NSF under grants CCF-1414030 and IIS-1250956 and by grants from Google  Danish Pruthi and Dheeraj Rajagopal are supported by the DARPA Big Mechanism program under ARO contract W911NF-14-1-0436 
Context 2: Introduction
How humans process language has become increasingly relevant in natural language processing since physiological data during language understanding is more accessible and recorded with less effort  In this work, we focus on eye-tracking and electroencephalography (EEG) recordings to capture the reading process  On one hand, eye movement data provides millisecond-accurate records about where humans look when they are reading, and is highly correlated with the cognitive load associated with different stages of text processing  On the other hand, EEG records electrical brain activity across the scalp and is a direct measure of physiological processes, including language processing  The combination of both measurement methods enables us to study the language understanding process in a more natural setting, where participants read full sentences at a time, in their own speed  Eye-tracking then permits us to define exact word boundaries in the timeline of a subject reading a sentence, allowing the extraction of brain activity signals for each word 
Human cognitive language processing data is immensely useful for NLP: Not only can it be leveraged to improve NLP applications (e g  barrett2016weakly for part-of-speech tagging or klerke2016improving for sentence compression), but also to evaluate state-of-the-art machine learning systems  For example, hollenstein2019cognival evaluate word embeddings, or schwartz2019inducing fine-tune language models with brain-relevant bias 
Additionally, the availability of labelled data plays a crucial role in all supervised machine learning applications  Physiological data can be used to understand and improve the labelling process (e g  tokunaga2017eye), and, for instance, to build cost models for active learning scenarios BIBREF0  Is it possible to replace this expensive manual work with models trained on physiological activity data recorded from humans while reading? That is to say, can we find and extract relevant aspects of text understanding and annotation directly from the source, i e  eye-tracking and brain activity signals during reading?
Motivated by these questions and our previously released dataset, ZuCo 1 0 BIBREF1, we developed this new corpus, where we specifically aim to collect recordings during natural reading as well as during annotation 
We provide the first dataset of simultaneous eye movement and brain activity recordings to analyze and compare normal reading to task-specific reading during annotation  The Zurich Cognitive Language Processing Corpus (ZuCo) 2 0, including raw and preprocessed eye-tracking and electroencephalography (EEG) data of 18 subjects, as well as the recording and preprocessing scripts, is publicly available at https://osf io/2urht/  It contains physiological data of each subject reading 739 English sentences from Wikipedia (see example in Figure FIGREF1)  We want to highlight the re-use potential of this data  In addition to the psycholinguistic motivation, this corpus is especially tailored for training and evaluating machine learning algorithms for NLP purposes  We conduct a detailed technical validation of the data as proof of the quality of the recordings 
Related Work
Some eye-tracking corpora of natural reading (e g  the Dundee BIBREF2, Provo BIBREF3 and GECO corpus BIBREF4), and a few EEG corpora (for example, the UCL corpus BIBREF5) are available  It has been shown that this type of cognitive processing data is useful for improving and evaluating NLP methods (e g  barrett2018sequence,hollenstein2019cognival, hale2018finding)  However, before the Zurich Cognitive Language Processing Corpus (ZuCo 1 0), there was no available data for simultaneous eye-tracking and EEG recordings of natural reading  dimigen2011coregistration studied the linguistic effects of eye movements and EEG co-registration in natural reading and showed that they accurately represent lexical processing  Moreover, the simultaneous recordings are crucial to extract word-level brain activity signals 
While the above mentioned studies analyze and leverage natural reading, some NLP work has used eye-tracking during annotation (but, as of yet, not EEG data)  mishra2016predicting and joshi2014measuring recorded eye-tracking during binary sentiment annotation (positive/negative)  This data was used to determine the annotation complexity of the text passages based on eye movement metrics and for sarcasm detection BIBREF6  Moreover, eye-tracking has been used to analyze the word sense annotation process in Hindi BIBREF7, named entity annotation in Japanese BIBREF8, and to leverage annotator gaze behaviour for coreference resolution BIBREF9  Finally, tomanek2010cognitive used eye-tracking data during entity annotation to build a cost model for active learning  However, until now there is no available data or research that analyzes the differences in the human processing of normal reading versus annotation 
Related Work ::: ZuCo1 0
In previous work, we recorded a first dataset of simultaneous eye-tracking and EEG during natural reading BIBREF1  ZuCo 1 0 consists of three reading tasks, two of which contain very similar reading material and experiments as presented in the current work  However, the main difference and reason for recording ZuCo 2 0, consists in the experiment procedure  For ZuCo 1 0 the normal reading and task-specific reading paradigms were recorded in different sessions on different days  Therefore, the recorded data is not appropriate as a means of comparison between natural reading and annotation, since the differences in the brain activity data might result mostly from the different sessions due to the sensitivity of EEG  This, and extending the dataset with more sentences and more subjects, were the main factors for recording the current corpus  We purposefully maintained an overlap of some sentences between both datasets to allow additional analyses (details are described in Section SECREF7) 
Corpus Construction
In this section we describe the contents and experimental design of the ZuCo 2 0 corpus 
Corpus Construction ::: Participants
We recorded data from 19 participants and discarded the data of one of them due to technical difficulties with the eye-tracking calibration  Hence, we share the data of 18 participants  All participants are healthy adults (mean age = 34 (SD=8 3), 10 females)  Their native language is English, originating from Australia, Canada, UK, USA or South Africa  Two participants are left-handed and three participants wear glasses for reading  Details on subject demographics can be found in Table TABREF4  All participants gave written consent for their participation and the re-use of the data prior to the start of the experiments  The study was approved by the Ethics Commission of the University of Zurich 
Corpus Construction ::: Reading materials
During the recording session, the participants read 739 sentences that were selected from the Wikipedia corpus provided by culotta2006integrating  This corpus was chosen because it provides annotations of semantic relations  We included seven of the originally defined relation types: political_affiliation, education, founder, wife/husband, job_title, nationality, and employer  The sentences were chosen in the same length range as ZuCo 1 0, and with similar Flesch reading ease scores  The dataset statistics are shown in Table TABREF2 
Of the 739 sentences, the participants read 349 sentences in a normal reading paradigm, and 390 sentences in a task-specific reading paradigm, in which they had to determine whether a certain relation type occurred in the sentence or not  Table TABREF3 shows the distribution of the different relation types in the sentences of the task-specific annotation paradigm 
Purposefully, there are 63 duplicates between the normal reading and the task-specific sentences (8% of all sentences)  The intention of these duplicate sentences is to provide a set of sentences read twice by all participants with a different task in mind  Hence, this enables the comparison of eye-tracking and brain activity data when reading normally and when annotating specific relations (see examples in Section SECREF4) 
Furthermore, there is also an overlap in the sentences between ZuCo 1 0 and ZuCo 2 0  100 normal reading and 85 task-specific sentences recorded for this dataset were already recorded in ZuCo 1 0  This allows for comparisons between the different recording procedures (i e  session-specific effects) and between more participants (subject-specific effects) 
Corpus Construction ::: Experimental design
As mentioned above, we recorded two different reading tasks for the ZuCo 2 0 dataset  During both tasks the participants were able to read in their own speed, using a control pad to move to the next sentence and to answer the control questions, which allowed for natural reading  Since each subject reads at their own personal pace, the reading speed between varies between subjects  Table TABREF4 shows the average reading speed for each task, i e  the average number of seconds a subject spends per sentence before switching to the next one 
All 739 sentences were recorded in a single session for each participant  The duration of the recording sessions was between 100 and 180 minutes, depending on the time required to set up and calibrate the devices, and the personal reading speed of the participants 
We recorded 14 blocks of approx  50 sentences, alternating between tasks: 50 sentences of normal reading, followed by 50 sentences of task-specific reading  The order of blocks and sentences within blocks was identical for all subjects  Each sentence block was preceded by a practice round of three sentences 
Corpus Construction ::: Experimental design ::: Normal reading (NR)
In the first task, participants were instructed to read the sentences naturally, without any specific task other than comprehension  Participants were told to read the sentences normally without any special instructions  Figure FIGREF8 (left) shows an example sentence as it was depicted on the screen during recording  As shown in Figure FIGREF8 (middle), the control condition for this task consisted of single-choice questions about the content of the previous sentence  12% of randomly selected sentences were followed by such a comprehension question with three answer options on a separate screen 
Corpus Construction ::: Experimental design ::: Task-specific reading (TSR)
In the second task, the participants were instructed to search for a specific relation in each sentence they read  Instead of comprehension questions, the participants had to decide for each sentence whether it contains the relation or not, i e  they were actively annotating each sentence  Figure FIGREF8 (right) shows an example screen for this task  17% of the sentences did not include the relation type and were used as control conditions  All sentences within one block involved the same relation type
Context 3: Winograd Schemas
A Winograd schema (Levesque, Davis, and Morgenstern 2012) is a pair of sentences, or of short texts, called the elements of the schema, that satisfy the following constraints:
The following is an example of a Winograd schema:
Here, the two sentences differ only in the last word: `large' vs  `small'  The ambiguous pronoun is `it'  The two antecedents are `trophy' and `brown suitcase'  A human reader will naturally interpret `it' as referring to the trophy in the first sentence and to the suitcase in the second sentence, using the world knowledge that a small object can fit in a large container, but a large object cannot fit in a small container (Davis 2013)  Condition 4 is satisfied because either a trophy or a suitcase can be either large or small, and there is no reason to suppose that there would be a meaningful correlation of `small' or `large' with `trophy' or `suitcase' in a typical corpus 
We will say that an element of a Winograd schema is “solved” if the referent of the pronoun is identified 
An example of a pair of sentences satisfying conditions 1-3 but not 4 would be
Since women cannot be carcinogenic and pills cannot be pregnant, the pronoun `they' in these sentences is easily disambiguated using selectional restrictions  This pair is therefore not a valid Winograd schema 
The Winograd Schema Challenge (WSC) is a challenge for AI programs  The program is presented with a collection of sentences, each of which is one element of a Winograd schema, and is required to find the correct referent for the ambiguous pronoun  An AI program passes the challenge if its success rate is comparable to a human reader  The challenge is administered by commonsensereasoning org and sponsored by Nuance Inc  It was offered for the first time at IJCAI-2016 (Morgenstern, Davis, and Ortiz, in preparation); the organizers plan to continue to offer it roughly once a year 
Winograd schemas as the basis for challenges for machine translation programs
In many cases, the identification of the referent of the prounoun in a Winograd schema is critical for finding the correct translation of that pronoun in a different language  Therefore, Winograd schemas can be used as a very difficult challenge for the depth of understanding achieved by a machine translation program 
The third person plural pronoun `they' has no gender in English and most other languages (unlike the third person singular pronouns `he', `she', and `it')  However Romance languages such as French, Italian, and Spanish, and Semitic languages such as Hebrew and Arabic distinguish between the masculine and the feminine third-person plural pronouns, at least in some grammatical cases  For instance in French, the masculine pronoun is `ils'; the feminine pronoun is `elles'  In all of these cases, the masculine pronoun is standardly used for groups of mixed or unknown gender 
In order to correctly translate a sentence in English containing the word `they' into one of these languages, it is necessary to determine whether or not the referent is a group of females  If it is, then the translation must be the feminine pronoun; otherwise, it must be the masculine pronoun  Therefore, if one can create a Winograd schema in English where the ambiguous pronoun is `they' and the correct referent for one element is a collection of men and for the other is a collection of women, then to translate both elements correctly requires solving the Winograd schema 
As an example, consider the Winograd schema:
If these sentences are translated into French, then `they' in the first sentence should be translated `elles', as referring to Jane and Susan, and `they' in the second sentence should be translated `ils', as referring to Fred and George 
A number of the Winograd schemas already published can be “converted” quite easily to this form  Indeed, the above example was constructed in this way; the original form was “Jane knocked on Susan's door, but she did not [answer/get an answer] ” Of the 144 schemas in the collection at http://www cs nyu edu/faculty/davise/papers/WinogradSchemas/WSCollection html there are 33 that can plausibly be translated this way  (#'s 3, 4, 12, 15, 17, 18, 23, 25, 26, 27, 42, 43, 44, 45, 46, 57, 58, 60, 69, 70, 71, 77, 78, 79, 86, 97, 98, 106, 111, 114, 127, 132, 144) — essentially any schema that involves two people as possible referents where the content of the sentences makes sense as applied to groups of people rather than individuals 
A similar device, in the opposite direction, relies on the fact that French does not distinguish between the possessive pronouns `his' and `her'  The pronouns `son' and `sa' are gendered, but the gender agrees with the possession, not the possessor  Therefore a Winograd schema that relies on finding a referent for a possessive pronoun can be turned into a hard pair of French-to-English translation problems by making one possible referent male and the other one female  For example, schema #124 in the online collection reads “The man lifted the boy onto his [bunk bed/shoulders] ” Changing the boy to a girl and translating into French gives “L'homme leva la fille sur [ses épaules/son lit superposé]”

ground_truth: Unanswerable

Unanswerable

prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. Did they experiment with this new dataset?
Context 2:  The blocks started with a practice round, which described the relation and was followed by three sample sentences, so that the participants would be familiar with the respective relation type 
Corpus Construction ::: Linguistic assessment
As a linguistic assessment, the vocabulary and language proficiency of the participants was tested with the LexTALE test (Lexical Test for Advanced Learners of English, lemhofer2012introducing)  This is an unspeeded lexical decision task designed for intermediate to highly proficient language users  The average LexTALE score over all participants was 88 54%  Moreover, we also report the scores the participants achieved with their answers to the reading comprehension control questions and their relation annotations  The detailed scores for all participants are also presented in Table TABREF4 
Corpus Construction ::: Data acquisition
Data acquisition took place in a sound-attenuated and dark experiment room  Participants were seated at a distance of 68cm from a 24-inch monitor with a resolution of 800x600 pixels  A stable head position was ensured via a chin rest  Participants were instructed to stay as still as possible during the tasks to avoid motor EEG artifacts  Participants were also offered snacks and water during the breaks and were encouraged to rest  All sentences were presented at the same position on the screen and could span multiple lines  The sentences were presented in black on a light grey background with font size 20-point Arial, resulting in a letter height of 0 8 mm  The experiment was programmed in MATLAB 2016b BIBREF10, using PsychToolbox BIBREF11  Participants completed the tasks sitting alone in the room, while two research assistants were monitoring their progress in the adjoining room  All recording scripts including detailed participant instructions are available alongside the data 
Corpus Construction ::: Data acquisition ::: Eye-tracking acquisition
Eye position and pupil size were recorded with an infrared video-based eye tracker (EyeLink 1000 Plus, SR Research) at a sampling rate of 500 Hz  The eye tracker was calibrated with a 9-point grid at the beginning of the session and re-validated before each block of sentences 
Corpus Construction ::: Data acquisition ::: EEG acquisition
High-density EEG data were recorded at a sampling rate of 500 Hz with a bandpass of 0 1 to 100 Hz, using a 128-channel EEG Geodesic Hydrocel system (Electrical Geodesics)  The recording reference was set at electrode Cz  The head circumference of each participant was measured to select an appropriately sized EEG net  To ensure good contact, the impedance of each electrode was checked prior to recording, and was kept below 40 kOhm  Electrode impedance levels were checked after every third block of 50 sentences (approx  every 30 mins) and reduced if necessary 
Corpus Construction ::: Preprocessing and feature extraction ::: Eye-tracking
The eye-tracking data consists of (x,y) gaze location entries for all individual fixations (Figure FIGREF1b)  Coordinates were given in pixels with respect to the monitor coordinates (the upper left corner of the screen was (0,0) and down/right was positive)  We provide this raw data as well as various engineered eye-tracking features  For this feature extraction only fixations within the boundaries of each displayed word were extracted  Data points distinctly not associated with reading (minimum distance of 50 pixels to the text) were excluded  Additionally, fixations shorter than 100 ms were excluded from the analyses, because these are unlikely to reflect fixations relevant for reading BIBREF12  On the basis of the GECO and ZuCo 1 0 corpora, we extracted the following features: (i) gaze duration (GD), the sum of all fixations on the current word in the first-pass reading before the eye moves out of the word; (ii) total reading time (TRT), the sum of all fixation durations on the current word, including regressions; (iii) first fixation duration (FFD), the duration of the first fixation on the prevailing word; (iv) single fixation duration (SFD), the duration of the first and only fixation on the current word; and (v) go-past time (GPT), the sum of all fixations prior to progressing to the right of the current word, including regressions to previous words that originated from the current word  For each of these eye-tracking features we additionally computed the pupil size  Furthermore, we extracted the number of fixations and mean pupil size for each word and sentence 
Corpus Construction ::: Preprocessing and feature extraction ::: EEG
The EEG data shared in this project are available as raw data, but also preprocessed with Automagic (version 1 4 6, pedroni2019automagic), a tool for automatic EEG data cleaning and validation  105 EEG channels (i e  electrodes) were used from the scalp recordings  9 EOG channels were used for artifact removal and additional 14 channels lying mainly on the neck and face were discarded before data analysis  Bad channels were identified and interpolated  We used the Multiple Artifact Rejection Algorithm (MARA), a supervised machine learning algorithm that evaluates ICA components, for automatic artifact rejection  MARA has been trained on manual component classifications, and thus captures a wide range of artifacts  MARA is especially effective at detecting and removing eye and muscle artifact components  The effect of this preprocessing can be seen in Figure FIGREF1d 
After preprocessing, we synchronized the EEG and eye-tracking data to enable EEG analyses time-locked to the onsets of fixations  To compute oscillatory power measures, we band-pass filtered the continuous EEG signals across an entire reading task for five different frequency bands resulting in a time-series for each frequency band  The independent frequency bands were determined as follows: theta$_1$ (4–6 Hz), theta$_2$ (6 5–8 Hz), alpha$_1$ (8 5–10 Hz), alpha$_2$ (10 5–13 Hz), beta$_1$ (13 5–18 Hz), beta$_2$ (18 5–30 Hz), gamma$_1$ (30 5–40 Hz), and gamma$_2$ (40–49 5 Hz)  We then applied a Hilbert transformation to each of these time-series  We specifically chose the Hilbert transformation to maintain the temporal information of the amplitude of the frequency bands, to enable the power of the different frequencies for time segments defined through the fixations from the eye-tracking recording  Thus, for each eye-tracking feature we computed the corresponding EEG feature in each frequency band  Furthermore, we extracted sentence-level EEG features by calculating the power in each frequency band, and additionally, the difference of the power spectra between frontal left and right homologue electrodes pairs  For each eye-tracking based EEG feature, all channels were subject to an artifact rejection criterion of $90\mu V$ to exclude trials with transient noise 
Data Validation
The aim of the technical validation of the data is to guarantee good recording quality and to replicate findings of previous studies investigating co-registration of EEG and eye movement data during natural reading tasks (e g  dimigen2011coregistration)  We also compare the results to ZuCo 1 0 BIBREF1, which allows a more direct comparison due to the analogous recording procedure 
Data Validation ::: Eye-tracking
We validated the recorded eye-tracking data by analyzing the fixations made by all subjects through their reading speed and omission rate on sentence level  The omission rate is defined as the percentage of words that is not fixated in a sentence  Figure FIGREF10 (middle) shows the mean reading speed over all subjects, measured in seconds per sentence and Figure FIGREF10 (right) shows the mean omission rates aggregated over all subjects for each task  Clearly, the participants made less fixations during the task-specific reading, which lead to faster reading speed 
Moreover, we corroborated these sentence-level metrics by visualizing the skipping proportion on word level (Figure FIGREF13)  The skipping proportion is the average rate of words being skipped (i e  not being fixated) in a sentence  As expected, this also increases in the task-specific reading 
Although the reading material is from the same source and of the same length range (see Figure FIGREF10 (left)), in the first task (NR) passive reading was recorded, while in the second task (TSR) the subjects had to annotate a specific relation type in each sentence  Thus, the task-specific annotation reading lead to shorter passes because the goal was merely to recognize a relation in the text, but not necessarily to process the every word in each sentence  This distinct reading behavior is shown in Figure FIGREF15, where fixations occur until the end of the sentence during normal reading, while during task-specific reading the fixations stop after the decisive words to detect a given relation type  Finally, we also analyzed the average reading times for each of the extracted eye-tracking features  The means and distributions for both tasks are shown in Figure FIGREF21  These results are in line with the recorded data in ZuCo 1 0, as well as with the features extracted in the GECO corpus BIBREF4 
Data Validation ::: EEG
As a first validation step, we extracted fixation-related potentials (FRPs), where the EEG signal during all fixations of one task are averaged  Figure FIGREF24 shows the time-series of the resulting FRPs for two electrodes (PO8 and Cz), as well as topographies of the voltage distributions across the scalp at selected points in time  The five components (for which the scalp topographies are plotted) are highly similar in the time-course of the chosen electrodes to dimigen2011coregistration as well as to ZuCo 1 0 
Moreover, these previous studies were able to show an effect of fixation duration on the resulting FRPs  To show this dependency we followed two approaches  First, for each reading task, all single-trial FRPs were ordered by fixation duration and a vertical sliding time-window was used to smooth the data BIBREF13  Figure FIGREF25 (bottom) shows the resulting plots  In line with this previous work, a first positivation can be identified at 100 ms post-fixation onset  A second positive peak is located dependent on the duration of the fixation, which can be explained by the time-jittered succeeding fixation  The second approach is based on henderson2013co in which single trial EEG segments are clustered by the duration of the current fixation  As shown in Figure FIGREF25 (top), we chose four clusters and averaged the data within each cluster to four distinct FRPs, depending on the fixation duration  Again, the same positivation peaks become apparent  Both findings are consistent with the previous work mentioned and with our findings from ZuCo 1 0 
Conclusion
We presented a new, freely available corpus of eye movement and electrical brain activity recordings during natural reading as well as during annotation  This is the first dataset that allows for the comparison between these two reading paradigms  We described the materials and experiment design in detail and conducted an extensive validation to ensure the quality of the recorded data  Since this corpus is tailored to cognitively-inspired NLP, the applications and re-use potentials of this data are extensive  The provided word-level and sentence-level eye-tracking and EEG features can be used to improve and evaluate NLP and machine learning methods, for instance, to evaluate linguistic phenomena in neural models via psycholinguistic data  In addition, because the sentences contains semantic relation labels and the annotations of all participants, it can also be widely used for relation extraction and classification  Finally, the two carefully constructed reading paradigms allow for the comparison between normal reading and reading during annotation, which can be relevant to improve the manual labelling process as well as the quality of the annotations for supervised machine learning 
Context 3:  In the first sentence “ses” is translated “his”, in the second “son” is translated “her”  The eleven Winograd schemas #'s 112, 117, 118, 119, 124, 125, 127, 129, 130, 131, and 143 can be converted this way 
Care must be taken to avoid relying on, or seeming to rely on, objectionable stereotypes about men and women  One mechanism that can sometimes be used is to include a potentially problematic sentence in both directions  For instance, schema 23 from the WSC collection can be translated into both “The girls were bullying the boys so we [punished/rescued] them” and “The boys were bullying the girls, so we [punished/rescued] them,” thus avoiding any presupposition of whether girls are more likely to bully boys or vice versa 
A historical note: Winograd schemas were named after Terry Winograd because of a well-known example in his doctoral thesis (Winograd 1970)  In his thesis, this example followed the above form, and the importance of the example was justified in terms of machine translation  Winograd's original schema was: “The city councilmen refused to give the women a permit for a demonstration because they [feared/advocated] violence", and Winograd explained that, in the sentence with `feared', `they' would refer to the councilmen and would be translated as `ils' in French, whereas in the sentence with `advocated', `they' would refer to the women and would be translated as `elles'  In the later versions of the thesis, published as (Winograd 1972), he changed `women' to `demonstrators'; this made the disambiguation clearer, but lost the point about translation 
Current state of the art
No one familiar with the state of the art in machine translation technology or the state of the art of artificial intelligence generally will be surprised to learn that currently machine translation program are unable to solve these Winograd schema challenge problems 
What may be more surprising is that currently (July 2016), machine translation programs are unable to choose the feminine plural pronoun even when the only possible antecedent is a group of women  For instance, the sentence “The girls sang a song and they danced” is translated into French as “Les filles ont chanté une chanson et ils ont dansé” by Google Translate (GT), Bing Translate, and Yandex  In fact, I have been unable to construct any sentence in English that is translated into any language using the feminine plural pronun  Note that, since the masculine plural pronoun is used for groups of mixed gender in all these languages, it is almost certainly more common in text than the feminine plural; hence this strategy is reasonable faute de mieux  (It also seems likely that the erroneous use of a feminine plural for a masculine antecedent sounds even more jarring than the reverse )
The same thing sometimes occurs in translating the feminine plural pronoun between languages that have it  GT translates the French word `elles' into Spanish as `ellos' (the masculine form)  Curiously, in the opposite direction, it gets the right answer; the Spanish `ellas' (fem ) is translated into French as `elles' 
Language-specific issues
The masculine and feminine plural pronouns are distinguished in the Romance languages (French, Spanish, Italian, Portuguese etc ) and in Semitic languages (Arabic, Hebrew, etc ) I have consulted with native speakers and experts in these languages about the degree to which the gender distinction is observed in practice  The experts say that in French, Spanish, Italian, and Portuguese, the distinction is very strictly observed; the use of a masculine pronoun for a feminine antecedent is jarringly wrong to a native or fluent speaker  “Les filles ont chanté une chanson et ils ont dansé” sounds as wrong to a French speaker as “The girl sang a song and he danced” sounds to an English speaker; in both cases, the hearer will interpret the pronoun as referrinig to some other persons or person, who is male  In Hebrew and Arabic, this is much less true; in speech, and even, increasingly, in writing, the masculine pronoun is often used for a feminine antecedent 
Looking further ahead, it is certainly possible that gender distinctions will be abandoned in the Romance languages, or even that English will have driven all other languages out of existence, sooner than AI systems will be able to do pronoun resolution in Winograd schemas; at that point, this test will no longer be useful 
In some cases, a translation program can side-step the issue by omitting the pronoun altogether  For example, GT translates the above sentence “The girls sang a song and they danced” into Spanish as “Las chicas cantaron una canción y bailaban” and into Italian as “Le ragazze hanno cantato una canzone e ballavano ” However, with the more complex sentences of the Winograd Schemas, this strategy will rarely give a plausible translation for both elements of the schema 
Other languages, other ambiguities
Broadly speaking, whenever a target language INLINEFORM0 requires some distinction that is optional or non-existent in source language INLINEFORM1 , it is possible to create a sentence INLINEFORM2 in INLINEFORM3 where the missing information is not explicit but can be inferred from background knowledge  Translating INLINEFORM4 from INLINEFORM5 to INLINEFORM6 thus requires using the background knowledge to resolve the ambiguity, and will therefore be challenging for automatic machine translation 
A couple of examples: The word `sie' in German serves as both the formal second person prounoun (always capitalized), the third person feminine singular, and the third person plural  Therefore, it can be translated into English as either “you”, “she”, “it”, or “they”; and into French as either `vous', `il', `elle', `ils', or `elles'  (The feminine third-person singular German `sie' can be translated as neuter in English and as masculine in French because the three languages do not slice up the worlds into genders in the same way ) Likewise, the possessive pronoun `ihr' in all its declensions can mean either `her' or `their'  In some cases, the disambiguation can be carried out on purely syntactic ground; e g  if `sie' is the subject of a third-person singular verb, it must mean `she'

ground_truth: No

prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. What language do they explore?
Context 3:  However, in many case, the disambiguation requires a deeper level of understanding  Thus, it should be possible to construct German Winograd schemas based on the words `sie' or `ihr' that have to be solved in order to translate them into English  For example,
Marie fand fünf verlassene Kätzchen im Keller  Ihre Mutter war gestorben 
(Marie found five abandoned kittens in the cellar  Their mother was dead )
vs 
Marie fand fünf verlassene Kätzchen im Keller  Ihre Mutter war unzufrieden 
(Marie found five abandoned kittens in the cellar  Her mother was displeased )
Another example: French distinguishes between a male friend `ami' and a female friend `amie'  Therefore, in translating the word “friend” into French, it is necessary, if possible, to determine the sex of the friend; and the clue for that can involve an inference that, as far as AI programs are concerned, is quite remote  Human readers are awfully good at picking up on those, of course  In general, with this kind of thing, if you want to break GT or another translation program, the trick is to place a large separation between the evidence and the word being disambiguated  In this case, at the present time, the separation can be pretty small  GT correctly translates “my friend Pierre” and “my friend Marie” as “mon ami Pierre” and “mon amie Marie” and it translates “She is my friend” as “Elle est mon amie”, but rather surpringly it breaks down at “Marie is my friend,” which it translates “Marie est mon ami ” As for something like “Jacques said to Marie, `You have always been a true friend,' ” that is quite hopeless  GT can surprise one, though, in both directions; sometimes it misses a very close clue, as in “Marie is my friend”, but other times it can carry a clue further than one would have guessed 
Acknowledgements
Thanks to Arianna Bisazza, Gerhard Brewka, Antoine Cerfon, Joseph Davis, Gigi Dopico-Black, Nizar Habash, Leora Morgenstern, Oded Regev, Francesca Rossi, Vesna Sabljakovic-Fritz, and Manuela Veloso for help with the various languages and discussion of gendered pronouns 
References
E  Davis, “Qualitative Spatial Reasoning in Interpreting Text and Narrative” Spatial Cognition and Computation, 13:4, 2013, 264-294 
H  Levesque, E  Davis, and L  Morgenstern, “The Winograd Schema Challenge,” KR 2012 
L  Morgenstern, E  Davis, and C  Ortiz, “Report on the Winograd Schema Challenge, 2016”, in preparation 
T  Winograd, “Procedures as a Representation for Data in a Computer Program for Understanding Natural Language," Ph D  thesis, Department of Mathematics, MIT, August 1970  Published as MIT AITR-235, January 1971 
T  Winograd, Understanding Natural Language, Academic Press, 1972 

ground_truth: English, French, German 

French, English, Spanish, Italian, Portuguese, Hebrew, Arabic