Conclusion	We have introduced BoolQ, a new reading comprehension dataset of naturally occurring yes/no questions. We have shown these questions are challenging and require a wide range of inference abilities to solve. We have also studied how transfer learning performs on this task, and found crowd-sourced entailment datasets can be leveraged to boost performance even on top of language model pre-training. Future work could include building a document-level version of this task, which would increase its difficulty and its correspondence to an end-user application.

Related Work	Yes/No questions make up a subset of the reading comprehension datasets CoQA  , QuAC  , and HotPotQA  , and are present in the ShARC  dataset. These datasets were built to challenge models to understand conversational QA (for CoQA, ShARC and QuAC) or multi-step reasoning (for HotPotQA), which complicates our goal of using yes/no questions to test inferential abilities. Of the four, QuAC is the only one where the question authors were not allowed to view the text being used to answer their questions, making it the best candidate to contain naturally occurring questions. However, QuAC still heavily prompts users, including limiting their questions to be about pre-selected Wikipedia articles, and is highly class imbalanced with 80% “yes" answers. The MS Marco dataset  , which contains questions with free-form text answers, also includes some yes/no questions. We experiment with heuristically identifying them in Section SECREF4 , but this process can be noisy and the quality of the resulting annotations is unknown. We also found the resulting dataset is class imbalanced, with 80% “yes" answers.

Training Yes/No QA Models	Models on this dataset need to predict an output class given two pieces of input text, which is a well studied paradigm  . We find training models on our train set alone to be relatively ineffective. Our best model reaches 69.6% accuracy, only 8% better than the majority baseline. Therefore, we follow the recent trend in NLP of using transfer learning. In particular, we experiment with pre-training models on related tasks that have larger datasets, and then fine-tuning them on our training data. We list the sources we consider for pre-training below. Entailment: We consider two entailment datasets, MultiNLI  and SNLI  . We choose these datasets since they are widely-used and large enough to use for pre-training. We also experiment with ablating classes from MultiNLI. During fine-tuning we use the probability the model assigns to the “entailment" class as the probability of predicting a “yes" answer. Multiple-Choice QA: We use a multiple choice reading comprehension dataset, RACE  , which contains stories or short essays paired with questions built to test the reader's comprehension of the text.

Multi-Step Transfer Results	Our best single-step transfer learning results were from using the pre-trained BERTL model and MultiNLI. We also experiment with combining these approaches using a two-step pre-training regime. In particular, we fine-tune the pre-trained BERTL on MultiNLI, and then fine-tune the resulting model again on the BoolQ train set. We found decreasing the number of training epochs to 3 resulted in a slight improvement when using the model pre-trained on MultiNLI. We show the test set results for this model, and some other pre-training variations, in Table . For these results we train five versions of each model using different training seeds, and show the model that had the best dev-set performance. Given how extensively the BERTL model has been pre-trained, and how successful it has been across many NLP tasks, the additional gain of 3.5 points due to using MultiNLI is remarkable. This suggests MultiNLI contains signal orthogonal to what is found in BERT's unsupervised objectives.

Types of Inference	Finally, we categorize the kinds of inference required to answer the questions in BoolQ. The definitions and results are shown in Table . Less than 40% of the examples can be solved by detecting paraphrases. Instead, many questions require making additional inferences (categories “Factual Reasoning", “By Example", and “Other Inference") to connect what is stated in the passage to the question. There is also a significant class of questions (categories “Implicit" and “Missing Mention") that require a subtler kind of inference based on how the passage is written.

In this paper we propose an alternative: we test models on their ability to answer naturally occurring yes/no questions. That is, questions that were authored by people who were not prompted to write particular kinds of questions, including even being required to write yes/no questions, and who did not know the answer to the question they were asking. Figure contains some examples from our dataset. We find such questions often query for non-factoid information, and that human annotators need to apply a wide range of inferential abilities when answering them. As a result, they can be used to construct highly inferential reading comprehension datasets that have the added benefit of being directly related to the practical end-task of answering user yes/no questions. Yes/No questions do appear as a subset of some existing datasets  ,  ,  . However, these datasets are primarily intended to test other aspects of question answering (QA), such as conversational QA or multi-step reasoning, and do not contain naturally occurring questions. We follow the data collection method used by Natural Questions (NQ)  to gather 16,000 naturally occurring yes/no questions into a dataset we call BoolQ (for Boolean Questions).

The BoolQ Dataset	An example in our dataset consists of a question, a paragraph from a Wikipedia article, the title of the article, and an answer, which is either “yes" or “no". We include the article title since it can potentially help resolve ambiguities (e.g., coreferent phrases) in the passage, although none of the models presented in this paper make use of them.

Entailment Results: The MultiNLI dataset out-performed all other supervised methods by a large margin. Remarkably, this approach is only a few points behind BERT despite using orders of magnitude less training data and a much more light-weight model, showing high-quality pre-training data can help compensate for these deficiencies. Our ablation results show that removing the neutral class from MultiNLI hurt transfer slightly, and removing either of the other classes was very harmful, suggesting the neutral examples had limited value. SNLI transferred better than other datasets, but worse than MultiNLI. We suspect this is due to limitations of the photo-caption domain it was constructed from. Other Supervised Results: We obtained a small amount of transfer using QQP and Y/N MS Marco. Although Y/N MS Marco is a yes/no QA dataset, its small size and class imbalance likely contributed to its limited effectiveness. The web snippets it uses as passages also present a large domain shift from the Wikipedia passages in BoolQ. Unsupervised Results: Following results on other datasets  , we found BERTL to be the most effective unsupervised method, surpassing all other methods of pre-training.

Yes/No QA has been used in other contexts, such as the templated bAbI stories  or some Visual QA datasets  ,  . We focus on answering yes/no questions using natural language text. Question answering for reading comprehension in general has seen a great deal of recent work  ,  , and there have been many recent attempts to construct QA datasets that require advanced reasoning abilities  ,  ,  ,  ,  . However, these attempts typically involve engineering data to be more difficult by, for example, explicitly prompting users to write multi-step questions  ,  , or filtering out easy questions  . This risks resulting in models that do not have obvious end-use applications since they are optimized to perform in an artificial setting. In this paper, we show that yes/no questions have the benefit of being very challenging even when they are gathered from natural sources. Natural language inference is also a well studied area of research, particularly on the MultiNLI  and SNLI  datasets. Other sources of entailment data include the PASCAL RTE challenges  ,  or SciTail  .

Analysis	In the following section we analyze our dataset to better understand the nature of the questions, the annotation quality, and the kinds of reasoning abilities required to answer them.

Each question is paired with a paragraph from Wikipedia that an independent annotator has marked as containing the answer. The task is then to take a question and passage as input, and to return “yes" or “no" as output. Figure contains some examples, and Appendix SECREF17 contains additional randomly selected examples. Following recent work  , we focus on using transfer learning to establish baselines for our dataset. Yes/No QA is closely related to many other NLP tasks, including other forms of question answering, entailment, and paraphrasing. Therefore, it is not clear what the best data sources to transfer from are, or if it will be sufficient to just transfer from powerful pre-trained language models such as BERT  or ELMo  . We experiment with state-of-the-art unsupervised approaches, using existing entailment datasets, three methods of leveraging extractive QA data, and using a few other supervised datasets. We found that transferring from MultiNLI, and the unsupervised pre-training in BERT, gave us the best results. Notably, we found these approaches are surprisingly complementary and can be combined to achieve a large gain in performance.

On the other hand, detecting when a yes/no question can be answered using these strategies seems much easier and more intuitive. Thus, having annotators answer pre-existing questions opens the door to building datasets that contain more inference and have higher quality labels. A surprising result from our work is that the datasets that more closely resemble the format of BoolQ, meaning they contain questions and multi-sentence passages, such as SQuAD 2.0, RACE, or Y/N MS Marco, were not very useful for transfer. The entailment datasets were stronger despite consisting of sentence pairs. This suggests that adapting from sentence-pair input to question/passage input was not a large obstacle to achieving transfer. Preliminary work found attempting to convert the yes/no questions in BoolQ into declarative statements did not improve transfer from MultiNLI, which supports this hypothesis. The success of MultiNLI might also be surprising given recent concerns about the generalization abilities of models trained on it  , particularly related to “annotation artifacts" caused by using crowd workers to write the hypothesis statements  . We have shown that, despite these weaknesses, it can still be an important starting point for models being used on natural data.

This work builds upon the Natural Questions (NQ)  , which contains some natural yes/no questions. However, there are too few (about 1% of the corpus) to make yes/no QA a very important aspect of that task. In this paper, we gather a large number of additional yes/no questions in order to construct a dedicated yes/no QA dataset.

Overall, our best model reaches 80.43% accuracy, compared to 62.31% for the majority baseline and 90% human accuracy. In light of the fact BERT on its own has achieved human-like performance on several NLP tasks, this demonstrates the high degree of difficulty of our dataset. We present our data and code at https://goo.gl/boolq.

We note that, although SciTail, RTE-6 and RTE-7 did not use crowd workers to generate candidate statements, they still use sources (multiple choices questions or document summaries) that were written by humans with knowledge of the premise text. Using naturally occurring yes/no questions ensures even greater independence between the questions and premise text, and ties our dataset to a clear end-task. BoolQ also requires detecting entailment in paragraphs instead of sentence pairs. Transfer learning for entailment has been studied in GLUE  and SentEval  . Unsupervised pre-training in general has recently shown excellent results on many datasets, including entailment data  ,  ,  . Converting short-answer or multiple choice questions into entailment examples, as we do when experimenting with transfer learning, has been proposed in several prior works  ,  ,  . In this paper we found some evidence suggesting that these approaches are less effective than using crowd-sourced entailment examples when it comes to transferring to natural yes/no questions. Contemporaneously with our work,  showed that pre-training on supervised tasks could be beneficial even when using pre-trained language models, especially for a textual entailment task. Our work confirms these results for yes/no question answering.

Annotation Quality	First, in order to assess annotation quality, three of the authors labelled 110 randomly chosen examples. If there was a disagreement, the authors conferred and selected a single answer by mutual agreement. We call the resulting labels “gold-standard" labels. On the 110 selected examples, the answer annotations reached 90% accuracy compared to the gold-standard labels. Of the cases where the answer annotation differed from the gold-standard, six were ambiguous or debatable cases, and five were errors where the annotator misunderstood the passage. Since the agreement was sufficiently high, we elected to use singly-annotated examples in the training/dev/test sets in order to be able to gather a larger dataset.

We hypothesize that a key advantage of MultiNLI is that it contains examples of contradictions. The other sources of transfer we consider, including the next-sentence-selection objective in BERT, are closer to providing examples of entailed text vs. neutral/unrelated text. Indeed, we found that our two step transfer procedure only reaches 78.43% dev set accuracy if we remove the contradiction class from MultiNLI, regressing its performance close to the level of BERTL when just using unsupervised pre-training. Note that it is possible to pre-train a model on several of the suggested datasets, either in succession or in a multi-task setup. We leave these experiments to future work. Our results also suggest pre-training on MultiNLI would be helpful for other corpora that contain yes/no questions.

BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questions	In this paper we study yes/no questions that are naturally occurring --- meaning that they are generated in unprompted and unconstrained settings. We build a reading comprehension dataset, BoolQ, of such questions, and show that they are unexpectedly challenging. They often query for complex, non-factoid information, and require difficult entailment-like inference to solve. We also explore the effectiveness of a range of transfer learning baselines. We find that transferring from entailment data is more effective than transferring from paraphrase or extractive QA data, and that it, surprisingly, continues to be very beneficial even when starting from massive pre-trained language models such as BERT. Our best method trains BERT on MultiNLI and then re-trains it on our train set. It achieves 80.4% accuracy compared to 90% accuracy of human annotators (and 62% majority-baseline), leaving a significant gap for future work.

Paraphrasing: We use the Quora Question Paraphrasing (QQP) dataset, which consists of pairs of questions labelled as being paraphrases or not. Paraphrasing is related to entailment since we expect, at least in some cases, passages will contain a paraphrase of the question. Heuristic Yes/No: We attempt to heuristically construct a corpus of yes/no questions from the MS Marco corpus  . MS Marco has free-form answers paired with snippets of related web documents. We search for answers starting with “yes" or “no", and then pair the corresponding questions with snippets marked as being related to the question. We call this task Y/N MS Marco; in total we gather 38k examples, 80% of which are “yes” answers. Unsupervised: It is well known that unsupervised pre-training using language-modeling objectives  ,  ,  , can improve performance on many tasks. We experiment with these methods by using the pre-trained models from ELMo, BERT, and OpenAI's Generative Pre-trained Transformer (OpenAI GPT) (see Section SECREF11 ).

Following previous work  , we also try building entailment-like training data from SQuAD 2.0  . We concatenate questions with either the correct answer, or with the incorrect “distractor" answer candidate provided by the dataset, and train the model to classify which is which given the question's supporting text. Finally, we also experiment with leveraging the long-answer portion of NQ, where models must select a paragraph containing the answer to a question from a document. Following our method for Multiple-Choice QA, we train a model to assign a score to (question, paragraph) pairs, apply the softmax operator on paragraphs from the same document to get a probability distribution over the paragraphs, and train the model on the negative log probability of selecting an answer-containing paragraph. We only train on questions that were marked as having an answer, and select an answer-containing paragraph and up to 15 randomly chosen non-answer-containing paragraphs for each question. On BoolQ, we compute the probability of a “yes" answer by applying the sigmoid operator to the score the model gives to the input question and passage.

Question Types	Part of the value of this dataset is that it contains questions that people genuinely want to answer. To explore this further, we manually define a set of topics that questions can be about. An author categorized 200 questions into these topics. The results can be found in the upper half of Table . Questions were often about entertainment media (including T.V., movies, and music), along with other popular topics like sports. However, there are still a good portion of questions asking for more general factual knowledge, including ones about historical events or the natural world. We also broke the questions into categories based on what kind of information they were requesting, shown in the lower half of Table . Roughly one-sixth of the questions are about whether anything with a particular property exists (Existence), another sixth are about whether a particular event occurred (Event Occurrence), and another sixth ask whether an object is known by a particular name, or belongs to a particular category (Definitional). The questions that do not fall into these three categories were split between requesting facts about a specific entity, or requesting more general factual information.

Finally, annotators mark whether the question's answer is “yes" or “no". Annotating data in this manner is quite expensive since annotators need to search entire Wikipedia documents for relevant evidence and read the text carefully. Note that, unlike in NQ, we only use questions that were marked as having a yes/no answer, and pair each question with the selected passage instead of the entire document. This helps reduce ambiguity (ex., avoiding cases where the document supplies conflicting answers in different paragraphs), and keeps the input small enough so that existing entailment models can easily be applied to our dataset. We combine 13k questions gathered from this pipeline with an additional 3k questions with yes/no answers from the NQ training set to reach a total of 16k questions. We split these questions into a 3.2k dev set, 3.2k test set, and 9.4k train set, ensuring questions from NQ are always in the train set. “Yes” answers are slightly more common (62.31% in the train set). The queries are typically short (average length 8.9 tokens) with longer passages (average length 108 tokens).

Transfer Learning Results	The results of our transfer learning methods are shown in Table . All results are averaged over five runs. For models pre-trained on supervised datasets, both the pre-training and the fine-tuning stages were repeated. For unsupervised pre-training, we use the pre-trained models provided by the authors, but continue to average over five runs of fine-tuning. QA Results: We were unable to transfer from RACE or SQuAD 2.0. For RACE, the problem might be domain mismatch. In RACE the passages are stories, and the questions often query for passage-specific information such as the author's intent or the state of a particular entity from the passage, instead of general knowledge. We would expect SQuAD 2.0 to be a better match for BoolQ since it is also Wikipedia-based, but its possible detecting the adversarially-constructed distractors used for negative examples does not relate well to yes/no QA. We got better results using QNLI, and even better results using NQ. This shows the task of selecting text relevant to a question is partially transferable to yes/no QA, although we are only able to gain a few points over the baseline.

Following what was done in SciTail  , we convert questions and answer-options to statements by either substituting the answer-option for the blanks in fill-in-the-blank questions, or appending a separator token and the answer-option to the question. During training, we have models independently assign a score to each statement, and then apply the softmax operator between all statements per each question to get statement probabilities. We use the negative log probability of the correct statement as a loss function. To fine-tune on BoolQ, we apply the sigmoid operator to the score of the question given its passage to get the probability of a “yes" answer. Extractive QA: We consider several methods of leveraging extractive QA datasets, where the model must answer questions by selecting text from a relevant passage. Preliminary experiments found that simply transferring the lower-level weights of extractive QA models was ineffective, so we instead consider three methods of constructing entailment-like data from extractive QA data. First, we use the QNLI task from GLUE  , where the model must determine if a sentence from SQuAD 1.1  contains the answer to an input question or not.

Then pool these embeddings by computing attention scores INLINEFORM2 , INLINEFORM3 , and then the sum INLINEFORM4 = INLINEFORM5 . Likewise we compute INLINEFORM6 from the premise. Classify: Finally we feed INLINEFORM0 into a fully connected layer, and then through a softmax layer to predict the output class. We apply dropout at a rate of 0.2 between all layers, and train the model using the Adam optimizer  . The learning rate is decayed by 0.999 every 100 steps. We use 200 dimensional LSTMs and a 100 dimensional fully connected layer.

Discussion	Why do natural yes/no questions require inference so often? We hypothesize that there are several factors. First, we notice factoid questions that ask about simple properties of entities, such as “Was Obama born in 1962?", are rare. We suspect this is because people will almost always prefer to phrase such questions as short-answer questions (e.g., “When was Obama born?"). Thus, there is a natural filtering effect where people tend to use yes/no questions exactly when they want more complex kinds of information. Second, both the passages and questions rarely include negation. As a result, detecting a “no" answer typically requires understanding that a positive assertion in the text excludes, or makes unlikely, a positive assertion in the question. This requires reasoning that goes beyond paraphrasing (see the “Other-Inference" or “Implicit" examples). We also think it was important that annotators only had to answer questions, rather than generate them. For example, imagine trying to construct questions that fall into the categories of “Missing Mention" or “Implicit". While possible, it would require a great deal of thought and creativity.

Data Collection	We gather data using the pipeline from NQ  , but with an additional filtering step to focus on yes/no questions. We summarize the complete pipeline here, but refer to their paper for a more detailed description. Questions are gathered from anonymized, aggregated queries to the Google search engine. Queries that are likely to be yes/no questions are heuristically identified: we found selecting queries where the first word is in a manually constructed set of indicator words and are of sufficient length, to be effective. Questions are only kept if a Wikipedia page is returned as one of the first five results, in which case the question and Wikipedia page are given to a human annotator for further processing. Annotators label question/article pairs in a three-step process. First, they decide if the question is good, meaning it is comprehensible, unambiguous, and requesting factual information. This judgment is made before the annotator sees the Wikipedia page. Next, for good questions, annotators find a passage within the document that contains enough information to answer the question. Annotators can mark questions as “not answerable" if the Wikipedia article does not contain the requested information.

We do find a correlation between the nature of the question and the likelihood of a “yes" answer. However, this correlation is too weak to help outperform the majority baseline because, even if the topic or type is known, it is never best to guess the minority class. We also found that question-only models perform very poorly on this task (see Section SECREF12 ), which helps confirm that the questions do not contain sufficient information to predict the answer on their own.

Recurrent Model	Our recurrent model is a standard model from the text pair classification literature, similar to the one used in the GLUE baseline  and the model from  . Our model has the following stages: Embed: Embed the words using a character CNN following what was done by  , and the fasttext crawl word embeddings  . Then run a BiLSTM over the results to get context-aware word hypothesis embeddings INLINEFORM0 and premise embeddings INLINEFORM1 . Co-Attention: Compute a co-attention matrix, INLINEFORM0 , between the hypothesis and premise where INLINEFORM1 , INLINEFORM2 is elementwise multiplication, and INLINEFORM3 , INLINEFORM4 , and INLINEFORM5 are weights to be learned. Attend: For each row in INLINEFORM0 , apply the softmax operator and use the results to compute a weighed sum of the hypothesis embeddings, resulting in attended vectors INLINEFORM1 . We use the transpose of INLINEFORM2 to compute vectors INLINEFORM3 from the premise embeddings in a similar manner. Pool: Run another BiLSTM over INLINEFORM0 to get embeddings INLINEFORM1 .

Shallow Models	First, we experiment with using a linear classifier on our task. In general, we found features such as word overlap or TF-IDF statistics were not sufficient to achieve better than the majority-class baseline accuracy (62.17% on the dev set). We did find there was a correlation between the number of times question words occurred in the passage and the answer being “yes", but the correlation was not strong enough to build an effective classifier. “Yes" is the most common answer even among questions with zero shared words between the question and passage (with a 51% majority), and more common in other cases.

Neural Models	For our experiments that do not use unsupervised pre-training (except the use of pre-trained word vectors), we use a standard recurrent model with attention. Our experiments using unsupervised pre-training use the models provided by the authors. In more detail: Our Recurrent model follows a standard recurrent plus attention architecture for text-pair classification  . It embeds the premise/hypothesis text using fasttext word vectors  and learned character vectors, applies a shared bidirectional LSTM to both parts, applies co-attention  to share information between the two parts, applies another bi-LSTM to both parts, pools the result, and uses the pooled representation to predict the final class. See Appendix SECREF18 for details. Our Recurrent +ELMo model uses the language model from  to provide contextualized embeddings to the baseline model outlined above, as recommended by the authors. Our OpenAI GPT model fine-tunes the 12 layer 768 dimensional uni-directional transformer from  , which has been pre-trained as a language model on the Books corpus  .

Our BERTL model fine-tunes the 24 layer 1024 dimensional transformer from  , which has been trained on next-sentence-selection and masked language modelling on the Book Corpus and Wikipedia. We fine-tune the BERTL and the OpenAI GPT models using the optimizers recommended by the authors, but found it important to tune the optimization parameters to achieve the best results. We use a batch size of 24, learning rate of 1e-5, and 5 training epochs for BERT and a learning rate of 6.25e-5, batch size of 6, language model loss of 0.5, and 3 training epochs for OpenAI GPT.

Question/Passage Only Results	Following the recommendation of  , we first experiment with models that are only allowed to observe the question or the passage. The pre-trained BERTL model reached 64.48% dev set accuracy using just the question and 66.74% using just the passage. Given that the majority baseline is 62.17%, this suggests there is little signal in the question by itself, but that some language patterns in the passage correlate with the answer. Possibly, passages that present more straightforward factual information (like Wikipedia introduction paragraphs) correlate with “yes" answers.

Sample Efficiency	In Figure 2, we graph model accuracy as more of the training data is used for fine-tuning, both with and without initially pre-training on MultiNLI. Pre-training on MultiNLI gives at least a 5-6 point gain, and nearly a 10 point gain for BERTL when only using 1000 examples. For small numbers of examples, the recurrent model with MultiNLI pre-training actually out-performs BERTL.

Randomly Selected Examples	We include a number of randomly selected examples from the BoolQ train set in Figure FIGREF19 . For each example we show the question in bold, followed by the answer in parentheses, and then the passage below.

Introduction	Understanding what facts can be inferred to be true or false from text is an essential part of natural language understanding. In many cases, these inferences can go well beyond what is immediately stated in the text. For example, a simple sentence like “Hanna Huyskova won the gold medal for Belarus in freestyle skiing." implies that (1) Belarus is a country, (2) Hanna Huyskova is an athlete, (3) Belarus won at least one Olympic event, (4) the USA did not win the freestyle skiing event, and so on. Work completed while interning at Google. Also affiliated with Columbia University, work done at Google. To test a model's ability to make these kinds of inferences, previous work in natural language inference (NLI) proposed the task of labeling candidate statements as being entailed or contradicted by a given passage. However, in practice, generating candidate statements that test for complex inferential abilities is challenging. For instance, evidence suggests  ,  ,  that simply asking human annotators to write candidate statements will result in examples that typically only require surface-level reasoning.

Question: how was the dataset built?
Answer: Questions are gathered from anonymized, aggregated queries to the Google search engine. Queries that are likely to be yes/no questions are heuristically identified: we found selecting queries where the first word is in a manually constructed set of indicator words and are of sufficient length, to be effective. Questions are only kept if a Wikipedia page is returned as one of the first five results, in which case the question and Wikipedia page are given to a human annotator for further processing. Annotators label question/article pairs in a three-step process. First, they decide if the question is good, meaning it is comprehensible, unambiguous, and requesting factual information. This judgment is made before the annotator sees the Wikipedia page. Next, for good questions, annotators find a passage within the document that contains enough information to answer the question. Annotators can mark questions as “not answerable" if the Wikipedia article does not contain the requested information. Finally, annotators mark whether the question's answer is “yes" or “no"