Introduction
Natural language based question answering (NLQA) not only involves linguistic understanding, but often involves reasoning with various kinds of knowledge. In recent years, many NLQA datasets and challenges have been proposed, for example, SQuAD BIBREF0 , TriviaQA BIBREF1 and MultiRC BIBREF2 , and each of them have their own focus, sometimes by design and other times by virtue of their development methodology. Many of these datasets and challenges try to mimic human question answering settings. One such setting is open book question answering where humans are asked to answer questions in a setup where they can refer to books and other materials related to their questions. In such a setting, the focus is not on memorization but, as mentioned in BIBREF3 , on “deeper understanding of the materials and its application to new situations BIBREF4 , BIBREF5 .” In BIBREF3 , they propose the OpenBookQA dataset mimicking this setting.
The OpenBookQA dataset has a collection of questions and four answer choices for each question. The dataset comes with 1326 facts representing an open book. It is expected that answering each question requires at least one of these facts. In addition it requires common knowledge. To obtain relevant common knowledge we use an IR system BIBREF6 front end to a set of knowledge rich sentences. Compared to reading comprehension based QA (RCQA) setup where the answers to a question is usually found in the given small paragraph, in the OpenBookQA setup the open book part is much larger (than a small paragraph) and is not complete as additional common knowledge may be required. This leads to multiple challenges. First, finding the relevant facts in an open book (which is much bigger than the small paragraphs in the RCQA setting) is a challenge. Then, finding the relevant common knowledge using the IR front end is an even bigger challenge, especially since standard IR approaches can be misled by distractions. For example, Table 1 shows a sample question from the OpenBookQA dataset. We can see the retrieved missing knowledge contains words which overlap with both answer options A and B. Introduction of such knowledge sentences increases confusion for the question answering model. Finally, reasoning involving both facts from open book, and common knowledge leads to multi-hop reasoning with respect to natural language text, which is also a challenge.
We address the first two challenges and make the following contributions in this paper: (a) We improve on knowledge extraction from the OpenBook present in the dataset. We use semantic textual similarity models that are trained with different datasets for this task; (b) We propose natural language abduction to generate queries for retrieving missing knowledge; (c) We show how to use Information Gain based Re-ranking to reduce distractions and remove redundant information; (d) We provide an analysis of the dataset and the limitations of BERT Large model for such a question answering task.
The current best model on the leaderboard of OpenBookQA is the BERT Large model BIBREF7 . It has an accuracy of 60.4% and does not use external knowledge. Our knowledge selection and retrieval techniques achieves an accuracy of 72%, with a margin of 11.6% on the current state of the art. We study how the accuracy of the BERT Large model varies with varying number of knowledge facts extracted from the OpenBook and through IR.
Related Work
In recent years, several datasets have been proposed for natural language question answering BIBREF0 , BIBREF1 , BIBREF2 , BIBREF8 , BIBREF9 , BIBREF10 , BIBREF11 , BIBREF12 , BIBREF13 and many attempts have been made to solve these challenges BIBREF7 , BIBREF14 , BIBREF15 .
Among these, the closest to our work is the work in BIBREF7 which perform QA using fine tuned language model and the works of BIBREF16 , BIBREF17 which performs QA using external knowledge.
Related to our work for extracting missing knowledge are the works of BIBREF18 , BIBREF19 , BIBREF20 which respectively generate a query either by extracting key terms from a question and an answer option or by classifying key terms or by Seq2Seq models to generate key terms. In comparison, we generate queries using the question, an answer option and an extracted fact using natural language abduction.
The task of natural language abduction for natural language understanding has been studied for a long time BIBREF21 , BIBREF22 , BIBREF23 , BIBREF24 , BIBREF25 , BIBREF26 , BIBREF27 , BIBREF28 . However, such works transform the natural language text to a logical form and then use formal reasoning to perform the abduction. On the contrary, our system performs abduction over natural language text without translating the texts to a logical form.
Approach
Our approach involves six main modules: Hypothesis Generation, OpenBook Knowledge Extraction, Abductive Information Retrieval, Information Gain based Re-ranking, Passage Selection and Question Answering. A key aspect of our approach is to accurately hunt the needed knowledge facts from the OpenBook knowledge corpus and hunt missing common knowledge using IR. We explain our approach in the example given in Table 2 .
In Hypothesis Generation, our system generates a hypothesis $\mathbf {H_{ij}}$ for the $i$ th question and $j$ th answer option, where $j \in \lbrace 1,2,3,4\rbrace  $ . In OpenBook Knowledge Extraction, our system retrieves appropriate knowledge $\mathbf {F_{ij}}$ for a given hypothesis $\mathbf {H_{ij}}$ using semantic textual similarity, from the OpenBook knowledge corpus $\mathbf {F}$ . In Abductive Information Retrieval, our system abduces missing knowledge from $\mathbf {H_{ij}}$ and $\mathbf {F_{ij}}$ . The system formulates queries to perform IR to retrieve missing knowledge $\mathbf {K_{ij}}$ . With the retrieved $i$0 , $i$1 , Information Gain based Re-ranking and Passage Selection our system creates a knowledge passage $i$2 . In Question Answering, our system uses $i$3 to answer the questions using a BERT Large based MCQ model, similar to its use in solving SWAG BIBREF29 .
Hypothesis Generation
Our system creates a hypothesis for each of the questions and candidate answer options as part of the data preparation phase as shown in the example in Table 2 . The questions in the OpenBookQA dataset are either with wh word or are incomplete statements. To create hypothesis statements for questions with wh words, we use the rule-based model of BIBREF30 . For the rest of the questions, we concatenate the questions with each of the answers to produce the four hypotheses. This has been done for all the training, test and validation sets.
OpenBook Knowledge Extraction
To retrieve a small set of relevant knowledge facts from the knowledge corpus $\mathbf {F}$ , a textual similarity model is trained in a supervised fashion on two different datasets and the results are compared. We use the large-cased BERT BIBREF7 (BERT Large) as the textual similarity model.
We train it on the semantic textual similarity (STS-B) data from the GLUE dataset BIBREF31 . The trained model is then used to retrieve the top ten knowledge facts from corpus $\mathbf {F}$ based on the STS-B scores. The STS-B scores range from 0 to 5.0, with 0 being least similar.
We generate the dataset using the gold OpenBookQA facts from $\mathbf {F}$ for the train and validation set provided. To prepare the train set, we first find the similarity of the OpenBook $\mathbf {F}$ facts with respect to each other using the BERT model trained on STS-B dataset. We assign a score 5.0 for the gold $\mathbf {\hat{F_i}}$ fact for a hypothesis. We then sample different facts from the OpenBook and assign the STS-B similarity scores between the sampled fact and the gold fact $\mathbf {\mathbf {\hat{F}_{i}}}$ as the target score for that fact $\mathbf {F_{ij}}$ and $\mathbf {H_{ij}}$ . For example:
Hypothesis : Frilled sharks and angler fish live far beneath the surface of the ocean, which is why they are known as Deep sea animals. Gold Fact : deep sea animals live deep in the ocean : Score : 5.0 Sampled Facts : coral lives in the ocean : Score : 3.4 a fish lives in water : Score : 2.8
We do this to ensure a balanced target score is present for each hypothesis and fact. We use this trained model to retrieve top ten relevant facts for each $\mathbf {H_{ij}}$ from the knowledge corpus $\mathbf {F}$ .
Question: .. they decide the best way to save money is ? (A) to quit eating lunch out (B) to make more phone calls (C) to buy less with monopoly money (D) to have lunch with friends Knowledge extraction trained with STS-B: using less resources usually causes money to be saved a disperser disperses each season occurs once per year Knowledge extraction trained with OpenBookQA: using less resources usually causes money to be saved decreasing something negative has a positive impact on a thing conserving resources has a positive impact on the environment
Table 3 shows a comparative study of our three approaches for OpenBook knowledge extraction. We show, the number of correct OpenBook knowledge extracted for all of the four answer options using the three approaches TF-IDF, BERT model trained on STS-B data and BERT model Trained on OpenBook data. Apart from that, we also show the count of the number of facts present precisely across the correct answer options. It can be seen that the Precision@N for the BERT model trained on OpenBook data is better than the other models as N increases.
The above example presents the facts retrieved from BERT model trained on OpenBook which are more relevant than the facts retrieved from BERT model trained on STS-B. Both the models were able to find the most relevant fact, but the other facts for STS-B model introduce more distractors and have lesser relevance. The impact of this is visible from the accuracy scores for the QA task in Table 3 . The best performance of the BERT QA model can be seen to be 66.2% using only OpenBook facts.
Natural Language Abduction and IR
To search for the missing knowledge, we need to know what we are missing. We use “abduction” to figure that out. Abduction is a long studied task in AI, where normally, both the observation (hypothesis) and the domain knowledge (known fact) is represented in a formal language from which a logical solver abduces possible explanations (missing knowledge). However, in our case, both the observation and the domain knowledge are given as natural language sentences from which we want to find out a possible missing knowledge, which we will then hunt using IR. For example, one of the hypothesis $\mathbf {H_{ij}}$ is “A red-tailed hawk is searching for prey. It is most likely to swoop down on a gecko.”, and for which the known fact $\mathbf {F_{ij}}$ is “hawks eats lizards”. From this we expect the output of the natural language abduction system to be $\mathbf {K_{ij}}$ or “gecko is a lizard”. We will refer to this as “natural language abduction”.
For natural language abduction, we propose three models, compare them against a baseline model and evaluate each on a downstream question answering task. All the models ignore stop words except the Seq2Seq model. We describe the three models and a baseline model in the subsequent subsections.
We design a simple heuristic based model defined as below: $ K_{ij} = (H_{ij} \cup F_{ij}) \setminus (H_{ij} \cap F_{ij}) \quad \forall j \in \lbrace 1,2,3,4\rbrace  $
where $i$ is the $i$ th question, $j$ is the $j$ th option, $H_{ij}$ , $F_{ij}$ , $K_{ij}$ represents set of unique words of each instance of hypothesis, facts retrieved from knowledge corpus $\mathbf {F}$ and abduced missing knowledge of validation and test data respectively.
In the Supervised Bag of Words model, we select words which satisfy the following condition: $ P(w_n \in K_{ij}) > \theta $
where $w_n \in \lbrace H_{ij} \cup F_{ij}\rbrace $ . To elaborate, we learn the probability of a given word $w_n$ from the set of words in $H_{ij} \cup F_{ij}$ belonging to the abduced missing knowledge $K_{ij}$ . We select those words which are above the threshold $\theta $ .
To learn this probability, we create a training and validation dataset where the words similar (cosine similarity using spaCy) BIBREF32 to the words in the gold missing knowledge $\hat{K}_i$ (provided in the dataset) are labelled as positive class and all the other words not present in $\hat{K}_i$ but in $H_{ij} \cup F_{ij}$ are labelled as negative class. Both classes are ensured to be balanced. Finally, we train a binary classifier using BERT Large with one additional feed forward network for classification. We define value for the threshold $\theta $ using the accuracy of the classifier on validation set. $0.4$ was selected as the threshold.
In the final approach, we used the copynet sequence to sequence model BIBREF33 to generate, instead of predict, the missing knowledge given, the hypothesis $\mathbf {H}$ and knowledge fact from the corpus $\mathbf {F}$ . The intuition behind using copynet model is to make use of the copy mechanism to generate essential yet precise (minimizing distractors) information which can help in answering the question. We generate the training and validation dataset using the gold $\mathbf {\hat{K}_i}$ as the target sentence, but we replace out-of-vocabulary words from the target with words similar (cosine similarity using spaCy) BIBREF32 to the words present in $H_{ij} \cup F_{ij}$ . Here, however, we did not remove the stopwords. We choose one, out of multiple generated knowledge based on our model which provided maximum overlap_score, given by $ overlap\_score = \frac{\sum _{i}{count ((\hat{H}_{i} \cup F_{i})\cap K_{i})}}{\sum _{i}{count(\hat{K_{i}})}} $
where $i$ is the $i$ th question, $\hat{H}_{i}$ being the set of unique words of correct hypothesis, $F_{i}$ being the set of unique words from retrieved facts from knowledge corpus $\mathbf {F}$ , $K_{i}$ being the set of unique words of predicted missing knowledge and $\hat{K_i}$ being the set of unique words of the gold missing knowledge .
To see if abduction helps, we compare the above models with a Word Union Model. To extract the candidate words for missing knowledge, we used the set of unique words from both the hypothesis and OpenBook knowledge as candidate keywords. The model can be formally represented with the following: $ K_{ij} = (H_{ij} \cup F_{ij}) \quad \forall j \in \lbrace 1,2,3,4\rbrace  $
Information Gain based Re-ranking
In our experiments we observe that, BERT QA model gives a higher score if similar sentences are repeated, leading to wrong classification. Thus, we introduce Information Gain based Re-ranking to remove redundant information.
We use the same BERT Knowledge Extraction model Trained on OpenBookQA data (section "Acknowledgement" ), which is used for extraction of knowledge facts from corpus $\mathbf {F}$ to do an initial ranking of the retrieved missing knowledge $\mathbf {K}$ . The scores of this knowledge extraction model is used as relevancy score, $rel$ . To extract the top ten missing knowledge $\mathbf {K}$ , we define a redundancy score, $red_{ij}$ , as the maximum cosine similarity, $sim$ , between the previously selected missing knowledge, in the previous iterations till $i$ , and the candidate missing knowledge $K_j$ . If the last selected missing knowledge is $K_i$ , then $ red_{ij}(K_j) = max(red_{i-1,j}(K_j), sim(K_i,K_j)) $ $\mathbf {K}$0
For missing knowledge selection, we first take the missing knowledge with the highest $rel$ score. From the subsequent iteration, we compute the redundancy score with the last selected missing knowledge for each of the candidates and then rank them using the updated $rank\_score$ . We select the top ten missing knowledge for each $\mathbf {H_{ij}}$ .
Question Answering
Once the OpenBook knowledge facts $\mathbf {F}$ and missing knowledge $\mathbf {K}$ have been extracted, we move onto the task of answering the questions.
We use BERT Large model for the question answering task. For each question, we create a passage using the extracted facts and missing knowledge and fine-tune the BERT Large model for the QA task with one additional feed-forward layer for classification. The passages for the train dataset were prepared using the knowledge corpus facts, $\mathbf {F}$ . We create a passage using the top N facts, similar to the actual gold fact $\mathbf {\hat{F}_i}$ , for the train set. The similarities were scored using the STS-B trained model (section "Conclusion" ). The passages for the training dataset do not use the gold missing knowledge $\mathbf {\hat{K}_i}$ provided in the dataset. For each of our experiments, we use the same trained model, with passages from different IR models.
The BERT Large model limits passage length to be lesser than equal to 512. This restricts the size of the passage. To be within the restrictions we create a passage for each of the answer options, and score for all answer options against each passage. We refer to this scoring as sum score, defined as follows:
For each answer options, $A_j$ , we create a passage $P_j$ and score against each of the answer options $A_i$ . To compute the final score for the answer, we sum up each individual scores. If $Q$ is the question, the score for the answer is defined as $ Pr(Q,A_i) = \sum _{j=1}^{4}score(P_j,Q,A_i) $
where $score$ is the classification score given by the BERT Large model. The final answer is chosen based on, $ A = \operatornamewithlimits{arg\,max}_A Pr(Q,A_i) $
In the first round, we score each of the answer options using a passage created from the selected knowledge facts from corpus $\mathbf {F}$ . For each question, we ignore the passages of the answer options which are in the bottom two. We refer to this as Passage Selection. In the second round, we score for only those passages which are selected after adding the missing knowledge $\mathbf {K}$ .
We assume that the correct answer has the highest score in each round. Therefore we multiply the scores obtained after both rounds. We refer to this as Weighted Scoring. We define the combined passage selected scores and weighted scores as follows : $ Pr(\mathbf {F},Q,A_i) = \sum _{j=1}^{4}{score(P_j,Q,A_i)} $
where $P_j$ is the passage created from extracted OpenBook knowledge, F. The top two passages were selected based on the scores of $Pr(\mathbf {F},Q,A_i)$ . $ Pr(\mathbf {F}\cup \mathbf {K},Q,A_i) = \sum _{k=1}^{4}{\delta * score(P_k,Q,A_i)} $
where $\delta =1$ for the top two scores and $\delta =0$ for the rest. $P_k$ is the passage created using both the facts and missing knowledge. The final weighted score is : $ wPr(Q,A_i) = Pr(\mathbf {F},Q,A_i) * Pr(\mathbf {F} \cup \mathbf {K},Q,A_i) $
The answer is chosen based on the top weighted scores as below: $ A = \operatornamewithlimits{arg\,max}_A wPr(Q,A_i) $
Table 4 shows the incremental improvement on the baselines after inclusion of carefully selected knowledge.
Passage Selection and Weighted Scoring are used to overcome the challenge of boosted prediction scores due to cascading effect of errors in each stage.
Question: What eat plants? (A) leopards (B) eagles (C) owls (D) robin Appropriate extracted Fact from $\mathbf {F}$ : some birds eat plants Wrong Extracted Fact from $\mathbf {F}$ : a salamander eats insects Wrong Retrieved Missing Knowledge: Leopard geckos eat mostly insects
For the example shown above, the wrong answer leopards had very low score with only the facts extracted from knowledge corpus $\mathbf {F}$ . But introduction of missing knowledge from the wrong fact from $\mathbf {F}$ boosts its scores, leading to wrong prediction. Passage selection helps in removal of such options and Weighted Scoring gives preference to those answer options whose scores are relatively high before and after inclusion of missing knowledge.
No Passage Selection and Weighted Scoring.
Dataset and Experimental Setup
The dataset of OpenBookQA contains 4957 questions in the train set and 500 multiple choice questions in validation and test respectively. We train a BERT Large based QA model using the top ten knowledge facts from the corpus $\mathbf {F}$ , as a passage for both training and validation set. We select the model which gives the best score for the validation set. The same model is used to score the validation and test set with different passages derived from different methods of Abductive IR. The best Abductive IR model, the number of facts from $\mathbf {F}$ and $\mathbf {K}$ are selected from the best validation scores for the QA task.
Abductive Information Retrieval
We evaluate the abductive IR techniques at different values for number of facts from $\mathbf {F}$ and number of missing knowledge $\mathbf {K}$ extracted using IR. Figure 2 shows the accuracy against different combinations of $\mathbf {F}$ and $\mathbf {K}$ , for all four techniques of IR prior to Information gain based Re-ranking. In general, we noticed that the trained models performed poorly compared to the baselines. The Word Symmetric Difference model performs better, indicating abductive IR helps. The poor performance of the trained models can be attributed to the challenge of learning abductive inference.
For the above example it can be seen, the pre-reranking facts are relevant to the question but contribute very less considering the knowledge facts retrieved from the corpus $\mathbf {F}$ and the correct answer. Figure 3 shows the impact of Information gain based Re-ranking. Removal of redundant data allows the scope of more relevant information being present in the Top N retrieved missing knowledge $\mathbf {K}$ .
Question: A red-tailed hawk is searching for prey. It is most likely to swoop down on what? (A) an eagle (B) a cow (C) a gecko (D) a deer Fact from $\mathbf {F}$ : hawks eats lizards Pre-Reranking $\mathbf {K}$ : red-tail hawk in their search for prey Red-tailed hawks soar over the prairie and woodlands in search of prey. Post-Reranking $\mathbf {K}$ : Geckos - only vocal lizards. Every gecko is a lizard.
Model Analysis
BERT Question Answering model: BERT performs well on this task, but is prone to distractions. Repetition of information leads to boosted prediction scores. BERT performs well for lookup based QA, as in RCQA tasks like SQuAD. But this poses a challenge for Open Domain QA, as the extracted knowledge enables lookup for all answer options, leading to an adversarial setting for lookup based QA. This model is able to find the correct answer, even under the adversarial setting, which is shown by the performance of the sum score to select the answer after passage selection.
Symmetric Difference Model This model improves on the baseline Word Union model by 1-2%. The improvement is dwarfed because of inappropriate domain knowledge from $\mathbf {F}$ being used for abduction. The intersection between the inappropriate domain knowledge and the answer hypothesis is $\mathbf {\varnothing }$ , which leads to queries which are exactly same as the Word Union model.
Supervised learned models The supervised learned models for abduction under-perform. The Bag of Words and the Seq2Seq models fail to extract keywords for many $\mathbf {F}-\mathbf {H}$ pairs, sometimes missing the keywords from the answers. The Seq2Seq model sometimes extracts the exact missing knowledge, for example it generates “some birds is robin” or “lizard is gecko”. This shows there is promise in this approach and the poor performance can be attributed to insufficient train data size, which was 4957 only. A fact verification model might improve the accuracy of the supervised learned models. But, for many questions, it fails to extract proper keywords, copying just a part of the question or the knowledge fact.
Error Analysis
Other than errors due to distractions and failed IR, which were around 85% of the total errors, the errors seen are of four broad categories.
Temporal Reasoning: In the example shown below, even though both the options can be considered as night, the fact that 2:00 AM is more suitable for the bats than 6:00 PM makes it difficult to reason. Such issues accounted for 5% of the errors.
Question: Owls are likely to hunt at? (A) 3:00 PM (B) 2:00 AM (C) 6:00 PM (D) 7:00 AM
Negation: In the example shown below, a model is needed which handles negations specifically to reject incorrect options. Such issues accounted for 1% of the errors.
Question: Which of the following is not an input in photosynthesis? (A) sunlight (B) oxygen (C) water (D) carbon dioxide
Conjunctive Reasoning: In the example as shown below, each answer options are partially correct as the word “ bear” is present. Thus a model has to learn whether all parts of the answer are true or not, i.e Conjunctive Reasoning. Logically, all answers are correct, as we can see an “or”, but option (A) makes more sense. Such issues accounted for 1% of the errors.
Question: Some berries may be eaten by (A) a bear or person (B) a bear or shark (C) a bear or lion (D) a bear or wolf
Qualitative Reasoning: In the example shown below, each answer options would stop a car but option (D) is more suitable since it will stop the car quicker. A deeper qualitative reasoning is needed to reject incorrect options. Such issues accounted for 8% of the errors.
Question: Which of these would stop a car quicker? (A) a wheel with wet brake pads (B) a wheel without brake pads (C) a wheel with worn brake pads (D) a wheel with dry brake pads
Conclusion
In this work, we have pushed the current state of the art for the OpenBookQA task using simple techniques and careful selection of knowledge. We have provided two new ways of performing knowledge extraction over a knowledge base for QA and evaluated three ways to perform abductive inference over natural language. All techniques are shown to improve on the performance of the final task of QA, but there is still a long way to reach human performance.
We analyzed the performance of various components of our QA system. For the natural language abduction, the heuristic technique performs better than the supervised techniques. Our analysis also shows the limitations of BERT based MCQ models, the challenge of learning natural language abductive inference and the multiple types of reasoning required for an OpenBookQA task. Nevertheless, our overall system improves on the state of the art by 11.6%.
Acknowledgement
We thank NSF for the grant 1816039 and DARPA for partially supporting this research.

Question: How is OpenBookQA different from other natural language QA?
Answer: in the OpenBookQA setup the open book part is much larger, the open book part is much larger (than a small paragraph) and is not complete as additional common knowledge may be required