Commonsense Reasoning with Implicit Knowledge in Natural Language

1 . Introduction . For an AI agent to reason about the everyday routine human activities , the agent needs to possess commonsense . Consequently , commonsense acquisition and reasoning are considered critical research challenges from the early days of AI [ McCarthy , 1959 ] . The need for commonsense reasoning is reemphasized recently [ Sap et al. , 2019a , Marcus and Davis , 2019 ] , particularly in NL understanding and QA . Several commonsense reasoning tasks have been proposed that study the different aspects of commonsense reasoning , such as abductive commonsense [ Bhagavatula et al. , 2019 ] , physical commonsense [ Bisk et al. , 2019 ] , and social commonsense [ Sap et al. , 2019b ] . QA systems approach solving tasks using large-pretrained transformers , such as BERT [ Devlin et al. , 2019 ] , or use complex knowledge fusion methods to perform QA [ Lin et al. , 2019 , Lv et al. , 2020 ] . In this paper , focusing on low resource use , we evaluate the use of smaller transformer language models and a small number of knowledge-rich natural language sentences , where relevant knowledge may be implicitly expressed . To understand what we mean by implicit knowledge , consider an example from [ Winograd , 1972 ] : Given the context “ The city councilmen refused the demonstrators a permit because they feared violence. ” , and the question “ Who is fearing violence ? ” , the correct answer is “ The city councilmen ” . An ∗ . Equal contribution unstructured retrieved ( through a web search engine ) knowledge [ Prakash et al. , 2019 ] for this context-question pair that can help answer this question correctly is : “ He also refused to give his full name because he feared for his safety. ” . We can use this knowledge to reason that the person who is refusing , is the one who is fearing . From this example , we can observe that the necessary commonsense knowledge to reason may be present in text in many cases but in an implicit way . Moreover , this knowledge is unstructured , and hence current state-of-the-art knowledge fusion methods are unable to utilize this knowledge without a method to represent it in a knowledge graph triple , as present in ConceptNet . Using natural language sentences ( as a source of knowledge ) at first glance appears similar to the application of evidence retrieval for open-domain question answering [ Yang et al. , 2018 , Clark et al. , 2018 , Kwiatkowski et al. , 2019 ] , where systems retrieve supporting evidence to be able to answer an open-ended question . However , there is a big difference as , unlike in evidence retrieval , the needed commonsense knowledge may not be explicitly available in unstructured knowledge corpora . Our approach is to reason-with-example , in contrast to reading comprehension with retrieved supporting paragraphs containing answers or explicit knowledge that lead to answers . Moreover , a high lexical overlap with a retrieved knowledge and context-question-answer does not mean it can be used to answer correctly . For example , another retrieved knowledge for the above question is : “ Demonstrators fear the retaliatory police violence. ” . An additional layer of complexity to commonsense reasoning with natural language is added because of such high lexical overlap but distracting sentences . We limit our study to two pre-trained transformers , namely BERT and RoBERTa . BERT and RoBERTa have been trained using 13GB and 160GB data , respectively . RoBERTa has the same architecture and parameter count but is trained with extensive hyper-parameter tuning and has a larger vocabulary ( 25K v/s 50K ) . These allow us to study the implicit commonsense reasoning ability with varying pre-training and vocabulary size . Larger pretrained transformers have been effectively shown to improve performance on downstream tasks , but training such models is resource-intensive . Hence we ask the following auxiliary question : To what extent can we improve a smaller transformer encoder ’ s performance ? Smaller in the sense of pre-training data , vocabulary size , and parameter tuning space . For addressing the above questions , we propose the following experimental framework . We categorize different unstructured knowledge sources and define a knowledge source preparation and retrieval component . We then propose three strategies of unstructured knowledge infusion . In the Revision strategy , we fine-tune the transformer on an unstructured knowledge source . In Openbook strategy , we choose a certain number of knowledge statements from the unstructured knowledge source that are textually similar to each of the dataset samples . Then we fine-tune the pre-trained transformer for the question-answering task . In the final strategy , we combine both the strategies mentioned above . We propose three strong baseline methods that utilize knowledge , concat , max , simple-sum , and an explainable reasoning model weighted-sum to combine and reason with multiple commonsense knowledge sentences . We evaluate our proposed framework on three public commonsense question answering datasets : AbductiveNLI ( aNLI ) [ Bhagavatula et al. , 2019 ] , PIQA [ Bisk et al. , 2019 ] and Social Interaction QA ( SIQA ) [ Sap et al. , 2019b ] . Our key findings are as follows : ( a ) Transformers can reason with implicit commonsense knowledge to some extent . We observe that transformers fail to answer questions through detailed error analysis even when sufficient knowledge is present with minimal distractors 30-50 % of the time . This observation shows the scope of future improvements . ( b ) Revision and Openbook Strategy improve commonsense reasoning performance , but the Revision strategy ’ s impact depends on how well-formed the unstructured knowledge corpus is . ( c ) Our knowledge retrieval and knowledge infusion methods improve accuracy over pre-trained transformers by 2-9 % . They are significantly effective over the smaller transformer encoders and approach larger pre-trained transformers , surpassing T5-11B [ Raffel et al. , 2019 ] by 4.14 % in aNLI and reducing the gap to 1.75 % in SIQA using RoBERTa . These methods should act as future baselines . In summary , our contributions are : ( a ) a thorough analysis of transformers ’ ability to perform commonsense reasoning with implicit knowledge on three different commonsense QA tasks using two transformer models . ( b ) four models representing four ways knowledge can be infused in transformer encoders . These methods apply to multiple commonsense reasoning tasks and improve performance over pre-trained transformers by 2-9 % in accuracy . ( c ) a detailed study to bridge the gap between smaller and larger pre-trained transformers . ( d ) an extensive investigation to study the impact of different knowledge sources and pre-training on such knowledge sources on commonsense QA tasks . 2 . MCQ Datasets . To study the extent of transformers ’ commonsense reasoning ability , we choose the following three datasets to evaluate our models , each with a different kind of commonsense knowledge . Figure 1 shows examples from each of the datasets with our retrieved commonsense knowledge sentences . Abductive NLI ( aNLI ) : This dataset [ Bhagavatula et al. , 2019 ] is intended to judge the potential of an AI system to do abductive reasoning to form possible explanations for a given set of observations . The task is to find which of the hypothesis options H1 , and H2 explains O2 where O1 should precede and O2 should succeed the hypothesis , given a pair of observations O1 and O2 . This task needs a commonsense understanding of which order sequence of events occurs . There are 169,654 train and 1,532 validation samples . The test set is blind . It has a generation task , but we restrict ourselves to the multiple-choice task . PIQA ( Physical Interaction QA ) : This dataset is created to evaluate an AI system ’ s physics reasoning capability . The dataset requires reasoning about physical objects and how we use them in our daily lives . The task is to predict the most appropriate choice to the goal G , given a goal G and a pair of choices C1 and C2 . There are 16,113 train and 1,838 validation samples . The test set is blind . SIQA : This dataset is a collection of instances about social interaction reasoning and the social implications of their statements . The task is to choose the correct answer option AOi out of three choices when given a context C of a social situation and a question Q about the situation . There are several question types in this dataset derived from Atomic inference dimensions [ Sap et al. , 2019a ] . A few of the Atomic inference dimensions are actor intention , actor motivation , effect on the actor , effect on others , etc . In total , there are 33,410 train and 1,954 validation samples . The test set is blind . 3 . Commonsense Knowledge Sources . 3.1 Knowledge Categorization for Evaluation . Directly Derived : Here the commonsense QA task is directly derived from the knowledge source , and hence using the same knowledge may make the task trivial . We test this scenario on the aNLI task with the following knowledge sources , ROCStories Corpus [ Mostafazadeh et al. , 2016 ] and Story Cloze Test , that were used in creating aNLI . Our motivation is to see how well the model can answer questions when given the “ same ” or similar implicit/explicit commonsense knowledge . Partially Derived : Here the commonsense QA task is not directly derived from an external knowledge source , and considerable human knowledge was used to generate the question-answers . In this case , we use SIQA as the task , which uses the Atomic [ Sap et al. , 2019a ] knowledge base as the source for social events , but has undergone sufficient human intervention to make the task non-trivial . During dataset creation , the human annotators were asked to turn Atomic events into sentences and were asked to create question-answers . Relevant : Here , the commonsense task is entirely created with human annotators ’ help without using a specific knowledge source . However , we guess knowledge sources that seem relevant through our QA pairs analysis . We evaluate this using PIQA as the commonsense task and WikiHow dataset [ Koupaee and Wang , 2018 ] as the “ relevant ” external knowledge source . 3.2 Knowledge Source Preparation . aNLI : To test our first category of external knowledge , we use the entire Story Cloze Test and ROCStories Corpus . We also prepare another source that contains knowledge sentences retrieved for the train set of aNLI from the first source . This knowledge source is created to ensure the task is not trivialized with knowledge leakage . We also create a knowledge source from multiple datasets such as MCTest [ Richardson et al. , 2013 ] , COPA [ Roemmele et al. , 2011 ] and Atomic , but not Story Cloze Test and ROCStories Corpus . These sources contain commonsense knowledge , which might be useful for the aNLI task . SIQA : We synthetically generate a knowledge source from the events and inference dimensions provided by the Atomic dataset [ Sap et al. , 2019a ] . The Atomic dataset contains events and eight types of if-then inferences ∗ . The total number of events is 732,723 . Some events are masked , which we fill by using a BERT and masked language modeling [ Devlin et al. , 2019 ] . We extend the knowledge source , and replace PersonX and PersonY , as present in ∗ . More details in Supplemental Materials . the original Atomic dataset , using gender-neutral names . These steps may approximate the steps taken by humans to generate QA pairs . PIQA : We use the Wikihow dataset for PIQA . It contains paragraphs ( 214,544 ) with detailed steps or actions to complete a task . We extract the title of each paragraph and split the paragraphs into sentences . The title is concatenated to each of the sentences . This preprocessing ensures that the task ’ s goal is present in each of the sentences . A Combined Commonsense Corpus is created which combines the partially related and relevant corpuses , for example , combining Wikihow , Atomic , MCTest .

Summary: This paper asks how to improve commonsense question answering by infusing or providing potentially useful knowledge to BERT-style models. Authors explore using finetuning on and/or retrieving of useful or related knowledge for questions, and experiment with multiple ways to combine the useful knowledge with the questions. Results show that simultaneously finetuning on related knowledge source and retrieving a few examples for each question improves performance the best.