Commonsense Reasoning with Implicit Knowledge in Natural Language

1 . Introduction . For an AI agent to reason about the everyday routine human activities , the agent needs to possess commonsense . Consequently , commonsense acquisition and reasoning are considered critical research challenges from the early days of AI [ McCarthy , 1959 ] . The need for commonsense reasoning is reemphasized recently [ Sap et al. , 2019a , Marcus and Davis , 2019 ] , particularly in NL understanding and QA . Several commonsense reasoning tasks have been proposed that study the different aspects of commonsense reasoning , such as abductive commonsense [ Bhagavatula et al. , 2019 ] , physical commonsense [ Bisk et al. , 2019 ] , and social commonsense [ Sap et al. , 2019b ] . QA systems approach solving tasks using large-pretrained transformers , such as BERT [ Devlin et al. , 2019 ] , or use complex knowledge fusion methods to perform QA [ Lin et al. , 2019 , Lv et al. , 2020 ] . In this paper , focusing on low resource use , we evaluate the use of smaller transformer language models and a small number of knowledge-rich natural language sentences , where relevant knowledge may be implicitly expressed . To understand what we mean by implicit knowledge , consider an example from [ Winograd , 1972 ] : Given the context “ The city councilmen refused the demonstrators a permit because they feared violence. ” , and the question “ Who is fearing violence ? ” , the correct answer is “ The city councilmen ” . An ∗ . Equal contribution unstructured retrieved ( through a web search engine ) knowledge [ Prakash et al. , 2019 ] for this context-question pair that can help answer this question correctly is : “ He also refused to give his full name because he feared for his safety. ” . We can use this knowledge to reason that the person who is refusing , is the one who is fearing . From this example , we can observe that the necessary commonsense knowledge to reason may be present in text in many cases but in an implicit way . Moreover , this knowledge is unstructured , and hence current state-of-the-art knowledge fusion methods are unable to utilize this knowledge without a method to represent it in a knowledge graph triple , as present in ConceptNet . Using natural language sentences ( as a source of knowledge ) at first glance appears similar to the application of evidence retrieval for open-domain question answering [ Yang et al. , 2018 , Clark et al. , 2018 , Kwiatkowski et al. , 2019 ] , where systems retrieve supporting evidence to be able to answer an open-ended question . However , there is a big difference as , unlike in evidence retrieval , the needed commonsense knowledge may not be explicitly available in unstructured knowledge corpora . Our approach is to reason-with-example , in contrast to reading comprehension with retrieved supporting paragraphs containing answers or explicit knowledge that lead to answers . Moreover , a high lexical overlap with a retrieved knowledge and context-question-answer does not mean it can be used to answer correctly . For example , another retrieved knowledge for the above question is : “ Demonstrators fear the retaliatory police violence. ” . An additional layer of complexity to commonsense reasoning with natural language is added because of such high lexical overlap but distracting sentences . We limit our study to two pre-trained transformers , namely BERT and RoBERTa . BERT and RoBERTa have been trained using 13GB and 160GB data , respectively . RoBERTa has the same architecture and parameter count but is trained with extensive hyper-parameter tuning and has a larger vocabulary ( 25K v/s 50K ) . These allow us to study the implicit commonsense reasoning ability with varying pre-training and vocabulary size . Larger pretrained transformers have been effectively shown to improve performance on downstream tasks , but training such models is resource-intensive . Hence we ask the following auxiliary question : To what extent can we improve a smaller transformer encoder ’ s performance ? Smaller in the sense of pre-training data , vocabulary size , and parameter tuning space . For addressing the above questions , we propose the following experimental framework . We categorize different unstructured knowledge sources and define a knowledge source preparation and retrieval component . We then propose three strategies of unstructured knowledge infusion . In the Revision strategy , we fine-tune the transformer on an unstructured knowledge source . In Openbook strategy , we choose a certain number of knowledge statements from the unstructured knowledge source that are textually similar to each of the dataset samples . Then we fine-tune the pre-trained transformer for the question-answering task . In the final strategy , we combine both the strategies mentioned above . We propose three strong baseline methods that utilize knowledge , concat , max , simple-sum , and an explainable reasoning model weighted-sum to combine and reason with multiple commonsense knowledge sentences . We evaluate our proposed framework on three public commonsense question answering datasets : AbductiveNLI ( aNLI ) [ Bhagavatula et al. , 2019 ] , PIQA [ Bisk et al. , 2019 ] and Social Interaction QA ( SIQA ) [ Sap et al. , 2019b ] . Our key findings are as follows : ( a ) Transformers can reason with implicit commonsense knowledge to some extent . We observe that transformers fail to answer questions through detailed error analysis even when sufficient knowledge is present with minimal distractors 30-50 % of the time . This observation shows the scope of future improvements . ( b ) Revision and Openbook Strategy improve commonsense reasoning performance , but the Revision strategy ’ s impact depends on how well-formed the unstructured knowledge corpus is . ( c ) Our knowledge retrieval and knowledge infusion methods improve accuracy over pre-trained transformers by 2-9 % . They are significantly effective over the smaller transformer encoders and approach larger pre-trained transformers , surpassing T5-11B [ Raffel et al. , 2019 ] by 4.14 % in aNLI and reducing the gap to 1.75 % in SIQA using RoBERTa . These methods should act as future baselines . In summary , our contributions are : ( a ) a thorough analysis of transformers ’ ability to perform commonsense reasoning with implicit knowledge on three different commonsense QA tasks using two transformer models . ( b ) four models representing four ways knowledge can be infused in transformer encoders . These methods apply to multiple commonsense reasoning tasks and improve performance over pre-trained transformers by 2-9 % in accuracy . ( c ) a detailed study to bridge the gap between smaller and larger pre-trained transformers . ( d ) an extensive investigation to study the impact of different knowledge sources and pre-training on such knowledge sources on commonsense QA tasks . 2 . MCQ Datasets . To study the extent of transformers ’ commonsense reasoning ability , we choose the following three datasets to evaluate our models , each with a different kind of commonsense knowledge . Figure 1 shows examples from each of the datasets with our retrieved commonsense knowledge sentences . Abductive NLI ( aNLI ) : This dataset [ Bhagavatula et al. , 2019 ] is intended to judge the potential of an AI system to do abductive reasoning to form possible explanations for a given set of observations . The task is to find which of the hypothesis options H1 , and H2 explains O2 where O1 should precede and O2 should succeed the hypothesis , given a pair of observations O1 and O2 . This task needs a commonsense understanding of which order sequence of events occurs . There are 169,654 train and 1,532 validation samples . The test set is blind . It has a generation task , but we restrict ourselves to the multiple-choice task . PIQA ( Physical Interaction QA ) : This dataset is created to evaluate an AI system ’ s physics reasoning capability . The dataset requires reasoning about physical objects and how we use them in our daily lives . The task is to predict the most appropriate choice to the goal G , given a goal G and a pair of choices C1 and C2 . There are 16,113 train and 1,838 validation samples . The test set is blind . SIQA : This dataset is a collection of instances about social interaction reasoning and the social implications of their statements . The task is to choose the correct answer option AOi out of three choices when given a context C of a social situation and a question Q about the situation . There are several question types in this dataset derived from Atomic inference dimensions [ Sap et al. , 2019a ] . A few of the Atomic inference dimensions are actor intention , actor motivation , effect on the actor , effect on others , etc . In total , there are 33,410 train and 1,954 validation samples . The test set is blind . 3 . Commonsense Knowledge Sources . 3.1 Knowledge Categorization for Evaluation . Directly Derived : Here the commonsense QA task is directly derived from the knowledge source , and hence using the same knowledge may make the task trivial . We test this scenario on the aNLI task with the following knowledge sources , ROCStories Corpus [ Mostafazadeh et al. , 2016 ] and Story Cloze Test , that were used in creating aNLI . Our motivation is to see how well the model can answer questions when given the “ same ” or similar implicit/explicit commonsense knowledge . Partially Derived : Here the commonsense QA task is not directly derived from an external knowledge source , and considerable human knowledge was used to generate the question-answers . In this case , we use SIQA as the task , which uses the Atomic [ Sap et al. , 2019a ] knowledge base as the source for social events , but has undergone sufficient human intervention to make the task non-trivial . During dataset creation , the human annotators were asked to turn Atomic events into sentences and were asked to create question-answers . Relevant : Here , the commonsense task is entirely created with human annotators ’ help without using a specific knowledge source . However , we guess knowledge sources that seem relevant through our QA pairs analysis . We evaluate this using PIQA as the commonsense task and WikiHow dataset [ Koupaee and Wang , 2018 ] as the “ relevant ” external knowledge source . 3.2 Knowledge Source Preparation . aNLI : To test our first category of external knowledge , we use the entire Story Cloze Test and ROCStories Corpus . We also prepare another source that contains knowledge sentences retrieved for the train set of aNLI from the first source . This knowledge source is created to ensure the task is not trivialized with knowledge leakage . We also create a knowledge source from multiple datasets such as MCTest [ Richardson et al. , 2013 ] , COPA [ Roemmele et al. , 2011 ] and Atomic , but not Story Cloze Test and ROCStories Corpus . These sources contain commonsense knowledge , which might be useful for the aNLI task . SIQA : We synthetically generate a knowledge source from the events and inference dimensions provided by the Atomic dataset [ Sap et al. , 2019a ] . The Atomic dataset contains events and eight types of if-then inferences ∗ . The total number of events is 732,723 . Some events are masked , which we fill by using a BERT and masked language modeling [ Devlin et al. , 2019 ] . We extend the knowledge source , and replace PersonX and PersonY , as present in ∗ . More details in Supplemental Materials . the original Atomic dataset , using gender-neutral names . These steps may approximate the steps taken by humans to generate QA pairs . PIQA : We use the Wikihow dataset for PIQA . It contains paragraphs ( 214,544 ) with detailed steps or actions to complete a task . We extract the title of each paragraph and split the paragraphs into sentences . The title is concatenated to each of the sentences . This preprocessing ensures that the task ’ s goal is present in each of the sentences . A Combined Commonsense Corpus is created which combines the partially related and relevant corpuses , for example , combining Wikihow , Atomic , MCTest .

Summary: The paper presents a detailed study on the commonsense reasoning ability of pre-trained Transformer-based LMs such as BERT and RoBERTa. The authors build an experimental framework for evaluating different models with different knowledge sources under different infusion strategies, e.g., continually pre-training (revision) or retrieval-based learning (openbook), or both. The authors present a few methods that can improve the results and reduce the gap between smaller LMs and larger ones.