Contextualized Scene Imagination for Generative Commonsense Reasoning

1 INTRODUCTION . Humans describe everyday scenes in natural language based on their understanding of common concepts encountered in their environment ( Tincoff & Jusczyk , 1999 ) . Analogously , the task of generative commonsense reasoning ( GCSR ) asks machines to generate a description of everyday situations based on a set of concepts and an initial context ( Liu et al. , 2020 ; Li et al. , 2021 ) . For example , given concept words { dog , frisbee , catch , throw } , a machine is expected to generate a plausible description , e.g. , “ A man throws a frisbee and his dog catches it in the air ” . Machines with GCSR skills would communicate fluidly with humans , e.g. , when summarizing a document by preserving its key details ( Sha , 2020 ) , composing a creative story according to a set of clues ( Yao et al. , 2019 ) , and generating a conversation reply that includes specified keywords ( Mou et al. , 2016 ) . GCSR poses three unique challenges for automatic text generation methods . To depict plausible scenes when composing sentences , machines require commonsense knowledge to reason about the relations between concepts and the affordances of objects ( e.g. , “ dog ” performs the action “ catch ” but not the action “ throw ” ) . Moreover , machines require a compositional generalization ability ( Keysers et al. , 2019 ) , i.e. , the ability to judge the plausibility of a new concept composition that has not been observed during training , and to identify concepts related to the scene that are not explicitly provided ( e.g. , “ person ” to perform “ throw ” in the above example ) . GCSR can be directly attempted by fine-tuning pre-trained text-to-text language models ( LMs ) ( Raffel et al. , 2019 ; Radford et al. , 2019 ) . While pre-trained LMs capture certain encyclopedic knowledge mentioned in text corpora ( e.g. , Wikipedia ) ( Petroni et al. , 2019 ) and can combine concepts in novel ways , they may generate grammatically fluent but implausible sentences that conflict with human common sense ( Lin et al. , 2020 ) . This is because LMs have no intrinsic mechanism to reason over high-level relations between concepts Zhou et al . ( 2020 ) . To close the knowledge gap , recent work augment LM input with knowledge graph triples ( e.g. , ( dog , CapableOf , catch ) ) retrieved 1Code and data used in our experiments can be found at https : //anonymous.4open.science/r/ ImagineAndVerbalize-EE98/ . from ConceptNet ( Liu et al. , 2020 ; Li et al. , 2020 ) , or prototype sentences that cover input concepts retrieved from external text corpora ( Fan et al. , 2020 ; Wang et al. , 2021 ) . However , despite the input augmentation , GCSR skills are implicitly learned based on the concept-text pairs in the training data , without explicit supervision . While some recent work propose content planning in story generation in the form of plots or scripts ( Yao et al. , 2019 ; Fan et al. , 2019 ) , only the narrative order of concepts are planned in those methods instead of their plausible roles and relations . Given the complexity of the GCSR task , machines need a direct mechanism to create a high-level relational representation of the provided concepts , which would allow them to judge the plausibility of their combination . In this paper , we propose to model an explicit scene imagination step which constructs a structured representation of a plausible scene based on input concepts and initial context . The scene imagination module formalizes the background knowledge required for the reasoning through a contextualized relational graph , called scene knowledge graph ( SKG ) . An SKG allows us to collect and harmonize diverse commonsense knowledge across resources and modalities into a comprehensive SKG distribution ( see Figure 1 for an illustration ) . We develop an imagine-and-verbalize framework : an imagination module learns to construct a contextualized SKG from input concepts and context by pretraining over a large amount of external SKGs ; a verbalization module learns to faithfully realize the imagined SKG into natural language by training over downstream datasets . By learning from a large number of diverse SKGs , our method is able to capture plausible relations between concepts . By integrating these SKGs with LMs , the imagination module is able to compose new objects in novel ways , and identify implicit concepts for a scene . Imagine-and-verbalize decomposes the challenging scene description task into two realistic tasks for which a wealth of training data can be collected , simultaneously enabling for effective and explainable GCSR . We experiment with two GCSR tasks and three scene graph resources , observing consistently better or competitive performance to SotA baselines . We find that ( 1 ) SKGs extracted from visual captions and story datasets are more helpful than other resources ; ( 2 ) our model can learn faster ( with less training data ) with the help of scene imagination ; and ( 3 ) the imagination module with a larger backbone LM demonstrates larger capacity in encoding commonsense knowledge . Our human evaluation study on the generated imagination indicates that these SKGs capture common sense and that the verbalization module generates the text by following the guidance of the imagination . 2 METHOD . Formally , in GCSR , we consider a list of concepts sets { x1 , x2 , ... , xK } and a textual context c ∈ C as input . Each concept set xi is unordered and consists of multiple concept words { xj } . A concept word xj ∈ X ( or concept for brevity ) is a commonly seen object ( nouns such as “ dog ” or “ frisbee ” ) or commonly performed action ( verbs such as “ throw ” or “ catch ” ) . The goal of GCSR is to generate K sentences { y1 , y2 , ... , yK } , each describing a plausible situation following human common sense for a concept set xi . The i-th sentence yi ⊂ Y should be generated using all concepts in xi . We consider two variants of GCSR : 1 ) concepts-to-sentence generation ( Lin et al. , 2020 ) , where no context is given ( i.e. , c is empty ) and only one concept set is provided ( K = 1 ) ; and 2 ) concepts-tostory generation task , where c is the leading sentence of a multi-sentence story and more than one concept sets are provided , each corresponding to one sentence to be generated ( K > 1 ) . Both tasks are evaluated by comparing the machine-generated text with human-generated ( gold ) references . 2.1 THE IMAGINE-AND-VERBALIZE APPROACH . Pre-trained LMs struggle with learning a generalizable mapping from concepts to plausible sentences solely based on the training data . Augmenting concepts with external knowledge to form the input X ′ and fine-tuning a pretrained LM to model P ( Y|C , X ′ ) ( Liu et al. , 2020 ; Fan et al. , 2020 ; Li et al. , 2021 ) alleviates this issue partially , while still learning a direct mapping of { C , X ′ } → Y . In this work ( Figure 1 ) , we decompose the GCSR task into two sub-tasks : contextualized scene generation ( imagination ) and scene-aware text generation ( verbalization ) . P ( Y|C , X ) = ∑ Z P ( Y|C , X , Z ) P ( Z|C , X ) , ( 1 ) where Z denotes the scene representation for the given concepts and context . The contextualized scene imagination module P ( Z|C , X ) aims to construct a multi-relational graph representation Z ( scene knowledge graph , or SKG ) that describes a plausible scene that involves all input concepts and corresponds to the provided context . To learn this module , we collect a diverse set of SKG instances from different resources and modalities to form a comprehensive distribution of scenes ( §2.2 ) . The imagination module is pre-trained over the collected scene instances and learns to generate SKGs depicting plausible day-to-day situation . The imagination module is based on a neural architecture , which enables it to generate concept compositions that might not have been observed during training ( §2.3 ) .2 We leverage the contextualized SKG for text generation with a verbalization module P ( Y|C , X , Z ) which takes the context , concepts , and the generated SKG as input , and composes a grammatical and plausible scene description in natural language ( §2.4 ) . To perform GCSR , where one or multiple concept sets are given , we apply the imagination module to sample zi . Since the marginalization over Z is generally intractable due to the complex structure of the SKGs , we only sample the most probable scene representation z∗i that maximizes P ( zi|c′ , xi ) , where c′ includes the given context c and the previously generated yj , ( j < i ) . We then apply the verbalization module to generate one sentence at a time by sampling from P ( yi|c′ , xi , zi∗ ) . Multiple sentences are generated by iteratively applying the imagination and verbalization modules . 2.2 IMAGINATION VIA GENERATING SKG . Imagination through SKGs We adopt the term “ scene graph ” from the computer vision community , and we generalize it to a novel relational schema that represents knowledge from multiple modalities . Our SKG is defined as a relational graph G = ( E , R ) that organizes a set of concepts in a coherent scene that follows common sense . The node set E of the graph includes both given and implicit concepts , while each relation ( edge type ) r ∈ R denotes how two concepts should be related . We follow the Abstract Meaning Representation ( AMR ) ( Banarescu et al. , 2013 ) schema to consider the core relations between two concepts , which corresponds to the commonsense knowledge required by GCSR . Table 7 in the appendix illustrates a few representative relations and their examples . Table 1 : Statistics of the SKG instances collected from different resources . Knowledge source # SKGs # Concepts Caption-AMR 584,252 22,961 Story-AMR 927,163 41,272 VG-SceneGraph 292,596 41,629 All 1,792,941 84,835 Collecting Diverse SKGs We consider two complementary modalities , text and vision , as some concepts and relationships are more likely to occur in one modality versus another . ( 1 ) Textual Modality : According to pragmatic principles of human language , people generally leave out expected details about common scenes ( Grice , 1975 ) . For this reason , we extract SKGs from visual captions and narrative stories , in which human annotators are asked to explicitly describe scenes that may happen using 2The imagination module can be further fine-tuned over the downstream datasets . descriptive language as shown in Figure 1 ( a , b ) . To extract an SKG out of these textual signals , we adopt the AMR parsing tool to transform each sentence into an AMR graph . This process yields a single SKG per sentence . For the story SKGs , we also keep the sentences ( up to 256 tokens ) that precede the sentence that corresponds to the SKG , as context c. ( 2 ) Visual Modality : Image captions focus on salient information and may not capture all useful visual signals . Thus , we also capture the scene structures directly from images , by using VisualGenome ( Krishna et al. , 2016 ) , a large-scale scene graph dataset annotated by humans . To adopt a unified SKG schema , we manually map the relations in scene graphs from VisualGenome to the ones used in textual SKGs . A full set of mapping rules can be found in the Appendix ( A.1 ) . The statistics of the SKGs collected from each resource/modality are summarized in Table 1 . We note that visual scene graphs may be biased towards knowledge about spatial relationships and object affordance , which further motivates our decision to extract SKGs from multiple modalities .

Summary: The paper introduces an Imagine and Verbalize two-step method for generative commonsense reasoning. The system first imagines a scene in the form of a linearized SKG and uses that as input for a second model that verbalizes it into (more) human readable text. The method is tested on the Concept2Sentence and Concept2Story tasks and compared to multiple baselines drawn from related works.