Contextualized Scene Imagination for Generative Commonsense Reasoning

1 INTRODUCTION . Humans describe everyday scenes in natural language based on their understanding of common concepts encountered in their environment ( Tincoff & Jusczyk , 1999 ) . Analogously , the task of generative commonsense reasoning ( GCSR ) asks machines to generate a description of everyday situations based on a set of concepts and an initial context ( Liu et al. , 2020 ; Li et al. , 2021 ) . For example , given concept words { dog , frisbee , catch , throw } , a machine is expected to generate a plausible description , e.g. , “ A man throws a frisbee and his dog catches it in the air ” . Machines with GCSR skills would communicate fluidly with humans , e.g. , when summarizing a document by preserving its key details ( Sha , 2020 ) , composing a creative story according to a set of clues ( Yao et al. , 2019 ) , and generating a conversation reply that includes specified keywords ( Mou et al. , 2016 ) . GCSR poses three unique challenges for automatic text generation methods . To depict plausible scenes when composing sentences , machines require commonsense knowledge to reason about the relations between concepts and the affordances of objects ( e.g. , “ dog ” performs the action “ catch ” but not the action “ throw ” ) . Moreover , machines require a compositional generalization ability ( Keysers et al. , 2019 ) , i.e. , the ability to judge the plausibility of a new concept composition that has not been observed during training , and to identify concepts related to the scene that are not explicitly provided ( e.g. , “ person ” to perform “ throw ” in the above example ) . GCSR can be directly attempted by fine-tuning pre-trained text-to-text language models ( LMs ) ( Raffel et al. , 2019 ; Radford et al. , 2019 ) . While pre-trained LMs capture certain encyclopedic knowledge mentioned in text corpora ( e.g. , Wikipedia ) ( Petroni et al. , 2019 ) and can combine concepts in novel ways , they may generate grammatically fluent but implausible sentences that conflict with human common sense ( Lin et al. , 2020 ) . This is because LMs have no intrinsic mechanism to reason over high-level relations between concepts Zhou et al . ( 2020 ) . To close the knowledge gap , recent work augment LM input with knowledge graph triples ( e.g. , ( dog , CapableOf , catch ) ) retrieved 1Code and data used in our experiments can be found at https : //anonymous.4open.science/r/ ImagineAndVerbalize-EE98/ . from ConceptNet ( Liu et al. , 2020 ; Li et al. , 2020 ) , or prototype sentences that cover input concepts retrieved from external text corpora ( Fan et al. , 2020 ; Wang et al. , 2021 ) . However , despite the input augmentation , GCSR skills are implicitly learned based on the concept-text pairs in the training data , without explicit supervision . While some recent work propose content planning in story generation in the form of plots or scripts ( Yao et al. , 2019 ; Fan et al. , 2019 ) , only the narrative order of concepts are planned in those methods instead of their plausible roles and relations . Given the complexity of the GCSR task , machines need a direct mechanism to create a high-level relational representation of the provided concepts , which would allow them to judge the plausibility of their combination . In this paper , we propose to model an explicit scene imagination step which constructs a structured representation of a plausible scene based on input concepts and initial context . The scene imagination module formalizes the background knowledge required for the reasoning through a contextualized relational graph , called scene knowledge graph ( SKG ) . An SKG allows us to collect and harmonize diverse commonsense knowledge across resources and modalities into a comprehensive SKG distribution ( see Figure 1 for an illustration ) . We develop an imagine-and-verbalize framework : an imagination module learns to construct a contextualized SKG from input concepts and context by pretraining over a large amount of external SKGs ; a verbalization module learns to faithfully realize the imagined SKG into natural language by training over downstream datasets . By learning from a large number of diverse SKGs , our method is able to capture plausible relations between concepts . By integrating these SKGs with LMs , the imagination module is able to compose new objects in novel ways , and identify implicit concepts for a scene . Imagine-and-verbalize decomposes the challenging scene description task into two realistic tasks for which a wealth of training data can be collected , simultaneously enabling for effective and explainable GCSR . We experiment with two GCSR tasks and three scene graph resources , observing consistently better or competitive performance to SotA baselines . We find that ( 1 ) SKGs extracted from visual captions and story datasets are more helpful than other resources ; ( 2 ) our model can learn faster ( with less training data ) with the help of scene imagination ; and ( 3 ) the imagination module with a larger backbone LM demonstrates larger capacity in encoding commonsense knowledge . Our human evaluation study on the generated imagination indicates that these SKGs capture common sense and that the verbalization module generates the text by following the guidance of the imagination . 2 METHOD . Formally , in GCSR , we consider a list of concepts sets { x1 , x2 , ... , xK } and a textual context c ∈ C as input . Each concept set xi is unordered and consists of multiple concept words { xj } . A concept word xj ∈ X ( or concept for brevity ) is a commonly seen object ( nouns such as “ dog ” or “ frisbee ” ) or commonly performed action ( verbs such as “ throw ” or “ catch ” ) . The goal of GCSR is to generate K sentences { y1 , y2 , ... , yK } , each describing a plausible situation following human common sense for a concept set xi . The i-th sentence yi ⊂ Y should be generated using all concepts in xi . We consider two variants of GCSR : 1 ) concepts-to-sentence generation ( Lin et al. , 2020 ) , where no context is given ( i.e. , c is empty ) and only one concept set is provided ( K = 1 ) ; and 2 ) concepts-tostory generation task , where c is the leading sentence of a multi-sentence story and more than one concept sets are provided , each corresponding to one sentence to be generated ( K > 1 ) . Both tasks are evaluated by comparing the machine-generated text with human-generated ( gold ) references . 2.1 THE IMAGINE-AND-VERBALIZE APPROACH . Pre-trained LMs struggle with learning a generalizable mapping from concepts to plausible sentences solely based on the training data . Augmenting concepts with external knowledge to form the input X ′ and fine-tuning a pretrained LM to model P ( Y|C , X ′ ) ( Liu et al. , 2020 ; Fan et al. , 2020 ; Li et al. , 2021 ) alleviates this issue partially , while still learning a direct mapping of { C , X ′ } → Y . In this work ( Figure 1 ) , we decompose the GCSR task into two sub-tasks : contextualized scene generation ( imagination ) and scene-aware text generation ( verbalization ) . P ( Y|C , X ) = ∑ Z P ( Y|C , X , Z ) P ( Z|C , X ) , ( 1 ) where Z denotes the scene representation for the given concepts and context . The contextualized scene imagination module P ( Z|C , X ) aims to construct a multi-relational graph representation Z ( scene knowledge graph , or SKG ) that describes a plausible scene that involves all input concepts and corresponds to the provided context . To learn this module , we collect a diverse set of SKG instances from different resources and modalities to form a comprehensive distribution of scenes ( §2.2 ) . The imagination module is pre-trained over the collected scene instances and learns to generate SKGs depicting plausible day-to-day situation . The imagination module is based on a neural architecture , which enables it to generate concept compositions that might not have been observed during training ( §2.3 ) .2 We leverage the contextualized SKG for text generation with a verbalization module P ( Y|C , X , Z ) which takes the context , concepts , and the generated SKG as input , and composes a grammatical and plausible scene description in natural language ( §2.4 ) . To perform GCSR , where one or multiple concept sets are given , we apply the imagination module to sample zi . Since the marginalization over Z is generally intractable due to the complex structure of the SKGs , we only sample the most probable scene representation z∗i that maximizes P ( zi|c′ , xi ) , where c′ includes the given context c and the previously generated yj , ( j < i ) . We then apply the verbalization module to generate one sentence at a time by sampling from P ( yi|c′ , xi , zi∗ ) . Multiple sentences are generated by iteratively applying the imagination and verbalization modules . 2.2 IMAGINATION VIA GENERATING SKG . Imagination through SKGs We adopt the term “ scene graph ” from the computer vision community , and we generalize it to a novel relational schema that represents knowledge from multiple modalities . Our SKG is defined as a relational graph G = ( E , R ) that organizes a set of concepts in a coherent scene that follows common sense . The node set E of the graph includes both given and implicit concepts , while each relation ( edge type ) r ∈ R denotes how two concepts should be related . We follow the Abstract Meaning Representation ( AMR ) ( Banarescu et al. , 2013 ) schema to consider the core relations between two concepts , which corresponds to the commonsense knowledge required by GCSR . Table 7 in the appendix illustrates a few representative relations and their examples . Table 1 : Statistics of the SKG instances collected from different resources . Knowledge source # SKGs # Concepts Caption-AMR 584,252 22,961 Story-AMR 927,163 41,272 VG-SceneGraph 292,596 41,629 All 1,792,941 84,835 Collecting Diverse SKGs We consider two complementary modalities , text and vision , as some concepts and relationships are more likely to occur in one modality versus another . ( 1 ) Textual Modality : According to pragmatic principles of human language , people generally leave out expected details about common scenes ( Grice , 1975 ) . For this reason , we extract SKGs from visual captions and narrative stories , in which human annotators are asked to explicitly describe scenes that may happen using 2The imagination module can be further fine-tuned over the downstream datasets . descriptive language as shown in Figure 1 ( a , b ) . To extract an SKG out of these textual signals , we adopt the AMR parsing tool to transform each sentence into an AMR graph . This process yields a single SKG per sentence . For the story SKGs , we also keep the sentences ( up to 256 tokens ) that precede the sentence that corresponds to the SKG , as context c. ( 2 ) Visual Modality : Image captions focus on salient information and may not capture all useful visual signals . Thus , we also capture the scene structures directly from images , by using VisualGenome ( Krishna et al. , 2016 ) , a large-scale scene graph dataset annotated by humans . To adopt a unified SKG schema , we manually map the relations in scene graphs from VisualGenome to the ones used in textual SKGs . A full set of mapping rules can be found in the Appendix ( A.1 ) . The statistics of the SKGs collected from each resource/modality are summarized in Table 1 . We note that visual scene graphs may be biased towards knowledge about spatial relationships and object affordance , which further motivates our decision to extract SKGs from multiple modalities .

Summary: In this work, the authors propose a novel framework for the generative commonsense reasoning (GCSR) tasks. GCSR are tasks where given a set of concepts (or a sequence of set of concepts), a model is required to generate sentences (or a short paragraph) that are both grammatically correct and plausible (follow commonsense).   The authors propose Imagine-and-Verbalize (I&V), which is a 2-layer system: 1. The imagination module is a transformer-based seq2seq model, takes a set of concepts (and sometimes text as context) as input, and generates a flattened graph (AMR tree, PENMAN serialization).  2. The verbalization module is another transformer-based seq2seq model, which takes the concepts, some context, and the generated scene knowledge graph (SKG, flattened) as input, and translate the scene information into natural language sentences (or iteratively, paragraphs).  On two GCSR datasets, the proposed method significantly outperform strong baselines and prior works.