Contextualized Scene Imagination for Generative Commonsense Reasoning
Authors: PeiFeng Wang
Jonathan Zamora
Junfeng Liu
Filip Ilievski
Muhao Chen
Xiang Ren
Venue: ICLR 2022 Poster (2022)
Keywords: Commonsense reasoning
constrained text generation
knowledge representation

Abstract: Humans use natural language to compose common concepts from their environment into plausible, day-to-day scene descriptions. However, such generative commonsense reasoning (GCSR) skills are lacking in state-of-the-art text generation methods. Descriptive sentences about arbitrary concepts generated by neural text generation models (e.g., pre-trained text-to-text Transformers) are often grammatically fluent but may not correspond to human common sense, largely due to their lack of mechanisms to capture concept relations, to identify implicit concepts, and to perform generalizable reasoning about unseen concept compositions. In this paper, we propose an Imagine-and-Verbalize (I\&V) method, which learns to imagine a relational scene knowledge graph (SKG) with relations between the input concepts, and leverage the SKG as a constraint when generating a plausible scene description. We collect and harmonize a set of knowledge resources from different domains and modalities, providing a rich auxiliary supervision signal for I\&V. The experiments demonstrate the effectiveness of I\&V in improving language models on both concept-to-sentence and concept-to-story generation tasks, while enabling the model to learn well from fewer task examples and generate SKGs that make common sense to human annotators.

Published as a conference paper at ICLR 2022
CONTEXTUALIZED SCENE IMAGINATION FOR GENERATIVE COMMONSENSE REASONING
Peifeng Wang 1.3 Jonathan Zamora Junfeng Liu';
Filip Ilievski', Muhao Chen Xiang Ren 2.3
DDepartment of Computer Science, University of Southern California
Department of Computer Science, University of California, San Diego IInorrmatinn Sciences Institute, University of Southern Californi [peifengw, fie, muhaoche, xiangren] @usc. edu, m.amoraauucs.eddy edu, ilievski (isii... ieiieee edd
ABSTRACT
Humans use natural language to compose common concepts from their environ- ment into plausible, day-to-day scene descriptions. However, such generative commonsense reasoning (GCSR) skills are lacking state-of-the-art text genera- tion methods. Descriptive sentences about arbitrary concepts generated by neural text generation models (e.g., pre-trained text-to-text Transformers) are often gram- matically fluent but may not correspond to human common sense, largely due their lack of mechanisms to capture concept relations, to identify implicit con- cepts, and to perform generalizable reasoning about unseen concept compositions. In this paper, we propose an Imagine-and Verhalize (I&V V) method, which learns imagine relational scene knowledge graph (SKG) with relations between the input concepts, and leverage the SKG as a constraint when generating plausible scene description. We collect and harmonize set of knowledge resources from different domains and modalities, providing a rich auxiliary supervision signal for I&V. The experiments demonstrate the effectiveness of V&V in improving lan- guage models on both concept-to-sentenc and concept-to-story generation tasks, while enabling the model to learn well from fewer task examples and generate SKGs that make common sense to human annotators
INTRODUCTION
Humans describe everyday scenes in natural language based on their understanding of common concepts encountered in their environment (Tincoff & Jusczyk 1999). Analogously, the task of generative commonsense reasoning (GCSR) asks machines to generate a description of everyday situations based on set of concepts and an initial context (Liu et al. 2020 et al., 2021p. For example, given concept words {dog. frisbee, catch, throw). a machine is expected to generate a plausible description, e.g., "A man throws frishee and his dog catches in the air". Machines with GCSR skills would communicate fluidly with humans, e.g., when summarizing a document by preserving its key details (Shal 2020), composing creative story according to set of clues (Yao) ct al. 2019), and generating a conversation reply that includes specified keywords (Mou et al. 2016).
GCSR poses three unique challenges for automatic text generation methods. To depict plausible scenes when composing sentences, machines require commonsense knowledge to reason about the relations between concepts and the affordances of objects (c.g.. "dog" performs the action "catch" but not the action "throw"). Moreover, machines require a compositional generalization ability (Key-) sers al., 2019) i.e., the ability to judge the plausibility of new concept composition that has not been observed during training, and to identify concepts related to the scene that are not explicitly provided (e.g., "person" to perform "throw" in the above example).
GCSR can be directly attempted by fine-tuning pre-trained text-to-text language models (LMs) (Raf-) fel et al. 2019 Radford et al. 2019). While pre-trained LMs capture certain encyclopedic knowl-
"Equal contributions
Cooee and data available httpp///ggtthib..cm/////ggggggg.

---

Published as a conference paper at ICLR 2022
Captions
Imagination Module (Our focus)
Verbalization Module
wormn riding bicycle along the river.
woman throws
dog
frisbee and a dog catches the air.
catch
Staries
frisbee
ConncppttSeeeennce
ARGI
When their wagon breaks they make their way through woods
ARG1
Input
Concepts: (dog, frisbee, throw, catch)
ARGO
throw
Output
woman throws frisbee dog it."
hut.
Concept2Story
Scene graphs
guitar
Scene Knowledge Graph
Context: "My sister anrived early help with the family B6Q."
playing
fedora
Concepts': : (everyone, arrive)
person
wearing
Learning to imagine
O
Learning to verbalize
Output
Concepts": (day,,
"Everyone else arrived soon after...
shoot)
day shooting off some fireworks."
Figure I: Overview of the proposed V&V method: (1) We leverage SKGs for unifying scene knowl- edge from different resources. (2) We pre-train contextualized imagination module to construct an SKG for set of concepts, based on the collected SKG instances. (3) At inference time, our verbalization module realizes the generated SKG into natural language.
edge mentioned in text corpora (e.g.. Wikipedia) (Petroni et al. 2019) and can combine concepts in novel ways, they may generate grammatically fluent but implausible sentences that conflict with hu- 2020). This is because LMs have no intrinsic mechanism to reason over high-level relations between concepts /hhou et al. (2020). To close the knowledge gap, re- cent work augment LM input with knowledge graph triples (e.g., (dog, CapableOf, catch)) retrieved from ConceptNet (Liu et al. 2020 et al. 2022)), (0000,)..). or prototype sentences that cover input concepts retrieved from external text corpora (Fan al. 2020 Wang al. 202T). However, despite the input augmentation, GCSR skills are implicitly learned based on the concept-text pairs in the training data, without explicit supervision. While some recent work propose content planning in story generation in the form of plots or scripts (Yao et al. 2019 Fan al. 2019), only the narrative order of concepts are planned those methods instead of their plausible roles and relations. Given the complexity of the GCSR task, machines need direct mechanism to create high-level relational representation of the provided concepts, which would allow them to judge the plausibility their combination.
man common sense (Lin al.
In this paper, propose to model an explicit scene imagination step which constructs struc- tured representation of a plausible scene based on input concepts and initial context. The scene imagination module formalizes the background knowledge required for the reasoning through contextualized relational graph, called scene knowledge graph (SKG). An SKG allows us to collect and harmonize diverse commonsense knowledge across resources and modalities into compre- hensive SKG distribution (see Figure [ for an illustration). We develop an imagine-and- framework: an imagination module learns construct contextualized SKG from input concepts and context by pretraining over a large amount of external SKGs; a verbalization module learns to faithfully realize the imagined SKG into natural language by training over downstream datasets. By learning from large number of diverse SKGs, our method is able to capture plausible relations between concepts. By integrating these SKGs with LMs, the imagination module is able com- pose new objects in novel ways, and identify implicit concepts for scene. imagine-and-verraali decomposes the challenging scene description task into two realistic tasks for which a wealth of training data can be collected, simultancously enabling for effective and explainable GCSR.
We experiment with two GCSR tasks and three scene graph resources, observing consistently better competitive performance to SotA basclines. We find that (1) SKGs extracted from visual cap- tions and story datasets are more helpful than other resources; (2) our model can learn faster (with less training data) with the help of scene imagination; and (3) the imagination module with larger backbone LM demonstrates larger capacity in encoding commonsense knowledge. Our human eval- uation study the generated imagination indicates that these SKGs capture common sense and that the verbalization module generates the text by following the guidance of the imagination.
2 METHOD
Formally, in GCSR, we consider a list of concepts sets {x', ,x? xx....,xx*, input. Each concept set x is unordered and consists of multiple concept words (I,). A concept
xx,, and a textual context c E
2

---

Published as a conference paper at ICLR 2022
word 2, E X' (or concept for brevity) is a commonly seen object (nouns such as "dog" "frisbee") or commonly performed action (verbs such as "throw" or "catch"). The goal of GCSR is to generate K' sentences (y' .y² ..... y.. ). each describing plausible situation following human common sense for a concept set x'. The i-th sentence y' C y should be generated using all concepts in x'.
We consider two variants of GCSR: 1) concepts- concepts-to-senteen to-seneence generation (Lin al. 2020), where no context is given (i.e., e is empty) and only one concept set is provided (K = =); and 2) concepts-to- story generation task, where e the leading sentence of multi-sentence story and more than one concept sets are provided, each corresponding to one sentence to be generated (K' > 1). Both tasks are evaluated by comparing the machine- gennrrttd text with suman-generatee (gold) references.
2.1 THE IMAGINE- AND- VERBALIZE APPROACH
Pre-trained LMs struggle with learning a generalizable mapping from concepts to plausible sen- tences solely based on the training data. Augmenting concepts with external knowledge to form the input X' and fine-tuning a pretrained LM to model P()IC, X') (Liu et al. 2020: Fan ct al. 2020 Li et al. 2021) alleviates this issue partially, while still learning a direct mapping of x', y In this work (Figure D]. decompose the GCSR task into two sub-tasks, namely contextualized scene generation (imagination and scene-aware text generation (verbalization):
P(VIC,X) Σ X.
where Z denotes the scene representation for the given concepts and context.
The contextualized scene imagination module P P(ZIC, X)) aims to construct a multi-relational graph representation Z (scene knowledge graph, or SKG) that describes plausible scene that involves all input concepts and corresponds to the provided context. To learn this module, we collect a diverse set of SKG instances from different resources and modalities to form comprehensive distribution of scenes ($2.)). The imagination module is pre-trained over the collected scene instances and learns generate SKGs depicting plausible day-to-day situation. The imagination module is based neural architecture, which enables it to generate concept compositions that might not have been observed during training ($2.3) 2 We leverage the contextualized SKG for text generation with verbalization module P() C, Z) which takes the context, concepts, and the generated SKG as input, and composes grammatical and plausible scene description in natural language 2..4.
To perform GCSR, where one multiple concepp sets are given, we apply the imagination module to sample z!. Since the marginalization over Z is generally intractable due to the complex structure of the SKGs, we only sample the most probable scene representation z that maximizes P(x |c', x*), where c' includes the given context с and the previously generated y", < i). We then apply the verbalization module to generate one sentence at time by sampling from P(y"lc', x''')")") Multiple sentences are generated by itcratively applying the imagination and verbalization modules.
2.2 IMAGINATION VIA GENERATING SKG
Imagination through SKGs We adopt the term "scene graph" from the computer vision com- munity, and we generalize it to novel relational schema that represents knowledge from multiple modalities. Our SKG is defined as relational graph G (E, R) that organizes set of concepts in coherent scene. The node set È of the graph includes both given and implicit concepts, while each relation (edge type) r E R denotes how two concepts should be related. We follow the Abstract Meaning Representation (AMR) (Banarescu et al., 2013) schema to consider the core relations be- tween two concepts, which corresponds to the commonsense knowledge required by GCSR. Table 7 in the appendix illustrates a few representative relations and their examples.
Collecting Diverse SKGs We consider two complementary modalities, text and vision, as some concepts and relationships are more likely to occur in one modality versus another. (1) Textual Modality: According to pragmatic principles of human language, people generally leave out pected details about common scenes (Grice 1975). For this reason, we extract SKGs from visual captions and narrative stories, in which human annotators asked explicitly describe scenes that
Thee imagination module can be further fine-tuned over the downstream datasets.
3

---

Published as a conference paper at ICLR 2022
Gold SKGs from external resources for continual pretraining
Graph linearization
"throw <:ARG0> woman <SEP> throw <:RGG>> frisbee
Transformer
(Optional) SKGs from task dataset fine-tuning
"Context: People are playing on the grass. Concepts: dog <SEP> frisbee <SEP> catch <SEP> throw"
Randomly ordered concepts
Continual pretraining and fine-tuning of the imagination module output linearized SKG based on a sequential input (context and concepts).
Figure 2:
may happen using descriptive language as shown in Figure Ta.b). To extract an SKG out of these textual signals, we adopt the AMR parsing tool to transform each sentence into an AMR graph. This process yields a single SKG per sentence. For the story SKGs, we also keep the sentences (up 256 tokens) that precede the sentence that corresponds to the SKG, as context (2) Visual Modality; Image captions focus on salient information and may not capture all useful visual signals. Thus, also capture the scene structures directly from images, by using VisualGenome (Krishna et al.) 2016), a large-scale scene graph dataset annotated by humans. To adopt a unified SKG schema, we manually map the relations in scene graphs from VisualGenome to the ones used in textual SKGs. full set of mapping rules can be found in the Appendix (A.T). The statistics of the SKGs collected from each resource/modality biased towards knowledge about spatial relationships and object affordance, which further motivates our decision extract SKGs from multiple modalities.
summarized in Table We note that visual scene graphs may be
2.3 LEARNING THE SCENE IMAGINATION MODULE
We describe how we pre-train the scene imagination model using multimodal SKG examples collected from diverse sources, and how we fine-tune the imagination module to downstream datasets.
Table I: Statistics of the SKG instances col- lected from different resources.
Knowledge source
SKGs
Concepts
straightforward way to construct SKG to retrieve instances that contains all the given concepts from the collected SKGs. However, performance of such method is limited by the coverage of the SKG collection and will fail when encountering novel concept composition. We propose to model P(Z C, X') with neural graph generator. Inspired by previous work (condi- tional) graph generation (You et al. 2018), we formulate SKG construction as an auto-regressive sequence generation task, where linearized SKG is generated sequentially conditioned the con- text, input concepts, and the graph sequence generated so far. Since the sequence generation problem can be efficiently tackled by pre-trained auto-regressive LMs (c.g., GPT-2 Radford al. (2019)). adopt LMs as the backbone of our imagination module (Bossclut et al. 2 et al. 2020).
Caption-AMR Story-AMR
584,252
22,961
927.163
41,272
VG SceneGraph
292,596
1,792.941
84,835
Linearized SKG Generation To form training instances for the imagination module, we treat the nodes in SKG instance as input concepts and the linearized SKG the target output (Figure 2
The input concepts are concatenated into sequence x [x1 T2, Zn), preceded by the context c' C. When is not given, we prepend the word "none" to the concept sequence. To linearize
an AMR-based SKG into sequence z [ =.. Am we adopt the PENMAN serialization format (Goodman) 2020) which converts AMR into a spanning tree over the graph. This format shown to be more suitable than other lincarization strategies like depth-first-searchh (DFS) in enabling LMs to learn the graph structure (Mager et al. 2020).
During training, we randomize the order of the concepts at every training step such that the graph generator learns to be invariant to concept order (Zhang et al. 2019). For each training instance, we randomly discard small subset of the SKG nodes (concepts) in each training epoch. This simulates the scenario in which a subset of the concepts that constitute a scene will be given, thus teaching the model infer implicit concepts for completing a plausible scene.
4

---

Published as a conference paper at ICLR 2022
and Fine-tuning With both the imput concepts (plus context) and the output graph lincarized as sequences based on the collected SKG instances, we continually pretrain an auto- regressive LM to generate z Transformer(c', x). The training objective is to maximize P(Z|C, x) by minimizing the negative log-likelihood:
Limagine
log P(zz|=et, x',x). x).
(2)
Our pre-trained imagination module generates an SKG on the fly, and it can be further fine-tuned on downstream datasets, when their distributions of context and concepts pretraining data (see Figure illustration). Since downstream datasets cannot be expected to have ground-truth SKGs paired with each training example, we apply the AMR parsing tool described in $2.2 on the training sentences to obtain silver-standard SKGs. We then follow the same training procedure continually pretrain the module into a customized imagination module for a specific downstream dataset.
different from the
2.4 SCENE-AWARE VERBALIZATION
Iterative Imagine-and-Vel Verbalize At the inference time, we apply the trained imagination mod- ule iteratively generate the most plausible SKG for each given concept set x', i.e., x²x arg maxx: P(z'|c', ,/,,'''' x²), where the context c' includes both the given context and the previously generated sentences {y²} (j < i). The generated SKG is used by the scene-aware verbalization module to model P() X, Z). The verbalization module generates the i-th sentence by sampling from P(y'lc', x², ")) Multiple sentences are generated iteratively by alternating between the scene imagination (to construct SKG) and verbalization (to produce the next sentence). See Figure for an illustration this iterative inference process.
Model Training Since both the linearized SKG (generated by the imagination module) and the tar- get sentences are sequences by nature, we design gener- ative model and learn this verbalization module by fine-tuning another pre-trained auto-regressive LM, i.., y* = Transformer(c! (c', x', z²), To form the input for generating the sentence y', we con- catenate the context c', the concept set sequence x" and z into one sequence illustrated Fig- ure 3 We then train the model to maximize P(YCC, Z) by minimizing the negative log- likelihood:
throws frisbee and dog catches it."
Verbalization Transformer2
P()'C, 2) 2)
"Context: People are playing on the grass. Concepts: dog <SEP> frisbee <SEP> catch <SEP> throw.
Relations: throw <:ARGO> woman <SEP> throw
<:ARG1> frisbee...
Linearized
Imagination Transformer1
[Context, previously generated sentences, concepts)
Lverbalize
log Bullubes c', x², x²,z") (3)
Figure 3:
Our I&V method iteratively applies imagination and the verbalization modules, by gen- erating one sentence in each iteration.
For each training instance (y², c', x*), we con- struct two types of SKG instances as the input z':
(1) We perform AMR parsing on y' to obtain silver-standard SKG: (2) We apply the trained imagi- nation module to generate a SKG z = arg max P(z , x'). where e' includes the given context and the ground-truth prefix sentences (y") (j i). We find it beneficial to train the verbalization module over these two types of SKGs as evidenced by our ablation study ( ($A.7). During inference, the SKG z' is generated by the imagination module, while c' includes the given context e and the previous sentences {y") (j < i) generated by the verbalization module.
3 EXPERIMENTAL. SETUP
Tasks & Datasets We consider two GCSR tasks: Concep12Sentence and Concept2Story. (!) Con- cept2Sentence is task of generating single sentence for given set of concepts and no context.
Our ablation study in Appendix A.6 shows that including all these elements input is helpful.
5

---

Published as a conference paper at ICLR 2022
We evaluate Concep12Sentence the CommonGen (Lin al., 2020) benchmark. Since the labels of the official test set are not publicly available, we submit our method to the leaderboard to obtain its performance. Notably, the concept sets in CommonGen's 's test set are novel and do not appear in the training set. We also create an in-house split of CommonGien to facilitate comparison between different variants of our method and the baselines.
(2) Concept2Story is a generalization of the con- cept2sentence task, where the goal is to generate a coherent story with K = 4 sentences given a set of concepts and an initial verbal context. We construct two benchmarks based on the Visual Story Telling (VIST) (Huang et al., 2016) and ROCStories (Mostafazadeh et al. (016) datasets. Follow- ing CommonGen, we conduct part-of-speech tagging over the sentences and further lemmatize the recognized verbs and nouns to obtain the concept sets.
Baselines (1) Concept2Sentence: We consider several recent submissions to the leaderboard of CommonGen that leverage auxiliary information for GCSR. KFCNet (Li et al., 202T). Re-T5 (Wang et at 2021), and EKI-BART (Fan et al.) 2020) are prototype-based models, which retrieve sentences containing as many input concepts as possible from external captions and NIJ datasets, and then use these sentences as auxiliary inputs. VisCTG (Feng et al., 202 is image-augmented model which retrieves images from Google by using concepts as a query, followed by an image caption- ing model that generates captions auxiliary inputs. KG-BART (Liu et al. 2020) is knowledge graph-augmented model which retrieves relations between concepts from ConceptNet as auxiliary inputs. SAPPHIRE (Feng al. 2021a) is keyword-based model which extracts keywords from sentences as auxiliary inputs only during training. We also compare to Node2Text, which fine-tunes pre-trained auto-regressive LM take the concatenation of concepts input and output the target sentences. (2) Concept2Story: We augment Node2Text with the iterative generation pipeline as in our method, which generates the next sentence given the provided context, previously generated sen- tences and the current concept set. In addition, we experiment with two representative methods from the controlled text generation literature. Plan-and-write (Yao et al. 2019) first generates storyline keywords, then uses the keywords to generate a story. We use the concept set and context to generate storyline keywords. Action-Plan (Fan et al., 2019) uses predicate- aragumet pairs as storyline. We adapt the KFCNet model to retrieve prototype sentences. All Concept2Story baselines are used in an iterative generation pipeline, to enable fair comparison to our method.
Evaluation Metric We evaluate systems against the K reference sentences pro- vided by a dataset, by measuring the similarities between the maahine-generater text and the gold references. Follow- ing CommonGen (Lin et al., (020), we adopt widely-used automatic metrics for evaluating text generation, which focus on (1) n n-gram overlap: BLEU (Papinenil jct al. 2002). ROUGE (Lin) 2004). and
Performance comparison with the top-ranked, published models on the official CommonGen test set. "Note that KFCNet uses much larger corpora (over 70M) retrieve prototypes and average less than concept the concept sets is not covered (Li al.. [02I], while we filter out any SKGs that contain concept sets that overlap with CommonGen datasct.
Table 2:
Model
BLEU-
CIDEr
SPICE
KFCNet Lii et al.
2021
43.62
18.85
33,91
RE-T5 Wang al. 2021
40.86
17.66
31.08
METEOR (Banerjee & Lavie 2005). ) and
VisCTG (reng el al. 212Tb
36.94
17.20
29.97
(2) concept association: CIDEr (Vedan- tam et al., 2015) and SPICE (Anderson jet al. 2016). Lin et al. (2020) reports that
SAPPHIKE Feng ot al
2021a)
37.12
16.90
29.75
KG-BART Liu et al. 2020
33.87
16.93
29.63
E.K-BART Fan ef al.
2020
35.95
17.00
29.58
SPICE yields the best correlation with hu- man judgments and thus we used as the main evaluation metric.
TS-base (our implementation)
33.81
15.79
28.34
Ta-large (our implementation)
32.85
15.76
28.38
TS-large (reported)
31.96
15.13
28.86
L&V (TS-base) 1&V (TS-large)
40.
17.44
30.57
40.57
17.71
31.29
4 RESULTS AND ANALYSIS
We design experiments answer the following questions: (1) Does contextualized scene imagi- nation improve the performance of GCSR models? (2) Does imagination allow GCSR models to learn with less data? (3) How does each source of scene knowledge for pretraining affect the GCSR performance? (4) Do generated SKGs make common sense and correspond to the generated text?
4.1
MAIN RESULTS
We compare our proposed approach with state-of-ther the-art text generation methods on two GCSR tasks to understand whether scene imagination helps GCSR. Table 2 shows the performance of dif-

---

Published as a conference paper at ICLR 2022
Table 3: Performance of the compared methods the Concept2Story tasks. Best results are bold-faced. We
mark them with an asterisk they exceed the second best with statistical significance (p-value < 0.05).
Concept2Story-V VIST
Concept2Story ROC
TS-hase
BART-large
TS-hase
BART-large
Model
BLEU-4
CIDEr
SPICE
BLEU-4
CIDE
SPICE
BLEU-4
CIDEr
SPICE
BLEU-4
CIDEr
SPICE
Node2Test
20.64
25.41
58.55
18.52
22.91
55.48
23.31
29.32
57.66
20.60
26.09
53.80
Keyword
16.75
56.23
15.62
20.86
55.49
27.05
22.14
49.52
Action-Plan
17.84
22.77
57.11
16.20
21.10
54.77
21.15
27.32
56.14
20.45
26.29
54,32
Prototype
20.28
25.05
58.17
22.81
26.93
58.84
23.59
29.48
57.68
26.76
31.60
58.35
I&V
21.05"
25.
59.21
22.45
26.80
59.11
26.77'
32.33"
60.63"
28.30"
33.40*
60.39'
ferent models on CommonGinn We have the following observations. First, V&V drastically improves the vanilla Ta-lerge model (Node2Text), demonstrating the effectiveness of the imagination module in GCSR. We also provide concrete examples in |A.4 which showcase how imagination fixes errors made by Node2Text. All these errors can be attributed to the fact that Node2Text does not properly capture the commonsense relations between concepts while our imagination module learns how con- cepts are related from indirect supervision. Second, our model outperforms other models using dif- ferent auxiliary inputs, including prototypes (Re-T5 and EKI-BART), knowledge facts (KG-BART) and images (VisCTG), showing the benefit of SKGs over these knowledge sources. Although our model under-performs KFCNet, our analysis in their work reveals that 97. 97.4% of the test cases have perfectly matched prototypes, i.e., sentences containing all the queried concepts. whether KFCNet is conducting commonsense reasoning or merely rephrasing the prototypes. Note that we filter out any collected SKGs that cover the concept sets from the downstream datasets. This ensures that the imagination module is examined with its compositional generalization.
thus unclear
Table sshwws the experimental results by V&V on the two Concept2Story datasets using T5-base and BART BART-large as the backend respectively. Among most evaluation metrics, our method outperforms Node2TText and baselines with other intermediate representations incorporated in the same backends. This demonstrates that our imagination module provide contextualized scene imagination that are more helpful in guiding long narrative generation.
4.2 PERFORMANCE ANALYSIS
How does the knowledge source affect GCSR? We perform an ablation study in order to under- stand how effectively each source of SKGs contributes the imagination. Specifically, we use each of the following SKG sources to pre-train an imagination module using T5-large as the backend: the silver-standard SKGs extracted from the training set from the downstream task (Task-AMR), and the external SKGs: Caption-AMR, Story-AMR, and VG-SceneGrraph ($2.2). For CommonGen, we do not further fine-tune the imagination module in order to distinguish the contributions from each knowledge source more clearly. For Concepi2Story (ROCstories), we conduct further fine-tuning using the task-AMR. Since this task provides the context as input, we find it helpful to adapt the imagination module with the task dataset.
The results are shown in Table 4 and we have the following observations. For CommonGen, the contribution comes mostly from the SKGs based on Caption-AMR while being less from VG- SceneGraph. This may due to the fact that VG-SeeneGrraph is biased towards spatial relations and attributes of objects. For Concept2Story, we find both Story-AMR and Caption-AMP ful for continual pretraining. The former teaches the model to generate contextualized imagination which is necessary for story generation in particular while the latter teaches the model about general commonsense knowledge. For both datasets, the imagination modules that are pre-trained over all the SKG instances yield significantly better results than the ones trained on the task-AMR datasets. This validates our intuition that different sources of SKGs contain complementary commonsense knowledge, and they should be used together for machine imagination.
be help-
How does the backbone LM size affect the module's performance? We also ablate the LM architecture of the imagination module and the verbalization module respectively to how our method work with different pre-trained LMs. For the imagination module, we use T5-base and Ta-gar This is to investigate how the capacity of LMs affects the learning of scene knowledge.
7

---

Published as a conference paper at ICLR 2022
Table 4: Performance of method using different SKG train the imagination module, with T5-large as the backbone L.M.
Table 5: SPICE performance of our
CommonGin (in-house)
Concept2Story--Ke
method using different sizes of T5 backbone for imagination module.
Knowledge Source
BLEU-4
CIDEr
SPICE
BLEU-4
CIDEr
SPICE
Task-AMR
28.87
15.74
31.22
23.14
29.25
57.91
Dataset Backbong LM
T5-base
TS-large
Caption-AMR
32.21
16.14
32.16
23.77
29.76
58.46
Story-AMR
23.73
13.51
27.53
24.17
30.10
58.59
CommonGenn (in (on-house)
32.00
33.49
VG Geeneeeerrpph
21.00
13.36
29.07
22,84
25.33
53.96
Concept2Story-R0
59.56
60.63
AII-SKG
33.27
16.95
33.49
26.77
32.33
60.63
NaceaTor::
34.5
IBV
SRICE
20.5
67 puras)
lor gprent
17.5
20
Nevici
Rodal Fel
Figure 4: Ablation study back- bone LM sizes of our verbalization module and Node2Text using the Concep12Story-ROC dataset.
50
ning Examples (V1ST)
700
5000
M
50 Training Exampies [ROC]
500
5000
MI
Traiking Examples (ComGen)
500
5200
Figure 5: Results (SPICE) of the low-resource experiment the three benchmark datasets different number training examples.
The results are shown in Table Compared to T5-large, we observe a slight performance drop for T5-base, which indicates that larger LMs are able to encode our rich set of SKG instances more expressive manner. For the verbalization module, we use BARTTbaae/large and T5-base/large. The results are shown in Figure 4 We observe that compare to baseline, our method consistently yields better performance regardless of what LM architecture is used.
Does imagination allow models to learn (faster) with less data? Next, we study how the indirect supervision provided the imagination module help the system effectively learn with limited task- specific training data. Accordingly, we conduct low-resource experiment where we randomly sample (50, 500, 5000) training and development examples from each dataset. For each data size, use 5 random seeds to obtain 5 different training and development splits. On each split, we train and test with random initialization of seeds, and we report the average on the total 15 ways of results. In this study, the imagination module is fixed untrainable after continual pretraining and is not fine-tuned over the sampled task datasets.
Figure 5 shows that model consistently outperforms the baselines, and the performance gain is larger when less training data are used. This indicates that rich sources of SKGs provide practical forms of indirect supervision complement limited task-specific training data. The robustness of our model in low-resource settings also justifies the need for including contextualized SKGs as an intermediate representation, which further enhances the verbalization module to generate plausible sentences even with little training data.
Is context helpful for imagination? To validate that the textual context, including the provided context as well as the previously generated sentences, is helpful for imagination in the Con- cept2Story task, we conduct an ablation study where we learn an uncontextualized imagination module which only takes concepts input. The final results VIST and ROC datasets are 47.32 and 45.18 (SPICE) respectively, which are much lower than the results from contextualized I&V (59.21 and 60.63). This demonstrates that the context is critical in generating SKGs which are more relevant to the story line and thus lead to better text generation.
4.3 HUMAN EVALUATION ON GENERATED SKGS
We conduct human evaluation on the SKGs generated by our imagination module to examine their quality. Annotators presented with the input concepts, the generated SKGs, the predicted sen-
8

---

Published as a conference paper at ICLR 2022
tences resulting from the corresponding SKGs and the ground-truth sentences for reference. For each dataset, 100 instances are randomly chosen for evaluation. Annotators are students majoring in computer science and not all of them know about AMR language prior to the human evaluation. To facilitate annotators' understanding of the evaluation task and AMR, we provide the detailed instruction and the examples of AMR relations. The annotators are asked to judge for: 1) Complete- ness, whether the SKG includes all the concepts (both given and implicit) to constitute coherent scene; 2) CommonSense, whether the SKG organizes the concepts sense; 3) Alignment, whether the generated sentence aligns with the SKG and 4) Similarity, whether the predicted sentence similar to any referenced sentence in semantic. Annotation based on a 3-point scale: a) 0 0- "I do not agree", b) 0.5 "I partially agree" and c) 1.0- "I fully agree".
way that follows common
shows the evaluation results where we get a fair level of agreement measured by Fleiss Kappa (k = 0.21). We observe that the generated SKGs are complete and follow common sense in high degree across three datasets, which learning useful common-
Table 6: Human evaluation on the generated SKGs regarding Completeness (COM), Commonsense (CS) and Align- ment (AL) and Similarity (SIM).
demonstrates the effectiveness sense knowledge with vast indirect supervision from differ- ent resources. Moreover, the SKGs are well-aligned with the generated text, which indicates that the verbalization module consistently follows the guidance of the imagination module when generating sentences. The moderate similarity scores validate that the generated text is gener- ally similar to the natural language sentences annotated by humans.
COM
CS
AL
SIM
CommonGen
97.30
90.15
89,90
88.30
VIST ROC
93.80
89.70
91.40
76.20
95.70
86.60
87.80
75.68
5 RELATED WORK
Knowledge- Enhanced GCSR Recent works (Liu et al. 2020 Li et al. 1 GCSR propose to retrieve external knowledge to enhance the text generation. Prototype- Prototype-based models, including EKI-BART (Fan al., 2020), Re-T5 (Wang al. 2021), and KFCNet (Li et al. 2021 retrieve massive prototype sentences from external corpora like visual captions and Wikipedia as auxiliary input to the L.M. Though the retrieved prototype sentences provide high coverage on the concepts, their model is supervised to compose sentences that are very similar to those existing prototypes. It thus unelear whether their models are conducting commonsense reasoning or only mimicking the prototypes. KG-BART (Liu et al. 2020) incorporates the embedding of relational facts about the concepts from ConceptNet into both the encoders and decoders of the BART architecture (Lewis) et al. 2020). As there could be multiple relations between two concepts, it is unclear how to select the relation that fits given context (Fadnis et al. (019)) Our imagination module infers the relations between concepts by taking all the concepts into consideration and organizes them in coherent way.
Content Planning Our method is also related to prior works (Goldfarb- Tarrant et al. 2020 that propose intermediate representations as a way to "plan ahead" before generating long narratives. Plan-and-write (Yao et al. (209)) generates chains of keywords as storyline, but do not consider relations between keywords (concepts) as we do. Action-plan (Fan et al. 2019) takes a step further with semantic role labeling, but still does not involve all the concepts by using Moreover, these methods are limited to obtaining supervision from task-specific a sentence. datasets, while we gather effective indirect supervision signals from rich multi-source, multi-modal SKG representations without the need for additional annotations.
6 CONCLUSIONS
This paper proposed to enhance neural architectures for GCSR with an intermediate imagination layer. We divided the GCSR process into two steps: imagination, which generated plausible scene knowledge graph for given set of concepts, and verbalization, which transformed this scene graph into a fluent sentence that corresponds to human common sense. The method was trained with diverse scene knowledge graphs derived from both text and vision modalities. Our experiments demonstrated the ability of the proposed method to perform GCSR effectively, by describing plau- sible scenes, and efficiently. by requiring less training data. The image caption graphs proved most beneficial to learn from. Future work should investigate the impact of imagination on interactive commonsense tasks, like dialogue generation, and include scene graphs from the audio modality.

---

Published as a conference paper at ICLR 2022
ACKNOWLEDGMENTS
We thank the anonymous reviewers and all the collaborators in USC INK research lab for their valuable feedback. This material is based upon work sponsored by the DARPA MCS program under Contract No. N66001 1924033 with the United States Office Of Naval Research.
REFERENCES
Peter Anderson, Basura Fernando, Mark Johnson, and Stephen Gould. Spice: Semantic propo- sitional image caption evaluation. In European conference on computer vision. pp. 382-398.
Springer, 2016.
Laura Banarescu, Claire Bonial, Shu Cai, Madalina Georgescu, Kira Griffitt, Ulf Hermjakob, Kevin Knight, Philipp Kochn, Martha Palmer, and Nathan Schneider. Abstract meaning representation for sembanking. In Proceedings of the 7th linguistic annotation workshop and interoperability with discourse, pp. 178-186. 186, 2013.
Satanjeev Banerjee and Alon Lavie. Meteor: An automatic metric for mt evaluation with improved correlation with human judgments. In Proceedings the acl workshop intrinsic and extrinsic evaluation measures for machine translation and/or summarization, pp. 65-72, 2005.
Antoine Bosselut, Hannah Rashkin, Maarten Sap, Chaitanya Malaviya, Asli Celikyilmaz, and Yejin Choi. COMET: Commonsense transformers for automatic knowledge graph construction. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguisties, pp. Florence, Italy, 2019. Association for Computational Linguistics. doi: 18.136/3/v1/ P19- 1470. URLthttss:/// www. ac web. wwtp://anncoooog///111-100100
Kshitij Fadnis, Kartik Talamadupula, Pavan Kapanipathi, Haque Ishfaq, Salim Roukos, and Achille Fokoue. Heuristics for interpretable knowledge graph contextualization. arXiv preprint arXiv:1911.02085 2019.
Angela Fan, Mike Lewis, and Yann Dauphin. Strategies for structuring story generation. arXiv preprint mrr:ivv:9220 01 109, 2019.
Zhihao Fan, Yeyun Gong, Zhongyu Wei, Siyuan Wang. Yameng Huang, Jian Jiao, Xuanjing Huang, Nan Duan, and Ruofei Zhang. An enhanced knowledge injection model for commonsense gener- ation. arXiv preprint arXiv:2012.00366 2020.
Steven Feng, Jessica Huynh, Chaitanya Prasad Narisetty, Eduard Hovy, and Varun Gangal. SAP- PHIRE: Approaches for enhanced concept-to-text generation. In Proceedings of the 14th Inter- national Conference on Natural Language Generation, pp. 212-225, Aberdeen, Scotland, UK August 2021a. Association for Computational Linguistics. URL htEtps://aclanthol.oo org/2021. inlg-1.21
Steven Y Feng, Kevin Lu, Zhuofu Tao, Malihe Alikhani, Teruko Mitamura, Eduard Hovy, and Varun Gangal. Retrieve, caption, generate: Visual grounding for enhancing commonsense text generation models. arXiv preprint arXiv:2109.03892, 2021b.
Seraphina Goldfarb-Tarrant Tuhin Chakrabarty, Ralph Weischedel, and Nanyun Peng. Content planning for neural story generation with aristotelian rescoring. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 4319-4338, Online, November 2020. Association for Computational Linguistics. doi: 10.18653/1/20000
amnlp-main.33 URL
org/2020 emalp maai.aannn...m. 351
Michael Wayne Goodman. Penman: An open-source library and tool for AMR graphs In Pro-
ceedings of the 58th Annual Meeting of the Association for Computational Linguistics: Sys- fem Demonstrations, pp. 312-319, Online, July 2020. Association for Computational Linguis- URL /ttp///ppp////aaaaaooooooo/////p
tics. doi: (al.:3/////2220alldddmmoooo
acl wew-deeess.... 35
H Paul Grice. Logic and conversation, syntax and semantics. Speech Acts, 3:41-58, 1975.
10

---

Published as a conference paper at ICLR 2022
Ting- Hao Huang, Francis Ferraro, Nasrin Mostafazadeh, Ishan Misra, Aishwarya Agrawal, Jacob Devlin, Ross Girshick, Xiaodong He, Pushmeet Kohli, Dhruv Batra, et al. Visual storytelling. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies, pp. 1233-1239, 2016.
Daniel Keysers, Nathanael Schärli, Nathan Scales, Hylke Buisman, Daniel Furrer, Sergii Kashubin, Nikola Momchev, Danila Sinopalnikov, Lukasz Stafiniak, Tibor Tihon, al. Measuring compo- sitional generalization: A comprehensive method on realistic data. In International Conference on Learning Representations. 2019.
Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A Shamma, Michael Bernstein, and Li Fei-Fei. Visual Connecting language and vision using crowdsourced dense image annotations. 2016.
genome:
URL https::/aaxxaai org/ /602. 07332
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. Bart: Denoising eqquence-t--seuuee sequence pre- training for natural language generation, translation, and comprehension. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 7871-7880 2020.
Haonan Li, Yeyun Gong, Jian Jiao, Ruofei Zhang, Timothy Baldwin, and Nan Duan. Kfenet: Knowl- edge filtering and contrastive learning network for generative commonsense reasoning. arXiv preprint arXiv:2109.06704 2021.
Yikang Li, Pulkit Goel, Varsha Kuppur Rajendra, Har Simrat Singh, Jonathan Francis, Kaixin Ma, Eric Nyberg. and Alessandro Oltramari. sexically-constraime text generation through common- sense knowledge extraction and injection. arXiv preprint urXiv:2012.108133 2020.
Bill Yuchen Lin, Wangchunshu Zhou, Ming Shen, Pei Zhou, Chandra Bhagavatula, Yejin Choi, and Xiang Ren. CommonGen: A constrained text generation challenge for generative commonsense reasoning. In Findings of the Association for Computational Linguistics: EMNLP 2020, pp- 1923-1840, Online, November 2020. Association for Computational Linguistics. URL https:
//www. www..we.bb... orgg htt.w//w...rgg//o//ooooooooo hnthooogyohoo20000 f findings emn monnpp.1. ppp...).))
Chin-Yew Lin. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out, pp. 74-81, 2004.
Ye Liu, Yao Wan, Lifang He, Hao Peng, and Philip S Yu. Kg-bart: Knowledge graph-augmented bart for generative commonsense reasoning. arXiv preprint arXiv:2009. 2020.
Manuel Mager, Ramón Fernandez Astudillo, Tahira Naseem, Md Arafat Sultan, Young-Suk Lee, Radu Florian, and Salim Roukos. Gpt-too: A language-modellfisst approach for amr-to-text generation. arXiv preprint arXiv:2005. 09 /23, 2020.
Nasrin Mostafazadeh, Nathanael Chambers, Xiaodong He, Devi Parikh, Dhruv Batra, Lucy Van- derwende, Pushmeet Kohli, and James Allen. A corpus and cloze evaluation for deeper under- standing of commonsense stories. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp- 839-849, San Diego, California, June 2016. Association for Computational Linguistics. doi: N..18653/11NN16 1098. URL htt.htttss//aclaatoooooy.oo org.No11o.....oo- 1017N66661988
Lili Mou. Yiping Song, Rui Yan, Ge Li, Lu Zhang, and Zhi Jin. Sequence backward and forward sequences: A content-introducing approach to generative short-text conversation. arXiv preprint 1607.00007.009700 2016.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pp. 311-318, 2002.
Fabio Petroni, Tim Rocktäschel, Sebastian Riedel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, and Alexander Miller. Language models as knowledge bases? In Proceedings of the 2019 Con- ference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP--JCNIPP) pp. 2463-2475 Hong Kong.
11

---

Published as a conference paper at ICLR 2022
China, 2019. Association for Computational Linguistics. doi: 1/1/655///1D11-12500 URL
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, llya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9, 2019.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. arXiv preprint arXiv:19I0. I0t::399000083,, 2019.
Lei Sha. condient-guideds unsupervised lexically constrained text generation. In Proceedings of the 2020 Conference on Empirical Methods Natural Language Processing (EMNLP), pp. 8692- 8703, 2020.
Ruth Tincoff and Peter W Jusczyk. Some beginnings of word comprehension in 6-month-olds. Psychological science, 10(2):122-77,, 1999.
Ramakrishna Vedantam, C. Lawrence Zitnick, and Devi Parikh. Cider: Consensus-based image description evaluation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2015.
Han Wang. Yang Liu, Chenguang Zhu, Linjun Shou, Ming Gong, Yichong Xu, and Michael Zeng. Retrieval enhanced model for commonsense generation. arXiv preprint arXiv:2105.11174, 2021.
Peifeng Wang, Nanyun Peng. Filip Ilievski, Pedro Szekely, and Xiang Ren. Connecting the dots: A knowledgeable path generator for commonsense question answering. arXiv preprint arXiv:2005.0069), 2020.
Lili Yao, Nanyun Peng, Ralph Weischedel, Kevin Knight, Dongyan Zhao, and Rui Yan. Plan-and- write: Towards better automatic storytelling. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pp. 3378-7385. 2019.
Jiaxuan You, Rex Ying, Xiang Ren, William Hamilton, and Jure Leskovec. Graphrnn: Generat- ing realistic graphs with deep auto-regressive models. In International conference machine learning, pp. 5708-5717. PMLR, 2018.
Yan Zhang, Jonathon Hare, and Adam Prügel-Bennett. Fspool: Learning set representations with featurewise sort pooling, In International Conference on Learning Representations, 2019.
Pei Zhou, Rahul Khanna, Seyeon Lee, Bill Yuchen Lin, Daniel Ho, Jay Pujara, and Xiang Ren. Rica: Evaluating robust inference capabilities based on commonsense axioms. arXiv preprint arXiv:2005.00782. 2020.
A APPENDIX
A.1 RULES FOR MAPPING VISUAL SCENE GRAPHS TO SKG
There are 3858 relation types in our processed VisualGenome dataset due to the noisy annotation. We map these relations into 8 relations. For relations that are annotated verbs by VisualGenome, we break the relationship (subject, relation, object) into (relation, :ARGO, subject) and (relation, ARRI, oject). For other popular relations, we conduct the following mapping,,subject, be, ob- (subect,, domain, object), (subject, displace, -sujec) possible, (sbbec), have/of, (subec)- part, (sujec),, with, location, object). The remaining relations that do not follow the above mapping criteria are mapped to an "other" relation. Note that the non- "other" relations make up 97.73% of the triplets in VisualGenome.
poss, object)
wwwbchindd////ndde//llon/g//////
object)
12

---

Published as a conference paper at ICLR 2022
Table 7: The most common relation types in SKG instances and their example triplets.
Relation types
Examples
ARGI
(play, ARGI, guitar)
ARGO
(play, ARGO, man)
ARG2
(ask, ARG2, girl) (play, Location, stage) (play, Time, sing) (down, Opl, stair) (dog, Part, ear)
Location
Time
Opl
Part
A.2 IMPLEMENTATION DETAILS
For the main experiments, we develop the imagination module by continually pre-train T5-large model over the caption, story, and vision SKGs. We then further adapt the imagination module over each task dataset annotated with the silver-standard SKGs by further fine-tuning, To train the verbalization module, we fine-tune T5-base and BART-large as two backend LMs. During training, both silver-standard SKGs and generated SKGs, while averaging the training loss associated with each of them. We use the Adam optimizer with weight decay le 2. We search the optimal hyper-parameters based on the perplexity
the development set, where the learning rate is chosen from {2e 6, le 5, 3e 5, le 4), the batch size is chosen from {8, 16, 32, 64, 128).
A.3 STATISTICAL ANALYSIS
Table 8: Statistical analysis (p-values) for the ablation study on the backend LM used by the ver- balization module and the low-resource experiment. < 0.01 indicates a significant improvement and 0.05 indicates fairly significant improvement, and NA indicates that our method does not outperform the best baseline.
BART-base
BART-large
T5-base
T5-base
0.01
0.01
0.01
0.01
# Training examples
CommonGen (in-house)
VIST
ROC
50
< 0.01
0.01
< 0.01
500
< 0.01
< 0.01
< 0.01
5000
0.01
NA
< <.0.1
All
NA
0.05
< 0.01
We conduct statistical significance analysis the experiments involving baselines, which include
the ablation study on the backend LM used by the verbalization module (Figure 4 and the low- resource experiment (Figure 5). The p-values
shown in Table 8 where 0.01 indicates
significant improvement and < 0.05 indicates fairly significant improvement, and NA indicates that method does outperform the best baseline.
A.4 QUALITATIVE ANALYSIS HOW IMAGINATION HELPS
We show how imagination can help generating sentences that follow common sense via qualitative analysis in Table 9 10 As comparison, we also show the results from the Node2Text baseline which does not imagines. We organize the results based on 5 (not necessarily exclusive) types of emros made by Node2Text, which include incorrect role attribution 4) failing infer the implicit concepts and 5) misunderstanding the relations between events.
agents, 2) actions 3) objects,
A.5 QUALITY EVALUATION ON THE GENERATED SKGS
Since there are no grountruth SKGs annotated in downstream datasets, we use the silver-standard SKGs reference give rough estimation of the quality of the generated SKGs. We focus
13

---

Published as a conference paper at ICLR 2022
Table 9: Qualitative analysis errors made without imagination and how imagination help fix the errors (Part 1). The left arrow <- indicates the key relations that fix the errors.
Error I (Incorrect Agent)
Example
Example 2
Input concepts
{owner, chase, dog, ball, throw)
[ski, rope, hold, boat, pull)
Text w/o imagination
The dog is chasing the ball and
A woman skis downhill she pulls
throwing it the owner.
boat holding rope.
Text w/ imagination
A dog chases ball being thrown by its owner.
A boat pulls skier who is holding rope.
Generated SKG
(chase, ARGO, dog),
(pull, ARGO, boat),
(chase, ARGI, ball),
(pull, ARGI, person),
(throw, ARGI, hall),
(ski, ARGO, person),
(throw, ARGO, owner) f-
(hold, ARGO, person),
(hold. ARGI, rope)
Error 2 (Incorrect Action)
Example
Example 2
Input concepts
{butter, pot, crack, egg, add)
{stand, tongue, stick)
Text w/o imagination
She adds eggs, crackers, and butter to a pot.
boy stands next to a stick of his tongue.
Text w/ imagination
You crack an egg and add hutter to a pot.
man stands with tongue stick- ing out.
Generated SKG
(crack, ARGO, you),
(stand, ARGI, man),
(crack, ARGI, egg), f-
(man, part, tongue), (stick, ARGO, man), (stick, ARGI, tongue), (stick, ARG2, out)
(add, ARGO, you).
(add, ARGI, butter),
(add, ARG2, pot)
Error 3 (Incorrect Object)
Example 1
Example 2
Input concepts
{hit, bottle, shoe, open, wall)
{wear, talk, phone}
Text w/o imagination
Someone opens his shoe and hits a bottle the wall.
A woman is wearing cell phone and talking the camera.
Text w/ imagination
A man opens a bottle and hits his shoe against a wall.
A man wearing glasses is talking on the phone.
Generated SKG
(open, ARGO, man), (open, ARG1, bottle),
(talk, ARGO, man).
(wear, ARGO, man),
(hit, ARGO, man).
(wear, ARGI, glasses), (- (talk, medium, phone)
(hit, ARGI, shoe), +- (shoe, poss, man). (hit, ARG2, against). (against, opl, wall)
call since the silver-standard SKGs may not cover all the plausible scenes. Our evaluation considers the following three metrics. 1) Average recall of the given concepts (Explicit Concepts) to examine whether an SKG contains all the given concepts. 2) Average recall of the implicit concepts (Implicit Concepts) to examine whether an SKG also contains implicit concepts. The implicit concepts for reference are the nodes from the stader-dtandard SKGs excluding the given concepts. 3) Average re- call of the relations (Relation) to examine the proportion of the referenced relations that covered by the generated SKGs. Here, a relation is considered as correct only if the head concept, relation and the tail concept all match the reference.
The results shown in Table II indicate a fairly good quality of the generated SKGs, which connect all the given concepts for over 99% of the cases and have a large overlap (over 68%) with the silver- standard SKGs. Note that the particular low recall on implicit concepts is due to the fact that there can be many different implicit concepts constitute complete SKG.
14

---

Published as a conference paper at ICLR 2022
Table 10: Qualitative analysis on errors made without imagination and how imagination can help fix the errors (Part 2). The left arrow <- indicates the key relations that fix the errors.
Error 4 (Implicit Concepts)
Example
Example 2
Input concepts
[fill, liquid, machine, bottle)
(lasso, catch, horse, animal, ride)
Text w/o imagination
A machine holding a bottle filled
Animals ride horse that caught lasso.
with liquid.
Text w/ imagination
A man holds bottle filled with liq- uid a machine.
A man riding horse to catch an an- imal with lasso.
Generated SKG
(hold, ARGO, man). (hold, ARGI, bottle), (fill, ARGI, bottle,
(ride, ARGO, man).
(ride, ARGI, horse), (ride, purpose, catch), (catch, ARGO, man), (catch, ARGI, animal), (catch, instrument, lasso)
(fill, ARG2, liquid), (hold, location, machine)
Error 5 (Event Relations)
Example 1
Example 2
Input concepts
(trick, perform, begin, stunt, dance}
{stir, pour, pot, ingredient, begin}
Text w/o imagination
A group of people begin performing
She begins stirring the ingredients in the pot and begins pouring them into the water.
stunt while performing a trick.
Text w/ imagination
A man performs stunts and tricks as he begins to dance.
He pours the ingredients into the pot and begins to stir them.
Generated SKG
(perform, ARGO, man),
(pour, ARGO, he.
(perform, ARGI, stunt), (perform, ARGI, trick), (perform, time, begin), +- (begin, ARGO, man), (begin, ARGI, dance), (dance, ARGO, man)
(pour, ARGI, ingredient), (pour, ARG2, pot).
(begin, ARGO, he), f-
(begin, ARGI, stir),
(stir, ARGO, he).
(stir, ARGI, ingredient)
Table 11: Quality evaluation (recall) on the generated SKGs with silver-standard SKGs as reference.
Dataset
Explicit Concepts
Implicit Concepts
Relation
CommonGen (in-house)
99.81
17.07
72.10
VIST
99.96
35.66
68.19
ROC
99.95
61.86
74.04
A.6 ABLATION STUDY ON INPUT TO V&V
For imagination, the inclusion of context helps the module generate contextualized SKG which is more relevant to the current storyline. To justify this design choice, we conduct an ablation study where we learn an uncontextualized imagination module which only takes concepts as input. The resulting SPICE scores are 47.32 and 48 on VIST and ROC datasets respectively, which are much lower than the results from contextualized I&V (59.21 and 60.63 in SPICE respectively). This demonstrates that the context is critical in generating relevant SKGs which lead to better text generation.
For verbalization, the textual context is important for narrative generation in keeping the storyline consistent. The concept input helps indicate what are the nodes while the SKG input is about the edges. We conduct an ablation study on what input to include for verbalization. The results in shsw that adding concepts as input generally helps improve the performance of our method while adding context is critical for story generation.
15

---

Published as a conference paper at ICLR 2022
Table 12: Ablation study what input is fed the verbalization module.
Input
CommonGen (in-house)
VIST
ROC
SKG-only
33.39
17.13
18.90
Concept + SKG
33.49
27.01
36.42
Context SKG
NA
57.99
58.06
Context Concept + SKG
NA
59.21
60.63
Table 13: Ablation study on using 1) silver-standard SKGs, 2) generated SKGs or 3) both train the verbalization module.
Input SKGs
CommonGen (in-house)
VIST
ROC
Silver-standard
33.19
53.26
60.55
Generated
32.56
58.34
59.82
Silver. Generated
33.49
59.21
60.63
Table 14: Ablation study on the design of the generation process.
Generation Process
VIST
ROC
All-at-once
57.16
54.83
Independent
27.01
36.42
Iterative (I&V)
59.21
60.63
A.7 ABLATION STUDY ON WHAT SKGS TO USE WHEN LEARNING VERBALIZATION
We conduct an ablation study where we use 1) silver-standard SKGs only, 2) generated SKGs only and 3) both types of SKGs during training the verbalization module. The results Table validates that using both types of SKGs yield the best performance of our method.
A.8 ABLATION STUDY ON CONCEPT-DROPOUT
We conduct ablation study where we do not drop any concepts when we train the imagination module. We then apply the imagination module on CommonGen and conduct the experiments. The final performance is 28.28 in SPICE while the system with the imagination module trained with concept-dropout achieves 33.49. This validates that dropping concepts is necessary since in the downstream tasks not all the concepts are provided and the model needs to infer the implicit ones.
A.9 ABLATION STUDY ON THE GENERATION PROCESS
We conduct an ablation study verify that the iterative generation process is effective than 1) generating all the sentences at once, and 2) generating each sentence independently. For baseline I), learn an uncontextualized imagination module which only takes concepts as input and does need the previous generated context. We apply the uncontextualized imagination module to generate all the SKGs at once and then the verbalization module generates all the target sentences once by taking the provided context, all the concept sets and all the SKGs as input. For baseline 2), we still use the contextualized imagination module to generate an SKG at a time. But we learn verbal- ization module which does not take the previously generated sentences input and thus generate each target sentence independently. We conduct the ablation study on the two datasets the con- cept2story task (we do not consider the CommonGen benchmark here since there is only one target sentence to be generated in CommonGen). The results the average SPICE scores from 3 runs shown in Table [4] The two baselines are both outperformed by iterative approach, which verifies that the previously generated sentences are important for both imagination and verbalization and thus the iterative generation process is necessary.
16

---

Published as a conference paper at ICLR 2022
Table 15: Evaluation results (SPICE) with V&V using silver-standard SKGs during inference.
SKGs (inference)
CommonGen (in-house)
VIST
ROC
Silver-standard
41.85
69.34
67.70
Generated
33.49
59.21
60.63
A.10 EXPERIMENTS WITH SILVER-STANDAS SKGS DURING INFERENCE
We report the "oracle" performance of our system using silver-standard SKGs during inference to estimate the upper-bound of method. The result is shown in Table I5
17


Reviews:
[{"review_id": "fhYs4G9_9LS", "reviewer": "Reviewer_9SQy", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths\n- The authors propose a novel imagine-and-verbalize framework: an imagination module learns to construct a contextualized SKG from input concepts; a verbalization module learns to faithfully realize the imagined SKG into natural language. The scene imagination module constructs a structured representation of a plausible scene and formalizes the background knowledge required for the reasoning.\n- The authors collect and harmonize knowledge resources from different domains and modalities, providing a rich auxiliary supervision signal for scene imagination.\n- The experiments on both concept-to-sentence and concept-to-story generation tasks demonstrate the effectiveness of the proposed framework.\n\nWeaknesses\n- The novelty of the whole method may be limited. On the one hand, previous work has used SRL as an intermediate representation, and this work simply replaces it with AMR. On the other hand, pretrained language models are directly adopted in the two stages of the proposed framework. Therefore, there is no innovation in technology.\n- The proposed method needs a good AMR parsing tool to get AMR graphs. But training the AMR tool requires expensive annotation data. Does the author check the parsing quality of the parsing AMR tool?\n- The evaluation of the generated AMR of the first stage was not given in the experiment. Will the generated AMR graph contain a lot of noise?\n- In human evaluation, the authors should also present results of baselines to better demonstrate the improvement of the proposed approach over the baselines.\n", "summary_of_the_paper": "Descriptive sentences about arbitrary concepts generated by neural text generation models are often grammatically fluent but may be not consistent with common sense. This show that such generative commonsense reasoning (GCSR) skills are lacking in state-of-the-art text generation methods. Therefore, the authors propose an Imagine-and-Verbalize (I&V) method, which learns to imagine a relational scene knowledge graph (SKG) with relations between the input concepts, and leverage the SKG as a constraint when generating a plausible scene description. Experimental results demonstrate the effectiveness of I&V in improving language models on both concept-to-sentence and concept-to-story generation tasks.", "summary_of_the_review": "Although the paper is not technically innovative, I do think AMR is a better form of intermediate representation. But my other worry is how to get a good AMR parsing tool, because there is relatively little annotated AMR-text parallel data, and I wonder if the trained AMR tools are ready for use. Overall, this work has a certain contribution. Therefore, I suggest accepting it.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "N6_m324apSe", "reviewer": "Reviewer_y2yo", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Overall this is an interesting paper, which shows convincing improvement over most baselines, and which seems to show more efficient learning relative to all selected baselines. The writing of the paper is on the whole clear, and the reasoning behind the use of the intermediate representations is logical and intuitive. \n\nThere are, however, a few things that remain a bit unclear, making it more difficult to assess exactly how much and why the imagination/SKG component contributes. First, I'm not seeing a description of what is meant by the \"vanilla T5-Large model\", though the improvement over that baseline is used as key evidence in favor of the importance of the imagination module. Does this baseline involve T5-Large being fine-tuned to map directly from concepts to sentences? Does it include the textual context input as well? Understanding these details more clearly will help in assessing what exactly has been demonstrated by the improvement over that baseline.\n\nAlong a similar line, I'm curious about the inclusion of the concepts and context as input both for the imagination module and for the verbalization module. Why is it necessary to include these at both stages? What does the performance look like if only the SKG is given as input to the verbalization module? \n\nMore generally, while it is nice to see the ablations on the specific data types and base model sizes, it seems that it would be more relevant to see some analyses that shed light on how exactly the intermediate representations contribute to improving the model performance. Perhaps an analysis of errors made in the absence of the imagination module that are not made in the presence of the imagination module, or something of this kind. I'm also wondering about the fact that the improvements over some of the strongest baselines are fairly small -- do we think that those models are learning something like these SKGs, but simply doing so in a different (potentially less efficient) way? Discussion of these considerations would improve the clarity of what is being contributed by the model's innovations and why.\n \nTypos / smaller notes:\n\np3 Pre-trainined\np8 moperformance\nFig 5 \"I&V\" vs \"Imagination\" label?", "summary_of_the_paper": "This paper proposes a model for generative commonsense reasoning, taking a set (or sets) of concepts as input and producing a sentence (or sentences) as output. The primary innovation introduced in this model is the use of an intermediate \"scene knowledge graph\" (roughly equivalent to an AMR of the corresponding sentence) which is output by an \"imagination module\" that takes concepts (and textual context) as input. This SKG is then input, along with the concepts and textual context, to a \"verbalization module\" which outputs a sentence (or sentences). The authors show that this model outperforms most of the selected baselines (on tasks of generating single sentences as well as stories), with the exception of KFCNet, which the authors argue is trained on prototypes from a much larger corpus, and is supervised to produce sentences like these prototypes, possibly limiting the model's need to generalize at test. They also do ablations on the type of training data, and the size of base LM, as well as showing that their model learns better with smaller amounts of training data. ", "summary_of_the_review": "Overall the paper is interesting and mostly clear, and it shows improvement over most baselines, with a fairly convincing argument that it is a more efficient learner with smaller amounts of training data. However, I have some lingering questions about what exactly has been demonstrated in terms of importance of the imagination module, and why certain inputs were included in the verbalization module -- and I would have liked to see more analysis discussion on exactly why the SKGs are improving performance, and how we can think about the way this compares to strong alternative models.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "IBMf4bWGGf", "reviewer": "Reviewer_6Z6c", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n\n- The core task is well defined and motivated. \"What is needed to get models to generate through commonsense reasoning.\"\n\n- The paper is well written overall and is easy to follow along, model design choices are mostly explained.\n\n- The paper compares to multiple existing methods and the line up of related work baselines strengthens the work.\n\n\nClarifications/Concerns/Weaknesses:\n\n- My reading of this is that two of the main contributions are in the iterative imagine and verbalize bits. Two critical ablations that don't show up is are those testing: (1) the two-step process: is the two step process even necessary? How does just crunching both transformers in Figure 3 into one perform? (2) the iterative process (Sec. 2.4): going one step further, its not clear that the iterative process helps by going one sentence at a time. E.g. ROC Stories is relatively small in terms of story length and should allow for giving all 4 sentences as input for context for generation.\n\n- The reasoning for the proposed model underperforming KFCNet (Li et al 2021) is also unconvincing. Are there exact numbers on how much test set concept coverage the KFCNet method has in their prototypes vs. your method?\n\n- In the \"Does imagination allow models to learn (faster) with less data?\" section, a methodology with multiple random seeds, splits etc etc. I'm unclear if this methodology of multiple random seeds etc is used for the rest of the experiments as well. Further, no mention of statistical significance is made (except in Table 1) - this would be necessary to back up some of the analysis especially given the rather close margin between multiple ablations/experiments.\n\n- Move more human subject studies details to the main paper from Appendix A2 (at the expense of some of the equations such as the autoregressive decoding loss and model implementation details perhaps). n=3 as well as the relatively low kappa's (on ROC Stories esp) makes the human studies portion impossible to draw conclusions from. Adding qualitative examples of the SKGs generated and the actual stories/sentences that are verbalized from those would *also* be necessary to make any meaningful conclusions.\n\n- I generally hate making novelty based arguments, but in this particular case - the method is very similar to many others that have come before, both in storytelling and other forms of generation. The plan with structured representations and then verabalize paradigm is very well know (as seen simply in the author's related work). Given this and the concerns regarding the experiments expressed earlier, I am not sure how this paper's contributions are situated. What is the contribution here? The imagine-and-verbalize paradigm? The way to imagine using SKGs by decoding linearized graphs? The unified schema of the SKGs themselves?\n\nMinor - Appendix A.1 - please add details on how many relations you're crunching into how many others (this is mostly a clarity issue)", "summary_of_the_paper": "The paper introduces an Imagine and Verbalize two-step method for generative commonsense reasoning. The system first imagines a scene in the form of a linearized SKG and uses that as input for a second model that verbalizes it into (more) human readable text. The method is tested on the Concept2Sentence and Concept2Story tasks and compared to multiple baselines drawn from related works.", "summary_of_the_review": "Overall I think that the paper is well written and has potential but is not ready to be published in its current stage with all the weaknesses mentioned.\n\n\n====Rebuttal Update====\nAs written out in the rebuttal response comment below, I am relatively satisfied by the changes the authors have made and recommend that this paper be accepted.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "C-N6mStZC56", "reviewer": "Reviewer_GjaV", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Strengths**\nThe task of generative commonsense reasoning is interesting but challenging. I like the idea of decomposing imagination and verbalization. The idea of using graphs as better representations (compared to just sets), and the idea of leveraging graph annotations from various modalities make a lot of sense to me. The paper is well written, the experiments are well designed.\n\nThe results are convincing, I appreciate the authors providing significance test in multiple tables, and details about their human evaluation. \"We conducted this study with 3 individuals who are studying Computer Science, though not all individuals were aware of SKG’s prior to the human evaluation.\" this is important information to me. \n\n**Questions and concerns**\n1. In Section 2.3, the authors mention that they randomize the order of concepts during training, does this really help in the GCSR tasks, for instance, compared to using the concept order provided by the dataset (or maybe other ordering strategies, like to sort the concepts so that the model always observe concepts in a consistent order). The concern comes from the question of, given a data point, are the concepts really a *set* or a *list*. It may depend on how the data is collected, for example, humans may have some common preferences extracting keywords from a data source (e.g., a story, an image). \n2. In the same paragraph, the authors mention that they randomly \"dropout\" a subset of the concepts to encourage generalization. Do the authors provide experiments discussing this?\n3. On the top of Section 3, the authors say that during the training of the verbalization module, they use both 1) silver-standard SKG and 2) imagination module-generated SKGs as input. Are there ablation studies on each of these two? \n4. In addition to the AMR tree, did the author try other semantic parsing methods, e.g., semantic role labeling? Do the authors expect to see similar improvement if use AMR? \n5. What are the evaluation scores if the system is equipped with silver-standard SKGs? I.e., testing the verbalization module with SKGs returned by a parser? Comparing this number with Table 2 and Table 3 may make the motivation of having a neural imagination module stronger.\n6. It would be great if a few examples of the model outputs/behaviors, or other qualitative analyses can be provided.\n7. (minor) AMR parsing will specify words (e.g., verbs) with their word sense. For instance, the verb \"want\" will be something like \"want-01\" in the tree. How do the authors deal with these?\n8. (minor) In appendix A.2, the authors describe details about human evaluation, I appreciate it. To me, I would also see a model's surprising score (sense-making output which being absent from the given concept set) since one of the motivations of using an imagination module is to \"imagine\".\n9. (minor, and maybe a bad question) Why not trying graph neural networks to encode/represent SKGs? What extra ability does the Language Model in verbalization module enable.\n", "summary_of_the_paper": "In this work, the authors propose a novel framework for the generative commonsense reasoning (GCSR) tasks. GCSR are tasks where given a set of concepts (or a sequence of set of concepts), a model is required to generate sentences (or a short paragraph) that are both grammatically correct and plausible (follow commonsense). \n\nThe authors propose Imagine-and-Verbalize (I&V), which is a 2-layer system:\n1. The imagination module is a transformer-based seq2seq model, takes a set of concepts (and sometimes text as context) as input, and generates a flattened graph (AMR tree, PENMAN serialization). \n2. The verbalization module is another transformer-based seq2seq model, which takes the concepts, some context, and the generated scene knowledge graph (SKG, flattened) as input, and translate the scene information into natural language sentences (or iteratively, paragraphs).\n\nOn two GCSR datasets, the proposed method significantly outperform strong baselines and prior works. ", "summary_of_the_review": "Overall I believe this is a good paper, to me it is certainly above the bar. While I think the paper can be further improved (see my questions above), I would suggest to accept this paper. \n\nBut of course, correct me if I misunderstood the paper. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}]

Author rebuttal:

Meta-review: Strengths:
* Strong results across two benchmarks
* Ablation study demonstrates importance of components
* Provides improvements especially in low resource settings
* Well-written paper

Weaknesses:
* Novelty of the method may be limited as previous works have explored structured outputs as intermediate plans
* Not clear method will extend to other domains as decent AMR parses are required to train the imagination module, which might work well on the datasets used (e.g., RocStories), but wouldn't work in settings with more complex language
Decision: Accept (Poster)