Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper introduces an Imagine and Verbalize two-step method for generative commonsense reasoning. The system first imagines a scene in the form of a linearized SKG and uses that as input for a second model that verbalizes it into (more) human readable text. The method is tested on the Concept2Sentence and Concept2Story tasks and compared to multiple baselines drawn from related works.
main_review: Strengths:

- The core task is well defined and motivated. "What is needed to get models to generate through commonsense reasoning."

- The paper is well written overall and is easy to follow along, model design choices are mostly explained.

- The paper compares to multiple existing methods and the line up of related work baselines strengthens the work.


Clarifications/Concerns/Weaknesses:

- My reading of this is that two of the main contributions are in the iterative imagine and verbalize bits. Two critical ablations that don't show up is are those testing: (1) the two-step process: is the two step process even necessary? How does just crunching both transformers in Figure 3 into one perform? (2) the iterative process (Sec. 2.4): going one step further, its not clear that the iterative process helps by going one sentence at a time. E.g. ROC Stories is relatively small in terms of story length and should allow for giving all 4 sentences as input for context for generation.

- The reasoning for the proposed model underperforming KFCNet (Li et al 2021) is also unconvincing. Are there exact numbers on how much test set concept coverage the KFCNet method has in their prototypes vs. your method?

- In the "Does imagination allow models to learn (faster) with less data?" section, a methodology with multiple random seeds, splits etc etc. I'm unclear if this methodology of multiple random seeds etc is used for the rest of the experiments as well. Further, no mention of statistical significance is made (except in Table 1) - this would be necessary to back up some of the analysis especially given the rather close margin between multiple ablations/experiments.

- Move more human subject studies details to the main paper from Appendix A2 (at the expense of some of the equations such as the autoregressive decoding loss and model implementation details perhaps). n=3 as well as the relatively low kappa's (on ROC Stories esp) makes the human studies portion impossible to draw conclusions from. Adding qualitative examples of the SKGs generated and the actual stories/sentences that are verbalized from those would *also* be necessary to make any meaningful conclusions.

- I generally hate making novelty based arguments, but in this particular case - the method is very similar to many others that have come before, both in storytelling and other forms of generation. The plan with structured representations and then verabalize paradigm is very well know (as seen simply in the author's related work). Given this and the concerns regarding the experiments expressed earlier, I am not sure how this paper's contributions are situated. What is the contribution here? The imagine-and-verbalize paradigm? The way to imagine using SKGs by decoding linearized graphs? The unified schema of the SKGs themselves?

Minor - Appendix A.1 - please add details on how many relations you're crunching into how many others (this is mostly a clarity issue)
summary_of_the_review: Overall I think that the paper is well written and has potential but is not ready to be published in its current stage with all the weaknesses mentioned.


====Rebuttal Update====
As written out in the rebuttal response comment below, I am relatively satisfied by the changes the authors have made and recommend that this paper be accepted.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: Descriptive sentences about arbitrary concepts generated by neural text generation models are often grammatically fluent but may be not consistent with common sense. This show that such generative commonsense reasoning (GCSR) skills are lacking in state-of-the-art text generation methods. Therefore, the authors propose an Imagine-and-Verbalize (I&V) method, which learns to imagine a relational scene knowledge graph (SKG) with relations between the input concepts, and leverage the SKG as a constraint when generating a plausible scene description. Experimental results demonstrate the effectiveness of I&V in improving language models on both concept-to-sentence and concept-to-story generation tasks.
main_review: Strengths
- The authors propose a novel imagine-and-verbalize framework: an imagination module learns to construct a contextualized SKG from input concepts; a verbalization module learns to faithfully realize the imagined SKG into natural language. The scene imagination module constructs a structured representation of a plausible scene and formalizes the background knowledge required for the reasoning.
- The authors collect and harmonize knowledge resources from different domains and modalities, providing a rich auxiliary supervision signal for scene imagination.
- The experiments on both concept-to-sentence and concept-to-story generation tasks demonstrate the effectiveness of the proposed framework.

Weaknesses
- The novelty of the whole method may be limited. On the one hand, previous work has used SRL as an intermediate representation, and this work simply replaces it with AMR. On the other hand, pretrained language models are directly adopted in the two stages of the proposed framework. Therefore, there is no innovation in technology.
- The proposed method needs a good AMR parsing tool to get AMR graphs. But training the AMR tool requires expensive annotation data. Does the author check the parsing quality of the parsing AMR tool?
- The evaluation of the generated AMR of the first stage was not given in the experiment. Will the generated AMR graph contain a lot of noise?
- In human evaluation, the authors should also present results of baselines to better demonstrate the improvement of the proposed approach over the baselines.

summary_of_the_review: Although the paper is not technically innovative, I do think AMR is a better form of intermediate representation. But my other worry is how to get a good AMR parsing tool, because there is relatively little annotated AMR-text parallel data, and I wonder if the trained AMR tools are ready for use. Overall, this work has a certain contribution. Therefore, I suggest accepting it.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper proposes a model for generative commonsense reasoning, taking a set (or sets) of concepts as input and producing a sentence (or sentences) as output. The primary innovation introduced in this model is the use of an intermediate "scene knowledge graph" (roughly equivalent to an AMR of the corresponding sentence) which is output by an "imagination module" that takes concepts (and textual context) as input. This SKG is then input, along with the concepts and textual context, to a "verbalization module" which outputs a sentence (or sentences). The authors show that this model outperforms most of the selected baselines (on tasks of generating single sentences as well as stories), with the exception of KFCNet, which the authors argue is trained on prototypes from a much larger corpus, and is supervised to produce sentences like these prototypes, possibly limiting the model's need to generalize at test. They also do ablations on the type of training data, and the size of base LM, as well as showing that their model learns better with smaller amounts of training data. 
main_review: Overall this is an interesting paper, which shows convincing improvement over most baselines, and which seems to show more efficient learning relative to all selected baselines. The writing of the paper is on the whole clear, and the reasoning behind the use of the intermediate representations is logical and intuitive. 

There are, however, a few things that remain a bit unclear, making it more difficult to assess exactly how much and why the imagination/SKG component contributes. First, I'm not seeing a description of what is meant by the "vanilla T5-Large model", though the improvement over that baseline is used as key evidence in favor of the importance of the imagination module. Does this baseline involve T5-Large being fine-tuned to map directly from concepts to sentences? Does it include the textual context input as well? Understanding these details more clearly will help in assessing what exactly has been demonstrated by the improvement over that baseline.

Along a similar line, I'm curious about the inclusion of the concepts and context as input both for the imagination module and for the verbalization module. Why is it necessary to include these at both stages? What does the performance look like if only the SKG is given as input to the verbalization module? 

More generally, while it is nice to see the ablations on the specific data types and base model sizes, it seems that it would be more relevant to see some analyses that shed light on how exactly the intermediate representations contribute to improving the model performance. Perhaps an analysis of errors made in the absence of the imagination module that are not made in the presence of the imagination module, or something of this kind. I'm also wondering about the fact that the improvements over some of the strongest baselines are fairly small -- do we think that those models are learning something like these SKGs, but simply doing so in a different (potentially less efficient) way? Discussion of these considerations would improve the clarity of what is being contributed by the model's innovations and why.
 
Typos / smaller notes:

p3 Pre-trainined
p8 moperformance
Fig 5 "I&V" vs "Imagination" label?
summary_of_the_review: Overall the paper is interesting and mostly clear, and it shows improvement over most baselines, with a fairly convincing argument that it is a more efficient learner with smaller amounts of training data. However, I have some lingering questions about what exactly has been demonstrated in terms of importance of the imagination module, and why certain inputs were included in the verbalization module -- and I would have liked to see more analysis discussion on exactly why the SKGs are improving performance, and how we can think about the way this compares to strong alternative models.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: In this work, the authors propose a novel framework for the generative commonsense reasoning (GCSR) tasks. GCSR are tasks where given a set of concepts (or a sequence of set of concepts), a model is required to generate sentences (or a short paragraph) that are both grammatically correct and plausible (follow commonsense). 

The authors propose Imagine-and-Verbalize (I&V), which is a 2-layer system:
1. The imagination module is a transformer-based seq2seq model, takes a set of concepts (and sometimes text as context) as input, and generates a flattened graph (AMR tree, PENMAN serialization). 
2. The verbalization module is another transformer-based seq2seq model, which takes the concepts, some context, and the generated scene knowledge graph (SKG, flattened) as input, and translate the scene information into natural language sentences (or iteratively, paragraphs).

On two GCSR datasets, the proposed method significantly outperform strong baselines and prior works. 
main_review: **Strengths**
The task of generative commonsense reasoning is interesting but challenging. I like the idea of decomposing imagination and verbalization. The idea of using graphs as better representations (compared to just sets), and the idea of leveraging graph annotations from various modalities make a lot of sense to me. The paper is well written, the experiments are well designed.

The results are convincing, I appreciate the authors providing significance test in multiple tables, and details about their human evaluation. "We conducted this study with 3 individuals who are studying Computer Science, though not all individuals were aware of SKG’s prior to the human evaluation." this is important information to me. 

**Questions and concerns**
1. In Section 2.3, the authors mention that they randomize the order of concepts during training, does this really help in the GCSR tasks, for instance, compared to using the concept order provided by the dataset (or maybe other ordering strategies, like to sort the concepts so that the model always observe concepts in a consistent order). The concern comes from the question of, given a data point, are the concepts really a *set* or a *list*. It may depend on how the data is collected, for example, humans may have some common preferences extracting keywords from a data source (e.g., a story, an image). 
2. In the same paragraph, the authors mention that they randomly "dropout" a subset of the concepts to encourage generalization. Do the authors provide experiments discussing this?
3. On the top of Section 3, the authors say that during the training of the verbalization module, they use both 1) silver-standard SKG and 2) imagination module-generated SKGs as input. Are there ablation studies on each of these two? 
4. In addition to the AMR tree, did the author try other semantic parsing methods, e.g., semantic role labeling? Do the authors expect to see similar improvement if use AMR? 
5. What are the evaluation scores if the system is equipped with silver-standard SKGs? I.e., testing the verbalization module with SKGs returned by a parser? Comparing this number with Table 2 and Table 3 may make the motivation of having a neural imagination module stronger.
6. It would be great if a few examples of the model outputs/behaviors, or other qualitative analyses can be provided.
7. (minor) AMR parsing will specify words (e.g., verbs) with their word sense. For instance, the verb "want" will be something like "want-01" in the tree. How do the authors deal with these?
8. (minor) In appendix A.2, the authors describe details about human evaluation, I appreciate it. To me, I would also see a model's surprising score (sense-making output which being absent from the given concept set) since one of the motivations of using an imagination module is to "imagine".
9. (minor, and maybe a bad question) Why not trying graph neural networks to encode/represent SKGs? What extra ability does the Language Model in verbalization module enable.

summary_of_the_review: Overall I believe this is a good paper, to me it is certainly above the bar. While I think the paper can be further improved (see my questions above), I would suggest to accept this paper. 

But of course, correct me if I misunderstood the paper.
review_title: null

Meta-review: Strengths:
* Strong results across two benchmarks
* Ablation study demonstrates importance of components
* Provides improvements especially in low resource settings
* Well-written paper

Weaknesses:
* Novelty of the method may be limited as previous works have explored structured outputs as intermediate plans
* Not clear method will extend to other domains as decent AMR parses are required to train the imagination module, which might work well on the datasets used (e.g., RocStories), but wouldn't work in settings with more complex language
Decision: Accept (Poster)