DIVE: Towards Descriptive and Diverse Visual Commonsense Generation
Authors: Jun-Hyung Park
Hyuntae Park
Youjin Kang
Eojin Jeon
SangKeun Lee
Venue: EMNLP 2023 Main (2023)
Keywords: Visual commonsense generation
descriptive and diverse text generation
commonsense inference
vision-language model

Abstract: Towards human-level visual understanding, visual commonsense generation has been introduced to generate commonsense inferences beyond images. However, current research on visual commonsense generation has overlooked an important human cognitive ability: generating descriptive and diverse inferences. In this work, we propose a novel visual commonsense generation framework, called DIVE, which aims to improve the descriptiveness and diversity of generated inferences. DIVE involves two methods, generic inference filtering and contrastive retrieval learning, which address the limitations of existing visual commonsense resources and training objectives. Experimental results verify that DIVE outperforms state-of-the-art models for visual commonsense generation in terms of both descriptiveness and diversity, while showing a superior quality in generating unique and novel inferences. Notably, DIVE achieves human-level descriptiveness and diversity on Visual Commonsense Graphs. Furthermore, human evaluations confirm that DIVE aligns closely with human judgments on descriptiveness and diversity.

DIVE: Towards Descriptive and Diverse Visual Commonsense Generation
Jun-Hyung Park Hyuntae Park Youjin Kang" Eojin Jeon" SangKeun Lee
BK21 FOUR R&E Center for Artificial Intelligence, Korea University, Seoul, Republic of Korea DDepartment of Artificial Intelligence, Korea University, Scoul, Republic of Korea
3 Department of Computer Science and Engineering, Korea University, Seoul, Republic of Korea {irish07, pht0639, yjkang10, skdlcm456, yalphy)@korea.. ac.
Abstract
Person2
Towards human-level visual understanding, vi- sual commonsense generation has been intro- duced to generate commonsense inferences be- yond images. However, current research on vi- sual commonsense generation has overlooked an important human cognitive ability: generat- ing descriptive and diverse inferences. In this work, propose novel visual commonsense generation framework, called DIVE, which aims to improve the descriptiveness and diver- sity of generated inferences. DIVE involves two methods, generic inference filtering and contrastive retrieval learning, which address the limitations of existing visual commonsense resources and training objectives. Experimental results verify that DIVE outperforms state-of- the-art models for visual commonsense gen- eration in terms of both descriptiveness and diversity, while showing a superior quality in generating unique and novel inferences. No- tably, DIVE achieves human-level descriptive- ness and diversity on Visual Commonsense Graphs. Furthermore, human evaluations con- firm that DIVE aligns closely with human judg- ments on descriptiveness and !
Person
C
(Event) Person2 stands beside Personl and listens intently (Place) music
(Type) Before, Person2 needed to
"walk Person!"
"have conversation" " "stand behind Person!"
Existing model
"meet Personl the music store" "hear Personl play guitar" "begin talking with Personl about music"
0. DIVE
"work in the music store with Personl"
"be interested in Personl's guitar playing" "see Personl playing" "listen the music"
Human
Figure Comparison commonsense inferences from models and humans. Blue words represent key details.
1 Introduction
well as the present intents of characters. Recent works on visual commonsense generation (Park et al., 2020; Xing et al., 2021) have progressed to develop vision-language models capable of gener- ating more plausible and relevant inferences.
Humans possess cognitive ability to reason about the rich and complex stories beyond a given visual scene, based on their background commonsense knowledge. Visual commonsense reasoning is a key to this cognition-level visual understanding (Zellers et al., 2019), which helps humans compre- hend the interactions around them. As research to- wards the human-level visual understanding of ma- chines, visual commonsense generation (Park et al., 2020) has been introduced. This challenging task aims generate textual commonsense inferences about potential antecedents and consequences, as
Despite considerable efforts visual common- sense generation, an important aspect of humans' innate cognitive ability has been overlooked previous studies: humans can make descriptive and diverse inferences by capturing important, spe- cific, and detailed information within visual scene. This ability necessary for making precise and in- formative inferences about various possible scenar- ios in the world, but it is lacking in existing models. Figure I illustrates a case where model-generaan inferences still fall short of humtnewritten infer-
"These authors contributed equally to this work. !Our code and dataset are available at https: //gi thub. com/ hon/Park-ing-- lot/DIVE -ooearrr-ing--otDDVV

---

ences in terms of descriptiveness and diversity. Ex- isting models often ignore key details in given scene, leading to the generation of similar and generic inferences such as "walk up to Person/" and "stand behind Person 1 that could happen in most contexts and provide minimal specific detail. In contrast, humans can create more descriptive and diverse inferences like "work in the music store with Person / " by considering the image's details, such as many instruments displayed behind and two men wearing employee ID cards.
generation quality of unique and novel inferences. We further conduct human evaluations on the plau- sibility, descriptiveness, and diversity of generated inferences, which confirm that DIVE aligns closely with humans' judgment of descriptiveness and di- versity. Our main contributions are as follows,
We propose a novel framework for visual com- monsense generation, called DIVE, which en- hances the capability of vision-language mod- els to generate descriptive and diverse infer- ences about visual scenes. To the best of our knowledge, this is the first work to address de- scriptiveness and diversity in visual common- sense generation, providing deep insights.
We observe that this deficiency can be largely attributed to the skewed distribution visual com- monsense resources. Such resources, typically crowd-sourced, often involve many generic infer- ences as labels, because humans may not use de- tailed information from given visual scene when annotating (Berg et al., 2012; Hessel al., 2022). For example, more than 60% of images in Visual Commonsense Graphs (VCG) (Park et al., 2020) involve generic inferences as labels?. As models repeatedly learn these generic inferences, they tend to generate inferences that vaguely describe a sit- uation, failing to capture detailed information that specifies given scene. This limitation restricts deeper understanding of visual information by existing vision-language models.
We develop generic inference filtering and contrastive retrieval learning methods facil- itate descriptive and diverse visual common- sense generation of vision-language models on the skewed visual commonsense resources.
Our extensive experiments verify that DIVE outperforms state-of-the visual common- sense generation models in terms of the de- scriptiveness and diversity on VCG.
In this paper, we introduce DIVE (Descriptive and dlverse Visual commonsense gEneration), a novel visual commonsense generation framework for improving the descriptiveness and diversity of commonsense inferences generated by vision- language models. Firstly, we construct a balanced visual commonsense graph from VCG, using carefully designed filtering method that removes generic inferences, with focus on the semantic concentration of images utilizing the representa- tions of CLIP (Radford et al., 2021). Furthermore, we propose a new contrastive retrieval learning method that facilitates a model to recognize spe- cific details of an image. To verify the efficacy of DIVE, conduct experiments VCG (Park al., 2020) with popular generative models (Lewis al.. 2020; Li al., 2022b). Our experiments verify that DIVE generates more descriptive and diverse inferences compared with state-of-the-art visual commonsense generation models. Notably, DIVE achieves human-level descriptiveness and di- versity scores on VCG, significantly improving the
Related Work
Visual commonsense reasoning. With the goal of reasoning beyond visual recognition, the commu- nity has actively explored several visual common- sense reasoning tasks. Zellers et al. (2019) have proposed a visual commonsense reasoning bench- mark to test if a model can identify an answer with rationale, given a question that requires thorough understanding of images based on commonsense knowledge. Hessel et al. (2022) have proposed abductive reasoning benchmark beyond literal image contents to evaluate the capacity of models to retrieve relevant inferences, localize evidence, and compare plausible inferences. et al. (2022a) have introduced a video QA benchmark that re- quires understanding of evidences and common- sense reasoning over time. Yu et al. (2022) have proposed an audiovisual commonsense reasoning benchmark focusing physical knowledge. which requires understanding on multi-sensory inputs. However, these benchmarks evaluate models in a question answering format, limiting the evaluation of the models' capability to generate commonsense inferences. To address this, Park et al. (2020) have
In VCG, 61% of images involve the 100 most frequent inference results as their lahels, which are predominantly generic, like "talk to Personl" and "cat dinner".

---

Agrement
Minimn Agreenzer
Octa clner
Image Projection Layer
Test Projection Layer
hooltingg
lock
Vislan- Langung Enender
Visua-I angrage Decader
Get clost
X
Filtoiin
look
Migh Prepuracy
Positro infommn
Speak
Lose somamil concestration
Postivve image
Negnive
bolling
Saepling Heages related cestsoo deseription
Staphhngg on inference uniguely relisedd the posttiv tauge
(a) Generie Inference Filtering
(b) Contrastive Retrieval Learning
Figure 2: Illustration of DIVE. (a) Generic Inference Filtering: Filtering inferences with high frequency and low semantic concentration of related images. (b) Contrastive Retrieval Learning: Learning maximize the agreement between pair of an image and its unique corresponding inference. Events and places are omitted for clarity.
introduced a visual commonsense generation task with VCG, which asks models generate infer- ences about potential antecedents, consequences, and the intents of people based on images and their corresponding textual descriptions. This requires models to generate free-form textual descriptions inference results, which more challenging and in line with the process of humans reasoning (Jung et al., 2022). As an approach to visual common- sense generation, Park et al. (2020) have extended pre-trained language models like GPT-2 (Radford al., 2019) by fine-tuning them process visual information. KM-BART (Xing al., 2021) has constructed vision-language datasets for continual pre-training of BART (Lewis et al., 2020), aiming to enhance its commonsense knowledge with ad- ditional data. Recent studies have proposed large- scale pre-trained vision-language models (Li et al., 2022b; Wang et al., 2022; Lu et al., 2023; Han al., 2023), which have shown promising results across various vision-langn Saaiunnl tasks. Several stud- ies have extended the modalities of rision-languee transformers to process audio and time information for a holistic understanding from all our senses (Zellers et al., 2022: Zong and Sun, 2023). In this work, based on VCG (Park et al., 2020) and vision-language models (Xing et al., 2021; Li et al., 2022b), propose novel framework focusing descriptive and diverse commonsense generation, which is important but overlooked in prior research on visual commonsense reasoning.
Descriptive and diverse text generation. Gen- erating meaningful and informative text has been challenging goal in many NLP fields including response generation, image captioning, and scene graph generation (Li et al., 2016; Dai and Lin, 2017: Zhao et al., 2017; :; Jiang and de Rijke, 2018; Luo et al., 2018: Wu et al., 2020; Tang et al., 2020). these works, several problems have been identified in current text generation frameworks. First, gener- ation models often prefer "safe" output sentences that very figh-frequeccy expressions, and they tend describe only the obvious facts while ig- noring key details (Li et al., 2016). Second, the standard training objective function for text gener- ation can promote the generation generic text without diversity (Li et al., 2016). Third, conven- tional metrics for text generation, such as BLEU and CIDEr, tend to give high scores to inferences with very common n-grams, as they prioritize n- gram overlap (Liu al., 2019). Thus, the first exploration of descriptive and diverse common- sense inferences, we propose filtering and training methods to prevent models from favoring "safe" sentences. Additionally, we construct datasets for evaluation, introduce various metrics, and conduct human evaluations to particularly evaluate descrip- tiveness and diversity of generated inferences.
3 Methodology
In this section, introduce DIVE, framework designed for descriptive and diverse commonsense generation. First, we propose a generic inference filtering method that balances the distribution

---

where F()) represents an image encoder, g(C)) is the set of images related commonsense descrip- tion Cj. When calculating the similarity, we utilize the average of feature representations the final hidden layer of CLIP (Radford et al., 2021). The measured value indicates how closely the related images lie in the feature space, that is, how specific an inference is.
(Event) 2 and I are sitting on the floor with (Place) lising
(Eventy stares and (Place) living
Dedore
See what is holding
Froqurncy: 3
Using this semantic concentration S(C₁) and the frequency [G())) of commonsense description Cj. we identify and filter out generie inferences for each commonsense description. Figure illustrates an example to measure the semantic concentration and frequency on visual commonsense graphs. In- spired by the frequency-basee filtering method for words (Mikolov et al., 2013), we calculate the fil- tering probability Py(Cg) for commonsense de- scription Cj, which is defined as follows:
Feature Space
Semantic concentration: 0.6
(Event) bending (Place) a drowing room
Figure 3: Illustration of measuring the semantic concen- tration and frequency of inferences.
t x S(C,) [g(C,)
P)(C)) -1
(2)
visual commonsense resources (Section 3.1). Then, we propose contrastive retrieval learning method that facilitates vision-language models to identify information specific to a given image (Section 3.2). Figure illustrates the overall procedure of DIVE.
where is a threshold. Finally, we deterministically filter [P(C))) inferences between Cý and related images with the lowest average similarity to the others.
3.2 Contrastive Retrieval Learning
3.1 Generic Inference Filtering
Although generic inference filtering can effectively reduce the skewness of the distribution, ob- serve that over-filtering negatively impacts the qual- ity of models' inference results, primarily due the reduction the number of training examples. Additionally, it has been noted that a specifically designed training objective function to improve descriptiveness and diversity is beneficial, as the standard generative objective function may lead to generic generation (Li et al., 2016; Luo et al.. 201 8). These two observations underscore the need to develop novel training methods that improve de- scriptiveness and diversity in visual commonsense generation, used in conjunction with the filtering method.
Since the skewed distribution of VCG can cause vision-language models to favor generic inferences, we construct a balanced VCG based on our generic inference filtering method. We identify generic in- ferences based their frequencies and how their related images are semantically concentrated, be- cause generic inference expected to be frequent and associated with a broader range of images.
Given a visual commonsense graph (I, E,, R), where I, E. P. and C denote sets of images, events, places, and commonsense descriptions, respectively, R is the set of edges in the graph comprising visual commonsense in- ferences Rij (1i, E,, Pt, Cy)- Here J; E E, P.C, E C, and a reasoning type r E (before, after, intent). Then, we measure the semantic concentration of images related to com- monsense description. Specifically, we measure the average cosine similarity of feature representations of the related images as follows:
Here we propose a new contrastive retrieval learning method that encourages models to gen- erate descriptive and diverse inferences about an image. The key motivation of our method is that models need to recognize detailed objects and inter- actions within images to generate descriptive and diverse inferences. Our method trains model retrieve the original image from which a given in- (1) ference derived, within a set of similar images,
S(C,)) Emeric,) Everie,) sim(F(x), [

---

in a contrastive manner. This approach is expected facilitate models to identify the differences in detailed objects and interactions among similar im- ages, thereby aligning with our motivation.
where is a non-negative hyper-paramettr for bal- ancing the objective functions. is noteworthy that we randomly select one inference for the loss cal- culation if multiple inferences are associated with an image hp- We construct for each example in batch and if a uniquely related commonsense de- scription does not exist, we exclude the contrastive loss.
First, we construct a set of similar images C ((C)) that share the same commonsense descrip- tion Cj as inference. To identify similar images, we sample images that share the same inference re- sult, based our intuition that they have semantic similarity that leads the same inference result. Subsequently, we identify a pair of an image and its corresponding commonsense description (htp sp), where hp E H and Sp is a commonsense descrip- tion uniquely related to hp among the images in H. We consider hp positive image and 8p as positive inference, while treating the other images h& E H negative images. We then define agreement function based on cosine similarity as follows:
Experiments
In this section, we demonstrate the effectiveness of DIVE by comparing with existing methods.
4.1 Experimental Setup
Dataset. We conduct the experiments the VCG dataset (Park et al., 2020), which is a large- scale visual commonsense graph. We train models with DIVE on the filtered VCG training set. addition, we evaluate models on the original VCG validation set, the unique VCG validation set, and the novel VCG validation set. The unique VCG validation set is subset of the original set that con- sists of inferences with commonsense descriptions that appear once the original set. The novel VCG validation set a subset of the original set that con- sists of inferences with commonsense descriptions that do not appear in the training set. We expect that the unique and novel subsets predominantly contain specific inferences, since they exclude du- plicate examples. For both subsets, we discard the inferences of images with fewer than five common- sense descriptions. The statistics of the dataset are reported in Appendix A.
(h.,s) exp(sim(Va. T.)).
(3)
where VA and T, denote the feature representations of an image h and text s from vision- languaee model being trained, which are extracted by av- eraging the output feature vectors from the final layers of the encoder and decoder, respectively. is worth noting that obtain the image and text representations by averaging the projected repre- sentations from the image encoder and text decoder a vision-lane languggg model, respectively. Finally, we define our contrastive retrieval loss as follows: o (hip 8p)
Chip Spr H) log
(4)
Σ o (hs, sp)
Based on the proposed method, we aim to train models to consider unique components that lead to specific inference results, rather than common components that lead to more generic inference results shared by multiple images.
Baselines. We mainly compare our results with those of VisualCOMET (Park al., 2020), KM- BART (Xing et al., 2021), and BLIP (Li al., 2022b). VisualCOMET (Park et al., 2020) ex- tends pre-trained GPT-2 model (Radford et al., 2019) with 126 million parameters to incorporate visual and textual information. KM-BART is based on pre-trained BART-base model (Lewis et al., 2020) with 141 million parameters and conducts additional pre-training with image captioning data. BLIP (Li et al., 2022b) is a pre-trained generative vasion-language transformer with 247 million pa- rameters. All the baselines are fine-tuned on VCG following the tieint settings specified in their original papers.
We integrate our contrastive retrieval loss with the original language modeling loss of visual com- monsense generation (Park et al., 2020). For a given image E H, we can identify correspond- ing ground-truth inference s = [wi, W2, wx) sequence of tokens. Then, the original loss is defined as follows:
- log Plus|weeih h). (5)
Elo
The final objective function is as follows:
C(hp, Sp. H) = Corg(hp: sp) + >Ceni(hp Sp: H).
Implementation details. We fine-tune both vision-language BART and BLIP on VCG using
(6)

---

Model
Length
Yngve
Dist-2
Dist-3
R@1
R@05
R
Entropy
Unique
Novel
VisualCOMET
4.733
7.68
127K
29.56
53.76
64.38
19.38
42.28
45.24
KM-BART
4.614
7.37
159K
37.38
62.03
71.75
18.76
57.61
38.57
BLIP
4.659
7.50
77K
174K
66.21
88.52
93.52
58.48
40.82
DIVERART (ours)
5.156
8,88
84K
207K
51.40
77.47
85.02
21.09
54.20
DIVERLIP (ours)
5.223
8.80
93K
221K
77.14
94.78
97.38
20.91
Human
4.858
8.15
93K
190K
20.71
74.34
54.98
Table 1: Evaluation of descriptiveness and diversity the original VCG validation set.
Model
Length
Yngve
Dist-2
Dist-3
R@i
R
R@
Entropy
Unique
Novel
VISUTICOMET
4.811
8.20
7.1K
10.1K
34.65
57.27
67.12
19.20
79.33
48.63
KM-BART
4.638
7.44
8.3K
11.9K
23.43
60.19
69.56
18.83
91.33
42.67
BLIP
4.646
7.38
8.2K
11.7K
66.74
87.42
92.03
18.43
89.44
43.45
DIVERART (ours)
5.169
8,94
10.1K
14.9K
31.78
74.55
82.36
21.09
96.88
56.74
DIVEBLIP (ours)
5.098
8.82
10.1K
15.0K
77.01
94.94
97.17
20.93
95.34
57.33
Human
5.792
10.39
14.9K
21.3K
26.11
100.0
100.0
Table 2: Evaluation of descriptiveness and diversity the unique VCG validation set.
Model
Length
Y Yngve
Dist-2
Dist-3
R@1
R@5
R@ 10
Entropy
Unique
Novel
VisualCOMET
4.788
8.26
4.6K
5.8K
46.46
69.77
78.79
19.52
71.55
45.75
KM-BART
4.637
7.41
6.6K
32.18
70.37
78.37
86.45
40.47
BLIP
4.614
7.47
5.0K
6.4K
72.93
92.18
95.63
18.84
84.61
41.20
DAVERART (ours)
5.165
8.85
8.1K
38.64
81.80
89.32
21.24
97.77
59.12
DIVERL/P (ours)
5.186
8,80
6.0K
8.1K
85.86
97.29
98.42
95.48
59.69
Human
5.515
11.32
9.5K
12.3K
24.44
100.0
100.0
Table 3: Evaluation of descriptiveness and diversity the novel VCG validation
our DIVE framework. DIVE with Dision-language BART (DIVEHARR) is based on the BART archi- tecture (Lewis et al., 2020) with several modifica- tions to process vision-language inputs consistent with KM-BART (Xing et al., 2021). We use pro- jected Region of Interest (Rol) feature representa- tions of image from the pre-trained Faster R- CNN (Ren al., 2015) as vision input vision- language BART. For each image, Faster R-CNN detects several objects and generates their bounding boxes and classification results as Rol features. We extract the feature representations fed into the final classification layer of Faster R-CNN. The extracted representations subsequently projected to fit the embedding dimension of the language mod- els. DIVE with BLIP (DIVEBLIP) uses the BLIP architecture (Li al., 2022b) while keeping the visual encoder layers frozen. We use the input for- mats consistent with Xing et al. (2021). We set the filtering threshold to 10 and extract two sampled images from each batch example for contrastive retrieval learning. We generate five inferences per example using nucleus sampling (Holtzman al., 2020) with p 0.9. All our experiments are con- ducted on six NVIDIA RTX A6000 GPUs. De- tailed training hyper-parameteer are specified in Appendix E.
Metrics. We employ a variety of automatic eval- uation metrics following previous works (Liu et al., 2019; Park et al., 2020) to evaluate the descrip- tiveness and diversity. We evaluate the sentence length (Length), syntactic complexity (Yngve (Yn- gve, 1960)), number of distinct n-grams in the whole generated inferences (Dist-2 and Dist-3 (Xu et al., 2018)), image retrieval performance given the generated inference (R@1 1, R@5, and R@ (Liu et al., 2019)), word-level entropy that mea- sures the average log probability of the uni-gram words of a generated inference in the training set (Entropy (Mou et al., 2016)), percentage of unique inferences within the generated inferences (Unique (Park al.. 2020)), and percentage of novel infer- ences that are seen in the training set (Novel (Park et al., 2020)). set of more descriptive and diverse inferences will show higher scores terms of the chove-mentionel metrics. We additionally employ conventional metrics to evaluate the qual- ity of generated inferences including BLEU (Pa- pineni et al., 2002), METEOR (Denkowski and Lavie, 2014), CIDEr (Vedantam al., 2015), and SPICE (Anderson al., 2016). However, conven- tional metrics are limited to evaluate accuracy of descriptive and diverse inferences (Li et al., 2016), because they give a high score to the inferences that

---

Model
B-2
M
DIVERART
Plausible
Deseriptive Lose
Diverse
VisualCOMET
13.50
11.37
18.28
5.53
Win Lose
Win
Win Lose
KM-BART
14.27
11.22
21.22
6.85
VisualCOMET
61.7 38.3
54.7 45.3
68.9 31.1
BLIP
13.93
11.33
20.81
6.90
KM-BART
59.8 40.2
56.0 44.0
56.7 43.3
DIVERART (ours)
13.33
11.48
20.26
7.33
DIVEBLIP (ours)
12.86
11.39
19.07
6.90
Table 7:
Human evaluation on the original VCG valida-
tion set.
Table 4: Evaluation of generation quality the original
VCG validation set.
baselines using conventional evaluation metries, shown in Tables 4-6. On the original VCG valida- tion sct, our models show a comparable quality to the baselines. However, since the results are de- rived from the original VCG validation set, which involves many generic inferences, these results may be affected by the limitations of the conventional metrics mentioned in Section 4.1. To more pre- cisely evaluate the generation quality of descriptive and diverse inferences, we further conduct experi- ments on the unique and novel VCG validation set. In these experiments, we observe that our DIVE models exhibit significantly better generation qual- ity than the baselines by improving the scores up to 17.3% in terms of SPICE. These results demon- strate that DIVE has better capability for gener- ating descriptive and diverse inferences compared with existing models.
Model
B-2
M
VisualCOMET
18.16
11.94
18.11
5.50
KM-BART
17.80
11.26
19.65
7.36
BLIP
16.69
11.11
17.92
7.03
DIVEBART (ours)
18.83
12.52
21.20
8.08
DIVEBLIP (ours)
17.96
12.32
19.70
7.45
Table 5: Evaluation of generation quality the unique VCG validation set.
Model
B-2
M
S
VisualCOMET
18.07
12.23
16.78
5.34
KM-BART
18.41
11.70
19.43
7.13
BLIP
17.15
11.66
17.66
6.82
DIVE (ours)
19.04
12.23
22.52
8.36
DIVEgLIP (ours)
18.01
11.91
20.17
7.54
Table 6: Evaluation of generation quality on the novel VCG validation set.
have many n-gram overlaps with ground-truth infer- ences, which are predominantly generic in visual commonsense resources. Thus, we further conduct human evaluations on the generated inferences.
4.3 Human Evaluation Results
We present human judgments on the plausibility, descriptiveness, and diversity the generated infer- ences. We conduct human evaluations with workers from Amazon Mechanical Turk3.
4.2
Main Results
We first evaluate the descriptiveness and diver- sity in visual commonsense generation of vision- language models. In Table 1, we compare our DIVE models with state-of-the-and visual common- sense generation models on the original VCG vali- dation set. We observe that our DIVE models out- perform the baselines in all evaluation metrics for descriptiveness and diversity. Particularly, DIVE models reach human- deseriptiveness and di- versity the original VCG validation set. These results confirm that our DIVE framework effec- tively augments vision- languaee models with the capability for generating descriptive and diverse commonsense inferences, showing significant im- provements over existing vision-language models. In addition, shown in Tables 2 and 3, our DIVE models consistently outperform the baselines in terms of descriptiveness and diversity the unique and novel VCG validation sets.
Following Xing et al. (2021), we generate 450 inferences with three different models, including DIVEBART, VisualCOMET (Park al., 2020). and KM-BART (Xing al., 2021), for pair-wise comparison. We generate five pairs of inferences for each inference type. For each example, we construct pairs of inferences generated by DIVE and one of the baselines, and sets of all inferences generated by each model. Then, we ask three an- notators to choose a better inference based on the following three metrics: 1) plausible: which infer- seems more plausible and reasonable to image, 2) descriptive: which inference explains the image more informatively and specifically, and 3) diverse: which set of inferences seems more diverse in meanings and expressions.
The results are shown in Table 7. For plausibility, our DIVEBART model outperforms the statc-of-
*We selected annotators who achieved 299% on the quali- fication HIT before. The workers were paid proportionally at $0.15 per example, which resulted in at least 616/r.
To evaluate the quality of visual commonsense generation, we compare our DIVE models with the

---

GIF
CRI.
SPICE
R@
Unique
0.00
VisualCOMET
51.40
76.09
KM-BART
48.87
73.49
DIVERART
32.93
56.56
Frequency Railo
DIVE
7.19
37.38
58.12
Table 8: Ablation study the original VCG validation set. GIF and CRL denote generic inference filtering and contrastive retrieval learning, respectively.
0.00
0.00
15
10
21
24
27
20
33
36
38
Entropy
the-art visual commonsense generation models in approximately 60% of cases. This result suggests that humans generally perceive inferences gener- ated by DIVE better than those from the baselines on the original VCG validation set, also revealing the limitations of conventional metrics for quality evaluation. For descriptiveness and diversity, observe consistent results of winning in approxi- mately 55% and 55% 70%, respectively. This demonstrates that DIVE aligns more closely with human judgments on descriptiveness and diversity.
Figure 4: Distribution of generated inferences in relation to word-level entropy.
5.2 Informativeness of Inferences
We analyze the amount of information contained in generated inferences by measuring word-level entropy (Mou et al., 2016). Figure shows the distribution of generated inference on the original VCG validation set in relation to word-level en- tropy. The y-axis represents the ratio of the number of generated inferences for the corresponding in- terval of entropy in the x-axis. Each value k in the X-axis represents the interval between k 1.0 and 1.0. We can observe that DIVE generates infer- ences with relatively high entropy, which implics the improvements their informativeness.
5 Analysis
In this section, we conduct analyses of the compo- nents and results of DIVEBART.
5.1 Ablation Study
To better understand the contributions of each com- ponent in DIVE to performance improvements, conduct ablation studies on generic inference filter- ing and contrastive retrieval learning. The results are shown in Table 8. We find that training models without our filtering method results in significant degradation in the R@ and Unique scores, which highlights that balancing the distribution of the vi- sual commonsense resources is crucial for generat- ing descriptive and diverse inferences. In addition, our contrastive retrieval learning method univer- sally improves the three metries when combined with the filtering method, showing its contributions the improvements in generation quality, descrip- tiveness and diversity. Nevertheless, the contrastive retrieval learning method degrades the performance when applied alone. We speculate that this be- cause wide range of images can be frequently sampled as negative ones if generic inferences are not climinated, failing to meet the motivation of the method that trains models to recognize the detailed differences among similar images. This observa- tion also shows that the components of DIVE are complementary to each other for the performance improvements.
5.3 Qualitative Analysis
We present qualitative examples of DIVE com- pared baselines in Figure 5. It demonstrates that DIVE can generate more descriptive and di- verse inferences compared to the baselines. As can be observed from the figure, DIVE effectively gen- erates unique and novel inferences applicable to the given situations, utilizing specific expressions related the image, such as "disturb", "thirst", etc.
In contrast, the baselines frequently generate sim- ple, generic, and seen descriptions. Interestingly, DIVE sometimes generates more descriptive and diverse inferences compared to human annotations like the Intent inferences in Figure (a). This fur- ther implies that existing automatic evaluation can underestimate the scores of DIVE due to lexical differences from human annotations.
Despite the promising results, DIVE generates some irrelevant inferences to the context even if the detailed information is explicitly given. This shows that vision language models still lack some commonsense knowledge and reasoning ability to accurately reason over recognized context.

---

Before
Intent
After
be exhausted
him
down chair fox
take
wake ruddznly
nap
be rupped by wake and starlled
Human
take break from stadies
relax in the citair
Person
have doown the choir
get nosl
wake when aprrescchigg
sick
go swway
taken tired.
over on
DIVE
nightts sleep nightt' sleep
by
fall asleep
wake refreshed
pit singlesece
nap same rest
he wolen by
decide down
fall sleap
KM-BART
get into connoorablle
waker fooling refreshed fall asleop
(Event) Personi is sleeping a chair
postiia
pit on lic
(Place) loft
Lae down
spead time away froms his fumily
call over
Ancoik
taok
VisuaCCOMFT
get into the choir be tired
fall asleep
sneak poeks at their phone
end the day quickly
noceive his phono call from be questioned by
2 a job
get soinz rest
Before
Intent
After
be too eshausted leave
chink some soda his
joke around with his friends
plop down as the chair for
get bullied by the other kids slart gailing hullied by and pet his drink pourral over his head
Human
nap
take a break from studies
relox in the chair
buy the cup
carry his purchases home
watch and pass by
Person3
purcitase drink the
spend break at the sture
make another drink.
Persond
notice
purie
sidewalk
drink
purchaases hoose
gosle
DIVE
sncpicioons
quench of'
OUT
pay cup drink
and
home with their purchases
dazide where wall line line buy the drink at the stone
lerve and go home home
leavee the store
tike a drink form the
leave for the day
go home and rest
KM-BART
grih a cup from the cuunter pay for his drink
waich movie store
gel trio the carr
(Event) Personă walks of the store holding drink
leave the store
buy food for his and date
buy alcobol
drink aicohol
talk with and
(Place) near shop entrance
VisuaCCOMET
purchase drink
got the drink from the baz
walk asto the store
queuch this thins goi smother
enjoy a drink
go home
enjoy a drisk with I and
greet and
walk past and
gou into siore
go home with and
Figure 5: Comparison of generation examples from DIVE, KM-BART (Xing et al., 2021), VisualCOMET (Park et al., 2020), and human annotations VCG validation set. We mark red the inference implausible and blue if the inference both unique and novel.
6 Conclusion
mation from visual commonsense graphs. Through extensive experiments on VCG, we have verified that DIVE is capable of generating descriptive and diverse inferences about visual scenes, significantly outperforming sate-of-the-ard visual commonsense generation models. Particularly, our human evalu- ations have confirmed that DIVE indeed captures specific visual information, leading to improve- ments in plausibility, descriptiveness, and diversity.
We have presented DIVE to improve the descrip- tiveness and diversity of vision-language models in visual commonsense generation. We have pro- posed generic inference filtering method to bal- ance the skewed distribution of visual common- sense resources, based the frequency and seman- tic concentration of images. In addition, proposed contrastive retrieval learning method to promote the descriptiveness and diversity of vision- language models, by leveraging the structural infor-
have

---

Limitations
Jesse Dodge, Amit Goyal, Xufeng Han, Alyssa C. Mensch, Margaret Mitchell, Aneesh Sood, Karl Stratos, and Kota Yamaguchi. 2012. Understand- ing and predicting importance in images. In 2012 IEEE Conference on Computer Vision and Pattern Recognition, pages 3562-3569.
While we have demonstrated that DIVE effec- tively improves the descriptiveness and diversity of generated inferences, there are some limita- tions that present promising avenues for future re- search. First, our filtering method leads to loss of training data, which can limit the capabilities of vision-language models. We plan to investigate data augmentation methods for visual common- sense generation, such as utilizing external data (Xing et al., 2021) or data generated by founda- tion models (West et al., 2022). In addition, as the first work that explores the descriptiveness and diversity in visual commonsense generation, have focused the evaluations on VCG, which is the most representative dataset for visual com- monsense generation. Nevertheless, the ability to capture detailed information from an image and to generate descriptive and diverse inferences would be significantly beneficial to various visual reason- ing and generation tasks (Hessel et al., 2022; Zang et al., 2021; You et al., 2022), as well as reason- ing tasks over audio and video (Yu al., 2022; Li al., 2022a). We thus plan investigate the efficacy DIVE on more diverse tasks and modal- ities. Finally, in Section 5.3, we have observed that DIVE occasionally generates irrelevant inferences to the given context possibly due to the lack of com- monsense knowledge and reasoning ability. Future work could focus on enhancing the commonsense knowledge and reasoning ability in vision-language models (Li al., 2022c; Han et al., 2023).
Bo Dai and Dahua Lin. 2017. Contrastive learning for image captioning. In Advances in Neural Informa- fion Processing Systems 30, pages 898-907.
Michael J. Denkowski and Alon Lavie. 2014. Meteor universal: Language specific translation evaluation for any target language. In Proceedings of the Ninth Workshop on Statistical Machine Translation, pages 376-380.
Seungju Han, Jack Hessel, Nouha Dziri, Yejin Choi, and Youngjae Yu. 2023. Champagne: Learning real- world conversation from large-scale web videos. In 2023 IEEE/CVF International Conference on Com- puter Vision, pages 15498-15509.
Jack Hessel, Jena D. Hwang, Jae Sung Park, Rowan Zellers, Chandra Bhagavatula, Anna Rohrbach, Kate Saenko, and Yejin Choi. 2022. The abduction of sherlock holmes: A dataset for visual abductive rea- soning. In European Conference on Computer Vision 2022, pages 558-575.
Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. 2020. The curious case of neural text de- generation. In The Eighth International Conference Learning Representations.
Shaojie Jiang and Maarten Rijke. 2018 Why are grquence-to-sequeeee models so dull? understanding the low- diversity problem of chatbots. In Proceed- ings of the 2018 EMNLP Workshop SCAI: The 2nd International Workshop on Search-Oriented Conver- sational AI, pages 81-86.
Acknowledgements
Yong-Ho Jung, Jun-Hyung Park, Joon-Young Choi, Mingyu Lec, Junho Kim, Kang-Min Kim, and SangKeun Lee. 2022. Learning from missing rela- tions: Contrastive learning with commonsense knowl- edge graphs for commonsense inference. In Findings of the Association for Computational Linguistics: ACL 2022, pages 1514-1523.
This work was supported by the Basic Research Program through the National Research Founda- tion of Korea (NRF) grant funded by the Korea government (MSIT) (202 A22300013)) and In- stitute of Information & Communications Technol- ogy Planning & Evaluation (IITP) grant funded by the Korea government (MSIT)
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, AbdeIrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zeullemyye.. 2020. BART: denoising sequence-to-sequeree pre-training for natural language generation, translation, and com- prehension. In Proceedings of the 58th Annual Meet- ing of the Association for Computational Linguistics. pages 787 7871-7880.
Artificial Intelligence Graduate School Program (Korea University)).
References
Peter Anderson, Basura Fernando, Mark Johnson, and Stephen Gould. 2016. SPICE: semantic propositional image caption evaluation. In European Conference on Computer Vision 2016, pages 382-398.
Jiangtong Li, Li Niu, and Liqing Zhang. 2022a. From representation to reasoning: Towards both evidence and commonsense reasoning for video question- answering. In 2022 IEEE/CVF Conference Com- puter Vision and Pattern Recognition, pages 21241- 21250.
Alexander C. Berg, Tamara L. Berg, Hal Daumé III,

---

Jiwei Li, Michel Galley, Chris Brockett, Jianfeng Gao, ob-
Jae Sung Park, Chandra Bhagavatula, Roozbeh Mot- taghi, Ali Farhadi, and Yejin Choi. 2020. Visual- comet: Reasoning about the dynamic context of still image. In European Conference on Computer Vision 2020, pages 508-524.
and Bill Dolan. 2016. A diversity-proman jective function for neural conversation models. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Compula- tional Linguistics: Human Language Technologies, pages 110-119.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison. Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Te- jani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. 2019. Pytorch: An imperative style, high-performance deep learning li- brary. In Advances in Neural Information Processing Systems 32, pages
Junnan Li, Dongxu Li, Caiming Xiong, and Steven C. H. Hoi. 2022b. BLIP: bootstrapping sanguage-mmae pre-training for unified vision-language understand- ing and generation. In International Conference on Machine Learning, volume 162 Proceedings of Machine Learning Research, pages 12888-12900.
Manling Li. Ruochen Xu, Shuohang Wang. Luowei Zhou, Xudong Lin, Chenguang Zhu, Michael Zeng, Heng Ji, and Shih-Fu Chang. 2022c. Clip-event: Connecting text and images with event structures. In 2022 IEFF/CVF Conference on Computer Vision and Pattern Recognition, pages 16408.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sas- try, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. 2021. Learn- ing transferable visual models from natural language supervision. In International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 8748-88888 -8763.
Lixin Liu, Jiajun Tang, Xiaojun Wan, and Zongming Guo. 2019. Generating diverse and descriptive im- age captions using visual paraphrases. In 2019 IEEE/CVF International Conference on Computer Vision, pages 4239 4248.
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. OpenAl blog.
llya Loshchilov and Frank Hutter. 2019. Decoupled weight decay regularization. In The Seventh Interna- tional Conference on Learning Representations.
Shaoqing Ren, Kaiming He, Ross B. Girshick, and Jian Sun. 2015. Faster R-CNN: towards real-time ob- ject detection with region proposal networks. In Ad- vances in Neural Information Processing Systems 28, pages 91 99.
Jiasen Lu, Christopher Clark, Rowan Zellers, Roozbeh Mottaghi, and Aniruddha Kembhavi. 2023. UNIFIED-IO: A unified model for vision, language, and multi-modal tasks. In The Fleventh International Conference on Learning Representations.
Kaihua Tang, Yulei Niu, Jianqiang Huang, Jiaxin Shi, and Hanwang Zhang. 2020. Unbiased scene graph generation from biased training. In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recog- nition, pages 3713-3722.
Ruotian Luo, Brian L. Price, Scott Cohen, and Gregory Shakhnarovich. 2018. Discriminability objective for training descriptive captions. In 2018 IEEE/CVF Conference on Computer Vision and Pattern Recog- nition, pages
Ramakrishna Vedantam, C. Lawrence Zitnick, and Devi Parikh. 2015. Cider: Consensus-based image de- scription evaluation. 2015 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4566-4573 4575.
Tomás Mikolov, Ilya Sutskever, Kai Chen, Gregory S. Corrado, and Jeffrey Dean. 2013. Distributed rep- resentations of words and phrases and their compo- sitionality. In Advances in Neural Information Pro- cexsing Systems 26, pages 1111-9119
Jianfeng Wang, Zhengyuan Yang. Xiaowei Hu, Linjic Li, Kevin Lin, Zhe Gan, Zicheng Liu, Ce Liu, and Lijuan Wang. 2022. GIT: A generative image-to-text transformer for vision and language. Transactions on Machine Learning Research, 2022.
Lili Mou, Yiping Song, Rui Yan, Ge Li, Lu Zhang, and Zhi Jin. 2016. Sequence to hackward and forward sequences: A content-introducing approach gen- erative short-text conversation. In Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers, pages 3349-3358.
Peter West, Chandra Bhagavatula, Jack Hessel, Jena D. Hwang, Liwei Jiang, Ronan Le Bras, Ximing Lu, Sean Welleck, and Yejin Choi. 2022. Symbolic knowledge distillation: from general language mod- els to commonsense models. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Hu- man Language Technologies, pages 4602-4625.
Kishore Papineni. Salim Roukos, Todd Ward, and Wei- Jing Zhu. 2002. Bleu: method for automatic evalu- ation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Compu- lational Linguistics, pages 31 1-318.

---

Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier- rie Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. 2020. Transform- ers: Sfate-of-the-ar natural language processing. In Proceedings of the 2020 Conference Empirical Methods in Natural Language Processing: System Demonstrations, pages 38 38-45.
Rowan Zellers, Jiasen Lu, Ximing Lu, Youngjae Yu. Yanpeng Zhao, Mohammadreza Salehi, Aditya Kusu- pati, Jack Hessel, Ali Farhadi, and Yejin Choi. 2022. MERLOT RESERVE: neural script knowl- edge through vision and language and sound. In 2022 IELF/CVF Conference Computer Vision and Pattern Recognition, pages 16354-16366.
Tiancheng Zhao, Ran Zhao, and Maxine Eskénazi. 2017. Learning discourse- diversity for neural dialog models using conditional variational autoencoders. In Proceedings of the 55th Annual Meeting of the As- sociation for Computational Linguistics, pages 654- 664.
Sixing Wu, Ying Li. Dawei Zhang, Yang Zhou, and Zhonghai Wu. 2020. Diverse and informative dia- logue generation with context-specifie commonsense knowledge awareness. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 58 1-5820.
Daoming Zong and Shiliang Sun. 2023, Mcomet: Mul- timodal fusion transformer for physical audiovisual commonsense reasoning. In Proceedings of the 37th AAAI Conference Artificial Intelligence, pages 16229.
Yiran Xing, Zai Shi, Zhao Meng, Gerhard Lakemeyer, Yunpu Ma, and Roger Wattenhofer: 2021. KM-- BART: knowledge enhanced multimodal BART for visual commonsense generation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, pages 535-
Jingjing Xu. Xuancheng Ren, Junyang Lin, and Xu Sun. 2018 ppoomoting GAN: A cross-entropy based generative adversarial network for diversified text generation. In Proceedings of the 2018 Con- ference Empirical Methods in Natural Language Processing, pages 3940-3949.
Victor H Yngve. 1960. model and an hypothesis for language structure. Proceedings of the American Philosophical Sociery, 144(5):4444666
Haoxuan You, Rui Sun, Zhecan Wang, Kai-Wei Chang. and Chih-Fu Chang. 2022. Find someone who: Vi- sual commonsense understanding human-centric grounding. In Findings of the Association for Com- putational Linguistics: EMNLP 2022, pages 5444 5454.
Samuel Yu, Peter Wu, Paul Pu Liang, Ruslan Salakhut- dinov, and Louis-Philippe Morency. 2022. PACS: dataset for physical audiovisual commonsense rea- soning. In European Conference Computer Vision 2022, pages 292-309.
Xiaoxue Zang, Lijuan Liu, Maria Wang, Yang Song. Hao Zhang, and Jindong Chen. 2021. Photochat: A human-human dialogue dataset with photo sharing behavior for joint image-text modeling. In Proceed- ings of the 59th Annual Meeting of the Association for Computational Linguistics and the 1 1th International Joint Conference Natural Language Processing, pages 6142-6155 6152.
Rowan Zellers, Yonatan Bisk, Ali Farhadi, and Yejin Choi. 2019. From recognition to cognition: Visual commonsense reasoning, In 2019 IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, pages 6720-6731.

---

Appendix
Training set #image
#inference
Original Filtered
47,595 47,595
1,174,063
We supplement our main content with dataset anal- ysis and additional experiments.
949,284
Table 10: Statistics of VCG training sets.
A Dataset Statisties
Validation set
#lmage 13,768
#Inference
Original Unique Novel
146.332
Table 9 presents statistics of the VCG training set, categorizing the inferences by types. It provides the number of inferences, the number of unique in- ferences, and the frequency of the 50 most frequent inferences, along with their ratios. It is evident that the 50 most frequent inference results, accounting for 0.1 of the total inferences, occupy a signifi- cant proportion of the entire dataset. Tables and show the comparison between the original and processed datasets.
7,067 1,109
3,485
Table 11: Statistics of VCG validation sets.
of diversity and descriptiveness, but the quality the generated results is poor. In addition, we ob- serve that beam search and greedy search exhibit poor results in generating descriptive and diverse sentences. Upon evaluation, we find that nucleus sampling is closer to the optimal decoding strat- egy when considering all the generation quality, descriptiveness, and diversity.
Before
After
Intent
Inferences
467,025
469,430
237,608
Unique inferences
270,419
282,893
166,466
25,103
24,894
8,012
Top-50 inferences (Ratio)
(5.38%)
(5.30%)
(3.37%)
C Similarity Evaluation Analysis
Table 9: Detailed statistics of the VCG training set
We report the results the similarity evaluation among various design choices including image- only, text-only, and combined image-text settings in Figure 7 and 8. These figures illustrate compar- isons of the top-2 similar and dissimilar images on two randomly sampled examples. We first verify that similarity metric only using an image is not over-influenced by visual similarity. Upon analyz- ing images categorized by our metric as similar and dissimilar, we are unable to identify any distinct visual patterns that distinguish them. In addition, despite every image in the VCG dataset having its own accompanying textual event descriptions, in- volving texts in similarity evaluation (solely relying texts combining them with images) can lead to over-influence by overlapping words and text lengths, resulting in leaving irrelevant images or filtering out relevant, descriptive images.
The bar charts in Figure 6 depict the 10 most frequent inference results for each inference type. along with their frequencies in the training set. These charts obviously show imbalance among the inference results, with "walk into the room", "talk to person", and "leave the room" being the most frequent ones. The images in VCG are de- rived from the Visual Commonsense Reasoning dataset (Zellers et al., 2019), which contains care- fully selected images depicting social interactions between least two people. In such context, the most frequent inferences indeed simple and generic, applicable most images in VCG portray- ing two individuals.
B Decoding Strategy
We primarily use nucleus sampling (Holtzman et al., 2020), powerful decoding method, to obtain more diverse and descriptive sentences as our objec- tive is to avoid generating generic inferences. We have also considered other options, such as greedy search, beam search, and top-k sampling, for gen- crating commonsense descriptions. The compari- results of different decoding strategies are pre- sented in Table 12. Except for greedy search, generate five sentences per example.
D Number of Similar Images
In our contrastive retrieval learning method, we ad- ditionally sample similar images for one example in batch. Specifically, for one example, we first find the images that are related to the same inference and then sample images and inference sentences uniquely related to those images. Therefore, the images that related least one generic infer- ence and one unique inference could be sampled. In our main experimental setup, we primarily sam- ple one image for each example, given that we filter
Compared with nucleus sampling, top-k sam- pling shows slightly better performance in terms

---

Model
Decoding Strategy
B-2
C
Entropy
Unique
Nucleus (p 0.9)
13.33
20.26
21.09
76.09
Nucleus (p 0.7)
15.78
25.05
19,99
63.06
DIVEBART
Top-k 50)
12.40
18.82
21.67
80.82
Beam (10 heams)
22.01
38.38
16.11
22.59
Greedy
21.82
37.75
17.84
40.70
Table 12: Results of using various decoding strategies the original validation set.
Before
B-2
Entropy
Unique
walk
Freezing
12,86
19.07
20,93
76.05
onter the room
Full fine-tuning
12.44
18.28
20.93
75.81
walk up person
down table
Table 14: Results of DIVEBLIP according to freezing
pat on unfarm
approach person
visual encoder on original validation set.
talk to person
B-2
C
Entropy
Unique
Intent
Freezing
18.83
21.20
20.93
95.34
talk to parson
Full fine-tuning
17.34
18.49
21.00
93.63
get away parson
gred
lome
Table 15: Results of DIVEBLIP according to freezing visual encoder on unique validation set.
ask parten rieesiinn
have conversation with person
spend time with porson
person attention
get
get drunk
B-2
Entropy
Unique
300
200
330
400
Freezing
19.04
22.52
21.23
95.48
After
Full fine-tuning
16.71
18.81
21.15
94.05
talk person
Table 16: Results of DIVEBLIP according to freezing visual encoder on novel validation set.
hug person
speak possn
walk away
kis
ask person question
E Training Setup
200
3000
3500
2000
In Table 13, we present hyperparameter settings for our models. In addition to that information, AdamW (Loshchilov and Hutter, 2019) our optimizer. We do not adopt any learning rate scheduler or gradient clipping technique. We use six NVIDIA RTX A6000 GPUs, and the whole training procedure can done within a day. We implement the model code using PyTorch (Paszke et al., 2019) and HuggingFace (Wolf et al., 2020) and we train the model in 16-bit bfloat preci- sion for efficiency. We freeze the visual encoders of both DIVEBART and DIVEBLIP. In the case of DIVEBLIPs empirical observations suggest that the frozen settings generally produce better per- formance compared to full fine-tuning settings. As shown in Table 14, 15 and 16, it is possibly due to better resistance to catastrophic forgetting of freez- ing settings. For DIVE we have not tuned the visual encoder (i.e., Faster R-CNN (Ren et al., 2015)), since its discrete sampling procedures pose training difficulties.
Figure 6: Top 10 inferences and their frequencies most frequently connected to different images for inference type in the training set of VCG
DIVESART DIVEBLIP BART-base BLIP-base 64, 128, 256 le-5, 2e-5, 3e-5, 4e-5, 5e-5 0.2, 0.3, 0.4, 0.5
Backbone
Batch Size
Learning Rate
X Filtering Threshold
Dropout rate
0.1
Epoch
20
Training Time (A6000
60h
50h
Table 13: Detailed settings for training DIVE frame- work. Bold text indicates the chosen hyperparameter among we tried.
out most generic inferences from the training set. For each image-inference pair in our training set, 38.8% of pairs have connected to more than one similar image, however, only 18.7% of pairs are connected to more than five similar images. As result, to make the model see more diverse images in one step, sampling one similar image for one example is the best option.
To fine-tune DIVE models, we empirically choose for the t and X, filtering threshold and loss balancing value, respectively. In Table 17, we report the results of DIVE with varying

---

R-2
Entropy
Unique
8
12,80
19.82
21.85
81.31
13.33
20.26
21.09
76.09
13.25
20.24
20.61
73.17
Table 17: Results of DIVEBART according to filtering
threshold (t) on original validation set.
filtering thresholds.
F Error Analysis
We provide several representative cases that show the efficacy and limitation of DIVE. In Figure 9 (a), we find that DIVE effectively generates descrip- tive and plausible inferences by recognizing the facial expressions of people in the image. More- over, also observe that the performance of DIVE can be underestimated by the conventional metrics focusing on n-gram overlaps due to simple human annotations. We further report one of the cases where DIVE generates incorrect inferences, shown in Figure 9 (b). In this example, although the detailed information such as cat held by Per- son2 and the people the train is explicitly given, DIVE generates some irrelevant inferences to the given context such "bring the cat the bath- room". This implies that existing vision language models still lack some commonsense knowledge and reasoning ability to accurately reason over rec- ognized context.

---

"engage in conversation"
Text
Image
Image
Text
(Event) 2 talicing to standing close him (Place) outside in the forest
explains somnething with hand gestures office
(Event)
is focused on chatting with bedroom
Remaining
(Eventy) tallking to while readying himsellf (Place) living
(Event) 2 is talking to while readying himself (Place)
(Event) talking to I while readying himself (Place) living
last im
(Event) 2 is female and Asian wearing pink layered shirts the desk speaking earnestly to (Place) in room
but stops huilding
(Event) sprawled the counter rending newspaper (Place) video store
(Place) a lango building
Filtered
wdderee
(Event) sprawled out the counter while reading the newspuper (Place)
(Eveat) walking holding hag (Place) in martal art studio
(Event) walking the bolding (Place) in mactial art studio
Figure 7: Results of various similarity evaluation for generic inference filtering.

---

"argue with 1"
Text
Image
Image Text
(Event)
looking at suspicion (Place) in office
(Rvent) listening explain something him (Place)
(Event) I quicklty leaves the room (Place) in office
house
Remaining
(Event) is getting quite annoyed with (Place) office
(Event) holding a pool cue in one hand (Place) in pool hall
(Event) leaning back while talking with (Place) in house
INTERMEDIATE SCRUTINY
(Event) tan black hair and mustache wearing shirt and suspenders staring (Place) in restaurant krtchen
(Event) together they (Place) mall
(Event) wearing shirt and suspenders staring at ith shookel angry expression (Place) in restrumntt kitchen
angry
Filtered
INTERMEDIATE SCRUTINY
(Event) is shirless holding a partially enien apple in one hand and (Place) a bodroom
(Event) blocking protect his quasteracck (Place) on
(Event) stand together they lice (Place) in mall
Figure 8: Results of various similarity evaluation for generic inference filtering.

---

Person 2
Person 2
Person 1
(Event)
Person leans back into Person2's arm as she closes her eyes
(Event) (Place)
Person2 the train holding her cat. (Type) Intent
on train
(Place)
on the sofa
(Type) Intent
feel close to 2 cared for by show how comfortable she is with 2 express her deep love for make 2 feel loved
be good mother bring her cat with her bring the cat the bathroom take care of her cat give her cat some exercise
DIVE
DIVE
feel save relax feel protected
keep the cat secure her lap watch in concern
Human
Human
(a)
(b)
Figure 9: Examples of generated inferences from DIVE on the original VCG validation We mark red the inference is implausible and blue if the inference is both unique and novel.

---

Before
Inteat
After
beoughi quasiiering
ground
quacetion
interppatal
yelled be suspended without
Human
calleed into 2's office sit down from the dest
bost'
be office onder what do sillen
security
Pensu12
by
askedd to sii down
calm
be calleed into 2% office tell to enter the room
be compliant with 2 zvoid provoking the man in the office
DIVE
be threatemed by
invite 2 conforence ronna
Flaish their conversation with 2
ask a quactioon sit in the cheir
antwwrr quiniions
talk about the incomiing threal leaiee the reom
luvve conversation zok 2 question ask something zivoid cansing pooblems
get from the desk
proside security for
KM-BART
assigned scarl
lave an argument with
SCLUPS...
contisue speaking
(Event) Personl calmly in their seat while looking Person2
have tell them they are going leave
things
guifight
(Place) office
VisualCOMET
Zaware of
hear something that made
do somaahhng make
understand way 're thene
ponder the situation
sland
minaway angur with 2
thom upset
get say
be appcoached by
Before
Intent
After
train fire the weapon
take 2's car
forcus en target
lock and load the wespoos
forece 2 out the car
fire wamiing shor into the air oar jack
Human
his alown
drive
get
like big shot something from the make sure the diiver was vextanel
car custndy
gus
Person
DIVE
pull from its holidur
concinue purous the
the intention
place
the head lito building
walk out onto the sidewall
make the
get out of cur exit building
looked
get the
(Event) Personl has gun right hand approaches
KM-BART
pick up passengers
secure the porimeter
seu the ce
brckk up his purtner who
un the car
(Place) on highway
know where was going sneak
dlive hume
diiic soat
get sttention
the hostage
from going
VisudCCMET
after
person lie's almed
get
get get the
get
(b)
Figure 10: Comparison generation examples from DIVE, KM-BART (Xing et al., 2021), VisualCOMET (Park al., 2020), and human annotations in VCG validation We mark red if the inference implausible and blue if the inference is both unique and novel.


Reviews:
[{"review_id": "5cGGjQ7jvc", "reviewer": "Reviewer_k6e2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate the thoughtful and valuable comments. We have endeavored to address all concerns in our response.\n\n**Reasons To Reject 1. Surprisingness of the filtering method.**\n\nA. We believe that our data filtering method has produced innovative results, since it is challenging to achieve actual improvements in descriptiveness and diversity through filtering methods. As evidence, simply filtering out the most frequent inferences leads to less descriptive generation in our human evaluation (49.2%-win rate vs. KM-BART). Furthermore, the removal of inferences generally leads to a noticeable degradation in generation quality due to the reduced training data. By focusing on the semantic concentration of images, our filtering method effectively identifies and removes generic inferences, thereby improving descriptiveness and diversity while minimizing generation quality degradation. We will clarify this point in the camera-ready version.\n\n**Improvement 1. Clarity of Figure 2.**\n\nA. We appreciate your feedback on this. We will incorporate additional illustrations of images accompanied by textual descriptions of events and tasks, in Figure 2 of the camera-ready version.\n\n**Improvement 2. Annotations in Figure 5a.**\n\nA. We appreciate your correction. In Figure 5a, the annotations of *Person 1* and *Person 2* are mistakenly swapped. This will be corrected in the camera-ready version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The authors provide insights on how to generate descriptive and diverse commonsense inferences about visual scenes as the first exploration. \n\nThe authors develop the framework to address generation issues on the skewed visual commonsense resources with comparable results. Overall the work is well-written and easy to follow. The method and experiment design are convincing.", "reasons_to_reject": "For the `descriptive and diverse` visual commonsense generation tasks, using data after filtering out the generic ones as the first step of solution seems not surprising. The ablation study results (Table 8) also show some more drops without generic inference filtering than without contrastive retrieval learning.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This work is addressing the issue of generating more descriptive and diverse inferences in visual commonsense generation tasks. Towards this, the paper proposes a framework called DIVE with 1) generic inference filtering method to balance the Visual Commonsense Graphs (VCG) dataset by removing high frequency and low semantic images, and 2) contrastive retrieval learning method to maximize the agreement during training. Evaluated on VCG, the framework performs SOTA results in terms of descriptiveness and diversity.", "excitement": 3, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "A. It would be great to make Figure 2 more clear with more illustrations on the images and text (e.g. events and tasks).\n\nB. Is the annotation of Person1 and Person2 reversed in Figure 5a?\n"}, "extra_text": {}}, {"review_id": "58cgnU0eGA", "reviewer": "Reviewer_Ukok", "summary": "", "questions": "1. L297: the other images $h_k$ are treated as negative. Isn’t it possible for any of them to be valid for the image $h_p$ as well? How would this be addressed? \n2. L445: 17.3% on which metric? (mention absolute points, not relative) \n3. Sec 3.1: the goal is to determine which images are “semantically concentrated” (L223). Wouldn’t the similarity metric defined be heavily influenced by visual similarity of images rather than semantic similarity, especially when using CLIP embeddings? I’d like the paper to contain an explanation of why this wouldn’t be the case, along with corresponding proof (even if in the appendix). \n    1. Follow-up: Don’t the images have a text description as well? Why wouldn’t you use them instead of / in addition to the images to gauge semantic similarity? \n4. Why do you freeze the visual features of DIVE-BLIP and not DIVE-BART? I may be missing something here because the DIVE-BART architecture is never actually discussed (listed in Reasons to Reject 1.5).\n5. A brief description of word-level entropy in Sec 5.2 (even if in the appendix) would be helpful.  \n6. Why do you use greedy decoding in Table 14 instead of nucleus sampling, as in the rest of the paper? Are the results similar? \n", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate the thoughtful and valuable comments. We have endeavored to address all concerns in our response.\n\n**Reasons To Reject 1. More information about the methodology and experiments**\n\nA. We will elaborate on the methodology and experiments in the camera-ready version.\n\n**- L289: Definition of similar images $H$**\n\n- We define “similar images” as those that share the same inference result. For instance, given the subgraph of Figure 2(a), three sets of similar images can be constructed: one set includes three images that share “See what 1 is holding”, another set comprises two images that share “Get a closer look at 1”, and the third set contains multiple images that share “Speak to 1”, as the inference results.\n\n**- L301-303: Details of extracting representations from a vision-language model**\n\n- We extract the representations using BART and BLIP models, which are simultaneously trained via contrastive retrieval learning. We input images and their corresponding events and places to the models' encoder, and their inference results to the models' decoder. Image representations $V_h$ and text representations $T_s$ are subsequently extracted by averaging the output feature vectors from the final layers of the model’s encoder and decoder, respectively.\n\n**- Eq 5: Incorrect loss equation**\n\n- Thank you for the correction. For an image $h \\in H$, we identify its corresponding ground-truth inference $s = [w_1, w_2, …, w_k]$ as a sequence of tokens. The loss is then computed conditioned on previous tokens in this sequence.\n\n**- Eq 6: Loss computation with multiple ground-truth inference results $s_p$s for any image $h_p$**\n\n- We randomly select one $s_p$ for the loss calculation if multiple $s_p$s are associated with an image $h_p$.\n\n**- Adaptation of GPT-2 and BART to support a vision input**\n\n- We use projected Region of Interest (RoI) feature representations of an image from the pre-trained Faster R-CNN [1] as a vision input to the language models. For each image, Faster R-CNN detects several objects and generates their bounding boxes and classification results as RoI features. We extract the feature representations fed into the final classification layer of Faster R-CNN. The extracted representations are subsequently projected to fit the embedding dimension of the language models.\n\n**Reasons To Reject 2. A random sample of outputs**\n\nA. Thank you for the comment. In addition to the reported samples, we will further report random samples of outputs in the Appendix of the camera-ready version. This [Link](https://ibb.co/M81TRXw) illustrates two randomly sampled outputs from DIVE and the baselines. \n\n**Question 1. L297: Negative images that are potentially valid**\n\nA. We apply the contrastive retrieval loss only when a single unique positive image is present. If there are multiple positive images, we exclude the contrastive retrieval loss from our training objective.\n\n**Question 2. L445: 17.3% on which metric**\n\nA. In Table 6, DIVE_BART achieves a SPICE score of 8.36, which represents an improvement of 1.23 points (17.3%) over KM-BART’s score of 7.13.\n\n**Question 3. Influences by visual similarity when using CLIP embeddings and reasons not to use text descriptions**\n\nA. We have verified that our similarity metric is not over-influenced by visual similarity. Upon analyzing images categorized by our metric as similar and dissimilar, we have not identified any distinct visual patterns that distinguish them. However, we have observed that solely relying on texts or combining them with images can lead to over-influence by overlapping words and text lengths, resulting in leaving irrelevant images or filtering out relevant, descriptive images. We will involve the similarity evaluation results and the comparison among the outcomes of image-only, text-only, and combined image-text settings of similarity evaluation in the camera-ready version. The [Link](https://ibb.co/VHbZFnV) illustrates comparisons of top-2 similar and dissimilar images by the similarity measurement strategies on two randomly sampled examples.\n\n**Question 4. Reasons to freeze the visual features of DIVE-BLIP and not DIVE-BART**\n\nA. We have frozen the visual encoders of both DIVE-BLIP and DIVE-BART. For DIVE-BLIP, our empirical observations indicate that freezing settings achieve better performance on average than the full fine-tuning settings, as shown in the following tables, possibly due to better resistance to catastrophic forgetting of freezing settings. For DIVE-BART, we have not tuned the visual encoder (i.e., Faster RCNN), since its discrete sampling procedures pose training difficulties. We will detail the training settings more comprehensively in the camera-ready version.\n\n- Evaluations on the original VCG validation set\n\n|     DIVE-BLIP    |     BLEU-2    |     METEOR    |     CIDER    |     SPICE    |     Unique    |     Novel    |     Entropy    |\n|---|---|---|---|---|---|---|---|\n|     Freezing    |     12.86    |     11.39    |     19.07    |     6.90    |     76.05    |     56.50    |     20.93    |\n|     Full fine-tuning    |     12.44    |     11.16    |     18.28    |     6.51    |     75.81    |     57.21    |     20.94    |\n\n- Evaluations on the unique VCG validation set\n\n|     DIVE-BLIP    |     BLEU-2    |     METEOR    |     CIDER    |     SPICE    |     Unique    |     Novel    |     Entropy    |\n|---|---|---|---|---|---|---|---|\n|      Freezing    |     18.83    |     12.52    |     21.20    |     8.08    |     95.34    |     57.33    |     20.93    |\n|     Full fine-tuning    |     17.34    |     11.98    |     18.49    |     7.01    |     93.63    |     58.03    |     21.00    |\n\n- Evaluations on the novel VCG validation set\n\n|     DIVE-BLIP    |     BLEU-2    |     METEOR    |     CIDER    |     SPICE    |     Unique    |     Novel    |     Entropy    |\n|---|---|---|---|---|---|---|---|\n|      Freezing    |     19.04    |     12.23    |     22.52    |     8.36    |     95.48    |     59.69    |     21.23    |\n|     Full fine-tuning    |     16.71    |     11.47    |     18.81    |     7.19    |     94.05    |     59.93    |     21.15    |\n\n**Question 5. Description of word-level entropy**\n\nA. We will elaborate on the word-level entropy, as well as the other metrics, in the camera-ready version. Word-level entropy measures the average log probability of the words in a generated inference. We use the unigram probability of a word, which refers to the frequency of the word divided by the total number of words in the training set.\n\n**Question 6. Greedy decoding in Table 14**\n\nA. We use greedy decoding in Table 14 because it shows the performance trend among different threshold values more distinctly. We also observe a similar trend using nucleus sampling. We provide the results of nucleus sampling on the original VCG validation set here, which will be reported in the camera-ready version.\n\n|     DIVE-BART    |     BLEU-2    |     METEOR    |     CIDER    |     SPICE    |     Unique    |     Novel    |     Entropy    |\n|------------------|---------------|---------------|--------------|--------------|---------------|--------------|----------------|\n|     t=5          |     12.80     |     11.47     |     19.82    |     7.39     |     81.31     |     60.70    |     21.85      |\n|     t=10         |     13.33     |     11.48     |     20.26    |     7.33     |     76.09     |     54.20    |     21.09      |\n|     t=20         |     13.24     |     11.33     |     20.24    |     7.10     |     73.17     |     51.58    |     20.61      |\n\n**Suggestion 1. Consideration of whether keeping more generic inferences around would be beneficial**\n\nA. We appreciate your suggestion. We will examine the impact of the amount of generic inferences in the future.\n\n**Improvements 1. Variable $t$ is defined in both L233 and Eq. 2.**\n\nA. Thank you for the correction. We will denote the reasoning type as $r$.\n\n[1] Ren et al., Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks, NIPS 2015.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The proposed framework improves both BLIP and a vision-language enhanced BART significantly across many metrics, both automatic (Tables 1-3) and human (Table 7). This shows the merit of the framework itself, which could potentially continue to be used as the underlying VL models improve. \n2. The experimentation is neat and thorough. The authors conduct evaluations using various metrics as well as a human evaluation, to confirm that the metrics used mirror human judgment. The appendix contains further experiments (although plugs in the main text would be useful). The ablation analysis is also well discussed. \n3. There (seemingly) has not been a large body of work in visual commonsense generation over the past few years. However, as VCG methods improve, visual commonsense reasoning shows more potential to improve. These generation methods could be used to create (perhaps slightly noisy) datasets at large scales, which could be used to train VL models on visual commonsense reasoning; or to create more challenging visual commonsense reasoning benchmarks for VL models (after human filtering). \n4. The paper is well written and easy to understand. The authors attach code. ", "reasons_to_reject": "1. More information about the methodology and experiments is needed. \n    1. L289: “we construct a set of _similar_ images H”. How is similar defined? \n    2. L301-303: “representations of an image and a text from a vision-language model”. Which vision-language model? Where in the model are the representations are extracted from?\n    3. Eq 5: The loss equation is incorrect: the original loss in Park et al 2020 calculates loss over the length of a string. Here, the loss is seemingly calculated over the number of inferences, where each is conditioned on the previous inferences (not the previous tokens of the string). This is probably a typo, but should definitely be corrected. \n    4. Eq 6: For each image $h_p$ in a batch, the authors construct $H$ and calculate the loss according to this equation. However, there are likely multiple $s_p$s for any $h_p$. Is the loss summed over all $s_p$s? \n    5. Very importantly, how are GPT-2 and BART adapted to support a vision input? Especially BART, which is used in these experiments as DIVE-BART – although KM-BART is cited, a brief discussion is definitely needed here (more than L362-379) so the readers know what the model looks like.\n2. A random sample of outputs would be appreciated, even if in the Appendix.\n", "questions_to_authors": "1. L297: the other images $h_k$ are treated as negative. Isn’t it possible for any of them to be valid for the image $h_p$ as well? How would this be addressed? \n2. L445: 17.3% on which metric? (mention absolute points, not relative) \n3. Sec 3.1: the goal is to determine which images are “semantically concentrated” (L223). Wouldn’t the similarity metric defined be heavily influenced by visual similarity of images rather than semantic similarity, especially when using CLIP embeddings? I’d like the paper to contain an explanation of why this wouldn’t be the case, along with corresponding proof (even if in the appendix). \n    1. Follow-up: Don’t the images have a text description as well? Why wouldn’t you use them instead of / in addition to the images to gauge semantic similarity? \n4. Why do you freeze the visual features of DIVE-BLIP and not DIVE-BART? I may be missing something here because the DIVE-BART architecture is never actually discussed (listed in Reasons to Reject 1.5).\n5. A brief description of word-level entropy in Sec 5.2 (even if in the appendix) would be helpful.  \n6. Why do you use greedy decoding in Table 14 instead of nucleus sampling, as in the rest of the paper? Are the results similar? \n", "paper_topic_and_main_contributions": "The paper proposes a new framework for visual commonsense generation that yields more diverse and descriptive outputs. The framework consists of two stages: generic inference filtering, and contrastive retrieval learning. They are complementary and improve the quality of the generated inferences, according to several metrics as well as a human evaluation. \n\nThe contributions of the paper are the new framework, as well as an extensive set of experiments and an analysis of the same.", "excitement": 4, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "The variable $t$ is defined as the reasoning type (L233) as well as the filtering threshold (Eq 2). "}, "extra_text": {"Missing_References": "A suggestion, rather than a missing reference: the current method seemingly filters generic inferences quite aggressively. Table 14 in the Appendix shows that tweaking the filtering threshold $t$ doesn’t impact the results too significantly; however, it may be worth considering whether keeping more generic inferences around would be beneficial --- in Dataset Cartography (Swayamdipta et al, 2022), \"easy\" inputs are shown to still have a specific/unique value during training. "}}, {"review_id": "txWbflACKO", "reviewer": "Reviewer_kR3u", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate the thoughtful and valuable comments. We have endeavored to address all concerns in our response.\n\n**Reasons To Reject 1. Need to highlight the reliance on an additional model**\n\nA. We agree with your comment. We will explicitly highlight the usage of CLIP for filtering in the Introduction section of the camera-ready version.\n\n**Reasons To Reject 2. Absence of reported results in Section 5.1**\n\nA. We appreciate the feedback. Section 5.1 is the discussion of the results shown in Table 8. We will clarify it in the camera-ready version.\n\n**Improvements 1. Consideration for including Appendix E in the main text.**\n\nA. We appreciate your suggestion. We will incorporate the Appendix E section into the main body by utilizing the additional page given on acceptance.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The focus of the paper on commonsense in multimodal settings can be of interest to the community.\n2. The heuristic proposed in the paper, aimed at filtering out generic descriptions, underscores the crucial role of data quality.\n3. The paper is well-written and provides thorough experiments with compelling results.\n", "reasons_to_reject": "Overall, the paper presents an intuitive method with sufficient empirical evidence to support its effectiveness. I don’t find any major reason that would stop me from recommending acceptance. Nevertheless, I do have the following minor concerns:\n\n1. The reliance on an additional model (CLIP as mentioned in the paper) for deriving feature representations to filter the dataset needs to be explicitly highlighted in the Introduction.\n\n2. No actual results were reported in Section 5.1, although the findings are described.\n", "questions_to_authors": "", "paper_topic_and_main_contributions": "The paper presents a new framework, namely DIVE, which aims to enhance the generation of descriptive and diverse commonsense descriptions based on given images. DIVE consists of two components: data cleaning, and a contrastive training method, which can be plugged into any existing model. The dataset in this task contains many generic descriptions that could potentially limit a model's learning capability. To overcome this problem, the paper introduces a measure that is contingent on the similarity of related images for a description to discard poor-quality instances from the training data. Subsequently, DIVE employs an InfoNCE loss to encourage models to generate descriptive and diverse text. Through empirical evaluation, the paper illustrates DIVE can improve existing models in various scenarios (Tables 1-6). Also, a human assessment, conducted by crowdworkers, confirms the superiority of DIVE over baselines. The paper also provides a qualitative analysis of generated descriptions to highlight where DIVE might generate less plausible text.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "I found Appendix E very interesting and hope the authors can make room for it in the main body."}, "extra_text": {}}, {"review_id": "rLtk4OC3KY", "reviewer": "Reviewer_3SLs", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 2, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate the thoughtful and valuable comments. We have endeavored to address all concerns in our response.\n\n**Reasons To Reject 1. Novel tasks but simple methods**\n\nA. We believe that our methodology is both novel and carefully designed. Throughout our research, we have explored a wide range of deep learning models and techniques, and chosen the optimal ones. We would like to highlight the following points.\n\n- We have introduced innovative training frameworks that integrate distinct filtering and training strategies. Notably, our filtering method is anchored on the principle of semantic concentration of images — a pioneering approach to identifying generic inferences. Moreover, we employ a contrastive retrieval learning method, uniquely leveraging the structural information from visual commonsense graphs.\n\n- We have considered various recent vision models including CLIP [1], ViT [2], BLIP [3], and Champagne [4] to quantify semantic concentration. Based on our qualitative evaluation, CLIP outperforms the rest in gauging the similarity.\n\n- We have considered several training techniques including direct fine-tuning and reinforcement learning, widely used in recent commonsense work [5, 6], for retrieval learning. However, direct fine-tuning shows performance degradation of about 10%, while reinforcement learning methods show considerable training instability.\n\n- Finally, our methodology outperforms contemporary visual commonsense generation models in terms of descriptiveness and diversity, as well as generation quality, which are substantiated through our extensive automatic and human evaluations.\n\nWe are always seeking to improve our methodology. If the reviewer can recommend any recent developments in deep learning that may be relevant to our research, we would greatly appreciate the guidance.\n\n[1] Radford et al., Learning transferable visual models from natural language supervision, ICML 2021.\n\n[2] Dosovitskiy et al., An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale, ICLR 2021.\n\n[3] Li et al., BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation, ICML 2022.\n\n[4] Han et al., CHAMPAGNE: Learning Real-world Conversation from Large-Scale Web Videos, ICCV 2023.\n\n[5] Liu et al., Rainier: Reinforced Knowledge Introspector for Commonsense Question Answering, ACL 2022.\n\n[6] Yu et al., Fusing Pre-trained Language Models with Multimodal Prompts through Reinforcement Learning, CVPR 2023.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The authors pay attention to scriptiveness and diversity in the visual commonsense generation, which are important aspects in this task.\n2. Writing is good and easy to understand.\n3. Experiment results and cases are abundant.", "reasons_to_reject": "While the task is fairly novel, the method is simple. Most of the deep learning achievements in recent years have not been used.", "questions_to_authors": "", "paper_topic_and_main_contributions": "The paper proposed a framework, named DIVE. DIVE improves the scriptiveness and diversity in visual commonsense generation by generic inference filtering and contrastive retrieval learning. ", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "yCwn17Od1Z", "reviewer": "Reviewer_TWj1", "summary": "", "questions": "", "limitations": "", "rating": 2, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate the thoughtful and valuable comments. We have endeavored to address all concerns in our response.\n\n**Reasons to Reject 1. Limited novelty and unclear reasons for diversity improvements by contrastive learning**\n\nA. We present an unprecedented filtering and learning methodology that utilizes the structure and semantics of visual commonsense graphs to improve descriptiveness and diversity in visual commonsense generation. We would like to highlight the following points.\n\n- Our filtering method is anchored on the principle of semantic concentration of images — a pioneering approach to identifying generic inferences. This is in contrast to related work, which has largely focused on data augmentation [1] or modifying objective functions [2].\n\n- We propose a novel contrastive retrieval learning method, uniquely leveraging the structural information from visual commonsense graphs. This method enforces the model to identify of specific differences among similar images, based on the structural information from visual commonsense graphs.\n\n- Indeed, our contrastive learning method exhibits significant performance improvements on filtered datasets, where contrastive sets are constructed from more semantically concentrated images, possibly leading to a more precise understanding of semantics.\n\n**Reasons to Reject 2. Inappropriate adopted metrics**\n\nA. We adopt the metrics that reflect important aspects of descriptive and diverse inferences, which are widely employed to evaluate the descriptiveness and diversity of generated texts [3, 4, 5, 6]. Furthermore, our human evaluations confirm that the proposed methods indeed capture specific visual information, leading to improvements in plausibility, descriptiveness, and diversity. If the reviewer can suggest additional metrics that would be more effective in measuring the descriptiveness and diversity of our generated inferences, we would be most grateful for such recommendations.\n\n**Reasons to Reject 3. Comparisons with unlikelihood training and contrastive decoding**\n\nA. We will investigate other training and decoding methods in the future. In this work, we have focused on mitigating the data skewness of visual commonsense resources towards generic inferences. Therefore, we have mainly compared our models with those trained on full data or augmented data, while keeping the original training and decoding method.\n\n\n[1] Liu et al., Generating diverse and descriptive image captions using visual paraphrases, CVPR 2019.\n\n[2] Luo et al., Discriminability objective for training descriptive captions, CVPR 2018.\n\n[3] Liu et al., Generating Diverse and Descriptive Image Captions Using Visual Paraphrases, ICCV 2019.\n\n[4] Park et al., VisualCOMET: Reasoning about the Dynamic Context of a Still Image, ECCV 2020.\n\n[5] Xing et al., KM-BART: Knowledge Enhanced Multimodal BART for Visual Commonsense Generation, ACL-IJCNLP 2021.\n\n[6] Mou et al., Sequence to Backward and Forward Sequences: A Content-Introducing Approach to Generative Short-Text Conversation, COLING 2016.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Improving the diversity and description of vision commonsense generation is an interesting topic.\n\n", "reasons_to_reject": "\n\n- The novelty of improving generation diversity by adjusting the training distribution and using contrastive learning to improve consistency is limited. Besides, why contrastive learning can improve generation diversity is not clearly stated.\n\n- The adopted metrics are NOT a good indicator for the claim \"descriptive and diverse inferences by capturing important, specific, and detailed information within a visual scene\", therefore whether the proposal really captures the specific visual commonsense knowledge, or just learns to produce more unique n-grams, remains unclear.\n\n- The diversity of conditional text generation is well-investigated, and experimental results can be improved by incorporating more comparisons, such as unlikelihood training and contrastive decoding.\n", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper aims to improve the diversity of vision commonsense generation, by first filtering generic descriptions to build a more balanced training dataset, then adding an extra contrastive objective to encourage the similarity of paired image-text representations.\nExperimental results show that the proposed method is effective regarding descriptiveness ( mainly measured by length, distinct n-grams) and generalization quality (measured by traditional n-gram-based metrics such as BLEU and SPICE).\n", "excitement": 2, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper introduces a novel framework for generating visual commonsense content, resulting in more diverse and detailed outputs. The experimentation is meticulously conducted, and the ablation analysis is well-explained. The proposed framework brings substantial improvements to both BLIP and a vision-language enhanced BART across various metrics, including both automated (Tables 1-3) and human evaluation (Table 7).

The reviewers find the paper well-written and clear, and they appreciate the innovative and robust approach. Following the rebuttal phase, all reviewers have voted in favor of accepting the paper. I recommend accepting the paper.
Decision: Accept-Main