Localized Symbolic Knowledge Distillation for Visual Commonsense Models
Authors: Jae Sung Park
Jack Hessel
Khyathi Chandu
Paul Pu Liang
Ximing Lu
Peter West
Youngjae Yu
Qiuyuan Huang
Jianfeng Gao
Ali Farhadi
Yejin Choi
Venue: NeurIPS 2023 poster (2023)
Keywords: multimodal
commonsense reasoning
instruction tuning
large language model
TL;DR: We extract commonsense knowledge as QAR for images using ChatGPT and distill visual commonsense knowledge to vision language model by training on the synthetic data. Critic model is additionally trained to select high-quality generations.

Abstract: Instruction following vision-language (VL) models offer a flexible
interface that supports a broad range of multimodal tasks in a zero-shot fashion.
However, interfaces that operate on full images do not directly enable the user to
“point to" and access specific regions within images. This capability is important
not only to support reference-grounded VL benchmarks, but also, for practical
applications that require precise within-image reasoning. We build Localized
Visual Commonsense model which allows users to specify (multiple) regions-
as-input. We train our model by sampling localized commonsense knowledge
from a large language model (LLM): specifically, we prompt a LLM to collect
commonsense knowledge given a global literal image description and a local
literal region description automatically generated by a set of VL models. This
pipeline is scalable and fully automatic, as no aligned or human-authored image
and text pairs are required. With a separately trained critic model that selects
high quality examples, we find that training on the localized commonsense corpus
expanded solely from images can successfully distill existing VL models to support
a reference-as-input interface. Empirical results and human evaluations in zero-shot
settings demonstrate that our distillation method results in more precise VL models
of reasoning compared to a baseline of passing a generated referring expression.

Localized Symbolic Knowledge Distillation for Visual Commonsense Models
Jae Sung Park', .aak Hessel', Khyathi Chandu', Paul Pu Liang*4, Ximing Lu Peter West! Youngjae Yu', Qiuyuan Huang?, Jianfeng Gao?, Ali Farhadi Yejin Choil
University of Washington 2 Allen Institute for Artificial Intelligence Microsoft Research *Carnegie Mellon University Yonsei University
Abstract
Instruction following vision-language (VL.) models offer flexible interface that supports a broad range of multimodal tasks in a zero-shot fashion. However, interfaces that operate on full images do not directly enable the user to "point to" and access specific regions within images. This capability is important not only to support reference-grounten grounded VL benchmarks, but also, for practical applications that require precise within-image reasoning. We build Localized Visual Commonsense models, which allow users specify (multiple) regions-as-input. We train our model by sampling localized commonsense knowledge from large language model (LLM)): specifically, we prompt LLM collect commonsense knowledge given global literal image description and a local literal region description automatically generated by set of VI. models. With a separately trained critic model that selects high-quality examples, we find that training on the localized commonsense corpus can successfully distill existing VI. models to support a reference-as-input interface. Empirical results and human evaluations in a zero- shot set up demonstrate that our distillation method results in more precise VL models of reasoning compared to baseline of passing generated referring expression to an LLM
Introduction
Large language models capable of efficiently performing wide array of tasks in zero-shot fashion. For text-only models, one commonly adopted interface is a flexible, language specifica- tion of inputs coupled with an imperative request, e.g., "[article text]. Summarize this article." Similarly, natural extension allowing visual inputs manifests as, e.g., "[image] Describe this image".
But, as models expand beyond text-only modalities, they should incorporate more flexible forms of user input as well. Allowing users specify individual objeccs/actors/regions within image part of the input query is an important challenge, e.g., the [image] [request.] interface above would directly user ask Why is [this person in the image] sad?. One option would simply require users specifically describe the piece of the image they are attempting to specify, e.g.. "[image] Edescriptions of specific region] [request]". But authoring concrete referring expressions is not only cumbersome, particularly for scenes with lots of objects (c.g., "the person the red jacket on the left of the scene with their arms crossed") but also challenging, even for humans: [ argue that good referring expression should both specify the reference precisely, but also, follow Grice's s maxim of Quantity, i.e., provide no extra information. Given this tension, many
Cooee will be released in https://gittub..cottttto http://hhhhb.oooo///mmmmaaaaas/s
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

popular referring expression datasets are gathered in a sophisticated "gamified" fashion 13 22
which aims to balance underspecification vs. verbosity.
We argue instead that users of vision-augmented LLMs should instead be able to pass localized visual references simply by "pointing" to regions within the image 4 focus on the region while interpreting the user's request in a more intuitive fashion, and provide more accurate and contextually relevant responses. By incorporating localized visual references, the model can better understand and interpret complex scenes, thereby improving its performance on tasks requiring a detailed understanding of the visual context.
48 40). This enables models to
We propose Localized Symbolic Knowledge Distillation (LSKD): the core idea is to provide literal descriptions of images to a large language model, and allow that model to connect-the-dok between these literal descriptors (e.g.. lists of objects) and a holistic perspective of the scene. Different from recent works which also distill from an L.M conditioned on visual descriptors symbolically 64 [2),
we additionally provide a localized reference to a particular region within the image and design prompts to encourage the LLM to generate commonsense inference about that specific region. After sampling, we train Localized Visual Commonsense models to generate commonsense triples conditioned on the image and the region directly; we show that this process effectively distills the LLM's capacity for global+local scene understanding highlighted by zero-shot results on localized visual reasoning benchmarks and human evaluation.
In summary, our main contributions are:
1. A new scalable framework that can generate reliable and localized visual commonsense statements.
2. The Localized Commonsense Knowledge Corpus: 1M localized commonsense inferences posed over 250K images. This dataset can be used to expand the capacity of existing vision+language models to incorporate references-as-inpur with no architectural modifications.
3. Achieving the SoTA zero-shot performance for three localized visual reasoning tasks.
4. Human evaluation results suggesting that a strong student model outperforms the teacher model in answering localized visual commonsense questions.
2 Distilling Localized Visual Commonsense from a LLM
Here, we describe our LSKD pipeline to distill visual commonsense from a LLM. Prior works have explored powerful LLM the teacher model (GPT-3, ChatGPT) apply knowledge distillation for language-only reasoning tasks [58 33 3) Multimodal inputs offer additional challenges in grounding regions to relevant texts. Our work addresses this challenge by automatically generating reliable and diverse knowledge statements for multimodal input, to further reason about regions within an image.
Figure D shows the overall framework of LSKE To learn from the LLM as our teacher model, we verbalize the image into a set of dense text statements generated by global descriptors that provide relevant, general overall semantics of the image, and local descriptors that talk about specific regions in the image. We then pass these automatically generated descriptions to LLM and prompt to mine localized, commonsense statements about the image at scale (See the Appendix for the exact prompt).
As LLMs comprehend multimodal input only through maahine--gneeaatee image-to-text verbalization, they are prone to hallucination and generation of inconsistent statements about the image. For instance, an incorrect verbalizer output, as Figure I1 might cause the LLM to produce visually incoherent statements like "[1] is holding surfboard".. To minimize errors in modality translation, we construct a critic model, trained on limited set of high-quality, hand- annotated instances to detect and remove such inconsistencies. This critic model mimics human judgment in evaluating the generated commonsense knowledge, so that we can intentionally oversample localized knowledge data, and utilize it to filter out non-relevant instances. Finally, we finetune vasion-language model on the high-quality synthetic data to facilitate zero-shot localized visual commonsense reasoning. We use 250K images in union of Visual Genome (26 and VCR 66 which include diverse set of social situations involving people and objects, as the seed images to collect the knowledge corpus. After filtering, we collect IM instances of Localized Commonsense Knowledge Corpus with information grounded to specific regions in the image (see Appendix A for more details).
For visualization purposes, we provide a shortened version of verbalizations. The full verbalization output with the prompt to call ChatGPT is shown in the Appendix.

---

Globall Descriptore
Place: place doacts
holding surfing board:
and sccneer asending man
Verbalisation
Coraaggt
Goncapte:: boand shoot, surfoounnt up paddie suriing, surfboard Shaper
Namatinna
board
left side
Descriptons
Proposaks
black shorts
10 0.54))
swaaring
(10.2, jo 0.60)
BLP- Royann
a whrlee auffooad the saed (3.0. D. 2050, (1.1, 0.98]
sunglasses woman's fooe j0,s2, 0.85,, (0.4, 0.881
a surfboord JD 64, 0.08), (3 09,9, 0.70g
1. Question: Whet the lgitiing
image?? Arswer: naturall light inside ounsidee
Guestion
Questton:: Dows
outside
GUP-2
4. Questian: What the litl doing in enege Answer hotciing surfboard
5. Quaedton:: Are thene any other peop beeides the man, woman, and gi Answer TO
6. Question: is the coean caln or wayy in the background? Answer wary
1M Localized Commonsense
ChaRGPT
R. Leoalined Knowledge Generasion with LIM
8. Superviaed Gritic Filtering
Knowledge Corpus
Queaion: Whr neghtt be (I)) and discwsing is they held the wefhces??
Knower and night dicceiing kow woth sorf
Add (theredd aed eccimpp pilese
Ratioeal:: una and woman ard beising aathbuad the gid stadig
hailing
they decwsiin
4. Locallned Knowledge Distillation
VL, models
Qucaion: Wast bnildine the image?
Amewrr Ill bokling sualbeadd innar.
Itatonaler by roceding on 131s sahrtty we interbar du Harry tntroaed in surting and Iaay be lasning from the slabs
BLIP-2
Localized Co Commonsense Knowleage Corpuz
Figure 1: Pipeline of our LSKD framework. 1) Diverse vision-language descriptors are used to verbalize images. 2) LLMs leverage the global and local descriptors to generate grounded commonsene knowledge. 3) We annotate a small subset of data to train a supervised critic model that can filter instances displaying incorrect visual details or incoherent reasoning. The critic model filters the rest of generated statements to finalize the data pool. 4) A multimodal model is finetuned on the synthetic data to support localized visual commonsense reasoning in a zero-shot manner.
2.1 Image to Text Verbalization
We first describe our methods for verbalizing (i.e., writing out in natural language) images/regions to text. Note that this step does not require images with text annotations for the target datasets, unlike prior work 6el , and can be applied any set of images. We focus deriving global image descriptions, local region descriptions, and dynamic question-answer pairs for each image. Figure gives schematic of our process which includes an example image verbalization.
the CLIP-ViTL model in shot fashion to extract basic concept information about the image using a template. We retrieve places from the Place365 77], objects from TencentML-Imagee (59]. and concepts from OpenImages 2 to arrive at global concepts. Specifically: we use the top places, the top 3 object labels, and the top 3 concepts. In addition concepts, also get narrative descriptions of the entire image. For this, we fine-tuned OFA-Huge (54) on the Localized Narratives 44 corpus, which pairs 849K images with multi-sentence descriptions (we ignore the mouse trace information and simply treat the task as image- image-to-text to-text captioning). We sample 5 localized narratives for each image using a temperature of 1.1.
Global descriptors: Image Verbalizers Following (6)
Local descriptors: Region Verbalizers. Global descriptors alone often fail to capture the intricate details of specific regions within an image, leading to a potential bottleneck in understanding scenes with more visual precision and enabling localized reasoning. We employ local descriptors that provide more grounded visual statements. To do so, we sample bounding box regions for the image using region proposal models from object detection literature
(32. We then train region captioning model that maps from (image. region) - description of the region. We fine-tuned the generative version of BLIP-2 with the FLAN-15-X xxll 7 backbone. We trained on datasets that provide descriptions of regions within images. combination of DeeCoCCORe(CCCO-ReeOOOO
[44] 37). Sherlock
Clues-only [9 (277K), and VisualGenome 26 (1 .6MM): all of these datasets provide descriptions
of given regions within images. Following [68 62 we render the bounding box in the image itself to
allow the model access to the bounding box 's location. More details of the local descriptors are in
Appendix
B
3

---

170
QA MSE
Rationale MSE
Precision
Recall
64.2
66.3
65.2
65.7
66.0
64.3
64,8
14
Table I:
Analysis of BLIP-2 based critic model. We see that adding the multi-class regression loss further improves the performance of critic model.
Figure 2: Precision of Critic Model with vary- ing threshold values to filter the corpus size. Precision increased significantly by using the supervised critic model to filter the corpus.
Dynamic descriptors: Q+A Verbalizers Finally, to support holistic understanding and enable models to dynamically probe for potentially missing context, we acquire more fine- aineggriined details about the scene using series of questions and answers. Following 73,, we prompt an LLM to ask short, simple questions conditioning on the global and local descriptors as context, and query BLIP-2 [29 in zero-shot fashion to answer the questions. We specifically collect 15 question/answer pairs for each image.
2.2 Localized Commonsense Knowledge Generation
For all experiments, we use ChatGPT our LLM though in principle, any instruction-tunet LLM could be used. We use question- aunwering-ratione representations flexible, and have been successfully adopted meaning representations in areas
(QAR) for knowledge representations. QAR
ranging from formal semantics 17 38 24] to commonsense reasoning [50 [6].
Given the verbalization of images, we prompt ChatGPT to come up with interesting and complex question with the possible answer that requires rationale to justify the reasoning. We support two versions of localized knowledge generation. One that refers to specific regions in the image either by their assigned numerical IDs and bounding box coordinates (e.g. What is [2] doing in the image?) for more precise localization, and one that uses descriptive phrases (e.g. What is [the woman wearing a pink shirt] doing in the image?) for more contextual and detailed references. Qualitatively, we observe that the LLM is able to connect the IDs and the region descriptions successfully, and create a convincing set of localized commonsense knowledge corpus. For each image, prompt ChatGPT three times to generate three unique QARs sequentially. We do this for ID-based and description-bart references (see Appendix for the prompts), and collect 18 localized instances per image.
2.3 Training the Critic Model
We train a supervised critic model to reflect the human acceptability of generated data. We allocate a subset of 20K statements train the critic model, and 4k for evaluation. The "accepted" instances should generally deliver the visually correct information and exhibit coherent reasoning. For each QAR, ask two human annotators to rate from to 3 (reject / maybe/ / accept) if 1) the QA displays visually correct information (QA rating), and 2) the rationale justifies the answer while being aligned with the image (QA- R rating f] We then assign binary label if at least one annotator has included reject for any of the two rating criteria. Using this labeling scheme, we found that only 45% of the instances are labeled as accepted, suggesting that aggressive filtering by the critic model required.
For the model, we use a stage-1 pre-trained BLIP2 the critique. Following their finetuning scheme retrieval tasks. we train the image encoder and Q-Former together, not freezing the weights. We add a linear layer to the image-text matching head that has been pre- pre-trained to capture the multimodal content, and train it perform the classification.
20 with VIT-G [2] image encoder to do
https://openas htth:/llog/chhtgge
The second criterion is automatically rejected if the QA has already rejected in the first pass

---

With Fitering (threshold 0.5)
Without Fibtring
3
Descriptors Used
Average Critic Score
2.41
2.34
Fall Descriptors
()) CLIP Concepts
52.1
58.4
1.97
1.87
(-) Localized Narratives
56.1
(-) Global Descriptors
54.3
Local Descriptors
49.8
QAs
49.0
Table 2: Ablation study of the descriptors. We remove one of the descriptors from full descriptors when calling ChatGPT gen- erate the corpus, and calculate the average critic score rate the generations (higher the better).
QA Correctness
Rationale Justification
Figure Human judgment of corpus with and without filtering. We get the average ratings in Likert scale (from to 3) from three human annotators.
We utilize the two rating criteria (QA and QA R) to further inform the critic model know what caused the humans to reject the QARs. We achieve this by multi-task training of critic model. The ratings containing reject are given the regression label of 0, while the average of two QA and QA R ratings is calculated to get the regression label AQ4 and -- Along with the binary classification loss, the image-text matching head further trained with mean squared error (MSE) losses with YQa and yQA--R- Table [T shows the performance of critic model on the above train and eval split. We empirically see that adding the multi-task loss (QS MSE and Rationale MSE) further helps the performance of classification.
Analysis of Supervised Critic How reliable is the critic model on filtering erroneous instances? In the annotation stage, we have observed that only 45% of the instances would be considered as valid by humans. We explore tuming different thresholds of critic model to filter the data (e.g. keep instances whose predicted scores are higher than the threshold), and see higher acceptability can be achieved with higher threshold. Figure 2 shows a plot of precision value (instances labeled as "accept") by the filtered corpus size. We see consistent trend where removing the corpus with more critical criteria yiclds higher acceptability. Specifically, it jumps from 45% of 70% acceptance if 20% are maintained by the critic model. We use this threshold value 0.8 to apply the critic model. Note that filtering the corpus randomly, on the other hand, doesn't 't have any influence on the acceptabilityy
In addition, we run human evaluation measure the acceptability of data with and without filtering. We collect 500 instances the same way critic model labels are collected: 1) is the QA visually correct? and 2) does the rationale justify the answer? Likert scores from [1-3] are calculated for each criteria (higher the better). Figure shoows the human evaluation results, and we see that the dataset with filtering more favored by humans than without filtering.
Are All the Descriptors Necessary? We run ablation studies of the descriptor components in the ChatGPT prompt and use the critic model to score the ChatGPT generations. We collect QAR instances for 500 images and calculate the average critic score, with higher score aligned with human preference. Table Salle@sshne shows the result when one of the descriptors is removed from full verbalizations. We see that using all descriptors provides the best results, and in fact the QA descriptor provides the biggest jump (from 49.0 to 58.4).
2.4 Training with the Localized Corpus
We explore the distillation of localized commonsense knowledge by finetuning discriminative and generative vision language model on our corpus. For the corpus that mentions IDs and bounding box coordinates, follow 68 62 I9 by directly drawing colored highlights around the regions in the images where the region IDs and highlights are consistent throughout the corpus (e.g. [0] always gets the color pink).
During training, we additionally apply region- based augmentation by reassigning the IDs with random order while still keeping consistent color coding (e.g. What might be [0] and [1] discussing?
5

---

What might be [1] and [3] discussing?). We similarly vary the number of regions to be shown in the image, while ensuring that the mentioned IDs are drawn in the image. With these tricks, the modifications are performed in the input image and text to enable localization, while the architecture and training objectives of the vision-language model remain unchanged.
We use the BLIP-2 29 as the vision and language backbone model. Given the recent success and efficiency of visual instruction methods,
74 29 I0, we freeze the weights of visual and language model and only train the Qformer 34 learns to map visual to text tokens. For discriminative tasks, we apply the stage pre-training objective with Image- mext Contrastive, Image-Teir Text Matching, and Image-Texx Text Grounding Losses. We further explore generative performance with the FlanT55xxL, [59 language model and Mini-GPT4 that tunes the Vicuma- 13b-v0 language model
$4
6 [2
understand
visual tokens. We refer to 29 for more training details.
3 Experiments & Results
We the OpenAl Chat API with gpt-3.5-tubro engine and temperature of 0.8 to prompt the LLM to collect knowledge data. The BLIP-2 critic model is trained with total batch size of 256, learning rate of le-5, max 10 epochs. The visual encoder (ViT-G) model is additionally trained instead of kept as frozen.
The discriminative BLIP2 is trained with 256 batch size and 128 max sequence length for le4 iterations. The BLIP-2 F5xxI and Mini-GPT4 models are trained with 64 batch size and 2e4 iterations. All models trained with learning rate of 1e-5, Adam optimizer [ linear warmup with cosine annealing. and image size of 480 using 80GB 4 A100 GPUS. We do not finetume the ViT or the language model, and only train the QFormrr shown by the success from prior work 20
34
3.1 Downstream Tasks
Localized Visual Commonsense Reasoning We evaluate on a set of visual commonsense reasoning tasks that involve identifying and referring specific regions in the image in a zero-shot setting. VCR [66] is a task that requires choosing the right answers for question (Q - A), and rationales justifying the answer (QA- R) from four multiple choice options. The results are combined with (Q AR) metric that requires selecting the right answer and rationale. VisualCOMET 41 is a commonsense knowledge graph of understanding specific people's s intent, and what they would do before and after, and adopt their Acc 55 task of retrieving ground truth inferences from 50 candidates. Sherlock [1 is a visual abductive dataset that includes the comparison evaluation of ranking of 10 text inference candidates aligned with human preference. All the aligned tasks require reasoning about specific regions or people in the image, and getting the image-text similarity score from model.
Non- Non-Localized Visual Reasoning We measure the effectiveness of the localized knowledge corpus on other vision-language tasks not limited to datasets with no bounding box annotations. We specifi- cally focus on ones that require high-level reasoning that would benefit from visual commonsense corpus. AOKVQA 447 requires outside world- horld-knowledge to answer questions and we evaluate on their multiple choice setting. SNLI-VE [6 is an inference based visual entailment that tests fine-grained image understanding. The task is to predict whether the image semantically entails the text, and specifically classify if the image-text is one of (entailment, neutral, contradiction J. Visual7W 751 is visual QA with focus on visual grounding, and we evaluate on the subset of telling questions that have textual answers (Telling QA).
Baseline models We include CLIP as our baseline as it has shown strong zero-shot generalization results for various image-text alignment tasks [45). Following [501, we exclude the question in the text input and acquire the image-text cosine similarity score to do the task. CLIP-Event is a
CLIP model pre-trained on their VOA dataset crawled from news websites [5I. BLIP is image-text alignment model trained with additional generation objective and boostrapped image captions (30). We lastly evaluate the zero shot performance of BLIP-2 20 varying the visual encoders before applying knowledge distillation. We do not draw bounding boxes in the image nor include id tags in the text description, as these models have not been pre- rre-trained in this domain.

---

Localized
Non-Localized
VCR
Sherlock Comparison
VisualCOMET Acc
AOKVOA
SNLI-VE Classificatinn
Visual 7 Telling QA
Approach
Q
QA
Q
Choice
CLIP-Event
H1
52.4
49.2
22.4
CLIP VIT-B-16"
45
54.8
48.6
26.6
9.9
33.0
58.3
36,0
65.5 9
CLIP WIT-L-14x336
55
56.3
51.3
29,9
10.9
34,8
61.0
31.9
66.7
20.1
31.3
61.3
34.2
BLIP-2 LTT V
29
52.3
48.1
25.3
18.7
36.7
65.0
31.7
73.6
BLIP2
28.0
39.0
68.0
BLIP-2 VIT- LSKD
59.0
56.4
33,4
29.7
40.3
68.9
40.3
79.5
Table 3: Zero-shot results on the localized and non-localized visual reasoning tasks. *Zero shot VCR results directly obtained from [56]. For CLIP, we follow 156] by omitting the question and having the answer (with rationale) as text imput calculate the image-text similarity. For BLIP-2, we maintain
the question text input as it improves the performance.
LSKD
(a) VCR - A
(b) VCR QA
Accurocy
Accurssy
22
50
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
3.6
0.8
1.0
Sherlock Companson
(d) Visua COMET HiT
Accuracy
Accuracy
28
38
26
0.0
0.2
0.A
0.8
0.8
1.0
0.0
0.2
0.4
0.8
0.9
1.0
Figure 4: Effect of data quality controlled by filtering threshold on different datasets. The x-axis shows the threshold for filtering and the y-axis is the accuracy metric in percentage. We compare training our model on the LLaVA-instruct dataset (red) and ours (blue).
3.2 Zero-Shot Visual reasoning results
Table shows the zero-shot results on the downstream tasks. For localized reasoning tasks, we first observe that scaling the visual encoder size (CL,P ViTB-16 vs ViT-L-114336; BLIP-2 ViT-L vs ViT-G) in general improves the performance. CLIP outperforms BLIP-2 on VCR tasks but fall short on Shlerock and VisualCOMET. After applying localized symbolic knowledge distillation (LSKD) to BLIP-2, there is consistent improvement over the BLIP-2 model on all downstream tasks (5.4% on VCR Q - AR, 10.2 on Sherlock Comparison, 1.3% on VisualCOMET Acc
For non-localized reasoning tasks, we observe similar pattern. Interestingly, applying LSKD improves the performance BLIP2 model further across all the tasks (AOKVQA, SNLI-VE. Visual7W) over the vanilla model, despite these tasks not being the primary target domain. This demonstrates that the advantages of distilling models with localized reasoning can be transferred to high-level visual commonsense tasks, thanks to the visual precision and enhanced reasoning abilities learned from the generated knowledge corpus.
Influence of Critic Filtering on Downstream Tasks How does the process of critic filtering influence the performance of downstream tasks? Keeping the size of the selected statements the same 300K, we select qualified knowledge statements with varying prediction thresholds. We also compare with training on the LLaVA-instruct dataset which similarly prompts an LLM (GPT-4) to generate complex questions using ground truth verbalizers 34. Figure 4 presents the resulting performances at these diverse thresholds across different datasets. Compared to LLaVA we observe that localized knowledge statements without filtering does not show any improvement for the downstream model, while any thresholding over 0.2 is consistently better than LLaVA across all datasets. For tasks that demand relatively moderate commonsense, such as VCR Q- and Sherlock Comparison, increasing the threshold consistently improves the model performance. For
7

---

Localized
Non-Localized
VCR
Sherlock Compurison
VisualCOMET
AOKVQA Muli.
SNLI-VE Clussfication
Visual 7 Telling QA
Dataset
Size
Annotator
Zero-Shot
NA
NA
28.0
19.5
39.0
68.0
33.4
77.1
Sherlock
[09
300K
Human
34.6
30.5
39.7
67.2
38.6
70.1
VisuaCOMETT
!1
1.2M
Human
31.8
25.3
50.2
68.5
35.6
70.8
LLAVA-Instruct
A
150K
GPT-
28.1
26.9
37.5
71.0
42.6
79.5
L.SKD (Ours)
130K
CharGPT
33.3
28.6
39.7
69.6
38.0
75.9
LSKD (Ours)
IM
CharGPT
33.4
29.7
40.3
68.9
40.3
79.5
Table 4: Ablations of BLIP-2 VIT-G trained with varying sources of visual-knowledge corpus annotated by humans and machines. We break down to visual reasoning tasks that require localized reasoning and those do not. Critic filtering is applied to the LSKD corpus (Ours).
tasks requiring higher degree of commonsense such VCR QA-R and VisualCOMET Hit@50, the performance increases until certain threshold and then fluctuates. We speculate that a more grounded critic model could potentially mitigate this fluctuation, and intend to investigate this in our future work. Overall, our findings suggest that higher thresholds (i.e., more critical filtering) tend yield superior quality generations, thereby enhancing the performance on downstream tasks.
3.3 Human vs Machine Annotated Corpus
Can training machine annotated corpus result in competitive performance with human annotations? In Table g we compare the performance of BLIP-2 VIT-G trained on existing human-ame annotated corpora with machine-annotated annotated corpus across various scales. First, we found that increasing the size our training corpus (150K vs IM) leads to consistent improvement across all tasks, indicating promising trend of scaling law for synthetic training data. Regardless of the size, training dataset yields considerable benefits over the zero-shot model on localized reasoning tasks.
Next, we observe that training on human annotated corpus vastly improves the performance of their relative tasks (e.g. training on VisualCOMET boosts performance from 39.0 to 50.2). However, this can lead to inferior results on other visual reasoning tasks than the zero-shot counterpart. For instance, the performance on Visual7W drops from 77.1 (Zero-shot) to 70.1 (Sherlock) and 70.8 (VisualCOMET). This suggests that human designed datasets may limit task generalization due to their lack diversity. Interestingly, we see that training the model our full LSKD corpus (IM) leads to uniform gains the zero-shot model across the tasks, and even outperforms the human annotation corpus for the non-localized tasks as well. This shows that machine-annon annotated datasets, when curated and scaled adequately, can indeed rival or surpass the performance of models trained on human-ane annattted corpora.
We directly compare training on ours and the LLaVA dataset. Regardless of our dataset scale, observe that LSKD filtering wins over training on the LLaVA corpus on localized reasoning benchmarks, even when using less powerful teacher model (ChatGPT vs GPT-4). This suggests that our creation of new localization corpus crucial to support the model with grounded reasoning. On the other hand, LLAVA wins non-localized reasoning tasks they aligned with the nature of training corpus. We thus observe that the appropriate application of the corpus can be task-dependent, and adopting selective approach towards generating the corpus may result in significantly enhanced performance across various benchmarks.
3.4 Localized Reasoning with Generative Models
We extend LSKD to train generative models that can refer and talk about highlighted regions image. We finetune BLIP-2 Flan'S and Mini-GPT4 and prompt them to answer questions from the VCR data. As there is no baseline zero-shot model that can reason about regions answer questions, we make direct comparison of the student LSKD model to the teacher LLM with access to verbalizations. We ask annotators on Amazon Mechanical Turk (AMT) platform to run head-to-head comparisons (with ties) on three criteria, if the answer delivers: 1) visually correct details, 2) informative and interesting information, and 3) content that sounds plausible. Finally, they select their overall preference. We take the majority vote of 3 annotators, and disregard the instance if there is no clear majority.

---

Model
Correctness
Informativeness
Plausibility
Overall
ChatGPT w/ Vebalizers
34.7
33,9
39.6
45.0
BLIP-2 (FlanT3Sxxr-1 -1Br--1BB-- LSKD
31.7
41.0
30.2
41.2
Tie
33.7
25.1
30.2
13.1
ChatGPT w/ Vebalizers Mini-GPT4 (Vicuna-13B) + LSKD
29.8
31.7
36.8
40.6
34.3
53.0
34.2
49.1
35.9
15.3
30.0
10.3
Table 5:
Human evaluation of generative models with LSKD vs Chat-GPT with verbalizers. Humans are asked to choose the better generation or tie if they share the same quality.
Table shows the human evaluation results. We observe that the LSKD generally wins informa- tiveness over ChatGPT, but not in plausibility. We see conflicting pattern in correctness and overall preference, where Mini-GPT4 is equipped with a more powerful language model that outperforms the teacher model while BLIP-2 falls short. Unlike previous language-basee distillation where relatively weak student model can outperform the teacher [58] K
we see that a strong student model may be required to outperform the teacher LLM in the multimodal domain.
Qualitative Results Figure 2 presents a comparative analysis of question-answering with ratio- nale results on VCR samples generated by ChatGPT, LLaVA Fell and Ours. Both Ground Truth (GT) and Ours consistently identify the correct entities, with Durs model often providing broader context, which is uncertain rare occasions (e.g. "likely the bride"). On the other hand, ChatGPT predominantly focuses on observable actions or states as described in the text context, occasionally resulting in the misidentification of the visual entities and their relations. In the third example in Figure "waiting for someone" focuses on the observable state "standing still", missing visual detail such cave, holding flame, and surrounding context. LLaVA, in contrast, generally provided broad overview while identifying a specific visual entity in most cases. However, it often struggled accurately identify specific entities within the complex scene (e.g. "holding wine glass" in Figure 5 (1), "cigarette" in Figure (3) ). Compare to LLaVA, Ours often aligned closely with GroundTruth and incorporated both actions and inferred knowledge in its answer. Overall, Ours delivers a more nuanced and contextually-riee response.
4 Related Work
Knowledge Distillation Recent research [ has extensively explored the use of language models knowledge bases, highlighting their potential in reasoning, explainability, and consistency, which can enhance downstream tasks by distilling knowledge from LMs. [15] demonstrated how knowledge augmentation explicitly from knowledge bases and implicitly from GPT-3 improved open-domain multimodal tasks. [33 showed that overgeneration with GPT-3 from exemplars to filter, as well reviewed by humans, is new and reliable way create an NLI dataset with human and Al collaboration. This setup also has the advantage of bringing forth cultural internalization via human collaboration
O Previous works have explored knowledge distillation in the multimodal domain by prompting the teacher LLM with human human-annotated annotated verbalizations [34, 74, [TO.. Our work is different in that generated localized commonsense descriptions and the knowledge generation framework can operate scale without the need for aligned descriptions.
Filtering filters the generated sentences using a classifier trained on original sentences and set of generated sentences. [55] used the same technique to filter out synthetic data created, which is of low quality. Large language models can be used refine the commonsense knowledge retrieved from web contents by filtering the data generated from these models They perform consolidation step that filters topical and relevant assertions based on OpenTE.
Multimodal commonsense reasoning requires more profound real-world knowledge, potentially spanning logical, causal, and temporal relationships between concepts. For example, elements of causal reasoning are required to answer the questions regarding images in VCR [66] and Visual- COMET 41, while other works have also introduced datasets with video and text inputs to test for
temporal reasoning (e.g., Social-IQ (55, MovieQA [5I]. MovieFIB [36] TVQA (28). Benchmarks for multimodal commonsense typically require leveraging external knowledge from knowledge
bases 49 or pretraining paradigms large-scale datasets [35 68).

---

Question Image
GT Answer
ChatGPT
LLaVA
Which purson make
Anewor: [6) is mos
Ammer:
(2)
Amswrr: In the scens, sccoral
Answer: the bridk in (5) sccam be the most likely person to make coldbruory toasi given the occaaion the working receprion
likely to make the loest.
women are gothered in room: and appear in celebratury maod.
Rationale; she
Rationale:
Radionale: The
Rationale: the
already holding wine chulics her hand.
dress holding glass
likely make
holding glass
charaaagni, which
caldbratory
siee
dessa supaeus die
communiy colibneory
holding glase
likaly
would
teests.
engaging in cheertul
appropeiate and celebratory way fir expras gracitude and appeeciation.
intenatiion with uther wumenn.
is j0) 's fanczion??
Anewert he's the diiver.
Anower: (0) might a groom or un importann gaest of the wollling panty
Answer: in the image, man suit and lus ridlng is larse-
Answer: (0) riding in hoose-
drown cariige, which suggesta that
drown carrage with a wersnn.
[9] is driver or rider.
Rationale: he's bolding whip the noins
Rationale:: (0) is seated in horee drawn
Rationale The marks role appears
Rationale: the fast thal (9) is siting driver's seat carragge
which
that carrage driver,
he holds some
storning homoodrasm carrigge.
suggesnn that in
direct
signifiiance weddiigg
carrage
prodlius.
(3)
What /s [1] doing?
?
Answer: [J] is conteuplaing somcthing.
Answer: [1] might be warting for something summume
Aaswer: (1] holding a flame and eiguncilc lighter in his lundl and
Answer:
The man holding
to be in that he may be on misiion or
euggeniing
uaing to light the cigurett..
warching for womethingg
Rationale: Ratiooall:: [1] is standing and soem ongaged any action.
Rationali He appears tack hand.
Rationale: baslding suggest
[I] looking down
exumiting somathing
mystery danges, while
prosenee of case suggods he may be seanching fir sirething
Answert she seems to be covering her
Answer: [0] might be covering hat face protect the bubly
Answer: The woman ou the left
Answer: (0) is hiding her dace from
Way is 10)
coveinng her fince, likely bussasee
the smoke and dist the
nose from somothing
from the cold westher.
she is oither shy mrying to hide her
backgroond, which suggeits
emotivone, or novtrng rnn to comething that happened....
serions sitution.
Rationalk: penson in
Rationall:: (0] is waterng
Rationake Strce she wilh
Rationale: [0] covering thais
[-] "1 look
hat and cost which indhortes
group
taey engaged possibie that
face, which implies that
[0]
weatler, and is
the background
probubly just
hobling
suciviity involscee stoeing
unbealthy chengeroos.
snell that only
that she trying protect the buby... ...
personal thoughts experiences.
smelling.
Figure 5: Qualitative examples comparing ChatGPT (the teacher model). LLAVA trained for complex visual reasoning [34], and ours. Each person referenced in the question has been assigned a unique number with a bounding box and their mention has been marked with a corresponding color. Any errors within the generated results are emphasized with a red highlight.
Region Understanding with Multimodal alignment Capturing elements across modalities that have a common meaning and is exemplified by tasks such as visual coreference resolution 125 42) visual referring expression recognition R multimodal question answering 19 66). and cross-modal retrieval T 43 Aligmment between modalities is challenging since it may depend on long-range dependencies, involves ambiguous segmentation (e.g.. words or utterances), and could be either one-to-one, many-to-man
or not exist at all. Resources for fine-grained alignment include Visual Genome 26 and dense captioning [21], diverse reasoning (33. Recent methods have adopted either generative or retrieval-based methods for alignment: generative methods create localized verbalizations of the region of interest 772 14 21 70), while retrieval aims to select the most
accurate caption for the region of interest despite possibly given only coarse-graines paired data of
captions for entire images [8
Conclusion
We present LSKD, method for sampling localized commonsense knowledge from large language model. With the help of supervised critic model aligned with human judgements, we create diverse, reliable 1M localized commmonsense corpus. Training on the resulting corpus supports models that can accept region references as input, which allows users to interact with specific parts of images by "pointing:" all without having to write out a referring expression explicitly. We show that training on our corpus improves the zero-shot performance of languag models for tasks requiring regions-as-input. Making the critic model more critical by strict thresholding improved performance further. We present a state of the art zero-short performance with our approach opening avenues for visual commonsense models with our localized commonsense knowledge corpus.
10

---

References
[1] Badr AlKhamissi, Millicent Li, Asli Celikyilmaz, Mona T. Diab, and Marjan Ghazvininejad. A review language models as knowledge bases. CoRR, abs/2204.06031, 2022.
[2] Ateret Tavor, Boaz Carmeli, Esther Goldbraich, Amir Kantor, George Kour, Segev Shlomov, Naama Tepper, and Naama Zwerdling. Do not have enough data? deep learning to the rescue! In The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirtt-Second Innovative Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium Educational Advances in Artificial Intelligence, EAAI 2020, New York, NY, USA, February 7-12, 2020, pages 7383-7390 AAAI Press, 2020.
[3] Chandra Bhagavatula, Jena D. Hwang, Doug Downey, Ronan Le Bras, Ximing Lu, Keisuke Sakaguchi, Swabha Swayamdipta, Peter West, and Yejin Choi. 12d2: Inductive knowledge distillation with neurologic and self-imitation. ArXiv, abs/2212.092466 2022.
[4] Richard A Bolt. "put-that-there" voice and gesture at the graphics interface. In Proceedings of the 7th annual conference on Computer graphics and interactive techniques, pages 262-270, 1980.
[5] Shizhe Chen, Yida Zhao, Qin Jin, and Qi Wu. Fine-grained video-text retrieval with hierarchical graph reasoning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10638-106477 2020.
[6] Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, lon Stoica, and Eric P. Xing. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality, March 2023.
[7] Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. Scaling instruction-finetune language models. arXiv preprint arXiv:2210. 11416, 2022.
[8] Volkan Cirik, Taylor Berg- Kirkpatrick, and Louis-Philippe Morency. Using syntax to ground referring expressions in natural images. In AAAI, volume 32, 2018.
[9] Cédric Colas, Tristan Karch, Clément Moulin-Friee Frier, and Pierre Yves Oudeyer. Vygotskian autotelic artificial intelligence: Language and culture internalization for numan-like AL. CoRR, abs/2206.01 01444,111111111011111111100010 2022.
[10] Wenliang Dai, Jumnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi. Instructblip: Towards general- purrpss lasion-l models with instruction tuning. arXiv preprint arXiv:2305. 06500, 2023.
[11] Robert Dale and Ehud Reiter. Computational interpretations of the gricean maxims in the generation of referring expressions. Cognitive science, 19(2):233 1995.
[12] Yuxin Fang, Wen Wang, Binhui Xie, Quan-Sen Sun, Ledell Yu Wu, Xinggang Wang, Tiejun Huang, Xinlong Wang, and Yue Cao. Eva: Exploring the limits of masked visual representation learning at scale. ArXiv, abs/221 1.2111..000 1.07636, 2022.
[13] Andrea Frome, Greg S Corrado, Jon Shlens, Samy Bengio, Jeff Dean, Mare' Aurelio Ranzato, and Tomas Mikolov. Devise: A deep visual-semantic embedding model. In Advances in neural information processing systems, pages 2 2121-2129, 2013.
[14] Kun Fu, Junqi Jin, Runpeng Cui, Fei Sha, and Changshui Zhang. Aligning where see and what to tell: Image captioning with region- region-based attention and acene-specific contexts. IEEE transactions on pattern analysis and machine intelligence, 39(12):2321-233 2016.
[15] Liangke Gui, Borui Wang. Qiuyuan Huang, Alexander Hauptmann, Yonatan Bisk, and Jianfeng Gao. KAT: A knowledge augmented transformer for vision and-aanddddaa and-language. In Marine Carpuat, Marie-Catherin de Marneffe, and Iván Vladimir Meza Ruíz, editors, Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACI. 2022, Seattle, WA, United States, July 10-15, 2022, pages 956-95669999 366.668.466 Association for Computational Linguistics, 2022.
11

---

[16] Agrim Gupta, Piotr Dollar, and Ross Girshick. Lvis: A dataset for large vocabulary instance segmentation. In Proceedings of the IEEE/CVE conference on computer vision and pattern recognition, pages 5356-5364, 2019.
[17] Luheng He, Mike Lewis, and Luke Zettlemoyer. Question-answer driven semantic role labeling: Using natural language to annotate natural language. In Proceedings of the 2015 conference on empirical methods in natural language processing, pages 643-653, 2015.
[18] Xiangteng He, Yuxin Peng, and Liu Xie. A new benchmark and approach for fine- ane-grained cross-media retrieval. In Proceedings of the 27th ACM international conference multimedia, pages 1740-1748, 2019.
[19] Jack Hessel, Jena D Hwang, Jae Sung Park, Rowan Zellers, Chandra Bhagavatula, Anna Rohrbach, Kate Saenko, and Yejin Choi. The abduction of sherlock holmes: A dataset for visual abductive reasoning. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XXXVI, pages 558-575. Springer, 2022.
[20] Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. The curious case of neural text degeneration, 2020.
[21] Justin Johnson, Andrej Karpathy, and Li Fei-Fei. Densecap: Fully convolutional localization networks for dense captioning. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4565-4574, 2016.
[22] Sahar Kazemzadeh, Vicente Ordonez, Mark Matten, and Tamara Berg. Referitgame: Referring to objects in photographs of natural scenes. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pages 787-798, 2014.
[23] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
[24] Ayal Klein, Eran Hirsch, Ron Eliav, Valentina Pyatkin, Avi Caciularu, and Ido Dagan. QASem Parsing: Text-to-text Modeling of QA-based Semantics. arXiv preprint arXiv:2205.11413 2022.
[25] Satwik Kottur, José MF Moura, Devi Parikh, Dhruv Batra, and Marcus Rohrbach. Visual coreference resolution in visual dialog using neural module networks. In ECCV, pages 153-169, 2018.
[26] Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanic Chen, Yannis Kalantidis, Li-Jia Li, David A Shamma, et al. Visual genome: Connecting language and vision using crowdsourced dense image annotations. International journal of computer vision, 123:32-73, 2017.
[27] Alina Kuznetsova, Hassan Rom, Neil Gordon Alldrin, Jasper R. R. Uijlings, Ivan Krasin, Jordi Bust-Tuset, Shahab Kamali, Stefan Popov, Matteo Malloci, Alexander Kolesnikov, Tom Duerig, and Vittorio Ferrari. The open images dataset v4. International Journal of Computer Vision, 128:1956 1881195-1988 1981, 2018
[28) Jie Lei, Licheng Yu, Mohit Bansal, and Tamara Berg. Tvqa: Localized, compositional video question answering. In EMNLP, pages 1369-1379, 2018
[29] Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: bootstrapping language- image pre-training with frozen image encoders and large language models. arXiv preprint arXiv:2301. /2597, 2023.
[30] Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation, 2022.
[31] Manling Li, Ruochen Xu, Shuohang Wang, Luowei Zhou, Xudong Lin, Chenguang Zhu, Michael Zeng, Heng Ji, and Shih-Fu Chang. Clip-event: Connecting text and images with event structures. 2022 IEEE/CVE Conference on Computer Vision and Pattern Recognition (CVPR), pages 16399- 16408, 2022.
12

---

[32] Yanghao Li, Hanzi Mao, Ross Girshick, and Kaiming He. Exploring plain vision transformer backbones for object detection. In Computer Vision-ECCC 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part IX, pages 280-296. Springer, 2022.
[33] Alisa Liu, Swabha Swayamdipta, Noah A. Smith, and Yejin Choi. WANLI: worker and Al collaboration for natural language inference dataset creation. In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang, editors, Findings of the Association for Computational Linguistics: EMNLP 2022, Abu Dhabi, United Arab Emirates, December 7-11, 2022, pages 6847.6.7.. Association for Computational Linguistics, 2022.
[34] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. arXiv preprint arXiv:2304.08848 08485, 2023.
[35] Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee. Vilbert: Pretraining task-agnostic visiolinguistic representations for iisoon-andlannguaee tasks, In Advances in Neural Information Processing Systems, pages 13-23, 2019.
[36] Tegan Maharaj, Nicolas Ballas, Anna Rohrbach, Aaron Courville, and Christopher Pal. A dataset and exploration of models for understanding video data through fill-n-thhe-llank question- answering. In CVPR, pages
2017.
[37] Junhua Mao, Jonathan Huang, Alexander Toshev, Oana Camburu, Alan I. Yuille, and Kevin Murphy. Generation and comprehension of unambiguous object descriptions. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 11 1--20, 2016.
[38] Julian Michael, Gabriel Stanovsky, Luheng He, Ido Dagan, and Luke Zettlemoyer. Crowdsoure- ing question-answer meaning representations. arXiv preprint arXiv: 2017.
[39] Tuan-Phong Nguyen, Simon Razniewski, Julien Romero, and Gerhard Weikum. Refined commonsense knowledge from large-scale web contents. CoRR, abs/2112.04596. 2021.
[40] Sharon Oviatt. Multimodal interfaces. The human-compptter interaction handbook, pages
439 458, 2007.
[41] Jac Sung Park, Chandra Bhagavatula, Roozbeh Mottaghi, Ali Farhadi, and Yejin Choi. Visu- alcomet: Reasoning about the dynamic context of still image. In European Conference on Computer Vision, pages 508-524. Springer, 2020.
[42] Jac Sung Park, Trevor Darrell, and Anna Rohrbach. Identity-aware multi-sentence video description. Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XXI 16, pages 360-378. Springer, 2020.
[43] Bryan A Plummer, Liwei Wang, Chris M Cervantes, Juan C Caicedo, Julia Hockenmaier, and Svetlana Lazebnik. Flickr30k entities: Collecting region-to-phen correspondences for richer image-to-sentence models. In ICCV, 2015.
[44] Jordi Pont-Tuset, Jasper Uijlings, Soravit Changpinyo, Radu Soricut, and Vittorio Ferrari. Connecting vision and language with localized narratives. In ECCV, 2020.
[45] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In Interna- lional Conference on Machine Learning, 2021.
[46] Nils Reimers and Iryna Gurevych. Sentence-bert: Sentence embeddings using siamese bert- networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, 11 2019.
[47] Dustin Schwenk, Apoory Khandelwal, Christopher Clark, Kenneth Marino, and Roozbeh Mottaghi. A-okvqa: A benchmark for visual question answering using world knowledge, 2022.
[48] J Siroux, M Guyomard, F Multon, and C Remondeau. Modeling and processing of the oral and tactile activities in the georal tactile system. International Conference on Cooperative Multimodal Communication CMC, volume 95, pages 287-298, 1995.
13

---

[49] Dandan Song, Siyi Ma, Zhanchen Sun, Sicheng Yang, and Lejian Liao. Kvl-bert: Knowledge enhanced visual-and- and-linguistic bert for visual commonsense reasoning. Knowledge-Baud Based Systems, 107:00, 2021.
[50] Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. CommonsenseQAA:
A question answering challenge targeting commonsense knowledge. arXiv preprint arXiv:1811.00937 2018.
[51] Makarand Tapaswi, Yukun Zhu, Rainer Stiefelhagen, Antonio Torralba, Raquel Urtasun, and Sanja Fidler. Movieqa: Understanding stories in movies through gvestion-answering In CVPR, pages 4631-4648 2016.
[52] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timo- thée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aur' elien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language models. ArXiv, abs/2302.13971. 2023.
[53] Luis Von Ahn, Ruoran Liu, and Manuel Blum. Peekaboom: game for locating objects in images. In Proceedings of the SIGCHI conference on Human Factors in computing systems, pages 65-64, 2006.
[54] Peng Wang, An Yang, Rui Men, Junyang Lin, Shuai Bai, Zhikang Li, Jianxin Ma, Chang Zhou, Jingren Zhou, and Hongxia Yang. Ofa: Unifying architectures, tasks, and modalities through a simple requence-to-sequeme learning framework. In International Conference on Machine Learning, pages 23318-23340 PMLR, 2022.
[55] Yufei Wang, Can Xu, Qingfeng Sun, Huang Hu, Chongyang Tao, Xiubo Geng, and Daxin Jiang. Promda: Prompt-based data augmentation for low-resource NLU tasks. In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio, editors, Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), ACL 2022, Dublin, Ireland, May 22-27, 2022, pages 4242-4255 Association for Computational Linguistics, 2022.
[56] Zhecan Wang, Noel C. Codella, Yen-Chun Chen, Luowei Zhou, Xiyang Dai, Bin Xiao, Jian- wei Yang. Haoxuan You, Kai-Wei Chang. Shih-Fu Chang, and Lu Yuan. Multimodal adaptive distillation for leveraging unimodal encoders for vision- Jangua-language tasks. ArXiv, fttp/220.14499 2022.
[57] Jason Wei, Maarten Bosma, Vincent Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoe V. Le. Finetuned language models are zero-shot learners. ArXiv, abs/2109.01652, 2021.
[58] Peter West, Chandra Bhagavatula, Jack Hessel, Jena D Hwang. Liwei Jiang, Ronan Le Bras, Ximing Lu, Sean Welleck, and Yejin Choi. Symbolic knowledge distillation: from general language models to commonsense models. arXiv preprint urXiv:2110.0710 2021.
[59] Baoyuan Wu, Weidong Chen, Yanbo Fan, Yong Zhang, Jinlong Hou, J. Liu, and T. Zhang. Tencent ml-images: A large-scale multi-label image database for visual representation learning. IEEE Access, 7:1 7:663 1726993,77 2019.
[60] Jialian Wu, Jianfeng Wang, Zhengyuan Yang, Zhe Gan, Zicheng Liu, Junsong Yuan, and Lijuan Wang. GRiT: a generative region-to-text transformer for object understanding. arXiv preprint arXiv:2212.00280. 2022.
[61] Ning Xie, Farley Lai, Derek Doran, and Asim Kadav. Visual entailment: A novel task for fine-grained image understanding, 2019.
[62] Yuan Yao, Ao Zhang, Zhengyan Zhang, Zhiyuan Liu, Tat-Seng Chua, and Maosong Sun. Cpt: Colorful prompt tuning for pre-trained vasion-language models. arXiv preprint arXiv:2109.11797 2021.
[63] Da Yin, Liunian Harold Li, Ziniu Hu, Nanyun Peng. and Kai-Wei Chang. Broaden the vision: Geo-diverse visual commonsense reasoning. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 2115-2129, 2021.
14

---

[64] Licheng Yu, Patrick Poirson, Shan Yang, Alexander C Berg, and Tamara L Berg. Modeling context referring expressions. In Computer EsioonEECV 2016: 14th European Conference, Amsterdum, The Netherlands, October 11-14, 2016, Proceedings, Part II 14, pages 69-85. Springer, 2016.
[65] Amir Zadeh, Michael Chan, Paul Pu Liang, Edmund Tong, and Louis-Philippe Morency. Social-iq: A question answering benchmark for artificial social intelligence. In CVPR, pages 8807-8817, 2019.
[66] Rowan Zellers, Yonatan Bisk, Ali Farhadi, and Yejin Choi. From recognition to cognition: Visual commonsense reasoning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 6720-6731, 2019.
[67] Rowan Zellers, Jiasen Lu, Ximing Lu, Youngjae Yu, Yanpeng Zhao, Mohammadreza Salehi, Aditya Kusupati, Jack Hessel, Ali Farhadi, and Yejin Choi. Merlot reserve: Neural script knowledge through vision and language and sound. 2022 JEEE/CVF Conference on Computer Vision and Puttern Recognition (CVPR), pages 16354-16366, 2022.
[68] Rowan Zellers, Ximing Lu, Jack Hessel, Youngjae Yu, Jae Sung Park, Jize Cao, Ali Farhadi, and Yejin Choi. Merlot: Multimodal neural script knowledge models. Advances in Neural Information Processing Systems, 34:23634-236333
2021.
[69] Andy Zeng, Adrian Wong, Stefan Welker, Krzysztof Choromanski, Federico Tombari, Aveek Purohit, Michael Ryoo, Vikas Sindhwani, Johnny Lee, Vincent Vanhoucke, et al. So- cratic models: Composing zero-shot multimodal reasoning with language. arXiv preprint arX5v:2204.00598, 2022.
[70] Yiwu Zhong, Jianwei Yang, Pengehuan Zhang, Chunyuan Li, Noel Codella, Liunian Harold Li, Luowei Zhou, Xiyang Dai, Lu Yuan, Yin Li, et al. Regionclip: Region- language-image pretraining. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16793-16803. 2022.
[71] Bolei Zhou, Aditya Khosla, Àgata Lapedriza, Antonio Torralba, and Aude Oliva. Places: 2016.
image database for deep scene understanding. ArXiv,
[72] Yuanen Zhou, Meng Wang, Daqing Liu, Zhenzhen Hu, and Hanwang Zhang. More grounded image captioning by distilling image-text matching model. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 4777-4786, 2020.
[73] Deyao Zhu, Jun Chen, Kilichheek Haydarov, Xiaoqian Shen, Wenxuan Zhang, and Mohamed Elhoseiny. Chatgpt asks, blip-2 answers: Automatic questioning towards enriched visual descriptions. ArXiv, abs/2303.061 06594, 2023.
[74] Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny, Minigpt-4: En- hancing rision-language understanding with advanced large language models. arXiv preprint arXiv:2304. 10592, 2023.
[75] Yuke Zhu, Oliver Groth, Michael Bernstein, and Li Fei-Fei. Visual7w: Grounded question answering in images, 2016.
15


Reviews:
[{"review_id": "PVarLwqWvY", "reviewer": "Reviewer_Cm2R", "summary": "This work presents a localized visual common sense model that can support region-level inputs for knowledge reasoning. Specifically, the proposed methods prompts large language models to collect commonsense knowledge from global image descriptions and local descriptions. A critic classifier trained over a small number of annotation data is used to select high-quality examples. The distilled localized common-sense corpus can be further used for fine-tuning vision-language models. The experimental results demonstrate improvement over baselines in the zero-shot setting. \n", "questions": "a). Can the collected 1M instances of localized commonsense knowledge be used for other relevant vision-language tasks? Will the knowledge further benefit those tasks? \n\nb). In the Section of “effect of filtering”, the comparison with LLaVa datasets seems unfair since the number of LLaVa is fixed. What is the number of training data for LLaVa? \n", "limitations": "n.a.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "a). Strong motivation; supporting region-level references and reasoning is important for many applications.\n\nb). Technical sound;  provide detailed and clear explanation of each component; experiments and ablation studies are comprehensive. \n\nc). Thorough evaluation; have human evaluations comparing zero-shot performance of localized reasoning with generative models. \n", "weaknesses": "a). A new way of distilling vision-language knowledge but no framework-level innovation compared with text only symbolic distillation. \n\nb). The effectiveness of supervised critic: as the validity of instance annotation is only 45% and the trained classifier can only achieve 0.66 on F1 score. It is unclear whether the filtering process can be effective in selecting high-quality examples.   \n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for agreeing that support for **region-level references and reasoning is important for many applications**, and our experiments and ablations are comprehensive that include human evaluations of generative models. We address the comments raised by the reviewer below.\n\n**1. A new way of distilling vision-language knowledge but no framework-level innovation compared with text only symbolic distillation.**\n\nText only symbolic knowledge distillation and its variants assume that the teacher LLM (e.g. GPT-3, ChatGPT) and student models have access to the same text input modalities to perform the knowledge distillation process. In comparison, our framework explores when the teacher and student are provided with varying levels of input modalities in that the teacher LLM does not support raw image as input, while the student multimodal model is not subjected to the same constraint. To deliver the raw image content interpreted by the teacher model, we use a range of automatic image descriptors to verbalize images as text modalities. Specifically, we introduce three types of descriptors (Global, Local, QAs) all of which have proven significant as per the response to Reviewer MRhi. We assert that this unique aspect of our framework is an innovative contribution that will enable the multimodal community to comprehensively understand effective strategies for distilling visual-language knowledge from LLMs using automated, partial image descriptions. \n\n**2. The effectiveness of supervised critic: as the validity of instance annotation is only 45% and the trained classifier can only achieve 0.66 on F1 score**.\n\nIn Line 170-178, we explore tuning thresholds for the critic model and keeping only the instances whose predictions are higher than the threshold value. Figure 2 shows that the model precision increases by increasing the threshold value. Specifically, the acceptability jumps 45% to 70% by setting the threshold as 0.8. This shows that applying the critic model alone can increase the dataset quality measured by humans. \n\nIn Figure 3, we show empirical benefits of the critic-filtering mechanism on the downstream tasks. We see that the performance strictly increases by applying the critic model than without (threshold of 0 in the left side of the graph).  \n\nLastly, we report human evaluation of filtered and unfiltered dataset in the main rebuttal. We show that humans prefer dataset with critic filtering based on visual correctness and justification by a large margin. \n\n**3. Can the collected 1M instances of localized commonsense knowledge be used for other relevant vision-language tasks? Will the knowledge further benefit those tasks?**\n\nTable 1 in the rebuttal shows the benefits of localized commonsense knowledge in other visual-reasoning tasks such as AOKVQA, SNLI-VE, and Visual7W. We see clear improvement from applying BLIP-2 with LSKD over BLIP-2 in their zero-shot performance. We will include these downstream task results in the final version.\n\n**4. In the Section of “effect of filtering”, the comparison with LLaVa datasets seems unfair since the number of LLaVa is fixed. What is the number of training data for LLaVa?**\n\nThe LLaVA dataset consists of 150K instructions on question-answers generated by GPT-4, while ours is generated by ChatGPT. In Figure 3 of the main paper, we have sampled the training data after filtering so that the number of instances is always fixed to be 300K (L236) and make comparison with the LLaVA. \n\nTo make a fair comparison with the LLaVA, we experiment with 150K training samples of our corpus with and without filtering. In Table 2 of rebuttal, we observe that **LSKD + filtering wins over training on the LLaVA corpus on localized reasoning benchmarks (VCR, Sherlock, and VisualCOMET), despite using the weaker teacher model**.  The results suggest that our creation of a new localization corpus is crucial to support the model with grounded reasoning. On the other hand, LLAVA wins on QA-based reasoning tasks as they are aligned with the nature of training corpus. We thus observe that the appropriate application of these corpuses can be task-dependent, and adopting a selective approach towards choosing the right toolset may result in significantly enhanced performance across various benchmarks.   "}, {"review_id": "C2ZpsBBHDq", "reviewer": "Reviewer_Ta54", "summary": "The paper introduces Localized Visual Commonsense models that enhance vision-language (VL) models by allowing users to specify specific regions within images. The authors train their model by collecting commonsense knowledge from a large language model (LLM) using global literal image descriptions and automatically generating local literal region descriptions from VL models. They also use a critic model to select high-quality examples. The results show that training on this localized commonsense corpus improves the performance of VL models of passing generated referring expressions to an LLM.", "questions": "My major concern with this work is how to guarantee the dataset quality since there are a lot of automatic generation methods in dataset construction. So I would like to hear more illustrations about that.", "limitations": "N/A", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "+ The paper is well-motivated. The introduction of Localized Visual Commonsense models represents a novel solution to build a more general multi-modal model.\n\n+ The constructed data is of large scale, which contains 1M samples over 250K images. \n\n+ The extensive experiments show that the proposed dataset can support model to perform better zero-shot inference.", "weaknesses": "- How does this Dataset reflect knowledge? Since ChatGPT's input is based on factual descriptions of images, how can we ensure the questions generated by ChatGPT are related to external knowledge? On the other hand, how do the authors define what are knowledge-related questions?\n\n- For some knowledge-related questions, it seems that ChatGPT needs to guess the state of the subject in the image. Could this process result in hallucination issues? I notice that there's a critic filtering in the method, which rates the results by training on annotated data. What types of patterns is this model expected to learn, and under what circumstances does it consider the generated data to be inadequate? From Section 2.3, it seems to be judging visual correctness, but how can the correctness of the generated knowledge be determined?\n\nFor example, in Fig1, the model assumes that [0] and [2] are teaching [1]. This is a guess made by ChatGPT, as no similar descriptions are generated by the visual models, indicating that they cannot capture this information. So how would the critic model determine if this kind of guess is based on reality or is just a hallucination?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the feedback and valid concerns regarding how to guarantee the dataset quality, which we address below.\n\n**1. How does this Dataset reflect knowledge? How do the authors define what are knowledge-related questions?**\n\nIn our work, we investigate extracting commonsense knowledge from LLMs that involves reasoning about everyday objects and concepts to make judgments and predictions about new situations. Our contribution is to extend this paradigm to handle and extract rich knowledge for the multimodal data. We do this by providing factual descriptions of objects in the image, and utilizing the LLM’s commonsense reasoning capabilities to extract complex relations and interactions among objects in the image. We hypothesize that ChatGPT can perform such tasks with localized information based on the findings from previous work:\n\n - Wu et. al. demonstrate that ChatGPT can solve challenging zero-shot multimodal tasks via sophisticated use of prompting and diverse vision experts, including image captioning and object detection models. These tasks include spatial/coordinate understanding (can provide valid localized information) and open-world concept understanding in the multimodal domain.\n - Bang et. al. perform evaluation of ChatGPT in commonsense reasoning and find that “ChatGPT does quite well not only in terms of answer accuracy but also in generating reasonable reasoning procedures to support its answer”.\n\nThese findings suggest that **one can indeed prompt LLMs to create commonsense, inference statements with reasonable reasoning procedures in the multimodal domain, if the appropriate descriptions of the images are provided**. Similarly, our dataset collection procedure follows this intuition: 1) The global, local descriptors and the QAs by themselves give a relatively surface-level understanding of the objects and concepts. 2) The LLMs can utilize the information to probe complex relations and interactions among the objects in the form of Question, Answer, and Rationale explaining the reasoning procedures (QAR).\n\nThis form of knowledge is in fact illustrated in Figure 7 of the supplemental, where we observe:\n- Localized descriptions of a man and woman standing next to each other and holding a surfboard.\n- Description of a girl is running in the sand towards the man, who is holding a surfboard. \n- Then, ChatGPT uses commonsense reasoning that the three hold close, family-like relationships, and predicts the father might be interested in teaching the girl how to surf.\n\nTo empirically validate if the dataset contains relevant knowledge-related questions, we have trained the state of the art zero-shot model on the localized corpus and evaluated on downstream tasks known to require localized, visual commonsense knowledge to achieve good performance (VCR, Sherlock, VisualCOMET). The results in Table 2 in the main paper and Table 1 in the rebuttal show that the model with LSKD provides improvements on various localized and reasoning based downstream tasks, suggesting that **ChatGPT is indeed capable of generating relevant knowledge of visual commonsense understanding**\n\n\n**2. How to guarantee the dataset quality since there are a lot of automatic generation methods in dataset construction? I would like to hear more illustrations about that.**\n\nAs pointed out by the reviewer, the **dataset quality is controlled by the supervised critic model trained to filter out irrelevant instances, including hallucination**. The main rebuttal reports empirical benefits of filtering and includes human evaluation of filtered data, which we summarize here:\n\nWe first argue that increasing the dataset quality should be transferred to their respective downstream performance. In Figure 3, we show empirical benefits of the critic-filtering mechanism on the downstream tasks. We see that the performance strictly increases by applying the critic model than without (threshold of 0 in the left side of the graph). Applying more strict threshold on the critic filter leads to near-linear increase in performance in downstream tasks. This suggests that the dataset quality can be controlled with the filtering mechanisms using the supervised critic model.\n\nIn Section 3 of the main rebuttal, we run human evaluations of instances with and without filtering based on visual correctness of QA pairs and justification of the rationale. We see that the data filtered by the critic model achieve higher ratings in both criteria by a large margin.\n\n**3. What types of patterns is the critic model expected to learn, and under what circumstances does it consider the generated data to be inadequate?**\n\t\nThe critic follows the patterns of human annotations on evaluating the generated QARs from ChatGPT. The criteria is based on the visual correctness of QAs and appropriate justification of rationales as described in $\\S$2.3. Unlike ChatGPT, the critic model is a vision language model with access to the image input to determine to eliminate cases of hallucination of objects and actions. By training on human labels, we hypothesize that the critic model is equipped with adequate visual discriminability to perform the appropriate filtering. \n\nWe show an illustration of the critic model prediction in Figure 1 of the main rebuttal. The critic model uses its visual understanding skills to give high-scores instances with correct visual information (First example: gold hook in region [4] is localized correctly with appropriate description of its significance; Second example: the tag in region [5] correctly indicates the brand of the stuffed animal). We see that instances with visually incorrect and hallucinated details (bear holding another toy, scowling expression of brown teddy bears) are given low scores.\n\n[1]: A Multitask, Multilingual, Multimodal Evaluation of ChatGPT on Reasoning, Hallucination, and Interactivity [Bang et. al.]\n\n[2]: MM-REACT: Prompting ChatGPT for Multimodal Reasoning and Action [Wu et. al.]"}, {"review_id": "6SCZ9p1MW5", "reviewer": "Reviewer_UzVU", "summary": "In this paper, authors argue that for multimodal LLMs, the previous input formulation is too rigid: either needs to specify the region models should focus on, or brings a verbose object description to refer to the region. A more natural referring expression strategy can help models better understand where the model should pay more attention to and the intent of the input questions. The proposed approach, called Localized Symbolic Knowledge Distillation (LSKD), aims to provide more natural referring expressions. It involves providing literal descriptions of images to a large language model and allowing the model to connect these descriptions with a holistic understanding of the scene. In addition, localized references to specific regions within the image are provided to encourage the model to generate commonsense inferences about those regions. This method effectively distills the large language model's capacity for global and local scene understanding. They demonstrate the effectiveness of this approach by achieving state-of-the-art zero-shot performance on localized visual reasoning benchmarks and conducting human evaluations.", "questions": "1. Is there any way to apply this method to the datasets with no bounding box annotations? That will further improve the scalability of this method.\n2. Is there any study to explore the correlation between the ration of error training instances and the downstream task performance?\n3. Could you also apply LSKD to models like CLIP? It seems that in Table 2, they just use BLIP-2 ViT-G to show the effectiveness of LSKD.\n4. More qualitative error analysis is welcome in the modified version.", "limitations": "The limitations about introducing errors in the intermediate steps are reasonable.", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Novel research question to distill visual commonsense: Distilling visual commonsense is an interesting question that previous works under-explore. The distilled visual commonsense could provide very useful information for understanding complex situations covered in datasets like VCR, which will make the reasoning more grounded and reliable. Also, this is the first work about automatically acquiring visual commonsense to my best knowledge. \n\nComprehensive experimental results on downstream tasks: I'm very glad to see the distilled knowledge could further help base models on downstream tasks like VCR, Sherlock and VisualCOMET. Also, the small models can even outperform ChatGPT with the help of LSKD when describing the useful information in localized reasoning process. Besides, authors design experiments to examine the effect of steps in LSKD generation methods, which further make the argument stronger.", "weaknesses": "Lack of comparison with Sherlock and VisualCOMET: Although the current experiments support that LSKD is helpful for downstream tasks, I'm still a bit confused if LSKD is better than Sherlock and VisualCOMET knowledge bases. These two datasets are both large-scale and of great quality. It's better to compare with competitive resources, instead of just showing that considering external knowledge will help. \n\nComparing VL models and ChatGPT is unfair: Although ChatGPT is prompted with verbalizers, it still can only accept text information, which may ignore many visual information in the images. Especially for datasets like VCR, answering questions may need a very careful observation to some detailed places, which is what VL models like BLIP-2 is better at.\n\nPresentation in introduction section is confusing: I don't quite understand the relation between the first and second half of the introduction section. For the first half, I think the authors are trying to say a better referring expression is important. But in the second half (which is the core part of the paper), they just focus on how to distill visual commonsense knowledge. They are a bit connected because both parts mention localized reference, but it has very loose connection with discussion about the quality of referring expression.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for positive feedback that we explore an interesting problem of localized commonsense reasoning that **previous works under-explore**, and acknowledge our novelty (**the first work about automatically acquiring visual commonsense**). The reviewer is **glad to see the distilled visual commonsense knowledge could further help base models** to make them **more grounded and reliable.**  We address the concerns raised in the weaknesses and questions.\n\n**Comparison with Sherlock and VisualCOMET**\n\nFor Sherlock and VisualCOMET, we fully acknowledge the large-scale and high-quality nature of these knowledge bases, and we agree with the reviewer that comparison with them is an imperative step. As discussed in the main rebuttal, we include results based on models trained on our corpus (Row 3) with these human-annotated corpora in Table 2 (Row 5 and 6). make the following observations:\n- Ours vs Sherlock (Row 3 and 5):  Sherlock yields better results in Sherlock and VCR, and falls short in others.\n- Ours vs VisualCOMET (Row 3 and 6): VisualCOMET yields better results in VisualCOMET, and falls short in others.\n\nNot surprisingly, evaluating the same task as the existing training corpus leads to higher performance than LSKD (e.g. training on VisualCOMET corpus yields better results in VisualCOMET data than LSKD). We observe that LSKD provides benefits in terms of better generalization across diverse visual reasoning tasks. In fact, training with human authored corpus leads to considerable drop than the zero-shot models in Visual7W tasks. Such drop from zero-shot model, however, is not observed when LSKD is applied due to their diverse nature of knowledge corpus.   \n \nWe also like to point out that another benefit of LSKD pipeline over the human authored corpus is their annotation cost and scalability. The cost of annotating human-written knowledge corpus is higher than that of making API calls and takes significantly more time and effort if one were to scale the dataset collection.\n\n\n**Comparing VL models and ChatGPT is Unfair**\n\nWe compare VL models and ChatGPT as a way to make a comparison between student and teacher models in the knowledge distillation pipeline. Despite the modality gap, we still consider ChatGPT with verbalizers a strong baseline, since the descriptors inform place information, specific descriptions of regions, and dense QAs to inform the visual content to help to answer the question. Prior works show that language models can display strong zero-shot visual reasoning [Zeng et. al., Zhu et. al., Wu et. al., Lu et. al.]  by utilizing the text descriptions from vision tools.  In response to Reviewer MrHi, we show that small and narrow objects are covered by the region proposal pipeline, suggesting that descriptors can cover specific details of the image. However, we do agree that there remains an information bottleneck in the ChatGPT evaluation when understanding the visual content and will address these concerns in the limitation section. \n\n**Introduction doesn't flow as well**: Thank you for the comment! We will fix the introduction to fit the flow between the need for localized visual commonsense models and distilling visual commonsense knowledge.\n\n**Apply to datasets with no bounding box annotations**: Yes! Please refer to the main rebuttal and updated results in Table 1. We see consistent improvements on datasets with no bounding box annotations (AOKVQA, SNLI-VE, and Visual7W).\n\n** Any correlation between training instances and downstream performance?**: In Figure 3 of the main paper, we find that there is a positive correlation between the filtering threshold value as data quality control and the downstream task performance. In other words, making this critic model more critical helps filter quality knowledge statements from large quantities (L233-235). Thus, even from extracting the generations with the same teacher model, we observe that decreasing the error of training instances with the critic filtering is crucial to see gains in downstream task performance.\n\n**LSKD to other models**: We thank the reviewers for the suggestion, and agree that LSKD can be applied to models like CLIP. We run experiments mainly on BLIP-2 ViT-G due to their superior zero-shot performance than CLIP. Due to limited time and computation resources in the rebuttal period, we have not included their results but will do so in the later discussions. \n\n**More qualitative error analysis**: In addition to Figure 4 of the main paper, we will add more qualitative analysis in the main paper. \n\n- [1]: Socratic Models: Composing Zero-Shot Multimodal Reasoning with Language [Zeng et. al.]\n- [2]: ChatGPT Asks, BLIP-2 Answers: Automatic Questioning Towards Enriched Visual Descriptions [Zhu et. al.]\n- [3]: Visual ChatGPT: Talking, Drawing and Editing with Visual Foundation Models [Wu et. al.]\n- [4]: Chameleon: Plug-and-Play Compositional Reasoning with Large Language Models [Lu et. al.]\n"}, {"review_id": "vrsVYJCJto", "reviewer": "Reviewer_MRhi", "summary": "This paper aims to build a instruction following model for **localized** visual commonsense reasoning. The work differs from existing works in that it is able to reason about localized image regions with boxes, without using complicated referring expressions. The paper first collects data by distilling LLMs, i.e., prompting GPT using verbalized image descriptions, which is a concatenation of 3 descriptors: global, local, and QA pairs. Then, the data collected in the first stage is filtered by a critic filter trained using human annotations. Finally, the filtered data is used to train a BLIP-2 model. Experiments are done on 3 datasets for the localized visual reasoning task, showing the performance of the proposed method.", "questions": "See weakness.", "limitations": "N/A.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper is clearly motivated and the problem studied is well described.\n- The dataset collection method is well designed. Using (global, local, QA) descriptors to prompt LLM in order to generate question, answer and rationales.\n- The effectiveness of the critic filtering is well-analyzed and clearly shown in Fig 3.\n- The paper is well written and easy to follow.\n", "weaknesses": "My major concern is about experiments. The current results may not be very extensive to show the effectiveness of the proposed method.\n- Box size is an important factor, since the task is localized reasoning. How does the model perform on large and small objects? How does this compare to other methods? First, it would be helpful to describe the distribution of the bbox sizes in the generated data. Second, in the final evaluation, it is also helpful to study the model performance change wrt different box sizes.\n- three descriptors (global, local, QA) are used to prompt LLMs. Is there a study to show the contribution of each of the three? Since QA is not as widely used as the other two, additional discussions/results showing its effectiveness would be good.\n- Very limited baselines are compared with. In table 2, only CLIP is compared. In table 3, only ChatGPT. More baselines should be considered. For example, for tab 2, while zero-shot baselines are hard to find, it is still beneficial to show some non zero-shot methods (methods on the leaderboard of these datasets). Although the model performance is not directly comparable, it can give readers some sense of how well the model performs.\n- (potential extension) While the contribution on the data collection/distillation part is strong, the contribution on the model part is weak. Basically, BLIP-2 is directly used and trained on the collected data. Could there be some model design specific for handling “localized” information? Moreover, the “localized” reasoning in the current paper is constrained to color-coded bbox, while more general forms like free-form masks can be a further extension.\n- The work is in principle similar to LLaVA, extending it to be “localized”. Could the authors discuss the difference with LLaVA in more detail?\n- The title suggests “symbolic”, how is this work related to “symbolic”? I feel this is mis-leading.\n- some typos: L168 QA-MSE, L146 QAR is not defined.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewers for acknowledging that “the dataset method collection is well designed” and “effectiveness of critic filtering is well-analyzed”. We address the concerns raised by the reviewers.\n\n**Bounding Box Size Distributions and Performance Analysis**\n\nFigure 2 in the rebuttal shows the distribution of normalized bounding box sizes in the filtered corpus, highlighting the width, height, and the area. We notice that almost 50% of the bounding boxes have the normalized area $\\lt$ 0.05, suggesting that small objects are well-covered in our corpus. The height shows more uniform distribution than the width, indicating that there are more bounding boxes with smaller widths and the width mainly clusters in the range of 0.1-0.2. This reveals that the corpus contains not just large and prominent objects, but also small or narrow objects that often require attentive vision models to recognize. \n\nWe use the Sherlock comparison task to study the model performance change w.r.t different bounding boxes as their dataset consists of single bounding boxes with diverse sizes. We measure the **Pearson’s correlation between the input bounding box size and the comparison accuracy**: $\\mathbf{\\rho}$ = -0.12 with p-value $\\lt$ 0.05. \n\nBased on the correlation, we see that the performance is actually higher for smaller objects. One might indeed initially think that larger bounding boxes would result in better performance, as they could potentially encompass more features or more of the object of interest. We hypothesize that the negative correlation is due to that:\n\n- **Specificity**: Smaller bounding boxes quite often are more specific in identifying the target objects, thereby reducing the complexity of the region and making it easier for the model to focus and reason.\n- **Clutter**: Larger bounding boxes might include more \"noise\" or irrelevant objects/background - which could mislead the model during the reasoning process as it gets distracted by extraneous details.\n\nThis is reflected in the dataset distribution as smaller objects are more covered in the dataset generation. The analysis on the bounding boxes sizes will be included in the final paper.\n\n**Contributions of Three Descriptors to prompt LLMs**\n\nWe run ablation studies of the descriptor components in the ChatGPT prompt by using the critic model to score the ChatGPT generations. We collect QAR instances for 500 images and calculate the average score with higher score aligned with human preference. **We see that using all descriptors provides the best results**, and in fact the QA descriptor provides the biggest jump (from 49.0 to 58.4). We will add these results in the final version.\n- Full verbalizations: 58.4\n- No Localized Narratives: 56.1\n- No CLIP: 52.1\n- No Global Descriptions: 54.3\n- No Region Descriptions: 49.8\n- No QA: 49.0\n\n**Limited Baselines**\n\nWe agree that limited baselines are compared due to difficulty of finding zero-shot baselines in the localized reasoning tasks. In Table 1 of the rebuttal, we additionally include the BLIP baseline results and the finetuned results that achieve top of the leaderboard on these dataests. The gap between finetuned and zero shot models suggests that localized visual commonsense is not present in pre-trained vision language models. This underscores the importance of our work in filling these gaps by adding localized visual commonsense to existing models.\n\n**Extension of Localization Approach** \n\nWe constrain our study to color-code bounding boxes following prior works [Zellers et. al., Hessel et. al.] known to effectively encode localized information. One way to improve the handling of “localized” information is to directly encode the information as floating points in the text input and output. Another extension is using the segmentation masks over the bounding boxes for more refined localization of the desired objects and regions. We leave it as future work to improve the localization aspect. \n\n**Difference with LLAVA** \n\n- The main difference between LLAVA and our pipeline:\n  - LLAVA uses GPT-4 as the teacher model, while we use ChatGPT.\n  - LLAVA relies on human annotations of captions and object bounding boxes in COCO images, while we utilize diverse, learned descriptors to extend to images without human annotations.\n  - We train additional supervised critic to filter irrelevant generations made by the teacher model to ensure higher dataset quality, while LLAVA does not have filtering mechanism.\n- As discussed in the main rebuttal and response to Reviewer Cm2R, we compare ours with LLaVA in the localized reasoning task on the BLIP-2 ViT-G model. We see that our corpus with critic filtering gives clear improvement in various localized reasoning tasks in Figure 3 of the main paper. Table 2 in the rebuttal shows more comparisons while fixing the dataset size to be the same. We see a clear improvement of using our corpus over LLAVA in localized visual commonsense reasoning tasks with the same dataset size (row 2 and 4 in Table 2), and the gap is increased with more training data (row 3 and 4 in Table 2).\n\n**How is this work related to symbolic?**\n\nThe “symbolic” terminology is introduced in [West et. al.] to denote that knowledge is distilled from the teacher to student model “symbolically as text”. Our framework is different in that we include verbalizers to prompt the teacher model to account for the modality gap as per response to Reviewer Cm2R. We will clarify this terminology in the introduction.\n\n**Typos**\n\nThank you for pointing out the typos. We will define QAR as (Question, Answer, Rationale) and fix any remaining typos accordingly in the final version.\n\n[1]: MERLOT: Multimodal Neural Script Knowledge Models [Zellers et. al.]\n\n[2]: The Abduction of Sherlock Holmes: A Dataset for Visual Abductive Reasoning [Hessel et. al.]\n\n[3]: Symbolic Knowledge Distillation: from General Language Models to Commonsense Models [West et. al.]\n"}, {"review_id": "J9uWEvtN1M", "reviewer": "Reviewer_uG1j", "summary": "This paper proposes a framework that can generate global, local, and dynamic descriptions. ChatGPT can generate question-and-answer pairs containing specific regions or descriptive phrases with the proposed tricks. A critic model is trained to filter the generated data. Finally, the localized corpus is used to fine-tune the vision language model.", "questions": "This paper proposes an impressive engineering framework. It would be better to analyze the impact of the quality and amount of the generated data on the final performance of the visual language model. ", "limitations": "I did not find a potential negative societal impact of this work.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. This paper proposes a way to acquire localized commonsense knowledge data and enable the vision and language model to answer localized visual commonsense questions. The workload of this work is impressive.\n\n2. The fine-tuned vision language model achieves promising zero-shot performance for three localized visual reasoning tasks.\n", "weaknesses": "1. The proposed framework is more like engineering work. The proposed method to generate localized data pairs is straightforward, and the way to enable the local-level question-answering ability is not novel, limiting its scientific contribution.\n\n2. There are too many tricks, including filtering by critic model and prompts for acquiring question-answering pairs, which decreases the reproducibility and robustness of the whole pipeline. Furthermore, the selection of the vision-language model and large language model, and hyperparameters for each model in each step increase the complexity of the proposed pipeline, and the final vision-language model may be sensitive to the generated data, which makes it hard to iterative improve the final model's performance by improving any step in this pipeline. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for acknowledging that the **workload is impressive** and **achieves promising zero-shot performance for three localized visual reasoning tasks.** We now address the reviewer's comments in the following section.\n\n**1. The proposed framework is more like engineering work. The proposed method to generate localized data pairs is straightforward, and the way to enable the local-level question-answering ability is not novel, limiting its scientific contribution.**\n\nPrevious models enabled local-level commonsense question-answering ability via training on human annotation dataset such as VCR, which cannot be scaled automatically. On the other hand, we explore novel ways of automatically extracting localized visual commonsense knowledge at scale, which requires a heavy engineering workload due to their nature of automated pipeline. In fact, reviewer UzVU states that we investigate a “novel research question to distill visual commonsense knowledge”, which is “an interesting question that previous works under-explore”. We are also “the first work about automatically acquiring visual commonsense”.  \n\nReviewer Cm2R has pointed out that our framework has limited innovation compared to the original symbolic knowledge distillation. Following our response, we argue that we investigate a different problem when the teacher and student are given different input modalities, where the teacher has no access to the raw image content. This leads to a new framework of utilizing automated image descriptors to capture the raw image content interpreted by the teacher model. Different from prior multimodal generation work such as LLaVA, we also generate localized data pairs to help grounding of vision language models. Given these differences, we believe our framework presents a novel approach to localized, visual commonsense understanding. \n\n**2. There are too many tricks, including filtering by critic model … which decreases the reproducibility and robustness of the whole pipeline. …It would be better to analyze the impact of the quality and amount of the generated data on the final performance of the visual language model**\n\nBased on our empirical studies, we find that the components of verbalizers and filtering by critic model are crucial to enable the knowledge distillation. \n\nWe first argue that **training the critic model is necessary to eliminate hallucinations and irrelevant generations to guarantee the dataset quality**. This is illustrated qualitatively in Figure 1 of the rebuttal where instances with incorrect visual details are given lower scores. Next, the human evaluation in Section 3 of the rebuttal clearly shows the benefits of filtered data based on their visual correctness (2.41 vs 1.97) and justification (2.34 vs 1.87). \n\nBased on these findings, we further analyze the impact of the quality of the generated data using the score from critic model.  Keeping the number of training data at 300K, we see if letting the critic model to be more critical would lead to increase in dataset quality. This can be controlled by the threshold value of critic model where we only keep the instances that score higher than the threshold.  Figure 3 shows the performance trend based on the threshold value, where we see a near-linear increase on the final performance with more aggressive filtering. This **implies that the *quality* of generated data could be controlled by the critic model**. \n\nIn Table 2 of the rebuttal, we evaluate the performance of the pipeline using varying amounts of generated data, specifically comparing the results from datasets of 150K and 1M in size. We see that **increasing the scale leads to better performance in all of localized visual reasoning tasks**. \n\nWe are also open to running any other analysis that reviewers would like us to investigate.  \n\n**3. Are prompts for acquiring question-answering pairs necessary?**\n\nIn response to Reviewer MRhi, we found that adding all three verbalizer descriptors yield the highest scores from the critic. For instance, the **adding the QA descriptor provides the biggest gain from 49.0 to 58.4**. This means that calling all the descriptors, including QA, is necessary to get more high-quality data as well.\n"}]

Author rebuttal:
We thank the reviewers for positive comments acknowledging that our work supporting region-level reasoning is "important and interesting tasks for many applications" (reviewer Cm2R), and "represents a novel solution to a more general multi-modal model" (reviewer Ta54). Reviewer UzVU notes that "this is the first work about automatically acquiring visual commonsense" that is "well-designed" (reviewer MRhi) and "workload is impressive" (reviewer uG1j).

We now address some common concerns shared by the reviewers.

**1. Generalization of localized reasoning corpus to other visual reasoning tasks**

In response to Reviewer Cm2R and UzVU, we measure the effectiveness of the localized knowledge corpus on other vision-language tasks not limited to datasets with no bounding box annotations. We specifically focus on ones that require high-level reasoning that would benefit from visual commonsense corpus:
  - AOKVQA [A]: that require outside world-knowledge to answer questions evaluated on multiple choice
  - SNLI-VE [B]: inference based visual entailment to test fine-grained image understanding
  - Visual7W [C]: Visual QA with focus on visual grounding and evaluations carried out on TellingQA.

Table 1 in the rebuttal displays the results of zero-shot and fine-tuned models on the considered visual reasoning tasks. We see that applying LSKD to BLIP-2 which is the best zero-shot model provides empirical improvement in every downstream tasks. This suggests that our corpus can be extended to improve on various reasoning tasks, not limited to localized reasoning, and further strengthens our contribution of corpus creation.

[A]: A-OKVQA: A Benchmark for Visual Question Answering using World Knowledge [Marino et. al.]

[B]: Visual Entailment: A Novel Task for Fine-Grained Image Understanding [Xie et. al.]

[C]: Visual7W: Grounded Question Answering in Images [Zhu et. al.]

**2. Ablation studies of localized corpus and comparison with existing, related corpus**

Several reviewers have requested to quantify the effect of filtering and dataset size, and make comparison of our localized commonsense knowledge (LCK) corpus with related resources. 

***Effect of Filtering on Downstream Performance***

In Figure 3 of the main paper, we have presented the effect of data quality controlled by filtering threshold set by the critic model. We keep the training dataset as 300K and sample knowledge statements scored higher than the threshold by the critic. We found that models trained without filtering (indicated by a threshold value of 0) always performed worse than filtered datasets at any varying threshold for downstream tasks. We also see near linear trend based on the threshold value, in which **increasing the dataset quality based on the supervised critic score can lead to increase in downstream performance**. 

***Comparison with existing corpus***

The LLaVA dataset consists of 150K instructions on question-answers, summary, and conversations generated by GPT-4 with no supervised critic. In **Table 2 of the rebuttal**, we report downstream results of BLIP-2 ViT-G trained with our localization corpus and LLAVA. To keep the comparison fair with LLAVA, we include results with same number of training data (150K). We notice that training on LLAVA corpus yields far less results on the localized reasoning benchmarks (VCR, Sherlock, VisualCOMET) than our subset of corpus. This is expected due to lack of support of localized reasoning in the LLAVA data. On the other hand, LLAVA provides benefits on other visual reasoning tasks, suggesting that the choice of corpus should be tailored to the specific task to optimize performance. We also notice that the critic filtering is again beneficial across all the tasks in the smaller training subset.   
 
Next, we compare with existing human-annotated visual commonsense knowledge corpus (reviewer MRhi). In Table 2 of the rebuttal, we observe that training on human annotated corpus vastly improves the performance of their relative tasks (e.g. training on VisualCOMET boosts the VisualCOMET performance from 39.0 to 50.2). However, we notice that this could lead to worse results than other visual reasoning tasks than the zero-shot counterpart (drops from 77.1 to 70.1/70.8 in Visual7W). This suggests that human-designed datasets may lead to less task generalization due to their lack of diversity, while such trend is not observed in LSKD.

**3. Ensuring Dataset Quality**

Figure 1 in the rebuttal shows qualitative examples to understand the patterns of critic model in distinguishing good and bad examples. We see that the model mostly relies on incorrect visual details (highlighted as red) lower than the correct instances. The third instance does not have glaring visual errors but are scored lower due to statement of "largest and most prominent regions", which is ambiguous but close to false. The critic model displays good calibrations in ordering the instances, such as giving the lowest score to the instance with the most visual error.

**Human Evaluation of Filtered Dataset**

We additionally run human evaluation to measure the acceptability of data with and without filtering. We collect 500 instances the same way critic model labels are collected in Sec 2.3: 1) is the QA visually correct? and 2) does the rationale justify the answer? Likert scores from [1-3] are calculated for each criteria (higher the better).

|                | With Filtering (threshold=0.8) | Without filtering |   |   |
|----------------|----------------------|-------------------|---|---|
| QA Correctness | 2.41 $\pm$ 0.74      | 1.97 $\pm$ 0.84   |   |   |
| Rationale Justification     | 2.34 $\pm$ 0.82      | 1.87 $\pm$ 0.87   |   |   |
|                |                      |                   |   |   |

The results further suggest that applying filtering ensures higher dataset quality not only measured by downstream task performances, but also confirmed by human judgment.


Meta-review: The work received primarily positive reviews from 4 reviewers, who found the idea novel and interesting. Questions were raised during the review period regarding data quality, generalization capabilities, ablation studies, etc., which were effectively addressed in the authors' responses to some extent. The authors are highly encouraged to incorporate details from the rebuttal into the final version for completeness.
Decision: Accept (poster)