ImageNetVC: Zero- and Few-Shot Visual Commonsense Evaluation on 1000 ImageNet Categories
Authors: Heming Xia
Qingxiu Dong
Lei Li
Jingjing Xu
Tianyu Liu
Ziwei Qin
Zhifang Sui
Venue: EMNLP 2023 Findings (2023)
Keywords: visual commonsense
large language model
visually-augmented language model

Abstract: Recently, Large Language Models (LLMs) have been serving as general-purpose interfaces, posing a significant demand for comprehensive visual knowledge. However, it remains unclear how well current LLMs and their visually augmented counterparts (VaLMs) can master visual commonsense knowledge. To investigate this, we propose ImageNetVC, a human-annotated dataset specifically designed for zero- and few-shot visual commonsense evaluation across 1,000 ImageNet categories. Utilizing ImageNetVC, we benchmark the fundamental visual commonsense knowledge of both unimodal LLMs and VaLMs. Furthermore, we analyze the factors affecting the visual commonsense knowledge of large-scale models, providing insights into the development of language models enriched with visual commonsense knowledge. Our code and dataset are available at https://github.com/hemingkx/ImageNetVC.

IMAGENETVC: Zero- and Few-Shot Visual Commonsense Evaluation on 1000 ImageNet Categories
Heming Xia 12; Qingxiu Dong 1.3, Lei Li Jingjing Xu',
Tianyu Liu', Ziwei Qin Zhifang Sui
National Key Laboratory for Multimedia Information Processing, Peking University
School of Software & Microelectronies, Peking University
"School of Computer Science, Peking University *Shanghai Al Lab AAlibaba (xiaheming, szf )ēpku. cespku.edeee edu. cn; {dqx, qinziwei jainzieei@@stu. eestu. pku. edu. cn
Abstract
the dog in pictune smillig or noc?
[s] Visual information specific image
Recently, Large Language Models (LLMs) have been serving general-purpose inter- faces, posing significant demand for compre- hensive visual knowledge. However, it remains unclear how well current LLMs and their vi- sually augmented counterparts (Val.Ms) can master visual commonsense knowledge. To in- vestigate this, propose IMAGENETVC, numan-annotated dataset specifically designed for zero- and few-shot visual commonsense evaluation across 1,000 ImageNet categories. Utilizing IMAGENETVC. we benchmark fundamental visual commonsense knowledge of both unimodal LLMs and VaLMs. Further- more, we analyze the factors affecting the vi- sual commonsense knowledge of large-scale models, providing insights into the develop- ment of language models enriched with vi- sual commonsense knowledge. Our code and dataset are available at https: /gi thub. com/ DemiiggxxImmgeNNe
Yes, it smiling.
What the Samoyydd?
(b) Visual commonoeesse most relevent images
imagie..... Samoyads and wiit
Figure I: Illustration of Visual Commonsense. Visual commonsense refers to the general visual knowledge that is commonly shared across the world, as opposed the visual information that is specific to a single image. Visual Commonsense can be captured through series of related images.
Norlund et al., 2021). As shown in Figure 1, vi- sual commonsense evaluation aims to evaluate the model's 's understanding of commonly shared human knowledge about generic visual concepts, includ- ing color (Bruni et al., 2012; Norlund et 2021; Zhang al., 2022a), spatial relations (Liu et al., 2022), relative sizes (Bagherinezhad al., 2016), etc. Despite their insightful investigations, these studies still have the following limitations from two sides: 1) data side: some research mines visual commonsense attributes based on frequency distri- butions in plain text corpora, which diverges from human visual perception and exhibits additional textual bias (Zhang et al., 2022a); 2) model side: most existing evaluations only focus on a specific model group, lacking a comprehensive exploration of various model families (Bagherinezhad et al., 2016; Norlund et al., 2021; Liu et al., 2022).
1 Introduction
With the breakthrough progress of Large Language Models (LLMs) in recent years (Brown et al., 2020; Zhang et al., 2022b), LLMs are gradually adopted as general-purpose, APT interfaces (e.g., ChatGPT)). addition to language, these intelligent agents, are further required to understand vision knowl- edge (Hao et al., 2022), especially the visual per- ception, which is crucial for real-world interactions such as commonsense reasoning (Talmor et al., 2019), recipe generation (Agarwal al., 2020), and robotic navigation (Shah al., 2022).
However, current studies lack a systematic evalu- ation on how well these widely-used LLMs and their variants are capable of visual understand- ing. Recent research proposes to evaluate the visual capability of models through visual com- monsense evaluation (Bagherinezhad et al., 2016;
In this work, we propose that similar to human beings, models can also answer intricate visual commonsense questions with related images (il- lustrated in Figure 1). To this end, we introduce IMAGENETVC, a unified zero- and few-shot visual commonsense benchmark incorporating multiple sources of images (c.g., ImageNet (Deng et al., 2009), search images, and synthetic images). From
"Co-first authors with equal contributions
'https://che //chat, orrcchaat.ppnii.....m openai. com

---

the data side, IMAGENETVC comprises 4,076 high- quality QA pairs, encompassing 1,000 ImageNet categories across diverse domains such as color, shape, material, component, etc. Moreover, as a human-annotated dataset, IMAGENETVC uti- lizes human visual perception to identify shared attributes across relevant images, avoiding textual bias and providing data that is more closely aligned with human knowledge. From the model side, be- sides unimodal LLMs, IMAGENETVC also enables the evaluation of various auggentee Lan- guage Models (VaLMs) to investigate the effect of visual grounding, which compensates for the lack of images in previous benchmarks.
sual data (e.g.. images) during pretraining restricts their visual commonsense capabilities (Li et al., 2023b). On the other hand, Visually-augmeette Language Models (VaLMs) have gained popularity by integrating visual information into LLMs (Tsim- poukelli et al., 2021; Alayrac et al., 2022). which enhance the visual understanding capabilities of language models (Yang et al., 2022; Wang et al., 2022).
Visual Commonsense Evaluation Visual com- monsense knowledge of visual concepts is a fun- damental and critical aspect of Al systems seeking to comprehend and reason about the world (Yao et al., 2022; Dong et al., 2022). Previously, sev- eral datasets have been proposed to address specific attributes of visual commonsense, including Memo- ryColors (Norlund al., 2021), ColorTerms (Bruni et al., 2012), RelativeSize (Bagherinezhad et al., 2016), and Spatial Commonsense (SpatialCS) (Liu et al., 2022). To evaluate general visual common- sense, Zhang et al. (2022a) introduced ViComTe, a template-based dataset consisting of various (sub- ject, object) pairs (such as (sky, blue)). However, its reliance pure textual input underestimates the visual capabilities of VaLMs. Furthermore, its utilization of template-base based formats and auto- matic extraction techniques leads to substandard data quality and inherent textual biascs.
With IMAGENET VC. we conduct extensive eval- uations on the most widely-used LLMs and VaLMs. We benchmark the visual commonsense capabili- ties of various LLMs such as OPT, LLaMA, and Falcon and assess the effect of visual grounding in VaLMs with multiple sources of relevant images. We further analyze the co-founding factors that may affect the visual commonsense capability of mod- els, such as model scale, in-context learning, and image sources. We highlight several experimental findings. These findings support the high value of our benchmark in assessing visual commonsense capabilitics.
Template- datasets yield artificially in- flated and unstable visual commonsense eval- uation, while manually constructed IMA- GENETVC provides evidence that visual com- monsense remains challenging for LLMs.
In this work, we introduce IMAGENETVC, human-
visual commonsense evaluation dataset that consists of 4K natural language QA pairs across various visual attributes, which sup- ports both LLM and VaLM evaluation with multi- ple sources of images. We present detailed compar- isons of VMAGENETVC with prior work in Table 1.
We discover that the acquisition of visual com- monsense is an emergent ability for LLMs. For instance, 1.3B could be a potential thresh- old for unimodal LLMs to emergent with vi- sual commonsense on the component.
3 IMAGENETVC
In-context learning enhances the understand- ing of visual commonsense tasks for both LLMs and VaLMs, not only reducing their variance across prompts but also calibrating the model confidence visual commonsense.
Starting from ImageNet, we construct our IMA- GENETVC dataset in multi-step crowd-sourcing pipeline, including annotator training, 2) com- monsense annotation, and 3) cross-check examina- tion. An overall demonstration of our annotation process is illustrated in Figure
2
Related Work
3.1 Image Source
Large Language Models Text-only Large lan- guage models (LLMs) have exhibited outstanding performance across various textual commonsense tasks, benefiting from their training on extensive textual data (Radford et al., 2019; Raffel et al., 2020; Brown et al., 2020). However, the lack of vi-
We selected ImageNet (Deng et al., 2009) as our im- age source because it covers large number of com- monly used objects in real-life situations, providing a diverse and representative image source. Addi- tionally, the unified image format ImageNet with

---

Gategory: Samoyed
C What the color Samoyed's body?
Ag White
QA
X
R
O3 What shape are Somoyed's ears?
Ay: Triangle
IMAGENET
Q3: is the dog the picture smiling or not?
Ast Yes
C. Ag
RA X
Q4: Do Samoyeds have spots on their bocks?
AAg: No
color
Shope
Oy
Axs
octers
Senata
Image Source
Phase I: Cummonsense Annotation
Phase 2: Cros-Check Examination
Figure 2: An overall demonstration of the construction procedures IMAGENETVC.
Human Annotation
Multi-attribute Evaluation
Support VaLM
Region-based Question
Natural Language
Dataset
#Category
#Test
MemoryColors (Norlund et al., 2021)
X
109
109
ColorTerms (Bruni al., 2012)
X
52
RelativeSize (Bagherinezhad et al., 2016)
X
486
SpatialCS (Liu et al., 2022)
X
59
1224
ViComTe (Zhang et al., 2022a)
X
X
3957
2223
IMAGENETVC (Ours)
1000
4076
Table I:
Features and statistical information of ImageNetVC and prior related datasets. The '# Category' column indicates the number object categories included, and Test' means the number of test samples in the dataset.
dimensions of 256x256 pixels facilitates annota- tors' understanding of images and reduces feature engineering. Specifically, we used the widely-used ImageNet (ILSVRC) 2012 subset,² consisting of 1.4 million images from 1,000 object categories.
color of Samoyed's body is white. They then cre- QA pair based on this feature if it aligns with their commonsense understanding of life, such What is the color of a Samoyed's 's body? White.
To ensure that the QA pairs reflect visual com- monsense rather than visual information tailored to specific images, annotators instructed focus on the visual features of each category rather than individual images. They are also provided with annotation examples and guidelines for rejection. The annotation UI and specifications for annotation can be found in Appendix A.
3.2 Prerequisite: Annotator Training
We posted online job listings on Amazon Mechani- cal Turk² and received over 500 applications from candidates with Bachelor's degrees or higher. To ensure dataset quality, we provided training with in- structions and guidelines and quick quiz assess candidate understanding. Only candidates with scores larger than 95% are hired.
3.4 Phase 2: Cross-Check Examination
3.3 Phase 1: Commonsense Annotation
The primary objective of the cross-check examina- tion phase to conduct rigorous screening and categorization of high-quality QA pairs that meet our requirements. This phase comprises two stages, In Stage 1, a category-level examination is per- formed, where three examiners are assigned to all annotated QA pairs in the same category. They are required to check all the pairs in the category based the annotation instructions, rectify any gram- matical errors, and eliminate low-quality or non- compliant pairs. Only the QA pairs that all three examiners approve are deemed acceptable. Stage involves a sample-level examination. Although the examination in Stage is efficient, examining
Figure 2 shows the commonsense annotation phase, where annotators are provided with category names and 50 randomly sampled images per category. They are instructed to form question and answer considering shared visual features of the images and their own commonsense knowledge. Visual features may be object-based, such as the color of a entire object, or region-basee such as the color specific object part. Annotators first identify a common visual feature of the category, such as The
Pimage-net, www.eett...o...eoooooooooo /etphhhleeeggs///////22222222 Phttps: //www mturk. com/

---

ImageNetVC Wen
Tie
ImageNetVC Lost
Color Shape Mater. Compo.
Others Total
Diversity
Labels
11
12
16
2
55
91
Categories
439
367
405
560
768
1000
Difficully
Samples
557
424
430
1114
1551 4076
Factuality
0%
20%
40%
100%
Table Statistical information of IMAGENETVC. Mater. and Compo. the abbreviations of Material and Component, respectively.
(a) Comparison with ViComTe
Diversity
57
10
3.3
Difficulty
45
18
37
ViComTe
Factuality
81
12
Accuracy (%)
ImageNetVC
0%
20%
40%
60%
80%
100%
40
(b) Comparison with ChatGPT Generated Data
30
Figure Human assessment of visual commonsense dataset from three aspects: diversity, difficulty, and fac- tuality. IMAGENETVO outperforms ViCom'Te in terms of diversity and difficulty, while also demonstrating perior factuality compared to ChatGPT generated data.
20
10
color
shape
material
Figure 3: Zero-shot performance distribution of GPT-
Neo- 1.3B across different prompts. Green triangles rep- resent mean results, black diamonds represent outliers. Compared the ceselate-bseed dataset, ViComTe, IM- AGENETVC demonstrates notably reduced evaluation variance across various prompts.
2021) with respective subsets of IMAGENETVC and ViComTe, a widely-used dataset, actoss differ- ent prompts.* Results in Figure 3 indicate that, template-based dataset, ViComTe exhibits severe prompt bias, with substantial evaluation variance across different prompts. E.g.. the model achieves only 2% accuracy with the prompt "X of shape Y" but achieves 61% score with "The shape of the X is Y". Besides, compared with ViComTe, IM- AGENETVC containing region-based questions is more challenging to models. For example, with the suitably selected prompt on the color subset, the model achieves 40% accuracy on ViComTe but only 28% accuracy on IMAGENETVC.
all QAs in one category simultaneously creates a misalignment with the final testing method (one- by-one QA) and introduces a distribution bias in the examination process. Therefore, in Stage 2, more thorough sample-level examination is carried out. Three examiners are randomly assigned QA pair with the corresponding category name from the entire dataset. They vote on whether to accept the QA pair and classify into the following five subsets: color, shape, material, component, and others. Only the sample that receives the majority vote is approved for acceptance.
We further conducted a human assessment be- tween ViComTe, IMAGENETVC, and QA pairs automatically generated from ChatGPT: Specifi- cally, we provided human annotators with sampled data from the two comparison datasets and asked them to vote for the better considering diversity, difficulty, and factuality. As depicted in Figure 4, in more than 84% of cases, IMAGENETVC C out- performs or matches the template-based ViComTTe in terms of diversity and difficulty, which is con- sistent with the results in Figure 3. Moreover, our dataset demonstrates notably higher factual correct- ness compared to the data automatically generated by ChatGPT in more than 81% of cases.
Our 60-day annotated dataset comprises 4,076 items (refer to Table from 1000 ImageNet cate- gories. It consists of 5 individual sub-tasks: color, shape, material, component, and others. More in- formation and examples of IMAGENETVC can be found in Appendix B. All pricing strategy details and hierarchical supervision process employed are elaborated in Appendix A.3 and A.4.
3.5 Dataset Evaluation
Unlike previous datasets which are template-based, IMAGENETVC comes from diverse real images as- sociated with numan-annotann descriptions, which can better represent real-world settings. To assess the strengths of our dataset, conduct automatic evaluation and human evaluation in this section.
To sum up, our data collection process ensured high-quality annotations with minimal bias and increased diversity, difficulty, and factuality, pro-
*Prompt details are provided Appendix C.I. Please refer to Appendix C.2 for the detailed process of our designed human assessment.
First, we implement GPT-Neo-11 -e0-1.BB (Black et al.,

---

viding a challenging dataset for advancing research in visual commonsense understanding.
model to calculate the score for the answer y E y:
8x x) = log PM (z: wes)
4 Experiments
log Par (ec(ar) et(wes))
Our experiments primarily focus on two types of language models: LLMs and VaLMs. Both models have demonstrated promising capabilities in under- standing visual information (Li et al., 2023b).
where M' denotes the transformer neural network in M, P(.) is the output probability given by the model. Then we obtain a probability distribution over all answer candidates using softmax:
Large Language Models We begin with the text- only setting, to explore the visual commonsense capability of LLMs learned from the textual corpus. We focus the dominant auto-regressive LLM family and benchmark a lot of model variants, in- cluding the GPT (Black et al., 2021; Wang and Ko- matsuzaki, 2021; Black al., 2022), OPT (Zhang et al., 2022b), LLAMA (Touvron et al., 2023), Fal- con (Almazrouei et al., 2023), and Pythia (Bider- man et al., 2023).
(alue)
g: (v
Ly' etla' a4)
We calibrate the prediction by normalizing the prob- ability distribution following Zhao et al. (2021), to mitigate the bias introduced by prompt formats as well as few-shot examples.
4.1.2 VaLM Evaluation
We incorporate two types of image sources addi- tional visual inputs for evaluating VaLMs: images retrieved from the web and synthetic images. We adopt Google Image Search to retrieve relevant im- ages and Stable Diffusion (Rombach et al., 2022) for image synthesis. Following Yang et al. (2022), for each QA pair, we utilize CLIP (Radford et al.. 2021) to sort images from these two sources based on their similarity with the question and then pre- serve top-K K images the final image sources. We mainly evaluate two types of VaLMs: prefix-based VaLMs and ensemble-based VaLMs.
Language Models In our experiments, mainly evaluate three widely-used open-source VaLMs: Z-LaVI (Yang et al., 2022), BLIP-2 (Li al., 2023a), and MAGMA (Eichen- berg et al., 2022). These VaLMs are mainly built on top frozen LLMs and incorporate diverse mech- anisms to integrate visual information. Further model details are provided in Appendix C.4.
4.1 Evaluation Methods
Given a QA pair with an image prefix- based VaLMs (e.g., BLIP-2 and MAGMA) first utilize visual encoder to transform the image into sequence of visual embeddings e, {el ...,."" Then, these embeddings are prefixed into the text embeddings of and put into the frozen LLM backbone to calculate the score:
Prasex-based VaLMs
In this work, we focus on evaluating the zero- and few-shot visual commonsense of LLMs and VaLMs IMAGENETVC. Following Schick and Schütze (2021) and Yang al. (2022), we treat the zero- shot evaluation as cloze test, transforming the QA pairs in IMAGENETVC into prompts like "[Ques- tion] The answer is [Answer]."'. n6 Formally, each QA pair is converted into a sequence of tokens x (zo. .... )n), }.n)inn in which is the answer. In the few-shot setting, examples with the same prompt concatenated before each QA pair.
a(y u. v. log PM (x: v. x.)
log Par (e:(2)) e (x(ee)))
The probability distribution with the image is calculated over all answer candidates, which is the same as Eq (1), If K images are provided, the final distribution will be averaged over images:
4.1.1 LLM Evaluation
Given LLM M, the sequence of input tokens T {mo, _n will first be mapped text em- beddings e; = (ex(2o),... ez(z), e { by the embedding layer e, E M. Then we utilize the
<qy x))
1 K
Σ
givl ( wx)
(2)
Following Dong et al. (2023), for prefix-based based VaLMs supporting few-shot evaluations, examples
'All the prompts utilized for the evaluation of M. and Ms are shown in Appendix C.I.

---

(a) Zero-Shot (LLM-7B)
[b) Few-Shot (LLM-78)
(c) Few-Shot (Best LLMs)
mater
hape
matergl
nape
materal
mnape
61
others
pefmponent
pomponnnt
0th
poneponent
(d) Zero-Shot (VGLM with OPT-7B)
(e) Zero-Shot (VaLM with GPT-j)
(f) Fow Few-Shot (VGLM with GPT-))
color
color
color
hape
077-78
GF75448
MACHA Iuto
others
cempoonen
Z4aM
others
63 Cemponent
lict
oth otheens
ponppntntt
DUP-2
9AGMA
MAGHA
Figure Radar plots for five individual sub-tasks in IMAGENETVC. We show evaluation results with four experimental settings: (a, b) Zero- and few-shot evaluation with LLMs-7B; (c) Few-shot evaluation with the best LLMs in their own model family; (d, e) Zero-shot evaluation with VaLMs and their frozen LLM backbones; (f) Few-shot evaluation with VaLMs. The numbers along the radio axis denote the mean Top-1 accuracy (%) of models over 5 different prompts. The detailed results for drawing these plots are shown in Appendix D.
with the same processing will be concatenated front of each QA pair. Since prefix-basee VaLMs utilize frozen LLM backbones, they can be regarded as conditional extension of text-only LLMs. Evaluations between these two model types facilitate thorough assess- ment of the effect of visual grounding on visual commonsense ability.
(o()), x000 |=
weighted over the output distributions of LLMs and CLIP:
< u. (1 w) g=(3/ ( m)) w go(2 v.
where w denotes the weight hyperparameter.
4.2 Experimental Details
We adopt Google Image Search to retrieve relevant images and utilize the newly released Stable Diffu- sion (Rombach et al., 2022) for image synthesis. Following Yang et al. (2022), for each QA pair in IMAGENETVC.
Ensemble-based VaLMs Given the input tokens a and multiple images v = [x00) / ensemble- based VaLMs (e.g.. Z-LaVI) utilize a frozen CLIP model, which contains a text encoder fe and a vi- sual encoder fe, to project the tokens æ and the image (88) into a shared representation space and compute the relevance score between them:
obtain 100 images with each of the two methods. These 200 images sorted using CLIP based on their similarity with the ques- tion. We preserve top-10 (K = 10) images for each QA pair as the final image sources. The other experimental details, such as the model implemen- tation, hyperparameters, and computing resources are presented in Appendix C.3.
s.(y (0 x) = cos [f(()) (o(o
Then, same as Eq (1) and Eq (2), the probability distribution over all answer candidates and across K images is obtained:
4.3 Main Results
The main evaluation results of LLMs and VaLMs on IMAGENETVC are shown in Figure 5. Here, highlight several interesting findings.
go v. Σ softmax (s.(2)
( )),,))))
where softmax(-) is a simplified denotation of Eq (1). The final ensemble score is calculated as
htttps: thub. comm/oopVis/stalleeoooo fisiii and we use the sd-v2-1 checkpoint.

---

Accurary (0-chat)
Pynta
46
55
Acurscy
50
23
12
66
Billions of Parameters
45
Accuracy (d0)
40-
30
Billons of Parameters
23 Billons of Parameters
Figure 6: Performance distribution of LLaMA the color subset. Models with ICL achieve higher perfor- mance and show reduced variance across prompts.
Figure Performance of LLMs the component sub- set of IMAGENETVC. LLMs with model size larger than 1.3B demonstrate emergent capabilities when soly- ing the component sub-task.
Falcon and LLaMA excel in all four presented LLM model families, especially on the color and component sub-tasks. As shown in Figure 5(a, b), Falcon and LLaMA consistently outperform OPT and GPT across various subsets with both experi- mental settings, despite the shape subset. Particu- larly, LLaMA achieves a zero-shot accuracy of 41 % on the color subset, surpassing GPT-J with con- siderable margin of 13%; Falcon yields the highest few-shot accuracy of 76% on the component subset, which amounts to 14% absolution improvement over OPT. We further present the few-shot results of the largest available LLMs in their own model family in Figure 5(c), where LLaMA-65B shows remarkable superiority over other counterparts.
rating visual information indeed helps to improve LLMs' visual commonsense capabilities. However, the results also show that the performance gains of VaLMs are small on some sub-tasks: both BLIP-2 and MAGMA only achieve an 0.5% accuracy im- provement on the shape sub-task, while Z-LaVI even has performance drops. This demonstrates that VaLMMs still have wide room for improvement. ICL capability of VaLMs should be further valued. As shown in Figure 5(f), MAGMA with ICL achieves consistent improvements across all subsets over itself and the few-shot results the frozen LLM backbone, indicating that ICL could also improve VaLMs' visual commonsense perfor- mances. However, ICL. has been somewhat under- investigated by previous VaLM research. For ex- ample, both Z-LaVI and BLIP-2 only support zero- shot evaluation, with the lack of ICL capability. We hope our research could draw more attention to future work on the ICL capability VaLMs.
In-context learning (ICL) not only improves the visual commonsense performance of LLMSs but also reduces their variance across different prompts. Comparing the results Figure 5()) and 5(b), we found that given few examples (i.e.. with ICL), LLMs achieve consistent and remark- able improvement over themselves. For instance, LLaMA-7B with ICL achieves average score of 62% across five sub-tasks, with 12% improve- ment over the zero-shot result. We further show the performance distribution LLaMA across differ- ent prompts in Figure 6, which illustrates that ICL not only improves the model's performance but also reduces its variance across different prompts. Further analysis is conducted in Section 5.
Analysis
We further investigate the factors influencing the visual commonsense capabilities of LLMs and VaLMs. For instance, we find that decent scale (e.g., 1.3B) could be potential threshold for text- only LLMs learn visual commonsense. We then analyze several influencing factors of VaLMs, such as image sources and image numbers.
VaLMs improve the visual commonsense abil- ity of their LLM backbones, despite small per- formance gains the shape subset. As depicted in Figure 5(d, e), BLIP-2 shows remarkable supe- riority over OPT on the color and material subset, with average accuracy improvements of 17% and 15%, respectively, which indicates that incorpo-
When (at what model scale) do text-only LLMss learn visual commonsense? We show the zero- and few-shot results of three LLM families the component subset in Figure Take the component sub-task as an example, we find that decent scale (c.g., 1.3B) could be starting point for LLMs to emerge with visual commonsense on the com-

---

perfecty Brotod
COL.
SHa.
MAT.
Cом.
ОTH.
Ave
aeroeluu
80
OPT-2.7B
25.8
39.9
40.2
51.3
48.1
41.1
Accurcc (%)
26.4
40.5
50.9
48.8
41.3
RANDOM
20.1
38.9
42.0
51.9
47.4
40.9
52.9
SYNTHESIS
43.2
39.5
59.2
53.8
51.3
49,4
RANK
44.7
40.3
61.9
$4.0
31.7
50.5
20
Table 3: Zero-shot results of BLIP-2 with OPT-2.7B on IMAGENETVC with various image sources. means no image is provided to the model.
60
100
(%)
Figure 8: Calibration results of LLaMA-7B on the com- ponent subset. ICL greatly enhances model calibra- tion, significantly boosting the correlation between con- fidence and accuracy from 0.57 0.98.
Accurasy
48
22
30
40
images
ponent": smaller models at sizes below 1.3B are unable to perform well on the task, with a perfor- mance close random guessing (i.c., -50% accu- racy); while models with size larger than 1.3B exhibit gradual performance improvements. For example, OPT-30B achieves 59% and 66% average accuracy with zero- and few-shot settings, respec- tively.
Figure 9: Average performance on IMAGENET with various numbers of top-ranked images. The results are obtained with BLIP-2 (OPT-2.7B).
VaLMs' visual commonsense demands relevant im- ages as input to be suitably stimulated. Among the various image sources, CLIP-ranked images yield the best performance, suggesting that align- ing images closely with the question facilitates the generalization of related visual commonsense by models. Thus, image source for our main experiment and analysis.
What is the effect of ICL on the calibration of LLMs? Ensuring the reliability of a model's pre- dictions is a crucial aspect of model evaluation, mis-calibrated models can lead to incorrect in- ferences and serious consequences in real-world applications (Guo et al., 2017). To this end, con- ducted calibration analysis evaluate whether the model confidence on visual commonsense reli- ably reflects the actual probability of the prediction being correct. Our analysis focuses on the calibra- tion of LLaMA-7B on the component subset of IMAGENETVC. Results in Figure 8 indicate that ICL significantly improves the calibration of the model, increasing the correlation between confi- dence and accuracy from 0.57 to 0.98. This consistent with the aforementioned findings, suggesting that ICL, improves the visual common- sense performance of LLMs.
ranked images the default
What the typical number of images required to capture visual commonsense? We show the model's average performance on IMAGENETVC with various numbers of images in Figure 9. The re- sults show that the model's performance increases as the number of top-ranked images gradually increases from age sources help the model to capture general sual commonsense. However, the improvement is marginal when the image number larger than 10.
10, indicating diverse im-
Visual Commonsense in Other Models
How do image sources influence VaLMs? Ta- ble shows the ablation results of the image sources used in VaLMs. As illustrraedd BLIP-2 with extra image sources (c.g., SEARCH) brings large improvements. This supports IMAGENETVC's mo- tivation, suggesting that previous visual common- sense evaluations undervalue VaLMs' potential, as
is worth noting that, as a general visual com- monsense dataset, IMAGENETVC supports vari- ous types of models and evaluation settings. Ex- cept for the evaluation setting in our main results, also evaluate several models in the setting of open-ended generation. Specifically, we select widely-used multimodal models, 12-in-1 1 (Lu al., 2020) and BLIP (Li al., 2022) that are finetuned on the VQAv2 dataset (Goyal et al., 2017) and the famous RLHF model, ChatGPT (gpt-3. (gpt-3.5-turie 5-turbo)
*Plcase note that, as the evaluated LLMs (OPT and Pythia) both rely on the Pile (Gao et al., 2021) as their pre-training corpus, our findings may not he generalized to other LLMss.

---

Models
COL.
SHA.
MAT.
CоM.
ОтH.
AvG
large amounts of text data. Thus, it remains un- clear whether fine-tuning on visual commonsense tasks would improve the performance of LLMs and
12-in-1
70.0
65.6
65.0
72.7
67.9
68.2
BLIP
77.0
70.0
73.9
71.5
69.7
72.4
CharGP
58.0
62.2
69.7
78.8
66.9
67.1
VsLMs. Therefore, this may not fully capture a model's ability to understand visual commonsense in real-world scenarios where prior knowledge may be available. Besides, although we explored the fac- tors that affect the visual commonsense knowledge of large models, it is still challenging to interpret how the models aequire this knowledge.
L
75.0
87.5
76.2
75.8
Humans
91.2
92.7
90.5
98.9
94.0
93.5
Table 4: Evaluation results of multimodal models and ChatGPT IMAGENETVC. We report Top-1 accuracy results obtained by open-ended generation. We also show human performance in the last row reference.
Overall, our study provides a foundation for future work in the field of visual commonsense knowledge and its applications. However, addi- tional research is necessary to address the afore- mentioned limitations and further advance the field.
for evaluation." 9 As illustrated in Table 4, the mul- timodal models finetuned on VQAv2 show strong performance on IMAGENETVC, especially on the color sub-task, with relatively small model scales (e.g., 583M of BLIP). ChatGPT with ICL achieves the best average accuracy score of 75.8% 8% across all compared models. However, it still has consider- able performance gap with humans, which has an average performance of 93.5%.
Acknowledgements
This paper is supported by the National Key Research and Development Program of China 2020AAA0106700 and NSFC project U19A2065.
7 Conclusion
References
In this paper, we introduced IMAGENETVC, com- prehensive Guman--nnooaale dataset for evaluat- ing visual commonsense using both textual and visual imputs. We conducted extensive experiments evaluate the visual commonsense of both uni- modal LLMs and VaLMs using IMAGENETVC. C. Our results demonstrate the varying degrees of vi- sual commonsense knowledge present in different models, as well as the factors that contribute to the acquisition and enhancement of this knowledge. Additionally, we offer insights into the emergent abilities of LLMs and the strengths of VaLMs in the realm of visual commonsense.
Yash Agarwal. Devansh Batra, and Ganesh Bagler. 2020. Building hierarchically disentangled lan- guage models for text generation with named enti- ties. In Proceedings of the 28th International Confer- ence on Computational Linguistics, COLING 2020, Barcelona, Spain (Online), December 8-13, 2020, pages 26-38. International Committee on Computa- tional Linguistics.
Cean-Baptiste Alayrac Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et 2022. Flamingo: a visual language model for few-shot learning. Advances in Neural Information Processing Systems, 35:23716 15::37166-2377 16776636.....
Limitations
Ebtesam Almazrouei, Hamza Alobeidli, Abdulaziz Al- shamsi, Alessandro Cappelli, Ruxandra Cojocaru, Merouane Debbah, Etienne Goffinet, Daniel Hes- low, Julien Launay. Quentin Malartic, Badreddine Noune, Baptiste Pannier, and Guilherme Penedo. 2023. Falcon-40B: an open large language model with statc-of-the-art performance. CoRR.
While our study provides valuable resources and insights into the visual commonsense knowledge of LLMs and VaLMs, several limitations need to be acknowledged. Firstly, due to the high cost of ragion-based human annotation, the IM- AGENETVC dataset only covers 1,000 ImageNet categories, which may not cover all real-world see- narios. Therefore, it is possible that models may perform differently on other types of images that are outside of the IMAGENETVC categories.
Hessam Bagherinezhad, Hannanch Hajishirzi, Yejin Choi. and Ali Farhadi. 2016. Are elephants bigger than butterflies? reasoning about sizes of objects. Proceedings of the Thirtieth AAAI Conference on Ar- rificial Intelligence, February 12-17, 2016, Phoenix, Arizona, USA, pages 1449-3456... AAAI Press.
Additionally, our study is limited to zero- and few-shot visual commonsense evaluation and only considers models that have been pretrained on
Stella Biderman, Hailey Schoelkopf. Quentin Anthony. Herbie Bradley, Kyle O'Brien, Eric Hallahan, Mo- hammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, Aviya Skowron, Lintang Sutawika, and Oskar van der Wal. 2023. Pythia:
"The evaluation details are illustrated in Appendix E.

---

suite for analyzing large language models across training and scaling. CoRR, abs/2304.01373.
MAGMA multimodal augmentation of generative models through adapter-basee finetuning. In Find- ings of the A Association for Computational Linguistics: EMNLP 2022, Abu Dhabi, United Arab Emirates, De- cember 7-11. 2022, pages 2416-2428. Association for Computational Linguisties.
Sid Black, Stella Biderman, Eric Hallahan, Quentin Authony, Leo Gao. Laurence Golding. Horace He, Connor Leahy, Kyle McDonell, Jason Phang, Michael Pieler, USVSN Sai Prashanth, Shivanshu Purohit, Laria Reynolds, Jonathan Tow, Ben Wang, and Samuel Weinbach. 2022. Gpt-neox-200 An open- autoregressive language model. CoRR, abs/2204.06745..
Leo Gao, Stella Biderman, Sid Black, Laurence Gold- ing, Travis Hoppe, Charles Foster, Jason Phang, Horace He. Anish Thite. Noa Nabeshima, Shawn Presser, and Connor Leahy. 2021. The pile: An 800gb dataset of diverse text for language modeling. CoRR, abs/2101 10100007.......
Sid Black, Leo Gao. Phil Wang, Connor Leahy, and Stella Biderman. 2021. GPT-Neo: Large
Scale Autoregressive Language Modeling with Mesh- Tensorflow.
Yash Goyal, Tejas Khot, Douglas Summers- Summers-Stay, Dhruv Batra, and Devi Parikh. 2017. Making the V VQA matter: Flevating the role of image understanding in visual question answering. In 2017 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017, Honolulu, HI, USA, July 21-26, 2017, pages 6325-62 6334. IEEE Computer Society.
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanic Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Fric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alee Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners. In Ad- vances in Neural Information Processing Systems 33: Annual Conference on Neural Information Process- ing Systems 2020, NeurlPS 2020, December 6-12, 2020, virtual.
Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q. Wein- berger. 2017. On calibration of modern neural net- works. In Proceedings of the 34th International Con- ference on Machine Learning, 1CML 2017, Sydney, NSW, Australia. 6-11 August 2017, volume 70 of Proceedings of Machine Learning Research, pages 1321-1330. PMLR.
Yaru Hao, Haoyu Song. Li Dong, Shaohan Huang, Zewen Chi, Wenhui Wang. Shuming Ma, and Furu Wei. 2022. Language models are general-purpose interfaces. arXiv preprint arXiv:2206.06336.
Elia Bruni, Gemma Boleda, Marco Baroni, and Nam- Khanh Tran. 2012. Distributional semantics in tech- nicolor. In The 50th Annual Meeting of the Asso- ciation for Computational Linguistics, Proceedings of the Conference, July 8- 14, 2012, Jeju Island, Ko- rea- Volume 1: Long Papers, pages 136-145. The Association for Computer Linguistics.
Junnan Li, Dongxu Li, Silvio Savarese, and Steven C. H. Hoi. 2023a. BLIP-2: bootstrapping language-imag pre-training with frozen image encoders and large language models. CoRR, abs/2301.12597.
Junnan Li, Dongxu Li, Caiming Xiong, and Steven C. H. Hoi. 2022. BLIP: bootstrapping language-imen image pre- training for unified vision-language understanding and generation. In International Conference on Ma- chine Learning. ICML 2022, 17-23 July 2022. Balti- more, Maryland, USA, volume 162 of Proceedings of Machine Learning Research, pages 12888-13 -29000 PMLR
Jia Deng, Wei Dong. Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. 2009. Imagenet: A large-scale hier- archical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 238-255. leee.
Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng, Zhiyong Wu, Baobao Chang, Xu Sun, Jingjing Xu, Lei Li, and Zhifang Sui, 2023. A survey for in-context learning. CoRR, abs/2230..
Lei Li, Jingjing Xu, Qingxiu Dong, Ce Zheng, Qi Liu, Lingpeng Kong, and Xu Sun. 2023b. Can lan- guage models understand physical concepts? arXiv preprint arXiv:2305. 14057.
Qingxiu Dong, Ziwei Qin, Heming Xia, Tian Feng, Shoujie Tong, Haoran Meng, Lin Xu, Zhongyu Wei, Weidong Zhan, Baobao Chang, Sujian Li, Tianyu Liu, and Zhifang Sui. 2022. Premise-based multimodal reasoning: Conditional inference on joint textual and visual clues. In Proceedings of the 60th Annual Meet- ing of the Association for Computational Linguistics (Volume I: Long Papers), ACL 2022, Dublin, Ireland, May 22-27, 2022, pages 932-946, Association for Computational Linguistics.
Xiao Liu, Da Yin, Yansong Feng, and Dongyan Zhao. 2022. Things not written in text: Exploring spatial commonsense from visual signals. of the 60th Annual Meeting of the Association for Computational Linguistics ( (Volume 1: Long Papers), ACL 2022, Dublin, Ireland, May 22-27, 2022, pages 2365- Association for Computational Linguis- tics.
Proceedings
Constantin Eichenberg, Sidney Black, Samuel Wein- bach, Letitia Parcalabescu, and Anette Frank. 2022.
Jiasen Lu, Vedanuj Goswami, Marcus Rohrbach, Devi Parikh, and Stefan Lee. 2020. 12-in-1: Multi-task

---

vision and language representation learning. In 2020 IEEE/CVE Conference on Computer Vision and Pat- tern Recognition, CVPR 2020, Seattle, WA, USA, June 13-19, 2020, pages 10434-10443. Computer Vision Foundation/ IEEE.
Hugo Touvron, Thibaut Lavril. Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurélien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023. Llama: Open and efficient foundation language models. CoRR, abs/2302. 13971.
Tobias Norlund, Lovisa Hagström, and Richard Johans- son. 2021. Transferring knowledge from vision to language: How to achieve it and how to measure it?
Maria Tsimpoukelli. Jacob Menick, Serkan Cabi, S. M. Ali Eslami, Oriol Vinyals, and Felix Hill. 2021. Multimodal few-shot learning with frozen language models. In Advances in Neural Information Pro- cessing Systems 34: Anmual Conference on Neural Information Processing Systems 2021, NeurlPS 1021, December 6-14, 2021, virtual, pages 200-212.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sas- try, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. 2021. Learn- ing transferable visual models from natural language supervision. In Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Research, pages 8748-8763. PMLR.
Ben Wang and Aran Komatsuzaki. 2021. GPT-J- 6B: A 6 Billion Parameter Autoregressive Lan- guage Model. https: //gi wtw....com/knggoooll.... fuss-ttranfformer -jax
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. OpenAl blog.
Weizhi Wang. Li Dong. Hao Cheng, Haoyu Song, Xi- aodong Liu, Xifeng Yan, Jianfeng Gao, and Furu Wei. 2022. Visually-augmente language modeling. arXiv preprint arXi:: 10178.
Colin Raffel, Noam Shazeer. Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text trans- former. J. Mach. Learn, Res., 21:140: 1-440:67.
Yue Yang, Wenlin Yao, Hongming Zhang, Xiaoyang Wang, Dong Yu, and Jianshu Chen. 2022. Z-lavi: Zero-shot language solver fueled by visual imagi- nation. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, EMNLP 2022, Abu Dhabi, United Arab Emirates, De- cember 7-11. 2022, pages 1186-1203, Association for Computational Linguistics.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björ Ommer. 2022. High- resolution image synthesis with latent diffusion mod- els. In IEEE/CVÉ Conference on Computer Vision and Pattern Recognition, CVPR 2022, New Orleans, LA, USA, June 18-24, 2022, pages IEEE
Yuan Yao, Tianyu Yu, Ao Zhang, Mengdi Li, Ruobing Xie, Cornelius Weber, Zhiyuan Liu, Haitao Zheng, Stefan Wermter, Tat-Seng Chua, et al. 2022. Visu- ally grounded commonsense knowledge acquisition. arXiv preprint
Timo Schick and Hinrich Schütze. 2021. Exploiting cloze-question for few-shot text classification and natural language inference. In Proceedings of the 16th Conference of the European Chapter of the As- sociation for Computational Linguistics: Main Vol- ume, EACL 2021, Online, April 19 23, 2021. pages 255-269. Association for Computational Linguistics.
Chenyu Zhang, Benjamin Van Durme, Zhuowan Li, and Elias Stengel- 2022a. Visual commonsense in pretrained unimodal and multimodal models. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies, NAACL 2022, Seattle, WA, United States, July 10-1 15, 2022, pages 5321-5335. Association for Computa- tional Linguistics.
Dhruv Shah, Blazej Osinski, Brian Ichter, and Sergey Levine. 2022. Lm-nav: Robotic navigation with large pre-trained models of language, vision, and action. 14-18 December 2022, Auckland, New Zealand, volume 205 of Proceedings of Machine Learning Research, pages 492-504. PMLR.
In Conference on Robot Learning,
Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona T. Diab, Xian Li, Xi Victoria Lin, Todor Mihaylov, Myle Ott, Sam Shleifer, Kurt Shus- ter, Daniel Simig. Punit Singh Koura, Anjali Srid- har, Tianlu Wang, and Luke Zettlemoyer. 2022b. OPT: open pre-trained transformer language mod- els. CoRR, Abs/2205.00068.
Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. 2019. Commonsenseqa: A question answering challenge targeting commonsense knowl- odge. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Tech- nologies, NAACI-HHL 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume 1 (Long and Short Papers), pages 4158, Association for Computational Linguistics.
Zihao Zhao, Eric Wallace, Shi Feng, Dan Klein, and Sameer Singh. 2021. Calibrate before use: Improv- ing few-shot performance language models. Proceedings of the 38th International Conference on

---

Machine Learning, ICML 2021, 18-24 July 2021, Vir- tual Event, volume 139 of Proceedings of Machine Learning Research, pages 12697-12706. PMLR.

---

Appendix
written before, vision features cannot be found in the images. or the QA is not about vision features, Additionally, the QA should conform to our com- monsense life, be strongly related to the category, and be about a specific feature of the category.
A Annotation Details
In this section, we will provide comprehensive overview of our annotation process, including the guidelines we follow, the user interface we use, the hierarchical supervision process we employ to ensure data quality, and our payment policy.
In summary, the annotation guidelines of IMA- CENETVC involve observing images of category, finding a vision feature, and creating a QA about the feature that conforms our commonsense of life and follows the rules outlined above. These guidelines ensure the accuracy and consistency of the annotations, making the dataset suitable for use in various applications
A.1 Annotation Guidelines
The annotation of IMAGENETVC involves observ- ing 20-50 images of a given category, finding a vision feature of the category, checking if it con- forms to most of the images and our commonsense of life, and then writing simple question-answer (QA) about this vision feature. The QA should contain one question and one correct answer.
A.2 Annotation
Figure shows the annotation user interface used in our human annotation process for model knowl- edge assessment. The interface consists of three main parts: the task instruction, the annotation pipeline, and the most common cases we reject, The task instruction provides clear guidance for the annotators how to write effective prompts to assess the model knowledge. The annotation pipeline displays the generated text by the model and allows the annotators refine their prompts until the generated text matches the expected target answer. The rejected cases section provides exam- ples of prompts that do not meet the criteria and serves as reference for the annotators to avoid such mistakes. The user interface design is intu- itive and user-friendly which greatly improves the efficiency and accuracy of the human annotation process.
The vision features can be object-based (such as the color, shape, material, and spotted/striped patterns of the whole object) region-based (such the color, shape, and material of certain part of the object). They are features that can be seen through the images.
The annotation pipeline involves looking at the 20-50 images given and finding a common vision feature of the category. For example, "The shape the dorsal fin of the tiger shark is triangle". The annotators check if this feature conforms to their commonsense of life and if it is written. Then, one QA is created, such "What is the shape of the dorsal fin of the tiger shark? Triangle".
The following rules must be followed during the annotation process:
The question should contain the name of the category. Otherwise, the submission will not be passed.
A.3 Hierarchical Supervision
To ensure high-quality annotation, we have imple- mented a hierarchical supervision process. During the annotation phase, examiners cross-check the an- notation results, and annotators who excessively rejected receive a warning. Those who exceed the warning limit are removed. Additionally, during the examination phase, a random sample check of the examination results is performed by five au- thors, and examiners with low-quality checks also receive a warning. The hicrarchical supervision process guarantees the high-quality execution of the entire annotation process.
If the annotator cannot think of a question that can be written or the images cannot be displayed, the annotator can skip this category.
The first letter of the question needs to be capitalized.
The end of the question needs to be a question mark.
Please do not write lots of Yes/No questions. These questions are more likely to be rejected. We encourage the annotators to write more diverse answers.
In the annotation examples, we describe how the correct QA is created. Annotators can write their own QA according to this pipeline. The rejected examples include cases where the QA has been
A.4 Payment Policy
We compensated the crowd workers with varying rates based on the workload and quality of their

---

Subset
Answer Candidates
Color
brown (tan). black, white, yellow (gold, golden). green, gray (grey), red, orange, blue, silver, pink
round (circle), rectangle (rectangular), triangle (triangular), square, oval, curved, cylinder, straight cone. curly. heart, star
Shape
metal (steel, iron), wood (wooden), plastic, cotton (yar, wool), glass, fabric (nylon, silk, rope) stone (rock), rubber, ceramic (porcelain), cloth (denim), leather, flour (dough, bread), paper. clay, wax concrete
Material
Component
yes, no
Others (yes/no)
yes, no
Others (numhers)
(two), 4 (four), 6 (six), 1 (onc), 8 (eight), 3 (three), (five)
long, small, short, large, forest (jungle, woods), water, ocean, big, tree (branch), ground, tall, wild outside, thin, head, thick, circle, brown, soft, land, nock, rough, chest, smooth, fur, hard, top, plants black, metal, books, vertical, lake (pond), grass, road, sky, front, kitchen, feathers, stripes, baby, hair feet, mouth, female, table
Others (others)
Table 5: The answer set of all subsets IMAGENETVC. Inside the parentheses attributes grouped into the same answer candidate.
work. In Phase workers were tasked with marizing shared visual characteristics from 50 im- ages and creating QA pairs. If their work was accepted in Phase 2, they would receive $0.50 for each sample. However, their work was re- jected, they would only carn $0.10 per sample. In Phase 2, annotators were paid $0.30 per sample for cross-checking tasks. On average, the annotators received approximately double the local minimum wage per hour.
Length Compartson
Others
21.1%
30.4%
20.4%
Relative Size
11.9%
16.2%
Counting
Living Environment
Figure 10: Detailed composition Others subset.
B Details of IMAGENETVC
B.1 Details of the Others subset
types of "yes/no", numbers, and other answers, spectively.
Annotated QA pairs that do not belong to the four specified sub-tasks (e.g., color, shape, material, and component) will be categorized into the Oth- ers subset. Therefore. The Others subset contains more diverse range of QA samples, which is more challenging. Figure 10 illustrates the de- tailed composition of QA types in the Others subset, which covers various topies such as length compar- ison (21%), relative size (20%), living cnvironment (16%), counting (12%), etc.
B.3 More Qualitative Examples
We present additional qualitative examples in Ta- ble 6, which compare the predictions made by var- ious models. The comparisons between OPT-7B and BLIP-2 demonstrate the effectiveness of in- corporating visual information in enhancing the visual commonsense capabilities of LLMs. How- ever, these leading models, including ChatGPT. also encounter difficulties in certain challenging cases, such as determining the color of a flamingo's beak tip. Besides, these examples highlight Chat- GPT's tendency to prioritize selecting the most commonly associated property of an object the answer, rather than considering the properties of the specific region in question. We hypothesize that this behavior may be attributed to the higher frequency of these common attributes co-occurring with the object in the pre- pre-training text corpus.
B.2 Answer Set
Considering that the results of open-ended genera- tion are uncontrollable, we evaluate all models with constrained decoding in main experiments. Ta- ble 5 shows the list of all possible answers in IMA- GENET VC. Besides, we noticed that LLMs tend predict "yes/no" or numerical answers when evalu- ated on the others subset. Thus, we split the others subset into three small test sets, containing answer

---

Related Images
Question Answer
What color is goldfinch's tail?
ОPT-7B: red x, BLIP-2: black v. ChatGPT: yellow X.
Related Images
Question Answer
What is the color of the flamingo's beak tip?
OPT-7B: green BLIP-2: red ChatGPT: pink (Ground Truth: black).
Related Images
Question
What the shape of the eyes of the jack-o" llantern?
Answer
OPT-7B: cylinder X, BLIP-2: square x, ChatGPT: triangle
Related Images
Question
What's the material of china cabinet? ОPT-7B: cotton X, BLIP-2: wood v. ChatGPT: ceramic X.
Answer
Related Images
Question
Does the koala have tail?
Answer
OPT-7B: yes X. BLIP-2: no / ChatGPT: yes
Table More qualitative examples comparing different models on IMAGENETVC. Here we demonstrate compar- isons with OPT-7B, BLIP-2, and ChatGPT.
Prompts
C.2 Details of Human Assessment
[QUESTION] (ANSWER).
[QUESTION] Answer: [ANSWER).
In the human assessment process of ImageNetVC, annotators were presented with pairs of mutually exclusive random 32 instances from both Ima- geNetVC and other datasets (e.g., ViComTe) each time. In the whole process of comparison eval- uation, conducted multiple rounds of human assessment based the dataset containing fewer test samples. Specifically, there are 556 compari- son pairs used for evaluation between ImageNetVC and ViComTe, and 510 pairs used for evaluation be- tween ImageNetVC and ChatGPT generated data.
[QUESTION] The answer is [ANSWER].
Question: [QUESTION] Answer: [ANSWER].
Question: [QUESTION] The answer (ANSWER].
Table 7: The prompts we utilize for LLM and VaLM evaluation on IMAGENETVC.
Experimental Details
C.1 Multiple Prompts
Annotators were instructed to evaluate these in- stances based on the overall quality of the data, choosing the better side when considering three fac- tors respectively: diversity, difficulty, and factuality. The final results were computed and demonstrated as percentages in Figure 4. For instance, the Diver- sity score depicted in Figure 4(a) signifies that in 86% of cases, the evaluators found ImageNetVC samples to either outperform or match those from
Table 7 shows all the intuitive prompt templates utilized to evaluate the LLMs and VaLMs in our main experiments. We do not tune the prompt for each subset in IMAGENETVC.
In our data quality experiments, we utilize the original prompts provided by Zhang et al. (2022a) for ViComTe evaluation while adopting the prompts in Table 7 for VMV.

---

COLOR
SHAPE
MATER.
COMPO.
OTHERS
Models
PParam
AVG
0-shot 5-shot
O-shot
5-shot
O-shot
5-shot
O-shot
5-shot
O-shot
5-shot
Random"
7.7
7.7
9.0
9.0
6.1
6.1
49.8
49.8
24.3
24.3
19.4
OPT-125m
125M
16.3
7.6
18.7
12,4
25.2
24.8
51.0
50.7
38.4
39.4
28.5
OPT-350m
350M
19.9
23.1
18.4
25.8
27.7
26.0
50.4
51.4
38.2
42.3
32.3
OPT-1.3B
1.3B
26.5
27.1
32.7
42.6
37.3
41.3
50.3
51.7
44.2
53.0
40.7
OPT
OPT-2.7B
2.7B
25.8
33.4
39.9
49.7
40.2
58.7
51.3
54.5
48.1
55.6
45.8
OPT-6.7B
6.7B
31.1
37.1
38.6
51.2
46.1
61.4
53.4
61.5
51.9
52.9
48.5
OPT-30B
30B
31.7
41.1
40.5
59.2
49.8
70.4
59.2
66.3
54.4
59.7
53.2
OPT-66B
66B
34.1
44.0
40.6
57.1
47.3
68.4
56.1
64.3
55.5
60.8
52.8
GPT-Neo-1.3B
1.3B
23.6
24.3
34.1
47.9
35.9
40.2
51.8
51.4
45.0
53.0
GPT
GPTT-No--2.78
2,7B
24.1
31.1
25.1
46.5
39.3
47.1
52.0
51.0
46.6
53.0
41.6
GPT-J-6B
6B
27.9
39.4
39.8
56.3
49.2
63.8
52.4
59.0
51.9
58.0
49.8
Pythia- 160m
160M
18.6
23.0
24.7
34.8
24.3
32.0
51.0
50.1
39.8
42.2
34.1
Pythia-410m
410M
11.2
25.8
20.9
39.1
36.8
44.2
50.7
51.2
43.8
47.4
37.1
Pythia-1.4B
1.4B
18.8
29.8
36.5
41.5
39.0
55.6
50.9
50.5
47.3
$2.4
42.2
Pythia
Pythia-2.8B
2,8B
18.6
34.0
31.6
53.5
43.2
46.1
$2.6
56.1
49.6
55.7
44.1
Pythia-6.9B
24.7
38.4
30.7
52.6
42.2
60.8
54.2
55.7
51.9
46.8
Pythia- 12B
12B
12B
27.6
43.3
34.6
47.7
46.3
63.8
56.7
63.4
54.2
62.4
50.0
Falcon-7B
7B
36.9
50.3
35.8
60.4
55.8
69.8
59.3
75.5
54.9
58.4
55.7
Falcon
Falcon-4OB
40B
46.4
56.2
42.2
62.5
57.5
71.9
69.0
79.6
59.9
68.9
61.7
LLaMA-7B
7B
41.0
52.8
34,9
54.4
50.9
69.9
64.3
72.4
57.0
58.9
55.7
LLaMA-13B
13B
41.9
55.7
40.8
65.9
52.0
72.1
59.0
79.2
58.5
64.2
58.9
LLaMA
LLaMA-30B
30B
46.6
60.5
41.5
52.7
58.2
72.0
68.5
82.8
59.4
70.3
61.3
L.AMA-65B
65B
50.8
64.0
46.7
70.4
57.5
71.8
71.3
84.0
61.1
73.0
65.1
Table Evaluation results large language models (LLMs) in IMAGENETVC. We report the mean Top-1 accuracy
(%) over 5 different prompts. ": We report random results over 5 different runs for comparison. We show the best results in boldface.
ViComTe concerning diversity.
#Extra Params
Visual Encoder
#Pretrained Images
Models
C.3 Model Implementation
Z-LaVI MAGMA
150M 400M
ViT-B/32
All the LLMs are implemented based the Hug- gingface "0 For VaLMs, utilize the official release of Z-LAVI"I MAGMA and BLIP-2 for evaluation. The CLIP model is adapted from the OpenAT's public 14 We utilize the vision- language pretrained checkpoints of VaLMs instead of tasc-fpe finetuned models to evaluate their zero- and few-shot capabilities. The details of VaLMs, including the visual encoder architecture and pretraining data, are included in Appendix C.4. All hyperparameters of VaLMs are the same that of their origin paper. We conduct our experiments on 6 NVIDIA A40 GPUs with 48GB memory.
R650x16
25M
BLIP-2
1.IB
VIT-G/14
129M
Table 9: Details of VaLMs evaluated IMAGENETVC.
Z.LaVI (Yang et al., 2022) introduces zero- shot framework that ensembles the solutions of LLMs and CLIP (Radford al., 2021) handle plain language tasks. The extra visual inputs of CLIP are obtained with web search and synthesis methods.
MAGMA (Eichenberg et al., 2022) adds ad- ditional adapter layers into frozen LLMs to augment them with visual capabilities. It also finetunes visual encoder to transform images into visual prompts as prefixes.
C.4 Model Details
We mainly conduct our evaluations with the follow- ing VaLMs in our experiments:
BLIP-2 (Li al., 2023a) introduces Query- ing Transformer to extract visual features from the frozen image encoder. then utilizes these features as visual prompts to augment frozen LLMs with visual information.
10
hhttps://hugginggface..
"https://gii htttt:///gggtuuu..oo////aY///as com//YueYANG1996/2-Laa
main/projects/blip2
hhtttss://githhe
wwtpthuu..cmmmooogeee/eeeeeeeeee com/openai/CLIP
We show the details VaLMs evaluated on IM-

---

Models
#Param.
COLOR
SHAPE
MATER.
COMPO.
OTHERS
Random
9.0
49.8
24.3
19.4
OPT-2.7B
2.7B
25.8
39.9
40.2
51.3
48.1
41.1
VaLM
(2.7B)
w/ Z-LaVI
2.9B
37.3 111.5
31.2 j8.7
46.6 76.4
51.5 10.2
46.4 L1.7
42.6 11.5
w/ BIJP-2
3.88
44.7
118.9
40.3 10.4
61.9 121.7
54.9 12.7
51.7 13.6
50.5 19.4
GPT-J-6B
6B
27.9
39.8
49.2
52.4
51.9
44.2
VaLM (6B)
N/ Z-LaVI
6.2B
38.2 f10.3
32.7 47.1
51.9 122.
52.5 10.1
48.6 L3.3
44.8 10.6
m/ MAGMA
6.48
42,9 115.0
40.3 10.5
60.9 111.7
54.6 12.2
53.9 12..0
50.5 16..3
OPT-6.7B
6.7B
31.1
38.6
46.1
53.4
51.9
44.2
(6.7B)
w/ Z-LaVI
6.9B
38.8 17.7
33.2 155.4
49.5
13.4
52.2 41.2
49.6 J2.3
44.7 10.5
W/ BIJP-2
7.8B
47.9 116.8
39.1 10.5
61.3 115.2
58.4
15.0
57.4
15.5
52.8 18.6
Table 10: Zero-shot probing results of visually-augmenteed language models (VaLMs) in IMAGENETVC. We report the mean accuracy (%) results in 5 different prompts. Numbers that are highlighted in orange represent the percentage of improvement and blue denotes percentage performance drop.
Prompt 1
E
Evaluation Details of Other Models
Answer List: [CANDIDATES]
This section provides evaluation details of VQA finetuned multimodal models and RLHF models.
[QUESTION] Please select the most possible answer from the above list. Please answer in one word.
Prompt 2
Multimodal Models For multimodal models such as BLIP, we adopt the evaluation settings used in VQAv2 (Goyal et al., 2017). Specifically, for each QA pair and its corresponding image, eval- uate the model using open-ended generation and obtain the output answer. Based on the experimen- tal settings outlined Section 1.2, we provide the top-10 10 ranked images for each QA pair and determine the final answer by majority prediction.
Answer List: [CANDIDATES]
[QUESTION] Please only print the answer selected the above list. answer in one word.
Prompt 3
[QUESTION] Please select the most possible answer from (CANDIDATES). Please answer one word.
Table II: The prompts utilize for ChatGPT evalua- tion on IMAGENETVC. "[CANDIDATESI" denotes the answer set the evaluated subset in IMAGENETVC, as shown in Table 5.
RLHF Models We evaluate ChatGPT with con- strained prompts and automatically compute the top-1 accuracy. The prompts utilized presented in Table 11.
AGENETVC in Table 9, including the extra model parameters (except the frozen LLM backbone), the architecture of the visual encoder, and the number of pretraining images. Since the VaLMs vary in implementation details (e.g., the visual encoder), we cannot make a direct (head-to-head) compar- ison between the VaLMs and leave it for future investigation.
D Details of Main Experimental Results
We provide detailed experimental numbers of our main results with LLMs and VaLMs in Table 8 and 10, respectively. For LLMs, show the zero- and few-shot evaluation results of OPT, GPT, Pythia, Falcon, and LLaMA across various model scales. For VaLMs,, we compare the performance zf BLIP-2, and MAGMA with their frozen LLM backbones.

---

Short Instruction:
Observe 20-50 images of given category, find vision feature of the given category, check If conforms most the images and our common sense life.
Then write simple QA about this vision feature.
The should contain
The vision feature can be:
Coarse-graines color, shape, material, spotted/stripd. of the whole object.
Fine- line-granne::o color, shape,
cortain onject.
Anywey, the vision features are features that we can see through the images. There are some features that can be referred to here.
The Annotation pipeline lo:
images below,
category. example,
Ibizan yellow
Chhck this feature conforms to common sense of life.
feature
is created. For "Whot the of Ibizan hound? yellow and white"!
The GA has been written before, Written OAs are shown below.
2. You can not find the vision features which the QA tells about in the images. Or the is not talking about vision features but just common sensess
Common Sense. For example, "Which country is rich coconuts? Thalland" It has nothing to do with visual features and can not be directly
Images.
Usage. "Can the be used pictures? yes", "Can the guiter be used music? yes", these questions telking about vision features.
images, esample, the planetanium very large? yes", the absolute size can not be directly inferred from images, so the "big/small" question not suitable.
which directly
cule?
For "What
the bottiecap? Not every bottienap
life
strongly elated
those apples all red these
hope the
attributes
of the eategory, not these images.
The QA is not about the specific feature of the category. For example, "Do the Chihuahus have legs? yes". We hope to get the specific feature of category, like
Chihuahua order diversity
wil not accept it again).
Click Annotation
Category: Samoyed
QAs written before:
Samoyed's bodly?
02 Does Samoyed have fur?
A2yes
Question
Anewer
Submit
Figure 11: Screenshot the IMAGENETVC annotation UI, featuring task instructions, the annotation pipeline, and the most common of a given category, and the task instruction guides the annotator to identify common vision feature and create a simple QA about The annotation pipeline includes checking the conformity to commonsense and avoiding pre-writted QAs. Annotations are focused visual features, commonsense con-visual attributes.
for rejecting prompts during the annotation process. The interface displays 20-50 images


Reviews:
[{"review_id": "92wmzZ3JGo", "reviewer": "Reviewer_xCkP", "summary": "", "questions": "Questions are mostly re-phrases of weaknesses:\nA. How do we agree that answers are valid for all classes? For example, the color of \"husky\" is \"black\" and \"white\", but some of them could have the color \"brown\". It would be good to get more qualitative examples of the collected data.\nB. What are some qualitative examples when LM and VaLM fail on? What is the missing gap between machine vs human performance of visual commonsense that should be worked on?\n ", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We deeply appreciate your effort in reviewing our paper and your acknowledgment of our paper’s contribution. We are very glad that you like our task design, in-depth analysis, and insightful findings.\n\nBelow we give point-to-point responses to each of your comments.\n\n***Q1: Using visual attributes such as color, material and shape seem to be far-fetched to be called as measuring \"visual commonsense\". They seem to be mostly \"factual knowledge\" that requires perception and recognition of objects, rather than requiring reasoning.***\n\nR1: We'd like to clarify that our work focuses on the atomic visual commonsense knowledge. Such fundamental commonsense serves as the basis for more intricate reasoning processes. Our work can be likened to a visual counterpart of ConceptNet [1], as we aim to establish a foundational understanding of the visual commonsense of PLMs and VaLMs.\n\nMore concretely, we follow ViComTe's definition [2] of \"visual commonsense\", i.e., knowledge about generic visual concepts such as \"knobs are usually round\", which allows for a straightforward assessment of the visual capabilities of both PLMs and VaLMs. We agree that more sophisticated benchmarks for visual commonsense reasoning (e.g., [3]) are also crucial for PLMs, but it is beyond the scope of our paper. We leave this insightful study for future investigation.\n\n***Q2: What are some qualitative examples of when LM and VaLM fail? What is the missing gap between machine vs human performance of visual commonsense that should be worked on?***\n\nR2: We appreciate your inquiry for more qualitative examples. In the final version of the manuscript, we will incorporate more qualitative examples and case studies that highlight instances where PLMs and VaLMs struggle, to better illustrate the limitations and gaps.\n\n***Q3: How do we agree that answers are valid for all classes? For example, the color of \"husky\" is \"black\" and \"white\", but some of them could have the color \"brown\". It would be good to get more qualitative examples of the collected data.***\n\nR3: We appreciate your inquiry for more qualitative examples of the collected data. We will illustrate more examples in the final manuscript. As demonstrated in Appendix A of our paper, during the annotation process, we asked the annotators to write visual features that match the majority of the given images and align with their commonsense of life. For example, \"The shape of the dorsal fin of the tiger shark is triangular\". To ensure the reliability and clarity of our dataset, questions with multiple possible (and confusing) answers (such as *the color of a cat's tail*) do not meet our requirements and will not be accepted during the examination.\n\nWe greatly appreciate the valuable feedback you provided, which significantly contributed to enhancing our work. We hope that the above clarifications will resolve your concerns. If you have any further questions, please feel free to let us know.\n\n**Reference**\n\n[1] Robyn et al. Conceptnet 5.5: An open multilingual graph of general knowledge. In AAAI 2017.\n\n[2] Zhang et al. Visual Commonsense in Pretrained Unimodal and Multimodal Models. In NAACL 2022.\n\n[3] Rowan et al. From Recognition to Cognition: Visual Commonsense Reasoning. In CVPR 2019.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Overall, the paper is well-written, running extensive studies in validating their dataset annotations and possible baselines related to their task with in-depth analysis. Here are its strengths:\n- demonstrates thorough annotation steps involving cross-check examination of collected QA pairs and their categorization.\n- runs another set of human evaluation to claim the value of their ImagenetVC dataset; e.g. Figure 4 shows that humans found the ImageNetVC dataset to be \"better\" in terms of diversity, difficulty, and factuality than previous similar dataset  (ViComTe) and machine generated (ChatGPT). This proves the need of human effort for the task of visual commonsense QAs, which is seldom explored in similar dataset papers. \n- explores performance of various, SoTA pre-trained text-only (e.g. LLAMA, ChatGPT) and vision-language models (BLIP-2), and runs analysis of which categories these models typically excel on. The results show that the task cannot be solved by pre-trained language models and that the image modality is still required to do achieve human-level performance. \n- shows insightful findings in their results such as: \n   - in-context learning (ICL) can lead to reduced variance across different prompts and improved calibration of language models (Figure 8), and improve the VaLM's visual commonsense performance.\n  - an ensemble of both generated and retrieved images as input could boost the performance of visual commonsense understanding. \n  - Scaling law of PLMs (Figure 7). \n- Shows that human performance is 93.5% v.s. the best model (in context learning with ChatGPT) is 75.8%, meaning that there is still a lot of room to improve on and this is a well-designed task. ", "reasons_to_reject": "- Using visual attributes such as color, material and shape seem to be far-fetched to be called as measuring \"visual commonsense\". They seem to be mostly \"factual knowledge\" that require perception and recognition of objects, rather than requiring reasoning.   \n- Lack of qualitative examples other than the Samoyed example. \n- Failure cases to understand what is the gap between current PLMs & VaLMs v.s. humans in visual commonsense.", "questions_to_authors": "Questions are mostly re-phrases of weaknesses:\nA. How do we agree that answers are valid for all classes? For example, the color of \"husky\" is \"black\" and \"white\", but some of them could have the color \"brown\". It would be good to get more qualitative examples of the collected data.\nB. What are some qualitative examples when LM and VaLM fail on? What is the missing gap between machine vs human performance of visual commonsense that should be worked on?\n ", "paper_topic_and_main_contributions": "The paper proposes a new dataset named ImageNetVC, designed for zero- and few-shot visual commonsense evaluation of Pretrained Language models (PLM) such as OPT, LLAMA, Falcon, and ChatGPT and Vision and Language models (VaLM) across the ImageNet categories. The evaluation is done as a QA set up with human annotation of QA pairs focused on color, shape, and material of classes.\nTo validate their dataset, the authors run another stage of cross-examine validation with human workers, and studies of human preference to show that their dataset is interesting and challenging than another visual commonsense dataset (ViComTe), and one generated by language models such as ChatGPT.  Extensive experiments are run to measure the factors that influence the performance of PLMs and VaLMs in their visual commonsense understanding, such as in-context-learning, image sources, and model size.\nHuman evaluation shows that their human annotation scheme provides much higher-quality than template and language model generated datasets, and the gap between machine-level (75.8%) and human-level (93.5%) of visual commonsense suggests that there is a good room of improvement to investigate further in this dataset.  Overall, the authors run thorough experiments to analyze possible factors of model performance and human evaluations to validate their dataset annotations. ", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "- Table and Radar plot showing LMs and VaLMs at the same time would helpful to see what are the categories that unimodal vs multimodal models excel on."}, "extra_text": {}}, {"review_id": "d5v2Rg1XzM", "reviewer": "Reviewer_eD4G", "summary": "", "questions": "See reasons to reject.", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank Reviewer eD4G for your review and are also glad you think our dataset is valuable and our findings are insightful. Below we would like to give detailed responses to each of your comments.\n\n&nbsp;\n\n***Q1: Comparison with previous works, especially ViComTe***\n\nR1: Yes, some previous works, such as ViComTe [1]*,* also worked on visual commonsense, which highlights the significance of this research problem. However, **one key difference between ImageNetVC and ViComTe is the way multimodal models are evaluated for visual commonsense**. ViComTe measures visual commonsense knowledge via textual frequency distributions over the corpora and evaluates multimodal models with **text-only inputs**, which has a misalignment with the pretraining data format (i.e. Text-Image pair) of multimodal models. Compared to ViComTe,  ImageNetVC proposes that **the visual commonsense knowledge in multimodal models demands relevant images as input to be suitably stimulated**, as we illustrate in **Table 3**. Equipped with diverse image sources (e.g., ImageNet, search images, synthetic images, and CLIP-ranked images), ImageNetVC provides **a fair VaLM evaluation with unified image sources** and facilitates future research on visual commonsense.\n\nIn addition, previous datasets have not yet addressed the following three critical challenges, including **the quality, difficulty, and reliability** of the datasets. We would like to further clarify the distinct contributions of ImageNetVC that set it apart from prior datasets, which are threefold.\n\n1. `Quality`: In contrast to ViComTe's automatically mined data from Visual Genome (VG) attributes, ImageNetVC directly leverages **human visual perception** to identify shared attributes in high-quality ImageNet images, **avoiding potential textual bias** and **providing data that has a closer alignment with human knowledge**. For example, we found low-quality mined samples in ViComTe, such as *(i, white)* and *(cat tail, black)*, which either contained non-visual objects or contained non-commonsense attributes. These two textual bias issues account for **7% and 34%** of all samples in ViComTe's color test set, respectively. Compared to ViComTe, the human annotation process (with quality verification) guarantees the high quality of our dataset.\n2. `Difficulty`: Another advantage of ImageNetVC over prior work is that it incorporates **region-based visual commonsense** **questions**, which poses **greater challenges** due to the scarcity of such long-tail knowledge in the natural language corpora and the requirement of **the localization capabilities of PLMs**. Previous work, such as ViComTe, only focuses on the attributes of the whole object, neglecting the rich and meaningful visual commonsense knowledge from a fine-grained perspective. For example, the ViComTe color test set has samples of the form *(pine tree, green)*, *(kitchen drawer, white)* and *(taxi cab, yellow)*, which do not explicitly ask questions about specific regions.\n3. `Reliability`: Furthermore, previous templated-based datasets, such as ViComTe, **exhibit severe prompt bias**, with substantial evaluation variance across different prompts (e.g., Accuracy varies from 2% to 61% as we illustrate in Figure 3). While ChatGPT can rewrite these template-based datasets, it may introduce **inconsistencies with the original data**. Figure 4(b) highlights potential risks associated with ChatGPT-generated results. We anticipate that large language models will enhance high-quality dataset development in the future. However, given the challenges of visual commonsense, manually constructed datasets are considered to be more reliable.\n\nAs mentioned by *[Reviewer xCkP](https://openreview.net/forum?id=u14dVx4rMW&noteId=92wmzZ3JGo)*, our validation results in Section 3.5 of the paper further confirm the strengths of ImageNetVC over previous datasets (such as ViComTe) in terms of **factuality, difficulty, and diversity,** *demonstrating the need of human effort for the task of visual commonsense* *QAs*. \n\n&nbsp;\n\n***Q2: It's not clear in the paper why category-level examination is necessary given that sample-level examination is gonna be conducted later. IMO, the sample-level examination is already enough.***\n\nR2: As agreed by *[Reviewer xCkP](https://openreview.net/forum?id=u14dVx4rMW&noteId=92wmzZ3JGo)*, we demonstrate a thorough annotation process involving cross-check examination of collected QA pairs and their categorization. Although sample-level examination can check each sample with high quality, it will consume more human resources and labeling costs. Therefore, we designed a **more efficient and cost-effective** stage - category-level examination, to filter out low-quality samples or those that do not meet our requirements.\n\n&nbsp;\n\n***Q3: In L222, I wonder what the average agreement rate is among all examples. Assume that 2/3 approval means 66.6% and 3/3 approval 100% for each sample.***\n\nR3: In the stage of sample-level examination, **the average agreement rate is 78%**. This means that 78% of all samples are accepted during Stage 2. Among all these accepted samples, we observed that **72%** of them received unanimous approval from the examiners. For samples with disagreement among the reviewers (i.e., 2/3 approval), we authors conducted additional screening to ensure the data quality. Besides, we have implemented a hierarchical supervision process, as illustrated in Appendix A.3, where we authors monitored the examination quality during the annotation process. We will add more details of our annotation process to the final version.\n\n&nbsp;\n\n***Q4: This paper only evaluates VaLM models. I wonder why not evaluate VLM models such as MiniGPT4 and LLAVA?***\n\nR4: ImageNetVC accommodates the evaluation of a wide range of models, including PLMs, VaLMs, and certainly VLMs and their instruction tuning backbones (e.g., Alpaca and Vicuna). However, we'd like to clarify that our main contribution is the constructed dataset and the research problem on the effectiveness of visual augmentation for PLMs. Therefore, we conduct our main experiments on PLMs & VaLMs and explore several co-founding factors that may affect the visual commonsense capability of models, such as model scale, in-context learning, and image sources. We have evaluated *LLaVA-13B* on ImageNetVC in the setting of open-ended generation. The results are illustrated below. We will also add the evaluation of *MiniGPT4 to the final version.*\n\n| Models     | Color | Shape | Material | Component | Others | Average |\n| ---------- | ----- | ----- | -------- | --------- | ------ | ------- |\n| Vicuna-13B | 57.3  | 64.2  | 71.9     | 77.8      | 67.3   | 67.7    |\n| LLaVA-13B  | 66.2  | 65.8  | 74.2     | 82.5      | 72.8   | 72.3    |\n\nThe experimental results show that LLaVA achieves consistent improvements over its text-only Vicuna backbone, especially on the \"Color\" sub-task, indicating that visual instruction tuning could further boost the visual commonsense capability of models. All these additional experimental results will be included in our final manuscript.\n\n&nbsp;\n\nWe sincerely hope that the above clarifications will resolve your concerns. Feel free to leave a comment if you have questions for further discussion.\n\n**Reference**\n\n[1] Zhang et al. Visual Commonsense in Pretrained Unimodal and Multimodal Models. In NAACL 2022.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. This paper introduced a new dataset about visual commonsense in 1000 categories of ImageNet. The data can be used to probe LLM and VaLM's understanding ability or knowledge of visual commonsense in a wide range of classes.\n2. The paper benchmarked several prevailing and strong LLMs and VaLMs and present several insightful observations.", "reasons_to_reject": "1. My main concern about the proposed dataset is that the type of knowledge it possesses is not very new from previous works, especially ViComTe. I went through examples of this paper's data and ViComTe's and didn't feel much difference. I.e., they all focus on different commonsense aspects of different objects, such as shape, material, and color, etc.  As for the differences listed in Tab1: (1). `Support VaLM`: I think other datasets can also be evaluated with VaLM. (2). `Region-based Question`: I think ViComTe also has region-based question? For example, in Fig1 of ViComTe, the shape or material of penguin all require locating at specific regions? (3). `Natural Language`: ViComTe uses a set of templates to generate data, which is also a kind of natural language. If the author is talking about whether the sentence is fluent and natural, I don't think it's a critical point because people can just rewrite template-based datasets using ChatGPT to make it natural and fluent.\n2. It's not clear in the paper why category-level examination is necessary given that sample-level examination is gonna be conducted later. IMO, the sample-level examination is already enough. \n3. In L222, I wonder what the average agreement rate is among all examples. Assume that 2/3 approval means 66.6% and 3/3 approval 100% for each sample.\n4. This paper only evaluates VaLM models. I wonder why not evaluate VLM models such as MiniGPT4 and LLAVA?", "questions_to_authors": "See reasons to reject.", "paper_topic_and_main_contributions": "This paper introduced a new dataset covering different types of visual commonsense in 1000 classes from ImageNet. Multiple popular LLMs and VaLMs are benchmarked on this dataset. The authors show comprehensive experimental comparison and analysis, as well as several interesting observations.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "N/A", "Typos_Grammar_Style_and_Presentation_Improvements": "N/A", "Justification_for_Ethical_Concerns": "N/A"}, "extra_text": {}}, {"review_id": "y5P6dED6AS", "reviewer": "Reviewer_MrcQ", "summary": "", "questions": "NA", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank Reviewer MrcQ for the positive feedback and we are grateful for the time you spent on our submission. We are also glad for the acknowledgment that our high-quality multimodal dataset is a good contribution to the research community, and our benchmarking results and paper writing are clear.\n\nOur response to your further comments is as follows:\n\n***Q1: The size of the dataset(4K) seems relatively low.***\n\nR1: Thanks for your advice! We are committed to further expanding the dataset's size in our ongoing and future research endeavors. For instance, the dataset could be further expanded by prompting large language models with in-context demonstrations and then filtering with the image-text matching model. By doing so, we aim to enhance the dataset's coverage and its ability to serve as a comprehensive benchmark for zero- and few-shot visual commonsense evaluation. We are glad to explore this in the future and will highlight this in our revision. \n\n***Q2: It might also be better to have the model performance/questions on more dimensions than simply color, material, shape, component and pattern (e.g open domain VQA style questions.)***\n\nR2: Thanks for your suggestion! The \"Others\" sub-task of ImageNetVC contains more diverse QA samples covering various topics such as length comparison (21%), relative size (20%), living environment (16%), counting (12%), etc. Visual commonsense knowledge assessment in open domain VQA style questions is a great point, and we are interested in constructing more challenging and large-scale datasets on visual commonsense.\n\nThank you very much for the constructive comments, which really helped us further improve our work. We hope our answers have addressed your concerns. If you have any further questions, please feel free to let us know.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The authors present a manually annotated dataset that they claim to have better alignment with QA and image pairs. They benchmark it with various popular Pretrained Language Models(PLMs) and visually augmented Language Models (VaLMs). High quality multimodal datasets and benchmarks on LLMs will be a good contribution to the research community. The authors provide clear benchmarking results and a commentary on how various LLMs behave (e.g., emergent properties) on their dataset. The paper is also well written and easy to follow.", "reasons_to_reject": "The size of the dataset(4K) seems relatively low. It might also be better to have the model performance/questions on more dimensions than simply color, material, shape, component and pattern (e.g open domain VQA style questions.)", "questions_to_authors": "NA", "paper_topic_and_main_contributions": "This paper introduces a dataset with s 4,076 high quality QA pairs, encompassing 1,000 ImageNet categories across diverse domains such as color,shape, material, component, etc. For a given QA pair the authors provide multiple relevant images in some cases.The authors use manual human annotations so the data is claimed to be of higher quality. The authors test their dataset on popular uni(PLMs) and multimodal (VaLMs) models on visual commonsense reasoning tasks (visual grounding). The authors benchmark popular LLMs with various configurations (model scale, in-contect learning, image sources) on the visual commonsense reasoning task using their dataset. The authors also detail the data annotation process.", "excitement": 4, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "NA"}, "extra_text": {}}]

Author rebuttal:

Meta-review: Based on the reviews and my perusal of the manuscript, the work clearly states its goal---manually curating a benchmark dataset for visual commonsense QA based on ImageNet. The authors have also prepared quite a thorough comparison with existing ViComTe dataset in terms of *diversity*, *difficulty*, and *factuality* (correctness). They also evaluate LLMs and VaLMs (Vision-augmented LMs) on the curated ImageNetVC dataset under zero- and few-shot settings. The authors did not perform evaluation on instruction-tuned VLMs, as pointed out by  eD4G. They subsequently provided the results in the rebuttal. Overall, ImageNetVC appears more difficult, factual, and diverse. They also provide many interesting observations and analyses w.r.t. various aspects of the image, the effect of in-context learning, emergence of visual commonsense with model scale, model calibration for ICL, lack of prompt bias of ImageNetVC samples over ViComTe, etc. 

# Pros
1. Overall, the authors seem to have tried their best to leave few questions unanswered in the manuscript.
2. The overall goal and the aim and conclusions of the experiments seem mostly clear to the reviewers, especially after the rebuttal.
3. This dataset could be significant in evaluating fundamental visual knowledge of LLM/VLMs, if not for more complex visual commonsense reasoning. According to the presented empirical human and automatic evaluation, ImageNetVC seems to be more challenging, yet factual and diverse dataset.

# Cons (The authors may want to consider some of these points in the future iterations)
1. The end product, the methods, and the evaluation are not particularly original. There is already ViComTe dataset with similar purpose. The zero- and few-shot evaluations of LLM and VLMs across various aspects is quite standard in NLP at this point. As pointed out by eD4G, the current manuscript claim this as novelty.
2. It is unclear how A/B human evaluation was performed in fig. 4 (by category?). How do you get the notion of parallel instances across the datasets? Also, diversity is a global aspect of the dataset, unlike factuality and difficulty that can be evaluated per QA pair. More details are required here.
3. The authors claim that PLMs may emerge to possess zero- and few-shot visual commonsense from 1.3B size onwards. However, such claim is made based on a sample size of only two. The authors also do not mention why it could be. Do they share any part of the pre-training dataset? That could lead to them having similar performance curves for visual commonsense. As far as I know, both at least share PILE as the pre-training dataset. More analysis is required.
4. More qualitative examples should be added as mentioned by reviewer xCkP.
5. The rationale and difference between the category- and instance-level filtering is not quite clear, even after rebuttal. Are you not looking at the all instances during category-level cross-checking? If so, why the instance-level cross-check once again? Can you elaborate on the distribution bias? Couldn't you have just done sample-level cross-check, followed by more annotation for categories with significantly fewer samples?
Decision: Accept-Findings