VIPHY: Probing “Visible” Physical Commonsense Knowledge
Authors: Shikhar Singh
Ehsan Qasemi
Muhao Chen
Venue: EMNLP 2023 Findings (2023)
Keywords: Commonsense
Evaluation
Vision-Language Model

Abstract: Vision-language models (VLMs) have shown remarkable performance on visual reasoning tasks (e.g. attributes, location). While such tasks measure the requisite knowledge to ground and reason over a given visual instance, they do not, however, measure the ability of VLMs to retain and generalize such knowledge. In this work, we evaluate VLMs' ability to acquire "visible" physical knowledge -- the information that is easily accessible from images of static scenes, particularly along the dimensions of object color, size, and space. We build an automatic pipeline to derive a comprehensive knowledge resource for calibrating and probing these models. Our results indicate a severe gap between model and human performance across all three dimensions. Furthermore, we demonstrate that a caption pretrained LM significantly outperforms VLMs on both size and spatial tasks -- highlighting that despite sufficient access to ground language with visual modality, they struggle to retain such knowledge.

VIPHY: Probing "Visible" Physical Commonsense Knowledge
Shikhar Singh Ehsan Qasemi Muhao Chen
University of Southern California
ssingh43@usc. edu
Abstract
Aision-lngguage models (VLMs) have shown remarkable performance on visual reasoning tasks (e.g. attributes, location). While such tasks measure the requisite knowledge to ground reason over given visual instance, they do not, however, measure the ability of VLMs to retain and generalize such knowledge. In this work, evaluate VLMs' ability to ac- quire "visible" physical knowledge- the infor- mation that easily accessible from images of static scenes, particularly along the dimensions of object color, size, and space. We build an automatic pipeline to derive a comprehensive knowledge resource for calibrating and probing these models. Our results indicate a severe gap between model and human performance across all three dimensions. Furthermore, we demon- strate that an LM tuned on the captions signifi- cantly outperforms VLMs on both size and spa- tial tasks highlighting that despite sufficient access to ground language with visual modal- ity, they struggle retain such knowledge. The dataset and code are available at ht https: //gi thub. com/ luka tuka-group/ /Vi Wiy...
Color: keyboard is white, black
Size: monitor larger than keyboard
VLM
Space: monitor is located above table
Figure I: We propose VIPHY for probing the ability to generalize visually accessible knowledge particularly along the dimensions of color, size, and space.
2010) have acknowledged the interdependency be- tween them, particularly the consolidation of se- mantic knowledge from episodic memories aids humans to acquire commonsense from experience. Pretrained language models (Devlin et al., 2019; Raffel ct al., 2020) have demonstrated the capacity to reason (Wang et al., 2019) and retain knowl- edge (Petroni et al., 2019; Da et al., 2021). Like- wise, vision- models (Lu et al., 2019; Rad- ford et al., 2021) driven by the availability of large- scale paired image-text datasets have shown strong performance on visual reasoning tasks (Antol al., 2015; Chen et al., 2015). While such tasks empha- size the model's ability to draw inferences from a specific visual instance primarily to ground enti- ties and reason about their attributes and relations, they do not, however, explicitly measure the con- solidation of such knowledge. In this work, we evaluate the model's ability to generalize aspects of grounding and reasoning tasks, regarded com- monsense knowledge.
1 Introduction
The ability to reason and acquire knowledge from experience, while being intuitive for humans, has been a long-standing challenge for AI agents (Mc- Carthy et al., 1960). Examples such as the color of grass, the relative position of monitor and ta- ble, are formally regarded as commonsense knowl- edge (Chi, 2005). The retention of such knowledge humans is achievable due to the presence of long-term memory, broadly classified into episodic and semantic memory (Tulving, 1972: Camina and Güell, 2017). While the former stores the infor- mation pertaining to personal events, the latter is geared towards general, decontextualizzed knowl- edge. Prior studies (Greenberg and Verfaellie,
Prior works have been largely directed towards probing language models pertaining to object prop- erties such as weight, size, speed, and affor-
'For instance, the memory of one's birthday cake episodic, whereas knowing that most birthdays include cake is part of semantic memory.
Counting bike wheels for different instances is an exam- ple of reasoning while generalizing that bikes have two wheels is consolidating across instances.

---

dance (Forbes and Choi, 2017: Forbes et al., 2019). Drawing upon the notion of world scopes (Bisk et al.. 2020a), we find that such datasets, albeit comprehensive across aspects of physical knowl- edge, are ideally suited for embodied agents capa- ble of interacting with the physical environment. This motivates us to develop resources that better align with the world scope of existing Al systems, primarily vision-language models.
WEW
Co [baseball, tonnis ball, footbal, baskotbal, snowball, ball
O: ball
Image-based Filtering
In this work, we introduce VIPHY, visible physical commonsense dataset designed probe aspects of physical knowledge that are casily acces- sible in images of static scenes. Therefore, can be argued that models pre-trained on such data have sufficient access to the "visible world". We build a large-scale dataset along three dimensions of ob- jects: (1) color, (2) size, and (3) space. In contrast prior works (Paik al., 2021; Zhang et al., 2022), we bypass crowdsourced annotations in favor of an automated pipeline to derive a resource span- ning 14k objects (30x) from raw images. This is achieved by extracting object subtypes informed by the visual context in images (c.g. kitchen sink). We leverage image data, along with existing vision- language and depth perception models to develop VIPHY.
C [baseball, tonnis ball, footbal)
naggion-assed Selection
football
Figure Subtype Selection Module: Given object o in image 7, assigns subtype 8, from candidate set Co.
knowledge. The contributions of this work can be summa- rized as follows: (1) We build a comprehensive dataset, covering multiple aspects of visually acces- sible knowledge ($3), which is developed through an automated pipeline to derive high-quality re- sources from images at scale ($2). (2) We conduct extensive benchmarking across several state-of-the the art language and vision-language models and find significant gaps in human performance (S4). (3) We demonstrate baseline tuned on the caption that significantly outperforms its vision-language counterparts - highlighting that despite access to images, VLMs struggle to consolidate such knowl- edge.
Beyond scale, we introduce resource for prob- ing spatial knowledge of common environments. Although one can reason along several types of spatial relations for a visual instance (e.g. a cat behind a laptop; Liu et al. (2022a)), we find that mapping them to commonsense knowledge is non- 3 We define spatial relations by selecting "ground" as the observer and specifying the relative elevation of objects under allocentric reference frame (Klatzky. 1998).
2 Pipeline
We probe state-of-the-art models on VIPHY, and find a significant gap across all three dimen- sions, compared to human performance. Previ- ous works (Paik et al., 2021; Liu et al., 2022b) have corroborated the improvements from language grounding towards acquiring visual knowledge our results, however, show more nuanced picture. While VLMs fare much better than LMS on recall- ing colors, the caption pretrained baseline (Zhang et al., 2022) significantly outperforms VLMs on both size and spatial inference tasks. This high- lights that despite access to visual modality, exist- ing VLMs strugglc to effectively consolidate such
We provide a conceptual overview of our pipeline for developing VIPHY, as illustrated in Fig. 3. Dur- ing the preprocessing stage, we build an internal database comprising object names and correspond- ing subtype candidates. Given image and object regions inpurt, substitute object names with their subtypes ($2.1), and compute the correspond- ing depth map. The processed data used to ex- tract color, size and spatial knowledge ($2.2).
2.1 Object Subtype
While object recognition datasets consider a wide range of objects, such tasks do not necessitate fine- grained categories (Zou et al., 2019). However,
'Due to challenges in specifying the reference frame of the ohserver, the canonical pose of the objects, and the situa- tional nature of a seenc.
4 Available from either manual annotations or object de- tection/segmmentation model (c.g. He et al. (2017)).

---

Source Data: Image,
[Object: Name Name--Box]
WF
Compute Depth
Select Subtype
Sublype Candidates
"bal" "Totball"
Processed Data:
Image, Depth, [Object / Subitype]
Spatial
Size
Color
PC 1313
plato?"
Cluster by Depth: Intra cluster relations
Cluster by Size: Inter cluster relations
Query VLM: cropped image
Figure 3: Pipeline Overview: The preprocessing step computes the image depth map. and re-annotates objects by selecting the best subtype from the set of candidates (KB). The color, size and spatial knowledge are then derived independently.
object subtypes inform attributes such color, and help contextualize objects in absence of visual sig- nals (e.g. office chair). Although subtypes are generally accessible from knowledge bases (KB), their coverage is often limited. We extend this def- inition to include objects defined by visual context indicating event, location, state, part, etc. (Ap- pendix Tab. 9). For subtype collection, we parse captions to build set of object names. We then employ suffix- based lexical matching derive sub- types for each object, and merge with hyponyms from knowledge base. The resulting data repre- sents mapping between the object name and its candidate subtypes.
candidates using visual features. It employs two- stage approach to filter candidates using image text, and select the best subtype with region-leve
features, as illustrated in Fig. 2. The visual and textual inputs are embedded using a dual stream vision-language model. Formally, given the visual feature of the image I, textual features of the ob- ject and subtype candidates Co, we extract the appropriate subtype as follows:
C=fole Cos sim(c, I) sim(o, 1)) Uo
Here, sim(.) is the cosine similarity. Intuitively, since the object name is independent of visual con- text, it serves as an anchor for excluding subtypes that do not align with the contextual cues. the next stage, we incorporate visual features of the object region Ou, to query from filtered candidate set C, and compute the best subtype 8,;
As goal is to derive object attributes and relations directly from images, we design a sub- type selection module to annotate the source im- age regions with the best subtype. This required since human annotators often abstract the object name to avoid redundancy when presented with vi- sual context (example in Appendix Fig. 12) con- gruent with the maxim of quantity (Grice, 1975). Likewise, existing object detectors are not suited for open-vocabulary and fine-grained classifica- tion (Minderer al., 2022).
So arg max sim(Ou; CEC
The preprocessed dataset comprises object-subtype mapping for every bounding box region in the im- age.
The module is designed to query from subtype
2.2 Knowledge Extraction
Given the image and depth map, along with object- subtype region annotations, we independently ex- tract color, size and spatial knowledge.
We report -60% object name overlap between Concept- Net KB (Speer et al., 2017) and our collection derived from dense captions in Visual Genome (Krishna et al., 2017).

---

Color Prior works (Paik al., 2021) have relied on human annotations acquire the color distri- bution of objects instead of inferring color from pixel values due to challenges such as lighting, shadow, segmentation, etc. However, we argue that large-scale availability of images can mitigate potential noise associated with automated extrac- tion. Given the ubiquity of color attribute in visual reasoning tasks (Antol al., 2015; Hudson and Manning, 2019), we find that VLMs pretrained on such datasets are reliable for inferring color from images. As object localization is decoupled from attribute recognition in the pipeline, the imput to the VLM is simply the cropped image region, queried with predefined textual prompt (detailed in $3.1).
Figure 4: Illustrates transitive spatial relation, computed across partitions (ordered by depth). The y-axis denotes elevation, while the z-axis indicates depth.
Size To derive size relations, we consider co- occurring objects in a scene. As objects in an image are expected to appear at varying depths, we ap- proximate perceived size by including scene depth. Given an image, depth map and object-region anno- tations as inputs, the objects are clustered by size- defined as the bounding box area scaled by mean depth of the region. The sorted partitions are then used to derive inter-cluster relations. The object pair relations are aggregated across images. The number of clusters are fixed for all instances.
3.1 Construction
Sources We leverage two datasets: (1) Vi- sual Genome (Krishna al., 2017), and (2) ADE20K (Zhou et al., 2017). The dense captions Visual Genome provide a broad coverage of object classes, making it a suitable resource for collecting subtype candidates. For extracting hyponyms from knowledge base, we acquire "is-a" relations from ConceptNet (Speer et al., 2017), and augment the subtype candidate set. We extract spatial relations from ADE20K, it provides images categorized by scene primarily indoor environments with high object density: (bedroom, bathroom, kitchen, living room, office/.
Spatial We define spatial knowledge as the rel- ative elevation between objects, for given scene type. To infer these relations directly from image, however, is challenging perspective projection of 3D world distorts the relative elevation due to variation in depth. We discount this distortion by partitioning the image by depth, and compute intra- cluster object relations, i.e. we discard the depth coordinate of objects that belong to the same clus- ter, and simply compare the relative elevation. The inter-cluster relations are derived transitively via overlapping partitions defined by objects with dual membership. as illustrated in Fig. 4. The spa- tial relations are aggregated across all images for a given scene type. We detail the specifics of map- ping object annotations to spatial relations Ap- pendix A.3.
Models To collect subtype candidates (as de- tailed in $2.1), we perform part-of-speech tag- ging to extract object names (noun) from cap- tion data, using LSTM-CRF (Akbik et al., 2018). Our subtype selection module is instantiated with UniCL (Yang al., 2022) designed for discrimi- native representations and broad semantic coverage of entities. To compute depth map from monoc- ular image, we use DPT (Ranftl et al., 2021). To infer object color from image region, we query OFA (Wang et al., 2022), using the prompt tem- plate: "what color is the <object>?", and map zero- shot predictions to basic color set (Berlin and Kay, 1991) detailed in Appendix A.4.
3 Dataset
3.2 Task Summary
This section details the specific data sources and models used develop VIPHY (53.1). We also report the dataset statistics and task format for each dimension ($3.2). Additional parameters related to dataset construction are provided Appendix A.I.
Table summarizes statistics for VIPHY, compris- ing the number of objects, classes and instances for each dimension. For multi-label tasks, we report the label cardinality, i.e. number of labels for a sam- ple. We also indicate the number of objects with

---

Subtype
objects
14k
70% test set.
objects with subtype
1.8k
subtype cardimality
7.73 (12.91)
Baselines We consider the following language (LM) and vision-language (VLM) models:
Color
objects (instances)
14k
classes
label cardinality
2,42 (1.37)
LMs: BERT (Devlin et al., 2019), RoBERTa (Liu al., 2019), DeBERTa (He et al., 2020) and
Size
objects
1.6k
instances
43k
relations
2
UnifiedQA (Khashabi et al., 2020).
Spatial
objects
300
instances
6.5k
VLMs: VisualBERT (Li al., 2019), ViLT (Kim al., 2021), CLIP (Radford et al., 2021) and FLAVA (Singh et al., 2022).
relations
3
label cardinality
1.28 (0.45)
5
Table Dataset statistics for VIPHY. The cardinality is reported with mean and standard deviation.
CapBERT In addition to the aforementioned baselines, we explore what degree does an LM pretrained only on image captions, encodes visual knowledge. Such a model effectively serves as a diagnostic baseline - ablating the grounding mech- anism in VLMs. We build CapBERT by pretraining BERT base? 7 on captions (4M) from COCO (Chen et al., 2015), CC3M (Sharma et al., 2018) and VG (Krishna et al., 2017) datasets. While contem- poraneous work by Zhang al. (2022) developed similar baseline, we find limited comparisons on color and no evaluation on their size dataset. Through extensive benchmarking, we are able to derive novel insights with CapBERT (54.1).
subtypes and subtype c Note that while we extract size relations and color attributes from same source images ($3.1), we ignore contextual subtypes for size relations, as they serve limited role towards informing object size (e.g. rain coat). However, as we only consider co-occurring ob- jects, we implicitly incorporate context for objects in comparison, i.e. help disambiguate word sense. We collect 7. smaller and 14.5k larger relations, and balance labels by including their complements. The label distributions of color dataset is provided in Appendix Fig. 10.
To evaluate under finetune setting, we train linear classifier on top of the model's output, while rest of the weights are frozen. We use Softmax Cross Entropy loss for single and multi- label setups, following Mahajan al. (2018). All probes are finetuned for 50 epochs, with batch size of 8, using Adam optimizer (Kingma and Ba, 2015) and a learning rate 10-4
Finetuning
The objective of VIPHY tasks is to measure the ability to generalize physical knowl- edge pertaining objects. To probe models with textual prompts, we map the raw distribution of labels (acquired by our pipeline) typical val- ues, as detailed in Appendix A.2. In the color task, objects can have multiple labels from the set of 11 basic colors as defined in Berlin and Kay (1991): (red, orange, yellow, brown, green, blue, purple, pink, white, gray, black). Like- wise, consider multiple labels for spatial re- lations from (below, above, similar level), condi- tioned on scene type as mention in $3.1. Lastly, size relations are mapped to a single label from { {smaller, larger).
Formulation
Prompts For probing LMs and VLMs, we pro- vide manually designed textual prompt as input to the model. The prompt templates for probing across color, size and spatial tasks, under zero-shot (7S) and finetune (FT) settings are given in Table 2. Besides models trained on the masked language objective the question-answeriin baseline (Uni- fiedQA) follows a common template for both ZS and FT settings.
4 Experiments
Metrics We introduce the following metrics for measuring multi-label task performance:
We evaluate several state-of-the-art models under zero-shot and finetune settings ($4.1), and conduct further analysis of model performance ($4.2). The datasets are partitioned into 20% train, 10% dev,
"Note that VLM baselines follow the base model size.
To fair comparisons, we build the base version. *CLIP being the exception, cannot evaluated under setting. Under FT, it uses EOS instead of CLS token.
Note: The suhtype cardinality is only reported for objects
with subtype.
"The prompt includes all classes as choices.

---

Task
Setting
Prompt
tasks, with VLMs (VisualBERT) performing worse than their LM counterparts underscoring the chal- lenges of manual prompting (Jiang et al., 2020). The best scoring baseline (UnifiedQA) falls at least 30% points below human scores.
Color
ZS
is of [KASK] color
FT
[CLS] color of
QA
What the color O? (a) (b)
Size
75
O₂ is [MASK] than O2 size
FT
[CLS] size On in comparison to O2 what is the size O₂ in comparison to
QA
Model
Color
Size
Spatial
O₂? (a) (b)
BERT laras
48,39
44.61
20.96
Spatial
ZS
in S, the O₂ is located [MASK] the O2
RoBERTaare
0,59
47.01
17.52
UnifiedQAuse
51.00
51.76
63.04
FT
[CLS] in S,, the O₁ is located in com- purison to O2
VisualBERT
9.06
24.91
9.57
QA
in S. where is O₂ is located in com- parison Oz? (b)..
Human
97.45
90.12
88.24
Table 3: Zero-shot results (R-Acc) across all tasks.
Table 2: Prompt templates across tasks and evaluation settings. Here, O, R and are placeholders for object, relation and scene type respectively.
When compared to zero-shot results, we report improved calibration under finetuned probing, as evident from results on color (Tab. 4), size (Tab. 6) and spatial tasks (Tab. 5). We find that VLMs score higher than LMs specifically their "caption-only" counterpart (CapBERT) on the color task. These results hint the role of color attribute in grounding entities. However, CapBERT outper- forms VLMs on both size and spatial tasks, indi- cating that despite access to visual representations, VLMs do not retain such relational knowledge as effectively. In particular, it highlights that position encoded visual inputs in VLMs remain insufficient towards consolidating spatial knowledge. Lastly, CapBERT outperforming other LMs likely due to the domain similarity between the pretraining source and the evaluation tasks
Finetune
Relaxed Accuracy (R-Acc) The prediction (P;) is accurate if the most probable label (1,) belongs to the set of ground-truth labels (T;).
(4 T.J arg max; P.(())
RA
D
iED
True Confidence (Conf) The sum of predicted probabilities for labels in the ground-truth set.
C= IE, P.(
D
iCD
Here, D denotes samples in the evaluation set. In addition to the aforementioned metrics, we also report the macro-averaged Fl-score (F1).
4.2 Analysis
Human Performance To provide an upper bound on VIPHY tasks, we use CoDa (Paik al., 2021) for color - computed over 432 overlapping objects, 10 For size and spatial tasks, we evalu- ate 100 relations with three external annotators II (crowdsourced) and report the average scores.
Color: Cardinality We further analyze model performance with respect to label cardinality (i.e. number of ground-trutt colors for an object), by grouping objects accordingly. As shown in Fig. 5, we report results for three baselines, their average, along with human scores. While the performance is expected to increase with the cardinality we notice an inconsistency between model and human scores. In particular, while difference in overall confidence scores (as inferred from Table 4) for human and the model average is -18%, the relative differences between the two ranges from ~122% (x = 6) to ~40% (x 1), where x-axis denotes the label cardinality. While color influences object
4.1 Results
Zero-Shot We report zero-shot performance us- ing R-Acc metric, across all tasks in Table 3. For spatial task, we only consider two labels from {above, below). due to the limitation of single word masking selected baselines. We observe significant variations in model performance across
10 The label distributions VIPHY CoDa are provided Appendix- - Fig. 10 and Fig. "Tee intettannotator agreement measure (Fleiss' Kappa) for size and spatial are 0.85 and 0.78, respectively.
"Foor instance, a prepositional phrase can convey both abstract (on schedule) and physical (on table) relations, with cuptions predominantly containing the latter. BR-Acc Conf are 1, when cardinality is 11.

---

Model
R-Ace
Conf
F1
BEKT
CLPP
CapBERT
70.45
58.55
40.91
BERT hose
66.87
55.59
30.94
RoBERTasry
55.95
49.28
20.93
UnifeedQAunee
62.34
DeBERTaa.
72.74
59.59
36.33
VisualBERT
66.22
50.99
24.46
ViLT
64.83
53.92
30.27
FLAVA
76.33
62.84
38.74
CLIP
79.96
65.50
49.54
Human cobs
97.45
78.65
68.82
90
Table 4: Color results.
Figure 6: Category-wist performance color (R-Acc). The dashed lines indicate average scores.
Model
R-Acc
Conf
F1
CapBERT
69.93
62.09
60.78
BERT hose
67.25
59.91
61.34
RoBERTauge
54.88
58.40
58.88
jects that belong to a common entity. We manually select category names from the object ontology pro- vided in Open Images dataset (Kuznetsova et al., 2020). We assign each object calegory by com- puting the most similar category. We use Sentence- BERT (Reimers and Gurevych, 2019) and thus ar- tificially convert each category to a sentence by concatenating the category name with few objects serving as semantic cues (Appendix Tab. 8). We re- port category- category-wise performance (R-Acc) for BERT and CLIP, provided in Fig. 6. We observe sig- nificant differences in performance between the two baselines, across categories. For animal and nature categories, BERT performs poorly with re- spect to the mean score. This difference widens for food, hinting at the effects of reporting bias as their typical colors are less likely to be written in text. The strong performance of CLIP for this category, however, indicates that such under reporting can be mitigated by visual modality. In contrast, for cate- gories such as electronics and vehicle, we observe that LMs perform well likely because color often plays a role in describing objects such as gadgets and cars.
UnifiedQAuge
62.04
DeBERTa..
62.30
61.27
60.54
VisualBERT
63.08
58.40
58.88
ViLT
65.78
60.28
59.80
FLAVA
63.71
61.06
60.56
CLIP
65.10
63.56
62.26
Human
88.24
81.22
Table 5: Spatial results.
4
Tuman
Avg
FLAAA
Size: Transitivity As the size dataset 14 is com- posed of frequently co-occurring object pairs, intend to evaluate model's ability infer relative size for objects linked transitively across scenes. We build new evaluation set comprising transitive relations from the standard size dataset, ensuring no overlapping instances between the two. The results (Tab. 6) indicate that LMs (on average) im- prove by significant margin, compared to VLMs. While the improvements on the evaluation set can
Figure 5: Effect of label cardinality (x-axis) color prediction, as measured by R-Ace and Conf. The Avg curves (black) indicate averuge model performance.
perception in humans (Gegenfurtner and Rieger, 2000), these results show that VL.Ms do not ascribe similar degree of saliency to color, especially for uni-color objects (i.e. cardinality of one).
Color: Category-wise To conduct error analysis on color task, we define categories by grouping ob-
For reference, the #instances for size evaluation sets are as follows - standard: 30k, subtype: 23k, transitive: 20k.

---

Model
Standard
Subtype
Transitive
tial reasoning in images, spanning 65 relation types. In contrast, VIPHY measures the ability to recall spatial commonsense. Additionally, whereas Liu et al. (2022b) have probed spatial knowledge for human-object interaction (224 instances) under 15 action types (e.g. driving, cooking), we consider the spatial layout of objects across scene types over 6k instances, independent of events.
CapBERT
83.69
79.14
91.82
BERT hase
78.35
72.28
77.29
RoBERTaann
65.23
57.12
69.31
UnificdQAuuee
62,20
60.66
90.78
DeBERTaad
74.73
66.88
69.79
L.werrage
69.37
64.23
74.54
ViSuaBBERT
76,99
64.00
77.69
ViLT
78.54
57.32
86.18
FLAVA
82.67
69.54
81.78
Vision-Language Resources While image clas- sification (Deng al., 2009) can be construed one of the earliest attempts at bridging vision and language, recent years have witnessed a plethora of resources. Visual reasoning tasks have been directed towards object attributes (Antol et al., 2015), activities (Chen et al., 2015), as well as social (Zellers et al., 2019) and temporal common- sense (Fu et al., 2022). Recently, VLMs (Lu et al., 2019; Li et al., 2020; Radford et al., 2021) have demonstrated strong performance These works evaluate the requisite knowledge reason about specific instance, VIPHY contrast probes the knowledge retained in the absence of visual context, i.e. generalized from instances.
CLIP
75.43
66.56
72.48
VLM 79.15
64.35
79.53
Human 90.12
Table 6: Size results reported across different evaluation sets, measured by accuracy (random baseline: 50%).
be partially attributed to objects on the relatively extreme ends of size clusters being paired up, they are able generalize on transitive relations.
such tasks.
Size: Subtypes While qualifiers denoting visual context tend to inform the typicality of color at- tribute, their effect size is likely inconsequential. Therefore, models should retain their performance with reference the standard set. We test this hy- pothesis by creating an evaluation set comprising contextual subtypes for objects in the standard test set. While the addition of subtypes leads to perfor- mance drop across all models (Tab. 6), we observe that LMs are more robust in comparison to VLMs.
Knowledge in LMs Recent advancements in lan- guage models (Devlin et al., 2019; Raffel et al., 2020), pretrained on large corpora, has led to significant improvements across several reason- ing tasks (Wang et al., 2019). Prior works have also highlighted the capacity of these models to acquire several types of knowledge such fac- tual (Petroni al., 2019; Roberts al., 2020), instructional (Huang al., 2022) and common- sense (Da et al., 2021). In this work, we study to what degree do their lasgunel analogs (VLMs) driven by the availability of massive paired image-text datasets, retain information that is easily accessible in images.
5 Related Works
Recent years have wit- nessed a renewed interest commonsense via nat- ural language benchmarks (Talmor et al., 2019; Singh et al., 2021). Specific works have evalu- ated the language models on their ability to reason about physical commonsense (Bisk et al., 2020b; Qasemi al., 2022), and identified reporting bias as potential bottleneck (Forbes al., 2019; Paik et al., 2021). In this work, we direct our focus to- wards languaee models pretrained on large paired image-text datasets, and evaluate them on vi- sually accessible commonsense knowledge. While prior works have probed knowledge pertaining to color (Paik et al., 2021; Mullenbach al., 2019) and size (Talmor et al., 2020), their coverage of objects is severely limited in comparison to VIPHY (30x). Beyond size and color, Zhang et al. (2022) also incorporated object shape and material. Recently, Liu et al. (2022a) have evaluated spa-
Physical Commonsense
Conclusion
We present VIPHY, a large scale resource for prob- ing "visible" physical knowledge - information eas- ily accessible from images of static scenes, across dimensions of color, size and space. We design an automated pipeline to extract and consolidate such knowledge facts from images, and introducc new resource for evaluating spatial knowledge of common environments. Our benchmarking eval- uation highlights a huge gap between model and human performance across all three tasks. Fur- thermore, while prior works have reported VLMs

---

be more effective, our caption pretrained base- line (CapBERT) significantly outperforms VLMs on the ability to recall size and spatial knowledge, These results underscore that despite access to vi- sual modality, existing VLMs struggle to retain visual knowledge as effectively.
since we automatically estimate object size from the image the estimations are limited by the qual- ity of bounding boxes and depth predictions from models.
References
Acknowledgement
Alan Akbik, Duncan Blythe, and Roland Vollgraf. 2018. Contextual string embeddings for sequence label- ing. In Proceedings of the 27th International Con- ference on Computational Linguisties, pages 1638- 1649, Santa Fe, New Mexico, USA. Association for Computational Linguistics.
We wish to acknowledge Bowen Zhang for his thoughtful comments on our paper. We also ap- preciate the constructive feedback from our anony- mous reviewers, which has played a vital role in improving and clarifying this paper. This work is supported by the DARPA MCS program under Con- tract No. N0.N66001 1924033 with the United States Office Of Naval Research.
Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Mar- garet Mitchell, Dhruv Batra, C. Lawrence Zitmick, and Devi Parikh. 2015. VQA: visual question an- swering. In 2015 IEEE International Conference on Computer Vision, ICCV 2015, Santiago, Chile, December 7-13, 2015, pages 2425-2433. IEEE Com- puter Society.
Ethical Implications
We build VIPHY from existing images from crowd- verified visual datasets which have been identi- fied to lack geographical diversity, often limited to scenes from Europe and North America (Shankar al., 2017). Furthermore, such datasets are sub- jected to several kinds of biases at different stages of collection and annotation such as selection bias, framing bias and observer bias (Fabbrizzi et al., 2022). Therefore, its likely that such biases will be reflected in our dataset as well. As we also re- port benchmarking results on VIPHY, the model performance may not be reflected as accurately on knowledge pertaining to different geographical and cultural backgrounds, studied by Yin al. (2021). Lastly, our proposed resource is limited to English, and thus excludes any considerations for multilingual models (Yin et al., 2022).
Brent Berlin and Paul Kay. 1991. Basic color terms: Their universality and evolution. Univ of California Press.
Yonatan Bisk, Ari Holtzman. Jesse Thomason, Jacob Andreas, Yoshua Bengio, Joyce Chai. Mirella Lap- ata, Angeliki Lazaridou, Jonathan May. Aleksandr Nisnevich, Nicolas Pinto, and Joseph Turian. 2020a. Experience grounds language. Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 8718-58 1--855, Online. Association for Computational Linguistics.
Yonatan Bisk, Rowan Zellers, Ronan LeBras. Jianfeng Gao, and Yejin Choi. 2020b. PIQA: reasoning about physical commonsense in natural language. In The Thirty-Fourth AAAI Conference on Artificial Intelli- gence, AAAI 2020, The Thirty-Second Innovative Ap- plications of Artificial Intelligence Conference, IAAI 2020. The Tenth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2020, New York, NY. USA, February 7-12, 2020, pages 7432- 7439. AAAI Press.
Limitations
We extract spatial knowledge directly from images, we assume that the camera's 's image plane is some- what orthogonal to the ground (transverse plane), and do not account for the edge cases where the image plane will be parallel, i.e. top view. For collecting subtype candidates, we parse Visual Genome (Krishna et al., 2017) captions to acquire object names. To assign object subtypes, we rely pretrained vision-language model (UniCL;Yang al., 2022) which can be a likely source of noise in the pipeline. Furthermore, quantifying their per- formance on subtype selection task beyond the scope of our work, due to unavailability of ground- truth annotations. In contrast to previous works,
Eduardo Camina and Francisco Güell. 2017. The roanatomical, neurophysiological and psychological basis of memory: Current models and their origins. Frontiers in pharmacology, 8:438.
Xinlei Chen, Hao Fang, Tsung-Yi Lin, Ramakr- ishna Vedantam, Saurabh Gupta, Piotr Dollár, and C Lawrence Zitnick. 2015. Microsoft coco captions: Data collection and evaluation server. arXiv preprint NN0::0504.003..
Michelene Chi. 2005. Commonsense conceptions of emergent processes: Why some misconceptions are robust. The journal of the learning sciences.
14(2):161-199.
Jeff Da, Ronan Le Bras, Ximing Lu, Yejin Choi, and Antoine Bosselut. 2021. Analyzing commonsense

---

emergence in few-shot knowledge models. In 3rd Conference on Automated Knowledge Base Construc- tion.
Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. 2020. Deberta: Decoding-enhanced bert with disentangled attention. arXiv preprint arXiv:2006.0360
Jia Deng. Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Fei-Fei Li. 2009. Imagenet: A large-scale hier- archical image database. In 2009 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR 2009), 20-25 June 2009, Miami, Florida, USA, pages 248-255. IEEE Computer Soci- ety.
Wenlong Huang, Pieter Abbeel, Deepak Pathak, and Igor Mordatch. 2022. Language models as zero-shot planners: Extracting actionable knowledge for em- badied agents. arXiv preprint arXiv:2201.07200
Drew A. Hudson and Christopher D. Manning. 2019. GQA: A new dataset for real-world visual reason- ing and compositional question answering. In IEEE Conference Computer Vision and Pattern Recogni- tion, CVPR 2019, Long Beach, CA, USA, June 16-20, 2019, pages 6700-6709 100006709.00 Computer Vision Founda- tion / IEEE
Jacob Devlin, Ming-Wei Chang. Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language under- standing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, Volume I (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
George F Jenks. 1967. The data model concept in sta- tistical mapping. International yearbook of cartogra- phy, 7:186-190.
Zhengbao Jiang, Frank F. Xu, Jun Araki, and Grabam Neubig. 2020. How can we know what language models know? Transactions of the Association for Computationall Linguistics, 8:423-438.
Simone Fabbrizzi, Symeon Papadopoulos, Firini Ntoutsi, and Ioannis Kompatsiaris. 2022. A survey on bias in visual datasets. Computer Vision and Im- age Understanding, page 103552.
Daniel Khashabi, Sewon Min, Tushar Khot, Ashish Sabharwal, Oyvind Tafjord, Peter Clark, and Han- naneh Hajishirzi. 2020. UNIFIEDQA: Crossing for- mat boundaries with a single QA system. In Find- ings of the Association for Computational Linguisties: EMNLP 2020. pages 1896-19077 Online. Association for Computational Linguistics.
Maxwell Forbes and Yejin Choi. 2017. Verb physics: Relative physical knowledge of actions and objects. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguisties (Volume
1: Long Papers), pages 266-276, Vancouver, Canada. Association for Computational Linguistics.
Wonjae Kim, Bokyung Son, and Ildoo Kim. 2021. Vilt: Vision-and-language transformer without convolu- tion or region supervision. In International Con- ference on Machine Learning, pages 5583-5594. PMLR.
Maxwell Forbes, Ari Holtzman, and Yejin Choi. 2019. Do neural language representations learn physical commonsense? In CogSci.
Xingyu Fu, Ben Zhou, Ishaan Chandratreya, Carl Von- drick, and Dan Roth. 2022. There's 's time and place for reasoning beyond the image. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1138-1149, Dublin, Ireland. Association for Computational Linguistics.
Diederik P. Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In 3rd Inter- national Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings.
Roberta L Klatzky. 1998. Allocentric and egocentric spatial representations: Definitions, distinctions. and interconnections. In Spatial cognition. pages 1-17. Springer.
Karl R Gegenfurtner and Jochem Rieger. 2000. Sensory and cognitive contributions of color to the recognition of natural scenes. Current Biology, 10( 1308058888.
Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin John- son, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis. Li-Jia Li, David A Shamma, et al. 2017. Visual genome: Connecting language and vi- sion using crowdsourced dense image annotations. International journal of computer vision, 123(1):32- 73.
Daniel I. Greenberg and Mieke Verfaellie. 2010. Inter- dependence of episodic and semantic memory: Evi- dence from ncuropsychology. Journal of the Interna- Lional Neuropsychological society, 16(5):748-755 7533.
Herbert P Grice. 1975. Logic and conversation. In Speech acts, pages 41-58. Brill.
Alina Kuznetsova, Hassan Rom, Neil Alldrin, Jasper Ui- jlings, Ivan Krasin, Jordi Pont-Tuset, Shahab Kamali, Stefan Popov, Matteo Malloci, Alexander Kolesnikov, et al. 2020. The open images dataset v4. Inter- national Journal of Computer Vision, 128(7):1956 1981.
Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross B. Girshick. 2017. Mask R-CNN. In HEEE Interna- Computer Vision, ICCV 2017,
tional Conference Venice, Italy October 22-29. 2017, pages 2980-2988. [EEE Computer Society.

---

Liunian Harold Li, Mark Yatskar, Da Yin, Cho-Jui Hsich, and Kai- Chang. 2019. Visualbert: A sim- ple and performant baseline for vision and language. arXiv preprint
Cory Paik, Stéphane Aroca-Ouelletee Alessandro Ron- cone, and Katharina Kann. 2021. The world of an octopus: How reporting bias influences language model's 's perception of color. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 823-835.
Xiujun Li, Xi Yin, Chumyuan Li, Pengchuan Zhang, Xiaowei Hu, Lei Zhang, Lijuan Wang. Houdong Hu, Li Dong, Furu Wei, et al. 2020. Oscar: Object- semantics aligned pre-training for vision-language tasks. In European Conference on Computer Vision, pages 121- Springer.
Fabio Petroni, Tim Rocktäschel, Sebastian Riedel. Patrick Lewis, Anton Bakhtin, Yuxiang Wu, and Alexander Miller. 2019. Language models as knowl- edge bases? In Proceedings of the 2019 Confer- ence on Empirical Methods in Natural Language Pro- cessing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages
Fangyu Liu, Guy Emerson, and Nigel Collier. 2022a. Visual spatial reasoning. arXiv preprint arXiv:2205.00366
Hong Kong, China. Association for Computational Linguistics.
Xiao Liu, Da Yin, Yansong Feng, and Dongyan Zhao. 2022b. Things not written in text: Exploring spatial commonsense from visual signals. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pages 2365-2376
Ehsan Qasemi, Filip llievski, Muhao Chen, and Pedro Szekely. 2022. Paco: Preconditions attributed to commonsense knowledge. In EMNLP-Findings.
Alec Radford, Jong Wook Kim, Chris Hallacy. Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sas- try, Amanda Askell, Pamela Mishkin, Jack Clark, et al. 2021. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, pages 8748-8763. PMLR.
Yinhan Liu, Myle Out, Naman Goyal, Jingfei Du, Man- dar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining ap- proach. arXiv preprint arxiv:
Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee. 2019. Vilbert: Pretraining task- agnostic visiolinguis- tic representations for vision-and- and-language tasks. In Advances in Neural Information Processing Systems 32: Annual Conference Neural Information Pro- cessing Systems 2019, NewrlPS 2019, December 8- 14, 2019, Vancouver, BC, Canada, pages 13-23.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, Peter J Liu, et al. 2020. Exploring the limits of transfer learning with a unified text-to-text trans- former. J. Mach. Learn. Res., 21(140):1-67.
René Ranftl, Alexey Bochkovskiy, and Vladlen Koltun. 2021. Vision transformers for dense prediction. In Proceedings of the IEEE/CVF International Confer- ence Computer Vision, pages 12179-12188.
Dhruv Mahajan, Ross Girshick, Vignesh Ramanathan, Kaiming He, Manohar Paluri, Yixuan Li, Ashwin Bharambe, and Laurens Van Der Maaten. 2018. Ex- ploring the limits of weakly supervised pretraining. In Proceedings of the European conference on com- puter vision (ECCV), pages 181-196.
Nils Reimers and Iryna Gurevych. 2019. Sentence- BERT: Sentence embeddings using Siamese BERT- networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natu- ral Language Processing (EMNLP-IJJCN) pages 3982-3992, Hong Kong, China. Association for Com- putational Linguistics.
John McCarthy et al. 1960. Programs with common sense. RLE and MIT computation center Cambridge, MA, USA.
Matthias Minderer, Alexey Gritsenko, Austin Stone, Maxim Neumann, Dirk Weissenborn, Alexey Doso- vitskiy, Aravindh Mahendran, Anurag Arnab, Mostafa Dehghami. Zhuoran Shen, et al. 2022. Sim- ple open-vocabulary object detection with vision transformers. arXiv preprint axX.v.2005.062330
Adam Roberts, Colin Raffel, and Noam Shazeer. 2020. How much knowledge can you pack into the param- eters of a language model? In Proceedings of the 2020 Conference on Empirical Methods in Nanural Language Processing (EMNLP), pages 5418-5426 Online. Association for Computational Linguistics.
James Mullenbach, Jonathan Gordon, Nanyun Peng, and Jonathan May. 2019. Do nuclear submarines have nuclear captains? a challenge dataset for com- monsense reasoning over adjectives and objects. In Proceedings of the 2019 Conference Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Lan- pages 6052-
Shreya Shankar, Yoni Halpern Fric Breck, James At- wood, Jimbo Wilson and D Sculley. 2017. No classi- fication without representation: Assessing geodiver- sity issues in open data sets for the developing world. arXiv preprint arxiv: 171 1111.0836..
Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. 2018. Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic im- age captioning. In Proceedings of the 56th Annual
guage Processing (EMNLP-IJCNN) 6058, Hong Kong, China. Association for Computa- tional Linguistics.

---

Meeting of the Association for Computational Lin- guistics ( (Volume 1: Long Papers), pages 2556-2565 Melbourne, Australia. Association for Computational Linguistics.
Jianwei Yang. Chunyuan Li. Pengchuan Zhang, Bin Xiao, Ce Liu, Lu Yuan, and Jianfeng Gao, 2022. Uni- fied contrastive learning in image-text-lahel space. In Proceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition, pages 19163- 19173.
Amanpreet Singh, Ronghang Hu, Vedanuj Goswami, Guillaume Couairon, Wojeiech Galuba, Marcus Rohrbach, and Douwe Kiela. 2022. Flava: A founda- tional language and vision alignment model. Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15638-15650.
Da Yin, Hritik Bansal, Masoud Monajatipoor, Liu- nian Harold Li, and Kai-Wei Chang. 2022. Geom- lama: Geo-diverse commonsense probing on multi- lingual pre-trained language models. In Proceedings of the 2022 Conference on Empirical Methods Natural Language Processing.
Shikhar Singh, Nuan Wen, Yu Hou, Pegah Alipoormo- labashi, Te-lin Wu, Xuezhe Ma, and Nanyun Peng. 2021. COM2SENSE: A commonsense reasoning benchmark with complementary sentences. In Find- ings the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 883-898, Online. Associa- tion for Computational Linguistics.
Da Yin, Liunian Harold Li, Ziniu Hu, Nanyun Peng, and Kai-Wei Chang. 2021. Broaden the vision: Geo- diverse visual commonsense reasoning. ings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 2115-2122
Proceed-
Rowan Zellers, Yonatan Bisk, Ali Farhadi, and Yejin Choi. 2019. From recognition to cognition: Visual commonsense reasoning. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, USA, June 16-20, 2019, pages 6720-6731. Computer Vision Foundation IEEE.
Robyn Speer, Joshua Chin, and Catherine Havasi. 2017. Conceptnet 5.5: open multilingual graph of gen- eral knowledge. In Proceedings of the Thirty-First AAAI Conference Artificial Inteliience, February 4-9, 2017, San Francisco, California. USA, pages 4444 4451. AAAI Press.
Chenyu Zhang, Benjamin Van Durme, Zhuowan Li. and Elias Stengel- Eskin. 2022. Visual commonsense in pretrained unimodal and multimodal models. Proceedings of the 2022 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies, pages 5321-5335, Seattle, United States. Association for Computational Linguistics.
Alon Talmor, Yanai Elazar, Yoav Goldberg, and Jonathan Berant. 2020. oLMpics-on what language model pre- pre-training captures. Transactions of the As- sociation for Computational Linguistics, 8:743-758.
Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. 2019. Co A ques- tion answering challenge targeting commonsense knowledge. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Tech- nologies, Volume I (Long and Short Papers), pages 4149-4158, Minneapolis, Minnesota. Association for Computational Linguistics.
Bolei Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso, and Antonio Torralba. 2017. Scene parsing through ADE20K dataset. In 2017 IEEE Conference on Computer Vision and Pattern Recog- nition, CVPR 2017, Honolulu, HI, USA, July 21-26, 2017, pages 5122-5130, IBEE Computer Society.
Zhengxia Zou, Zhenwei Shi, Yuhong Guo, and Jieping Ye. 2019. Object detection in 20 years: A survey. arXiv preprint arXiv: 1905.55055
Endel Tulving. 1972. Episodic and semantic memory. Organization of memory.. pages xiii, 423-xiii, 423. Academic Press, Oxford, England.
Alex Wang. Yada Pruksachatkun, Nikita Nangia, Aman- preet Singh. Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. 2019. Superglue: A stickier benchmark for general-purpose language understand- ing systems. In Advances Neural Information Processing Systems 32: Annual Conference on Neu- ral Information Processing Systems 2019, NeurIPS 2019, December 8- 14, 2019, Vancouver, BC, Canada, pages 3261-3275.
Peng Wang, An Yang. Rui Men, Junyang Lin, Shuai Bai, Zhikang Li. Jianxin Ma, Chang Zhou, Jingren Zhou, and Hongxia Yang. 2022. Ofa: Unifying ar- chitectures, tasks, and modalities through simple sequence-to-sequence learning framework. In Inter- national Conference on Machine Learning, pages 23318-23340. PMLR.

---

Appendix
A.4 Prediction to Basic Color
We use OFA in our pipeline, and map generated text to the basic color set as shown in Table 7. If the model predicts multiple colors, we assign each of them to the object instance.
A Implementation Details
A.1 Pipeline Parameters
To cluster objects for computing relative size, we use Jenks Natural Breaks (Jenks, 1967), with #clus- ters = 5 following the manual groupings of object sizes in Liu et al. (2022b). We also experimented with #clusters 3, but qualitatively observed less optimal clusters. In spatial module, we create partitions of uniform size and overlap.
Basic Color
Raw Predicted Terms
Yellow
gold, golden, blonde, beige, peach, cream wooden, tan, beige, bronze, copper grey, silver, metal, steel
Brown
Gray
Pink
peach
Purple
violet
A.2 Typical Labels
Red
maroon
Green
teal
We derive typical labels from the raw label proba- bilities (C) by filtering classes per predefined threshold Pmin. that can be interpreted as either noise or rare occurrence. Formally, we apply the fil- ter as follows: C p)lp Pminn (c,p) E C}. Here, Pmin is defined as:
Blue
teal, turquoise
Table 7: Mapping between raw predictions (OFA) and basic color terms.
A.5 Few-Shot Performance
10%
4 |C| < 11 |C| =
We report few-shot results for color (Fig. 8) and size (Fig. 9) task, by building variants of the train set with respect to the percentage of samples. The improvement in VLMs and CapBERT likely in- dicates prompt adaptation due to their relatively weaker linguistic ability compared to LMs.
20%
30%
|C| = 2
The resulting distribution is re-normalized and the filtering step is applied recursively.
A.3 Defining Spatial Relations
Our objective to map the raw coordinates in an image for two objects to discrete relations. We de- fine simple set of rules to convey above and simi- lar level, from object annotations. Given bounding box or polygon mask, first compute its centroid along with the lowest point. We then compare the y- coordinates of objects as illustrated Fig. 7. an object's lowest point above the other's centroid, we map it to above, else similar level.
%45ample
Figure 8: Few-shot performance color with respect to percentage of samples in the train set.
0, = O2
O < O2
Figure 7: Illustrates definition of spatial relation from raw annotations.

---

BdIT
Category
Objects
VIST
Debiiils
person
man. woman, lady. buy, girl, child hand, mouth, hair, teeth, skin
FLAIA
human body
CUP
CapRERT
animal
bird, fish, reptile, insect, cattle, pet
animal body
paw, hooves, tail, mane, fur, whisker sky, tree, flower, sea, beach, snow vegetable, fruit, dessert, snack table, chair, shelif, bed, cabinet fridge, knife, bowl, sink towel, shower, toothbrush, toilet helmet. racket, ball, glove, skateboard hat, shirt, pant, skirt, shoc, glasses laptop, mouse, printer, cell, projector bike, car, bus, truck, hoat, plane traffic, sign, crosswalk, hydrant, pole store, building, school, airport, hridge lamp, paper, trashean, bug, extinguisher entity, object, thing, stuff, item
nature
food
furniture
kitchenware
bathroom
sports
clothing
electronics
vehicle
street
construction
26
utility
miscellancous
Figure Few-shot performance on size.
Table 8: Categories and objects used semantic cues. We map them to sentence with "<Category>: <Objects>" template.
Context
Examples
12
Event
wedding cake, bathing soap
Location
kitchen sink, street lamp
State
hare tree, sliced apple
Part
piano key, bike wheel
Table 9: Context-based subtype examples
Bryy
Figure 10: Color distribution objects in VIPHY and CoDa.
Regions
orange
food in the sink
round fruit in sink
Way
lightt hitting the sink
15
26
middle fruit slnk
marks in the sink
nine fruit sink
round center of fruit
10
silver object above sink
orange right
Figure 12: A sample image and corresponding captions from Visual Genome dataset. Illustrates how humans omit the subtype kitchen sink, when annotating images.
Figure 1l: Color distribution with respect label cardi- nality in VIPHY and CoDa.

---

704
Figure 13: Scene-wise wise performance on spatial task, with dashed lines indicating the overall scores. The sample distribution is 12%, 7%, 42%, 34%, 2%.
44 4 of fM00 H # 1th 1N " #t gft 44
Figure 14: Spatial relation distribution for object pairs in kitchen scene from VIPHY.

---

sports equipmmrt (1299)-
stroce (1399)
wehicle (999)
naturr (996)
clothing (895)
fumiiture (899)
food (0%)
kitch hecherrreee (59%)
bathroom (595)
electronica (095)
animaal body (495)
construction (399)
person (39%)
misccllonccuz (295)
human body (399)
utiltyy herms (196)
animall
srugory
Figure 15: Color distribution for object categories as defined in $4.2. The parenthesized values indicate the percentage of samples in each category.
Figure 16: Color distribution for 90 objects from VIPHY, sorted by entropy.


Reviews:
[{"review_id": "YLJLgfUy0o", "reviewer": "Reviewer_btZN", "summary": "", "questions": "Question A: In the evaluation method related to unmasking a token, is there any method to deal with synonymous tokens and prevent them from polluting the evaluation score?\n\nQuestion B: Does the analysis and conclusion still hold for recent models like OFA? And what about even more recent models such as LLaVA (which is within 3 months and is totally okay to ignore this part)?\n\nQuestion C: What are the strengths of this dataset compared to previous smaller, manually collected datasets? Can you illustrate these strengths with examples?", "limitations": "", "rating": 2, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate your review and detailed comments on our work. Regarding your presentation suggestions, we will add more examples from the data to the appendix. Please find responses to your other points below.\n\n### Additional VLMs\nWe agree with the reviewer that the evaluation could be more comprehensive by including more recent VLMs. However, the goal of our study is to evaluate the effectiveness of access to visual modality in VLMs in the context of visible commonsense reasoning. We show that the model trained on captions (CapBERT) outperforms the VLMs that we evaluated. This suggests that access to visual modality is not always necessary for good performance on visible commonsense reasoning tasks. \nWe believe that the four VLMs that we evaluated represent a diverse range of architectures and capabilities and that our results are still valid even if we do not include the most recent models.\nWe would be happy to include more recent VLMs in our evaluation in future work. However, we believe that our current results are still valuable and contribute to the understanding of the role of visual modality in VLMs for visible commonsense reasoning.\n\n### Contribution and Strength of Resource\nOur first contribution is the automated pipeline that bypasses crowdsourced annotations to cover 30 times more objects than any existing resource. The only limitation of our pipeline is the existence of a high-quality bounding box and object detection that given current SoTa is cheap to acquire, hence ViPHy can scale up to new objects without the need for expensive human supervision.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "A. The proposed dataset is relatively large in size and  might be a helpful resource to the community.\nB. The writing is clear.", "reasons_to_reject": "A. The evaluation lacks inclusion of recent models, which may limit the comprehensiveness of the assessment. Among the four VLMs evaluated, only one is from 2022, while the rest are from earlier years. Including more recent, potentially stronger models like OFA, Unified-IO etc., would make the analysis and conclusions more robust and up to date.\n\nB. Despite the author's emphasis on the dataset's size as a novelty, previous benchmarks have derived similar conclusions and analyses using smaller, manually curated datasets as partially noted in the Related Works section. The paper does not clarify how utilizing a larger, potentially noisier dataset adds unique value or insights to the community.", "questions_to_authors": "Question A: In the evaluation method related to unmasking a token, is there any method to deal with synonymous tokens and prevent them from polluting the evaluation score?\n\nQuestion B: Does the analysis and conclusion still hold for recent models like OFA? And what about even more recent models such as LLaVA (which is within 3 months and is totally okay to ignore this part)?\n\nQuestion C: What are the strengths of this dataset compared to previous smaller, manually collected datasets? Can you illustrate these strengths with examples?", "paper_topic_and_main_contributions": "This paper proposes a large, machine-generated dataset for evaluating Vision-language Models (VLMs) performance related to object color, size and spatial relationships. The result shows limitations of existing VLMs especially in spatial relation reasoning tasks.", "excitement": 2, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "[1] Lu, Jiasen, et al. \"Unified-io: A unified model for vision, language, and multi-modal tasks.\" arXiv preprint arXiv:2206.08916 (2022).", "Typos_Grammar_Style_and_Presentation_Improvements": "It would be more illustrative to include some sampled responses from the models in the body or appendix of the paper."}, "extra_text": {}}, {"review_id": "Jt3PfSkiHJ", "reviewer": "Reviewer_jjqR", "summary": "", "questions": "A. How do you validate that the instances are \"correct\"? For example, for size questions, suppose you see a zoomed-in cup in front of you and part of a car visible in the background. Running the label construction pipeline that relies on depth estimation would still consider the cup as \"larger\" than the \"car\". This is most likely the reason for relatively low oracle score in human performance in Table 5, where certain instances simply contain labels that are not true based on the human level of commonsense understanding.\n\nB. What is the human acceptability / validation rate of collected instances?\n\nC. How does this work provide more value than ViComte which follow similar automatic pipeline and same data source (Visual Genome)? \n\nD. What is the human performance when they are not looking at the image?\n\nE. What is the performance of current SoTA LLMs such as ChatGPT?\n\n  \n\n", "limitations": "", "rating": 2, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate your review. Please find responses to your points below.\n### No Human Validation\nWe agree with your point that any automated pipeline is susceptible to error. However, as shown in the human evaluations of Table 3, the data generated by ViPHY is of high quality. We have taken steps to mitigate the effects of noise on the data, such as feeding the bounding box of each object to the OFA model, which helps to improve the accuracy of color extraction. Additionally, the evaluation focuses on transitivity through pair-wise comparison, which minimizes the impact of size errors. Finally, the depth grouping algorithm ensures that objects are only compared in relatively equal depth, which eliminates cases where relative proximity results in an inflated size measurement.\nWe did not feel it would be interesting to the general community to include a detailed ablation study and opted for an overall human evaluation of the data.\n\n\n### Unclear Motivation\nThe main strength of our first contribution, the ViPHy pipeline, is its scalability to new objects and sizes in all three tasks (30 times more objects than any existing resource) while preserving a reasonable level of quality for visible commonsense reasoning (human evaluation in Table 3). The only limitation of ViPHy's automatic pipeline is its reliance on bounding boxes (Figure 3).\nAlthough there are available resources on visible commonsense knowledge, such as Visual Genome and Winoground, they are all created using direct human annotations, which makes them both expensive to create and unscalable to new objects. In contrast, ViPHy's automatic pipeline is significantly more scalable and cost-effective.\nWe respectfully disagree with your formulation that our \"dataset is constructed based on VisualGenome\". We solely use the images from VisualGenome due to their high-quality of bounding boxes. However, ViPHy is not limited to using VisualGenome images. In principle, ViPHy can be used to reproduce any resource, such as VisualGenome, without the need for costly human annotations.\n\n\n### Limited Coverage\nWe agree with your suggestion of using other setups, such as counterfactuals, for evaluating \"visible commonsense\" understanding. However, we believe that our approach of focusing on the simplest aspects of visible commonsense is a valid one, as it allows us to identify the shortcomings of SoTA VLMs in reasoning with these basic concepts.\nWe also respectfully disagree with your formulation that color, size, and spatial knowledge are not part of commonsense knowledge. In fact, numerous prior works have classified these concepts as part of commonsense knowledge (Ilievski et. al. \"Dimensions of commonsense knowledge.\" Knowledge-Based Systems 229 (2021)). Therefore, we believe that our work provides a valuable contribution to the field of commonsense reasoning by demonstrating the importance of these seemingly simple concepts.\n\n\n### CapBERT and Text-bias in data\n\nWe respectfully point out a misunderstanding of our work. All of the analyses in Section 4.1 are conducted in a text-only setup. In this work, we first use an automatic pipeline to extract visible commonsense knowledge from images. We then use the extracted knowledge (not the images) to probe two groups of models: language models (LMs) that are only trained with text modality, and visual language models (VLMs) that observe the world during their training through both visual and textual modalities. The goal of this experiment is to evaluate and compare VLMs' understanding of visible commonsense knowledge. We will make sure to clarify this in the final version.\nTherefore, we respectfully disagree with your argument that there is a text-only bias in our work and your statement that \"visual understanding is not required\".\nWe also note that you have mentioned that models like CapBERT have been previously used in similar setups. However, we believe that our work is novel as our evaluation is on a larger scale and on the three dimensions: color, size, and space.\n\n\n### LLMs\nWe agree with your point on the need for the addition of LLMs such as ChatGPT as a baseline. Unfortunately, we were not able to finalize such results during the rebuttal period but they will be added to the final manuscript.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": " - Introduces an innovative pipeline to automatically construct visual physics commonsense dataset that involves use of diverse tools: parser, depth estimation, vision language models, bounding box areas.   \n- Presents empirical findings that the vision language models show far worse performance in VIPHY than humans and calls for the need to address visible physics common understanding of such models to the NLP community.  \n- Demonstrates an in-depth evaluation of the performance of current vision language models, taking into account various factors such as types of questions, color categories, and label cardinality.", "reasons_to_reject": "- No human validation of automated process which might be prone to error due to the following reasons:\n  - Colors are derived from prediction of VL model, specifically OFA that might not yield a correct answer. \n  -  Size relies on depth estimation models and bounding box areas to get the label. How do you account for the  case when smaller objects are zoomed in for size questions (See the example in Question A)?\n  - Spatial: how do you validate that the objects are similar level with one another vs \"smaller/larger\". What if objects are in front of / behind one another, and have different levels of bounding box coordinates? \n\n- Unclear motivation of dataset collection for visible physics commonsense v.s. prior work:\n  - Dataset is constructed based on VisualGenome, which already include color and spatial information (above, under, behind, in front of, etc) in their scene graphs attributes and relation labels, and size can be trivially inferred from object labels most of the times. Why can we not just evaluate on a subset of VisualGenome scene graph tasks that deal with colors and spatial information?\n  - Winoground [1] was carefully hand-curated by expert annotators instead of relying on model-based predictions, and include far rich annotations testing visual-language compositionality that actually require looking at images to do well.\n  - ViComte [2] already covers similar and diverse visual commonsense knowledge, such as color, shape, material, size, and visual co-occurrence in their dataset. How does the authors' work provide more benefits over this dataset?   \n \n- Limited coverage of \"visible commonsense\" understanding. Color seems to be purely vision task, and size and spatial most of the times can be inferred from text information. There are far more interesting cases that test such knowledge, such as counterfactual reasoning asking (what would happen if this action is applied to the object in this image).      \n- The findings of CapBERT is not novel as it is already explored in Zhang et. al [2]. and the paper simply runs the same set up to validate their dataset. \n  - This in fact further highlights that this dataset is essentially prone to **severe text biases, in which visual understanding is not required to perform well for size and spatial questions**. It is difficult to claim that the dataset tests *visible* physics commonsense knowledge if such information is not needed. \n- Language model experiments are not as comprehensive and do not not cover performance of stronger LLMs such as ChatGPT. It would be more interesting for the language community to understand in their physics commonsense understanding skills.\n- Experiment results illustrate that the task does not have much too improve on from using current SoTA models. Color questions are extracted from OFA models, and size + spatial questions do not require visual understanding to perform well. Couldn't then the task be solved by 1) run OFA (which is not evaluated on this dataset) for color questions, 2) run LLMs such as GPT-4 for size + spatial reasoning tasks.\n- More qualitative examples of when images are important to do well in the task. \n\n[1]: Winoground: Probing Vision and Language Models for Visio-Linguistic Compositionality [Thrush et. al.]\n\n[2]: Visual Commonsense in Pretrained Unimodal and Multimodal Models [Zhang et. al.]", "questions_to_authors": "A. How do you validate that the instances are \"correct\"? For example, for size questions, suppose you see a zoomed-in cup in front of you and part of a car visible in the background. Running the label construction pipeline that relies on depth estimation would still consider the cup as \"larger\" than the \"car\". This is most likely the reason for relatively low oracle score in human performance in Table 5, where certain instances simply contain labels that are not true based on the human level of commonsense understanding.\n\nB. What is the human acceptability / validation rate of collected instances?\n\nC. How does this work provide more value than ViComte which follow similar automatic pipeline and same data source (Visual Genome)? \n\nD. What is the human performance when they are not looking at the image?\n\nE. What is the performance of current SoTA LLMs such as ChatGPT?\n\n  \n\n", "paper_topic_and_main_contributions": "VIPHY is a dataset measuring vision language models' understanding of visible physics commonsense knowledge, specially on color, size, and spatial relations of objects in the image. The dataset collection pipeline is done automatically without human annotation which first involves extracting object names and subtypes from existing captioning dataset, such as Visual Genome and ADE20K. Then, VLMs are used to infer colors object regions, size and spatial relations are inferred based on the ratio of depth estimation and bounding box area. Experiments probing the vision language model in their benchmark show a huge gap between model and human performance across all the tasks. In addition to vision-language model, the authors explore training a language only model on captioning dataset which achieve superior results in inferring size and spatial knowledge than VL models. ", "excitement": 2, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "gLjmaSAkQi", "reviewer": "Reviewer_2fNQ", "summary": "", "questions": "I appreciate this work, and I really hope the authors can address my concerns in the rebuttal, especially:\n\n**Q1**: More implementation details of the proposed framework. For example, what object detectors do the authors use for cluster relation? What depth estimation method does the authors use?\n\n**Q2**: More evidence or justification, to bridge the gap between 'Physical Commonsense Knowledge' and more rigid definition from image formulation and preception. \n\n**Q3**: Some other presentation issues, listed in the presentation section.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate your review and your detailed comments for improving this work. Please find responses to your points below.\n### Implementation Details\nWe respectfully want to point out L257 “To compute depth map from monocular image, we use DPT (Ranftl et al., 2021)” which addressed your point on depth estimation. Regarding the object detector, we relied on the bounding boxes provided by the visual genome L238.\n\n\n### Related studies from CV\nWe appreciate your feedback on the parallel terms from the computer vision community. In this work, we have focused on a simple subset of visible commonsense that can be easily inferred from static images. However, we acknowledge that the common sense associated with the physical attributes of objects is more extensive than the three categories we have considered. We will update the manuscript to include more relevant literature on \"physical knowledge.\"", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- This work proposes a new task to estimate the attributes such as size, color and space from images, by leveraging VLM and other vision tools.\n\n- The experiments are very extensive and solid. \n\n- This paper is well-written and easy-to-follow. \n\n- The authors provide both source code and data along with the submission, which I appreciate a lot. I hope it can be made publicly avaible, once published.", "reasons_to_reject": "- Some implementation details inside the proposed framework are not very informative from the submission. For example, what object detectors do the authors use for cluster relation? What depth estimation method does the authors use?\n\n-  As I am not an expert in NLP but in Computer Vision, especially with some low-level vision background, the claim of 'Physical Commonsense Knowledge' does not seem very comfortable for me. In fact, from the image preception and image formulation prespective, the physicial attributes may actually contain more attributes than color, space and size. \nSo, it would be highly appreciated if the authors can enhance this part, and provide some references to justify that, these three key factors can also align with the definition of physcial knowledge in vision community. \n\n\n- Some other minor issues on presentation, please refer to the 'presentation improvement' section.", "questions_to_authors": "I appreciate this work, and I really hope the authors can address my concerns in the rebuttal, especially:\n\n**Q1**: More implementation details of the proposed framework. For example, what object detectors do the authors use for cluster relation? What depth estimation method does the authors use?\n\n**Q2**: More evidence or justification, to bridge the gap between 'Physical Commonsense Knowledge' and more rigid definition from image formulation and preception. \n\n**Q3**: Some other presentation issues, listed in the presentation section.", "paper_topic_and_main_contributions": "This paper studies an interesting field, how to preceive the physical commonsense knowledge from images via vision-language model.\nFrom the technique side, the proposed framework first need to estimate both depth information and fine-grained categories from the input image. \nThen, the depth and object size information is used for clustering, and the color information is acuqired from VLM.\nThe authors leverage ADE20K and Visual Genome for the experiments, which are evaluated on multiple state-of-the-art methods on both zero-shot and fine-tuning settings.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "N/A", "Justification_for_Ethical_Concerns": "No ethical issues."}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "There are indeed some presentation issues.\n\n- First, and the most important one. Page1, below the title. Why 'Anonymous ACL submission'?\n\n- Each equation does not assign its number.\n\n- From the reviewer's background, it would be better to present each function operation by using \\rm function in latex. \n\n- Fig.4 is not very informative. What does each color mean is not reflected in either legend or caption."}}, {"review_id": "OtEuVrwFTd", "reviewer": "Reviewer_Vefi", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate your review. Please find responses to your points below.\n\n### Latest VLMS\nWe agree with the reviewer that the evaluation could be more comprehensive by including more recent VLMs. However, the goal of our study is to evaluate the effectiveness of access to visual modality in VLMs in the context of visible commonsense reasoning. We show that the model trained on captions (CapBERT) outperforms the VLMs that we evaluated. This suggests that access to visual modality is not always necessary for good performance on visible commonsense reasoning tasks. \nWe believe that the four VLMs that we evaluated represent a diverse range of architectures and capabilities and that our results are still valid even if we do not include the most recent models.\nWe would be happy to include more recent VLMs in our evaluation in future work. However, we believe that our current results are still valuable and contribute to the understanding of the role of visual modality in VLMs for visible commonsense reasoning", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The problem about physical knowledge acquisition and reasoning of VLM is generally interesting.\n2. The work contributed a new dataset in this line of work.\n", "reasons_to_reject": "Latest VLMs such as InstructBLIP, LLaVA not included for experiment which makes the claim that existing VLMs struggle to effectively consolidate physical knowledge less appealing.  ", "questions_to_authors": "", "paper_topic_and_main_contributions": "This work explores the problem of physical knowledge acquisition of VLMs. \nMain contributions:\n1. Proposed an automatic pipeline for physical knowledge data generation \n2. Constructed VIPHY dataset for physical knowledge based on existing Genome and ADE20K dataset\n3. Evaluated a list of LMs and VLMs and reported performance comparison \n", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: **Summary:**
This paper proposes a large, machine-generated dataset, called VIPHY, aimed at evaluating Vision-language Models (VLMs) performance related to object color, size and spatial relationships. The result shows limitations of existing VLMs especially in spatial relation reasoning tasks. The automatic pipeline extracts object names and subtypes from existing captioning dataset, such as Visual Genome and ADE20K.
The authors evaluate the results on multiple state-of-the-art methods on both zero-shot and fine-tuning settings. Experiments probing the vision language model in their benchmark show a huge gap between model and human performance across all the tasks. In addition to vision-language model, the authors explore training a language only model on captioning dataset which achieve superior results in inferring size and spatial knowledge than VL models.

**Strengths:**
The reviewers agree on the following strengths:
1. The authors have introduced a substantial dataset that holds the potential to become a valuable resource for the community.
2. Their work presents an innovative pipeline for the automatic construction of a visual physics commonsense dataset, leveraging diverse tools such as a parser, depth estimation, vision language models, and bounding box areas. Building upon these foundations, the contribution introduces a novel task: estimating attributes like size, color, and space from images.
3. The authors provide empirical evidence demonstrating that vision language models perform considerably worse in VIPHY compared to humans. This underscores the imperative need to address the common understanding of visible physics in such models within the NLP community.
4. Lastly, the authors have thoughtfully included both source code and data with their submission. It would be worthy if these resources could be made publicly available upon the paper's publication.

**Weaknesses:**
Reviewers have identified the following weaknesses in the paper:
1. The work lacks the inclusion of recent models, potentially limiting the comprehensiveness of the assessment. The absence of stronger models like OFA and Unified-IO harms the robustness of the analysis. Moreover, this absence weakens the claim that existing Visual Language Models (VLMs) struggle to effectively integrate physical knowledge. Actually, it would be more intriguing for the language community to gain insights into the capabilities of Large Language Models (LLMs) in the domain of physics commonsense understanding.
2. The paper does not clearly articulate how utilizing a larger, “potentially noisier” dataset adds insights to the community. In fact, previous benchmarks have reached similar conclusions and analyses using smaller, manually curated datasets.
3. Reviewers have emphasized the absence of human validation in the automated process, which may introduce errors for several reasons. Additionally, the paper has limited coverage of "visible commonsense" understanding.
4. The existence of CapBERT highlights that the dataset may be prone to significant text biases. This means that visual understanding may not be necessary for size and spatial questions, leading doubt on the claim that the dataset effectively tests visible physics commonsense knowledge.

**Author-Reviewer discussion and acknowledgment:**
Reviewers raised various questions and concerns to which the authors responded during the rebuttal and discussion phase, outlining planned improvements and engaging in further discussions with the reviewers. All reviewers have responded and acknowledged the authors' arguments, while still pointing out some weaknesses in the paper.

**Conclusion:**
The writing is clear, and the paper is well-written and easy to follow. However, reviewers suggest that the authors address the identified typos. Furthermore, reviewers recommend that the authors include additional references and improve the paper based on the points raised during the discussion phase.
Decision: Accept-Findings