Can Language Models Understand Physical Concepts?
Authors: Lei Li
Jingjing Xu
Qingxiu Dong
Ce Zheng
Xu Sun
Lingpeng Kong
Qi Liu
Venue: EMNLP 2023 Main (2023)
Keywords: large language models
embodied concept understanding

Abstract: Language models (LMs) gradually become general-purpose interfaces in the interactive and embodied world, where the understanding of physical concepts is an essential prerequisite.
However, it is unclear whether LMs can understand physical concepts in the human world. 
To investigate this, we design a benchmark VEC that covers the tasks of (i) Visual concepts, such as the shape and material of objects, and (ii) Embodied Concepts, learned from the interaction with the world such as the temperature of objects. 
Our zero (few)-shot prompting results show that
the understanding of certain visual concepts
emerges as scaling up LMs, but there are still
basic concepts to which the scaling law does not apply.
For example, OPT-175B performs close to humans with a zero-shot accuracy of $85$\% on the material concept, yet behaves like random guessing on the mass concept.
Instead, vision-augmented LMs such as CLIP and BLIP achieve a human-level understanding of embodied concepts.
Analysis indicates that the rich semantics in visual representation can serve as a valuable source of embodied knowledge. Inspired by this, we propose a distillation method to transfer embodied knowledge from VLMs to LMs, achieving performance gain comparable with that by scaling up parameters of LMs $134\times$. Our dataset is available at https://github.com/TobiasLee/VEC.

Can Language Models Understand Physical Concepts?
Lei Li', Jingjing Xu², Qingxiu Dong', Ce Zheng', Xu Sun', Lingpeng Kong', Qi Liu' SShanghai AI Lab
'The University of Hong Kong
*Peking University
nlp. ppp.lieei@gmail... com {jingjingxu, zce1 ace1112zslx, xusun)@pku. (euuu))@pu.euu.cc edu.
daxēstu.pka edu.cn cn
(1pk, liuqi jecs. hku. hk
Abstract
these tasks, e.g., producing correct instructions for cleaning the coke requires understanding the visual characteristics
Language models (LMs) gradually become general-purpose interfaces in the interactive and embodied world, where the understand- ing of physical concepts essential prereq- uisite. However, it is unclear whether LMs can understand physical concepts in the human world. To investigate this, we design bench- mark VEC that covers the tasks of (i) Visual concepts, such as shape and material of objects, and (ii) Embodied Concepts, learned from the interaction with the world such as the temperature objects. Our zero (few)-shot prompting results show that the understanding certain visual concepts emerges as scaling up LMs. but there are still basic concepts to which the scaling law does not apply. For example, OPT-175B performs close to humans with zero-shot accuracy of 85% the material con- cept. yet behaves like random guessing on the mass concept. Instead, vision-augmented I.Ms such CLIP and BLIP achieve a human-level understanding of embodied concepts. Analysis indicates that the rich semantics visual rep- resentation serve valuable source of embodied knowledge. Inspired by this, we pro- pose distillation method transfer embodied knowledge from VLMs to LMs, achieving per- formance gain comparable with that by scaling up parameters of LMs 134x.
coke can, as well as physical properties such as hardness. However, still re- mains unclear whether current LMs can understand basic physical concepts (Driess et al., 2023).
To answer the question, we first define an evalua- tion suite of physical concepts covering visual and embodied concepts. Specifically, visual concepts examine knowledge that can be gained via visual perception, including generic visual concepts, such as color, shape, and material of common objects, and spatial perception, which focuses on the rela- tionship between visual stimuli, i.e., relative size and height of objects. The ability to deal with vi- sual concepts serves as the basis for understanding real-world scenes to perform further instruction. Embodied concepts examine knowledge that quires more interaction and multimodal sensory ex- perience in the embodied world, including knowl- edge about the mass, temperature, and hardness of objects, c.g., ice is colder than water. Understand- ing embodied concepts is essential for an embod- ied agent to make correct choices when translating language into actions (Bisk et al., 2020a). We com- pose Visual and Embodied Concepts evaluation benchmark VEC, with examples shown in Table 1.
With the benchmark, we examine wide range of LMs. We cover masked language models and causal language models text-only LMs, includ- ing BERT (Devlin et al., 2019) and RoBERTa (Liu et al., 2019b), GPT (OPT)-family (Radford et al., 2019; Zhang et al., 2022b) with parameters rang- ing from 125M to 175B, LLaMA-1/2 (Touvron et al., 2023a,b) and Vicuna (Chiang et al., 2023). Furthermore, humans understand the world by learning from multiple modalities, especially using the visual modality (Bloom, 2002), we are inter- ested in whether the vision supervision in recent iision-aggmented language models (VLMs) (Chen et al., 2019: Radford et al., 2021; Wang et al., 2022;
1 Introduction
With the emergent capabilities such as arith- metic (Brown et al., 2020; Wei et al., 2022) and multi-step reasoning (Chowdhery et al., 2022) brought by large-scale pre-training, language mod- els (LMs) are gradually becoming unified inter- faces (Hao et al., 2022), capable of instructing em- bodied robots for high-level tasks such as clean- ing the spilled coke in interactive and embodied environments (Ahn et al., 2022). Understanding physical concepts is an essential prerequisite for
OOu dataset is available at TebiasLee/VEC.

---

Conepet Category
Instance
Label
# of Examples
Color
h: melon, green, f2r black
green
574
Shape
h: lemon, ty: triangle, 12: round
round
140
Visual Concepts
Material
h: guitar, &1: wood, 12: glass h: ant, r: larger than, t: bird
wood
284
Size
false
500
Height
h; bottle, r: shorter than, truck
true
500
Mass
h: wooden spoon, heavier than, t: toaster
false
654
Embodied Concepts
Temperature
h: ice, colder than, t: water
true
422
Hardness
h: pearl, r: softer than, t: glass
true
1,016
Table 1: The illustration of VEC benchmark. We design two forms of probing tasks. The former (Color, Shape and Material) asks models to make a choice between two tail options given the head object. The latter (Size, Height, and embodied concepts) requires LMs to judge whether the relation valid given the head the tail.
Madureira, 2021) could also facilitate the under- standing ability of embodied concepts. CLIP (Rad- ford et al., 2021) and BLIP (Li et al., 2022a) chosen representatives of VLMs for evaluation, due their promising performance and the abil- ity to deal with textual-only inputs. To eliminate the effects of training corpus (Tan and Bansal, 2020), we train BERT, OPT, and CLIP on the same caption dataset with similar Transformer model (Vaswani et al., 2017) from scratch for a fair evaluation. Furthermore, as previous studies have shown that prompting methods that fit the pre- training paradígm could better elicit the knowledge learned from LMs (Petroni al., 2019; Schick and Schütze, 2021a; Brown al., 2020), we adopt orc-trrineed-bjccctvve style promoting methods to narrow the gap between probing and pre-training.
average accuracy gain of 3.38, comparable to the 4.46 gain achieved by scaling the model parameters 134x. Nevertheless, the improved LMs still exhibit great gaps with humans, indicating great potential for further advancements.
VEC Benchmark
Our VEC benchmark aims evaluate the under- standing of physical concepts of LMs. Inspired by the world scope definitions by Bisk et al. (2020a), we divide physical knowledge into visual knowl- edge and embodied knowledge. The former are visual properties that can be acquired via visual perception, while the latter focus on knowledge that requires multimodal sensory interaction.
2.1 Visual Concepts
Our zero (few)-shot results on the VEC bench- mark show that: (i) Moderate-sized LMs such as BERT and RoBERTa exhibit a random- under- standing of both visual and embodied concepts. (ii) A decent visual understanding of specific concepts emerges LMS scale up, while they still struggle to understand the embodied knowledge with perfor- mance slightly better than random guessing. (iii) image-grounder caption text-only pre-training, in- struction tuning, and visual supervision could pro- vide performance gain regarding visual concepts, yet only the last one enhances the understanding of embodied knowledge of LMs.
Perception is necessary for language learning be- cause it forms the basis for many of our semantic axioms (Bisk et al., 2020a). Among the various types of perception, visual concepts model vast- ness of experiences in the world that cannot be stated by text alone (Harnad, 1990). In this work. consider evaluating the visual understanding ability of LMs by examining their performance on various visual concepts. Specifically, we com- bine the recently proposed visual knowledge prob- ing datasets, including Spatial Commonsense (Liu et al., 2022a) and ViCom'Te (Zhang et al., 2022a). The combined dataset requires not only understand- ing various generic visual concepts including color, shape, and material, but also understanding the lationship between common objects, such size and height. For generic visual concepts. i.e. color, shape, and material identification, we define an an- swer selection game: selecting a correct value from two options for the attribute given an object. For
We further investigate the source of embodied knowledge in VLMs. A study demonstrates that embodied knowledge in the VLM of CLIP is potentially rooted in the rich semantics of im- age representations. We thus devise a knowledge distillation method to transfer the learned embod- ied knowledge in VLMs into LMs, resulting in an

---

example, given head object banana, the model should pick the ground-trun tail answer yellow in- stead of an alternative option such as black. For vi- sual relationships, i.e., size and height understand- ing, we define a comparison game: LMs need to perform a comparison between different objects. For example, given a head entity ant and a tail entity bird, the LM is asked to compare the size of two objects and makes a prediction between the correct relation smaller and the false one larger.
vapor is 100 We convert the object with temper- ature annotations into pairs, and each pair contains two objects and the corresponding temperature re- lation. For example, (ice, colder than, water vapor). The temperature gap between two ob- jects must be greater than difference threshold, which is loosely set to 10 for assurance of ther- mal perception for human (Jones, 2009). The final Temperature dataset consists of 422 pairs in total.
Hardness is a measure of the resistance to localized plastic deformation in ma- terial science. For example, hard metals such titanium are harder than soft minerals such talc. Humans can perceive the hardness of different ma- terials in interaction with the environment by us- ing tactile organs like fingers (Gueorguiev et al., 2016). To investigate whether LMs capture hard- ness knowledge, we build a Hardness dataset by collecting the Mohs hardness scores of 25 objects from Wikipedia. We define the task in com- parison format. For example, (talc, softer than, titanium). Each pair contains two ob- jects. The gap between two objects is greater than the threshold for human-level understanding. The final dataset contains 1, 016 pairs.
Hardness Dataset
2.2 Embodied Concepts
The embodied concepts refer physical realities of objects, e.g., mass, and temperature, which in- fants could learn by interacting with the environ- ment (Gopnik et al., 1999). This kind of knowledge is the basis of intelligence and enables agent mod- els to explore challenging tasks in physical environ- ments. We are curious about whether current LMs capture embodied knowledge via large-scale pre-training. In this work, we define embodied knowledge as the knowledge that requires multi- modal sensory interaction with the environments beyond visual perception. We construct embod- ied knowledge evaluation datasets regarding basic physical properties including mass, temperature, and hardness.
3 Prompting Methods
Mass Dataset We build the Mass dataset by trans- forming the Image2Mass dataset curated by Stan- dley al. (2017), which annotates 56 common objects with corresponding weights. The most weight object in the dataset is a red Lego brick, weighing 0.026 lbs, and the heaviest ob- ject is a 2.664 lbs drill. Directly asking the LM for the absolute mass of objects can be challeng- ing (Wallace et al., 2019). We define the task in a comparison format. Specifically, each compar- ison pair contains two objects with weight gap greater than 1 lbs. The threshold is set according the FeberrFechner laws (Fechner et al., 1966) to guarantee that the mass difference is perceivable for humans. We build 654 triplets such as (hair dryer, heavier than, red Lego brick) for evaluation.
Recent studies have shown that prompting methods that fit the pre-training paradigm are more effective than other possible prompting methods (Petroni et al., 2019; Schick and Schütze, 2021 Following these studies, we design specific prompts for LMs with different objectives.
Prompting Masked Language Models Follow- ing PET (Schick and Schütze, 2021a,b), we probe the masked language models by converting knowl- edge facts into a question-answeringg form. For example, size knowledge fact (coin, smaller than, table) is converted into sentence with a special mask token: Question: is a coin smaller than a table? Answer: [MASK]. We also explored other prompts, such as Is a coin [MASK] than table. However, our experiments show that a question-answeriig form can better induce models to generate answers and avoid the influence of tokenization of different LMs. Given masked inputs, the model is asked to predict the probabilities of the mask token over two choices,
Temperature Dataset We design a temperature probing dataset by collecting the temperature of 22 common objects from Wikipedia. For exam- ple, the ice is C,C, and the temperature of water
hhttps: htttps://en.wikipedda.. www.wwwiiiiia...mm wttp:/.iiiiaaa.rr/iiiiiiiie wwt..riii//rddrr..oofr magni Sagnitude_(temeeee (temperrrteeerrrurrrr
thttpss//en.wikipddda.... .rr wrg/wiki./i/oooooooo wowsssscale_aeooo
mineral
mineral_hardness hardness

---

No
Question: the water colder than the frying oil?
(MASK].
BERT
Yes: 13.1%
Probabilty Distribution of [MASK]
5; The water colder than the frying oil.
Perplexity 5, 87.22
OPT
Syi The water hotter than the frying oll.
Perplexity of Sy: 69.19
og This is photo of the water.
catk02, 0.64
CLIP Text Encoder
Attribute: photo object.
og: This photo of frying
Figure 1: An illustration of prompting methods. For BERT-like models with a masked language head, we convert the knowledge fact a question and perform prediction with the head over yes or no. For OPT models, we evaluate the perplexity of different assertions and take the one with lower perplexity as valid fact. For CLIP, we devise matching- -based probing framework.
i.e., yes for confirming the knowledge fact is valid for unrcasonable assertion. We observe that in specific LMs. the prediction toward some answers as investigated by Zhao et al. (2021). We calibrate the prediction by normalizing the probabilitics according to an estimated prior following Zhao et al. (2021).
sentence representation without any pre-trained lan- guage heads. To probe the learned knowledge in VLMs such CLIP, we design matching-based prompting method. more detail, for the size fact stated before, we first obtaín two object de- scriptions 01 photo of a coin, and 02 photo of table. These two sentences are encoded to get the corresponding object vectors via the CLIP language encoder:
be biased
Prompting Causal Language Models Differ- ent from BERT, there is no special [MASK] to- ken in causal language models like GPT (Radford al., 2019). Therefore, introducing special to- ken would result an inconsistency between pre- training and evaluation. To remedy this, for each knowledge fact, we state it in natural sentences ac- cording to prompting templates and evaluate the sentence perplexity as the proxy metric. Specif- ically, for size-property evaluation, we convert it into valid knowledge assertion s1 A coin is smaller than table, and an invalid by replacing the relation with the antonym adjective s2 = A coin is larger than table. The sentence with lower perplexity is then chosen as the predicted one. We evaluate the perplexity of each sentence s = (wo, uf,,
01 02 CLIP(o1), CLIP
We then derive an attribute sentence a = a photo of small object, and encode it to an attribute adjective vector with the language encoder:
CL.
The prediction then performed by comparing the cosine similarity cos(o), a) and cos(o2, a).* The object with higher similarity with the attribute de- scription is adopted as the answer, i.e., a coin is smaller than table, if cos(o1, a) > cos( a). Otherwise, assume that the model thinks the reversed relation holds. We can also adopt the antonym adjective large for getting the attribute vectors. The results of the best-performing adjec- tive words for CLIP are reported and we discuss the influence of adjective options $ 4.3.
Wp) as:
(s = PMM))
PM (wx wa.w.. wk-1)
where PM denotes the conditional word probability the causal language model to be probed and n is the number of tokens in s. We compare the perplexity PPL(s1) and (P2() and choose the sentence with lower PPL as more valid assertion and calculate the prediction accuracy accordingly.
Experiments
4.1 Experimental Settings
We cover two kinds of LMs, text-only LMs and visual-augmented LMs. Text-only LMs include BERT-base/large (Devlin et al., 2019),
Models
Prompting Vision-augmented Language Models CLIP Unlike text-only LMs that support word predictions, the text encoder CLIP only has one
"The matching- prompting also applies to the pooled embedding of BERT, yet the results exhibit great variance as shown in Appendix A.

---

Model (ff of Param.)
Color
Shape
Size
Height
Material
Average
BERT (63M)
56.05
53.21 1-79
10.34 1.
50.1611.0
55.35
53.02
OPT yiC (333)
65.21 115.27
51.25 18.00
50.50
49.96 11.36
81.41 11.50
59.67
CCC (63M)
68.21
67.21 47.68
62.64 22.64.....
54.04
62.92
63.00
BERT-base (110M)
49.29 11.80
52,
4..222
49. L.0
50.56 1:0.
48.08 L:2
50.00
BERT-large (340M)
49.36.11.8
51,21 1:5
49.26
49.08
1.233
49.72
49.73
RoBERTa-base (125M)
49.07 #1.02
49.36
50.32
49.58 4:0.49
49,86 1:444
49.64
RoBERTa-large (355M)
49.66+0.54
50.68 +1.48
50.54
50.14
50.00
50.20
OPT (125M)
70.02.19.50
37.32.16.00 46
45.98 1:4.
56.76.1 1:1
82,43 1:220
62.50
OPT (1.38)
76.92 76.92...97
65.00 1..0
51.12 1:2 .6
57.82 1:4
85.63 1:3.99
67.30
OPT (13B)
79.62 +5.22
62,50++.444
57.56 (566-6.60
54.58 4-4.6a
88.38 4-314
68.53
OPT (175B)
83.10 13.13
65.7147..54
59.18 19.05
55.84
85,49 1:2.01
69.87
LLaMa-1 (7B)
63.94114.87
66.19
16..1912.30 1:236
65.91
1:9 86
50.00.1...00 00
66.76.13.
62.56
Vieuna-vI. (7B)
64.31 40044
73.33 4:2.88
62.50 1: 800.80
50.02-to.11
68.3
+:375
63.69
LLaMa-1 (13B)
66.27 +3.89
62.38 42236
63.14 11..00
50.16
65.46++299
61.48
Vicuna-v1.3 (13B)
66. +0.00
67.38 k:2.99
64.35
#-3..2
50.92+2.00 +2.0a
68.52+o.50 +0.00
63.46
LLaMM-22 (7B)
63.73 1:309
65.24 1.4.2
61.88 19.37
50.02 06
66.34 L33
61.44
LLaMa-2-Chad (7B)
60.99 +5.18
70.95
4-277G
68.03 +3.993.
51.72++2.00
67.39 +-4.18
63.82
LLaMa-2 (13B)
66.59 +3.40
62.38 23.21
68.20
+: 11.51
50.10 1:0.88
66.73 2:3.
62.80
LLaMa-2-Chat (13B)
64.04444.41
70.71 1.77
70.68 +8.79
51.18 +1.61
67.96+44.6
64.91
(63M)
80.07 #2.07
84,43 422.77
61.4 61.40 Li
62.28 4-6.40
80.07 #2.57
73.94
(63M)
81.48 1:44+2..00
84.07 07:+2.34
76.92=1.81
68.12
81.48
78.35
CLIP-ViT/L-14 (123M)
80.33 13:61
85,00
63.96
60.72
80.33
74.21
BLIP-base (138M)
82.60
84,86
76.00=a.40
69.84
19.84.2..0
80.67 1.22
78.79
Table 2: Zero-shot probing results visual datasets. Models with the YFCC-15M subscript represents that these models are trained from scratch on YFCC-15M data. Scaling OPT-family brings clear improvements on size and color datasets. The scaling law fails on the height dataset.
RoBERTa-base/laree (Liu et al., 2019b) for masked language models, and OPT models with parame- ters ranging from 125M to I 175B. We further in- corporate recent variants of causal language mod- els into evaluation, including LLaMA-1/2 (7B and 13B) (Touvron et al., 2023a), Vicuna models (7B and 13B, v1.3) (Chiang et al., 2023) trained with the instruction tuning dataset, and LLaMa- 2 Chat models (7B and 13B) (Touvron et al., 2023b) trained with supervised fine-tuning and RLHF (Ouyang et al., 2022). For VLMs, include the text encoders of CLIP-ViT-L/14 (Radford et al., 2021) as base and large version, respectively. We also include an enhanced VLM with masked language mod- eling as self-supervision, DeCLIP-ViT-B/32 (Li et al., 2022b) and BLIP, a boosted VLM by uni- fying multi-mode modal understanding and generation tasks (Li et al., 2022a).5 Since directly compar- ing the VLMMs and text-only LMs be unfair due to the difference in model configuration and training corpus (Tan and Bansal, 2020), we re- re-train CLIP, BERT, and GPT from scratch with a similar Transformer model on the same text corpus, the
caption dataset in the YFCC-15M dataset (Thomee et al.. 2016). All models are trained for 32 epochs. The only difference between these models is the pre-training objective. Detailed model and training settings are elaborated in Appendix B.
Prompts We manually write several prompts (at least 4 prompts for each task) to eliminate the side- effect of the expression variations and report the averaged accuracy. Besides, the variance across different prompts could also serve indicator to evaluate the robustness of learned knowledge facts. We report the averaged performance over multiple prompts for all models. All used prompts can be found in Appendix C.
and
4.2 Main Findings
The ability of certain visual concepts emerges as scaling up LMs, but there are still basic vi- sual concepts where the scaling law fails. The evaluation results on visual datasets are shown in Table 2. Interestingly, with the scaling up of fam-la models, the prediction accuracy in- creases obviously on specific visual concepts such as color and size. On material and color, the largest OPT-175B model even achieves better results than VLMs of
'https: / /htg:///ggggggaaa ace.. cwtSaaesoooe.e//////cooeooc Alip-itm-le happ-tt-basee-oo base-coco coco
which are augmented

---

Model Param.)
Mass
Temperature
Hardness
Average
50.73
19.50.1.
50.91 1.04
50.38
GPT mPC (63M)
50.02 10.00
57.73 a.2.2.220
50.04 12.98
52.61
CLIPycc-sss(63M)
67.45+4..10
64.83
62.22
64.83
BERT-base (110M)
50.35 10.00
49.67 1:0.00
50.20
50.07
BERT-large (340M)
49.97.11.5
49.83 3:000.0000
49.98 1:0.00
49.93
RoBERTa-base (125M)
49.65 40.01
50.00
48. 48.04 1:200
49,23
RoBERTa-larg (355M)
50.08
50.07
49.95
50.03
OPT (125M)
50.00.0.000
54,53 14433
2.:2.45
50.23
OPT (1.3B)
50.05 10.10
50.90 j.5. us
53.03 1:2.
51.33
OPT (13B)
50, 14-40.6
51.85 46.34
52.38 4-3.09
51.46
OPT (175B)
50.21 20.24
59.83 a.a
57.33 4/3.01
55.79
LLaMa-1 (7B)
54.88 1:240
60.69 1:4 34
51.97 1:2.84
55.84
Vicuna-v1.3 (7B)
54.23 4:1.78
58.85 4.5.4-4.300
54.42 2.422.--.00
55.83
LLaMa-1 (13B)
53.69 53.69+3.81 +3.81
50.76 8.660
33.94+-4.46
52.80
Vicuna-v1.3 (13B)
56,90 +-300
53.32
55.50
55.24
aa (7B)
54.01 1:4.47
56.87
55.22
55.37
LLaMa-2-Cher (7B)
52.51 ++4.88
61.99.
55.65 4-5.28
56.72
LLaMa-2 (13B)
53.38 12.10
57.54
53.01 4:4.77
54.64
LLaMa-2-Chat (13B)
54.13 +2.78
56.68 +6002
54.12++4.64
54.98
CLIP- -2 (63M)
65,20 4-470
60,28
59.43
61.64
DeCLIP-ViT/B-32 VIT/B-32 (63M)
54.95
68.58 4.
61.10+4.14
61.54
CLIP-VIT/L-14 (123M)
73.15
65.88
69.57 1:226
69.53
BLIP-base (138M)
83.94
74.98 5:5.00
56.93 36.93+5.50
71.95
Table Zero-shot results embodied datasets. LMs struggle understand embodied knowledge, including OPT (175B) and visual-augmented LMs, with 71.95 as the best average performance.
Model
Mass
Temperature
Hardness
Avg.
couta
69.57.g0.0
76.16
Mess
BERT-bas
64.72
55.62
s1.80
57.38
Probing Task
BERT large
aa
52.73 g
57.46
RoBtRTa-base
60.24:5.04
60.27 44
30.44.1.18
56.98
Hardners
RoBERTa-larg 61.18
58.28
50.56
56.67
Table 4: The few-shot results of BERT variants. With 16 instances, the fine-tuned BERT variants are still worse than zero-shot risual-augment LMs.
Temperature
Prediction Accuracy (96)
Figure 2: Few-shot results of OPT-175B with in- stances demonstration embodied tasks.
with vision supervision and are supposed to per-
form better (Zhang et al., 2022a; Liu et al., 2022b). A potential reason is that the combination of color and material frequently occurs (e.g., red apples) in raw texts, and these co-occurrence statisties well captured by large LMs. The significant per- formance improvements after training on visual- grounded text corpus YF5C-15M validate this ex- planation. Besides, 1PT-13B and LLaMa-1 13B models excel in different visual concepts, with OPT-13B performing well on material concepts and LLaMa-1 13B relative size comparisons, likely due to the difference of pre-training corpus distri- bution. However, increasing LMs 175B brings negligible improvements in the Height dataset, indi- cating that there still remain visual concepts where the scaling law does not hold even though these concepts can be easily captured by humans.
LMs exhibit poor understanding of embod- ied concepts. As shown in Table 3, the scaling law fails again on the embodied concepts, as all LMs, including OPT-175B and variants trained with captions data, perform poorly. Among LMs, the LLaMa series shows a better performance embodied concepts, yet still reaches a plateau of around 55% overall accuracy. We further conduct few-shot prompt evaluation for OPT models by constructing the inputs with A 16 randomly sam- pled instances and adopt PET (Schick and Schütze, 2021a) for masked language models. The results are illustrated Figure and Table 4, respectively. We find that while the performance is boosted, the average results are still worse than the CLI--VIT/L-

---

100
100
Prediction Accuracy (8%)
Material Hardiness
80
Count
60
40
60
50 Color
Shape
Size
Helgght Materia
Mass
Temp.
Hardness
Task
0.2
0.3
0.6 0.7
1.0
Correct Ratio
H4000 P00
(PT-1746
DACELP A/12
Comparison between the best-performing models and human annotators on sampled subsets of VEC. The pest-performing LMs and VLMs achieve close-to-human results on visual datasets, yet far lag behind humans embodied datasets.
Figure 3:
Figure Entity correct ratio histograms of Mass and Material datasets across different prompts. BERT could make consistent correct predictions for specific entities, and the bell curve on hardness indicates chal- lenging for BERT to understand embodied concepts.
Instruction tuning enhances proficiency in both visual and embodied concepts. After post- training with the instruction tuning dataset, Vicuna models display enhanced proficiency both visual and embodied concepts, with larger LLMs demon- strating more significant improvement. For in- stance, when using LLaMa-1 (13B) baseline model, the average accuracy in three embodied tasks rises from 52.8 to 55.2. Moreover, LLaMa- 2-Chat models, which are further trained with a supervised instruction tuning dataset and RLHF techniques, show consistent accuracy gains in both visual and embodied concept tasks as well. How- ever, disentangling the influence of instruction tun- ing and RLHF on these models presents a challenge as they are intertwined. Nevertheless, a clear per- formance gap still remains between more recent LMs and VLMs, indicating the significance of vi- sual supervision.
14 model without any demonstration, which only utilizes 0.08% parameters of OPT-175B. These findings show that visual supervision can help learn embodied knowledge, but there is still large gap between the best results of existing LMs with hu- man performance.
Compared with human annotators, OPT-175B and VLMs achieve competitive performance re- garding visual concepts, yet they exhibit great gaps with humans on embodied concepts. We conduct a human evaluation to better understand the performance of different models. Specifically, we randomly sample 100 examples for each task and ask three volunteers label these examples. The annotators achieve substantial agreements all the tasks with Cohen's kappa (Cohen, 1960) larger than 0.7, except for the Hardness dataset with a moderate к = 0.52. The comparison with best- performing models, i.e., OPT- 175B, CLIP-VIT/L. 14 and DeCLIP is illustrated in Figure 3. We find that (i) Regarding visual concepts, both OPT and CLIP-like models perform closely to human an- notators. CLIP and DeCLIP even outperform the human annotators on the shape task, which is poten- tially due the noise introduced by the automatic construction of the dataset (Zhang et al., 2022a). Overall, the close-to-human results indicate that visual knowledge can be effectively acquired by large-scale cross-modal pre- treinii or even text- only pre-training with sufficient parameters. (ii) Regarding embodied concepts, the best-performing CLIP-V1T-L/14 model still has absolute 18.5% accuracy gap with the humans. The clear perfor- mance gaps reveal that there is still a long way to go in equipping LMs and VLMs with embodied knowledge.
4.3 Analysis
Does BERT behave similarly regarding visual and embodied concepts? The overall prediction results of BERT-like models in the visual and em- bodied world are both a random level. We inves- tigate this question result by first checking whether BERT models perform consistently level for all the entities in the dataset. We compute the entity correct ratio among different prompts for the objects in different datasets and compare the distribution on different tasks with the BERT model trained YFCC-15M dataset. As illustrated in Figure in the Material identification task, there are entities that the model that could provide consis- tent correct predictions. However, the distribution of the Hardness dataset in embodied evaluation ex- hibits a bell curve, i.e., most entities are predicted
guessing

---

Probabillty of gijject
Proba of Heavy Objec
a St 2t
photo clyeet. 59 (sid. 2.00%)
shoto of light weight ouject 35.63% (std) 2.68%)
0.
85
30c
Plephant:
4015
Stoni Toy
20r1
20:3
Clip Feather 1e-4
A phote of hard object. 64.00% (std: 1.20%)
A photto of hoavy DOjeCt 6b 20% jutdl 4.75%)
Figure 5: Case study showing that the image represen- tations in CLIP exhibit embodied knowledge. (Left) The probability an image being classified as "hot" increases the ice melts being heated in a boiler in a video. (Right) The probability of an image being classified as "heavy" along with corresponding mass annotation.
Top-5 retrieved images and the prediction accuracy with different attribute prompts. The accuracy drops when the text inputs contain ambiguous words and compound words, the retrieved images are biased toward specific meanings.
Figure 6:
ever, our preliminary study in Appendix F shows that vanilla alignment on the predicted word distri- butions could not be effective. Inspired by our case study showing that the rich embodied knowledge contained in the representations, we utilize Neu- ron Selectivity Transfer (Huang and Wang, 2017) which transfers the inner states such as spatial acti- vation patterns of teacher neurons to student neu- rons, by aligning the token representations of the last layer between the teacher and student language models, which is implemented as a squared maxi- mum mean discrepancy (MMD) with polynomial kernel to measure the distance between the activa- tion patterns of student neurons and teacher neu- rons. The total training objective of the language model is a combination of the original language modeling loss and the MMD loss with balanc- ing coefficient. We refer readers to Appendix E for more details. As shown in Table the dis- tillation provides a performance boost on embod- ied concepts understanding, e.g., learning from a CLIP-ViT-LA teacher model achieves improve- ment that is comparable with that brought by scal- ing the model parameter 134x from OPT-1.3B to OPT-175B. It validates our assumption and in- dicates that future studies could utilize the richer representations in VLMs for improving LMs, yet the gap between distilled LM and VLM suggests that there is still room for advancement.
correctly at a random-chance level. The distribu- tions of other tasks show similar results and can be found in Appendix D. These results suggest that BERT learns visual knowledge for certain entities yet indeed struggles regarding embodied concepts.
Exploring learned embodied knowledge in im- age representations. We are interested in how the VLMs of CLIP learn embodied knowledge. A potential answer is that the images contain rich se- mantics regarding embodied knowledge such as the heat of the object, and the knowledge can be prop- agated to the VLMs via the contrastive learning ob- jective. To examine this, we perform a study by calculating the attribute similarities among the images. We first take clips from video of heating a pile of ice and then perform binary classifi- cation by calculating the cosine similarities with text prompts a photo of a hot object. and a photo of a cold object for each frame. The left of Fig- ure shows that the probability of hot object increases during the heating procedure. Similarly, we perform binary classification over heavy and light- objects ranging from an elephant to a feather. The right of Figure 5 shows that the image representations are aware of the mass of dif- ferent objects. This qualitative study shows that image representations are the potential source of embodied knowledge.
VLMs perform poorly when dealing with am- biguous text descriptions. During our prelimi- nary study, observe that VLMs of CLIP perform relatively poorly for specific adjectives such as hard. To further investigate this issue, we examine the retrieved images using prompts with different attribute adjectives on the CC12M dataset (Chang-
Transferring embodied knowledge from VLMS to LMs. We further verify whether the learned embodied knowledge in CLIP could be transferred text-only models. Specifically, we perform knowledge distillation (Hinton al., 2015) by treat- ing the original text-only language model as stu- dent, and the CLIP text encoder as a teacher model providing the learned embodied knowledge. How-
"Only OPT is adopted for experiments as the CLII encoder cannot deal with the mask tokens introduced in BERT.

---

pinyo et al., 2021). Our results, illustrated in Fig- ure 6, revealed that for the prompt a photo of a hard object, the retrieved images were mostly about ab- stract and difficult learning materials, with only one rock image related to the attribute of hardness. Additionally, for the prompt light-weight, the re- trieved images are biased towards meanings related to lighting bulbs and light-toned colors. These ob- servations demonstrate that handling semantic biguity remains a challenge for VLMs (Ren al., 2023), suggesting that future improvements may incorporate more language-side supervision into the text encoder VLMs (Li et al., 2022b).
Model
Mass
Temperature
Hordness
Avg.
CLIP-VITR-52 (E1)
65.
00.28.40:8 ss
-9.48-2.00
61
CLU--VITI-1441))
73.15
66, (5.88
69.57.2.26
69.53
OPT-L.3B
50.05 10
40.90
53.03
51.33
scale up to 13B
50.14
$1.85
52.38
$1.46
(+0.13)
scale up to
50.21
27.38
$5.79
(44.46)
50,02
57.73
50.04
52.61
Distill Distill w/ T2
+4.01
53.23
$2.96 (+0.35)
54.27
60.78
52.91
55.99 (+3,35)
Table 5: Results of embodied distillation. Transferring embodied knowledge from CLIP-ViT to OPT brings a gain of 3.38 points, which is comparable with the improvements by scaling the model from 1.3B to 175B.
learning transferable image representations with large-scale image-text pairs, triggering a series of variants for further improvements (Jia et al., 2021; Li et al., 2022b; Yao et al., 2022; Li et al., 2021, 2022a). Our study uses VLMs of CLIP and BLIP to investigate the impact of visual supervision on understanding physical concepts and our results suggest that visual supervision is crucial for LMs to understand embodied concepts, which can be utilized to enhance the text-ony LMs.
5 Related Work
Probing Language Models Understanding what LMs know after large-scale pre-training is an active research area (Rogers et al., 2020). Various prob- ing methods have been developed (Tenney et al., 2019b; Petroni et al., 2019), and investigations show that LMs capture linguistic (Tenney et al., 2019a; Liu et al., 2019a), factual (Petroni et al..
2019; Roberts et al., 2020; Dai al., 2022), com- monsense knowledge (Wang al., 2019; Forbes et al., 2019), and even acquire grounded con- cepts (Patel and Pavlick, 2022). For VLMs, stud- ies demonstrate their potential in acquiring spa- tial commonsense (Zhang et al., 2022a; Liu et al., 2022a; Xia et al., 2023) and color perception (Ab- dou et al., 2021), yet performing worse on NLU tasks (Tan and Bansal, 2020) and achieving sig- nificant on lexical grounding (Yun al., 2021). In this paper, we investigate the ability of LMs to understand physical concepts. Different from PIQA (Bisk et al., 2020b) consists of questions re- quiring physical commonsense reasoning, our VEC benchmark examines the understanding ability of the fundamental physical concepts. The evaluation on the VEC benchmark demonstrates that text-only LMs can learn specific visual concepts after scaling up while struggling with the embodied concepts.
Conclusion
In this puper, we introduce VEC for evaluating the understanding of physical concepts in LMs. Our results show that large LMs understand specific visual concepts but struggle with embodied knowl- edge. VLMs instead perform much better in both the visual and the embodied world, indicating that visual signals are vital for understanding hicai concepts. Further analysis suggests that transfer- ring the VLM representations to LMs effectively boosts embodied concepts understanding, shedding light on directions for improving LMs.
Limitations
Limited Scopes of Physical Concepts In this work, we focus on evaluating certain physical prop- erties such as color, mass, temperature, and hard- ness. These properties are chosen because they can be measured using well-established metrics and are easily sensed by humans. However, this selection introduces bias in our approximation of embodied knowledge. Despite this bias, our results are still sufficient to demonstrate the poor performance of current text-only language models (LMs) in under- standing embodied concepts. We suggest that in- corporating vision supervision could help improve the understanding of embodicd concepts. Addi- tionally, our current benchmark only examines the
Vision- Languaee Pre-training Unifying cross- modal representations via vision-language pre- training has achieved promising progress. Pilot studies adopt masked reconstruction to learn shared representations across modalities from mixed vi- sual and language inputs (Li et al., 2019; Tan and Bansal, 2019; Su et al., 2020; Chen al., 2019; al., 2020). CLIP (Radford et al., 2021) in- troduces a contrastive language-iman ingguege-mmae pre-training framework, utilizing language as supervision for

---

fine-grained understanding ability of specific phys- ical concepts, while neglecting the more complex physical understanding that involves multiple in- teractions or observations within a single example. Developing a dataset that encompasses composi- tional physical concepts holds promise for future research.
Michael Ahn, Anthony Brohan, Noah Brown, Yevgen Chebotar, Omar Cortes, Byron David, Chelsea Finn, Keerthana Gopalakrishnan, Karol Hausman, Alex Herzog, et al. 2022. Do i can, not say: Ground- ing language in robotic affordances. ArXiv preprint. abs/2204.01699
Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang. Junyang Lin, Chang Zhou, and Jingren Zhou. 2023, Qwen-vl: A frontier large vision-language model with versatile abilities. ArXiv preprint, abs/2308. 20s62
Limited Adoption of VLMs While there are many multi-modal models available, we restrict our investigation to visual-linguistic models (VLMs) based on CLIP and its variants. We choose CLIP for its superior image representation perfor- mance and support for text-only encoders. Since our evaluation focuses on language-oriented tasks, we require models that can handle inputs con- sisting of pure text. Consequently, VLMs like UNITER (Chen et al., 2019), which require multi- modal inputs, are not considered. CLIP is selected as a representative work of VLMs for evaluation. However, it is important to note that the findings from CLIP may not readily generalize to other V+L models, as CLIP utilizes large dataset of million-level image-text pairs collected from the web, which could be significant source of em- bodied knowledge itself. Furthermore, there have been recent proposals for VLM models with vari- ous architectures and pre-training recipes, such as VLM (Wang et al., 2022), UniT (Hu and Singh, 2021), ViLT (Kim et al., 2021), and FLAVA (Singh et al., 2022), as well as vision-enhancel multi- modal agents like InstructBLIP (Dai et al., 2023), Qwen- VL (Bai et al., 2023), and Ying-VLM VLM (Li al., 2023). These models have shown promis- ing performance in both cross-modal and single- modality tasks, and we look forward to evaluating these advanced models in our benchmark in the future.
Yonatan Bisk, Ari Holtzman, Jesse Thomason, Jacob Andreas, Yoshua Bengio, Joyce Chai, Mirella Lap- ata, Angeliki Lazaridou, Jonathan May. Aleksandr Nisnevich, Nicolas Pinto, and Joseph Turian. 2020a. Experience grounds language. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 5718-8735
Yonatan Bisk, Rowan Zellers. Ronan LeBras. Jianfeng Gao, and Yejin Choi. 2020b. PIQA: reasoning about physical commonsense natural language. In The Thirty-Fouu Fourt--Fourthao AAAI Conference on Artificial Intelli- gence, AAAI 2020, The Thirty-Second Innovative Ap- plications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2020, New York, NY, USA, February 7-12, 2020, pages 7432- 7439.
Paul Bloom. 2002. How children learn the meanings of words.
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin. Scott Gray, Benjamin Chess, Jack Clark, Christopher Bemnrr Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners. In Ad- vances in Neural Information Processing Systems 33: Annual Conference on Neural Information Process- ing Systems 2020, NeurIPS 2020, December 12, 2020, virtual.
Acknowledgments
We thank all the anonymous reviewers for their constructive comments, Yuxuan Fan, Shuhuai Ren and Sijie Cheng for their valuable suggestions in preparing the manuscript.
Soravit Changpinyo, Piyush Sharma, Nan Ding, and Radu Soricut. 2021. Conceptual 12m: Pushing web- scale image-text pre-training recognize long-tail visual concepts. In IEEE Conference on Computer Vision and Puttern Recognition, CVPR 2021, virtual, June 19-25, 2021, pages 3558-3568.
References
Yen-Chun Chen, Linjie Li, Licheng Yu, Ahmed El Kholy, Faisal Ahmed, Zhe Gan, Yu Cheng, and Jingjing Liu. 2019. Uniter: Learning universal image- text representations. ArXiv.
Mostafa Abdou, Artur Kulmizev, Daniel Hershcovich, Stella Frank, Ellie Pavlick, and Anders Søgaard. 2021. Can language models encode perceptual struc- ture without grounding? case study in color. In Pro- ceedings of the 25th Conference on Computational Natural Language Learning, pages 109-132.
Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng. Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan

---

Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, lon Stoica, and Eric P. Xing. 2023. Vicuna: An open- source chatbot impressing gpt-4 with 90%* chatgpt quality.
Yaru Hao, Haoyu Song, Li Dong, Shaohan Huang. Zewen Chi, Wenhui Wang, Shuming Ma. and Furu Wei. 2022. Language models are general-purpose interfaces. ArXiv preprint, 20.22226600636..
Aakanksha Chowdhery, Sharan Narang. Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung. Charles Sutton, Sebastian Gehrmann, et al. 2022. Palm: Scaling language modeling with pathways. ArXiv preprint, abs/2204.02311.
Stevan Harnad. 1990. The symbol grounding problem. Physica D: Nonlinear Phenomena.
Geoffrey Hinton, Oriol Vinyals, Jeff Dean, et al. 2015. Distilling the knowledge in neural network. ArXiv preprint, Abss/5503.0253.1
Jacob Cohen. 1960. coefficient agreement for nominal scales. Educational and psychological mea- surement, 20(1):37-44 -6..
Ronghang Hu and Amanpreet Singh. 2021. Unit: Mul- timodal multitask learning with unified transformer. In 2021 IEEE/CVF International Conference on Com- puter Vision, ICCV 2021, Montreal, QC, Canada, October 10-17, 2021, pages 1419-1429.
Damai Dai, Dong, Yaru Hao, Zhifang Sui, Baobao Chang, and Furu Wei. 2022. Knowledge neurons in pretrained transformers. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 8493 8502.
Zehao Huang and Naiyan Wang. 2017. Like what you like: Knowledge distill via neuron selectivity transfer. ArXiv preprint, abs/1777.01219.
Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi. 2023. In- structblip: Towards general-purpose vision-language models with instruction tuning. ArXiv preprint, abs/2305.06500.
Shankar Iyer, Nikhil Dandekar, and Kornél Csemai. 2017. Quora question pairs. First Quora Dataset Release: Question Pairs.
Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc V. Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. 2021. Scaling up vi- sual and vision-language representation learning with noisy text supervision. In Proceedings of the 38th In- ternational Conference on Machine Learning, ICML 2021, 18 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Research, pages 4904-4916.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language under- standing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Tech- nologies, Volume I (Long and Short Papers), pages 4171 1186.
Lynette Jones. 2009, Thermal touch. Scholarpedia.
Danny Driess, Fei Xia, Mehdi S. M. Sajjadi, Corey Lynch, Aakanksha Chowdhery, Brian Ichter, Ayzaan Wahid, Jonathan Tompson, Quan Vuong, Tianhe Yu, Wenlong Huang, Yevgen Chebotar, Pierre Ser- manet, Daniel Duckworth, Sergey Levine, Vincent Vanhoucke, Karol Hausman, Marc Toussaint, Klaus Greff, Andy Zeng. Igor Mordatch, and Pete Florence. 2023. Palm-e: An embodied multimodal language model. volume abs/2303.00 003378..
Wonjae Kim, Bokyung Son, and IIdoo Kim. 2021. Vilt: Vision-and labion-and-langusae -anguae transformer without convolu- tion or region supervision. In Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Research, pages 5583-5594.
Junnan Li. Dongxu Li, Caiming Xiong, and Steven C. H. Hoi. 2022a. BLIPP hootstrapping language-image pre-training for unified vision-language understand- ing and generation. In International Conference on Machine Learning, ICML 2022, 17-23 July 2022, Bal- timore, Maryland, USA, volume 162 of Proceedings of Machine Learning Research, pages 12888 -12900.
Gustav Theodor Fechner, Davis H Howes, and Ed- win Garrigues Boring. 1966. Elements of psy- chophysics, volume 1.
Maxwell Forbes, Ari Holtzman, and Yejin Choi. 2019, Do neural language representations learn physical commonsense? CogSci.
Junnan Li, Ramprasaath R. Selvaraju, Akhilesh Gotmare, Shafiq Joty. Caiming Xiong. and Steven Chu-Hong Hoi. 2021. Align before fuse: Vision and language representation learning with momentum distillation. In Advances in Neural in- formation Processing Systems 34: Annual Confer- ence Neural Information Processing Systems 2021, NeurIPS 2021, December 6-14, 2021, viriual, pages 9694-9705.
Alison Gopnik, Andrew N Meltzoff, and Patricia K Kuhl. 1999. The scientist in the crib: Minds, brains, and how children learn.
David Gueorguiev, Séréna Bochereau, André Mouraux, Vincent Hayward, and Jean-Louis Thonnard. 2016. Touch uses frictional cues to discriminate flat materi- als. Scientific reports.

---

Lei Li, Yuwei Yin, Shicheng Li, Liang Chen, Peiyi Wang, Shuhuai Ren, Mukai Li, Yazheng Yang, Jingjing Xu, Xu Sun, Lingpeng Kong, and Qi Liu. 2023. MT large-scale dataset towards multi- modal multilingual instruction tuning. ArXiv preprint, abs/2306.0438)
2022. Training language models to follow instrue- tions with human feedback. Advances in Neural Information Processing Systems, 35:27730-27744,
Roma Patel and Ellie Pavlick. 2022. Mapping language models to grounded conceptual spaces. In The Tenth International Conference on Learning Representa- tions, ICLR 2022, Virtual Event, April 25.29, 2022.
Liunian Harold Li, Mark Yatskar, Da Yin, Cho-Jui Hsieh, and Kai-Wei Wei Chang. 2019. Visualbert: A sim- ple and performant baseline for vision and language. ArXiv.
Fabio Petroni, Tim Rocktäschel, Sebastian Riedel. Patrick Lewis, Anton Bakhtin, Yuxiang Wu, and Alexander Miller. 2019. Language models as knowl- edge bases? In Proceedings of the 2019 Confer- ence on Empirical Methods in Natural Language Pro- cessing and the 9th International Joint Conference on Natural Language Processing WEMNL--JCNNLL pages 2463-2473.
Xiujun Li. Xi Yin, Chunyuan Li. Xiaowei Hu, Pengchuan Zhang, Lei Zhang, Lijuan Wang, Houdong Hu, Li Dong, Furu Wei, Yejin Choi, and Jianfeng Gao. 2020. Oscar: Object-semantics aligned pre-training for vision-language tasks. In Proc. of ECCV.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sas- try, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. 2021. Learn- ing transferable visual models from natural language supervision. In Proceedings of the 38th International Conference Machine Learning, JCML 2021, 18-21 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Research, pages 8748-8763.
Yangguang Li, Feng Liang, Lichen Zhao, Yufeng Cui, Wanli Ouyang, Jing Shao, Fengwei Yu, and Jun- jie Yan. 20226. Supervision exists everywhere: A data efficient contrastive language-iman image pre-training paradigm. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022.
Nelson F. Liu, Matt Gardner, Yonatan Belinkov, Matthew E. Peters, and Noah A. Smith. 2019a. Lin- guistic knowledge and transferability of contextual representations. In Proceedings of the 2019 Confer- ence of the North American Chapter of the Associ- ation for Computational Linguistics: Human Lan- guage Technologies, Volume 1 (Long and Short Pa- pers). pages 1073-1094.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019. Language models are unsupervised multitask learners. OpenAl blog.
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Perey Liang. 2016. SQuAD: 100,000+ questions for machine comprehension of text. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2383-2392.
Xiao Liu, Da Yin, Yansong Feng, and Dongyan Zhao. 2022a. Things not written in text: Exploring spatial commonsense from visual signals. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). pages 2376..
Shuhuai Ren, Lei Li, Xuancheng Ren, Guangxiang Zhao, and Xu Sun. 2023. Delving into the open- ness of CLIP. In Findings of the Association for Computational Linguisticx: ACI. 2023, pages 9587- 9606.
Xiao Liu, Da Yin. Yansong Feng, and Dongyan Zhao, 2022b. Things not written in text: Exploring spatial commonseuse from visual signals. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers). pages
Adam Roberts, Colin Raffel, and Noam Shazeer. 2020. How much knowledge can you pack into the param- eters of a language model? In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 5418-5426.
Anna Rogers, Olga Kovaleva, and Anna Rumshisky. 2020. A primer in BERTology: What we know about how BERT works. Transactions of the Association for Computational Linguistics, 8:842-866.
Yinhan Liu, Myle Oit, Naman Goyal, Jingfei Du, Man- dar Joshi, Danqi Chen, Omer Levy, Mike Lewis. Luke Zettlemoyer, and Veselin Stoyanov. 2019b. RoBERTa: A robustly optimized BERT pretraining approach. ArXiv preprint.
Timo Schick and Hinrich Schütze. 2021a. Exploiting cloze-questions for few-shot text classification and natural language inference. In Proceedings of the 16th Conference the
Brielen Madureira. 2021. Flamingos and hedgehogs in the croquet-ground: Teaching evaluation of NLP systems for undergraduate students. In Proceedings of the Fifth Workshop on Teaching NIP. pages 87-91.
the European Chapter of the Asso- ciation for Computational Linguistics: Main Volume, pages 259.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al.
Timo Schick and Hinrich Schütze. 202 It's not just size that matters: Small language models are also few- shot learners. In Proceedings of the 2021 Conference

---

of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2339-2352.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023a. Llama: Open and effi- cient foundation language models. ArXiv preprint, abs/2302.1397
Amanpreet Singh, Ronghang Hu, Vedanuj Goswami, Guillaume Couairon, Wojciech Galuba, Marcus Rohrbach, and Douwe Kiela. 2022. Flava: A founda- tional language and vision alignment model. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages
Hugo Touvron, Louis Martin, Kevin Stone, Peter Al- bert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023b. Llama 2: Open founda- tion and fine-tuned chat models. ArXiv preprint. abs/2307. 002888988.
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang. Christopher D. Manning, Andrew Ng, and Christopher Potts. 2013. Recursive deep models for semantic compositionality over sentiment treebank. In Proceedings of the 2013 Conference on Empiri- cal Methods in Natural Language Processing, pages 1631-1642.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Pro- cessing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pages 5998-60090 9988-008.
Trevor Standley, Ozan Sener, Dawn Chen, and Silvio Savarese. 2017. image2mass: Estimating the mass of an object from its image. In Proceedings of the 1st Annual Conference on Robot Learning.
Eric Wallace, Yizhong Wang. Sujian Li, Sameer Singh, and Matt Gardner. 2019. Do NLP models know num- bers? probing numeracy in embeddings. In Proceed- ings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th Inter- national Joint Conference on Natural Language Pro- cessing (EMNLP-IJCNLP), pages 5307-5315.
Weijie Su, Xizhou Zhu, Yue Cao, Bin Li, Lewei Lu, Furu Wei, and Jifeng Dai. 2020. VL-BERT: pre- training of generic visual-linguistic representations. In 8th International Conference on Learning Repre- sentations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020.
Cunxiang Wang. Shuailong Liang, Yue Zhang, Xiaonan Li, and Tian Gao. 2019. Does it make sense? and why? a pilot study for sense making and explana- tion. In Proceedings of the 57th Anmual Meeting of the Association for Computational Linguistics, pages 4020-4026.
Hao Tan and Mohit Bansal. 2019. LXMERT: Learning cross-modality encoder representations from trans- formers. In Proceedings of the 2019 Conference on Empirical Methods Natural Language Processing and the 9th International Joint Conference on Natu- ral Language Processing (EMNLP-JJCNLP), pages 5100-5111.
Zirui Wang, Jiahui Yu, Adams Wei Yu, Zihang Dai. Yu- li Tsvetkov, and Yuan Cao. 2022. Simvlm: Simple visual language model pretraining with weak super- vision. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022.
Hao Tan and Mohit Bansal. 2020. Vokenization: Im- proving language understanding with contextualized, sssual-roouned supervision. Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2066-2080.
Tan Tenney, Dipanjan Das, and Ellie Pavlick. 2019a. BERT rediscovers the classical NLP pipeline. In Proceedings of the 57th Annual Meeting of the Asso- ciation for Computational Linguistics. pages 4593- 4601.
Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph. Sebastian Borgeaud. Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, Ed H. Chi, Tatsunori Hashimoto, Oriol Vinyals, Percy Liang, Jeff Dean, and William Fedus. 2022. Emer- gent abilities of large language models. Transactions on Machine Learning Research.
lan Tenney, Patrick Xia, Berlin Chen, Alex Wang. Adam Poliak, R. Thomas McCoy, Najoung Kim, Benjamin Van Durme, Samuel R. Bowman, Dipan- jan Das, and Ellie Pavlick. 2019b. What do you learn from context? probing for sentence structure 7th Inter-
Adina Williams, Nikita Nangia, and Samuel Bowman. 2018. A broad-coverage challenge corpus for sen- tence understanding through inference. In Proceed- ings of the 2018 Conference of the North American Chapter of the Association for Computational Lin- guistics: Human Language Technologies, Volume I (Long Papers), pages 1112-1122.
contextualized word representations. national Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019.
Bart Thomee, David A Shamma, Gerald Friedland, Ben- jamin Elizalde, Karl Ni, Douglas Poland, Damian Borth, and Li-Jia Li. 2016. 10cm:00m: The new data in multimedia research. Communications of the ACM, 59(2)66473.
Heming Xia, Qingxiu Dong, Lei Li, Jingjing Xu, Ziwei Qin, and Zhifang Sui. 2023. Imagenetve: Zero-shot visual commonsense evaluation on 1000 imagenet categories. arXiv preprint 23Xiv:230S.55023

---

Lewei Yao. Runhui Huang, Lu Hou, Guansong Lu. Minzhe Niu, Hang Xu, Xiaodan Liang, Zhenguo Li, Xin Jiang, and Chunjing Xu. 2022. FILIP: fine- grained interactive language-image pre-training. In The Tenth International Conference on Learning Rep- resentations, ICLR 2022, Virtual Event, April 25-29, 2022.
with the YFCC-15M dataset, they are adopted the same LM architecture as the VLM of CLIP-ViT/B- 32, and the only difference is the pre-training ob- jective, as shown in Table 7. All these models are optimized using an Adam optimizer with a learning rate set to 1c-4, linearly increased at the first 2000 steps. The batch size is 2018 and all models are trained with 32 epochs. We use 1% of the data for evaluation, and the final OPT-YFCC15M model gets 31.9 validation perplexity.
Tian Yun, Chen Sun, and Ellie Pavlick. 2021. Does uusion-and-languaa pretraining improve lexical grounding? In Findings of the Association for Com- putational Linguistics: EMNLP 2021, pages 4357- 4366.
C Details of Prompts
Chenyu Zhang, Benjamin Van Durme, Zhuowan Li, and Filias Stengel-Fsskin. 2022a. Visual commonsense pretrained unimodal and multimodal models. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Compula- tional Linguistics: Human Language Technologies, pages 5321-5335.
We provide the used prompts for evaluating differ- ent models based on their pre-training objectives. Examples of Head, Rel and Tail of each dataset shown in Table 1. Due to the sensitivity of language models to prompts, we provide diverse prompts for each model on each task.
Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher De- wan, Mona Diab, Xian Li, Xi Victoria Lin, Todor Mi- haylov, Myle Ott, Sam Shleifer, Kurt Shuster, Daniel Simig, Punit Singh Koura, Anjali Sridhar, Tianlu Wang, and Luke Zettlemoyer. 2022b, Opt: Open pre-trained transformer language models.
Prompts for Masked Language Models A [MASK] token is placed in the prompt and the mod- els are asked to predict the probabilities of the [MASK] token. To avoid multiple mask tokens in prompts, we follow Schick and Schütze to convert knowledge fact into cloze-question. For exam- ple, temperature fact (water, colder than, frying oil) can be converted into Q: is the water colder than frying oil? A: [MASK]!. The models need to choose the token yes or no to fill the mask.
Zihao Zhao, Eric Wallace, Shi Feng, Dan Klein, and Sameer Singh. 2021. Calibrate before use: Improv- ing few-shot performance of language models. In Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Vir- tual Event, volume 139 of Proceedings of Machine Learning Research, pages 12697-12706.
Appendix
Prompts for Causal Language Models As there is no special [MASK] token during the pre-training of causal language models, we do not use [MASK] tokens in prompts for causal language models. For Color, Shape and Material datasets of the vi- sual concepts we construct two prompts for (Head, Tail;) and (Head, Taily); while for other datasets, we construct two prompts for (Head, Rel, Tail) and (Head, Rel', Tail) where Rel' is the antonym rela- tion of Rel. The prediction is based on the prompt with lower perplexity.
A Matching- Prompting Results for BERT
We perform the identical prompting method and templates with CLIP VLMs for LMs of BERT. RoBERTa is discarded as there is no pooled em- bedding during its pre-traine pre-training for representing the sentence. As shown in Table 9, the performance is still close to random guessing baseline. Besides, the higher variances on all the tasks compared to VLMs of CLIP indicate that this method does not LMs of BERT well, validating our prompting de- sign choice in the main paper to fit the pre-training paradigm.
Prompts for CLIP Following Radford et al. (2021), we use prompts like a photo of here. As the language encoder of CLIP encodes sentences to vector and can evaluate similarities between sentences. We use an attribute prompt like photo of cold object and construct same prompts for objects (water and frying oil) in the knowledge fact. We can determine the colder object if the prompt of this object has a higher similarity to the attribute prompt.
B Model Configurations
Here we provide the detailed configurations of eval- uated LMs and VLMs in our main paper. For the vanilla model, there model configurations are listed in Table For the models pre-trained from scratch

---

Model
Hidden Layers
Hidden Size
Attention Heads
Total Parameters
BERT-base
768
110M
BERT-large
24
1,024
16
340M
RoBERTa-base
768
125M
RoBERTa-large
24
1,024
16
355M
OPT-125M
12
768
12
125M
OPT-1.3B
2,048
1.3B
OPT-13B
40
5,120
40
13B
OPT-175B
12,288
175B
CIJP-ViT//B32
12
512
63M
DeCLIP-ViT/B-32
12
512
63M
CLIP-ViT/L14
768
12
123M
BL.I-hhase
12
768
12
138M
Table 6: Detailed configuration of models evaluated in the paper.
Model
Training Objective
Training Dataset
BERTYFCC-1N
Masked Language Modeling (MLM)
Captions in YFCC-15M
1SC
Causal Language Modeling (MLM)
Captions in YFCC-15M
CLIP CLCPYYCC-ISM
Contrastive Image-text Matching (CIM)
Image- Text Pair in YFCC-15M
Table Pre-training objectives and corpus comparison of YFCC- 15M models evaluated in the main paper.
Model
SST-2
QQP
QNLI
MNLI mm)
Avg.
BERT (Wiki)
90.13
83,20
87.57
78.90/ 80.05
83.97
DistilledOscar
89.33
67.98
82.48
74.46 / 74.82
77.81
VLM-BERT-buse
90.60
90.10
89.47
81.57 / 82,43
86.83
BLM--RoBKKTa-baee
90.13
88.44
87.91
80.37 / 80.43
85.46
Model
Color
Shape
Size
Height
Material
Mass
Temperature
Hardness
Avg.
BERT (Wiki)
49.41
48.07
51.70
49.46
52.39
48.85
$1.07
52.34
50.41
DistilledOscar
49.97
53.61
49.07
49.80
51.46
$1.22
47.94
51.23
50.54
LLM--ERR--aal base
50.69
50.07
51.00
50.92
53.89
44.83
50.64
49.22
50.16
49.53
51.21
49.00
49.22
49.54
49.92
51.11
49.63
49.90
Table 8: Fine-tuned accuracy of other visual-informed pre-trained language models on NLU tasks and zero-shot results regarding the physical concepts.
Model
Color
Shape
Size
Height
Material
Mass
Temperature
Hardness
Avg.
VITP-VVT/L.-14
80.33
85.00
63.9626.10
60.72
80.33
73. 73.15
65. 85.88
69.57
72.37
Thise Pooled
43.19
49.64
66. 16.10-7a0
65.48 65.48_6.08
46.55
52.32
56.59
55.67 55.51 45.50
BERT-large Pooled
44.74
56.93
53.80 5.00
54.84 :8.01
52.18 1..88
57.92.7.9
51.90.a.5
56.22
53.57
Table Zero-shot results of BERT models with pooled output sentence embedding VEC benchmark.
Entity Analysis
curves, indicating the poor understanding ability BERT on embodied concepts.
In our main paper, we investigate the random-level performance of BERT models by exploring the cor- rect ratio over different prompts. We provide full histogram plots of all tasks in Figure 7, 8, 9,10, 11, 12, 13, and 14. It be found that for visual concepts tasks such material and shape, there entities that BERT could produce consistent cor- rect prediction across different prompts. However, for all embodied tasks, the histograms exhibit bell
E Embodied Knowledge Transfer
We provide implementation details here for the knowledge transfer experiments from VLMs LMs. Specifically,
take the VLM teacher model (e.g., the text encoder of the CLIP model) and the LM as a student model (c.g., the OPT language model). Given a text x from the training

---

dataset D, we transfer the sequential activation pat- terns of (x) Rlaixd S(x) Rla|xd where T(x) and (x) denote the last hidden representa- tions of the VLM and the LM, respectively. is the number of hidden units. The squared maximum mean discrepancy (MMD) with kernel trick (Huang and Wang, 2017) is adopted to measure the distance between the activation patterns:
Size
Count
0.50
Correct Ratio
MMD (x)
[S(x) S(x)a8]
d2
Figure 7: Histogram of entity correct ratio across differ- ent prompts on the Size dataset.
1=
d
h
d2
Shape
j= ø²=1
50
Count
50-
We adopt a polynomial kernel k(x:y) (xTy + c) with p = and c The MMD objective LMDD is minimized along with the origi- nal language modeling objective LLM:
0.0
0.2
0.4
0.5
0.8
1.0
Correct Ratio
L - Lim + 3LMMD
Figure 8: Histogram of entity correct ratio across differ- ent prompts on the Shape dataset.
where 3 weighting factor set to to achieve a balance between objectives.
Color
F Evaluation and Distillation with Oscar
We examine whether other vanilla distillation from traditional V+L pre-training methods brings gains regarding visual and embodied knowledge. Specifi- cally, following Zhang al. (2022a), we distill the knowledge of Osear (Li et al., 2020) into BERT model by performing knowledge distillation (Hin- ton et al., 2015) on the image-caption pair dataset. Specifically, the paired text and image are fed into the Oscar model for getting the vision-aware vo- cabulary distribution, and student BERT model is performing masked language modeling on the text data only and learns from the soft labels pro- vided by the Oscar teacher model. The distillation results in a DistilledOscar model supporting text- only inputs. We also evaluate VLM-BERT learned via Vokenziation (Tan and Bansal, 2020), which devises a fine-grained token-voken matching frame- work to utilize visual supervision. The models evaluated the four largest datasets in GLUE, including SST-2 (Socher al., 2013), QQP (lyer al., 2017), QNLI (Rajpurkar et al., 2016) and MNLI (Williams et al., 2018) for stable results. As shown in Table 8, DistilledOscar performs worse
Count
20-
0.0
0.2
0.4
0.6
0.8
2.0
Correct Ratio
Figure 9: Histogram of entity correct ratio across differ- ent prompts on the Color dataset.
than the vanilla BERT in both NLU tasks and prob- ing tasks regarding visual and embodied knowl- edge. Besides, while VLM-BERT achieves im- provements NLU tasks, it still performs the random level on the probed tasks. These indicate that not all VLMs could learn embodied knowledge and it is non-trivia to distill the visual supervision from VLMs to LMs via purely language modeling.

---

Table 10: Prompts for Masked Language Models
Model
Task
Prompt
is the [Hedd]] [Rel] than the [Tail]? [MASK]]
[Head] [Rel] the [Tail]] [MASK].
is [Head)] [Rel] than [Tail]? [MASK]]
[Head] [Rel] than [Tail]? [MASK].
is [Head]] [Rel]] compared with ["Tai]]? [MASK].
Size, Height, Temperature, Weight, Hardness
is (Head] [Rei] compared with ["aiil]] [MASK]!
compared with [TTail), [[eead]
[MASK].
compared with [TRail], is [Head]] [Rel]] [MASK]!
(Head) usually [Rel] rhan [Tail]? [MASK].
is [[lead] usually [Rel] than [Tail?? [MASK]]
can [Headd] be color [Tail]? [MASK]!
[Head] be color [Tail]? [MASK].
[Head] [Taiil] [MASK]
is the color a [[lead] [Tail]? [MASK].
[Hlead]
IMASK].
Color
is [Heud] [Tail]? [MASK]]
is (Head) typically (Tbil] [MASK]
is [[lead] typically [Tail? IMASK]!
Q: [Head] of color [TRil]] [MASK]
Question: [Head] color [Thil]]
[MASK].
[Head] be the shope of [Tail]] [MASK].
can [[lead] be the shape of [Tail]? [MASK]]
have shape [Tall
BERT ROBERE
does the [Head] have a shape [Tail]? (MASK]!
is (Head) [Tail]] (MASK).
Shape
is [[lead] [Taill? [MASK]]
Q: [Head] of [Tail]? A: [MASK]
Question: [Head] [Tail]?
[MASK].
[Tail] [Elead]? [MASK].
(Head) typically [Tbill] [MASK].
be made [Tail?? IMASK!!
can [Head] be made [Tail]? [MASK].
[[lead] made of [Tail? IMASK!!
is [Head] made [Tait]] [MASK].
Material
is [Tail]] the nocesssry material for making [Head]] [MASK].
necessary
for making [Head]? [MASK]
does [Hleod] consist of [Tai]] (MASK).
(Hend) made ['ailly [MASK].
Q: [Head]] made [Tail]? A: [MASK].
Question: is [Head] made [Rai]] Answer: [MASK].
Height
Material
10
140
120
100
6-
Count
0
Correct Ratio
Correct Ratio
Figure 10: Histogram of entity correct ratio across dif- ferent prompts on the Height dataset.
Figure Histogram entity correct ratio across dif- ferent prompts the Material dataset.

---

Table 11: Prompts for Causal Language Models
Model
Tack
Prompt
the [fledd] [Rel] than the [Tail].
[Head] [Rel] than [Tail]]
acutally, the [3eadd] is [Rel] than the [Tail).
acutally, [Heud] is [Rel] than [Tail].
il well-know that [Head] is [Rel] thmm [Tail]]
Size, Height, Temperature, Weight, Hardness
[Head] indeed [Rel] than [Tail]
[[lead]
[Rell]
compared with the [Head], the [Tail]] [Rel].
al(an) (Head] is [Rel] rhan allan) [Tail]]
yes. [[lead] is [Rel] than [Tail].
[Head] can be the color (Tail).
[Head] can be of color (Tai).
a(an) [Head] [Tail]
the color of [Head]] is [Tail).
[[eadd]
Color
[Head] is [Tail]]
what color the [Head]] [Tail]-
[Hedd''s color is [Taill.
usually. [Head] [Tail].
OPT
(Head) typically [Tail).
[Head] usually [Tail].
what the shape of [llaad] [Tail].
Shape
[Head] typically [Tail]]
(Head)s is shape is [Tail].
[Head] made of [Tail].
[[eadd] made of [Tail].
[Head] consists of [Tail].
msin material [Head] is [Tall.
necessary material making [Head].
Material
the [eedd consists of [Tail]
[Head]]
[Tal]].
the [Head] built with [Tail].
[elad contains [Tail].
[Head] made up [Tai]]
Table 12: Prompts used for VLMs of CLIP.
Model
Tisk
Prompt
photo (leadd/[Attribttt)
plutoo of the [Heud]]] Auribuie]
blurry
[Headd[AAribbee))
good photo of [Head//i Auribbttee
paintig of [Head/]A Atribute).
photo
(trriuut
close photo of Antiiue/.
bright photo the [[triii/)trrbbuee]]]] photo of one [Headj]] |l Atrrbute].
low resolution
Atributt)
Temperature
Hardness
10-
8
6-
Count
Count
2.
0.40
0.45
0.50
0.35
0.50
0.425
0.525
0.550
0.575
Correct Ratio
Correct Ratio
Figure 12: Histogram of entity correct ratio across dif- ferent prompts on the Temperature dataset.
Figure 13: Histogram of entity correct ratio across dif- ferent prompts the Hardness dataset.

---

Mass
17.5
15.0-
12.5
10.0
755
5.-
0.40
0.45
0.50
0.35
0.00
0.06
0.70
Correct Ratio
Figure 14: Histogram of entity correct ratio across dif- ferent prompts on the Mass dataset.


Reviews:
[{"review_id": "WWg7LVHlgF", "reviewer": "Reviewer_zxLC", "summary": "", "questions": "The questions are mostly the same as the weakness of this paper. I think addressing these weaknesses will help increase the impact of this paper.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We deeply appreciate your insightful comments and the recognition you have given to our work. Your positive assessment of the VEC benchmark, experimental design, findings, and knowledge distillation exploration is truly inspiring. We are grateful for your valuable feedback, and we have provided detailed responses to your comments below. \n\n\n## Evaluation results with LLaMa-1/2, Vicuna and LLaMa-2-Chat\n\nThanks for acknowledging the results obtained from the OPT-175B model, and we are happy to incorporate more results with recent LLMs.\n\nWe compare the performance of LLaMA 1/2 models (7B and 13B), Vicuna models (7B, 13B, v1.3) trained with the instruction tuning dataset, and LLaMa-2 Chat models (7B and 13B) trained with SFT and RLHF, using the same setup as OPT models, as they are all casual language models.\n\n**Results on Visual Datasets**\n\n| Model           | Color           | Shape          | Size            | Height         | Material       | Average |\n| --------------- | --------------- | -------------- | --------------- | -------------- | -------------- | ------- |\n| LLaMa-1-7B     | 63.94  $\\pm$4.87   | 66.19  $\\pm$2.36  | 65.91  $\\pm$9.86   | 50.00  $\\pm$0.00  | 66.76  $\\pm$3.88  | 62.56   |\n| Vicuna-7B-v1.3  | 64.31  $\\pm$5.44   | 73.33  $\\pm$2.88  | 62.50  $\\pm$8.80   | 50.02  $\\pm$0.11  | 68.31  $\\pm$3.75  | 63.69   |\n| LLaMa-1-13B   | 66.27  $\\pm$3.89   | 62.38  $\\pm$2.36  | 63.14 $\\pm$11.06   | 50.16  $\\pm$0.43  | 65.46  $\\pm$2.95  | 61.48   |\n| Vicuna-13B-v1.3 | 66.11  $\\pm$5.62   | 67.38  $\\pm$2.69  | 64.35  $\\pm$13.62  | 50.92  $\\pm$2.53  | 68.52  $\\pm$5.60  | 63.46   |\n| LLaMa-2-7B     | 63.73 $\\pm$ 3.09 | 65.24 $\\pm$ 4.22 | 61.88 $\\pm$ 9.37 | 50.02 $\\pm$ 0.06 | 66.34 $\\pm$ 3.25 | 61.44 |\n| LLaMa-2-7B Chat | 60.99 $\\pm$ 5.18 | 70.95 $\\pm$ 2.76 | 68.03  $\\pm$ 9.91 | 51.72  $\\pm$ 2.20 | 67.39  $\\pm$ 4.13 | 63.82 |\n| LLaMa-2-13B     | 66.59 $\\pm$  3.40 | 62.38 $\\pm$ 3.21 | 68.20  $\\pm$ 11.51 | 50.10  $\\pm$ 0.18 | 66.73 $\\pm$   3.99 | 62.80 |\n| LLaMa-2-13B Chat | 64.04 $\\pm$ 4.41 | 70.71$\\pm$ 1.75 | 70.68 $\\pm$  8.79 | 51.18 $\\pm$  1.61 | 67.96  $\\pm$ 4.63 | 64.91 |\n| OPT-13B | 79.62  $\\pm$5.28 | 62.50 $\\pm$6.44 | 57.56 $\\pm$6.60 | 54.58 $\\pm$4.53 | 88.38 $\\pm$3.14 | 68.53 |\n| BLIP | 82.60 $\\pm$5.50 | 84.86 $\\pm$2.80 | 76.00 $\\pm$6.40 | 69.84 $\\pm$7.76 | 80.67  $\\pm$1.24 | 78.79 |\n\n\n\n**Results on Embodied Datasets**\n\n| Model           | Mass           | Temperature    | Hardness       | Average |\n| --------------- | -------------- | -------------- | -------------- | ------- |\n| LLaMa-1-7B      | 54.88  $\\pm$2.49  | 60.69  $\\pm$4.35  | 51.97  $\\pm$2.84  | 55.84   |\n| Vicuna-7B-v1.3  | 54.23  $\\pm$1.78  | 58.85  $\\pm$4.36  | 54.42  $\\pm$6.42  | 55.83 |\n| LLaMa-1-13B   | 53.69  $\\pm$3.81  | 50.76  $\\pm$8.69  | 53.94  $\\pm$4.45  | 52.80 |\n| Vicuna-13B-v1.3 | 56.90  $\\pm$3.53  | 53.32  $\\pm$6.47  | 55.50  $\\pm$5.73  | 55.24   |\n| LLaMa-2 7B       | 54.01 $\\pm$4.47  | 56.87  $\\pm$6.25  | 55.22   $\\pm$5.89  | 55.37   |\n| LLaMa-2 7B Chat  | 52.51 $\\pm$4.83  | 61.99  $\\pm$3.93  | 55.65  $\\pm$  5.28 | 56.72   |\n| LLaMa-2 13B | 53.38$\\pm$ 2.10  | 57.54  $\\pm$ 7.51 | 53.01 $\\pm$ 4.57   | 54.64   |\n| LLaMa-2 13B Chat | 54.13 $\\pm$2.73  | 56.68 $\\pm$ 6.02  | 54.12  $\\pm$ 4.64  | 54.98   |\n| OPT-13B | 50.14  $\\pm$0.36 | 51.85  $\\pm$6.34 | 52.38  $\\pm$3.09 | 51.46 |\n| BLIP | 83.94 $\\pm$ 2.59 | 74.98 $\\pm$ 5.60 | 56.93 $\\pm$ 5.56 | 71.95 |\n\n- Comparison of LLaMA series models with OPT-13B: Our observations reveal that **OPT-13B and LLaMa 13B models demonstrate proficiency in understanding distinct visual concepts**. OPT-13B shines in the material concept, whereas LLaMa 13B performs better in relative size comparison tasks. This discrepancy may stem from the distribution of the pre-training corpus, as evidenced by the noticeable accuracy improvement after training on the YFCC-15M visual-grounded text corpus. Furthermore, **LLaMa models exhibit better performance in embodied concepts**, manifesting an initial understanding of specific concepts, such as a ~60% accuracy rate on the temperature task of LLaMa-1-7B.\n\n- Impact of Instruction Tuning: Post-training with the instruction tuning dataset, **Vicuna models display enhanced proficiency in both visual and embodied concepts**, with larger LLMs demonstrating a more significant improvement. For instance, when using LLaMa 13B as a baseline model, the average accuracy in three embodied tasks rises from 52.8 to 55.2. This encouraging result suggests an alternative approach to enhancing embodied understanding.\n\n- Comparison of LLaMa-2 and LLaMa-2-Chat models: Chat models, further trained with the supervised instruction tuning dataset and RLHF techniques, consistently show accuracy improvements in both visual and embodied concept tasks. However, disentangling the influence of instruction tuning and RLHF on these models presents a challenge as they are intertwined.\n\n- Performance gap between instruction-tuned, RLHF LLMs and VLMs: **A clear performance disparity still exists between recent LLMs incorporating instruction tuning/RLHF and VLMs**. Nonetheless, we observed that in certain tasks, such as object hardness, LLaMa-2-7B performs comparably to the powerful BLIP VLM.\n\nWe will integrate these revised results to align better with the ongoing advancements in LLM research.\n\n\n### Details of Dataset Construction\n\n\nWe appreciate your insightful feedback. To address your suggestion, we are keen to expand our dataset details in the paper.\n\nIn terms of visual datasets, we've chosen the Size and Height tasks from the Spatial Commonsense [1] dataset. These tasks delve into the relative spatial relationships among 25 everyday objects. Additionally, we've included the Material, Color, and Shape tasks from the ViComTe dataset [2]. We primarily utilize the test split of each task to evaluate the LLMs.\n\nConcerning embodied datasets, we gather annotations for objects with attributes of interest from academic databases or Wikipedia. This approach yields 56, 22, and 25 different objects in the mass, temperature, and hardness datasets, respectively. Pairs of objects are formed by establishing a significant gap for the particular attribute.\n\nIn our revised version, we plan to provide these extended dataset details and offer more statistics, including the distribution of objects, to portray a more comprehensive picture of the dataset.\n\n\n\n### Developing Instances involving Compositional Physical Concepts\n\nWe appreciate your comment regarding the potential development of a more complex embodied dataset that incorporates multiple physical interactions or observations within a single example. \n\nOur research primarily aims to examine the fine-grained understanding ability of specific embodied concepts, such as mass and temperature, which we observed large language models struggle with.  \n\nWe also agree that developing a dataset encompassing compositional physical concepts very interesting and look forward to exploring this avenue in future research. \n\n\n\n[1] Things not Written in Text: Exploring Spatial Commonsense from Visual Signals, ACL 2022\n\n[2] Visual Commonsense in Pretrained Unimodal and Multimodal Models, NAACL 2022", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The paper introduces a novel benchmark evaluation suite (VEC) that covers a wide range of physical concepts, including both visual and embodied attributes. This benchmark fills a gap in the field by providing a standardized framework to evaluate language models' understanding of physical attributes, which is essential for advancing research in this area.\n\n2. The paper conducts a thorough evaluation of various types of language models, including text-only LMs like BERT and VL models like CLIP and BLIP. \n\n3. This direction is very relevant as next step for grounded LMs or LLMs.\n\n4. The paper addresses many intriguing properties of LMs including its physical ability or understanding of embodiment ab. It is interesting to see that *\"the ability of certain visual concepts emerges as scaling up LMs, but there are still basic visual concepts where the scaling law fails\"*, which gives strong motivation for grounded LMs.", "reasons_to_reject": "1. The paper might have benefited from testing on open-source LLMs such as llama or instruction tuned models, but this could be attributed to the rapidly evolving landscape of LLMs and the timeline of the experiments of this paper. Given the dynamic nature of this field, including open-source LLMs could have enhanced the paper's relevance to the community and ensured its alignment with the latest developments. Still, it is worth acknowledging the largest model scale used is OPT(175B).\n2. The paper can benefit from allocating more paragraphs to how they develop the dataset, which seems to be the core contribution of this paper. Moreover, it is intuitively ideal to develop a more complex embodied dataset in which a single example may involve a series of embodied or physical interaction/observation. Current paper usually addresses single embodied attribute like mass/temperature.", "questions_to_authors": "The questions are mostly the same as the weakness of this paper. I think addressing these weaknesses will help increase the impact of this paper.", "paper_topic_and_main_contributions": "This paper examines the capacity of language models (LMs) to comprehend physical concepts in the real world. The authors create the VEC benchmark, encompassing visual and embodied attributes. Visual concepts encompass properties like color, shape, and material, while embodied concepts involve mass, temperature, and hardness. The study evaluates different LMs, including text-based ones like BERT and GPT (OPT)-family, as well as vision-augmented models like CLIP and BLIP.\n\nThe paper's contributions are as follows:\n\n**VEC Benchmark:** Introducing the VEC benchmark enables the assessment of LMs' grasp of physical attributes, covering both visual and embodied concepts.\n\n**Understanding with Scaling:** The findings reveal that as LMs scale, they gain better understanding of certain visual concepts. However, understanding embodied knowledge remains a challenge, despite performing better than random guessing. This underscores the need for advancing embodied concept comprehension.\n\n**Vision-Augmented LMs:** Vision-augmented models like CLIP and BLIP achieve human-level comprehension of embodied concepts, indicating the significance of visual data in understanding physical attributes.\n\n**Knowledge Distillation:** Inspired by vision-augmented LMs, a knowledge distillation method is proposed, transferring embodied knowledge to standard LMs. This yields performance gains comparable to extensive parameter scaling.\n\nIn summary, the paper investigates LMs' grasp of physical concepts, demonstrating improved comprehension of visual concepts with scaling. However, challenges persist in understanding embodied concepts. Vision-augmented LMs exhibit promise, and a knowledge distillation approach is proposed to bridge the gap between standard and vision-augmented LMs' comprehension of physical attributes.", "excitement": 4, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "No", "Typos_Grammar_Style_and_Presentation_Improvements": "Some minor typos but overall good presentation."}, "extra_text": {}}, {"review_id": "B1SXFNwzP5", "reviewer": "Reviewer_kY1c", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable feedback. We appreciate your recognition of our interesting topic, well-designed task format, and solid experimental results.\n\nIn response to your comment suggesting exploring recent LLMs, we would like to provide additional information regarding the evaluation results. We select LLaMa-1/2 (7B and 13B) as a representative of well-performing LLMs, Vicuna (v1.3, 7B and 13B) to explore the effect of instruction tuning, and LLaMa-2 Chat is also incorporated for exploring the impact of RLHF. We exclude commercial LLM APIs such as ChatGPT because of their intransparency and the potential issue of data contamination.\n\n\n**Results on Visual Datasets**\n\n| Model           | Color           | Shape          | Size            | Height         | Material       | Average |\n| --------------- | --------------- | -------------- | --------------- | -------------- | -------------- | ------- |\n| LLaMa-1-7B     | 63.94  $\\pm$4.87   | 66.19  $\\pm$2.36  | 65.91  $\\pm$9.86   | 50.00  $\\pm$0.00  | 66.76  $\\pm$3.88  | 62.56   |\n| Vicuna-7B-v1.3  | 64.31  $\\pm$5.44   | 73.33  $\\pm$2.88  | 62.50  $\\pm$8.80   | 50.02  $\\pm$0.11  | 68.31  $\\pm$3.75  | 63.69   |\n| LLaMa-1-13B   | 66.27  $\\pm$3.89   | 62.38  $\\pm$2.36  | 63.14 $\\pm$11.06   | 50.16  $\\pm$0.43  | 65.46  $\\pm$2.95  | 61.48   |\n| Vicuna-13B-v1.3 | 66.11  $\\pm$5.62   | 67.38  $\\pm$2.69  | 64.35  $\\pm$13.62  | 50.92  $\\pm$2.53  | 68.52  $\\pm$5.60  | 63.46   |\n| LLaMa-2-7B     | 63.73 $\\pm$ 3.09 | 65.24 $\\pm$ 4.22 | 61.88 $\\pm$ 9.37 | 50.02 $\\pm$ 0.06 | 66.34 $\\pm$ 3.25 | 61.44 |\n| LLaMa-2-7B Chat | 60.99 $\\pm$ 5.18 | 70.95 $\\pm$ 2.76 | 68.03  $\\pm$ 9.91 | 51.72  $\\pm$ 2.20 | 67.39  $\\pm$ 4.13 | 63.82 |\n| LLaMa-2-13B     | 66.59 $\\pm$  3.40 | 62.38 $\\pm$ 3.21 | 68.20  $\\pm$ 11.51 | 50.10  $\\pm$ 0.18 | 66.73 $\\pm$   3.99 | 62.80 |\n| LLaMa-2-13B Chat | 64.04 $\\pm$ 4.41 | 70.71$\\pm$ 1.75 | 70.68 $\\pm$  8.79 | 51.18 $\\pm$  1.61 | 67.96  $\\pm$ 4.63 | 64.91 |\n| OPT-13B | 79.62  $\\pm$5.28 | 62.50 $\\pm$6.44 | 57.56 $\\pm$6.60 | 54.58 $\\pm$4.53 | 88.38 $\\pm$3.14 | 68.53 |\n| BLIP | 82.60 $\\pm$5.50 | 84.86 $\\pm$2.80 | 76.00 $\\pm$6.40 | 69.84 $\\pm$7.76 | 80.67  $\\pm$1.24 | 78.79 |\n\n\n\n\n\n**Results on Embodied Datasets**\n\n| Model           | Mass           | Temperature    | Hardness       | Average |\n| --------------- | -------------- | -------------- | -------------- | ------- |\n| LLaMa-1-7B      | 54.88  $\\pm$2.49  | 60.69  $\\pm$4.35  | 51.97  $\\pm$2.84  | 55.84   |\n| Vicuna-7B-v1.3  | 54.23  $\\pm$1.78  | 58.85  $\\pm$4.36  | 54.42  $\\pm$6.42  | 55.83 |\n| LLaMa-1-13B   | 53.69  $\\pm$3.81  | 50.76  $\\pm$8.69  | 53.94  $\\pm$4.45  | 52.80 |\n| Vicuna-13B-v1.3 | 56.90  $\\pm$3.53  | 53.32  $\\pm$6.47  | 55.50  $\\pm$5.73  | 55.24   |\n| LLaMa-2 7B       | 54.01 $\\pm$4.47  | 56.87  $\\pm$6.25  | 55.22   $\\pm$5.89  | 55.37   |\n| LLaMa-2 7B Chat  | 52.51 $\\pm$4.83  | 61.99  $\\pm$3.93  | 55.65  $\\pm$  5.28 | 56.72   |\n| LLaMa-2 13B | 53.38$\\pm$ 2.10  | 57.54  $\\pm$ 7.51 | 53.01 $\\pm$ 4.57   | 54.64   |\n| LLaMa-2 13B Chat | 54.13 $\\pm$2.73  | 56.68 $\\pm$ 6.02  | 54.12  $\\pm$ 4.64  | 54.98   |\n| OPT-13B | 50.14  $\\pm$0.36 | 51.85  $\\pm$6.34 | 52.38  $\\pm$3.09 | 51.46 |\n| BLIP | 83.94 $\\pm$ 2.59 | 74.98 $\\pm$ 5.60 | 56.93 $\\pm$ 5.56 | 71.95 |\n\nOur key findings are:\n\n- **OPT-13B and LLaMa 13B models each excel in different visual concepts**, with OPT-13B performing well on material concepts and LLaMa 13B on relative size comparisons. This is likely due to the pre-training corpus distribution. LLaMa models also show better performance in embodied concepts, with ~60% accuracy on certain tasks.\n- **Instruction tuning improves the performance of Vicuna models in both visual and embodied concepts**, with more significant gains for larger LLMs. For instance, the average accuracy in three embodied tasks improves from 52.8 to 55.2 when using LLaMa 13B as a baseline.\n- LLaMa-2-Chat models, further **trained with a supervised instruction tuning dataset and RLHF techniques, show consistent accuracy gains** in both visual and embodied concept tasks. However, the individual effects of instruction tuning and RLHF are unclear as they are implemented together.\n- **A clear performance gap still remains between more recent instruction-tuned and RLHF LLMs, and VLMs. However, in certain tasks like object hardness, LLaMa-2-7B performs comparably to the powerful BLIP VLM**.\n\nWe will incorporate these updated results to provide a more aligned investigation with recent progress of LLMs.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Probing LM with physical concepts is a novel and interesting topic, it provides insights of how to further improve and align current LMs (especially LLMs) to real world.\n\n- The task format is well-designed, and the two kinds of concepts are defined clearly.\n\n- probing methods for different LMs and VLM are well-designed, and the experiments are solid.", "reasons_to_reject": "- The models used in this paper mainly are OPT and Encoder-only LMs, I think it would be more interesting and meaningful to explore more recent state-of-the-art LLMs such as ChatGPT and Llama, due to current LLMs are aligned with human instructions, they might can gain better or near to human performance.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper proposes an evaluation suite of physical concepts covering visual and embodied concepts. They probe text-only LMs and VLMs in this benchmark. They find that the understanding of certain visual concepts emerges as scaling up LMs, and vision-augmented LMs can gain better performance compared with text-only LMs. Based on this, they propose a distillation method to transfer embodied knowledge from VLMs to LMs to gain better performance.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "S5JvySdgeo", "reviewer": "Reviewer_MVNy", "summary": "", "questions": "Please see the sub-bullets in the Reasons to reject.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We deeply appreciate your thoughtful and constructive review. Your recognition of our benchmark's combination of existing and new datasets, as well as your acknowledgment of our solid experiments and clear findings, is greatly encouraging. We're also pleased that you found our paper well-written and easy to follow. Thank you for your time and valuable inputs, and the responses are provided below.\n\n## Evaluation Results with Recent LLMs\n\nWe supplement the results with LLaMa-1/2, Vicuna and LLaMa-2-Chat for a better understanding the scaling and the effect of SFT and RLHF. Commercial APIs such as ChatGPT and GPT-4 are excluded to avoid potential data contamination due to the intransparent usage of user data.\n\n\n**Results on Visual Datasets**\n\n| Model           | Color           | Shape          | Size            | Height         | Material       | Average |\n| --------------- | --------------- | -------------- | --------------- | -------------- | -------------- | ------- |\n| LLaMa-1-7B     | 63.94  $\\pm$4.87   | 66.19  $\\pm$2.36  | 65.91  $\\pm$9.86   | 50.00  $\\pm$0.00  | 66.76  $\\pm$3.88  | 62.56   |\n| Vicuna-7B-v1.3  | 64.31  $\\pm$5.44   | 73.33  $\\pm$2.88  | 62.50  $\\pm$8.80   | 50.02  $\\pm$0.11  | 68.31  $\\pm$3.75  | 63.69   |\n| LLaMa-1-13B   | 66.27  $\\pm$3.89   | 62.38  $\\pm$2.36  | 63.14 $\\pm$11.06   | 50.16  $\\pm$0.43  | 65.46  $\\pm$2.95  | 61.48   |\n| Vicuna-13B-v1.3 | 66.11  $\\pm$5.62   | 67.38  $\\pm$2.69  | 64.35  $\\pm$13.62  | 50.92  $\\pm$2.53  | 68.52  $\\pm$5.60  | 63.46   |\n| LLaMa-2-7B     | 63.73 $\\pm$ 3.09 | 65.24 $\\pm$ 4.22 | 61.88 $\\pm$ 9.37 | 50.02 $\\pm$ 0.06 | 66.34 $\\pm$ 3.25 | 61.44 |\n| LLaMa-2-7B Chat | 60.99 $\\pm$ 5.18 | 70.95 $\\pm$ 2.76 | 68.03  $\\pm$ 9.91 | 51.72  $\\pm$ 2.20 | 67.39  $\\pm$ 4.13 | 63.82 |\n| LLaMa-2-13B     | 66.59 $\\pm$  3.40 | 62.38 $\\pm$ 3.21 | 68.20  $\\pm$ 11.51 | 50.10  $\\pm$ 0.18 | 66.73 $\\pm$   3.99 | 62.80 |\n| LLaMa-2-13B Chat | 64.04 $\\pm$ 4.41 | 70.71$\\pm$ 1.75 | 70.68 $\\pm$  8.79 | 51.18 $\\pm$  1.61 | 67.96  $\\pm$ 4.63 | 64.91 |\n| OPT-13B | 79.62  $\\pm$5.28 | 62.50 $\\pm$6.44 | 57.56 $\\pm$6.60 | 54.58 $\\pm$4.53 | 88.38 $\\pm$3.14 | 68.53 |\n| BLIP | 82.60 $\\pm$5.50 | 84.86 $\\pm$2.80 | 76.00 $\\pm$6.40 | 69.84 $\\pm$7.76 | 80.67  $\\pm$1.24 | 78.79 |\n\n\n\n**Results on Embodied Datasets**\n\n| Model           | Mass           | Temperature    | Hardness       | Average |\n| --------------- | -------------- | -------------- | -------------- | ------- |\n| LLaMa-1-7B      | 54.88  $\\pm$2.49  | 60.69  $\\pm$4.35  | 51.97  $\\pm$2.84  | 55.84   |\n| Vicuna-7B-v1.3  | 54.23  $\\pm$1.78  | 58.85  $\\pm$4.36  | 54.42  $\\pm$6.42  | 55.83 |\n| LLaMa-1-13B   | 53.69  $\\pm$3.81  | 50.76  $\\pm$8.69  | 53.94  $\\pm$4.45  | 52.80 |\n| Vicuna-13B-v1.3 | 56.90  $\\pm$3.53  | 53.32  $\\pm$6.47  | 55.50  $\\pm$5.73  | 55.24   |\n| LLaMa-2 7B       | 54.01 $\\pm$4.47  | 56.87  $\\pm$6.25  | 55.22   $\\pm$5.89  | 55.37   |\n| LLaMa-2 7B Chat  | 52.51 $\\pm$4.83  | 61.99  $\\pm$3.93  | 55.65  $\\pm$  5.28 | 56.72   |\n| LLaMa-2 13B | 53.38$\\pm$ 2.10  | 57.54  $\\pm$ 7.51 | 53.01 $\\pm$ 4.57   | 54.64   |\n| LLaMa-2 13B Chat | 54.13 $\\pm$2.73  | 56.68 $\\pm$ 6.02  | 54.12  $\\pm$ 4.64  | 54.98   |\n| OPT-13B | 50.14  $\\pm$0.36 | 51.85  $\\pm$6.34 | 52.38  $\\pm$3.09 | 51.46 |\n| BLIP | 83.94 $\\pm$ 2.59 | 74.98 $\\pm$ 5.60 | 56.93 $\\pm$ 5.56 | 71.95 |\n\nBased on the updated results, we provide point-to-point response for the raised questions.\n\n**Question: Do the results hold for newer models?**\n\n**Response**:\n\nWe find that OPT-13B and LLaMa 13B excel in understanding different visual concepts, and LLaMa series models perform slightly better regarding the embodied concepts. However, there is still a clear performance gap between the text-only LLMs and VLMs (such as BLIP). Besides, there are specific tasks, such as relative height recognition, that model scaling does not help. These findings are consistent with our previous investigation with OPT-series models and show that pure-text models still struggle with physical concepts, especially for the embodied concepts.\n\n**Question: To what extent can these pure-text models understand the physical world?**\n\n**Response**: Based on our updated results, pure-text models demonstrate a commendable understanding of visual concepts, including aspects such as relative size comparison and material recognition. They also exhibit an introductory level of comprehension of embodied concepts like temperature, as evidenced by an approximate 60% accuracy rate in temperature understanding tasks.\n\n**Question: What causes the performance boost in these newer models if scaling is not the sole reason? (Could it be RLHF, and if so, why?)**\n\n**Response**: We have observed that Vicuna models, after undergoing instruction tuning with SFT data, exhibit enhanced performance in understanding physical concepts. This improvement is particularly noticeable in larger language models. For instance, when compared to the base LLaMa 13B model, the average embodied accuracy of Vicuna-13B increases from 52.80 to 55.24. This suggests that a combination of instruction tuning and model size scaling could contribute to the performance boost.\n\nAs for the role of RLHF, while LLaMa-2-chat consistently outperforms LLaMa-2, the gap is not substantial enough to conclusively attribute the improvement to RLHF. Given the absence of LLMs solely incorporating RLHF, we leave a more comprehensive investigation of RLHF's impact for future work.\n\n\n\n\n## Evaluation Setup of CLIP \n\n**Question: On the Evaluation Difference Between VLMs (i.e., CLIP) and PLMs (BERT, OPT)**\n\n**Response**: Yes, your understanding of the matching-based method is accurate. As outlined in lines 225-228 of our paper, all prompting methods are tailored to best fit the respective pre-training paradigms, a strategy recent studies have corroborated as effective. Given this, we argue that our comparison is fair, as these prompting methods all align with the pre-training objectives used to extract learned knowledge from the pre-training phase. Furthermore, we applied matching-based prompting methods to BERT's pooled embedding, as referenced in Appendix A and noted in the footnote on Page 4. Our findings indicate that employing a matching-based method for BERT yields marginally superior results but with greater variances across prompts. However, a significant performance gap between CLIP and BERT persists, aligning with our main paper's results.\n\n\n**Question: On the correlation between the cosine similarity and understanding of specific concepts**\n\n**Response**: Thank you for posing such as interesting question. To begin, it's key to emphasize that both CLIP and language models like BERT or GPT fundamentally operate as statistical models. They generate predictions based on the patterns they've learned from their training data, thereby mimicking a form of understanding that is more about recognizing these patterns than exhibiting true comprehension in the human sense. This understanding ability is demonstrated by co-occurrence relationships between objects. For instance, if `water` frequently co-occurs with `cold` and `frying oil` does so less often, indicated by the cosine similarities between sentences, the model might infer a relative temperature difference based on these associations. To verify this, we compute the text embeddings for `a photo of water` and ` a photo of frying oil` with the text encoder of CLIP-ViT-L/14, coupled with a set of temperature sentences describing `a photo of an object at X degrees Celsius`. Here, `X` represents a range of common temperatures including `[0, 10, 20, 50, 100, 200]`. We find that the sentence representation of the water photo most closely aligns with 10 degrees Celsius, while the sentence representation for the frying oil photo is closest to 50 degrees Celsius. This outcome suggests that the model uses co-occurrence patterns to infer a relative temperature difference between objects.\n\n  \n  \n\nWe hope these explanations clarify our evaluation setup and findings, and we would gladly incorporate this discussion in our revision.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The proposed benchmark is a good combination of previous word and newly collected datasets, covering both visual and embodied concepts;\n- The experiments are solid with clear findings and insights. The finding that we can effectively distill embodied knowledge from VLMs to less capable LLMs is quite inspiring;\n- The paper is well-written and easy-to-follow;", "reasons_to_reject": "- Lacking a comparison or case study on recent LLMs (models with instruction-following SFT or RLHF), such as ChatGPT, GPT-4, and Llama-family.\n\n  - Based on my own observations, when compared with GPT-3 era models (analyzed in this paper), newer models like ChatGPT contain much richer physical commonsense knowledge and can perform complex reasoning about physical interactions. As a result, some of the conclusions presented in this paper may not hold true for these newer models. A particular question that I expected to see in this paper was: to what extent can these pure-text models understand the physical world? Furthermore, what causes the performance boost in these newer models if scaling is not the sole reason? (Could it be RLHF, and if so, why?)\n\n- The evaluation setting for CLIP is quite different from BERT and OPT, which might be argued whether it is an apples-to-apples comparison.\n    - If I understand correctly, the matching-based method for CLIP essentially compares the level of correlation between an object and an attribute instead of directly comparing two objects.\n    - A smaller cosine score between \"This is a photo of the water\" and \"Attribute: This is a photo of a cold object\" versus \"This is a photo of frying oil\" can only indicate that \"water\" is more likely to be associated or co-occur with \"cold\" compared to \"oil\"; however, can we say that the model understands \"water\" is \"colder\" than \"frying oil\"?", "questions_to_authors": "Please see the sub-bullets in the Reasons to reject.", "paper_topic_and_main_contributions": "The paper investigated language models' understanding of physical concepts and designs a benchmark called VEC. The main contributions of this paper:\n\n- conducted in-depth analysis with different types of LLMs and VLMs on their understanding of visual concepts and embodied concepts;\n- provided insights on what kind of physical concepts are well-captured and struggled by LLMs; showing evidence that vision-language pretraining helps learn embodied knowledge\n- showed that distilling knowledge from VLM to LLM significantly boost performance on understanding embodied concepts", "excitement": 4, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a new evaluation benchmark for of the ability for language models to represent real-world physical concepts, including color, shape, material, mass, temperature, hardness, etc. The paper also performs thorough evaluation of existing text-only and vision-and-language models, including (in the rebuttal) updated results with more recent models that became available after the submission deadline. The paper finds that both scaling and knowledge distillation from vision-language to text-only models improve performance for visual concepts (but not embodied concepts), and that models that use visual features perform better than text-only models. The more recent models also perform better than those available at submission time.
Decision: Accept-Main