Visually-Augmented Language Modeling
Authors: Weizhi Wang
Li Dong
Hao Cheng
Haoyu Song
Xiaodong Liu
Xifeng Yan
Jianfeng Gao
Furu Wei
Venue: ICLR 2023 poster (2023)
Keywords: visually-grounded language modeling
visual commonsense reasoning
pre-trained visually-augmented language model

Abstract: Human language is grounded on multimodal knowledge including visual knowledge like colors, sizes, and shapes. However, current large-scale pre-trained language models rely on the text-only self-supervised training with massive text data, which precludes them from utilizing relevant visual information when necessary. To address this, we propose a novel pre-training framework, named VaLM, to Visually-augment text tokens with retrieved relevant images for Language Modeling. Specifically, VaLM builds on a novel latent text-image alignment method via an image retrieval module to fetch corresponding images given a textual context. With the visually-augmented context, VaLM uses a visual knowledge fusion layer to enable multimodal grounded language modeling by attending on both text context and visual knowledge in images. We evaluate VaLM on various visual knowledge intensive commonsense reasoning tasks, which require visual information to excel. The experimental results illustrate that VaLM outperforms all strong language-only and vision-language baselines with substantial gains on reasoning object commonsense including color, size, and shape.

Published as a conference paper at ICLR 2023
VISUALLY-AUGMENTE LANGUAGE MODELING
Weizhi Wang Li Dong', Hao Cheng', Haoyu Song', Xiaodong Liu', Xifeng Yan', Jianfeng Gao', Furu Wei University of California, Santa Barbara *Microsoft Research wei www.zwwwwguuus..eeee. edu, (lidongl, haocheng emicrosoft.com
ABSTRACT
Human language is grounded on multimodal knowledge including visual knowl- edge like colors, sizes, and shapes. However, current large-scale pre-trained language models rely on text-only self-supervised training with massive text data, which precludes them from utilizing relevant visual information when neces- sary. To address this, we propose novel pre-training framework, named VALM, auggeel text tokens with retrieved relevant images for Language Modeling. Specifically, VALM builds on a novel latent text-image alignment method via an image retrieval module fetch corresponding images given a tex- tual context. With the auggented context, VALM uses visual knowledge fusion layer to enable multimodal grounded language modeling by attending to both text context and visual knowledge in images. We evaluate VALM on various visual snowledge-intensive commonsense reasoning tasks, which require visual information excel. The experimental results illustrate that VALM outperforms all strong language-only and iision-langusal baselines with substantial gains in rea- soning object commonsense including color, size, and shape. Our code is available at htttts:://
INTRODUCTION
Large-scale pre- pre-trained language models (PLMs) have achieved great success in promoting state of the art on various natural language understanding and generation tasks (Devlin al. [019 Radford et al.) 2019 Liu et al. 2019: Yang et al. 2019, Brown et al. 2020 Wang et al. 2022). PLM self-supervision training largely benefits from harvesting local context information the pre-training corpus. To further strengthen such contextual supersuupe recent seminal works, e.g. GPT-3 (Brown et al.) 2020 and Megatron-LM (Narayanan et al., 202I), focus on increasing the model size and the scale of pre-training corpus. With billions of parameters, these tremendous PLMs exhibit incredible ability as zero-shot or few-shot learners. More remarkably, PLMs can achieve human-parii -ariiay--aatty performance on various downstream tasks, even without any task task-specific supervision. Another major research line of PLMS is to enhance the language model with auxiliary knowledge (Wei et al. (202)). including entity knowledge YYu al., 2020). relational knowledge (Zhang al. 2019: Qin et al. (021).
text chunk (Lewis et al., 2020: Wu et al. 2022, Borgeaud et | 22021). etc. The incorporation of
various knowledge resources to PLMs mitigates the drawbacks of local contextual attention, bringing
additional relevant global context that benefits both language understanding and generation tasks.
Since current unimodal PLMs lack visual knowledge grounding, they inevitably suffer from the hallucination problem, which refers to the inconsistent or false statements generated by PLMs with respect the world knowledge (Logan et al. 2 For instance, the PLMs may predict the color of the sky as red only due to the statistical contextual correlations between the token "color" and "red" in the pre-training corpus, neglecting the commonsense facts.
In this paper, we propose novel framework to enable language model pre-training to take full advantage of both local text context and corresponding visual knowledge. Recent work on joint vision- language model (VLM) pre-training (Su al. 2020 Tan & Bansal 2020) relies explicit alignments between text and image, e.g. supervised image captioning data, which limits the cross-modalle fusion during fine-tume tuning/inference over text without accompanying images. As a consequence, later in experiments (section 3). those prominent VLMs are found to achieve unsatisfactory performance on visual knowledge-intensvy commonsense reasoning tasks. Instead, we design flexible text-image

---

Published as a conference paper at ICLR 2023
alignment mechanism via image retrieval module that gathers related images for each token as visual augmentation. To achieve better uinuuae-vision vision grounding. we propose a visual knowledge fusion layer enable joint attention across visually- -augmented context including both textual tokens and retrieved images. Based on this, we build up Tisueely-augmented Language Model, VALM,, with flexible on-the-fly visual knowledge enhancement.
We evaluate the effectiveness of the proposed VALM on various commonsense reasoning and language-only benchmarks. Experimental results demonstrate that our model consistently outper- forms the unimodal and multimodal baselines in terms of object commonsense reasoning. Remarkably, our method substantially improves + +14.50% +17 17.80% and 1.68% accuracy on MEMORY COLOR, RELATIVESIE and OBJECTSHAPE datasets, respectively. Additional experiments on natural lan- guage understanding tasks also validate that the proposed framework could be helpful to improve the fundamental natural language understanding capability of PLMs.
language modeling
Our contributions are summarized as follows:
We propose a novel disually-augmentee casual language model, VALM, to enable the language model to utilize visual knowledge flexibly and effectively. Through the proposed visual knowledge fused language modeling, VALM is capable of accomplishing tasks with the high demand of cross-modality knowledge, such as visual commonsense reasoning.
We design framework to construct flexible on-the-fly fly text-image alignments and fuse augmented images into the context of language modeling. We implement an image retrieval module to query token-level representation in large-scale cached image database and retrieve its nearest neighbors as the augmentation. With the proposed visual knowledge fusion layer, VALM can effectively take full advantage of both language information from local text context and visual information from retrieved images.
Experimental results demonstrate that VALM effectively alleviates the hallucination problem of PLMs via introducing visual knowledge language model pre-training. VALM achieves significant performance improvements in inferring the commonsense object properties.
2
METHODS
We propose a novel multi-modal pre-trained language model, which is augmented with retrieved images, named VALM. The architecture of VALM is presented in Figure VALM augments each token in pre- pre-training text corpus with k retrieved related images. VALM uses an image retrieval module retrieve corresponding images for each token. The image retrieval module deploys pre-trained CLIP model, which is capable of unifying the textual query and image candidates into joint embedding space. VALM constructs a cached large-scale image knowledge base using image encoder of CLIP, and uses the contextual representation of each token as textual query to search its nearest neighbors in image knowledge base. With the help of the unified text and image embedding space provided by CLIP, the image nearest neighbors are taken augmented images of each token to construct text and image alignments. We then propose a essual-knowledge fusion layer to enable learned hidden state to attend to both texts and augmented images.
2.1
VALM:
LANGUAGE MODELING
Given an input text sequence {x, N the embedding layer first encodes input vector {x4 | into embedding space and outputs the initial hidden state H to the successive Transformer decoder layers. Then the proposed VALM model encodes HD into visual knowledge fused contextual representations difference levels H (H') via I. Transformer decoder layers and one special visual knowledge fusion layer. Each Transformer decoder layer is identical to Vaswani al. (2017). which outputs the contextual representations at different semantic levels given the representation from the previous layer H' Layer, (H' ).) [].
The visual knowledge fusion layer is proposed as a variant of the Transformer decoder layer to incorporate visual knowledge in contextual learning via joint attention both text contexts and augmented images. The visual knowledge fusion layer is injected in the second-to-lase layer of VALM. The visual knowledge is stored in corresponding augmented image representations, obtained from
2

---

Published as a conference paper at ICLR 2023
The
Knowledge
Pix x
sky
K
saftmax loyor
Visual-L anguage Attention
Feed-Forward
h
SelfAAtendon Layer
Feed Fonwand Noural Network
Images
Knowledge Layer
CLIP Text Encoder
Topk Imagge
The color of sky in
x [
Food Forward Neural Network
Testan
Vooll Knowlelgg
Self Ceft-Atention Layer
Baon
CLIP Image
Text-to-lmage Retrieval
sky
C Disg
Figure 1: Overview of ausmenty- language modeling (VALM). We conduct dense retrieval get top-k images for the input context each time step. Then the visual knowledge fusion layer attends to both text tokens and retrieved images. The vision-language fused representation is fed back Transformer for language modeling.
image retrieval module {[zy) fx.( Then the visual knowledge fusion layer takes the input including both contextual representation of the previous layer and augmented image sets and outputs a visual-knowledge fused contextual representation H4-1 = VisualLayer (H,--2 {Zij
1C )= =)) IN
Finally, the output contextual representations are passed into the output projection layer and softmax function is used to compute the token probability P(x,(x,,... .x,1)) softmax softmax(WFF (WH' + 6).
We conduct generative unsupervised pre-training (Radford al., 2019) for VaLM on a large-scale text corpus. The training objective of VALM is the standard left-to-righl language modeling objective, which maximizes the likelihood of the next word token based on the left context:
max
log P(x:|x),.... (x;-1);
(I)
where x represents a sentence randomly sampled from the large-scale pre-training text corpus D.
2.2
IMAGE RETRIEVAL
The visual knowledge corresponding specific token is stored in its correlated images. Therefore, to prepare the fused visual knowledge, VALM deploys an image retrieval module to retrieve augmented images, denoted (.) In order to achieve multi-modality text-image retrieval, it is of great importance to building up a discriminator to assess the correlation of every image in the extremely large-scale open image knowledge bases to the specific text representation. CL.IP (Radford et al.) 2021) proposed a simple-yet-effective method to connect images and texts into unified multi-modal embedding space. We directly deploy the pre-trained CLIP model to encode the images and texts to enable nearest neighbor text-image retrieval. Specifically, the pre-trained CLIP model we use in constructing the image retrieval module includes ResNet-50x 16 (He encoder and a Transformer (Vaswani et al.] (017)) model as a text encoder. Here, we only use the CLIP model as the backbone of our image retrieval module, and the CLIP parameters are not updated during the pre-training process of VALM.
(016) model as an image
Image Knowledge Base Creation. The image knowledge base the retrieval module of set of image keys, which are the high-level visual representations of images. Given an image z E D Dimg- such visual representation can be obtained via forwarding image z to the pre-trained CLIP image encoder. Then the whole image knowledge base (Z) is constructed by taking the output hidden state fo, (x) as image keys: = UzeDimg (fo, (z)). where f, represents the image encoder parameters.
cache

---

Published as a conference paper at ICLR 2023
Textual Query. We take the contextual representation of each token as the query in the nearest neighbor search. For each sentence x E D. the contextual representation of i-th token is computed via for (x<x), where 0r represents the text encoder parameters. As the input sequence length of VALM generally exceeds the input length limitation of 75 tokens of CLIP text encoder, the long context X
[|t i- < 75, cut off into a context-chunk y; for fitting in CLIP text encoder: y; X
2 75,
where L is the index of the closest stop character before i-th token. Then the textual query for i-th token computed as its context-chunk representation as so (y:).
NN Text-Image Retrieval. The retrieval module uses the contextual representation to search the cached image knowledge base (Z) and retrieves k nearest neighbor image keys w.r.t. dot product distance. As the pre-trained CLIP model has learned a joint embedding space for text and image domain, the retrieved images { 1==1 are thus regarded as the top-k relevant images to the query.
2.3 VISUAL KNOWLEDGE FUSION
With the help of the image retrieval module, each token in the pre-training corpus is augmented with k corresponding images, and these augmented images are represented in the joint embedding space with texts. Then the augmented image representations are directly treated as auxiliary "context" in the learning process.
As the conventional Transformer decoder layer uses the multi-head self-attention (Vaswani et al. 2017) to learn the contextual representation, we extend it joint-attention mechanism and propose a novel visual knowledge fusion layer to enable each token to attend to both contexts and retrieval images jointly. In addition, due to the inconsistency in magnitude and distribution between contextual hidden states and retrieved image representations, we apply Layer Normalization (Ba et al.) (2016) on retrieved K. image representations to alleviate such inconsistency, denoted as I.Nimg- Assume that the hidden state output for i-th token is h, and the corresponding retrieved images are (z.g 1 the
hidden state H is computed as:
Q H'-2W0 +02, K H + V 2 + +bV
(2)
kis = Lining Zikk WK + bK
img Vik = LNimg (zix) W + bK
(3)
Q,K
exp (e:)
6 Vd
(4)
Σ 1j=1 exp (ety) + Σk= exp (e#t)
Q T
exp (eix)
6sk
Vd
O4k
Σj expee (et)
Σx=1 exp (eix)
(5)
H - = a. V Σ ask Viki
(6)
where Q knt V RRE K, RI Cs,a; a, E Rix The hidden state output from the previous layer H is linearly projected into contextual queries, keys, and values Q, K, V separately. IC is the number of retrieved images for each token, and E is the embedding dimension for both context and image representations. In order to generate image-specific attention keys and values, we adopt image-specific bias biKnng "//ng in linear projections and reuse the contextual projection weights W V to generate image-specific attention keys and values. Moreover, it is vital to mention that the image-specific attention keys and values are distinct for each query token, which is highly different from self attention where the contextual keys and values are kept the same for each token. A secondary subscript & is used to denote different image representations for the i-th token.
3 EXPERIMENTS
3.1 PRETRAINING SETUP
Text Corpus. We use the English corpus of CC-100 (Conneau et al. (020) as the pre-training text corpus for both VALM and baseline GPT-2". CC-100 corpus is of the largest high-quality
4

---

Published as a conference paper at ICLR 2023
Task
Example Prompt
Object Pair
Answer
Object Color Reasoning
The color of [object] [answer] The shape of [cbject] is [answer] Is [mtell]] larger than [Item2]? [answerl
the sky apple (sofa, car)
blue
Object Shape Reasoning
round
Object Size Reasoning
Yes
Table 1: Evaluation examples of object color, shape, and size reasoning.
web-crawl text data. The English monolingual dataset of CC-100 contains about 55 billion tokens, stored in 301 GiBs disk storage. Due to the limitation of computing resources, we only consume 15% of CC-100 English monolingual corpus for pre-training VALM and bascline GPT-2'.
Image Data. We use the LAION Open Image Dataset (Schuhmann al. 2021) as the image knowledge base for dense retrieval. To the best of our knowledge, the LAION Open Dataset is one of the world's largest openly available image-tett-pair dataset with 400 million samples. Due the disk space limitation, we randomly select half of LAION images for the dense text-image retrieval, which 200M images in total.
The proposed model deploys transformer decoder architecture with 124M trainable parameters. Hyperparameter setting and training details are presented in Appendix B.I
Pre-training Hyperparameters.
Retrieval Module. For the implementation of the dense text-image retrieval module, we use the faiss (Johnson et al. (021)) toolkit to construct the efficient index. The faiss index contains the whole 200M image keys and provides the efficient nearest neighbor search. For efficiency purposes, quantize all image keys 32 bytes. fai ss index stores image keys in clusters to speed up the search, which requires the additional training process to learn the cluster centroids. We use 10M keys for learning 131k cluster centroids and search 32 clusters find the nearest neighbors during inference. We load the faiss index to GPU to achieve efficient dense text-image retrieval.
3.2 VISUAL KNOWLEDGE INTENSIVE TASKS
The visual information stored in retrieved images can play useful role in providing relevant visual knowledge to help language models perform better grounded commonsense reasoning. Such helpful visual information can be colors, positions, sizes, spatial relations, etc. The task of object commonsense reasoning requires language models to predict the correct visual property for a given object. To excel these tasks typically require models to capture and utilize intensive visual knowledge without any explicit text demonstrations or external knowledge bases. Due to reporting biases, such descriptive text of object properties rarely appears in text corpora, likely making this type of knowledge absent from language models. Thus, those visual mnwwledge-intensive tasks are likely challenging for both language models and vision-language models.
We first compared VALM and recent baselines on four object commonsense reasoning datasets, MEMORYCOLOR (Norlund et al., 2021), COLORTERMS (Bruni et al. 2012). OBJECTSHAPE (Zhang et al. 2022a) reasoning, and RELATIVESIZE (Bagherinezhad et al. (016). In addition, we use another physical interaction question answering dataset (PIQA) (Bisk et al.) 2020). to evaluate whether such visual commonsense knowledge could be implicitly encoded and utilized in the question answering we provide examples for different visual commonsense reasoning tasks.
process. In Table [
HEMORYYCOLOR and COLORTERMS Dataset. The memory color of a concrete object the typical color an object appears in, e.g. the color of banana is mostly memorized as yellow. Norlund let al. 202T) proposed this dataset for evaluating visual knowledge transfer in multi-modal language models. The dataset contains 109 objects paired with their memory color labels, illustrating picture, and a descriptor. The COLORTERMS dataset also contains a list of common items manually labeled with their commonsense color. Both datasets hold set of 11 color labels.
OBJECTSHAPE Dataset. Zhang al. (022a)) proposed visual commonsense dataset with set of object attributes like shape. The dataset of object shapes contains 140 objects with their shape label. The ORIECTSHAPE dataset consists of 12 shape categories.
5

---

Published as a conference paper at ICLR 2023
RELATIVESIZE Dataset. Bagherinezhad et al. (2016) proposed the RELATIVESIZE dataset, which includes a total of 486 object pairs between 41 physical objects. The task of object size reasoning requires the model predict the size relations between two given objects, e.g,, an ant is smaller than an elephant. The size information is again rarely included and described in text, while it is much easier capture from the images. We convert the size relation reasoning task into binary question-answering form with "Yeo" answers.
PHYSICAL INTERACTION QUESTION ANSWERING. Physical Interaction Question Answering (PIQA) is proposed and designed to investigate the physical commonsense knowledge of existing language models (Bisk et al. 2020). Completing such question answering tasks requires the language model to effectively utilize physical commonsense knowledge, i.e. knowledge of basic properties of the objects (flexibility, curvature, and being porous). Language models are supposed to first achieve the perception of objects and later encode such physical knowledge into the language modeling process. Each data sample in PIQA contains one goal and two solutions. The model is supposed to figure out and predict the more reasonable and appropriate solution between two candidates.
Evaluation Setting. We evaluate VALM and all baseline methods in zero-shot manner without any task-specific tuning. Specifically, VALM takes the input consisting of textual prompts and objects during inference and predicts the property label as the last token. The prompts used in evaluating object color, shape, and size reasoning performance are listed in Appendix Table [T] We use the top-1 accuracy as the evaluation metric and compute the average accuracy of all listed prompts to increase evaluation robustness. For PIQA, we follow [Shwartz et al. 2020) to use the cross-entropy loss as the scorer for each potential solution score(si; CE Dt. Sty s.yD. [0, 1]. Then the solution with lower scores is selected as the prediction. The classification accuracy is used as the evaluation metric.
Baselines. We consider both pretrained language-only and vision-language models as baselines. In particular, three strong language models are considered for comparison with VALM, including 1) GPT-2" (Radford et al. 2019): 2) BERT Devlin et al. (2019); and 3) CaptionBERT (Zhang et al. 2022a)), a pre-trained auto-encoding language model on Oscar's (Li et al. 2020) caption- -aased text data. Here, GPT-2" is re-implemented and trained from scratch using the identical training data, hyper-parameter settings, and model size as the proposed VALM. Additionally, we also compare VALM with prominent vision-language models, including I) OSCAR (Li et al.) 2020). pre-trained vision-language model with learned representations that capture channel-invarian factors (i.e. object tags) at the semantic level; 2) VisualBERT (Li et al., 2019). vision-languar model with learned joint contextualized representations across vision and language: 3) CLIP (Radford et al. 2021). a vision-language system with one image encoder and one text encoder which are mapped into same cross-modal embedding space. We directly use OSCAR and VisualBERT as auto-encoding language models for zero-shot evaluations. For CLIP, we first retrieve the corresponding image using the concatenated query prompt and the given object. Then, the dot-product similarity of the retrieved image vector and the candidate-awe aware text vector (including the query prompt, the given object, and one candidate label) is used to rank. Finally, the top-ranked candidate label is regarded as the prediction for evaluation.
Results. The main results on four object commonsense reasoning datasets are summarized in Table
The two variants of VALM (KC = 4, 8) significantly outperform all considered language models and vision-language models on object color and shape reasoning datasets, with an improvement of +14.50%, + +13.56%, and 1.66% on MEMORYCOLOR respectively. Moreover, the proposed VALM with IC = 4 achieves an encouraging result with +17.00 accuracy gain over the best baseline, VisualBERT on RELET The substantial improvements on these datasets demonstrate that VALM takes full advantage of visual knowledge (object visual property) to complete the corresponding visual commonsense reasoning. Surprisingly, the zero-shot evaluation results of all auto-encoding language models and vision-languag models are below 40% accuracy on object color and shape reasoning datasets. Although pretrained with aligned text-image pairs, those linguaelanguage models cannot effectively utilize relevant visual knowledge from their jointly contextualized languae representations. Among language models, the rego-segressvve PLMs significantly outperform auto- outo-encoding PLMs, suggesting that auto-regressi regressive PLMs are likely better at zero-shot reasoners. We also observe that retrieving more images for each token results in performance drop for object size and shape reasoning. We attribute the degradation
COLORTERMS, and OBEECTSHAPE

---

Published as a conference paper at ICLR 2023
Color (ACC1)
Shape (ACC))
Size (ACC1)
Model
K
MEMORYCOLOR
COLORTERMS
OBJECTSHAPE
RELATIVESIZE
GPT-2
NIA
44.14%
39.10%
51.09%
47.22%
BERT
N/A
24.34%
26.33%
31.86%
34.78%
CaptionBERT
N/A
24.84%
28.40%
38.14%
66.05%
CLIP
N/A
26.25%
23.08%
13.66%
47.99%
OSCAR
N/A
20.32%
16.86%
33.14%
50.14%
VisualBERT
N/A
26.68%
38.02%
11.14%
67.23%
VALM
53.99%
52.66
62.77%
85.03%
VALM
58.64%
50.19%
59.41%
62.35%
Table 2: Accuracy on object commonsense reasoning datasets. GPT-2" is the re-implemented model with identical pre-training data and hyper-parameter settings to VALM. XC represents the number of images augmented to each token. Best performance is marked with bold.
Model
GPT-2*
BERT
CaptionBERT
VALM (K=4)
VALM (K=8)
PIQA (ACCT)
62.53%
54.73%
53.97%
64.31%
64.64%
Table 3: Accuracy on Physical- Intercci-nntrraationn Question-Answering benchmark.
the increased noise brought by augmenting with more images which causes model confusion when differentiating relevant visual information from irrelevant one.
PIQA is more challenging task that requires the model reason useful implicit object properties and utilize these commonsense in the question answering process. The results on PIQA are presented in Table is shown, VALM outperforms all baseline language models with +2.1 11% accuracy improvement. The two variants of VALM achieve almost identical performance because the selection for the correct solution is based on the language modeling perplexity, indicating that the two variants demonstrate similar language modeling capability.
3.3 NATURAL LANGUAGE UNDERSTANDING AND LANGUAGE MODELING TASKS
The casual language modeling pre-training task enables PLMs to naturally perform natural lan- guage understanding (NLU) and long-text modeling. Therefore, the zero-shot natural language understanding and language modeling performance are widely adopted to evaluate the capability of PLMS (Radford et al. 2019). Here, we evaluate VaLM and the most relevant language model baseline GPT-2* on four NLU datasets, SST-2 (Socher et al.) (013). MPQA (Wiebe et al., (005).. DBPeida (Auer et al.) 2007), and AGNews (Zhang et al. 2015). The prediction accuracy is used as the evaluation metric. In addition, following Radford et al. (2019), Wikitext- 103 (Merity et al.) 2017) and Lambada corpus (Paperno al.) (0166) are considered to study the language modeling performance in a zero-shot manner. We report perplexity for two corpora and also report last-word prediction accuracy for Lambada corpus.
The results on natural language understanding are summarized in Table 4 easy to see that VALM achieves decent improvements all four NLU tasks, indicating that the cross-modality knowledge learned in our model is likely helpful for typical natural language understanding. Thus, aisgeney-augmented language modeling framework can be further explored enhance the natural language understanding ability of PLMs. Table illustrates the results of language modeling tasks. Again, VALM slightly improves the perplexity on both datasets, +0.68 on Wikitext- 103 and +0.08 on Lambda. A similar trend is observed for the final word prediction accuracy on Lambada. Different from previous visual knowledge intensive commonsense reasoning tasks (subsection 3.2). we find that VALM models with different numbers of retrieved images (K 8 vs K. 4) perform similarly on the intrinsic language modeling task, suggesting that VALM can effectively ignore irrelevant visual information when the task is unlikely to benefit from visual knowledge. In other words, visual commonsense reasoning tasks require more fine-graine -ne-grr fusions of text and image, i.e. locating the text object in the image set, extracting relevant vision information, and verbalizing reasoning output. contrast, a certain portion of text from general language modeling corpora s is probably not visually related. Thus, only coarse-grained fusion is sufficient here (e.g. deciding if the image set is
7

---

Published as a conference paper at ICLR 2023
SST-2
MPQA
DBPedia
AGNews
Model
K
ACC1
ACC1
ACCT
ACC1
Majority GPT-2"
N/A
50.90%
50.00%
9.4%
25.0%
N/A
68.04%
71.25%
67.20%
53.51
VALM
70.12%
78.70%
72.27%
53.81%
VALM
8
67.33%
77.35%
68.48%
59.77%
Table 4: Zero-shot evaluation results on natural language understanding tasks (SST-2, MPQA, DBPedia, AGNews). Majority: majority class.
Wikitext-103 PPI.
Lambada PPI.].
Lambada ACC1
Model
K.
GPT-2"
N/A
36.44
42.46
42.17%
VALM
4
35.78
42.51 42.38
42.65%
VALM
8
35.76
42.87%
Table 5: Zero-shot evaluation results on language modeling tasks. We report perplexity (PPL) on Wikitext- 103 and Lambada and final word prediction accuracy (ACC) on Lambada.
useful), making the language modeling evaluation less affected by the retrieval noise from augmented images.
3.4 ABLATION STUDIES
So far, we empirically verify the effectiveness and superiority of VALM in utilizing visual knowledge for both visual knowledge-intensin tasks and traditional language understanding and modeling. To figure out how the visual information takes effect in model, we focus two questions here: I) the model capable using the retrieved image representations as "auxiliary" contexts? What is the effect of disabling such retrieved image representations during inference? To evaluate this, design an ablation model which set IC 0 and disables image retrieval and fusion during inference. 2) Does the model learn leverage visual knowledge in the retrieved images? What is the effect of directly augmenting randomly-retrieved image representations during inference? Thus, an ablation model which retrieves random images as augmentations during inference is used for probing. The results of the two ablation models, Randomly-Retrieww and Disable-Retriewe during the inference stage, are listed in the first two rows of Table 6 As we can see, both changes to the image retrieval result in noticeable performance degradation on all evaluation tasks. In particular, we find that disabling the image retrieval and augmenting no image during inference also makes huge difference to the language modeling perplexity on two corpora, which is more related to pure text corpus rather than augmented images. Therefore, suggests that VALM is able to effectively capture rich semantics from both pretraining sources, i.c. text corpus well augmented images. In other words, the improved zero-shot task transfcrability of VALM relies on visual information from augmented images, which complements the linguistic knowledge learned via text-based self-supervised learning. ablation model further illustrate that achieving the capability
The results of the Randomly-Retrian of integrating visual knowledge cannot be realized by only augmenting unrelated images to language models, while only context-relevant images can make a true difference.
VALM proposes novel visual knowledge fusion layer with joint context-image attention mech- anism as key component to fuse visual knowledge into the language model. The separate linear projection layers are regarded as important components map contexts into different embedding spaces for attention keys and values. Therefore, the proposed joint self-attention mechanism naturally holds three variations to generate image keys and values: establish image-specific linear projections, reuse contextual linear projections, and only use specific linear bias for augmented images. We conduct the ablation study to evaluate the effect of these three alternatives on image linear projec- tions. The results in demonsttettttttttee that adopting image-specific projection bias outperforms directly sharing the contextual projection bias. Introducing additional image-specific linear projection weights does not lead to further performance improvement. Thus, we take the strategy of only adding additional linear bias for augmented images and reuse contextual linear weights in generating visual attention keys and values for implementation convenience and parameter efficiency.
8

---

Published as a conference paper at ICLR 2023
MemoryColor ACC1
ColorTerms ACC1
RelativeSize ACC*
Wikitext-103 PPL4
Lambada PPL.
Lambada
Ablation Model
A
ACCT
Hundomly-Retriess
41.48%
38.46% 40.53%
49.63%
37.89
43.56
41.35%
0
43.12%
59.36%
39.22
44.59
41.20%
W.Y.V &... mma ima W.R. b b..
48.62%
41.86%
83.54%
35.95
42.28
42.67%
4
47.85%
46.30%
62.04%
35.95
42.33
41.74%
WKv." B... (VALM) 4
53.99%
52.66%
85.03%
35.78
42.51
42.65%
ing
Table 6: Ablation studies the effects of Handomly-Retriesd and Disabling-Retrievall during inference stage (Upper part). Second ablation study on the effects of introducing extra image-specific attention key and value projection weights Whey or bias K.V. in Equation 3 for augmented images. The proposed model VALM is shown in the last row which introduces only tmage-speciff bias and reuses contextual weight in attention key and value projection layers.
RELATED WORK
Pre-trained Language Models. Pre-trained language models (PLMs) revolutionized NL.P research.
Enabled by attention mechanism (Bahdanau et al., 2015) and Transformer architecture (Vaswani et al. 2017). the fate-of-thheart PLMs. including BERT (Liu et al. 2019), GPT (Radford et al. 2018 2019). RoBERTa (Liu et 2019), ELECTRA (Clark al. 2020), T5 (Raffel et al. (020). and OPT ZZhngg et al., 2022b), have become the dominant approach in NLP tasks via the paradigm of
pre-training on large-scale text corpora and fine-tuning on downstream tasks. With the exponential scaling up of model size, surprising fact emerged that the PLMs like GPT-3 (Bronn et al. 2020) can work as few-shot or zero-shot learners.
Vision-Language Models. Vision-language tasks are at the intersection area of both modalities of vision and language, like visual-question answering (Agrawal al. 2013), and image caption- ing (Chen et al. a (015). ViL-BERT (Lu et al.) (201)) firstly proposed to generate image region features via object detection and then learn joint multi-modal representations via an interacted two-stream model. OSCAR (Li et al., 2020) proposed to introduce object tags detected in images as anchor points to solve the issue of high demand for image-text alignments. Another significant pathway for VLMs is to construct unified embedding space for texts and images and use textual prompts to extract task-specific labels during inference, which the representative models are CLIP (Radford and ALIGN (Jia et al. 2021).
al. 2021
Visually- Grouudee Language Learning. Visually-grounded language learning is an emerging research topic in vision-language learning, in which the proposed VALM can be categorized in this area with other prior works like Vokenization (Tan Bansal 2020), VidLanKD (Tang et al. (021). and iACE (Lu et al., (022). Visual information and know ledge can be memorized by the PLMs via fusion layer or concatenated inputs. However, extracting and utilizing the visual information efficiently and effectively is still difficult for uni-modal language models. Vokenization concatenated tokens and rokan-related images as "vokens", transferring sentence-level caption text to token-level "voken" with Vokenizer model.
CONCLUSION
In this paper, we propose a multi-modal framework VALM to enable auto-regressive language modeling to effectively utilize visual knowledge. Specifically, an effective text-to-image retrieval module is designed to construct latent text-image alignments for visually-grounder language model- ing. Empowered by pre-training. VALM achieves improved zero-shot task transfer on downstream tasks. Experiments on various visual Knowledge-intensin tasks demonstrate the effectiveness of our model over recent vision- langua-a models. VALM also achieves decent improvements over language models on multiple representative natural language understanding tasks. For future work, we plan to adapt the model architecture to encoder-only and encoder-decoder Transformer backbones. We are also interested in more input modalities for VALM.

---

Published as a conference paper at ICLR 2023
ACKNOWLEDGEMENTS
We would like to thank the anonymous reviewers for the helpful comments. We appreciate Zewen Chi and Hangbo Bao for the fruitful discussions, and Yaru Hao for helpful suggestions on evaluation benchmarks.
REFERENCES
Aishwarya Agrawal, Jiasen Lu, Stanislaw Antol, Margaret Mitchell, C. Lawrence Zitnick, Devi Parikh, and Dhruv Batra. VQA: Visual question answering. International Journal of Computer Vision, 123:4-31, 2015.
Saced Anwar, Muhammad Tahir, Chongyi Li, Ajmal S. Mian, Fahad Shahbaz Khan, and Abdul Wahab Muzaffar. Image colorization: A survey and dataset. ArXiv, abs/2008.00774, 2020.
Sören Auer, Christian Bizer, Georgi Kobilarov, Jens Lehmann, Richard Cyganiak, and Zachary Ives. Dbpedia: A nucleus for web of open data. In The semantic web, pp. 722-735. Springer, 2007.
Jimmy Ba, Jamie Ryan Kiros, and Geoffrey E. Hinton. Layer normalization. ArXiv, abs/1607..commm 667706450, 2016.
Hessam Bagherinezhad, Hannaneh Hajishirzi, Yejin Choi, and Ali Farhadi. Are elephants bigger than butterflies? reasoning about sizes of objects. ArXiv, abs/ hess/6022.075,, 2016.
Damitry Bahdanau. Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. CoRR, abs/ 1es91449.77, 2015.
Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, et al. Piqu: Reasoning about physical commonsense in natural language. In Proc. AAAI, 2020.
Sebastian Borgeaud, Arthur Mensch, Jordan Hoffmaann, Trevor Cai, Eliza Rutherford, Katie Mil- lican, George van den Driessche, Jean- -aptiste Lespiau, Bogdan Damoc, Aidan Clark, Diego de Las Casas, Aurelia Guy. Jacob Menick, Roman Ring. T. W. Hennigan, Saffron Huang, Lorenzo Maggiore, Chris Jones, Albin Cassirer, Andy Brock, Michela Paganini, Geoffrey Irving, Oriol Vinyals, Simon Osindero, Karen Simonyan, Jack W. Rae, Erich Elsen, and L. Sifre. Improving language models by retrieving from trillions of tokens. ArXiv, abs/21 12.04426. 2021.
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, T. Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeff Wu. Clemens Winter. Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. ArXiv, abs/2005.14115 2020.
Elia Bruni, Gemma Boleda, Marco Baroni, and Nam Khanh Tran. Distributional semantics in technicolor. In ACL, 2012.
Xinlei Chen, Hao Fang. Tsung- Yi Lin, Ramakrishna Vedantam, Saurabh Gupta, Piotr Dollár, and C. Lawrence Zitnick. Microsoft COCO captions: Data collection and evaluation server. ArXiv, abs/1554.00032, 2015.
Kevin Clark, Minh-Thang Luong, Quoc V. Le, and Christopher D. Manning. Electra: Pre-training text encoders discriminators rather than generators. ArXiv, has/2003.10555: 2020.
Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Fran- cisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. Unsupervised cross-lingual representation learning at scale. In ACL, 2020.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In NAACL, 2019.
10

---

Published as a conference paper at ICLR 2023
Ruiqi Guo, Philip Sun, Erik Lindgren, Quan Geng, David Simcha, Felix Cher, and Sanjiv Kumar. Accelerating large-scale inference with anisotropic vector quantization. In International Conference on Machine Learning, pp. 3887-3896. PMLR, 2020.
Kaiming He, X. Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. 2016 IEEE Conference Computer Vision and Pattern Recognition (CVPR), pp. 770-778, 2016.
Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoe Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and vision-language representation learning with noisy text supervision. In JCMI, 2021.
Jeff Johnson, Matthijs Douze, and Hervé Jégou. Billion-scale similarity search with gpus. IEEE Transactions on Big Data, 7:535-547, 2021.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. CoRR, abs/1412.6990, 2015.
Patrick Lewis, Ethan Perez, Aleksandara Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Kuttler, Mike Lewis, Wen tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. Retrieval-augmented generation for knowledge-intensive nlp tasks. ArXiv, abs/2005. 2020.
Liunian Harold Li, Mark Yatskar, Da Yin, Cho-Jui Hsieh, and Kai-Wei Chang. Visualbert: A simple and performant baseline for vision and language. arXiv preprint 1rX3v:190..03557, 2019.
Xiujun Li, Xi Yin, Chunyuan Li, Xiaowei Hu, Pengchuan Zhang, Lei Zhang, Lijuan Wang, Houdong Hu, Li Dong, Furu Wei, Yejin Choi, and Jianfeng Gao. Oscar: Object-semantics aligned pre- training for vision-language tasks. In ECCV, 2020.
Yinhan Liu. Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. RoBERTa: A robustly optimized bert pretraining approach. ArXiv, abs/1907.1 1692, 2019.
Robert Logan, Nelson F. Liu, Matthew E. Peters, Matt Gardner, and Sameer Singh. Barack's 's wife hillary: Using knowledge graphs for fact-aware language modeling. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 9971-1971, Florence, Italy, July 2019. Association for Computational Linguistics. doi: 10.186533v1/PP 1/t5///1/P99111111 1598.
Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee. Vilbert: Pretraining task-agnostic visiolinguistic representations for iision-and-languun tasks. Advances in neural information processing systems, 32, 2019.
Yujie Lu, Wanrong Zhu, Xin Wang, Miguel P. Eckstein, and William Yang Wang. Imagination- augmented natural language understanding. ArXiv, abs/2204.08535, 2022.
Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models. ArXiv, htss/669907843.. 2017.
Deepak Narayanan, Mohammad Shoeybi, Jared Casper, Patrick LeGresley, Mostofa Ali Patwary, Vijay Anand Korthikanti, Dmitri Vainbrand, Prethvi Kashinkunti, Julie Bernauer, Bryan Catanzaro, Amar Phanishayee, and Matei A. Zaharia. Efficient large-scale language model training on gpu clusters using megatron-im. Proceedings of the International Conference for High Performance Computing, Nenworking. Storage and Analysis, 2021.
Tobias Norlund, Lovisa Hagström, and Richard Johansson. Transferring knowledge from vision to language: How to achieve it and how to measure it? ArXiv, abs/2109.11321 2021.
Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross. Nathan Ng, David Grangier, and Michael Auli. fairseq: A fast, extensible toolkit for sequence modeling. In NAACL, 2019.
Denis Paperno, Germán Kruszewski, Angeliki Lazaridou, Quan Ngoc Pham, Raffaella Bernardi, Sandro Pezzelle, Marco Baroni, Gemma Boleda, and R. Fernández. The lambada dataset: Word prediction requiring broad discourse context. ArXiv, hiss/600.000331 2016.
11

---

Published as a conference paper at ICLR 2023
Yujia Qin, Yankai Lin, Ryuichi Takanobu, Zhiyuan Liu, Peng Li, Heng Ji, Minlie Huang, Maosong Sun, and Jie Zhou. Erica: Improving entity and relation understanding for pre-trained language models via contrastive learning. In ACL, 2021.
Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving language under- standing with unsupervised learning. 2018.
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. 2019.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and llya Sutskever. Learning transferable visual models from natural language supervision. In ICML, 2021.
Colin Raffel, Noam M. Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter Liu. Exploring the limits of transfer learning with unified text-to-text transformer. ArXiv, abs/1910. 10683, 2020.
Christoph Schuhmann, Richard Vencu, Romain Beaumont, Robert Kaczmarczyk, Clayton Mullis, Aarush Katta, Theo Coombes, Jenia Jitsev, and Aran Komatsuzaki. Laion-400m: Open dataset of clip-filterd 400 million image-text pairs. ArXiv, abs/2111.0211 2021.
Rico Sennrich, Barry Haddow, and Alexandra Birch. Neural machine translation of rare words with subword units. ArXiv, abs/1508 .00900, 2016.
Vered Shwartz, Peter West, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Unsupervised commonsense question answering with self-talk. arXiv preprint arXiv:2004. 05483,, 2020.
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning, Andrew Ng, and Christopher Potts. Recursive deep models for semantic compositionality over sentiment treebank. In Proceedings of the 2013 conference on empirical methods in natural language processing, pp. 1631-1642, 2013.
Weijie Su, Xizhou Zhu, Yue Cao, Bin Li, Lewei Lu, Furu Wei, and Jifeng Dai. VI-bert: Pre-training of generic visual-linguistic representations, ArXiv, 0bs/1998.00530 2020.
Hao Tan and Mohit Bansal. Vokenization: Improving language understanding via contextualized, visually-grounded supervision. In EMNLP, 2020.
Zineng Tang, Jaemin Cho, Hao Tan, and Mohit Bansal. Vidlankd: Improving language understanding
via video-distilled knowledge transfer. Advances in Neural Information Processing Systems, 34: 14468-24481, 2021.
Ashish Vaswani, Noam M. Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. In NIPS, 2017.
Weizhi Wang, Zhirui Zhang, Junliang Guo, Yinpei Dai, Boxing Chen, and Weihua Luo. Task-oriented dialogue system as natural language generation. In Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 2698-2703, 2022.
Xiaokai Wei, Shen Wang, Dejiao Zhang, Parminder Bhatia, and Andrew O. Arnold. Knowledge enhanced pretrained language models: A compreshensive survey. ArXiv, abs/21 10.08455, 2021.
Janyce Wiebe, Theresa Wilson, and Claire Cardie. Annotating expressions of opinions and emotions in language. Language resources and evaluation, 39(2): 2200 2005.
Yuhuai Wu, Markus N. Rabe, DeLesley S. Hutchins, and Christian Szegedy. Memorizing transformers. ArXiv, 0tts/2233.08913. 2022.
Zhilin Yang, Zihang Dai, Yiming Yang. Jaime G. Carbonell, Ruslan Salakhutdinov, and Quoe V. Le. XLNet: Generalized autoregressive pretraining for language understanding. In NeurlPS, 2019.
Donghan Yu, Chenguang Zhu, Yiming Yang, and Michael Zeng. Jaket: Joint pre-training of knowledge graph and language understanding. ArXiv, abs/2010.00796, 2020.
12

---

Published as a conference paper at ICLR 2023
Chenyu Zhang, Benjamin Van Durme, Zhuowan Li, and Elias Stengel-Eskin. Visual commonsense in pretrained unimodal and multimodal models. arXiv preprint arXiv:2205.0/850 2022a.
Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.0100 2022b.
Xiang Zhang, Junbo Zhao, and Yann LeCun. Character-level convolutional networks for text classification. Advances in neural information processing systems, 28, 2015.
Zhengyan Zhang, Xu Han, Zhiyuan Liu, Xin Jiang, Maosong Sun, and Qun Liu, Ernic: Enhanced language representation with informative entities. In ACL, 2019.
13

---

Published as a conference paper at ICLR 2023
A ADDITIONAL RESULTS
A.1 TIME-COST EFFECTS OF RETRIEVAL AND IMAGESET SIZE
Introducing efficient image retrieval on GPU brings a linear increase in inference time cost (about 2.1 times of text-only GPT-2" baseline), shown in Table 7 This cost is negligible with larger-size language models because the model forward cost will increase many times while the retrieval cost will change with the model size. The retrieval cost can be further improved by searching fewer clusters or decreasing the number of encoding bytes for approximate image keys, with a minor trade-off on the performance. Moreover, efficient nearest neighbor search is an active research area (Guo et 2020) and we could try other efficient search tools to accelerate the retrieval.
As the introduced retrieval time cost is proportional to the size of imageset for dense retrieval, we provide more details on the relationship between retrieval time cost and imageset size, presented T Concluded from Table 7 there no significant performance decrease with the smaller imageset size from the original 200M down to 10M. As the 10M set is still large and sufficient for providing enough visual knowledge, we will consider deploying a 10M size imageset to train VALM for potential real-time industry applications.
Image Size
Color (ACC1)
Shape (ACC1) OBJECTSHAPE
Size (ACC1)
Timecost (GPT2' as 1x)
MEMORYCOLOR
COLORTERMS
RELATIVESIZE
200M
53.99%
52.66%
62.77%
85.03%
2.06x
100M
53.50%
49.71%
61.39%
81.84%
1.91x
10M
51.79%
47.49%
62.18%
82.15%
1.79x
1M
51.87%
46.31% %
48.51
82.35%
1.74x
Table Accuracy on object commonsense reasoning datasets of VALM (K with variants of retrieval imageset size. K. represents the number of images augmented to each token.
A.2 COMPARISONS WITH ADDITIONAL STRONG BASELINES
We compare VALM with Vokenization Tan Bansal 2020) on four tasks, and the results are shown in Table In addition, we evaluate the performance of large language models on the Sosuall-knowlldg---teeesssi We evaluate the OPT (1.3B parameters) (Zhang al. 2022b) model these visual-knowledge- intensive tasks and the results are presented in Table 8 VALM(124M parameters) significantly outperforms the OPT- on four datasets, which further demonstrates the challenge of solving those wwwwkedge-inneeeeeeooeee
tasks for stronger and more fair comparisons.
tasks and the effectiveness of our method.
Color (ACC1)
Shape (ACCT) OBJECTSHAPE
Size (ACCT) RELATIVESIZE
Model
KC
MEMORYCOLOR
COLORTERMS
OPT-1.3B
N/A
39.25%
41.03%
19.21% %
50.78%
Vokenization
N/A
14.18%
19.97%
48.35%
43.28%
VALM VALM
53.99%
52.66 %
62.77% 59.41%
85.03% 62.35%
8
58.64%
50.19%
Table Accuracy on object commonsense reasoning datasets. K. represents the number of images augmented to each token. Best performance is marked with bold.
A.3 SCALING EFFECT OF VALM
We train the 355M model (GPT-2 Medium Size) of VALM (k=8) to evaluate the effects of scaling up model parameters. The results presented in Table and improved on four visual knowledge-intente -nnteneivve datasets. We will seck more computation resources to train large size VALM models.
and the model performance is significantly
14

---

Published as a conference paper at ICLR 2023
Color (ACC1)
Shape (ACC1)
Size (ACC1)
Model
K
MEMORYCOLOR COLORTERMS
OBJECTSHAPE
RELATIVESIZE
'ALM-124M
58.64%
50.19%
62.35% 59.41%
VALM-355M
65.82%
55.36%
70.0%
72.79%
Table 9: Accuracy object commonsense reasoning datasets. K. represents the number of images augmented to each token. Best performance is marked with bold.
A.4 ABLATION STUDY K.
We further conduct another ablation study by setting the number of augmented images XC = 1 for VALM, which is very similar to the CLIP (Radford et al. (02T)) inference. The results are presented in Table VALM (k=1) significantly outperforms CLIP in all Assual-nnowleger validating the effectiveness our method.
-etelldde--itensveee intensive tasks,
Color (ACC1)
Shape (ACC1)
Size (ACC1) RELATIVESIZE
Model
MEMORYCOLOR
COLORTERMS
OBJECTSHAPE
CLIP
N/A
26.25%
23.08% 46.01 %
13.66% 64.55%
47.9 47.99%
VxL.M
52.43%
86.73%
Table 10: Accuracy on object commonsense reasoning datasets. K. represents the number of images augmented each token.
A.5 CASE STUDIES
We provide case study the object color reasoning task VALM. order reason the correct commonsense color of objects sky and parsley, VALM takes the input combination of the prompt and the object as "the color [object] is". Then we present the retrieval results of top-4 corresponding images to the textual query in Figure
color
parsley
0.2
0.1
(a)
color
sky
(b)
Figure 2: The attention matrix visualization given the query prompt "the color of [object] is" for VALM. VALM achieves accurate image retrieval of top-4 images corresponding the objects of sky and parsley as augmented images, shown in the horizontal index of each subfigure.
A.6 COLORIZATION EFFECT
We conduct another interesting ablation case study to evaluate the effect of image color changes in the object color reasoning task. Specifically, VALM predicts the color label of an apple as red based on the commonsense in both contexts and retrieved images. The original prediction probability distribution is presented in Blue Bars of Figure b). Then we replace the retrieved images with IC unusual images of green apples in TRIECTCOL.ORIZATUUU dataset (Anwar al. 2020). shown in
15

---

Published as a conference paper at ICLR 2023
Figure B a). The predicted probability distribution for 11 color types given replaced colorization objects is presented in Orange Bars TigureBib) We could observe a clear probability increase in the color type of green and decrease in that of red, which is confronted with the colorization process. This ablation study demonstrates VALM is capable of extracting useful visual knowledge from retrieved object images and inferring correct semanties based that. Given retrieved object images in different colors, VALM could extract the correct color knowledge and adopt it in its semantic inference stage.
(a) Images for green apples in OBJJCTCOL.ORIZZATTI dataset as replacements.
class
0.200
VaLM
VaLM_ Colorization
0.175
0.150
0.125
prob
0.050
0.025
0.000
red
whiteorangegreen wiitterangeeree-yyloopppppp black blue brown pink grey color
(b) Probability Distribution Visualization for retrieved images and colorization images.
Figure 3: The visualization of the predicted probability distribution 11 object color types with retrieved images and colorization images, respectively. The adopted prompt for reasoning the object color of an apple is "the color of [object] is".
B EXPERIMENTAL DETAILS
B. PRE-TRAINING HYPERPARAMETERS AND TRAINING DETAILS
The implementation of models and all experiments are based on the fai rseq (Ott at. 2019 toolkit.
The proposed model deploys transformer decoder architecture with 124M trainable parameters, in
which Mayer 12, Thhead 12, dembed 768. We deploy Adam (Kingma & Ba 2015) (8)
0.9, B2 0.98) optimizer and train all models with Ir = 0.0005, t tarmuu 4000, dropout =
0.1, bsz 128. len 512. The layer normalization over the retrieved image keys is initialized with of 0.00001. VALM reuses the identical lower-cased byte pair encoding (BPE) (Sennrich et al. (016)) representation with 49152 vocab size of CLIP text encoder. The proposed VALM and re-implemented GPT-2" are trained for 500k steps using 16 Nvidia Tesla V100-SXM2-32GB GPUs. The encoded 200M image knowledge base takes up 274GiBs disk storage and the trained faiss approximate retrieval index takes another 14GiBs storage.
B.2 PROBE TEMPLATES
We present all zero-shot query prompts and labels for object commonsense reasoning datasets and 4 natural language understanding benchmarks in Tabele 11]
16

---

Published as a conference paper at ICLR 2023
Task
Prompt
Labels
What the color [DESCRIPTOR] [ITEM]? It [Label]
What the colour [DESCRIPTOR] [ITEM] ? A: is [Label]
What is the color of [DESCRIPTOR] [ITEM]] It [Label]
What is the colour of [DESCRIPPORI [ITEM]? [Label]
(red, white, orange, green,
Object Color Reasoning
The color of [DESCRIPTOR] (TTEM) is [Label]
blue, yellow, purple, black,
The usual color of [DESCRIPTOR] [ITEM] is [Label]
pink, grey, brown]
[DESCRIPTOR] [ITEM] usually has the color of [Label]
What usual color of [DESCRIPTOR] [TTEM]? [Lahel]]
What typical color [DESCRIPTOR] [ITEM]? [Label]
Is [ITEMA] larger than [ITEMB]? [Label]
[ITEM] can be shape [Label]
[cross, heart, octagon, oval, polygon, rectangle, rhombus, round, semicirele, square, star, triangle)
Object Shape Reasoning
[ITEM] has shape [Label]]
[ITEM] is shape [Label]
The shape [ITEM] can [Label]
The shape the [ITFM] [T.abel]
Is [TTEMA] larger than [ITEMB]] [Label]
Is [ITEMA] taller than [[TEMB]] [Label]
Object Size
Is [ITEMA] higher than [ITEMB]] [Tahel]
YYes, No)
Reasoning
[TEMA] is larger than [TTEMB], is true? [Label]
[TEEMA] taller [ITEMB]] true? [Label]
[ITEMA] larger than [ITEMB], is true? [Lahel]
SST-2 MPQA
Review: [Sentence] Sentiment: [Tabel]
(Positive, Negative) (Positive, Negative] (company, school, ... book] [worl,, ... technology]
Review: [Sentenee] Sentiment: [Label]
DBPedia
input:
[Sentence] type: [Label]
AGNews
input:
[Sentence] type: [[ahee]]
Table 11: The prompts and prediction labels used to query the model predictions on the zero-shot evaluation of 4 object commonsense reasoning and 4 natural language understanding benchmarks. The labels for DBPedia are (company, school, artist, athlete, politics, transportation, building, nature, village, animal, plant, album, film, book) and the labels for AGNews are (orll,, sports, business. technology).
17


Reviews:
[{"review_id": "x1wvct0mAUB", "reviewer": "Reviewer_FCDh", "summary": "", "questions": "", "limitations": "", "rating": 10, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "\nThis paper improves a language model's performance on pure language tasks about visual concepts by augmenting its internal representation with dynamically retrieved images.\n\n(motivation)\nGlobal context (external knowledge about entities, relations, etc.) has been incorporated into pre-trained language models (PLMs), but so far it has not been visual. This paper incorporates external visual knowledge into their PLM (called VaLM) to\n1) prevent VaLM from hallucinating inconsistent statements and\n2) give VaLM visual knowledge that is harder to obtain from text.\n\n(approach)\nVaLM is a standarder decoder style PLM, used like any standard PLM and trained on the CC-100 dataset (text only). The difference is that the second to last self attention layer is replaced with a Visual Knowledge Fusion Layer (VKFL). First the VKFL uses CLIP to retrieve K (4 or 8) relevant images from a database of 200M natural images (from LAION) where relevance is according to the text input VaLM has seen so far. Second the VKFL fuses these relevant images with VaLM's hidden state at the previous layer to produce the output hidden state - the tokens are the same, they just have additional visual knowledge.\n\n(experiments)\nResults show that\n1. When asked to complete text prompts about the typical color, shape, and relative size of objects, VaLM outperforms both PLMs and pretrained vision-language models by a large margin.\n2. VaLM outperforms baselines at pysical commensense QA (PIQA).\n3. VaLM is at parity with and sometimes better than baselines at traditional language understanding and language modeling tasks.\n4. Ablations show disabling image retrieval or retrieving random (not relevant) images hurts performance.\n5. Ablations also justify design decisions in the attention mechanism.\n\nVaLM improves language modeling performance by adding global image context internally.", "summary_of_the_review": "\nThe novelty, potential significance, quality, clarity, and potential reproducibility of this work are all high, so it is a very strong submission.\n", "strength_and_weaknesses": "\nStrengths\n===\n\nIncorporating visual background knowledge in a language only model makes a lot of sense and seems novel. Furthermore, VaLM is a simple and effective realization of the idea in practice.\n\nThe writing is clear and straightforward, without frills.\n\n\n\nWeaknesses\n===\n\nThere are couple areas where the model should have been evaluted:\n\n* There is limited analysis of the images retrieved by the retrieval module. Using the figure 1 example, the idea is that it will say the sky is blue because the retrieved images tend to be blue. Are the retrieved images consistent with its answer? That is, given that it says the sky is blue, does it actually retrieve images where the sky is blue? If the retrieved images are manually substituted with alternative images where the sky is green then does VaLM say the sky is green? It would be good to have a systematic evaluation of the retrieved images, but it would also help to simply provide examples.\n\n* A related concern is about the images recalled for each piece of a sentence. As I understand it, the visual knowledge fusion layer recalls a different set of K images for each sentence part (e.g. different images for \"The color\", \"The color of\", \"The color of sky\", etc.). How do the recalled images vary over the course of the sentence? Do they stay the same when non-semantic words like \"of\" are used? Do they change appropriately when semantic words like \"sky\" are used?\n\n* There is no evaluation of how this impacts the model's runtime. How long does a forward pass take when generating every word requires a kNN lookup?\n\n\nThere are also some points where the presentation could be clearer:\n\n* The motivation in 3.2 (\"object properties rarely appears in text corpora\") seems like it should also belong in the introduction. To me this is a key reason to expect this approach to be helpful.\n\n* The text doesn't mention what the \"Majority\" row in Table 4 means.\n\n\nFinally, here are some minor suggestions and comments:\n\n* The attention mechanism over K retrieved images is novel, but a fairly straightforward extension of the attention mechanism once one has already decided to retrieve relevant images. The major contribution is in retrieving relevant images.\n\n* Can VaLM be used as a vision-language model? Essentially, what if it was given vision and langauge tasks (e.g., VQA) and then the image retrieval module was replaced with a module that simply returned the image(s) associated with the VL task example? Would it perform well at the VL task? In general, does it treat the retrieved images as global / abstract context or does it also consider them as specific context local to those images?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "se5ACUT53z", "reviewer": "Reviewer_9KzG", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "To augment language models with relevant visual information, a visually-augmented language model (VALM) is proposed in this paper. The core idea is to retrieve relevant images by CLIP model and then fuse them to the second last Transformer layer of PLM.", "summary_of_the_review": "My main concern is about the lack of experimental comparison with previous methods and the efficiency problem. So my initial rating is borderline reject.\n\n---------- After rebuttal --------\nMy main concerns have been solved and I'd like to raise the score from borderline below to borderline above.", "strength_and_weaknesses": "Strengths:\n1. The research problem of how to inject visual information into language modeling for visual-demand tasks is interesting. The method is intuitive and novel to a certain degree.\n2. In many datasets, the performance of the proposed model outperforms compared methods/baselines \n\nWeaknesses:\n1. Indexing images, retrieving images and fusing image features to PLM all take extra computation and time as opposed to original PLM, especially retrieving I think. The analysis of additional training and inference time brought by the proposed method should be added.\n2. Lack of experimental comparison against recent visual-augmented language model works. For example, Vokenization and iACE.\n3. The database image quality matters for the retrieval and downstream task performance. But the ablation/analysis is missing. I mainly have two questions: (1) Does the dataset have to be an image-text dataset, such as ImageNet? Theoretically, it can also be an image-only dataset since CLIP is already trained. (2) Does the size of the database matter? How about we only take 100M, 10M, or even 1M data from Laion?\n4. Some unclear illustrations. (1) In Sec2.3, is z_ij the image feature output from the same CLIP image encoder used in retrieval? (2) In the 3rd last sentence of Sec3.4, shouldn't it be \"adopting image-specific bias outperforms directly sharing the bias\"?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "hJ-lqv3w8d", "reviewer": "Reviewer_j6aD", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work proposes a novel architecture to do visual-augmented language modeling. Before each prediction of the next word, this architecture queries the most relevant images w.r.t. the current received part of the sentence using a pretrained CLIP model and a large image database. The authors trained this model on a large text corpus. They then showed that this model outperforms other models on multiple visual-language benchmarks, including Memory Color, Object Shape, Relative Size, and ColorTerms, by noticeable margins. The authors further show some ablation studies that the use of the image-retrieval module is helpful.", "summary_of_the_review": "This work proposes a new architecture that achieves better performance than other models on the benchmarks they tested. The current version needs more justification about how real this improvement is and how innovative this architecture is.", "strength_and_weaknesses": "Strength:\n\nThis work proposes a new architecture for augmenting language modeling with auxiliary visual cues. The authors tested multiple evaluation datasets for their proposed and other candidate models. The improvement in these datasets compared to the other models tested in this work is significant. The authors also show some ablation studies that the proposed image retrieval module helps the performance.\n\nWeakness:\n\nThe biggest worry I have is about the evaluation in this work, which is critical to be resolved before I can fully back the acceptance of the paper. The pure language model tested in this work is a GPT-2 retrained by the authors on the same text corpus. And the other visual augmented models are all pretrained models from other papers. None of these comparisons can be perfectly fair on the training datasets. However, it seems that the model size is controlled, as this model uses a pretrained CLIP model during its training and is augmented by a large image database. If the training dataset can never be perfectly controlled, why not try models trained on much larger text corpus, such as the pretrained OPT models? Besides this, the numbers on these benchmarks seem to be also lower than the numbers I can find in other papers. For example, in the paper “Transferring Knowledge from Vision to Language: How to Achieve it and how to Measure it?” (about the Memory Color dataset), Table 4 shows numbers much higher than the numbers reported in this paper. Can the authors explain this difference?\n\nAnother minor issue about the evaluation is: what would be the current up-limit on these evaluation datasets from pure-language models? Have people tested the largest pretrained models on these datasets? Are these problems really hard for the models to resolve?\n\nTo help correctly evaluate the innovation of this work, can the authors also comment on how different this fusion layer proposed here is from the fusion layer in the Google Flamingo paper?\n\nThis is more of a question instead of an issue. It would be great to see how reducing the number of images in the auxiliary image database will influence the performance. I would imagine that having these images during the real-time inference makes the inference very slow, about which I cannot find any time estimation from the paper. So this could be an issue for real-world applications. To be clear, I don’t think this issue needs to be addressed in this work right now, but I want to get a sense of how annoying this is.\n\nFinally, it would be good to know how the scaling will influence the results. Will large models like 350M make the performance better?\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "8sqgxzaVdv", "reviewer": "Reviewer_f9V4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a pre-training framework, called VALM, to jointly train on image-text data. The novelty of this work, compared to previous works in similar field, is how the image-text pairs are created. While previous works use pre-curated image-text aligned pairs, this work instead uses images retrieved using text as a query and then jointly fuses them through attention layers. The claim is that this will help the model perform better on tasks requiring visual commonsense reasoning.", "summary_of_the_review": "Overall, quite an interesting and novel idea in the space of vision-language models. The claims are verified by empirical results. VALM outperforming on tasks compared to other vision-language models is a strong result. The weakness related to understanding why this method works better than other vision-language models and whether frozen image retrieval will limit the method's effectiveness is something the reviewer would like to understand and engage with the authors further.", "strength_and_weaknesses": "**Strengths**:\n\n-- The empirical results in the paper are quite strong and surpass pre-trained text models (GPT) as well as vision-language models (VisualBERT) on 4 reasoning task datasets and several language understanding/modeling tasks. While the differences in language-only tasks is small, the gains compared to baselines in visual reasoning tasks is large as claimed.\n\n-- The proposed model (esp visual knowledge fusion component) is novel and elegant and simple, which will serve as motivation for following works.\n\n-- The paper is clearly written and easy to follow.\n\n**Weaknesses**:\n\n-- The main weakness in the method is the use of frozen image retrieval component. The concern is that if this component is not end-to-end trained with the rest of the model, will the model quality be limited by the quality of image retrieval. While the reviewer acknowledges that empirical results show a large gap between CLIP and VALM on all tasks, it is worth wondering if the gains are due to embedding multiple images with text in VALM (compared to single image-text pair used originally in CLIP). An ablation with different values of k (number of images retrieved) will be helpful here.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster