Visually-augmented pretrained language models for NLP Tasks without Images
Authors: Hangyu Guo
Kun Zhou
Xin Zhao
Qinyu Zhang
Ji-Rong Wen
Venue: Submitted to ICLR 2023 (2023)
Keywords: Visually-augmented pretrained language models for NLP tasks without images

Abstract: Although pre-trained language models (PLMs) have shown impressive perfor-
mance by text-only self-supervised training, they are found lack of visual se-
mantics or commonsense, e.g., sizes, shapes and colors of commonplace objects.
Existing solutions often rely on explicit images for visual knowledge augmenta-
tion (requiring time-consuming retrieval or generation), and they also conduct the
augmentation for the whole input text, without considering whether it is actually
needed in specific inputs or tasks. To address these issues, we propose a novel
visually-augmented fine-tuning approach that can be generally applied to various
PLMs or NLP tasks, without using any retrieved or generated images, namely
VAWI. Specifically, we first identify the visually-hungry words (VH-words) from
input text via a token selector, where three different methods have been proposed,
including syntax-, attention- and learning-based strategies. Then, we adopts a
fixed CLIP text encoder to generate the visually-augmented representations of
these VH-words. As it has been pre-trained by visual-language alignment task on
large-scale corpus, it is capable of injecting visual semantics into the aligned text
representations. Finally, the visually-augmented features will be fused and trans-
formed into several pre-designed visual prompts based on VH-words, which can
be inserted into PLMs to enrich the visual semantics in word repersentations. We
conduct extensive experiments on ten NLP tasks, i.e., GLUE benchmark, Com-
monsenseQA, CommonGen and SNLI-VE. Experimental results show that our
approach can consistently improve the performance of BERT, RoBERTa, BART
and T5 at different scales, and outperform several competitive baselines signifi-
cantly. Besides, the generated visual prompts of our framework can also be used
for parameter-efficient tuning, which boosts the performance of T5-3B. We will
make our code, data, and models publicly available.

Under review as a conference paper at ICLR 2023
VISUALLY-AUGMENTER MODELS FOR NLP TASKS WITHOUT IMAGES
PRETRAINED
LANGUAGE
Anonymous authors Paper under double-blind review
ABSTRACT
Although pre-trained language models (PLMs) have shown impressive perfor- mance by text-only self-supervised training, they are found lack of visual se- mantics or commonsense, e.g., sizes, shapes, and colors of commonplace objects. Existing solutions often rely on explicit images for visual knowledge augmenta- tion (requiring time-consuming retrieval or generation), and they also conduct the augmentation for the whole input text, without considering whether it actually needed in specific inputs or tasks. To address these issues, we propose a novel iisualy-auumenter fine-tuning approach that can be generally applied to various PLMs NLP tasks, without using any retrieved or generated images, namely VAWI Specifically, we first identify the visually- words (VH-words) from input text via token selector, where three different methods have been proposed, including syntax-, attention- and learning-hased strategies. Then, we adopt a fixed representations of these
CLIP text encoder to generate the VH-words. As has been pre-trained by vision-language alignment task on the large-scale corpus, it is capable of injecting visual semantics into the aligned text representations. Finally, the visually formed into the pre-designed visual prompts based on VH-words, which can be inserted into PLMs to enrich the visual semantics in word representations. We conduct extensive experiments on ten NLP tasks, i.e., GLUE benchmark, Com- monsenseQA, CommonGen, and SNLI-VE. Experimental results show that our approach can consistently improve the performance BERT, RoBERTa, BART, and T5 different scales, and outperform several competitive basclines signifi- cantly. Besides, the generated visual prompts of our framework can also be used for parameter-efficient tuning, which boosts the performance of T5-3B. We will make our code, data, and models publicly available.
features will be fused and trans-
INTRODUCTION
Recent years have witnessed the success of pre-trained language models (PLMs), such as GPT- 3 (Brown et al., 2020) and T5 (Raffel et al. 2020), in a variety of natural language process (NLP) tasks. Since these PLMs are mostly trained on text-only corpus via self-supervised pre-training they have been shown lack of visual commonsense (Liu et al., 2022) and real-world knowledge (Zhang et al., 2022). As a result, PLMMs can't well solve visually related language tasks] e.g., answering the color and size of common things, especially those requiring complex commonsense knowledge.
To alleviate this problem, existing works mainly enhance PLMs by infusing the visual information. Typically, given text input, these studies firstly augment the visual information from retrieved or generated images about the input and then leverage their visual representations to improve PLMs on NLP tasks. Such approach leads to where they adopt either augmente pre- training (Tan & Bansal 2020 Wang et al. 2022) visually risueely-agmmented fine-tuning techniques (Lu et al. 2022). Despite the effectiveness, there two major shortcomings in these methods. First, it is very costly to retrieve or generate high-quality images that are paired with the input. These methods often rely on pre-lcarned complementary re- trievers or generators, and also require time-consuming inference for obtaining the images, which
pre-trdined language models ( (VaLMs),
In this work, we mainly consider the text-only NL.P tasks that may need the visual information as comple- mentary, but the visual-language tasks with images.

---

Under review as a conference paper at ICLR 2023
largely limits the applicabiliity of this approach. Second, as the augmented visual information comes from retrieved or generated images, it is inevitable to involve irrelevant or redundant visual informa- tion. If simply integrating the augmented visual information, the original text representations might be affected or even "polluted". Increasing evidence shows that the visual information is not always useful for NLP tasks (Dai al., 2022). and sometimes leads to performance degradation.
Considering these issues, we aim to develop a more efficient and effective way to visually augment the PLMs and the solution is twofold:
Firstly, we don't explicitly produce (retrieve or generate) the images but instead generate visually- aligned representations of the text on-the-fly. Recent studies (Radford et al., 2021] Jia et al. 202T) have shown that the vision-language pre-trained models (VL-PTMss) can well learn the alignment between the representations of texts and images from large-scale text-image pairs. Thus, our idea is to employ the output representations of a text from VL-PTMMs' text encoders as surrogate for the visual representations of related images.
Such way is simple and efficient: we can only keep the text encoder of VL-PTM to produce the visually-aligned representations of texts, getting rid of the complicated image retrieval or genera- tion process. It is widely recognized that there is a large semantic gap between different modali- ties (Liang et al. 2022). Our method can alleviate this issue to some extent since the visual augmen- tations are derived from the text representation itself, i.e., visually-aligned text representations from VL-PTMs.
Secondly, instead of directly feeding visual augmentations into the PLM, we propose to use the augmented visual information only when actually needs. In fact, for a text imput of NLP task, PLMs are not always hungry for the visual background knowledge it, especially for visually-irrelevant expressions. Unlike previous works which inject visual infor- mation into text (Tan & Bansal 2020; Wang et al., 2022) from the whole, we consider identifying visually-hungry words (those that require visual knowledge to derive complete semantics) from the text input, and only infuse the visual augmentations through these trigger words. We conduct vi- sual augmentations at the word level, because it is more flexible and controllable, considering the augmented information often irrelevant or noisy.
effectively understand
To this end, in this paper, we propose general Visuall-AAggenneee fine-tuning approach to improv- ing PLMs for NLP tasks Without Images, namely VAWI. Our approach consists of three ingredients, namely visually-hungry words extraction, visual knowledge augmentation, and esually-enhancee fine-tuning- Given the text input from a NI.P task, we first extract the visually-hungry words (VH- words) from the input sentence. As the annotations of VH-words propose three strategies to automatically extract the VH-words, relying on the syntax trees, attention distributions of the VL-PTMs' text encoder, and adaptive learnable module, respectively. Then, based on the extracted VH-words, we leverage the text encoder of CLIP (Radford et al.) 2021) (be- ing fixed in our approach), VL-PTM that has been pre-trained millions of text-image pairs, to encode the VH-words for obtaining their visually-aligned representations. Finally, we design visually-enhanced fine-tuning strategies to infuse the visually-aligned representations into PLMs. For small PLMs. we directly incorporate these visual representations to enrich the word embed- dings and fine-tune the parameters of the PLM and our approach. For large-scale PLMs, we also propose carameter-efficien prompt-tuning strategy that only tunes very few parameters in our approach, with the PLM being frozen.
generally unavailable, we
To summarize, our approach provides an adaptive, flexible and efficient way to leverage visual infor- mation for enhancing text-based PLMs. For verifying the effectiveness of our framework VAWI, we test it on four PL.Ms (i.e,, BERT, BART, RoBERTa, and T5) at different scales (i.e., 110M, 340M, 3B), and conduct extensive experiments in natural language understanding. commonsense reason- ing, and text generation tasks. Experimental results show that our VAWI can boost the performance of these PLMs significantly, Le.. 3.11%, 2.54%. and 2.16% absolute improvements on the com- monsenseQA task using RoBERTa-base, RoBERTa-large, and T5-3b, respectively. Besides, VAWI can outperform (or be on par with) several competitive baselines that adopt complicated visually- augmented methods. Additionally, VAWI further improves the performance of VL-PTM (i.e. ALBER et al. 2021 on the cross-modal reasoning task by enhancing its text-encoder.
2

---

Under review as a conference paper at ICLR 2023
2
RELATED WORK
In this section, we illustrate related work from the following three perspectives, namely pre-trained language models, vision-language pre-trained models and visually-augmented language model.
Pre-trained Language Models. Recent years have witnessed the success of pre-trained language models (PLMs) (Devlin et al. 2019; Radford et al. 2019 Lewis al. 2020 Raffel et al.) 2020)_ After pre-trained on large-scale corpus, PLMs can be fine-tuned on multiple NLP tasks and achieve remarkable performance. Typically, PLMs incorporate the Transformer architecture (Vaswani al.) 2017) and require large-scale corpus. With the increasing scale of the model parameters and pre- training corpus, PLMs can even achieve human performance (Devlin al., 2019 Radford et al. 2019). However, since PLMs are just pre- tre-trained with text-only data, they may sufter from the reporting bias problem (Gordon & Van Durme 2013 Paik et al. [021] Zhang al. 2022). where the frequency distribution of visual commonsense in the text may not reflect the real-world distribution of the commonsense. Existing works have also found that such a problem can not be addressed by enlarging the model or pre-training corpus (Paik al. 2021 Zhang et al.) (022)) In this work, we aim to reduce the influence of the reporting bias problem on PLMs by adding visual knowledge during fine-tuning.
Vision- Vision-Language Pre- Trained Models. To better accomplish the vision-language tasks, vision- language pre-trained models (VL-PTMs) (Su et al., 2019 Lu et al., 2019 Li et al. 2021) become hot point in recent years, which require large-scale maxge-ext pairs for pre-training. Existing VL- PTMs fall into two categories based on the way of modeling
language interaction. The first category of models (Su et 2019 Lu et al.) 2019 Li et 2001)) adopts an explicit vision-language interaction layer to fuse the text embeddings and image features using single-stream or dual-stream network structure. These models are more suitable for the tasks that need to capture fine-grained semantic interactions between vision and language (e.g., NLVR, VQA, and VE), and have achieved
remarkable performance on them (Li al. 2021 Alayrac et al. 2022). The second category of models (Radford et al. 2021 Dia et al.] (2021) | incoporates separate encoders to independently model the vision and language information, and relies pre-training tasks (e.g., cross-modal contrastive learning) to align their representations into the same space. Such a way is capable of producing enriched single-modal representations and has achieved remarkable performance on both vision and image-text retrieval tasks (Radford et al. 2021).
Visually-augmentt pre-trained language model (VaLM) (Wang et al. 2022) has become emerging research topic that aims to in- troduce the visual information into PLMs for alleviating the reporting bias problem. Existing VaLMs can be categorized into visually-augmented pre-training and fine-tuning methods. Visually- augmented pre-traie vre-training approaches (Tan & Bansal 2020; Wang et al., 2022) have to continually pre-train the PLMs with the visual information. They mostly retrieve the visual information related to input tokens or sentences from a large-scale images dataset and revise the masked language fine-tuning methods (Lu) et al. 2022) introduce the visual information into PLMMs during the fine-tuning stage. These methods also leverage the image retrieval or generation models augment the visual information and design a special fusion module to inject it into PLMs. However, cxisting VaLM approaches mostly need to retrieve or generate visual information for utilization. Such a way is not only time-consuming, but may also involve unrelated or noisy information into PLMs, leading to performance degradation. In this work, we aim to first detect the visually-hungry words from the text, and then utilize a VL-PTM to generate their visually-aligned representations without the usage of external images or generation models. As a comparison, our approach more flexible and efficient to leverage the visual information for enhancing text- PLMs.
Tisually-Augmente
Language Model.
model task for better capturing the visual semantics.
3 METHOD
In this section, we firstly introduce the task setting, and then describe our proposed visual augmen- tation approach for infusing visual knowledge into PLMS during fine-tuning.
3

---

Under review as a conference paper at ICLR 2023
Input: "He flies laughter.
Cader
trilokong
Wanaly arpuntttc Pepramnddii
Sestincal
Fond VL-PIN Toxt Earoder Igg
II
Reformulation Layer
Vinals Eshassed Word Enhnlligg
PLM
NLL Heal
hughhtrr
Input: "Hle flic kite with laughter"
Be
gate
Viaally sagannte
H
Syntax 2ased Straloyy
0illed
Layad VMM
The
Test Encolic
(itte)
Croa l tenfre
0
Inset ofte Cosital Froweiinge
Inightr
with
nows's FI-. towader
HI
sill pasiion
(lughhtd)
Gaabd Softrin
Hieallyyllaee
11.38
doadh-ssgeendd represbdionn
Loacing lased Stratgg
Houalyy-hugrr Words Extroction
Visual Knowledge Augmentation
Visualy-enhanred Vine-buning
Figure I: The illustration of our VAWI approach, consisting of hungry words extraction, visual knowledge augmentation and essuall-enhanee fine-tuning.
3.1 TASK SETTING AND SOLUTION OVERVIEW
This work aims improve the fine-tuning performance pre-trained language models (PLMs) on NLP tasks by leveraging the related visual information without images. For NL. task, set of labeled texts (<r, yi) are available, where r; is the i-th data consisting of sequence of words, denoted as z; = {wI, U22 .... Wm). and y, is the ground-truth output. Note that we don't specify the form of yi. which can be a discrete label (classification), continuous value (regression) or a text sequence (generation), since approach is generally applicable to various settings (either natural language understanding or generation tasks).
To solve the target task, assume that text-based PLM is given (either for understanding gen- eration). Let f denote a PLM parameterized by 9PLM that has already been pre-trained on general- purpose large-scale text data. We mainly focus the encoder of the PLM, which learns the input embedding via the "[CLS]" token. Given the labeled training data, we can train the PLM using specific loss function (e.g., cross-entropy loss) and further solve the target task. However, existing works TTan & Bansal 2020: Zhhag et al.) 2022) have revealed that PLMs may be unaware of vi- sual knowledge that is explicitly mentioned in the pre-trained text-only data (e.g., the shape of coins and the color of the sky), leading to the lack of world commonsense and generating wrong statements.
In this work, we focus devising efficient and effective way to infuse such visual knowledge into PLMs during fine-tuning. Our approach based on visually-hungry words (abbreviated as VH- words), which require the visual information to derive complete semantic representations. The over- all illustration of our approach is shown in Figure [ which consists three important ingredients, namely VH-words extraction, visual knowledge augmentation, and visually-enhanced fine-tuning. Given the input text z; and its label yı, we first detect and extract a set of VH-words. Then, we adopt visual knowledge augmentation module enhance the visual background knowledge of their to- kens and generate their visually-alignes representations. Finally, we infuse the visually-aligned text representations into the PLM to improve its fine-tuning performance, where consider both the general fine-tuning of small PLMS and the parameter-efftcient fine-tuning of large-scale PLMs.
3.2 VISUALLY-HUNGRS WORDS EXTRACTION
In our approach, visually-hungry words (VH-words) are the trigger units for visual augmentations, requiring visual knowledge for deriving complete semantic representations. For example, some visually-related words (e.g.. color, shape and object) may not be well learned by the text-only PLM. Therefore, we propose to first detect the VH-words from the input text, and then inject the proper visual knowledge that they are hungry for into the PLM. However, the annotations about Vo-words generally not available in NL. datasets. To address this problem, devise three different
4

---

Under review as a conference paper at ICLR 2023
strategies to extract the VH-words from the input text, including two feature- strategies based on syntax tree and attention distribution of PLMs, and a learnable model-based strategy.
Syntax-based Strategy. In natural language, entity words and descriptive words usually convey more visual semantics than others. For example, for the sentence "Ie is eating a green apple", where underlined words are more related to visual semantics. Such words are mostly nouns or adjectives in the input text, which can be detected by syntactic analysis. Therefore, we design a rule- based strategy that leverages the syntactic information of the words in s for VH-words extraction. Concretely, we first delete all stop words in text and then adopt an open-resource toolkit SPAcy convert the input text into a syntax dependency tree. Based on the syntax tree, we extract the words that have a particular part of speech (POS), e.g., nouns or adjectives, W(VH), In this way, we can efficiently extract the VH-words from input text by using fast parser toolkit. However, as such strategy mainly relies on the syntax information, may select some nouns adjectives that are not explicitly related to visual semantics (e.g, sweet and bitter). In the following, we will introduce two improved strategies to alleviate this problem.
the VH-words denoted by
Visually-enhanced Attention Based Strategy. The attention-based strategy utilizes the attention distribution of VL-PTM to detect the VH-words. Since VL-PTMs (Radford et al.) (022)) are pre- trained on large-scale image-text pairs, their text encoders are able focus more on the words cor- responding to some specific visual concepts in an image, which are likely to be VH-words. Inspired by this idea, can use the attention scores calculated by the text encoder of VL-PLMs to select the Worwords Specifically, we adopt the text encoder of CLIP (Radford et al.) 202I). a VL-PTM that has been pre-trained on millions of image-text pairs, to help extract the VH-words. As CLIP adopts an autoregressive GPT-2 model as the text encoder, we calculate the average attention scores between each token and the " "[EOS]" token on the last multi-head self-
layer, denoted as Swy Then, we select the top-K K ranked words according to {sw.} as the VH-words W((H), In this way, we can leverage the intrinsic knowledge encoded by the VL-PTM to extract VH-words.
Learning-based Strategy. Considering that diverse PLMs and NLP tasks may be hungry for the different complementary visual information, we devise learning-based strategy that can adaptively extract VH-words according to task requirements. Concretely, we add a parameterized VH-words extractor layer for the PLM, which can be updated by gradient-based optimization algorithms fit the need for some specific task. Given the imput text Iú we first leverage the PLM and a text encoder of VL-PPMM (i.e., CLIP (Radford al. 2021) produce the contextualized representations of the contained words in Ti- Then, we concatenate the representations of each word from the two models and utilize MLP layer to project into the score Swy
Su; MLP (") M))
where h and has are the output word representations from the PI.M and VL.PPTM, respectively, and scores Sw, are calculated by the learned model based on the supervision information from down- stream tasks. Based on the scores of all words, we incorporate the gumbel-softmax function (Jang ett al. (016)) to extract the top-k words as the VH-words in a differentiable way. In this way, the gradients of the fine-tuned tasks can be back-propagated to the extractor layer, which learns to adap- tively select the more suitable VH-words.
3.3 VISUAL KNOWLEDGE AUGMENTATION
Existing works (Lu et al. 2022 Wang et al. 2022) mainly utilize image retrieval or generation module to augment related visual knowledge. Such way is come-consuning and may also involve noisy images that will influence the task performance. Inspired by recent works that show the effective iisual-language alignment in VL-PTMs (Radford et al.) 202T Li et al., (02T), we utilize the visually-aligned text encoders as the visual augmentation representations of VH-words. As the text encoders have been aligned to the image encoders during pre-training on large-scale image-text pairs, their output textual representations can be used surrogates of visual augmentations based on real images related to the input text. As will be shown in experiments (Section 4), this approach is not only efficient but very effective for downstream NLP tasks.
https: httpss///ppayyyoop /
5

---

Under review as a conference paper at ICLR 2023
Based on the extracted VH-words, first add prefix text in the image caption style before the VH-words, e.g., "a photo of: to compose the input text x'. Then, we utilize the text encoder of CLIP (Radford al., 2021) to encode x' and obtain the contextualized word representations as the visually-aligned representations H E IR where l is the sequence length of x' and is the embedding size. To effectively reserve the visual semantics of VH-words,
obtain the visual representation of each VH-word in W)(H)) For example, given sentence "a green apple and a big orange", we enhance the visual knowledge of underlined words with visually-alignes representations H.z, respectively. Therefore, we incorporate a reformulation layer to aggregate the visually-aligned representation H., into the visually-augmented representations of these VH-words. However, the positions of the VH-words vary from sentence sentence. Therefore, nosition-awart attention mechanism in the reformulation layer to inject position information into H. for obtaining the visual representation of each VH-word via H Specifically, we leverage a soft position embedding matrix E e R to reserve the position information of VH-words, where I is and perform cross-attention between and the visual representations as:
design
the number of
Q- = E,
K= HK + V=
H= W bY,
QK Vd
H. = softmax
VV,
(3)
H (h1, h2. h2..... h
where h, E Rd, K, V Rexd H. E Rixd is the obtained aisuelly-uummented representations of VH-words, which can be leveraged for augmenting the visual knowledge of the PLM during fine-tuning. h, the visual representation of the i-th VH-word in WVCH), which are sorted by the position of VH-word in x'. Note that in Eq. 2 we only use the position information to set the query matrix Q, which is more efficient than standard self-attention (Vaswani et al. 2017). and the visual semantics are mainly captured and injected through the key and value matrices.
3.4 VISUALLY-ENHANNEE FINE-TUNING
representations of VH-words (i.e., H, in Eq 4 propose a visually-cnhanced fine-tuning strategy to inject the captured visual knowledge. Here, we consider two cases: (1) full-parameter fine-tuning for small PLMs, and (2) parameter-efficient prompt-tuning for large-scale PLMs. Before introducing the learning method, we simply review the parameters to consider, consisting of the parameters in the underlying PLM (denoted by p and the parameters from our approach (denoted by Oref). Note that we will always fix the text encoder from CLIP.
After obtaining the
Fine-tuning for Small PLMs. For small PLMs, we can perform full call-parameter fine-tuning, which updates both Optm and θref. Specifically, given the sosually-augmentiee representations H, of VH- words, we directly incorporate them into the embedding layer of the PLM. For a certain VH-word, insert its issuall-augmeened representation after the original word embedding, which leverages the visual semantics to enrich the word representations.
Prompt-tuning for Large-Scale PLMs. For large-scale PLMs, we fix the parameters in it, i.e., ppm and employ cirrameter-efficien prompt-tuning way to optimize it on downstream NLP tasks. Concretely, given the visually- augmentd representations H, of VH-words, we directly insert them before the key, value of every layer of PLM, respectively. Then, following the typical prompt-tuning paradigm (Li Liang 2021), we only tune the parameters of the reformulation layer (i.e., Oref) to control the soft prompts.
Our approach can be generally applied to various PLMs (c.g., BERT (Devlin et al. 2019). BART (Lewis et al. 2020), T5 (Raffel et al. (0200) and NLP tasks (natural language understanding and text generation). Unlike other complicated esuuall-augmenttr methods (Tan & Bansal 2020
Wang et al., 2022), it is more efficient, without the explicit need of external images generation model; and meanwhile, it only introduces a relatively small number of parameters (Eq. 2). which easier to learn.

---

Under review as a conference paper at ICLR 2023
Table 1: Performance comparison of different methods on NLU tasks, the BEST results are high- lighted in bold. +None denotes that we directly fine-tune the backbone without adding visual infor- mation. SBS, VABS, and LBS represent using the syntax- -bsed strategy, issuall-enhannee attention based strategy, and learning-based strategy in our approach, respectively. The results of VOKEN and iACE on GLUE are reported from Lu et al. (2022).
Base Model
Method
SST-2
QNLI
QQP
MNLI
MRPC
STS-B
Avg.
CLIP
+None
73.3
74.5
72.8
68.4
74.3
73.8
72.85
BLIP
+None
76.3
77.4
78.8
72.5
77.8
76.4
76.53
ALBEF144
+None
78.9
78.2
79.4
73.4
76.5
77.5
77.31
+None
89.3
87.9
87.2
79.4
81.7
84.4
84.98
+VOKEN
92.2
88.6
88.6
82.6
83.5
86.0
86.83
+iACE
91.7
88.6
89.1
82.8
85.8
86.6
87.43
BERT have
+VAWI-SBS
92.9
88.4
89.6
82.2
85.5
86.9
87.58
VAWI-VABS
92.7
88.9
89.5
82.7
85.8
87.2
87.80
VAWI-LBS
92.4
89.1
89.7
83.0
85.6
86.9
87.78
+None
89.2
87.5
86.2
79.0
81.4
85.4
84.78
+VOKEN
90.5
89.2
87.8
81.0
87.0
86.9
87.06
+iACE
91.6
89.1
87.9
82.6
87.7
86.9
87.63
RoBERTabce Boose
+VAI-SBS
91.4
89.4
87.7
82.2
88.2
87.7
87.76
VAWI-VABS
91.7
89.1
87.9
82.6
88.3
88.1
87.95
VAWI-LBS
91.6
90.6
87.9
82.4
88.5
88.3
88.21
4 EXPERIMENTS
4.1 EXPERIMENTAL SETUP
Datesets. We conduct experiments four types of NLP tasks. (1) Nature Language Understand- ing (NLU) task: we extract datasets from the popular GLUE benchmark (Wang et al., 2018); (2) Commonsense reasoning task: we select CommonsenseQA Talmor et al., 2019). 5ay multi- ple choice QA dataset that requires commonsense knowledge; (3) Text generation task: we select CommonGen (Lin al., 2019b), constrained text generation task about generative commonsense reasoning. (4) Cross-modal reasoning task: we select SNLI-VE (Xie et al. 2019), to evaluate the capacity of predicting whether the image semantically entails the text.
4.2 MAIN EXPERIMENTAL RESULTS
In this part, we conduct series of experiments on NLU, commonsense reasoning, text generation, and cross-modal commonsense reasoning tasks.
Evaluation NLU Tasks. We present the experimental results of different methods 6 NLU tasks in Table T First, we observe that VL-PPT's perform worse than PLMs, a possible reason is that they have been continually pre-trained on large-scale image-text pairs, which may cause the catastrophic forgetting problem. Second, VaLMs (i.e., VOKEN, iACE ,nd VAWI) achieve better performance over PLMs. As VaLMs infuse external visual knowledge into the PLMs, they can help the PLMs better understand the background knowledge of some words (e.g., color, shape, and size of objects). Between the two VaLM baselines, iACE is slightly better. This is because iACE is enhanced based on VOKEN and incorporates an image generation model, so it produces more visual information to utilize. However, the generated images inevitably contain noise and redundant information, which limits the performance gain of iACE.
Finally, by comparing our approach with all baselines, it obvious that VAWI performs consistently better than them on the six datasets. In our approach, we adopt efficient and effective way that augments the words from the input text. Benefiting from pre-training encoder of CLIP has been well aligned with the semantic space of images, so that it can generate high-quality augmente representations of the VH-words enrich them. Such way not
representations using the text encoder of CLIP to encode the VH-
large-scale image-text pairs, the text

---

Under review as a conference paper at ICLR 2023
only saves the costs of time and computation but also reduces the influence of inevitable noise from retrieved or generated images. is worth noting that our approach only needs to add 5.09M learnable parameters to the RoBERTa-base, far less than the 192.37M learnable parameters of iACE, but achieves better performance. Additionally, among three VH-words extraction strategies, LBS slightly outperforms others in most NLU tasks. The reason is that LBS incorporates learnable model-based strategy to select the VH-words. Such a way can adaptively extract proper VH-words with the consideration of the intrinsic knowledge of the PLMs. However, LBS will increase the computation cost due to its involved learnable VH-words extractor layer. Therefore, for efficiency, in the following experiments, we utilize the SBS strategy in our approach for comparison.
Evaluation on Commonsense Reasoning Tasks. Following existing works (Lin et al. 2019a), also rely on rule-based strategy to extract the examples containing visible objects, to construct a new dataset called CommonsenseeQA-33
It consists of 2,903 and 341 examples in the training set and dev set, respectively. In this way, we can ensure that most of the instances in this dataset require the visual information to enrich the commonsense knowledge. Based on the CommonsenseeA and CommonsenseQA-3K, we also report the results with different amounts of training data, to further evaluate the performance of different methods in the few-shot setting..
We show the experimental results of different methods on the commonsense reasoning tasks in Table 2 We can also see that with the help of the visual information from either retrieved images or our VAWI-SBS, the performance of PLMs can be improved significantly. It indicates that the visual information is indeed helpful improve PLMs for understanding commonsense knowledge. Besides. our approach outperforms the method using retrieved images from search engines. Our approach omits the image retrieval process due to its inevitably involved noise, and relies on the text encoder of CLIP to augment the visual representations. Such way can guarantee the relevance between the augmented visual knowledge and the text input, reducing the influence of retrieved noisy images and redundant information. Furthermore, we also perform parameter-efficient tuning on T5-3B-encode with our approach and boost its performance. It shows that our approach is able to be applied to large-scale PLMs to meet their thirst for visual information.
Table 2: Performance comparison on CommonsenseQA-33 and CommonsenseQA with different amounts of training data. We report the average performance on the dev set over three runs, and the BEST results are highlighted in bold. +Images denotes that we add retrieved images about the VH-words using web search engines, and encode them via CLIP-VIT.
CommonsenneoQA-3k
CommonsensecQA
Base Model
Method
5%
10%
20%
100%
5%
10%
20%
100%
NNone
41.88
46.04
50.58
61.88
44.88
50.04
57.08
67.90
RoBERTa
+Images
42.37
48.09
52.81
64,22
45.72
51.17
58.96
69.64
+VAWI-SBS
42.94
49.27
53.97
65.10
46.51
52.44
59.87
71.01
+None
48.39
56.30
59.06
74.19
51.24
59.95
65.52
76.65
RoBERTatarge
+Images
49.55
57.78
61.29
75.61
52.18
60.93
66.08
78.39
+VAWI-SBS
50.27
58.17
62.22
76.54
52.98
61.97
67.40
79.19
+None
70.16
73.02
75.04
81.81
71.99
75.27
77.72
82.40
T5-3B
+Images
70.96
73.60
75.91
82.40
72.87
76.17
78.71
83.64
VAWI-SSS+PTT
71.52
74.19
76.49
83.61
73.58
73.58
79.66
84.56
Evaluation on the Text Generation Task. As shown in previous experiments, it is useful to improve the performance of VAWI on commonsense reasoning and nature language understanding tasks. Here, we would like to study the effectiveness of our approach on the text generation task (i.e., CommonGenn using large PLMs. As shown in Table our model VAWI also consistently boosts the performance of BART-Large and T5-3b among all metrics. It further shows that our approach can also improve PLMs on the text generation task. As comparison, images are not very helpful and even cause performance degradation. The reason may be that the text generation task is more sensitive the inevitable noise from the retrieved images. Finally, the parameter-efficien tuning strategy of our approach also achieves comparable performance with the full-parameter tuning. It indicates that our parameter-efficien strategy is able efficiently optimize
can see that the retrieved
8

---

Under review as a conference paper at ICLR 2023
Table 3: Performance comparison on CommonGen. We also show the performance of parameter- efficient tuning of our approach, denoted as +PET. The BEST results are highlighted in bold.
Method
Base Model
BLUE-3
BLUE-4
METOR
Rouge-L
CIDER
SPICE
+None
42.80
32.42
31.36
57.57
16.56
32.94
+Images
42.67
32,67
32.12
57.46
16.78
32.81
BART-large
VAWI-SBS
44.56
34.17
32.47
58,46
17.23
33,67
WWWI-SBS+PET
43.12
33.76
32.20
58.12
16.91
33.17
+None
45.92
35.92
33,02
58.57
17.71
33.51
+Images
45.69
35.50
33.55
58.94
17.51
32.91
T5-3b
+VAWI-SBS
47.67
37.54
33.41
59.94
18.34
34.67
+ +VWWI-SBS+PTT
47.40
37.36
33.71
59.78
18.18
34.17
the parameters of large-scale PLMs, and shows promising future apply our approach to much larger PLMs, e.g., GPT-3.
Evaluation on the Cross-modal Commonsense Reasoning Task. To verify the generality of our method, we further evaluate the effectiveness of our VAWI improving VL-PTM (ie., LL jet al. (02TT), and conduct experiments on a cross-modal reasoning dataset, SNLI-VE. Concretely, implement approach on ALBEF by inserting the the VH-words embeddings of the text encoder before the multimodal encoder, and keeping others unchanged. As shown in Table 4 our VAWI also improve the performance of ALBEF using different amounts of training data. It further shows the generality of our approach in VL-PTMM, as also provide rich information to enhance the text encoder of VL-PTM, helping it better perform cross-modal reasoning.
representations after
Table 4: Results the test set of SNLI-VE task. The BEST results are highlighted in bold.
SNLI-VE
Method
10%
20%
50%
100%
ALBEF
65.46
67.52 68.23
75.47
ALBEF+ VAWI +SBS
65.94
76.14
81.64
CONCLUSION
In this paper, proposed a general aisually-augmented fine-tuning approach that be applied to a variety PLMs and NLP tasks, without using any retrieved or generated images, namely VAWI. Specifically,
first identified and extracted the visually-hungry words (VH-words) from input text via a token selector, where three different methods have been proposed, including syntax-, attention- and learning-based strategies. Then, we adopted fixed CLIP text encoder to generate the aisueeey-augmented representations of these VH-words. As it has been pre-trained by visual- language alignment task on large-scale corpus, it is capable of injecting visual semantics into the aligned text representations. Finally, we transformed the ssuulll--ugmenned features into several pre-designed visual prompts based on VH-words, and inserted them into PLMs to enrich the visual semantics of word representations in PLMs. Extensive experiments conducted on 10 NLP tasks, i.e., GLUE benchmark, CommonsenseQA. CommonGen, and SNLI-VE show that our approach can consistently improve the performance of BERT, RoBERTa, BART, and T5 at different scales, and outperform several competitive baselines significantly. Besides, the visual prompts our framework also be used for parameter-efficient tuning, which boosts the performance of T5-3b.
In the future, we will investigate more effective approaches to extract hisual-hungry words from the input text, and incorporate more effective VL-PTMs augment the visual representations. Besides, we will also devise more effective augmente pre-training strategies without images, to further improve the performance of PL.ss.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Cean-Baptisst Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katie Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. arXiv preprint arXiv:2204. 14/98, 2022.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
Wenliang Dai, Lu Hou, Lifeng Shang. Xin Jiang, Qun Liu, and Pascale Fung. Enabling multimodal generation on clip via vision-language knowledge distillation, arXiv preprint arXiv:2203.06 06386, 2022.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep
bidirectional transformers for language understanding. In NAACL, 2019.
Jonathan Gordon and Benjamin Van Durme. Reporting bias and knowledge acquisition. In Pro-
ceedings of the 2013 workshop on Automated knowledge base construction, pp. 25-30, 2013.
Eric Jang, Shixiang Gu, and Ben Poole. Categorical reparameterization with gumbel-softmax. arXiv preprint arXiv: /61 144, 2016.
Chao Jia, Yinfei Yang. Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun-Hsuan Sung. Zhen Li, and Tom Duerig. Scaling up visual and vision-language representation learning with noisy text supervision. International Conference on Machine Learning, pp. 4904 4916. PMLR, 2021.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. Bart: Denoising training for natural language generation, translation, and comprehension. In ACL, 2020.
pre-
Junnan Li, Ramprasaath Selvaraju, Akhilesh Gotmare, Shafiq Joty, Caiming Xiong, and Steven
Chu Hong Hoi. Align before fuse: Vision and language representation learning with momentum distillation. Advances in neural information processing systems, 34:9694-9705, 2021.
Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language- image pre-training for unified vasion-language understanding and generation. arXiv preprint 2XXv:2201.12088 2022.
Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation. arXiv preprint wwXhv:2101.001900 2021.
Weixin Liang, Yuhui Zhang, Yongchan Kwon, Serena Yeung, and James Zou. Mind the gap: Un- derstanding the modality gap in multi-modal contrastive representation learning. arXiv preprint arXiv:2203.02053, 2022.
Bill Yuchen Lin, Xinyue Chen, Jamin Chen, and Xiang Ren. Kagnet: Knowledge-aware graph networks for commonsense reasoning. arXiv preprint 100iv:190,.02151, 2019a.
Bill Yuchen Lin, Wangchunshu Zhou, Ming Shen, Pei Zhou, Chandra Bhagavatula, Yejin Choi, and Xiang Ren. Commongen: A constrained text generation challenge for generative commonsense reasoning, arXiv preprint arXiv: 19110337 2019b.
Xiao Liu. Da Yin, Yansong Feng, and Dongyan Zhao. Things not written in text: Exploring spatial commonsense from visual signals. arXiv preprint arXiv:2203.08075, 2022.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zeullemoyer, and Veselin Stoyanov. RoBERTa: A robustly optimized bert pretraining approach. ArXiv, abs/1990..16e 11692, 2019.
Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee. Vilbert: Pretraining task-agnostic visiolin- guistic representations for vision-and and-language tasks. Advances neural information processing systems, 32, 2019.
10

---

Under review as a conference paper at ICLR 2023
Yujie Lu, Wanrong Zhu, Xin Eric Wang, Miguel Eckstein, and William Yang Wang. Imagination- augmented natural language understanding. arXiv preprint arXiv:2204.08533, 2022.
Cory Paik, Stéphane Aroca-Ouellttte, Alessandro Roncone, and Katharina Kann. The world of an octopus: How reporting bias influences a language model's perception of color. arXiv preprint arXiv:2110.08182. 2021.
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. 2019.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, pp. -763. PMLR, 2021.
Colin Raffel, Noam M. Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with unified text-to-text transformer. ArXiv, Abs/1910.10068 2020.
Weijie Su, Xizhou Zhu, Yue Cao, Bin Li, Lewei Lu, Furu Wei, and Jifeng Dai. VI-bert: Pre-training of generic visual-linguitn representations. arXiv preprint 1rXiv:1908.08550 2019.
Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. Commonsenseqa: A question answering challenge targeting commonsense knowledge. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguisties: Human Lan- guage Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume I (Long and Short Papers). pp. 4149-4158, 2019.
Hao Tan and Mohit Bansal. Vokenization: Improving language understanding with contextualized, visual-grounded supervision. arXiv preprini arXiv:2010.06775, 2020.
Ashish Vaswani, Noam M. Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. In NIPS, 2017.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. Glue: A multi-task benchmark and analysis platform for natural language understanding. arXiv preprint arXiv:1804.074555 2018.
Weizhi Wang, Li Dong, Hao Cheng, Haoyu Song, Xiaodong Liu, Xifeng Yan, Jianfeng Gao, and Furu Wei. Visually-augmentedd language modeling. arXiv preprint arXiv:2205. 10178, 2022.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowiez, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, 2020.
Ning Xie, Farley Lai, Derek Doran, and Asim Kadav. Visual entailment: A novel task for fine- grained image understanding. arXiv preprint 1901:0190 2019.
Jianwei Yang. Chunyuan Li, Pengchuan Zhang, Bin Xiao, Ce Liu, Lu Yuan, and Jianfeng Gao. Uni- fied contrastive learning in image-text-labed space. In Proceedings of the IEEE/CVF Conference Computer Vision and Patter Recognition, pp. 19163-19173, 2022.
Chenyu Zhang, Benjamin Van Durme, Zhuowan Li, and Elias Stengel- -skknn. Visual commonsense in pretrained unimodal and multimodal models. arXiv preprint arXiv:2205.0185. 2022.
11

---

Under review as a conference paper at ICLR 2023
A EXPERIMENTAL SETUP
Implementation Details. We implement all baselines and our method based Huggingface Trans- formers (Wolf et al. 2020). For all baselines, we set their papers. In our approach, we leverage the text encoder of CLIP (ViT-B/32) to implement the learnable model-based VH-words extractor and generate the visual representations of VH-words in the visual knowledge augmentation module. The hidden size of visual representations is set 512. For differ- ent NLP tasks, we tune the number of visually hungry words in {2. 3,4, 5). During fine-tuning, we perform parameter-efficient tuning on T5-3b and BART-Large, and full-parameter tuning on other PLMs. For all tasks and all backbones, we utilize Adam as the optimizer, set the learning rate to 2e-5, weight decay to 0.01, and linear warmup for the first 6% steps. For GLUE, GommonGen, and SNLI-VE datasets, we fine-tune our model for epochs with a batch size of 32. For Common- senseQA, we tune our model for epochs with a batch size of 32. We use the cross-entropy loss for classification and mean squared error loss for regression. All experiments are conducted on V100 GPU.
according to the original
Baseline Models. We compare approach with the following baselines, including pre- trained language models (PLMs), visual-language pre-trained models (VL-PTMss) and visually-augmented
pre-trained language modes (V)LMs). (1) PLMs: We choose BERT (Devlin et al. (019).
RoBERTa (Liu et al. 2019), BART (Lewis et al. 2020). T5 (Raffel et al., 2020) as the PLM back-
bones, and directly fine-tune them as baselines. (2) VL-PTMs: We select ALBEF (Li et al. 2021),
BLIP (Li et al. (022). and CLIP (Radford et al. 2021 as baselines, which have been tre-traned on large-scale image-text pairs via cross-modal contrastive learning. (3) VaLMs: we select VO- KEN (Tan & Bansal 2020) and iACE (Lu et al. 2022), which introduce the visual information into PLMs by pre-training on retrieved images and fine-tuning on generated images, respectively.
B ABLATION STUDY
B. ABLATION STUDY ON VISUAL KNOWLEDGE AUGMENTATION
In this part, we conduct series experiments to verify whether the improvement of our approach derives from the augmented visual knowledge about the VH-words.
The Effect of the Source of Visual Representations. We first propose three variants that incor- porate powerful PLMs, iee,, RoBERTa-base, T5-Large, and T5-3b respectively, to replace the text encoder of CLIP in our framework. We also replace the generated visual representations from the text encoder of CLIP with random noise, to investigate the importance of the visual representations. As shown in Table 5 we can see that our approach is better than all the variants, even T5-3b with billion-scale parameters. It indicates that CL.IP-base more effective to augment visual knowledge to improve the performance of PLMs. Besides, our approach also outperforms the variant using random noise as the visual representation, showing the worse performance among all the variants. It also shows the importance of visual representations, as they indeed contain the visual knowledge that the PLM is hungry for.
Table 5: Performance comparison of different sources of visual representation in our approach. The base model is RoBERTa-base.
Source of visual representation (Params)
CSQA-3k
CSQA
SST-2
QQP
STS-B
QNLI
Random Noise (0M)
61.59
66.78
89.13
86.27
85.13
87.22
RoBERTa-large (355M)
61.18
67.17
89.43
86.53
85.60
87.77
TS-large-encoder (375M)
62.21
67.87
89.71
86.67
86.40
87.94
3b-encoder (1500M)
63.10
68.42
90.24
86.96
86.93
88.21
CLIP- base (52M)
65.10
71.07
91.41
87.72
87.67
89.40
The Effect of the Stronger VL-PTMss our work, we choose CLIP-base to enhance PLMs, as it has been pre-trained large-scale image-text dataset. Generally, a stronger VL.-PTM would
12

---

Under review as a conference paper at ICLR 2023
be more promising to further improve the performance. Here, replace CLIP-baae model with some stronger VL-PTMs, e.g,, ALBEF (Li et al. 2021). UniCL-base Yang et al. 2022). and CLIP-large. Concretely, ALBEF leverages more pre-traiming tasks (e.g., MLM, ITM. and ITC), UniCI. utilizes more high-quality pre-training data, and CLIP-large increases the scale of model parameters. We evaluate the above variations on CSQA-3K, QQP, and SST-2, and the results shown in Table [ We can see that UniCL and CLip-large outperform CLIP-base. It indicates that the VL-PTMs with larger scale of model parameters or more high-quality pre-training data are more capable of augmenting useful visual knowledge for PLMs. Considering the efficiency, CLIP- base is also a good choice in our approach, and we will investigate more proper VL-PTMs in the future.
Table 6: Performance comparison of visual representations from different VL-PTMs in our ap- proach. The base model is RoBERTa-base.
The text encoder of different VL.PPMMs (Params)
CSQA-3k
SST-2
QQP
Random Noise (0M)
61.59
89.23
86.21
ALBEF (110M)
63.34
90.72
87.17
CLIP-base (52M)
65.10
91.41
87.72
UniCL-base (52M)
65,98
91.75
88.07
CLIP-large (123M)
66.27
92.10
88.31
The Effect of the Pre-trained Dataset of VL-PTMs. We notice that the pre-training dataset of VL-PTMs is different from PLMs. Here, investigate whether the captions or images from the large-scale image-text pairs contribute more to the performance gain of our approach. To verify it, pre-train new PLM only using the captions data. Following the setting of ALBER, the pre-trained parameters of BERT to initialize this model and only extract the captions from the pre-training data of ALBEF (totally 14.5M sentences). After pre-training convergence, we utilize this model to replace CLIP-base in our approach and keep other settings unchanged. We conduct experiments on commonsense reasoning and NLU tasks evaluate its ef- fectiveness for augmenting visual knowledge. As shown in Table 7 we can see that such variation underperforms ALBEF and our approach, and even leads to performance degradation on CSQA task. It indicates that during pre- pre-training the image data is an important resource for learning visual knowledge VL-PTMs. Only text data (i.e., captions) can not provide sufficient visual knowledge that PLMs are hungry for. Therefore, after pre-learned on large-scale text-image pairs, CLIP can absorb the useful visual knowledge from the images and inject them into PLMs in our approach. further indicates that the improvement of our method is due to the involvement of the visual infor- mation about the VH-words.
utilize
these captions until
Table 7: Performance comparison of visual representations pre-trained using different pre-training data in our approach. The base model is RoBERTa-base.
The text encoder of different VI.-PTMss (Params)
CSQA-3k
CSQA
SST-2
STS-B
MNLI
None
61.59
67.90
89.23
85.46
79.06
BERT pre-trained captions (110M)
62.17
67.56
89.58
85.73
79.24
ALBEF (110M)
63.64
68.47
90.72
87.17
80.86
CLie-base (52M)
65.10
71.07
91.41
87.73
82,27
B.2 ABLATION STUDY ON VISUALLY-EHHANCED FINE-TUNING
Different Insertion Positions of Visual Representations. In our visually-enhanced fine-tuning framework, we insert the visual representation of the VH-word after its original word embedding. propose three variants of it that do not insert, insert all visual repre-
To verify its effectiveness, sentations of VH-wordd before and after the input text, respectively. As shown in Table g we can observe that all these variants would lead to performance decrease. It demonstrates that proper position to insert the visual representation is important for the utilization of augmented visual rep- resentations. By inserting them after the word embeddings of corresponding VH-words, PLMs can
13

---

Under review as a conference paper at ICLR 2023
effectively aggregate the visual representations to enrich the word representations, leading to better performance on downstream NLP tasks.
Table 8: Performance comparison w.r.t. different insertion positions of visual representations. The base model is RoBERTa-base.
CSQA-3k
Insert Positions
5%
10%
20%
100%
Not insert
41.88
46.04
50.58
61.88
Before input text After input text After the VH-words
39.77
44.86
57.47
40.23
45.67
58.08
42.94
49.27
53.97
65.10
C FURTHER ANALYSIS
The Number of VH-words. Our approach has an important such the number of VH-words. VH-words can supply visual knowledge that PLMs may be hungry for. Here, we
required to tune,
would like to study whether more VH-words are better improve performance. We conduct experiments on the
CSQA-3k
QOP
QQP and CSQA-3K datasets using RoBERTa-base as the backbone, and present the results in Figure 2 We can see that with the increase of the number of VH-words, the performance gain of our approach first increases and then decreases. A possible reason is that too many VH-words may also introduce noisy redundant information (e.g., not very relevant words), which would also influence the fine-tuning performance. Instead, it is also more efficient to select few VH-words (e.g.. two words for CSQA-3k) for deploying our approach in large-scale PLMs.
Accuracy Gain
2.0
1.5
1.0
2
VH-words Numbers
Figure 2: Performance comparison w.r.t. different numbers of VH-words.
The Computation Latency of the Proposed Methods. In our VAWI, we fix the model parameters of CLiP-base to preserve the visual knowledge. Such a way can also
decrease the computation costs during training and inference. To verify we report the mean training and inference latency batch on the CSQA-3k dataset of our method and baselines on RTX3090 GPU, where all these methods utilize RoBERTa-base as the backbone. As shown in the Table 9 can that our proposed VAWI-SBS and VAWI-ABS would not increase the latency too much. For VAWI-LBS, as requires a PLM and a VL-PTM to adaptively select the VH-words, it will relatively increase the much latency. As shown in Table achieve comparable performance in NLU datasets. Therefore, it is more efficient and effective to select the SBS and ABS variations in our approach. Despite it, we can see that all our variants own less latency than iACE, since our approach does not require a time-consuming image generation process. And shown in Table T approach can also achieve better performance.
that all the three variants
Table 9: The computation latency during training and inference.
Computation Costs
RoBERTa-base
+ +VWI-SBS
+VAWI-BS
+ +VAI--LBS
+Voken
+iACE
Training Time (s)
0.506
0.587 0.241
0.680
0.893
0.506
1.138
Inference Time (s)
0.182
0.308
0.486
0.182
0.512
The Effeet of the Improper Visually-hungry Words. To analyze how the quality of the VH-words CSQA-3K and
affects the performance of our approach, we further conduct the experiments two NLU tasks SST-2 and QQP from GLUE, to show the effect of insufficient VH-words on our model performance. After extracting the VH-words, remove part of them and only randomly
14

---

Under review as a conference paper at ICLR 2023
that with the decreasing of the sampling probability, the performance of our approach degrades gradually. indicates that not enough VH-words would degrade the performance of our approach.
sample 0%, 20%, and 50% VH-words for augmentation. As shown in Table 10
Table 10: The effect of the improper visually-hungry words. The base model is RoBERTa-base.
The proportion of correct visual words
CSQA-3K
SST-2
QQP
0 %
61.60
89.57
87.63
20 %
62.17
89.44
87.40
50 %
64.22
91.73
89.20
100
65.10
92.93
89.74
None
61.88
89.23
86.21
The Interpretability of Augmented Embeddings. In this part, we show how our augmented em- beddings infuse visual knowledge into the PLM. Concretely, we show the attention distributions of PLM (i.e., RoBERTa-base) in the last few layers before and after infusing iisuall--ugmeentr representations on CSQA. As shown in Table T1 we can see that the [CISS] tokens pay more at- tention to the visually-hungry words (VH-words) and their the VH-words also pay more attention to their Assualy-augmentee representations, shows that representations indeed provide useful knowledge, which guides the
representations, and
the injected PLM to focus on more important tokens and also improves the representations of the Wo-words and the [CLS] token.
Case Study of Extracted Visually-hungry Words. In this part, we show the visually-hungry words (ordw)rds) extracted by syntax-, attention- and seaming-based strategies in the Table [2] Table [3] Table 14 and Table 15] We can see that the three strategies would extract slightly different VH- words. The reason that the three strategies are based on different techniques identify the VH- words. As we can see, the cases show that most of the extracted VH-words by our strategies are generally related to some visual semantics, e,g, spider, two eyes. Although such VH-words can not perfectly cover all the visual semantics, they actually contain most of the important words that the PLMs may be hungry for, e.g., red and yellow. Besides, we can also see that the VH-words extracted by three strategies may perfectly align with human judgment. In fact, it is also hard for humans to determine proper rules to identify VH-words, egg., people, human, and water. In addition, as the learned knowledge of PLM a black box, is also difficult for humans judge the usefulness of our extracted VH-words for PLMs.
15

---

Under review as a conference paper at ICLR 2023
Table 11: The attention maps of the self-attention layers RoBERTa-base and our approach.
RoBERTa-Base
VAWI
Self-Attention, Layer12, Head5
Self-Attention, Layer12, Head5
uno
10
Whem
Wheee
0.55
0.35
(hrel
human
0.33
hunnan
luman
hol
0.25
0.25
dip
ipight
catch
water
0.15
0.5
sutly
0.13
diy
0.0
0.5
waterb
[SE7)
Self-Attentonn, Layer2, Head8
Self-Attention, Layer12, Head8
wren
gic.
valte
See
bay
522)
Where
W7em
rvg
haman
0.8
human
hid
hold
to
water
-3
sumyy
car
-31
Naterta
Inateraal)
Table 12: The first instance from the CommonsenseQQ dataset. The extracted visually-hungry words are highlighted in green.
Input
Input sentence: Unlike spider and his many sight seers, people only have what? eyes.
Syntax-based Strategy
Unlike spider and his many sight seers, people only have what? two eyes
Attention-based Strategy
Unlike spider and his many sight seers, people only have what? two eyes.
Attention-bann Strategy
Unlike spider and his many sight seers, people only have what? two eyes.
16

---

Under review as a conference paper at ICLR 2023
Table 13: The second instance from the CommonsenseeQA dataset. The extracted risually-hungr words are highlighted green.
Input
Input sentence: Where on a river can a human hold a cup upright to catch water on sunny, clear day? waterfall.
Syntax-based Strategy
Where on a river can a human hold cup upright to catch water on a sunny, clear day? waterfall.
Attention-basce Strategy
Where on a river can a human hold cup upright to catch water on a sunny, clear day? waterfall.
Attention-base Strategy
Where on a river can a human hold cup upright to catch water on a sunny, clear day? waterfall.
Table 14: The instance from the SST-2 dataset. The extracted risually-hunge words are highlighted in green.
Input Input sentence: the mesmerizing performances of the leads keep the film grounded and keep the audience riveted.
Syntax-based Strategy
the mesmerizing performances of the leads keep the film grounded and keep the audience riveted.
Attention-based Strategy
the mesmerizing performances of the leads keep the film grounded and keep the audience riveted.
Attention- based Strategy the mesmerizing performances of the leads keep the film grounded and keep the audience riveted.
Table 15: The instance from the QQP dataset. The extracted visually-hungry words are highlighted in green.
Input
Input sentence: How do sell dry Moringa leaves powder Indian market? Can use the moringa leaves that are already starting to turn yellow or yellowish?
Syntax-baser Strategy
How do sell dry Moringa leaves powder Indian market? Can use the moringa leaves that are already starting to turn yellow or yellowish?
Attention-banl based Strategy
How do I sell dry Moringa leaves powder in Indian market? Can I use the moringa leaves that are already starting to turn yellow or yellowish?
Attention- based Strategy How do sell dry Moringa leaves powder in Indian market? Can use the moringa leaves that are already starting to turn yellow or yellowish?
17


Reviews:
[{"review_id": "y_EYuyzuSzZ", "reviewer": "Reviewer_4pzW", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposed to inject visual information into pre-trained language models without retrieved or generated images. Instead, it detects visually-hungry words and generates their visual representation by CLIP text encoder, and injects them back into a pre-trained language model. Experiments are done in various datasets and recent methods are compared.", "summary_of_the_review": "Overall, I like the straightforward idea and it also works well. Some details and experiments are missing but I still lean to accept it.", "strength_and_weaknesses": "Strengths:\n1. Compared to recent works of Visually-augmented PLM, this work doesn't need to retrieve or generate images, which is much faster in training/inference and more straightforward in methodology.\n2. Experiment section covers many different datasets and previous methods. The proposed method can outperform others and baselines by a satisfactory margin.\n\nWeaknesses:\n1. Some detail of the visually-hungry word extraction is not clear to me. \n- a) In the learning-based strategy, when back-propagation, will the VL-PTM (CLIP in your case) and PLM all be updated? Or just the MLP is updated?  \n- b) In those three strategies, will the visual bi-gram or phrase be detected one unit? For example, in the case of 'He is eating a green apple', 'green apple' should be detected as one phrase instead of two words. In the following CLIP text encoding, 'green apple' should also be encoded as in one sentence instead of encoding 'green' and 'apple' separately. \n2. In experiments, why T5-3B model is missing in NLU tasks while it's used in other tasks?\n3. The details about the experimental setting of `+retrieved images` in Tab2, 3 are missing.  What's the image database? How many images are retrieved for each VH token? How is it different from other retrieving-based methods such as iACE?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "srVgUI5MUV", "reviewer": "Reviewer_TuCV", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors demonstrate that performance on\nGLUE/Commongen/CommonsenseQA can be improved by selectively\nincorporating textual CLIP representations. This method outperforms\nprior vision-for-language methods like Vokenization. Ablations in the\nappendix substituting random/T5/etc. features for the CLIP features\nsuggest that this effect is not due to architecture choices, but\nsomething unique about the CLIP textual features.", "summary_of_the_review": "\nOverall --- the authors make some promising experimental steps towards\nan exciting goal: showing that \"visual knowledge\" can be transferred\nto text-only models with performance improvements for downstream\ntasks.  However, the argument is incomplete: there are plausible\nalternative hypotheses that could explain CLIP features helping\nlanguage models beyond \"visual knowledge:\" and, some experiments that\ncould quantify additional aspects of the potential visual knowledge\ntransfer are missing.\n\n=== after response:\n\nThe new experiments the authors ran to address my core concern are interesting initial steps, and I raised my score accordingly. However, I am still not 100% convinced that visual information is the reason why CLIP base features are better. 400M image captions is quite a lot of novel data, which might be the bottleneck in the scaling regime the authors consider in their additional creative experiment (first one described in response to YcbG). It's a great initial type of experiment to run, but, because the authors are quite ambitious in their work, the extraordinary claim of visual knowledge transfer, for me, requires more evidence than just this initial result on ALBEF.", "strength_and_weaknesses": "The question the authors address is one of the \"holy grails\" of\nvision-and-language research --- it seems like visual information\nshould be able to augment a text only model's capacity (sometimes\ncalled \"vision-for-language\"). Yet, the results to date haven't been\ndefinitive. Here, the authors results suggest that\nthe visually grounded textual representations of CLIP can do just that.\nThis method outperforms Vokenization and iACE, which are two prior\nvisual knowledge transfer to text-only tasks works, which is\nimpressive.\n\nMy biggest concern is that, despite the presented experiments, I am\nnot convinced that visual information is the reason why performance\nimproves. The authors presented a compelling case that CLIP textual\nfeatures uniquely improve performance vs. text-only alternatives. But\n--- does this really mean that visual knowledge transfer is occurring?\nFor example, what visual knowledge is required for SST-2, i.e., which\ninstances does the model improve performance over, and are those one's\nextra \"visual\"? The authors allude to color/shape/size information\nbeing what is transferred, but the experiments do not complete that\nargument.\n\nIt's a very difficult argument to make, and I commend the authors for\ntheir ambition in providing additional results in this sometimes\ndifficult-to-analyze domain. I appreciate the ablations the authors\nran in the appendix. And yet, despite these efforts, the core\nconclusion of the paper requires that visual knowledge is why CLIP\ntext representations do better --- I am still somewhat unconvinced.\n\nAlternate explanations that are not possible to rule out include:\n\n- CLIP's pretraining dataset ImageWebText, contains different text\n  compared to the usual language-only models. Maybe the text-only\n  information there is why they help? Prior work on commongen suggests\n  incorporation of additional textual information can improve performance:\n\n@inproceedings{wang-etal-2021-retrieval-enhanced,\n    title = \"Retrieval Enhanced Model for Commonsense Generation\",\n    author = \"Wang, Han  and\n      Liu, Yang  and\n      Zhu, Chenguang  and\n      Shou, Linjun  and\n      Gong, Ming  and\n      Xu, Yichong  and\n      Zeng, Michael\",\n    booktitle = \"Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021\",\n    month = aug,\n    year = \"2021\",\n    address = \"Online\",\n    publisher = \"Association for Computational Linguistics\",\n    url = \"https://aclanthology.org/2021.findings-acl.269\",\n    doi = \"10.18653/v1/2021.findings-acl.269\",\n    pages = \"3056--3062\",\n}\n\n- CLIP's contrastive objective at the level of sentences is different\n  enough from language modeling objectives such that any contrastive\n  model's features might add predictive accuracy.\n\n- The authors ran a large number of permutations of their model ---\n  could it be that the performance gains of a few accuracy points are\n  due mostly in part to better hyperparameter optimization?\n\nWhile definitively making the visual knowledge transfer case is\nperhaps beyond the scope of this work --- I nonetheless could envision\na number of missing additional experiments in this setup:\n\n- Which words does the model select as \"visually hungry\"? Do they\n  align with the syntax-based method? Do they align with human judgments\n  of visual-ness, e.g., from:\n\nDouglas L. Nelson, Cathy L. McEvoy, and Thomas A.\nSchreiber. 2004. The University of South Florida free association,\nrhyme, and word fragment norms.  Behavior Research Methods,\nInstruments, & Computers, 36(3):402–407.\n\n- Which instances across the NLP tasks does performance improve most\n  for? Are these cases where we would expect visual information to\n  help?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "OPR7VvtG74y", "reviewer": "Reviewer_YcbG", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In this paper, the authors proposed to leverage pretrained vision-language models like CLIP to augment the pretrained language models with visual knowledge by injecting embeddings generated from the text encoder in CLIP for words that may be lack of visual knowledge. Effectiveness of the propopsed method is verified on various benchmarks on natural language understanding, commonsense reasoning, text generation and visual entailment.", "summary_of_the_review": "Although the reviewer thinks the proposed idea might be interesting to the community, the analysis on the real source of performance gain is not enough to really support the claim.", "strength_and_weaknesses": "Pros:\n1. The proposed idea is quite intuitive and straightforward.\n2. The evaluation is extensive on ten NLP tasks.\n\nCons:\n1. Insufficient analysis to identify the true contribution of performance gain.\na. The baselines in Table 5 should be actually included for the other evaluations as it is important to know whether the proposed method is only significantly effective on other tasks/datasets.\nb. It is unclear whether stronger VL models lead to better performance when used in the proposed method.\nc. Most importantly, despite the performance gain of CLIP text encoder as reported in Table 5, it is unclear whether this really comes from \"seeing the visual world\". What if the knowledge source actually comes from the captions? Therefore, it is important to add baselines to justify this problem. For example, using image captions to pretrain a language model as the knowledge source; retrieve some image captions as the knowledge source.\n2. Additionally, could the augmented embeddings be interpreted? Are there any evidence from the interpretation really also supporting the claims?\n3. The extra computation and latency introduced by the proposed method should be also provided to help the audience understand better the trade-off between visual augmentation and efficiency.\n\n\nMinor:\nSecond to last paragraph in introduction,  \"improveing\".", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "CnbbwNIBqqu", "reviewer": "Reviewer_A4xX", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes to integrate visual information into pre-trained language models during fine-tuning when applied to pure-language tasks. Different from previous work that either uses retrieved or generated images, they propose to leverage the CLIP text encoder to obtain the image-aligned text representations of certain input words, and then feed the language models with these additional representations when fine-tuned on language tasks. They demonstrate improvements over baselines on text classification, commonsense classification/generation, and visual entailment tasks.\n\nSpecifically, they propose three strategies to extract \"visually-hungry words\" (VH words): 1) using spacy to extract the POS of each word and treating nouns and adjectives as VH words; 2) using the attention scores of the CLIP text encoder and selecting the top-k words with the highest attention scores with the EOS token; 3) training a neural network with Gumbel-Softmax to select the VH words automatically.\n\nAfter they extract the VH words, they use the CLIP text encoder with prompts to get the image-aligned representations of these words and feed them to their language models.", "summary_of_the_review": "Overall, while the idea of using an image-aligned text encoder to provide visual information for language tasks is interesting and may have potential, the currently proposed methods are not technically sound and the evaluation settings are questionable (the details of these are listed in \"strength and weakness\"), therefore, I am leaning towards a rejection of the paper.", "strength_and_weaknesses": "Strengths:\n1. The paper focuses on how to improve the model performance on pure-language tasks by introducing visual information, which is an important research direction yet has not been widely studied yet.\n2. The idea of not using either retrieved or generated images for integrating visual information is novel and interesting, as it has been pointed out by the paper that sometimes the original text representations can be polluted using previous methods.\n3. They conduct experiments across various tasks and demonstrate improvements over their baselines.\n\nWeaknesses:\n1. Their ways of extracting \"visually-hungry\" words are not technically sound and sometimes even arbitrary to me. For example, for their syntax-based strategy, they simply use an external tool to extract all the nouns and adjectives from sentences. As mentioned in the paper, not all nouns and adjectives are related to visual semantics, and using external tools can cause error propagation issues, yet this is their default strategy for some tasks as the other methods is computationally more costly or achieves similar performance. More technically sound approaches should be proposed, and the authors should conduct qualitative and/or quantitative analyses on the extracted VH words.\n2. Their evaluation settings are questionable. It is unclear why they chose to report numbers on 6 GLUE tasks whereas most previous work would report results on 8 or 9 GLUE tasks. In addition, for GLUE tasks, they use a relatively weak baseline from Tan et al. [2020], but they should also include the baselines from the original BERT/RoBERTa papers and build their models on top of them so that the readers can be more clear about how their models perform.\n3. There are few analyses on why their model can achieve improvements over pure-text models. They should include more analyses in this regard. For example, they can include ablation studies on whether the improvements are because of ensembling different text encoders, on what kinds of sentences/tasks their model can outperform the baselines the most, what if there are no visually-hungry words or their models extract wrong VH words, etc.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject