Visually-augmented Pretrained Language Models for NLP Tasks without Images
Year: 2023

Abstract:
Although pre-trained language models (PLMs) have shown impressive performance by textonly self-supervised training, they are found lack of visual semantics or commonsense. Existing solutions often rely on explicit images for visual knowledge augmentation (requiring time-consuming retrieval or generation), and they also conduct the augmentation for the whole input text, without considering whether it is actually needed in specific inputs or tasks. To address these issues, we propose a novel Visually-Augmented fine-tuning approach that can be generally applied to various PLMs or NLP tasks, Without using any retrieved or generated Images, namely VAWI. Experimental results show that our approach can consistently improve the performance of BERT, RoBERTa, BART, and T5 at different scales, and outperform several competitive baselines on ten tasks. Our codes and data are publicly available at https://github.com/RUCAIBox/VAWI.

heading: 1 Introduction
text: Recent years have witnessed the success of pretrained language models (PLMs) (Qiu et al., 2020; Zhao et al., 2023), such as GPT-3 (Brown et al., 2020) and T5 (Raffel et al., 2020), in a variety of natural language process (NLP) tasks. Since these PLMs are mostly trained on text-only corpus via self-supervised pre-training, they have been shown lack of visual commonsense (Liu et al., 2022) and real-world knowledge (Zhang et al., 2022). As a result, PLMs can’t well solve visually related language tasks 1, e.g., answering the color and size of common things, especially those requiring complex commonsense knowledge.
To alleviate this problem, existing works mainly enhance PLMs by infusing visual information. Typ-
∗ Equal contributions. † Corresponding authors.
1In this work, we mainly focus on text-only NLP tasks that may benefit from external visual information, rather than visual-language tasks involving images.
ically, given a text input, these studies firstly augment the visual information from retrieved or generated images about the input and then leverage their visual representations to improve PLMs on NLP tasks. Such an approach leads to visuallyaugmented pre-trained language models (VaLMs), where they adopt either visually-augmented pretraining (Tan and Bansal, 2020; Wang et al., 2022) or visually-augmented fine-tuning (Lu et al., 2022). Despite the effectiveness, there are two major shortcomings in these methods. First, these methods often rely on pre-learned complementary retrievers or generators, and also require time-consuming inference to retrieve or generate proper images that are paired with the input. The above costly conditions largely limit the applicability of these approaches. Second, the retrieved or generated images are inevitable to involve irrelevant or redundant visual information. If simply integrating them, the original text representations might be affected.Increasing evidence shows that the visual information is not always useful for NLP tasks (Dai et al., 2022), and sometimes leads to performance degradation.
Considering these issues, we aim to develop a more efficient and effective way to visually augment the PLMs and the solution is twofold: • Firstly, we don’t explicitly produce (retrieve or generate) the images but instead generate visually-aligned representations of the text onthe-fly. Recent studies (Radford et al., 2021; Jia et al., 2021) have shown that the vision-language pre-trained models (VL-PTMs) can well learn the alignment between the representations of texts and images from large-scale text-image pairs. Thus, our idea is to employ the output representations of a text from VL-PTMs’ text encoders as a surrogate for the visual representations of related images.
Such a way is simple and efficient: we can only keep the text encoder of a VL-PTM to produce the visually-aligned representations of texts, getting rid of the complicated image retrieval or generation
14912
process. It is widely recognized that there is a large semantic gap between different modalities (Liang et al., 2022). Our method can alleviate this issue to some extent since the visual augmentations are derived from the text representation itself.
• Secondly, instead of directly feeding visual augmentations into the PLM, we propose to use the augmented visual information only when it is actually required. In fact, for a text input of a NLP task, PLMs are not always hungry for the visual background knowledge to effectively understand it, especially for visually-irrelevant expressions. Unlike previous works which inject visual information into a text (Tan and Bansal, 2020; Wang et al., 2022) from the whole, we consider identifying visuallyhungry words (those that require visual knowledge to derive complete semantics) from the text input, and only infuse the visual augmentations through these trigger words. We conduct visual augmentations at the word level, because it is more flexible and controllable, considering the augmented information is often irrelevant or noisy.
To this end, in this paper, we propose a general Visually-Augmented fine-tuning approach to improving PLMs for NLP tasks Without Images, namely VAWI. Our approach consists of three ingredients, namely visually-hungry words extraction, visual knowledge augmentation, and visuallyenhanced fine-tuning. Given the text input from a NLP task, we first extract the visually-hungry words (VH-words) from the input sentence. As the annotations of VH-words are generally unavailable, we propose three strategies to automatically extract the VH-words, relying on the syntax trees, attention distributions of VL-PTMs, and an adaptive learnable module, respectively. Then, based on the extracted VH-words, we leverage the text encoder of CLIP (Radford et al., 2021) (being fixed in our approach), a VL-PTM that has been pre-trained on millions of text-image pairs, to encode the VHwords for obtaining their visually-aligned representations. Finally, we infuse the visually-aligned representations into PLMs, and consider the general and parameter-efficient fine-tuning strategies for small and large PLMs, respectively.
To verify the effectiveness of our framework VAWI, we test it on four PLMs (i.e., BERT, BART, RoBERTa, and T5) at different scales (i.e., 110M, 340M, 3B), and conduct extensive experiments in natural language understanding, commonsense reasoning, and text generation tasks. Ex-
perimental results show that our VAWI can boost the performance of these PLMs significantly, i.e., 3.11%, 2.54%, and 2.16% absolute improvements on the commonsenseQA task using RoBERTa-base, RoBERTa-large, and T5-3b, respectively. Besides, VAWI can outperform (or be on par with) several competitive baselines that adopt complicated visually-augmented methods.

heading: 2 Related Work
text: Pre-trained Language Models. Recent years have witnessed the success of pre-trained language models (PLMs) (Devlin et al., 2019; Radford et al., 2019). After pre-trained on the large-scale corpus, PLMs can be fine-tuned on multiple NLP tasks and achieve remarkable performance. However, since PLMs are just pre-trained with text-only data, they may suffer from the reporting bias problem (Gordon and Van Durme, 2013; Paik et al., 2021; Zhang et al., 2022), where the frequency distribution of visual commonsense in the text may not fully reflect the real-world distribution of the commonsense. Existing works have also found that such a problem can not be well addressed by enlarging the model or pre-training corpus (Paik et al., 2021; Zhang et al., 2022). In this work, we aim to alleviate this problem by adding visual knowledge on PLMs during fine-tuning.
Vision-Language Pre-Trained Models. To better accomplish the vision-language tasks, visionlanguage pre-trained models (VL-PTMs) (Su et al., 2019; Lu et al., 2019) become a hot point in recent years, which require large-scale image-text pairs for pre-training. Existing VL-PTMs fall into two categories based on the way of modeling visionlanguage interaction. The first category of models (Lu et al., 2019; Li et al., 2021) adopts an explicit vision-language interaction layer to fuse the text embeddings and image features. These models are more suitable to capture fine-grained semantic interactions between vision and language.The second category of models (Radford et al., 2021; Jia et al., 2021) incorporates separate encoders to model the vision and language information, and relies on pre-training tasks (e.g., cross-modal contrastive learning) to align their representations into the same latent space. Such a way is capable of producing enriched single-modal representations.
Visually-Augmented Language Model. To introduce visual information into PLMs, visually-
augmented language model (VaLM) (Wang et al., 2022) has become an emerging research topic. Existing VaLMs can be categorized into visuallyaugmented pre-training and fine-tuning. Visuallyaugmented pre-training approaches (Tan and Bansal, 2020; Zhu et al., 2022) continually pretrain PLMs with the retrieved visual information related to input tokens or sentences and also revise the masked language model task for better capturing the visual semantics. Visually-augmented fine-tuning method (Lu et al., 2022) introduces the visual information into PLMs during fine-tuning. These methods also leverage the image retrieval or generation models to augment the visual information and design a special fusion module to inject it into PLMs. However, existing VaLM approaches mostly need to retrieve or generate visual information for utilization. Such a way is time-consuming, and may involve unrelated or noisy information into PLMs, leading to performance degradation. In this work, we aim to first detect the visually-hungry words from the text, and then utilize a VL-PTM to generate their visually-aligned representations without the usage of external images or generation models. As a comparison, our approach is more flexible and efficient to leverage visual information for enhancing text-based PLMs.

heading: 3 Method
text: In this section, we firstly introduce the task setting, and then describe our proposed visual augmentation approach for infusing visual knowledge into PLMs during fine-tuning.

heading: 3.1 Task Setting and Solution Overview
text: This work aims to improve the fine-tuning performance of pre-trained language models (PLMs) on NLP tasks by leveraging the related visual information without images. For a NLP task, a set of n labeled texts {⟨xi, yi⟩} are available, where xi is the i-th text data consisting of a sequence of words, denoted as xi = {w1, w2, ..., wm}, and yi is the ground-truth output, which can be a discrete label (classification), a continuous value (regression) or a text sequence (generation).
To solve the target task, we assume that a textbased PLM is given (either for understanding or generation). Let f denote a PLM parameterized by θPLM that has already been pre-trained on generalpurpose large-scale text data. Given the labeled training data, we can train the PLM using a specific
loss function (e.g., cross-entropy loss) and further solve the target task. However, existing works (Tan and Bansal, 2020; Zhang et al., 2022) have revealed that PLMs may be unaware of visual knowledge that is not explicitly mentioned in the pre-trained text-only data (e.g., the shape of coins and the color of the sky), leading to the lack of world commonsense and generating wrong statements.
In this work, we focus on devising an efficient and effective way to infuse such visual knowledge into PLMs during fine-tuning. Our approach is based on visually-hungry words (abbreviated as VH-words), which require visual information to derive complete semantic representations. The overall illustration of our approach is shown in Figure 1. Given the input text xi and its label yi, we first detect and extract a set of VH-words. Then, we adopt a visual knowledge augmentation module to enhance the visual background knowledge of their tokens and generate their visually-aligned representations. Finally, we infuse the visuallyaligned text representations into the PLM to improve its fine-tuning performance, where we consider both the general fine-tuning of small PLMs and the parameter-efficient fine-tuning of largescale PLMs.

heading: 3.2 Visually-Hungry Words Extraction
text: In our approach, visually-hungry words (VHwords) are the trigger units for visual augmentations, requiring visual knowledge for deriving complete semantic representations (e.g., color, shape, and object). Therefore, we propose to first detect the VH-words from the input text, and then inject the proper visual knowledge that they are hungry for into the PLM. However, the annotations about VH-words are generally not available in NLP datasets. To address this problem, we devise three different strategies to extract the VH-words from the input text, including two feature-based strategies based on syntax tree and attention distribution of PLMs, and a learnable model-based strategy.
Syntax-based Strategy. In natural language, entity words and descriptive words usually convey more visual semantics than others. For example, for the sentence “He is eating a green apple”, where underlined words are more related to visual semantics. Such words are mostly nouns or adjectives in the input text, which can be detected by syntactic analysis. Therefore, we design a rule-based strategy that leverages the syntactic information for
VH-words extraction. Concretely, we first delete all stop words in a text and then adopt an openresource toolkit SPACY 2 to convert the input text into a syntax dependency tree. Based on the syntax tree, we extract the words that have a particular part of speech (POS), e.g., nouns or adjectives, as the VH-words denoted by W(V H). In this way, we can efficiently extract the VH-words from input text by using a fast parser toolkit.
Visually-enhanced Attention Based Strategy. The attention-based strategy utilizes the attention distribution of a VL-PTM to detect the VH-words. Since VL-PTMs (Radford et al., 2021) are pretrained on large-scale image-text pairs, their text encoders can focus more on the words corresponding to some specific visual concepts in an image, which are likely to be VH-words. Inspired by it, we use the attention scores calculated by the text encoder of VL-PLMs to select the VH-words. Specifically, we adopt the text encoder of CLIP (Radford et al., 2021), a VL-PTM that has been pre-trained on millions of image-text pairs, to help extract the VH-words. As CLIP adopts an autoregressive GPT2 model as the text encoder, we calculate the average attention scores between each token and the “[EOS]” token on the self-attention layer, denoted as swi . Then, we select the top-K ranked words according to {swi} as the VH-words W(V H).
Learning-based Strategy. Considering that diverse PLMs and NLP tasks may be hungry for different complementary visual information, we
2https://spacy.io/
devise a learning-based strategy that can adaptively extract VH-words according to task requirements. Concretely, we add a parameterized VH-words extractor layer for the PLM, which can be updated by gradient-based optimization algorithms to fit the need for some specific task. Given the input text xi, we first leverage the PLM and a text encoder of a VL-PTM (i.e., CLIP (Radford et al., 2021)) to produce the contextualized representations of the contained words in xi. Then, we concatenate the representations of each word from the two models and utilize a MLP layer to obtain the score swi :
swi = MLP([h (P ) wi ;h (V ) wi ]) (1)
where h(P )wi and h (V ) wi are the output word representations from the PLM and VL-PTM, respectively, and scores swi are calculated by the learned model based on the supervision information from downstream tasks. Based on the scores of all words, we incorporate the gumbel-softmax function (Jang et al., 2016) to extract the top-k words as the VHwords in a differentiable way. In this way, the gradients of the fine-tuned tasks can be back-propagated to the extractor layer, which learns to adaptively select the more suitable VH-words.

heading: 3.3 Visual Knowledge Augmentation
text: Existing works (Lu et al., 2022; Wang et al., 2022) mainly utilize image retrieval or generation module to augment related visual knowledge. Such a way is time-consuming and may also involve noisy images.Inspired by recent works that show the effective visual-language alignment in VL-PTMs (Rad-
ford et al., 2021; Li et al., 2021), we utilize the visually-aligned text encoders to generate the visual augmentation representations of VH-words. As the text encoders have been aligned to the image encoders during pre-training, their output textual representations can be used as surrogates of visual augmentations based on real images related to the input text. As will be shown in experiments (Section 4), this approach is not only efficient but very effective for downstream NLP tasks.
Based on the extracted VH-words, we first add a prefix text in the image caption style before the VH-words, e.g., “a photo of: ”, to compose the input text x′. Then, we utilize the text encoder of CLIP (Radford et al., 2021) to encode x′ and obtain the contextualized word representations as the visually-aligned representations Hx ∈ Rk×d, where k is the sequence length of x′ and d is the embedding size. Next, we incorporate a reformulation layer to aggregate and strengthen the visually-aligned representation Hx into the visually-augmented representations of these VHwords. As the positions of the VH-words vary from sentence to sentence, we design a positionaware attention mechanism in the reformulation layer to inject position information into Hx for obtaining the visual representation of each VH-word. Specifically, we first leverage a soft position embedding matrix E ∈ Rl×d to reserve the position information of VH-words, where l is the number of VH-words. Then, we perform the cross-attention between it and the visual representations as:
Q = E, K = HxW K + bK , (2)
V = HxW V + bV , (3)
Hv = softmax( QK⊤√
d )V, (4)
H⊤v = [h1,h2, ...,hl], (5)
where hi ∈ Rd, K,V ∈ Rk×d. Hv ∈ Rl×d is the obtained visually-augmented representations of VH-words, which is leveraged for augmenting the visual knowledge of the PLM. hi is the visual representation of the i-th VH-word in W(V H). Note that in Eq. 2 and 3, we adopt an efficient way that only uses the position information to set the query matrix Q, and the visual semantics are mainly captured and injected through the key and value matrices.

heading: 3.4 Visually-Enhanced Fine-tuning
text: After obtaining the visually-augmented representations of VH-words (i.e., Hv in Eq. 5), we propose
a visually-enhanced fine-tuning strategy to inject the captured visual knowledge. Here, we consider two cases: (1) full-parameter fine-tuning for small PLMs, and (2) parameter-efficient prompt-tuning for large-scale PLMs. Before introducing the learning method, we simply review the parameters of our approach, consisting of the parameters in the underlying PLM (Θplm), the VL-PTM (Θvlp) and the parameters of the reformulation layer (Θref ). Note that we will always fix Θvlp in our approach.
Fine-tuning for Small PLMs. For small PLMs, we can perform full-parameter fine-tuning, which updates both Θplm and Θref . Specifically, given the visually-augmented representations Hv of VHwords, we directly incorporate them into the embedding layer of the PLM. For each VH-word, we insert its visually-augmented representation after the original word embedding, to leverage the visual semantics to enrich the word representations.
Prompt-tuning for Large-Scale PLMs. For large-scale PLMs, we fix the parameters in it, i.e., Θplm, and employ a parameter-efficient prompttuning way to optimize it on downstream NLP tasks. Concretely, given the visually-augmented representations Hv of VH-words, we directly insert them before the input representations of every layer of PLMs. Then, following the typical prompttuning paradigm (Li and Liang, 2021), we only tune the parameters of the reformulation layer (i.e., Θref ) as the soft prompts to adapt all the model into the fine-tuning task.
Our approach can be generally applied to various PLMs (e.g., BERT (Devlin et al., 2019), BART (Lewis et al., 2020), T5 (Raffel et al., 2020)) and NLP tasks (natural language understanding and text generation). Unlike other complicated visuallyaugmented methods (Tan and Bansal, 2020; Wang et al., 2022), it is more efficient, without the explicit need of external images or generation model; and meanwhile, it only introduces a small number of parameters (Eq. 3), which are easier to learn.

heading: 4 Experiments

heading: 4.1 Experimental Setup
text: Datesets. We conduct experiments on four types of tasks. (1) Natural Language Understanding (NLU): we extract 6 datasets from the GLUE benchmark (Wang et al., 2018); (2) Commonsense reasoning: we select CommonsenseQA (Talmor et al.,
2019), a 5-way multiple choice QA dataset that requires commonsense knowledge; (3) Text generation: we select CommonGen (Lin et al., 2019b), a constrained text generation task about generative commonsense reasoning. (4) Cross-modal reasoning: we select SNLI-VE (Xie et al., 2019), to evaluate the capacity of predicting whether the image semantically entails the text.
Baseline Models. We compare our approach with the following baselines, including pre-trained language models (PLMs), visual-language pre-trained models (VL-PTMs), and visually-augmented pretrained language modes (VaLMs). (1) PLMs: We choose BERT (Devlin et al., 2019), RoBERTa (Liu et al., 2019), BART (Lewis et al., 2020), T5 (Raffel et al., 2020) as the PLM backbones, and directly fine-tune them as baselines. (2) VL-PTMs: We select ALBEF (Li et al., 2021), BLIP (Li et al., 2022), and CLIP (Radford et al., 2021), which have been pre-trained on large-scale image-text pairs. (3) VaLMs: we select VOKEN (Tan and Bansal, 2020) and iACE (Lu et al., 2022), which introduce the visual information into PLMs by pre-training on retrieved images and fine-tuning on generated images, respectively.
Implementation Details. We implement all methods based on Huggingface Transformers (Wolf et al., 2020). For all baselines, we set their hyper-
parameters according to their papers. In our approach, we leverage the text encoder of CLIP (ViTB/32) to implement the learnable model-based VHwords extractor and generate the visual representations of VH-words in the visual knowledge augmentation module. The hidden size of visual representations is set to 512. For different NLP tasks, we tune the number of visually hungry words in {2, 3, 4, 5}. During fine-tuning, we perform parameterefficient tuning on T5-3b and BART-Large, and full-parameter tuning on other PLMs. For all tasks and all backbones, we utilize Adam as the optimizer, set the learning rate to 2e-5, weight decay to 0.01, and a linear warmup for the first 6% steps. For GLUE, GommonGen, and SNLI-VE datasets, we fine-tune our model for 3 epochs with a batch size of 32. For CommonsenseQA, we tune our model for 10 epochs with a batch size of 32. We use the cross-entropy loss for classification and the mean squared error loss for regression.

heading: 4.2 Main Experimental Results
text: In this part, we conduct a series of experiments on NLU, commonsense reasoning, text generation, and cross-modal commonsense reasoning tasks.
Evaluation on NLU Tasks. We present the experimental results of different methods on 6 NLU tasks in Table 1. First, we observe that VL-PTMs perform worse than PLMs, a possible reason is that
they have been continually pre-trained on largescale image-text pairs, which may cause the catastrophic forgetting problem. Second, VaLMs (i.e., VOKEN, iACE, and VAWI) achieve better performance over PLMs. As VaLMs infuse external visual knowledge into the PLMs, they can help the PLMs better understand the background knowledge of some words (e.g., color, shape, and size of objects). Between the two VaLM baselines, iACE is slightly better. This is because iACE is enhanced based on VOKEN and incorporates an image generation model, so it produces more visual information to utilize. However, the generated images inevitably contain noise and redundant information, which limits the performance gain of iACE.
Finally, by comparing our approach with all baselines, it is obvious that VAWI performs consistently better than them on the six datasets. In our approach, we adopt an efficient and effective way that augments the visually-augmented representations using the text encoder of CLIP to encode the VH-words from the input text. Benefiting from pre-
training on large-scale image-text pairs, the text encoder of CLIP has been well aligned with the semantic space of images, so that it can generate high-quality visually-augmented representations of the VH-words to enrich them. Such a way not only saves the costs of time and computation but also reduces the influence of inevitable noise from retrieved or generated images. Additionally, among three VH-words extraction strategies, LBS slightly outperforms others in most NLU tasks. The reason is that LBS incorporates a learnable model-based strategy to select the VH-words. Such a way can adaptively extract proper VH-words with the consideration of the intrinsic knowledge of the PLMs. However, LBS will increase the computation cost due to its involved learnable VH-words extractor layer. Therefore, for efficiency, in the following experiments, we utilize the SBS strategy in our approach for comparison.
Evaluation on Commonsense Reasoning Tasks. Following existing works (Lin et al., 2019a), we
also rely on a rule-based strategy to extract the examples containing visible objects, to construct a new dataset called CommonsenseQA-3K. It consists of 2,903 and 341 examples in the training set and dev set, respectively. Based on the CommonsenseQA and CommonsenseQA-3k, we also report the results with different amounts of training data, to further evaluate the performance of different methods in the few-shot setting.
As shown in Table 2, we can also see that with the help of the visual information from either retrieved images or our VAWI-SBS, the performance of PLMs can be improved significantly. It indicates that visual information is indeed helpful to improve PLMs for understanding commonsense knowledge. Besides, our approach outperforms the method using retrieved images from search engines. Our approach omits the image retrieval process due to its inevitably involved noise, and relies on the text encoder of CLIP to augment the visual representations. Such a way can guarantee the relevance between the augmented visual knowledge and the text input, reducing the influence of retrieved noisy images and redundant information. Furthermore, we also perform parameter-efficient tuning on T53B-encoder with our approach and boost its performance. It shows that our approach is able to be applied to large-scale PLMs to meet their thirst for visual information.
Evaluation on the Text Generation Task. As shown in previous experiments, it is useful to improve the performance of VAWI on commonsense reasoning and nature language understanding tasks. Here, we would like to study the effectiveness of our approach on the text generation task (i.e., CommonGen) using large PLMs. As shown in Table 3, our model VAWI also consistently boosts the performance of BART-Large and T5-3b among all metrics. It further shows that our approach can also improve PLMs on the text generation task. As a comparison, we can see that the retrieved images are not very helpful and even cause performance degradation. The reason may be that
the text generation task is more sensitive to the inevitable noise from the retrieved images. Finally, the parameter-efficient tuning strategy of our approach also achieves comparable performance with the full-parameter tuning. It indicates that our parameter-efficient strategy is able to efficiently optimize the parameters of large-scale PLMs, and shows a promising future to apply our approach to much larger PLMs, e.g., GPT-3.
Evaluation on the Cross-modal Commonsense Reasoning Task. To verify the generality of our method, we further implement our VAWI on a VLPTM (i.e., ALBEF (Li et al., 2021)), and conduct experiments on a cross-modal reasoning dataset, SNLI-VE. Concretely we implement our approach on ALBEF by inserting the visually-augmented representations after the VH-words embeddings of the text encoder before the multimodal encoder, and keeping others unchanged. As shown in Table 4, our VAWI can also improve the performance of ALBEF using different amounts of training data. It further shows the generality of our approach in VL-PTMs, as it can also provide rich information to enhance the text encoder of VL-PTM, helping it better perform cross-modal reasoning.

heading: 4.3 Ablation Study
text: In this part, we conduct a series of experiments to verify whether the improvement of our approach derives from the augmented visual knowledge about the VH-words. More ablation studies are shown in Appendix A.
The Effect of the Source of Visual Representations. We first propose three variants that incorporate powerful PLMs, i.e., RoBERTa-base, T5Large, and T5-3b respectively, to replace the text encoder of CLIP in our framework. We also replace the generated visual representations from the text encoder of CLIP with random noise, to investigate the importance of the visual representations. As shown in Table 5, we can see that our approach is better than all the variants, even T5-3b with billionscale parameters. It indicates that CLIP-base is more effective to augment visual knowledge to improve the performance of PLMs. Besides, our approach also outperforms the variant using random noise as the visual representation, showing the worse performance among all the variants. It also shows the importance of visual representations, as they indeed contain the visual knowledge that the
PLM is hungry for.
The Effect of the Stronger VL-PTMs. In our work, we choose CLIP-base to enhance PLMs, as it has been pre-trained on a large-scale imagetext dataset. Generally, a stronger VL-PTM would be more promising to further improve the performance. Here, we replace our CLIP-base model with some stronger VL-PTMs, e.g., ALBEF (Li et al., 2021), UniCL-base (Yang et al., 2022), and CLIP-large. Concretely, ALBEF leverages more pre-training tasks (e.g., MLM, ITM, and ITC), UniCL utilizes more high-quality pre-training data, and CLIP-large increases the scale of model parameters. We evaluate the above variations on CSQA3k, QQP, and SST-2, and the results are shown in Table 6. We can see that UniCL and CLIP-large outperform CLIP-base. It indicates that the VLPTMs with the larger scale of model parameters or more high-quality pre-training data are more capable of augmenting useful visual knowledge for PLMs. Considering the efficiency, CLIP-base is also a good choice in our approach, and we will investigate more proper VL-PTMs in the future.

heading: 5 Conclusion
text: In this paper, we proposed a general visuallyaugmented fine-tuning approach that can be applied to a variety of PLMs and NLP tasks, without using any retrieved or generated images, namely VAWI. Specifically, we first identified and extracted the visually-hungry words (VH-words) from input text via a token selector, where three different
methods have been proposed, including syntax-, attention- and learning-based strategies. Then, we adopted a fixed VL-PTM text encoder to generate the visually-augmented representations of these VH-words. As it has been pre-trained by visuallanguage alignment tasks on the large-scale corpus, it is capable of injecting visual semantics into the aligned text representations. Finally, we transformed the visually-aligned features into visuallyaugmented features by reformulation layer based on VH-words, and inserted them into PLMs to enrich the visual semantics of word representations in PLMs. Experimental results on 10 NLP tasks show that our approach can consistently improve the performance of BERT, RoBERTa, BART, and T5 at different scales, and outperform several competitive baselines significantly. Besides, the visual prompts of our framework can also be used for parameter-efficient tuning, which can boost the performance of large language models, such as T5-3b.
Limitations
An important limitation of our approach VAWI is the need for extracting visually-hungry words (VHwords) as the trigger to inject visual knowledge into PLMs. In real-world applications, it is hard to obtain the annotations of VH-words. Therefore, we propose three VH-words extraction strategies. However, the three strategies may be not always proper for all NLP tasks, and we rely on the experimental results to select the best one among them. Besides, we adopt the text encoder of CLIP as the
VL-PTM for generating the visually-aligned representation. As a pre-trained model, CLIP also may contain biases learned from the pre-training corpus, which may result in improper biased prediction on some NLP tasks.

heading: Acknowledgement
text: This work was partially supported by National Natural Science Foundation of China under Grant No. 62222215, Beijing Natural Science Foundation under Grant No. 4222027, and Beijing Outstanding Young Scientist Program under Grant No. BJJWZYJH012019100020098. Xin Zhao is the corresponding author.

heading: A Ablation Study
text: A.1 Ablation Study on Visual Knowledge Augmentation
The Effect of the Pre-trained Dataset of VLPTMs. We notice that the pre-training dataset of VL-PTMs is different from PLMs. Here, we investigate whether the captions or images from the large-scale image-text pairs contribute more to the performance gain of our approach. To verify it, we pre-train a new PLM only using the captions data. Following the setting of ALBEF, we utilize the pretrained parameters of BERT to initialize this model and only extract the captions from the pre-training data of ALBEF (14.5M sentences in total). After pre-training on these captions until convergence, we utilize this model to replace CLIP-base in our approach and keep other settings unchanged. We conduct experiments on commonsense reasoning and NLU tasks to evaluate its effectiveness for augmenting visual knowledge. As shown in Table 7, we can see that such a variation underperforms ALBEF and our approach, and even leads to performance degradation on the CSQA task. It indicates that during pre-training the image data is an important resource for learning visual knowledge in VL-PTMs. Only text data (i.e., captions) can not provide sufficient visual knowledge that PLMs are hungry for. Therefore, after pre-learned on largescale text-image pairs, CLIP can absorb the useful visual knowledge from the images and inject them into PLMs in our approach. It further indicates that the improvement of our method is due to the involvement of the visual information about the VH-words.
A.2 Ablation Study on Visually-enhanced Fine-tuning
Different Insertion Positions of Visual Representations. In our visually-enhanced fine-tuning framework, we insert the visual representation of the VH-word after its original word embedding. To verify its effectiveness, we propose three variants of it that do not insert, insert all visual representations of VH-words before and after the input text, respectively. As shown in Table 8, we can observe that all these variants would lead to a performance decrease. It demonstrates that a proper position to insert the visual representation is important for the utilization of augmented visual representations. By inserting them after the word embeddings of
corresponding VH-words, PLMs can effectively aggregate the visual representations to enrich the word representations, leading to better performance on downstream NLP tasks.

heading: B Further Analysis
text: The Frozen CLIP’s Text Encoder. In the experiment presented in Table 1, we directly finetuned CLIP and the results indicate that the performance of VL-PTMs’ text encoder is unsatisfactory when directly fine-tuned on NLP tasks. In our VAWI, we fix the model parameters of CLIPbase’s text encoder to preserve the visual knowledge. Hence we also conduct experiments on four NLU tasks from GLUE using frozen CLIP. Specially, we fix CLIP-base’s text encoder and only fine-tuned added 4 transformer layers above it. As shown in Table 9, we can see that CLIP’s performance under this setting is better than that of directly full-parameter fine-tuning CLIP and also underperforms RoBERTa and BERT. It indicates that fixing CLIP is more suitable for NLP tasks, and shows the rationality of VAWI settings that always fix the CLIP’s text encoder in VAWI to preserve CLIP’s knowledge.
The Computation Latency of the Proposed Methods. In our VAWI, we fix the model parameters of CLIP-base to preserve the visual knowledge. Such a way can also decrease the computation costs during training and inference. To verify it, we report the mean training and inference latency per batch on the CSQA-3k dataset of our method and baselines on RTX3090 GPU, where all these methods utilize RoBERTa-base as the backbone. As shown in Table 10, we can see that our proposed VAWI-SBS and VAWI-VABS would not increase the latency too much. For VAWI-LBS, as it requires a PLM and a VL-PTM to adaptively select the VH-words, it will relatively increase the latency. As shown in Table 1, we can see that all the three variants achieve comparable performance in 6 NLU datasets. Therefore, it is more efficient and effective to select the SBS and VABS variations in our approach. Despite it, we can see that all our variants own less latency than iACE, since our approach does not require a time-consuming image generation process. And as shown in Table 1, our approach can also achieve better performance.
The Effect of the Improper Visually-hungry Words. To analyze how the quality of the VH-
words affects the performance of our approach, we further conduct the experiments on CSQA-3K and two NLU tasks SST-2 and QQP from GLUE, to show the effect of insufficient VH-words on our model performance. After extracting the VHwords, we remove part of them and only randomly sample 0%, 20%, and 50% VH-words for augmentation. As shown in Table 11, we can see that with the decreasing of the sampling probability, the performance of our approach degrades gradually. It indicates that not enough VH-words would degrade the performance of our approach.
The Number of VH-words. Our approach has an important hyper-parameter required to tune, such as the number of VH-words. VH-words can supply visual knowledge that PLMs may be hungry for. Here, we would like to study whether more VH-words are better to improve performance. We conduct experiments on the QQP and CSQA-3K datasets using RoBERTa-base as the backbone, and present the results in Figure 2. We can see that with the increase of the number of VH-words, the performance gain of our approach first increases and then decreases. A possible reason is that too many VH-words may also introduce noisy or redundant information (e.g., not very relevant words),
Correct VH-words proportions CSQA-3k SST-2 QQP
which would also influence the fine-tuning performance. Instead, it is also more efficient to select a few VH-words (e.g., two words for CSQA-3k) for deploying our approach in large-scale PLMs.
Case Study of Extracted Visually-hungry Words. In this part, we show the VH-words extracted by syntax-, attention- and learning-based strategies in Table 12, Table 13, Table 14 and Table 15. We can see that the three strategies would extract slightly different VH-words. The reason is that the three strategies are based on different techniques to identify the VH-words. As we can see, the cases show that most of the extracted VH-words by our strategies are generally related to some visual semantics, e.g., spider, two eyes. Although such VH-words can not perfectly cover all the visual semantics, they actually contain most of the important words that the PLMs may be hungry for, e.g., red and yellow. Besides, we can also see that the VH-words extracted by our three strategies may not perfectly align with human judgment. In fact, it is also hard for humans to determine proper rules to identify VH-words, e.g., people, human, and water. In addition, as the learned knowledge of PLM is a black
box, it is also difficult for humans to judge the usefulness of our extracted VH-words for PLMs.
The Interpretability of Augmented Embeddings. In this part, we show how our augmented embeddings infuse visual knowledge into the PLM. Concretely, we show the attention distributions of a PLM (i.e., RoBERTa-base) in the last few layers before and after infusing visually-augmented representations on CSQA. As shown in Table 16, we can see that the [CLS] tokens pay more attention to the VH-words and their visually-augmented representations, and the VH-words also pay more attention to their visually-augmented representations. It shows that the injected visually-augmented representations provide useful knowledge, which guides the PLM to focus on more important tokens and also improves the representations of the VH-words and the [CLS] token.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:
text: 3 A1. Did you describe the limitations of your work?
Section 6.
3 A2. Did you discuss any potential risks of your work? The potential risks can be found in Section 6.
3 A3. Do the abstract and introduction summarize the paper’s main claims? The main claims can be found in Abstract and Section 5.
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts? We introduce the dataset and pre-trained models used and the baselines in Section 4.
3 B1. Did you cite the creators of artifacts you used? We cite the creators of artifacts we used in Section 4.1.
3 B2. Did you discuss the license or terms for use and / or distribution of any artifacts? We use the default license of all artifacts in Section 4.
3 B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? We use all artifacts with their default intended use in Section 4.
B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? Not applicable. Left blank.
B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? Not applicable. Left blank.
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. Except for the dataset we created ourselves in Section 4, the relevant statistics is not reported in any other datasets we use. The dataset we use exactly follows the amount and train/test/dev splits of data in the original dataset paper.
C 3 Did you run computational experiments? Our computational experiments for evaluating our method can be found in Section 4.
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.
3 C1. Did you report the number of parameters in the models used, the total computational budget (e.g., GPU hours), and computing infrastructure used? We report the number of parameters in the models used in a few experiments, which can be found in Section 4 and Appendix. In addition, we report the total computation budget in Section C of the Appendix.
3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found hyperparameter values? We talk about them in Section 4.1.
3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? We report descriptive statistics about our results, which can be found in Section 4.
3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? We report these important settings in Section 4.
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants? Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? No response.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? No response.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? No response.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? No response.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? No response.

heading: Future Work
text: Considering the efficiency, CLIP-base is also a good choice in our approach, and we will investigate more proper VL-PTMs in the future