Visually-Enhanced Phrase Understanding
Year: 2023

Abstract:
Large-scale vision-language pre-training has exhibited strong performance in various visual and textual understanding tasks. Recently, the textual encoders of multi-modal pre-trained models have been shown to generate highquality textual representations, which often outperform models that are purely text-based, such as BERT. In this study, our objective is to utilize both textual and visual encoders of multimodal pre-trained models to enhance language understanding tasks. We achieve this by generating an image associated with a textual prompt, thus enriching the representation of a phrase for downstream tasks. Results from experiments conducted on four benchmark datasets demonstrate that our proposed method, which leverages visually-enhanced text representations, significantly improves performance in the entity clustering task.1

heading: 1 Introduction
text: Recent advances in vision-language pre-training have seen the successful alignment of visual and linguistic inputs through the implementation of cross-modal pre-training objectives, such as language modeling and contrastive learning (Lu et al., 2019; Radford et al., 2021). These pre-trained models have shown impressive performance on downstream vision-language tasks, validating their crossmodal capabilities (Su et al., 2019).
While most previous studies focused on multimodal tasks, researchers have shown that pretrained cross-modal encoders are equally proficient at uni-modal language understanding, matching the performance of pre-trained text encoders. Lu et al. (2022) were the pioneers in utilizing machine abstract imagination from pre-trained crossmodal encoders, demonstrating improvement on general NLU tasks. Yan et al. (2022) established
∗Equal contribution 1Source code: https://github.com/MiuLab/VisualLU
that the text encoder of CLIP (Radford et al., 2021) surpasses models designed for producing phrase representations, including Phrase-BERT (Wang et al., 2021) and UCTopic (Li et al., 2022a). They hypothesized that the visual supervision during pre-training empowers CLIP to produce visuallygrounded phrase representations, beneficial for language-only tasks. Such a phenomenon aligns with neuroscience studies, demonstrating that visual and linguistic semantic representations are coordinated in the human brain (Popham et al., 2021).
Despite the strong performance of the previous method, it only utilized the text encoder of a crossmodal pre-trained model. In contrast, our study aims to exploit its multi-modal representation capacity, incorporating both text and image encoders. We introduce a visually-enhanced phrase understanding framework to exploit multiple modalities for uni-modal tasks. Our framework comprises a text-to-image generator and a text-image crossmodal encoder. We employ a text-to-image generator to produce visual cues for a textual candidate. Subsequently, the generated image and the textual prompt are processed by the cross-modal encoder to create visually-enhanced phrase embeddings. Unlike Lu et al. (2022), our method does not require supervised data for downstream tasks, making it more scalable. Our approach also differs from VOKEN (Tan and Bansal, 2020), as they generated visual cues in tokens and processed the signal solely on the language side, whereas we employ representations directly from different modalities. Therefore, our model can capture more abstract concepts from images, enhancing generalizability.
We evaluate our approach on four benchmark phrase understanding datasets. The experiments demonstrate that our proposed visual enhancement significantly outperforms all text-only baselines, demonstrating that abstract visual concepts can provide complementary cues for text understanding.
5879

heading: 2 Method
text: Our proposed method is illustrated in Figure 1, where we first generate images associated with phrases using a text-to-image diffusion model. Following this, we utilize pre-trained text and image encoders to construct visually-enhanced phrase embeddings for downstream understanding tasks.

heading: 2.1 Text-To-Image Model
text: Recently, text-to-image models have attracted significant interest. Among these, diffusion models have played an important role in text-to-image generation, showing impressive performance. To more effectively generate visual cues associated with texts, this study adopts stable diffusion (Rombach et al., 2022) as our image generation model.
During the training phase, an image autoencoder is trained using an extensive image database. A time-conditional U-Net (Long et al., 2015) forms the core of the diffusion model, learning to denoise image latent representations incrementally.
In the sampling procedure, we first obtain a text prompt and derive a text embedding from the text encoder. Subsequently, we use Gaussian noise as the latent representation, and progressively denoise the latent representation via the diffusion model and a scheduler algorithm. Ultimately, an image is generated by reconstructing the latent representation through the image decoder.

heading: 2.2 CLIP (Contrastive Language-Image Pretraining)
text: CLIP (Radford et al., 2021) is a large-scale visionlanguage pre-training model using contrastive learning, which achieves remarkable performance in zero-shot image classification tasks. Given a
batch of data D, CLIP jointly trains an image encoder and a text encoder to maximize the similarities of |D| paired text-image representations while minimizing the similarities of other (|D|2−|D|) unpaired text-image representations. Given the weak alignment between texts and images, this study employs the pre-trained CLIP text encoder Etext and image encoder Eimage to extract meaningful cues from different modalities. Our experiments focus on showing that the pre-trained CLIP encoders provide superior visual enhancement for texts, compared to separately pre-trained text and image encoders.

heading: 2.3 Visually-Enhanced Multimodal Representation
text: Given a text sequence with an entity candidate phrase p, we design our text prompt as “A photo of <p>”, a proven effective default template that delivers robust zero-shot classification performance (Radford et al., 2021). As depicted in Figure 1, we initially use the text prompt to generate a text-associated image with the text-to-image model G. Following this, we employ the pre-trained text and image encoders of CLIP to extract corresponding representations ri(p) and rt(p) as follows.
rt(p) = Etext(“A photo of p”)
ri(p) = Eimage(G(“A photo of p”))
Lastly, we concatenate the two embeddings originating from different modalities to create visuallyenhanced phrase embeddings, which potentially capture richer and more comprehensive information and thus benefit downstream tasks.

heading: 3 Experiments
text: To evaluate whether our visually-enhanced phrase embeddings provide improved semantic cues, we conduct a series of experiments focused on entity clustering, as our primary task is to categorize entity candidates with similar concepts only based on phrase representations in an unsupervised fashion.

heading: 3.1 Setup
text: Our experiments are conducted on four diverse datasets, each with annotated entities from various domains:
• CoNLL2003 (Sang and De Meulder, 2003) comprises 20,744 sentences, incorporating four types of entities: persons (PER), organizations (ORG), locations (LOC), and miscellaneous names (MISC). • BC5CDR (Li et al., 2016) is formed from 1,500 PubMed articles and contains chemical and disease entities. • W-NUT 2017 (Derczynski et al., 2017) is collected from public platforms, including YouTube and Twitter, with a focus on identifying previously unseen entities in emerging discussions. It includes six types of entities. • MIT-Movie (Liu et al., 2013) contains 12,218 sentences featuring title and person entities.
Following previous research (Xu et al., 2017; Li et al., 2022b; Yan et al., 2022), we implement K-means clustering on the cross-modal representations to perform unsupervised phrase understanding tasks. In this setup, the number of clusters is set to the number of classes present in the dataset.
The Hungarian algorithm (Papadimitriou and Steiglitz, 1998) is employed to optimally allocate each cluster to a class.
To evaluate the quality of the representations and compare them fairly with the previous work, we employ accuracy (ACC) and normalized mutual information (NMI) as our evaluation metrics. The results reported are averages over five separate clustering runs. For our proposed image and text-image approaches, we conduct runs over three seeds for diffusion models to generate images.

heading: 3.2 Baselines
text: We position our model in comparison to various language models and phrase understanding models to validate the effectiveness of our cross-modal framework. The used representations are the same as described in the prior work.
• BERT/RoBERTa are well-established pretrained language models (Devlin et al., 2019; Liu et al., 2019) capable of distilling intrinsic patterns from input texts into meaningful representations.2 • LUKE (Yamada et al., 2020) enhances RoBERTa by introducing entity embeddings to the input, as well as an entity-aware attention mechanism.3 • Phrase-BERT (Wang et al., 2021) refines BERT using a contrastive objective to gen-
2We take the embedding of [CLS] with further processing as phrase representations.
3We take the last layer of Transformer as our phraseassociated representations.
erate more powerful phrase representations.4
• UCTopic (Li et al., 2022a) employs an unsupervised contrastive learning strategy, with LUKE serving as the foundational model, to create robust and context-aware embeddings.5 • CLIP Text (Yan et al., 2022) leverages the text encoder of CLIP for understanding.6

heading: 3.3 Results
text: The evaluation results are presented in Table 1. Our proposed visually-enhanced representations outperform all baselines on the CoNLL2003 and MIT-Movie datasets, while achieving competitive performance on the BC5CDR and W-NUT 2017 datasets. Moreover, solely utilizing image representations encoded from generated images yields a higher average ACC than all the baselines. This suggests that the visual signal offers valuable cues for enhanced phrase understanding. Hence, we conclude that integrating different modalities can effectively augment phrase representations. For a more granular understanding, we provide detailed scores across multiple turns in Table 2. The lower standard deviation of our proposed text-image approach indicates superior stability.

heading: 3.4 Analysis of Different Encoders
text: To further investigate whether the CLIP encoders, pre-trained jointly, are more effective for visual enhancement, we compare them with image and
4We take the average of the last layer in Transformer as our phrase-associated representations.
5We take the pooling of the entity-associated vectors based on https://github.com/JiachengLi1995/UCTopic.
6We take [EOT] of the last Transformer layer’s output as phrase representations.
text encoders that have been pre-trained individually. Table 3 presents the experimental results, where we substitute the text and image encoders of CLIP with RoBERTa-base and ViT-B/32 respectively. We notice that phrase representations augmented by ViT-B/32 outperform textual representations, which suggests the richness of information drawn from multiple modalities. It is evident that CLIP encoders surpass individually pre-trained encoders, implying that text and image encoders, when pre-trained together, can more effectively enrich phrase representations by integrating text and image at the representation level.

heading: 3.5 Contextual Prompt
text: Previous work (Yan et al., 2022) demonstrated that enriching phrase candidates with a large pre-trained language model can yield more domain-specific keywords for textual prompts. Specifically, given a phrase p, the prompt “p is a [MASK]” is fed into a language model, which in turn returns the top K predictions {m1,m2, . . . ,mK} for the [MASK] token. Subsequently, we formulate the contextual prompt as “A photo of p, a m1,m2, . . . ,mK .” In this paper, we set K to 3 for the contextual prompts. Table 1 shows that the addition of such contextual prompts enhances the performance of text-only baselines.
We further probe into whether a contextual prompt can boost our performance and present the results in Table 4. Our observation is that utilizing contextual prompts for text embeddings yields comparable performance, indicating that our visual cues already encompass the domain-specific signal. We hypothesize that generating images from
contextual prompts may introduce more noise, resulting in difficulty encoding effective visual representations for phrase understanding. Notably, our baseline setting already achieves significantly improved performance compared with earlier work utilizing additional keywords, demonstrating the informativeness of our cross-modal representations.

heading: 3.6 Qualitative Analysis
text: To further examine how our visual cues enhance text understanding, we present several generated images along with their understanding results in Figure 2. Previous work, CLIP Text, incorrectly classifies “Mpumulanga” and “Golan” as PER (persons). However, with the visual cues generated in our model, shown in Figure 2(a-b), we can correctly classify them as LOC (locations). The images generated by our model, displayed in Figure 2(c-f), further enrich the phrase representations and better understand the concepts. This demonstrates the effectiveness of our multi-modal framework.
However, there are cases where the generated
image may lead to incorrect categorization, as is the case with “BAYERISCHE VEREINSBANK” in Figure 2(g). The image misled the categorization process, changing the cluster from the correct classification (ORG, or organization) to an incorrect one (LOC, or location). Figure 2(h) displays an instance where the generated image does not provide useful visual information for an unusual entity, and the incorrect classification (group) persists. Therefore, there is still room for enhancement in future work.

heading: 4 Conclusion
text: This work presents a multi-modal framework that leverages a text-to-image model to bridge between language and visual modalities for enhancing text comprehension. The model effectively transforms text inputs into coherent images, enriching phrase representations by merging outputs from different modalities. Experimental results show our framework surpassing robust phrase understanding models across diverse domains.
Limitations
Due to the maximum input length constraint of both the CLIP text encoder and the text-to-image model, we are unable to process long texts. We are interested in exploring alternative prompt configurations to circumvent this limitation. Our methodology is readily extendable to these settings, making it an intriguing area of study.
Ethics Statement
Our approach leverages a pre-trained text-to-image model to visually enhance representations. However, the text-to-image model may carry over biases and improper content from its training data. This necessitates additional analyses to safeguard against any undue influence of these biases on our method.

heading: Acknowledgements
text: We thank the reviewers for their insightful comments. This work was financially supported by the Young Scholar Fellowship Program by the National Science and Technology Council (NSTC) in Taiwan, under Grants 111-2222-E-002-013-MY3 and 111-2628-E-002-016.

heading: A Datasets
text: • CoNLL2003: This dataset comprises 20,744 sentences with four distinct types of entities - persons (PER), organizations (ORG), locations (LOC), and miscellaneous names (MISC). Our experiments utilize 30,027 entities that are labeled as PER, ORG, or LOC.
• BC5CDR: This dataset features 1,500 PubMed articles that are populated with chemical and disease entities, adding up to a total of 28,354 entities. • W-NUT 2017: This dataset is an accumulation of data collected from public platforms like YouTube and Twitter, with a focus on distinguishing previously unseen entities within emerging discussions. It includes six types of entities: person, location, group, corporation, creative_work, and product. The dataset contains a total of 3,890 entities. • MIT-Movie: This dataset includes 12,218 sentences populated with title and person entities, accounting for a total of 9,920 entities.
B Implementation Details
In our work, we use the Huggingface models to generate all the representations:
• BERT/RoBERTa: We take pooler_output as the representations, where pooler_output is the classification token after processing through a linear layer and an activation function.7 The linear layer weights are learned by next sentence prediction during pre-training. • LUKE: entity_last_hidden_states is used as the representation, which is the last hidden states of the input entity.8 • Phrase-BERT: Phrase representations can be easily acquired by calling model.encode().9 • UCTopic: We obtain the phrase representations with the released source code.10 • CLIP: pooler_output is taken as the representation for both the text encoder11 and the image encoder.12

heading: C Pre-trained Models
text: For the pre-trained CLIP model, we adopt the version ViT-B/32, which consists of a ViT-B/32 image encoder and a 12-layer Transformer text encoder.
7https://huggingface.co/docs/transformers/ main_classes/output#transformers.modeling_ outputs.BaseModelOutputWithPooling.pooler_output
8https://huggingface.co/docs/transformers/ model_doc/luke#transformers.LukeModel
9https://huggingface.co/whaleloops/ phrase-bert
10https://github.com/JiachengLi1995/UCTopic/ blob/main/clustering.py#L43
11https://huggingface.co/docs/transformers/ model_doc/clip#transformers.CLIPTextModel
12https://huggingface.co/docs/transformers/ model_doc/clip#transformers.CLIPVisionModel
For the text-to-image diffusion model, we use stable diffusion v2-base13 trained on the subset of LAION-5B (Schuhmann et al., 2022) in our experiments.
D Inference Details
We conduct our experiments on single V100 GPU. • Generation time of stable diffusion v2-base
with respect to inference steps is elaborated in Appendix E. • Each clustering experiment takes no more than 10 minutes to run.
D.1 Licenses • BERT (Apache License Version 2.0) • RoBERTa (MIT License) • LUKE (Apache License Version 2.0) • Phrase-BERT (T License) • UCTopic (MIT License) • CLIP (MIT License) • vit-base-patch32-224-in21k (Apache License
Version 2.0) • stable-diffusion-2 (CreativeML Open
RAIL++-M License)

heading: E Efficiency vs. Efficacy
text: Results over different inference steps of stable diffusion v2-base are shown in Table 5. It took 0.84 seconds per image for inference step 10, 2.02 seconds per image for inference step 30, and 3.24 seconds per image for inference step 50. The balance between efficiency and efficacy depends on application usage.
13https://huggingface.co/stabilityai/ stable-diffusion-2-base
ACL 2023 Responsible NLP Checklist

heading: A For every submission:

heading: 3 A1. Did you describe the limitations of your work?
text: section: Limitations (between Conclusion and Reference)

heading: 3 A2. Did you discuss any potential risks of your work?
text: Ethnics Statement

heading: 3 A3. Do the abstract and introduction summarize the paper’s main claims?
text: section: Abstract, 1. Introduction
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts? section: 3.1 3.2 Appendix

heading: 3 B1. Did you cite the creators of artifacts you used?
text: section: 3.1 3.2 Appendix Reference

heading: 3 B2. Did you discuss the license or terms for use and / or distribution of any artifacts?
text: section: Appendix
7 B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? We have followed the licenses and don’t have a conflict with the artifacts’ intended use.
7 B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? We have checked that we don’t use images related to people in our paper.
7 B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? We have referenced the datasets by citing them and attaching the URLs in the paper.
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. section: Appendix
C 3 Did you run computational experiments? section: 3. Experiments

heading: 3 C1. Did you report the number of parameters in the models used, the total computational budget
text: (e.g., GPU hours), and computing infrastructure used? Appendix
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.

heading: 3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found
text: hyperparameter values? section: 3. Experiments

heading: 3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary
text: statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? section: 3. Experiments

heading: 3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did
text: you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? section: Appendix
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants? Left blank.

heading: D1. Did you report the full text of instructions given to participants, including e.g., screenshots,
text: disclaimers of any risks to participants or annotators, etc.? Not applicable. Left blank.

heading: D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students)
text: and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? Not applicable. Left blank.

heading: D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to
text: crowdworkers explain how the data would be used? Not applicable. Left blank.

heading: D4. Was the data collection protocol approved (or determined exempt) by an ethics review board?
text: Not applicable. Left blank.

heading: D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data?
text: Not applicable. Left blank.

heading: Future Work
text: Therefore, there is still room for enhancement in future work