Learning the Visualness of Text Using Large Vision-Language Models
Authors: Gaurav Verma
Ryan A. Rossi
Christopher Tensmeyer
Jiuxiang Gu
Ani Nenkova
Venue: EMNLP 2023 Main (2023)
Keywords: text visualness
vision-language models
multimodal learning
TL;DR: We propose the task of predicting sentence visualness, curate a human-annotated dataset, and develop a fine-tuning strategy to predict sentence visualness using large vision-language models.

Abstract: Visual text evokes an image in a person's mind, while non-visual text fails to do so. A method to automatically detect visualness in text will enable text-to-image retrieval and generation models to augment text with relevant images. This is particularly challenging with long-form text as text-to-image generation and retrieval models are often triggered for text that is designed to be explicitly visual in nature, whereas long-form text could contain many non-visual sentences. To this end, we curate a dataset of 3,620 English sentences and their visualness scores provided by multiple human annotators. We also propose a fine-tuning strategy that adapts large vision-language models like CLIP by modifying the model's contrastive learning objective to map text identified as non-visual to a common NULL image while matching visual text to their corresponding images in the document. We evaluate the proposed approach on its ability to (i) classify visual and non-visual text accurately, and (ii) attend over words that are identified as visual in psycholinguistic studies. Empirical evaluation indicates that our approach performs better than several heuristics and baseline models for the proposed task. Furthermore, to highlight the importance of modeling the visualness of text, we conduct qualitative analyses of text-to-image generation systems like DALL-E.

Learning the Visualness of Text Using Large Vision-Language Models
Gaurav Verma
Ryan A. Rossi
Georgia Institute of Technology gverma@gatech. edu
Adobe Research
ryrossi@adobe.com com
Christopher Tensmeyer
Jiuxiang Gu
Ani Nenkova
Adobe Research tensmeye@adobeeccm com
Adobe Research
Adobe Research
i jigu@adobe. com
nenkova@adobe.com com
Abstract
Our contributions
Visual text evokes an image in person's 's mind, while non-visual text fails to do so. A method automatically detect visualness in text will enable texi-to-image retrieval and generation models to augment text with relevant images. This is particularly challenging with long-form text as text-to-image generation and retrieval models are often triggered for text that is de- signed to be explicitly visual in nature, whereas long-form text could contain many non- sentences. To this end, curate dataset of 3,620 English sentences and their visual- ness scores provided by multiple human an- notators. We also propose fine-tuning strat- egy that adapts large lasoun-a models like CLIP by modifying the model's contrastive learning objective to map text identified as non- visual to a common NULL image while match- ing visual text to their corresponding images in the document. We evaluate the proposed ap- proach on its ability to (i) classify visual and non-visual text accurately, and (ii) attend over words that are identified as visual in psycholin- guistic studies. Empirical evaluation indicates that our approach performs better than several heuristics and baseline models for the proposed task. Furthermore, highlight the importance of modeling the visualness of text, we conduct qualitative analyses of text-to-image generation systems like DALL-E.
Whan the pardene ofter cown fiist apotar
"viuua teat"
fouum partorming the
meneurrs
teet Impur
Junann arroaned
buned identfying visual best
case Ludgeentt
nakon against you by derbut complent.
Impss
haner-asirtttd 1.87
mnon-rtuull baxt"
Figure 1: Overview of the sentence visualness identifi- cation task, along with motivating downstream appli- cation (passive generation of relevant images).
not evoke visual image. To this end, we introduce the task of identifying sentence visuahness-a term we use interchangeably with imageability- as a necessary first step toward connecting long-form textual documents with relevant visual assets, with- out having to manually find visual sentences. In other words, to work effectively with long-form text without relying on manual input, text-to-image generation models like Stable Diffusion, DALL-E, and Imagen (Saharia al., 2022) would benefit from inferring text visualness before they can gen- erate images to embellish textual documents. In Figure 1, we demonstrate the need with some ex- amples: text identified to have low visualness leads to irrelevant generations from DALL--E, while text identified to have high visualness leads to the gen- eration of relevant images.
1 Introduction
People typically communicate knowledge and information textually, but most prefer to con- sume visually rich content. Text-to-image genera- tion/retrieval models could augment text with ap- propriate images, aiding the creation of appealing and cany-to-und documents. Models like DALL-E (Ramesh al., 2022) and Stable Diffu- sion (Rombach et al., 2022) work phenomenally well for input text that carefully constructed to clicit images. However, they cannot handle long- form text with a mix of sentences that may or may
Prior approaches for quantifying the visu- alness of text operate on a word or phrase level (Deschacht and Moens, 2007; Jeong et al., 2012) and leverage lexicons that contain human- assigned world-level imagcability scores (Louis and Nenkova, 2013). However, besides being lim- ited in their coverage, our experiments also show that word phrase-level visualness cannot be ag- gregated to quantify sentence-level visualness.
To this end, in this work, we curate a corpus of 3,260 sentences in English paired with their human

---

ratings for visualness, as well as a noisy but large corpus of 48,077 automatic alignments between text and visual assets long-form documents. The textual part of the resulting alignment pairs can be used as examples of visual and non-visual sen- tences. We propose a strategy to fine-tune vision- language models like CLIP, allowing classification inferences over text-only inputs. Our objective also ensures that the learned embeddings remain usable for downstream text-to-image retrieval.
the task-specific dataset (Mittal al., 2022: Sarto et al., 2022). Our work differs from existing work in that the input is only text, requiring us to adapt large vision-language models to not rely on both modalities during inference. We propose a fine- tuning strategy that does not involve additional ar- chitectural components (and parameters) top of pre-trained CLIP architecture and yet effectively adapts CLIP for learning text visualness. Our task can be considered a precursor tasks like text-to- image retrieval and generation, where images are only retrieved or generated for visual text. Further, since reusability of representation is a desirable property (Yosinski et al., 2014; Long et al., 2015) we aim to preserve the reusability of text embed- dings learned for the visualness categorization task for downstream tasks like text-to-image retrieval. Visualness of words: The visualness of text has been studied multiple prior works but at word or phrase level. Colthcart (1981) curated the MRC Psycholinguistic Database comprising human rat- ings for imageability of 3769 words, which were later expanded using automated methods by Louis and Nenkova (2013). Beyond word-level visual- ness, some studies have focused automated quantification of phrase-level visualness (Jeong et al., 2012; Deschacht and Moens, 2007). Our work focuses on learning sentence-level visualness instead of word or phrasc-level visualness. While it is possible to aggregate word-level and phrase-level visualness scores to obtain sentence- scores, it is unclear how accurate and generalizable these techniques are. We design multiple baselines that aggregate word-level scores to obtain sentence- level visualness and contrast the performance of such approaches with our proposed approach.
We compare the performance of our proposed approach against several heuristic and model-based baselines. Our extensive evaluation suggests that our fine-tuning strategy leads to the most accu- rate visual and non-visual text classifier. Finally, conduct several analyses to glean insights into the model's 's learned attention mechanism, text-to- image retrieval abilities, and downstream text-to- image generation capabilities.
In sum, our key contributions are:
We propose the task of identifying the visualness of sentence and curate a dataset by crowdsoure- ing annotations for English sentences.
We develop training objective that fine-tunes large vision-language models for the task of text visualness identification.
Quantitative and qualitative experiments demon- strate the effectiveness of our fine-tuning approach in identifying visual text over several competitive baselines, while preserving downstream text-to- image retrieval performance.
2 Related Work
Fine-tuning vision-language models for down- stream tasks: Large vision- vision-language models like CLIP (Radford et al., 2021), UNITER (Chen et al., 2020), and ALIGN (Jia et al., 2021) have demon- strated remarkable performance on downstream tasks via transfer learning or fine-tuning. How- ever, such downstream tasks assume both text and image as input to determine similarity or gener- ate/retrieve the other modality for every instance of the corresponding modality; for instance, vi- sual question answering (Antol et al., 2015), cap- tion generation (Xu et al., 2015), and cross- modal retrieval (Wang et al., 2016). Fine-tuning large vasion-language models on such downstream tasks involves adding components to the encoders' ar- chitecture and training additional parameters on
3 Text Imageability Dataset (TImeD)
Our proposed fine-tuning approach follows multi- stage training of large rision-languagy model CLIP (Radford et al., 2021). In the first stage, we conduct large-scale fine-tuning, followed by fine- tuning on a relatively smaller annotated corpus in the second stage. We first discuss the curation of a large-scale corpus that comprises automatically- assigned and distant labels and then describe the curation of the non-visual sentences.
corpus of visual &
3.1 Fine-tuning with automatic labels
The formulation of the training objective (discussed later) requires positive examples comprising vi-
Project wehpage: https://gaurrav22vemm. ti thub. io/text- visullsssssllesss/

---

Categury
Example text from TiMED
Rio
the snow has malled and the gras not only looks dirarr, but il saggy.
6.88
openation
six zipper scuc on chest.
A 6.44
When the gardens open, just after dawn, the first uppear are the joggers and the silest figuees performing the intricute mineuvers of tai chi,
Visual
He removed the bos, placed next girhago and put gurhage inside the can.
5,88
Rui, after running only the fins 500 meters, realived tlut the injury that scomod so insigiifiaai woukd only prewett him from winning the mos. but also from finishing
There's 's only way prove wrong.
#= 1.22
For more infocmation or to schodule as outreach, please call (999) 123- 4567 or email e wesiiiweebee..... com.
= 1.55
Non-visual
In case of your failuru anower, jalgmmnnt will be taken agains by defaslt for the rolicf demardal the complaint.
1.67
25% quorum member votes
district needed to condact district delegate elections October.
p- 1.77
Colliers Insernational makes so guarantees, representatioss or warranties of any kind, expressed or implicd, regarding the information including. but 2.00
limited to, samaantcc
accaracy raliability
3. Romam discrsees book Cliin State Foobball: The Forgotion Down which on extersive anchival reseanch to tell the mntuld story of the carly 2.34
days of football Ohio tlagship public university
Remember be sure set your clocks back
before you go bed Saturday, November
Ambignous
That is the most impottan thing in my life tolay: Jesus.
2.20
-hiidron & parenis will get hear author Geonge McClements ned his book Rulin' Dinos with Buck Brunco.
2.14
Fiuncial Peace University nine-lesson cluss taught by finaccial expert Dave Runsey through entertuining videos with an deydh workhook, that will teach you how to take control of your money.
2.16
Table Qualitative examples of visual and non- non-visual text from the human-annotann annotated subset of the Text Imageability Dataset (based on the average of annotator ratings), and text with high ambiguity (based on the standard deviation of annotator ratings).
sual text and paired images as well negative examples that comprise non-visual text. To cre- ate corpus like this, we: (i) leverage image-text co-occurrences in documents to develop a self- supervised approach, and (ii) use image-text simi- larity scores obtained using CLIP as priors to con- struct a large training corpus. We start with 450,000 publicly available PDFs referenced in the Common Crawl corpus and identify pages within those PDFs that include images. We use a proprietary doc- ument object detection tool like Fitz² to extract paragraphs and images from the document pages.
with a common NULL image. The thresholds Tpos and T They are chosen conservatively only include top or bottom k % sentences from the entire corpus, respectively. This limits the noise in our training corpus for adapting the CLIP model for scoring text visualness. In our experiments, we set Tpos to be 0.35 to consider top 1% sentences as visual and Theg to be 0. 0.18 8 to consider bottom 5% sentences as non-visual. Our automaticalll-llaelee corpus com- prises 15,359 visual sentences, the corresponding images, and 32,718 non-visual sentences.
We do sentence segmentation for the identified paragraphs using NLTK Tokenizer (Loper and Bird, 2002). To map the images in the page to sentences, we compute CLIP similarity scores be- tween each image-sentent pair in given page. Based on the distribution of image-sentence simi- larity scores across all the pages in our corpus, we set two thresholds, Tops and neg. A sentence in a page is considered a positive example (visual text) if its similarity with any of the images in the page is greater than Tpos Similarly, chosen negative ex- amples have similarity values less than They with all images within the same page. Sentences with an image similarity value greater than Tos are as- sociated with the most similar image in the same page, while the negative examples are associated
3.2 Human-annotated dataset
For the human-annotatee visual and non-visual ex- amples, we start with another 200,000 PDFs dis- tinct from those used for the automated assignment of labels. To focus on natural images rather than in- fographics and academic figures, we filtered these documents to only include brochures, flyers, and magazines. For the resulting 35,432 documents, we adopted the same policy as that for curating the automatically-lablled dataset (selecting top 1% and bottom 5% sentences based on similarity val- ues). We then recruited annotators rate the visual- ness of the resulting 3,620 sentences after manually anonymizing any personal information.
We recruited annotators on Amazon Mechani- cal Turk (AMT). We randomly ordered the 3,620 examples and, for each example, we asked nine an- notators provide response on 7-point Likert scale for the following question: "Do you agree that the sentence below evokes an image or picture in your mind?" A response of indicated strong disagreement, while 7 indicated strong agreement.
"e choose to work with PDF documents rather than web- pages because (i) PDFs have natural demarcations the form of pages (whereas webpages often contain long-running text with complex image-text interactions), and (if) images within a page are likely to be related to selected text fragments within the same page.
Phttps: http:/ggtbb.cmm///pgg///// cww/pyympdf//yyMePe...

---

We also inserted some attention-check examples (5%; = 181) to ensure the annotators read the text carefully before responding. These checks ex- plicitly asked the annotators to mark a randomly chosen score on the Likert scale regardless of the actual content. We discarded the annotations from annotators who did not correctly respond to all the attention-check examples and re-collected more re- sponses iteratively. Appendix A.3 provides more details about the filters used for recruiting the an- notators and the annotation interface.
(MAA NOT
2-14 sablly nost fris
upproound
frsttd soainding Entind
thin eacomess angunn
OnE not
Tr
T2
Tien
Tи
hTs
Tw
...
heat::
:
..e
To
Irage
If a majority of annotations (i.e.. at least 5 out of 9) were 1, 2, or 3, we considered the example to be non-visual (n 2108). Similarly, visual ex- amples had majority of 5, 6, or responses (n 1132). We considered examples that did not have a clear majority or majority of responses of 4 (i.e., 'Neutral' on the Likert scale) as ambiguous and neutral, respectively. Table 1 shows illustrative examples of vi visul,, non-visual, and ambiguous text from our numan-annotatte corpus.
MULL
Figure 2: Our approach to predicting sentence visual- ness, with a fine-tuning strategy where visual text is matched with its corresponding image while nin-visual text is matched with fixed NULL image.
text (text that does not evoke image). In line with the original training objective, we further train the CLIP model to match text that identified as visual with the corresponding image. We adapt the CLIP training to match text that is identified as non-visual with a single NULL image (see Fig. 2). Matching visual text with the corresponding image while non-visual text to a NULL image not only encourages the model to distinguish between visual and non-visual text, but also allows it anchor non- vosuvl text in the common NULL image that can be used during inference without having access to a potentially paired image. Formally, the adapted training objective is given as,
For 27. 27.1% of the examples only most of the 9 annotators disagreed with the labels decided based on the process described above. 10.5% of the sentences were assigned a neutral or ambigu- ous class. Inter-annotator agreement measured by Krippendorff's a was 0.446. This inter-annotator agreement value is in similar range to what is observed for other language-related tasks that in- volve assessment of text by experts dimensions like coherence, likability, relevance, and even gram- mar (Karpinska et al., 2021). For brevity, we refer to the curated dataset as TIMED, short for Text Imageability Dataset.
4 TIP-CLIP for Scoring Text Visualness
C
1
Σ
log
ΣN capl (15. (1)-1//)))
(1 (5,TT)//
2N
j=1
Background: The CLIP model (Radford et al.. 2021) jointly trains image and text encoders to pre- dict the correct pairing between images and textual descriptions. In batch size of N images and N texts (N2 possible image-text pairings), the objec-
N
cxpl(TE: TB)/T)
log
2N
Σ expl( TB) /T)
k=1
tive function ensures that the cosine similarity be-
st. 1
Je
Bull
m E (i.e., non-visual)
tween the embeddings of correct image-text pairs maximized while the cosine similarity between the
if E V (i.e., visual).
m=
(1) Here, N denotes the number of examples in batch, I and T denote the embeddings of the m-th pair of image and text that are normal- ized have unit fn-norm, respectively, such that m E {1,..., N). (...) represents the inner prod- uct, and 7 is the trainable temperature parameter. y and are the set of examples in the current batch
(N2 N) incorrect image-text pairs is minimized. The encoders are trained over large multimodal dataset of ~ 400 million image-text pairs.
Updated training objective: When predicting text visualness, the goal to assign higher score to text that is visual (evokes a concrete image for the person reading it) and a lower score for non-visual

---

that belong to non-visual and visual categories, re- spectively. Finally, / denotes the embedding of the NULL image. During inference, we compute the cosine similarity between the representation of a given text with the representation of the NULL image; non-visual texts will have high similarity with the NULL image. Conversely, the visualness score S of any text with embedding T can be ob- tained using
'44 GPUs, for 5 epochs, and a learning rate ini- tialized at 5 10-5 and optimized using Adam optimizer (Kingma and Ba, 2014). Following this, for the second stage, we further fine-tune the same model for 2 epochs using the same ob- jective and the train set of human-annotated TIMED. 4 The hyyer-parametten are selected by performing grid search while observing performance on the vali- dation set of TIMED. Based on the performance on the validation set of TIMED, we set the thresh- old S (Eq. 2) to be 0.79 to categorize text as visual or non-visual. We refer to the model trained using our fine-tuning strategy as TIP-CLIP Text Imageability Predictor CLIP, and report performance on the test set of TIMED.
but this time using
S = 1- (TUUL T*)
(2)
For the NULL image, we create an RGB image of size (221, 221, 3) in which each pixel value is chosen randomly (see Figure 2). However, experi- ments with different types of NULL images indicate that the choice of null image does not affect the model's s performance: see Appendix A.
5.1 Baselines
An alternative formulation for adapting the CLIP training objective could have been match visual text with a single image while matching non-visual text with single NULL image. However, this formu- lation of the training objective similar to binary classification and does not enforce a contrastive objective for the positive examples. Matching vi- sual text with its corresponding image instead of a common image for all visual text affords text em- beddings that can be used for downstream tasks like text-to-image retrieval; we provide empirical evidence for worse text-to-image retrieval perfor- mance with the alternative formulation in Results.
We investigate the performance of TIP-CLIP against several heuristics and baseline models.
Random: The random baseline generates predic- tions via prior class probabilities in the training set. Average MRC-I score: We consider the image- ability scores of 3,769 words in the MRC lexicon and normalize them to be [0, 1]. For each exam- ple, we take the average of the imageability scores of the unique words; vocabllvyya words are assigned a score of O. We lowercase the words the MRC lexicon as well as the input text. Based on this average score, we categorize an example as visual or non-visual by setting the decision boundary 0.17. The threshold is chosen to opti- mize performance on the validation set of TIMED. Concentration of Visual Genome objects (VG- Objects): The Visual Genome dataset comprises 75,729 objects, along with annotations for their at- tributes and object-object relations (Krishna et al.. 2017). Based on the heuristic that a mention of a visual object in the text can trigger imageability, we quantify the concentration of Visual Genome objects by computing the fraction of unique object mentions tokenized text with respect to the num- ber of total unique words within the input text. We set the threshold to 0.5 based on the performance on the validation set.
5 Training details and Baselines
Train, test, & validation splits: Recall that our fine-tuning approach requires paired images for vi- sual sentences only during training time and not during inference time; the model needs only text input during inference. Of the 1132 visual sen- tences in the human-annotate set of TIMED, we assign 515 examples that had automatically de- termined corresponding image to the training set, and the remaining were randomly assigned to the test set (n = 517) and validation set (n = 100). The 2108 non- visual sentences were randomly split into the training (n = 980), test (n = 928),
and validation set (200). All three sets maintain bositive:ne negativeeeiveeee ve class ratio of ~ 0.5.
Expanding the MRC lexicon using word embed- dings: The coverage of the MRC lexicon is poor be- cause it contains only 3,769 words. We expand this
For the first stage of training, we fine-tune the CLIP model (ViT/B-32) on the proposed objec- tive (see Eq. 1) using the 48,077 examples with automatic labels. This training is done on Tesla
"The CLIP model has maximum context length of 77 tokens (about 50 words). Fewer than 1% of the training exam- plcs are truncated to fit this context length.

---

list using semantic similarity between distributed representations of words (300-dim word2vee vec- tors trained on Google News corpus). For each word w in the word2vec (Mikolov et al., 2013) vo-
MODELS
Fi
PRECISION
RECALL f
Acc.
Random
0.531
0.531
0.531
0.577
MRC-I
0.584
0.599
0.583
0.644
VG- Objects
0.606
0.610
0.605
0.646
cabulary of pre-trained representations that does
MRC-I+ w2v
0.638
0.637
0.639
0.667
not occur in the MRC lexicon, we compute its co- sine similarities with all the words in the MRC lex- icon to identify the most semantically similar word that exists in MRC, given by WMRC and its similar- ity with w given as (simmaa). We assign the word w an imageability score of sim X scorewaae> where scorewwee the normalized imagcability score of w's most similar word WMRC. Based on the performance on the validation set, the decision boundary for average imageability score of input text is set 0.17. This baseline propagation ap- proach is highly effective in quantifying word-lewe imageability as the Pearson's correlation coefficient between the assigned visualness score and the aver- age AMT rating of humans is 0.735 (p < 0.001); see Appendix A.2 for details.
BERT
0.753
0.766
0,789
0.756
CLIP TIP-CLIP (Ours)
0.694
0.695
0.701
0.712
0.865
0.858
0.873
0.871
Table 2: Evaluation human- annotttde test of TIMED. Reported F,, Precision, and Recall values are macro-averages across the two classes (visual and non-visual).
and classification accuracy. Table show the re- sults for this evaluation. We observe that our pro- posed two-stage fine-tuning strategy leads to the best-performing model (TIP-CLIP). In comparison, the pre-trat pre-trained CLIP model demonstrates notably weaker performance on the task of distinguishing visual text from non-visual text. Interestingly, fine- tuned BERT performs reasonably well on the task, considerably better than the CLIP model. Using the average imageability scores from MRC provides better-than--andonn performance but is severely subpar to models like CLIP, BERT, and TIP-CLIP. Using word2vec embeddings expand the cover- age of the MRC lexicon (i.e., MRC-1 + w2v) leads to a boost in performance. However, collectively, the lacking performance of MRC-I and MRC-I + w2v demonstrates that word-level imagcability does not translate to sentence-level imageability to a great extent. Notably, in terms of baselines that aggregate word-level attributes, VG-Objects provides the best estimate of sentence-level image- ability by quantifying the concentrations of visual objects in the input sentence.
Fine-tuned BERT classifier: We fine-tune a BERT HuggingFace (De- vlin al., 2018; Wolf et al., 2020)) for the clas- sification task of visual versus non-visual text detection. Similar to our proposed model, we adopt a two-stage fine-tuning approach with the BERT classifier (adding a classification layer to BERT for the first input token's ([CLS]) representation). We first fine-tune the model using the automatically labeled dataset followed by fine-tuning on the train- ing set of the numan-cuuater TIMED. For the first stage, we fine-tune the model for 7 epochs with a learning rate initialized at 5 x 10-5 using a batch size of 32 while setting other default. We fine-tune the model for 3 epochs for the second stage with the same hyperparameters. Pre-trained CLIP model: We use the pre-trained CLIP model (ViT/B-32) to obtain similarity scores between the embeddings of the NULL image (used for the fine-tuning of our model) and the input text, We then use 1 (TULL T as an estimate of the visual score of text (sec Eq. 2). Based on the per- formance on the TIMED validation set, we set the threshold for S to be 0.83.
model (bert-basee tastt--bb---ucseed
to
Correlation of attention Weights with MRC im- ageability scores: Attention mechanisms could be taken as proxies for explainability (Wiegreffe and Pinter, 2019; Chefer al., 2021). Since the fine- tuned BERT, pre-trained CLIP, and our TIP-CLIP are attention-based models, we compute the corre- lation between average word-level attention scores (obtained from the last layer) on a given dataset with the imageability scores assigned by humans in the MRC lexicon. We compute these values for two datasets MSCOCO dataset (Vinyals al., 2016) and the test set of TIMED. We only consider words that occur more than once in the specific cor- pus. Table 3 shows that TIP-CLIP attention scores correlate the most with MRC imageability scores,
6 Results and Analyses
Evaluation on held-out test set of TIMED: We first evaluate the baselines and our approach on the test set of the human-annan annotated TIMED, com- puting macro- averrged F,, precision, recall scores,

---

MODELS
MSCOCO
TIMED
as the standard pre-trained CLIP model.
BERT
0,461 in 344)
0.326 (n= 294)
Effect on text-to-image retrieval: We aim to ana- lyze the re-usability of learned embeddings by the TIP-CLIP model for the text-to-image retrieval task. To this end, we consider the 515 visual examples from the test set of TIMED and, for each visual example, we rank the 515 corresponding images based on the cosine similarity between the image and text embeddings obtained from the TIP-CLIP model. We compute the Mean Reciprocal Rank (MRR) and contrast it with the MRR obtained us- ing the pre-trained CLIP embeddings. As expected, CLIP achieves a near-perfect MRR of 0.989. The
CLIP
0.448 in 344)
0,283**0 (n 294)
TIP-CLIP (Ours)
0,497-** 344)
0.367 =
Table 3: Correlation between MRC Imageability scores and model attention-scones for BERT, CLIP, and TIP- CLIP. denotes the number of overlapping words across vocabularies; *** denotes < 10
MODELS
A1
PRECIsION
RECALL 1
ACC. 1
BERT (auto-labeled)
0,714
0.701
0,716
0,700
RERT (lumun-labelaa)
0.53
8.766
0.789
BERT (auto human laboled)
0.734
0,783
0,797
0.771
0.701
TIP-CLI (aute-bbekal)
0.751
0.763
0.791
0.788
TIP-CUI mumann-lbbleed
0.10
0.817
proposed fine-tuning objective does not severely impact the reusability of embeddings obtained from TIP-CLIP for retrieval, and results in an MRR of 0.937. This comparison evaluates the retrieval ca- pabilities of TIP-CLIP against that of the CLIP model because the correspondence between visual text and images was established using similarities between CLIP embeddings."
TIP CLPP (auto human labeled)
0,865
0,858
0.873
0.871
Table 4: Ablation studies to understand the benefits of two-stage fine-tuning. The presented results are on the human- anmaattn test set of TIMED. Reported values are macro-averages of class-wise Fi, precision, and recall, and overall classification accuracy.
followed by the fine-tuned BERT's attention scores. The trends are consistent across both datasets. The relative ordering of models in terms of the corre- lation of their attention scores with MRC image- ability scores follows the same order as their per- formance the test set of TIMED. However, all correlation scores are in the low range, indicating a non-trivial relationship between sentence- and word-level imageability. The same trends hold for propagated visualness scores; see App. A.4. We also analyze the reason behind higher correlation scores on MSCOCO with respect to the TIMED corpus in Appendix A.4.
The downside of an alternate training objec- tive: Recall that our fine-tuning strategy involves matching visual text with its corresponding im- age and matching non-visual text with the NULL image. With only the classification of visual and non-visual text in mind, an alternate fine-tuning strategy would have been to match all the visual examples with one common image while match- ing all the non-visual text with the common NULL image. The major downside of this approach is that while leads to an effective classifier after two-stage fine-tuning, demonstrating a compara- ble F1 score of 0.842 as the TIP-CLIP model, it performs poorly on the text-to-image retrieval task with MRR of 0.014. Overall, while the alternate entirely classification-basse training objective per- forms at par with the proposed TIP-CLIP model on the classification task, the resultant embeddings demonstrate poor reusability for downstream tasks like text-to-image retrieval.
Effect of multi-stage training: We conduct abla- tions to isolate the effect of two-stage training. In Table 4, we show that BERT and TIP-CLIP can learn to distinguish visual and non-visual text even when fine-tuned only using the automatically labeled data. However, for both models, the gains from fine-tuning only on smaller, human-labeled data are notably higher. Furthermore, we find the proposed two-stage fine-tuning (i.e., training on automatically labeled data followed by human- labeled data) to be most effective, leading to gain of over 2 and 5 absolute Fi points over training only on human-labeled data for BERT and TIP- CLIP models, respectively. Additionally, for given training strategy, our proposed fine-tuning of TIP-CLIP demonstrates better performance than the corresponding fine-tuned BERT model as well
Properties of the new embedding space: In Figure 3 we visualize the embedding space of the learned embeddings using t-SNE (Van der Maaten and Hinton, 2008), Alongside visual and non-visual sentences from the test set of TIMED,
"To automatically establish a correspondence between visual text and images, we enforce that the most similar image for a text should exist on the same page of the PDF. Therefore, it is possible that the CLIP similarity of text may he higher for a different image, resulting in an MRR slightly less than 1.0 (i.c., 0.989).

---

image 1
Iriegea vigual text
Nonvisual 1est
NULL image
visual bost
NULL
Images for vasual food
Non-dteua tect
NULL Image
Visual
Varul lood
Visual text
(a) CLIP embeddings
(b) TIP-CLIP embeddings
(c) Alt. formulation embeddings
Figure 3: L-SNE visualization of embeddings learned by (a) CLIP, (b) TIP-CLIP - using contrastive and adapted contrastive learning objective, respectively, & (c) model trained using alternative formulation solely focusing classification. The plotted data points are from the TIMED test set. The observed "gap" in image & text spaces has been studided by Liang et al. (2022).
Inpul 201t
Origiaa imag CLP startze TECCLP stantion
Insat sot
OVEP olggeei CUT otteraat TIP anartinn
CUP frandly
ingrediant the
8. DOOR as juica S. unch
que amaarpeie many plcasil
CL---L--- dog menddy SUBB an wy ingredient chare
1 BATE 91 price of& S- O sNCT
CLI- conmen innonstt's 'n vite Loady likes
dong WoRks and one mil Bmil ties coage
(kins shore trea
TIP CLI- the comeoniion ios. rirrasza''
felowing CANoAAL BEOME and rnd Bruth coagh.
Figure 4: Comparing the attention maps over input text and images for CLIP and TIP-CLIP. For text, darker shade of green demonstrates greater attention by the model. For images, red demonstrates the greatest attention in the heatmap. Image best viewed with zoom.
(+) DALL-E generations for non-visual text
(b) DALL-E ge generations for visual text
There's only one way
now the snow has meted and the gr only looks
wrong
dreary. soggy.
cass of your fallaw to answer, jucgment will be
Dicieit 23
Zeitrn The
Thit Dechonaf
ACHPIPACT
When the gardhers upen, just after dawn, the first to appear joggara in sikent tigures performing
taken against you dufaul for roliet demanded compaint.
haddl ucoteatur
PIEER moar RESSUTES
wetesee May
rccuonnfing
DEEOUTOOATIO
Collerss Interaaionaal macess no gue
after
first
warrantee regarding ninn the informtion indluling warranses of content, accuracy and eilibiiity.
COLIRTERS
COLLEETIAL COUTTEANTS
COUCTERS
500 moter's, he realbed that the lijaly that suereed se
ATTT
CAATEEATTER
insignitant would not unly
fivihing
Figure 5: Examples of DALL-E generations for non-visual and visual text.
we also plot the embeddings of images correspond- ing to the visual sentences, and the embedding(s) of the NULL image(s). First off, we observe that the embeddings in Figure 3a and 3b from CLIP and TIP-CLIP are different in that the TIP-CLIP em- beddings demonstrate better distinguishability be- tween visual and non-visual text. In Figure 3c we observe that the alternative formulation pushes the NULL embeddings to the periphery of the image embeddings' cluster from a near-center location in Figures 3a and 3b. The text embeddings demon- strate notable distinguishability in Figure 3c too. We believe that the alternative classification-only formulation causes distortion in the latent space that causes drastic modification of text-only embed- dings, making them useless for downstream text-to- image retrieval, as demonstrated empirically earlier. However, our proposed objective TIP-CLIP pre-
serves reusability for downstream tasks by main- taining semantic relevance between learned image and text embeddings.
6.1 Qualitative Analysis
In this section, we conduct two qualitative analy- ses: (i) contrasting the attention mechanisms for CLIP and TIP-CLIP, and (ii) the role of distinguish- ing visual and non-vi sual text in downstream text-to-image generation using systems like DALL- (Ramesh et al., 2021).
A Attention map visualization: To contrast the mechanism by which CLIP and TIP-CLIP mod- els match input text with their corresponding im- age, we visualize and contrast the attention maps for both models. We adopt the sfate-of-the-nn ap- proach to explain multimodal Transformers (Chefer

---

al., 2021). In Fig. 4 show 4 illustrative visual sentences from the test of TIMED along with their corresponding images. Focusing on text, we observe that TIP-CLIP has greater tendency to attend to visual aspects the text; for instance, words like 'islands,' 'lakes," "anglers' are attended to greater extent by TIP-CLIP than CLIP. In im- ages, we observe small changes in attention maps across CLIP and TIP-CLIP; for instance, while the CLIP attention is focused on the Common Loon, TIP-CLIP also attends to the 'lake.' The qualitative analysis of visualization maps reinforces that the matching process for text and images undergoes small changes to accommodate greater attention to visual aspects in the text.
In the future, we will study alternate objectives for learning text visualness while ensuring that the learned representations are transferable to related downstream tasks. We are also interested in us- ing measures relating to the quality of the images generated from text-to-image generation systems to decipher signals about the visualness of input text, enabling the creation of auto-labeled exam- ples. As the aggregation of word-level visualness scores leads to poor predictability of sentence-level visualness, future work could aim to understand what linguistic factors (like compositionality) pre- cipitate sentence-level visualness.
8 Limitations and Broader Perspective
Downstream text-to-image generation: In Fig. 5 we show the generations obtained using DALL- E for text that categorized as non-visual and visual in our dataset. We observe that for non-visual text, the images produced by DALL- E show poor relevance to the text. However, for visual text the generated images demonstrate great relevance to the input text.
Limitations: As the first study on predicting sentence-level visualness, we focus on fine-tuning representative vision-and-language (CLIP) and language-only (BERT) encoders. Future studies can extend our experiments to explore the bene- fits of using other encoders to model text visual- ness. Our curated TIMED dataset only covers the English language. The notion of visualness can vary across languages and we encourage future search contrast visualness in the context of the English language with that in other non-English languages. Additionally, since US-based crowd workers provided our ground-truth annotations for visualness, the dataset reflects a predominantly Western-centree view of text visualness. It is un- clear how visualness in the text is perceived across different cultures. To this end, we acknowledge that our work and artifacts reflect West-centric views of visualness in the English language and encourage cross-lingual and cross-cultural extensions.
Triggering text-to-image generation models like DALL-E for visual text crucial to effectively use such systems in a passive setting. For instance, the authors should only be recommended to add visual assets in relevant places (i.e., for visual sen- tences) while working with long-form documents; triggering image generations for non-visual sen- tences could cause sub-optimal experiences. Thus, our contributions focus on distinguishing visual text from non-visual text the necessary first step.
7 Conclusion and Future Work
We propose the task of predicting the visualness of text and curate Human-annotated dataset of sentence-level visualness scores. Additionally, propose two-stage fine-tuning objective for the task that involves training on a distantly supervised corpus followed by a smaller human-annotated cor- pus. Comparisons with several baselines demon- strate the effectiveness of our approach in distin- guishing visual and non-visual text. We analyze the attention weights and downstream text-to- eext-to-image re- trieval capabilities of the model. Qualitative analy- sis of attention weights over textual input reinforces that our model attends to visual words to greater extent. In closing, we show qualitative examples how predicting text visualness can make text-to- image generation more effective.
Broader Social Impact, Annotations, and Datasets: The authors do not foresee any nega- tive social impacts of this work. However, our model can inherit the known biases in underly- ing models like CLIP and BERT (Agarwal et al., 2021; Garimella et al., 2021). The documents from which our datasets are curated are publicly avail- able and are mentioned in The Common Crawl corpus (https: httttp::/ccommncrrt wwwwooowwwwwooroooooooooooo org/); we abide by their terms of use. We manually anonymize instances of PII in the sentences that are annotated using Amazon Mechanical Turk and check for po- tentially offensive content. The recruited annota- tors are from the United States and are paid an hourly rate of 12 USD.

---

References
Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6900 1412.6980.
Sandhini Agarwal, Gretchen Krueger, Jack Clark, Alec Radford, Jong Wook Kim, and Miles Brundage. 2021. Evaluating CLIP: Towards Characterization of Broader Capabilities and Downstream Implications. arXiv preprint arXiv:2108.02818.
Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin John- son, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A Shamma, al. 2017. Visual genome: Connecting language and vi- sion using crowdsourced dense image annotations. International Journal of Computer Vision, 123:32-
Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Mar- garet Mitchell, Dhruv Batra, Lawrence Zitnick, and Devi Parikh. 2015. VQA: Visual question answering. In Proceedings of the IEEE International Conference on Computer Vision, pages 1425-2433.
73.
Weibin Li, Qiwei Zhong, Qingyang Zhao, Hongchun Zhang, and Xiaonan Meng. 2021. Multimodal and contrastive learning for click fraud detection. urXiv:2105.03567
Hila Chefer, Shir Gur, and Lior Wolf. 2021. Generie attention-model explainabilityy for interpreting bi- modal and encoder-decoder transformers. In Pro- ceedings of the IEFF/CVF International Conference on Computer Vision, pages 397-406.
Victor Weixin Liang, Yuhui Zhang, Yongchan Kwon, Serena Yeung, and James Y Zou. 2022. Mind the gap: Understanding the modality gap multi-modal con- trastive representation learning. Advances in Neural Information Processing Systems,
Yen-Chun Chen, Linjie Li. Licheng Yu, Ahmed El Kholy, Faisal Ahmed, Zhe Gan, Yu Cheng, and Jingjing Liu, 2020. Uniter: Universal image-text representation learning. In European Conference on Computer Vision, pages 104-120. Springer.
17625.
Mingsheng Long, Yue Cao, Jianmin Wang, and Michael Jordan. 2015. Learning transferable features with deep adaptation networks. International Confer- ence on Machine Learning, pages 97-105. PMLR.
Max Coltheart, 1981. The MRC psycholinguistic database. The Quarterly Journal of Experimental Psychology Section A, 33(4)-497-505
Edward Loper and Steven Bird. 2002. NLTK: the natu- ral language toolkit. In Proceedings of the ACL-02 Workshop on Teffective Tools and Methodologies for Teaching Natural Language Processing and Compu- tational Linguistics- ainguistics-ollume 1, pages 63-70.
Koen Deschacht and Marie Francine Moens. 2007. Text analysis for automatic image annotation. In Proceed- ings of the 45th Annual Meeting of the Association of Computational Linguistics, pages 1000-1007.
Jacob Devlin, Ming-War Meng Chang, Kenton Lee, and Kristina Toutanova. 2018. BERT: Pre-training of deep bidirectional transformers for language under- standing, arXiv preprint arXiv: 1800004
Annie Louis and Ani Nenkova. 2013. What makes writing great? first experiments on article quality prediction in the science journalism domain. Trans- actions of the Association for Computational Linguis- tics, 1:341-352.
Aparna Garimella, Akhash Amarnath, Kiran Kumar, Akash Pramod Yalla, N Anandhavelu, Niyati Chhaya, and Balaji Vasan Srinivasan. 2021. He is very intel- ligent, she is very beautiful? on mitigating social biases in language modelling and generation. In Findings of the Association for Computational Lin- guistics: ACI-IICNIIP 2021, pages 4534-4545.
Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg Cor- rado, and Jeff Dean. 2013. Distributed representa- tions of words and phrases and their compositionality. Advances in Neural Information Processing Systems, 26.
Jin-Woo Jeong, Xin-Jing Wang, and Dong-Ho Lee. 2012. Towards measuring the visualness of a concept. In Proceedings of the 21 ACM International Con- ference on Information and Knowledge Managemens. pages 2415-2418.
Anshul Mittal, Kunal Dahiya, Shreya Malani, Janani Ramaswamy, Seba Kuruvilla, Jitendra Ajmera, Keng- hao Chang, Sumeet Agarwal, Purushottam Kar, and Manik Varma. 2022. Multi-modal extreme classifi- cation. In Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition, pages 12393 12402.
Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoe Le, Yun-Hsuan Sung. Zhen Li, and Tom Duerig. 2021. Scaling up visual and vision-language representation learning with noisy text supervision. In International Conference on Machine Learning. pages 4904-4916. PMLR.
Allan Paivio, John C Yuille, and Stephen A Madigan. 1968. Concreteness, imagery, and meaningfulness values for 925 nouns. Journal of Experimental Psy- chology, 76( 76(1p2).1.
Marzena Karpinska, Nader Akoury, and Mohit lyyer. 2021. The perils of using mechanical turk to evaluate open-ended text generation. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 1265-1285.
Jeffrey Pennington, Richard Socher, and Christopher D Manning. 2014. GloVe: Global vectors for word rep- resentation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Process- ing (EMNLP), P),, pages 1532-1543.

---

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sas- try, Amanda Askell, Pamela Mishkin, Jack Clark, al. 2021. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, pages 8748 8763.. PMIR.
Oriol Vinyals, Alexander Toshev, Samy Bengio, and Du- mitru Erhan. 2016. Show and tell: Lessons learned from the 2015 mscoco image captioning challenge. IEEE Transactions on Puttern Analysis and Machine Intelligence, 39(4):652-663.
Kaiye Wang, Qiyue Yin, Wei Wang, Shu Wu, and Liang Wang. 2016. A comprehensive survey cross- modal retrieval. arXiv preprint FiXX:6667.06215
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. 2022. Hierarchical text- conditional image generation with clip latents. arXiv preprint: 2204.06/25.
Sarah Wiegreffe and Yuval Pinter. 2019. Attention is not not explanation. In Proceedings of the 2019 Confer- ence on Empirical Methods in Natural Language Pro- cessing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 11-20.
Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. 2021. Zero-shot text-to- text-to-image gen- eration. In International Conference on Machine Learning, pages 9821-8831, PMLR.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier- ric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, et al. 2020. Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Lan- guage Processing: System Demonstrations, pages 38 45.
Shivaen Ramshetty, Gaurav Verma, and Srijan Kumar. 2023. Cross-modal attribute insertions for assess- ing the robustness of iisionaadd-aaguuae learning. arXiv preprint arXiv:2306.1106.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser. and Bjöm Ommer. 2022. High- resolution image synthesis with latent diffusion mod- cls. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10684-10695.
Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, Aaron Courville, Ruslan Salakhudinov. Rich Zemel, and Yoshua Bengio. 2015. Show, attend and tell: Neural image caption generation with visual attention. In International Conference on Machine Learning, pages 2048- 2057- PMI.R.
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L. Denton, Kam- yar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, al. 2022. Photo- realistic lext-to-image diffusion models with deep language understanding. Advances in Neural Infor- mation Processing Systems, 35:36479-360 -34944.
Jason Yosinski, Jeff Clune, Yoshua Bengio, and Hod Lipson. 2014. How transferable are features in deep neural networks? Advances in Neural Information Processing Systems, 27.
A
Appendix
Sara Sarto, Marcella Comia, Lorenzo Baraldi, and Rita Cucchiara. 2022. Sarrivval-augmented transformer for image captioning. In Proceedings of the 19th International Conference Content-bact based Multime- dia Indexing, pages 1-7,
A.1 Effect of the NULL Image
Since all the non-visual sentences in the training corpus are mapped to common NULL image, we aim to see the effect of the chosen NULL image on the results. Recall that the NULL image used for our main experiments was obtained by creat- ing an RGB image in which each pixel value is chosen randomly. We perform the same process with a different random seed to generate another NULL image. Additionally, we use a natural image as another alternative for the NULL image. These images are shown in Figure 6. We then evaluate the resulting models the Humman-annottee test of TIMED. Table 5 shows that the performance of the models is not dependent on the choice of the NULL image. We also find no dependence between the choice of the NULL image and the performance on downstream to-image retrieval.
Krishna Srinivasan, Karthik Raman, Jiecao Chen, Michael Bendersky, and Marc Najork. 2021. Wit: Wikipedia- basee image dataset for multimodal multilingual machine learning. Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 2443-2449.
Laurens Van der Maaten and Geoffrey Hinton. 2008 Visualizing data using t-SNE Journal of Machine Learning Research, 911).
Gaurav Verma, Vishwa Vinay, Ryan Rossi, and Sri- jan Kumar. 2022. Robustness of fusion-bass based mul- timodal classifiers to cross-modal content dilutions. In Proceedings of the 2022 Conference on Empiri- cal Methods in Natural Language Processing, pages 360-374.

---

(a) Original NULL image
(b) NULL image with diff. seed
(c) Natural NULL image
Figure 6: Various NULL images used to study the effect of the chosen image on the text visualness identification task and the downstream text-to- ext-to-image retrieval task.
VARIANTS
F1
PRECISION
RECALL f
Acc. f
MRR
TIP-CLIP (Original - Fíg. 6a)
0.865
0,858
0.873
0.871
0.937
TIP TIP-CIIP (w/ diff. seed Fig. 6b)
0.867
0.854
0.875
0.872
0.934
TIP- TIP-CLIP (natural image Fig. 6c)
0.861
0.855
0.876
0.872
0.939
Table 5: Effecc of the choice of the NULL image on categorizing the human-annotated test set of TIMED and downstream text-to-image retrieval, Reported Fi, Precision, and Recall values are macro-averages across the two classes (visual and non-visual).
Category
Example words (assigned score)
martini, crabmeat, tceth, oysters, mosquitos, bracelets, motorboat, dia- monds, squirrels, cigurettes, beaches, trumpets, dolphin, caramel, cattle, portobello, libraries, chimpanzee, snorkeling, sailboat, harmonica
High imageability
reassure, militancy, inhumanly, catalyses, industrial, peacefulness, hand- woven, neurosurgery, overwashed, whooper, smails, preeminence, recluse, entrepreneur, character, insufficient, paladin, impersonal, deviously, re- cover
Medium imagcability
politologist, psycholinguistic, requirements, confirmatory, terseness, pre- formulation, offender, controversial, unhealable, monoculuralism, mis- erable, reprogrammabiiity. this, participate, attractive, determinant, dises- tablishment
Low imageability
Table 6: Qualitative examples of words that are assigned scores the high (2 0.7), medium (e (0.3, 0.7)), and low (I 0.3) range using the word2vee embedding- ambedddng--basd based propagation methodology.
A.2 Assessment of word-level imageability score propagation
words and normalized them to be E [0, 1]. We com- pute the Pearson's 's correlation coefficient between (a) the average ratings for MRC words and the nor- malized imageability scores, and (b) the average ratings for word2vec words and the imageability scores assigned via embedding- based propagation. The correlation between MRC imageability scores and average annotators' ratings 0.870 (p 0.001) and the correlation between scores assigned via our propagation method and average annota- tors' ratings is 0.735 (p < 0.001). This high posi- tive correlation coefficient between assigned image- ability scores and human -ernen-eeeeeiee ratings demon- strates the effectiveness of our adopted propagation method. We also note that the inter-annotator agree-
We randomly selected 500 words from the MRC lexicon and 500 words from the word2vec vocabu- lary that did oceur in the MRC lexicon. Each word was shown to 9 annotators using Amazon Mechanical Turk to seek responses to the follow- ing question: "Do you agree that the word below evokes an image or picture in your mind?" The annotators were instructed to respond on 7-point Likert scale, where 1 denoted strong disagreement and 7 denoted strong agreement. Please Ap- pendix A.3 for details about the instructions, demo- graphic filters, and compensation.
We average the ratings for all the annotated

---

ments for the ratings for MRC words and word2vec words, as computed using Krippendorf's a (ordinal measure), were 0.626 and 0.584, respectively. Overall, this assessment illustrates the validity of propagating word-level imageability scores us- ing embedding-basee semantic similarities. More broadly, the aim of adopting this approach is to expand the coverage of MRC lexicon. Qualita- tively, we observe that words like 'gotcha' (0.33) and 'presbyterian' (0.61) are assigned meaning- ful imageability scores, demonstrating expansion along time and domains. As a point of difference between human ratings and assigned scores, we notice that the propagation approach assigned a high imageability score to words like 'qawwali" (0.60) while the human annotators did not, possi- bly due to lack of sociocultural context. In Table 6 we show illustrative words that are assigned high (2 0.7), medium (E (0.3, 0.7)), and low (< 0.3) imageability scores using our propagation method.
scores which lead broader coverage in terms of vocabulary. As seen in Table 7, we observe the same trends as with MRC imageability scores, al- beit with slightly lower values of correlation scores. To analyze the alignment between learned atten- tion scores for various models, we compute the correlation between average attention scores across different models. Pearson's 's correlation coefficients in Table 8 show that all the model attention scores have moderate correlation with each other. Why are correlation scores higher for MSCOCO than for TIMED?: An interesting trend across Ta- ble 3 and 7 is that the correlation scores are consis- tently higher, across all the models under consid- eration, for the MSCOCO dataset than the test set of TIMED, We note that, on average, MSCOCO has caption length of 11.4 whereas the TIMED dataset has an average sentence length of 20.6, with greater concentration of objects from the Visual Genome objects-6.7 6.7 (58.7%) objects per example versus 8.4 (40.7%) objects per example). For our TIP-CLIP model, these objects acquire an average of 63.2% attention scores across all the MSCOCO examples, whereas they only acquire 37.1% of attention scores, average, across the examples in the TIMED test Overall, these results demonstrate that the TIP-CLIP model at- tends over words the MSCOCO corpus in an object-targeted manner but the attention is rela- tively diffused in the TIMED corpus. Combined with the observation that MRC imageability scores are higher for concrete objects (Paivio et al., 1968), this explains why the correlation scores are consis- tently higher on MSCOCO than TIMED.
A.3 Details about MTurk Experiments
For all our annotation tasks, we recruited annota- tors using Amazon Mechanical Turk. We set the criteria to 'Master' annotators with at least a 99% approval rate and were located in the United States. To further ensure the quality of annotations, re- quired the annotators to have at least 5000 accepted annotations in the past. The rewards were set by assuming an hourly rate of 12 USD for all the anno- tators. We show the annotation interfaces in Figure 7. In addition, the annotators were informed that the aggregate statistics of their annotations would be used and shared as part of academic research.
We also inserted some "attention-checl exam- ples during the annotation tasks to ensure the an- notators read the text carefully before responding. This was done by asking the annotators to mark a randomly chosen score on the Likert scale re- gardless the actual content. We discard the an- notations from annotators who did not correctly respond to all the attention-check examples and re-collect annotations for the affected samples.
MODELS
MSCOCO
TIMED
BERT
0.434
0.301
CLIP
0.429
0.262
TIP-CLIP (Ours)
0.465
0.338
Table 7: Pearson's 's correlation coefficient between prop- agated imageability scores (using word2vec) and model attention-scores, denotes 0.001
Effect of length on the correlation between at- tention and MRC-I scores: We categorize the sen- tences in the test set of TIMED into short (S 10;
A.4 Further analyses on the correlation between atfention scores and word-level visualness scores
304), medium (E (10, 20); = 505), and long (2 20; n 606) sentences based on word counts. However, we did not find notable vari- ation in the correlation scores between the atten- tion weights of the TIP-CLIP model and MRC Imageability scores. Pearson's 's correlation coeffi-
We compute the Pearson's correlation coefficient between a model's average attention scores over words and the visualness score assigned using our propugation method. However, unlike Table 3, this time, we consider the propagated imageability

---

using our proposed approach is to conduct evalu- ations on out-of-domain (OOD) datasets. To this end, we curate a social media dataset by scraping Twitter. We start with the Wikipedia-basee Image Text Dataset (WIT) (Srinivasan et al., 2021) and query Twitter using the Wikipedia page title trieve posts in English that are with and without images. We require that the retrieved post con- tains the page title string to ensure topical simi- larity between posts with and without images. To remove examples with irrelevant images, we dis- (a) Interface to collect sentence- visualness scores
Instructions
Do you agree that the sactence below erckes image picture you elle?
Tead the sertonne and reseet whether in vouu Dies the seseense evsbe unagg the reater's mond?
Steee
You shoun You rouund thooghh the that repord ahother not agrae
Select an option
Strongly
word bermar consdrrd stval the pettines has foe to mediu Wituiress.
Disagreo
Somewnet deaggee
"ext iiuatiess
poltue roder's Consiler Sullwing
Neutral
Since 31 exckes the Iaag a flowe nader's rind the sepores thoud indicaio ageement the cuntion 7. Broogly agres
slovig
agre
Agren
Broe 52 doss mode image noder cinel the repuoee shuud indouee agreeeen quedior 1. Stuingly desggns
Strongly Agres
(b) Interface to evaluate word-lever visualness scores assigned by the propagation method
Instructions
Do you agroe tha the upor bulow drohas isape or pistane ln jour cone?
Read
whather aieuad
vinaa,, strongly
Spaad]
You will shown word. You rqqureed to read't and then respond whether agroe vieuai nosts
Sclvel cptlen
Worths viuathess catted extent to which meckes image pictue mind Considrr fallowiig exampiec: consent Since WT
should indrade agraement question Sinongly tresight Since dows image wader's resporse should agremeet queators Strengll disagres
Oregre
She
though
rocggrzz
Ser
seongy Agel
Figure 7: Interface for our annotation tasks on Amazon Mechanical Turk. For each of the annotations task, we also show the instructions provided to the annotators.
MODELS
BERT
CLIP
TIP-CLIP
BERT
CLIP
0.552
TIP-CLIP (Ours) 0.631
0.57
Table 8: Pearson's is correlation coefficient between word- level attention scores of various models for the TIMED test set. *** denotes p 0.001
MODELS
F
PRECISION
RECALL 1
Acc. f
Randem
0.503
0.503
0.503
0.505
MRC-T
0.470
0.472
0.472
0.470
VG-Objects
0.536
0,541
0.539
0.548
MRC-I + w2v
0,501
0,502
0.504
0,502
card posts with CLIP-similarity lower than 0.70 between the Twitter post's image and the corre-
MRC.I+ GloVe (Twitter)
0.516
0,518
0.520
0.519
BERT
0.612
0.634
0.624
0.618
sponding image on Wikipedia. Consequently, we obtain dataset of Twitter posts containing men- tions of 1185 Wikipedia topics, 7844 Twitter posts with images, and 7248 Twitter posts without im- ages. The posts with and without images are tied by common Wikipedia topics.
CLIP
0.644
0.645
0.645
0.644
TIP-CLIP (Ours)
0.693
0.694
Table 9: Out of domain evaluation the Twitter dataset. Reported F,. Precision, and Recall values are macro-averages across the two classes (visual and non-visual).
We hypothesize that the text in Twitter posts that mention a certain topic and contain an image is more visual than text in Twitter posts that men- tion the same topic and do not contain any images. To test this hypothesis, we randomly sample 40 Wikipedia topics and present the associated text with (n 264) and without images (n = 241) to human annotators. In an AMT survey that fol- lows the design for curating TIMED, we find that the average annotator rating for the text from Twit- ter posts without images is 2.306 1#1336 while that for text from Twitter posts with images is
cient was 0.33, 0.35, and 0.37 for short, medium, and long sentences, respectively. We observed the same trend for the fine-tuned BERT model and the pre-trained CLIP model.
A.5 Out-of-Domain Generalization
Robustness of vision-language models has been the subject of investigation in several prior works (Verma et al., 2022; Ramshetty et al., 2023; Li et al., 2021). A critical assessment of the ro- bustness and generalizability of the models trained

---

4.304 (#1.273). We observe the inter-annotator agreement of 0.413, which similar to that ob- served while curating TIMED. For out of the 10 Wikipedia topics, the annotators provided higher imageability rating to text originally associated with an image on Twitter than text not associated with an image. Overall, the AMT survey validates our hypothesis by demonstrating that text in Twitter posts with images is perceived more visual than text in Twitter posts without images, modulo the topic common across the posts.
200
CLIP
175
BERT scores
Number of exxmppee
150
TIP-CLPP scores
100
50
25
-3
-2 -1 standardized 'visual' score
2 3
Figure 8: Distribution of standardized visualness scores for ambiguous examples (i.e., (v /a/a, where e is the original visualness score, μ and a are the mean and standard deviation of the distributions, respectively). We contrast the predicted visualness scores by fine-tuned BERT, pre-trained CLIP, and our TIP-CLIB models.
We now ask the question: how well the models considered in our work categorize Twitter text with images as visual and Twitter text without images non-visual? We first adapt the thresholds used to classify text using various methods by rumning evaluation randomly sampled validation set of 100 Twitter examples, 50 from each category. The thresholds are set as follows: MRC-I: 0.19; VG-Objects: 0.52; MRC-I + w2v: 0.17: MRC-I + GloVe: 0.32%; CLIP: 0.87; TIP-CLIP: 0.74. Using these threshold values, we categorize the rest of the Twitter dataset (n = 14, 992) into visual and non-visual categories. The random baseline uses uniform sampling.
and our proposed TIP-CLIP model score them on imageability. We compute the imageability score using Equation 2 for CLIP and TIP-CLIP, while treating fine-tuned BERT's prediction probability score as its imagcability score for a given exam- ple. To appropriately compare the distribution of imageability scores across these three models, we standardize the values by computing 2-scores (i.e., 2; is transformed into z; = (x; p)/o; where x; is the original value, and are mean and stan- dard deviation of the distribution that 2; belongs to). In Figure 8, we show that while CLIP and TIP- CLIP imageability scores are distributed normally around their respective means, BERT imageability scores are bimodal with peaks close one stan- dard deviation away from their mean. This demon- strates that if the models were to be used for scoring text imageability, as opposed to categorizing text into vi vual and non-visual categories, CLIP and TIP-CLIP models will provide more reasonable middle-level scores for ambiguous text, whereas scores from BERT would either be higher or lower. We attribute this to how the underlying models are trained and how the consequent imageability scores are computed. While the BERT model trained solely for the classification task that emphasizes discriminative encoding and the predicted proba- bility score is used the imageability score, the distribution is bimodal. However, CLIP and TIP- CLIP are trained using image-text matching (the former, entirely; the latter, to some extent), and imageability scores are computed as the distance between the NULL image and input text.
Table 9 shows the results for this out-of-domain evaluation. First, we note that all models undergo a severe drop in performance on the OOD dataset, indicating that the notion of sentence-level image- ability is strongly tied to the domain. Our proposed TIP-CLI model demonstrates better OOD gener- alization capabilities than all the considered base- lines. It is noteworthy that the fine-tuned BERT model performs poorly on the OOD dataset than the standard pre-trained CLIP model. The aggre- gation of word-level imageability scores provides a worse-than-randee random estimate of sentence-level imageability on the OOD dataset.
A.6 Predictions on Ambiguous Sentences
Recall that while curating TIMED, we combined examples without clear majority from the anno- tators (n 378) and those with majority votes for the 'Neutral' category (n 2) into single category called ambiguous. We revisit these exam- plcs to analyze how the most competitive basclines
"Since we are operating with the Twitter domain, we de- sign a version of the propagation method where MRC Im- ageability scores are propagated in the Glo VloVeeembedding space, where the GloVe embeddings are learned on Twitter cor- pus (Pennington et al., 2014). We use 200-dimensional GloVe vectors trained on 2 billion Twitter posts with a vocabulary size of 1.2 million.


Reviews:
[{"review_id": "IqhUge3vMg", "reviewer": "Reviewer_4r5P", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their comments. We clarify the two aspects of the work the reviewer mentioned: the significance of the work and the use of the CLIP model as one of the baselines.\n\n**Significance of the work**: The reviewer mentions that for the work to be considered significant, it needs to demonstrate better quality of images generated by text-to-image models. We believe this to be orthogonal to our key contributions in the work. We state that identifying text visualness is a necessary _first step_ toward connecting long-form textual documents with relevant visual assets, which could be done using retrieval or generation as the subsequent steps (lines 43-50). This motivation is independent of the image-generation capabilities of the models like DALL-E. As an example, let us consider that there are 1000 sentences in a long-form document. If the document can be enriched using 5 or 10 images, it would help to know which portions of the document are visual and which are not. This process of identifying visual text at a sentence level is what our work contributes to, which is an orthogonal first step to leverage the capabilities of text-to-image generation (and retrieval) models in a _targeted_ manner. Therefore, we disagree with the reviewer’s assessment that the work needs to demonstrate the increased quality of images generated by text-to-image models. Using qualitative analysis, we demonstrate that separating visual and non-visual text as the first step could avoid generating absurd images for non-visual text (Figures 4 and 6).\n\n**CLIP as a baseline**: The reviewer mentions their concern about using pre-trained CLIP as one of the baselines. Given the training objective, the score $1 - <I_{null}, T>$ demonstrates how dissimilar text $T$ is from the $NULL$ image, which is being used as a measure of its visualness. As such, there is no requirement for this score to correlate with $<I_{actual}, T>$, where $I_{actual}$ represents the true image corresponding to visual text $T$. This is what the reviewer expresses their concern about. In fact, given the problem formulation, $I_{actual}$ is not known during inference time. During inference, the task is to categorize $T$ into visual or non-visual using the similarity with a static $NULL$ image. Once the text is identified as visual, we show in Section 6 (under ‘Effect on text-to-image retrieval’) that our proposed fine-tuning preserves the retrieval capabilities of the CLIP model. \n\nWe request the reviewer to reconsider their scores based on these clarifications.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "(a) The paper propose a new task of identifying the visualness of a sentence and curate a new dataset accordingly.\n\n(b) The experiment is comprehensive and the result analysis is detail.", "reasons_to_reject": "(a) The paper lacks clarity in demonstrating the significance of its work and fails to outline the potential benefits that recognizing visual information in text could bring to other applications. As for the image generation model mentioned in the introduction, it can generate reasonable images for visual text. However, even if the visualness of the text is recognized, it cannot change the quality of generated images. Therefore, there's a need for further clarification regarding the importance of this task.\n\n(b) The utilization of the pre-trained CLIP model as a baseline in Line 443 raises concerns. Representing the visual score of text through the 1-<I, T> measure seems unreasonable. While a high score of 1-<I, T> indicates low similarity with a NULL image, it doesn't necessarily correlate to high similarity between the text and an actual visual image.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper curate a dataset containing English sentences and their visualness score. It also proposes a fine-tuning strategy to identify whether a text is visualness and its corresponding image.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "ijWqanbT73", "reviewer": "Reviewer_B9nx", "summary": "", "questions": "Question A: Why did you chose BERT-base over BERT-large as your baseline?\nQuestion B: Latest cited work on visualness in the paper dates back to 2013. Wondering whether there aren't any other work since then?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their thoughtful comments and feedback. In this rebuttal, we present some results that demonstrate that BERT-base is a good representative baseline model for text-only classifiers for the task of visual text identification. Below, we also show the performance of the RoBERTa (base and large) encoders (Liu et al., 2019) for an exhaustive comparison with our proposed TIP-CLIP model.\n\n| Model                              | F1 Score | Precision | Recall | Accuracy |\n| ---------------------------------- | -------- | --------- | ------ | -------- |\n| BERT-base (auto + human-labeled)    | 0.774    | 0.783     | 0.797  | 0.771    |\n| BERT-large (auto + human-labeled)   | 0.786    | 0.791     | 0.799  | 0.782    |\n| RoBERTa-base (auto + human-labeled) | 0.794    | 0.798     | 0.801  | 0.786    |\n| RoBERTa-large (auto + human-labeled)| 0.801    | 0.799     | 0.811  | 0.793    |\n| TIP-CLIP (auto + human-labeled)     | 0.865    | 0.858     | 0.873  | 0.871    |\n  \n\nSince the performance on our task is similar with BERT-base and BERT-large models and since the overall trends remain the exact same with both the model variants as well as with the RoBERTa encoders (please refer to Tables 2 and 4 for complete results), we perform the subsequent experiments in the paper with a more efficient (in terms of training time and cost) model – i.e., BERT-base. As per the reviewer’s suggestion, we will include this information in the appendix.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper is very well written. The task of identifying the visualness of a text phrase is defined and a relevant solution is proposed along with a dataset. ", "reasons_to_reject": "A stronger baseline (such as BERT-large instead of BERT-base) could be selected.", "questions_to_authors": "Question A: Why did you chose BERT-base over BERT-large as your baseline?\nQuestion B: Latest cited work on visualness in the paper dates back to 2013. Wondering whether there aren't any other work since then?", "paper_topic_and_main_contributions": "In this paper, the authors propose the task of predicting the visualness of a given text. They also propose a fine tuning strategy that utilize large visual language models to solve the problem. Moreover, a dataset is curated and published for the visualness problem. The method is explained in detail, experiments and results are provided.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "I didn't identify any missing references.", "Typos_Grammar_Style_and_Presentation_Improvements": "Figure 4-A: TIS-CLIP -> TIP-CLIP "}, "extra_text": {}}, {"review_id": "HloAXxGlmA", "reviewer": "Reviewer_dbaf", "summary": "", "questions": "1. In the paper, DALL-E was used as a tool for quantitative analysis experiments, providing a small amount of generated content for both visual and non-visual text. However, it seems that there is no visual quantification of the generated images for the TIMED test set and out-of-domain data. It is important to ensure that the fine-tuning strategy for the two stages of CLIP not only optimizes the classification performance of visual and non-visual sentences, but also ensures the generalization performance of CLIP and the generation performance of text-to-image under an open vocabulary setting. Additional explanations may be needed.\n\n2. When doing filtering, CLIP sets Tpos to 0.35 and Tneg to 0.18. Based on previous experience, a filtering threshold of 15% is already high enough for CLIP. Why did you adopt the positive sample filtering strategy of 0.35 and the negative sample threshold strategy of 0.18? Is it because taking only the bottom and top parts makes this threshold more effective?\n\n3. Although the appendix mentions the possibility of conducting a differential analysis of text visual entailment at the word and sentence levels in the future, would there be any consideration for conducting a small experiment to support the importance of the superior sentence-level mentioned in the paper？", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their constructive comments. We also thank the reviewer for noting the novelty of the task and the usefulness of the curated dataset for future research. In the rebuttal, we answer the questions and provide additional clarification for the motivation of the work.\n\n**Motivation of the work**: The reviewer mentions that the analysis conducted using generative models such as DALL-E is insufficient to prove that identifying text visualness can effectively help downstream image retrieval and generation quality. We wish to clarify that the motivation of the work is to identify text visualness as the first step of the process that could connect long-form documents with visual assets. Once the visual text has been identified, existing text-to-image retrieval and generation methods can be triggered just for visual text. To this end, our contributions are orthogonal to the motivation of improving image retrieval and generation quality. Instead, we argue that retrieving and generating images only for visual text will enable a more efficient and targeted usage of these models, as non-visual text seldom needs visual assets to be retrieved or generated. However, our analysis demonstrates that once the CLIP model has been fine-tuned to identify visual text from non-visual text, it preserves the retrieval capabilities of the pre-trained CLIP model. Additionally, in Figures 4 and 6, we qualitatively show how DALL-E generates absurd images for non-visual text, demonstrating the reason why distinguishing visual and non-visual text is beneficial. \n\n\n**(A1) Question about the images in the TIMED test set**: It seems that the reviewer considers that we used DALL-E to generate images for visual and non-visual text in our data. We wish to clarify that this is not the case. To construct the large auto-labeled training corpus, the text and the images were taken from publicly available PDFs using CLIP-based similarity scores (heuristic: visual text is likely to have higher similarity with the image within the same PDF page, whereas non-visual text is likely to be dissimilar to the image). Recruited crowd workers manually annotated a subset of this large-scale auto-labeled data, which makes up the TIMED train and test set (please see Sections 3.1 and 3.2). None of the content was generated using DALL-E. To clarify further, in the analysis, Figures 4 and 6 aim to demonstrate how DALL-E generates absurd images for non-visual text and relevant images for visual text, thereby motivating why the distinction between visual and non-visual text is important.\n\n**(A2) Clarification question about the thresholds for filtering the auto-labeled data**: We thank the reviewer for this insightful question. As the reviewer notes, the reason why the thresholds T_pos and T_neg were set to 0.35 and 0.18, respectively, is to limit the noise in the auto-labeled data. We only consider top and bottom k % sentences (based on similarity scores) to obtain about 15,000 positive and 32,000 negative examples (lines 197-207).\n\n**(A3) Clarification about the importance of sentence-level visualness**: We believe both word-level and sentence-level visualness are important, depending on the application. For instance, if the subject of analysis is web search queries, modeling visualness at a word or phrase level would be more beneficial. However, for long-form documents, which is the focus of our work, modeling sentence-level visualness will be more direct. In Table 2, we show that aggregating word-level visualness is not an effective strategy to predict sentence-level visualness – please see MRC-I, VG-Objects, and MRC-I + w2v baselines. In the discussion of future work, we indicate that this could be due to compositionality (lines 658-662) and could be the focus of a future investigation. While existing works have studied word- or phrase-level visualness (please see ‘Visualness of words’ under Related Work), our work focuses on modeling sentence-level visualness.  \n\nRegarding **reproducibility**, we mention in lines 698-700 that we will release the code and human-annotated data upon acceptance. We will include the links to the data and code repositories in upcoming versions of the paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Strengths:\n1. Novel task: The paper introduces a new task of identifying the visualness of text, which is important for various applications like text-to-image retrieval and generation.\n2. Dataset creation: The authors curate a dataset with human annotations for visualness in English sentences, which can be useful for future research in this area.\n3. Fine-tuning approach: The paper proposes a fine-tuning strategy for large vision-language models like CLIP, enabling them to identify visual text while preserving downstream performance.\n4. Experimental validation: The authors provide quantitative and qualitative experiments to demonstrate the effectiveness of their approach compared to competitive baselines.", "reasons_to_reject": "The paper focuses on the contrastive learning paradigm between visually rich and visually impoverished text, which can better uncover which visual information in text is truly aligned with images. However, one weakness of this paper is that only a small amount of quantitative analysis was conducted using generative models such as DALL-E，which is not sufficient to prove that the text-visual semantic discrimination method can effectively help downstream task scenarios, such as image-text retrieval and text-based image generation quality. \n", "questions_to_authors": "1. In the paper, DALL-E was used as a tool for quantitative analysis experiments, providing a small amount of generated content for both visual and non-visual text. However, it seems that there is no visual quantification of the generated images for the TIMED test set and out-of-domain data. It is important to ensure that the fine-tuning strategy for the two stages of CLIP not only optimizes the classification performance of visual and non-visual sentences, but also ensures the generalization performance of CLIP and the generation performance of text-to-image under an open vocabulary setting. Additional explanations may be needed.\n\n2. When doing filtering, CLIP sets Tpos to 0.35 and Tneg to 0.18. Based on previous experience, a filtering threshold of 15% is already high enough for CLIP. Why did you adopt the positive sample filtering strategy of 0.35 and the negative sample threshold strategy of 0.18? Is it because taking only the bottom and top parts makes this threshold more effective?\n\n3. Although the appendix mentions the possibility of conducting a differential analysis of text visual entailment at the word and sentence levels in the future, would there be any consideration for conducting a small experiment to support the importance of the superior sentence-level mentioned in the paper？", "paper_topic_and_main_contributions": "The paper \"Learning the Visualness of Text Using Large Vision-Language Models\" addresses the problem of identifying the visualness of text, which is the ability of a text to evoke an image or picture in a person's mind. This is particularly important for tasks such as text-to-image retrieval and generation, where models like DALL-E and Stable Diffusion can benefit from inferring text visualness before generating images to embellish textual documents.\n\nThe main contributions of the paper are as follows:\n1. The authors propose the task of identifying the visualness of a sentence and curate a dataset by crowdsourcing annotations for English sentences\n2 They develop a training objective that fine-tunes large vision-language models, such as CLIP, for the task of text visualness identification\n3. Quantitative and qualitative experiments demonstrate the effectiveness of their fine-tuning approach in identifying visual text over several competitive baselines, while preserving downstream text-to-image retrieval performance.\n", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "KA9hx7yFAx", "reviewer": "Reviewer_3qDT", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their comments. In this rebuttal, we present counter-arguments for the reviewer’s stated reasons for rejection: the small size of the manually annotated corpus and the simplicity of the approach, and provide details about code and data sharing.\n\n**Relatively small-sized manually annotated corpus**: The manually annotated corpus (n = 3,620) is not small compared to other manually annotated corpora that are now part of standard benchmark datasets. For instance, the GLUE benchmark (Wang et al., ICLR 2019) includes STS-B and RTE datasets that have the same order of examples. The same applies to the AdvGLUE benchmark (Wang et al., NeurIPS 2021), with 5 tasks with fewer manually verified examples. Additionally, large vision-language and language models demonstrate lesser dependence on fine-tuning corpus size. We believe the manually annotated corpus is appropriately sized to enable effective training and evaluation for the proposed task. We empirically demonstrate this in Table 4, where the human-labeled data alone can be used to train a state-of-the-art TIP-CLIP model without using the auto-labeled data.\n\n**The simplicity of the approach**: The proposed fine-tuning approach is intentionally simple and yet effectively performs the task when compared to the baselines. We believe that effective and simpler approaches are preferred as they are less brittle and easy to deploy in real-world settings. \n\n**Code and data-sharing**: We mention in lines 698-700 that we will release the human-annotated data and code associated with our work to aid reproducibility and encourage future work on the topic. Upon acceptance, we will include the links to the data and code repositories.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. This paper introduces a novel task aimed at identifying the visualness of a sentence.\n2. To achieve this objective, this paper compiles a corpus consisting of 3,260 English sentences, each paired with human ratings for visualness. Additionally, it gather a noisy yet extensive corpus of 48,077 automatic alignments between text and visual assets found in long-form documents.", "reasons_to_reject": "1. The overall size of the corpus is relatively small, especially the high-quality portion that is manually annotated.\n2. The innovation is somewhat lacking, and the proposed fine-tuning strategy is relatively simple.\n3. No mention about publishing the code and dataset.", "questions_to_authors": "", "paper_topic_and_main_contributions": "The objective of this paper is to determine the visualness of sentences, for which it creates a dataset by gathering annotations through crowdsourcing for English sentences. To accomplish this, it devises a training objective that fine-tunes large vision-language models specifically for identifying the visual aspect of text. Through both quantitative and qualitative experiments, it shows the remarkable effectiveness of its fine-tuning method compared to various competitive baselines in recognizing visual text, all the while maintaining the performance of downstream text-to-image retrieval.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper looks at the task of determining if a give phrase or text evokes visual imagery or not. They motivate this by stating that understanding the visualness of text can be a precursor step to help generation models  and retrieval models (focus on just the visual text). They curate a dataset of 3620 sentences for this task, and fine tune large vision language models like CLIP to identify visual text. 

The reviewers were able to understand the contributions of the paper and 2 of 4 reviewers found it exciting, and all reviewers thought at least the major claims in the paper supported sufficiently (soundness=3). There were some clarifications regarding better ways of quantifying the visualness, and also corrections in the baseline formulations that the reviewers suggested. The authors have responded to those, and indicated they will correct their work based on the feedback. Overall the paper will improve and benefit from the feedback suggested by the reviewers.
Decision: Accept-Main