Learning the Visualness of Text Using Large Vision-Language Models
Authors: Gaurav Verma
Ryan A. Rossi
Christopher Tensmeyer
Jiuxiang Gu
Ani Nenkova
Venue: Submitted to ICLR 2023 (2023)
Keywords: text visualness
vision-language models
multimodal learning
natural language processing
deep learning

Abstract: Visual text evokes an image in a person's mind, while non-visual text fails to do so. A method to automatically detect visual text will unlock the ability to augment text with relevant images, as neural text-to-image generation and retrieval models operate on the implicit assumption that the input text is visual in nature. We curate a dataset of 3,620 English sentences and their visualness scores provided by multiple human annotators. Additionally, we use documents that contain text and visual assets to create a distantly supervised corpus of document text and associated images. We also propose a fine-tuning strategy that adapts large vision-language models like CLIP that assume a one-to-one correspondence between text and image to the task of scoring text visualness from text input alone. Our strategy involves modifying the model's contrastive learning objective to map text identified as non-visual to a common NULL image while matching visual text to their corresponding images in the document. We evaluate the proposed approach on its ability to (i) classify visual and non-visual text accurately, and (ii) attend over words that are identified as visual in psycholinguistic studies. Empirical evaluation indicates that our approach performs better than several heuristics and baseline models for the proposed task. Furthermore, to highlight the importance of modeling the visualness of text, we conduct qualitative analyses of text-to-image generation systems like DALL-E. We release the curated dataset and code.

Under review as a conference paper at ICLR 2023
LEARNING THE VISUALNESS OF TEXT USING LARGE VISION-LANGUAGE MODELS
Anonymous authors Paper under double-blind review
ABSTRACT
Visual text evokes an image person's mind, while non-visual text fails to do so. A method to automatically detect visual text will unlock the ability to augment text with relevant images, as neural text-to-image generation and retrieval models operate the implicit assumption that the input text is visual in nature. We curate a dataset of 3,620 English sentences and their visualness scores provided by multiple human annotators. Additionally, we use documents that contain text and visual assets to create a distantly supervised corpus of document text and associated images. We also propose fine-tuning strategy that adapts large vision- language models like CLIP that assume a one-to-one correspondence between text and image to the task of scoring text visualness from text input alone. Our strategy involves modifying the model's contrastive learning objective to map text identified as non-visual to common NULL image while matching visual text to their corresponding images in the document. We evaluate the proposed approach on its ability to (i) classify visual and non-visual text accurately, and (ii) attend over words that are identified as visual in psycholinguistic studies. Empirical evaluation indicates that our approach performs better than several heuristics and baseline models for the proposed task. Furthermore, to highlight the importance of modeling the visualness of text, we conduct qualitative analyses of text-to-image generation systems like DALL-E. We release the curated dataset and code.
INTRODUCTION
People typically communicate knowledge and information textually, but most prefer visually rich content. Text-to-image generattion/retrivall models could augment text with appropriate associated images, aiding the creation of appealing and easy-to-unde understtdd documents. Recent models like DALL-E (Ramesh et al. 202 la 2022) and Stable Diffusion (Rombach et al. 2022) work phenom- enally well for input text that is carefully constructed to elicit images. However, they cannot handle long text that may or may not evoke a visual image. We introduce the task of quantifying sentence visualness-a term we use interchangeably with imageability- as a necessary first step toward con- necting textual documents with visual assets. Consider the following two examples: "The flower- heads of Haemanthus coccineus ... with scarlet spathe valves on them like bright shaving brushes, make it striking plant" (V) and "A copyright notice is a notice of statutorily prescribed form that informs users of the underlying claim to copyright ownership in published work" (V). While V evokes an image in the reader's mind, V will be considered vonuvllual by most.
Vision-language models like ViLBERT (Lu et al., 2019), CL.P (Radford et al. 2021) and UNITER Chen et al., 2020) have achieved remarkable performance on tasks like Visual Ques- tion Answering (VQA) (Antol et al. 2013). cross-modal retrieval (Wang et al.) 2016), and Visual Commonsense Reasoning (VCR) (Zellers et al. 2019), but is not clear how well these models can distinguish visual text from non-visual text. Text-to-image generation models like Stable Diffusion, DALL-E, and Imagen (Saharia et al. 2022 would benefit from inferring text visualness before they can generate images to embellish textual documents. In Figure Ia we demonstrate the need with some examples: text identified to have low visualness leads to irrelevant generations from DALL-E, while text identified to have high visualness leads to the generation of relevant images.
Prior approaches quantifying the visualness of text operate on a word or phrase level (Deschacht & Moens 2007 Jeong et al. 2012) and leverage lexicons that contain auman-assigned world-level im-
Project wehpage: redacted for anonymization

---

Under review as a conference paper at ICLR 2023
Haza BEG
bed uyaelly idlnnd Shore lies our them blind
colleer Intemnat small makes to guaiartnne, repoosettiionn wanantio of any kand, aspressed
tead image
Our contributions
of trese successfi sustulllgggnnn anglers
When pardemn just ofter diem, fist aspear
'Visual hext"
red drun deaght.
...
way
Fnti monouwers of
forming the
text mpput
non-v tect
T.TT
h.T
h,TT
1,.
Vousless score =
human avserumed
CLIP model fine- -uned
idenittying
Tu
but
In Sakure answer Jadgmem be for the ralie denaa rood in compottt.
T9kt input
:
:
NUJ Irago
Image encoder
het
b-s
by
TT
Troe visuall boxt"
visuiicsa scare 1.87
hu
Tan
teat-tp-lmmag GALL-E)
Clocen 2014
NL. Irage
(a) Sentence visualness task
(b) Technical solution for sentence visualness prediction
Figure 1: (a): The visual text identification task, along with motivating downstream application. (b): Our approach to predicting sentence visualness, with a fine-tuning strategy where visual text is matched with its corresponding image while non-visual text matched with a fixed NULL image.
agcability scores (Louis & Nenkova 2013). However, such techniques are limited in their coverage and may not translate well to handle sentence-level visualness.
We curate a corpus of 3,260 sentences in English paired with their human ratings for visualness, as well as noisy-but-large corpus of 48,077 automatic alignments between text and visual assets in documents, including a NULL non-visual image. The textual part of the resulting alignment pairs can be used as examples of visual and non-visual sentences. We propose fine-tuning strategy for vision-language models like CLIP that allows classification inferences over text-only inputs. Our proposed objective also ensures that the learned emheddings remain usable for downstream tasks like text-to-imant to-igee retrieval. We compare the performance of our proposed approach against several heuristic and model-based baselines. Our extensive evaluation suggests that our fine-tuning strategy leads to the most accurate visual and non-visual text classifier. Finally, we conduct several analyses to glean insights into the model's learned attention mechanism, text-to-image retrieval abilities, and downstream text-to-image generation capabilities.
RELATED WORK
There are two research themes related to our work: (i) large vision- laiiun-languag models and their adap- tations to downstream multimodal tasks, and (ii) understanding and quantifying visualness of words.
Fine-tuning Vision-Language Models for Downstream Tasks: Vision-Language models aim to process and relate information across the visual and language modalities (Baltrušaitis et al.) 2018 Yuan et al. 2021; Radford et al.) [222]: Lu et al.] [2199] Tan & Bansal, (2019). Large models like CLIP (Radford et al.] 2021), UNITER (Chen et al.J (020), and ALIGN (Jia et al.] 2021 have demonstrated remarkable performance on downstream tasks via transfer learning or fine-tuning. However, such downstream tasks assume both text and image input to determine similarity or generate/retrieve the other modality for every instance of the corresponding modality; for instance, visual question answering (Antol al. 2015), caption generation (Xu al. 2013), and cross-modal retrieval (Wang et al. (016). Fine-tuning large rision-language models on such downstream tasks involves adding components to the encoders' architecture and training additional parameters on the task-specific dataset; the additional components could be fusion layers with cross-attention for mul- timodal classification (Mittal et al.) (022), or Transformer-basse
generation module for caption generation (Sarto et al. (022). Transferability and reusability of models and their learned represen- tations to downstream tasks and other domains are also a desirable properties (Yosinski et al. 2014 Long et al. 2015), especially in light of catastrophic forgetting (Goodfellow al. 2013).
Our work differs from existing work in that the input is only text, requiring us to adapt large vision- language models to not rely both modalities during inference. We propose fine-tuning strategy that does not involve additional architectural components (and parameters) on top pre-traines CLIP architecture and yet effectively adapts CLIP for learning text visualness. Our task can be
2

---

Under review as a conference paper at ICLR 2023
considered a precursor to tasks like text-to-image retrieval and generation, where images are only retrieved or generated for visual text. Further, we aim to preserve the reusability of text embeddings learned for the visualness categorization task for downstream tasks like text-to-image retrieval.
Visualness of Words: The visualness of text has been studied in multiple prior works but at a word phrase level. Coltheart (198T) curated the MRC Psycholinguistic Database comprising human ratings for word-level imageability. Since the lexicon only contains scores for 3769 words, the limited coverage of these visualness ratings has been a major limitation. Louis & Nenkova (203)) address this challenge by assuming that visual tags for images tend to co-occur with other visual terms. They use topic modeling over image tags and consider tags co-oceurring in the same topic as visual words in the MRC lexicon to be visual. Beyond word-level visualness, some studies have focused on phrase-level visualness. For instance, [eeon et al. (2012) quantify the visualness of a concept like 'round table' and 'red tomato' by measuring the "visual purity" and entropy of the clusters of images retrieved for that concept. In the same vein, Deschacht Moens] (2007) quantify the visualness of an entity mention on Wikipedia by computing its synsets similarity with a collection of 25 synsets that are manually labeled for their visualness (In WordNet (Miller 1995), synset is a collection of words that have close meaning and that represent underlying concept).
Our work focuses on learning sentence-level visualness instead of word or phrase-level visualness. While it is possible to aggregate word-level and phrase-level visualness scores to obtain sentence- level scores, it is unclear how accurate and generalizable these techniques are. We design multiple baselines that use word-level visualness scores to quantify sentence-level visualness and contrast the performance of such approaches with our proposed approach.
TEXT IMAGEABILITY DATASET (TIMED)
Our proposed fine-tuning approach follows multi-stage training of large vision-language model CLIP (Radford et al. 1021 In the first stage, we conduct large-scale self-supervised fine-tuning, followed by fine-tuning on a relatively smaller annotated corpus in the second stage. We first discuss the curation of large-scale corpus that comprises cammattially-sssgnnne and distant labels and then describe the curation of the human-labeled corpus of visual and non-visus sentences.
3.1 DATASET FOR FINE-TUNING WITH AUTOMATIC LABELS
As we will discuss in the following section, the formulation of the training objective requires positive examples comprising visual text and paired images as well as negative examples that comprise non- visual text. To create a corpus like this, we: (i) leverage image-tex co-occurrences in documents to develop a self-supervised approach, and (ii) use image-text similarity scores obtained using CLIP as priors to construct a large training corpus. We start with 450,000 publicly available PDFs referenced the Common Crawl corpus and identify pages within those PDFs that include images
2 We use a
document object detection tool like Fitz to extract paragraphs and images from the document pages.
We do sentence segmentation for the identified paragraphs using NLTK Tokenizer (Bird 2006). To map the images in the page to sentences, we compute CLIP similarity scores between each image- sentence pair in a given page. Based on the distribution of imaage-sentence similarity scores across all the pages in our corpus, we two thresholds, Tpos and Tneg- A sentence in page is considered positive example (visual text) if its similarity with any of the images in the page is greater than Tpos- Similarly, chosen negative examples have similarity values less than Taeg with all images within the same page. Sentences with an image similarity value greater than Tpos are associated with the most similar image in the same page, while the negative examples are associated with common NULL image. The thresholds T Tpos and T Tneg are chosen conservatively to only include top or bottom l % sentences from the entire corpus, respectively. This limits the noise in our training corpus for adapting the CLIP model for scoring text imageability. In our experiments, we set T pos to be 0.35 to consider top 1% sentences as visual and Tneg to be 0.18 to consider bottom 5% sentences as non-visual. Our automatiially-llaelee corpus comprises 15,359 visual sentences, the corresponding images, and 32,718 18 vis-vl sentences.
We choose to work with PDF documents rather than webpages because (i) PDFs have natural demarcations the form of pages (whereas webpages often contain long- onggrunning text with complex image-text interactions), and (ii) images within page are likely to be related to selected text fragments within the same page. https:/ //gi thub. com/ http://ymmdff//////o/e/
3

---

Under review as a conference paper at ICLR 2023
Table 1: Qualitative examples of visual and non-visual text from the human- annotated subset of the Text Imageability Dataset (based on the average of annotator ratings), and text with high ambiguity (based on the standard deviation of annotator ratings).
Categury
Example text
p/o
now the sew has mehed and the grass not only locks dreary. but it sogay:
8.85
The oper kcft sis-iinh sipper scar us his cland.
2 6.55
Visual
When the gunder ugen, jus after dawa, the find appee are the jusgen and the viluu figuro perfennng the intisate museuvers of ai chi
He removed the box placed it next to the prtege can, and put his partage inside the on.
/ 6.64
P 5.85
Hut, after nimning only the tine 500 meters he nodimed ther injry that wormed insigifiont woold not only present him from winning
P 5.00
race, but also fuiiaing
only way prove weon
#=
please call (999) 123 4367 eaail emull website.com. d:fusht the zelief demanded the complaine Callien intersa makes gareantess, experectttions warmatic
information schedlle
1.55
Non-riual
A 25%
member voice each didrael is rasdial to cominet dieeic
judgment
taken
electims in Octsher.
1.67
1.77
kind, cogrosee implisi, reponding the infermation inchud-
2.00
ing bet not linited to, warmantes of comeer. accuncy and relishiiiy.
JI Roman disnsoare his bosk Ohis State Forbbol: The Foeguaass Dawn which drws on estimene andivaal reacandh tn tell the untola viory of
a 2:31
the eatty days of football at Ohio as flaghii public university
Kemember to be sure to set your clocks back hour before you go to bed on Sanunday, November Ird.
(1 2.25
Ambiggouu
Thes the moai impontani thing my life soday:
Childces& parems
Univenett author Geoeze McClements
book Riãn' Dines Buck Bruzco,
0-
nught finaessal repert Ramey through enterniiing videon in-dephh you taktmil your sunry.
worbbork
3.2 HUMAN-ANNOTATE, DATASET
For the -anantan visual and non-visual examples, we start with another 200,000 PDFs dis- tinct from those used for the automated assignment of labels. To focus on natural images rather than infographies and academic figures, we filtered these documents to only include brochures, flyers, and magazines. For the resulting 35,432 documents, we adopted the same policy as that for curating the automatically- labeled dataset (selecting top 1% and bottom 5% sentences based on similarity values). We then recruited annotators to rate the visualness of the resulting 3,620 sentences after manually anonymizing any Personal Identifiable Information (PII) instances.
We recruited annotators on Amazon Mechanical Turk (AMT). We randomly ordered the 3,620 ex- amples and, for each example, we asked nine annotators to provide a response on a 7-point Likert scale for the following question: "Do you agree that the sentence below evokes image or picture in your mind?" A response of indicated strong disagreement, while 7 indicated strong agreement. We also inserted some attention-check examples (5%; n 181) to ensure the annotators read the text carefully before responding. These checks explicitly asked the annotators to mark a randomly- chosen score the Likert scale regardless of the actual content. We discarded the annotations from annotators who did not correctly respond to all the attention-check examples and re-collected more responses iteratively. Appendix provides details about the demographie filters for the recruited annotators, compensation, and the annotation interface.
If majority of annotations (i.e., at least 5 out of 9) were 1, or 3, we considered the example to be non-vis visull 1 (n = 2108). Similarly, vi sual 1 examples had majority of 5, 6, or 7 responses (n = 1132). We considered examples that did not have a clear majority or majority of responses of 4 (i.e.. 'Neutral' on the Likert scale) as amb iguuss and neut ral, respectively. Table [ shows illustrative examples of v suall 1, non-vi sual, and ambi ambiguous guous text from our dataset.
For 27.1% of the examples only at most 1 of the 9 annotators disagreed with the labels decided based on the process described above. Only 10.5% of the sentences were assigned neutral or ambiguous class. Inter-annotator agreement measured by Krippendorff's s a was 0.446. Krippendorf's ' a quan- tifies the degree of agreement beyond that by chance (i.e., observed disagreement over expected disagreement). Since the expected disagreement is strongly influenced by the ratio of values in the reliability matrix, the value is inherently small in our case as the annotator responses are skewed towards labels like 'Somewhat agree,' 'Disagree,' and 'Completely disagree' than the others. This inter-annotator agreement value is in a similar range to what is observed for other language-related tasks that involve assessment of text by experts on dimensions like coherence, likability, relevance, and even grammar (Karpinska et al. 2021). For brevity, we refer to the curated dataset as TIMED, short for Text Imageability Dataset.
4 TIP-CLIP FOR SCORING TEXT VISUALNESS
Background: The CLIP model (Radford al., 2021 jointly trains image and text encoders to pre- dict the correct pairing between images and textual descriptions. In batch size of N images and N texts (N2 possible image-text pairings), the objective function ensures that the cosine similarity between the embeddings of correct image-text pairs is maximized while the cosine similarity be- tween the (N22 N) incorrect image-text pairs is minimized. The encoders are trained over large multimodal dataset comprising about 400 million image-text pairs.
Updated training objective: When predicting text visualness, the goal is to assign a higher score to text that is visual (evokes concrete image for the person reading it) and lower score for non-visual
4

---

Under review as a conference paper at ICLR 2023
text (text that does not evoke an image). In line with the original training objective, we further train the CLIP model to match text that is identified as visual with the corresponding image. We adapt the CLIP training to match text that is identified non-visual with single NULL image (see Fig. Tb Matching visual text with the corresponding image while non-visual text to a NULL image not only encourages the model distinguish between visual and non-visual text, but also allows it to anchor non-visual text in the common NULL image that can be used during inference without having access to a potentially paired image. Formally, the adapted training objective is given as,
L =
1 2N
Σ log
( ( DK= expliyy, E T2)/T) ( 2N
Σ log
crpliTE TT)TT)
Σj= expl TE)TT)
Lmults U m
if m E V (i.e., non-visual)
such that, m
(1)
if E V (i.e., visual).
Here, N denotes the number of examples in batch, i and T denote the embeddings of the m-th pair of image and text that are normalized to have unit f--nrrm, respectively, such that m E (1....,N} N). (..)) represents the inner product, and r is the trainable temperature parameter. V and V are the set of examples in the current batch that belong non-visual and visual categories, respectively. Finally, Iaw denotes the embedding of the NULL image. During inference, we com- pute the cosine similarity between the representation of given text with the representation of the NULL image: non- non-visual texts will have a high similarity with the NULL image. Conversely, the visualness score S of any text with embedding T can be obtained using
S = 11- (IRELL: T
(2)
For the NULL image, create RGB image of size (224, 224,3) in which each pixel value is chosen randomly (see Figure lb). However, we experiment with different types of NULL images and find that the choice of null image does not affect the model's performance; see Appendix [A.]
An alternative formulation for adapting the CLIP training objective could have been to match vi- sual text with single image while matching non-visual text with single NULL image. However, this formulation of the training objective is similar to binary classification and does not enforce a contrastive objective for the positive examples. Matching visual text with its corresponding image instead of a common image for all visual text affords text embeddings that can be used for down- stream tasks like text-to- ti-agege- retrieval; we provide empirical evidence for worse text-to-image to-image retrieval performance with the alternative formulation in the Results section.
5 TRAINING DETAILS AND BASELINES
Train, test, & validation splits: Recall that our fine-tuning approach requires paired images for visual sentences only during training time and not during inference time; the model needs only text as input during inference. Of the 1132 visual sentences in the annottaed set of TIMED, we assign 515 examples that had an automatically determined corresponding image to the training set, and the remaining were randomly assigned to the test set (n = 517) and validation set (n = 100). The 2108 non-visual sentences were randomly split into the training (n = 980), test = 928), and validation set (200). All three sets maintain
negat ivve class ratio of ~ 0.5.
For the first stage of training, we fine-tune the CLIP model (ViT/B-32) on the proposed objective (see Eq. [ using the 48,077 examples with automatic labels. This training is done on Tesla T4 GPUs, for 5 epochs, with a batch size of 32, and a learning rate initialized at x 10-5 and optimized using Adam optimizer (Kingma & Bal 2014). Following this, for the second stage, we further fine- tune the same model for 2 epochs using the same objective and hyper-parameters, but this time using the train set of human-annotated TIMED The hyper-parameters are selected by performing grid search while observing performance on the validation set of TIMED. Based on the performance on the validation set of TIMED, set the threshold of S (Eq. 2 to be 0.79 to categorize text as vi risull 1 or non-vi sual. 1. We refer to the model trained using our fine-tuning strategy as TIP- CLIP, short for Text Imageability Predictor CLIP, and report performance the test set of TIMED.
"The CLIP model has a maximum context length of 77 tokens (about 50 words). Fewer than 1% of the training examples across both stages of the training are truncated fit this context length.
5

---

Under review as a conference paper at ICLR 2023
Table 2:
Evaluation human- annotated test set of TIMED. Reported Fi, Precision, and Recall values are macro-averages across the two classes (visual and non-vi sual).
MODELS
F
PRECISION f
RECALL
ACC. f
Random
0.531
0.531
0.531
0.577
MRC-I
0.584
0.599
0.583
0.644
VGiOOjjects
0.606
0.610
0.605
0.646
MRC-I + w2v
0.638
0.637
0.639
0.667
BERT
0.774
0.783
0.797
0.771
CLIP
0.694
0.695
0.701
0.712
TIP-CLIP (Ours)
0.865
0.858
0.873
0.871
5.1 BASELINES
We investigate the performance of TIP-CLIP against several heuristics and baseline models. Random:
The random baseline generates predictions via prior class probabilities in the training set. Average MRC-I score: We consider the imageability scores of 3,769 words the MRC lexicon and normalize them to be E [0, 1]. For each example, we take the average of the imageability scores of the unique words; out-of-vocabulary words are assigned a score of We lowercase the words in the MRC lexicon well as the input text. Based on this average score, categorize example as either visual or non-visual by setting the decision boundary as 0.17. The threshold chosen to optimize the performance on the validation set of TIMED.
Concentration of Visual Genome Objects (VG-Objects): The Visual Genome dataset comprises 75,729 objects, along with annotations for their attributes and object-object relations (Krishna al.) 2017]. Based on the heuristic that mention of a visual object in the text can trigger imageability, we quantify the concentration of Visual Genome objects by computing the fraction of unique object mentions in tokenized text with respect to the number of total unique words within the input text. We set the threshold to 0.5 based on the performance on the validation set.
Expanding the MRC lexicon using word embeddings: The coverage of the MRC lexicon poor because it contains only 3,769 words. We expand the list of word-level human-assigned imageabil- ity scores using semantic similarity between distributed representations of words [ For each word w in the word2vec (Mikolov et al. 2013) vocabulary of pre-trained representations that does not occur in the MRC lexicon, we compute its cosine similarities with all the words in the MRC lexicon to identify the most semantically similar word that exists in MRC, given by WMRC and its similarity with w given as (simmaax) We assign the word w an imageability score of simmar X score where scoreuxme is the normalized imageability score of w's most similar word WMRC. Based on the performance on the validation set, the decision boundary for average imagcability score of input text is set as 0.17. This baseline propagation approach is highly effective in quantifying word-level imageability as the Pearson's 'ss correlation coefficient between the assigned visualness score and the average AMT rating of humans is 0. 0.735 < 0.001);
Appendix A.2 for details.
Fine-tuned BERT classifier: We fine-tune a BERT model (bert www.bbbaseeuusseeeeeeee on Hugging- Face (Devlin al., 2018 Wolf et al., 2020 for the binary classification task of visual versus non-visual text detection. Similar to our proposed model, we adopt two-stage fine-tuning approach with the BERT classifier (adding a classification layer to BERT for the first input to- ken's ([CLS]) representation). We first fine-tune the model using the automatically labeled dataset followed by fine-tuning on the training set of the human-curated TIMED. For the first stage, we fine-tune the model for 7 epochs with a learning rate initialized at 5 x 10 5 using batch size of 32 while setting other stage with the same hyperparameters (chosen based on the performance on TIMED validation set).
to default. We fine-tune the model for 3 epochs for the second
Pre-trained CLIP model: We use the pre-trained CLIP model (VTT/B-32) to obtain similarity scores between the embeddings of the NULL image (used for the fine-tuning of our model) and the input text. We then use 1 - (1 T') as estimate of the visual score of text (see Eq. 2) Based on the performance on the TIMED validation set, we set the threshold for to be 0.83.
6 RESULTS AND ANALYSES
Evaluation on held-out test set of TIMED: We first evaluate the baselines and our approach on the test set of the numan-annotated TIMED, computing macro-averaged Fi, precision, recall scores, and classification accuracy. Table 2 show the results for this evaluation. We observe that our proposed
Wee experiment with 300-dimensional word2vec vectors trained on the Google News corpus, comprising 3 million words and phrases.

---

Under review as a conference paper at ICLR 2023
Table 3: Correlation between MRC Imageability scores and model attention-scores for BERT, CLIP.
and TIP-CLIP denotes the number of overlapping words across vocabularies; *** denotes p 10-3
MODELS
MSCOCO
TIMED
BERT
0.461*0* (n = 344)
0.326 = 294)
CLIP
0.448 (n 344)
0.283*** = 294)
TIP-CLIP (Ours)
0.497 (n = 344)
0.367 (n 294)
Table 4: Ablation studies to understand the benefits of two-stage fine-tuning. The presented results are on the human-annotated test set of TIMED. Reported values are macro-averages of class-wise Fi. precision, and recall, and overall classification accuracy.
MODELS
Fi f
PRECISION
RECALL 1
Acc.
BERT (auto-lubeled)
0.714
0.704
0.716
0.710
BERT (human-labeled)
0.753
0,766
0.789
0.756
BERT (auto human-labeled)
0.774
0.783
0.797
0.771
CLIP
0.694
0.695
0.701
0.712
TIP-CLIP (auto-labeled)
0.751
0.763
0.791
0.748
TIP-LIIP (human-labelld)
0.810
0.807
0.815
0.820
TIP-CIIP (auto numan--abbleed
0.865
0.858
0.873
0.871
two-stage fine-tuning strategy leads to the best-performing model (TIP-CLIP) In comparison, the pre-trained CLIP model demonstrates notably weaker performance on the task of distinguishing vi- sual text from non-visual text. Interestingly, fine-tuned BERT performs reasonably well on the task, considerably better than the CLIP model. Using the average imageability scores from MRC provides better-than thaeer--ha-rrndom random performance but is severely subpar to models like CLIP, BERT, and TIP-CLIP. Using word2vec embeddings to expand the coverage of the MRC lexicon (i.e., MRC-I w2v) leads to a boost in performance. However, collectively, the lacking performance of MRC-I and MRC-I w2v demonstrates that word-level imageability does not translate to sentence-level imageability to a great extent. Notably, in terms of baselines that aggregate word-level attributes, VG-Objects provides the best estimate of sentence-level imageability by quantifying the concentrations of visual objects in the input sentence.
Correlation of Attention Weights with MRC Imageability Scores: Attention mechanisms could be taken as proxies for explainability (Wiegreffe & Pinter, 2019 Chefer et al. 202I). Since the fine-tuned BERT, pre-trained CLIP, and our TIP-CLIP are attention based models, we compute the correlation between average word-level attention scores (obtained from the last layer) on given dataset with the imageability scores assigned by humans in the MRC lexicon. We compute these values for two datasets- the MSCOCO dataset (Vinyals et al. 2016) and the test set of TIMED. We only consider words that occur more than once in the specific corpus. Table shows that TIP- CLIP attention scores correlate the most with MRC imageability scores, followed by the fine-tuned BERT's s attention scores. The trends are consistent across both datasets. The relative ordering of models in terms of the correlation of their attention scores with MRC imageability scores follows the same order as their performance on the test set of TIMED. However, all correlation scores are in the low range, indicating a non-trivial relationship between sentence- and word-level imageability,
The same trends hold for propagated visualness scores, albeit with slightly lower values of the correlation scores (see Appendix A.4). We also analyze the reason behind higher correlation scores on MSCOCO with respect to the TIMED corpus in Appendix A.4
Effect of multi-stage training: We conduct ablations isolate the effect of two-stage training. In Table 4 we show that BERT and TIP-CLIP can learn to distinguish visual and non-vi -visual text even when fine-tuned only using the automatically labeled data. However, for both models, the gains from fine-tuning only on smaller, we find the proposed two-stage fine-tuning (i.e., training on automatically labeled data followed by human-labeled data) to be most effective, leading gain of over 2 and 5 absolute Fi points over training only on human-labeled data for BERT and TIP-CLIP models, respectively. Additionally, for given training strategy, our proposed fine-tuning of TIP-CLIP demonstrates better performance than the corresponding fine-tuned BERT model as well as the standard pre-trained CLIP model.
data are notably higher. Furthermore,
Effect on Text-to- Image Retrieval: We aim to analyze the re-usability of learned embeddings by the TIP- model for the text-to-image retrieval task. To this end, we consider the 515 visus visual examples from the test set of TIMED and. for each vi visual example, we rank the 515 corresponding images based on the cosine similarity between the image and text embeddings obtained from the TIP-CLIP model. We compute the Mean Reciprocal Rank (MRR) and contrast with the MRR
7

---

Under review as a conference paper at ICLR 2023
Narrecozal food
NULLimoge
Imaggee
NILL KAAO
mage 2
Imapes
visual lout
Non-daaoo
NULL Image
Visall
Visual leal
(a) CLJP emheddings
(b) TIP-CLIP embeddings
(c) Alt. formulation embeddings
Figure 2: t-SNE visualization of embeddings learned by (a) CLIP, (b) TIP-CLIP - using contrastive and adapted contrastive learning objective, respectively, & (c) model trained using alternative for- mulation solely focusing on classification. The plotted data points are from the TIMED test set.
obtained using the pre-trained CLIP embeddings. As expected, CLIP achieves a near-perfect MRR of 0.989. The proposed fine-tune fine-tuning objective does not severely impact the reusability of embeddings obtained from TIP-CLIP for retrieval, and results in an MRR of 0.937. This comparison evaluates the retrieval capabilities of TIP-CLIP against that of the CLIP model because the correspondence between visual text and images was established using similarities between CLIP embeddings O
The downside of an alternate training objective: Recall that our fine-tuning strategy involves matching visual text with its corresponding image and matching non-visie vvisual text with the NULL image. With only the classification of viv risual and non- fine-tuning strategy would have been to match all the visual examples with one common image while matching all the non-visual text with the common NULL image. The major downside of this approach is that while it leads to an effective classifier after two-stage fine-tuning, demonstrating a comparable Fj score of 0.842 as the TIP-CLIP model, it performs poorly on the text-to-image re- trieval task with an MRR of 0.014. Overall, while the alternate entirely dussificationn-asee training objective performs at par with the proposed TIP-CLIP model on the classification task, the resultant embeddings demonstrate poor reusability for downstream tasks like text-to- text-to-image retrieval.
text in mind, an alternate
Properties of the new embedding space: In Figure we visualize the embedding space of the leamed embeddings using t-SNE (Van der Maaten & Hinton 2008 Alongside v visual and non-visual sentences from the test set of TIMED, we also plot the embeddings of images cor- responding to the visual 1 sentences, and the embedding(s) of the NULL image(s). First off, we observe that the embeddings in Figure 2ajand and 2b from CL.IP and TIP-CLIP are different in that the TIP-CLIP embeddings demonstrate better distinguishability between visual and non-vi sual text. In Figure 2c we observe that the alternative formulation pushes the NULL embeddings to the pe- riphery of the image embeddings' cluster from a near-center location in Figures 2a and 26) The text embeddings demonstrate notable distinguishability in Figure [gguree2ctooe too. We believe that the alternative classification-only formulation causes distortion in the latent space that causes drastic modification of text-only embeddings, making them useless for downstream text-to-image retrieval, as demon- strated empirically earlier. However, our proposed objective in TIP-CLIP preserves reusability for downstream tasks by maintaining semantic relevance between learned image and text embeddings.
6.1 QUALITATIVE ANALYSIS
In this section conduct two qualitative analyses: (i) contrasting the attention mechanisms for CLIP and TIP-CLIP, and (ii) the role of distinguishing vi sual 1. and non-v v sual 1 text in down- stream text-to-image generation using systems like DALL-E RRamesh al. 2021b).
Attention Map Visualization: To contrast the mechanism by which CLIP and TIP-CLIP models match input text with their corresponding image, we visualize and contrast the attention maps for both models. We adopt the state-of-the-art approach to explain multimodal Transformers (Chefer al. 202T). In Figure 3 we show 4 illustrative vi sisull sentences from the test set of TIMED along with their corresponding images. Focusing text, we observe that TIP-CLIP has greater tendency to attend to visual aspects in the text; for instance, words like 'christmas, 'islands,' 'lakes,' 'anglers' are attended to a greater extent by TIP-CLIP than CLIP. In images, we observe small changes in attention maps across CLIP and TiP-CLIP: for instance, while the CLIP attention is focused on the Common Loon, TIP-CI CIP- also attends the 'lake.' It is worth noting that the proposed fine-tuning
'While establishing the correspondence between vi visual text and images, we enforce the constraint that the most similar image for text should exist on the same page of the PDF Therefore. it is possible that while ranking all the images in the test set, the CLIP similarity of text may be higher for different image, resulting in MRR slightly less than 1.0 (i.e., 0.989).
8

---

Under review as a conference paper at ICLR 2023
orginiinge CLP stariie
sroning our ci BREER
3722000 COuchh
wrrge atreuphor Tery disio TON
oui dag Handy /RBR mas irgndie
& I: of 5 EITT.
CLIS) falowing peggee
GLT ser
verren de Buai fiee sheppe
leageR auccouild
Figure 3: Comparing the attention maps over input text and images for CLIP and TIP-CLIP. For text, a darker shade of green demonstrates greater attention by the model. For images, red demonstrates the greatest attention in the heatmap. Image best viewed with zoom.
[yj DALL E generations for non-visual testt
(6) DALL E generations for sisual bect
Themtore onty way
TOA the arow hos rabved sreay, soage
unang
your feluce answet jalg the demanded campairt
DiclaR
Zeim The TV ucoteatur weaconfing
XCHPIPACT
pardens just after dawn the fist appner fguiss parforiigg the ofralich
Ondiare oram mikes reproeeeeiee
523 rator rencale thei Dhc Ricw weald priverg vhriingth sack
naturzo eny iemmatiio
CM
sccusicy and walaaiig.
Figure 4: Examples of DALL-E generations for non non-visus visual and visual text.
objective that TIP-CLIP follows closely related to the original contrastive objective for training CLIP - both encourage the matching of correct image-text pairs for visual sentences but TIP- CLIP additionally encourages matching of non-visual text to the NULL image. The qualitative analysis of visualization maps reinforces that the matching process for text and images undergoes small changes to accommodate for greater attention to visual aspects in the text.
Downstream Text-to-Image Generation: In Figure we show the generations obtained using DALL-E for text that is categorized as pon-viee sual 1. and visual in our dataset. We observe that for non-visus 1 text, the images produced by DALL-E show poor relevance to the text. However, for visual text the generated images demonstrate great relevance to the input text. Qualitatively, if the text contains declarative information, DALL-E generates text-heavy images (last two examples in TigureFa)) For vi susull text, we observe that visual concepts like 'melted snow on grass,' "Tai chi,' 'joggers in the garden,' and 'running in a race, are well represented in the generated images.
Triggering image-to-text generation models like DALL-E for the text that is identified as visual crucial to effectively use such systems in a passive setting. For instance, while working with long-form documents, the authors should only be recommended to add visual assets in relevant places (i.c.. for visual sentences). Triggering image generations for non-visual sentences could cause suboptimal user experiences by recommending irrelevant images. To this end, our contributions focus on distinguishing visual text from non-visual text as the necessary first step.
TIP-CLIP also demonstrates the best oft-of-domain (Twitter) generalizability compared to the base- lines considered here; see Appendix [A3] more details. We also analyze the predictions of com- petitive models the ambiguous sentences in TIMED in Appendix A.6
7 CONCLUSION AND FUTURE WORK
We propose the task of predicting the visualness of text and curate a human- annottaed dataset of sentence-level visualness scores. Additionally, we propose two-stage fine-tuning objective for the task that involves training on distantly supervised corpus followed by a smaller human-annotated corpus. Comparisons with several baselines demonstrate the effectiveness of our approach in dis- tinguishing visual and non-visual text. Furthermore, analyses of attention weights for our model indicate greater correlation with word-level imageability scores than other attention-based base- lines. The embeddings from our approach are transferable to downstream text-to-image retrieval. Qualitative analysis of attention weights over textual input reinforces that our model attends vi- sual words to greater extent. In closing, we show qualitative examples of how predicting text visualness can make text-to-image generation more targeted and effective.
In future, we aim to study alternate objectives for learning text visualness while ensuring transferable representations for more downstream tasks; our current experiments demonstrate the ineffectiveness of the binary classification formulation on this front as it shows poor text-to-image retrieval capabili- ties. As the aggregation of werd-level visualness scores leads to poor predictability of sentence- level visualness. future work could aim to understand the compositionality in language that precipitates visualness at the sentence level. Additionally, we will study in detail how text visualness impacts the quality and relevance of images generated using systems like DALL-E and Stable Diffusion.

---

Under review as a conference paper at ICLR 2023
8 ETHICS STATEMENT AND REPRODUCIBILITY
The authors do not foresee any negative social impacts of this work. However, our model can inherit the known biases underlying models like CLIP and BERT (Agarwal et al. 2021 Garimella al, 2021). The documents from which our datasets curated publicly available and are mentioned in The Common Crawl corpus (https: : /ommonoomaaa..... awll org/). We manually anonymize in- stances of Personal Identifiable Information in the sentences that are annotated using Amazon Me- chanical Turk. The recruited annotators are from the United States and are paid at an hourly rate of 12 USD. We intend release the human-annotated dataset to aid future research the topic and the source code for fine-tuning CLIP for the task of visual text identification.
REFERENCES
Sandhini Agarwal, Gretchen Krueger, Jack Clark, Alec Radford, Jong Wook Kim, and Miles Brundage. Evaluating clip: towards characterization of broader capabilities and downstream im- plications. arXiv preprint arXiv:2108.02818, 2021.
Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Zit- nick, and Devi Parikh. Vqa: Visual question answering. In Proceedings of the IEEE international conference on computer vision, pp. 2425 -223,, 2015.
Tadas Baltrušaitis, Chaitanya Ahuja, and Louis-Philippe Morency. Multimodal machine learning: A survey and taxonomy. IEEE transactions pattern analysis and machine intelligence, 41()): 423-443, 2018.
Steven Bird. Nltk: the natural language toolkit. In Proceedings the the COLING/ACL 2006 Interactive Presentation Sessions, pp. 69-72, 2006.
Hila Chefer, Shir Gur, and Lior Wolf. Generic attention-model explainability for interpreting bi- modal and encoder-decoder transformers. In Proceedings of the IEEE/CVF International Confer- on Computer Vision, pp. 397-4996 2021.
Yen-Chun Chen, Linjie Li, Licheng Yu, Ahmed El Kholy, Faisal Ahmed, Zhe Gan, Yu Cheng, and Jingjing Liu. Uniter: Universal image-text representation learning. In European Conference on Computer Vision, pp. 104-120. Springer, 2020.
Max Coltheart. The psycholinguistic database. The Quarterly Journal of Experimental Psy- chology Section A, 33(4):497-505, 1981.
Koen Deschacht and Marie Francine Moens. Text analysis for automatic image annotation. In Proceedings of the 45th annual meeting of the association of computational linguistics, pp. 1000- 1007, 2007.
Jacob Devlin, Ming-Wir Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. 2018. URL htlltttss///xxi..co
04805)
Aparna Garimella, Akhash Amarnath, Kiran Kumar, Akash Pramod Yalla, Anandhavelu, Niyati Chhaya, and Balaji Vasan Srinivasan. He is very intelligent, she is very beautiful? mitigating social biases in language modelling and generation. In Findings of the Association for Computa- tional Linguistics: ACL-IJCNLP 202/, pp. 4534 4545, 2021.
Ian Goodfellow, Mehdi Mirza, Da Xiao, Aaron Courville, and Yoshua Bengio. An empiri- cal investigation of catastrophic forgetting in gradient- based neural networks. arXiv preprint 132iv:1312.6211, 2013.
Woo Jeong, Xin-Jing Wang, and Ho Lee. Towards measuring the visualness of cept. In Proceedings of the 21st ACM international conference on Information and knowledge management, pp. 2415-2418, 2012.
10

---

Under review as a conference paper at ICLR 2023
Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun- Sung, Zhen Li, and Tom Duerig. Scaling up visual and usion-language representation learning with noisy text supervision. In International Conference on Machine Learning, pp. 4904-4916. PMLR, 2021.
Marzena Karpinska, Nader Akoury, and Mohit lyyer. The perils of using mechanical turk to evaluate open-ended text generation. Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 1265-1285, 2021.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint 1412.6900, 2014.
Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A Shamma, et al. Visual genome: Connecting lan- guage and vision using crowdsourced dense image annotations. International journal of computer vision, 123(1))32-73, 2017.
Mingsheng Long, Yue Cao, Jianmin Wang, and Michael Jordan. Learning transferable features with deep adaptation networks. In International conference on machine learning, pp. 97-105, PMLR, 2015.
Annie Louis and Ani Nenkova. What makes writing great? first experiments on article quality prediction in the science journalism domain. Transactions of the Association for Computational Linguistics, 1:341-352, 2013.
Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee. Vilbert: Pretraining task-agnostic visiolin- guistic representations for vision-and-language tasks. Advances in neural information processing systems, 32, 2019.
Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg Corrado, and Jeff Dean. Distributed representa- tions of words and phrases and their compositionality. Advances in neural information processing systems, 26, 2013.
George A Miller. Wordnet: A lexical database for english. COMMUNICATIONS OF THE ACM, 38 (11):39, 1995.
Anshul Mittal, Kunal Dahiya, Shreya Malani, Janani Ramaswamy, Seba Kuruvilla, Jitendra Ajmera, Keng-hao Chang, Sumeet Agarwal, Purushottam Kar, and Manik Varma. Multi-modal extreme classification. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.
2022.
Allan Paivio, John Yuille, and Stephen A Madigan. Concreteness, imagery, and meaningfulness values for 925 nouns. Journal of experimental psychology, 76(12)): 1968.
Jeffrey Pennington, Richard Socher, and Christopher D Manning. Glove: Global vectors for word representation. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pp. 1532 1543, 2014.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, pp. 8748-88777 8763. PMLR, 2021.
Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alee Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. In International Conference on Machine Learming, pp. 8821-8831 PMLR, 2021a.
Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. In International Conference on Machine Learning, pp. 8821-882 PMLR, 2021b.
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text- conditional image generation with clip latents. arXiv preprint arXiv:2204.06 06125, 2022.
11

---

Under review as a conference paper at ICLR 2023
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. Proceedings of the IEEE/CVF Con- ference Computer Vision and Pattern Recognition, pp. 10684-10695, 2022.
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Seyed Kam- yar Seyed Ghasemipour, Burcu Karagol Ayan, S Sara Mahdavi, Rapha Gontijo Lopes, et al. Photorealistic text-to-image diffusion models with deep language understanding. arXiv preprint arXiv:2205.11487, 2022.
Sara Sarto, Marcella Corna,, Lorenzo Baraldi, and Rita Cucchiara, Setrivval-augmentee transformer for image captioning. arXiv preprint arXiv:2207.13162, 2022.
Krishna Srinivasan, Karthik Raman, Jiecao Chen, Michael Bendersky, and Mare Najork. Wit: Wikipedia-based image text dataset for multimodal multilingual machine learning. arXiv preprint arXiv:2103.01913, 2021.
Hao Tan and Mohit Bansal. Lxmert: Learning cross-modality encoder representations from trans- formers. In Proceedings of the 2019 Conference Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP- IJCNLP), pp. 3100-5111, 2019.
Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research, 9(11), 2008
Oriol Vinyals, Alexander Toshev, Samy Bengio, and Dumitru Erhan. Show and tell: Lessons learned from the 2015 mscoco image captioning challenge. IEEE transactions on pattern analysis and machine intelligence, 3944)655--663, 2016.
Kaiye Wang, Qiyue Yin, Wei Wang, Shu Wu, and Liang Wang. A comprehensive survey cross- modal retrieval. arXiv preprint 1607v/6607.660 2016.
Sarah Wiegreffe and Yuval Pinter. Attention is not not explanation. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNL),, pp. 11-20, Hong Kong, China, November 2019. Association for Computational Linguistics. doi: 10. 10L.1//5////D1--I100 1002.
URL
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, et al. Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 conference on empirical methods in natural language processing: system demonstrations, pp. 38-45, 2020.
Kelvin Xu, Jimmy Ba. Ryan Kiros, Kyunghyun Cho, Aaron Courville, Ruslan Salakhudinov, Rich Zemel, and Yoshua Bengio. Show, attend and tell: Neural image caption generation with visual attention. In International conference machine learning, pp. 2048-2057. PMLR, 2015.
Jason Yosinski, Jeff Clune, Yoshua Bengio, and Hod Lipson. How transferable are features in deep neural networks? Advances neural information processing systems, 27, 2014.
Xin Yuan, Zhe Lin, Jason Kuen, Jianming Zhang, Yilin Wang, Michael Maire, Ajinkya Kale, and Baldo Faieta. Multimodal contrastive training for visual representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6995-7004, 2021.
Rowan Zellers, Yonatan Bisk, Ali Farhadi, and Yejin Choi. From recognition to cognition: Visual commonsense reasoning. Proceedings of the IEEE/CVF conference on computer vision and pultern recognition, pp. 6720-6731, 2019.
12

---

Under review as a conference paper at ICLR 2023
A APPENDIX
A.1 EFFECT OF NULL IMAGE
Since all the non-visual sentences in the training corpus are mapped to a common NULL image, we aim to see the effect of the chosen NULL image on the results. Recall that the NULL image used for our main experiments was obtained by creating an RGB image in which each pixel value chosen randomly. We perform the same process with a different random seed to generate another NULL image. Additionally, we use natural image as another alternative for the NULL image. These images shown in Figure [ We then evaluate the resulting models on the human-annokated test set TIMED. Table shows that the performance of the models is not dependent on the choice of the NULL image. We also find dependence between the choice of the NULL image and the performance on downstream text-to-image retrieval.
A.2 ASSESSMENT OF WORD-LEVEL IMAGEABILITY SCORE PROPAGATION
We randomly selected 500 words from the MRC lexicon and 500 words from the word2vec vocab- ulary that did not occur in the MRC lexicon. Each word was shown annotators using Amazon Mechanical Turk to seek responses to the following question: "Do you agree thal the word below evokes an image or picture your mind?" The annotators were instructed respond on 7-point Likert scale, where 1 denoted strong disagreement and 7 denoted strong agreement. Please see Appendix A.3 for details about the instructions, demographic filters, and compensation.
We average the ratings for all the annotated words and normalized them to be E [0, 1]. We compute the Pearson's correlation coefficient between (a) the average ratings for MRC words and the normal- ized imageability scores, and (b) the average ratings for word2vec words and the imageability scores propagation. The correlation between MRC imageability scores and average annotators' ratings is 0.870 (p < 0.001) and the correlation between scores assigned via our propagation method and average annotators' ratings 0.735 (p < 0.001). This high positive correlation coefficient between assigned imageability scores and temman-perceived ratings demon- strates the effectiveness our adopted propagation method. We also note that the inter-annotator agreements for the ratings for MRC words and word2vee words, computed using Krippendorf's a (ordinal measure), were 0.626 and 0.584, respectively.
assigned via embedding-
Overall, this assessment illustrates the validity of propagating word-level imageability using embedding-basee based semantic similarities. More broadly, the aim of adopting this approach is to ex-
(a) Original NULL image
(b) NULL image with diff. seed
(c) Natural NULL image
Figure 5: Various NULL images used to study the effect of the chosen image on the text visualness identification task and the downstream text-to-image retrieval task.
Table 5: Effect of the choice of the NULL image categorizing the human-annotated test set of TIMED and downstream text-to-image retrieval. Reported Fi, Precision, and Recall values are macro-averages across the two classes (visual and non (aan-visuall
VARIANTS
F t
PRECISION f
RECALL
Acc. t
MRR 1
TIP-CLIP (Original Fig.
5ah
0.865
0.858
0.873
0.871
0.937
TIP-CLIP (w/ diff. seed Fig.
5b
0.867
0.854
0.875
0.872
0.934
TIP-CLIP (natural image Fig.
0.861
0.855
0.876
0.872
0.939
13

---

Under review as a conference paper at ICLR 2023
Table 6: Qualitative examples of words that are assigned scores in the high (2 0.7), medium (E (0.3, 0.7)), and low (s 0.3) range using the word2vec embedding-based propagation methodology.
Category
Example words (assigned score)
martini, crabmeat, teeth, oysters, mosquitos, bracelets. motorboat, diamonds, squirrels, cigarettes, beaches, trumpets, dolphin, caramel, cattle, portobello, libraries, chimpanzee, snorkeling, sailboat, harmonica
High imageability
reassure, militancy, inhumanly, catalyses, industrial, peacefulness, handwo- ven, neurosurgery, overwashed, whooper, snails, preeminence, recluse, en- trepreneur, character, insufficient, paladin, impersonal, deviously, recover
Medium imageability
politologist, psycholinguistic, requirements, confirmatory, terseness, prefor- mulation, offender, controversial, unhealable, monoculturalism, miserable, re- programmability, this, participate, attractive, determinant, disestablishment
Low imageability
(a) Interface to collect sentence-lever visualness scores
Instructions
Hut austence belon
the deeenn and whoch ee the
recipe the abcegle maat the santriie HBLL meant you carapes
Select an option
aelu was redacin bodee sssicil of kakay Chinsaak poout Bhe Stowing AC
dedernnnii of
Sinnny shagree
Cornowhri coagre
le parsa Gurader
Naurrl
light
Commocher
uchedee sase caggrger tv dudiiaaaatn
Agrse
Seongy
sole viberdie of be suney vieter you poing chertor cumppee net wooghte twnn aroourtte
(b) Interface to evaluate word-level visualness scores assigned by the propagation method
Instructions
Stort
Your reporat ba mociddddi
"hu willie shoee en You are reg rouured nas aua repport neuemer not sgran with soea
Belout
Eenrgy
1811 conee Shor 10 encees mmape of concet repprres aee intiude severn wr with the aerin To Oeongly nerees
Redicia
Note: thane auta lec exampine anteateed thi egs suy to bid whither Jae sraing stantte to naanciess rart you meaagti stane athin you noriautte Parts
Mergo Ape
Figure 6: Interface for our annotation tasks on Amazon Mechanical Turk. For each of the annota- tions task, we also show the instructions provided to the annotators.
pand the coverage of MRC lexicon. Qualitatively, we observe that words like 'gotcha' (0.33) and
'presbyterian' (0.61) are assigned meaningful imageability scores, demonstrating expansion along
time and domains. As a point of difference between human ratings and assigned scores, we notice
that the propagation approach assigned a high imageability score to words like 'qawwali' (0.60) while the human annotators did not, possibly due to lacking sociocultural context. Table 6 we show illustrative words that are assigned high (2 0.7), medium (E 0073,, and low (S 0.3) imageability scores using our propagation method.
A.3 DETAILS ABOUT MTURK EXPERIMENTS
For all our annotation tasks, we recruited annotators using Amazon Mechanical Turk. We set the criteria to 'Master' annotators with at least a 99% approval rate and were located in the United States. To further ensure the quality of annotations, we required the annotators to have at least 5000 accepted annotations in the past. The rewards were set by assuming an hourly rate of 12 USD for all the annotators. We show the instructions and the annotation interfaces in Figure 6
For our human evaluations, we also inserted some citeention-checke examples during the annotation tasks to ensure the annotators read the text carefully before responding. This was done by asking the annotators to mark a randomly- chosee score on the Likert scale regardless of the actual content. We discard the annotations from annotators who did not correctly respond to all the attention-check examples and re-collect annotations for the affected samples.
A.4 FURTHER ANALYSES ON THE CORRELATION BETWEEN ATTENTION SCORES AND WORD-LEVEL VISUALNESS SCORES
We compute the Pearson's 's correlation coefficient between a model's 's average attention scores over words and the visualness score assigned using our propagation method. However, unlike Table
14

---

Under review as a conference paper at ICLR 2023
Table Pearson's 's correlation coefficient between propagated imageability scores (using word2vec) and model coanttio-scoorss *** denotes p < 0.001
MODELS
MSCOCO
TIMED
BERT
0.434***
0,301
CLIP
0.429
0.262
TIP-CLIP (Ours)
0.465.**
0.338
Table 8: Pearson's correlation coefficient between word-level attention scores of various models for the TIMED test set. *** denotes < 0.001
MODELS
BERT
CLIP
TIP-CLIP
BERT
CLIP
0,552
TIP-CLIP (Ours)
0.63 nee 0.57|
this time, we consider the propagated imageability scores which lead to broader coverage in terms of vocabulary. As seen in Table 7 we observe the same trends as with MRC imageability scores, albeit with slightly lower values of correlation scores.
To analyze the alignment between learned attention scores for various models, we compute the cor- relation between average attention scores across different models. Pearson's 's correlation coefficients Table show that all the model attention scores have a moderate correlation with each other.
Why are correlation scores higher for MSCOCO than for TIMED?:
An interesting trend across Table and 7 that the correlation scores are consistently higher, across all the models under consideration, for the MSCOCO dataset than the test set of TIMED. We note that, on average, MSCOCO has caption length of 1.44 whereas the TIMED dataset has an average sentence length of 20.6, with greater concentration of objects from the Visual Genome objects- 6.7 (58. 7%) objects per example versus 8.4 (40.7%) objects per example). For our TIP-CLIP model. these objects acquire average of 63.2% attention scores across all the MSCOCO examples, whereas they only acquire 37.1% 1% attention scores, on average, across the examples in the TIMED test set. Overall, these results demonstrate that the TIP-CLIP model attends object-targeted manner but the attention is relatively diffused in the TIMED corpus, Combined with the observation that MRC imageability scores are higher for concrete objects (Paivio et al. 1968), this explains why the correlation scores are consistently higher on MSCOCO than on TIMED.
words in the MSCOCO corpus in
Effect of length on the correlation between attention and MRC-I scores: We categorize the sentences in the test set of TIMED into short (< 10; n = 304), medium (E (10, 20); = 505), and long (z 20; n 606) sentences based on word counts. However, we did not find a notable variation in the correlation scores between the attention weights of the TIP-CLIP model and MRC Imageability scores. Pearson's correlation coefficient was 0.33, 0.35, and 0.37 for short, medium, and long sentences, respectively. We observed the same trend for the fine-tuned BERT model and pre-trained CLIP model.
Table Out of domain evaluation on the Twitter dataset. Reported Fi, Precision, and Recall values macro-averages across the two classes (visual and bon-visual).
MODELS
PRECISION f
RECALL
Acc. 1
Random
0.503
0.503
0.503
0.505
MRC-1 VG-Objects
0.470
0.472
0.472
0.470
0.536
0.541
0.539
0,548
MRC-I+ + w2v
0.501
0.502
0,504
0.502
MRC-1 + GloVe (Twitter)
0.516
0.520
0,519
BERT
0.612
0.634
0.624
0.618
CLIP
0.644
0.645
0.645
0.644
TIP-CLIP (Ours)
0.696
0.693
0.691
0.694
15

---

Under review as a conference paper at ICLR 2023
A.5
-DMMAIN GENERALIZATION
critical assessment of the robustness and generalizability of the models trained using our proposed approach is to conduct evaluations on out-of-domain (OOD) datasets. To this end, we curate a social media dataset by scraping Twitter. We start with the Wikipedia-based Image Text Dataset (WTT) (Srinivasan et al., 202T) and query Twitter using the Wikipedia page title to retrieve posts in English that are with and without images. We require that the retrieved post contains the page title string to ensure topical similarity between posts with and without images. To remove examples with irrelevant images, we discard posts with CLIP--imilarrtt lower than 0.70 between the Twitter post's image and the corresponding image on Wikipedia. Consequently, we obtain a dataset of Twitter posts containing mentions of 1185 Wikipedia topics, 7844 Twitter posts with images, and 7248 Twitter posts without images. The posts with and without images are tied by common Wikipedia topics.
We hypothesize that the text in Twitter posts that mention a certain topic and contain an image are more visual than text in Twitter posts that mention the same topic and do not contain any images. To test this hypothesis, we randomly sample 40 Wikipedia topics and present the associated text with (n = 264) and without images (n = 241) to human annotators. In AMT survey that follows the design for curating TIMED, we find that the average annotator rating for the text from Twitter posts without images is 2.306 (=13369) while that for text from Twitter posts with images is 4.304 (+1.273). We observe the inter-annotator agreement of 0.413, which is similar to that observed while curating TIMED. For 34 out of the 40 Wikipedia topics, the annotators provided higher imageability rating to text originally associated with an image on Twitter than text not associated with an image. Overall, the AMT survey validates our hypothesis by demonstrating that text in Twitter posts with images is perceived as more visual than text in Twitter posts without images, modulo the topic is common across the posts.
We now ask the question: how well the models considered in our work categorize Twitter text with images as visual and Twitter text without images as non-visual? We first adapt the thresholds used to classify text using various methods by running
evaluation on a randomly sampled validation set of 100 Twitter examples, 50 from each category. The thresholds are set as follows: MRC-I: 0.19; VG-Objects: 0.52; MRC-I + w2v: 0.17: MRC-I GloVe: 0.32] CLIP: 0.87: TIP-CLIP: 0.74. Using these threshold values, we categorize the rest of the Twitter dataset (n 14, 992) into visual and non-visual categories. The random baseline uses uniform sampling.
Table 9 shows the results for this out-of-domain evaluation. First, we note that all models undergo severe drop in performance on the OOD dataset, indicating that the notion of sentence-level im- ageability is strongly tied to the domain. Our proposed TIP-CLIP model demonstrates better OOD generalization capabilities than all the considered baselines. It is noteworthy that the fine-tuned BERT model performs poorly on the OOD dataset than the standard pre-trained CLIP model. The aggregation of word-level imageability scores provides worse-than-random estimate of sentence- level imageability on the OOD dataset.
A.6 PREDICTIONS ON AMBIGUOUS SENTENCES
Recall that while curating TIMED, we combined examples without a clear majority from the an- notators (n 378) and those with majority votes for the 'Neutral' category (n = 2) into a single category called ambi We revisit these examples to analyze how the most-competitive base- lines and our proposed TIP-CLIP model score them on imageability. We compute the imageability score using Equation 2 for CL.IP and TIP-CLIP, while treating fine-tuned BERT's prediction proba- bility score as its imageability score for a given example. To appropriately compare the distribution of imageability scores across these three models, we standardize the values by computing s-scores (i.e.. Dy is transformed into 2, = (z; w)/o; where x; is the original value, j and a are mean and standard deviation of the distribution that 2; belongs to). In Figure 7 show that while CLIP and TIP-CLIP imageability scores are distributed normally around their respective means, BERT imageability scores are bimodal with peaks close one standard deviation away from their mean. This demonstrates that if the models were to be used for scoring text imageability, as opposed to categorizing text into vi visual. 1 and non- vi sual categories, CLIP and TIP-CLIP models will pro-
"Snne we are operating with the Twitter domain, we design a version of the propagation method where MRC Imageability scores are propagated in the GloVe-embedding space. where the Gl embeddings are learned on Twitter corpus (Pennington et al. 2014). We use 200-dimensional GloVe vectors trained on 2 billion Twitter posts with a vocabulary size of 1.2 million.
16

---

Under review as a conference paper at ICLR 2023
vide more reasonable middle-level scores for ambiguous text, whereas scores from BERT would either be higher or lower. We attribute this to how the underlying models are trained and how the consequent imageability scores computed. While the BERT model is trained solely for the clas- sification task that emphasizes discriminative encoding and the predicted probability score is used imageability score, the distribution is bimodal. However, CLIP and TIP-CLIP trained using image-text matching (the former, entirely; the latter, to some extent), and imageability scores are computed the distance between the NULL image and input text.
200
CLIP scores
175
Number exammpls
150
TIP-CLIP scores
75
50
25
-3
-2
standardized 'visual' score
Figure 7: Distribution of standardized visualness scores for ambiguous examples (i.e., (v- p\/0, where v is the original visualness score, μ and the mean and standard deviation of the distri- butions, respectively). We contrast the predicted visualness scores by fine-tuned BERT, pre-trained CLIP, and our TIP-CLIP models.
17


Reviews:
[{"review_id": "Meg5HMwX-Z", "reviewer": "Reviewer_gb6o", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper is very interesting. It studies the visualness of text based on a simple example. Some words can incur our imagination for the corresponding image, while the others might not. With this motivation, the author proposed a new dataset called TimeD with both manually labeled data and the automatically model labeled data. The author also proposed a fine-tuning strategy that tunes the CLIP model to predict the visualness given a text. The whole paper is very fun to read.", "summary_of_the_review": "I think the paper is interesting and the idea is novel. However I have concerns with the proposed fine-tuning stage approach. I am happy to raise my rating if my concern is resolved during rebuttal. For this stage, I will give the rating of 5.", "strength_and_weaknesses": "The idea is very interesting and novel to me. I like the motivation example and the dataset collection part. Especially using the CLIP to automatically labels the relatedness between image and text and the human annotated data to further enhance the labeling quality is pretty clever. \n\nBut I have concerns with the fine-tuning stage.\n1. The negative text is aligned with a NULL image. Given a batch of positive and negative text, and a set of positive images and NULL images. I wonder are the NULL images identical to each other?\n2. If no, during different training iterations, suppose we sample a text which has been sampled before, would the NULL images looks different comparing to the previous iteration?\n3. If all the negative images looks the same in the training batch, would the contrastive learning loss work? Basically the negative text is asked to aligned with a NULL image and not aligned with the same NULL image in this batch.\n4. If all the negative images are different in the training batch, I doubt the contrastive learning would work. Basically, the logic follows. The negative text is asked to be close to one of the NULL image, but be far away from the other NULL images.\n5. As the TIP-CLIP is always fine-tuned on the pre-trained CLIP model. I wonder how much would the pre-trained CLIP contribute? Did the author try to train from scratch on the proposed dataset? Due to lack of the data, this might be done with a smaller model.\n6. For implementation detail, the model is fine-tuned on a batch size of 32 with one GPU. As the CLIP would achieve quite low performance on smaller batch size. I wonder if this is a typo or this is designed on purpose?\n\nFor Fig 1.a, the figure is rendered incorrectly on my PDF reader (I tried multiple ones) and my web browser. Only part of the image has been rendered. I wonder is this a PDF issue or the figure was drew on that way on purpose.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "Fli4Tp9Te9f", "reviewer": "Reviewer_oEJV", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper aims at learning whether a certain text is visualizable or not as generative models such as DALLE-2 are becoming very popular. To this effort, the authors curate a dataset called Text Imageability Dataset (TIMED) which contains 3620 sentences with label for whether it is a visual or non-visual text. The authors also curate an automatic dataset with similar labels and show that when a customized CLIP model is trained on a combination of these, the model is able to perform better than a lot of other baselines. Overall, this paper focuses on a niche area of research of visualness of text and provides a solution for it but there a lot of caveats as discussed in rest of the review.", "summary_of_the_review": "The paper focuses on a niche problem but doesn't discuss its significance and how it can help us build better model instead focuses on how to solve the problem. There is a lack of ablation study around backbones, objectives and text encoders which makes it hard to take away any significant lessons. The limitations imposed by the compounding effect of filtering using CLIP are prevalent and need to be disentangled. Given these concerns, I suggest a rating of ", "strength_and_weaknesses": "### Strengths\n- The paper provides a new dataset for evaluation and an automatic dataset for training a model for the task of testing the visualness.\n- The paper also provides a method to train CLIP model to detect whether a text is visual or not. \n- Provides ablations on the choice of the training objective for the purpose of testing visualness.\n- Clearly written and easy to read.\n- Ablations on MRC-I score and efforts to improve them using word embeddings.\n- Disentangles the effect of multi-stage training and shows that both datasets are needed to reach optimum performance.\n\n### Weaknesses\n- The motivation of problem statement is weak. The paper doesn't focus on why we need to solve the problem but mostly focuses on whats and the hows. Is it even needed to solve this problem? There is no discussion on how this approach can actually improve the current generative models by incorporating the visualness into the pipeline. The data stable diffusion is trained on (LAION) is already filtered out using CLIP scores which makes this process redundant. The question is whether we want to do automatic filtering on the fly during the training or beforehand as done in stable diffusion. I also don't understand why a human will ask the model to generate a non-visual text. Overall, I am not sure why this problem is important to solve.\n- I expected more analysis on the different backbones. For example, DeCLIP [1] performs better overall, FLAVA [2] and Unified-IO [3] have a better text encoder. It is hard to take away any conclusions from a single backbone test.\n- BERT (full) performs better than TIP-CLIP model when only trained on auto-labeled data. It is unclear whether the difference is due to pretraining objective? Maybe using VisualBERT-style [4] model would have disentangled that?\n- The downstream impact of the pretraining CLIP as TIP-CLIP-style objective is unclear as the only evaluation that has been conducted is on text-to-image retrieval and that too again on a subset that has been already filtered out by CLIP. There is a compounding effect of problems here. A clear comparison would be on rather MS-COCO or Flickr30k retrieval along with other tasks used in CLIP paper. The claim that the model has almost same performance on retrieval would strengthen if we have those results.\n- No other stronger text backbones have been used except BERT. Comparisons with RoBERTa/T5 would be useful to take a call.\n- Another compounding effect in the dataset is the fact that CLIP was used to filter out the matching images with custom threshold. \n- It is unclear how scene text fits into this picture. If the models get good at generating scene text, they can directly replicate non-visual text as scene text on a pdf page or a bland background. Overall, visualness is a subjective term changing from person to person and formulation of the problem.\n\n[1] Li, Yangguang, et al. \"Supervision exists everywhere: A data efficient contrastive language-image pre-training paradigm.\" arXiv preprint arXiv:2110.05208 (2021).\n\n[2] Singh, Amanpreet, et al. \"Flava: A foundational language and vision alignment model.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022.\n\n[3] Lu, Jiasen, et al. \"Unified-io: A unified model for vision, language, and multi-modal tasks.\" arXiv preprint arXiv:2206.08916 (2022).\n\n[4] Li, Liunian Harold, et al. \"Visualbert: A simple and performant baseline for vision and language.\" arXiv preprint arXiv:1908.03557 (2019).", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "3FtxHGTRZx", "reviewer": "Reviewer_yCLp", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors construct a dataset of sentences paired with a “visualness” label, indicating if the sentence evokes visual imagery. One part of this dataset is automatically synthesized by processing PDFs and computing CLIP scores between sentences and images in the PDFs. Another part of the dataset consists of sentences (also sourced from PDFs), that are then labeled by AMT workers for “visualness.” \n\nThe authors then use this dataset to further finetune a CLIP model to match “non-visual” sentences with a “NULL” image. This model is compared to various strong baselines, and shown to be the best of all considered methods. Various ablations and analysis are also conducted, such as measuring the correlation of BERT and CLIP models trained on the dataset to MRC scores, and ablating out automatic or human-labeled portions of the dataset for finetuning. Qualitative analyses of the attention of TIP-CLIP are also presented, as well as DALLE generations conditioned on “visual” and “non-visual” text.", "summary_of_the_review": "This paper poses an interesting problem: measuring the \"visualness\" of a sentence. The dataset presented is novel, but I have some slight concerns on the TIP-CLIP model and the dataset construction (see weaknesses section) that I would like the authors to address.", "strength_and_weaknesses": "Strengths:\n\nComprehensive analysis of TIP-CLIP, with good ablations of training methods, good comparison to existing MRC scores, and interesting visualizations. Appendix section contains even more analyses of the dataset, the ablations, the NULL image, and the TIP-CLIP model.\n\nThe baselines seem quite strong, and TIP-CLIP is able to outperform all of them. \n\nA dataset of sentences with \"visualness\" labels is novel (to the best of my knowledge). \n\nWeaknesses:\n\nIt's not clear to me that the data collection procedure should produce sentences that are really \"non-visual\" if the criteria for \"non-visualness\" is \"CLIP similarity with all images on page less than T_{neg}.\" For example, it seems possible to me that I can have a very visual sentence that just happens to not match any image on a page in a pdf.\n\nThe novelty and usefulness of the TIP-CLIP model is a tad unclear. It's just CLIP trained with an additional NULL image tag, and as mentioned in the paper one could easily just train a binary classifier. The justification for using TIP-CLIP rather than a binary classifier is that TIP-CLIP gives us representations that can be used for image retrieval, but TIP-CLIP actually underperforms CLIP in this task (MRR of 0.937 against 0.989). Relatedly, it seems that a binary classifier was trained and the F_1 score is presented, but not any other results. Could this be added to table 2?\n\nThe baselines are all very reasonable, but I'm surprised that the pre-trained CLIP model performs so well. My understanding is that it's a vanilla CLIP model and has therefore never seen the NULL image, which is why I find it interesting that it should perform so well. Is my understanding correct? And if so how come this model does so well?\n\nThe analysis of the TIP-CLIP model is well done, but I have a few questions about the qualitative results. In figure 2 the image embeddings and the text embeddings in the T-SNE plot seem disjoint, whereas I was under the impression that the contrastive loss used to train the CLIP model would bring these two clusters together. Is this simply how CLIP representations look? Or does this indicate a problem with the dataset? Or perhaps this is a problem with T-SNE hyperparameters?\n\nAdditionally, how was the attention calculated for figure 3 and 4? Anecdotally, I think attention in language models are all over the place, so I don't know how much weight should be put into these visualizations.\n\nVery small nitpick: I^e_m in equation 1 is defined self-referentially", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject