ICC: Quantifying Image Caption Concreteness for Multimodal Dataset Curation (2024)

Abstract: Web-scale training on paired text-image data is becoming increasingly central to multimodal learning, but is challenged by the highly noisy nature of datasets in the wild. Standard data filtering approaches succeed in removing mismatched text-image pairs, but permit semantically related but highly abstract or subjective text. These approaches lack the fine-grained ability to isolate the most concrete samples that provide the strongest signal for learning in a noisy dataset. In this work, we propose a new metric, Image Caption Concreteness (ICC), that evaluates caption text without an image reference to measure its concreteness and relevancy for use in multimodal learning. Our unsupervised approach leverages strong foundation models for measuring visual-semantic information loss in multimodal representations. We demonstrate that this strongly correlates with human evaluation of concreteness in both single-word and caption-level texts. Moreover, we show that curation using ICC complements existing approaches: It succeeds in selecting the highest quality samples from multimodal web-scale datasets to allow for efficient training in resource-constrained settings.

heading: 1 Introduction
text: Pre-training large vision-language models (VLMs) on web-crawled datasets consisting of imagecaption pairs has become the standard practice in achieving state-of-the-art results in vision-andlanguage tasks such as image captioning and multimodal representation learning. However, raw web data are often noisy and contain many low-quality samples, which impair VLMs’ learning in terms of quality and efficiency (Li et al., 2022; Schuhmann et al., 2022; Radenovic et al., 2023). While various factors impact data quality, we focus on semantic noise, characterized by analyzing the meaning of data items rather than, e.g., identifying low resolution images or quantifying token repetitions.
Existing datasets are commonly filtered using VLMs such as CLIP (Radford et al., 2021) to identify image-text semantic misalignments (Sharma et al., 2018; Schuhmann et al., 2022), i.e. captions irrelevant to their images; using rule-based proxies such as measuring the complexity of captions via semantic parsing (Radenovic et al., 2023); or removing images that contain text that overlaps with the caption (Maini et al., 2023). However, these approaches fail to identify captions that are highly abstract and may contain subjective, non-visual information, despite being semantically aligned with the image and having a sufficiently complex grammar. Figure 1 shows examples of such imagecaption pairs. A caption such as “It does not look like something I would want to eat” is semantically related to the image, yielding high CLIP similarity, but contains subjective details which provide a confounding signal when training VLMs (See also Figure 2). A model trained to generate such captions from images may learn to hal-
11048
lucinate details, e.g., liking a certain type of food in our example, which are not visually grounded and are highly subjective. Similarly, such imagecaption pairs provide a weaker signal for representation learning than images with visually concrete captions (e.g. “A sandwich sits on a small blue plate”), which may impede the learning process – particularly in a resource-restricted setting where data or compute is limited.
Thus, we suggest filtering image captions by their visual concreteness, referring to the extent to which a text describes visual aspects of a scene in a manner that can be vividly imagined (Schwanenflugel, 2013; Hessel et al., 2018)1. This contrasts with abstract text, which may correspond to many possible visual interpretations or include subjective information. We show that this new dimension of textual quality enables selecting imagecaption pairs that provide a strong supervision signal for vision-and-language tasks, particularly in resource-constrained settings where training directly on noisy web-scale multimodal data fails to converge to a satisfactory solution in a limited number of iterations.
We propose the Image Caption Concreteness (ICC) metric for quantifying the visual concreteness of image captions calculated from text alone, i.e., without an image reference. We measure concreteness using unsupervised autoencoding pipelines with visual-semantic information bottlenecks. Specifically, we use a visual-bottleneck autoencoder that leverages text-to-image generative models’ competence and a semantic-bottleneck autoencoder that identifies how well a large language model (LLM) recovers the input caption from its semantic CLIP embedding. As these models require costly inference through large generative models, they cannot feasibly run on a large scale; therefore, our ICC metric is distilled from these pipelines, enabling fast, computationally-efficient inference.
In our experiments, we demonstrate that when dealing with limited training iterations, employing ICC for filtering multimodal datasets leads to enhanced performance in image captioning and representation learning. Moreover, our results indicate a strong correlation between ICC and both
1Some works have treated this as roughly synonymous with imageability (visual association), while others use concreteness to refer more generally to association with sensory experiences of all types (Richardson, 1975; Khanna and Cortese, 2021). Our work focuses on the visual modality.
single-word concreteness and caption text scores. Stated explicitly, our contributions are as follows: (1) We propose the ICC metric distilled from foundation VLM models with a novel combination of unsupervised autoencoding pipelines; (2) we show that ICC highly correlates to human concreteness judgements of caption texts; (3) we demonstrate that ICC succeeds in selecting a core of samples from web-scale image-caption datasets for vision-and-language tasks, with superior downstream performance to existing filtering methods.

heading: 2 Method
text: Given an image caption (of an unseen image), we aim to predict its degree of visual concreteness. Our underlying assumption is that more visually concrete text can be mapped to a visual representation with less information loss. Conversely, we expect that visually abstract or subjective text cannot be converted to or from a visual representation without significant information loss, since it does not clearly describe a well-defined image.
As an example, consider the text “Wheat bread is always the healthy choice for lunchtime” in Figure 2b. The notion of wheat bread being a healthy choice is inherently non-visual and is unlikely to be directly depicted in an image. Therefore, this information is likely to be lost in an autoencoding process that includes an image as the bottleneck, when the encoded image is decoded back to the textual modality.
We model this effect with multimodal autoencoders (Kamath et al., 2023; Yang et al., 2023). In our setting, we use multiple autoencoder components that convert text to and from visual-semantic representations using foundation VLMs, and quantify the information loss of this process as a proxy for visual concreteness. While these autoencoders provide a strong signal, they are composed of slow, computationally-intensive large generative models making inference infeasible on a large scale. Therefore, we distill their scores into a small model which allows for an efficient calculation of the ICC scores.
We proceed to describe our proposed visualbottleneck autoencoder and semantic-bottleneck autoencoder components, and their distillation into the final ICC metric. See Figure 3 for an overview of our full pipeline.
Visual-Bottleneck Autoencoder (VBA). Since a caption represents an image, we construct the VBA by using an image as an intermediate representation via which textual information passes. In particular, we concatenate a frozen text-to-image model (Stable Diffusion 2, Ramesh et al., 2022) and a frozen captioning model (BLIP-2, Li et al., 2023) as shown in Figure 3 (bottom left). This autoencoding pipeline measures text concreteness by encoding and decoding a caption, followed by measuring semantic fidelity in reconstruction using BERTScore (F1) (Zhang et al.,
2019). We note that this pipeline contains no trained parameters as it concatenates pretrained, frozen models.
While the VBA pipeline is a simple and intuitive way of enforcing a visual bottleneck, it may sometimes produce sub-optimal reconstructions even for highly visual texts due to its inherently lossy nature. For example, the caption “a small black french bulldog” in Figure 3 may be reconstructed by the VBA from the generated image to “a dog with a white chest", which is relatively semantically far from the original caption and thus results in a relatively low reconstruction score of 0.6 for a concrete caption. This stems from the dense information content of generated images, which may contain details (such as the dog’s white chest) which were not mentioned explicitly in the original caption, and from the tendency of the captioning decoder to focus on different details than those used to generate the image. To alleviate this issue, we proceed to propose a complementary method using a stronger prior on caption semantics.
Semantic-bottleneck Autoencoder (SBA). Motivated by findings that CLIP embeddings encode visual information in text and particularly concreteness (Alper et al., 2023), we construct an autoencoding pipeline with CLIP text embeddings as a semantic information bottleneck, as shown in Figure 3 (top left). We extract visual information from the CLIP text embedding space by utilizing a frozen
LLM (Llama-2-7b, Touvron et al., 2023), by training a linear layer that converts CLIP text encoder’s output to inputs for the LLM. The training objective aims at reconstructing the input captions via a token-wise cross-entropy objective. By keeping the encoder backbone (CLIP) frozen, this introduces an information bottleneck preventing faithful reconstruction of abstract texts.
After training the SBA over image–caption pairs, we use it for measuring text concreteness by encoding and decoding the text followed by measuring reconstruction fidelity. To measure preservation of fine-grained textual details, we quantify this fidelity via per-character edit distance (Levenshtein et al., 1966), standardized by caption length, as detailed in Appendix A.1.
This pipeline generally succeeds in reconstructing highly concrete text (such as “A small black french bulldog” shown in the top left part of Figure 3). However, the strong textual prior of the SBA may also leak information about abstract and subjective captions as well (e.g. the abstract caption “A nice location” yields a relatively high reconstruction score of 0.4), limiting its correlation with visual concreteness. Overall, the SBA and VBA provide complementary scores, where each correlates more strongly to visual concreteness in different cases. Therefore, they perform most strongly when combined together, as we explicitly verify in our ablations in Section 4. We also show qualitative examples in figures 8 and 9 in the appendix.
ICC Distillation. Using the aforementioned pipelines to quantify the concreteness at scale is not feasible, as this requires running large models (e.g., diffusion models, LLMs) with billions of parameters for many forward passes per instance (up to dozens of forward passes for the diffusion models inference and for the LLM and captioning model decoding). This requires more than 1,000 GPU hours for a dataset of 1M samples. Therefore, we assemble SBA and VBA reconstruction scores over a relatively small collection of image-caption pairs and distill their aggregated values into our final ICC score. This enables efficient inference that can easily run on a large scale, with over a hundred times faster inference time and much less compute required. Specifically, we train a small text encoder model (Liumm et al., 2019) to predict a logit-linear combination of the SBA and VBA scores, computed as described in the appendix.
Implementation Details of ICC Construction. For the construction of our ICC score, we use a subset of CC3M (Sharma et al., 2018) composed of 595K image-caption pairs, introduced by Liu et al. (2023) and designed to have wider concept coverage. We take a subset of 476K samples for training the linear layer of the SBA, and train for 2 epochs with a batch size of 128 and learning rate of 2e-3 with cosine scheduling function. The remaining 118K samples are used for generating reconstruction scores through the VBA and the trained SBA. For each input caption, we generate five reconstructed captions using beam search (five beams) with the VBA’s captioner and the SBA’s LLM and then choose the reconstructed caption with the highest similarity to the source caption. By generating the reconstructions and measuring the reconstruction fidelities, we obtain a dataset of 118K captions and corresponding reconstruction scores. We standardize by caption length to disentangle the dependency of the reconstruction scores to the caption length (i.e., forcing the same distribution of scores for all caption lengths), as described in the appendix. We train a small language model (DistillRoberta-Base) to predict the combined scores on these samples with a Mean Squared Error objective. This final distilled model is used for generating the ICC scores.

heading: 3 Results
text: We turn to show ICC’s benefit in data curation for downstream tasks (Section 3.1), followed by its correlation to human judgement (Section 3.2).

heading: 3.1 VLM Dataset Curation
text: Experimental Settings. We investigate the effect of ICC and other filtering methods for curating a core of high-quality image-caption pairs from large multimodal datasets, comparing their effects on downstream task performance – both discriminative (representation learning) and generative (image captioning). We follow similar settings as described in the Datacomp (Gadre et al., 2023) benchmark’s filtering track2, with the following modifications to model the resource-limited setting: given a training dataset comprised of M samples, the downstream model is constrained to train for exactly N ≪ M iterations over the filtered subset of the dataset. This contrasts with
2As opposed to the BYOD track which allows for modifying the samples, for instance by using synthetic captions.
the original Datacomp setting where N = M, which requires significant compute for a web-scale dataset. Our formulation tests the ability of filtering methods to curate high-quality core subsets of such datasets. Our initial subset of LAION-400M is composed of M = 8M samples and we fix N = 2M training iterations. To verify the robustness of our method, we measure downstream performance over visually grounded benchmarks across three different sizes of filtering.
We compare to four existing filtering methods – CLIPScore (Hessel et al., 2021), Complexity and Action (CA) (Radenovic et al., 2023)3, TMARS (Maini et al., 2023), and PACScore (Sarto et al., 2023). CA is a rule-based filtering method which aims to retain only sufficiently complex captions that also contain an action, based on semantic parsing. T-MARS filters multimodal datasets by removing samples whenever an image includes text that overlaps significantly with the caption. PACScore trains a CLIP-based model with positive-augmented contrastive learning approach, showing improved correlations with human intuition in scoring image-caption pairs. As opposed to these methods, we focus on filtering according to the concreteness of image captions.
Captioning Models. In Table 1 we show quantitative results of applying ICC filtering on top of standard CLIPScore filtering over the subset of LAION-400M for training a captioning model. The captioning model used is an encoder-decoder architecture with a pretrained Swin (Liu et al., 2021) vision encoder and GPT-2 (Radford et al., 2019) text decoder. We use a batch size of 100, and learning rate of 2e-5 with a cosine scheduler. We test our approach over two standard captioning benchmarks datasets – MS-COCO (Lin et al., 2014) and NoCaps (Agrawal et al., 2019), across multiple captioning metrics (Papineni et al., 2002; Banerjee and Lavie, 2005; Lin, 2004; Vedantam et al., 2015; Anderson et al., 2016; Zhang et al., 2019; Wada et al., 2024). As illustrated in the table, filtering with ICC outperforms by a large margin the alternative filtering methods for captioning given a fixed number of desired samples and training iterations. Note that unlike other methods, ICC is directly aligned with the captioning objective, as a captioning model should generate visuallygrounded concrete text. This may explain the large
3Using our re-implementation, as there is no publicly available code.
gap in performance between ICC and other filtering baselines. We show qualitative comparison between captioning models trained with different filtering methods in Figure 4, exemplifying how filtering with ICC promotes more concrete and accurate captioning.
Image-Text Representation Learning. We also perform a representation learning experiment by training a dual text and image encoder model on LAION-400M filtered with different methods. Table 2 reports text-to-image retrieval over standard held-out retrieval benchmarks, namely MSCOCO (Lin et al., 2014) and Flickr30K (Plummer et al., 2015). The model is initialized from pretrained vision and text encoders (ViT-base, BERT-Base) (Dosovitskiy et al., 2010; Devlin et al., 2018), as suggested by Zhai et al. (2022). We use a batch size of 128, learning rate of 2e-5 with a cosine scheduling function. All other filtering methods in the table are identical to the ones in the captioning setting. As illustrated in the table, ICC yields superior performance for this task, showing that our method selects samples which provide better signals for downstream retrieval applications in this setting.
We note that although prior work has found filtering methods such as CLIPScore (Hessel et al., 2021) to be beneficial (Gadre et al., 2023), we find that it fails to significantly improve (or even degrades) results in the case of selecting a small core of samples. This accords with previous work showing that applying filtering to LAION-400M with CLIP degrades the performance (Maini et al., 2023) in some of the benchmarks, likely due to high-scoring images containing literal text that overlaps with the caption.

heading: 3.2 Concreteness Correlation
text: Table 3 shows the correlations of different concreteness estimation methods to ground-truth concreteness scores on both single-word and captionlevel benchmarks. We compare ICC to three baselines. The first baseline is zero-shot probing of CLIP through Stroop probing (SP) as proposed by Alper et al. (2023). The second baseline is aveCLIP (Wu and Smith, 2023), a learned metric quantifying concreteness at the sentence level, which generates multiple images from a caption and measures the average CLIP-similarity between the text and generated images. Due to its high computational cost, we evaluate it on a ran-
dom subset of each benchmark (as described in the appendix). Finally, we compare to GPT-3.5Turbo and GPT-4o (Achiam et al., 2023), used in the zero-shot setting by prompting them to provide concreteness scores. The prompts used are detailed in the appendix.
Correlation to Word Concreteness. We first validate our metric by measuring it on the dataset introduced by Hessel et al. (2018). This consists of 39,954 English unigrams and bigrams coupled with human-labelled concreteness scores on a scale from 1 (abstract) to 5 (concrete), averaged over annotators. To compare with prior work, we only use unigram nouns, totaling 14,562 items. As seen in Table 3, ICC outperforms prior dedicated methods for measuring word concreteness, while performing competitively with the proprietary and much larger GPT-4o.
Correlation to Caption Concreteness. We manually annotate concreteness scores for 500 captions from LAION-400M (Schuhmann et al., 2022), selected to cover a wide variety of levels of concreteness. As seen in Table 3, ICC outperforms existing methods in this setting by a large margin, demonstrating its advantage in selecting the most concrete image captions.

heading: 4 Ablations
text: Distillation Concreteness Effect. Although the distillation procedure is necessary to make inference feasible with respect to runtime, we provide further motivation by measuring the effect of distillation on the correlation to ground-truth annotations of concreteness scores in Table 4. As can be seen, the distillation improves correlations values, providing further motivation beyond compu-
tational efficiency and simplifying the inference of our ICC model. We hypothesize that this improvement is due to smoothing of noisy reconstruction of the VBA and SBA by the distillation process.
Distillation Speed-up. We ablate the speed-up provided by the distillation phase by running the SBA, VBA and the distilled ICC on the same hardware settings (an Nvidia A6000 GPU), the same batch size of 1 and the same caption samples. We find that the SBA and VBA process 0.45 and 0.2 samples per second respectively, and the distilled score processes 45 samples per second. Note that the time it would take to generate scores for our 8M subset of LAION-400M dataset is approximately 11,000 GPU hours for the VBA and 5,000 GPU hours for the SBA compared to just 50 GPU hours using the distilled ICC. Additionally, for a batch size of 1, the distilled model takes less than 700 MB of GPU memory compared to 13GB and 14GB for the VBA and SBA respectively.
Use of Both SBA and VBA Scores. We also ablate the use of both SBA and VBA scores for downstream captioning model training in Table 5. In the figure, we show captioning metrics (CIDEr and SPICE) of a model trained on a distilled version of each of the scores in isolation, compared to the combined ICC metric which outperforms both.
ICC Model Component Ablations. In Table 6, we ablate the effect of various design choices in the ICC pipeline by evaluating their effects on caption concreteness prediction (using the benchmark described in Section 3.2). In particular, we test different LLM sizes (Zhang et al., 2024; Geng and Liu, 2023) in the SBA pipeline, different captioning model architectures in the VBA pipeline, and the similarity measure used in each pipeline (edit distance vs. BERTScore). To identify the effect of each component, we evaluate SBA and VBA predictions in isolation (without combining or distilling them). As is seen in the table, our chosen LLM and captioning model perform comparably
to the alternative models tested, showcasing the robustness of the VBA and SBA across model sizes. Moreover, while the simple edit distance similarity measure performs acceptably for the SBA pipeline, the BERTScore similarity measure produces significantly better correlations in the VBA pipeline, matching the intuition that the VBA is inherently lossy with respect to the precise form of texts and must rely on a more semantic measure to properly detect abstract sentences.

heading: 5 Related Work
text: Evaluating Text Concreteness. Word concreteness is a topic of interest in cognitive science (Paivio et al., 1968; Richardson, 1975; Schwanenflugel, 2013; Khanna and Cortese, 2021), and a number of works have studied auto-
matic prediction of word concreteness using machine learning (Hill et al., 2014; Hill and Korhonen, 2014; Hessel et al., 2018; Rabinovich et al., 2018; Charbonnier and Wartena, 2019; Alper et al., 2023). However, little attention has been paid to measuring concreteness at the caption or string level. Shi et al (2019) define concreteness of constituents by matching them to images for learning syntactic representations without explicit supervision; as was later shown, the signal of noun concreteness plays a key role in the model’s syntactic predictions (Kojima et al., 2020). Most similar to us is Wu and Smith (2023), who generate multiple images for each caption and average the CLIP similarity scores over all the images to produce a caption-level concreteness score. Other text evaluation metrics compare to reference texts (Gehrmann et al., 2023) or a reference image (Hessel et al., 2021), while we are interested in the inherent quality of text in isolation (namely, its visual concreteness).
Multimodal Dataset Curation. Due to the highly noisy nature of Internet multimodal data, prior works have filtered using approaches such as rulebased text parsing (Radenovic et al., 2023), using CLIP similarity to detect misaligned text-image
pairs (Schuhmann et al., 2022), de-duplicating semantically similar content (Abbas et al., 2023), and removing samples with text that overlap with the image (Maini et al., 2023). A number of prior works have also proposed replacing or augmenting multimodal datasets with synthetic samples (Li et al., 2022, 2023; Fan et al., 2023; Lai et al., 2023; Nguyen et al., 2023). By contrast, we do not require modifying the given dataset and identify semantically infelicitous captions allowed by prior methods. Our work also contrasts with dataset distillation, which has been applied to multimodal dataset curation (Wu et al., 2023); while dataset distillation methods select samples to explicitly optimize a chosen downstream objective, we focus on the simpler and more general task of identifying samples of inherently poor quality.

heading: 6 Conclusion
text: We present a new metric for measuring the visual concreteness of image captions without an image reference. By leveraging strong foundation models, we quantify visual-semantic information loss in an unsupervised manner and find that this highly correlates with human concreteness judgments. Our results demonstrate that ICC is effective at selecting a core of high-quality image-caption samples from web-scale multimodal datasets for training models in the resource-constrained setting. We foresee the use of ICC in additional tasks requiring the curation of web-scale multimodal data, where high-quality, visually-concrete text is needed.

heading: Limitations
text: While our method manages to detect visually concrete captions well, it lacks sensitivity to grammatical structure, which might cause it to label oddly phrased captions as concrete. For instance, consider the caption: “a computer near a tree with a boy next to a table with a keyboard”. This cap-
tion is highly concrete and gets a high ICC score of 1.0. However, removing all object relations from the caption produces the following: “computer tree boy table keyboard” which results in a relatively minor decrease of the ICC score to 0.89. Such low-quality captions might have a negative impact on tasks such as image captioning where the model must learn to output grammatically correct English sentences which should ideally describe relevant fine-grained relations between entities. We hypothesize that this behavior stems from the dataset used to train the distillation model (CC3M) which is not likely to include such oddly phrased captions, and so these non-grammatical structures are not learned. We hypothesize that training over a dataset with higher caption diversity will likely alleviate this issue.
In addition, due to limited computational resources, our experiments were conducted on a relatively small scale of 8 million sample initial training dataset based on LAION-400M. We expect that increasing the scale and the filtered dataset proportionally will result in a performance improvement in the downstream model performance.
However, we leave verifying this as well as testing the effect of ICC filtering on other downstream tasks such as VQA and caption ranking to future work.
Finally, while our method detects and filters an important category of noise in multimodal datasets, we note that abstract captions such as those in Figure 2 may contain important information which our method discards. Future work might instead extract the relevant visual information from such captions, to avoid losing the information signal in such items. We also note that such captions often contain external or subjective information which could be of interest to tasks such as news image captioning or multimodal sentiment analysis, where external context is of interest. To identify such cases, further work might enhance the interpretability of our method to explore why a caption is or is not concrete.

heading: Ethics Statement
text: Models trained on multimodal Internet data may inherit biases from their training data. Our method is not designed to filter potentially harmful image descriptions; moreover, such biases are also present in the models used as part of our pipeline (CLIP, generative models) and thus our model may possibly inherit or amplify these issues for downstream tasks. We anticipate further research into such biases and guidelines needed before putting these models into deployment.

heading: Acknowledgements
text: This work was partially supported by the KLA Foundation and Google. We thank Yonatan Bitton and Keren Ganon for their helpful feedback.

heading: Appendix A Implementations Details

heading: A.1 Standardizing By Caption Length
text: We aim to have reconstruction scores that are only dependent on the concreteness of captions and not on the length of the captions for both the SBA and VBA. In Figure 5, we show the distribution of the edit-distance based reconstruction similarities of the SBA before and after standardization per caption length. We can see in Figure 5a that there is a strong dependency on caption length, which we would like to avoid.
More specifically, we force the reconstruction similarity distribution to be distributed according to LN (µ = 0.5, σ = 1), where LN denotes a Logit-Normal distribution. The normalization is performed by standardizing the logit of the similarities (defined by ln( 11−p)) for each caption length, and then taking the inverse logit. We can see in Figure 5b that short captions are reconstructed more easily compared to longer ones, and that normalization by caption length successfully disentangles the reconstruction scores from the caption length dependency.
A.2 ICC Distillation We distill the knowledge obtained by the two pipelines described in the paper in a two-stage manner. Firstly, we distill the VBA and SBA scores into two distinct DistilRoBERTa (Liumm et al., 2019) models. We then collect a small subset of 244 captions, sampled to have approximately uniform joint distribution of scores, and annotate the concreteness scores of these captions. This is showcased in Figure 6. We regress over these samples to get the optimal weights.

heading: A.3 Caption Concreteness Benchmark Distribution
text: Our aim is to have a small, yet diverse set of samples that represent the wide diversity of possible captions. Since Laion-400M is very noisy and only a small portion of it includes highly concrete captions, we curate our captions to achieve a balanced distribution of concreteness scores, as illustrated in Figure 7. As seen there, the concreteness of the benchmark’s captions is evenly distributed between abstract and concrete concepts.

heading: A.4 Zero-Shot CLIP Concreteness Score
text: We adapt the Stroop Probing method (Alper et al., 2023) for estimating text concreteness. While
Alper et al. (2023) test this on single words, we adapt this method to captions by replacing the empty slot in prompts with a caption rather than a single word. We use their prompts, omitting those which do not match the context of an entire caption being inserted in the masked slot (i.e., omitting the prompts “Alice giving the [*] to Bob” and “Bob giving the [*] to Alice”).

heading: A.5 GPT Prompts
text: The following prompts were used to extract concreteness scores for image captions4 from GPT-3.5 and GPT-4o: System: “You are an expert visual reasoner, capable of understanding the visual concretess of image captions. A visually concrete caption is a caption that is highly visual, and can be vividly imagined.” User: “Provide a numerical score on a scale of 1-5, when 1 is non-visual and 10 is highly visual caption for the following caption : <caption>. Only provide the numerical score and nothing else.”
Note that we experimented with three different ranges of [1-N] of concreteness scores in our prompts: N=3, N=5 and N=10. We found that N=5 yielded the best results.

heading: A.6 aveCLIP Word Concreteness
text: Since aveCLIP requires generating many images per word or caption, we found that running aveCLIP over the entire word concreteness dataset is not feasible due to runtime constraints. Therefore, we evaluate its performance on a random subset of 150 words/captions.

heading: A.7 Training Hyperparameters and
text: Additional Information
SBA. We train the linear layer of the SBA with a batch-size of 128, learning rate of 2e-3 with cosine scheduler and a warm-up ratio of 0.03, and train for a two epoch over a single Nvidia-A6000 GPU. All other hyperparameters are set to the defaults of the HuggingFace Trainer API.
VBA Text-To-Image. For the image generation of the diffusion model in the VBA, we use guidance scale of 9 and 20 inference steps.
4To get the concreteness scores of words, we used the same prompts with “word” instead of “caption” in the appropriate places.

heading: A.8 Model Checkpoints Used
text: We detail here all the checkpoints that were used in our experiments. All model checkpoints are taken from the Hugging Face Model Hub5. For the SBA, we used:
• openai/clip-vit-large-patch14 (only the text encoder) • meta-llama/Llama-2-7b
For the VBA, we used:
• stabilityai/stable-diffusion-2 • Salesforce/blip2-opt-2.7b
For the distilled model, we used:
• distilroberta-base 5https://www.huggingface.co/models
For training a captioning model, we used:
• microsoft/swin-base-patch4 -window7-224-in22k • gpt2
For training a dual-encoder model, we used:
• bert-base-uncased • google/vit-base-patch16-224

heading: A.9 Finding the Score Combination Parameters
text: To compute the combination parameters of the SBA and VBA scores, we label 244 captions, sampled uniformly over VBA and SBA scores, with concreteness scores in the range 0–3. We use logistic regression to find the parameters a, b, c of σ(a · V BA+ b · SBA+ c), where σ(x) = 1
1+e−x
is the sigmoid function, such that the output will approach 1 for concrete captions and 0 for abstract ones. We label concrete captions as captions with concreteness above the median score in the labeled dataset and abstract captions as captions with a score below this median. We visualize the annotated samples and the regression line a · V BA+ b · SBA+ c = 0 in Figure 6. The parameters found and used in our ICC are a = 13.2, b = 3.6 and c = −9.4. As seen in the figure, both scores contribute to the optimal predicted concreteness score, validating the importance of using both SBA and VBA components together in our full pipeline.

heading: Appendix B Additional Qualitative
text: Examples
We visually show examples of each of the scores’ weaknesses and the way they compliment each other. In Figure 8, we show examples of concrete captions, the reconstructed captions by VBA and SBA, and the different scores of each of them. The first four rows exemplify why VBA may fail to reconstruct some concrete captions. For instance, the caption “a nurse mopping a surgeon’s brow during an operation in an operation pub” was reconstructed to “two people in protective gear” which bears relatively low semantic similarity to the original caption. These cases mainly stem from the inherent difficulty of reconstructing (through a captioning model) from an image the exact caption from which the image was generated, as there may be many possible such captions. In this case, the use of SBA helps determining that the caption is concrete.
In a complementary manner, we show in Figure 9 examples of abstract captions. In this figure, the first four rows demonstrate that using SBA alone is also not enough, as it is sometimes able to reconstruct abstract captions due to the higher semantic information that is contained in the CLIP embeddings. In this scenario, VBA compensates for these failures, as it is very unlikely to reconstruct abstract text.
These qualitative examples further illustrate the benefit of using both VBA and SBA. Indeed, in
Figures 8–9, it can be observed that ICC reflects the advantages of both pipelines by generating low scores for abstract captions, and high scores for concrete ones in a consistent manner.
crete captions and the final distilled ICC scores. We mark by red low reconstruction scores which correspond to unsuccesfull detection of the concrete captions. As illustrated above, VBA yields generally less consistent scores for concrete captions (see the text for further discussion). Nonetheless, our final distilled scores correctly identify these captions as concrete ones, obtaining high ICC scores over these captions.
stract captions and the final distilled ICC scores. We mark by red captions which were reconstructed well (note that in the case of abstract captions, high scores correspond to unsuccessful detections of the abstract captions). As illustrated above, SBA yields generally less consistent scores for abstract captions (see the text for further discussion). Nonetheless, our final distilled scores correctly identify these captions as abstract ones, obtaining low ICC scores over these captions.

heading: Future Work
text: However, we leave verifying this as well as testing the effect of ICC filtering on other downstream tasks such as VQA and caption ranking to future work. Finally, while our method detects and filters an important category of noise in multimodal datasets, we note that abstract captions such as those in Figure 2 may contain important information which our method discards. Future work might instead extract the relevant visual information from such captions, to avoid losing the information signal in such items. We also note that such captions often contain external or subjective information which could be of interest to tasks such as news image captioning or multimodal sentiment analysis, where external context is of interest. To identify such cases, further work might enhance the interpretability of our method to explore why a caption is or is not concrete