No Detail Left Behind: Revisiting Self-Retrieval for Fine-Grained Image Captioning
Authors: Manu Gaur
Darshan Singh S
Makarand Tapaswi
Venue: Accepted by TMLR ()

Abstract: Image captioning systems are unable to generate fine-grained captions as they are trained on data that is either noisy (alt-text) or generic (human annotations). This is further exacerbated by maximum likelihood training that encourages generation of frequently occurring phrases. Previous works have tried to address this limitation by fine-tuning captioners with a self-retrieval (SR) reward. However, we find that SR fine-tuning has a tendency to reduce caption faithfulness and even hallucinate. In this work, we circumvent this bottleneck by improving the MLE initialization of the captioning system and designing a curriculum for the SR fine-tuning process. To this extent, we present (1) Visual Caption Boosting, a novel framework to instill fine-grainedness in generic image captioning datasets while remaining anchored in human annotations; and (2) BagCurri, a carefully designed training curriculum that more optimally leverages the contrastive nature of the self-retrieval reward. Jointly, they enable the captioner to describe fine-grained aspects in the image while preserving faithfulness to ground-truth captions. Our approach outperforms previous work by +8.9% on SR against 99 random distractors (RD100) (Dessi et al., 2023); and +7.6% on ImageCoDe. 

Additionally, existing metrics to evaluate captioning systems fail to reward diversity or evaluate a model's fine-grained understanding ability. Our third contribution addresses this by proposing self-retrieval from the lens of evaluation. We introduce TrueMatch, a benchmark comprising bags of highly similar images that uses SR to assess the captioner's ability to capture subtle visual distinctions. We evaluate and compare several state-of-the-art open-source MLLMs on TrueMatch, and find that our SR approach outperforms them all by a significant margin (e.g. +4.8% - 7.1% over Cambrian) while having 1-2 orders of magnitude fewer parameters. We also outperform vanilla SR by +14.4% to +19.5%.

Published in Transactions on Machine Learning Research (01/2025)
No Detail Left Behind:
Revisiting Self-Retrieval for Fine-Grained Image Captioning
Manu Gaur
manuggurrwrrkk@@aaal.cco
CVIT, IIT Hyderabad, India
Darshan Singh
dorshon hit.aammiiinggerrreeerrr.......
CVIT, IIT Hyderabad, India
Makarand Tapaswi
makkarond.topasuiiiiii
CVTT, IIT Hyderabad, India
Reviewed on OpenReview: https://ppperrrvaaaeee/r////oots
Abstract
Image captioning systems are unable generate fine-grained captions they are trained on data that is either noisy (alt-text) or generic (human annotations). This is further exacerbated by maximum likelihood training that encourages generalion of frequently occurring phrases. Previous works have tried to address this limitation by fine-tuning captioners with a self- retrieval (SR) reward. However, we find that SR fine-tuning has a tendency to reduce caption faithfulness and even hallucinate. In this work, we circumvent this bottleneck by improving the ML.E initialization of the captioning system and designing a curriculum for the SR fine-tuning process. To this extent, we present (1) Visual Caption Boosting, novel framework to instill ino-grainedness in generic image captioning datasets while remaining anchored in human annotations; and (2) BagCurri, carefully designed training curriculum that more optimally leverages the contrastive nature of the self-retrieval reward. Jointly, they enable the captioner to describe fine-graines aspects in the image while preserving faithfulness to ground-truth captions. Our approach outperforms previous work by +8.9% on SR against 99 random distractors (RD100) (Dessi et al. 2023); and +7.6% on ImageCoDe.
Additionally, existing metries to evaluate captioning systems fail to reward diversity or evaluate model's fine-grained understanding ability. Our third contribution addresses this by proposing self-retrieval from the lens of evaluation. We introduce TrueMatch, benchmark comprising bags of highly similar images that uses SR to assess the captioner's ability to capture subtle visual distinctions. We evaluate and compare several state-of-tho-art open-source MLLMs on TrueMatch, and find that our SR approach outperforms them all by a significant margin (e.g. +4.8% - 7.1% over Cambrian) while having 1-2 orders of magnitude fewer parameters. We also outperform vanilla SR by +14.4% to +19.5%.
1 Introduction
Image captioning, or generating natural language image descriptions, has witnessed remarkable progress over the last decade. Today's captioning systems composed of sophisticated deep learning architectures (Mokady
et al. 2021 Stefanini et al. 2022 Dai et al. 2024 Liu et al.) 2023) trained vast datasets (Lin et al. 2014
Sharma et al. 2018 Thomee et al. 2016 Desai et al. 2021 Schuhmann et al., 2022). However, even with these advances, approaches often generate generic captions that are unable to differentiate similar images (see Figure M violating the fundamental purpose of a caption: to facilitate accurate and efficient communication
of visual content (Fisch et al.] 2020 Kreiss et al. [2022] Dessi et al. 2023 Das et al. 2017). We attributc the shortcomings of image captioning systems to three key factors: (1) the nature of their training data,
(2) captioning evaluation metrics, and (3) the maximum likelihood estimation (MLE) training approach.

---

Published in Transactions on Machine Learning Research (01/2023)
COCO MLE
dog jumping in the air to catch a frisbee.
A dog jumping in the air to catch frisbee.
A dog is holding frisbee in it's mouth.
COCO SR (CVPR 2023)
A cly jumping in the air to catchh a frisbee.
A dog jumping in the air to eatch frisbee.
dog and a frisbee grassy field.
brown dog jumps up to catch white frishee in its mouth us it flies through the air in a grassy field, surrounded by trees.
brown dog jumps to catch a blue Prishee in its mouth in a grassy field, with fence in the background serving boundary for the area.
small brown and white dog stands
a grasy field next green Frisbee, bolding it in its mouth and appears to be enjoying playing with the grassy field.
For similar images, captioning systems struggle to generate meaningful captions that uniquely describe each image. In this example, COCO MLE: A model trained on COCO with MLE generates the same generic caption for the first. two images. COCO SR (Dessi el. al., 2023): While the self-retrieval (SR) objective may help, the COCO captions not rich enough to generate salient visual details. OUR SR: Our improved data and training recipe results in fine-grained, and therefore discriminant captions.
Figure 1:
I. Challenges with the training data. Training datascts may be divided into two: curatod datasets (COCO (Lin et al. 2014), Flickr30k (Plummer et. al. 2015)) or large-scale alt-text data (e.g. CC3M (Sharma et al. 2018)) While alt-text is noisy and may be unaligned with the image, hlmman-annotate (COCO) may be generic [Korublith et al. 2023) and lack world knowledge (Bavishi et al. 2023) as annotators describe visual concepts in simplistic manner (e.g. labeling a Golden Retriever as a dog). Recently, foundation models are being used to enhance large-scale alt-text. data by making them denser (Doveh et al. 2024 Urbanek et al.) 2024). However, such methods are prone to inherit biases (e.g. gender, geography
captions
present in foundation models (Hall et al. 2023 Sirotkin et al., 2022 Basu et al. 2023 Salman ct al. 2022
and exhibit verbose language modeling priors (Liu et al.) 2024.
To address these data inadequacies, we propose Visual Caption Boosting (VCB), model agnostic framework designed to generate dense captions that holistically capture different aspects of the image (objects, altributes, relations, scene, etc.) while remaining anchored in human annotations (see Section 2). In brief, multiple human annotated captions are blended together using a Large Language Model (LLM) and expanded with an image description generated from a Multinodal Large Language Model (MLLM). In case of conflicting visual details, we prompt the LLM prefer the blended caption over the description from the MLL.M Thus, VCB creates fine-grained captions that are grounded in human annotations, enabling rich and informative datasets to train image captioning systems.
Current metrics can be broadly classified into veference-based and reference-free. Reference-based metrics such as BLEU (Papineni et al. 2002), CIDEr (Vedantam et al. 2015). and SPICE (Anderson et al. 2016) rely on comparisons to ground-truth (GT) captions that may not include image details beyond salient. objects. Although these metrics evaluate grammatical correctness, they tend to penalize captions that are more specific than the ground-truth, often favoring generic descriptions (Wang et al., 2020). Reference-[ree metrics like CLIPScore (Hessel et al.) 2021)) alleviate this issue by directly measuring image-text similarity, but may fail to assess discriminativeness of the captions. These limitations necessitate the development of new evaluation strategies that incentivize the models to produce fine-grained captions.
II. Image Caption Evaluation.
In this work, we ask what makes a "fine-grained" or "good" description when evaluating captioning systems? We posit that a caption should be succinct, but enable a listener to identify the target image from a bag of images with similar visual elements (Liu et al., 2018). To this extent, we propose to evaluate captioning systems through the lens of self-vetrieval (SR). We measure the ability to retrieve the target image based on the generated caption within bag of highly similar distractor images.
2

---

Published in Transactions on Machine Learning Research (01/2025)
Traditionally, improvements in text-to-image (T2I) retrieval have focused on either enhancing the scoring
functions (Faghri et al. 2017) or making denser, more detailed text queries (Tai et al. 2023 Li et al. 2024 Singla et al. 2024 Shvetsova et al. 2024). In this work, we introduce a novel perspective: by curating sets of similar images, we can use T2I retrieval as a tool for evaluating whether the text query (caption) captures fine-grained visual details necessary for retrieving the target image. Specifically, we const.ucc bags of highly similar images, designed such that uniquely describing each image within a bag demands capturing different. facets of visual understanding. For example, retrieving the middle image in Figure [ incorporate information about attributes (blue frisbee) or background objects (fence). Note, each caption is generated without looking at the other images in the bag. Thus, we use discriminability as proxy to measure how well a model describes fine-grained components in an image.
requires the caption to
While previous works have incorporated SR as a reward signal during training (Dessi et al. 2023) to our knowledge, we are the first to curate bags of similar images and leverage SR to evaluate fine-grained understanding cxhibited by captioning systems. To this end, we introduce TrueMatch, a benchmark of image sets with varying size. TrueMatch offers comprehensive evaluation framework to assess the ability of captioning systems to capture various aspects of visual discrimination such as positioning, action, orientation.
III. Guiding captioners away from their language modeling priors. MLE training incentivizes captioning systems to overuse common concepts and statistically probable phrases when describing visually similar images. While previous works have optimized self- reff-retrieval (SR) to learn discriminant captioners (Luo let al. 2018: Liu et al., 2018: Cho et al.] 2022; Dessi et al. 2023), ; the models are first trained via MLE on generic caption datasets (e.g. COCO). We find that this is suboptimal and show that SR. fine-tuning is sensitive to initialization: it is necessary to start with a captioning system that captures fine-grained details in order to better preserve faithfulness to GT captions. In fact, we discover propensity of captioning systems to hallucinate (Section 4.4) when trained via SR on generic captions (Dessi et al. 2023 Thus, current SR approachees face two distinct challenges: a trade-off between retrieval performance and caption faithfulness, and sub-optimal fino-grained retricval performance.
To enhance SR's ability to instill fine-grained visual information, we (1) fine-tune both components (language and visual) of the captioning system (Section 4.5), (..),, and (2) mine multiple hard negatives (visually similar sets of images) to create training bags. We also introduce curriculum learning approach that progressively increases the bag size during training to leverage the contrastive nature of our retrieval-based reward (Section 4.6). Our carefully designed curricuuu enables SR to leverage the rich initialization provided by VCB, improving retrieval performance as well as the faithfulness of the generated captions. Through our training procedure, we are able to circumvent the aforementioned trade-off and achieve substantial performance gains, surpassing (Dessi et al., 2023) by 20% on TrueMatch without making any modifications to the model architecture or reward. Our results demonstrate that well-crafted training paradigm achieves performance comparable to MLLMs that are orders of magnitude larger. Figure [ shows images of a bag from TrueMatch and captions generated by the same model architecture trained with different datasets (COCO us. Ours) and paradigms (MLE us. Vanilla SR us. Our SR).
Key contributions. We address three challenges plaguing current image captioning systems: dala, evaluation, and MLE training. (1) We identify the inadequacies of image captioning datasets in Section 2 and propose Visual Caption Boosting, a novel caption enhancement strategy that leverages LLMs and MLLMs to generate dense, informative, and unbiased captions that are anchored in human annotations.
(2) We introduce
TrueMatch (Section 3), benchmark consisting of curated bags of highly similar images that uses self-retrieval (SR) to assess the ability of captioning systems to capture fine-grained visual distinctions. (3) In Section 4 we leverage the rich data and effective evaluation introduced in the previous two sections to train finegrrained captioning systems with SR. We offer new insights into SR fine-tuning through extensive ablations revealing its sensitivity to MLE initialization, failure to preserve caption faithfulness, and tendency to hallucinate. (4) Finally, we design a simple "plug and play" training recipe that enables SR to improve caption faithfulness while significantly outperforming vanilla SR (Dessi et al., (023) on TrueMatch and achieving state-of- state-of-the-art results on ImageCoDe (Krojer et al. 2022 anooher benchmark for fine-grained image retrieval.
3

---

Published in Transactions on Machine Learning Research (01/2025)
In this image. there group of zebras standing near a body water, such river lake. The zebras appear to be grazing or drinking from water. There also bird flying overhead. which adds to the natural setting of image. The zebras and the bird seem to be coexisting peacefully in their respective habitats
MLLM
A herd of zebras is gathered around watering hole in the wild. where they appear to grazing. bird flies overhead, coexisting peacefully with the zebras in their respective
Some zebiä chilling in with bind fyyng over.
Visual Caption
LLM (Anchored Expander)
A herd of zebra on the plains at Wetering hole.
Several zebras are grazing and standing near watering hole the wild, with a bird flying overhead
group of zebras and birds are gatherad around water
Holistic Caption
LLM (Blender)
There a herd of zebras slanding around.
Blended Caption
There one sewaral rabnes grazing Bear the watel as hidd fles over
Figure 2: Example of Visual Caption Boosting transforming the original human annotated captions (left) to a Holistic Caption. First, an LLM blends the human annotations to create a Blended Caption. Next, an MLLM generates dense visual caption that may be noisy. Finally, we create a Holistic Caption by instructing the LLM incorporate fine-grained details from the Visual Caption with the Blended Caption, while staying anchored in human annotations in case of conflicts. Specific prompts are shared in Appendix A.1 The colors indicate various concepts racted from the human annotations or the visual caption. The red underlined text (illustrated by us for ease of understanding) indicating hallucinations or verbose text, is ignored in the Holistic Caption as we anchor it to human annotations.
2 Improving Datasets through Visual Caption Boosting
Training with dense ground-truth descriptions benefits vision-language models (Doveh et al.] 2024 Urbanek
et al. 2024 Lai et al.) 2023). Although existing image captioning datasets like COCO (Lin et 2014)
provide multiple annotations per image, they do not clicit detailed captions. This leads to generic annotations such "There is herd of zebras standing around" " (see Figure 2 that ignore finer visual concepts and fail to describe the image holistically. This bottleneck reflects in trained captioning systems as well.
To address this, recent works leverage foundation models to synthctically expand visual information within captions (Doveh et al.
2024 Li al.
2024 Singla el al. 2024 Lai et al.) 2023 Jiao el al. 2024), instilling
a wider range of visual details. However these methods are prone to inherit biases present in foundation models (Sirotkin et al.
2022 Basu et al. 2023 Salman al. 2022). Moreover, the descriptions generated by some of these methods are excessively long making them susceptible to hallucinations (Favero et al., 2024 They also pose challenges to training captioning systems as they exceed the token capacity of current VL.Ms (e.g. CLIP with 77 tokens (Urbanek et al. 2024]). Nevertheless, while individual COCO captions are sparse, we find that they describe complementary facets of the image e.g. "walering hole", "bird flying over", "herd of zebras" (Figure 2. This also corroborates findings of Ye et al. (2023) that annotators of different cultural backgrounds describe distinct visual concepts when viewing the same image.
Building upon these findings, we introduce Visual Caplion Boosting (VCB), novel two-stage approach to enrich the training data with dense, more informative captions that encourage captioning systems learn and generate rich descriptions. VCB leverages foundation models and the diverse perspectives offered by human annotators to generate rich descriptions while being anchored in human annotations.
Step 1. BlendCap leverages an off-the-shelf LLM to create a blended caption that combines multiple facets of visual information that the human annotators describe. Figure Saaure2sti shows an example of how few captions can be blended together into comprehensive description of the image using our method. Notably, prompt the LIM to minimize redundant information resulting in succinet descriptions.
4

---

Published in Transactions on Machine Learning Research (01/2025)
Step 2. HolisticCap extends BlendCap by incorporating a fine-grained visual description produced by an MLLME Specifically, we prompt the I.M to instill the visual caption into the blended caption while preferring human-grounded BlendCap in case of confficting visual information. As seen in Figure 2 this enables the LLM to ignore hallucinations present in Visual Caption such as "river or lake " and "drinking", as they conflict with BlendCap's description like "walering hole" " and "gmaznn and slanding" Additionally, the anchoring of semantic visual information in human annotations encourages the LLM to eliminate verbose tendencies of MLLMs, producing rich and succinct captions that capture fine-grained details. Although recent works (Urbanek et al. 2024 Lai et al. 2023) address similar challenges, they have some drawbacks they anchor synthetic visual information in alt-text (Lai et al. 2023) or depend on manual annotation for incorporating dense visual information (Urbanek et al. 2024).
Anchored captions hallucinate less. We conduct a human study to measure the degree of hallucinations in BlendCap and HolisticCap (sce Appendix A.2). We find that BlendCap introduces negligible hallucinations and that anchoring VisualCap in human annotations significantly (41.6%) reduces hallucinations in HolisticCap.
In summary, Visual Caption Boosting is able to efficiently instill fine-grained visual information into image caption datasets while being less prone to hallucinations. Appendix [A.] presents the LLM and MLLM prompts and Appendix A.3 analyzes the distribution of the lengths of the generated captions.
3 TrueMatch: Fine-grained Evaluation through Self-Retrieval
Existing self-retrieval (SR) approaches require models to select the target image from a set of N random distractor images in the dataset (Dessi ct al. 2023). However, randomly chosen distractors often have simple differences (e.g. the primary object or scene), making easy for captioning systems to distinguish between them. This evaluation is suboptimal as they neither encourage the model to generate detailed captions nor do they evaluate fine-grained abilities of captioning systems.
In this section, we present the SR setup used in our work. We propose TrueMatch, a benchmark of carefully curated bags of highly similar images that enables SR to evaluate whether captioning systems capture different facets of fine-grained visual discrimination. The results in Section 3.2 show that most captioning systems (including MLLMs) struggle to generate captions that would allow distinguishing fine-grained visual details.
Self-retrieval setup. Within bag of images B (or minibatch during training), we require the generated caption for image to retrieve itself (image i) from B. Note, B contains i and a set of visually similar distractor images DD The caption e and all images in B are encoded using CLIP text and vision encoders respectively and ranking is performed by computing the cosine similarity sim(c, i) in the CLIP embedding space. Consequently, the caption is deemed to be good ( sim(c, > sim(c, (c f'), Vi' E D.
3.1 Benchmark Creation
Given a dataset, the benchmark creation process involves curating bags of highly similar images. We use the 10,000 images from COCO's validation and test sets (Karpathy & Fei- 2015) for our benchmark.
Creating candidate bags of images. We use the fine-grained descriptions generated through HolisticCap and encode visual and textual features in the CLIP embedding space. We treat each multimodal embedding (concatenation of the two modalities) as a query and use simple nearest neighbour search to create bags of highly similar images. Algorithm [ provides details on the bag creation process, especially for creating bags of variable size. Having bags of varying size facilitates SR evaluation at multiple levels of difficulty as increasing the bag size limits the number visual concepts that can be used to uniquely describe each image. This is different from previous works that typically use image pairs (Jhamtani & Berg- KirrgKKirpattrcck 2018 Park et all 2019: Tong et al. 2024b) random distractors (Dessi et al. 2023) to evaluate captioners.
Automated bag curation. We compute intra-bag similarity a, an average pairwise cosino-similarity bel ween multimodal embeddings of all images in the bag to quantify the difficulty of uniquely describing each image in the bag (details in Algorithm 2 To create TrueMatch, we sort the list of bags created
We adopted TnstructBI.IP for this work, see http:://weeniineeeooausaeeeeeeer
5

---

Published in Transactions on Machine Learning Research (01/2025)
above in the descending order based on and use set V to keep track of images that are included in TrueMatch. Going down the sorted list, bag is added to the benchmark only if none of its images exist in V. This ensures that only the hardest bags with highest visual similarity are added to the benchmark for each visual concept in the embodding space. Bags within TrueMatch unlike previous works involving image sets, are fine-grained (Dunlap et al. 2024) and caplure different aspects of visual discrimination beyond object positioning and negation (Krojer et al. 2022 Jhamtani & Berg-Kirkpatinn
2018 Park et al. 2019).
A manual filtering pass is performed to retain only those bags that would require captioning systems to capture some aspect of tie-grained visual discrimination. For bag size 3, this reduces the number of bags in TrucMatch from 680 to 254. Some examples of selected and discarded bags are shown in Figure 10
3.2 Experiment 1: Results on TrueMatch
Table 1: Recall@1 for several open-source captioning models, MLLMs, and SR-based methods on TrueMatch. The number of bags in #3 is 254, #5 104, and #7 is 93.
We evaluate several open-source captioning ap- proaches, MLLMs, and SR trained models on TrueMatch. Recall@1 (R&1) is reported in Table [ for bag sizes 3, 5, and 7.
TrueMatch
Method
Params
#3
#5
#7
A Approaches lack fino-grained details. Cap- tioning models, irrespective of their size, strug- gle to capture fine-grained visual details leading to poor performance TrueMatch.
Random chance
33.3
20.0
14.3
OFA Wang et al.
2022
180M
50.4
36.3
37.2
ClipCap (ookady et al.
2021
240M
50.8
36.5
33.8
CoCa Yu et 2022
640M
52.2
40.4
38.2
Doing more with less. Although billion- parameter MLLMs have achieved impressive results (Liu et al. 2023 Tong et al. 2024a), Ta- ble I demonstrates that they still struggle with fine- Sue-grained visual discrimination. In fact Dis- criTune (Dessi et al., 2023), that trains ClipCap with the vanilla SR setup matches InstructBLIP despite being two orders of magnitude smaller. Cambrian-1 is the best-performing open-source model and although it surpasses DiscriTune, our proposed approach outperforms by a significant. margin. This demonstrates the effectiveness of both: Trtc for evaluating captioning systems, and our approach of using SR to improve captioning (Section 4
PaliGemma-224 Beyer et al. PaliGemma-444 InstructBLIP (Dai et al. LLaVA 1.6
2022
3B
48.3
35.6
32.7
Beyer et al.
2022
3B
49.3
38.5
33.3
2024
13B
53.7
42.7
42.1
(Liu et al.)
2023
34B
57.9
46.9
47.9
Cambrian-1
Tong et al.
2024a
3B
58.3
48.8
52.8
Cambrian-1
Tong
2024a
8B
60.6
53.3
53.1
Discritune Ours (best)
Dessi et al.
2023
240M
53.3
42.3
38.4
240M
67.7
58.7
57.9
3.3 Experiment 2: Benchmarking Visual Caption Boosting with Relf-Retrieval
We evaluate the quality of captions adopted in VCB in Table Along with TrueMatch, we also adopt the SR evaluation strat- egy of Dessi et al. (2023) with 99 random distractors (RD100). On RD100, BlendCap outperforms original COCO captions by large margin of 8% on R@1 and 6.2% on ClipScore, confirming that human annotations capture complementary visual aspects of the same image. However, this sizeable gap shrinks to 2.5% on TrueMatch (bags of size #3). This con- firms that even though annotated captions blendod together work better, they inherently lack fine-grained details (see Fig- ure 2) necessary to perform well on TrueMatch. HolisticCap, on the other hand, yields remarkable performance gains over COCO and BlendCap across both setups: RD100 and all bag sizes of TrueMatch. This demonstrates the effectiveness of VCB in instilling fine-grained information into standard image captioning datasets.
Table 2: R@I scores for COCO and VCB captions evaluated on RD100 (100 random distractors) and TrueMatch.
RD100
True #5
Caption
R@1 ClipSe
#3
#7
COCO VisualCap
80.9
26.4
51.1
41.3
39.6
86.9
32.2
53.7
42.7
42.1
BlendCap
88.9
32.6
53.6
44.8
43.4
HolisticCap
91.3
33.4
57.5
48.1
49.1
We study the effectiveness of anchoring visual captions producing more succinct (sce Appendix A.3) and fine- fnn-grained captions. Table 2 shows that standalone VisualCap (generated using the MLLM) slightly worse than BlendCap. However, anchoring VisualCap in human annotations to create HolisticCap results

---

Published in Transactions on Machine Learning Research (01/2025)
in +4% improvement over BlendCap on TrueMatch. Additionally, by drawing comparisons to Table we observe that HolisticCap achieves performance par with larger (LLaaa 1.6) and newer (Cambrian-1-33 MLLMs compared to InstructBLIP.
Finding 1. Self-retrieval with TrucMatch evaluates models' ability to produce fine-grained captions and Visual Caption Boosting is a promising way to enrich captioning datascts with granular details.
In summary, SR within TrueMatch enables fine-grained evaluation of captioning systems, probing different. facets of visual discrimination. requires captions to be unique even for semantically similar images, thereby addressing the limitations of traditional captioning metrics.
4 Improving Captioning Systems with Relf-Retrieval as Training Objective
In this section, present key insights improving training with self-retrieval (SR), resulting in significant performance gains over previous works (Dessi et al. 2023 We begin by outlining the method (Section 4.1
and experimental setup (Section 4.2p. Our experiments reveal that captioners trained with SR are highly sensitive to their MLE initialization (Section 4.3). In fact, we discover a trade-off between caplion faithfulness and retrieval performance plaguing captioning systems fine-tuned with previous SR approaches (Dessi et al. 2023 Liu et al. 2018). To address this we: (1) initialize our model with more detailed captions from HolisticCap. (2) line-tune the visual encoder with SR (Section 4.5), and (3) mine hard training bags and
design curriculum over bag sizes (Section 4.6). Finally we show that training strategy is complementary
to CIDEr optimization (Rennie et al. 2017 resulting in further improvements (Section 4.8
4.1 Methodology
Model architecture. Similar to Dessi et al. 2023). adopt ClipCap (Mokady et al. 2021 a lightweight simplification of modern MLLMs (e.g. LLa VA, InstructBLIP, Cambrian). ClipCap connects pretrained visual encoder (CLIP [Radford et al., 2021]) to a pretrained language model (GPT-2 (Radford et al. 2019) through a simple MLP adapter. The adapter is tasked with mapping the rich visual embeddings from CLIP into a fixed number of prefiz tokens. These tokens capture essential visual information and guide the language model towards generating an image-conditioned caption.
Training this captioning system has two steps: (1) model protraining with maximum likelihood estimation (MLE), and (2) model fine-tuning by maximizing the SR reward with REINFORCE (Williams) 1992). We outline both training objectives, followed by experimental investigation of their properties and limitations.
Maximum Likelihood pretraining models the training data distribution. Specifically, captioning models are often teacher-forced to learn the word (token) distribution that maximizes the log-likelihood of the ground-truth captions given an input image. However, this results in strong language modeling priors resulting in generic captions that use a small vocabulary (see Section
Maximizing self-retrieval with Reinforce. A fine-tuning step that optimizes metric (e.g. CIDEr) is popular in training captioners (Rennie et al. 2017). We adopt Dessi et al. (2023)s 's contrastive formulation of SR that compares the generated caption against distractors, providing an optimal learning signal:
esim(e,))
Reward R(e, i, D) = log Σ ErEDU(D esim(e.)))
(1)
We use REINFORCE optimize the captioning system by estimating the gradient of the expected reward E RR(e, i, D log P(cli; since backpropagation is broken due to auto-regressive sampling. Further
discussion on using REINFORCE and variance reduction of the gradient estimates is in Appendices C.1 and C.2
4.2 Experiment Setup and Metrics
We present results for all our experiments on ClipCap. During MLE pretraining, the ClipCap MLP adapter is trained from scratch, CLIP is frozen, and GPT-2 is fine-tuned. When fine-tuning with REINFORCE, ClipCap's
7

---

Published in Transactions on Machine Learning Research (01/2025)
MLP adapter is trained normally while CLIP and GPT-2 are fine-tuned with LoRA adapters (Hu et al. 2021 with rank 32. The model is trained with the AdamW optimizer (Loshchilov & Hutter 2018) on single A6000 GPU. Further details about the hyperparameters are provided in Appendix C.3
We evaluate the quality of captions with NLG metrics: BLEU-4 and CIDEr, keeping the reference set consistent with the training distribution. On RD100, a setup with 99 random distractors (Dess) et al. 2023). we use R0I and ClipScore. On TrueMatch, we report R@1.
4.3 Experiment 3: Improving SR with Richer MLE Initialization
Following the findings of Section 3.3 we use TrueMatch to investigate the impact of different MLE initializations
SR fine-tuning in Table 3 We study SR. in the same setting previous work (Dessi et al.
2023
(1) we
only fine-tune the language model (SR-L), but using LoRA adapters; and (2) until Section
4.6
we use 99
random distractors to fine-tune with the SR reward.
MLE pretraining yields generic captions. Table B rows 1-3 report the performance of MLE training captioning datasets with increas- ing level of detail. We observe that improving the training captions re- sults in a significant. performance boost. Training with HolisticCap beats COCO by +8.2% on RD100 R@1 and +5.1% on TrueMatch #3. However, comparing performance against the ground-truth Holistic- Cap directly (from Table p we see
Table 3: Results for different training methods and captions show the effectiveness of improved initialization with HolisticCap and SR-L.
MLE Dataset
RD100
TrueMatch
Training CIDEr
R@I
ClipSc
#3
#5
#7
COCO
107.8
76.5
29.9
50.8
36.5
33.8
2 BlendCap
MLE
80.2
84.9
31.6
51.8
42.5
42.2
3
HolisticCap
26.6
84.7
31.7
55.9
43.7
42.2
COCO
108.2
83.7
30.2
53.3
42.3
38.4
BlendCap
SR-L
80.1
87.5
31.7
54.7
47.5
44.2
6 HolisticCap
27.8
87.6
31.9
59.1
47.7
46.5
that MLE training on HolistieCap yields -6.6% on RD100 R@1 and -1.6% on TrueMatch #3. This indicates
that even though HolisticCap improves performance, MLE training (i) generates generic descriptions that may not be aligned with the image content, and (ii) leads to object hallucinations (see Appendix C.7
Self-retrieval fine-tuning benefits from VCB. SR-L reduces MLE's preference towards statistically probable phrascs rosulting in more discriminant captions, reduced object hallucinations and consistent improvements in RD100 and TrueMatch across datasets (compare rows 1-4, 2-5, 3-6). Providing a richer initialization to SR-L through VCB enhances ine-grained visual discrimination. Row 6 outperforms DiscriTune (row 4) (Dessi et al. 2023) with +3.9% RD100 and +5.8% Truhh #3. This illustrates the importance of a good MLE initialization for SR and the effectiveness of VCB.
Longer reference sets hurt CIDEr. We see a significant. drop in CIDEr scores for BlendCap and HolisticCap compared to COCO in Table Upon further analysis, we believe that CIDEr rewards brevity. Appendix C.6 shows some valid short captions that unfortunately obtain scores close to 0.
Self-retrieval unlocks latent semantic information when fine-tuning the LLM. Although BlendCap does not capture any new information that is not present in one of the COCO annotations, being descriptive is sufficient to achieve superior retrieval performance against random distractors (BlendCap improves over COCO by +8% on RD100 R@1, Table D. Further, MLE pretraining is able to model these data distributions as the performance gap is maintained at +8.4% on RD100 R@1 between rows 1 and in Table 3 The model trained on COCO, due to the MLE objective, generates sparse captions that resemble the independent. annotations of COCO. Thus, even though the semantic information in the embedding spaces of both models (trained on COCO or BlendCap) are similar, their ability to access this information is bottlenecked by the MLE objective, leading to a large gap between COCO and BlendCap on RD100 in Table 3
Interestingly, SR-L fine-tuning narrows this gap dramatically from +8.4% to a mere +3.8% (rows 4, 5). This indicates that SR fine-tuning teaches the captioner to uncover semantic information within its embedding space, even when it was initially obscured by MLE. Furthermore, the mostly unchanged CIDEr scores hint that the captioner remains faithful to the GT captions for both COCO (rows 1, 4) and BlendCap (rows 2, 5).
8

---

Published in Transactions on Machine Learning Research (01/2025)
This is remarkable, it demonstrates that SR fine-tuning unlocks latent semantic information and steers the language model while preserving the initial data distribution.
Finding 2. Self-rctrioval steers the language model to generate discriminative captions by uncovering latent semantic information that was initially obscured by MLE.
4.4 Trade-off plaguing SR fine-tuning: Retrieval Performance vs. Caption Faithfulness
While SR fine-tuning removes language modeling priors from captioners by making them more discriminant, observe that they have tendency to become less faithful to the GT captions upon extended training. We
contrast these two lendencies of SR fine-tuning in Figure 3 comparing CIDEr and RD100 R@1 scores. We train for 100 epochs compared to the default 20 epochs used for all other experiments.
While SR. performance contimually improves (+7.9%) with exlended training, a corresponding degradation in CIDEr (-15.4%) is observed. This trade-off reveals that SR has the capacity to clicit better retrieval from the captioning sys- tem, even al the cost, of generaling fower-qualily coplions. We further investigate with qualitative examples in Ap- pendix C.& Dessi et al. (2023) attribute this deterioration to the model diverging from the GT distribution during SR-L. fine-tuning and argue that it is desirable. While this is partially true (see Figure 7 our investigation also reveals that captioners trained with vanilla SR-L, in a dash to enhance retrieval performance, build a propen- sity to hallucinate attributes (see Figure 8 resulting in poor CIDEr scores. This tendency is only exacerbated with extended training resulting in further degradation of NLG metrics and increased object hallucinations (see Appendix C.7). These findings reveal an important bot- tleneck in SR fine-tuning
Tradeoff between RD100 and CIDEr scores
RD100 R@
105
CIDEr
100
80
20
100
Epoch
Figure 3: RD100 R@1 continually increases while
CIDEr degrades when fine-tuning ClipCap with SR-L on COCO for 100 epochs.
Finding 3. Captioners fine-tuned with SR-L suffer from a tradc-off between retrieval performance and faithfulness, often hallucinating details and deviating from GT captions with extended training.
Finally, the continued improvement in RD100 R&1 over 100 epochs indicates that vanilla. SR-L fine-tuning fails to saturate SR's capacity to instill fine-grained visual discrimination in captioning systems. This motivates us to explore better SR fine-tuning strategies while mitigating their propensity to hallucinate. Since a higher learning rate destabilizes the policy gradients, we turn to: (1) fine-tuning the visual encoder (CLIP) with SR; and (2) adopting curriculum over bags of hard negatives.
4.5 Experiment 4: Fine-tuning CLIP with Reff-Retieeval
We present comprehensive analysis of fine-tuning language and vision modules of the captioning system with LoRA in Table 4 Fine-tuning CLIP (SR-V) makes the captions discriminative. For example, with BlendCap MLE pretraining, SR-V fine-tuning results in +2.6% RD100 R@1 and +6.6% on TrueMatch #3 (row 4, 5). However this comes at the cost of deteriorating NLG metrics (-7.7% CIDEr), suggesting that SR-V is unable to preserve the faithfulness of generated captions. We further verify that even after extended SR-L fine-tuning to match SR-V's retrieval scores, SR-L has a higher CIDEr (see Appendix C.4p. This suggests that CIDEr deteriorates due CLIP fine-tuning and not due NLG metrics penalizing discriminative captions (Wang et al. 2020). This is notable bottleneck fine-tuning CLIP with SR: while it enables superior retrieval performance, it makes the captioner less faithful to the GT captions.
9

---

Published in Transactions on Machine Learning Research (01/2025)
Further qualitative analysis of the captions generated by SR-L and SR-V in Figure g reveal that captioners fine-tuned with SR-V struggle to bind attributes to the correct objects. For instance, while SR-V captures the red color of the forklift, it binds it with the incorrect object (train). This is likely due to two reasons: (1) the retrieval objective treats the caption as a bag of words and does not promote compositionaliy; and
(2) CLIP has been shown to struggle with attribute binding (Lewis el al. 2022 Hsieh et al. 2024).
RD100 is suboptimal for training fine-grained caption- ers. On TrueMatch, while SR- V fine-tuning provides substan- tial improvements over SR-L for COCO and BlendCap (rows 1-2, 4-5), we observe relatively smaller gains for HolisticCap (rows 7-8). This indicates that the vanilla SR- L objective of retrieving against 99 random distractors (RD100) is
Table 4: Ablation for fine-tuning different modules with SR: GPT-2
(SR-L), CLIP (SR-V), both (SR-LV). Row 1 is Dessi et al. 2023
Bold
indicates best, italics is second. (x) presents improvement against SR-L.
MLE Dataset
NLG
RD100
TrueMatch #5
Training
C
R@1
#3
#7
COCO
SR-L
108.2
83.7
53.3
42.3
38.4
COCO
SR-V
97.2
86.1
(2.4)
59.3
(6.0)
49.2
(6.9)
45.0 (6.6)
3
COCO
SR-LV
96.6
87.4
(3.7)
59.6
(6.3)
51.2
(8.9)
47.2 (8.8)
BlendCap
SR-L
80.1
87.5
54.7
47.5
44.2
BlendCap
SR-V
72.4
90.1
(2.6)
61.3 (6.6)
52.9
(5.4)
51.0 (6.8)
not challenging enough to provide a good learning signal for models pretrained on fine-gres fine-grained Holis- ticCap. While SR-LV does better than SR-V on HolisticCap (rows 8-9), the gap between BlendCap
BlendCap
SR-IV
72.7
90.7
(3.2)
61.3 (6.6)
52.3
(4.8)
50.1 (5.9)
HolisticCap
SR-L
27.8
87.6
59.1
47.7
46.5
HolistieCap
SR-V
25.6
90.2 (2.6)
61.9
(2.8)
52.5
(1.8)
51.5 (5.0)
HolisticCap
SR-IV
26.5
90.7 (3.1)
62.6
(3.5)
52.7
(5.0)
51.2 (1.7)
(row 6) and HolisticCap (row 9) still remains small. To address this, we devise a curriculum with hard negatives, discussed in the next section, to increase the difficulty of SR fine-tuning objective.
Finding 4. Fine-tuning the visual module with self-retrieval instills fine-grained visual details in the captioning system, while increasing hallucinations and struggling with attribute binding.
Among methods in Table 4 SR-LV demonstrates the best retrieval performance while also preserving CIDEr slightly better than SR-V. Thus, we adopt SR-LV for subscquent experiments with training curriculum.
4.6 Experiment 5: Self- Reff-Retieeva Fine-tuning with Bags and Curriculum
The above SR experiments were trained using 99 random distractors (Dessi et al. 2023). In- stead, in this experiment, we fine-tune with bags of highly similar images within a train- ing minibatch (see Appendix B.2). Retrieving against multiple hard negatives flattens the soft- max distribution resulting in a stronger learning signal. We also propose curriculum over bag sizes (BagCurri) to more optimally leverage the contrastive reward (sce Appendix B.3) SR fine-tuning with BagCurri instills fine-grained visual details the model, essential for dis- criminating highly similar images.
Table 5: Ablation of fine-tuning with the bag curriculum (BagCurri) as compared against vanilla SR (SR-L). Row is Dessi et al. (002))
MLE Dataset
Bag
NLG
RD100
TrueMatch
Training Curri
CIDEr
R@1
#3
#7 #5
SR-L SR-L
108.2
83.7
53.3
42.3
38.4
2 COCO
103.4
84.5
53.7
46.2
41.5
SR-V
85.8
87.4
63.1
51.9
49.3
SR-LV
V
84.2
85.8
67.2
56.3
53.6
SR-L
27.8
87.6
59.1
47.7
46.5
6
Holistic-
SR-L
31.3
88.5
58.4
50.0
49.5
Cap
SR-V
25.8
90.6
64.0
54.6
54.1
8
SR-LV
29.7
91.3
65.2
57.1
57.1
SR-V versus SR-L with our bag curricu- lum. We evaluate the impact of fine-tuning
different components of the captioning system (CLIP and GPT-2) with BagCurri compared to vanilla SR-L in Table 5 We find that SR-V fine-tuning with our curriculum forces CLIP to learn fine-grained visual features resulting in a substantial +5% to +11% improvement over vanilla SR-L (rows us. 1, 7 us. 5) on TrueMatch. Interestingly, SR-V with BagCurri renders COCO (row 3) comparable to HolisticCap (row 7) TrueMatch #3. However, these disproportionate gains for COCO come at the expense of CIDEr
10

---

Published in Transactions on Machine Learning Research (01/2025)
scores, which plummet -22.4% compared to SR-L. Further qualitative analysis of the generated captions shows that SR-V fine-tuning with BagCurri amplifies the trends observed in Section 4.5 while it boosts retrieval performance, it exacerbates the inability of the captioner to bind attributes to the correct objects and increases hallucinations (see Figure Dp. In contrast, BagCurri may be too challenging for SR-L fine-tuning, as evident by the small improvements of +0% to +-4% over vanilla SR-L (rows u8. 1, us. 5) on RD100 or TrucMatch. Nevertheless, similar to the findings of Scction 4.3 we find that fine-tuning SR-L with BagCurri benefits heavily from the rich initialization provided by VCB. While the CIDEr score drops for COCO when using SR-L with BagCurri (-4.8% row 2 us. 1), for HolisticCap we observe improvement (+3.5% row 6 08. 5).
Best of both worlds: SR-LV with BagCurri. The above findings indicate that BagCurri has the capacity to enhance both: (1) visual discrimination ability of the captioner with SR-V; and (2) adherence to GT captions with SR-L when provided with rich initialization. Motivated by this, we initialize the captioner with HolisticCap and fine-tune both the language and visual components (SR-LV) with BagCurri (row 8), retaining the best of both worlds. DR-IV-BBaCurri further enhances E'sV's already impressive performance TrueMatch (achieving +6-10% over vanilla SR-L) while also improving the CIDEr score over vanilla SR-L by +1.9% (rows 5, 8). This is momentous, as without modifying the model's architecture or reward function, we outperform vanilla SR-L (Dessi et al., 2023) by an average +15% across all bag sizes on TrueMatch (row 1, 8), while circumventing the caption faithfulness trade-off plaguing current SR approaches.
Importance of the curriculum. In Appendix B.3 we investigate the role our curriculum plays in overcoming the retrieval and caption quality trade-off compared to directly using bags as hard negatives. Table 10 reveals that BagCurri is responsible for preserving caption faithfulness, as indicated by decrease in CIDEr when bags are used without curriculum.
Finding 5. Providing rich initialization with VCB and SR-IV fine-tuning with BagCurri instills fine-grained visual details in the captioner while preserving its faithfulness to GT captions.
4.7 Generating Diverse Captions
Table 6: Number of words with frequency >= 5 on the COCO test set for captioners trained with differ- ent optimization strategics, modules, and datascts.
We evaluate how VCB and SR increase caption diver- sity by measuring the number of words that appear 2 5 times on the COCO test set. In Table b we observe clear increase in the number of unique words as we progress from COCO to BlendCap to HolistieCap, across all optimization and training strategies. Furthermore, within each individual dataset, there is a substantial increase in the usage of diverse vocabulary with transi-
Self Retrieval
BBagCuri SR-LV
Dataset
MLE
SR-L
SR-V
SR-LV
COCO BlendCap
428
689
719
762
770
1220
1306
1345
1366
1435
HolisticCap
1431
1535
1606
1628
1732
tions from MLE to SR-L, SR-V, and SR-IV. Notably, SR-LV with BagCurri yields siguificant improvementts over MLF: relative +80 on COCO, +17.6% on BlendCap, and +21% on HolisticCap. This demonst rates the effectiveness of VCB and our curriculum in guiding the captioning system away from the language modelling priors and improving caption diversity.
4.8 Experiment 6: Combining Self- Retrieval and CIDEr Optimization
CIDEr optimization with REINFORCE is commonly used to improve captioning system's adherence to the GT captions (Rennie et al. 2017.. As the final experiment, we present results of combining with SR in Table 7
CIDEr optimization (rows 3, 8) improves faithfulness to GT captions compared to MLE pretraining (rows 1, 6): CIDEr improves +10.7% on COCO and +29.1% on HolisticCap. Importantly, CIDEr optimization on top of HolisticCap MLE pretraining provides +4.8% on TrueMatch #7 (rows 6, 8), while we see only a smaller +2.5% with COCO captions. Even without SR fine-tuning, the larger improvement with HolisticCap emphasizes the importance of MLE initialization with captions containing fine-grained visual details (VCB). Furthermore, compared to vanilla SR-L (rows 2, 7), CIDEr opt. (rows 3, 8) fairs worse on both RD100 and Truc This presents an opportunity to optimize both rewards simultancously (Luo et al.) 2018
11

---

Published in Transactions on Machine Learning Research (01/2025)
CIDEr meets SR. Building upon the find- ings of previous sections, we push the bound-
Table 7: Impact of combining SR with CIDEr optimization.
Results are presented for various combinations of RL rewards. C: CIDEr, SR: Self- Retrieval, and BC: SR with BagCurri. Row
aries of SR by explicitly adding CIDEr to our discriminative reward = SR + A CIDEr. We conduct an ablation over various values (see Appendix C,5), and select 0.5 for our experiments.
Mokady et al.
(2021). row 2 is Dessi al. 2023 and row
is Rennie et al. (2017 with ClipCap.
ML. Dataset
Reward
NTG CIDEr
RD100
TrueMatch
SR
BC
R@1
#3
#5
#1
With HolisticCap, we that joint opti- mization (row 10) outperforms the most dis- criminant SR model (row 9) achieving im- provements on metrics: +8.3% on CIDEr, +1.3% on RD100 R&1, and +0.8% to +2.5% on TrucMatch. Conversely, on COCO cap- tions (rows 4, 5): we observe that adding CIDEr optimization adversely affects SR met- rics on TrueMatch with -5.1% to -7.6% per- formance drops. This aligns with previous findings and underscores the importance of initialization during SR fine-tuning.
107.8
76.5
50.8
36.5
33.8
SR-L
108.2
83.7
53.3
42.3
38.1
COCO
118.5
79.1
50.7
37.3
36.3
4
SR-LV v
84.2
85.8
67.2
56.3
53.6
SR-LV v 92.8
89.0
59.6
51.2
47.2
26.6
84.7
55.9
43.7
42.2
7
SR-T.
27.8
87.6
59.1
47.7
46.5
Holistic-
55.7
86.9
57.6
47.1
47.0
Cap
SR-LV
29.7
91.3
65.2
57.1
57.1
10
/ SR-IX V
38.0
92.6
67.7
58.7
57.9
Table 7 row 10 is our best model that also achieves highest performance on TrueMatch as compared several MLLLMs that are orders of magnitude larger and trained more data (see Table
Table 8: Zero-shot evaluation
5 Generalization Experiments
the ImageCoDe validation set (R@1). Rows 2 and 3 are from Dessi
In this section, we evaluate the generalization of our approach Image- CoDe (Krojer et al. 2022), benchmark for text-based image retrieval where image sets are created with 10 sequential video frames. Different from TrueMatch, ImageCoDe focuses on fine-grainedness found across video frames such as negation, occlusion, etc. We follow the same ex- perimental setup adopted by Dessi et al. (2023). Table S shows that SR fine-tuning of both the vision and text modules with our bag curriculum (SR-LV BagCurri) achieves state-of-the-an results. In fact, it outper- forms Dessi et al. (2023) 's work using SR-L after MLE pretraining COCO (row 2, +.6%) or Conceptual Captions, a 30x larger dataset (row 3, +1.7%).
ct al. 2023
Dataset
Method
R@l
COCO COCO
MLE
28.8
SR-L
30.3
ConCap
SR-L
36.2
4
MLE
31.9
5
HolisticCap
SR-L
33.0
SR-IV
37.9
BagCurri
6 Conclusion
We investigated self-retrieval (SR) as way to comprehensively improve current captioning systems on all three fronts: (1) data, (2) evaluation, and (3) training. (1) We proposed Visual Caption Boosting (VCB), a model agnostic framework to instill fine- ane-grained visual information into generic image captioning datasets while remaining anchored in human annotations.
(2) Furthermore, we created TrucMatch, a benchmark comprised of highly similar image bags. SR evaluation with TrueMatch enabled fine-grained evaluation of captioning systems and unlike traditional captioning metrics encouraged diversity in generated captions. (3) We adopted SR fine-tuning and showed that the rich MLE initialization provided by VCB is important. We uncovered tradc-off between caption faithfulness and retricval performance plaguing current SR approaches and designed a training recipe to address it. Specifically, we employed a curriculum over the number of hard negatives to more optimally leverage the SR reward, and fine-tune both the visual and language components of the captioning system. We demonstrated that this training recipe and VCB circumvents the aforementioned trade-off. With the same model architecture, our approach achieved significant performance improvements vanilla. SR (Dess) et al. 2023) across a dataset with 99 random distractors, and set the state-of-the-art.
12

---

Published in Transactions on Machine Learning Research (01/2025)
ImageCoDe Krrojer al. 2022 On TrueMatch, approach outperformed powerful MLLMs that are 1-2 orders of magnitude larger and trained on large datasets, while improving over vanilla SR by up to +15%.
7 Limitations and Future Scope
Self-rctrioval (SR) is heavily reliant on a scorer to identify and retrieve the image that best matches the generated text. We employ CLIP as scorer, which has been shown to struggle with capturing extremely fine tne-grained and compositional visual details. However, it is fair to expect that the reliability of SR evaluation should improve steadily over time with better vision-language models.
Additionally, as Visual Caption Boosting uses LLMs and MLLMs for generating fine-grained descriptions, it is prone to hallucinations. However, as shown, our anchoring strategy mitigates this to a large extent.
As future work, exploring further applications of the SR fine-tuning presents a promising research avenue. MLE initialization with improved captions also a recurring theme and played an important role in our work this too deserves further investigation while training ML.LMs.
Acknowledgments
This project was supported in part by funding from SERB SRG/2023 0223/002544 and an Adobe Research gift. We thank Shivanshu Sharma for partial assistance with compute. We thank Takshmipathi Balaji, Haran SK Raajesh, Naren Akash RJ, and Vansh Agarwal for assistance with the human study.
References
Peter Anderson, Basura Fernando, Mark Johnson, and Stephen Gould. SPICE: Semantic Propositional Image Caption Evaluation. In European Conference on Computer Vision (ECCV), 2016.
Abhipsa Basu, R Venkatesh Babu, and Danish Pruthi. Inspecting the Geographical Representativenes iveness of
Images from Text-to-lmage Models. In International Conference on Computer Vision (ICCV), 2023.
Rohan Bavishi, Erich Elsen, Curtis Hawthorne, Maxwell Nye, Augustus Odena, Arushi Somani, and Sagnak Taşrlar. Introducing our multimodal models, 2023. URL htttss//www.ddee..aa///gggyyyyy
Lucas Beyer, Xiaohua Zhai, and Alexander Kolesnikov. Big Vision. httts:///gthubb.commooogleeerree big_vision 2022.
Jacmin Cho, Scunghyun Yoon, Ajinkya Kale, Franck Dernoncourt, Trung Bui, and Mohit Bansal. Fine- grained Image Captioning with CLIP Reward. In Findings of North American Chapter of Association of Computational Linguisties (NAACL), 2022.
Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale N Fung, and Steven Hoi. ILI:: Towards Gencral-purpose Vision-Language Models with Instruction Tuning. In Advances in Neural Information Processing Systems (NeurIPS), 2024.
Abhishek Das, Satwik Kottur, José MF Moura, Stefan Lee, and Dhruv Batra. Learning Cooperative Visual Dialog Agents with Deep Reinforcement Learning. In International Conference on Computer Vision (ICCV), 2017.
Karan Desai, Gaurav Kaul, Zubin Trivadi Aysola, and Justin Johnson. RedCaps: Web-curated image-text data created by the people, for the people. In Advances in Neural Information Processing Systems (NewrIPS), 2021.
Roberto Dessi, Michele Bevilacqua, Elconora Gualdoni, Nathanaäl Carraz Rakotonirina, Francesca Franzon, and Marco Baroni. Cross-Domaa Image Captioning With Discriminative Finetuning. In Conference on Computer Vision and Patlern Recognition (CVPR), 2023.
13

---

Published in Transactions on Machine Learning Research (01/2025)
Sivan Doveh, Assaf Arbelle, Sivan Harary, Roei Herzig, Donghyun Kim, Paola Cascante-Bonile Amit Alfassy, Rameswar Panda, Raja Giryes, Rogerio Feris, et al. Dense and Aligned Captions (DAC) Promote Compositional Reasoning in VL Models. In Advances in Neural Information Processing Systems (NeurIPS), 2024.
Lisa Dunlap, Yuhui Zhang, Xiaohan Wang, Ruiqi Zhong, Trevor Darrell, Jacob Steinhardt, Joseph E Gonzalez, and Serena Yeung-Levy Describing Differences in Image Sets with Natural Language. In Conference on Computer Vision and Pallern Recognition (CVPR), 2024.
Fartash Faghri, David J Fleet, Jamie Ryan Kiros, and Sanja Fidler. +SE++: Improving -isaal-Smmatti Embeddings with Hard Negatives. In Brilish Machine Vision Conference (BMVC), 2017.
Alessandro Favero, Luca Zancato, Matthew Trager, Siddharth Choudhary, Pramuditha Perera, Alessandro Achille, Ashwin Swaminathan, and Stefano Soatto. Multi-modal Hallucination Control by Visual Information Grounding. In Conference on Computer Vision and Pattern Recognition (CVPR), 2024.
Adam Fisch, Kenton Lee, Ming-Wor Chang, Jonathan H Clark, and Regina Barzilay. CapWAP: Image Captioning with Purpose. In Empirical Methods in Natural Language Processing (EMNLP), 2020.
Mclissa Hall, Laura Gustafson, Aaron Adcock, Ishan Misra, and Candace Ross. Vision-Languaae Models Performing Zero-Shot Tasks Exhibit Gender-based Disparities. In Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi. CLIPScore: A Relerence-free Evaluation Metric for Image Captioning. In Empirical Methods in Natural Language Processing (EMNLP), 2021.
Cheng- Yu Hsieh, Jieyu Zhang, Zixian Ma, Aniruddha Kembhavi, and Ranjay Krishna, SugarCrepe: Fixing Hackable Benchmarks for Vision-Language Compositionality. In Advances in Neural Information Processing Systems (NewrIPS), 2024.
Edward J Hu, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. LoRA: Low-Rank Adaptation of Large Language Models, In International Conference on Learning Representations (ICLR), 2021.
Harsh Jhamtani and Taylor Images. In Empirical Methods in Natural Longuage Processing (EMNLP), 2018.
Learning to Describe Differences Between Pairs of Similar
Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7B. Xixiu preprint ar Xix:23310 2.0223100000 06825, 2023.
Qirui Jiao, Daoyuan Chen, Yihum Huang, Yaliang Li, and Ying Shen. img-Diff Contrastive Data Synthesis for Multimodal Large Language Models. arXic preprint or orXin:2400 04594, 2024.
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling Laws for Neural Language Models. arXiv preprint "Xix:2001008366 2020.
Andrej Karpathy and Li Fei-Fei. Deep Visual-Semantic Alignments for Generating Image Descriptions. In Conference on Computer Vision and Pottern Recognition (CVPR), 2015.
Simon Kornblith, Lala Li, Zirui Wang, and Thao Nguyen. Guiding Image Captioning Models Toward More Specifie Captions. In International Conference on Computer Vision (ICCV), 2023.
Elisa Kreiss, Fei Fang, Noah Goodman, and Christopher Potts. Concadia: Towards Inago-Based Text Generation with a Purpose. In Empirical Methods in Natural Language Processing (EMNLP), 2022.
Benno Krojer, Vaibhav Adlakha, Vibhav Vincet, Yash Goyal, Edoardo Ponti, and Siva Reddy. Image Retrieval from Contextual Descriptions. In Association of Compulational Lánguistics (ACL), 2022.
14

---

Published in Transactions on Machine Learning Research (01/2025)
Zhengfeng Lai, Haotian Zhang, Wentao Wu, Haoping Bai, Alcksei Timofeev, Xianzhi Du, Zhe Gan, Jiulong Shan, Chen-Nee Chuah, Yinfei Yang, et al. VeCLIP: Improving CLIP Training via Visual-enriched Captions. arXiv preprint arXic:2310.07633 2023.
Martha Lewis, Nihal V Nayak, Peilin Yu, Qinan Yu, Jack Mcrullo, Stephen H Bach, and Ellic Pavlick. Does CLIP Bind Concepts? Probing Compositionality Large Image Models. In European Chapler of the Association for Computational Linguistics (EACL), 2022.
Xianhang Li, Haoqin Tu, Mude Hui, Zeyu Wang, Bingchen Zhao, Junfei Xiao, Sucheng Ren, Jieru Mei, Qing Liu, Huangjie Zheng, et al. What If We Recaption Billions of Web Images with LLaMA-3? arviv preprint arXiv:2406.00 2400204468888 2024.
Tsung-Y Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft COCO: Common Objects in Context. In European Conference on Computer Vision (ECCV), 2014.
Hanchao Liu, Wenyuan Xue, Yifei Chen, Dapeng Chen, Xiutian Zhao, Ke Wang, Liping Hou, Rongjun Li, and Wei Peng. Survey on Hallucination in Large Vision-Tanguage Models. Xiv preprint or (0i0:2402.00023, 2024.
Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jac Lee. Improved Baselines with Visual Instruction Tuning. In Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
Xihui Liu, Hongsheng Li, Jing Shao, Dapeng Chen, and Xiaogang Wang. Show, Tell and Discriminate: Image Captioning by Self-retrieval with Partially Labeled Data. In European Conference on Computer Vision (ECCV), 2018.
Πya Loshehilov and Frank Hutter. Decoupled Weight Decay Regularization. In International Conference Learning Representations (ICLR), 2018.
Ruotian Luo, Brian Price, Scout Cohen, and Gregory Shakhnarovich. Discriminability Objective for Training Descriptive Captions. In Conference on Computer Vision and Pattern Recognition (CVPR), 2018.
Ron Mokady, Amir Hertz, and Amit H Bermano. ClipCap: CLIP Prefix for Image Captioning. arXiv preprint arXiv:2111.09734 2021.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. BLEU: A Method for Automatic Evaluation of Machine Translation. In Association of Compulational Linguistics (ACL), 2002.
Dong Huk Park, Trevor Darrell, and Anna Rohrbach. Robust Change Captioning. In International Conference on Computer Vision (ICCV), 2019.
André Susano Pinto, Alexander Kolesnikov, Yuge Shi, Lucas Beyer, and Xiaohua Zhai. Tuning Computer Vision Models with Task Rewards. In International Conference on Machine Learning (ICML), 2023.
Bryan A Plummer, Liwei Wang, Chris M Cervantes, Juan Caicedo, Julia Hockenmaier, and Svetlana Lazebník. Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Sentence Models. In International Conference Computer Vision (ICCV), 2015.
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and llya Sutskever. Language Models are Unsupervised Multitask Learners. 2019.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning Transferable Visual Models From Natural Language Supervision. In International Conference on Machine Learning (ICML), 2021.
Steven J Rennie, Etienne Marcheret, Youssef Mrouch, Jerret Ross, and Vaibhava Goel. Self Caff-Critiaal Sequence Training for Image Captioning. In Conference on Computer Vision and Pattern Recognition (CVPR), 2017.
15

---

Published in Transactions on Machine Learning Research (01/2025)
Anna Rohrbach, Lisa Anne Hendricks, Kaylee Burns, Trevor Darrell, and Kate Sacnko. Object Hallucination in Image Captioning. In Empériaal Methods in Natural Language Processing (EMNLP), 2018.
Hadi Salman, Saachi Jain, Andrew Nyas, Logan Engstrom, Eric Wong, and Aleksander Madry. When Does Bias Transfer in Transfer Learning? arXiv preprint XrXi::2007. 22002227.0084,, 2022.
Gabriel Oliveira dos Santos, Esther Luna Colombini, and Sandra Avila. CIDEr-R: Robust Consensus- based
Image Description Evaluation. 2021.
Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. LAION-5B: An Open Large-scale Dataset for Training Next Generation Image-Text Models. In Advances in Newral Information Processing Systems (NewrIPS), volume 35, 2022.
Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. Conceptual Captions: A Cleaned, Hypernymed, Image Alt-text Dataset for Automatic Image Captioning. In Association of Computational Linguistics (ACL), 2018.
Nina Shvetsova, Anna Kukleva, Xudong Hong, Christian Rupprecht, Bernt Schicle, and Hilde Kuehne. How CoCCototi Prompting L.MS to Transform Video Annotations at. Scale. In European Conference on Computer Vision (ECCV), 2024.
Vasu Singla, Kaiyu Yue, Sukriti Paul, Reza Shirkavand, Mayuka Jayawardhana, Alireza Ganjdanesh, Heng Huang, Abhinav Bhatele, Gowthami Somepalli, and Tom Goldstein. From Pixels to Prose: A Large Dataset. of Dense Image Captions. arXiv preprint arXiv:2406. 10328, 2024.
Kirill Sirotkin, Pablo Carballeira, and Marcos Escudero- Viñolo. A Study the Distribution of Social Biases in Self-Supervised Learning Visual Models. In Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
Mattco Stefanini, Marcella Cornia, Lorenzo Baraldi, Silvia Cascianelli, Giuseppe Fiameni, and Rita Cucchiara. From Show to Tell: A Survey on Deep Learning- Based Image Captioning. IEEE Transactions on Paltern Analysis and Machine Intelligence (PAMI), 45(1):539 559, 2022.
Bart Thomee, David A Shamma, Gerald Friedland, Benjamin Elizalde, Karl Ni, Douglas Poland, Damian Borth, and Li-Jia Li. YFCC100M: The New Data in Multimedia Research. Communications of the ACM, 59(2):64-73, 2016.
Shengbang Tong, Ellis Brown, Penghao Wu, Sanghyun Woo, Manoj Middepogu, Sai Charitha Akula, Jihan Yang, Shusheng Yang, Adithya lyer, Xichen Pan, et al. Cambrian-l: A Fully Open, Vision-Centric Exploration of Multimodal LLMs. arXiv preprint 2xx::2406.16861 2024a.
Shengbang Tong, Zhuang Liu, Yuexiang Zhai, Yi Ma. Yann LeCun, and Saining Xie. Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMs. In Conference Computer Vision and Pattern Recognition (CVPR), 2024b.
Jack Urbanck, Florian Bordes, Pictro Astolfi, Mary Williamson, Vasu Sharma, and Adriana Romero-Soriano. A Picture is Worth More Than 77 Text Tokens: Evaluating CLIP-Style Models on Dense Captions. In Conference on Computer Vision and Pattern Recognition (CVPR), 2024.
Ramakrishna Vedantam, C Lawrence Zätnick, and Devi Parikh. CIDEr: Consensus- Image Description Evaluation. In Conference on Computer Vision and Pattern Recognition (CVPR), 2015.
Peng Wang, An Yang. Rai Men, Junyang Lin, Shuai Bai, Zhikang Li, Jianxin Ma, Chang Zhou, Jingren Zhou, and Hongxia Yang. OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to- Sequence Learning Framework. In International Conference on Machine Learning (ICML), 2022.
Zeyu Wang, Berthy Feng, Karthik Narasimhan, and Olga Russakovsky. Towards Unique and Informative Captioning of Images. In Furopean Conference on Computer Vision (ECCV), 2020.
16

---

Published in Transactions on Machine Learning Research (01/2025)
Ronald J Williams. Simple Statistical Gradient-Following Algorithms for Connectionist Reinforcement Learning. Machine learning, 256, 1992.
Andre Ye, Sebastin Santy, Jena D Hwang, Amy Zhang, and Ranjay Krishna. Computer Vision Datasets and Models Exhibit Cultural and Linguistic Diversity in Perception. arXiv preprint SSK3:2310.143300 2023.
Jiahui Yu, Zirui Wang, Vijay Vasudevan, Legg Yeung, Mojtaba Seyedhosseini, and Yonghui Wu. CoCa: Contrastive Captioners are Mmago-Text Foundation Models. Transactions on Machine Learning Research (TMLR), 2022.
17

---

Published in Transactions on Machine Learning Research (01/2025)
A Visual Caption Boosting
A.1 Prompting Strategy
In this section, we discuss prompting strategies used to produce BlendCap, the visual caption, and HolisticCap. We provide the system prompt along with the in-context examples we utilize. In order to generate captions that holistically describe the image, we use the following two step process:
(a) BlendCap. First, prompt Mistral-7B (Jiang al., 2023) to reduce redundancy, preserve all factual information obtained from human annotated captions, and not introduce any new information other than what is mentioned in the reference set. This ensures that the blended caption remains factual and anchored to the human annotations:
System prompt You are given a reference set of 5 image captions. Create a holistic caption to describe the same image by incorporating relevant information from all the reference captions. Ensure that the generated caption minimizes redundancy by excluding overlapping information found in the reference set. Do not include additional information and adjectives other than what is given in reference set. Stop after the first period and do not ask any additional questions. Just complete the text. Three examples are shown below
In context examples
User Reference caption 1: A plate full of food with an assortment of food on it. Reference caption 2: There is meat and vegetables on a white and brown plate. Reference caption
plate on a table that hasfood it.
Reference caption 4: A plate with some steak, carrots, and sliced fried potatoes.
Reference caption 5: A plate of food on a table.
Assistant A white and brown plate on table with an assortment of steak, carrots, sliced fried potatoes, and vegetables.
User Reference caption
park bench on the side of lake.
Reference caption 2: lone bench sits atop hill looking over the river.
Reference caption 3: wooden bench sitting on top of sandy beach.
Reference caption 4: A bench on a river bank in the countryside.
Reference caption 5: A wood bench is sitting in front of river. Assistant wooden bench sits atop a hill in front of river overlooking the water and the surrounding countryside.
User Reference caption
bathroom with toilet sitting next sink.
Reference caption 2: white sink and toilet in room.
Reference caption 3: A bathroom with mirror, sinks, toilet and toilet roll.
Reference caption 4: A bathroom that has toilet sink and mirror in it.
Reference caption 5: A bathroom with a toilet next to sink.
Assistant A bathroom with toilet next to a white sink, mirror and toilet roll.
(b) Visual Caption. We concurrently use InstructBLIP (Dai al. 2024 produce dense visual descriptions for the given image. We find that the following simple prompt performs well.
System prompt Briefly describe the content of the image.
(c) HolisticCap We then prompt Mistral-7B to expand the blended caption with the visual description generated by the MLLM. Notably, the LLM prompted to preserve human annotated information when there is conflict of information between BlendCap and Visual Caption. Only "few-shot" examples are shown here for brevity.
18

---

Published in Transactions on Machine Learning Research (01/2025)
System prompt : In this task, your objective is to improve "Correct caption" by incorporating any additional information present in "New caption".. If there are conflicting details between the two captions regarding the objects or their attributes, strictly use information from "Correct caption". "New caption" can have incorrect information sometimes but "Correct caption" will always be correct. The focus is on transferring missing information, including adjectives and nouns, from "New caption" to "Correct caption". Refrain from seeking further clarification and explanations; simply generate the caption. Three examples are shown below
context examples
User Correct caption: A reaches up with tennis racquet to hit approaching ball in tennis court.
New noisy caption: In this image, a woman is playing tennis on a purple court. She is wearing white shirt and a blue short, and she is holding a tennis racket. The tennis ball is in the air as she prepares to hit it with her racket.
Assistant man wearing a white shirt and blue shorts reaches up with tennis racquet hit approaching ball in a purple tennis court.
User Correct caption: A white fire hydrant sits in front of an old couch on a sidewalk in front of a house.
New noisy caption: In this image, there is a black fire hydrant sitting on the sidewalk front of a brick building. The fire hydrant is positioned next to a couch, which is placed on the sidewalk front of the building.
Assistant white fire hydrant sits in front of old couch sidewalk in front of brick house.
User Correct caption: A young boy poses next to a wall with writing on it, smiling and holding bags.
New noisy caption: In this image, a young girl is sitting in front of a graffiti-covered wall, wearing a red shirt. She is holding a box crayons and smiling at the camera. Assistant young boy wearing red shirt poses next to graffiti-covered wall with writing it. He is holding bags of crayons and smiling at the camera.
A.2 Human Study: Quantifying Hallucinations VCB
We conduct human study to investigate the exlent of hallucinations in BlendCap and HolisticCap. We randomly sample 60 images from MSCOCO and for each image-caption pair, human is asked to count the number of ballucinations present in the caption. Three types of hallucinations are considered: (1) objects, (2) attributes, and (3) egregious statements, i.e., a glaringly incorrect statement that unabbiguousy false.
BlendCap contains only object and attribute hallucinations. Out of the hallucinations, are due incorrect COCO human annotations and only (3.3%) are induced by the LLM. This demonstrates the reliability of BlendCap in combining human annotations into denser caption.
HolisticCap contains only 9 instances of hallucinations (4 object and 5 attribute) across 7 captions, compared to VisualCap's 16 instances (across 12 captions). Additionally, VisualCap contained 3 egregious statements that were removed in HolisticCap. Furthermore, 7/60 captions in HolisticCap contain hallucinations, reflecting a 41.6% reduction compared VisualCap's 12/60 captions. This demonstrates VCB's effectiveness anchoring the MLLM descriptions (Visual Captions) into human annotations (BlendCap) to ereate HolisticCap.
A.3 Token Statistics for Captions
We present some caption statistics for Visual Caption Boosting compared original COCO captions in Table g. While Visual Captions longer than HolisticCap, they may contain erroneous details and
19

---

Published in Transactions on Machine Learning Research (01/2025)
verbose language modelling priors shown in Figure 2 Both BlendCap and HolisticCap generally more shows the histogram of the number of tokens,
descriptive than COCO, containing more words. Figure 4 tokenized using the CLIP tokenizer, for COCO, BlendCap and HolisticCap.
HolistiCap BlendCap
Table 9: Statistics of different captions. The mean and standard deviation of the number of words and tokens (post CLIP tokenization) are reported. Note the 4x difference between the original COCO and HolisticCap. Also note that the verbosity of Visual Captions is reduced (by about 10) after combining with BlendCap to create HolisticCap.
200K
Number Captions
COCO
150K
100K
50K
20
40
60
80
100
Number of Words
Dataset
Words/ Cap
Toks/Cap
COCO
10.5 1244
13.5 122.7
Figure 4: Histogram number of words for COCO, BlendCap, and HolisticCap. HolisticCap is more descriptive with 41.5 words per caption average, while COCO only has 10.5 words average.
BlendCap
20.1 16.0
25.4 17.0
HolisticCap
41.5 #13.1
50.2 #15.4
Visual Caption
51.1 #14.4
59.2 +128
Bag Creation, Curation, and Curriculum Details
B.1 Bag Creation and Curation
Algorithm D presents the process for creating bags of highly similar images that is used for both TrueMatch (Section 3.1 and SR fine-tuning (Section 4.6).
We take the bags created by Algorithm 1 on the validation and test sets of COCO (10,000 images) and Algorithm 2 to curate bags of highly similar images for TrueMatch. The curated bags only consider the bag with the highest intra-bag similarity for given visual concept.
B.2 Training Bags used for Self-Retrieval
We utilize our bag creation algorithm (Algorithm 1 to create training bags. However, for each row in the sorted cosine similarity matrix D only consider the top 200 retrievals, í.e. DC:, :200]. Each training bag of size is comprised of given query image and the top s-1 images it retrieves from D such that none of the retrieved images have been added to any other bag before. This ensures that all the bags are unique and each image is seen exactly once in an epoch.
Unlike TrucMatch where we select the hardest bag corresponding to specific visual concepts in the CLIP embedding space, we relax the constraint of only sampling the sub-cluster (bag of images) with the highest intra-cluster similarity.
20

---

Published in Transactions on Machine Learning Research (01/2025)
Algorithm Candidate Bag Creation
1: Input:
Output bag size s.
Set of images, each with M captions: {(i1, ((..,).... ."). (iN.
Output:
List of T bags B = [b1, b2,.. by] where each bag b C I.
= [a) .2,..., .T] are the intra-bag similarities.
3: Multimodal feature extraction:
4: for each image (i.ct,...,cc ca)) do
Compute image embedding CLIP CLIP. encoode_imaaee(( Compute text embedding t = .M Σ hulPeencoe__eexeeeeeeee Compute multimodal embedding m concat(z, t), m Rd.
6:
end for
9: Compute the cosine similarity matrix E NxN based on multimodal embeddings [mi.... m.]
10: Bag Creation:
11: I
12: for each image i,, r= [I,..,,N do
13: Create bag b, with i, and corresponding top-scoring s-1 images from row D,.
14: Compute intra- bag similarity between i, and s-1 images in the bag: a, = mean; coos(m,)))) my)).
15:
Update B + [B, br] and A + [A,a,]
16: end for
Algorithm TrucMatch: Automated Bag Curation
1: Input: List of T bags = [bi...., br] and corresponding intra-bag similarities A [au,..., aT]
Output: A curated list bags with highly similar images used in TrueMatch.
Benchmark Creation:
4: Sort all bags B in descending intra-bag similarity A.
5: Initialize the set of visited images V = { and curated list of bags Q = |]
6: for each bag ó B do
7: Pann= = then
Update + V Update Q [0,
10: end if
11: end for
21

---

Published in Transactions on Machine Learning Research (01/2025)
Table 10: Self-retrieval fine- tuninn of GPT-2 and CLIP (SR-LV) for ClipCap that is MLE initialized with HolisticCap. Values indicate R@I. (Green) numbers indicate absolute improvement and (red), a degradation.
TrueMatch
Method
CIDEr
RD100
#3
#5
#7
1
Rand
26.5
90.2
62.6
52.7
51.2
2
Bag2
26.0 (0.5)
90.5
64.8
54.0
55.1
3
Bag3
26,0 (0.5)
90.7
64.8
55.4
55.3
Bag5
25.2 (1.3)
90.9
65.0
55.8
52.8
5
Bag7
25.7 (0.8)
90.8
65.1
56.7
55.3
6
Bag10
25.3 (1.2)
90.9
65.1
55.8
54.1
7
Bagl5
24.9 (1.6)
91.0
66.5
59.2
56.5
8
Bag20
24.8 (1.7)
91.1
66.5
58.8
57.3
9. BagCurri 29.7 (3.2)
91.3
65.2
57.1 57.1
B.3 BagCurri: Designing Self-Retrieve Learning Curriculum with Bags
We design curriculum that varies the bag sizes during training, gradually increasing the bag size with each epoch as shown in Figure 5] Since the reward is the cross-entropy of matching the generated caption with the target image Dessi et al. (2023), mining multiple hard negatives yields a flatter softmax distribution leading to stronger learning signal. Hence, increasing the bag size introduces makes the task of SR more challenging. However, is important increase bag sizes gradually to ensure that the task not too hard, and prevent. collapse of caption faithfulness (Section 4.6).
To assess the role our curriculum plays in preventing this col- lapse, we conduct an ablation study. Specifically, we compare our curriculum against training with bags of a fixed size for the entire SR fne-tuning process. Table I0 demonstrates that our carefully designed curriculum is responsible for preserving cap- tion faithfulness. While training with hard negatives without curriculum improves retrieval performance 2-8), it fails to maintain the CIDEr score. In contrast, BagCurri not only yields substantial gains over training with random distractors (row 1) but also enhances CIDEr by +3.2%. This highlights the effectiveness of our carefully designed curricu- lum preventing the collapse of NLG metrics while improving retrieval performance. Note, this is different to previous SR presented in Section 4.4
Bag
16
14
TrueMatch (rows
Epach
approaches (Dessi et al. 2023
Figure Our curriculum over bag sizes during training.
C
Self- Retrieval Fine-tuning
C.1
Discussion on Optimization Strategies
Similar to (Pinto al. )023), we find that both optimization strategies, MLE and REINFORCE, discussed
Section 4.1 have complementary strengths and weaknesses for the task of image captioning.
MLE. Training the captioner MLE results in powerful models capable of capturing complex data distributions. The task of next-token prediction with current language models is both efficient (due to teacher forcing) and scalable (Kaplan et al. 2020 However, it creates captioning systems that prefer generic captions [Wang el al. 2020). reusing a small vocabulary to describe different images with fine-grained visual differences (see Figure L and Section 4.3
Reinforce. While fino-tuning with SR alleviates these issues (Section 4 it comes with its own set of challenges. Not only are the tokens sampled sequentially but optimizing REINPORCE is unstable, necessitating
22

---

Published in Transactions on Machine Learning Research (01/2025)
smaller learning rates. This results in extremely slow training process. Notably, using REINFORCE from scratch is infeasbile due to the vast action space of sampling token and reward sparsity. However, using a pretrained MLE model provides a good initial sampling strategy and requires relatively few optimization steps to perform well on SR. This consistent with previous works such as Rennic et al. (2017).
In Reinforcement Learning jargon; the gradients computed with REINFORCE are used directly optimize the policy network (captioning system) in prioritizing actions (next-token prediction) that minimize the negative expected reward P--))( ((,,
C.2 REINFORCE baseline
The variance of the gradient estimate in REINFORCE is commonly reduced by subtracting a baseline b from the reward R. This stabilizes training by reducing the variance of the estimated gradient. We adopt running mean past rewards as our baseline similar to Dessi et al. (2023 as they find that outperforms computing the reward with greedy decoding as a baseline. The latter requires sampling two outputs for one training input, one estimate the gradient and other compute the baseline. However, when the reward includes CIDEr, we adopt groedy baseline.
C.3 Overview of Hyperparameters
We provide a brief overview of hyperparameters for different optimization stages: MLE (Table 11 and REINFORCE (Table 12h. Notably, CIDEr optimization for HolisticCap is done with learning rate 10
Table 12: REINFORCE optimization settings.
Base params
Table 11: MLE pretraining settings.
Batch size:
100 constant
Schedule:
MLE Prefraining
Total steps:
23 000
Batch size:
40
Warmup steps:
Schedule:
Linear Decay
SR reward
Learning-rates
2 10
Learning-rate:
10-8 CIDEr reward
Total steps:
30 000
Warmup steps:
Learning-rate:
1 10
CIDEr SR reward
Learning-rate:
1 10
C.4 Fine- tine-tuning CLIP hurts Caption Faithfulness
Table 13: Extended SR-L fine-tuning until its R@1 on RD100 becomes equal to SR-V V fine-tuning R@1 when trained for 20 epochs.
Table shows that SR-V V is substantially more discriminative com- pared to SR-L. However unlike SR-L. SR-V fails to preserve caption faithfulness through early stopping. To verify that the observed deterioration isn't due to the tendency of NLG metrics to penalize discriminability (Wang al. 2020), train SR-L until it achieves the same RD100 score as SR-V for each datasct. This ensures a fair comparison of both methods' abilities to preserve CIDEr, as presented in Table 13 We clearly see that SR-L (col 2) attains a significantly higher CIDEr score compared to SR-V (col 3) for all caption types, demonstrating that the deterioration in NLG metrics observed in Table 4 is directly caused by CL.P fine-tuning.
Data COCO BlendCap
MLE SR-L SR-V
107.8 104
97.2
80.2
77.3
72.4
HolisticCap
26.6
26.4
25.6
23

---

Published in Transactions on Machine Learning Research (01/2025)
383443
470005
410428
Caption spacious and well-appointed bathroom features two sinks, a bathlub, and mirror.
Caption baby wearing pajamas sits bed while using laptop computer.
Caption Three sheep. of which adults and baby. stand together in grassy field.
CIDEr:
CIDEr
CIDEr 0
Figure Failure of CIDEr. COCOIDs in the test where CIDEr fails to correctly evaluate captions against HolisticCap reference sct.
C.5 Scaling CIDEr during SR optimization
The weighting of the rewards when jointly optimizing CIDEr and SR (Luo al., 2018) determines the trade-off between laithfulness and discriminative ability of the captioning system. We MLEE pretrain the captioner with HolisticCap and fine-tune both the LLM and CLIP components (SR-LV) with BagCurri. Table I4 presents an ablation over different values X while scaling CIDEr in the joint. reward R SR CIDEr. As expected, increasing í.e. heavily weighing CIDEr leads to improved caption faithfulness. Interestingly, using smaller however doesn't always improve TrueMatch scores, X 0.7 (row 5) outperforms X 0.1 (row 2). We adopt À = 0.5 for our experiments (Section 4.8) as it results in the best per- forming model achieving an optimal balance between retrieval performance and caption faithfulness.
Table 14:
Ablation over different lambda values for HolisticCap pretraining and SR- LV with BagCurri fine-tuning. We choose A=0.5 our best model.
NLG
RD100
TrueMatch
A CIDEr
R@1
#3
#5
#7
0
29.7
91.3
65.2
57.1
57.1
0.1
29.8
92.4
61.8
54.0
55.1
3 0.3
33.3
92.0
68.2
61.2
57.1
0.5
38.0
92.6
67.7
58.7
57.9
5
0.7
41.4
92.7
68.1
58.3
56.8
92.5
66.5
56.2
56.2
C.6 CIDEr Struggles with Longer Descriptions
Notably, observe that the scale of CIDEr scores for MLE pretrained models vary significantly across COCO, BlendCap, and HolisticCap (rows 1-3, Table 3) Consistent with Santos et al. (2021), we find that CIDEr rewards brevity, while reference sets containing longer captions have substantially lower scores. While the inefficacy of MLE in modeling longer descriptions does contribute to lower CIDEr scores, we also find that CIDEr fails to effectively evaluate MLE pretraining with HolisticCap, even assigning scores of to correct captions, as illustrated in Figure
C.7 CHAIR: Quantifying Object Hallucinations during Extended SR-L Fine-tuning
We generate captions for all the 10,000 images of MSCOCO test and validation set and compute object level hallucinations using CHAIR (Rohrbach et al. 2018). CHAIR compares model-generated captions with human annotations and provides 2 scores CHAIR, and CHAIRs to quantify object and sentence level hallucinations respectively:
CHAIRs = [captions with hallucinated objects)|
| (hallucinated objects)
CHAIR, = Hall objects mentioned]]
(2)
[all captions}|
24

---

Published in Transactions on Machine Learning Research (01/2025)
Table 15: Evaluating object hallucinations. We use CHAIR to study the extent of object hallucinations induced by MLE training and SR-L fine-tuning.
Dataset
Method
CHAIR,
CHAIRs
Hallucinated Objects
CIDEr
MLE
11.66
8.38
1255
107.8
COCO
SR-L
9.79
7.08
1033
108.2
SR-L 100 epochs
9.63
7.33
1102
92.4
MLE training encourages the captioner to generate generic phrases for different images that may not be aligned with the image content, resulting in increased object hallucinations.
SR-L fine-tuning may struggle with attribute binding, but it generates captions that are discriminant and specific to the image, resulting in signilicant reduction in object hallucinations.
Extended fine-tuning with SR-L for 100 epochs in addition to reducing caption faithfulness, also increases both the absolute count and percentage of hallucinated objects.
C.8 Investigating Caption Unfaithfulness in SR Fine-tuning
To improve our understanding of the trade-off presented Section 4.4 during SR fine-tuning, we further investigate the deterioration the NLG metrics. Dessi et al. (2023) attribute this deterioration to the model diverging from the GT distribution during SR-L fine-tuning. They argue it be desirable property il allows the captioner to generalize to wider range of captioning distributions. We investigate this claim by qualitatively analyzing image-caption pairs (COCO test set) where SR-L exhibits poor CIDEr scores compared to MLE. We find several image-caption pairs (somc are shown in Figure 7 corroborating this claim, highlighting the inadequacies of reference-based metrics. For instance, more discriminative caption "A woman serving a hot dog to a man" scores lower on CIDEr than a more generic one, "A couple of people standing front a food truck". However, our analysis also reveals that models fine-tuned on SR, in an attempt enhance retrieval, have a propensity to hallucinate which results poor CIDEr scores (see Figure [ This differs from the findings of Section 4.3 since the model is incorporating non-factual information the generated captions instead of semantic details. We observe this tendency under the experimental setting as (Dessì et al.] 2023), noting modest dip in NLG metrics due to early stopping, However, we find this effect exacerbates with longer training durations, as illustrated by the dramatic drop in CIDEr scores in Figure 3
25

---

Published in Transactions on Machine Learning Research (01/2025)
E
cocold: 482436
cocold: 222757
cocold: 558137
cocold: 472216
cocold: 10428
MLE: man standing in front
MLE: A view of empty window
MLE: couple of people that slanding front food truck
MLE: group of people down
MLE: bathroom with sink
covered slope
CIDEr: 56
CIDEr:
109
SR: A black white photo of people in restaurant
room window and black and while floor
woman serving a dog to man
group women cross country sking in competitin
gallery filled with portrails of men and women
CIDEr:
CIDEr: 12
CIDEr: 11
CIDEr: 8
CIDEr:
Figure 7: Low CIDEr due to generalization. COCOIDs in test set, where COCO SR fine-tuned caption scores poorly on CIDEr compared to COCO ML.E trained model due to generated captions being different. from COCO's ground truth distribution.
cocoid: 503311
cocoid: 310757
cocoid: 186747
cocoid: 328344
cocoid: 375078
MLE: A beach fying kite
MLE: brown dog standing next yellow dog
MLE: A tall giraffe standing next tree
MLE: close of orange on table
MIL:: A scissors sitting on top table
CIDEr:
CIDEr: 92
CIDEr: 107
CIDEr;
CIDEr:
SR: Two brown dogs standing next to yellow frisbee
SR: Two girafles standing next to each other tree
SR: A pair of sciesors and an table
SR: black and while stylized black and white stylized styized
windsurting the ocean
CIDEr:
CIDEr: 14
CIDEr: 24
CIDEr:
CIDEr:
Figure 8: Unfaithfulness through hallucination. Some COCOIDs from the test set where SR fine-tuned captioner scores poorly on CIDEr compared to COCO MLE trained model due to nonfactual information.
26

---

Published in Transactions on Machine Learning Research (01/2025)
cocoid: 536589
cocoid: 72397
cocold: 560756
cocoid: 137727
SR-L: A bicycle is parked next to boat on dock.
SR-L: A blue fire hydrant sitting on the side of a street.
SR-L: A living room filled with furniture and couch.
SR-L: A blue and grey passenger train parked on the tracks at a train station, crane nearby.
CIDEr: 77
CIDEr: 275
CIDEr: 83
CIDEr: 22
SR-V: Two bicycles that are next each other.
A blue fire hydrant sitting on the side of a street.
SR-V:
SR-V: A living room filled with red chairs and orange walls.
SR-V: A and orange train sits on the tracks train station, ready passengers to board
CIDEr:
CIDEr: 275
CIDEr: 7
CIDEr: 13
SR-V curri: bicycle and a displayed display case.
SR-V curri: Aluue and blue blue blue and yellow blue and yellow blue and yellow blue and yellow blue...
SR-V curri: Three orange chairs sitting next to each other room.
SR-V curri: and orange parked to each other train tracks.
CIDEr: 12
CIDEr: 0
CIDEr: 0
CIDEr:
Figure SR-V fine-tuned captioner struggles with attribute binding and hallucination. Training with BagCurri (SR-V curri) worsens this tendency.
Verified Bags in TrueMatch
Bags Discarded with Manual Verification
Figure 10: Manual filtering of bags in TrueMatch. We manually remove bags that do not capture some aspect of fine-grained visual discrimination, even they represent the same visual concept.
27


Reviews:
[{"review_id": "YbGqOILKcz", "reviewer": "Reviewer_Zz97", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "- Strengths\n  - The paper addresses the crucial challenge of fine-grained caption generation in vision-language understanding. \n  - The proposed approach demonstrates strong performance across benchmarks, supported by comprehensive ablation studies and empirical analyses. \n  - The thorough evaluation protocol effectively validates the framework's robustness, while providing insightful qualitative examples of the model's capabilities.\n\n- Weakness\n  - A significant concern regarding this submission is the potential for hallucinations throughout the pipeline. The multi-stage data generation process, which incorporates both LLM and MLLM components, may introduce hallucinations despite the anchoring mechanism to blended captions. Such data-level hallucinations could compromise the data quality and consequently affect the validity of the entire experimental protocol. \n  - A notable limitation lies in the evaluation methodology, which relies heavily on CLIP score-based retrieval. Since CLIP treats captions as bag-of-words, this metric may not adequately capture the semantic relationships and intended meanings in the generated captions, potentially providing an incomplete assessment of the model's true performance. This limitation is particularly problematic in the context of fine-grained vision-language understanding, where precise semantic relationships and nuanced details are crucial for evaluation.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper proposes solutions to improve fine-grained image captioning, which often yields generic descriptions due to noisy training data and MLE training methods. The authors introduce Visual Caption Boosting to enhance captions with detailed descriptions and BagCurri, a training curriculum that optimally utilizes self-retrieval (SR) rewards. They also present TrueMatch, a new evaluation metric to assess a model's ability to distinguish subtle visual details. Overall, the approach enhances caption specificity and accuracy, outperforming existing methods on various benchmarks.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- I recommend revising the statement about self-retrieval guiding the captioner away from its language modeling priors (Finding 2). It would be more accurate to describe self-retrieval as \"unlocking\" or \"unleashing\" the potential for descriptive captioning. This emphasizes how it harnesses latent semantic information, enabling more nuanced and fine-grained captions.\nAdditionally, the results for maximum likelihood estimation (MLE) are not as low as implied; MLE maintains a respectable performance level. Acknowledging this balance would provide a clearer picture of the effectiveness of the different training approaches.\n\n- As evidenced by Finding 3, the fine-tuning process with SR-L exhibits a tendency toward hallucination generation. The paper would benefit from a quantitative assessment of these hallucination phenomena using established vision-language hallucination metrics, providing a more rigorous evaluation of the framework's reliability.", "broader_impact_concerns": "Since this framework addresses fine-grained image description generation, there are concerns regarding the potential inclusion of undesirable content in the generated responses. Furthermore, the framework may inherit inherent biases from the pretrained LLMs used in its pipeline, raising important considerations about the model's fairness and reliability in real-world applications.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "oRgDApmwcS", "reviewer": "Reviewer_Wasn", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n=======\n- Well written paper with lots of insights and contributions that improve performance.\n- The dataset and metric would be easy to use if published (as in these are simple to adopt). The training pipeline is complicated but the complexity is justified by the empirical evidence and trade-offs involved.\n- The discussion balances well the several metrics that need to be tracked and provides an interpretation for the reader that is consistent throughout.\n\nWeaknesses\n=========\n- Will the HolisticCap dataset and the bags used for TrueMatch and for SR finetuning be made public?\n- Will any code be published?\n\nClarifying questions\n===============\n- How was manual filtering of the bags in section 3.1 done? Examples in the appendix of what was filtered and not filtered would be helpful.\n- The bag curation in section 3.1 does a sort of non-maximum supression to avoid reusing images and picking the most challenging bags. The last line additionally claims that the bags capture different aspects of visual discrimination beyond object, positioning and negation. How is this achieved?\n- In section 3.3, 'we observe that HolisticCap, which uses InstructBLIP': How does HolisticCap use InstructBLIP? I didn't understand this part.\n- Section 4.2 says that LoRA tuning is done, but section 4.3 starts with finetuning the langauge model. When the manuscript says SR-X is X being LoRA tuned?\n- In Table 1, is \"Ours (best)\" HolisticCap + SR-LV + Bag-Curri + CIDEr optimization?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The manuscript proposes 3 changes the pipeline for training a captioning model, towards the goal of obtaining fine-grained detailed captions that are also \"correct\".\n\n1. The training data is augmented using LLMs and MLLMs to combine multiple human captions per image into one single detailed caption. This makes the training data itself capture the sort of fine-grained details one desires from captioning models. The end result is the HolisticCap variation of COCO captioning data.\n\n2. A new evaluation metric called TrueMatch that builds on self-retrieval based evaluation. The contribution here is to mine similar bags of examples of different sizes so that the self-retrieval task is challenging from a fine-grained perspective.\n\n3. The training pipeline is improved in many ways: \n- Phase 1 trains on HolisticCap data to get a strong initial model.\n- Phase 2 fine-tunes using the self-retrieval objective on challenging bags. Here the contribution is to fine-tune all model components and use a curriculum of self-retrieval bags.\n\nThe paper analyzes weaknesses in the existing approach and illustrates how the several contributions work together to solve it.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Discussed in the previous box.", "broader_impact_concerns": "A limitations section is included but there is no broader impact statement. COCO includes the person category so some discussion is needed as captioning people with images requires more care with regards to model fairness.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "kLfPezNEnR", "reviewer": "Reviewer_tNaH", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "This is a fantastically written paper. It has virtually impeccable English and it was a pleasure to read something that has obviously been curated and prepared with careful attention to detail. \n\nI have some thoughts I jotted down during reading, based on a selected figure or two (mainly Figure 1 as it sets up the conceptual reason for the proposed work, while all follow-up work is carefully addressing issues raised from this point).\n\nFigure 1 is intended to give a concrete example to illustrate the current “problem” and how their approach solves it. The COCO MLE versions give the same caption to the same image (as they are all visually similar). The COCO SR parrots the COCO MLE version, but gets the 2nd picture wrong (there are not two people in this image). The proposed method (3rd set of captions) gets more details correct by mentioning the presence of gloves and the colour of the bike. However, it feels like the captions are divergent for the sake of text diversity alone. Why in some cases but not others could you call them motorcyclists? Intuitively, I feel like if I wanted a system to caption some images, the visual similarity of the images would make me prefer a system that was broadly consistent. If I wanted to use a machine learning system based on such captions, I’d be a bit concerned there was a tendency to mention specific items in images for some captions but not others (the presence of clothing description in one caption, but not the others). This means for the sake of being ‘identifiable’ (describing an image in one way that isn’t applied to the others) it could result in a patchwork-like situation where different details are highlighted in different images to pass the task. \n\nI think whether this is desirable or not depends on your use ase. If my system should learn many details across many images, the extra details in some images probably is desirable and useful if this approach was used to create training data. However, on an individual case-by-case basis if this was part of a final user-facing system.\n\nThis sort of sets me up to wonder just how much of a problem this actually is, with this being the flagship example the authors have chosen to promote. It is nice to see that the only caption for all variants that doesn’t automatically assume the presence of a man is a caption proposed in the authors current work (3rd caption). I just don’t think the authors are being fair when they imply the MLE captions don’t meaningfully capture the semantics of each image.\n\n---\n\nIn Figure 2, how do you derive what is a hallucination? Specifically, the example includes underlined text (red line) and the system should default to human annotations in these cases and not include them in the holistic caption. I wasn’t sure how the system did this and would like to know. Is this derived when the MLLM visual description includes details not present in ANY of the human annotator captions or is there some sort of different system?\n\n----\n\nYou can always design a benchmark that will emphasise particular properties of a proposed system, so evaluation on TrueMatch over the larger MLLMs is not painting the full picture, unless that benchmark encompasses most things that are desirable in a captioning system. I didn't get the sense that is what TrueMatch claimed to be, so I think the authors could couch some descriptions a bit better with this in mind. I got the sense the reported better metrics on their own design of a benchmark was perhaps implying more of an external validation than is perhaps warranted. However, this derives from my earlier questions around Figure 1. All in all, this is quite a minor point.\n\n----\n\nIn conclusion, this is an exceptionally strong submission and it was a pleasure to read. My comments above relating to potential weaknesses are relatively minor. Overall, it's a very impressive piece of work and I am happy to have had the opportunity to review it.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "There are three main contributions proposed in this work.  The first and second are a proposal to implement a curriculum for training image captioning models via self-retrieval in a way that targets the claimed observation that such methods are prone to hallucinations, with human-anchored captions, via a framework the authors call Visual Caption Boosting. The authors also propose a new benchmark which contains very similar pictures and forces models to be able to discern small details to match images to descriptions.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "My requested changes are non-critical and more in aid of increasing the submissions readability. I would be happy to see the submission accepted as is, if there were not time/interest available in making the changes.\n\nFigure 2 gets referred to prior to the definitions of ‘HolisticCap’ and ‘BlendCap’ but these are used in the figure caption. This happens a bit too early and is a bit confusing. I would recommend either amending the caption to define them or use those terms in the figure somewhere. It’s not an impediment to understanding in any strong sense, but it’s not clear when reading whether the specific term there directly refers exactly to what’s implied in the image or could be a derived system that is yet to be explained. I think the easiest solutions is to say ‘holistic / blended captions’ (not ‘HolisticCap’ / ‘BlendCap’) in the figure description, for clarity, and then define those other terms later on the same page.", "broader_impact_concerns": "The authors do well to consider potential broader impact concerns and I was impressed by how this was presented in the submission and how cultural issues are taken into consideration (in terms of what it means for different cultures to notice and report in annotations of visual images).", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: During the review-author discussion period, all the reviewers agreed that this paper solves an interesting problem and is well-written. I also think that the contribution of the current paper is sufficient to be published as a TMLR paper.
Decision: Accept (poster)