Seeing What’s Wrong: A Trajectory-Guided Approach to Caption Error Detection
Authors: Gabriel Afriat
Ryan Lucas
Xiang Meng
Yufang Hou
Yada Zhu
Rahul Mazumder
Venue: ICLR 2026 Poster (2026)
Keywords: Image-Caption Alignment
Error Detection
Caption Trajectory
Primary area: applications to computer vision, audio, language, and other modalities
TL;DR: We introduce TRACED, a model-agnostic, efficient, and interpretable framework that leverages caption trajectories to improve error detection on image–caption datasets

Abstract: Error detection is critical for enhancing multimodal dataset reliability and downstream model performance. Existing error filters, while increasingly powerful, typically rely on a single similarity score per image–caption pair. This is limiting: captions with subtle errors (e.g., mislabeled objects, incorrect colors, or negations) can still score highly, while correct but imprecisely worded captions may score poorly. To address this, we introduce the notion of a caption trajectory: an ordered sequence of captions produced by iteratively editing a caption to maximize an image-text relevance score. This trajectory carries rich signals for error detection. Correct captions typically stabilize after minor edits, while erroneous captions undergo substantial improvements. Building on these insights, we introduce TRACED, a cost-efficient and model-agnostic framework that leverages trajectory statistics for more accurate caption error detection. Beyond detection, TRACED also serves as an interpretable tool for identifying the origins of errors. We further demonstrate that, in the case of error correction, this interpretable token-level error information can be provided to VLMs to enhance the alignment score of the generated captions. On MS COCO and Flickr30k, TRACED achieves up to 2.8% improvement in accuracy for error detection across three noise types. Our code is available at https://github.com/mazumder-lab/TRACED.

Published as a conference paper at ICLR 2026
SEEING WHAT'S WRONG: A TRAJECTORY-GUDDED APPROACH TO CAPTION ERROR DETECTION
Gabriel Afriat', Ryan Lucas', Xiang Meng', Yufang Hou Yada Zhu?, Rahul Mazumder'
"Massachusetts Institute of Technology. PIT:U Interdisciplinary Transformation University Austria, 3BBM Research
fafriatg, ryanlu, mengx, rahulmaz Jemit edu, yufang. houê it it,u.at, yzhuuus 1bm, com
ABSTRACT
Error detection is critical for enhancing multimodal dataset reliability and down- stream model performance. Existing error filters, while increasingly powerful, typically rely on a single similarity score per image-caption pair. This is limiting: captions with subtle errors (e.g., mislabeled objects, incorrect colors, negations) can still score highly, while correct but imprecisely worded captions may score poorly. To address this, introduce the notion of caption trajectory: an or- dered sequence of captions produced by iteratively editing a caption to maximize an image-text relevance score. This trajectory carries rich signals for error detec- tion. Correct captions typically stabilize after minor edits, while erroneous cap- tions undergo substantial improvements. Building on these insights, we introduce TRACED, cost-efficient and model-agnostic framework that leverages trajectory statistics for more accurate caption error detection. Beyond detection, TRACED also serves as interpretable tool for identifying the origins of errors. We further demonstrate that, in the case of error correction, this interpretable token-level er- ror information can be provided to VLMs to enhance the alignment score of the generated captions. On MS COCO and Flickr30k, TRACED achieves up to 2.8% improvement in accuracy for error detection across three noise types. Our code is available at https:::/ /ttpttttttuuu///ut////////t com/ ma zumder lab/ lab/TRACED TRACED
INTRODUCTION
Vision models have achieved remarkable success across diverse applications, including visual un- derstanding (Dosovitskiy et al. 2021), multimodal reasoning (Alayrac et al.] 2022). and content generation (Esser et al., 2024).
These models require extensive training on massive datasets, often containing millions of image-caption pairs (Ordonez et al. 2001] Lin et al. 2014 Russakovsky et al.)
2015 Schuhmann et al. 202T [Bann et al. 2021 Changpinno et al. 202
Li et al. 2022). However, many rely on pre- pre-training with web-scraped (Radford et al.) 2021 Li al. 2021 Lin et al. 2024)
or even synthetic data (Li et al./ [2022] 2023 Hammoud et al. 2024)). These datasets often contain significant errors (Northcutt al.] 202 Liao et al. 2021 Zhang al., (025), which not only hampers model convergence during training but can also reinforce biases and reduce generalization capabilities. Recent studies have demonstrated that removing incorrect image-caption pairs (Zhang et al. 2025 errors is essential for boosting data quality and training better models.
2022) can substantially improve model performance. Therefore, detecting such
As manual annotation is infeasible at scale, many works have proposed automated error detection. Existing error detection methods typically rely on assigming quality or similarity score to each image-caption pair, using either model confidence (Pleiss et al. 2020; (Swayamdipta et al. 22200 Northeutt al. 2021a), neighborhood consistency (Bahri et al.) 2020 Zhu et 2002 [002
2025), or multimodal alignment (Radford et al.] [202] Li et al. 2hu these existing methods are increasingly powerful, they typically rely on a single similarity score per image-caption pair. This poses key limitation: not all errors equally detectable. Some captions may mostly align with the image but include subtle mistakes-incorrect object labels, color description, negation that still yield high similarity scores. Conversely, correct caption might receive a low score if the image is difficult to describe or the wording is imprecise (see Figure I]. In both cases, relying single similarity score lead to unreliable error detection.
al.
Zhang et al. (025). While

---

Published as a conference paper at ICLR 2026
Nolay Caption: man is standing front of a brick storefront wearing jacket.
Correct Caption: Vehictes street near green traffic light.
BLIP Aligmment Score: error)
BLIP Alighment Score: 0.44 (error)
Figure I: Left: the BLIP-based alignment score (ITM block) is high (above 0.5), likely because the caption is mostly correct except for single erroneous word ("no"). Right: the Blae--based alignment score is low (below 0.5) even though the caption is correct.
In this paper, we propose novel approach that leverages caption improvement trajectories for more accurate error detection. Our key insight is that the potential for improvement varies significantly between correct and incorrect captions, a pattern alignment scoring functions we evaluated. Specifically, when starting with an accurate caption, iter- ative attempts to improve yield minimal gains in similarity scores. In contrast, incorrect caption presents substantial improvement potential. We formalize these intuitions by generating a sequence of increasingly refined captions for each image-caption pair and analyzing the resulting trajectory. Rather than making error detection decisions based on a single similarity score, our method exam- ines the pattern of improvement across the entire sequence. Importantly, this trajectory- -bsed frame- work is model -ageot-c and can be combined with existing state-of satt-oof-the-are the--rr-ttt art error detection baselines to enhance their performance.
observe consistently across the state-of sate-of-the-- off-the-art art
To evaluate error detection in image captioning, synthetic noise is typically injected through caption swaps (Zhang et al., 2025). More sophisticated swaps (e.g.. between captions sharing nouns or metadata) can create harder cases but still often yield captions that unrelated to the image and easy to detect. To test our framework under more challenging and realistic conditions, we introduce new form of fine-grained noise generated with GPT-4o-mini (OpenAl) 2024). By prompting the model to make minimal yet semantically significant alterations, this noise yields captions that remain plausible but contain subtle errors, making them harder detect (see left example in Figure [
We further show that our error detection framework has broader applicability. In particular, it pro- vides interpretable insights into the origin of errors, which can then be leveraged guide VLM toward potential error sources, thereby correcting them and enhancing the alignment score of the edited captions.
Our contributions are as follows:
1.
We introduce Error Detection), based on the novel idea of creating caption trajectories. By iteratively improving captions through token replacements and deletions, we generate sequence of captions and analyze both their alignment with the corresponding image and the semantic changes between iterations. This trajectory-basee enables more accurate identification of mismatched image-caption pairs. TRACED is cost- efficient and interpretable. It is also flexible and can be applied on top of many existing етror detection methods to enhance their performance.
error detection framework called TRACED (Trajectory Creation for
approach provides richer signals and
2. We evaluate how TRACED improves the performance of several state-of-the-art error de- tection methods, including CLiP (Radford al., (022). LEMON (Zhang et al. 2025), and BLIP (i et al., 2022). Our experiments contain various types of label noise, including traditional random caption swaps and more challenging type of synthetic noise we gen- erated by prompting GPT-4o-mini (OpenAl 2024). Compared to the existing benchmarks, this novel type of noise consists of plausible yet incorrect captions designed to better re- flect real-world annotation errors. On average across all noise types, TRACED consistently improves detection Accuracy by up to 2.5% on MS COCO (Lin et al. 2014), 2.8% on Flickr30k (Plummer et al.] 2015), and 2.4% on MM-IMDb (Arevalo et al. (017).
3. We show how TRACED can be used to provide interpretable outputs and identify specific misaligned tokens in erroneous captions. On Intern VL.3 models, we evaluate the impact
2

---

Published as a conference paper at ICLR 2026
of this interpretable token-level error information on caption correction. We show that this information can be used to improve the alignment of the generated captions, and observe an improvement of up to 14.5% in the BLIP-all alignneet score for the corrected captions using TRACED compared to unguided caption correction.
RELATED WORK
Handling Noise in Vision Datasets. Vision datasets often contain labeling errors that degrade model performance Zhang et al.] 2021 Northcutt al. 2021b; Liao al. 2021 Zhang et al., 2025). To address this problem, two main research directions have emerged: (i) learning with noisy labels by adapting the loss function reducing the influence of likely corrupted pairs (Natarajan eet al. 2013; [Bennouna et al. 2023; Arazo et al. 2019 Huang et al. (023), and (ii) data cleaning, which aims detect and remove mislabeled samples (Grivas et al. 2020 Zhang et al. 2025). Our
work follows the second line, improving the filtering of noisy image-captinn pairs.
Error Detection for Classification Datasets. Label noise can be detected through various ap- proaches. Confidence-based methods such as Confident Learning (Northcutt et al., 2021a). AUM (Peciss et al., 2020). and Dataset Cartography (Swayamdiptaa al.) 2020) flag mislabeled samples based on model confidence. Neighbor-basee approaches, including Deep k-NN (Bahri et al. (020) and SimiFeat (Zhu et al.) (2022), detect label noise by checking agreement with nearest neighbors in an embedding space. With the emergence of foundation models, new stronger baselines for la- bel error detection have appeared. Liang et al. (2024) and Kang et al. (2022) propose leveraging CLIP (Radford et al. (021). pretrained on 400M image-text pairs, to score image-label consistency. Building on this, LEMoN (Zhang et al. 2025) introduces neighborhood based method that aggre- gates relevance scores from multimodal nearest neighbors to improve error detection in classification and image captioning datasets. outperforms prior confidence- and neighborhood haaebbbroood-baeedd methods, making it strong baseline, which further enhance with trajectory- hased framework.
Error Detection for Image Captioning. In this paper, we focus on error detection in image caption- ing, more challenging task than image classification, it requires deeper semantic understanding of both language and visual content. To improve caption quality, BLIP (Li et al., 2022) builds on
CLIP by learning a shared image-text embedding space but also by training classifier distinguish high-quality from noisy image-caption pairs. Although not originally intended for error detection, Zhang et al., (2025) show that BLIP's filtering component performs very well in identifying misla- beled image-caption pairs on the dataset it was fine-tuned on. We therefore also examine how our framework can further enhance ILII on caption error detection.
Evaluation via Synthetic Noise Injection. Error detection methods are often evaluated by in- jecting synthetic label noise into clean datasets. Prior work has studied symmetric noise, where labels are randomly swapped (Pleiss et al. 2020 Kang al.) 2022). asymmetric noise, where labels are replaced with semantically similar ones via a transition matrix (Northcutt et al.] 2021a), and settance-depndeent noise, where incorrect labels depend on instance features (Liang et al. 2024 Zhu et al. 2002).. These noise models, however, are designed for classification tasks. Zhang et al. (2025) extend noise modeling image captioning via random caption swaps, swaps between cap- tions with shared nouns, and swaps within the same category using metadata. While more realistic, these approaches still replace entire captions, producing descriptions that can be unrelated the im- age. In practice, noise is often subtler: annotators may describe the correct image but misrepresent specific elements, and provide captions that are mostly accurate yet partially wrong. In this paper, we introduce another type of noise for image captioning that aims at capturing this fine- tne-grained form of caption noise and evaluate our framework in this more challenging setting.
Error Correction for Image Captioning. Prior work on caption correction has followed two main directions: (a) structured editing frameworks, where models are trained to add or delete words in a sentence (Wang et al.) 2022), and (b) dedicated correction models designed to fix caption (Sam-) mani & Melas-Kyriazz 2020 Huang et al., 2024 Berger et al. 2025). Our framework has direct applications to the second line of work. Early efforts used LSTMs for correction (Sammani &
Melas- Kyriazi 2020), while later work fine-tuned small VLMs (Berger et al. 2025) or leveraged
large closed-source models (Huang et al. 2024) such GPT-4 (OpenAl et al. 2024 These stud- ies highlight the effectiveness of transformer basee architectures for caption correction, with large
models offering strong but costly performance, and smaller VL.Ms providing efficient altemn-
3

---

Published as a conference paper at ICLR 2026
tive for large-scale dataset cleaning. In this paper, study both settings: evaluate TRACED on Intern the leading open-source VLM under 20B parameters on the OpenVLM Leader- board (Duan al. 2024), and smaller fine-tuned versions of Intern
3 TRACED: A TRAJECTORY-BASHE FRAMEWORK FOR ERROR DETECTION
To address the limitations of single-score image-caption alignment methods, we propose TRACED, trajectory-based framework that leverages iterative caption refinement for improved error detec- tion. TRACED iteratively modifies the caption to increase its alignment with the image and tracks how alignment evolves across these edits. This produces caption trajectory, i.c. a sequence of increasingly refined captions, which we use as a signal for error detection. Our core insight is as follows: (i) If the original caption is correct, alignment scores should improve only slightly, and edits will leave the meaning largely intact. (ii) If the caption is incorrect, alignment can typically be improved substantially- requiring major revisions. By capturing how easily and meaningfully caption can be improved, TRACED provides a richer and more interpretable signal than any single similarity score. TRACED is flexible and can be integrated with any existing ssoring-based error de- tection method. TRACED can be used for interpretability: using trajectory statistics, tokens that are likely incorrect in a caption. Another application of TRACED is caption correction. The interpretable information the origin of the error can be provided to VLMs to guide caption correction and improve the alignment score of the generated captions.
identify
3.1 TRAJECTORY GENERATION AND ASSESSMENT
Trajectory Creation and Evaluation. Let X' denote the set of captions and y the set of images. We assume access to a relevance scoring function:
X x R
(x, v) + s(x,
image-caption pair, with higher values indicating stronger alignment. The choice of is flexible: it may represent the matching probability in BLIP (Li et al. 2022), the cosine similarity of CLIP image and text embeddings (Kang et al., 2022).
This function assigns real-valued relevance score
or a multimodal similarity metric like LEMoN (Zhang et al. 2025).
To capture how a caption evolves during the procedure, we define trajectory evaluation function:
x y T+1
Rd (zo NT, 3) + e(sor..., Tr,y)
e:
where T + is the trajectory length and is the di- mensionality of the trajectory representation used for error detection.
Algorithm 1 Trajectory Creation and Eval- uation
A simple choice of e is the concatenation of relevance scores: e(ro...., 2r,9) v) = [s s)...)))......... (v).
Input: initial caption x, image y, scoring function 8, evaluation function e, number of exploration steps T, number of caption candidates N at each exploration step Initialize Zo
Another interesting metric to keep track of is the semantic similarity between the caption at step and the original (potentially noisy) caption ro. de- noted (o)) This captures the degree of semantic change introduced at each step.
for 1. = to do Generate candidate alternatives: x, " Select best candidate:
(N) (
In this paper, we focus on these two key signals and construct the following evaluation function:
je + arg maxje[N] s(a(), ..)
"T......,,T.... 9) = [s( y)................... s(2T, v),
(it)
c(21, zo). ... 2o)]
Set I4 + xt
end for Output: e(zo, NT,
While any metrics can be used here, we observe in Appendix that relying solely ple is suboptimal. Using both consistently better. Given access to s and e, TRACED constructs and evaluates caption trajectory described in Algorithm [
for exam-
4

---

Published as a conference paper at ICLR 2026
Trajectory Evaluation
Trajectory Creation
some students outside school buildings in the frost.
Learning
bukissten
Train model trajectories for error detection
some students outside school buildings the frost.
outsido school buildings in the grass.
TRACED Pipeline for error detection on an example from Flickr30k (Plummer et al, 2013). Given noisy image-caption pair, caption trajectory is generated by iteratively maximizing a relevance scoring function 8. The trajectory then evaluated using various alignment metrics, which serve as features distinguish between correct and incorrect image-caption pairs.
Figure 2:
Learning From Trajectories. We apply Algorithm [ to each image-caption pair in the dataset. From the resulting trajectory embeddings, train a classifier distinguish between correct and erroneous pairs. The overall framework is described Figure
3.2 CAPTION EXPLORATION
critical component Algorithm I is the generation of candidate captions each step. We explore and evaluate several strategies for this purpose:
Elimination (Elim). This simple and efficient method generates candidates by removing one token at a time from the current caption. Formally, for a caption x = "W,,,..;W; with tokens, we set N L in Algorithm and produce L candidates:
(wI...,, ..01....,................. W;-1, Wi+1s WL)
This strategy computationally cheap: it requires only forward passes through the scoring function in Algorithm and no gradient computations.
Greedy Coordinate Descent (GCD). Inspired by Zou et al. (2023), this method aims to find im- proved captions by replacing individual tokens with alternatives that increase the relevance score For each token in caption of length L. we consider the top-K K muudient-guides replacements, leading to a candidate pool size K L. Since this often too large to evaluate exhaustively, we randomly sample N token replacements from this space.
Fast GCD (FGCD). To balance the efficiency and quality of the caption trajectory, we introduce hybrid strategy that combines Elimination with Greedy Coordinate Descent (GCD). We first apply the Elimination method to identify the token whose removal most improves the relevance score. Then, we explore only the top-K replacements for that specific token, reducing the search space K candidates. This approach requires only one gradient computation and K KLL forward passes per iteration in Algorithm 1 significant reduction compared to the KL evaluations needed for full exploration. Moreover, by focusing on the most impactful token, we promote more effective substitutions than would be achieved by randomly sampling from a large candidate pool.
The full algorithm descriptions provided in Appendix A.2
3.3 INTERPRETABILITT
Examining the caption trajectory can help identify the source of the error. As shown in Figure 3 the first tokens whose removal or replacement leads to the greatest improvement in alignment score often correspond to the source of the misalignment.
In this example, the initial alignment score from BLIP's classifier 0.55, indicating a 55% prob- ability that the image-caption pair is correct. Relying on this score would result in misclassify- ing the image-caption pair. However, the trajectory shows that a meaningful semantic change can
5

---

Published as a conference paper at ICLR 2026
increase the alignment score around 99.4%, in- dicating that the original caption was likely erro- neous.
Caption
Ramaved word
Trajectory
mam
On the contrary, for correct captions, the improve- ments in alignment scores are often associated with minor semantic changes, as seen on Figure 6 in Appendix A.3 Similar observations can be made when using the GCD and Fast GCD al- gorithms. Wrong captions are often improved through substitutions with semantically different words, and correct captions tend to refined with minor edits. Full GCD and Fast GCD trajectories are provided Figures 7 and Appendix A..3
standing in frant brick storefront
weering,
man standing
brick storafront
Cornect Casption: A man is standing storefront weering black jacket.
wearing jacket
man standing in front brick
Noiey Caption: A man standing staretront wearing no jacket.
storefront wearing jacket
of
3.4 APPLICATION TO CAPTION CORRECTION
Simiterits optenno S -ontontti--sa
Aligemment Coptoon Nigun--nne
Zero-shot Correction. The trajectories produced by TRACED contain rich information, particularly about the origin of errors. While interpretability can also be derived from GCD and FGCD (by ex- adopt the
amining which tokens replaced), Elimination Algorithm because is both computa- tionally efficient and achieves performance parable or even superior to GCD and Fast GCD, suggesting that the extracted signals are of high quality. In this procedure, we remove one token at a time and observe the change alignment score: increases suggest that the token is likely incorrect (it is better to remove it), while decreases sug- gest that it is likely correct. This token-level er- ror localization is then provided to the VL.M in the prompt, illustrated in Figure 4 We also investigate the use of Chain-of-Thoure (CoT) prompting to test whether reasoning over the flagged words can further aid correction. The exact prompts used provided in Appendix A.17 In experiments, find that both the number of correction prompts and the way the token-level error information is provided play an important role in the final alignment scores. More information the exact procedure used is available Appendix A.5
Elimination Step
Elimination Step
Figure 3: TRACED offers interpretability on a Flickr30k example (Plummer al., 2015). identifying "no" as the source of misalignment. The BLIP-based alignment score (ITM block) peaks at step 6, where the caption accurately matches the image. Removing "no" leads to a notable decline in semantic alignment in the caption trajectory.
Correction After Fine-tuning. For large-scale data cleaning, smaller VLMs are advantageous due their lower cost and faster inference. When sufficient data is available, small model can be fine-tuned for caption correction using larger model as teacher. Given the alignment score gains observed for Intern when applying TRACED in the zero-shot setting (see Figure
caption
Error Detection
3 Error Correction
No Error or max iterations?
No
Caption: tennis player the court with racket.
VLM
Yes
A male tennis player is on the court with a racket
Men are the with rackets.
Repeat step ②
Flagged words: "femnle"
Using flagged words flagged words
Correct
Incorrect
Figure 4: TRACED correction pipeline on example from MS-COCO (Lin et al. 2014). The
Elimination procedure identifies words whose removal increases the alignment score (in red on the bar plot). These flagged words then provided hints to the VLM, and the process is repeated until no further errors are detected or the maximum number of iterations is reached. Following the hint-guided path (blue) leads captions with fewer errors compared to the unguided path (orange).

---

Published as a conference paper at ICLR 2026
fine-tune ViternVL3-133 captions corrected by Intem using TRACED 's token-level guidance in the prompts. The resulting model is referred to as InterVLL3-IB-FT At inference time, isolate the contribution of TRACED by comparing two use-cases the same fine-tuned model: (i) TRACED-guided inference, which supplies TRACED trajectory information to the VLM, and (ii) a baseline Intern VI33-B-FT model, which only the image and noisy caption. This design controls for distillation effects and measures the incremental benefit of TRACED at inference. Additional details on the fine-tuning process of AnteeVVL3-11 is available in Appendix A.6
EXPERIMENTS
4.1
SETUP
All experiments conducted using 4 NVIIIA L40 AA GPUs, each with 40GB of memory. TRACED is highly parallelizable (see Appendix(A.7] Thus, the datasets are split into 4 subsets, with each GPU processing one subset independently. Sentences are processed in batches of size 128 on each GPU. TRACED is implemented using PyTorch (Paszke et al. 2019). Details on the computation overhead are provided in Appendix [A.8] In general, TRACED efficient and scalable: with BLIP (TTM), the most computationally expensive baseline, TRACED-BLIP classifies 1,000,000 image-caption pairs in roughly 6.5 hours on 4 L40 GPUs using the Elimination algorithm.
4.2 BASELINES AND DATASETS
We evaluate TRACED against common eтror detection baselines on benchmark datasets by injecting noise into previously clean captions, with the goal of identifying erroneous image-caption pairs.
Baselines. We consider BLIP (Li et al. 2022), LEMoN (Zhang et al., 2025 and CLIP (Radford et al. 2021 Kang et al. 2022). CLIP uses cosine similarity in a joint embedding space, L.MooN ag- gregates CL.PP scores from nearest neighbors, and BI.IP combines contrastive learning (ITC block) to learn a shared image-text embedding space with a classification head (ITM block) for alignment prediction. LEMoN supports two versions: FIX (default hyperparameters) and OPT (hyperparame- ters tuned via validation).
Integration with TRACED. We apply TRACED on top of each of these baselines by using their respective alignment scores as the scoring function during trajectory construction. For BLIP, we evaluate TRACED using both the ITC and ITM modules. For LEMoN, apply our method to both the FIX and OPT variants. For CLIP, we use the standard cosine similarity between image and text embeddings.
Datasets. We evaluate the impact of TRACED on LEMoN and CLIP using Flickr30k (Plummer
al. 2015), MS COCO (Lin et al. 2014) and MM-IMDb (Arevalo al. 2017). For Flickr30k and MS COCO, we use the standard Karpathy split (Karpathy & Li, 2015). For MM-IMDb, we adopt the same random 80/10/10 train- vaaii split as Zhang al. (2025).
For BLIP, finetuned models are publicly available only for Flickr30k and MS COCO. Therefore, we evaluate the improvements from TRACED on these two datasets only.
Noise Types. We evaluate TRACED's improvements under three types of synthetic label noise, introducing 50% erroneous image-caption pairs for each seed:
Random noise: A subset of captions randomly replaced with others from the dataset.
Noun noise: Captions are swapped with others that share at least one noun, introducing partial semantic overlap.
noise: Captions are minimally perturbed using gpt-4o-mint to introduce sub- tle semantic inconsistencies, as described in Appendix A.4 Some illustrative examples of the generated errors are provided in Appendix A.16
Due to the higher cost of generating fine- einiindd noise using the ChatGPT API, we limit its use to Flickr30k and MS- COCO. For both random and noun noise, we follow the methodology introduced in Zhang et al. (2025)
7

---

Published as a conference paper at ICLR 2026
4.3 EXPERIMENTAL DETAILS FOR TRAJECTORY CONSTRUCTION AND LEARNING FRAMEWORK
Trajectory Generation Hyperparameterss The trajectory generation hyperparameters for Elimination, GCD, and Fast GCD are detailed in Appendix A.9
Table I: Comparison of TRACED with baselines. Mean accuracy and mean aceuracy improvement of TRACED vs. baselines, averaged over seeds and noise types (noun, random for MM-IMDB; noun, random, fine-grained for Flickr30k and MS- COCO 50% noise), with standard errors.
Trajectory Evaluation Metries. For the align- ment score s(x, v). we use the scoring function of the baseline being evaluated either CLIP, LEMoN, BLIP. For the semantic similarity No), we compute the cosine similarity be- tween the embeddings of 2 and Zg- When the baseline is BLIP, we use its ITC block to ex- tract embeddings. For CLIP and LEMoN, use CLIP embeddings.
DATASET
METHOD
ALGOR- ITHM
Acc. (%)
IMPROVE- MENT (%)
ELIM
89.5 0.2
1.3 0.2
TRACED-
BLIP (ITM)
FGCD
89.2 = 0.2
0.8 0.2
GCD
88.8 0.2
0.2
BLIP (ITM)
88.5 0.3
0.0
Learning Procedure Details. Once the tra- jectory embeddings are constructed, they can be used as features predict whether given image-caption pair contains an error. While any standard classification model could be applied at this stage, we use XGBoost and CART due their simplicity, as our primary goal is to demonstrate the effectiveness of our approach. More sophisticated models could be explored to further improve the performance gap between TRACED and the original baseline.
TRACED-
ELIM
88.1 0.2
0.9
BLIP (ITC)
FGCD
88.1 0.1
0.3
GCD
88.0 0.2
0.7
BLIP (ITC)
87.4 0.3
0.0
ELIM
85.6 33.6=10 0.4
0.2
TRACED- LEMoNoPr
FLICKR- 30K
FGCD
85.5 0.3
0.2
GCD
85.7 0.4
2.1
LEMoNoer
84.3 0.3
0.0
TRACED-
BLIM
85.0 0.3
1.7
0.5
LEMoNNIX
FGCD
85.0 0.3
1.7
0.1
GCD
85.6 0.4
2.6
2.6
LEMONBI
0.3
0.0
85.7 0.1
0.4
TRACED-
CLIP
FGCD
85.5 0.1
2.2
GCD
85.5=0a 0.3
2.5
For datasets that the Karpathy split, combine the original training and validation sets. We then perform 3-fold grid-search cross- validation to select the best model and hyperpa- rameters. The complete grid searches are pro- vided in Appendix A.9 The best-perforning model (XGBoost CART) and its correspond- ing hyperparameterss are selected based on the highest cross-validation AUC score.
CLIP
83.8 0.2
0.0
ELIM
79.0 0.0
1.4
TRACED-
LEMONoFT
FGCD
0.1
0.1
GCD
78.3 0.1
0.1
LEMONgeT
77.9 = 0.2
0.0
TRACED-
ELIM
78.3 0.1
2.4 0.2
MM-
LEMoNaIX
FGCD
77.2 172-0.. 0.1
0.9 0.2
IMDB
GCD
77.6
1.2
LEMONBIX
76.5 =.1 0.1
0.0
TRACED-
ELIM
= 1.8
CL.IP
FGCD
77.5 0.1
0.2
GCD
77.8 =.8 0.1
0.9
CLIP
77.2 0.1
0.0
4.4
MAIN RESULTS AND ANALYSIS
ELIM
90.5 0.2
1.7 0.1
TRACED-
BLIP (ITM)
FGCD
89.8 0.1
0.1
Error Detection. The results error detection are presented Table I
GCD
89.7 0.1
0.1
where accuracy scores and accuracy improvements are averaged over all applicable noise types and random seeds. TRACED yields consistent and significant gains over each baseline, highlighting its effective- ness for error detection.
BLIP (ITM)
89.1 0.1
0.0
0.0
ELIM
88.7 = 0.1
1.0
TRACED-
BLIP (ITC)
FGCD
88.4 0.0
1.1
GCD
0.1
0.0
BLIP (ITC)
87.4 0.1
0.0
+ 0.0
TRACED- LEMoNoPT
ELIM
85.0 0.3
1.6
0.0
MS-
FGCD
84.6 0.3
1.0
COCO
GCD
84.5 0.3
1.0
0.1
Table that the Elimination algorithm often generates more informative trajectories for error detection compared to GCD and Fast GCD, We attribute this to two main factors.
LEMoNoPT
0.3
0.0
0.0
ELIM
84.3 0.1
2.3
0.1
TRACED-
LEMoNpIX
FGCD
83.9 0.2
1.8
GCD
83.9 0.3
1.8
LEMONBI
82.6 0.1
0.0
ELIM
84,5 0.2
2.2
TRACED-
The Elimination algorithm progressively removes words from the caption, producing a trajectory in which the alignment score typically increases before decreasing. Unlike GCD and FGCD, which replace some tokens increase alignment, Elimination reflects
CLIP
FGCD
GCD
83.7 0.2
83.8 0.1
1.2
1.6
CLIP
82.7 0.2
0.0
8

---

Published as a conference paper at ICLR 2026
Baseline
TRACED
Flickr30k
MS COCO
Alignment Score
(Higher is bettei
0.8
+14.5%
Alignment Score
(Higheri is better)
+13.0%
0.7
+3.3%
+11.4%
+10.4%%
0.6
+2.0%
0.6
0.5
0.4
IVL3-18
IM3-18-FFT
IVL3-148 MVL--148
COT
IVL3-18
MM3-18-FT
IML3-148 IML3-148 CoT
Figure 5: Impact TRACED on BLIP alignment scores of the corrected captions from Intern 1B, IntemVLL3-1B-FFF and VL.311133114 (with CoT prompting for the 14B model) Flickr30k and MS COCO. Results are shown after five fixing steps with two fixing prompts at each step (see Appendix [A.5). and averaged over three seeds with standard errors. 50% of the samples are corrupted using fine-grained noise. VL3 models are abbreviated as IVL3 in the plot.
both the positive and negative contributions of each individual word, showing which align with the image and which do not.
The Elimination algorithm operates in much more constrained search space, which introduces form of regularization. In contrast, GCD and Fast GCD allow broader substitutions, sometimes leading to non-meaningful token replacements that nonetheless increase the aligmment score.
detailed breakdown by noise type is provided Table (Appendix A.110.. The largest gains from TRACED occur under the fine-grained noise setting, where subtle word changes make detection especially challenging and baseline methods struggle most. Under this regime, TRACED achieves up 7.5% improvement in accuracy compared to the baseline. These results highlight TRACED strength in handling more realistic and semantically nuanced errors.
Importance of N, T' and c. Appendix A.13 and Appendix 14. analyze the effect of the number of candidate edits N, the trajectory length T,, and the semantic similarity metric on the performance of TRACED. We find that TRACED primarily sensitive T: increasing T lead significant and consistent gains in accuracy. However, only a few exploration steps are typically required to locate the erroneous token (Figure 3). and small values of 7 already achieve near-optimal performance.
Performance of TRACED across caption lengths and noise levels. Appendix A.15 reports the performance of TRACED sentence lengths. Appendix(2. further evaluates TRACED under varying noise levels, with results indicating consistent gains over the baseline.
function of caption length, showing consistent improvements across
Application of TRACED on Error Correction. Figure 5 shows the impact of TRACED on the correction performance of Intern VL, including with fine-tuning (Intern IL33-IB-TT) and internV33-144 with and without CoT prompting. We omit CoT results for the IB variants here, as this smaller model exhibits limited reasoning capabilities. We observe that TRACED consistently enhances the BLIP alignment score of the generated captions, regardless of model scale. Notably, for VL.-11VL3311BB performance improves further when combined with CoT prompting, suggesting that token-level error information is highly informative optimize over the alignment metric. Our sults show that our procedure can successfully optimize caption quality with respect to an alignment metric. In particular, our approach is metric-agnostic and could also benefit future scoring methods.
Downstream captioning performance. Appendix A. evaluates the impact of using TRACED to identify erroneous samples for filtering or correction prior fine-tuning BLIP-2 Li al. (203)). Both approaches improve BLIP-2's captioning performance, with correction yielding the largest gains, and TRACED consistently outperforming baseline methods. Overall, on Flickr30k, our pro- cedure yields up to 1-point improvement in BLEU-4 over training on the noisy data.
4.5 IMPORTANCE OF THE TRAJECTORY
To assess the importance of the caption trajectory for effective error detection, compare the full TRACED trajectory to three simplified single point variants: (i) using only the first step (s(z), y); note that o0) provides no additional signal, (ii) using only the last step (s(#T, y) and

---

Published as a conference paper at ICLR 2026
cxrr, 2o), and (iii) using the mean of all alignment and similarity values across the trajectory (7 Ei=o s( y) and + Ei= c( kg)). Table 2 reports the mean percent change in test ac- curacy for each variant, relative to using the complete trajectory.
Table Mean percent improvement in Test Acc. when using only the first step, last step or mean trajectory alone in TRACED, compared to using the whole trajectory. Experiments are conducted on Flickr30k using the Elimination algorithm. Results are averaged over 3 seeds and all 3 noise types (50% noise), with standard errors reported.
FIRST STEP
LAST STEP
MEAN TRAJECTORY
METHOD
BLIP (ITM)
-1.25 + 0.23
-43.07 = 0.61
-4.90 0.16
BLIP (ITC)
-0.85
-40.54
-5.48 0.30
LEMONOPT
0.17
-38.67
0.09
LEMON
-1.62
-38.18 0.42
+
CLIP
-2.22 0.35
-38.49 + 0.30
-5.37 + 0.02
Retaining only the first step causes the mildest decline. Using only the last step yields the sharpest drop (over 38% in all cases), and averaging over the trajectory performs slightly better but still falls far behind the full sequence. These results highlight the importance of modeling the trajectory, which captures how alignment evolves and provides richer information than single-point summary.
4.6 MAXIMIZING OR MINIMIZING THE SCORING FUNCTION?
TRACED, we proposed to generate the trajectories by maximizing the image-caption alignment score at each step. To test whether the opposite strategy is also effective, compare against variant that minimizes instead. Table reports the mean percent improvement in test accuracy when using the minimization approach, relative to maximization.
Table 3: Mean percent improvement in Test Acc. when generating TRACED's trajectory by mini- mizing s rather than maximizing it. Experiments are conducted on Flickr30k using the Elimination algorithm. Results are averaged over 3 seeds and all noise types (50% noise), with standard errors.
BLIP (TTM)
BLIP (ITC)
LEMONpIx
LEMONoPT
CLIP
0.76
-0.46
-0.70 +0.24
-0.58
00.44
10.12
#0.17
10.34
10.15
Across all baselines, maximizing the alignment score yields modest but consistent improvements over minimization. This suggests that constructing trajectories toward higher-scoring captions, rather than worse ones, provides a more reliable signal for detecting inconsistencies.
CONCLUSION
We presented TRACED, a flexible and efficient framework for image-caption error detection. By iteratively improving captions and analyzing alignment and semantic similarity over time, TRACED extracts rich signals that help distinguish between correct and erroneous image-caption pairs. Our framework can be applied top of existing error detection methods such as CLIP, LEMoN and BLIP, consistently boosting their performance across multiple datasets and noise types. We also introduced a new fine-grained noise generation process that reflects real-world annotation errors and provides a more challenging benchmark for evaluation. Beyond improving error detection, TRACED can be used for interpretability by revealing which parts of caption contribute most to misalignment. We show that this token-level error information can effectively guide VLMs for caption correction, producing captions that are better aligned with their corresponding images.
10

---

Published as a conference paper at ICLR 2026
REFERENCES
lean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, Roman Ring, Eliza Rutherford, Serkan Cabi, Tengda Han, Zhitao Gong, Sina Samangooei, Marianne Monteiro, Jacob Menick, Sebastian Borgeaud, Andrew Brock, Aida Nematzadeh, Sahand Sharifzadeh, Mikolaj Binkowski, Ricardo Barreira, Oriol Vinyals, Andrew Zisserman, and Karen Simonyan. Flamingo: vi- sual language model for few-shot learning. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances in Neural Information Processing Systems, 2022. URL /ttp:::///ppee//////e/ee www.eeeeeeveeeeeeeeeeeeeeeee net wtt.wewweeeeeeeeeeeeeeeeeeee forum? i d=EbMu mAbPbs
https:
Eric Arazo, Diego Ortego, Paul Albert, Noel O'Connor, and Kevin Meguinness. Unsupervised label noise modeling and loss correction. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th International Conference on Machine Learning. volume 97
of Proceedings of Machine Learning Research, pp. 312-321. PMLR, 09-15 Jun 2019. URL https::/proceedingg htt.:/ad/a...mm......ee/////aaaa
John Arevalo, Thamar Solorio, Manuel Montes Gómez, and Fabio A. González. Gated multimodal 11wp..102..0000 019922
units for information fusion, 2017.
Dara Bahri, Heinrich Jiang, and Maya Gupta. Deep k-NN for noisy labels. In Hal Daumé III and Aarti Singh (eds.). Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pp. 540-550. PMLR, 13-18 Jul 2020. URL https: /tttpp:://procceccngg.. mlr
PMLR,
html
Max Bain, Arsha Nagrani, Gäl Varol, and Andrew Zisserman. Frozen in time: A joint video and image encoder for end-to-end retrieval. In ICCV, pp. 1708-1718, 2021. URL htttt:://dddo
Amine Bennouna, Ryan Lucas, and Bart Van Parys. Certified robust neural networks: Generalization and corruption resistance, 2023. httppp://aaxxi..rrr..//. abs//3/3..//.//////////////3aa3/ 02251
Uri Berger, Omri Abend, Lea Frermann, and Gabriel Stanovsky. Improving image captioning by mimicking human reformulation feedback at inference-time, 2025. URL https://arxiss
org/abs/250 04513
Soravit Changpinyo, Piyush Sharma, Nan Ding, and Radu Soricut. Conceptual 12m: Push- ing web-scale image-text pre-training to recognize long-tail visual concepts. In CVPR, pp. 3558 3568, 2021. URL /tttps://ppeaccees: thecvf www/ htte::ccoocooooooet http:ppooyoooooepuaaaaaaaaaa Pre Tee-Traiiinngg____eooooo
Web Scale Taaee___maeee__eeee Text_ html
Tail caapaa_______2_______
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszko- reit, and Neil Houlshy. An image is worth 16x16 words: Transformers for image recogni- tion at scale. In International Conference on Learning Representations, 2021. URI. https: //openzeview.net wtt.:n/e///oommittomoooom wic.eiiid--iccbTiYYYY
Haodong Duan, Junming Yang, Yuxuan Qiao, Xinyu Fang, Lin Chen, Yuan Liu, Xiaoyi Dong, Yuhang Zang, Pan Zhang, Jiaqi Wang, al. Vimevalkit: An open-source toolkit for evaluat- ing large multi-modality models. In Proceedings of the 32nd ACM International Conference on Multimedia, pp. 11 11198-112000 2024.
Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, Dustin Podell, Tim Dockhor,, Zion En- glish, and Robin Rombach. Scaling rectified flow transformers for high-resolution image syn- thesis. In Forty-fisı International Conference on Machine Learning, 2024. URL https: //ppen/eview.ntt//oorrirr
Andreas Grivas, Beatrice Alex, Claire Grover, Richard Tobin, and William Whiteley. Not a cute stroke: Analysis of rule- and neural network-hased information extraction systems for

---

Published as a conference paper at ICLR 2026
brain radiology reports. In Eben Holderness, Antonio Jimeno Yepes, Alberto Lavelli, Anne- Lyse Minard, James Pustejovsky, and Fabio Rinaldi (eds.), Proceedings of the 11th Interna- tional Workshop Health Text Mining and Information Analysis, pp. 24-37, Online, Novem- ber 2020. Association for Computational Linguistics. doi: 10. 00.18655/11/2020.oooih htttps://acllntthoo..e www.oog..oorog/ooo 1ouh.
URI.
Hasan Abed Al Kader Hammoud. Hani Itani, Fabio Pizzati, Adel Bibi, and Bernard Ghanem. Syn- thCLIP: Are we ready for fully synthetic CLIP training? In Synthetic Data for Computer Vision Workshop @ CVPR 2024, 2024. URL https: /ttpss////perreeee.ee net forum?id-
oKWYycMSrf]
Kung- Huang. Mingyang Zhou, Hou Pong Chan, Yi Fung, Zhenhailong Wang, Lingyu Zhang, Shih-Fu Chang, and Heng Ji. Do LVLMs understand charts? analyzing and correct- ing factual errors in chart captioning. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Findings of the Association for Computational Linguistics; ACL 2024, pp. 730-749, Bangkok, Thailand, August 2024. Association for Computational Linguistics. doi: 1/2284 findings- aadings-acll.1. URL httph::///// http://aalaaattohooooooo org/ Fin.ings-aiiiinnncc.cccc.
Runhui Huang, Yanxin Long, Jianhua Han, Hang Xu, Xiwen Liang, Chunjing Xu, and Xiaodan Liang. Nlip: Noise-robust language-image pre-training. In AAAI, pp. 926-934, 2023. URL
https::/ddo..rg//11 1609 vaa....7.....5
Woo- VoooYoung Kang, Jonghwan Mun. Sungjun Lee, and Byungseok Roh. Noise- learning from web-crawled text data for image captioning. CoRR, abs/2212.13563 2022. URLhttpst
//doi.org/10- 48550 rxiv. 2212.13563
Andrej Karpathy and Fei-Fei Li. Deep visual-semantic alignments for generating image descrip- tions. In CVPR, pp. 3128-3137, 2015. URL
2015.
7298932
Junnan Li. Ramprasaath R. Selvaraju, Akhilesh Deepak Gotmare, Shafiq Joty, Caiming Xiong, and Steven Hoi. Align before fuse: Vision and language representation learning with momentum distillation. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, 2021. URthttts:///ppeeeeer/
net forum?
id id=OJLakwi XSbx
Junnan Li, Dongxu Li, Caiming Xiong, and Steven C. H. Hoi. Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation. In ICML., pp. 12888- 12900, 2022. htt.:httt/////ooooo
ml prwss/vssvvvv.. /2/ 11t./.12.......m html
Junnan Li, Dongxu Li, Silvio Savarese, and Steven C. H. Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. CoRR, abs/2301.12597, 2023. (ttsst /doii org/10. 48550/ /a150//rriv..mm 2301 12597
Chao Liang, Linchao Zhu, Humphrey Shi, and Yi Yang. Combating label noise with a general surrogate model for sample selection. International Journal of Computer Vision, 133:3166-3179, 12 2024. doi: 10.1007/s 12631//1112--2222222222
Thomas Liao, Rohan Taori, Inioluwa Deborah Raji, and Ludwig Schmidt. Are we learning yet?
meta review of evaluation failures across machine learning. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2), 2021. https:
//openreview net. /wte/foorrre unrormm?ii wwwiidddmdamm 2.3 ducsSMMEK
Bin Lin, Yang Ye, Bin Zhu, Jiaxi Cui, Munan Ning, Peng Jin. and Li Yuan. Video-LLaVA: Learning united visual representation by alignment before projection. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp. 5971-5984, Miami. Florida, USA, November URL
2024. Association for Computational Linguisties. doi: 10.110...... 101118861110/// h/2024.cmmmpmmmmaii. orwph//2..ll.r..2222o..... p- inaaaiii... 342/
Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. Microsoft coco: Common objects in context. In David Fleet, Tomas Pajdla, Bernt Schiele, and Tinne Tuytelaars (eds.), Computer Vision ECCV 2014, pp- 740-755, Cham, 2014. Springer International Publishing. ISBN 978-3-319-10602-1.
12

---

Published as a conference paper at ICLR 2026
Nagarajan Natarajan, Inderjit S Dhillon, Pradeep K Ravikumar, and Ambuj Tewari. Learn- ing with noisy labels. In C.J. Burges, L. Bottou, M. Welling, Z. Ghahramani, and K.Q. Weinberger (eds.). Advances in Neural Information Processing Systems, volume 26. Curran Associates, Inc., 2013. URL https:
wttp:///wnnnnnnn.e.eeeeeeeeeeeee f
Curtis Northcutt, Lu Jiang, and Isaac Chuang. Confident learning: Estimating uncertainty in dataset labels. J. Artif. Int. Res., 70:1373 1411, May 2021a. ISSN 1076-9757. doi: 10. 10.1613/jair.1 1.1225..
1613 URL https: http://ioooogg////////oo / jair. 12125
Curtis G Northeutt, Anish Athalye, and Jonas Mueller. Pervasive label errors in test sets destabilize machine learning benchmarks. In Thirty-fifth Conference on Neural Information Processing Sys- tems Datasets and Benchmarks Track (Round 1), 2021b. URL bectpps:/epeereviiiie n
Corum??d=xxcDDxeNNLe
OpenAl.
gpt Gpt-4o-mini. 2024
Available
https://openat: http:pppenna...mcn.nneeeeeeeeeee wtt.:/cm//nini.eee
OpenAl, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad. Ilge Akkaya, Floren- cia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, Red Avila, Igor Babuschkin, Suchir Balaji. Valerie Balcom, Paul Baltescu, Haiming Bao, Moham- mad Bavarian, Jeff Belgum, Irwan Bello, Jake Berdine, Gabriel Sernadett-Shapirg Christopher Berner, Lenny Bogdonoff, Oleg Boiko, Madelaine Boyd, Anna-Luisa Brakman, Greg Brock- man, Tim Brooks, Miles Brundage, Kevin Button, Trevor Cai, Rosie Campbell, Andrew Cann, Brittany Carey, Chelsea Carlson, Rory Carmichael, Brooke Chan, Che Chang, Fotis Chantzis, Derek Chen, Sully Chen, Ruby Chen, Jason Chen, Mark Chen, Ben Chess, Chester Cho. Casey Chu, Hyung Won Chung, Dave Cummings, Jeremiah Currier, Yunxing Dai, Cory Decareaux, Thomas Degry, Noah Deutsch, Damien Deville, Arka Dhar, David Dohan, Steve Dowling, Sheila Dunning, Adrien Ecoffet, Atty Eleti, Tyna Eloundou, David Farhi, Liam Fedus, Niko Felix, Simón Posada Fishman, Juston Forte. Isabella Fulford, Leo Gao, Elie Georges, Christian Gib- son, Vik Goel, Tarun Gogineni, Gabriel Goh, Rapha Gontijo-Lopes, Jonathan Gordon, Morgan Grafstein, Scott Gray, Ryan Greene, Joshua Gross, Shixiang Shane Gu, Yufei Guo, Chris Hal- lacy, Jesse Han, Jeff Harris, Yuchen He, Mike Heaton, Johannes Heidecke, Chris Hesse, Alan Hickey, Wade Hickey. Peter Hoeschele, Brandon Houghton, Kenny Hsu, Shengli Hu, Xin Hu, Joost Huizinga, Shantanu Jain, Shawn Jain, Joanne Jang, Angela Jiang, Roger Jiang, Haozhun Jin, Denny Jin, Shino Jomoto, Billie Jonn, Heewoo Jun, Tomer Kaftan, Lukasz Kaiser, Ali Ka- mali, Ingmar Kanitscheider, Nitish Shirish Keskar, Tabarak Khan, Logan Kilpatrick, Jong Wook Kim, Christina Kim, Yongjik Kim, Jan Hendrík Kirchner, Jamie Kiros, Matt Knight, Daniel Kokotajlo, Łukasz Kondraciuk, Andrew Kondrich, Aris Konstantinidis, Kyle Kosic, Gretchen Krueger, Vishal Kuo, Michael Lampe, Ikai Lan, Teddy Lee, Jan Leike, Jade Leung, Daniel Levy, Chak Ming Li, Rachel Lim, Molly Lin, Stephanie Lin, Mateusz Litwin, Theresa Lopez, Ryan Lowe, Patricia Lue, Anna Makanju, Kim Malfacini, Sam Manning, Todor Markov, Yaniv Markovski, Bianca Martin, Katie Mayer, Andrew Mayne, Bob McGrew, Scott Mayer McKinney, Christine McLeavey, Paul McMillan, Jake McNeil, David Medina, Aalok Mchta, Jacob Menick, Luke Metz, Andrey Mishchenko, Pamela Mishkin, Vinnie Monaco, Evan Morikawa, Daniel Mossing, Tong Mu, Mira Murati, Oleg Murk, David Mély, Ashvin Nair, Reiichiro Nakano, Ra- jeev Nayak, Arvind Neelakantan, Richard Ngo. Hyeonwoo Noh, Long Ouyang, Cullen O' O'Keefe, Jakub Pachocki, Alex Paino, Joe Palermo, Ashley Pantuliano. Giambattista Parascandolo, Joel Parish, Emy Parparita, Alex Passos, Mikhail Pavlov, Andrew Peng, Adam Perelman, Filipe de Avila Belbute Peres, Michael Petrov, Henrique Ponde de Oliveira Pinto, Michael, Pokorny, Michelle Pokrass. Vitchyr H. Pong, Tolly Powell, Alethea Power, Boris Power, Elizabeth Prochl, Raul Puri, Alec Radford, Jack Rae, Aditya Ramesh, Cameron Raymond, Francis Real, Kendra Rimbach, Carl Ross, Bob Rotsted, Henri Roussez, Nick Ryder, Mario Saltarelli, Ted Sanders, Shibani Santurkar, Girish Sastry, Heather Schmidt, David Schnurr, John Schulman, Daniel Sel- sam, Kyla Sheppard, Toki Sherbakov, Jessica Shich, Sarah Shoker, Pranav Shyam. Szymon Sidor, Eric Sigler, Maddie Simens, Jordan Sitkin, Katarina Slama, Ian Sohl, Benjamin Sokolowsky, Yang Song, Natalie Staudacher, Felipe Petroski Such, Natalie Summers, Ilya Sutskever, Jie Tang, Nikolas Tezak, Madeleine B. Thompson, Phil Tillet, Amin Tootoonchian, Elizabeth Tseng, Pre- ston Tuggle, Nick Turley, Jemry Tworek, Juan Felipe Cerón Uribe, Andrea Vallone, Arun Vi- jayvergiya, Chelsea Voss, Carroll Wainwright, Justin Jay Wang, Alvin Wang, Ben Wang, Jonathan
13

---

Published as a conference paper at ICLR 2026
Ward, Jason Wei, CJ Weinmann, Akila Welihinda, Peter Welinder, Jiayi Weng, Lilian Weng, Matt Wiethoff, Dave Willner, Clemens Winter, Samuel Wolrich, Hannah Wong, Lauren Work- man, Sherwin Wu, Jeff Wu, Michael Wu, Kai Xiao, Tao Xu, Sarah Yoo, Kevin Yu, Qiming Yuan, Wojciech Zaremba, Rowan Zellers, Chong Zhang, Marvin Zhang, Shengjia Zhao, Tianhao Zheng, Juntang Zhuang, William Zhuk, and Barret Zoph. Gpt-4 technical report, 2024. URL
https://arxia /ttp:////axvvv//
Vicente Ordonez, Girish Kulkarni, and Tamara Berg. Im2text: Describing images using 1 mil- lion captioned photographs. In J. Taylor, R. Zemel, P. Bartlett, F. Pereira, and K.Q. Weinberger (eds.), Advances in Neural Information Processing Systems, volume 24. Curran Associates, Inc., 2011. URL htttps://proceedi wwtp:/seerrrrepppppppeeeeeeeeeee cctpppprrrpppppprppppppppppp.. semppr__flls_ files/ fileee///ll/////////////eeeeeeal http:////....aaoaoooeeeaaaaaaaaa Pappee
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang. Zachary Vito,o Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang. Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high-performance deep learning library. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Bue E. Fox, and R. Garnett (cds.). Advances in Neural Information Processing Systems, volume 32. Cur- ran Associates, Inc., 2019. URL UtL/tttss//ppooeeetiee (itee///flll//// /ttee///02....ooo222ffff///22222
n nwwripe wtwpiiiss.ccpppaeeeeeeeeeeeee ccppaper.ppppp..pppppppppp files/
pdf
Geoff Pleiss, Tianyi Zhang, Ethan Elenberg, and Kilian Q Weinberger. ing mislabeled data using the area under the margin ranking. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in Neural In- formation Processing Systems. volume 33, pp. 17044-17056. Curran Associates. Inc., 2020. URL. Itttps::/ ngs new.rnnerrriie httpps....pppeee cwtpsper_rpcpeeeeeepe fi fiw.filll/eeeeee files/paper/20000 fil
Identify-
Bryan A. Plummer, Liwei Wang, Chris M. Cervantes, Juan C. Caicedo, Julia Hockenmaier, and Svetlana Lazebnik. Flickr30k entities: Collecting region-to-phree correspondences for richer image-to-sentence models. In 2015 IEEE International Conference on Computer Vision (ICCV), doi: 10. 109/ICCV.00553000
pp. 2641-2649, 2015.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning. volume 139 of Proceedings of Machine Learning Research, pp. 8748-8763. PMLR, 18-24 Jul 2021. URL. https: sttps://proceed:: wwwpooceedinng.... ml www.s/ressvv... /v13s/ radford21.. html
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision (IJV),, 1 15(3):211-252 2015. doi: 10. 1007/s 1263-015-0816-y.
Fawaz Sammani and Luke Melas-Kyriaze Show, edit and tell: A framework for editing image cap- tions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2020.
Christoph Schuhmann, Richard Vencu, Romain Beaumont, Robert Kaczmarczyk, Clayton Mullis, Aarush Katta, Theo Coombes, Jenia Jitsev, and Aran Komatsuzaki. Laion-400m: Open dataset of clip-filtered 400 million image-text pairs, 2021. URL
02114
Swabha Swayamdipta, Roy Schwartz, Nicholas Lourie, Yizhong Wang, Hannaneh Hajishirzi, Noah A. Smith, and Yejin Choi. Dataset cartography: Mapping and diagnosing datasets with training dynamics. In Bonnie Webber, Trevor Cohn, Yulan He, and Yang Liu (eds.), Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing Online, November 2020. Association for Computational Linguis- tics. doi: 18.186 wtt.//65.1111200... 1/2020.emmlp-main 746. URL TRLhttps http:::///aaaaatllloroooooo org.20000.. eemn p ma i n. 7467]
(EMNLP), pp.
14

---

Published as a conference paper at ICLR 2026
Zhen Wang, Long Chen, Wenbo Ma, Guangxing Han, Yulei Niu, Jian Shao, and Jun Xiao. Explicit image caption editing. In Computer Vision ECCV 2022: 17th European Conference, Tel Aviy, Israel, October 23-27, 2022, Proceedings, Part XXXVI, pp. 113-129, Berlin, Heidelberg, 2022. doi: (0.1007/978-3-03122000 7. URL https:
Springer-Verlag Verlag. ISBN
//doi.org// 0.
-
Yuheng Zha, Yichi Yang. Ruichen Li, and Zhiting Hu. AlignScore: Evaluating factual consistency with a unified alignment function. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 11328-111111 11488, Toronto, Canada, July 2023. Association for Computational Linguistics. URL htapss//aclantthllee org/
2023 1023.acl-10 www.acllloon...... 634
Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning (still) requires rethinking generalization. Commun. ACM, 64(3):107-115, Febru- ary 2021. ISSN 0001-0782.
doi: 10. 10.1445/344677. URL /tt.:tts////0...rr/111....
3446776
Haoran Zhang, Aparna Balagopalan, Nassim Oufattole, Hyewon Jeong, Yan Wu, Jiacheng Zhu, and Marzych Ghassemi. LEMon: Label error detection using multimodal neighbors. In Forty-second International Conference on Machine Learning, 2025. URL
net./
Jinguo Zhu, Weiyun Wang, Zhe Chen, Zhaoyang Liu, Shenglong Ye, Lixin Gu, Hao Tian, Yuchen Duan, Weijie Su, Jie Shao, Zhangwei Gao, Erfei Cui, Xuehui Wang, Yue Cao, Yangzhou Liu, Xingguang Wei, Hongjie Zhang, Haomin Wang, Weiye Xu, Hao Li, Jiahao Wang, Nianchen Deng, Songze Li, Yinan He, Tan Jiang, Jiapeng Luo, Yi Wang, Conghui He, Botian Shi, Xingcheng Zhang, Wenqi Shao, Junjun He, Yingtong Xiong, Wenwen Qu, Peng Sun, Penglong Jiao, Han Lv, Lijun Wu. Kaipeng Zhang, Huipeng Deng. Jiaye Ge. Kai Chen, Limin Wang, Min Dou, Lewei Lu, Xizhou Zhu, Tong Lu, Dahua Lin, Yu Qiao, Jifeng Dai, and Wenhai Wang. In- ternv13: Exploring advanceé training and test-time recipes for open-source multimodal models, 2025. URL httpt::///// arxiv. org/ abs/ 2504.10477
Zhaowei Zhu, Zihao Dong, and Yang Liu. Detecting corrupted labels without training model to predict. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song. Csaba Szepesvari, Gang Niu, and Sivan Sabato (eds.), Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pp. 27412-27427. PMLR, 17-23 Jul 2022. URL htttps://procettt
mlr http://s.m.....eeee////////////e 2hwp11/22zhhu2
Andy Zou, Zifan Wang, Nicholas Carlini, Milad Nasr, J. Zico Kolter, and Matt Fredrikson. Universal and transferable adversarial attacks on aligned language models, 2023. URL phttps://arx:
org/abs/ 2307. 15043
A APPENDIX
A.1 CONTRIBUTION OF IMAGE-CAPTION ALIGNMENT AND CAPTION-CAPTION SIMILARITY METRICS
To isolate the contribution of each trajectory evaluation metric, we conduct ablation studies using TRACED with either the alignment score s or the semantic similarity score alone. Table reports the mean percent change in test accuracy when using one of the two metrics alone, relative to using both jointly.
Across all baselines, using either metric in isolation results in consistent and significant drop in performance. The alignment score s alone is much more informative, likely because a notable in- crease in alignment often signals an error in the original caption. In contrast, the semantic similarity score e is less useful on its own, as captions along the trajectory may differ substantially from the original, reducing its standalone discriminative power. However, combining s and consistently yields the best performance: captures the degree of alignment improvement, while e indicates whether that improvement involves substantial semantic change or only minor rephrasing.
15

---

Published as a conference paper at ICLR 2026
Table 4: Mean percent improvement in Test Acc. when using either s or alone in TRACED, compared to using both jointly. Experiments are conducted on MS-COCO using the Elimination algorithm. Results are averaged over seeds and all noise types (50% noise), with standard emross reported.
ALIGNMENT IMAGE-CAPTION
SIMILARITY CAPTION-CAPTION
МETHOD
s(x,
c(11,30)
BLIP (ITM)
-0.41 0.13 0.13
-6.03 0.11
BLIP (ITC)
-0.59 0.12
-10.18 0.16
LEMONOPT
-0.55 + 0.12
-7.22 0.17
LEMoNFIX
-0.41 0.21
-6.44
CLIIP
-0.00 + 0.04
0.19
A.2 EXPLORATION ALGORITHMS DETAILS
The Elimination algorithm iteratively removes the token whose deletion increases in alignment score the most, until no tokens remain.
Algorithm Elimination Algorithm
Input: initial caption x
Note (Wi,.... WL) with w wL the tokens in caption for i = to L do
2.(() <- (Wi, ......., W Wi+1.-... WL)
end for Output: (x(11, 140)
The Greedy Coordinate Descent (GCD) algorithm perturbs the caption by replacing individual to- kens. For each position, it selects top-K promising replacements based on the gradient of the align- ment score. subset of candidate captions is then generated by sampling token replacements at random.
Algorithm 3 Greedy Coordinate Descent (GCD)
Input: Initial caption
(Wi...., Wy),, image y. scoring function s, evaluation function e, number of candidates N, top- K promising replacements per position
Let V be the vocabulary, and e(v) the embedding of token v E for j 1 to L do
Compute top- K replacements for Wjo:
X, < Top-K { Velws,)osa s(x, 2)² (e(v) e(w,)) (
end for for
to N do Uniform (f1,...,I]) LT) Uniform(2)
w
(W1, W/ w/, Wi+l.-.... W1)
end for Output: (1) (N))
This algorithm is inspired by the GCD method proposed by Zou et al. (003), which was originally developed for adversarial attacks large language models. In work, adapt this approach for the purpose improving image captions.
The Fast GCD algorithm is Algorithm to identify the token position jo E [L] that most negatively impacts alignment. Gradient- based substitution is then restricted to this single position. Unlike full GCD, which randomly sam-
efficient alternative full GCD. It first applies the Elimination
16

---

Published as a conference paper at ICLR 2026
ples N captions from pool of K x L candidates (N < L).,, Fast GCD can exhaustively evaluate all K candidate replacements at position Jo- This approach enables to find better token substitutions using reduced number of forward passes through the alignment scoring function
Algorithm Fast Greedy Coordinate Descent (Fast GCD)
Initial caption æ (wr),-... Wr.), image y, scoring function 8, evaluation function e, top-K promising replacements per coordinate
Input:
Let V be the vocabulary and e(v) the embedding of token E V
Run Elimination Algorithm: {I((e.),,. ................................ x (ee,2)) < Elim (x) Select most promising coordinate: ju < arg max s(x V)
Compute top-K replacements for Wjo:
Loo & Top-K { (Ve(w))) s((z (e(v) e(ws,)) u vev
for w' 22 do
x.(w') (wi...., 30 1, w', 1jo+1...., wL)
end for
Output: (xlw²) w E X,0}
A.3 ADDITIONAL EXAMPLES CAPTION TRAJECTORIES
Figure iillustrates the behavior TRACED using the Elimination Algorithm on an example with correct caption.
Caption Trajectory
Removed word
1. vahicles light.
street
vohicles
near a traffiifftic trafficc light
3. vehicles on neer tratfic
Comeet Cartion: Vehicles street traffic light.
Alignmene copetor-image
Elimination Step
Elimination
Figure TRACED improves the BLIP-based image-caption alignment score (ITM) on an MS COCO example Lin et al. (2014), with minimal semantic change in the revised captions, suggesting the original pair is likely accurate.
We then show in Figures and the trajectories obtained with GCD and Fast GCD on the examples in Figures 3 and 6 respectively.
17

---

Published as a conference paper at ICLR 2026
Caption Trajectory
word
Trajactory
war
Allommett ceptersimap
1. man is standing in fromn of storefront
1. man staanding in front of brick storefiont wearing
no
nO
wearing
2. msnits standling in front brick starafiont waswing jacket
2. standing in tront brick storefront weeriing the jacket.
GCD
Similattt mplinot
Sinttartty Captannt Monantt-cppppee
his standing in atorefront
standiing in
some brick
sto storefront wearing Jacket
wearing thecost.
GCD step
Fast GCD Step
Figure 7: Caption trajectories using GCD (left) and Fast GCD (right) for the example Figure In both cases, TRACED identifies "no" as the source of misalignment and further improves the caption's 's alignment with the image.
Caption
Changed wond
Caption
Chanand wond
Trajectory
Trajectory
1
Afgrament captomuum
vehicies greanish light.
Aignmmnt
vehicles crossroads near green traffic light.
greeni
crossr
vehloles undemeath
2. vehloles street greenish light.
under
GCD Steg
Fast GCD
Similaity captionts-co camnnnts-capbbrii capbon)0)
green traffic light.
3. v cressroeds
3. vehicies on street approsching an traffic light.
which est
appro
undemeath green traffic
aching
GCD Step
OCD Step
Figure 8: Caption trajectories using GCD (left) and Fast GCD (right) for the example in Figure 6 In both cases, TRACED improves the caption's alignment with the image using only minor semantic edits.
A.4 NEW BENCHMARK DATASET CREATION
Prior work on error detection image captioning introduces noise via full caption swaps as a means of constructing evaluation benchmarks (Zhang et al. 2025). However, such swaps replace the entire caption, often resulting in text unrelated to the original. contrast, real-world annotation errors can be more subtle, with annotators correctly describing To better capture fine-grainn ine-grained noise, we propose a new approach for constructing a challenging benchmark by modifying only a few words within each caption. More specifically. for each original caption, we leverage a large language model to generate K 20 variants that maintain the same structure but introduce small semantic errors. The exact prompt is provided Appendix AA.
image but misrepresenting specific details.
While many generated options are useful, some may be too similar to the initial caption. To filter these, we apply Alignscore (Zha al.) (203), a factual consistency metric based on fine-tuned natural language inference model. Alignscore assigns low scores to captions that either omit key information or contradict the original caption. The selected variants thus differ meaningfully in content while remaining structurally close, effectively modeling fine-ge ene-grained semantic noise. To increase variability across seeds, we select the least-aligned sentences for each sample.
18

---

Published as a conference paper at ICLR 2026
Figure illustrates this generation and filtering process.
A man standing ladder.
Sentences
Alignscore between original and generated
generoted using
A.5 CAPTION CORRECTION HYPERPARAMETERS
got-4o-mini
sentence
Sentences
Scores
man is standing on a chair.
0.43
Figure the general caption correction framework used in this paper, where TRACED applied to improve the alignment metric of corrected sentences and fix the potential er- rors. The way trajectory information is pro- vided, however, can vary. In our experiments, we find that repeating the same prompt across multiple iterations often improves performance, as the model progressively refines the caption. We also observe that asking the model first to fix the caption without any guidance (to avoid bias from our signals) and then with our token- level error information appears to be the best strategy.
woman standing ladder.
man is standling lng on ladder
0.8
carefully.
A man is standing small ladder.
0.84
Only the top-2 sentences
kept
A man is standing on a chalrr.
woman standing ladder.
Figure 9: Fine-grained noise generation pipeline. Given an original caption, an LLM generates 20 variants. Alignscore then evaluates the factual consistency of each variant with respect to original. The least aligned (lowest-scoring) sen- tences are selected as fine-grained noisy captions.
The main hyperparameters of correction framework are:
The number of fixing steps: how many times the procedure of error detection followed by error correction is applied. At each step, new flagged words are obtained with TRACED.
The number of correction attempts per fixing step, denoted as k: at each fixing step, how many times the VLM is asked to revise the caption (using the same flagged words).
The correction strategy: the manner in which token-level information from TRACED is provided to the VLM.
In our experiments, we evaluate the following correction strategies:
Without TRACED:
the model is asked 1: times to correct the caption without any guidance
from our interpretable error detection framework. In this case, the prompt Figure [3 is used with [1] empty.
With TRACED only: the model receives token-level error information from our framework and is asked to fix the caption with this informtion & times. Here, the prompt in Figure is used with [1] containing guidance on the errors in the caption.
Without then with TRACED: the model performs k -- correction steps without token-level guidance ([1) empty), followed by final step where TRACED provides token-level error information ([1) filled).
Figure 10 highlights the impact of correction strategies. Supplying token-level error information to the VLM in direct, naive way can sometimes perform worse than providing no guidance all. In contrast, with an improved strategy, both the 1B and 14B models are able to leverage the information provided to generate captions that align more closely with the image. Interestingly, we find that only two prompts are often sufficient to consistently surpass the bascline which doesn't use TRACED. This is why we show results for two fixing iterations (Without-then-with TRACED (2) and Without TRACED only (2)) after five fixing steps in Figure 5
A.6 FINE-TUNING INTERNVL3-IB.
To construct a fine-tuning dataset for In3ern VL3-1B, 1B, we apply the correction procedure described in Section 3.4 on the train and validation sets of MS COCO and Flickr30k using Intern VL3-14L...BBB with 50% fine-grained noise. For MS COCO, we sub-sample the train set and keep the first 30% (27,594 samples).
19

---

Published as a conference paper at ICLR 2026
InternVL3-14B (with CoT)
InternVL3-18
InternVL3-14B
BLIP Alighment Score
Results on COCO
0.7
Correction Strategy
0.65
Without and then with TRACED
Without and then with TRACED (3)
with TRACED
Without then TRAGED
0.5
TRACED only
With TRACED only
Fixing Step
Fixing Step
Fixing Step
With TRACED only (3)
With TRACED only
BLIP Alighment Score
Results on Flickr30k
With TRACED only
Without TRACED
0.6
Without TRACED only (2)
Without TRACED only (3)
0.1
Without TRACED only (5)
5
Fixing Step
Fixing Step
Fixing Step
Figure 10: Caption correction results for InternVL3-11 and Intern VL3-14B (with and without CoT prompting) across fixing steps. We compare three correction strategies: (1) without TRACED, (2) with TRACED only, and (3) without TRACED for k iterations followed by one iteration with TRACED. Numbers in parentheses indicate the number of times k the VLM is asked fix the caption at each fixing step. BLIP alignment scores (ITM block) on the corrected sentences are averaged over 3 seeds with standard errors.
For each dataset and seed, Intern V is trained using cross-entropy loss (next token prediction) for five epochs, and the best checkpoint selected based on validation correction performance (measured the number of errors detected by TRACEDD-LLIP-ITM, The hyperparameters (learning rate, optimizer, etc.) as in the original VL. paper were used (Zhu et al. 2025) This model is denoted Intern 1I---BBFT.
A.7 PARALLELIZATION BENEFITS
TRACED applies the trajectory creation and evaluation from Algorithm independently to each sentence in the dataset. This enables efficient large-scale parallelization. Our method benefits from both intra-GPU and multi-GPu parallelism: given access to GPUs, the dataset can be split into subsets processed in parallel, with each GPU handling batches of samples.
A.8 COMPUTATION OVERHEAD
Table 5 reports the computation time required by TRACED and the original baselines to process 1,000 image-caption pairs on single L40 GPU. Baseline models such as BLIP, LEMoN, and CLIP are very fast as they require only a single forward pass per pair. Despite performing multiple model evaluations construct trajectories, all variants of TRACED, including Elimination, Fast GCD, and GCD, remain practical and scalable. Among the proposed methods, Elimination is the most efficient, offering substantial speed advantages while maintaining among best performance. Fast GCD achieves strong balance between speed and trajectory quality. For example, when scaled to 1,000,000
pairs using BLIP (ITM), the most expensive baseline, and 4 L40 GPUs, Elimination completes in approximately 6.5 hours and Fast GCD takes about 2.6 days.
As described Appendix A.7 thanks to the high degree of parallelism in our method, leveraging more GPUs substantially ly further reduce total processing time.
We want to emphasize that TRACED needs to applied only once to identify and filter out incor- rect image-caption pairs. This one-time computational cost is reasonable for generating cleaner dataset that can be reused across various downstream tasks, including pre-training, fine-tunin fine-tuning, and evaluation.
20

---

Published as a conference paper at ICLR 2026
Table Computation time comparison across algorithms. Reported times (in seconds) corresponds to the duration required to process 1,000 sentences with a single L40 GPU, including both trajectory exploration and alignment score evaluation.
METHOD
ALGORITHM
COMPUTATION TIME (8)
BLIP (ITM)
1.82:00111 92.53 + 1.16
ELIMINATION FAST GCD
TRACED- BLIP (ITM)
905.22 0.99
GCD
1617.06 0.13
BLIP (ITM)
0.24
ELIMINATION
49.05 = 0.28
TRACED- BLIP (ITM)
FAST GCD
389.19 0.51
GCD
688.90 = 0.55
LEMONOPT
3.13 0.08
ELIMINATION FAST GCD
43.77 + 0.59
TRACED-
451.28 + 0.43
LEMONOPT
GCD
799.03 1.79
LEMoNFIX
+.19
ELIMINATION
43.44 + 0.44
TRACED-
FAST GCD
452.48 = 1.07
LEMoNAiX
GCD
802.13 + 1.76
CLIP
2,40 0.07
ELIMINATION FAST GCD
43.10 0.57
TRACED-
444.80 + 0.55
CLIP
GCD
788.97 0.29
A.9. HYPERPARAMETERS
Trajectory Generation Hyperparameters. Depending on the exploration strategy, the caption trajectory generation from Seporittm[iivolllee few hyperparameters:
Elimination Algorithm: We set T L and N -ات11 where L the caption length. The algo- rithm removes one token at a time, selecting the one whose removal most improves the alignment score 8, and continues until there no token in the sentence.
GCD Algorithm: We use 7 = 10, K = 128, and N = 256.
10, k 128 and promising replacements for the single token identified via Elimination Algorithm.
Fast GCD Algorithm: We
K 128 since explore all K
Grid Searches. The hyperparameter grids used for model selection are as follows:
XGBoost hyperparameters:
max_depth E (3, 4,
learning.rate (0.01, 0.05, 0.1, 0.5)
n_estimators (50, 100, 200, 400)
CART hyperparameters:
max.depth E {1,5, 10, +00}
A.10 RESULTS PER NOISE TYPE
We present Table 6 the impact of TRACED on various baselines across the three noise types evaluate. TRACED consistently improves performance across all baselines and noise settings. Notably, the gains are more substantial for noise types that are harder to detect. For example, improvements modest for random noise, where baselines already achieve over 97% Accuracy
21

---

Published as a conference paper at ICLR 2026
Flickr30k and MS COCO. On the contrary, improvements much pronounced the Fine- Grained noise and on MM-IMDb, which present more challenging errors for the existing methods.
Table Comparison of TRACED with baselines. "Elim" " and "FGCD" denote Elimination and Fast GCD, respectively. Results averaged over 3 seeds for each noise type (50% noise). We report mean accuracy and mean accuracy improvement compared to the baseline, with standard errors.
RANDOM
NOUN
FINE-GRAINED
DATASET
METHOD
ALG.
Acc. (%)
IMPROVE-
Acc. (%)
IMPROVE-
Acc. (%)
IMPROVE- (%)
(%)
(%)
98.2
93.8
3.3
TRACED- BLIP (ITM)
FGCD
98.3 0.1
0.5 0.0
93.8 = 0.3
0.3 0.1
75.4 + 0.4
1.7 + 0.7
GCD
98.1
0.2
0.3 0.1
93.6 + 0.3
0.0
74.5 0.5
0.5 0.5 0.7
BLIP (TT)
9
0.0
93.6 0.3
0.0
74.2 +4.200..
0.0
ELIM
97.8
0.1
93.4 0.2
1.1
0.4
0.3
TRACED-
BLIP (ITC)
FGCD
97.9 0.1
0.2
93.0 0.3
0.2
0.3
1.9
GCD
97.8 0.1
0.1
93.1 0.5
0.6
7.00
1.5
BLIP (ITC)
97.7 0.1
0.0
92.5
0.0
72.0
0.0
ELIM
97.5 0.1
0.0
90.8
0.1
1.4
68.5
3.6
FLICKR
TRACED- LEMONOFT
FGCD
97.5 0.1
-0.0
89.7 +
0.6
6.9
5.0
GCD
97.3 0.0
-.2 0.1
90.0 + 0.3
0.9
69.8 1.0
5.5
LEMONorT
97.5 0.1
89.2 0.3
0.0
6..1
0.0
TRACED-
ELIM
97.7 = 0.1
0.5
89.7 1
0.2
6..7
1.1
LEMONAI
FGCD
97.1 0.2
-0.1
89.5 0.3
0.2
6.6
5.1
97.0 0.2
-0.1 0.2
90.0
7.2
LEMONFIX
97.2 0.0
0.0 0.0
89.3
0.0
0.6
0.0
TRACED-
ELIM
97.6 0.1
0.4 0.1
90.7 = 0.2
1.6
6..
6.2
CLIP
FGCD
97.3 0.1
0.1 0.1
8944
0.1
0..
7.5 0.7
GCD
89.9
CLIP
97.2 0.1
0.0
89.3 + 0.3
0.0 0.0
64.8 0.6
0.0
TRACED-
ELIM
81.4 0.3
1,3 0.1
76.6 - 0.3
0.5
LEMoNoPT
FGCD
80.6 0.3
0.3 0.1
75.5 + 0.3
0.1
0.2
75.8
0.4
LEMoNorr
80.3 0.3
0.0 0.0
75.4
0.0
ELIM
80.9 0.2
2.0 0.1
75.7
2.8
TRACED-
MM- IMDB
LEMONHIX
FGCD
80.0 0.1
0.1
74.4
L.
GCD
80. 0.I
1,0
0.3
1.4
LEMONFIX
79.3 0.2
0.0 0.0
73.6 0.3
0.0
TRACED-
ELIM
80.9 O.3
1.1
76.1 0.2
2.1
CLIP
80. 0.2
0.5
7..9
0.2
GCD
80.3 0.1
0.7 0.2
7.1
1.2
CLIP
79.7 0.3
0.0 0.0
74.6 0.2
0.0
TRACED- BLIP (ITM)
ELIM
98.9 0.2
0.4 0.0
92.1
0.5
8..4
4.4
98.8
92.1
0.0
2.3
GCD
98.9 0.1
0.4 0.1
92.1
0.1
0.3
1.5
BLIP (ITM)
98.5 0.1
0.0 0.0
91.7 0.1
0.0
0.0
7.2
0.0
ELIM
98.7 0.0
0.1
9..8
0.9
76.8
4.4
TRACED- (ITC)
FGCD
98.5 0.1
0.1
0.1
7.3
4.1
GCD
98.6 0.1
0.1 0.1
9.2
0.2
0..
2.6
BLIP (ITC)
98.5 0.1
0.0 0.0
0.0
0.0
7..6
0.0
TRACED- LEMoNorT
ELIM
97.8 0.0
0.1
86.3
1.4
7..8
3.2
MS-
FGCD
97.7 0.1
0.0 0.1
85.5 0.4
0.4
7..4
2.2
COCO
GCD
97.7 0.1
-0.0 0.1
85.2
0.1
7.6
3.1
LEMONorT
97.7 0.1
0.0 0.0
85.1 + 0.4
0.0 0.0
6..
0.0
97.7 O.T
0.0
85.6
0.9
69.6
3.9
TRACED-
LEMONFIX
FGCD
97.8 0.1
0.1 0.0
84.6 0.3
1.8 0.3
6.73
0.6
GCD
97.7 = 0.1
0.1
84.2
1.3
6..7
3.9
LEMONEX
97.7
0.0
83.1 0.5
0.0
6.61
0.0
97.7 0.1
0.2100
85.8 0.2
2.3
69.9
0.2
TRACED-
CLIP
FGCD
97.5 0.1
0..0
84.6 0.1
0.3
69.0
0.6
GCD
97.6 0.1
0.1
84.4
0.2
0.7
69.3
4.0
CLIP
6.36
0.0
A.11 PERFORMANCE OF TRACED ACROSS NOISE LEVELS
The main results of the paper (Table I and Table 6 are reported for noise ratio 50%. In this section, we evaluate the performance TRACED across additional noise levels-10% 20%, 30%, and 40%. Because these settings introduce class imbalance, we report Test AUC scores in Table Across all noise levels, TRACED consistently outperforms the baselines. As seen previously, the
22

---

Published as a conference paper at ICLR 2026
Noise type
Method
10%
20%
30%
40%
TRACED BLIP (ITM)
83.5 1.5
82.6 = 0.8
84.0 1.3
84.4
BLIP (TTM)
80.8 1.0
79.1 + 0.6
8.1
8.74
Fine-grained
BRACED-LEMoNON
76.4
75.4 2.1
7.70
7.
LEMoNOFT
7.1
70.7
72.1 7.8
TRACED-CLIP
76.0 1.8 7.
7.04
75.8 -
CLIP
7.5
7..0
1.3
7.2
TRACED-BLIP (ITM)
99.1 99.1 0.3
08.8=022
98.5 9.2 0.2
BLIP (ITM)
99.0 0.3
98.8 0.2
98.
9.2
Noun
TRACED-LEMONOTT
96.2
95.9
0.3
0.2
LEMoNOPT
95.7 0.5
95.3
94.5
TRACED-CLIP
96.4 0.3
96.3 + 0.4
96.2
95.5 0.2
CLIP
95.5 0.3
95.6 0.7
0.3
94.7
TRACED-BIE -LIP (ITM)
99.7
99.6 0.1
99.8
99.8 0.1 99.7
BLIP (ITM)
99.6 0.3
99.7 0.1
99.7
Random
TRACED-LEMONORT
99.5 1 0.1
99.6 0.2
9..
99.5 0.1 9.14
LEMoNoPT
99.4 0.1
99.5 = 0.2
0.3
TRACED-CLIP
99.3 0.2
99.6 = 0.1
9.2
99.6 = 0.1 99.5 = 0.1
CLIP
99.3 + 0.2
99.5 = 0.2
99.5
Table 7: Comparison of TRACED, using the Elimination Algorithm, against baseline models across noise levels (10%, 20%, 30%, and 40%) Flickr30k. Test AUCs averaged over three seeds with standard errors.
gains are especially substantial under fine-grained noise, where baseline methods tend to struggle the most.
A.12 DOWNSTREAM CAPTIONING PERFORMANCE
In this section, we evaluate the impact of our filtering and correction procedure on the downstream performance of captioning model, BLIP-2 Li et al. (2023). particular, we fine-tune BLIP-2 on (i) the Flickr30k datasets containing 50% fine-grained noise and (ii) cleaned datasets obtained by removing sentences flagged as incorrect by BLIP (the strongest baseline) by TRACED-BLIP. In addition to filtering, we also test the correction strategy, where sentences predicted as wrong by TRACED-BLIP are replaced with corrected versions produced either by Intern alone or using the token-level correction procedure described in Section 3.4.
We fine-tune BLIP-2 using LoRA (rank = 4) and report BLEU-4, ROUGE, and CIDEr scores. We train the models for 3 epochs, with learning rate of le-5, AdamW and using early stopping. The table below summarizes the results:
Method
BLEU-4 (%)
ROUGE (%)
CIDEr (%)
Noisy
31.5 0.2
56.5 - 0.1
69.2 0.4
Filtering with BLIP
30.8
56.1 0.3
67.9 + 1.3
Filtering with TRACED-BLIP Ideal Filtering
31.
56.6 0.1
69.6 0.2
3.1
56.8 = 0.1
70.2 0.4
Correction with Intern VL3-1B
32.0
56.8 0.0
70.2 0.4
Correction with Intern VL3-1B + TRACED
32.5 0.1
56.9 = 0.1
70.4 + 0.5
Table Impact of filtering and correction, using either the baseline TRACED, on the downstream
caption quality of BLIP-2. In both correction settings, corrections are applied the same samples predicted as erroncous by TRACED-BLAN isolating the effect of TRACED 's interpretable tokens in the correction pipeline.
23

---

Published as a conference paper at ICLR 2026
Filtering with TRACED-BLIP achieves performance very close to Ideal Filtering and substantially outperforms the baseline (BLIP-filae filterrii across all metrics. Applying caption correction to the samples flagged erroneous by TRACED-BLIP yields additional gains in BLEU-4, ROUGE, and CIDEr, and incorporating TRACED 's interpretable tokens into the correction process further boosts downstream caption quality. Overall, our TRACED- correction pipeline improves BL.IP-2's captioning performance by to I point in BLEU-4, 0.4 in ROUGE, and 1.2 in CIDEr compared to training on the noisy dataset.
A.13 IMPACT OF N AND T ON TRACED
Algorithm relies on two key hyperparameters: N, the number of candidate edits considered at each step, and T, the trajectory length. In this section, we analyze the sensitivity of TRACED to these hyperparameters.
Impact of N. For the Elimination algorithm. N inherently small: by construction it matches the number of tokens in the caption and decreases by one each iteration, making exploration progressively faster. Consequently, we focus our analysis on the role of in the GCD algorithm, and report in Figure 11 the impact of varying N on performance on Flickr30k.
0.91
0.87
0.90
0.89
0.86
0.85
0 0.88
0.85
Test
0.84
0.84
0.87
0.83
32
64
128
296
512
32
128
256
512
32
64
128
296
512
N.
N
TRACED-BLIP (TTM)
'RACHDDLEMONOPT
TRACED-CLIP
Figure 11: Impact of varying N on the performance of TRACED when applied to BLIP (TTM), LEMONOPT, and CLIP using the GCD algorithm. Mcan accuracies Flickr30k over 3 seeds and noise types (noun, random and fine-grained at 50% noise), are reported with standard errors.
Impact of T. We evaluated the influence of trajectory length to determine how much information gained from longer versus shorter paths. We used the Elimination algorithm to conduct this experiment. The results on Flickr30k are presented in Figure 12 We observe that larger values of T generally lead to improved performance, particularly for CLIP and LEMoNOPP. However, a single step of Elimination (I' = 1) is often sufficient to achieve near-optimal, and sometimes even optimal, performance. This behavior can be explained by the fact that in many cases, only one or a few exploratory steps required for procedure identify incorrect token along the caption trajectory (see Figure] and therefore determine whether the caption is correct or not.
0.91
0.87
AiC
0.85
0.89
0.85
0.85
0.88
0.84
0.83
0.82
0.82
TRACED-BLE (ITM)
TRACED-LEMONOFI
TRACED-CLIP
Impact of varying T the performance of TRACED (Elimination algorithm) when applied to BLIP (ITM), LEMONOPT, and CLIP. Mean accuracies on Flickr30k over seeds and noise types (noun, random and fine-grained 50% noise), reported with standard emror..
Figure 12:
A.14 IMPORTANCE OF THE SEMANTIC SIMILARITY SCORE
We additionally study the impact of the semantic similarity function e on the performance of TRACED. For BLIP (ITM), compared using the ITC component (as described in Section 4.3)
24

---

Published as a conference paper at ICLR 2026
with using CLIP. For both LEMoNopr and CLIP, performed the other substitution and evalu- ated whether replacing CLIP with BLIP (ITC) affected performance. The resulting comparisons are provided below.
Noise Type
c= CLIP
BLIP (ITC)
fine-grained
76.0 0.2
7.3
BLIP (TTM)
93.7 + 0.3
933
random
98.0 - 0.1
9.1
fine-granee aineggrained
68.5 1.0
6.02
LEMoNorT
noun
90.8 = 0.1
9.2
random
97.5 0.1
97.4
fine -ne-grained
68.9 = 0.4
70.0 0.2
CLIP
noun random
90.7 0.2
90.9 + 0.3
97.6 0.1
97.5 + 0.3
Table 9: Impact of changing e on the performance of TRACED. Mean accuracies on Flickr30k for each noise type are computed over seeds with standard errors.
While the choice of BLIP (ITC) seems preferable for BLIP (TTM), the choice of does not seem to have a decisive impact on the final performance, and both CLIP and BLIP (ITC) appear to be effective metrics e for error-detection.
A.15 PERFORMANCE OF TRACED ACROSS CAPTION LENGTH
To assess whether the effect of TRACED on the baseline depends caption length, we compute test accuracies for both TRACED and the baselines across caption-length bins on MS COCO. The corresponding results are shown in Table I0 We find that TRACED consistently improves upon the different baselines across all caption-length ranges, indicating that the trajectory-guidee approach is beneficial regardless of the caption length.
Noise type
Method
(6.999, 9.0]
(9.0, 10.0]
(10.0. 11.0]
(11.0, 14.0]
(14.0, 34.0]
TRACED-BLIP (TTM)
80.5 + 0.5
80.9 + 1.2
80.1 + 0.5
80.2 0.2
79.1 + 0.4
BLIP
76.5 4: 0.3
78.3 4 0.9
76.8 - 0.7
77.8 0.4
74.4 + 2.0
Fine-grained
TRACED--LMMONor
70.5 + 0.9
70.4 0.9
71.1 + 0.5
71.2 0.5
71.3 1.5
LEMON_OPT
68.6
68.5 1.0 1.0
68.0 + 1.3
69.6 1.0
68.1
TRACED-CLIP
70.3
68.9
69.8
0.6
68.5
CL.IP
66.7 0.4
66.8 0.9
65.0 0.8
68.1 0.3
65.4 2.0
TRACED-BLIP (ITM)
91.9 0.3
92.6 12.60000
92.6 0.1
91.6
91.7 + 0.6 0.9
BLIP
91.7
91.8 0.1
92.1 0.2
0.4
Noun
86.0 - 0.5
86.3 0.7
86.4 0.8
86.6
87.8 1.2
LEMON OPT
84.1 1 0.3
85.4
85.6 0.7
85.8 1.8400 0.2
88.3 + 1.1
TRACED-CLIP
85.6 1 0.3
86.2
85.6 + 0.7
85.3 0.0 8
87.3 + 2.1
CLIP
83.1 + 0.2
83.9 03.9 0.7
84.3 1 0.4
86.0 + 1.2
TRACED-BLE (ITM)
98.8 9.0
99.1
9.2
98.5 + 0.5
BLIP
9.1
98.7
98.1
98.5
98.5
Random
TRACED-LEMONoRT
9.1
97.8 0.1
98.7 9..4
LEMON_OPE
9.2
9..1
97.7
97.5
08.0
TRACED-CLIP
07.7
08.2
97.6 0.2
9.2
98.0 + 0.2
CLIP
97.5
9.1
97.4 0.1
97.2
98.0 0.2
Table 10: Comparison of TRACED, using the Elimination Algorithm, with baseline models across caption-length bins MS COCO. The bins correspond to the (0%, 25%], (25%, 50%], (50%. 75%]. (75%, 95%], and (95%, 100%] percentiles caption length. For each bin, test accuracies are averaged over three seeds, with standard errors reported.
25

---

Published as a conference paper at ICLR 2026
A.16 EXAMPLES ERRORS WITH FINE-GRAINED NOISE
We show below list of examples of noisy captions generated described in A.4
Wrong object/clement
Wrong action
True: black and white dog is running through the grass. Noisy: A black and white cat is running through the grass.
True Girl wearing blue shirt and black shorts plays trumped out- side.
The girl wearing blue shirt and black shorts sings out- side.
Noisy:
True: Several students waiting
outside an igloo.
Noisy: Several students waiting outside car.
True: One man holds another man's head down and prepares punch face.
True: man in white shirt stands high up on scaffolding. Noisy: A woman in red shirt stands high up scaffolding.
Noisy: One man holds another man's head down while his friends cheer him
True: A group of people are hik- ing up icy hillside. Noisy: group of people ski- ing down an icy hillside.
Wrong negation
True: Three people sitting an outside pienie hench with an umbrella.
Noisy:
Two people are sitting inside pienic table without an brella.
Wrong number
True: A group of spectators watch men's sand volleyball game.
True: A man is standing in front of brick storefront wearing black jacket.
Noisy: solitary spectator watches a men's sand volleyball game.
Noisy: A man standing in front brick storefront wearing no jacket.
True: Shaft of light a cave shows three spelunkers. Noisy: A shaft of light in cave reveals spelunker.
True:
A playing acous- guitar and singing with a group of people behind him including a woman who is singing along
True: Person standing on rocky edge of water with hilly land in background.
not playing any instrument and speaking with a group people behind him in- cluding a woman who is taking notes.
Noisy:
Noisy: A group of people standing on the sandy beach with flat land the background.
Table 11:
Examples true versus noisy captions with fine- mne-ggrined noise from our new benchmark dataset, derived from Flickr30k, illustrating errors in the objects, negations, actions and quantities.
26

---

Published as a conference paper at ICLR 2026
Wrong object/element
Wrong action
True: guy that is riding his bike
True Little girl smiles for the
train Noisy: A guy that is riding his bike next to a bus.
she her sandwich Noisy: Little girl smiles for camera as she drinks her juice.
True: An old black and white photo of Pennsylvania Avenue. Noisy: An old black and white photo Fifth Avenue.
True:
couple riding motorey- ele down street.
A couple walking down street.
Noisy:
True: a giraffe bending down to drink from body of water
True: cat that is eating some kind of banana
Noisy: A giraffe stretching its neck to from tree.
Noisy: dog that cating some kind of banana.
Wrong negation
Wrong number
True: Many people gather around building with clocks
True: photo of a train heading down the tracks
Noisy: Many people gather
Noisy: photo multiple trains heading down the tracks.
around a building without clocks.
True: Two friends are skatehoard- ing down the street their next destination.
Noisy: Three friends are skate- boarding down the street to their next destination.
True: pencil is sitting on a ruler with pair of scissors.
Noisy: A pencil is sitting ruler without any scissors.
True: group three people talk-
ing to each other on sidewalk with a skateboard
True: pole that has sign on it
Noisy: A pole that has sign
Noisy: A group of four people talking each other the side- walk with bicycle.
it.
Table 12: Examples of true versus noisy captions with fine-grained noise from our new benchmark dataset, derived from MS COCO, illustrating emrors in the objects, negations, actions and quantities.
27

---

Published as a conference paper at ICLR 2026
Wrong color
Distributed errors
A boy wearing or- ange Doritos jersey jumps up the
True:
True:
A bride in a light pink dress poses for a picture with male relatives and is be- ing photographed by a man in a cream shirt with white pants.
Noisy: A girl wearing a blue Doritos jersey jumps up in the air.
Noisy: A bride in light pink dress poses for a picture with her siblings and is being photographed by girl in green shirt with a floral skirt.
True: A band is of four members includ- ing woman and three men playing their instruments with an open guitar case front them.
True: A brown dog about to catch green Frisbee. Noisy: A black dog about to eatch Frisbee.
Noisy: An orchestra of ten members in- cluding woman and nine men are playing their instruments with open violin case in front of them.
True: A man in brown building raft.
Noisy: man in red building a raft.
True: A young man in a gray tee-shirt and gray sweatpants stands by metal tiered shelf in an industrial kitchen, holding the
top edge of the metal structure, with
leg resting on knee the other leg.
Noisy: A young woman in a blue tee-shirt
and black sweatpants stands by a metal
tiered shelf in industrial kitchen, hold- ing the bottom edge of the metal struc- ture, with one leg resting on the knee of the other leg.
Table 13: Examples of true versus noisy captions with fine- fine-grained noise from our new benchmark dataset, derived from Flickr30k, illustrating color errors and distributed errors.
Wrong color
Distributed errors
A young man wear- ing black attire and flow- ered tie is standing smiling-
True:
True: Woman taking picture of someone standing behind sculpture and child p ushing another woman towards the sculp- ture.
Noisy: A young man wearing blue attire and a flowered tie is standing and smiling.
Noisy: A woman taking picture of dog standing behind sculpture and child pushing another sculpture.
away from the over body water. Noisy: A flock of ravens that are cawing together above forest.
True: Several birds that are flying together
True: A small blue plane sitting on top of field. Noisy: large red plane sitting top of field.
True: white flowers in a
with arranged leaves Noisy: Red flowers in a vase with arranged leaves.
True: woman holding blue birthday
cake with stars and candles on an- other woman in front of the cake. Noisy: A woman holding green birthday cake with stars and candles and other woman in front cake.
Table 14: Examples true versus noisy captions with fine- mne-grained noise from our new benchmark dataset, derived from MS COCO, illustrating color errors and distributed errors.
28

---

Published as a conference paper at ICLR 2026
A.17 PROMPT FOR CAPTION CORRECTION
We present the prompts used for caption correction in Section 3.4 To isolate the effect of TRACED, employ identical prompts across settings, with the only modification being the addition of sentence specific to our framework ([1] in Figures 13 and 14). For instance, in Figure [1] is replaced with: "The following words are wrong: aa 'player. 'female'. 'a', 'rack', 'is". 'the'. Other words might be wrong too."
<limage|>
### Task:
You are an expert caption editor.
Please check the caption factual or accuracy. [1]
caption inaccurate,
using only original or minimal substitutions
**Aooid adding new details descriptions. Keep the revised caption short and concise."
the caption is accurate, return exactly as is. Don't make any changes.
Respond with only the corrected caption, enclosed quotation marks.
Caption:
'A female tennis player the court with racket."
### Corrected Caption:
Figure 13: Prompt used for caption correction. In the case of TRACED, token-level error information provided in [1].
<limage|>> Task:
an expert caption editor.
Please check the caption for factual or visual accuracy. [1]
the caption is inaccurate, rewrite using only the original words or minimal substitutions to make it accurate.
*Avoii adding new details or descriptions. Keep the revised caption short and concise,*A
the caption accurate,
exactly Don't make any changes.
Think step by step, explain why there one multiple errors, and then provide the corrected caption, enclosed in quotation marks, Don't output anything after the corrected sentence.
#*# Caption:
"A female player
with racket."
Figure 14:
Prompt used for caption correction using Chain-of- Chain-of-Thought prompting. In the case of TRACED, token-level error information is provided in [1].
29

---

Published as a conference paper at ICLR 2026
A.18 PROMPT FOR
NOISE TYPE
We use the prompts in Figures 15 and 16 to generate 20 candidate noisy captions for each caption in the MS COCO (Lin et al. 2014) and Flickr30k (Plummer et al. 2015) datasets.
Sentence Variation Generstor
given input sentence, generate up to 20 variations that have similar structure but convey clearly diffferent meanings.
Follow these systematic modification
## Analysis Requirements
1. First, identify the basic structure of the sentence
2. Identify all components: subject, predicate, object (if any), attributives (if any), adverbials (if any), and clauses (if any)
3. Create variations by modifying one or two components per variation
Component Modification Guidelines
### Subject Modifications (1-2 variations)
Change the quantity the subject: e.g., "Aman" "Two men': 'A group of people" + "One person" Change the subject itself: e.g,, man' + "A woman"; 'A person" "An animal"; "A group of students" "A group of police
officers"
"Examplas:**
Original:
patient carefully."
Variation: "The examnined the patient carefully. (Changed subject identity)
Variation: "Several doctors examined the patient carefully." (Changed subject quantity)
#*# Predicate Modifications (1-2 variations)
Replace the verb with an unrelated verb: e.g., "standing" "sitting"; "waving" + "running"
Ensure object present) is also modified to new context
v Examppess:**
Original: "The chef prepared a delicious meal for the guests."
Variation: "The chef served delicious meal for the guests." (Changed verb)
Variation: "The chef ruined delicious meal for the guests." (Changed verb to opposite meaning)
нмН Object Modifications (1-2 variations)
Replace noun object with different noun: ensure
context significantly original
object complement, modify expresa opposite completely different meaning
"Examplss:**
Original: "She bought a new car with her bonus."
Variation: "She bought a new house with her bonus." (Changed object noun)
Variation: "She bought an old car with her bonus.' (Changed object attribute to opposite)
Attributive Modifications (1-2 variations)
serving attributives, replace contextually appropriate words that convey completely different meanings
For adjectives
For numerical attributives, change the quantity
For prepositional phrases or infinitives, modify to maintain context while expressing significantyy different meaning
*Eaamplas:**
Original: building corner was recently renovated."
recently renovated." (Changed attributive adjective) Variation: "The tall building in the downtown
Variation: 'The historic building the
recently renovated." (Changed attributive prepositional phrase)
Figure 15: First part of the prompt used to create the fine- grained noise using gpt-4o-ming
A. 19
USE OF LLM
LLMs were used polish the writing of this paper.
30

---

Published as a conference paper at ICLR 2026
heh Adverbial Modifications (1-2 variations)
For time and place adverbials, change complstely different times locations
For manner and degree adverbials, change the adverb to its antonym or to complately different adverb
For reason, result, condition adverbials, modify the corresponding clause
"Examples:**
Original: "They quickly finished their homework before dinner."
Variation: "They slowly tinished their homework before dinner." (Changed manner adverbial to opposite)
Variation: "They quickly finished their homework after midnight." (Changed time adverbial)
#*# Clause Modifications (1-2 variations)
Identify the components within the clause modify them according to the guidelines above
v "Eamppess***
Original: "She said that she would
the party she finished her work."
Variation: "She said that she would skip the party if she finished her work." (Changed predicate in the clause)
Variation:
"She said that she would come to the party unless she finished her work." (Changed condition in the adverbial
clause)
Important Requirements
1. Each variation should differ from the original in 1-2 components only
2. Modifications must be significant enough to clearly change the meaning of the sentence
3. The modified sentence must maintain grammatical correctness and contextual coherence
4. If the original sentence is too short to generate 20 variations, provide as many as reasonably possible
5. Consider the context of the sentence and ensure modifications are contextually appropriate
Number each variation sequentially (1-20)
Output Format
1. [Modified sentence
2. [Modified sentence
20. [Modified sentence 20]
Original:
(sentence)
Figure 16: Second part of the prompt used to create the fine-grained noise using gpt-4o-mini.
31


Reviews:
[{"review_id": "EBhCbPlpSU", "reviewer": "Reviewer_QGRZ", "summary": "This paper proposes TRACED (Trajectory Creation for Error Detection), a framework for detecting errors in image-caption datasets. Unlike existing methods that rely on a single similarity score, TRACED iteratively edits captions to create a \"trajectory\" and analyzes how alignment scores and semantic similarity evolve. The key insight is that correct captions stabilize after minor edits, while erroneous captions show substantial improvement potential.\n\n**Main contributions:**\n1. A novel trajectory-based approach using token deletion (Elimination) and replacement (GCD, Fast GCD)\n2. A model-agnostic framework applicable to existing methods (CLIP, LEMoN, BLIP)\n3. Introduction of fine-grained noise generated by GPT-4o-mini, more realistic than random/noun swaps\n4. Up to 2.8% AUC improvement on MS COCO, Flickr30k, and MM-IMDb\n5. Token-level error localization for interpretability and caption correction, achieving up to 14.5% alignment score improvement", "questions": "1. Can you provide diverse examples showing different error types (object, color, number, relationship, distributed errors) and failure cases beyond the repeated \"no jacket\" example?\n\n2. How did you validate that GPT-4o-mini generated noise reflects real annotation errors? Can you compare against datasets with actual annotation errors (e.g., Conceptual Captions)?\n\n3. Can you evaluate caption correction using independent metrics (CIDEr, SPICE) or human evaluation, not just BLIP scores? Does \"correction\" fix factual errors or just adjust to BLIP preferences?\n\n4. Why are critical ablation results (Table 4 on noise types, A.12 on trajectory importance, A.13 on maximization vs. minimization) in the appendix rather than the main text? Can you provide specific numbers showing fine-grained noise improvements in the main paper?\n\n5. How does the method perform at realistic noise rates (10%, 20%) rather than 50%? What about very long/short captions, ambiguous images, or subjective descriptions?\n\n6. Why not consider token addition, only deletion/replacement? What is the sensitivity to N and T? Why does maximization outperform minimization (A.13 only says \"seem to be obtained\")?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. **Novel and intuitive approach**: The trajectory-based idea is creative and well-motivated. The insight that correct captions require minimal edits while erroneous ones show substantial improvement is clear and compelling. The framework is model-agnostic and consistently improves multiple baselines (CLIP, LEMoN, BLIP).\n\n2. **Practical value with interpretability**: Beyond detection, TRACED provides token-level error localization and enables caption correction. The 14.5% alignment score improvement on InternVL3-14B demonstrates practical utility.\n\n3. **More realistic evaluation**: The fine-grained noise benchmark better reflects real annotation errors than wholesale caption swaps. The finding that TRACED's largest gains occur under fine-grained noise (where baselines struggle most) validates the approach's practical relevance.", "weaknesses": "1. **Limited demonstration of generality (Critical)**: The same \"no jacket\" example appears repeatedly (Figures 1, 3, and text), raising cherry-picking concerns. The paper lacks diverse examples showing different error types (object misidentification, color errors, numerical errors, relationship errors, distributed errors) and failure cases. This is essential for convincing readers of the method's generality, especially given concerns about ambiguous images, subjective captions, and varying caption lengths.\n\n2. **Circular reasoning and evaluation concerns**: Caption correction is evaluated using BLIP alignment scores while being optimized with BLIP-based TRACED. Independent metrics (METEOR, CIDEr, SPICE) or human evaluation are needed to verify whether \"correction\" actually fixes factual errors rather than simply adjusting to BLIP's preferences. Additionally, critical results are relegated to the appendix: noise-type breakdown (Table 4), trajectory importance (A.12), and maximization vs. minimization (A.13) should be in the main text to support key claims.\n\n3. **Insufficient validation of fine-grained noise**: While conceptually appealing, the fine-grained noise lacks validation against real annotation errors. The use of GPT-4o-mini raises reproducibility concerns, and the omission of MM-IMDb due to cost limits generalizability claims. The 50% noise rate is also unrealistic, performance at lower rates (10%, 20%) is needed. Furthermore, key questions remain unaddressed: Why is maximization better than minimization? What about token addition (not just deletion/replacement)? What is the sensitivity to hyperparameters N and T?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "LzTzAQ57l5", "reviewer": "Reviewer_ZqLU", "summary": "This paper aims to detect and correct errors in vision-text data pairs, which is an interesting issue. To tackle this issue, a trajectory-guided method, named TRACED, is proposed, which edits the caption iteratively. This is a plug-and-play framework for different scoring multi-modality models. Experiments show that such a detection and correction manner is helpful and useful.", "questions": "1.\tI cannot fully understand Table 1. Take the Flickr30K dataset as an example. What does the AUC here mean? Then what about the improvement? What’s the baseline result to compute the improvement? \n2.\tFor each image-text data pair, how many noised captions does the pipeline generate? \n\n\n------\nThis motivation is interesting to me. I will be happy to raise my score if my concerns can be solved.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1.\tThe motivation of this paper is interesting.\n2.\tThe proposed TRACED model is effective in detecting caption errors and finding the most influential tokens in a sentence.\n3.\tThe experimental results demonstrate the effectiveness of this model.", "weaknesses": "1.\tFrom my perspective, it’s true that current vision-text training data may contain mistakes, which affects the model's learning. But the authors don’t provide the number and the proportion of mislabelled instances of a specific dataset, like MS-COCO, Flickr30K. Hence, I have doubts about whether the pre-trained datasets have this issue. \n2.\tTable 1 is not easy to read and follow. \n3.\tIn Appendix A.4, the authors propose to construct a new benchmark, but I cannot see more details about this benchmark. \n4.\tThis paper is able to detect errors in captions. Hence, I think it would be better to represent how the correction procedure improves the multi-modal models’ understanding performance, such as the image-text retrieval task, the captioning task, etc. For now, Figure 12 still evaluates the data itself with different models after correction.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ce8u1CCy1a", "reviewer": "Reviewer_TbST", "summary": "The paper proposes **TRACED**, a model-agnostic framework for detecting erroneous image–caption pairs by analyzing a **trajectory** of edits applied to the caption that monotonically seek to improve an image–text relevance score $s$. Instead of trusting a single similarity score, TRACED constructs a sequence of candidate captions via token deletions and/or replacements, tracks how (s) evolves, and also measures semantic drift relative to the original caption. A classifier is then trained on trajectory features to predict whether the original pair is correct. The authors evaluate TRACED on MS-COCO, Flickr30k, and MM-IMDb, showing consistent AUC gains over CLIP, BLIP, and LEMoN baselines; they also introduce a “fine-grained” noise process (minimal GPT-4o-mini edits) to stress subtle errors. Beyond detection, the trajectory highlights token sources of error; these token-level hints can guide a VLM to produce higher-scoring corrected captions.", "questions": "N/A", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. Moving from a single score to **trajectory statistics** is intuitive and broadly compatible with popular alignment functions (CLIP cosine, BLIP ITC/ITM, LEMoN). The paper’s Algorithm-1-style procedure and the use of both alignment evolution and semantic similarity are well-motivated. \n2. Trajectories naturally localize problematic tokens (e.g., removing “no” resolves a negation error). This is a useful diagnostic for dataset cleaning pipelines and for prompting downstream caption correction.\n3. Supplying token-level “flagged words” to a VLM improves BLIP alignment of corrected captions (especially with larger models + CoT prompting), supporting the claim that trajectories provide actionable signals, not just detection.", "weaknesses": "1. TRACED *optimizes and evaluates* using alignment metrics like BLIP ITM and CLIP similarity. While convenient, this risks **circularity**: if trajectories explicitly maximize (s), improved AUC and “correction quality” may partly reflect over-optimization of the very metric used for labels/evaluation rather than true semantic correctness.\n2. A large part of the evaluation hinges on **synthetic** perturbations, including the proposed fine-grained GPT-4o-mini edits. While this nicely induces subtle errors, it is still simulated. It remains unclear how well TRACED generalizes to **real, organically noisy** web-scale datasets.\n3. We’re told using both alignment and semantic-similarity signals beats using either alone, and that Elimination often wins; however, a fuller **ablation** would help isolate what matters most: trajectory length (T), candidate count (N), semantic similarity choice, and the classifier architecture.\n4. While the paper argues scalability, **single-pass** baselines are very cheap; TRACED adds multiple forward passes per pair. The reported 6.5h/1M pairs (BLIP-ITM + Elimination on 4×L40) is reasonable, but not trivial for very large corpora; clearer guidance on **cost/quality trade-offs** (when Elimination suffices vs. when Fast GCD pays off) would aid adoption.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "fcHq1GvS2E", "reviewer": "Reviewer_PQfZ", "summary": "This paper proposes TRACED, a new method for caption error detection. By looking at caption trajectories – iterative edit traces starting from the original caption – TRACED looks for atomic edits that greatly improve an image-text alignment score, finding that these frequently correspond to fine-grained misalignments between the original caption and image. Results show that TRACED is effective at identifying and correcting caption errors, potentially complementing existing methods.", "questions": "How are flagged words (as in Fig 4) determined? Is it a fixed threshold applied to changes in s(...) or c(...)?\n\nOther works explore caption filtering, recaptioning, or error correction showing improvement on downstream VLM performance. Have you tested this? How does your method compare, or is it complementary to these approaches?\n\nIs the interpretability application (Sec 3.3) purely impressionistic, or have you tested that TRACED flagged errors correlate statistically with human judgements of the source of misalignments?\n\nIt seems like Elimination is usually more effective than the more complex GCD-based methods while being much more efficient. What is the motivation for testing GCD? Could it be improved to be competitive or does it have other conceptual or practical benefits?\n\nIn Tab 3, Fast GCD is approximately x240 slower than elimination, vs. L1608 where it is about x9 slower. Is this correct?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The idea of using caption trajectories to identify fine-grained misalignments is clever and appears novel.\n\nThe framework is model-agnostic, making it flexible and potentially applicable to a wide range of settings and improvements in vision-language models.\n\nThe results show a modest but significant improvement in error detection using TRACED relative to baseline methods.\n\nOverall the paper is clearly written.", "weaknesses": "The evaluation seems partly circular, as the results in Sec 4.4 show that TRACED, which is designed based on single-token edits, improves error detection for synthetic edits which may be largely single-token based (L367–374). In addition TRACED relies on an automatic alignment score using models which may be biased towards the types of errors generated synthetically. This seems partly in line with Table 4 where improvements for random caption substitution are mostly very small.\n\nThere are some evident limitations of the types of trajectories used which are not clearly discussed. Conceptually, since VLMs like CLIP and BLIP struggle with compositionality, it is not clear that they will be robust to fine-grained compositional misalignments (e.g. “a jacket” vs. “no jacket”). [1] Since trajectories only use single token edits, it is not clear how or if TRACED could handle misalignments due to multi-token words for phrases. \n\nOptimization-based edits lead to non-fluent captions, as seen in Fig 6–8 (“vehicles on near …”, “thecoat”, “vehiclesl” etc.), making it unclear whether the trajectory signal is fully meaningful. Flagged words include function words (L931, e.g. “is”). Limitations and/or robustness to these aspects should be explicitly discussed.\n\nThe improvements in error detection seem modest (Tab 1), raising questions about whether it is worth the added computational overhead. While the paper claims the method is “efficient” in several places, some of the runtimes reported in Tab 3 are rather high (although I acknowledge that this is a one-time cost).\n\n[1] Yuksekgonul et al. When and why vision-language models behave like bags-of-words, and what to do about it? ICLR 2023", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)