LEMoN: Label Error Detection using Multimodal Neighbors
Authors: Haoran Zhang
Aparna Balagopalan
Nassim Oufattole
Hyewon Jeong
Yan Wu
Jiacheng Zhu
Marzyeh Ghassemi
Venue: ICML 2025 poster (2025)
Keywords: label error detection
noisy labels
image captions
Primary area: social_aspects->robustness
TL;DR: We propose a method to automatically identify label errors in image captions using the multimodal neighborhood of image-caption pairs.

Abstract: Large repositories of image-caption pairs are essential for the development of vision-language models. However, these datasets are often extracted from noisy data scraped from the web, and contain many mislabeled instances. In order to improve the reliability of downstream models, it is important to identify and filter images with incorrect captions. However, beyond filtering based on image-caption embedding similarity, no prior works have proposed other methods to filter noisy multimodal data, or concretely assessed the impact of noisy captioning data on downstream training. In this work, we propose, theoretically justify, and empirically validate LEMoN, a method to identify label errors in image-caption datasets. Our method leverages the multimodal neighborhood of image-caption pairs in the latent space of contrastively pretrained multimodal models to automatically identify label errors. Through empirical evaluations across eight datasets and twelve baselines, we find that LEMoN outperforms the baselines by over 3% in label error detection, and that training on datasets filtered using our method improves downstream captioning performance by more than 2 BLEU points over noisy training.

LEMON: Label Error Detection using Multimodal Neighbors
Haoran Zhang
Aparna Balagopalan
Nassim Oufattole Marzyeh Ghassemi
Hyewon Jeong'
Yan Wu
Jiacheng Zhu'
Abstract
CIPAR1O
CIFAR1OO
MSCOCO
Flickr30k
Large repositories of image-caption pairs are es- sential for the development of vision-language models. However, these datasets are often ex- tracted from noisy data scraped from the web, and contain many mislabeled instances. In order to improve the reliability of downstream models, it is important identify and filter images with in- correct captions. However, beyond filtering based on image-caption embedding similarity, no prior works have proposed other methods to filter noisy multimodal data, or concretely assessed the im- pact of noisy captioning data on downstream train- ing. In this work, we propose, theoretically jus- tify, and empirically validate LEMON, method to identify label errors in image-caption datasets. Our method leverages the multimodal neighbor- hood of image-caption pairs in the latent space of contrastively pretrained multimodal models to au- tomatically identify label errors. Through empir- ical evaluations across eight datasets and twelve baselines, find that LEMON outperforms the baselines by over 3% in label error detection, and that training on datasets filtered using our method improves downstream captioning performance by more than BLEU points over noisy training.
Automobile
Leopard
is plane the front view
boy in red obart playing ball.
Samples from classification and captioning datasets discovered to be mislabeled by our method.
Figure 1:
problematic in settings such as healthcare, where the relia- bility of downstream models may depend on the quality of data used for pretraining (Chen et al., 2024; Liu et al., 2023; Longpre et al., 2023).
Identifying and correcting label errors in existing datasets scale would lead to more reliable and accurate models in the real world (Zhu et al., 2022; Vasudevan et al., 2022; Liao et al., 2021; Beyer et al., 2020). However, given the large size of such datasets, manual detection of errors is practi- cally infeasible. This is evidenced by the growth of models trained noisy data with the web (Li al., 2022; Wang et al., 2022a; Liu al., 2024), or with model generated pseudo-labels (Menghini et al., 2023; Lai et al., 2023).
Machine learning (ML) based approaches to automatically identifying label errors have also been proposed in prior work (Pleiss et al., 2020; Swayamdipta et al., 2020; Liang et al., 2023; Bahri al., 2020; Zhu et al., 2022; Northcutt et al., 2021a). However, we identify critical limitations: (1) the majority of such works unimodal: i.e., they only utilize image-based representations in detection strategies, and (2) many of the best-performing approaches depend on having access to a model already trained on the down- stream tasks interest (Pleiss et al., 2020; Swayamdipta et al., 2020). We hypothesize that applying a neighborhood- based approach to multimodal representations the form of image-tex pairs can improve label error detection without requiring task-specific training, which may be costly and/or domain specific for some datasets.
1. Introduction
Machine learning datasets used to train and finetune large vision, language. and vision-language models frequently contain millions of labeled instances (Schuhmann et al., 2021; Li et al., 2022; Wang et al., 2022a; Changpinyo et al., 2021). Prior work highlights that some instances in such datasets may be mislabeled (Northeutt al., 2021b; Luc- cioni & Rolnick, 2023; Liao al., 2021; Beyer al., 2020; Plummer et al., 2015), seen in Figure 1. This is especially
Equal contribution "asssacuseetts Institute of Technology. Correspondence to: Haoran Zhang <haoranz mit.cdu,, Aparna Balagopalan <apamab @ mit.edu>.
Additionally, a common assumption made in prior works is that each label one-of-k classes (Bahri et al., 2020; Zhu et al., 2022). The vast majority of label error detection meth-
Proceedings of the nd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

LEMON: Label Error Detection using Multimodal Neighbors
ods proposed in prior works are hence for classification datasets. In contrast, datasets used to train large vision- language models contain natural language labels such as image captions (Li et al., 2022: 2023; Wang et al., 2022a). Methods to filter out instances with noisy labels- - e.g- based on the similarity of image and caption representations have been utilized in prior work with some success (Li et al., 2022; Kang et al., 2023) for such datasets. However, to the best of our knowledge, no prior works have pro- posed or rigorously compared methods to identify errors in datasets with natural language labels, or assessed the impact of detection on downstream tasks like image captioning.
2. Related Works
Label Noise Detection Noisy and incorrect labels (Beyer et al., 2020) in training data may lead to decreased or "desta- bilized (Northcutt et al., 2021a; Luccioni & Rolnick, 2023) performance on downstream tasks (Chen et al., 2023; North- cutt et al., 2021b). Two orthogonal approaches can be taken to reduce the adverse effects of such labels: developing methods to learn in the presence of label errors (Cui et al., 2020; Natarajan et al., 2013; Huang et al., 2023) referred to as "noise robust" training, and/or detecting and filtering out instances with label errors (Zhu et al., 2024). In this work, we focus on the latter direction. This approach may be preferable, as identifying mislabeled samples is more flexible, with applications beyond just removing these sam- ples for downstream model training. For example, accurate label error detection can uncover systematic errors or biases in datasets (Rottmann & Reese, 2023), and these insights can then be used to guide higher-quality data collection practices (Bernhardt et al., 2022). This is especially impor- tant for practitioners releasing datasets intended for model evaluation (Northcutt et al., 2021 Schubert et al., 2024).
In this work, we propose LEMoN Label Error detec- tion using Multimodal Neighbors, a method for multi- modal label error detection which can be applied image- text pairs in datasets such MSCOCO (Lin et al., 2014). While prior techniques utilize unimodal neighbors for la- bel error detection, LEMMNN leverages multi-modal neigh- borhoods derived using contrastively pretrained vision- language models such as Contrastive Language-Iman Image Pre- training (CLIP) (Radford et al., 2021). Specifically, in addi- tion to considering pairwise image-text distances, we also retrieve nearest neighbors in the image and text space as illustrated in Figure 2. This is motivated by the rich neigh- borhood geometry in the joint embedding space of mul- timodal models (Liang et al., 2022; Schrodi et al., 2024). We then compute distance scores with neighbors in each modality and combine these into a single score measuring the likelihood of a label error, with the intuition that higher discordance (or higher distance) with neighbors indicates higher chance of label error. We validate the utility of these scores across eight datasets, including one in a healthcare setting, and compare to over ten baselines.
Prior approaches (Swayamdipta et al., 2020; Bahri et al., 2020; Pleiss et al., 2020; Northcutt et al., 2021a; Liang et al., 2023; Wu et al., 2020; Kim et al., 2021) for automatic label error detection include relying on the training dynamies of task-specific downstream models (Swayamdipta et al., 2020) and neighborhood-baset strategies (Bahri et al., 2020; Grivas et al., 2020). Some of these techniques are fully supervised (Northcutt et al., 2021a; Chen et al., 2023) or unsupervised (Pleiss et al., 2020; Swayamdipta et al., 2020; Grivas et al., 2020; Bahri et al., 2020), use pre-trained gen- crative models (Gertz al., 2024) or are fully training-free free approaches (Zhu al., 2022; Liang et al., 2023). Previous approaches for label error detection closest this work in- cludes deep k-nearest neighbor (deep k-NN) methods using k-NN entropy on vector space embeddings (Bahri et al., 2020; Grivas et al., 2020) and SimiFeat (Zhu et al., 2022) which employs a local neighborhood-base based voting or rank- ing for noise identification. In contrast to these methods, our work enhances label noise detection by harnessing in- formation across multiple data modalities, such as image and text.
Our key contributions and findings are as follows!:
We propose LEMON, novel, theoretically justified multimodal method capable of detecting label errors in large image-caption datasets (Section 3).
We show that LEMoN outperforms all training- free baselines for label error detection in three out of four classification datasets by up to 3.4% AUROC, and in three out of four captioning datasets by up to 6.3%
AUROC (Section 1.)).
We demonstrate that LEMON improves performance downstream classification and captioning models by filtering out data predicted to be label errors. (Sec- tion 6.2).
Contrastive Learning Contrastive learning is a repre- sentation learning method, relying on positive and nega- tive pairs of data instances (Chen et al., 2020: Misra & Maaten, 2020; Balestriero et al., 2023) to learn an embed- ding space. The core idea is to embed similar data points (positive pairs) closer together than dissimilar data points (negative pairs) (Schroff et al., 2015; Sohn, 2016; Oord et al., 2018). In this work, we primarily utilize pre-trained models that use the CLIP loss (where the pre- pre-training objective is to predict which text caption is paired with which image)
Finally, we verify that the predictions generated by LEMON are meaningful through a real world analysis of LEMON on existing datasets without known label errors (Section 6.5).
'Code: htttps://gtthbb..
2

---

LEMON: Label Error Detection using Multimodal Neighbors
bullet
Dvo passing
sail
plane Facting
Tex Manifold
y mhis plane
This ib. plane Eron the Front view
the Exont view
Image Manifold
Y YnO plane geat
Awa
Tex Manifold
about to tape ott
Xmi
Xmn
Image Manifold
(a)
(b)
Figure 2: Outline of LEMON, our proposed method for multimodal label error detection. We demonstrate LEMoN on a real sample from the MSCOCO dataset, where an image of train (x) is mislabeled as y "This is plane from the front view". (a) We compute the simple CLIP similarity dmm y). We then find the nearest neighbors of x in the image space (Xn,) and compute the distance between the corresponding texts and y to compute the score component Sn. (b) To compute the score component Sm. we find the nearest neighbors of y in the text space (ymz), and compute the distance between the corresponding images and x.
for jointly embedding image and text data (Radford et al., 2021).
timodal models, often with the goal of improving repre- sentation learning (Huang et al., 2024; Li et al., 2021) or retrieval (Thomas & Kovashka, 2020; 2022). The closest related work is Thomas & Kovashka (2022), where the au- thors use the semantic neighborhood of multimodal models to identify samples with high semantic diversity using text- based neighbors of neighbors. Importantly, the objective of their work is different from ours, which leads their proposed discrepancy and diversity scores to lack signal for label error in our setting. We further clarify this in Appendix B, and empirically compare against their discrepancy score baseline. We believe our work is the first to use multimodal neighbors for label error detection.
Image Captioning The goal of image captioning is to describe given image (Fu et al., 2024) in natural language. Prior approaches for caption generation have included su- pervised training of end-to-end models from scratch (Wang et al., 2022b; Lin et al., 2022: Hu et al., 2023; Xu et al., 2015; Fu et al., 2024). More recently, vision-language mod- els pretrained on large datasets of noisy image-caption pairs extracted from the web (Li et al., 2022; 2023; Wang et al., such 2022a) such as CC12M (Changpinyo et al., 2021) - have been utilized for captioning. Some of the pretraining tasks include image-text contrastive learning, image-text match- ing, and/or retrieval (Li et al., 2022), as well as general pur- pose text generation conditioned on an input image (Wang et al., 2022a). Given that datasets for training such large models are noisy (Kang et al., 2023), several steps have been utilized in prior work to filter out noisy captions during training. The most common strategy involves computing the similarity between representations of the image and caption text using another pretrained model (e.g., CLIP) prior to training (Kang al., 2023). Another approach in training the BLIP (Li et al., 2022) model is to synthetically generate noisy captions and train classifier to distinguish between high quality captions and noisy captions with a cross-entropy loss (Li et al., 2022). To the best of our knowl- edge, no previous work has conducted a comprehensive comparison of various strategies for label error detection in captioning datasets.
3. LEMON: Label Error Detection using Multimodal Neighbors
We are given a dataset D = {x consisting of two modalities x E X and E y. For example, X may rep- resent the set of all natural images, and ) may represent the set of all English text, or a restricted subset such as (cat, dog, We assume the existence of, but not ac- cess to, an oracle I' X x y + {0, which is able to assign binary mislabel indicator 2; (x,, y:) to each sample in D. Here, 2; = 1 indicates that the sample is mislabeled, and z = 0 indicates that the sample is correctly labeled. Our goal is to output a score s E R with some model = f(x, y) such that:
AUROC- = [[x,,))) a --)1====)) f(x.)) x')l
is maximized. Prior works have alternatively aimed to max- imize the F1 score, optimizing over a threshold t:
Multimodal Neighborhood Methods Previous stud- ies (Li et al., 2021; Thomas & Kovashka, 2020; Huang et al., 2024; Liang et al., 2022; Cai et al., 2023) have examined the geometry of neighborhood spaces in mul-
2. = 2
Fl = max tER
1) 1|=
3

---

LEMON: Label Error Detection using Multimodal Neighbors
Here, we build on prior work for label error detection in unimodal data (Bahri al., 2020; Zhu et al., 2022) and propose method for f based on nearest neighbors, sum- marized in Figure 2. Suppose we have a query sample y). Define r) := {a' E X : dx(xxx'), S r). the ball of radius r around x, and Bry,r) similarly, Let nx(x) := irf(r: : [B(x, r) n DI 2 ki), the minimum ra- dius required to encompass at least kt neighbors. Then, we define (xx,, Xnyy ) := Fxx,Tx(()) n 'D the top& k nearest neighbors of x. and {ym (Ymi:-mgy ... Ymy) := n 'D the top & nearest neighbors of y². We assume that the neighbors are sorted in order of ascending distance, e.g. dx(x,, xx,) 2 dxx (x, X.,)
downweight those instances.
The third term is analogous to Sn, but uses neighbors of y:
8, y, D)
Σ Xm,)) -imdollamm - dammn (8m, ym,)
(3)
where (Xmy (Xmy=mmy) E DD and 1im,, 72,m 2 0 are hyperpa- rameters. Crucially, note that notationally, Xa; * Xmy. and Yn, # Ymy- Specifically, Yny corresponds to the y modality of nearest neighbors taken in X space, and ym; corresponds to the nearest neighbors of y taken in y space.
We note that our method is generalization of several prior methods. When / = = 0, the method is equivalent to CLIP similarity (Liang et al., 2023). When 8 is large, Tin = T2nx = = = 0, and d(y. = ly : the method is equivalent to Deep kNN (Bahri et al., 2020). An algorithm outline and high-level description of the method can be found in Appendix C.
If y is a small discrete set, we could choose dly, y') 1 lysy'. If X or y' are unstructured or high dimensional, we assume access to multimodal encoders he (hg ,,)), ) where 1/ X' + IRd and lig 3 + Rd. Here, he may be a CLIP model (Radford et al., 2021) trained on a large internet corpus, or, as we show later, it may be sufficient to train he from scratch only on D. Then, we could naturally use simple distance metries in the embedding space, such as the cosine distance dx (x, x') deoa (x), hi (x'))
Our method contains several hyperparameters:
k 3, 7, Ti, 72. Tl.m. and 72 When there is a validation set with known mislabel flags, we perform a grid search over k, and use numerical optimization methods to search for an optimal value of the remaining hyperparameters which maximize label error detection performance on this set, which we describe further in Section 5.2. We refer to our method in this setting as LEMON We will empirically show that only a few hundred labeled validation samples may be sufficient to achieve optimal performance in this setting.
1 (<xx">>>" "A"" (x')
1 [ 2x IN (x'Illa Our proposed score is the linear combination of three terms:
s = y( = dmm
478m (x, y, D),
where 3,7 2 are hyperparameters. Here, dmm xx y) ==
deca(ft (hz (x), (onhv((())) (y)) is the multimodal distance, which has been shown empirically to provide meaningful signal in prior label error detection work (Liang et al., 2023; Kang et al., 2023). We thus use this distance as the basis, and augment it with two additional terms based on nearest neigh- bors:
When there is no labeled validation set available, we will show that our method is fairly robust to these hyperparam- eter choices, and that choosing set of reasonable fixed values for these hyperparameters yields nearly comparable results. We refer to our method in this setting as LEMoN pxx
sx D) = 1/b 3 dy(y. (addx(xx)))))))))))))))))))))))) (hnddwaa -Taddmm (K, Noy).
4. Theoretical Analysis
(2)
where (nz) E DD and TI.n, T2. 2 0 are hyperparam- eters. This corresponds to finding the nearest neighbors of x in X' space, then averaging the distance between their corresponding modality y and y. We weight this average with two additional terms. The Ti,n term corresponds to downweighting neighbors which are far from x. Intuitively, this is useful when & is too large for x and not all neighbors are relevant, and can be thought of as an adaptive k. The T2.n term corresponds to downweighting neighbors which are themselves likely to be mislabeled. If (Xny: Yng) vng)inn is itself mislabeled, then dy yn,) would contribute an erroneous signal to whether (x, y) is mislabeled, and we thus want to
We show that our multimodal KNN scores (Equations (2) and (3)) provide signal for label error. Suppose there exists "paraphrase function" H: ) - P(Y'), where P denotes the powerset, such that for a particular sample (z, y) with H(v) = (M1. ). (x, Th) is considered correctly labeled for all Ø. E H(y)). Informally, H. outputs the set of all possible captions which correctly describe x. Similarly define I (z), which outputs the set of images with identical semantics as x5
Assumption 1 (Structure of H. Jy:
Let (x'. be an arbitrary sample. If of & H(y), then x' & (()).
"One could take, for any i, (xx = (x, D' == DA ())
WW will use a subseript ny to index nearest neighbors in X, and subscript m, for neighbors in y.
Let (x', be an arbitrary mislabeled sample. Then, Vyy" E H(y'), x" & H(r').
4

---

LEMON: Label Error Detection using Multimodal Neighbors
Assumption 2 (Distribution of Distances); Let () be a randomly drawn sample.
5.1.1. NOISE TYPES
In ci rirrl0 and eifar100, we utilize a dataset collected in prior work (Wei et al., 2021) with human mislabels (hu- man). We also follow prior work (Zhu et al., 2022) in experimenting with class symmetric (sym.) and class asym- metric (asym.) synthetic noise. For stanfordCars and mini imagenet, we use datasets from Jiang et al. (2020), which contain noise from real-world (real) web annotators
'' & J(X) dx(X,X)) id M(11.0) A X E [(X) : Xx de ([2,, oz).
We empirically validate this assumption in Appendix A.2.
(Ym Ym,} denote the nearest neighbors of Y in the text space. Let R (YY( n (Y) = Sr, a random variable. Suppose that 1 (Xma, Ym.) is mislabeled]] = p is constant for all samples in the support of (X,Y).
Let (Y)
k fi :
For the four captioning datasets, we devise several ways to inject synthetic noise of prevalence p. The simplest way is to randomly select p fraction (random) of the samples and assign their text modality to be that of another random caption. In datasets where additional metadata is avail- able (mscoco: object category, mm mmimcb: genre of movie, mimi cexr: disease label), we can randomly swap the cap- tion with that of another sample from the same category (cat). Finally, in all captioning datasets except m mimica micexr, we tag each token of each caption with its part-of-speech us- ing SpaCy (Honnibal & Montani, 2017), and then randomly assign a selected sample's text modality to be from another sample with at least one noun in common (noun). Dataset processing details are in Appendix D.
Let (X) = = Er (2.Y,,ENN(1 (N dx( Xm.), which is identical to the proposed Equation (3) with 71 T2 = O.
Theorem 4.1 (AUROC of KNN Score). Let (X, Y) be randomly selected correctly labeled sample, and (X', Y") a randomly selected incorrectly labeled sample. Under Assumptions I and 2:
P(Sm((',))) > Sm (),)
= =1 -
where P
E[ p11 122, o
[S pilile + Varl<r)(1- ()))))(((()( p1)3)
1/2
These noise types are intended simulate an array of realis- tic label corruptions in the real world. As such, the resulting synthetic dataset may not have an exact noise level p, as e.g. randomly selected caption may actually be correct for the image, as well as due to noise in the base datasets, which we explore in Section 6.5. Unless otherwise stated, results shown in the main paper are for the bolded noise type in Table 1, with 40% synthetic noise. Additional results for other noise types can be found in Appendix I.
and 4 is the Gaussian CDF.
This provides an expression for the detection AUROC of the score Sm- The same expression can be derived for S, by symmetry.
Lemma 4.2 (Non-random Signal of KNN Score). If the
following three conditions hold: (1) p < 1, (2) B[(r] > O,
(3) pi > U2. Then, ((S... (X', Y') > Sm (), > 0.5.
Under these mild conditions, Sm, our proposed multimodal neighborhood score, provides a better than random signal at detecting mislabeled samples. The proof can be found in Ap- pendix 1. We additionally provide a theorem showing that embedding models trained via the contrastive multimodal objective are natural noisy label detectors in Appendix A.3.
5.2. Model Selection and Evaluation
We run LEMON on each dataset, using the training split of each dataset compute nearest neighbors. In classification datasets, use the discrete metric dy 1yzy' In all other cases and for dx, we utilize cosine or euclidean distance computed in the embedding space of a pretrained CLIP model, selecting the best distance metric on the vali- dation set for L.EMON OpT- and keeping the distance as the cosine distance for LEMON In mimiccxr, we use BiomedCLIP (ViT-B/16) (Zhang et al., 2023b), and we use OpenAl CLIP VIT-B/32 (Radford et al., 2021) for all other datasets. A full list of hyperparameters for our method and the baselines are in Appendix G.
5. Experimments
5.1. Datasets
We evaluate our method using eight datasets, as shown in Table 1. Four datasets (cifarld ci faffal100) 00, stanfordCarss mini ImageNet) are label error detec- tion datasets from the classification setting. The four remain- ing datasets are image captioning datasets. For mscoco and ckrkkkk we use the Karpathy split (Karpathy & Fei-Fei, 2015). The remaining datasets were randomly split into: training or reference set for the label detection method (80%), validation set for hyperparameter selection (10%), and test set for performance evaluation (10%).
For LEMONOPT- we select the hyperparameter combina- tion that maximizes F1 on a labeled validation set. We report the AUROC, macro-averaged AUPRC, and Fl for this model. For LEMON Rx, we fix the hyperparameters at the following reasonable values: A 30, B = 7 = 5,
Tia Ti.m = 0.1, and T2n T2 = 5. We report AU-
5

---

LEMON: Label Error Detection using Multimodal Neighbors
Table I: Classification and captioning datasets. n is the number of samples. In the main paper, results shown are for the bolded noise type with 40% noise level for synthetic noise. Results for remaining settings can be found in the appendices.
Dataset
Domain
Noise Types
Train
Validation
Test
Image
Text
citar10
40,000
5,000
5,000
Natural images
Object labels
[Jumman (Wei et al., 2021), sym, aryn.)
cilar100
40,000
5,000
5,000
Natural images
Object labels
(human (Wei et al., 2021), sym., asym.) (reat)
miniImagelStt (Jiang et al., 2020)
49,419
24,710
24,710
Natural images
Object labels
stantordCars (Jiang et al., 2020)
13,501
6,751
6,752
Car images
Car year and model
(reat)
macoco (Lin al., 2014)
82,783
5,000
5,000
Natural images
Captions
{oar. noun, random)
flickr30k (Young al., 2014)
29,000
1,014
1,000
Natural images
Captions
(noun. random)
mmindb (Arevalo et al., 2017)
15,552
2,608
7,799
Movic Posters
Piot summaries
[eat. nown, random)
mimiccxr (Johnson el al., 2019)
368,909
2,991
5,159
Chest X rays
Radiology reports
{eat.. random)
ROC and AUPRC, as the F1 requires additional information to compute a threshold for the score. We recognize that access to such a validation set in LEMoNofr may be un- realistic, but we will empirically show that (1) our method is fairly robust to selection of these hyperparameters, (2) only a few hundred labeled samples may be sufficient to select these hyperparameters, (3) using LEMoNnx with the fixed hyperparameter setting described above achieves nearly comparable results, and (4) hyperparameters opti- mized dataset with synthetic noise may transfer well to real datasets.
be used directly in this setting. Next, to adapt classification baselines to captioning, we embed the captions using the corresponding CLIP text encoder, and then use K-means clustering to assign the text caption into one of 100 clusters. We then apply Deep k-NN (Bahri et al., 2020) and Confi- dent Learning (Northcutt et al., 2021a), using the cluster ID as the discretized class. Finally, we adapt VDC (Zhu et al., 2024) to the captioning setting using open source models. Further details can be found in Appendix E.
6. Results
We repeat each experiment three times, using different ran- dom seed for the noise sampling (for human and real noise, we use a different random data split). Performance metrics shown are test-set results averaged over these three runs, with error bounds corresponding one standard deviation.
6.1. LEMON Outperforms Baselines on Label Error Detection
In Table 2, we show the performance of LEMoN against the baselines for label error detection on four classification datasets. We find that our method performs on-par with, or outperforms, existing baselines which do not require classifier training on all classification datasets. Two downstream-taas specific approaches (AUM and Datamap) outperform most training-free fare models (partic- ularly on i far100), but LEMON performs comparably and even outperforms them in two datasets. Similar re- sults are also observed on the two synthetic error types (see Appendix Table I.2). We find that LEMoN performs almost comparably with LEMONorT, and still beats almost all basclines.
Classification
Baselines We compare our method versus previous state- of-the-art in both the classification and captioning settings. We additionally adapt several baselines from the classifi- cation setting to the captioning setting. We briefly list the baselines here, and detailed description is in Appendix E.
In the classification setting, we experiment with the following baselines which require training a classifier on the particular dataset: AUM (Pleiss et al., 2020), Datamap (Swayamdipta et al., 2020), and Confident Learning (Northcutt et al., 2021a), and the following base- lines which do not require classifier training: Deep k- NN (Bahri et al., 2020), Simil (Zhu et al., 2022)-Votinn and Ranking, discrepancy in the image space (Discrepancy) RIS from Thomas& & Kovashka (2022)), CLIP Similar- ity (Kang et al., 2023), and CLIP Logits (Liang et al., 2023; Feng et al.).
Captioning In Table 3, we find that our method outper- forms existing neighhorhood and similarity- baselines on three datasets. In two datasets, our model underperforms the oracle (CapFilt) which has been pre-trained on clean captions from the mscoco dataset. Results for synthetic error types show similar trends (see Appendix I.2).
In the captioning setting, we compare our method with LLaVA (Liu et al., 2024) prompting and CapFilt (Li et al., 2022). We note that the latter can be viewed as an oracle for natural image captioning, as it has been trained in a supervised manner on clean mscoco data. CLIP Similar- ity (Kang et al., 2023). Diserepancy (Thomas & Kovashka, 2022), and Datamap (Swayamdipta et al., 2020) can also
Label Error Detection Performance Across Noise Levels In Figure I.I, we show the performance of LEMoN ver- sus the CLIP similarity baseline on mscoco and mmimabb varying the level of the synthetic noise. We find that L.MMNN performs better uniformly across noise levels.
6

---

LEMON: Label Error Detection using Multimodal Neighbors
Table 2: Label error detection performance across classification datasets. We separate AUM, Datamap, and Confident
learning, as they require training a classifier from scratch. Bold denotes best score within each training approach. A full version of this table with AUPRC can be found in Appendix I.I.
Method
Training- Free
cifar10
cifar100 100
mini ImageNet
stanfordCars
AUROC
AUROC
AUROC
AUROC
F1
AUM
98.3 (0.1) 92.9 (0.1)
92.3 (0.3)
81.1 (0.3)
83.1 (0.2)
68.3 (0.4)
(2.3)
(3.1)
Datamap
X
98.2 (0.1)
(0.5)
91.8 (0.3)
80.8 (0.5)
85.1 (0.3)
(0.2)
72.2 (1.7)
50.4 (2.1)
Confident
89.6(1.))
(1.7)
78.6 (0.44
73.7 (0.5)
59.5 (0.7)
37.7 (1.5)
60.7 (0.3)
39.9 (0.6)
CLIP Logits
95.5 (0.2)
(0.6)
84,9 (0.7)
(0.9)
90.0 (0.2)
(0.2)
68.8 (0.1)
(0.5)
CLIP Sim.
92.2 (0.2)
82.3
80.8 (0.9)
(1.1
89.3 (0.2)
76.1 (0.4)
69.8 (0.4)
(0.5
Simifeut-V V
(0.3
(0.5)
(0.1)
(0.2)
(0.2)
54.8 (0.3)
(1.6) (1.)
Simifeat-R
88.8 (0.1)
(0.9
(0.1)
76.1 (0.1)
(0.5 (0.3)
(0.3)
(0.5
Discrepuncy
77.1 (1.9)
(2.2)
(1.5)
(8.9
(..0 (0.))
(0.7
(0.7
Deep K-NN
97.8 (0.1)
87.4 (0.3) (0.6)
(0.3)
71.5 (0.5 (0.2
(0.6)
49.1 (0.6)
L.MoN,ex (Ours)
97.7 (0.2)
(0.1)
88.9 (0.7)
(0.6)
(0.2)
74.7 (0.2)
72.6 (0.7)
47.7
LEMoN (Ours)
98.1 (0.0) 92.0
90.8 (0.0)
78.4 (0.0)
(0.4)
76.9 (0.2)
73.1 (0.5)
51.3 (0.5)
Table 3: Label error detection performance on captioning datasets. We separate CapFilt as it has been trained on clean mscoco data. Bold denotes best (highest) score. A full version of this table with AUPRC can be found in Appendix 1.2.
Method
Flickr3Ck
macoce
mmimdbb
mimicexr
AUROC
F1
AUROC
AUROC
AUROC
F1
LLaVA
79.3 (0.8)
(1.1
(0.1)
74.9 (0.3)
(0.2)
(0.1)
$3.9 (0.5)
(0.1)
Datamap
80.6 (1.3)
(0.6)
(0.4)
(0.3)
(0.5)
(0.5 (0.2)
(0.1)
Discrepancy
73.0 (0.6)
(0.3)
72.7 (0.3)
(0.3)
57.8 (0.4)
(7.2
60.0 (0.7)
(0.1
VDC
92.9 (1.1)
81.1 (1.6)
94.1 (0.2)
80.5 (0.4
69.3 (0.6)
50.8 (0.4)
(0.0
Deep K-NN
71.1 (0.4)
59.2
(0.4)
(0.8)
(0.4)
58.3 (0.4)
62.9 (0.4)
59.2
Confident
63.1 (0.9)
(0.9)
(0.5 (0.5)
61.8 (1.8 (1.1
58.1
CLIP Sim.
94.8 (0.5)
84.2 (0.9)
93.8 (0.2)
(0.4
85.1
72.7
(0.1
(9.2
LEMONms (Ours)
93.6 (0.2)
95.6 (2.0
(0.3)
66.3 (0.4)
LEMoNerr (Ours)
94.5 (0.2)
(L.4)
87.0 (0.2)
(6.0
73.5 (0.3)
(1.6
61.1
CapFilt (Supervised Training)
98.6 (0.1)
93.1 (0.7)
(0.0)
95.4 (0.4)
82.7 (0.7)
71.3
49.7 (0.3)
57.0 (0.0)
Size of Labeled Validation Set In Appendix Figure I.3, we examine how varying the size of the labeled valida- tion set impacts the performance of LEMON We find that in all four captioning datasets, having about 100-500 labeled examples is sufficient to tune hyperparameters in oF to outperform L.EMN nx- In the three datasets where LEMONax underperforms the CLIP similarity base- line, we find again that having 100-500 labeled validation samples is sufficient for tuning LEMoNoer to perform on par with this baseline.
results in an AUROC drop of only 1.6% on average (std 1.3%), with a worst-case AUROC drop of 4.1% across all 18 dataset and noise type combinations. Thus, even when labeled validation set is not available, LEMONNx with reasonable hyperparameter settings is able to outperform most baselines which do use such information.
6.2. Filtering Mislabeled Data Improves Downstream Performance
Classification To assess the impact of label error detection on the performance of the downstream classification tasks, we filter out samples from the training set with mislabel scores in the top q percentile. We vary q. train ViT (Dosovit- skiy et al., 2020) models on the filtered dataset, and evaluate the downstream test accuracy using clean data. We compare the performance of LEMON with all training-free base- lines that produce a continuous score (i.e. all except Sim- ifeat and Confident). In Figure 3, we find that training with LEMONorT filtered samples leads to the highest accuracy on cifarl0 and cifar100 after removing more than 30% of the data. Training with LEMONorT filtered samples is also on par with baselines on the other datasets (either the best or within I % of the best baseline) as shown in Ap-
Robustness to Hyperparameters Here, we test the ro- bustness of our method when there is no labeled validation set available. First, in Appendix 1.4, we visualize the F1 of the selected score when varying & and y, keeping all other hyperparameters at their selected optimal values. We find that for most datasets and noise types, there is a reasonably large space of such hyperparameters, bounded away from the origin, which achieves close to optimal performance.
Next, we compare the performance of LEMoNopT and LEMON nx with hyperparameters described in Section 5.2 across all datasets in Table I.8. We find that when there is no labeled validation set available, using these hyperparameters
7

---

LEMON: Label Error Detection using Multimodal Neighbors
cifar10
cifar100
6.3. Ablations
0.95
0.75
In Table I.10, we show the performance of our method after ablating each component. We find that mislabel detection performance almost decreases monotonically additional components until we reach the CLIP Similarity baseline. We find that ablating the Ti and T2 terms results in a performance loss of about 1%. In Table 1.11, we examine the performance of each of the three components of our score and their combinations. We find that dmm is the most critical term. Of the two nearest neighbors terms, we find that Sn (nearest image neighbors) is more important in general, though this is highly dataset dependent, e.g. error detection in mm mil relies much more on neighbors in the text space than the image space, while the opposite is true for mscoco.
Accuracy
0.90
Accuracy
0.70
remove
0.65
LEMeN (Ound)
Deep kNN Sim
0.60
Cis Legit Dicrepancy
0.80
40
60
80
100
40
60
80
100
%aData retained
Vaatt retainne
Figure 3: Downstream classification accuracy on cifarl (left) and i farl CO (right) with LEMoNorn orr with human noise versus the baselines. Note that the noise prevalence is 40% in both datasets.
Table 4: Downstream captioning performance when remov- ing 40% samples with highest mislabel scores. We find that filtering noisy data with LEMoNorT improves captioning.
6.4. External Pretraining May Not Be Required
Medical Images Thus far, all of the results for LEMON (and CLIP Similarity) have utilized CLIP models which have been pretrained on external datasets (e.g. PMC-15M in the case of BiomedCLIPP). Here, we examine whether we can achieve comparable performance by pretraining CLIP from scratch only on the noisy data. We select mimicext has the most samples out of all captioning datasets. Similar to CheXzero (Tiu et al., 2022), we pretrain a CLIP ViT B/16 from scratch on the mimiccxr training set with 40% noise. We train this model for 10 epochs with a batch size of 64, and do not do any model selection or early stopping. We then apply LEMON and the CLIP similarity baseline using this model, for the same noise level and noise type. We present our results in Table 5. Surprisingly, we find that pretraining CLIP only on noisy data from MIMIC-CXR actually outperforms BiomedCLIP. This could be attributed to the pretraining domain (chest X-rays and radiology notes) matching the inference domain exactly (Nguyen et al., 2022). As an upper bound, we evaluate the same methods using CheXzero (Tiu et al., 2022), which has been pretrained on clean SIMIC-CXR data. We find that, as expected, it far outperforms this baseline. We conclude that, for large noisy datasets, pretraining a CLIP model from scratch could be a viable solution, though pretraining on clean data from the same domain is certainly superior.
Datnset
Method
B@4
CIDER
ROUGE
No Filtering
31.6 (0.6)
(1.6
51.4 (0.4)
CLJP Sim.
32.7 (0.3)
79.6 (0.4)
5288 (0.1)
flickr30k
LEMoNoo
32.1 (0.6)
(L.3)
52.5 (0.2)
Clean
33.3
(1.9
53.2 (0.3)
Filtering
(6.0.
118.9 (0.3)
56.8 (0.1)
it
CLIP Sim.
39.5 (0.1)
130.4 (0.3)
59.3 (0.0)
nacoc0
LEMoNoor
39.6 (0.1)
130.8 (0.1)
59.4 (0.0)
Clean
(0.6)
131.2 (0.2)
59.5 (0.1)
pendix 1.14. We also show that filtering data in this manner does not reduce classifier robustness (Appendix I.16).
Captioning We finetune a pre-trained Generativelm- age2Text (GIT) (Wang et al., 2022a) model generate captions. Given the large size of the model, use the parameter-efficient Low-Rank Adaptation (LoRA) (Hu et al., 2021) for all captioning models. We train models with clean data, noisy captions (No Filtering), and by filtering data detected as being mislabeled by a label detection method. In Table 4, we compare results of using either our model or a strong baseline (CLIP Sim.) for filtering data, as measured by the BLEU-4 (Papineni al., 2002), CIDER (Vedantam et al., 2015), and ROUGE (Lin, 2004) scores. In all cases, we filtered out the top 40% percentile of data predicted to be mislabeled (i.c., equal to the expected prevalence of noisy data). We find that (1) filtering out data predicted to be mislabeled helps recover performance as compared to train- ing on fully clean data along multiple metrics, and (2) our method performs comparably to the baseline in improving downstream results, with both filtering techniques perform within I score point of each other. However, as training with a fully clean dataset only outperforms training with fully noisy (i.e. no filtering) dataset by 2-3 BLEU-4 points, the range of potential improvement for any filtering method is limited.
Web-Scale Corpus Motivated by this result, we conduct large scale experiment on the CC3M dataset (Changpinyo et al., 2021), which contains 2.9 million valid URLs to image-caption pairs. We pretrain CLIP from scratch on this dataset, then use this CLIP model to filter samples in the original dataset using LEMON Fi and the CLIP simi- larity bascline. We select the I million samples with the lowest mislabel scores from each method, and pretrain an- other CLIP from scratch on this clean subset. We evaluate the resulting model on zero-shot classification using the
8

---

LEMON: Label Error Detection using Multimodal Neighbors
Table 5: Performance of LEMON for label error detection versus the CLIP similarity baseline on mimicexz, when external pretrained models may not be available. Biomed- CLIP (Zhang et al., 2023a) is trained on a large corpus of biomedical image-text pairs. We find that pretraining only noisy data from MIMIC-CXR outperforms Biomed- CLIP, though pretraining on clean mimi ccxr data (as in CheXzero (Tiu et al., 2022)) does perform better.
Table 6: We manually label 200 images from real-world datasets that each method identifies as the most likely to be mislabeled and show the percentage (%) of times where it is actually mislabeled. Numbers in parentheses are 95% confidence intervals from a binomial proportion.
CLIP Sim.
Ours
eifarld
(3.2)
10.0 (4.2)
cifar100
(4.3)
20.5 (5.6)
#lickr30k
(6.5)
41.0 (6.8)
Random Noise
Cat. Noise
m.cccco
(5.5)
25,5 (6.0)
AUROC
F1
AUROC
F1
Clip Sim.
66.8 (0.)
60.1 (0:4)
64.1 (0.4)
59.2 (0.0)
BIIMedCLIP
LEMoNNa: (Ours)
69.5 (0.7)
66.3 (0.4)
L.M (Ouns)
(1.7)
63.5 (0.8)
70.4 (1.6)
61.1 (0.8)
and filtering data mislabels at scale. Through experiments
Clip Sim.
(0.1)
66.8 (0.3)
76.5 (0.5)
64.4 (0.5)
On Noisy Data
CLIP Pretrain
Lorr (Ours)
LEx (Ours)
80.5 (0.1)
80.0 (0.9)
67.7 (0.5)
77.2 (0.8)
77.0 (0.5)
64.6 (0.3)
multiple datasets with synthetic and real-world noise, we demonstrated LEMoN's effectiveness in detecting label errors and improving downstream model performance.
Clip Sim.
(0.2)
79.5 (0.2)
88.4 (0.6)
76.1 (0.7)
CheXzero
LEMoNax (Ours)
(0.1)
88.4 (0.7)
LEMON.,r (Ours)
91.8((0.2)
80.9 (0.6)
88.5 (0.4)
76.2
Limitations Our work has several limitations. As we pri- marily rely on existing open-sourced datasets, some parts of these datasets may have been used as training data in pre- trained models. We specifically chose pretrained models that take care not to include the test sets of such datasets. Further, we run experiments on real-world healthcare dataset with access controls (mimiccx: to verify our results. Next, we assume that there exists an oracle binary indicator for whether sample is mislabeled. As we saw in practice, real-world mislabels contain much more uncertainty and ambiguity, e.g. due to blurry images (Gao et al., 2017; Basile et al., 2021; Gordon et al., 2021; 2022). Evaluating the effectiveness of our score as a measure of this uncer- tainty is an area of future work.
VTAB benchmark (Zhai et al., 2019). We find filtering with LEMoN marginally outperforms the baseline on average zero-shot accuracy, though both underperform pretraining on the full corpus. Full details are in Appendix 1.9. We additionally conduct an experiment on Datacomp (Gadre et al., 2024) in Appendix I.10.
6.5. World Analysis
We conduct preliminary study of LEMON on real datasets without known label errors. We run LEMON Fix and the CLIP similarity baseline on cirr11, ei farloo, Blickr30 r30k, and mscoco. As no laheled validation set is available, we use optimal hyperparameters from models previously run on each dataset with synthetic noise from Section 6.1 (Appendix I.11). For each dataset, we select the top 200 images from the validation and test splits with the highest mislabel scores. We then manually annotated each sample to determine whether it was mislabeled. Crucially, during labeling, images were randomly selected, so the la- beler is unaware of whether the candidate image originated from the baseline our method. We find that our method outperforms the baseline for every dataset (Table 6). Exam- ples of real-world mislabels are also in Figures 1 and I.5. We present a further comparison of our identified error sets in ci fafarl 0 and c far OC with crowd-sourced labels (Northcutt et al., 2021b) in Appendix I.13.
Acknowledgments
This work was supported in part by a National Science Foun- dation (NSF) 22-586 Faculty Early Career Development Award (#2339381), a Gordon & Betty Moore Foundation award, and a Google Research Scholar award. We would like to thank Walter Gerych and Olawale Salaudeen for their valuable feedback.
Impact Statement
In this work, we have studied the identification and removal of samples deemed to be mislabels. We recognize that not all such samples are truly erroneous, and that these false positives may not be uniformly distributed across the space of images and text. For example, automated detectors such as LEMoN may flag legitimate but atypical examples, such as rare concepts, under-represented languages or dialec- tic phrases, and images from minority classes or groups. Removal of these samples may then lead to downstream models with certain biases. To alleviate these issues, we encourage practitioners to use LEMoN and similar tools
7. Conclusion
In this work, we proposed LEMON, method that leverages the neighborhood structure of contrastively pretrained mul- timodal embeddings to automatically identify label errors in image datasets with natural language text labels.
Our approach a promising step to automatically detecting

---

LEMON: Label Error Detection using Multimodal Neighbors
surface candidates for expert review, rather than unsupervised pruning tool, We view the ability manually examine such samples strength of the filtering approach, over loss-based based approaches, for learning under label noise.
Chen, T., Kornblith, S., Norouzi, M., and Hinton, G. A simple framework for contrastive learning of visual rep- resentations. In International conference machine learning. pp. 1597-1607. PMLR, 2020.
Cui, Z.., Zhang, Y., and Ji, Q. Label emror correction and gen- eration through label relationships. In Proceedings of the AAAI Conference Artificial Intelligence, volume 34, pp. 3693-3700, 2020.
References
Arevalo, J., Solorio, T., Montes-y Gómez, M., and González, F. A. Gated multimodal units for information fusion. 2017.
arXiv preprint
Dai, W., Li, J., Li, D., Tiong, A., Zhao, J., Wang, W., Li, B., Fung, P., and Hoi, S. Instructblip: Towards general- purpose vision-language models with instruction tuning. arxiv 2023. ar preprint arXiv:2305.06500 2, 2023.
Bahri, D., Jiang, H., and Gupta, M. Deep k-nn for noisy labels. In International Conference on Machine Learning. pp. 540-550. PMLR, 2020.
Diffenderfer, J., Bartoldson, B., Chaganti, S., Zhang, J., and Kailkhura, B. A winning hand: Compressing deep networks can improve out-of-distribution robustness. Ad- vances in neural information processing systems, 34:664- 676, 2021.
Balestriero, R., Ibrahim, M., Sobal, V., Morcos, A.. Shekhar, S., Goldstein, T., Bordes, F., Bardes, A., Mialon, G., Tian, Y. et al. A cookbook of self-superviser learning, arXiv preprint arXiv:2304. 122 10, 2023.
Basile, V., Cabitza, F., Campagner, A., and Fell, M. To- ward perspectivist turn in ground truthing for predictive computing. arXiv preprint (tt::2/09.0422) 04270, 2021.
Dosovitskiy. A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., et al. An image worth 16x 16 words: Transformers for image recognition at scale. In International Conference Learning Representations, 2020.
Bernhardt, M., Castro, D. C., Tanno, R., Schwaighofer, A., Tezcan, K. C., Monteiro, M., Bannur, S., Lungren, P., Nori, A., Glocker, B., et al. Active label cleaning for improved dataset quality under resource constraints. Nature communications, 13(1):1161 1661, 2022.
Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Yang, A., Fan, A., et al. The Ilama 3 herd of models. arXiv preprint arXiv:2407.2178. 2024.
Beyer, L., Hénaff, O. J., Kolesnikov, A., Zhai, X., and Oord, A. v. d. Are we done with imagenet? arXiv preprint DXXi:2006.071590 2020.
Feng. C., Tzimiropoulos, G., and Patras, Clipeleaner: Cleaning noisy labels with clip. In ACM Multimedia 2024.
Cai, S., Qiu, L., Chen, X., Zhang, Q. and Chen, L. Semantic-enhanced image clustering. In Proceedings of the AAAI conference on artificial intelligence, volume 37, pp. 0869668677777 2023.
Fu, Z., Song, K., Zhou, L., and Yang, Noise im- age captioning with progressively exploring mismatched words. In Proceedings of the AAAI Conference on Artifi- cial Intelligence, volume 38, pp. 12091-12099, 2024.
Changpinyo, S., Sharma, P., Ding, N., and Soricut, R. Con- ceptual 12m: Pushing web-scale image-text pre-training to recognize long-tail visual concepts. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 3558-3568, 2021.
Gadre, Y., Ilharco, G., Fang, A., Hayase, J., Smyrnis, G.,
Nguyen, T., Marten, R., Wortsman, M., Ghosh, D., Zhang, J., et al. Datacomp: In search of the next generation of multimodal datasets. Advances in Neural Information Processing Systems, 36, 2024.
Chen, H., Wang, J., Shah, A., Tao, R., Wei, H., Xie, X., Sugiyama, M., and Raj. B. Understanding and mitigating the label noise in pre-training on downstream tasks. arXiv preprint 2309 17002, 2023.
Gao, B.-.., Xing, C., Xie, C.-W., Wu, J., and Geng, X. Deep label distribution learning with label ambiguity. IEEE Transactions on Image Processing, 26(6):2825- 2838, 2017.
Chen, S., Gallifant, J., Gao, M., Moreira, P., Munch, N., Muthukkumar, A., Rajan, A., Kolluri, J., Fiske, A., Hast- ings, J., et al. Cross-care: Assessing the healthcare im- plications of pre-training data on language model bias. arXiv preprint arXiv:2405.0 055666 2024.
Gertz, R. J., Dratsch, T., Bunck, A. C., Lennartz, S., Iuga, A., Hellmich, M. G., Persigehl, T., Pennig, L., Gietzen, C. H., Fervers, P., et al. Potential of gpt-4 for detecting errors in radiology reports: Implications for reporting accuracy. Radiology, 311(1):e2221) 2024.
10

---

LEMON: Label Error Detection using Multimodal Neighbors
Ghasemi, A. and Zahediasl, S. Normality tests for statistical analysis: guide for non-statisticians. International jour- nal of endocrinology and metabolism, 10(2):486, 2012.
Jiang. L.,, Huang, D., Liu, M., and Yang. W. Beyond syn- thetic noise: Deep learning on controlled noisy labels. In International conference on machine learning, pp. 4804- 4815. PMLR, 2020.
Gordon, M. L., Zhou, K., Patel, K., Hashimoto, T., and Bernstein, M. The disagreement deconvolution: Bring- ing machine learning performance metries in line with reality. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems, pp. 1-14, 2021.
Johnson, A. E., Pollard, T. J., Greenbaum, N. R., Lun- gren, M. P., Deng, C.-y., Peng, Y., Lu, Z., Mark, R. G., Berkowitz, S. J., and Horng, S. Mimic-cxr-jpg, a large publicly available database of labeled chest radiographs. arXiv preprint arXiv: (at:::991007444 2019.
Gordon, M. L., Lam, M. S., Park, J. S., Patel, K., Hancock, J., Hashimoto, T., and Bernstein, M. S. Jury learning: In- tegrating dissenting voices into machine learning models. In Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems, pp. 1-19, 2022.
Kang, W., Mun, J., Lee, S., and Roh, B. Noise-aware learning from web-crawled image-text data for image captioning. In Proceedings of the IEEE/CVF Interna- tional Conference on Computer Vision, pp. 2942-2952, 2023.
Grattafiori, A., Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Vaughan, A., et al. The Ilama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024.
Karpathy, A. and Fei-Fei, L. Deep ments for generating image descriptions. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 1128-3133 2015.
align-
Grivas, A., Alex, B., Grover, C., Tobin. R., and Whiteley, W. Not cute stroke: analysis of rule-and neural network- based information extraction systems for brain radiology reports. In Proceedings of the 11th international work- shop on health text mining and information analysis. pp. 24 37, 2020.
Kim, T., Ko, J., Choi, J., Yun, S.-Y., et al. Fine samples for learning with noisy labels. Advances in Neural Informu- tion Processing Systems, 34:24137-244
2021.
Lai, Z., Vesdapunt, N., Zhou, N., Wu, J., Huynh, C. P., Li, X., Fu, K. K., and Chuah, C.-N. Padelip: Pseudo- labeling with adaptive debiasing in clip for unsupervised domain adaptation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 16155- 16165, 2023.
Hendrycks, D. and Dietterich, Benchmarking neural network robustness common corruptions and perturba- tions. In International Conference on Learning Represen- tations, 2018
Li, J., Li, D., Xiong, C., and Hoi, S. Blip: Bootstrapping language Language-mmage pre-training for unified rision-languag understanding and generation. In International confer- ence on machine learning, pp. 12888-1228 -19900 PMLR, 2022.
Honnibal, M. and Montani, I. spaCy 2: Natural language understanding with Bloom embeddings, convolutional neural networks and incremental parsing. To appear, 2017.
Li, Li, D., Savarese, S., and Hoi, Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In International conference machine learning, pp. 19730-19742. PMLR, 2023,
Hu, E. J., Wallis, P., Allen-Zhu, Z., Li, Y., Wang. S., Wang, L., Chen, W., et al. Lora: Low-rank adaptation of large language models. In International Conference on Learn- ing Representations, 2021.
Li, Y., I Liang, F., Zhao, I.., Cui, Y., Ouyang, W., Shao, J., Yu, F,, and Yan, J. Supervision exists everywhere: A data ef- ficient contrastive language-image pre- pre-training paradigm. In International Conference on Learning Representations, 2021.
Hu, J. C., Cavicchioli, R., and Capotondi, A. Exploiting multiple sequence lengths in fast end to end training for image captioning. In 2023 IEEE International Confer- ence on Big Data (BigData), pp. 2173-2182. IEEE, 2023.
Huang. B., He, F., Wang. Q., Chen, H., Li, G., Feng, Z., Wang, X., and Zhu, W. Neighbor does matter: Global positive-negative sampling for vision-language pre-training. In ACM Multimedia 2024, 2024.
Liang, C., Zhu, L.,, Shi, H., and Yang, Y. Combating label noise with general surrogate model for sample selection. arXiv preprint arXiv:2310 10463, 2023.
Liang, V. W., Zhang, Y., Kwon, Y., Yeung, S., and Zou, J. Y. Mind the gap: Understanding the modality gap in modti contrastive representation learning. Ad- vances in Neural Information Processing Systems, 35: 17612-17625, 2022.
Huang, R., Long, Y., Han, J., Xu, H., Liang, X., Xu, C., and Liang, X. Nlip: Noise-robust language-imas pre- training. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pp. 926 926-934, 2023.

---

LEMON: Label Error Detection using Multimodal Neighbors
Liao, T., Taori, R., Raji. I. D., and Schmidt, L. Are we learning yet? meta review of evaluation failures across machine learning. In Thirty-fifth Conference on Neural In- formation Processing Systems Datasets and Benchmarks Track (Round 2), 2021.
Natarajan, N., Dhillon, I. Ravikumar, P. K., and Tewari, A. Learning with noisy labels. Advances in neural infor- mation processing systems, 26, 2013.
Nguyen, T., Ilharco, G., Wortsman, M., Oh, S., and Schmidt, L. Quality not quantity: On the interaction between dataset design and robustness of clip. Advances in Neural 2022.
Lin, C.-Y. Rouge: A package for automatic evaluation of summaries. In Text summarization branches oul, pp. 74-81, 2004.
Information Processing Systems,
Northcutt, C.,, Jiang, I.., and Chuang, Confident learn- ing: Estimating uncertainty in dataset labels. Journal of Artificial Intelligence Research, 70: 70:1333-1411 2021a.
Lin, K., Li, L.,, Lin, C.-C., Ahmed, F., Gan, Z., Liu, Z.,, Lu, Y., and Wang, L. Swinbert: End-to-end transformers with sparse attention for video captioning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 17949 17958, 2022.
Northcutt, C. G., Athalye, A., and Mueller, J. Pervasive la- bel errors in test sets destabilize machine learning bench- marks. arXiv preprint arXiv: 103.v 10.. /4749, 2021b.
Lin, T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ra- manan, D.., Dollár, P., and Zitnick, C. L. Microsoft coco: Common objects in context. In Computer Vision-ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12. 2014, Proceedings. Part V 13, pp. 740- 755. Springer, 2014.
Oord, A. v. d., Li, Y., and Vinyals, O. Representation learn- ing with contrastive predictive coding. arXiv preprint arXiv: ftf7::8800033445 2018.
Papineni, K., Roukos, S., Ward, T., and Zhu, W.-. Bleu: a method for automatic evaluation of machine transla- tion. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pp. 311-318, 2002.
Liu, B., Bubeck, S., Eldan, R., Kulkarni, J., Li, Y., Nguyen, A., Ward, R., and Zhang, Y. Tinygsm: achieving, 80% on gsmök with small language models. arXiv preprint arXiv:2312.0922 2023.
Pleiss, G., Zhang. T., Elenberg, E., and Weinberger, K. Q. Identifying mislabeled data using the area under the mar- gin ranking. Advances in Neural Information Processing Systems, 13:17044-70056, 2020.
Liu, H., Li, C., Wa, Q.. and Lee, J. Visual instruction tun- ing. Advances in neural information processing systems, 36, 2024.
Plummer, B. A., Wang, L., Cervantes, C. M., Caicedo, J. C., Hockenmaier, J., and Lazebnik, S. Flickr30k entities: Collecting region-to-phrase correspondences for richer image- image-to-sentence models. In Proceedings of the IEEE international conference on computer vision, pp. 2641- 2649, 2015.
Longpre, S., Yauney, G., Reif, E., Lee, K., Roberts, A., Zoph, B., Zhou, D., Wei, J., Robinson, K., Mimno, D., et al. A pretrainer's guide to training data: Measuring the effects of data age, domain coverage, quality, & toxicity. arXiv preprint 230.:2333333399 2023.
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp.
Loshchilov, and Hutter, F. Decoupled weight decay reg- ularization. In International Conference on Learning Representations, 2018.
PMLR, 2021.
Luccioni, A. S. and Rolnick, D. Bugs in the data: How ima- genet misrepresents biodiversity. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pp. 11382-143990 2023.
Rennie, J., Marcheret, E., Mrouch, Y., Ross, J., and Goel, V. Self-critical sequence training for image captioning. In Proceedings of the IEEE conference on computer vision and paltern recognition, pp. 7008-7024, 2017.
Menghini, C., Delworth, A., and Bach, S. Enhancing clip with clip: Exploring pseudolabeling for limited-label prompt tuning. Advances in Neural Information Pro- cessing Systems, 36:60984-61007, 2023.
Ridnik, T., Ben-Baruch E., Noy, A., and Zelnik-Manor, L. Imagenet-21k pretraining for the masses. Proceedings of 35th Conference on Neural Information Processing Systems, Track on Datasets and Benchmarks, 2021.
Misra, I. and Maaten, L. v. d. Self-supervised learning of pretext-invariant representations. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 6707-6717 2020.
Rottmann, M. and Reese, M. Automated detection of la- bel errors in semantic segmentation datasets via deep learning and uncertainty quantification, In Proceedings
12

---

LEMON: Label Error Detection using Multimodal Neighbors
of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 3214-3223, 2023.
Vasudevan, V., Caine, B., Gontijo Lopes, R., Fridovich- Keil, S., and Roelofs, R. When does dough become bagel? analyzing the remaining mistakes imagenet. Advances in Neural Information Processing Systems, 35: 6720-6734, 2022.
Russakovsky, O., Deng. J., Su, H., Krause, J.. Satheesh, S., Ma, S., Huang, Z. Karpathy, A., Khosla, A., Bern- stein, M., et al. Imagenet large scale visual recognition challenge. International journal of computer vision, 115: 211-252, 2015.
Vedantam, R., Lawrence Zitnick, C., and Parikh, D. Cider: Consensus- based image description evaluation. In Pro- ceedings of the IEEE conference on computer vision and pattern recognition, pp. 4566-45777 2015.
Schrodi, S., Hoffmaann, D. T., Argus, M., Fischer, V., and Brox. T. Two effects, one trigger: On the modality gap, object bias, and information imbalance in contrastive vision-language representation learning, arXiv preprint arXiv:2404.07988, 2024.
Wang. J., Yang, Z., Hu, X., Li, L.,, Lin, K., Gan, 7., Liu, 7.,, Liu, C., and Wang. L. Git: A generative image-to-text transformer for vision and language. Transactions on Machine Learning Research, 2022a.
Schroff, F., Kalenichenko, D., and Philbin, J. Facenet: A unified embedding for face recognition and clustering. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 81 815-823, 2015.
Wang, Y., Xu, J., and Sun, Y. End-to-end transformer based model for image captioning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pp. 2585-2594, 2022b.
Schubert, M., Riedlinger, T., Kahl, K., Kröll, D., Schoenen, S., Segvić. S., and Rottmann, M. Identifying label errors in object detection datasets by loss inspection. In Proceed- ings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 4582-4591 2024.
Wei, J., Zhu, Z., Cheng, H., Liu, T., Niu, G., and Liu, Y. Learning with noisy labels revisited: A study using real-world human annotations. arXiv preprint arXiv:21 10.1100....,.. 2021.
Wu, P., Zheng, S., Goswami, M., Metaxas, D., and Chen, C. A topological filter for learning with label noise. Ad- vances neural information processing systems, 33: 11382-21393, 2020.
Schuhmann, C., Vencu, R., Beaumont, R., Kaczmarczyk, R., Mullis, C., Katta, A., Coombes, T., Jitsev, J., and Komatsuzaki, A. Laion-400m. Open dataset of clip- filtered 400 million image-text pairs. arXiv preprint arXiv:2111.02111 2021.
Xu, K., Ba, J., Kiros, R., Cho. K., Courville, A., Salakhudi- nov, R., Zemel, R., and Bengio, Y. Show, attend and tell: Neural image caption generation with visual atten- tion. In International conference on machine learning. pp. 2048-2057. PMLR, 2015.
Sohn, K. Improved deep metric learning with multi-class n-pair loss objective. Advances in neural information processing systems, 29, 2016.
Young, P., Lai, A., Hodosh, M., and Hockenmaier, J. From image descriptions to visual denotations: New similarity metries for semantic inference over event descriptions. Transactions of the Association for Computational Lin- guistics, 2:67-78, 2014.
Swayamdipta, S., Schwartz, R., Lourie, N., Wang, Y., Ha- jishirzi, H., Smith, N. A., and Choi, Y. Dataset cartog- raphy: Mapping and diagnosing datasets with training dynamics. arXiv preprint 17779, 2020.
Thomas, C. and Kovashka, A. Preserving semantic neigh- borhoods for robust cross-modal retrieval. In Computer Vision-ECCV 2020: 16th European Conference, Glas- gow, UK, August 23-28, 2020, Proceedings, Part XVIII 16, pp. 317-335. Springer, 2020.
Zhai, X., Puigcerver, J., Kolesnikov, A., Ruyssen, P., Riquelme, C., Lucic, M., Djolonga, J., Pinto, A. S., Neu- mann, M., Dosovitskiy, A., et al. The visual task adapta- tion benchmark. 2019.
Zhang, S., Xu, Y., Usuyama, N., Bagga, J., Tinn, R., Preston, S., Rao, R., Wei, M., Valluri, N., Wong, C., et Large- scale domain-specifie pretraining for biomedical vision- language processing. arXiv preprint 2rX.v:2303.009/5. 2(3):6, 2023a.
Thomas, C. and Kovashka, A. Emphasizing complementary samples for non-literal mross- retrieval. In Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4632-4641, 2022.
Tiu, E., Talius, E., Patel, P., Langlotz, C. P., Ng, A. Y., and Rajpurkar, P. Expert-level detection of pathologies from unannotated chest x-ray images via self-supervised learning. Nature Biomedical Engineering, 6(12):1399- 1406, 2022.
Zhang, S., Xu, Y., Usuyama, N., Xu, H., Bagga, J., Tinn, R., Preston, S., Rao, R., Wei, M., Valluri, N., et al. Biomed- clip: a multimodal biomedical foundation model pre- trained from fifteen million scientific image-text pairs. arXiv preprint arXiv:2303.0m 15 2023b.
13

---

LEMON: Label Error Detection using Multimodal Neighbors
Zhu, Z., Dong, Z., and Liu, Y. Detecting corrupted labels without training model to predict. In International con- ference machine learning, pp. 27412-27427. PMLR, 2022.
Zhu, Z., Zhang, M., Wei, S., Wu, B., and Wu, B. Vdc: Versa- tile data cleanser based visual-linguisic inconsistency by multimodal large language models. In The Twelfth International Conference on Learning Representations, 2024.
14

---

LEMON: Label Error Detection using Multimodal Neighbors
A. Theoretical results
A.1. Proof: Theorem 4.1
Suppose that ςy is distributed such that supp(k(y(11p)) (1- S {0, L....,k, For a correctly labeled sample (X, Y), we have that k<y (1 - p) of the neighbors are relevant and have correct labels, and so each contribute (x(X, X) 5m 5m,,, Y), and all remaining samples are either incorrectly labeled, are not relevant Y, and so each contribute dx (X, X'). Since S. (X; Y) is the sum of iid Gaussians, it is also Gaussian, with:
(1- [[Sm(X,))) - (ElEId(X, + d(X, - +... d(X, X RCY (1 (1
= =1(sr](1 p)u22 + E[Gy] p))m
= [1(r](1- (P) m1) + 11
(X, = E[Var S. (X, 1) (X,
= = h2
Var (dxx, X1) + ... + d(X, (ECy d(x, +... d(X, x - Msr)
Var (X, Y |]
B (Sr p)oz + (1 -))oo)}) + Var( (2 (212-)220)))) - -11) + m1)
-
(B[r](L p)oz (1 E[ - p)) + Var(Sy) (((11ppp(pp))) m1)²
Similarly,
S(X')') N( of
Putting all together:
Y') - 5m, 11 -00(0(22 -(=))
Where f [1 () u2), (B(s))(1 p)o (2 El - p))oa} + Var(Sr)(1 p)2 -1)33
and & is the Gaussian CDF. Note that Var(Sr) is finite as Sr is bounded by [0, 1]. Setting > 0 gives Lemma 4.3.
A.2. Empirically Validating Assumption 2
1.2.1. CLASSIFICATION AT DATASET LEVEL
To empirically validate Assumption 2, we first utilize the training sets from the original CIFAR-10 and CIFAR-100 datasets. As these are classification datasets, we naturally define J as: 42 (x1) O Y/2 i.c. all images with the same label are paraphrases. We encode these images using the image encoder from OpenAl CLIP ViT-B/32 (Radford et al., 2021), and utilize the cosine distance as dx- We compute pairwise distance between all 40,000 samples, and categorize these distances into either x' E JIz) or x' & J(r)). We plot a histogram of these distances in Figure A.1. Visually, both of these distributions appear be normal, and also observe that μ4 >> μ42 from Lemma 4.2. We then run a Shapiro- test on all four distributions to test for normality, randomly subsampling 100 samples, the Shapiro- Wilk test not suitable for large sample sizes (Ghasemi & Zahediasl, 2012). We find that in all four cases, the null hypothesis cannot be rejected (p > 0.05), and the test statistics are all greater than 0.97, indicating high degree of normality.
A.2.2. CAPTIONING AT PER- SER-LEMPLE LEVEL
We verify Assumption at a sample-wise wise level, for the analagous H((Y), by conducting an experiment on captions from fli ck 300k. We select 20 random captions, then prompt Llama 3.1 3.1-8B-instruct (Grattafiori et al., 2024) to generate 50
15

---

LEMON: Label Error Detection using Multimodal Neighbors
8
X(E
X(E
6
X' X
6
X'EII()
Density
Density
4
4
2
2
0.0
0.2
0.4 d(x, X')
0.6
0.0
0.2
0.4
0.6
d(x, X')
(a) CIFAR-10
(b) CIFAR-I00
Figure A.1: Histogram of cosine distances in the CLIP image embedding space
paraphrasings of each caption (via sampling with temperature of 1), corresponding 50 samples from H(Y) ) for each caption. For the samples Y' & H(Y), we randomly select 50 other captions from the dataset. To match the support of the Gaussian, we take the distance function to be the log cosine distance (note that this does not change the ordering of the score across samples). We compute this distance using the text encoder from OpenAl CLIP ViT-B/32 (Radford et al., 2021), and plot histograms for each caption. We visualize our results in Figure A.2. Running the same Shapiro-Wi Wilk test from Section A.2.1, we find that of the positive samples, 8/20 are Gaussian, and of the negative samples, 16/20 are Gaussian. Thus, there is some evidence the Gaussianity assumption holds for natural language and complex paraphrase functions.
A.3. Second Theorem
We demonstrate that the embedding models trained via the contrastive multimodal objective are natural noisy label detectors. Theorem A.1 (Contrastive Multimodal Embedding Models Detect Noisy Labels). Let ) IR and consider training dataset D. Suppose that laz X + Rdd is embedding function, and hy y - IRd is a Lipschitz continuous embedding function with constant Ly > O, meaning that for all y.v e.
- n/2( S Lyly 5/l.
For an input E X and its corresponding positive label y E 3, let be random variable drawn from a normal distribution: 7) (N,00 22). Define noisy label yy' + 1, Let dmm (u, v) = ||u v112. which is proportional to /deas v) when 2Φ
||u||| = ||ll| 1. Then, with probability at least 5(c)
where >> 0 and & is the cumulative distribution
function of the standard normal distribution, the following inequality holds:
drma 11gg (x). 125((() (v')) 2 dmm (Rt² (at), byg (i((v) Ly
When Ly is small, this means that the score for the mislabeled sample cannot be much lower than the score for the positive pair with high probability. Thus, we can see that multimodal embeddings are inherently capable of detecting mislabeled pairs, ensuring the distance between the embeddings of positive pairs smaller than that of negative pairs. This motivates the use of d in LEMON and in prior work (Kang et al., 2023; Liang al., 2023).
Proof: Since b is Lipschitz continuous with constant Ly, for any y, v/ E V. we have:
| (v') ny (1)l s Lyly' yl Ly In|
Let demmm (u, w) = ||u 2/12 be the Euclidean distance. Note that when ||u||2 = ||0||2 = (as in our experiments), we have that I|u 21|2 = 22 u v) / 2dess (u, v), and so the two distances provide the same ordering of scores. Applying
16

---

LEMON: Label Error Detection using Multimodal Neighbors
man white hoodie is sitting hamboo scaffolding rear the roof of building holding spade.
female soccer player yellow uniferm in soccer field.
Men dressed up blue uniforms standing formation.
Aman white shirt and spron cuts bird.
girls in green and coach, wearing legs
Woman standing large wearing apron.
group smilling eating.
Little
laughing
The man on the yellow basketball beam tries to score while the men on the opposing team try to block his shat.
A few men stand the backg mund of fruit market illuminated by red light.
Two young men play electric gultiars on stage.
A group of people playing on beach.
Density
Y'EH(X)
Y'εH(X)
Three outside cafe.
the road (hem.
water muddy beach.
sitting
ball flying Lowards him.
detterming ined soccer player kicks the soccer ball sway from the apposing player, near a geal past, while the opposing player stays focused ball.
Man with outstretched arms standing on stone base
Shaft of light in cave shows spelunkers
brown dag is panting on grass during sunny day.
d(Y, Y"
Figure A.2: Histogram of log-cosine distances in the CLIP text embedding space
17

---

LEMON: Label Error Detection using Multimodal Neighbors
the triangle inequality, we get:
dram (bz(2)), Ai (2)(v)) 2 dmm (hz my (v) - 1 - -
When |n| S c, and substituting from Equation (4), follows that:
dmm (1 hig Rg((0) 2 dmm (kz(2), (z). Fig (v)) Lye
Since 17 N/(0, the probability that by) S is:
P(nl =1 2Φ (-2) 5(c),
where & is the cumulative distribution function of the standard normal distribution.
Thus, with probability at least 8(e), we have:
dmm (1)( (gg((()) 2 dmm (h Rig (v)) Lye
When Ly is small, this means that the score for the mislabeled sample cannot be much lower than the score for the positive pair with high probability.
B. Comparison with Thomas & Kovashka (2022)
The goal of Thomas & Kovashka (2022) to identify samples with semantic diversity, which is different from our goal of identifying mislabeled examples. As such, their proposed scores (i.e. YDIS and TDIV) may not be effective in identifying mislabeled samples, As an example, consider the score rpis, which computes the similarity between the original caption, and the captions of its second-degres neighbors text-space. Given a particular caption, e.g. "This is plane from the front view" in Figure 2, it could have second-degree neighhors in text-space that are semantically very similar to this caption (e.g. "A plane facing the viewer"). However, only computing the distance of these captions in text space does not provide any signal for whether the image is correctly paired to the caption. Similarly, the DDV scores also would not necessarily work, as the closeness of neighbors to each other in either modality do not provide signal for whether the original sample is mislabeled.
However, the score from Thomas Kovashka (2022) that would intuitively provide a signal for mislabeling r which computes second-degree neighbors in text space, then examines similarity between images. This is essentially the sum over dx [x, Xmy) terms in Equation (3), but using accond-degree neighbors instead of nearest neighbors. In addition, Equation (3) contains two additional weighting terms (which we show improve label error performance in our ablation experiments). Finally, our proposed score contains the sum of two additional terms, which are not explored in Thomas & Kovashka (2022).
We compare the performance of our method against the r score in the main paper. and show performance of all four individual scores and two combined scores from Thomas & Kovashka (2022) in Appendix I.8.
18

---

LEMON: Label Error Detection using Multimodal Neighbors
C. LEMOO Algorithm
Algorithm 1: LEMON: Label Error Detection Using Multimodal Neighbors
Input: Dataset D (x)
y:) Multimodal encoders hi hgg Distance functions dx. dy
Hyperparameters: k, 8, 7, T1,m, T2mn 71,m, 72.m
Output: Scores { N
Cache embeddings hg (x,) and by (y:) for (x,, D kD (x.) (xr)
Cache d (x). = [lng (x.)lla (1 (y.)l2
for (x..V:) E ;
for i 1 N do
Find indices {ng) of ls nearest neighbors of from D \ ((x+,)+) using dx; // dx can use cached hig
Find indices {my} of nearest neighbors of y, from D y(), using dy // dy can use cached by
Compute 8n,s := 1 Ei= (y', W NnyJe T.adx(((xx)))) T2 ad (Xn,:
Compute 5m, = 3 2i=1 dx (x,, Xm, )e - (y - mm (Xmy smy)
:::= dma (i) Ban, 78m.
return
For each image-caption pair in the dataset, we first compute how similar the image and caption are to each other using a pre-trained CLIIP model (dmm)), which gives basic measure how well they match. To compute 8ms we compute the nearest neighbors of the caption among other captions in the dataset. For each neighbor, we look how similar their corresponding image is to the original image. The intuition is that if sample is correctly labeled, the image should be similar to images of other samples with similar captions. We weight each neighbor based how close it is to our original sample and how well-matched the neighboring pairs themselves are. Finally, we repeat this for nearest neighbors in the image space to get sp. LEMON is then the weighted sum of these three scores.
Table C.I: Notation and definitions used in Section 3.
Symbol/Notation
Meaning
D
Dataset consisting of samples (x
x, X
First modality and its corresponding space (e.g., images)
y.y
Second modality and its corresponding space (e.g., text)
f
Oracle function that assigns a binary mislabel indicator 2
2
Mislabel indicator for sample i (2; 1 if mislabeled, 2, otherwise)
y(x,y) =
Model output score
dx, dy
Distance functions in N' and y spaces
r(
Ball of radius centered at x in X' space
Byy,r)
Ball of radius centered at y in ) space
(x(x)
Radius such that the ball ((x, r) contains at least k neighbors
Xny
Nearest neighbor j in space Nearest neighbor j in y space
Ymn
hg (hi) 12
Multimodal encoder mapping X' and y to Rd
dmm (x. y)
Multimodal distance between and y Score component based x's neighbors, see Equation (2). Score component based y's neighbors, see Equation (3). Hyperparameters weighting 8,, and 8m
(X, y; D
8m Xx y, D)
B.Y
Ti,ns T2, T1,ms T2.m
Hyperparameters for weighting terms in 8n and Number of nearest neighbors
19

---

LEMON: Label Error Detection using Multimodal Neighbors
D. Data Processing
D.I. Classification
We utilize CIFARION (cifar10) and CIFARIOON (cifar100) object detection (Zhu et al., 2022) datasets for all classification-bant based experiments. Each image is associated with label indicating the primary object present the image. These datasets contain 50,000 image-label pairs, with a clean and noisy label available per image. The noisy labels examples of real human errors within the dataset. Further, we also generate synthetically noised labels as deseribed in the main text. All images resized to 224x224, center cropped, and normalized using mean and standard deviations corresponding to CLIP during the pre-processing stage. These two datasets are released under the Creative Commons Attrbution-NonCommerii 4.0 license.
For mini ImageNet and stanfordCars, we use the "red" datasets from Jiang et al. (2020), which contain noise from real-world web annotators. We split the full dataset (containing all annotations) into 75%/12.5%%/2 12.5% train/val/test sets, stratifying by the mislabel flag. The annotations are licensed by Google under CC BY 4.0 license, and the images are under CC BY 2.0 license.
To generate the "text" modality for these classification datasets, we utilize the label name correspond to each class. For example, class O in ci cifarl0 is "airplane", and this is the caption associated we associate with all images of that class. In contrast to the caption-bas based datasets, there will be multiple k-nearest neighbors in the text modality with zero distance (i.e., with the same class label).
D.2. Captioning
We preprocess MSCOCO (Lin et al., 2014) and Flickr30k (Young et al., 2014) by using the Karpathy split (Karpathy & Fei-Fei, 2015), and then selecting one random annotation from the ones available. For the MMIMDB dataset (Arevalo et al., 2017), we utilize the plot outline as the text, and use the dataset splits provided. For MIMIC- (Johnson al., 2019), use all images in the database and the provided data splits, and extract the findings and impression sections from the radiology note for the text modality. Images were normalized and transformed using the same procedure described above.
For downstream captioning, we use the pre-trained tokenizer and image processor corresponding to the pre-trained model (GIT (Wang et al., 2022a)) to pre-process image and captions.
Note that flickr is available under Flickr terms of use for non-commercial research and/or educational purposes mscoco available under Creative Commons Attribution 4.0 License. mmi mdb available for personal and non- commercial use". Finally, mimiccxr is available under the PhysioNet Credentialed Health Data License 1.5.0%.
E. Baseline Methods
E.1. Classification
TRAINING-DEPENDENT
AUM (Pleiss et al., 2020): This model assumes access to a classifier that can predict the class that an image likely belongs to. Then, the margin of difference between the prediction probability from the trained classifier for the assigned class and the class with the (next) highest probability is computed and averaged over training epochs. This score is thresholded to identify potential label errors.
Datamap (Swayamdipta al., 2020): Similar to AUM, this method requires training classifier the dataset. In this baseline, it is assumed that instances with label errors are 'hard to learn', and thus low confidence in prediction throughout training epochs. To produce single score, we combine the mean and standard deviation of the probability associated with the assigned class into single score?.
Confident Learning (Northcutt et al., 2021a) is designed to identify labeling errors in classification datasets by modeling
potttps:/sshanes cs. wtw.:aaannnnn.....ooii....ooo... ht..iimmos...tt///ettttttttoooo
/httpe://dcveloper imdb. com www//oon-oommereeeooooooooooceec http:///ooooooooorreeeaaaaaaaaaa
org/ content /ttp://wwwoooo/iioww--eeeeeeeeee 0.00/
"We experimented with different strategies, and the square root of the product of the mean and (1-standard deviation) and (1-mcan) standard deviation led to comparable, high validation Fl scores.
20

---

LEMON: Label Error Detection using Multimodal Neighbors
the relationship between true class labels and noisy ones. It sets thresholds for each true-noisy label pair. Using these thresholds, the model employs predicted class probabilities rank predictions for each class, filtering the noisy data.
TRAINING-FREE
CLIP Logits (Liang et al., 2023): CLIP is used as a zero-shot classifier to obtain the softmax-based probability for the assigned class. This value is then thresholded to identify label errors. Recently, (Feng et al.) used a similar zero-shot prediction jointly with a semi-supervised training approach for learning in the presence of label noise.
CLIP Similarity (Kang et al., 2023): The distance (either euclidean or cosine) between image and text embeddings from CLIP are computed and thresholded.
Deep k-NN (Bahri al., 2020): The proportion of k nearest neighbors with the same label is computed for each image of interest. Prior works have utilized different representations for obtaining neighbors, including logits and representations from pre-trained (Zhu et al., 2022) vision models. Here, we use pre-trained representations from CLIP.
SimiFeat (Zhu et al., 2022) uses nearby features to detect noisy labels under the assumption that local groups of features share clean or noisy labels. SimiFeat-V uses local voting and SimiFeat-R leverages ranking to detect noisy labels based on HOC estimator. The binary outputs produced are used for all score computations. Note that the difference between Simifeat-V V and deep k-NN is in the data processing and augmentation.
Discrepancy (Thomas & Kovashka, 2022) finds second-degres nearest neighbors in the text space, then computes the average distance of these neighbors to the original sample in image space. We utilize the same CLIPP model to compute semantic distance here as in LEMON.
Note that AUM and Datamap use the entire dataset as the reference set for label error detection, whereas SimiFeet uses both the train and test sets as reference sets, following the implementation in their original papers (Pleiss et al., 2020; Swayamdipta et al., 2020; Zhu et al., 2022).
E.2. Captioning
PRE-TRAINED OR SUPERVISED
LLaVA (Liu al., 2024): We prompt LLaVA (v1.6-vicuna-13b) with the following prompt: The proposed caption for this image is "()". Is this caption correct? Only answer with "Yes" or "No". We examine the probability distribution over the first non-special token, and find the likelihood of the token with the highest probability. If the corresponding token in lower case starts with "yes", we return 1- this probability as the mislabel score. Otherwise, we return the probability.
VDC (Zhu et al., 2024): As the original VDC paper does not explore the captioning setting, we make the following modifications to adapt it to our setup:
As we only utilize open-source models in our method, to preserve fairness, we also implement VDC with open- source models. Specifically, we use Generation (VQG) and Visual Answer Evaluation (VAE) stages (note that the VDC paper uses the OpenAl APD), and InstructBIIIP. PatruuttlIP--Viuuna-7 (Dai et al., 2023) as the VL.LM in the Visual Question Answering (VQA) stage (as in the VDC paper).
(Dubey et al., 2024) for the LLM in the Visual Question
In the VQG stage, instead of generating specific questions for each class, we generate six specific questions for each caption. We slightly modify the VQG prompt (Table 8 in the VDC paper) to adapt it to captioning, and omit providing the label set, as the set of all possible captions is infinite. We keep the two general questions used in the VDC paper.
CapFilt (oracle-like): We generate predictions using a pre-trained model trained to distinguish between high-quality MSCOCO and noisy synthetic captions (Li et al., 2022). This forms an oracle-like, supervised baseline. We use pre-trained checkpoint (pre-trained on the train split of MSCOCO)² for producing prediction scores.
'Note that this score can only take value in {0, 1/k, 2/k,..,, 1}.
https:: //tgggiiiiii//ee wag.ace.ee... acee.oo c0/ Sa Salesfafee mealesffrree force/teee /b1 ip- tm- wtw.tmm---eoooee b-s--aasecc-- coco
21

---

LEMON: Label Error Detection using Multimodal Neighbors
UNSUPERVISED
Datamap: We compute the causal language modeling loss across training epochs and compute the product the mean and variance in loss across epochs. That is, expect captioning loss for instances with label enrors to be consistently high. We train captioning models for epochs, with LoRA rank set to 4, and maximum length of corresponding to maximum model length 10 for the finetuning task.
Confident Learning:
We adapt this approach for dual-modality datasets, such as image-text pairs, by clustering text embeddings to serve as class labels for noise detection.
DOWNSTREAM-TASK UNAWARE
Deep KNN: We cluster captions similar to confident learning, adapting classification baseline.
CLIP Similarity: This is the same setup as classification.
Discrepancy: This is the same setup as classification.
F. Compute Setup
We our experiments on a shared Slurm cluster. Most experiments used one RTX A6000 with 48 GB VRAM, 10 CPU cores of Intel Xeon Ice Lake Platinum 8368, and 50 GB RAM.
G. Hyperparameters in Label Error Detection
The hyperparameters in each case were selected based on the validation set F1-score. Note that LEMON does not require hyperparameter tuning. Baseline code included in the supplementary material. For SimiFeat-V and -R, we use the official open-sourced implementation directly.
G.1. Classification
The search space for each method:
1. AUM, Datamap: learning rate {5e 5, 5e 6). training for epochs E (5, 10)
2. Confident learning: learning rate E (5e 6, 5e 5). upto 30 epochs with early stopping with patience of 10.
3. CLIP Sim.: cosine distance metric, no other hyperparameters
4. CLIP Zero shot: distance metric
5. Discrepancy: & E 21,2,5, 5, 10, 15, 20, 30, 50}
6. deep k-NN: k cosine distance metric
7. Simifeat: we set A: 10 following the original paper (Zhu et al., 2022).
G.2. Captioning
For most baselines requiring a class index-obtained by clustering captions-we set the number of clusters to be 100.
1. LLaVA: Small amount of prompt tuning. The optimal prompt selected was The proposed caption for this image is "'''. 'Norr,'
this caption correct? Only answer with "Yes!! or
2. Confident learning: learning rate E (5e 5e 5). upto 30 epochs with early stopping with a patience of 10, number of clusters for captions!2
10This longer than captions the train sets of all datasets except medical dataset.
" Note that we experiment with training for fewer epochs to avoid memorization, following (Pleiss al., 2020).
For eexr, we use 10 clusters.
22

---

LEMON: Label Error Detection using Multimodal Neighbors
3. Discrepancy: & E (1, 10, 20, 30,50
4. deep k-NN: & 21, 10, 15, 20, 30, 50}, number of text clusters
5. VDC: no tunable hyperparameters
G.3. Our Method
We search the following hyperparameters for our LEMONopT:
1. h E (1,2,5, 10, 20, 30, 50)
2. Distance metric (either cosine or cuclidean)
3. 8. 7: Fi.n 72.n. Tim, 72mm We take the hyperparameter which achieves the best validation set Fl from these two strategies: (1) Using Scipy's mi nimi ze function, with initial guess (1, 1,,,, 1), and with explicit bounds. (2)
Using a grid search with the following grid:
B E {0,5, 10, 15, 100}
7 E 10, 15, 100}
Tl,ns T2,n: T1,,, T2.m E {0, 5,
G.4. Optimal Hyperparameters
Optimal hyperparameters for classification datasets can found in Table 1. and optimal hyperparameters for captioning datasets can be found in Table G.2.
Table G.1:
Optimal hyperparameters for methods shown in Table L.I. Note that Simifeat, VDC, CLIP Sim., and LEMON have no tunable hyperparameters.
cifar10
cifar100
mini miniImageNet
stanfordCars
LR 5E-6
LR = 5E-5
LR = 5E=
LR = 5E-5
AUM
Epochs =
Epochs =
Epochs 10
Epochs 10
LR= 5E-6
=R= = 5B-5
=R=
LR = 5E-5
Datamap
Epochs =
Epochs =
Epochs = 10
Epochs = 10
LR=5c-06
LR=5c-06
LR=5c-05
LR=5e-05
Confident
Epochs=30
Epochs=30
Epochs=30
Epochs=30
Batch size= 128
Batch size=128
Batch size=128
Batch size=128
CLIP Logits
Cosine distance
Cosine distance
Cosine distance
Cosine distance
Discrepancy
k=20
k=50
k=30
k=20
k=50
k=20
k=50
k=30
Deep k-NN
cosine distance
cosine distance
cosine distance
cosine distance
k=50
k=20
k=50
k=15
cosine distance
cosine distance
Euclidean distance
Euclidean distance
3 = 20
B = 2.14
B = 0.664
B = 0.631
Y =
7 = -0.024
? = 0.395
7 = 0.431
LEMON Nor
TIn 0
Jim -1.71
7I, 1.91
JI. 0.898
72n 5
F2nx 4.85
22n 1.04
22 -0.192
Tim 0
Tim = -0.068
1.m = 1.00
Tim = 0.0
72m 5
22m = -0.019
72m = 1.35
72.m = 00.01
23

---

LEMON: Label Error Detection using Multimodal Neighbors
Table G.2: Optimal hyperparameters for methods shown in Table 3. Note that LLaVA, VDC, CLIP Sim. and LEMON have no tunable hyperparameters.
flickr30k
mscoco
mmimdb
mimi ccxr
Batch size = 16
Batch size 16
Batch size = 16
Batch size = 16
Datamap
Epochs =
Epochs = 3
Epochs =
Epochs = 3
LoRA rank =
LoRA rank =
LoRA rank
LoRA rank 4
Discrepancy
k=5
k=10
k=10
k=10
k=50
k=50
k=20
k=50
Deep k-NN
n_cluster=100
n_cluster=100
n.cluster=100
n.cluster=100
LR=5e-06
LR=5e-06
LR=5e-05
LR=5e-06
Epochs=30
Epochs=30
Epochs=30
Epochs=30
Confident
Batch size= 128
Batch size=128,
Batch size=128
Batch size=16
n_cluster=10
n.cluster=100
n.cluster=10
n.cluster=10
k=30
k=30
k=10
k=30
cosine distance
cosine distance
Euclidean distance
cosine distance
3 = 0.092
= 5.324
8 = 1.001
3
Y = 0.177
7 11.057
7 = 1.202
? = 10
LEMON
Ti, 0.274
7I, 5.143
Th. 0.983
7I,
72, = 0.074
72, 10.498
22n 1.000
72, 10
TImm 0.072
71.m -7.233
TIm
= =4.450
71m 5
72m = 0.0
72.m 15.637
2.m = =1.080
72m = 10
H. Hyperparameters in Downstream Models
H.1. Classification
We train Vision Transformer (ViT)- based image classification (Dosovitskiy et al., 2020) model pre-trained on ImageNet- 21k (Ridnik et al., 2021) and fine-tuned on ImageNet 2012 (Russakovsky et al., 2015) with an additional linear layer. We add linear layer above the last hidden state. We train for up to 30 epochs with an initial learning rate of and early stopping with a patience of 4. We tune learning rates on the fully noisy set, for learning rate in E {5e - 4, le - 3, le - 5] (with cosine annealing learning rate scheduling).
H.2. Captioning
The hyperparameter tuning grid for the captioning m are: learning rate of le - 4 le - 5, batch size: 16, maximum number epochs: 10. The model checkpoint from the epoch with the lowest validation loss used for caption generation test time. For text generation, optimizer (Loshchilov & Hutter, 2018), with cosine scheduling for learning rate with 1000 warmup steps. We tune LoRA rank in {4,16} based on BLEU-4 scores on the validation set. We also experimented with a lower learning rate of le 5 on the fully noisy set, and observed higher performance (in terms of validation BLEU-4) with a higher learning rate of 1e - 4 on fl flickr3Ck.
beam search with 4 beams, following Wang et al. (2022a). We use the AdamW
Here, for finding best epoch during training, we assume that validation loss is correlated with language model quality. We leave strategies such as self-critical training validation (Rennie et al., 2017) future work. We verified that retaining the model from the last epoch leads to similar trends in results both LEMON and the baseline perform within 1 point of each other on both datasets (similar to the results with loss-based checkpointing), and filtering out noisy data improves downstream captioning performance. Note that pre-conference, we benchmarked results without loss masking of padded tokens during training and observed similar result trends.
Table I.9 for models trained up to 10 epochs. We find that
"https://huge www.cc..c...ooo..o.....o..o "https: http:/::////gg/n////// co/mi wttp://///....oo..ott.o.t.t..ta.
htt.face.co/oooooo co/googl wwweeeittaaeeeeeoooeee 2-224
24

---

LEMON: Label Error Detection using Multimodal Neighbors
100%
100.0%
CUP Sim.
AUPRC
80%
AUPRC
95.0%
F1 Score
80%
Score
90.0%
Ours
90.0%
60%
85.0%
CLP
E1 80.0%
40%
80.0%
Our
40%%
80%
60%
20%
40%
60%
80%
40%
60%
80%
Noise
Noise
(a) AUPRC on mmimch
(b) AUPRC on macoco
(c) on mm mnimddb
(d) Fl on mscoco
AUROC
AUROC
95.00%
86 10.00%
CLIP Sim.
24.00%
85. 00%
20%
40%
60%
80%
20%
(e) AUROC on mmimcclo
(I) AUROC on mscoco
Figure 1.1: Test-set performance of LEMON OpT compared the CLIP Similarity for varying levels of the synthetic noise.
I. Additional Experimental Results
L1. Label Error Detection in Classification Settings
Full results on classification datasets using the noise types bolded in Table (including AUPRC) be found in Table I.1.
The performance of all basclines and our method on the two types of synthetic errors shown in Table I.2, all noise level of 40% (comparable to the amount of error in the noisy CIFAR datasets).
1.2. Label Error Detection in Captioning Settings
Full results classification datasets using the noise types bolded in Table 1 (including AUPRC) can be found in Table I.3.
Results on the remaining synthetic noise types (at 40%) can be found in: flickr30k: Table I.4, mscoco: Table I.5, mmimdb: Table 1.6, and mimic c- cxr: Table 1.7. Across all datasets and noising types, we find that our model outperforms other oraeeecccess baselines.
I.3. Varying Noise Level
We show the AUROC for varying noise levels in Figure I.I.
1.4. Robustness to Hyperparameters
We show the test-set F1 of LEMON for varying 3 and y, keeping all other hyperparameters at their fixed optimal values, in
Figure 1.2. In Table I.8, we show the performance of LEMoN when hyperparameters are fixed (at & = 30, cosine distance, 3 = 7 = 5, Ti,n T1.m = 0.1, and T2,n = 72.m = 5) versus when they are optimized using labeled validation set. Note that not computed as it requires external information to select a threshold.
1.5. Ablations of our Method
Ablations of our method can be found Table I.10 and Table I.11.
L6.. Runtime Comparison
We compare the runtime of LEMON with baselines in Table 1.12.
25

---

LEMON: Label Error Detection using Multimodal Neighbors
155
3.13
0.94
2,63
5:26
0.94
1033
9:30
181
0.92
13:16
15.79
12.5
21.05
14.00
2368-
20:31
3421
0.88
21.88
2686
25.0
0.86
26:36
44.74
28.13
0.86
47.37
50.0
1.75
3.5
5.25
7.01
12.20
14.01
15.76
17.51
19.27
22.17
2452
26
28
29.7
33.28
0.0
1.0%
2.11
3.16
6.3
737
8.42
9.47
10.55
12.3
13 1368--
74.74
15.79
17.89
18.99
20.
(a) cifar10, asymmetric noise
(b) cifar10, symmetric noise
0.0
0.0
3.16
0.88
1.05
3.16
12.63
421
15.79
5:6
22:11
737
0.84
25.26
8.42
28/42
9.47
0.82
34.74
37.89
1243
41.05
0.80
13.68
47.37
0,80
50.53
0.78
1684
$3.60
17.89
60.0
20.0
1.05
2.11
3.16
4.23
5.28
6:22
737
8.42
9.47
10.53
11.88-
11.8-
14.74
15.79
16.04
0.0
1,05
2.11
3.16
5.26
6.32
2.37
8.42
9.47
12.63
13
14
15 79
17.09
18:95
20.0
(c) cirar1, real noise
(d) cifar100, asymmetric noise
0.0
0.90
3.16
4.74
4.74
6.32
632
0.80
11.05
12.6
12.3
0,72
14.21
1421
0.88
0,70
1895
18.5
2051
0.84
23
2368
25.2
2526
0.66
30.0
0.82
30.0
0.64
105
2.11
3.16
4.21
5.26
6.22
737
8.42
9447
10 53
S8
6R
74.74-
25 79
16. 04
18.95
20.0
105
2.11
3.16
2.26
6.32
7.37
4242
51
58
61
68
15 7M
$7.09
18.99
20.0
(e) farr100, symmetric noise
(f) ciaar100, real noise
1.64
0.0
0.9800
2.1
4.91
6.54
433
3226
11.45
0.9750
652
7.61
16.36
17.99
0.9700
11.05
10.77
0.82
21.27
0.80
22.9
16.3
26.17
27.81
0.9650
17.39
31.08
20.65
523
6-9--
8.71
10.45
12.19
13.0K
15.08
17.42
19.36
22.65
24.23
2613
77.01
19.01
31.35
33.1
0.0
1.99
3:3-
507
10:14
11:31
13:52
15,21
(0.28
21.97-
23.06
25.35
77.04
42
32.11
mscoco, random noise
mscoco, noise
26

---

LEMON: Label Error Detection using Multimodal Neighbors
0.0
1.74
348
38.36
855
24:21
8.60
22.16
68.32
77.37
0.970
13.9
15.64
110.53
99.47
18:11
12158
0.965
20.85
2258
18
754-7
26.06
27.8
176
0.960
33.01
210.0
0.955
105
5.78
7.7
9.63
11.50
13.00
19.29
21.18
75.
25.00
30.
52.73
$6.15
36.39
1899
25.6
$1588
37.89
44.22
26.8
63.1
75.79
R2.1
94.7
10 00
107.3
113
:20.0
ii
noise
Flickr30k, random noise
0.54
1.07
-0.5
2.11
1.05
421
526
2.88
3.2
632
4.22
5.36
0.75
1053
8.42
9.47
13.8-
0.60
75
1474-
0.65
0.59
1759
0.55
9.11
18.95
0.58
0.60
0.0
0.51
1.02
1.52
2.03
2.54
2.99
1.57
5.59
6.6
7.11
7,02
6.32
8,63
9.65
LO0
2,11
3116
6.32
7.37
8.42
10.53
12.33
1368
14.74
15.79
10.95
20.0
(k) Elickr3 Ck, noun noise
(I mimiccxx. random noise
0.0
1555
0.0
3.11
0.78
3.16
8:1
6.2
1.17
0.77
6.32
737
12
0.76
13.99
10.53
11558
18:65
17.1
1368
2021
14.74
15
180
24.77
17.89
18.95
0.575
26442
27.9
0.72
0,71
116
4.74
6.37
7,88
9.4
05
12.63
1422
10 79
96
20 53
22.11
23
15.25
28
28-42
30.0
157
5:14
4.7
7.84
941
90
12.54
34.11
20.30
21.99
03.52
55,00
26:06
28:23
19.79
(m) mimi ccxr, cat noise
mimiccxr,
(n) mm mdb, random noise
0.0
0.70
123
4.74
0.68
1.24
633
2.46
0.70
11.05
4.3
12.1
14.21
491
553
0.68
17337
6.75
18.55
7.37
7.88
0.60
0.64
23.88
951
2522
0.58
052
0.62
30.0
0.58
11.66
109
211
3.16
4.21
5,26
6.22
8-42
3.47
JA.53 18
13.09
14.74
15.79
16.00g
9..99
18.30
200
0.0
1.77
191
218
101
808
2.72
6.26
8.9
10.17
10.0
11.44
12 20.08
(o) mmimdb, noun noise
(p) mmimdb, noise
Figure 1.2: Fl of our method for varying and y, keeping all other hyperparameters their fixed optimal values.
27

---

LEMON: Label Error Detection using Multimodal Neighbors
Table L1: Label error detection performance across classification datasets, for the bolded noise types in Table 1. We separate AUM, Datamap, and Confident learning, as they require training classifier from scratch. Bold denotes best score within each training approach.
Dataset
Method
Training-free
AUROC (%)
AUPRC (%)
F1 (%)
AUM
98.3 (0.1)
97.9 (0.1)
92.9 (0.1)
Datamap
X
98.2 (0.1)
97.6 (0.1)
92.2 (0.5)
Confident
89.6 (1.4)
86.1 (1.8)
88.2 (1.7)
CLIP Logits
95.5 (0.2)
93.9 (0.3)
86.2 (0.6)
cifar10
CLIP Sim.
92.2 (0.2)
90.2 (0.4)
82.3 (0.3)
Simifeat-V
90.8 (0.3)
88.0
88.2 (0.5)
Simifeat-R
88.8 (0.1)
85.1 (0.5)
85.9
Discrepancy
77.1 (1.9)
70.4 (2.7)
66.4 (2.2)
Deep K-NN
97.8 (0.1)
(0.2)
91.4 (0.6)
LEMON FI (Ours)
97.7 (0.2)
96.8 (0.3)
90.9 (0.1)
LEMONoPT (Ours)
98.1 (0.0)
97.4 (0.1)
92.0 (0.2)
AUM
92.3 (0.3)
90.0 (0.5)
81.1 (0.3)
Datamap Confident
91.8 (0.3)
(0.5)
80.8 (0.5)
78.6 (0.4)
68.8 (0.9)
73.7 (0.5)
CLIP Logits
84.9 (0.7)
80.3 (1.2)
72.0 (0.9)
cifar100
CLIP Sim.
80.8 (0.9)
75.2
68.7 (1.1)
Simifcat-V V
79.5 (0.1)
71.1 (0.2)
73.3 (0.2)
Simifcat-R
81.4 (0.1)
70.5 (0.5)
76.1 (0.1)
Discrepancy
66.0 (1.5)
57.4 (2.3)
58.9 (0.8)
Deep k-NN
87.4 (0.3)
77.9
75.7 (0.3)
LEMON (Ours)
88.9 (0.7)
(1.1)
75.4
LEMoN (Ours)
90.8 (0.0)
(0.3)
78.4
AUM
83.1 (0.2)
(0.5)
68.3 (0.4)
Datamap
x
85.1 (0.3)
70.1 (0.8)
70.6 (0.2)
Confident
59.5 (0.7)
(0.8)
37.7 (1.5)
CLIP Logits
90.0 (0.2)
80.9 (0.5)
77.1 (0.2)
mini ImageNet
CLIP Sim.
89.3 (0.2)
80.7 (0.3)
76.1 (0.4)
Simifeat-V V
68.1 (0.2)
52.9 (0.4)
54.8 (0.3)
Simifcat-R R
65.1 (0.3)
48.6 (0.5)
Discrepancy
79.5 (0.2)
(0.9
(4.0
Deep k-NN
83.2 (0.2)
70.9
68.5 (0.2)
LEMON (Ours)
89.5 (0.2)
81.5 (0.3)
70.7
LEMON (Ours)
90.0 (0.4)
79.7 (6.1)
76.9 (0.2)
AUM
70.4 (2.3)
42.0 (1.0)
47.2 (3.1)
Datamap
X
72.2 (1.7)
39.5 (0.2)
50.4 (2.1)
Confident
60.7 (0.3)
29.9 (0.2)
39.9 (0.6)
CLIP Logits
68.8 (0.1)
39.7 (0.9)
47.3 (0.5)
stanfordCars
CLIP Sim.
69.8 (0.4)
40.7 (1.0)
(0.5)
Simifeat-V V
63.6 (1.3)
43.5 31.4
Simifeat-R
66.8 (0.3)
34.9
(7.9
Discrepancy
65.7 (0.3)
33.3 (0.6)
44.3 (0.7)
Deep k-NN
71.5 (0.6)
41.7 (0.7)
49.1 (0.6)
LEMONax (Ours)
72.6 (0.7)
44.9 (1.4)
47.7
LEMON OPT (Ours)
73.1 (0.5)
40.5 (0.4)
51.3
28

---

LEMON: Label Error Detection using Multimodal Neighbors
Table 1,2: Label error detection performance on synthetic errors for classification datasets
AUPRC
mean
std
mean
std
mean
std
Dataset
Flip Type
Method
AUM
93.7
0.6
86.6
0.6
86.6
1.1
Datamap
93.6
0.4
86.3
0.8
0.9
Confident
87.3
4.7
78.0
7.3
84.7
5.7
CLIP Logits
98.8
0.2
97.9
0.3
93.2
0.4
aymumetric
CLIIP Sim.
97.0
0.2
95.3
0.1
89.4
0.5
Simifeat--
Simifeat-R
71.4
1.0
57.6
1.3
64.3
1.4
Discrepancy
63.2
3.3
1.4
58.5
1.2
Deep K-NN
85.2
0.7
66,2
0.9
79.5
0.8
LEMoNN.
97.5
0.2
94.8
0.6
90.2
0.8
cifae10
L.MoNorr
98.8
0.2
97.8
0.5
93.9
0.3
Datamap
99.8
0.0
99.7
0.D
98.0
0.2
Confident
CLIP Logits
98.5
0.0
97.9
0.1
92.2
0.1
symmetric
CLIP Sim.
97.1
0.1
95.9
0.2
89.5
0.1
Simifeat-V
96.5
0.2
93.9
0.5
94.3
0.2
Simifeat-R
95.4
0.1
91.8
0.2
0.2
Discrepancy
84.5
2.3
78.2
3.0
73.2
1.1
K-NN
LEMoNox
99.5
0.1
99,2
0.1
95.9
0.3
LEMONorT
99.6
0.1
99.4
0.1
96.8
0.2
AUM
82.4
2.0
67.5
2.6
75.2
1.5
Datamap
Confident
71.7
2.5
56.6
3.D
67.3
2.7
CLIP Logits
96.6
0.3
94.8
0.5
88.3
0.7
asymmetric
95.1
93.1
0.5
0.6
Simifeat--
65.4
1.0
52.8
1.2
$7.0
1.3
Simifeat-IR
60.1
0.5
45.9
0.7
$6.2
1.3
Discrepancy
62.0
0.6
50.8
1.2
58.1
0.9
Deep k-NN
62.6
0.3
47.7
0.8
63.2
0.5
LEMoNns
0.3
0.4
oifar100
L.MoN
96.7
0.3
95.3
0.4
$8.4
0.5
AUM
99.2
0.3
98.9
0.4
95.2
1.1
Datamap
99.2
0.4
98.8
0.7
95.0
1.5
Confident
87.8
0.6
77.8
1.0
85.2
0.5
CLIP Logits
96.8
0.1
95.2
0.3
89.1
0.4
symmetric
Simifeat--V
CLIP Sim.
95.5
0.2
93.5
0.3
86.6
0.9
Simifeat-R
90.5
0.3
79.8
1.1
84.1
0.9
Discrepancy
82.8
0.9
75.8
1.0
70.7
0.5
Deep K-NN
96.7
0.1
91.7
0.3
91.0
0.4
LEMoN
98.4
0.1
97.7
0.2
92.0
0.1
LEMoNort
99.0
0.0
98.7
0.1
29

---

LEMON: Label Error Detection using Multimodal Neighbors
Table 13: Label error detection performance captioning datasets, for the bolded noise types in Table 1.
Dataset
Method
AUROC (%)
AUPRC (%)
F1 (%)
LLaVA
79.3 (0.8)
58.5 (0.2)
65.0 (1.1)
Datamap
80.6 (1.3)
63.3
66.6 (0.6)
Discrepancy
(0.6)
51.8
59.3
VDC
92.9(1.1)
(0.3)
(1.6
flickr30k
Deep k-NN
71.1 (0.4)
(1.0
59.2
Confident
63.1 (0.9)
42.1
54.0
CLIP Sim.
94.8 (0.5)
92.8 (0.5)
84.2
LEMoN Fx (Ours)
(0.2)
92.0
LEMON (Ours)
(0.2)
83.6 (1.4)
CapFilt (Oracle)
98.6
98.1 (0.1)
93.1 (0.7)
LLaVA
80.3 (0.1)
63.4 (0.3)
74.9 (0.3)
Datamap
85.8 (0.4)
80.0
74.3 (0.3)
Discrepancy
72.7 (0.3)
67.2
62.5
VDC
94.1 (0.2)
91.8
86.3
mscoco
Deep k-NN
76.6
70.3
(0.5
Confident
71.5 (0.5)
56.4
(6.5
CLIIP Sim.
93.8 (0.2)
93.0
84.5
LEMON ni (Ours)
(0.1)
91.3
LEMoN (Ours)
95.6 (0.2)
94.6 (0.3)
87.0 (0.0)
CapFilt (Oracle)
99.3
99.1 (0.0)
95.4 (0.4)
LLaVA
58.4 (0.2)
46.4 (0.2)
(0.1)
Discrepancy
57.8 (0.4)
46.1 (0.9)
57.4 (0.2)
VDC
80.5 (0.3)
(0.1
(0.3
Datamap
50.5)
57.2 44.1
mdb
deep k-NN
61.2 (0.4)
58.3 (7.2
Confident
52.8
(0.6
(1.8
CLIP Sim.
85.1
77.8
72.7
LEMON Fi (Ours)
(0.3)
77.7 (0.8)
LEMON opr (Ours)
86.0
(0.6)
(0.3)
CapFilt
82.7 (0.7)
73.3
(0.3)
LLaVA
(0.5)
42.7 (0.7)
(0.1)
Datamap
51.0
40.5
(0.0)
Discrepancy
60.0 (0.7)
(0.5)
(0.1)
VDC
50.8 (0.4)
40.3 (0.2)
(0.0)
mimicexr
deep k-NN
62.9 (0.4)
48.0
(0.1)
Confident
61.8 (0.3)
47.0
58.1 (0.6)
CLIP Sim.
64.1 (0.4)
51.7
59.2
LEMON FI (Ours)
66.3 (0.4)
54.6
LEMoN oPr (Ours)
70.6
60.4
61.1
CapFilt
49.7 (0.3)
40.0 (0.2)
57.0
30

---

LEMON: Label Error Detection using Multimodal Neighbors
Table 14:: flickr30x: Label Error Detection
Noise Type
AUROC
AUPRC
mean
mean
mean
LLaVA
79.3
0.8
58.5
0.2
65.0
Datamap
63.3
66.6
Discrepancy
73.0
0.5
59.2
1.8
59.0
0.3
VDC
81.1
Deep KNN
71.1
0.4
52.0
1.0
59.2
0.8
Coulident
63.1
0.9
42.1
54.0
CLIP Sim.
94.8
0.5
92.8
84.2
L.MoN
93.6
0.2
92.0
0.2
83.4
0.6
LEMoNort
83.6
flickr30k
CapFin
Datamap
90.5
0.8
83.9
78.7
Discrepancy
83.8
1.2
75.3
72.2
VDC
98.2
0.2
96.4
92_1
Deep KNN
81.1
1.6
65.3
1.7
72.7
1.2
random
Confident
CLIP Sim.
99.5
0.1
99.3
0.1
95.7
0.5
L.MoNNIx
96.3
LEMoNort
99.5
0.2
99.4
0.3
96.3
1.0
CapFilt
99.9
0.0
99.8
97.9
0.2
Table I.:: msccoco: Label Error Detection
Dataset
Noise Type
Method
AUROC
AUPRC
F1
std
LLaVA
74.9
Datamap
85.8
0.4
80.0
0.5
74.3
0.3
Discrepancy
VDC
94.1
0.2
91.8
0.2
86.3
0.4
Deep KNN
76.6
70.3
67.5
cal
Confident
71.5
0.5
56.4
66.5
0.5
CLIP Sim.
93.8
0.2
93.0
84.5
0.4
LEMoNnx
LEMoNort
95.6
0.2
94.6
0.3
87.0
0.2
CapFitt
99.3
0.0
99.1
0.0
95.4
0.4
LLaVA
79.4
0.2
61.3
0.3
72.6
0.2
Datamap
82.3
73.8
69.4
Discrepancy
72.4
64.0
60.0
VDC
91.9
0.5
88.3
0.7
82.7
0.5
m500co
Deep KNN
65.8
noun
Confident
69.8
1.1
52.8
1,5
63.3
1.3
CLIP Sim.
92.1
0.2
90.5
80.8
LEMoN...
90.4
0.5
89.5
80.2
0.5
LEMoNort
92.9
91.5
82.3
CapFilt
98.7
0.2
98.4
0.2
93.6
0.5
LLaVA
82.6
0.3
65.1
0.6
76.7
0.2
Datamap
Discrepancy
90.8
0.4
84.2
0.6
80.1
0.9
VDC
Deep KNN
94.5
0.4
88.4
87.9
0.7
Confident
85.0
0.8
71.4
81.8
0.9
CLIP Sim.
99.4
97.1
LEMONas
99.5
0.2
99.4
0.1
97.3
0.2
LEMoN
CapFilt
31

---

LEMON: Label Error Detection using Multimodal Neighbors
Table 16: mmimdlo: Label Error Detection
Type
AUROC
AUPRC
mean
mean
mean
LLaVA
58.4
0.2
46.4
0.2
58.5
0.1
Datamap
55.4
0.5
44.1
57.2
Discrepancy
57.8
0.4
46.1
0.9
57.4
0.2
67.1
deep k-nn
61.2
0.4
47.2
0.5
58.3
0.4
Confident
41.4
51.8
CLIP Sim.
85.1
0.3
77.8
72.7
L.
84.3
0.3
77.7
69.9
0.8
LEMoNort
79.4
73.5
CapFin
71.3
55.2
Datamap
53.9
0.7
38.9
53.9
0.2
Discrepancy
58.8
1.0
43.3
$4.6
amimdb
VDC
79.0
0.2
62.3
65.9
0.4
deep k-nn
61.4
0.1
44.2
0.3
55.8
0.3
noum
Confident
CLIP Sim.
82.8
0.4
72.8
0.5
68.4
1.1
LEMON
LEMoNort
84.2
0.5
75.9
0.5
69.5
0.4
CapFilt
79.9
0.1
66.2
67.1
0.3
LLaVA
58.5
46.7
58.5
Datamap
55.1
0.5
43.7
0.6
$7.0
0.3
Discrepancy
VDC
82.7
0.2
69.6
0.4
72.1
0.4
deep k-nn
49.0
random
Confident
52.0
1.0
41.0
0.5
52.1
3.1
CLIP Sim.
S8.1
82.0
75.7
LEMoN
73.8
LEMoNort
89.3
0.9
84,2
1.3
77.0
1.2
CapFilt
84.9
0.4
76.4
0.7
73.1
0.3
Table 1.7: mimicexr: Label Error Detection
cexr:
Dataset
Noise Type
Method
AUROC
AUPRC
F1
mean
sid
mean
std
menn
std
LLaVA
53.9
0.5
42.7
0.7
$7.0
0.1
Datamap
51.0
0.5
40.5
0.7
57.0
Discrepancy
60.0
0.7
50.2
0.5
57.2
VDC
50.8
0.4
40.3
0.2
$7.0
0.1
k-NN
62.9
0.3
Confident
61.8
0.3
47.0
0.2
58.1
0.6
CLIP
64.1
0.4
0.5
0.0
LEMoNay
66.3
0.4
$4.6
0.5
55.5
0.3
LEMON
70.4
1.6
60.4
1.6
61.1
0.8
mimicoxe
CapFilt
49.7
0.3
40.0
0.2
$7.0
0.0
LLaVA
50.8
0.4
40.6
0.2
$7.1
0.0
Datamap
Discrepancy
62.5
0.5
$2.2
1.0
$7.1
0.5
VDC
0.9
random
k-NN
66.6
0.6
53.8
1.1
59.4
0.2
Confident
65.0
1.0
49.5
0.7
61.6
1.1
CLIP Sim.
66.8
0.8
$4.4
0.4
LEMON mx
69.5
0.7
57.8
1.0
$7.7
1.2
LEMON
CapFilt
32

---

LEMON: Label Error Detection using Multimodal Neighbors
Table 1.: We show the AUROC and AUPRC LEMoN when we search for the optimal hyperparameters using a labeled validation set (LEMONorT) and when we use fixed hyperparameters (LEMON ex: = 30, cosine distance, = 71,n = 7i,m 0.1, and 72.n 72.m = 5). The mean gap in AUROC is -1.6 (1.3), and the mean gap in AUPRC -1.6 (2.4).
Note that F1 is not computed as it requires external information select threshold.
AUROC
AUPRC
Dataset
Noise Type
LEMoNorr
Gap
LEMoNor
LEMoN
Gap
asymmetric
98.8 (0.2)
97.5 (0.2)
(0.1)
97.8 (0.5)
94.8 (0.6)
(0.1)
cifar10
real
98.1 (0.0)
97.7 (0.2)
-0.5 (0.2)
(0.4
96.8 (0.3)
-0.5 (0.2)
symmetric
99.6 (0.1)
99.5 (0.1)
-0.2 (0.1)
99.4
99.2 (0.1)
-0.2 (0.1)
asymmetric
96.7 (0.3)
94.9 (0.3)
(0.1)
95.3 (0.4)
92.1 (0.4)
3.2 (0.1)
cifar100
real
90.8 (0.0)
88.9 (0.7)
(0.7
(0.3)
(1.1)
-2.8 (0.9)
symmetric
99.0 (0.0)
98.4 (0.1)
-0.7 (0.1)
98.7 (0.1)
97.7 (0.2)
(..1
mini
human
90.0 (0.4)
89.5 (0.2)
-0.5 (0.2)
79.7 (3.0)
81.5 (0.3)
+1.8 (2.7)
StanfordCars human
73.1
72.6 (0.7)
-0.5 (0.5)
40.5 (0.4)
44.9
+4.4 (1.0
noun
94.5 (0.2)
93.6 (0.2)
-0.9 (0.3)
92.8 (0.3)
(2.0)
-088 (0.1)
flickr30k random
99.5 (0.2)
(0.2)
-0.0 (0.1)
99.4)
99.3 (0.2)
-011 (0.2)
cat
70.4 (1.6)
66.3 (0.4)
-4.1 (1.5)
61.6
54.6 (0.5)
(1.5)
mimicexr random
73.7 (1.7)
69.5 (0.7)
-4.1 (1.5)
64.1 (2.2)
57.8 (1.0)
66.3 (1.7)
cat
86.0 (0.1)
84.3 (0.3)
(0.3
(0.6)
77.7 (0.8)
(1.2
mmincb noun
(0.5)
$2.1 (0.4)
-2.1
(0.5)
72.7 (0.6)
(..5
random
89.3 (0.9)
87.6 (0.1)
(0.8)
84.3
(0.9
-2.2
cat
95.6 (0.2)
92.0
(..1
(4.3
91.8 (0.3)
2.8
mscoco
noun
92.9 (0.5)
90.4
-255
(0.5)
(0.4)
(2.3
random
99.6 (0.1)
99.5 (0.2)
-0.1 (0.0)
99.5
99.4
-0.1 (0.0)
Table 1.9: Downstream captioning performance when removing 40% samples with highest mislabel scores, and models are trained without early stopping for 10 epochs. We find that filtering noisy data with LEMONoeT improves captioning.
Dataset
Method
B64
CIDER
ROUGE
No Filtering
31.7 (0.2)
(0.6)
(0.1)
CLIP Sim.
(0.2)
80.6 (0.)
f1 tckr30k
LEMoNar:
32.5
79.2 (0.6)
52.7 (02)
Clean
(3.4)
(1.8(
53.1 (0.2)
Filtering
(0.3)
(0.6)
(0.2)
CLIP Sim.
39.6 (0.1)
130.8 (0.2)
(0.0)
mscoco
LEMoN
(0.6)
131.0 (0.1)
(9.4
(9.6
(0.3)
(9.1
Table I.10: Performance of our method after ablating various components. We find that mislabel detection performance almost decreases monotonically as we remove additional components, with the exception of two metries on mmimde mimm where one ablation is statistically comparable the original method.
mmimdb
mscoCO
AUROC
AUPRC
F1
AUROC
AUPRC
F1
LEMON (Ours)
86.0 (0.1)
79.4
73.5 (0.3)
95.6 (0.2)
94.6 (0.3)
87.0 (0.2)
T1
35.3
(1.1
72.9 (0.5)
94.6
93.8 (0.4)
85.2 (0.5)
T2
(0.1)
78.6 (0.5)
73.3 (0.3)
94.7
93.8
85.4 (0.8)
-T1, T2
85.4 (0.2)
(0.7
73.0 (0.7)
94.7 (0.3)
93.8 (0.5)
$5.3
-8n
86.1 (0.2)
79.6
73.7 (0.2)
94.6 (0.3)
93.6 (0.5)
84.7.0.7)
-8m
85.3 (0.2)
77.9
73.1 (0.4)
94.9 (0.2)
94.0 (0.4)
86.5 (0.6)
Sn, Sm
(CLIP Sim.)
85.1 (0.3)
77.8 (0.7)
72.7 (0.6)
93.8 (0.2)
93.0 (0.4)
84.5 (0.4)
33

---

LEMON: Label Error Detection using Multimodal Neighbors
Table 111: AUROC of label error detection for each component of our score. We find that dmm the most critical
component of the score. Of the two nearest neighbor terms, we find that 8n (nearest image neighbors) is the more important term for most datasets.
cifar10
cifar100
ImageNeL
sLanIordCars
Flick:30k
macoco
mimi
mmn (CLIP Sim.)
92.2 (0.2)
80.8 (0.1)
89.3 (0.2)
69.8 (0.4)
94.8 (0.5)
93.8 (0.2)
85.1 (0.3)
64.1 (0.4)
Sm
80.2 (1.1)
65.4 (2.0)
80.8 (0.3)
66.1 (0.6)
(3.0
75.9 (0.2)
60.4 (0.7)
59.7 (0.5)
Su
98.1 (0.0)
88.4(0.1) (0.1)
84.3 (0.2)
73.0 (0.6)
69.9 (2.3)
76.3 (0.8)
53.7 (0.2)
57.7 (0.7)
dmmm 5m
92.5 (0.5)
81.3 (1.1)
89.6 (0.2)
70.0 (0.6)
95.0 (0.5)
94.6 (0.3)
86.0 (0.2)
64.5 (0.6)
Su + 8p
98.0 (0.2)
88.8 (0.2)
84.5 (0.4)
73.0 (0.7)
83.2 (0.5)
86.1 (0.6)
61.0
64.7 (1.0)
dymn 8n
98.2 (0.1)
90.7 (0.2)
90.0 (0.4)
73.1 (0.5)
94.9 (0.3)
94.9 (0.2)
85.3 (0.2)
64.4 (2.0)
8t. + 8p, (LEMoN)
98.1 (0.0)
90.8 (0.0)
90.0 (0.4)
73.1 (0.5)
(0.2(
95.6 (0.2)
86.0 (0.0)
(1.6)
Table 1.12: Average (~) per-sample runtime (milliseconds) of each method for label error detection. Standard deviation across 3 random data seeds are shown in parentheses. Experiments were run using job scheduling with GTX A6000 or A100 GPUs.
ci cifar10
cirr100
mini ImageMet
stanfordCars
mscoco
flickr30k
mimicexr
runimcbb
LEMoN
10.1 (0.5)
(..)
(1.6)
(1.0)
18.8
31.9)
52.2 (2.7)
21.1 (1.4)
CLIP Sim.
(0.0)
2.9 (0.0)
6.5
6.8 (0.1)
(0.1
9.8 (0.0)
17.1 (0.0)
24.7 (1.0)
Deep KNN
(1.4)
6.3 (1.8)
(1.8)
5.4 (0.4)
(1.4)
337.6 (3.3)
29.6
Datamap
19.0 (0.7)
19.3 (0.5)
39.2 (1.4)
41.2
30.2)
34.3
47.1
52.5
CapiFilt
13.5 (0.1)
14.8
29.6 (0.0)
31.6
VDC
4460 (880)
5160 (503)
7932 (3.4)
4672 (357)
34

---

LEMON: Label Error Detection using Multimodal Neighbors
1.7. Varying Validation Set Size
In Figure I.3, we examine the effect of varying validation set size (by random subsampling) LEMON OPT
0.9
0.94
AUROC
AUROC
0.8
LEMoNopt opt
0.92
LEMoNopt
LEMoN fix
LEMoN
0.7
CLIP Sim.
0.90
CLIP Sim.
101
102
103
10a
102 |Val Setl
10
|Val Setl
(a) mscoco, noise
(b) flickr30k, noise
0.9
0.70
AUROC
0.8
AUROC
0.65
LEMoN opt
LEMON LEMoN CLIP Sim.
0.7
0.60
LEMON fix
CLIP Sim.
0.55
0.6
101
102
103
101
102
103
|Vll Set]
|Val Set|
(c) mmimdb, noun noise
(d) mimim coxr, cat noise
Figure I.3: Test-set AUROC of mislabel detection with varying size of the labeled validation set for LEMONorr. Note that LEMoNnx and CLIP Sim. do not have any hyperparameters and such do not rely a labeled validation set.
L8. Empirical Comparison with Thomas & Kovashka (2022)
In Table I.13, compare the performance of LEMoNort against the four individual scores and two combined scores proposed in Thomas & Kovashka (2022), using the datasets and noise types shown in Table 1. For the Comb- Val strategy, as there are four terms, we sweep over weights in {1, 2, 3, 4, 5,44, selecting the best combination using labeled validation set, identically to LEMON. For the Comb-Stat strategy, we use the mean and standard deviations, as in Equation (8) in Thomas & Kovashka (2022). We find that none the combined scores significantly outperform r This is because in both combination strategies, non-zero weight placed on the other terms, which essentially adds noise the final score without contributing any signal.
I.9. Real-World Web Scale Corpus (CC3M)
conduct an experiment of LEMONBx on CC3M (Changpinyo et al., 2021), a large web-scraped dataset of images and annotations, where we demonstrate the utility of LEMoN filtered data on CLIP pretraining. We download CC3M, which contains 2.9 million valid URLs image-caption pairs. We then pretrain CLIP model (ViT-B/16) from seratch this dataset for 20 epochs, with batch size of 128, and using cyclic learning rate scheduler with learning rate of 10-4
We then use this CLIP model as the basis to compute distances for LEMoN using the reasonable hyperparameters from the main paper: 1 = 30, cosine distance, Ti,x = Ti,m 0.1, and T2,x = 72, = 5. We then select the I million samples with the lowest mislabel scores, filtering out the 1.9 million samples most suspected to mislabels. We pre-train another CLIP model from scratch on this subset using the same architecture and setup as above. We evaluate the resulting model on
35

---

LEMON: Label Error Detection using Multimodal Neighbors
Table 1.13: Comparison of label error detection performance of LEMoN versus baselines from Thomas & Kovashka (2022).
Dataset
Metric
X
Tpis TX
TDiv
Comb- Val
Comb-Stat
LEMoNorr
AUROC
77.1 (1.9)
48.2 (1.2)
50.3 (3.5)
(1.9)
59.6 (19.5)
59.6 (19.5)
98.1 (0.0)
cifar10
AUPRC
70.4 (2.7)
41.2 (1.1)
41.6
(211)
51.6 (20.5)
51.6 (20.5)
97.4 (0.1)
66.4 (2.2)
58.4 (0.8)
58.4
(8.8
62.9 (8.0)
62.9 (8.0)
92.0 (0.2)
AUROC
49.7 (1.5)
(1.1
(1.9
51.5 (6.2)
51.8 (7.0)
90.8 (0.0)
cifar100
AUPRC
57.4 (2.3)
41.3
42.6
40,9 (1.1)
41.9 (5.6)
42.3 (6.3)
87.4 (0.3)
Fl
(0.8)
57.2
(0.)
57.1
57.8 (0.7)
57.8 (0.7)
78.4 (0.0)
AUROC
79.5 (0.3)
(0.5
48.3 (0.2)
(5.4
76.6 (0.3)
90.0 (0.4)
miai L
AUPRC
(0.4)
(2.5)
46.3
(0.2)
59.8 (0.2)
61.7 (0.3)
79.7 (3.1)
F1
(0.1)
50.9
(3.7
(0.1)
60.5 (0.4)
61.2 (0.6)
76.9 (0.2)
AUROC
65.7
50.8 (1.1)
51.9
50.1 (0.5)
62.0 (0.1)
64.1 (0.1)
73.1 (0.5)
stanfordCars
AUPRC
(0.7 (0.6)
(0.3)
20.2)
30.1 (0.2)
31.9 (0.4)
40.5 (0.4)
F1
(0.7)
38.0
38.0
38.2
41.4
43.3 (0.2)
51.3 (0.5)
AUROC
(0.6)
53.3 (1.4)
(2.9)
52.9 (2.2)
63.9
70.5 (0.2)
94.5 (0.2)
flickr30k
AUPRC
(1.8)
37.1 (1.8)
32.8 (1.9)
(0.8
(2.2)
53.7 (2.7)
92.8 (0.3)
F1
(0.3)
50.9 (0.3)
50.8 (0.4)
50.8 (0.3)
(3.7
62.6 (3.3)
83.6 (1.4)
AUROC
60.0 (0.7)
49.6 (0.4)
49.7
(1.3)
(3.6
52.5 (4.7)
70.4 (1.6)
mimiccxr
AUPRC
50.2 (0.5)
39.3 (0.5)
(0.1)
39.4
40.5 (2.7)
42.2 (4.8)
60.4 (1.6)
(0.1)
(7.0)
(7.0
(0.1)
57.2
57.3 (0.1)
61.1 (0.8)
AUROC
57.8 (0.4)
50.1 (0.4)
(0.4)
50.4
53.4)
54.7 (2.6)
86.0 (0.1)
mmi mdb
AUPRC
46.1 (0.9)
40.2 (0.6)
38.9 (0.5)
40.2
(3.0)
42.0 (3.6)
79.4 (0.6)
F1
57.4 (0.2)
57.1 (0.0)
57.1
(0.0)
56.5 (1.8)
56.6 (2.0)
73.5 (0.3)
AUROC
72.7 (0.3)
48.5
48.8 (2.9
49.7
50.1 (0.7)
95.6 (0.2)
mScoco
AUPRC
67.2 (0.4)
39.1 (0.5)
42.3 (1.0)
(9.4)
38.9 (0.3)
39.5 (0.2)
94.6 (0.3)
FI
62.5 (0.3)
57.0
57.1 (0.0)
57.0
57.3 (0.1)
57.3 (0.1)
87.0 (0.2)
36

---

LEMON: Label Error Detection using Multimodal Neighbors
Table I.14: Performance of each method on the Datacomp (Gadre et al., 2024) small benchmark from the filtering track. As of 2024/11/14, only 9.96M images ("Data Available") out of 12.8M are accessible. We compare the performance of LEMONNFP versus the CLIIP score baseline after filtering to 3.5M images.
Method
ImageNet
ImageNet Dist. Shifts
VTAB
Retrieval
Avg (38 datasets)
Data Available (9.96M Samples)
LEMoNmx
0.045
0.053
0.188
0.116
0.168
CLIP score
0.043
0.049
0.177
0.119
0.160
No filtering
0.025
0.033
0.145
0.114
0.132
Basic filtering
0.038
0.043
0.150
0.118
0.142
Text-based
0.046
0.052
0.169
0.125
0.157
From Gadre et al. (2024) (12.8M Samples)
Image-based
0.043
0.047
0.178
0.121
0.159
LAION-2B filtering
0.031
0.040
0.136
0.092
0.133
CLIP score
0,051
0.055
0.190
0.119
0.173
Image-hased score
score
0..099
0.045
0.162
0.094
0.144
zero-shot classification using the VTAB benchmark (Zhai al., 2019), and compare it with CLIP models trained using data filtered to million examples using the CLIP similarity baseline, and the original unfiltered model.
In Table 1.1,, we find that LEMONnx marginally outperforms the CLIP similarity baseline on average zero-shot accuracy, though both underperform pretraining on the full corpus. Similar results can be found for few-shot linear probing (Table 1.16) and full finetuning (Table I.17). One likely explanation of this is that although large proportion of images in the CC3M dataset are technically "mislabelled" that the caption is not precisely correct description of the image, some substrings of these noisy captions may, on aggregate, contain useful word associations which the model learns, and thus may be useful to downstream tasks.
We examine images of images selected to be mislabels by our method in Figure 1.4. We find that our method identifies images that are completely mislabeled one cause of which is images changing after they have been indexed. In addition, our method also identifies samples which are ambiguous or imprecise.
I.10. Real-World Web Scale Corpus (Datacomp)
We conduct an experiment of LEMoNAx on Datacomp (Gadre et al., 2024). We use the small dataset from the filtering track, which originally consisted of 12.8M images. As these images are accessed directly from the web, only 9.96M images were able to be downloaded as 1024/11//4, We apply LEMON ex to this dataset using OpenAl CLIP ViT-L/14 embeddings provided by Datacomp. We select the 3.5M images with lowest mislabel scores, and use the default hyperparameters from Datacomp train CLIP model, and evaluate it on the same 38 zero-shot classification datasets. We compare with filtering using only the CLIP score (equivalent to CLIP Sim.) to the same number of images. In Table I.14, find that given the available images, LEMONax outperforms the baseline on average, and on three of four individual evaluations. However, neither method outperforms the scores reported in the original paper due to their dataset being larger.
1.1.. Hyperparameters Used for Real-World
We show the hyperparameters used for the real-world experiment in Table 1.18. We use k 30, cosine distance, and these hyperparameters, which originate from a hyperparameter search on synthetically noised data. We note that fl r fickk300 has some negative hyperparameters, which we attribute to overfitting to relatively small validation set during hyperparameter selection.
I.12. Examples of Detected Real Label Errors
We show additional examples of label errors Figure 1.5.
37

---

food101 LEMON: Label Error Detection using Multimodal Neighbors
Table 1.15: Zero-shot accuracy (%) of various CLIP models on the VTAB benchmark (Zhai al., 2019). CLIP models
(ViT-B/16) are pretrained from scratch subset of CC3M (Changpinyo al., 2021) which has been filtered to million samples using LEMONBx and the CLIP similarity bascline, using version of CLIP pretrained the entire dataset.
CLIP Sim.
LEMoN
Unfiltered
caltech101
28.25
28.99
51.43
cifar100
11.02
6.79
18.65
clevr closest object distance
18. 11
22.58
25.76
clevr.count.all
12.98
12.65
12.05
dmlab
14.78
16.22
16.62
poorite__abel_orrinnnee
2.44
1.34
1.98
dsprites label position
3.06
3.20
3.13
Aoprites_abell. Iabelly-positioon
3.11
2.89
3.20
dtd
6.60
3.94
12.34
eurosat
14.37
22.07
9.93
flowers
6.11
5.19
6.83
food101
4.94
5.31
9.02
pets
7.63
4.69
8.23
sun397
13.89
14.22
24.02
svhn
7.80
12.35
8.00
Average
10.34
10.83
14.08
Table 1.16: Few-shot linear-probing accuracy (%) using 5 samples per class of various CLIP models the VTAB benchmark.
CLIP Sim.
LEMoN FIx
Unfiltered
caltech101
53.50
53.45
60.82
cifar100
18.66
17.11
23.59
clevr.closest.objcct.disttt
23.39
25.48
25.97
flevr_count all
19.19
20.43
23.22
dmlab
18.43
21.01
19.72
dsprites label orientation
9.11
8.40
9.91
Esprites_label. dabel_x_pon __position
3.78
4.52
4.20
dsprites label yy position dtd
6.93
7.90
8.20
28.88
29.84
38.78
eurosat
66.17
65.20
64.76
flowers
57.31
54.53
62.35
11.94
13.50
17.89
pets
18.92
17.12
24.42
sun397 svhn
21.43
21.60
32.53
12.05
12.60
12.99
Average
24.65
24.85
28.62
38

---

LEMON: Label Error Detection using Multimodal Neighbors
Table 1.17: Full finetuning linear-probing accuracy (%) using 5 samples per class of various CLIP models the VTAB benchmark.
CLIP Sim.
LEMoN FI
Unfiltered
caltech101
65.91
65.94
74.80
c0far100
49.74
48.53
56.54
clevr closest object distance
52.35
55.83
53.74
clevr_count.all
55.03
54.05
57.99
dmlab
39.10
39.31
43.00
dsprites_label laapites_labee_oriennntee
38.58
38.65
42.75
dsprites label position
27.08
36.61
32.78
dsprites_label yacobelllyppsiiiee position
46.31
47.63
49.56
dtd
44.20
42.07
51.91
eurosat
92.70
92.09
93.00
flowers
65.72
63.65
72.45
food101
47.75
48.62
55.23
pets
40.86
40.69
51.57
sun397
49.58
49.45
57.84
svhn
38.08
39.25
42.74
Average
50.20
50.83
55.73
fresh milk in the glass on colour background, illustration
a very young baby girl playing with toys in white studio
portrait stock photo
homeyhomey.dul
homes for sale and luxury real estate including horse farms and property in areas
park covered in yellow leaves and lined with tall trees turning bright yellow during an autumn day
tangled tree roots forest trail
face of people stock
begin your exercise with jump rope easy and funny
evil looking person sitting atop a hay bale royalty free
photo
Figure 1.4: Sample images and captions from CC3M which have been identified as mislabeled by LEMoN
39

---

LEMON: Label Error Detection using Multimodal Neighbors
Table 1.18: Hyperparameters used for the real-world experiment. We use A: 30, cosine distance, and the hyperparameters below, which originate from hyperparameter search synthetically noised data.
71,m
72,m
71,m2
72.m
cifar10
20
10
5
cifar100
15
0
0
5
0
mscoco
5.324
11.057
5.143
10.498
7.233
15.637
mmimdb
15
5
10
10
flickr30k
0.092
-0.177
-0.274
-0.074
-0.072
0.000
mimiccxr
10
10
CIFAR10
CIFAR100
MSCOCO
Flickr30k
This is plane from the front view
boy in red shirt
Automobile
Leopard
playing ball.
The is sitting in the Are you coming wi th dirt near metal fence.
Airplane
Bus
for a cup of coffee?
Horse
Lobster
A WOMAND IN ALL BLACK
À young girl celebrating her tean after winning world series in the finals held texas.
BEHIND TO WHITE DOGS
house in small constraining carriage.
The policeman driving down the street.
Deer
Camel
Figure L.5: Example images in each dataset identified by our method to be mislabels, and labeled as errors by a human annotator.
40

---

LEMON: Label Error Detection using Multimodal Neighbors
I.13. Comparison with Northcutt al., 2021 (Northeutt al., 2021b)
In Northcutt et al., 2021 (Northcutt et al., 2021b), the authors utilized confident learning (Northeutt al., 2021a) identify suspected errors in the test sets of ci far10 and ci cifarl00 They then obtained human labels for each suspected error using Amazon Mechanical Turk, and confirmed the image be mislabel if at least 3 of 5 workers stated This amounts to 54 confirmed mislabels in ei farl0 (out of 221 suspected), and 585 confirmed mislabels in cifarl00 00 (out of 1650 suspected). In this section, we compare the performance of LEMON RI versus the CLIP similarity baseline on this set. As this set a subset the images identified to be mislabels by confident learning, we are not able to compare our model performance with confident learning itself. In addition, this presents a pessimistic view (lower bound) of the performance of our method, as there are many images identified by LEMON which are mislabeled, but were not selected by confident learning in (Northcutt al., 2021b). We demonstrate examples of these images in Figure I.6.
In Table 119,, we compare the performance of LEMON with the CLIP similarity baseline on the error set from Northcutt et al., 2021 (Northcutt et al., 2021b). First, we compute the mean ranking of all enror set samples as ranked by each method, out of 10,000 test-set samples. We find that our method ranks error set samples higher on average than the baseline, though the variance is large. Next, we subset to the top -CL Set- ranked samples for each method, and compute the percentage of which are actually the error set. We note that this precision metric is upper bounded by the precision of the reference method (confident learning). Again, we find that LEMoNnx outperforms the baseline, and is able identify more actual label errors than CLIP similarity this threshold.
idx 3309
idx = 4175
idx 7524
label deer
label cat
label cat
CIFAR10
idx 5031 label = camel
idx = 5681 label seal
idx 9269 label pear
CIFAR100
Figure 1.6: Demonstrative examples of mislabeled samples in cifar10 and cifar100 which have been identified by our method in the top CCL Set-, but was not identified by confident learning in Northcutt al., 2021 (Northcutt et al., 2021b) and thus was not part of their error set.
I.14. Downstream Classification with Label Error Detection-based Filtering
Here, show the impact of filtering out different proportions of the training data based label error predictions, and obtaining test performance.
1. AVERAGE ACCURACY
We observe that the gap in performance is low between LEMoNorT and the best method (less than 1% on stanfordCars) terms of downstream accuracy (see Figure I.7). LEMON is the best method on ImageNet.

---

LEMON: Label Error Detection using Multimodal Neighbors
Table 1.19: Comparison of LEMON (Ours) with the CLIP similarity baseline on the human labeled error set from Northcutt et al., 2021 (Northcutt et al., 2021b). In this prior work, the authors used confident learning to identify |CL Set candidate label errors in e farrl0 0 and c far100, |Error Set of which are confirmed be mislabels by Mechanical Turkers. Mean Ranking denotes the average ranking of all error set samples as ranked by each method. Precision @ Top [CL Set involves taking the top |CL Set] samples ranked by each method, and computing the percentage of which are in the error set. Note that each dataset's 's test set consists of 10,000 samples. Numbers in parentheses represent one standard deviation.
Mean Ranking
Precision @ Top JCL Set
Dataset
CL Set]
Error Set]
LEMoN nx
CLIP Sim.
Oracle
LEMON
CLIP Sim.
ci far10
275
54
1269.7 (1905.1)
2681.0 (2507.1)
19.64%
6.55% 14.41%
1.45%
cifar100
2235
585
2357.5 (1981.5)
3642.1 (2719.5)
26.17%
10.16%
stanfordCars
minilmageNet
0.7
0.9
Accuracy
Accuracy
0.8
0.6
LEMoN (Ours)
0.7
Deep K-NN
0.5
Clip Logits Discrepancy
0.6
40
60
80
100
40
60
80
100
%Data retained
%Data retained
Figure 1.7:
Downstream accuracy on stanfordCars and mini ImageNet.
L15. Area Under Test Error vs % Data Retained Curve
We compute the area under the test error (i.e., 1-accuracy) vs % data retained curve in Table 1.20. Note that the minimum data retained is 30% (i.e., the minimum amount of data required for training the downstream model).
On ei farl0 and mini ImageNet, we observe that LEMoN performs the best in terms of AUC (i.e., lowest test error). On stan forrCa rs, Deep k-NN performs better. However, the gap in performance is low between LEMoN and the best method all datasets.
Table 1.20: Area under the curve: test error vs % data retained for all four classification datasets. Lower better, and bold denotes best method.
Method
eifar16
stanfordCars
Tmsgeeett
CLIP Sim.
4.58
16.89
24.52
8.84
CIJIP Logits
4.54
16.90
25.72
8.27
Discrepancy
5.98
20.22
26.57
13.07
Deep NN
Ours
4.23
17.02
24.48
7.91
L.16. Out-of-Domain Robustness
We report the test performance on an Out-of-Domain (OOD) dataset CIFAR-I0C (Hendrycks Dietterich, 2018), when models are trained and validated the cifar10 noisy train set. The CIFAR-10C dataset contains 19 corruptions applied to the ci cirarl0 test set, with varying severity of corruption. Then, robustness is measured the average test top-1 class accuracy performance on the 10CAR-10C dataset (across all corruption types and severities), following prior work (Diffenderfer et al., 2021). We observe that our method outperforms the CLIP similarity baseline on robustness, when percentage data retained is less than 60%.
42

---

LEMON: Label Error Detection using Multimodal Neighbors
0.86
Clip Sim.
LEMoN (Ours)
0.84
Robustness
0.82
0.80
0.78
40
60
80
100
% %aata Retained
Figure 1.8: Downstream accuracy CIFAR-1OC, averaged across all corruption types.
43


Reviews:
[{"review_id": "KOpcBNOdGS", "reviewer": "Reviewer_qofm", "summary": "This paper presents LEMoN, a method for detecting label errors in image-text pair datasets. The authors define a scoring function in the CLIP embedding space that combines the pairwise image-text distance with distances to nearest neighbors in both the image and text modalities. Specifically, the score integrates multimodal distances and distance scores with neighbors in each\nmodality (text, image). The proposed method is theoretically justified and empirically validated across eight classification and captioning datasets (including the healthcare dataset), consistently outperforming existing baselines. The authors conduct comprehensive ablation studies, with a strong focus on demonstrating the method’s robustness to variations in hyperparameters.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the insightful review and constructive feedback!\n\n> For instance, BLIP-based filtering methods may not be prohibitively expensive, as they can be fine-tuned on a domain-specific dataset (e.g., movie or biomedical domains) with relatively modest computational costs. Such fine-tuning could potentially outperform the proposed method—as suggested by results on the MS-COCO dataset. A direct comparison of computational overhead between LEMoN and these BLIP-based approaches would strengthen the paper’s claims. Furthermore, CapFilt appears to perform reasonably well even in a zero-shot setting (i.e., without fine-tuning on MS-COCO), wondering about this simple zero-shot CapFilt result. Clarifying the necessity of LEMoN in contrast to these simpler alternatives would help reinforce its practical relevance.\n\n*CapFilt Inference Runtime*: We compare the inference time (per sample runtime, milliseconds) of LEMoN with CapFilt using the same setup as Table I.11. We find that the two methods have generally comparable inference runtimes.\n|           | mscoco      | flickr30k   | mimiccxr     | mmimdb      |\n| :-------- | ----------: | ----------: | -----------: | -----------: | \n| LEMoN     | 18\\.8 (1.8) | 35\\.9 (1.2) | 52\\.2 (2.7)  | 21\\.1 (1.4) |\n| CapFilt | 21\\.4 (9.9)  | 28.7 (23.8)  | 31\\.6 (0.2)   | 33\\.8 (3.0)   |\n\n*Advantages of LEMoN over CapFilt*: We clarify that for the results reported in the paper, we utilized the pretrained “Salesforce/blip-itm-base-coco” checkpoint. This model was trained on the **clean** training split of MSCOCO, which is why we refer to it as the \"oracle\". This training includes minimizing image-text matching loss, which is a binary classification objective designed to predict whether a caption matches a given image. LEMoN does not ever need access to the clean dataset (except optionally for hyperparameter tuning), only the noisy data. As such, we clarify that CapFilt is not applied in a “zero shot” setting, especially not for MSCOCO. \n\nFurther, we note that CapFilt appears to be more domain specific than LEMoN. As CapFilt has been trained on clean MSCOCO, it does well on MSCOCO and Flickr30k (both contain COCO‐style captions), but does worse than LEMoN on mmimdb.\n\n\n> At first glance, LEMoN's strong performance on mimiccxr seems to highlight its generalizability and ease of adaptation to out-of-domain data—a potential advantage over BLIP-based methods. However, since the proposed method also relies on a domain-specific encoder (BiomedCLIP), the comparison may not be entirely fair. I believe the advantage over the BLIP variant should be more clearly explained —particularly through comparisons with zero-shot CapFilt and fine-tuned CapFilt (e.g., on domain-specific datasets), along with an analysis of their relative computational costs. In addition, exploring whether the proposed scoring function could be integrated into or combined with existing methods like CapFilt might potentially enhance its practical utility.\n\n\nWe clarify that all other baselines on MIMIC-CXR except CapFilt do also utilize BiomedCLIP (where applicable), and LEMoN outperforms all such baselines. We highlight that we have also explored label error detection without an external domain-specific encoder for MIMIC-CXR in Table 4. Finally, as the reviewer points out, one can leverage representations from BLIP for label error detection with LEMoN, and LEMoN's score could also be combined with other mislabel scores (e.g. through ensembling). We highlight this as one area of future work.\n\n> Finally, many recent vision-language pipelines rely on synthetically generated high-quality captions (e.g., post-BLIP processing). In such scenarios, the role and added benefit of LEMoN seems less obvious.\n\nWe highlight that such vision-language pipelines assume that high quality synthetic caption generators already exist. LEMoN is designed to improve these synthetic caption generators by providing them with better real training data to start with (e.g. Section 6.2). Additionally, LEMoN may be used to detect errors in synthetic captions as well, thus adding the potential of improving the filtering component of such vision-language pipelines.\n\n\n> believe the paper includes relevant citations overall; however, the main topic is also closely related to the issue of false negatives in vision-language models. The authors may consider to include references such as [Chun et al., ECCV 2022] and [Byun et al., MAFA, CVPR 2024]\n\nThank you for these references – we will add them in the revised version of the paper!", "claims_and_evidence": "1. I believe the proposed method to be well-motivated and theoretically sound, supported by Theorem 4.1 and Lemma 4.2. The empirical results are strong across several datasets, and the experiments are particularly thorough in evaluating the robustness to hyperparameter choices—a critical aspect given the number of hyperparameters involved.\n\n2.  While the proposed method demonstrates strong performance compared to existing baselines, its practical advantages remain somewhat unclear.\n- For instance, BLIP-based filtering methods may not be prohibitively expensive, as they can be fine-tuned on a domain-specific dataset (e.g., movie or biomedical domains) with relatively modest computational costs. Such fine-tuning could potentially outperform the proposed method—as suggested by results on the MS-COCO dataset. A direct comparison of computational overhead between LEMoN and these BLIP-based approaches would strengthen the paper’s claims.\nFurthermore, CapFilt appears to perform reasonably well even in a zero-shot setting (i.e., without fine-tuning on MS-COCO), wondering about this simple zero-shot CapFilt result. Clarifying the necessity of LEMoN in contrast to these simpler alternatives would help reinforce its practical relevance.\n- At first glance, LEMoN's strong performance on mimiccxr seems to highlight its generalizability and ease of adaptation to out-of-domain data—a potential advantage over BLIP-based methods. However, since the proposed method also relies on a domain-specific encoder (BiomedCLIP), the comparison may not be entirely fair. I believe the advantage over the BLIP variant should be more clearly explained —particularly through comparisons with zero-shot CapFilt and fine-tuned CapFilt (e.g., on domain-specific datasets), along with an analysis of their relative computational costs. In addition, exploring whether the proposed scoring function could be integrated into or combined with existing methods like CapFilt might potentially enhance its practical utility.\n- Finally, many recent vision-language pipelines rely on synthetically generated high-quality captions (e.g., post-BLIP processing). In such scenarios, the role and added benefit of LEMoN seems less obvious. \n\n3.  Limited impact on CC3M. I believe one of the most practical use cases for label error correction lies in large-scale web-crawled image-text datasets. However, the marginal performance gain of the proposed method over the baseline—especially when it underperforms the default (unfiltered) setting—raises concerns about its practical effectiveness.\n\nI believe that points 2 and 3 must be thoroughly addressed in the rebuttal.", "methods_and_evaluation_criteria": "I wrote them above", "theoretical_claims": "I checked them", "experimental_designs_or_analyses": "It seems that experimental dI wrote them above.", "relation_to_broader_scientific_literature": "I believe the", "essential_references_not_discussed": "believe the paper includes relevant citations overall; however, the main topic is also closely related to the issue of false negatives in vision-language models. The authors may consider to include references such as [Chun et al., ECCV 2022] and [Byun et al., MAFA, CVPR 2024]", "other_strengths_and_weaknesses": "Wrote them above", "other_comments_or_suggestions": "Wrote them above", "questions_for_authors": "Wrote them above", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ZsJUj8msK0", "reviewer": "Reviewer_XaUo", "summary": "The paper presents LEMoN, a method to detect label errors in paired image-text data by using a pretrained CLIP model. Given a dataset of image-text pairs, LEMoN constructs a score $f(x,y)$ which is a weighted combination of CLIP score of $(x,y)$ and two nearest neighbor based intra-modal scores. The intuition is that if $(x,y)$ is mislabeled, then i) captions corresponding to images similar to $x$ will be mismatched with $y$, and ii) the corresponding images of captions similar to $y$ are far away from $x$. The paper provides a theoretical justification for this scoring function. Experiments are performed on 4 classification datasets and 4 image captioning datasets (with artificial noise added) where the proposed scoring outperforms relevant baselines in detecting noisy samples. The paper also reports downstream classification and captioning performance following filtering. Experiments are also performed on real world datasets CC3M and Datacomp.\n\n\\### update after rebuttal: I have gone through the rebuttal and the comments of other reviewers, and have raised my score to 3. The rebuttal addressed a few of my concerns regards design choices and comp. with LNL algorithms, which I hope will be incorporated in the revision. However, echoing reviewer qofm, I have concerns over the downstream applicability of LEMoN (based on perf. on CC3M and Datacomp).", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the insightful review and constructive feedback!\n\n> Empirical performance on realistic noise. Downstream performance on CC3M and Datacomp is almost the same as Clip Similarity baseline. \n\nFirst, we would like to emphasize that we evaluate our method against the baselines on several other datasets with human noise, including CIFAR-10N, CIFAR-100N [1], and StanfordCars and MiniImageNet [2]. These datasets contain noise from human annotations collected on Amazon Mechanical Turk and the Google Cloud Data Labeling Service respectively. Second, we would like to note that, in addition to CC3M and Datacomp, we also observe increased downstream performance of training on LEMoN filtered datasets in CIFAR-10N and CIFAR-100N (Figure 3) and mscoco (Table 3).\n\n> Empirical performance on realistic downstream tasks is unsatisfactory. Perhaps fine-tuning under limited data (few-show training samples) is better suited to evaluate these methods?\n\nThank you for this suggestion! We have conducted additional experiments for these CC3M-pretrained models by linear probing on the VTAB benchmark, first in the few-shot setting where we select 5 random samples per class, and next where we finetune on the standard training split of each dataset. [Our results can be found here](https://postimg.cc/VdpjZjHS). Overall, we find similar trends as the zero-shot setting, where LEMoN marginally outperforms the baseline, with both underperforming the model that has been pretrained on the whole corpus.\n\n\n> Simpler ways to incorporate k-NN information. For ex. average CLIP score between the caption $y$ and neighboring images of $x$ (and vice versa). This avoids the $\\tau_2$ hyperparam, but ignores paired information of neighbors.\n\n> As described in weaknesses, a discussion on simpler ways to incorporate k-NN information would be informative.\n\nThank you for this suggestion! We have added this alternate way of integrating neighbor information as suggested by the reviewer:\n\n$s(x, y) = d_{mm}(x, y) + \\beta s_n (x, y, \\mathcal{D}) + \\gamma s_m(x, y, \\mathcal{D})$\n\nwith \n$s_n(x, y, \\mathcal{D}) = \\frac{1}{k} \\displaystyle\\sum_{j=1}^k d_{mm}(x_{n_j}, y)e^{-\\tau_{1, n} d_{\\mathcal{X}}(x, x_{n_j})}$\n\nand $s_m(x, y, \\mathcal{D}) = \\frac{1}{k} \\displaystyle\\sum_{j=1}^k d_{mm} (x, y_{m_j})e^{-\\tau_{1, m}d_{\\mathcal{Y}} (y, y_{m_j}) }$\n\nWhich drops the $\\tau_2$ hyperparameter as the reviewer suggested. To maintain fairness, we use the same model selection strategy and hyperparameter grid for remaining hyperparameters as $\\text{LEMoN}\\_{\\text{opt}}$.  We evaluate this alternate neighborhood score versus LEMoN, and [these results can be found here](https://postimg.cc/BXnj5XSj). We find that LEMoN outperforms this alternate neighbor method on the majority of datasets.\n\nFinally, we note that we have also discussed some alternate ways of integrating neighbor information in Appendix B when we compare LEMoN conceptually and empirically with a baseline which uses semantic neighborhood information for a different purpose. We will add more exposition to this discussion in the revision.\n\n\n\n\n> It is not clear if filtering noisy data is preferred to techniques that learn in the presence of label noise. Empirical comparison with [2] on captioning datasets would strengthen the paper.\n\n> When are filtration based approaches preferred over techniques that learn with label noise?\n\n\nIn our view, noise-robust training algorithms are a disjoint field of work from noisy label identification. In particular, identifying noise labels is a more flexible approach, with applications beyond just removing these samples for downstream model training. By identifying mislabeled samples, we can also characterize systematic errors or biases in datasets (such as in Figures I.4 and I.5), which can then be fixed, both by repairing existing data and improving future data collection practices. This is especially important for practitioners looking to release high-fidelity datasets for others to train (and especially evaluate) on, as mislabeled samples in test sets have been shown to destabilize ML benchmarks [3]. We will add some discussion on this to the revised version of the paper.\n\nFinally, per-sample mislabel identification methods such as LEMoN can also be deployed to flag incorrect human inputs in an online setting. One particular example might be flagging simple mistakes made by radiologists when writing notes from chest X-rays (as motivated by our MIMIC-CXR setting). \n\n\n[1] Learning with Noisy Labels Revisited: A Study Using Real-World Human Annotations. ICLR 2022.\n\n[2] Beyond Synthetic Noise: Deep Learning on Controlled Noisy Labels. ICML 2020.\n\n[3] Pervasive Label Errors in Test Sets Destabilize Machine Learning Benchmarks. NeurIPS 2021.", "claims_and_evidence": "The paper theoretically justifies their score by deriving an expression for the detection AUROC. Extensive empirical evidence on noise simulated datasets is also provided for the same. However, it is not clear if this performance is translated to datasets with realistic noise, as evidenced by the CC3M experiment.", "methods_and_evaluation_criteria": "Experiments are performed on 8 datasets covering both classification and captioning. The paper evaluates efficacy of the proposed score on label error detection (AUROC & F1 score) as well as downstream impact of filtering noisy samples.", "theoretical_claims": "Theorem 4.1 (AUROC of k-NN score) is not specific to the proposed score, and may be valid for other kinds of scoring (one such alternative is proposed below). I have not checked correctness of Thm A.1 in the appendix.", "experimental_designs_or_analyses": "Yes, the experimental design is sound", "relation_to_broader_scientific_literature": "The paper extends existing work on filtering noisy labels to incorporate nearest-neighbor consistency in a multimodal fashion. This is a novel contribution. The paper also performs thorough empirical analysis. However there is no empirical comparison with a related body of work on learning with noisy correspondences [1,2].\n\n[1] Radenovic, Filip et al. “Filtering, Distillation, and Hard Negatives for Vision-Language Pre-Training.” 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2023): 6967-6977.   \n[2] Huang, R., Long, Y., Han, J., Xu, H., Liang, X., Xu, C.,and Liang, X. Nlip: Noise-robust language-image pretraining. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pp. 926–934, 2023", "essential_references_not_discussed": "Some discussion on methods that learn from noisy correspondences is missing [1,3, 4]\n\n[1] Radenovic, Filip et al. “Filtering, Distillation, and Hard Negatives for Vision-Language Pre-Training.” 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2023): 6967-6977.   \n[3] Andonian, Alex et al. “Robust Cross-Modal Representation Learning with Progressive Self-Distillation.” 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2022): 16409-16420.   \n[4] Chen, Hao et al. “Understanding and Mitigating the Label Noise in Pre-training on Downstream Tasks.” ArXiv abs/2309.17002 (2023): n. pag.", "other_strengths_and_weaknesses": "Strengths:   \n\\+ Preprocessing large scale web-crawled datasets is an essential step in the training/fine-tuning large foundational models. The paper explores a novel neighborhood consistency based approach to filter out noisy correspondences from datasets, which can help in improving downstream performance.   \n\\+ The proposed approach is motivated well, and the paper is well written. However, some further discussion is needed to understand why simpler cross modal alternatives are not explored (see weaknesses below).  \n\\+ Empirical evaluation is thorough, and the proposed scoring rule outperforms baselines in detecting noisy samples (improved F1 and AUROC metrics) for both classification and captioning datasets synthetically augmented with noise.\n\nWeaknesses:      \n\\- Empirical performance on realistic noise. Downstream performance on CC3M and Datacomp is almost the same as Clip Similarity baseline. Although a human study on noise detection is performed in Section 6.5, it is not clear if the proposed score translates to downstream performance.      \n\\- Simpler ways to incorporate k-NN information. For ex. average CLIP score between the caption $y$ and neighboring images of $x$ (and vice versa). This avoids the $\\tau_2$ hyperparam, but ignores paired information of neighbors. It is not clear to me if the proposed approach is the optimal way of adding multimodal consistency. A clarification of the same would be appreciated.   \n\\- It is not clear if filtering noisy data is preferred to techniques that learn in the presence of label noise. Empirical comparison with [2] on captioning datasets would strengthen the paper.", "other_comments_or_suggestions": "\\- In L1731 should refer to table I.14   \n\\- In table I.14, unfiltered outperforms both kinds of filtering on average", "questions_for_authors": "1. As described in weaknesses, a discussion on simpler ways to incorporate k-NN information would be informative.\n2. When are filtration based approaches preferred over techniques that learn with label noise?\n3. Empirical performance on realistic downstream tasks is unsatisfactory. Perhaps fine-tuning under limited data (few-show training samples) is better suited to evaluate these methods?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "kjztQgdbMr", "reviewer": "Reviewer_9xRU", "summary": "This paper tries to apply a neighbor-based noisy sample detection method to a multimodal dataset (image-text pairs dataset) with the help of a pre-trained vision-language model. The authors also provide theoretical proof to show that their method has better noise detection capability than random detection. Experiments on various datasets have shown the proposed method's efficacy and robustness to hyper-parameters.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the insightful review and constructive feedback!\n\n> In appendix A.2, authors only present the visualization results for classification tasks, which is a much easier case than tasks involving image captioning, where the text can be natural language and have more diverse space on J(x).\n\n> For assumption 2, though appendix A.2 shows the distribution of the whole dataset, it is much better to show the sample-wise visualization results, since assumption 2 is a sample-wise claim.\n\nThank you for pointing this out! To address these two concerns simultaneously, we conduct an experiment on captions from the flickr30k dataset. We select 20 random captions, then use Llama 3.1-8B-instruct to generate 50 paraphrasings of each caption (via sampling with temperature), corresponding to 50 samples from $\\mathcal{H}(Y)$ for each caption. For the samples $Y' \\not\\in \\mathcal{H}(Y)$, we randomly select 50 other captions from the dataset. To match the support of the Gaussian, we take the distance function to be the log cosine distance (note that this does not change the ordering of the score across samples). We compute this distance using the text encoder from OpenAI CLIP ViT-B/32, and plot histograms for each caption. [The results are shown here](https://postimg.cc/HVpKsCp0). Running the same Shapiro-Wilk test from Appendix A.2, we find that of the positive samples, 8/20 are Gaussian, and of the negative samples, 16/20 are Gaussian. Thus, there is some evidence the Gaussianity assumption holds for natural language and complex paraphrase functions.\n\n> the conclusion on \"our proposed multimodal neighborhood score, provides a better than random signal at detecting mislabeled samples\" is tricky. As long as there are slight distribution differences over two distributions (say, with different $\\mu$) in appendix A.2, this conclusion still holds, but does not provide significant insight into how the detection method is good enough.\n\nWe emphasize that Lemma 4.2 is a specialization of Theorem 4.1 meant to demonstrate that only loose conditions are necessary to obtain non-random signal. Our Theorem 4.1 provides the exact expression for the AUROC of the score as a function of the distribution parameters.\n\n> Could authors please show the experiment results on directly setting γ1=γ2=0 in the algorithm?\n\nWe have provided results for setting $\\tau_1 = \\tau_2 = 0$ in Table I.9.\n\n> However, it seems that the impact of noise level is not presented in the theoretical part, and authors directly claim that the proposed method can achieve better noise detection performance than the random signal case. \n\nThe impact of noise level in the neighbors is accounted for in Theorem 4.1 via the $p$ term.\n\n> Notice the experiment in Figure I.1 in Appendix for different noise levels, please explain the phenomenon and tendency on F1/AUROC on mother datasets. \n\nTo better match the theoretical setting, we [examine the performance of individual scores](https://postimg.cc/ct7mG6gq) $s_n$ and $s_m$, without the $d_{mm}$ term and with $\\tau_1 = \\tau_2 = 0$. Looking at the influence of $p$ in Theorem 4.1, we find that as $p \\rightarrow 1$, the AUROC approaches 0.5. As $p \\rightarrow 0$, taking distribution parameters to be fixed (i.e. $\\mu_1, \\sigma_1$, etc), the AUROC approaches a fixed constant, which, under the assumptions of Lemma 4.2, is greater than 0.5.\n\nFor $p \\in (0, 1)$, the function is strictly decreasing in $p$. Thus, from the theory, we would expect the AUROC to be strictly decreasing with higher noise rate, going down to 0.5 for $p = 1$. Empirically, we do observe the decrease in AUROC, with a faster decrease for mscoco than mmimdb (which according to the theory is due to dataset specific parameters like $\\mu$'s, $\\sigma$'s, and the moments of $\\zeta$).  \n\nRegarding variance: we would like to note that the result of our Theorem 4.1 is for the \"population\" AUROC without finite sample considerations. In practice, as in our experiments, AUROC is estimated using finite samples from a fixed dataset. The variance that is observed is due to the variance of this statistical estimator. The variance of empirical AUROC is related to the variance of a Mann-Whitney U statistic, and has been characterized in [1]. This statistical variance is independent of our theorem.\n\nFinally, our theory does not provide an explanation for the F1 score. This is tricker to characterize theoretically, as F1 is computed given a particular threshold on the score. This threshold is selected to be the one that maximizes the F1, and the F1 is a non-concave function of this threshold.\n\n> Essential References Not Discussed\n\nThank you for pointing these out. We have added [2] and [3] respectively to address these.\n\n[1] Confidence Intervals for the Area under the ROC Curve. NeurIPS 2004.\n\n[2] Deep k-NN for Noisy Labels. ICML 2020.\n\n[3] Detecting Corrupted Labels Without Training a Model to Predict. ICML 2022.", "claims_and_evidence": "- assumption 2: in appendix A.2, authors only present the visualization results for classification tasks, which is a much easier case compared with tasks involving image captioning, where the text can be natural language and can have more diverse space on $\\mathcal{J}(x)$. \n- For assumption 2, though appendix A.2 shows the distribution of the whole dataset, it is much better to show the sample-wise visualization results, since assumption 2 is a sample-wise claim. Say, a clean sample $X$ can belong to a Gaussian distribution with a lower mean value compared with its noisy version or noisy neighbors, however, does this claim still hold when comparing $X_1$ and $X_2$ that $X_1 \\neq X_2$ and have even different $Y_1$ and $Y_2$? \n- the conlcusion on \"our proposed multimodal neighborhood score, provides a better than random signal at detecting mislabeled samples\" with $\\mathbb{P}\\left(S_m\\left(X^{\\prime}, Y^{\\prime}\\right)>S_m(X, Y)\\right)>0.5$ is tricky. As long as there are slight distribution differences over two distributions (say, with different $\\mu$) in appendix A.2, this conclusion still holds, but does not provide significant insight into how the detection method is good enough. \n- In the theoretical part, authors try to set $\\gamma_1=\\gamma_2=0$ for the proof, and the final AUROC result is still higher than the random signal case. Could authors please show the experiment results on directly setting $\\gamma_1=\\gamma_2=0$ in the algorithm?\n- for different noise ratio experiments. For the neighbor-based intuition of the proposed method, and that the neighbor is from the original dataset, my concern for the authors: when the noise level increases, the neighbor pairs can also contain a lot of noisy pairs, which makes the algorithm not that reliable since it heavily relies on the quality of neighbor pairs. However, it seems that the impact of noise level is not presented in the theoretical part, and authors directly claim that the proposed method can achieve better noise detection performance than the random signal case. Thus, the impact of noise level in the neighbor pairs should also be considered in the theoretical part, and how this noise level impacts the final AUROC.\n- Notice the experiement in Figure I.1 in Appendix  for different noise levels, please explain the phenomenon and tendency on F1/AUROC on mother datasets. It seems when noise level increases, the F1 will drop, and AUROC becomes very unstable (large variance). Could you please combine these observation into the theoretical part? or how to understand this phenominon based on the theoretical part.", "methods_and_evaluation_criteria": "Yes. Authors follows the classic noise annotation setting from image clasificaiton task for vision-language datasets, and also follows the experiment setting from previous noisy vision-language related work.", "theoretical_claims": "Yes. I have check the theoretical part. Please check the feedback in __Claims And Evidence__.", "experimental_designs_or_analyses": "I have checked the experiment design. Related concern:\n- In appendix A.2, authors only present the visualization results for classification tasks, which is a much easier case than tasks involving image captioning, where the text can be natural language and have more diverse space on $\\mathcal{J}(x)$. \nFor experiments on different noise levels, please explain the phenomenon and tendency on F1/AUROC on the mother datasets. It seems that when the noise level increases, F1 drops, and AUROC becomes very unstable (large variance). Could you please combine these observations into the theoretical part or explain how to understand this phenomenon based on the theoretical part?", "relation_to_broader_scientific_literature": "- This paper tries to apply neighbor-based noisy sample detection method for multimodal dataset (image-text pairs dataset) with the help of pre-trained vision-language model. \n- Though their previous work uses neighbor-based methods for unimodal datasets with noise, this paper is the first to apply neighbor-based methods on multimodal settings. \n- Authors also provide theoretical proof to show their method has better noise detection capability than the random detection.", "essential_references_not_discussed": "- Line 065 left column \"While prior techniques utilize unimodal neighbors for label error detection\", please add reference for this sentence.\n- Line 131 right column \"Prior works have alternatively aimed to maximize the F1 score\", please add reference", "other_strengths_and_weaknesses": "__Strengths:__\n- clearly written, easy to follow, and understand the paper\n- Originality: Though I got insights from many related previous works, the originality of this paper is enough for publication.\n\n__Weakness:__ please check the __Experimental Designs Or Analyses__ and __Claims And Evidence__.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "Please check the __Experimental Designs Or Analyses__ and __Claims And Evidence__.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "HAetmpeHla", "reviewer": "Reviewer_bu6V", "summary": "LEMON is a method designed to identify mislabeled image-caption pairs in large vision-language datasets, which often contain noisy data scraped from the web. Unlike previous approaches that rely solely on image-caption embedding similarity for filtering, LEMON leverages multimodal neighborhood information in the latent space of contrastively pretrained models to detect label errors. The authors theoretically justify and empirically validate LEMON across eight datasets and ten baselines, demonstrating that it improves label error detection by over 3% and enhances downstream captioning performance by 2 BLEU points.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the insightful review and constructive feedback!\n\n> I noticed that the most recent dataset used was published in 2020. Is the method still effective on datasets published in the last two years? Do mislabeled data still exist in these recent datasets?\n\nWe clarify that we evaluated our method on CC3M [1] (from 2021) in Appendix I.9 and the DataComp benchmark [2] (from 2023) in Appendix I.10. As we have motivated in the introduction, and also has been highlighted in many prior works [2-4], the issue of mislabeled data is only growing with time due to the use of billion-sample scale datasets collected from scraping the web.\n\nFinally, we note that the published work which the reviewer later references [5] uses no datasets from later than 2015.\n\n\n\n> The author compares the method with LLaVA but does not provide fine-tuned results for LLaVA. Since it is known that mislabeled data degrade model performance, and LLaVA does not address mislabeled data through fine-tuning, the comparison with large models lacks persuasiveness.\n\n> The study does not provide results on fine-tuning large models such as LLaVA. Since mislabeled data can degrade model performance, it is important to examine whether filtering with LEMON improves the performance of fine-tuned large models. A comparison with fine-tuned LLaVA would strengthen the persuasiveness of the results.\n\n\nWe have already conducted several experiments showing that filtering with LEMoN improves the performance of downstream large models. In particular, we have finetuned GenerativeImage2Text models in Section 6.2, pretrained CLIP models on MIMIC-CXR in Section 6.4, pretrained CLIP models on CC3M in Appendix I.9, and pretrained CLIP models on DataComp in Appendix I.10. We believe this sufficiently addresses the reviewer's concern.\n\n\n\n> The author only considers image and text modalities. However, could the method generalize to more widely used modalities such as video and audio? Can relevant experiments be provided to support this?\n\n> Additionally, the current work focuses on image and text modalities. Expanding the study to include other widely used modalities, such as video and audio, would help demonstrate the generalizability of the method. Conducting experiments on multimodal datasets beyond image-caption pairs could further validate LEMON’s effectiveness in broader applications\n\nWe believe that demonstrating LEMoN's effectiveness on image-text pairs is a sufficient contribution, and extending it to other modalities is out of scope for this paper. We will note this as an area of future work in the revision.\n\n\n\n> While the paper claims novelty in using multimodal scoring for label noise detection, a similar approach has recently been explored in \"VDC: Versatile Data Cleanser based on Visual-Linguistic Inconsistency by Multimodal Large Language Models,\" ICLR 2024.\n\n\nWe have already compared our method to this baseline in Table 2. We strongly dispute the claim that VDC is a \"similar approach\" to LEMoN. VDC entirely relies on prompting LLMs and VLLMs. In contrast, our method does not utilize any prompt engineering, and instead utilizes the neighborhood information in image and text representations of contrastively pretrained models. As a result, not only does VDC perform worse empirically (Table 2), it also has much higher runtime (Table I.11). We emphasize that VDC does not utilize multimodal neighbors, or even embeddings at all, in any form.\n\n\n\n[1] Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Concepts. CVPR 2021.\n\n[2] DataComp: In search of the next generation of multimodal datasets. NeurIPS 2023.\n\n[3] Multimodal datasets: misogyny, pornography, and malignant stereotypes. arXiv:2110.01963.\n\n[4] What's In My Big Data? ICLR 2024.\n\n[5] VDC: Versatile Data Cleanser based on Visual-Linguistic Inconsistency by Multimodal Large Language Models. ICLR 2024.", "claims_and_evidence": "YES", "methods_and_evaluation_criteria": "YES", "theoretical_claims": "YES", "experimental_designs_or_analyses": "YES", "relation_to_broader_scientific_literature": "YES", "essential_references_not_discussed": "YES", "other_strengths_and_weaknesses": "Strengths:\n-    The author utilizes multimodal neighbor detection to identify mislabeled data, a simple and effective method that is easy to follow.\n-    The author employs theoretical analysis to prove the effectiveness of the LEMON method.\n-    Extensive experiments demonstrate that mislabeled data can degrade model performance, providing significant insights for future research.\n\nWeaknesses:\n-    I noticed that the most recent dataset used was published in 2020. Is the method still effective on datasets published in the last two years? Do mislabeled data still exist in these recent datasets?\n-    The author compares the method with LLaVA but does not provide fine-tuned results for LLaVA. Since it is known that mislabeled data degrade model performance, and LLaVA does not address mislabeled data through fine-tuning, the comparison with large models lacks persuasiveness.\n-    The author only considers image and text modalities. However, could the method generalize to more widely used modalities such as video and audio? Can relevant experiments be provided to support this?\n-    While the paper claims novelty in using multimodal scoring for label noise detection, a similar approach has recently been explored in \"VDC: Versatile Data Cleanser based on Visual-Linguistic Inconsistency by Multimodal Large Language Models,\" ICLR 2024.", "other_comments_or_suggestions": "The study does not provide results on fine-tuning large models such as LLaVA. Since mislabeled data can degrade model performance, it is important to examine whether filtering with LEMON improves the performance of fine-tuned large models. A comparison with fine-tuned LLaVA would strengthen the persuasiveness of the results.\n\nAdditionally, the current work focuses on image and text modalities. Expanding the study to include other widely used modalities, such as video and audio, would help demonstrate the generalizability of the method. Conducting experiments on multimodal datasets beyond image-caption pairs could further validate LEMON’s effectiveness in broader applications", "questions_for_authors": "None", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: We have a rather solid consensus (4 weak accepts) towards acceptance of the paper. The AC agrees, having checked the paper, reviews, and rebuttals.
Decision: Accept (poster)