LEMoN: Label Error Detection using Multimodal Neighbors

## Abstract

Abstract Large repositories of image-caption pairs are essential for the development of vision-language models. However, these datasets are often extracted from noisy data scraped from the web, and contain many mislabeled examples. In order to improve the reliability of downstream models, it is important to identify and filter images with incorrect captions. However, beyond filtering based on image-caption embedding similarity, no prior works have proposed other methods to filter noisy multimodal data, or concretely assessed the impact of noisy captioning data on downstream training. In this work, we propose LEMoN , a method to automatically identify label errors in multimodal datasets. Our method leverages the multimodal neighborhood of image-caption pairs in the latent space of contrastively pretrained multimodal models. We find that our method outperforms the baselines in label error identification, and that training on datasets filtered using our method improves downstream classification and captioning performance.

## 1 Introduction

Machine learning datasets used to train and finetune large vision, language, and vision-language models frequently contain millions of labeled instances [, , , ]. Given the large scale of modern datasets, ensuring the correctness of labels is often impossible [, , , , ], as seen in Figure 1. This is especially critical in settings such as healthcare, where the reliability of downstream models may depend on the quality of data used for pretraining [, , ].

Figure: Figure 1: Samples from classification and captioning datasets discovered to be mislabeled by our method.
Refer to caption: x1.png

Identifying and correcting label errors in existing datasets at scale would lead to more reliable and accurate models in the real world [, , , , ]. However, given the large size of such datasets, manual detection of such errors is practically infeasible. This is evidenced by the growth of models trained on noisy data with the web [, , ], or with model generated pseudo-labels [, ]. Machine learning-based label error detection methods have thus been proposed to automate this process [, ].

Several approaches to automatically identifying label errors have been proposed in prior work [, , , , , ]. However, we identify two critical limitations: (1) a majority of such works are *unimodal*: i.e., they only utilize image-based representations and detection strategies, and (2) some of the most performant models require access to the predictions from a model trained for several iterations on the downstream task of interest [, ].
We hypothesize that applying a neighborhood-based approach to rich, multimodal representations in the form of image-text pairs can improve label error detection without requiring costly task-specific training.

Additionally, a common assumption made in prior works is that each label is one-of-k classes [, ]. In contrast, datasets used to train large vision-language models contain natural language labels such as image captions [, , ]. Some methods to filter out instances with noisy labels – e.g., based on the similarity of image and caption representations – have been utilized in prior work with some success [, ]. However, to the best of our knowledge, no prior works have proposed or rigorously compared methods to identify errors in settings with natural language labels, or assessed the impact of detection on downstream tasks like image captioning.

In this work, we propose LEMoN– Label Error detection using Multimodal Neighbors – a method for multimodal label error detection, which can be applied to image-text pairs in datasets such as MSCOCO []. While prior techniques utilize unimodal (i.e., image) neighbors for label error detection, LEMoN leverages multi-modal neighborhoods derived using contrastively pretrained vision-language models [].
Specifically, in addition to considering pairwise image-text distances, we also retrieve nearest neighbors in the image and text space as illustrated in Figure 2. We then compute distance scores with neighbors in each modality and combine these into a single score measuring the likelihood of a label error, with the intuition that higher discordance (or higher distance) with neighbors indicates a higher chance of label error.

We first show that LEMoN performs on par with prior works in the classification setting, without requiring any information about the downstream task of interest, outperforming all task-unaware baselines in the classification setting by over 4% F1-score (Section 6.1). We then extend baselines to the image captioning setting, and find that LEMoN outperforms all baselines in captioning setups that do not require supervised training by over 3% F1-score on average (Section 6.1). Filtering out data predicted to be label errors by LEMoN improves downstream classification and caption generation performance (Section 6.2) over training on noisy labels, and nearly recovers the performance obtained on training with clean data. Finally, through ablations and a real-world evaluation (Section 6.5), we verify that the predictions generated by LEMoN are meaningful. In summary, LEMoN is able to surface label errors in image captioning datasets, resulting in improvements to downstream use(^†^†Code: [https://github.com/MLforHealth/LEMoN](https://github.com/MLforHealth/LEMoN)).

Figure: (a)
Refer to caption: x2.png

## 2 Related Works

##### Label Noise Detection

Noisy and incorrect labels [] in training data may lead to decreased or “destabilized” [, ] performance on downstream tasks [, ]. Two orthogonal approaches can be taken to reduce the adverse effects of such labels: developing methods to learn in the presence of label errors [, , ], or detecting and filtering out instances with label errors []. In this work, we focus on the latter approach. Prior approaches [, , , , , , ] for automatic label error detection include relying on the training dynamics of task-specific downstream models [] and neighborhood-based strategies [, ]. Some of these techniques are fully supervised [, ] or unsupervised [, , , ], use pre-trained generative models [] or are fully training-free approaches [, ]. Previous work closest to this work includes deep k-nearest neighbor (deep k-NN) methods using k-NN entropy on vector space embeddings [, ] and SimiFeat [] which employs a local neighborhood-based voting or ranking for noise identification. Building upon these methods, our work enhances label noise detection by harnessing information across multiple data modalities, such as image and text.

##### Contrastive Learning

Contrastive learning is a representation learning strategy that contrasts positive and negative pairs of data instances [, , ] to learn an embedding space. The core idea is to embed similar data points (positive pairs) closer together than dissimilar data points (negative pairs) [, , ]. Specifically, paired observations, created using multiple data modalities or augmented/mined views of the same object, are contrasted against negative observation pairs. In this work, we utilize pre-trained models that use the CLIP loss for jointly embedding image and text data []. Utilizing contrastive learning for multimodal learning has shown promise in zero-shot and few-shot image classification [, , , ], image captioning [, ], and retrieval tasks [].

##### Image Captioning

The goal of image captioning is to describe a given image [].
Prior works for image captioning have included training end-to-end models from scratch in a supervised way [, , , , ]. More recently, pretrained vision-language models trained on large datasets of noisy image-caption pairs extracted from the web [, , ] – such as Conceptual Captions 12M [] – have been utilized for captioning. Some of the pretraining tasks include image-text contrastive learning, image-text matching, and/or retrieval [], as well as general purpose text generation conditioned on an input image. Given that datasets for training such large models are noisy [], several steps have been utilized in prior work to filter out noisy captions during training. The most common strategy involves computing the similarity between representations of the image and caption text using another pre-trained model (e.g., CLIP) prior to training []. Another approach in training the BLIP [] model is to synthetically generate noisy captions and train a classifier to distinguish between high quality captions (e.g., from MSCOCO) and noisy captions with a cross-entropy loss []. However, no prior works, to the best of our knowledge have compared different strategies for label error detection comprehensively. In this work, we systematically introduce meaningful errors in captioning datasets and compare seven methods for automatic label error detection.

## 3 LEMoN : Label Error Detection using Multimodal Neighbors

We are given a dataset $\mathcal{D}=\{(\mathbf{x},\mathbf{y})_{i=1}^{N}\}$ consisting of two modalities $\mathbf{x}\in\mathcal{X}$ and $\mathbf{y}\in\mathcal{Y}$. For example, $\mathcal{X}$ may represent the set of all natural images, and $\mathcal{Y}$ may represent the set of all English text, or a restricted subset such as $\{\texttt{cat},\texttt{dog},...\}$. We assume the existence of, but not access to, an oracle $f^{*}:\mathcal{X}\times\mathcal{Y}\rightarrow\{0,1\}$, which is able to assign a binary mislabel indicator $z_{i}=f^{*}(\mathbf{x}_{i},\mathbf{y}_{i})$ to each sample in $\mathcal{D}$, without ambiguity. Here, $z_{i}=1$ indicates that the sample is mislabeled, and $z_{i}=0$ indicates that the sample is correctly labeled. Our goal is to output a score $s\in\mathbb{R}$ with some model $s:=f(\mathbf{x},\mathbf{y})$
such that

$$ $\text{AUROC}=\mathbb{E}_{\begin{subarray}{c}(\mathbf{x},\mathbf{y})\sim\mathbb {P}(\cdot|z=1)\\ (\mathbf{x}^{\prime},\mathbf{y}^{\prime})\sim\mathbb{P}(\cdot|z=0)\end{ subarray}}[\mathbf{1}_{f(\mathbf{x},\mathbf{y})\geq f(\mathbf{x}^{\prime}, \mathbf{y}^{\prime})}]$ $$

is maximized. Prior works have alternatively used the F1 score of mislabeling for evaluation, optimizing over a threshold $t$:

$$ $\text{F1}=\max_{t\in\mathbb{R}}\frac{2\cdot\mathbb{P}(z=1|s\geq t)\cdot\mathbb {P}(s\geq t|z=1)}{\mathbb{P}(z=1|s\geq t)+\mathbb{P}(s\geq t|z=1)}$ $$

Here, inspired by prior work [, ], we propose a method for $f$ based on nearest neighbors, summarized in Figure 2. Suppose we have a query sample $(\mathbf{x},\mathbf{y})$(^1^11One could take, for any $i$, $(\mathbf{x},\mathbf{y}):=(\mathbf{x},\mathbf{y})_{i},D^{\prime}:=D\setminus\{( \mathbf{x},\mathbf{y})_{i}\}$). Define $B(\mathbf{x},r):=\{x^{\prime}\in\mathcal{X}:d_{\mathcal{X}}(\mathbf{x},\mathbf
{x}^{\prime})\leq r\}$, and $B(\mathbf{y},r)$ similarly. Here, $r_{k}(x):=\inf\{r:|B(\mathbf{x},r)\cap\mathcal{D}|\geq k\}$. Then, we define $\{\mathbf{x}_{n_{1}},\mathbf{x}_{n_{2}},...,\mathbf{x}_{n_{k}}\}:=B(\mathbf{x}
,r_{k}(x))\cap D$ the top $k$ nearest neighbors of $\mathbf{x}$, and $\{\mathbf{y}_{m_{1}},\mathbf{y}_{m_{2}},...,\mathbf{y}_{m_{k}}\}:=B(\mathbf{y}
,r_{k}(y))\cap D$ the top $k$ nearest neighbors of $\mathbf{y}$(^2^22We will use a subscript $n_{j}$ to index nearest neighbors in $\mathcal{X}$, and subscript $m_{j}$ for neighbors in $\mathcal{Y}$.). We assume that the neighbors are sorted in order of ascending distance, so for e.g. $d_{\mathcal{X}}(\mathbf{x},\mathbf{x}_{n_{2}})\geq d_{\mathcal{X}}(\mathbf{x},
\mathbf{x}_{n_{1}})$.

If $\mathcal{Y}$ is a small discrete set, we could choose $d(\mathbf{y},\mathbf{y}^{\prime})=\mathbf{1}_{\mathbf{y}=\mathbf{y}^{\prime}}$. If $\mathcal{X}$ or $\mathcal{Y}$ are unstructured or high dimensional, we assume access to multimodal encoders $h_{\theta}=(h^{\mathcal{X}}_{\theta},h^{\mathcal{Y}}_{\theta})$, where $h^{\mathcal{X}}_{\theta}:\mathcal{X}\rightarrow\mathbb{R}^{d}$ and $h^{\mathcal{Y}}_{\theta}:\mathcal{Y}\rightarrow\mathbb{R}^{d}$. Here, $h_{\theta}$ may be a CLIP model [] trained on a large internet corpus, or, as we show later, it may be sufficient to train $h_{\theta}$ from scratch only on $\mathcal{D}$. Then, we could naturally use simple distance metrics in the embedding space, such as the cosine distance $d_{\mathcal{X}}(\mathbf{x},\mathbf{x^{\prime}})=d_{\text{cos}}(h^{\mathcal{X}}
_{\theta}(\mathbf{x}),h^{\mathcal{X}}_{\theta}(\mathbf{x}^{\prime}))=1-\frac{h
^{\mathcal{X}}_{\theta}(\mathbf{x})\cdot h^{\mathcal{X}}_{\theta}(\mathbf{x}^{
\prime})}{||h^{\mathcal{X}}_{\theta}(\mathbf{x})||_{2}\cdot||h^{\mathcal{X}}_{
\theta}(\mathbf{x}^{\prime})||_{2}}$. Our proposed score is the linear combination of three terms:

$$ $s=f(\mathbf{x},\mathbf{y})=d_{mm}(\mathbf{x},\mathbf{y})+\beta s_{n}(\mathbf{x },\mathbf{y},\mathcal{D})+\gamma s_{m}(\mathbf{x},\mathbf{y},\mathcal{D})$ (1) $$

Where $\beta,\gamma\geq 0$ are hyperparameters.
Here, $d_{mm}(\mathbf{x},\mathbf{y}):=d_{\text{cos}}(h_{\theta}^{\mathcal{X}}(\mathbf
{x}),h_{\theta}^{\mathcal{Y}}(\mathbf{y}))$ is the multimodal distance, which has been shown to provide a meaningful signal in prior label error detection work [, ]. We thus use this distance as the basis, and augment it with two additional terms based on nearest neighbors:

$$ $s_{n}(\mathbf{x},\mathbf{y},\mathcal{D})=\frac{1}{k}\sum_{j=1}^{k}d_{\mathcal{ Y}}(\mathbf{y},\mathbf{y}_{n_{j}})e^{-\tau_{1,n}d_{\mathcal{X}}(\mathbf{x}, \mathbf{x}_{n_{j}})}e^{-\tau_{2,n}d_{mm}(\mathbf{x}_{n_{j}},\mathbf{y}_{n_{j}})}$ (2) $$

Where $(\mathbf{x}_{n_{j}},\mathbf{y}_{n_{j}})\in\mathcal{D}$, and $\tau_{1,n},\tau_{2,n}\geq 0$ are hyperparameters. This corresponds to finding the nearest neighbors of $\mathbf{x}$ in $\mathcal{X}$ space, then averaging the distance between their corresponding modality in $\mathcal{Y}$ and $\mathbf{y}$. We weight this average with two additional terms. The $\tau_{1,n}$ term corresponds to downweighting neighbors which are far from $\mathbf{x}$. Intuitively, this is useful when $k$ is too large for $\mathbf{x}$ and not all neighbors are relevant and can be thought of as an adaptive $k$. The $\tau_{2,n}$ term corresponds to downweighting neighbors which are themselves likely to be mislabeled. If $(\mathbf{x}_{n_{j}},\mathbf{y}_{n_{j}})$ is itself mislabeled, then $d_{\mathcal{Y}}(\mathbf{y},\mathbf{y}_{n_{j}})$ would contribute an erroneous signal to whether $(\mathbf{x},\mathbf{y})$ is mislabeled, and we thus want to downweight those instances.

The third term is analogous to $s_{n}$, but uses neighbors of $\mathbf{y}$:

$$ $s_{m}(\mathbf{x},\mathbf{y},\mathcal{D})=\frac{1}{k}\sum_{j=1}^{k}d_{\mathcal{ X}}(\mathbf{x},\mathbf{x}_{m_{j}})e^{-\tau_{1,m}d_{\mathcal{Y}}(\mathbf{y}, \mathbf{y}_{m_{j}})}e^{-\tau_{2,n}d_{mm}(\mathbf{x}_{m_{j}},\mathbf{y}_{m_{j}})}$ (3) $$

Where $(\mathbf{x}_{m_{j}},\mathbf{y}_{m_{j}})\in\mathcal{D}$, and $\tau_{1,m},\tau_{2,m}\geq 0$ are hyperparameters. Crucially, note that $\mathbf{x}_{n_{j}}\neq\mathbf{x}_{m_{j}}$, and $\mathbf{y}_{n_{j}}\neq\mathbf{y}_{m_{j}}$. Specifically, $\mathbf{y}_{n_{j}}$ corresponds to the $\mathcal{Y}$ modality of nearest neighbors taken in $\mathcal{X}$ space, and $\mathbf{y}_{m_{j}}$ corresponds to the nearest neighbors of $\mathbf{y}$ taken in $\mathcal{Y}$ space.

We note that our method is a generalization of several prior methods. When $\beta=\gamma=0$, the method is equivalent to CLIP similarity []. When $\beta$ is large, $\tau_{1,n}=\tau_{2,n}=\gamma=0$, and $d(\mathbf{y},\mathbf{y}_{n_{j}})=\mathbf{1}_{\mathbf{y}=\mathbf{y}_{n_{j}}}$, the method is equivalent to Deep kNN [].

Our method contains several hyperparameters: $k,\beta,\gamma,\tau_{1,n},\tau_{2,n},\tau_{1.m}$, and $\tau_{2,m}$. When there is a validation set with known labels, we perform a grid search over $k$, and use numerical optimization methods to search for an optimal value of the remaining hyperparameters which maximize performance on this set, which we describe further in Section 5.2. We refer to our method in this setting as LEMoNopt.

When there is no labeled validation set available, we will show that our method is fairly robust to these hyperparameter choices, and that choosing a set of reasonable fixed values for these hyperparameters yields nearly comparable results. We refer to our method in this setting as LEMoNfix.

## 4 Theoretical Analysis

In this section, we present a theoretical analysis of the impact of noisy labels in multimodal learning. Specifically, we illustrate that multimodal embeddings are robust when there is noise in one modality, which consequently can be leveraged as noisy detectors in our proposed method.

Let $\mathcal{X}$ and $\mathcal{Y}$ denote the input space of two modalities.

The objective of multimodal representation learning is to find an encoder for each modality: $h_{\theta}^{\mathcal{X}}:\mathcal{X}\to\mathbb{R}^{d}$, and $h_{\theta}^{\mathcal{Y}}:\mathcal{Y}\to\mathbb{R}^{d}$ simultaneously, which is usually realized by minimizing over multimodal contrastive learning loss such as CLIP loss [].
We define the multimodal distance as $d_{mm}(h_{\theta}^{\mathcal{X}}(x),h_{\theta}^{\mathcal{Y}}(y))$ which measures the distance the embeddings from two modalities respectively. To further proceed with our analysis, we first define $y\in Y$ to be the correct label for $x$ and $y^{\prime}\in\mathcal{Y}$ be the noisy label obtained by flipping $y$ with probability $p$, chosen uniformly at random from the set of incorrect labels $\mathcal{Y}\setminus\{y\}$. Additionally, we assume $h_{\theta}^{\mathcal{X}}:\mathcal{X}\mapsto\mathbb{R}^{d}$ and $h_{\theta}^{\mathcal{Y}}:\mathcal{Y}\mapsto\mathbb{R}^{d}$ to be Lipschitz continuous with Lipschitz constants $L_{\mathcal{X}}$ and $L_{\mathcal{Y}}$.

###### Proposition 4.1 .

Let the objective function $\mathcal{L}(h_{\theta}^{\mathcal{X}},h_{\theta}^{\mathcal{Y}})$ be the CLIP loss:

$$ $L(h_{\theta}^{\mathcal{X}},h_{\theta}^{\mathcal{Y}},\delta)=-\frac{1}{2}\left[ \frac{1}{N}\sum_{i=1}^{N}\ln\frac{\exp(h_{\theta}^{\mathcal{X}}(x_{i})^{\top}h _{\theta}^{\mathcal{Y}}(y_{i})/\delta)}{\sum_{k=1}^{N}\exp(h_{\theta}^{ \mathcal{X}}(x_{k})^{\top}h_{\theta}^{\mathcal{Y}}(y_{i})/\delta)}+\frac{1}{N} \sum_{i=1}^{N}\ln\frac{\exp(h_{\theta}^{\mathcal{X}}(x_{i})^{\top}h_{\theta}^{ \mathcal{Y}}(y_{i})/\delta)}{\sum_{k=1}^{N}\exp(h_{\theta}^{\mathcal{X}}(x_{i} )^{\top}h_{\theta}^{\mathcal{Y}}(y_{k})/\delta)}\right]$ $$

Then the multimodal representation loss is robust to label noise and satisfies the following property:

$$ $|L(h_{\theta}^{\mathcal{X}}(x),h_{\theta}^{\mathcal{Y}}(y))-L(h_{\theta}^{ \mathcal{X}}(x),h_{\theta}^{\mathcal{Y}}(y^{\prime}))|\leq C\|h_{\theta}^{ \mathcal{Y}}(y)-h_{\theta}^{\mathcal{Y}}(y^{\prime})\|_{2}$ $$

where $C$ is a constant.

The above proposition indicates that the CLIP loss change is bounded by the embedding difference, since it inherently provides robustness to noisy labels due to the properties of cosine similarity and Lipschitz continuity of the embedding functions. Such robustness enables a multimodal contrastive learning method to train meaningful embeddings under the presence of noisy input [, ]. We will empirically validate this in Section 6.4, by pretraining CLIP from scratch on noisy input.

Moreover, we demonstrate that the embedding models trained via the contrastive multimodal objective are natural noisy label detectors.

###### Proposition 4.2 (Contrastive multimodal embedding models detect noisy labels) .

Suppose $\mathcal{Y}=\mathbb{R}$. For a training dataset $\mathcal{D}$, Suppose $\hat{h}^{\mathcal{X}}_{\theta}(x)$ and $\hat{h}^{\mathcal{Y}}_{\theta}(y)$ are the two embeddings that minimize the empirical CLIP objective on this dataset. Then, for an input $x$ and its corresponding positive counterpart $y$, assume there exists a noisy label $y^{\prime}$ such that $y^{\prime}=y+\eta$ where $\eta\sim\mathcal{N}(0,\sigma^{2})$ and $|\eta|\geq\epsilon$. Then the following holds with high probability at least $1-\delta$, where $\delta$ is a function of $\epsilon$.

$$ $d_{mm}(\hat{h}^{\mathcal{X}}_{\theta}(x),\hat{h}^{\mathcal{Y}}_{\theta}(y)) \leq d_{mm}(\hat{h}^{\mathcal{X}}_{\theta}(x),\hat{h}^{\mathcal{Y}}_{\theta}(y ^{\prime}))$ $$

Therefore, we can see that multimodal embeddings are inherently capable of detecting mislabeled pairs, ensuring the distance between the embeddings of positive pairs is smaller than that of negative pairs. This motivates the use of $d_{mm}$ in LEMoN as well as in prior work [, ]. Proofs can be found in Appendix A.

###### Proposition 4.1 .

###### Proposition 4.2 (Contrastive multimodal embedding models detect noisy labels) .

## 5 Experiments

### 5.1 Datasets

We evaluate our method using six datasets, as shown in Table 1. Two datasets (cifar10 and cifar100) are classic label error detection datasets from the classification setting. The four remaining datasets are image captioning datasets. For mscoco and flickr30k, we use the Karpathy split []. In the remaining datasets, we randomly split each dataset into three parts in an 80-10-10 ratio: training or reference set for the label detection method, validation set for the hyperparameter selection, and test set for testing label error detection performance.

**Table 1: Classification and captioning datasets. $n$ is the number of samples (rounded). In the main paper, results shown are for the bolded noise type with 40% noise level unless otherwise stated. Performance on remaining noise types can be found in the appendices.**
| Dataset | $n$ | Domain | Noise Types |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
|  | Train | Validation | Test | Image | Text |  |
| cifar10 | 40k | 5k | 5k | Natural Images | Object Labels | {human  [], sym., asym.} |
| cifar100 | 40k | 5k | 5k | Natural Images | Object Labels | {human [], sym., asym.} |
| mscoco [] | 83k | 5k | 5k | Natural Images | Captions | {cat., noun, random} |
| flickr30k [] | 29k | 1k | 1k | Natural Images | Captions | {noun, random} |
| mmimdb [] | 16k | 3k | 8k | Movie Posters | Plot Summaries | {cat., noun, random} |
| mimiccxr [] | 370k | 3k | 5k | Chest X-rays | Radiology reports | {cat., random} |

#### 5.1.1 Noise Types

In cifar10 and cifar100, we utilize a dataset collected in prior work [] with human mislabels. We also follow prior work [] in experimenting with class symmetric (sym.) and class asymmetric (asym.) synthetic noise. For the four captioning datasets, we devise several ways to inject synthetic noise of prevalence $p$. The simplest way is to randomly select $p$ fraction of the samples and assign their text modality to be that of another random caption. In datasets where additional metadata is available (mscoco: object category, mmimdb: genre of movie, mimiccxr: disease label), we can randomly swap the caption with that of another sample from the same category. Finally, in all captioning datasets except mimiccxr, we tag each token of each caption with its part-of-speech using SpaCy [], and then randomly assign a selected sample’s text modality to be from another sample with at least one noun in common. Additional dataset processing details can be found in Appendix B.

Our motivation for these noise types is to simulate an array of realistic label corruptions that one might face in the real world. We recognize that the resulting synthetic dataset may not have exact noise level $p$, as e.g. a randomly selected caption may actually be correct for the image, as well as noise in the base datasets, which we explore in Section 6.5. Unless otherwise stated, results shown in the main paper are for the bolded noise type in Table 1, with 40% synthetic noise. Additional results for other noise types can be found in the appendices.

### 5.2 Model Selection and Evaluation

We run LEMoN on each dataset, using the training split of each dataset to compute nearest neighbors. In cifar10 and cifar100, we use the discrete metric $d_{\mathcal{Y}}(\mathbf{y},\mathbf{y^{\prime}})=\mathbf{1}_{\mathbf{y}=\mathbf
{y^{\prime}}}$. In all other cases and for $d_{\mathcal{X}}$, we utilize cosine or euclidean distance computed in the embedding space of a pretrained CLIP model, selecting the best distance metric on the validation set for LEMoNopt, and keeping the distance as the cosine distance for LEMoNfix. In mimiccxr, we use BiomedCLIP (ViT-B/16) [], and we use OpenAI CLIP ViT-B/32 [] for all other datasets. A full list of hyperparameters for our method and the baselines can be found in Appendix E.

For LEMoNopt, we select the hyperparameter combination that maximizes F1 on a labeled validation set. We report the AUROC, AUPRC, and F1 for this model. For LEMoNfix, we fix the hyperparameters at the following values: $k=30$, $\beta=\gamma=5$, $\tau_{1,n}=\tau_{1,m}=0.1$, and $\tau_{2,n}=\tau_{2,m}=5$. We report AUROC and AUPRC, as the F1 requires additional information to compute a threshold for the score.
We recognize that access to such a validation set as in LEMoNopt may be unrealistic, but we will empirically show that (1) our method is fairly robust to selection of these hyperparameters, (2) choosing a fixed reasonable hyperparameter setting as in LEMoNfix achieves nearly comparable results, and (3) hyperparameters optimized on a dataset with synthetic noise may transfer well to real datasets.

We repeat each experiment three times, using a different random seed for the noise sampling (for human noise, we use a different random data split). Performance metrics shown are test-set results averaged over these three runs, with error bounds corresponding to one standard deviation.

##### Baselines

We compare our method versus previous state-of-the-art in both the classification and captioning settings. As multimodal label error detection is fairly new, we additionally adapt several baselines from the classification setting to the captioning setting. We briefly list the baselines here, and a detailed description is in the Appendix C. In all cases, bold denotes best within each training approach.

##### Classification

In the classification setting, we experiment with the following baselines which require training a classifier on the particular dataset: AUM [], Datamap [], and Confident Learning [], and the following baselines which do not require classifier training: Deep k-NN [], SimiFeat [], CLIP Similarity [], and CLIP Logits [].

**Table 2: Label error detection on classification datasets using human noise.**
| Dataset | Method | Training-free | AUROC (%) | AUPRC (%) | F1 (%) |
| --- | --- | --- | --- | --- | --- |
| cifar10 | AUM | ✗ | 98.3 (0.1) | 97.9 (0.1) | 94.0 (0.2) |
| Datamap | 98.2 (0.1) | 97.6 (0.1) | 93.4 (0.5) |  |  |
| Confident | 93.7 (0.4) | 89.4 (0.6) | 92.7 (0.5) |  |  |
| CLIP Logits | ✓ | 95.5 (0.2) | 93.9 (0.3) | 88.0 (0.5) |  |
| CLIP Sim. | 93.8 (0.1) | 92.4 (0.2) | 86.9 (0.4) |  |  |
| Simifeat-V | 89.6 (0.4) | 84.4 (0.9) | 88.0 (0.5) |  |  |
| Simifeat-R | 89.5 (0.4) | 84.3 (0.8) | 87.9 (0.4) |  |  |
| Deep k-NN | 96.2 (0.2) | 93.8 (0.4) | 89.3 (0.4) |  |  |
| LEMoNfix (Ours) | 97.7 (0.2) | 96.8 (0.3) | - |  |  |
| LEMoNopt (Ours) | 98.1 (0.0) | 97.4 (0.1) | 93.1 (0.2) |  |  |
| cifar100 | AUM | ✗ | 92.2 (0.3) | 89.9 (0.5) | 83.9 (0.4) |
| Datamap | 91.7 (0.4) | 89.4 (0.5) | 83.6 (0.3) |  |  |
| Confident | 74.1 (1.7) | 59.3 (2.2) | 69.3 (2.0) |  |  |
| CLIP Logits | ✓ | 84.9 (0.7) | 80.3 (1.2) | 75.5 (0.5) |  |
| CLIP Sim. | 78.5 (0.6) | 72.1 (0.7) | 69.2 (1.3) |  |  |
| Simifeat-V | 77.9 (0.4) | 66.0 (0.6) | 73.1 (0.6) |  |  |
| Simifeat-R | 77.9 (0.7) | 65.9 (0.8) | 73.2 (0.7) |  |  |
| Deep k-NN | 79.2 (0.8) | 66.0 (1.2) | 69.9 (0.2) |  |  |
| LEMoNfix (Ours) | 88.9 (0.7) | 84.6 (1.1) | - |  |  |
| LEMoNopt (Ours) | 90.8 (0.0) | 87.4 (0.3) | 81.3 (0.2) |  |  |

##### Captioning

In the captioning setting, we compare our method with LLaVA [] prompting (v1.6-vicuna-13b), and CapFilt []. We note that the latter can be viewed as an oracle for natural image captioning, as it has been trained in a supervised manner on clean mscoco data.
CLIP Similarity [] and Datamap [] can also be used directly in this setting. Finally, to adapt classification baselines to captioning, we embed the captions using the corresponding CLIP text encoder, and then use the K-Means Clustering method to assign the text caption into one of 100 clusters. We then apply Deep k-NN [] and Confident Learning [], using the cluster ID as the discretized class.

## 6 Results

### 6.1 LEMoN Outperforms Baselines on Label Error Detection

##### Classification

In Table 2, we show the performance of LEMoN against the baselines for label error detection on classification datasets. We find that our method outperforms existing baselines which do not require classifier training on both CIFAR-10 and CIFAR-100. Two downstream-task specific approaches AUM and Datamap outperform all training-free models (particularly on CIFAR-10), but LEMON performs comparably (within 2%) to these methods, and is consistently within the top-3 methods. Similar results are also observed on the two synthetic error types (see Appendix Table G.1). We find that LEMoNfix performs almost comparably with LEMoNopt.

**Table 3: Label error detection performance on captioning datasets.**
| Dataset | Method | AUROC (%) | AUPRC (%) | F1 (%) |
| --- | --- | --- | --- | --- |
| flickr30k | LLaVA | 79.3 (0.8) | 58.5 (0.2) | 65.0 (1.1) |
| Datamap | 53.4 (2.5) | 38.3 (1.3) | 27.1 (1.0) |  |
| Deep k-NN | 70.2 (0.9) | 51.1 (1.0) | 64.4 (1.5) |  |
| Confident | 61.6 (0.4) | 40.6 (0.6) | 54.2 (0.7) |  |
| CLIP Sim. | 94.8 (0.5) | 92.8 (0.5) | 88.1 (0.7) |  |
| LEMoNfix (Ours) | 93.6 (0.2) | 92.0 (0.2) | - |  |
| LEMoNopt (Ours) | 94.5 (0.2) | 92.8 (0.3) | 87.7 (0.9) |  |
| \cdashline2-6 | CapFilt (Oracle) | 98.6 (0.1) | 98.1 (0.1) | 94.8 (0.5) |
| mscoco | LLaVA | 80.3 (0.1) | 63.4 (0.3) | 74.9 (0.3) |
| Datamap | 50.4 (0.6) | 40.5 (0.5) | 28.6 (0.0) |  |
| deep k-NN | 75.9 (0.4) | 69.3 (0.6) | 72.2 (0.5) |  |
| Confident | 66.4 (1.2) | 52.1 (1.2) | 58.9 (1.5) |  |
| CLIP Sim. | 93.8 (0.2) | 93.0 (0.4) | 87.5 (0.3) |  |
| LEMoNfix (Ours) | 92.0 (0.1) | 91.8 (0.3) | - |  |
| LEMoNopt (Ours) | 95.6 (0.2) | 94.6 (0.3) | 89.3 (0.2) |  |
| \cdashline2-6 | CapFilt (Oracle) | 99.3 (0.0) | 99.1 (0.0) | 96.2 (0.3) |
| mmimdb | LLaVA | 58.4 (0.2) | 46.4 (0.2) | 58.5 (0.1) |
| Datamap | 50.1 (0.3) | 39.9 (0.3) | 28.7 (0.1) |  |
| deep k-NN | 60.0 (0.6) | 46.3 (0.4) | 41.9 (3.7) |  |
| Confident | 52.8 (0.8) | 41.4 (0.4) | 53.6 (0.7) |  |
| CLIP Sim. | 85.1 (0.3) | 77.8 (0.7) | 74.5 (0.3) |  |
| LEMoNfix (Ours) | 84.3 (0.3) | 77.7 (0.8) | - |  |
| LEMoNopt (Ours) | 86.0 (0.1) | 79.4 (0.6) | 76.3 (0.1) |  |
| \cdashline2-6 | CapFilt | 82.7 (0.7) | 73.3 (1.2) | 71.6 (0.8) |
| mimiccxr | LLaVA | 53.9 (0.5) | 42.7 (0.7) | 28.7 (0.1) |
| Datamap | 49.9 (1.2) | 39.8 (1.2) | 28.5 (0.0) |  |
| deep k-NN | 58.2 (0.3) | 44.5 (0.1) | 38.8 (3.5) |  |
| Confident | 60.1 (0.3) | 45.6 (0.3) | 59.4 (0.1) |  |
| CLIP Sim. | 64.1 (0.4) | 51.7 (0.5) | 48.6 (3.4) |  |
| LEMoNfix (Ours) | 66.5 (0.2) | 54.8 (0.4) | - |  |
| LEMoNopt (Ours) | 70.4 (2.3) | 60.3 (2.3) | 57.0 (1.6) |  |
| \cdashline2-6 | CapFilt | 49.2 (0.3) | 39.3 (0.6) | 28.5 (0.0) |

##### Captioning

In Table 6.1, we find that our method outperforms existing neighborhood and similarity-based baselines. Our model underperforms a fully supervised oracle-like model (CapFilt), where the supervision and training objective includes distinguishing between accurate and incorrect captions. Results for synthetic error types show similar trends (see Appendix G.2).

##### Label Error Detection Performance Consistent Across Noise Ranges

In Figure 6.1, we show the performance of LEMoN versus the CLIP similarity baseline on mscoco and mmimdb, varying the level of the synthetic noise. We find that our method performs better uniformly across noise levels.

##### Robustness to Hyperparameters

Here, we test the robustness of our method when there is no labeled validation set available. First, in Appendix G.4, we visualize the F1 of the selected score when varying $\beta$ and $\gamma$, keeping all other hyperparameters at their selected optimal values. We find that for most datasets and noise types, there is a reasonably large space of such hyperparameters, bounded away from the origin, which achieves close to optimal performance.

Next, we compare the performance of LEMoNopt and LEMoNfix with hyperparameters described in Section 5.2 across all datasets in Table G.6. We find that when there is no labeled validation set available, using these hyperparameters results in an AUROC drop of only 1.7% on average (std = 1.3%), with a worst-case AUROC drop of 3.9% across all 16 dataset and noise type combinations. Thus, even when a labeled validation set is not available, LEMoNfix with reasonable hyperparameter settings is able to outperform most baselines which do use such information.

### 6.2 Filtering Mislabeled Data Improves Downstream Performance

##### Classification

To assess the impact of label error detection on the performance of the downstream classification tasks, we filter out samples from the training set with mislabel scores in the top $q$ percentile. We vary $q$, train ViT [] models on the filtered dataset, and evaluate the downstream test accuracy using clean data. In Figure 3, we find that training with LEMoNopt filtered samples leads to the highest downstream accuracy on cifar10, and is competitive with another baseline on cifar100, after removing more than 20% of the data. Surprisingly, LEMoN outperforms AUM and Datamap on downstream accuracy, despite these methods achieving higher label error detection performance.

Figure: Figure 3: Downstream classification accuracy on cifar10 (left) and cifar100 (right) with human noise of LEMoNopt versus the baselines. Note that the noise prevalence is 40% on both datasets.
Refer to caption: x4.png

##### Captioning

We finetune a pre-trained Huggingface checkpoint (^3^33[https://huggingface.co/microsoft/git-base](https://huggingface.co/microsoft/git-base)) of a transformer decoder conditioned on CLIP image and text tokens – the GenerativeImage2Text (GIT) [] model – to generate captions. Note that this model is pre-trained on mscoco, and evaluated on the Karpathy test split following Wang *et al.* []. Given the large size of the model, we use the parameter-efficient Low-Rank Adaptation (LoRA) [] for all captioning models with rank $r=4$. We train models with clean data, noisy captions (No Filtering), and by filtering data detected as being mislabeled by a label detection method. In Table 4, we compare results of using either our model or a strong baseline (CLIP Sim.) for filtering data, as measured by the BLEU averaged over 1 to 4 n-grams (B@4) [], CIDER [], and ROUGE [] scores. In all cases, we filtered out the top-40% percentile of data predicted to be mislabeled (i.e., equal to the expected prevalence of noisy data). Experimental results with varying thresholds are in the appendix. We find that (1) filtering out data predicted to be mislabeled helps recover performance as compared to training on fully clean data along multiple metrics, and (2) our method performs comparably to the baseline in improving downstream results.

### 6.3 Ablations

In Table G.7, we show the performance of our method after ablating each of its components. We find that mislabel detection performance almost decreases monotonically as we remove additional components until we reach the CLIP similarity baseline. We find that ablating the $\tau_{1}$ and $\tau_{2}$ terms results in a performance loss of about $1\%$. Ablating either $s_{n}$ or $s_{m}$ entirely also causes a similar drop, though the dependence of each dataset on each term is different, e.g. error detection in mmimdb relies much more on neighbors in the text space than the image space, while the opposite is true for mscoco.

### 6.4 Is External Pretraining Required?

**Table 5: Performance of LEMoN versus the CLIP similarity baseline on mimiccxr, when external pretrained models may not be available. BiomedCLIP [] is trained on a large corpus of biomedical image-text pairs. We find that pretraining only on noisy data from MIMIC-CXR outperforms BiomedCLIP, though pretraining on clean mimiccxr data (as in CheXzero []) does perform better.**
|  |  | Random Noise | Cat. Noise |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- |
|  |  | AUROC | AUPRC | F1 | AUROC | AUPRC | F1 |
| BiomedCLIP | Clip Sim. | 66.8 (0.8) | 54.4 (0.9) | 54.3 (1.0) | 64.1 (0.4) | 51.7 (0.5) | 48.6 (3.4) |
| LEMoNfix (Ours) | 69.5 (0.7) | 57.8 (1.0) | - | 66.5 (0.2) | 54.8 (0.4) | - |  |
|  | LEMoNopt (Ours) | 73.1 (0.9) | 63.0 (2.0) | 63.1 (3.6) | 70.4 (2.3) | 60.3 (2.3) | 57.0 (1.6) |
| CLIP Pretrain On<br>Noisy Data | Clip Sim. | 78.8 (0.1) | 73.4 (0.5) | 70.7 (0.5) | 76.5 (0.5) | 71.2 (0.4) | 67.9 (0.7) |
| LEMoNfix (Ours) | 80.5 (0.1) | 76.1 (0.5) | - | 77.0 (0.5) | 72.4 (0.3) | - |  |
|  | LEMoNopt (Ours) | 80.5 (0.1) | 76.7 (0.3) | 72.8 (0.7) | 77.2 (0.8) | 72.4 (0.6) | 68.7 (0.2) |
| CheXzero | Clip Sim. | 90.8 (0.0) | 89.5 (0.0) | 82.9 (0.2) | 88.4 (0.6) | 86.4 (0.7) | 79.8 (0.7) |
| LEMoNfix (Ours) | 91.4 (0.1) | 90.4 (0.0) | - | 88.4 (0.7) | 87.0 (0.6) | - |  |
|  | LEMoNopt (Ours) | 91.6 (0.3) | 90.5 (0.4) | 84.4 (0.5) | 89.0 (0.3) | 87.0 (0.6) | 80.9 (0.6) |

Thus far, all of the results for LEMoN (and CLIP Similarity) have utilized CLIP models which have been pretrained on external datasets (e.g. PMC-15M in the case of BiomedCLIP). Here, we examine whether this is necessary, or whether we can achieve comparable performance by pretraining CLIP from scratch only on the noisy data. We select mimiccxr as it has the most samples out of all captioning datasets. Similar to CheXzero [], we pretrain a CLIP ViT B/16 from scratch on the mimiccxr training set with 40% noise. We train this model for 10 epochs with a batch size of 64, and do not do any model selection or early stopping. We then apply LEMoN and the CLIP similarity baseline using this model, for the same noise level and noise type. We present our results in Table 5. Surprisingly, we find that pretraining CLIP only on noisy data from MIMIC-CXR actually outperforms BiomedCLIP. This could be attributed to the pretraining domain (chest X-rays and radiology notes) matching the inference domain exactly []. As an upper bound, we evaluate the same methods using CheXzero [], which has been pretrained on clean MIMIC-CXR data. We find that, as expected, it far outperforms our method. We conclude that, for large noisy datasets, pretraining a CLIP model from scratch could be a viable solution, though pretraining on clean data from the same domain is certainly superior.

### 6.5 Real-World Analysis

We conduct a preliminary study of LEMoN on real datasets without known label errors. We run LEMoNfix and the CLIP similarity baseline on cifar10, cifar100, flickr30k, and mscoco. As no labeled validation set is available, we use optimal hyperparameters from models previously run on each dataset with synthetic noise from Section 6.1 (shown in Appendix G.8). For each dataset, we select the top 200 images from the validation and test splits with the highest mislabel scores. We then manually annotated each sample to determine whether it was mislabeled. Crucially, during labeling, images were randomly selected, so the labeler is unaware of whether the candidate image originated from the baseline or our method. We present the accuracy of each method in Table 6. We find that our method outperforms the baseline for every dataset, though we recognize that this is a small-scale study and that many images are ambiguous. Examples of real-world mislabels can be found in Figures 1 and G.5. We present a further comparison of our identified error sets in cifar10 and cifar100 with a prior work [] which obtained crowd-sourced labels for these datasets in Appendix G.10.

**Table 6: We manually label 200 images from real-world datasets that each method identifies as the most likely to be mislabeled and show the percentage (%) of times where it is actually mislabeled. Numbers in parentheses are 95% confidence intervals from a binomial proportion.**
|  | CLIP Sim. | Ours |
| --- | --- | --- |
| cifar10 | 5.5 (3.2) | 10.0 (4.2) |
| cifar100 | 11.0 (4.3) | 20.5 (5.6) |
| flickr30k | 32.5 (6.5) | 41.0 (6.8) |
| mscoco | 19.5 (5.5) | 25.5 (6.0) |

## 7 Conclusion

In this work, we proposed LEMoN, a novel method that leverages the neighborhood structure of contrastively pretrained multimodal embeddings to automatically identify label errors in image-text datasets. Through experiments on multiple datasets with synthetic and real-world noise, we demonstrated LEMoN’s effectiveness in detecting label errors and its ability to improve downstream model performance when used as a dataset filtering tool.

##### Limitations

In this work, we primarily rely on existing open-sourced datasets. While some parts of these datasets may have been used as training data in large pretrained vision-language models, we specifically chose pre-trained models that take care not to include the test sets of such datasets. Further, we run experiments on a real-world healthcare dataset (mimiccxr) to verify our results.

Additionally, we have not assessed label error detection on synthetic instance-dependent [] noise, which is an area of future work. Finally, in our evaluations, we assume that there exists an oracle binary indicator for whether a sample is mislabeled. As we saw in practice, real-world mislabels contain much more uncertainty and ambiguity, e.g. due to blurry images and differing interpretations of text [, , , , , ]. Evaluating the effectiveness of our score as a measure of this uncertainty, in the case of a non-binary target, is an area of future work.

## Appendix A Theoretical results

### A.1 Proposition 4.1

Assumptions:

- 1.
Noise Model: Let $y\in\mathcal{Y}$ be the correct text label and $y^{\prime}\in Y$ be the noisy label obtained by flipping $y$ with probability $p$, chosen uniformly at random from the set of incorrect labels $\mathcal{Y}\setminus\{y\}$. The noisy text embedding is denoted as $g^{\prime}(y)$.
- 2.
Embedding Functions: Assume $f:\mathcal{X}\to\mathbb{R}^{d}$ and $g:\mathcal{Y}\to\mathbb{R}^{d}$ are Lipschitz continuous functions with Lipschitz constants $L_{f}$ and $L_{g}$, respectively. This means:
$\displaystyle\left\lVert f(x_{1})-f(x_{2})\right\rVert$
$\displaystyle\leq L_{f}(\left\lVert x_{1}-x_{2}\right\rVert),\quad\forall x_{1
},x_{2}\in\mathcal{X}$
$\displaystyle\left\lVert g(y_{1})-g(y_{2})\right\rVert$
$\displaystyle\leq L_{g}(\left\lVert y_{1}-y_{2}\right\rVert),\quad\forall y_{1
},y_{2}\in\mathcal{Y}$
- 3.
Robust Loss Function: Let $L$ be a robust loss function that satisfies the following property:
$\displaystyle|L(f(x),g(y))-L(f(x),g^{\prime}(y))|\leq C(\left\lVert g(y)-g^{
\prime}(y)\right\rVert)$
where $C$ is a constant that depends on the specific choice of $L$.

Theorem: Under the above assumptions, the multimodal distance $D(f(x),g(y))=\left\lVert f(x)-g(y)\right\rVert$ is robust to noise in the text modality, satisfying:

$$ $\displaystyle|D(f(x),g^{\prime}(y))-D(f(x),g(y))|\leq L_{g}C^{\prime}p$ $$

where $C^{\prime}$ is a constant that depends on the properties of the embedding space and the set of incorrect labels.

Proof:

- 1.
By the triangle inequality:
$\displaystyle|D(f(x),g^{\prime}(y))-D(f(x),g(y))|\leq\left\lVert g^{\prime}(y)
-g(y)\right\rVert$
- 2.
Since $g$ is Lipschitz continuous and $y^{\prime}$ is chosen uniformly at random from the set of incorrect labels $\mathcal{Y}\setminus\{y\}$:
$\displaystyle\mathbb{E}[\left\lVert g(y^{\prime})-g(y)\right\rVert]$
$\displaystyle\leq L_{g}\mathbb{E}[\left\lVert y^{\prime}-y\right\rVert]$
$\displaystyle\leq L_{g}\frac{1}{|Y|-1}\sum_{y^{\prime}\in Y\setminus\{y\}}
\left\lVert y^{\prime}-y\right\rVert$
$\displaystyle\leq L_{g}C^{\prime}$
where $C^{\prime}$ depends on the properties of the embedding space and the set of incorrect labels.
- 3.
By the linearity of expectation and the noise model
$\displaystyle\mathbb{E}[\left\lVert g^{\prime}(y)-g(y)\right\rVert]$
$\displaystyle=p\mathbb{E}[\left\lVert g(y^{\prime})-g(y)\right\rVert|y^{\prime
}\neq y]+(1-p)\mathbb{E}[\left\lVert g(y)-g(y)\right\rVert|y^{\prime}=y]$
$\displaystyle\leq p\mathbb{E}[\left\lVert g(y^{\prime})-g(y)\right\rVert]$
$\displaystyle\leq L_{g}C^{\prime}p$
- 4.
Combining steps 1 and 3:
$\displaystyle|D(f(x),g^{\prime}(y))-D(f(x),g(y))|$
$\displaystyle\leq\left\lVert g^{\prime}(y)-g(y)\right\rVert$
$\displaystyle\leq\mathbb{E}[\left\lVert g^{\prime}(y)-g(y)\right\rVert]$
$\displaystyle\leq L_{g}C^{\prime}p$

Therefore, the multimodal distance is robust to noise in the text modality, with the deviation bounded by $L_{g}C^{\prime}p$.

### A.2 Proposition 4.2

Consider the case where there is a batch of two pairs of samples. $\{(x,y),(x,y^{\prime})\}$, where $(x,y)$ are labelled as positive pairs and $(x,y^{\prime})$ are negative pairs, with $\|y-y^{\prime}\|\geq\epsilon$.
The empirical CLIP loss for this objective is

$$ $\displaystyle\mathcal{L}_{CLIP}=-2\log\frac{\exp(h_{x}h^{\top}_{y})}{\exp(h_{x }h^{\top}_{y})+\exp(h_{x}{h^{\prime}}^{\top}_{y})}$ $$

Since we assume an optimal embedding that satisfies the CLIP objective to be $\hat{h}^{X}(x_{i}):=x_{i}$, $\hat{h}^{Y}(y_{j}):=x_{i}$ if $i=j$. Also this embedding model gives all embeddings with norm $1$, $\|\hat{h}^{x}_{\theta}(x)\|=\|\hat{h}^{y}_{\theta}(y)\|=1$. It can be shown that the above encoders minimize the above CLIP objective.
Recall that the noisy label is considered as a negative pair $(x,y^{\prime})$ where $y^{\prime}=y+\eta$ and $\eta\sim\mathcal{N}(0,\sigma^{2}I)$ with $\|\eta\|\geq\epsilon$.

Given the above encoders, the cosine similarity for positive pairs is given as
cos(^h^x_θ(x), ^h^y_θ(y)) = ^h^x_θ(x) ⋅^h^y_θ(y) = 1
And for the noise label
cos(^h^x_θ(x), ^h^y_θ(y’)) = ^h^x_θ(x) ⋅^h^y_θ(y’) = ^h^x_θ(x) ⋅(y + η)

Since $y^{\prime}=y+\eta$, we can find expectation $\mathbb{E}[\hat{h}^{x}_{\theta}(x)\cdot\eta]=0$ and variance $\text{Var}(\hat{h}^{x}_{\theta}(x)\cdot\eta)=\sigma^{2}\|\hat{h}^{x}_{\theta}(
x)\|^{2}=\sigma^{2}$, therefore $\mathbb{E}[\hat{h}^{x}_{\theta}(x)\cdot\hat{h}^{y}_{\theta}(y^{\prime})]=\hat{
h}^{x}_{\theta}(x)\cdot y=1$.

Bounding the Cosine Similarity:
The cosine similarity for negative pairs becomes:

$$ $\text{cos}(\hat{h}^{x}_{\theta}(x),\hat{h}^{y}_{\theta}(y^{\prime}))=\hat{h}^{ x}_{\theta}(x)\cdot(y+\eta)=1+\hat{h}^{x}_{\theta}(x)\cdot\eta$ $$

For the similarity to be less than 1:

$$ $\hat{h}^{x}_{\theta}(x)\cdot\eta<0$ $$

We want:
P [ ^h^x_θ(x) ⋅η< 0 ] = 1 - P[ ^h^x_θ(x) ⋅η≥0 ]

Using Chebyshev’s Inequality:
For $\hat{h}^{x}_{\theta}(x)\cdot\eta$:
P [ ^h^x_θ(x) ⋅η≥ϵ] ≤σ2ϵ2
Hence,
P [ (^h^x_θ(x) ⋅η< ϵ) ] ≥1 - σ2ϵ2

Using Chebyshev’s inequality, we have:
P [ cos(^h^x_θ(x), ^h^y_θ(y)) > cos(^h^x_θ(x), ^h^y_θ(y’)) ] ≥1 - σ2ϵ2

Therefore, we can show that given a pair of multimodal samples $(x,y)$ and another one with noisy label $(x,y^{\prime})$, the cosine similarity of the noisy labels will be smaller with a high probability. It is worth noticing that the existence of optimal multimodal encoders is a relatively strong assumption. Our theoretical analysis provides a clear implication for our proposed method and, in return, is empirically verified by our experimental results.

## Appendix B Data Processing

### B.1 Classification

We utilize CIFAR10N (cifar10) and CIFAR100N (cifar100) object detection [] datasets for all classification-based experiments. Each image is associated with a label indicating the primary object present in the image. These datasets contain 50,000 image-label pairs, with a clean and noisy label available per image. The noisy labels are examples of real human errors within the dataset. Further, we also generate synthetically noised labels as described in the main text. All images are resized to 224x224, center cropped, and normalized using mean and standard deviations corresponding to CLIP during the pre-processing stage.

Note that the two datasets are released under the Creative Commons Attribution-NonCommercial 4.0 license.

### B.2 Captioning

We preprocess MSCOCO [] and Flickr30k [] by using the Karpathy split [], and then selecting one random annotation from the ones available. For the MMIMDB dataset [], we utilize the plot outline as the text, and use the dataset splits provided. For MIMIC-CXR [], we use all images in the database and the provided data splits, and extract the findings and impression sections from the radiology note for the text modality. Images were normalized and transformed using the same procedure described above.

For downstream captioning, the pre-trained tokenizer and image processor corresponding to the pre-trained model (GIT []) is to pre-process image and captions.

Note that flickr30k is available under Flickr terms of use for non-commercial research and/or educational purposes(^4^44[https://shannon.cs.illinois.edu/DenotationGraph/](https://shannon.cs.illinois.edu/DenotationGraph/)). mscoco is available under Creative Commons Attribution 4.0 License. mmimdb is available for personal and non-commercial use(^5^55[https://developer.imdb.com/non-commercial-datasets/](https://developer.imdb.com/non-commercial-datasets/)). Finally, mimiccxr is available under the PhysioNet Credentialed Health Data License 1.5.0(^6^66[https://physionet.org/content/mimic-cxr/view-license/2.0.0/](https://physionet.org/content/mimic-cxr/view-license/2.0.0/)).

## Appendix C Baseline Methods

### C.1 Classification

#### Training-dependent

AUM []: This model assumes access to a classifier that can predict the class that an image likely belongs to. Then, the margin of difference between the prediction probability from the trained classifier for the assigned class and the class with the (next) highest probability is computed and averaged over training epochs. This score is thresholded to identify potential label errors.

Datamap []: Similar to AUM, this method requires access to a pretrained classifier. In this baseline, it is assumed that instances with label errors are ‘hard to learn’, and thus low confidence in prediction throughout training epochs. To produce a single score, we combine the mean and standard deviation of the probability associated with the assigned class into a single score(^7^77We experimented with different strategies, and the square root of the product of the (1-mean) and (1-standard deviation) and mean and (1-standard deviation) led to comparable, high validation F1 scores.).

Confident Learning [] is designed to identify labeling errors in classification datasets by modeling the relationship between true class labels and noisy ones. It sets thresholds for each true-noisy label pair. Using these thresholds, the model employs predicted class probabilities to rank predictions for each class, filtering out the noisy data.

#### Training-free

CLIP Logits []: CLIP is used as a zero-shot classifier to obtain the softmax-based probability for the assigned class. This value is then thresholded to identify label errors.

CLIP Similarity []: The distance (either euclidean or cosine) between image and text embeddings from CLIP are computed and thresholded.

Deep k-NN[] The proportion of $k$ nearest neighbors with the same label is computed for each image of interest. Prior works have utilized different representations for obtaining neighbors, including logits and representations from pre-trained [] vision models. We find that pre-trained representations from CLIP outperformed logits from a zero-shot CLIP classifier [].

SimiFeat [] uses nearby features to detect noisy labels under the assumption that local groups of features share clean or noisy labels. SimiFeat-V [] uses local voting and SimiFeat-R leverages ranking to detect noisy labels based on HOC estimator. The binary outputs produced are used for all score computations.

### C.2 Captioning

#### Pre-trained or Supervised

LLaVA []: We prompt LLaVA (v1.6-vicuna-13b) with the following prompt: The proposed caption for this image is "{}". Is this caption correct? Only answer with "Yes" or "No". We examine the probability distribution over the first non-special token, and find the likelihood of the token with the highest probability. If the corresponding token in lower case starts with “yes”, we return $1-$ this probability as the mislabel score. Otherwise, we return the probability.

CapFilt (oracle-like): We generate predictions using pre-trained model trained on distinguishing between high-quality MSCOCO and noisy synthetic captions []. This forms an oracle-like, fully supervised baseline.

#### Unsupervised

Datamap: We compute the cross-entropy across training epochs and compute the ratio of the mean and variance in loss across epochs. That is, we expect captioning loss for instances with label errors to be consistently high.

Confident Learning: We adapt this approach for dual-modality datasets, such as image-text pairs, by clustering text embeddings to serve as class labels for noise detection.

#### Downstream-task Unaware

Deep KNN: We cluster captions similar to confident learning, adapting classification baseline.

CLIP Similarity: This is the same setup as classification.

## Appendix D Compute Setup

We run our experiments on a shared Slurm cluster. Each experiment used one RTX A6000 with 48 GB VRAM, 10 CPU cores of Intel Xeon Ice Lake Platinum 8368, and 50 GB RAM.

## Appendix E Hyperparameters in Label Error Detection

### E.1 Classification

- 1.
AUM, Datamap: learning rate $\in\{5e-5,5e-6\}$
- 2.
Confident learning: learning rate $\in\{5e-7,5e-6,5e-5\}$, upto 30 epochs
- 3.
CLIP Zero shot: distance metric (either cosine or euclidean)
- 4.
deep k-NN: representation type, $k$, distance metric (either cosine or euclidean)

### E.2 Captioning

For most baselines requiring a class index–obtained by clustering captions–we set the number of clusters to be 100.

- 1.
LLAVA: Small amount of prompt tuning
- 2.
Confident learning: learning rate $\in\{5e-7,5e-6,5e-5\}$, upto 30 epochs, number of clusters
- 3.
CLIP Zero shot: distance metric (either cosine or euclidean)
- 4.
deep k-NN: representation type, $k\in\{1,2,5,10,15,20,30,50\}$, distance metric (either cosine or euclidean)

### E.3 Our Method

We search the following hyperparameters for our LEMoNopt:

- 1.
$k\in\{1,2,5,10,15,20,30,50\}$
- 2.
Distance metric (either cosine or euclidean)
- 3.
$\beta,\gamma,\tau_{1,n},\tau_{2,n},\tau_{1,m},\tau_{2,m}$: We take the hyperparameter set which achieves the best validation set F1 from these two strategies: (1) Using Scipy’s minimize function, with initial guess $(1,1,...,1)$, and with no explicit bounds. (2) Using a grid search with the following grid:
•
$\beta\in\{0,5,10,15,...,100\}$
•
$\gamma\in\{0,5,10,15,...,100\}$
•
$\tau_{1,n},\tau_{2,n},\tau_{1,m},\tau_{2,m}\in\{0,1,5,10\}$

## Appendix F Hyperparameters in Downstream Models

### F.1 Classification

We finetune a Vision Transformer (ViT) [] model pre-trained on ImageNet-21k [] at and fine-tuned on ImageNet 2012 []. We vary the learning rate to $3e-3,1e-3$ and $3e-4$, maximum number of $15$ epochs and report average results across the learning rates for all results in the main text.

### F.2 Captioning

The hyperparameters for the captioning model are: learning rate : $1e-5$, batch size: 16 (and 20 for mimiccxr), maximum number of epochs: 5. The model checkpoint from the epoch with lowest validation loss is used for caption generation at test time. For LoRA, we use a rank of 4 (we also experimented with ranks of 2,8, and 16 and did not see much variance in results for flickr30k). For text generation, we use top-k sampling witj $k=50$ and temperature of 0.75. We use the AdamW optimizer [], with cosine scheduling for learning rate with 1000 warmup steps.

## Appendix G Additional Experimental Results

### G.1 Label Error Detection in Classification Settings

The performance of all baselines and our method on the two types of synthetic errors are shown in Table G.1, all at a noise level of 40% (comparable to the amount of error in the noisy CIFAR datasets).

**Table G.1: Label error detection performance on synthetic errors**
|  |  |  | AUROC (%) | AUPRC (%) | F1 (%) |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  |  |  | mean | std | mean | std | mean | std |  |
| Dataset | Flip Type | Method |  |  |  |  |  |  |  |
| cifar10 | asymmetric | AUM | 93.7% | 0.6% | 86.7% | 0.7% | 88.8% | 0.9% |  |
| Confident | 96.2% | 0.8% | 91.3% | 1.6% | 95.0% | 1.0% |  |  |  |
| CLIP Logits | 98.8% | 0.2% | 97.9% | 0.3% | 94.3% | 0.4% |  |  |  |
| CLIP Sim. | 98.2% | 0.2% | 97.1% | 0.3% | 93.4% | 0.1% |  |  |  |
| Datamap | 93.7% | 0.5% | 86.5% | 0.9% | 88.3% | 1.1% |  |  |  |
| Simifeat-V | 55.7% | 0.6% | 69.1% | 0.5% | 60.6% | 0.7% |  |  |  |
| Simifeat-R | 69.3% | 0.2% | 56.0% | 0.4% | 60.8% | 0.3% |  |  |  |
| Deep k-NN | 78.7% | 1.9% | 58.4% | 2.0% | 74.9% | 2.1% |  |  |  |
| LEMoNfix | 97.5% | 0.2% | 94.8% | 0.6% | - | - |  |  |  |
|  | LEMoNopt | 98.8% | 0.2% | 97.8% | 0.5% | 94.9% | 0.3% |  |  |
| symmetric | AUM | 99.8% | 0.0% | 99.7% | 0.1% | 98.4% | 0.3% |  |  |
| Confident | 97.6% | 0.4% | 94.1% | 1.3% | 96.8% | 0.7% |  |  |  |
| CLIP Logits | 98.5% | 0.0% | 97.9% | 0.1% | 93.4% | 0.1% |  |  |  |
| CLIP Sim. | 97.9% | 0.0% | 97.1% | 0.2% | 92.5% | 0.3% |  |  |  |
| Datamap | 99.8% | 0.0% | 99.7% | 0.0% | 98.3% | 0.2% |  |  |  |
| Simifeat-V | 95.4% | 0.2% | 90.6% | 0.5% | 94.3% | 0.3% |  |  |  |
| Simifeat-R | 90.6% | 0.5% | 95.4% | 0.1% | 94.3% | 0.3% |  |  |  |
| Deep k-NN | 98.7% | 0.1% | 97.2% | 0.3% | 95.1% | 0.4% |  |  |  |
|  | LEMoNfix | 99.5% | 0.1% | 99.2% | 0.1% | - | - |  |  |
|  |  | LEMoNopt | 99.6% | 0.1% | 99.4% | 0.1% | 97.3% | 0.2% |  |
| cifar100 | asymmetric | AUM | 79.0% | 1.1% | 65.1% | 1.3% | 71.3% | 1.3% |  |
| Confident | 63.0% | 1.9% | 48.4% | 1.1% | 59.0% | 1.5% |  |  |  |
| CLIP Logits | 96.6% | 0.3% | 94.8% | 0.5% | 90.1% | 0.7% |  |  |  |
| CLIP Sim. | 94.7% | 0.5% | 92.7% | 0.7% | 87.3% | 0.4% |  |  |  |
| Datamap | 74.2% | 1.2% | 59.2% | 1.4% | 66.0% | 1.2% |  |  |  |
| Simifeat-V | 64.5% | 1.0% | 50.4% | 1.1% | 56.9% | 1.5% |  |  |  |
| Simifeat-R | 64.7% | 0.8% | 75.5% | 1.5% | 57.2% | 1.1% |  |  |  |
| Deep k-NN | 58.0% | 2.0% | 44.8% | 1.6% | 43.0% | 1.1% |  |  |  |
| LEMoNfix | 94.9% | 0.3% | 92.1% | 0.4% | - | - |  |  |  |
|  | LEMoNopt | 96.6% | 0.3% | 95.1% | 0.2% | 90.0% | 0.5% |  |  |
| symmetric | AUM | 99.1% | 0.4% | 98.8% | 0.6% | 95.6% | 1.2% |  |  |
| Confident | 88.3% | 0.9% | 75.3% | 1.7% | 85.3% | 1.2% |  |  |  |
| CLIP Logits | 96.8% | 0.1% | 95.2% | 0.3% | 90.7% | 0.4% |  |  |  |
| CLIP Sim. | 95.1% | 0.3% | 93.2% | 0.5% | 87.6% | 0.0% |  |  |  |
| Datamap | 99.1% | 0.4% | 98.7% | 0.7% | 95.9% | 1.2% |  |  |  |
| Simifeat-V | 87.3% | 0.7% | 75.3% | 1.4% | 84.4% | 0.9% |  |  |  |
| Simifeat-R | 87.4% | 0.6% | 75.5% | 1.5% | 84.6% | 0.9% |  |  |  |
| Deep k-NN | 91.4% | 0.3% | 81.1% | 1.1% | 84.9% | 0.3% |  |  |  |
|  | LEMoNfix | 98.4% | 0.1% | 97.7% | 0.2% | - | - |  |  |
|  |  | LEMoNopt | 99.0% | 0.0% | 98.7% | 0.1% | 95.1% | 0.1% |  |

### G.2 Label Error Detection in Captioning Settings

We compare the performance of our method to strong baselines on all types of synthetic noising (at 40%) on four datasets: flickr30k G.2, mscoco G.3, mmimdb G.4, and mimic-cxr G.5.
Across all datasets and noising types, we find that our model outperforms other non-oracle/supervised baselines.

**Table G.2: flickr30k: Label Error Detection**
| Dataset | Noise Type | Method | AUROC | AUPRC | F1 |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  |  |  | mean | std | mean | std | mean | std |  |
| flickr30k | noun | LLAVA | 79.3% | 0.8% | 58.5% | 0.2% | 65.0% | 1.1% |  |
|  | captfilt | 98.6% | 0.1% | 98.1% | 0.1% | 94.8% | 0.5% |  |  |
|  | Datamap | 53.4% | 2.5% | 38.3% | 1.3% | 27.1% | 1.0% |  |  |
|  | Deep kNN | 70.2% | 0.9% | 51.1% | 1.0% | 64.4% | 1.5% |  |  |
|  | Confident | 61.6% | 0.5% | 40.6% | 0.6% | 54.3% | 0.8% |  |  |
|  | CLIP Sim. | 94.8% | 0.5% | 92.8% | 0.5% | 88.1% | 0.7% |  |  |
|  | LEMoNfix | 93.6% | 0.2% | 92.0% | 0.2% | - | - |  |  |
|  | LEMoNopt | 94.5% | 0.2% | 92.8% | 0.3% | 87.7% | 0.9% |  |  |
| random | LLAVA | 81.3% | 1.0% | 65.6% | 1.4% | 72.2% | 1.1% |  |  |
|  | captfilt | 99.9% | 0.0% | 99.8% | 0.0% | 98.3% | 0.2% |  |  |
|  | Datamap | 50.0% | 1.4% | 40.5% | 1.3% | 29.6% | 0.9% |  |  |
|  | Deep kNN | 80.4% | 1.9% | 64.6% | 2.2% | 71.6% | 1.2% |  |  |
|  | Confident | 68.5% | 1.8% | 52.0% | 1.5% | 66.3% | 1.6% |  |  |
|  | CLIP Sim. | 99.5% | 0.1% | 99.3% | 0.1% | 96.4% | 0.4% |  |  |
|  |  | LEMoNfix | 99.4% | 0.2% | 99.3% | 0.2% | - | - |  |
|  |  | LEMoNopt | 99.5% | 0.2% | 99.4% | 0.3% | 96.9% | 0.8% |  |

**Table G.3: msccoco: Label Error Detection**
| Dataset | Noise Type | Method | AUROC | AUPRC | F1 |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  |  |  | mean | std | mean | std | mean | std |  |
| mscoco | cat | LLAVA | 80.3% | 0.1% | 63.4% | 0.3% | 74.9% | 0.3% |  |
|  | captfilt | 99.3% | 0.0% | 99.1% | 0.0% | 96.2% | 0.3% |  |  |
|  | Datamap | 50.4% | 0.6% | 40.5% | 0.5% | 28.6% | 0.0% |  |  |
|  | Deep kNN | 75.9% | 0.4% | 69.3% | 0.6% | 72.2% | 0.5% |  |  |
|  | Confident | 66.4% | 1.2% | 52.1% | 1.2% | 58.9% | 1.5% |  |  |
|  | CLIP Sim. | 93.8% | 0.2% | 93.0% | 0.4% | 87.5% | 0.3% |  |  |
|  | LEMoNfix | 92.0% | 0.1% | 91.8% | 0.3% | - | - |  |  |
|  | LEMoNopt | 95.6% | 0.2% | 94.6% | 0.3% | 89.3% | 0.2% |  |  |
| noun | LLAVA | 79.4% | 0.2% | 61.3% | 0.3% | 72.6% | 0.2% |  |  |
|  | captfilt | 98.7% | 0.2% | 98.4% | 0.2% | 94.9% | 0.4% |  |  |
|  | Datamap | 51.2% | 1.5% | 39.2% | 1.5% | 27.6% | 0.1% |  |  |
|  | Deep kNN | 75.3% | 1.1% | 67.3% | 0.8% | 72.0% | 0.2% |  |  |
|  | Confident | 64.6% | 1.1% | 48.4% | 1.1% | 55.6% | 1.9% |  |  |
|  | CLIP Sim. | 92.1% | 0.2% | 90.5% | 0.2% | 84.8% | 0.7% |  |  |
|  | LEMoNfix | 90.4% | 0.5% | 89.5% | 0.4% | - | - |  |  |
|  | LEMoNopt | 92.9% | 0.5% | 91.5% | 0.5% | 86.1% | 0.3% |  |  |
| random | LLAVA | 82.6% | 0.3% | 65.1% | 0.6% | 76.7% | 0.2% |  |  |
|  | captfilt | 99.9% | 0.0% | 99.9% | 0.0% | 99.1% | 0.1% |  |  |
|  | Datamap | 50.5% | 0.3% | 40.7% | 0.5% | 28.6% | 0.0% |  |  |
|  | Deep kNN | 93.3% | 0.4% | 85.1% | 0.5% | 88.1% | 0.6% |  |  |
|  | Confident | 83.5% | 1.5% | 69.4% | 2.3% | 80.2% | 1.6% |  |  |
|  |  | CLIP Sim. | 99.5% | 0.1% | 99.4% | 0.1% | 97.6% | 0.1% |  |
|  |  | LEMoNfix | 99.5% | 0.2% | 99.4% | 0.1% | - | - |  |
|  |  | LEMoNopt | 99.6% | 0.1% | 99.5% | 0.1% | 97.9% | 0.1% |  |

**Table G.4: mmimdb: Label Error Detection**
| Dataset | Noise Type | Method | AUROC | AUPRC | F1 |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  |  |  | mean | std | mean | std | mean | std |  |
| mmimdb | cat | LLAVA | 58.4% | 0.2% | 46.4% | 0.2% | 58.5% | 0.1% |  |
| captfilt | 82.7% | 0.7% | 73.3% | 1.2% | 71.6% | 0.8% |  |  |  |
| Datamap | 50.1% | 0.3% | 39.9% | 0.3% | 28.7% | 0.1% |  |  |  |
| Deep kNN | 60.0% | 0.6% | 46.3% | 0.4% | 41.9% | 3.7% |  |  |  |
| Confident | 52.8% | 0.8% | 41.4% | 0.4% | 53.6% | 0.7% |  |  |  |
| CLIP Sim. | 85.1% | 0.3% | 77.8% | 0.7% | 74.5% | 0.3% |  |  |  |
| LEMoNfix | 84.3% | 0.3% | 77.7% | 0.8% | - | - |  |  |  |
|  | LEMoNopt | 86.0% | 0.1% | 79.4% | 0.6% | 76.3% | 0.1% |  |  |
| noun | LLAVA | 59.1% | 0.3% | 44.2% | 0.6% | 55.2% | 0.2% |  |  |
| captfilt | 79.9% | 0.1% | 66.2% | 0.4% | 70.0% | 0.3% |  |  |  |
| Datamap | 49.4% | 0.5% | 36.6% | 0.7% | 28.0% | 1.6% |  |  |  |
| Deep kNN | 57.5% | 0.2% | 41.0% | 0.4% | 41.8% | 0.9% |  |  |  |
| Confident | 52.1% | 2.2% | 38.0% | 1.3% | 50.3% | 1.6% |  |  |  |
| CLIP Sim. | 82.8% | 0.4% | 72.8% | 0.5% | 72.7% | 0.4% |  |  |  |
| LEMoNfix | 82.1% | 0.4% | 72.7% | 0.6% | - | - |  |  |  |
|  | LEMoNopt | 84.4% | 0.2% | 75.9% | 1.2% | 75.2% | 0.1% |  |  |
| random | LLAVA | 58.5% | 0.8% | 46.7% | 0.5% | 58.5% | 0.1% |  |  |
| captfilt | 84.9% | 0.4% | 76.4% | 0.7% | 73.6% | 0.2% |  |  |  |
| Datamap | 50.5% | 0.2% | 40.3% | 0.3% | 28.7% | 0.2% |  |  |  |
| Deep kNN | 62.0% | 0.4% | 47.4% | 0.4% | 47.3% | 1.0% |  |  |  |
| Confident | 52.9% | 1.8% | 41.5% | 0.9% | 54.1% | 2.1% |  |  |  |
|  | CLIP Sim. | 88.1% | 0.1% | 82.0% | 0.2% | 78.2% | 0.9% |  |  |
|  | LEMoNfix | 87.6% | 0.1% | 81.9% | 0.3% | - | - |  |  |
|  |  | LEMoNopt | 89.4% | 0.3% | 84.1% | 0.8% | 80.1% | 0.4% |  |

**Table G.5: mimiccxr: Label Error Detection**
| Dataset | Noise Type | Method | AUROC | AUPRC | F1 |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  |  |  | mean | std | mean | std | mean | std |  |
| mimiccxr | cat | LLAVA | 53.9% | 0.5% | 42.7% | 0.7% | 28.7% | 0.1% |  |
| captfilt | 49.2% | 0.3% | 39.3% | 0.6% | 28.5% | 0.0% |  |  |  |
| Datamap | 49.9% | 1.2% | 39.8% | 1.2% | 28.5% | 0.0% |  |  |  |
| Deep kNN | 58.2% | 0.3% | 44.5% | 0.1% | 38.8% | 3.5% |  |  |  |
| Confident | 60.2% | 0.3% | 45.6% | 0.3% | 59.4% | 0.1% |  |  |  |
| CLIP Sim. | 64.1% | 0.4% | 51.7% | 0.5% | 48.6% | 3.4% |  |  |  |
| LEMoNfix | 66.6% | 0.2% | 54.8% | 0.4% | - | - |  |  |  |
|  | LEMoNopt | 70.4% | 2.3% | 60.3% | 2.3% | 57.0% | 1.6% |  |  |
| random | LLAVA | 50.8% | 0.4% | 40.6% | 0.2% | 57.1% | 0.0% |  |  |
| captfilt | 50.8% | 0.4% | 40.5% | 0.7% | 28.6% | 0.0% |  |  |  |
| Datamap | 50.3% | 0.9% | 39.9% | 0.5% | 28.6% | 0.0% |  |  |  |
| Deep kNN | 60.4% | 0.3% | 46.1% | 0.2% | 44.8% | 4.9% |  |  |  |
| Confident | 61.1% | 0.7% | 46.3% | 0.5% | 60.7% | 0.5% |  |  |  |
| CLIP Sim. | 66.8% | 0.8% | 54.4% | 0.9% | 54.3% | 1.0% |  |  |  |
|  | LEMoNfix | 69.5% | 0.7% | 57.8% | 1.0% | - | - |  |  |
|  |  | LEMoNopt | 73.1% | 0.9% | 63.0% | 2.0% | 63.1% | 3.6% |  |

### G.3 Varying Noise Level

We show the AUROC for varying noise levels in Figure 6.1.

Figure: (a) AUPRC on mmimdb
Refer to caption: x5.png

Figure: (e) AUROC on mmimdb
Refer to caption: x8.png

Figure: (f) AUROC on mscoco
Refer to caption: x9.png

### G.4 Robustness to Hyperparameters

Figure: (a) cifar10, asymmetric noise
Refer to caption: x10.png

Figure: (c) cifar10, real noise
Refer to caption: x12.png

Figure: (d) cifar100, asymmetric noise
Refer to caption: x13.png

Figure: (e) cifar100, symmetric noise
Refer to caption: x14.png

Figure: (f) cifar100, real noise
Refer to caption: x15.png

Figure: (g) mscoco, random noise
Refer to caption: x16.png

Figure: (h) mscoco, cat noise
Refer to caption: x17.png

Figure: (i) mscoco, noun noise
Refer to caption: x18.png

Figure: (k) flickr30k, noun noise
Refer to caption: x20.png

Figure: (l) mimiccxr, random noise
Refer to caption: x21.png

Figure: (m) mimiccxr, cat noise
Refer to caption: x22.png

Figure: (n) mmimdb, random noise
Refer to caption: x23.png

Figure: (o) mmimdb, noun noise
Refer to caption: x24.png

Figure: (p) mmimdb, cat noise
Refer to caption: x25.png

### G.5 Downstream Classification

We evaluated the downstream classification accuracy from filtering out the predictions from each method and plotted the downstream accuracy on testing set against accuracy, F1 score, precision, recall and AUROC of the predictions on training set, as seen in Figure G.3.

Figure: Figure G.3: Downstream Classification Accuracy for different methods The methods labeled with $*$ do not have raw scores and thus different threshold applied.
Refer to caption: x26.png

### G.6 Downstream Captioning

The impact of varying the amount of data filtered is shown in Figure G.4. We hypothesize that the mscoco performance is in low % data retained because the base-model is pretrained on mscoco.

Figure: (a) Filtering on flickr30k
Refer to caption: x27.png

### G.7 Ablations of our Method

Ablations of our method can be found in Table G.7.

**Table G.7: Performance of our method after ablating various components. We find that mislabel detection performance almost decreases monotonically as we remove additional components, with the exception of two metrics on mmimdb where one ablation is statistically comparable to the original method.**
|  | mmimdb | mscoco |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
|  | AUROC | AUPRC | F1 | AUROC | AUPRC | F1 |
| LEMoNopt (Ours) | 86.0 (0.1) | 79.4 (0.6) | 76.3 (0.1) | 95.5 (0.1) | 94.5 (0.3) | 89.3 (0.3) |
| $-\tau_{1}$ | 85.3 (0.3) | 78.2 (1.1) | 75.4 (0.5) | 94.6 (0.3) | 93.8 (0.4) | 88.0 (0.5) |
| $-\tau_{2}$ | 85.4 (0.6) | 77.1 (2.4) | 75.4 (0.2) | 94.7 (0.3) | 93.6 (0.5) | 87.7 (0.8) |
| $-\tau_{1},\tau_{2}$ | 85.4 (0.2) | 78.1 (0.7) | 75.2 (0.3) | 94.7 (0.3) | 93.8 (0.5) | 88.0 (0.8) |
| $-s_{n}$ | 86.1 (0.3) | 79.6 (0.5) | 76.1 (1.1) | 94.6 (0.3) | 93.6 (0.5) | 87.5 (0.6) |
| $-s_{m}$ | 85.3 (0.3) | 77.9 (0.7) | 75.5 (0.4) | 94.9 (0.2) | 94.0 (0.4) | 89.0 (0.6) |
| $-s_{n},s_{m}$ (CLIP Sim.) | 85.1 (0.3) | 77.8 (0.7) | 74.5 (0.3) | 93.8 (0.2) | 93.0 (0.4) | 87.5 (0.3) |

### G.8 Hyperparameters Used for Real-World

We show the hyperparameters used for the real-world experiment in Table G.8. We use $k=30$, cosine distance, and these hyperparameters, which originate from a hyperparameter search on synthetically noised data. We note that flickr30k has some negative hyperparameters, which we attribute to overfitting to a relatively small validation set during hyperparameter selection.

**Table G.8: Hyperparameters used for the real-world experiment. We use $k=30$, cosine distance, and the hyperparameters below, which originate from a hyperparameter search on synthetically noised data.**
|  | $\beta$ | $\gamma$ | $\tau_{1,n}$ | $\tau_{2,n}$ | $\tau_{1,m}$ | $\tau_{2,m}$ |
| --- | --- | --- | --- | --- | --- | --- |
| cifar10 | 20 | 10 | 0 | 5 | 0 | 5 |
| cifar100 | 15 | 0 | 0 | 5 | 0 | 0 |
| mscoco | 5.324 | 11.057 | 5.143 | 10.498 | 7.233 | 15.637 |
| mmimdb | 15 | 5 | 5 | 10 | 5 | 10 |
| flickr30k | 0.092 | -0.177 | -0.274 | -0.074 | -0.072 | 0.000 |
| mimiccxr | 5 | 10 | 5 | 10 | 5 | 10 |

### G.9 Examples of Detected Real Label Errors

We show additional examples of label errors in Figure G.5.

Figure: Figure G.5: Example images in each dataset identified by our method to be mislabels, and labeled as errors by a human annotator.
Refer to caption: x29.png

### G.10 Comparison with Northcutt et al., 2021 [ 50 ]

In Northcutt et al., 2021 [], the authors utilized confident learning [] to identify suspected errors in the test sets of cifar10 and cifar100. They then obtained 5 human labels for each suspected error using Amazon Mechanical Turk, and confirmed the image to be a mislabel if at least 3 of 5 workers stated so. This amounts to 54 confirmed mislabels in cifar10 (out of 221 suspected), and 585 confirmed mislabels in cifar100 (out of 1650 suspected). In this section, we compare the performance of LEMoNfix versus the CLIP similarity baseline on this set. As this set is a subset of the images identified to be mislabels by confident learning, we are not able to compare our model performance with confident learning itself. In addition, this presents a pessimistic view (lower bound) of the performance of our method, as there are many images identified by LEMoN which are mislabeled, but were not selected by confident learning in []. We demonstrate examples of these images in Figure G.6.

In Table G.9, we compare the performance of LEMoNfix with the CLIP similarity baseline on the error set from Northcutt et al., 2021 []. First, we compute the mean ranking of all error set samples as ranked by each method, out of 10,000 test-set samples. We find that our method ranks error set samples higher on average than the baseline, though the variance is large. Next, we subset to the top |CL Set| ranked samples for each method, and compute the percentage of which are actually in the error set. We note that this precision metric is upper bounded by the precision of the reference method (confident learning). Again, we find that LEMoNfix outperforms the baseline, and is able to identify more actual label errors than CLIP similarity at this threshold.

Figure: Figure G.6: Demonstrative examples of mislabeled samples in cifar10 and cifar100 which have been identified by our method in the top |CL Set|, but was not identified by confident learning in Northcutt et al., 2021 [] and thus was not a part of their error set.
Refer to caption: x30.png

**Table G.9: Comparison of LEMoNfix (Ours) with the CLIP similarity baseline on the human labeled error set from Northcutt et al., 2021 []. In this prior work, the authors used confident learning to identify |CL Set| candidate label errors in cifar10 and cifar100, |Error Set| of which are confirmed to be mislabels by Mechanical Turkers. Mean Ranking denotes the average ranking of all error set samples as ranked by each method. Precision @ Top |CL Set| involves taking the top |CL Set| samples as ranked by each method, and computing the percentage of which are in the error set. Note that each dataset’s test set consists of 10,000 samples. Numbers in parentheses represent one standard deviation.**
|  |  |  | Mean Ranking | Precision @ Top |CL Set| |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Dataset | |CL Set| | |Error Set| | LEMoNfix | CLIP Sim. | Oracle | LEMoNfix | CLIP Sim. |
| cifar10 | 275 | 54 | 1269.7 (1905.1) | 2681.0 (2507.1) | 19.64% | 6.55% | 1.45% |
| cifar100 | 2235 | 585 | 2357.5 (1981.5) | 3642.1 (2719.5) | 26.17% | 14.41% | 10.16% |

## Peer Reviews

confidence: 3
contribution: 2
initial_rating: 6
presentation: 3
questions: see the weaknesses
soundness: 3
strengths: 1. The research problem is important. The image-caption datasets are widely used to train multimodal models. Detecting the label errors in these datasets is important for downstream tasks.
2. The idea of using pre-trained multimodal models is well-motivated and the theoretical analyses are reasonable. 
3. This paper is very well organized and written in general.
summary: This paper proposes a label error detection method for multimodal datasets. Specifically, the authors first use pre-trained vision-language models to extract image-caption embeddings. Then they leverage the distance of multi-modal neighborhoods to detect the label error of image-caption datasets. This paper also provides a theoretical analysis of the feasibility of the proposed method.
weaknesses: 1. The details of the application on the unimodal dataset need to be clarified. How to define the nearest neighbors of text in unimodal datasets like CIFAR10/100？
2. Figure 3 can be improved. These lines overlap too much and are difficult to distinguish.
3. In the downstream captioning task, the improvements over CLIP similarity seem trivial. Can the authors provide some analysis of the possible reason？

---

confidence: 3
contribution: 2
initial_rating: 6
presentation: 3
questions: 1. In section 3, a few symbols are not explained, like r, D, k etc. It's better to split the section 3 into multiple sub sections to explain each term separately
2. Is there any explanation that why using pure CLIP similarity performs better than LEMoN on Flickr30k?
soundness: 3
strengths: This paper introduces a novel approach that uses multimodal nearest neighbors to assess the relevance between images and captions, providing both theoretical justification and empirical validation for the proposed method. The experiments evaluate its effectiveness in detecting label errors and its impact on downstream classification and captioning models.
summary: This paper proposed a new way to filter noisy multimodal data. Besides of using image-caption embedding similarity, the new approach leverages the multimodal neighborhood of image-captions pairs to identify label error. The method demonstrates improvements in label error detection and enhances performance on downstream captioning tasks.
weaknesses: The current experiments lack the breadth needed to fully demonstrate the impact of adding nearest neighbor terms. It would be beneficial to include a comparison using only single-side nearest neighbor term, and to present the actual values of all three terms for clearer insight.

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: 1. It took some time to understand the intuition behind Eq. 2. I think it is better to provide high-level ideas of what Eq. 2 is computing.
soundness: 3
strengths: 1. They provide a new metric to detect label noise in image-text data, which is reasonable. Also, their experiments verify that the proposed metric outperforms an existing metric in detecting errors in their settings. 
2. Their writing and presentations are clear, and mostly easy to follow. 
3. Their approach includes some hyper-parameters, but the robustness to such parameters is also investigated. 
4. They conduct a wide rage of experiments which can be insightful for readers.
summary: This paper presents an approach to detect misalignment between image-text pairs to clean image-text datasets. To achieve it, they propose a new metric that considers the distance between the image-text pairs and neighbors in image and text space. 

The high-level idea of their metric to compute label-error score is as follows. 
1. Given an image-text pair that is neighboring a target pair, if the text is far from the target text, but the image is close to the target image, the target pairs can be inconsistent. 
2. The neighboring pair used above can be also mismatched. To account for such a case, they weigh the score by using the similarity between the neighboring pair. 

Empirically, they evaluate the effectiveness of the proposed metric by image-classification dataset, an image-caption dataset such as COCO, and a medical image-report dataset. For evaluation in the image-text dataset, they randomly inject noise into the supervision of the dataset, e.g., replacing object names. Overall, their metric seems to be better than existing metrics in detecting noises, but the improvement was marginal in image-captioning.
weaknesses: I am concerned that the experiments are not so focused on noisy image-caption datasets although their motivation is to handle issues of noisy data collected from the web. 

1. Their approach seems to be mainly designed to detect label errors in image-caption datasets. However, the effectiveness of applying filtering to such a dataset seems to be marginal according to Table 4. I think label-error identification is proven to be effective by improving performance on downstream tasks. In this sense, the effectiveness of the proposed approach is not proven enough. 

2. They conduct experiments on COCO and Flickr to show the effectiveness of the image-caption dataset. Then, the effectiveness of their metric is verified only on the synthetic noise they created. However, there can be more diverse types of noise in real image-caption data collected from the web. For example, some captions might focus on a specific aspect of the image while others have details. According to the experiments, it is not clear how their metric behaves in such cases. 

3. Also, according to their appendix, their metric does not show much gain in the CC3M, which is webly collected. I actually feel that the authors had to conduct more analysis on this kind of dataset since their main motivation seems to detect errors on this kind of dataset, rather than image-classification dataset.

---

confidence: 5
contribution: 2
initial_rating: 3
presentation: 3
questions: 1. Which implementations of the baseline methods were used, and how were their hyperparameters selected?
2. How does LEMoN compare to VDC [1] in terms of strengths and weaknesses?
3. What is the inference speed of LEMoN relative to other methods? How does it scale with larger datasets, and is it feasible to apply this method to billion-scale datasets?
4. Why does LEMoN not show improvements in terms of the area under the accuracy/filter-out-rate curve?
soundness: 2
strengths: 1. The work introduces an original scoring method for label noise detection.
2. The proposed method is intuitive and clearly explained.
3. The method exhibits notable improvements across most experiments involving synthetic label noise, especially among training-free approaches.
summary: The paper presents LEMoN, a novel approach designed to identify inconsistencies in image-caption pairs, with a focus on label noise detection. The proposed method demonstrates improved detection performance in classification and captioning tasks. Additionally, the paper offers theoretical justifications to support the proposed cross-modal scoring method.
weaknesses: 1. The theoretical justifications provided in the paper contain several significant flaws, limiting their effectiveness as a core claim:
   - Theorem 4.1 appears to contain contradictory conditions, where the variable $\eta$ is defined as normal but also subject to the constraint $|\eta| > \epsilon$.
   - In Proposition A.1, Part 3 (line 885), the inequality does not hold, as the condition on $y' \ne y$ cannot be omitted. This leads to the conclusion $\mathbb{E} \le p\mathbb{E}$ with $p < 1$, implying that $\mathbb{E} = 0$.
   - There is frequent interchange between labels and embeddings, which results in incorrect conclusions. For example, in line 913, the embedding of $y'$ is replaced with label $y$ and an additive term $\eta$. However, according to Assumption 1 (line 855), both $y'$ and $y$ should be labels, not embeddings.
   - The expectation operator is lost when transitioning from the equation in line 915 to the one in line 917.
   - In the proof of Theorem 4.2 (line 954), the variance of $\frac{1}{k} E$ should be expressed as $\frac{1}{k^2} Var E$, rather than $\frac{1}{k} Var E$.
2. While the paper claims novelty in applying multi-modal scoring to label noise detection, this approach has recently been explored in [1].
3. The provided source code does not include implementations of the baseline methods. As a result, it remains unclear how the hyperparameters for these baseline methods were tuned, particularly given that the authors introduced a new set of synthetic datasets.
4. Some previous works [2] evaluate the area under the accuracy/filter-out-rate curve on real datasets, which provides a better understanding of filtering quality in real-world applications. The authors address this metric in Appendix I.12, where the results suggest that Deep k-NN may be a more effective alternative. However, these results are presented for only two datasets, limiting the generalizability of the findings.

[1] Zhu, Zihao, et al. "VDC: Versatile Data Cleanser based on Visual-Linguistic Inconsistency by Multimodal Large Language Models." ICLR 2024.

[2] Bahri, Dara, et al. "Deep k-nn for noisy labels." ICML 2020.

Meta-review: The paper proposes a mechanism for coping with label/caption noise. The reviewers praise the intuitive method and extensive experiments. However they raise concerns about effectiveness (e.g. improvement over using CLIP, results in Table 4), applicability (to real-world noise) and novelty (e.g. wrt to an ICLR 2024 paper). No reviewer score exceeds 6 and some 6-scoring reviewers voice numerous substantial concerns (even after the rebuttal).

Decision: Reject