One-step Noisy Label Mitigation

## Abstract

Abstract Mitigating the detrimental effects of noisy labels on the training process has become increasingly critical, as obtaining entirely clean or human-annotated samples for large-scale pre-training tasks is often impractical. Nonetheless, existing noise mitigation methods often encounter limitations in practical applications due to their task-specific design, model dependency, and significant computational overhead. In this work, we exploit the properties of high-dimensional orthogonality to identify a robust and effective boundary in cone space for separating clean and noisy samples. Building on this, we propose One-step Anti-Noise (OSA), a model-agnostic noisy label mitigation paradigm that employs an estimator model and a scoring function to assess the noise level of input pairs through just one-step inference, a cost-efficient process. We empirically demonstrate the superiority of OSA, highlighting its enhanced training robustness, improved task transferability, ease of deployment, and reduced computational costs across various benchmarks, models, and tasks. Our code is released at https://github.com/leolee99/OSA .

## 1 Introduction

Noise mitigation aims to handle the detriment of noisy labels encountered during the training process. The advancement of large-scale pre-training has significantly increased data scale to the trillion level. Much of this data is sourced from the internet, inevitably introducing considerable noise, which severely impedes the training process. This poses a substantial challenge for robust model training in various tasks, such as cross-modal matching [, ], image-classification [, ], and image-retrieval [].

Traditional noise mitigation approaches encounter several limitations that constrain their practical applicability: 1) Task specificity: Existing methods [, , ] are tailored to specific tasks, limiting their applicability across different tasks. 2) Model dependency: Most noise mitigation techniques [, ] are tightly coupled with specific models, requiring extensive modifications for adaptation to different models. 3) Computational cost: Numerous existing methods necessitate dual-model collaborations [, ] or multiple training passes [], i.e., they require at least two backward passes per training step, effectively doubling the computational expense and substantially increasing the training burden (see Figure. 1a).

To tackle these challenges, we use an external estimator to assess the noise level of each sample, ensuring a model-agnostic approach. This estimator adjusts the training loss by reducing the influence of noisy samples, driving their weights toward zero.
Furthermore, multimodal pre-trained models have demonstrated remarkable task transferability due to their strong semantic capabilities.
For instance, CLIP [] unifies the paradigms of image-text retrieval and image classification through a shared embedding space (see Figure. 1b). It converts category labels into sentences, maps them into the shared embedding space, and then calculates the cosine similarity with the image representation to perform image classification. Inspired by this, we leverage multimodal pre-trained models as estimators and apply the shared embedding space to enable task transfer. In this case, only one additional inference process is required for each sample, significantly reducing the computational overhead compared to performing an extra backward pass.

Nonetheless, this paradigm introduces a new challenge: how to accurately identify noise based solely on cosine similarity scores generated by estimators. An ideal solution is to find a decision boundary that separates clean samples from noise and accurately handles overlapping samples near the boundary. Existing methods [, , , ] typically attempt to build this boundary within the loss space, an isotropic space with uniform distribution, which creates only a narrow gap between noisy and clean samples. Moreover, the coarse handling of overlaps by integrating multi-model predictions often results in an unstable decision boundary. In contrast, the shared embedding space of pre-trained models is a high-dimensional, anisotropic space with an imbalanced distribution. Thus, a consideration is whether the properties of imbalanced anisotropic space can help to identify a more precise and robust decision boundary.

Figure: (a) Multiple backwards enhancing cost
Refer to caption: x1.png

In this work, we delve into the decision boundary of pre-trained models employed as estimators to accurately differentiate between clean and noisy samples. We first investigate the cosine similarity distributions of clean and noisy samples, calculated using the multimodal pre-trained models CLIP [] and ALIGN [], on two datasets with a 50% noise ratio: MS-COCO [] and SDM, as shown in Figure. 1c-1f. SDM is a dataset with the images generated by the Stable Diffusion Model (SDM) [] in some uncommon styles (see illustrations in Figure. 4). It is designed to explore how well-pre-trained models can distinguish unfamiliar domains that they rarely encounter during training. There are two interesting observations in Figure. 1c-1f: (1) The clean and noisy distributions of the same model on different datasets have a similar intersection point, suggesting the existence of a natural and stable boundary in distinguishing between clean and noisy samples. (2) The overlaps between the clean distribution and noisy distribution are minimal even in the unfamiliar domain dataset, indicating this boundary has strong potential for distinguishing between clean and noisy samples.

Building upon these two observations, we conduct an in-depth investigation and make the following contributions:

1. We figure out the origin of the intersection, attributing it to the shift in the orthogonal boundary induced by the cone effect. Furthermore, we provide a theoretical framework that proves and elaborates the stability and precision of this boundary in separating noisy and clean samples.

2. We provide a detailed explanation of the reliability of pre-trained models in general noise recognition, even in unfamiliar domains, grounded in the analysis of the pre-training process.

3. Build on this, we introduce One-Step Anti-Noise (OSA), a general model-agnostic paradigm for noise recognition that requires only one-step inference. Specifically, we utilize a pre-trained model as the estimator to maintain a shared embedding space. A scoring function, designed based on the properties of high-dimensional orthogonality, is then used to accurately handle overlaps by directly assigning a learning weight to each sample’s loss according to its cosine similarity.

4. We conduct comprehensive experiments across a variety of challenging benchmarks, models, and tasks, demonstrating the effectiveness, generalization capabilities, and efficiency of our method.

## 2 Boundary Principle Analysis

In Figure. 1c-1f, we observe a natural boundary emerging in the pre-trained model’s ability to distinguish between clean and noisy samples. In this section, we explain the principle of boundary forming from high-dimensional perspectives, and how robust it is in general noise mitigation.

### 2.1 Hypothesis: Interaction Boundary is Shifted from Orthogonal Boundary

We first elaborate on the gap extent between the positive and negative sides kept by the orthogonal boundary. Then, we present the reasoning behind the hypothesis that the intersection boundary in Figure. 1 is a shifted orthogonal boundary in the cone space.

##### The orthogonal boundary largely separates the positive and negative sides.

High-dimensional orthogonality is a general phenomenon caused by dimension disaster, where the angles between randomly selected vectors typically approximate 90 degrees, suggesting the cosine similarity that trends toward zero. For instance, in a 1024-dimensional space, the probability of two random vectors having a cosine similarity within $[-0.1,0.1]$ is approximately 99.86% (details are provided in Appendix. C.1). In this case, a natural boundary of cosine similarity zero forms, capably separating the positive side and negative side with a huge gap.

**Table 1: The mean and variance of cosine similarity between randomly generated pairs.**
| Model | Mean | Var |
| --- | --- | --- |
| CLIP | 0.215 | 0.024 |
| ALIGN | 0.087 | 6e-4 |

##### Cone effect may induce orthogonal boundary shift.

Recent literature [, , ] has demonstrated that the cone effect is a general phenomenon in deep neural networks, where the learned embedding subspace forms a narrow cone and the orthogonal boundary encounters a positive shift. Based on this, a hypothesis is that the interaction boundary in Figure. 1 is the shifted orthogonal boundary. To prove this, we simulate the process of selecting random vectors in high-dimensional space and randomly generate thousands of pairs mapped into the shared embedding space. We find that all similarity of these random vector pairs tends to a fixed value, with the low-variance cosine similarity almost lying in the middle of clean and noise distributions (see Table. 1).
An interesting phenomenon is that if we compare the mean with the intersection points in Figure. 1c-1f, we find they are almost exactly the same. This suggests that the interaction boundary is highly likely to be a shifted orthogonal boundary in cone space.

### 2.2 Theoretical verification of the Interaction Boundary Origin

Here, we theoretically investigate whether the origin of the interaction boundary is a shifted orthogonal boundary. We first show that (i) contrastive learning separates clean and noisy samples on opposite sides of the orthogonal boundary and (ii) The relative relationships of pairs’ cosine similarity stays unchanged after transmitting into the narrow cone space. Based on (i) and (ii), we can confirm that the intersection boundary at the center of the clean and noisy distributions is the shifted orthogonal boundary.

##### Contrastive learning empowers the separation of clean and noisy samples.

For an initialized model intending to learn an embedding space, both clean and noisy samples are treated as orthogonal random vectors since lacking semantic perception ability in the initial space. During contrastive training process, given $N$ sample pairs $\{(x_{i},y_{i})\}^{N}_{i=1}$, the embedding space is optimized through the cross-entropy loss (Eq. 1).

$$ $\mathcal{L}_{ce}=\frac{1}{N}\sum^{N}_{i=1}\log\frac{\exp(m_{ii})}{\sum^{N}_{j= 1}\exp(m_{ij})},$ (1) $$

where $M\in\mathbbm{R}^{N\times N}$ represents the cosine similarity matrix of $N$ sample pairs during training process. Each element $m_{ij}\in M$ denote the cosine similarity between $x_{i}$ and $y_{j}$. The diagonal elements $m_{ii}$ denote the cosine similarities of positive pairs, while the non-diagonal elements $m_{ij}$ represent the cosine similarities of negative pairs.

To minimize $\mathcal{L}_{ce}$ during training, two subprocesses occur: the diagonal elements of the matrix (i.e., clean pairs) are optimized to the positive side of the orthogonal boundary, while the non-diagonal elements (equivalent to noise pairs) are optimized to the negative side. Consequently, the distributions of these two types of samples are on opposite sides of the orthogonal boundary.

##### Relative relationship unchanged in transmitting process.

We study how the boundary shifts from the entire space to the narrow cone in the neural network. The following theorem shows that the cosine similarity will be proportionally scaled to the target narrow cone, while still maintaining a boundary with properties similar to the orthogonal boundary. In other words, vectors with cosine similarity smaller than the orthogonal boundary in the original space remain smaller than the shifted boundary in the narrow cone space, while those larger remain larger.

###### Theorem 1 (Proportional shift of boundary) .

Let $\mathbbm{R}^{d_{in}}$ be the original space before being transmitted in a neural network. Suppose $u,v\in\mathbbm{R}^{d_{in}}$ are any two random vectors with $\cos(u,v)\approx 0$. $u_{c},v_{c}\in\mathbbm{R}^{d_{in}}$ is a pair of clean vectors with $\cos(u_{c},v_{c})>0$, while $u_{n},v_{n}\in\mathbbm{R}^{d_{in}}$ is a noisy pair with $\cos(u_{n},v_{n})<0$. Given a Neural Network $F(x)={f_{t}(f_{t-1}(\dots f_{2}(f_{1}(x))))}\in\mathbbm{R}^{d_{out}}$ with $t$ layers. $f_{i}(x)=\sigma_{i}(\mathbf{W}_{i}x+\mathbf{b}_{i})$ denotes $i^{th}$ layer, where $\sigma(\cdot)$ indicates activation function. $\mathbf{W}_{i}\in\mathbbm{R}^{d^{i}_{out}\times d^{i}_{in}}$ is a random weight matrix where each element $\mathbf{W}_{i}^{k,l}\sim\mathcal{N}(0,1/d_{out}^{i})$ for $k\in\left[d^{i}_{out}\right]$, $l\in\left[d^{i}_{in}\right]$, and $\mathbf{b}_{i}\in\mathbbm{R}^{d^{i}_{out}}$ is a random bias vector such that $\mathbf{b}^{k}_{i}\sim\mathcal{N}(0,1/d^{i}_{out})$ for $k\in\left[d^{i}_{out}\right]$. Then, there always be a boundary $\beta$, satisfying:

$$ $\cos(F(u_{n}),F(v_{n}))<\cos(F(u),F(v))\approx\beta<\cos(F(u_{c}),F(v_{c})) \text{.}$ (2) $$

Theorem. 1 shows that the relative relationship of pairs in the original entire space, will not change after transmitting to the narrow cone space of the trained model, and there is always a boundary $\beta$ concentrated on most random vectors. In Appendix. C.2, we provide a detailed statement and proof of the Theorem.

###### Theorem 1 (Proportional shift of boundary) .

### 2.3 Qualitative analysis of robustness and applicability

Next, we perform a qualitative analysis to explore (i) the robustness and generality of the boundary in distinguishing between clean and noisy samples, and (ii) how the boundary’s properties can be leveraged to achieve more reasonable and precise overlap handling.

##### How about the boundary robustness even in unfamiliar domains?

Although the boundary’s ability to distinguish clean and noisy samples is proven, its robustness and generality still require further exploration. For practical pre-training, it must maintain accuracy and robustness even in unfamiliar domain datasets. Since the capabilities of the pre-trained model are difficult to quantify, we conduct a qualitative analysis from the perspective of pre-trained model inference. The models pre-trained on millions of samples already possess somewhat semantic understanding capabilities. Given a positive pair from an unseen domain, due to the contrastive learning process during pre-training, it still has a strong likelihood of moving toward the positive side of the boundary, while the negative pair tends toward the negative side. Although the cosine similarity difference might be slight, as we have shown in Section. 2.1, the boundary constructs a significant gap from the perspective of high-dimensional orthogonality.

##### How to handle the overlaps through imbalanced probability?

Due to the properties of orthogonal boundary, as cosine similarity decreases and approaches zero from the positive side, the probability of positive samples sharply decreases. Therefore, we can design a scoring function to annotate the cleanliness of samples. This function should satisfy two requirements: for samples with cosine similarity less than or equal to zero, which are almost certainly noise, the function should assign them a weight of zero. For samples with cosine similarity greater than zero, the function gradient should increase rapidly as the cosine similarity moves further from zero.

## 3 Method

In this section, we present our One-Step Anti-Noise (OSA) paradigm with a workflow shown in Figure. 2. We first define the pair-based noise mitigation tasks for image-text matching, image classification, and image retrieval tasks in Sec. 3.1. Consequently, the detailed description of OSA is clarified in Sec. 3.2.

### 3.1 Task Definition

Let $\mathcal{D}=\{(x_{i},y_{i},c_{i})\}_{i=1}^{N}$ denote a paired dataset, where $(x_{i},y_{i})$ represents the $i$-th pair in the dataset, and $c_{i}$ indicates a noise label for that pair. Specifically, when $c_{i}=0$, $(x_{i},y_{i})$ forms a correct (paired) match, while $c_{i}=1$ denotes an incorrect (unpaired) match. The objective of noise mitigation in contrastive learning is to construct a shared embedding space that brings $x_{i}$ and $y_{i}$ closer when $c_{i}=1$. In different tasks, $x_{i}$ and $y_{i}$ are distinct data types. For instance, in the image-text retrieval task, $x_{i}$ and $y_{i}$ represent images and texts, respectively. In the image classification task, $x_{i}$ and $y_{i}$ represent images and categories, respectively. In the image retrieval task, $x_{i}$ and $y_{i}$ represent images and relevant images, respectively. The paired sample $(x,y)$ could be encoded into a shared embedding space by corresponding encoders $\phi_{x}(\cdot)$ and $\phi_{y}(\cdot)$. Afterward, the cosine similarity $s(x,y)$ is calculated through Eq. 3 as semantic relevance of $(x,y)$ to guide the training.

$$ $s(x,y)=\frac{\phi_{x}(x)}{\left\|\phi_{x}(x)\right\|}\cdot\frac{\phi_{y}(y)}{ \left\|\phi_{y}(y)\right\|}\text{.}$ (3) $$

### 3.2 One-step Anti-Noise

The workflow of our noise mitigation approach OSA is depicted in Figure. 2. Initially, we utilize an estimator model to encode the input pair to a shared embedding space and continue to compute the cosine similarity between the paired embedding. Afterward, the cosine similarity is converted to a cleanliness score $w_{i},(0\leq w_{i}\leq 1)$ through a scoring function designed based on orthogonal properties (Section. 2.3). This score quantifies the clean degree of the sample, the smaller $w_{i}$ is, the noisier the sample.

During the target model training phase, this cleanliness score is used as a weight, directly multiplied by the loss of the corresponding sample to facilitate selective learning. This noise mitigation process, being solely dependent on the estimator model, is readily adaptable to the training of various target models by simply adding an extra coefficient to the loss function, ensuring the model-agnostic property. Therefore, the key of our noise mitigation approach revolves around the estimator model and noise score assessment.

Figure: Figure 2: The workflow of OSA. In the anti-noise process, there are two phases: Scoring Phase and Training Phase. In the Scoring Phase, a pair is mapped to a shared embedding space by estimators. Then the cosine similarity is transformed to a weight $w$ by a scoring function. In the Training Phase, the weight $w$ is directly multiplied with the loss to instruct the optimization.
Refer to caption: x7.png

#### 3.2.1 Estimator Model

##### Estimator model selection.

In our approach, the Estimator Model must satisfy two critical requirements: 1) effectively mapping input pairs into a unified embedding space and 2) possessing basic semantic understanding capabilities. To meet these requirements, we employ CLIP [], a commonly used multimodal pre-trained models, as our estimator model. It is equipped with a text encoder $\phi_{t}(\cdot)$ and an image encoder $\phi_{v}(\cdot)$, enabling it to perform basic zero-shot tasks efficiently.

##### Domain adaptation (Optional).

While we have performed a qualitative analysis of the zero-shot pre-trained model’s robustness on out-of-domain data in Section. 2.3, and shown strong robustness for edge cases in Figure. 1, considering the domain diversity in real-world scenarios, we provide an optional Domain Adaptation (DA) approach to enhance the estimator model’s adaptability when encountering edge domains. Following NPC [], we first employ a Gaussian Mixture Model (GMM) coupled with strict selection thresholds to ensure the absolute cleanliness of the chosen samples. We afterward implement a warm-up phase with few steps, allowing the estimator model to better understand the semantics of the target domain. Notably, this trick is only optional for our methods. Through multiple experiments, we found that even without domain adaptation, the zero-shot CLIP model performs exceptionally well across various scenarios.

#### 3.2.2 Noise Score Assessment

##### Spatial Debiasing.

The cone effect phenomenon has been demonstrated as a general phenomenon for deep neural networks, typically resulting in a narrow embedding space that causes a shift of space center to a narrow cone center []. Specifically, when paired randomly generated inputs are mapped into a shared embedding space through model encoders, the resultant vectors exhibit an average cosine similarity that deviates from zero and tends to another fixed angle. To counteract this shift and mitigate its impact on the estimator’s ability to accurately recognize noises through high-dimensional orthogonality, a random sampling method is developed. We begin by constructing $K$ random sample pairs $\mathcal{R}=\{(x_{j},y_{j})\mid j=1,2,\dots,K\}$ and processing them through the estimator’s encoder to generate a set of vectors. Then the average cosine similarity among these vectors will be calculated as the space shift $\beta$ through:

$$ $\beta=\frac{\sum^{K}_{j=1}s(x_{j},y_{j})}{K}.$ (4) $$

##### Scoring Function.

After spatial debiasing, we employ a scoring function $w(\cdot)$ to evaluate the cleanliness of the input pair $(x,y)$. In section. 2.3, we have elaborate how to handle overlaps based on the orthogonal boundary property. For an estimator model trained on millions of samples using contrastive learning, clean pairs (diagonal elements) are optimized to positive side, while noise pairs (non-diagonal elements) are optimized to negative side. Given unfamiliar pairs, the model also tends to map clean pairs towards positive and noisy pairs towards negative. Despite the potentially slight similarity difference between clean and noisy pairs, high-dimensional orthogonality ensures a substantial gap between them.
In this case, a negative cosine similarity $s(x,y)$ computed by the estimator, indicating the pair is almost certainly noise, should be assigned a score of zero. For samples with $s(x,y)$ greater than zero, the probability of the sample being positive sharply decreases as the cosine similarity approaches zero from the positive side. Therefore, the function gradient should increase rapidly as the cosine similarity moves further from zero. To systematically score the noise, we design the scoring function as:

$$ $w(x,y,\beta)=\left\{\begin{aligned} &0&,&s(x,y)-\beta\leq 0\\ &-(s(x,y)-\beta)^{2}(s(x,y)-\beta-1)&,&\textit{otherwise}\hfill\\ \end{aligned}\right.$ (5) $$

##### Re-weight Training.

After scoring, the target model can selectively learn from the samples by re-weighting the loss. Noise samples with smaller weights will have a reduced impact on model updates and will be effectively mitigated. For a sample $(x,y)$, let $\mathcal{L}_{x,y}$ denote its loss, the re-computed loss $\mathcal{L}_{re}$ is defined as:

$$ $\mathcal{L}_{re}=w(x,y,\beta)\times\mathcal{L}_{x,y}\text{.}$ (6) $$

## 4 Experiments

In this section, we present experiments on multiple datasets with label noise, demonstrating the effectiveness of our methods. Firstly, we describe the datasets, metrics, and implementation details.
Then, we report our results on several downstream tasks.
Lastly, we conduct ablation studies to show how each part of our method contributes and examine how these parts interact. The literature involved in our experiments and richer related work are detailed in Appendix. B.

### 4.1 Evaluation Setting

In this section, we briefly introduce the datasets and evaluation metrics used in the experiments. For more dataset and implementation details, please refer to Appendix. A.

##### Datasets.

We evaluate our method on three downstream tasks with noisy labels, including one multimodal task and two visual tasks. For the cross-modal matching task, we perform experiments on the MSCOCO [] and Flickr30K [] datasets. Following NPC [], we further carry out evaluations on a real-world noisy dataset CC120K. For image classification tasks, experiments are conducted under three subsets of WebFG-496 []—Aircraft, Bird, and Car.
For image retrieval tasks, we conduct experiments on the CARS98N dataset under PRISM [] setting.

##### Evaluation Metrics.

For the image-text matching task, the recall value of the top-K retrieved results (R@K) is used. For classification tasks, accuracy serves as the evaluation metric. For the image retrieval task, we use Precision@1 and mAP@R for evaluation.

### 4.2 Comparisons with State of The Arts

**Table 2: Comparison on noisy MS-COCO.**
| Noise ratio | Method | MS-COCO 1K | MS-COCO 5K |  |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| i2t | t2i | i2t | t2i |  |  |  |  |  |  |  |  |  |  |
| R@1 | R@5 | R@10 | R@1 | R@5 | R@10 | R@1 | R@5 | R@10 | R@1 | R@5 | R@10 |  |  |
| 0% | VSE$\infty$ | 82.0 | 97.2 | 98.9 | 69.0 | 92.6 | 96.8 | 62.3 | 87.1 | 93.3 | 48.2 | 76.7 | 85.5 |
| PCME++ | 81.6 | 97.2 | 99.0 | 69.2 | 92.8 | 97.1 | 62.1 | 86.8 | 93.3 | 48.1 | 76.7 | 85.5 |  |
| PAU | 80.4 | 96.2 | 98.5 | 67.7 | 91.8 | 96.6 | 63.6 | 85.2 | 92.2 | 46.8 | 74.4 | 83.7 |  |
| NPC | 82.2 | 96.5 | 98.7 | 68.3 | 92.0 | 98.7 | 65.4 | 87.3 | 93.1 | 48.5 | 75.4 | 84.4 |  |
| CLIP | 80.1 | 95.7 | 98.2 | 67.1 | 91.4 | 96.6 | 62.9 | 84.9 | 91.6 | 46.5 | 73.8 | 82.9 |  |
| +OSA | 82.2 | 96.5 | 98.7 | 68.8 | 92.1 | 96.7 | 65.6 | 86.8 | 92.9 | 49.1 | 76.2 | 84.8 |  |
| ALIGN | 84.9 | 97.3 | 99.0 | 70.5 | 92.8 | 97.2 | 69.6 | 89.9 | 94.5 | 50.5 | 77.5 | 85.7 |  |
|  | +OSA | 85.3 | 97.4 | 99.0 | 71.4 | 93.1 | 97.3 | 69.8 | 89.9 | 94.8 | 51.4 | 78.2 | 86.3 |
| 20% | VSE$\infty$ | 78.4 | 94.3 | 97.0 | 65.5 | 89.3 | 94.1 | 58.6 | 83.4 | 89.9 | 45.0 | 72.9 | 81.7 |
| PCME++ | 78.4 | 95.9 | 98.4 | 64.9 | 90.8 | 96.1 | 57.7 | 83.9 | 91.0 | 43.2 | 72.3 | 82.4 |  |
| PAU | 78.2 | 95.2 | 98.1 | 64.5 | 90.0 | 95.4 | 59.3 | 82.9 | 90.4 | 44.2 | 71.3 | 81.3 |  |
| NPC | 79.9 | 95.9 | 98.4 | 66.3 | 90.8 | 98.4 | 61.6 | 85.4 | 91.6 | 46.0 | 73.4 | 82.9 |  |
| CLIP | 76.0 | 94.3 | 97.5 | 63.4 | 89.0 | 94.8 | 55.3 | 79.1 | 86.9 | 41.0 | 68.8 | 79.3 |  |
| +OSA | 81.6 | 96.2 | 98.5 | 68.9 | 92.0 | 96.6 | 65.8 | 86.4 | 92.5 | 48.7 | 76.1 | 84.5 |  |
| ALIGN | 79.4 | 95.7 | 98.2 | 66.2 | 90.8 | 96.1 | 60.9 | 84.5 | 91.0 | 46.3 | 73.6 | 82.3 |  |
|  | +OSA | 85.1 | 97.4 | 99.1 | 70.9 | 93.0 | 97.3 | 69.7 | 90.0 | 94.7 | 50.9 | 77.8 | 86.2 |
| 50% | VSE$\infty$ | 44.3 | 76.1 | 86.9 | 34.0 | 69.2 | 84.5 | 22.4 | 48.2 | 61.1 | 15.8 | 38.8 | 52.1 |
| PCME++ | 74.8 | 94.3 | 97.7 | 60.4 | 88.7 | 95.0 | 52.5 | 79.6 | 88.4 | 38.6 | 68.0 | 79.0 |  |
| PAU | 76.4 | 94.1 | 97.6 | 62.3 | 88.5 | 94.6 | 57.3 | 81.5 | 88.8 | 41.9 | 69.4 | 79.6 |  |
| NPC | 78.2 | 94.4 | 97.7 | 63.1 | 89.0 | 97.7 | 59.9 | 82.9 | 89.7 | 43.0 | 70.2 | 80.0 |  |
| CLIP | 73.9 | 93.0 | 97.2 | 60.1 | 87.3 | 94.0 | 54.1 | 78.5 | 86.6 | 39.7 | 67.2 | 77.5 |  |
| +OSA | 80.4 | 96.2 | 98.6 | 67.8 | 91.6 | 96.4 | 64.0 | 85.5 | 91.9 | 47.9 | 74.6 | 83.8 |  |
| ALIGN | 78.0 | 95.8 | 98.5 | 65.4 | 90.3 | 96.0 | 60.1 | 84.3 | 91.2 | 45.2 | 72.8 | 82.1 |  |
|  | +OSA | 84.3 | 97.0 | 98.9 | 70.0 | 92.5 | 97.0 | 68.5 | 89.2 | 94.2 | 50.0 | 77.0 | 85.4 |

##### Results on MSCOCO.

To fairly demonstrate the effectiveness of our method, we compare OSA with various robust learning image-text matching approaches using the same ViT-B/32 CLIP as backbone, including VSE$\infty$ [], PCME++ [], PAU [], NPC []. Besides, we separately employ OSA on both CLIP [] and ALIGN []. The results in Table. 2 show that OSA outperforms all previous approaches on all metrics with a huge gap. In the more challenging MS-COCO 5K set with 50% noise ratio, OSA surpasses the SOTA method NPC in the R@1 for both image-to-text (i2t) and text-to-image (t2i) matching by 8.6% and 7.0%, respectively. Another phenomenon is that as the noise ratio increases from 0% to 50%, all other methods encounter severe performance drop, with an averaging drop of 5.05% for NPC across four R@1 metrics. In contrast, OSA exhibits only a slight decrease of 1.275%, showcasing the accuracy and robustness of OSA in anti-noise tasks.

**Table 3: Comparison on noisy Flickr30K.**
| Method | Noise ratio | i2t | t2i | Noise ratio | i2t | t2i |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| R@1 | R@5 | R@10 | R@1 | R@5 | R@10 | R@1 | R@5 | R@10 | R@1 | R@5 | R@10 |  |  |  |
| NCR | 0% | 77.3 | 94.0 | 97.5 | 59.6 | 84.4 | 89.9 | 20% | 73.5 | 93.2 | 96.6 | 56.9 | 82.4 | 88.5 |
| DECL | 79.8 | 94.9 | 97.4 | 59.5 | 83.9 | 89.5 | 77.5 | 93.8 | 97.0 | 56.1 | 81.8 | 88.5 |  |  |
| BiCro | 81.7 | 95.3 | 98.4 | 61.6 | 85.6 | 90.8 | 78.1 | 94.4 | 97.5 | 60.4 | 84.4 | 89.9 |  |  |
| NPC | 87.9 | 98.1 | 99.4 | 75.0 | 93.7 | 97.2 | 87.3 | 97.5 | 98.8 | 72.9 | 92.1 | 95.8 |  |  |
| CLIP | 86.2 | 97.6 | 99.2 | 72.9 | 92.3 | 96.0 | 82.3 | 95.5 | 98.3 | 66.0 | 88.5 | 93.5 |  |  |
| +OSA | 88.6 | 97.7 | 99.3 | 75.6 | 93.6 | 96.8 | 88.9 | 97.7 | 99.1 | 75.6 | 93.3 | 96.9 |  |  |
| NCR | 40% | 68.1 | 89.6 | 94.8 | 51.4 | 78.4 | 84.8 | 60% | 13.9 | 37.7 | 50.5 | 11.0 | 30.1 | 41.4 |
| DECL | 72.7 | 92.3 | 95.4 | 53.4 | 79.4 | 86.4 | 65.2 | 88.4 | 94.0 | 46.8 | 74.0 | 82.2 |  |  |
| BiCro | 74.6 | 92.7 | 96.2 | 55.5 | 81.1 | 87.4 | 67.6 | 90.8 | 94.4 | 51.2 | 77.6 | 84.7 |  |  |
| NPC | 85.6 | 97.5 | 98.4 | 71.3 | 91.3 | 95.3 | 83.0 | 95.9 | 98.6 | 68.1 | 89.6 | 94.2 |  |  |
| CLIP | 76.2 | 93.3 | 96.5 | 59.4 | 85.0 | 90.9 | 66.3 | 87.3 | 93.0 | 52.1 | 78.8 | 87.4 |  |  |
| +OSA | 87.3 | 97.6 | 99.3 | 74.2 | 93.1 | 96.7 | 87.2 | 98.1 | 99.6 | 74.4 | 92.9 | 96.4 |  |  |

##### Results on Flickr30K.

To further demonstrate the generalization ability of OSA, we evaluate on the Flickr30K dataset and compare with several anti-noise methods, including NCR [], DECL [], BiCro [], and NPC []. The results are presented in Table. 3 of Appendix. It is evident that OSA consistently outperforms all models on the R@1 metric. Notably, compared with the baseline CLIP, training with OSA at a 60% noise ratio achieves 20.9% R@1 improvement for i2t and a 22.3% R@1 improvement in t2i, further indicating the effectiveness of OSA on noise mitigation. Additionally, OSA demonstrates similar noise robustness on the Flickr30K dataset as observed on MSCOCO, with only 1.4% R@1 drop on i2t and 1.2% R@1 drop on t2i ranging from 0% noise to 60% noise, while all of the other anti-noise approaches hardly resist the detriment from high-ratio noise. All of these results demonstrate the effectiveness and robustness of OSA on anti-noise tasks.

##### Results on CC120K.

To further verify the reliability of OSA in real scenarios, we conduct evaluations on a large-scale real-world noisy dataset, CC120K, with 3%-20% noise ratio. The results shown in Table. 5 indicate that OSA outperforms the current state-of-the-art method NPC, even in larger-scale real-world domains. This demonstrates the feasibility and generality of OSA even in practical training scenarios.

**Table 4: Comparison on real-world noisy dataset CC120K.**
|  | i2t | t2i |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
| Method | R@1 | R@5 | R@10 | R@1 | R@5 | R@10 |
| NPC | 71.1 | 92.0 | 96.2 | 73.0 | 90.5 | 94.8 |
| CLIP | 68.8 | 87.0 | 92.9 | 67.8 | 86.4 | 90.9 |
| +OSA | 73.1 | 92.2 | 95.7 | 73.9 | 91.2 | 94.7 |

##### Results on Other Downstream Tasks.

To validate the transferability of OSA across different tasks, we evaluate it on two additional tasks: image classification and image retrieval. The results are presented in Table. 5. The baseline method for both tasks leverages contrastive learning. In the image classification task, OSA outperforms the baseline by 7.74%, 8.21%, and 4.28% on the Aircraft, Bird, and Car subsets, respectively. In the image retrieval task, OSA improves performance by 6.76% in precision and 6.83% in mAP. These improvements demonstrate the strong task transferability and generality of OSA.

### 4.3 Target Model-Agnostic Analysis

OSA is an architecture-agnostic paradigm that can be easily adapted to various models. To verify its model-agnostic property, we evaluate it across models with different architectures. Subsequently, we apply it to other anti-noise models to demonstrate its generalization capability in noise mitigation.

##### Architecture-agnostic Analysis.

The effectiveness of OSA on Vision Transformer (ViT) has been proven in Section. 4.2. We further explore the generality of OSA on target models with other architectures. Specifically, we deploy OSA above the VSE++ [] model with two different architecture types: ResNet-152 [] and VGG-19 []. These two architectures have revealed significant sensitivity and vulnerability to noise []. In this experiment, all estimator models employ zero-shot CLIP and we utilize the original VSE++ as our baseline. The results in Table. 6 indicate a significant performance degradation emerged for the baseline methods in noisy setting, while a stable performance is achieved after employing OSA. The stable performance on these two noise-vulnerable architectures fully demonstrates that OSA possesses the architecture-agnostic property.

**Table 6: The results of the target model with different architectures on noisy MSCOCO.**
| Noise ratio | Method | Architecture | MS-COCO 1K | MS-COCO 5K |  |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| i2t | t2i | i2t | t2i |  |  |  |  |  |  |  |  |  |  |  |
| R@1 | R@5 | R@10 | R@1 | R@5 | R@10 | R@1 | R@5 | R@10 | R@1 | R@5 | R@10 |  |  |  |
| 0% | Baseline | ResNet-152 | 58.9 | 86.9 | 93.8 | 44.2 | 77.9 | 88.3 | 34.9 | 64.3 | 76.1 | 23.3 | 50.9 | 64.2 |
| +OSA | 58.9 | 86.2 | 93.7 | 44.3 | 77.9 | 87.9 | 35.0 | 64.1 | 76.0 | 23.5 | 50.8 | 63.9 |  |  |
| Baseline | VGG-19 | 49.6 | 79.4 | 89.1 | 38.0 | 72.9 | 84.7 | 26.9 | 54.2 | 66.8 | 18.7 | 43.8 | 56.8 |  |
| +OSA | 50.1 | 80.0 | 89.3 | 38.3 | 73.0 | 84.6 | 26.6 | 54.4 | 67.4 | 18.8 | 43.9 | 57.3 |  |  |
| 20% | Baseline | ResNet-152 | 45.8 | 70.3 | 83.7 | 36.1 | 68.4 | 79.7 | 26.0 | 48.4 | 58.3 | 18.3 | 42.0 | 54.0 |
| +OSA | 58.1 | 86.1 | 93.2 | 43.4 | 76.8 | 87.2 | 33.7 | 62.6 | 74.5 | 22.5 | 49.7 | 62.8 |  |  |
| Baseline | VGG-19 | 33.2 | 67.1 | 81.5 | 25.9 | 58.0 | 71.4 | 13.7 | 35.0 | 49.2 | 10.7 | 29.9 | 41.9 |  |
| +OSA | 49.3 | 79.1 | 88.6 | 37.2 | 71.9 | 83.8 | 25.2 | 53.3 | 65.3 | 17.9 | 42.6 | 55.9 |  |  |
| 50% | Baseline | ResNet-152 | 28.4 | 61.2 | 75.2 | 5.2 | 14.0 | 19.5 | 11.0 | 31.0 | 43.6 | 1.6 | 6.0 | 9.2 |
| +OSA | 55.0 | 84.0 | 92.0 | 40.7 | 74.7 | 85.6 | 30.8 | 60.2 | 72.3 | 20.9 | 46.6 | 60.0 |  |  |
| Baseline | VGG-19 | 2.5 | 9.8 | 16.2 | 0.1 | 0.5 | 1.0 | 0.5 | 2.5 | 4.4 | 0.0 | 0.1 | 0.2 |  |
| +OSA | 47.1 | 77.7 | 87.6 | 35.7 | 70.3 | 82.8 | 24.0 | 51.5 | 64.0 | 16.9 | 40.8 | 54.2 |  |  |

**Table 7: The results of other methods employing OSA on MSCOCO 1K.**
|  |  | i2t | t2i |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Noise Ratio | Method | R@1 | R@5 | R@10 | R@1 | R@5 | R@10 |
| 0% | NPC | 82.2 | 96.5 | 98.7 | 68.3 | 92.0 | 98.7 |
| +OSA | 82.4 | 96.4 | 98.6 | 68.5 | 91.8 | 98.7 |  |
| 20% | NPC | 79.9 | 95.9 | 98.4 | 66.3 | 90.5 | 98.4 |
| +OSA | 81.2 | 96.0 | 98.6 | 66.9 | 91.2 | 98.6 |  |
| 50% | NPC | 78.2 | 94.4 | 97.7 | 63.1 | 89.0 | 97.7 |
| +OSA | 79.3 | 95.6 | 98.2 | 66.8 | 90.8 | 98.2 |  |

##### Adaptability to Other Anti-Noise Models.

Theoretically, OSA can be adapted to any target model, providing noise resistance. However, can OSA further enhance the robustness of models specifically designed for noise mitigation? To investigate this, we applied OSA to the current state-of-the-art model, NPC []. As shown in Table. 7 of Appendix, even for noise-mitigating models, OSA consistently improves training robustness. This finding further demonstrates the broad adaptability of OSA across different model types.

### 4.4 Estimator Model Analysis.

The estimator model is the basis of OSA’s anti-noise capability. In this section, we explore the impact of different estimator models on noise mitigation, and examine the impact of domain adaptation in noise mitigation. In Table. 8 of Appendix, we investigate four types of estimators: “None” refers to training CLIP directly without using OSA. “CLIP (w/o DA)” and “ALIGN (w/o DA)” represent using CLIP and ALIGN without domain adaptation as estimators, respectively, i.e., zero-shot CLIP and ALIGN. “CLIP (w DA)” indicates the CLIP with domain adaptation. The target models are all CLIP. We can observe that both of CLIP and ALIGN as estimators significantly enhance the target model performance stability when learning with noise, indicating that the choice of estimator is very flexible. Both CLIP and ALIGN demonstrate exceptional performance when served as estimators. The other phenomenon is that the zero-shot CLIP model shows comparable performance to the domain-adapted CLIP with a even better performance at lower noise ratios. This indicates that zero-shot CLIP, as an estimator, already performs exceptionally well in noise mitigation. The domain adaptation is unnecessary. This further enhances the deployment convenience of OSA.

**Table 8: Ablation study of estimator type on noisy MS-COCO.**
| Noise ratio | Estimator | MS-COCO 1K | MS-COCO 5K |  |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| i2t | t2i | i2t | t2i |  |  |  |  |  |  |  |  |  |  |
| R@1 | R@5 | R@10 | R@1 | R@5 | R@10 | R@1 | R@5 | R@10 | R@1 | R@5 | R@10 |  |  |
| 0% | None | 80.1 | 95.7 | 98.2 | 67.1 | 91.4 | 96.6 | 62.9 | 84.9 | 91.6 | 46.5 | 73.8 | 82.9 |
| CLIP (w/o DA) | 82.6 | 96.7 | 98.7 | 68.5 | 92.1 | 96.7 | 66.2 | 87.0 | 93.3 | 48.6 | 75.7 | 84.8 |  |
| ALIGN (w/o DA) | 81.9 | 96.7 | 98.7 | 68.9 | 92.2 | 96.9 | 64.8 | 86.6 | 92.7 | 49.0 | 75.9 | 84.7 |  |
| CLIP (w DA) | 82.2 | 96.5 | 98.7 | 68.8 | 92.1 | 96.7 | 65.6 | 86.8 | 92.9 | 49.1 | 76.2 | 84.8 |  |
| 20% | None | 76.0 | 94.3 | 97.5 | 63.4 | 89.0 | 94.8 | 55.3 | 79.1 | 86.9 | 41.0 | 68.8 | 79.3 |
| CLIP (w/o DA) | 81.8 | 96.1 | 98.7 | 68.2 | 91.9 | 96.5 | 64.8 | 86.6 | 92.3 | 48.3 | 75.4 | 84.1 |  |
| ALIGN (w/o DA) | 81.2 | 96.0 | 98.6 | 67.7 | 91.5 | 96.4 | 64.8 | 86.2 | 92.3 | 47.8 | 74.9 | 83.9 |  |
| CLIP (w DA) | 81.6 | 96.2 | 98.5 | 68.9 | 92.0 | 96.6 | 65.8 | 86.4 | 92.5 | 48.7 | 76.1 | 84.5 |  |
| 50% | None | 73.9 | 93.0 | 97.2 | 60.1 | 87.3 | 94.0 | 54.1 | 78.5 | 86.6 | 39.7 | 67.2 | 77.5 |
| CLIP (w/o DA) | 79.6 | 95.6 | 98.4 | 65.9 | 90.8 | 95.9 | 62.4 | 84.8 | 90.8 | 45.7 | 73.1 | 82.5 |  |
| ALIGN (w/o DA) | 80.4 | 95.6 | 98.3 | 66.0 | 90.5 | 95.8 | 62.0 | 84.9 | 91.8 | 45.7 | 73.2 | 82.5 |  |
| CLIP (w DA) | 80.4 | 96.2 | 98.6 | 67.8 | 91.6 | 96.4 | 64.0 | 85.5 | 91.9 | 47.9 | 74.6 | 83.8 |  |

### 4.5 Noise Assessment Accuracy

##### Noise Detection Accuracy Analysis.

To figure out how accurate OSA is in recognizing noise, we evaluate the accuracy and recall on CLIP without Domain-Adaptation (w/o DA) and CLIP with Domain-Adaptation (w DA) on noisy MSCOCO. We utilize zero as the threshold to roughly divide pairs into noise and clean sets, respectively. Concretely, we classify scores less than or equal to 0 as noise, and scores greater than 0 as clean. The Accuracy means the proportion of the clean pairs correctly classified into the clean set, while the Recall indicates the noisy pairs correctly classified into the noisy set. The results presented in Table. 11 indicates the powerful noise recognizing capability of OSA. The remarkable performance on CLIP (w/o DA) fully demonstrates the generality of OSA.
Another notable phenomenon is that all recall scores converge towards 100, indicating that OSA achieves greater accuracy in noise detection. This suggests that OSA can almost entirely eliminate the impact of noise on training.

**Table 9: Mean Noise Rank Comparison between OSA and NPC.**
| Noise Ratio | Method | Mean Noise Rank$\uparrow$ | Optimal Rank | Noise Number | Sample Number |
| --- | --- | --- | --- | --- | --- |
| 20% | NPC | 1641.3 | 1815.5 | 370 | 2,000 |
| OSA | 1809.1 | 1815.5 | 370 | 2,000 |  |
| 50% | NPC | 1456.2 | 1524.0 | 953 | 2,000 |
| OSA | 1520.7 | 1524.0 | 953 | 2,000 |  |

##### Noise Re-weighting Accuracy Comparison.

Some anti-noise methods, like NPC, also employ loss re-weighting for optimization. To assess whether our method assigns relatively smaller weights to noise than these methods, we first analyze the weights generated by NPC and OSA. Due to differences in weight scales across methods, a direct comparison is unfair. Therefore, to unify the scale, we adopt a ranking-based approach, sorting weights in descending order and calculating the Mean Noise Rank. This metric evaluates whether smaller weights are consistently assigned to noisy samples relative to clean ones. Our experiments use 2,000 randomly selected samples from the MSCOCO dataset under two noise conditions: 20% noise (370 noisy samples) and 50% noise (953 noisy samples). The theoretical optimal Mean Noise Ranks, where all noisy weights are ranked last, are 1815.5 and 1524.0, respectively. Results presented in Table. 9 of Appendix show that OSA achieves a higher Mean Noise Rank compared to NPC, demonstrating greater accuracy in re-weighting. Moreover, OSA’s rankings are nearly optimal (20% noise: 1809.1 for OSA versus 1815.5 optimal; 50% noise: 1520.7 for OSA versus 1524.0 optimal). This near-perfect alignment indicates that OSA effectively places almost all noisy samples behind the clean ones.

**Table 10: ACC and recall of noise detection.**
| Estimator Type | Noise Ratio | Acc | Recall |
| --- | --- | --- | --- |
| CLIP (w/o DA) | 0.2 | 93.88 | 97.49 |
| CLIP (w DA) | 0.2 | 97.68 | 97.18 |
| CLIP (w/o DA) | 0.5 | 93.91 | 99.35 |
| CLIP (w DA) | 0.5 | 98.14 | 99.24 |

### 4.6 Computational Cost Analysis

##### Cost in Pre-training.

To evaluate the practicality of OSA in a real-world pre-training scenario, we estimate the additional computational cost for processing 1 billion data points. Using an NVIDIA RTX 3090 with an inference batch size of 4096, utilizing approximately 24 GB of GPU memory, processing the MS-COCO dataset consisting of 566,435 pairs takes approximately 153 seconds. At this inference rate, processing 1 billion data points would require approximately 75 hours on a single RTX 3090. This cost is negligible within the context of large-scale pre-training, especially when leveraging multiple GPUs for parallel inference.

##### Time Cost Comparison.

To further examine the computational efficiency of OSA compared to other anti-noise techniques, we evaluate training time against two representative approaches: CLIP and NPC. CLIP, which serves as the baseline, is trained directly without any additional technique. NPC, the current state-of-the-art, also uses CLIP as its backbone but applies an anti-noise technique by estimating the negative impact of each sample, necessitating double backward passes. The training time comparison, presented in Table. 11, shows that our method introduces only a minimal increase in training time compared to direct training, requiring just one-tenth of the additional time needed by NPC. This highlights the efficiency of OSA, making it well-suited for large-scale robust training.

## 5 Conclusion

##### Broader Impacts.

In this work, we investigated the possibility of anti-noise in practical large-scale training. We introduced a novel model-agnostic anti-noise paradigm with advantages such as task transferability, model adaptability, and low computational overhead. By leveraging the properties of high-dimensional spaces, we found a robust and effective boundary for distinguishing between noisy and clean samples. Through rigorous theoretical analysis and comprehensive experimentation, we validated the efficacy and robustness of OSA for general noise mitigation. Although our primary objective is to adapt to practical large-scale training, OSA also achieves SOTA performance in standard anti-noise settings. To the best of our knowledge, this is the first work to explore anti-noise in practical large-scale training scenarios, as well as the first to propose a general anti-noise approach.

##### Limitations and Future Works.

Limited by the significant computational cost of pre-training, it is difficult for us to evaluate in a real pre-training process. Instead, we simulate large-scale pre-training processes to the greatest extent possible, such as evaluating on the real-world noisy dataset CC120K, which shares similar domains with mainstream pre-training datasets like CC4M and CC12M. Exploring the broad domain adaptability of OSA in real pre-training scenarios will be a valuable direction for future work.

## Appendix

## Appendix A Details of Implementation and Datasets

##### Dataset Details.

MSCOCO is widely used for noisy cross-modal matching, with each image accompanied by five descriptive captions.
Following the setting of [], we utilize 113,287 images for training, 5,000 for validation, and 5,000 for testing.
The Flickr30K dataset encompasses 31,783 image-text instances, each image paired with five textual annotations. Adhering to the NCR [], we use 29,783 images for training and 1,000 images each for validation and testing.
Regarding noise splits, following the NCR categorization, we conduct experiments at noise ratios of 0%, 20%, 40%, and 60%.
CC120K is a real-world multimodal noisy dataset collected by [] from the Internet, with about 3%-20% noise ratio. There are 118,851 image-text pairs for training, 1,000 for validation, and 1,000 for testing.

The Aircraft, Bird, and Car we used in the image classification task are three non-overlapping subsets of the WebFG-496 [] dataset.
WebFG-496 consists of 53,339 images, totaling 496 subcategories. This dataset is annotated using a webly supervised approach, which leverages resources from web search engines (e.g., Google Image Search Engine, Bing Image Search Engine) to expand the annotated image dataset.

For the image retrieval task, we conduct experiments on the CARS98N dataset under PRISM’s setting []. We utilize 9,558 car-related images sourced from text-based searches on Pinterest as the training set, and employ the remaining 98 categories from CARS, unsearched on Pinterest, as a clean test set.
The dataset’s noise is inherently real-world, with its creators estimating a noise ratio of approximately 50%.

##### Implementation Details.

To demonstrate the effectiveness of the OSA, we incorporate an estimator, built around the core of CLIP, and re-weighting operations based on the Estimator’s outcomes into numerous downstream tasks. In the principal task of cross-modal image-text retrieval, we employ CLIP with ViT-B/32 as the baseline and target model by default. All experiments are conducted on a single RTX 3090 GPU using the AdamW optimizer. During both training phases, the model is trained for five epochs with a batch size of 256 and 500 warmup steps.

For the image classification task on the WebFG dataset, we align with the field’s prevalent models for a fair comparison by employing the ResNet-50 model enhanced by CLIP for feature extraction and the CLIP image encoder as our estimator. Training and testing are executed on single RTX 3090 GPU, with an input image resolution of $448\times 448$. The batch size and initial learning rate are specified as 64 and 1e-5, respectively. In the first phase, the estimator is trained with data modeled by a Gaussian Mixture Model (GMM), which considers the classification and matching losses of all training samples, with the GMM probability threshold of 0.95. The classification task leverages the CLIP protocol, where a fixed prompt (“This is a picture of”) is prepended to category texts.

For the image retrieval task, we use CLIP ViT-B/32 as the baseline, with a batch size set to 128, an initial learning rate of 5e-6, and the number of epochs set to 10. Following the setup of the PRISM [], we set the parameter for sampling positive examples by the random sampler of the dataloader to 4, and adjust the number of positive examples sampled per epoch to one-fourth of the original parameter according to the increase in batch size. In this task, we also adopt a two-stage training approach. The strictly clean in-domain training data for the first stage is obtained using a GMM model with a probability setting of 0.8.

## Appendix B Related work

### B.1 Noise Mitigation in Cross-Modal Matching

The cross-modal matching task [, , , , ] serves as a fundamental component in multimodal learning.
However, the inherent difference in information density between these modalities leads to high annotation costs and inconsistent annotation quality, rendering cross-modal tasks particularly vulnerable to label noise.
Some approaches explicitly identify and correct noisy samples through cross-prediction between concurrently trained dual models [, , ], while others [, ] implicitly estimate the probability of sample noise, reducing its training impact by adjusting the loss function. NCR [] employs the memorization capacity of its counterpart model for simple clean samples to rectify the output results.
BiCro [] utilizes the consistency of similarity score distributions from a Siamese model ensemble on noisy data, alongside anchors modeled on the loss distribution via a Beta-Mixture-Model (BMM), to filter out noisy samples.
NPC [], deviating from the dual-model training schemes, introduces a two-stage single-model training approach that reduces training overhead by replacing two backward passes with one forward and one backward pass. Specifically, the first stage estimates the impact of potentially noisy samples on model performance by constructing a high-quality clean sample bank; the second stage then utilizes these estimates to reweight the loss function.
However, current methods for distinguishing clean from noisy samples rely on numerous hyperparameters that are closely linked to dataset size and model capacity. This dependency not only limits their adaptability to various downstream tasks but also makes them challenging to deploy in real-world applications.

### B.2 Noise Mitigation in Image Classification

Image classification is vulnerable to training data noise, due to varied noise types and strong model memorization.
Noise in datasets manifests in two primary forms: synthetic alterations and those arising from real-world scenarios. The former typically involves shuffling the labels of a subset of the data or retaining the labels while introducing corresponding category images from external datasets. The latter entails substituting images for a random selection of data points with those sourced from image search engines.
Existing approaches are categorized based on their operational focus: loss correction [, , , , , , , , , , ] and sample selection [, , , , ]. Loss correction methods typically incorporate a regularization term into the loss function, implicitly reweighting clean and noisy samples within the loss. Sample selection strategies, in contrast, explicitly differentiate between clean and noisy samples, applying distinct processing to each category during loss computation.
Representative for the loss correction category, [] aims to generalize ordinary Cross-Entropy loss and MAE loss by setting the loss threshold to iid and ood noisy samples.
DivideMix [] concurrently trains two networks, each utilizing the data partitioning from the other network to distinguish between clean and noisy samples based on loss values, thereby mitigating the influence of confirmation bias inherent within each network.
PNP [] framework employs a unified predictive network to estimate the in-distribution (iid), out-of-distribution (ood), and clean probabilities for a given sample. Co-training trained on a sample that has a lower loss, and with the different predictions by its siamese network.

### B.3 Noise Mitigation in Image Retrieval.

Although image retrieval tasks focus on pairwise relationships, the noise predominantly originates from image categorization errors. Analogous to image classification tasks, this can be bifurcated into in-domain [] and open-set noise []. In terms of task configuration, noise retrieval typically operates at the category level, treating images within the same category as positive instances.
PRISM [] tries to find noisy image samples by finding the outliers score in the whole similarity matrix from the same category. The generalization ability of the image feature is ensured by a broader query bank restored multi-view of it.
TITAN [] utilizes prototypes to be representative of the anchor of the clean and noisy samples and then generates synthetic samples by a combination of prototypes for substitution of noisy samples.
T-SINT [] utilizes more negative samples by the interaction between noisy samples and negative samples that belong to another category.

## Appendix C Proofs

### C.1 Proof of High-dimensional Orthogonality

Suppose $u,v\in\mathbbm{R}^{d}$ are any two random vectors. The cosine similarity $\cos(u,v)\sim\mathcal{N}(0,d^{-1})$. The probability that $\cos(u,v)$ is within a specific range $\left[-a,a\right]$ is denoted as:

$$ $P(-a\leq\cos(u,v)\leq a)=\Phi\left(\frac{a}{\varsigma}\right)-\Phi\left(\frac{ -a}{\varsigma}\right),$ (7) $$

where $\Phi$ represents the CDF of the standard normal distribution, and $\varsigma=\frac{1}{\sqrt{d}}$ is the standard deviation of the cosine similarity.
When $d=1024$ and $a=0.1$, there are

$$ $\varsigma=\frac{1}{\sqrt{1024}}=\frac{1}{32},$ (8) $$

and

$$ $P(-0.1\leq\cos(u,v)\leq 0.1)=\Phi\left(\frac{0.1}{1/32}\right)-\Phi\left(\frac {-0.1}{1/32}\right)\approx 0.9986.$ (9) $$

### C.2 Proof of Theorem 1

In the Section. 2.2, we propose that Theorem 1 about the relative relationship of pairs in the original entire space, will not change after transmitting to the narrow cone space of the trained model, and there is always a boundary $r$ concentrated on most random vectors.

To prove this Theorem, we first introduce a useful lemma of monotonicity of cosine similarity proposed by [], indicating that the cosine similarity between two vectors increases with a high probability after one feedforward computation consisting of a linear transformation and ReLU computation.

###### Lemma 1 .

Suppose $u,v\in\mathbbm{R}^{d_{in}}$ are any two fixed vectors such that $\left\|u\right\|=r\left\|v\right\|$ for some $r>0$, $\mathbf{W}\in\mathbbm{R}^{d_{out}\times d_{in}}$ is a random weight matrix where each element $\mathbf{W}_{k,l}\sim\mathcal{N}(0,d_{out}^{-1})$ for $k\in\left[d_{out}\right]$, $l\in\left[d_{in}\right]$, and $\mathbf{b}\in\mathbbm{R}^{d_{out}}$ is a random bias vector such that $\mathbf{b}_{k}\sim\mathcal{N}(0,d^{-1}_{out})$ for $k\in\left[d_{out}\right]$. If $\cos(u,v)<(\frac{1}{2}(r+\frac{1}{r}))^{-1}$, then the following holds with probability at least $1-O(1/d_{out})$.

$$ $\cos(\sigma(\mathbf{W}u+\mathbf{b}),\sigma(\mathbf{W}v+\mathbf{b}))>\cos(u,v).$ (10) $$

###### Proof of Theorem. 1 .

Let $\mathbbm{R}^{d_{in}}$ be the original space before being transmitted in a neural network. Suppose $u,v\in\mathbbm{R}^{d_{in}}$ are any two random vectors with $\cos(u,v)\approx 0$. $u_{c},v_{c}\in\mathbbm{R}^{d_{in}}$ is a pair of clean vectors with $\cos(u_{c},v_{c})>0$, while $u_{n},v_{n}\in\mathbbm{R}^{d_{in}}$ is a noisy pair with $\cos(u_{n},v_{n})<0$. Given a Neural Network $F(x)={f_{t}(f_{t-1}(\dots f_{2}(f_{1}(x))))}\in\mathbbm{R}^{d_{out}}$ with $t$ layers. $f_{i}(x)=\sigma_{i}(\mathbf{W}_{i}x+\mathbf{b}_{i})$ denotes $i^{th}$ layer, where $\sigma(\cdot)$ indicates activation function. $\mathbf{W}_{i}\in\mathbbm{R}^{d^{i}_{out}\times d^{i}_{in}}$ is a random weight matrix where each element $\mathbf{W}_{i}^{k,l}\sim\mathcal{N}(0,1/d_{out}^{i})$ for $k\in\left[d^{i}_{out}\right]$, $l\in\left[d^{i}_{in}\right]$, and $\mathbf{b}_{i}\in\mathbbm{R}^{d^{i}_{out}}$ is a random bias vector such that $\mathbf{b}^{k}_{i}\sim\mathcal{N}(0,1/d^{i}_{out})$ for $k\in\left[d^{i}_{out}\right]$. We would like to prove that there are always be a boundary $\beta$, satisfying:

$$ $\cos(F(u_{n}),F(v_{n}))<\cos(F(u),F(v))\approx\beta<\cos(F(u_{c}),F(v_{c})),$ (11) $$

which is equivalent to proving.

$$ $\cos(f_{i}(u_{n}),f_{i}(v_{n}))<\cos(f_{i}(u),f_{i}(v))\approx\beta_{i}<\cos(f _{i}(u_{c}),f_{i}(v_{c})),$ (12) $$

where $\beta_{i}$ is the boundary of $i^{th}$ layer.

We first consider the cosine similarity between $u$ and $v$ as:

$$ $\cos(u,v)=\frac{u\cdot v}{\|u\|\|v\|}.$ (13) $$

After a linear transformation of $i^{th}$ layer, the cosine similarity of $\cos(\mathbf{W}_{i}u+\mathbf{b}_{i},\mathbf{W}_{i}v+\mathbf{b}_{i})$ denotes:

$$ $\cos(\mathbf{W}_{i}u+\mathbf{b}_{i},\mathbf{W}_{i}v+\mathbf{b}_{i})=\frac{( \mathbf{W}_{i}u+\mathbf{b}_{i})\cdot(\mathbf{W}_{i}v+\mathbf{b}_{i})}{\| \mathbf{W}_{i}u+\mathbf{b}_{i}\|\|\mathbf{W}_{i}v+\mathbf{b}_{i}\|}.$ (14) $$

Since $\mathbf{b}_{i}$ has a mean of zero and is independent from $\mathbf{W}_{i}u$ and $\mathbf{W}_{i}v$, the expectation of $\mathbf{b}_{i}$ and $(\mathbf{W}_{i}u+\mathbf{b}_{i})\cdot\mathbf{W}_{i}v+\mathbf{b}_{i})$ can be signified as:

$$ $\mathbb{E}\left[\mathbf{b}_{i}\right]=0,$ (15) $$

$$ $\mathbb{E}\left[(\mathbf{W}_{i}u+\mathbf{b}_{i})\cdot(\mathbf{W}_{i}v+\mathbf{ b}_{i})\right]=\mathbb{E}\left[(\mathbf{W}_{i}u\cdot\mathbf{W}_{i}v)\right]= \sum^{n}_{i=1}\sum^{n}_{i=1}\frac{1}{d^{i}_{out}}u_{k}v_{k}=\frac{1}{d^{i}_{ out}}(u\cdot v).$ (16) $$

Additionally, we have

$$ $\|\mathbf{W}_{i}u+\mathbf{b}_{i}\|^{2}=\mathbf{W}_{i}u\cdot\mathbf{W}_{i}u+2 \mathbf{W}_{i}u\cdot\mathbf{b}_{i}+\mathbf{b}_{i}\cdot\mathbf{b}_{i}.$ (17) $$

Due to $\mathbbm{b}^{k}\sim\mathcal{N}(0,1/d^{i}_{out})$, as $d^{i}_{o}ut$ increases, the term of $2\mathbf{W}_{i}u\cdot\mathbf{b}_{i}$ and $\mathbf{b}_{i}\cdot\mathbf{b}_{i}$ become negligible, which implies

$$ $\|\mathbf{W}_{i}u+\mathbf{b}_{i}\|^{2}\approx\mathbf{W}_{i}u\cdot\mathbf{W}_{i }u=\sum_{i=1}^{n}(\mathbf{W}_{i}u)^{2}.$ (18) $$

Therefore, the expectation of $\|\mathbf{W}_{i}u+\mathbf{b}_{i}\|^{2}$ is approximate to

$$ $\mathbb{E}\left[\|\mathbf{W}_{i}u\|^{2}\right]=\sum_{k=1}^{n}u_{k}^{2}\frac{1} {d^{i}_{out}}=\frac{\|u\|}{d^{i}_{out}},$ (19) $$

and

$$ $\displaystyle\cos(\mathbf{W}_{i}u+\mathbf{b}_{i},\mathbf{W}_{i}v+\mathbf{b}_{i})$ $\displaystyle\approx\frac{\mathbb{E}[\mathbf{W}_{i}u\cdot\mathbf{W}_{i}v]}{ \sqrt{\mathbb{E}[\|\mathbf{W}_{i}u+\mathbf{b}_{i}\|^{2}]\mathbb{E}[\|\mathbf{W }_{i}v+\mathbf{b}_{i}\|^{2}]}}$ (20) $\displaystyle=\frac{\frac{1}{d^{i}_{out}}(u\cdot v)}{\sqrt{\frac{1}{d^{i}_{out }}\|u\|^{2}\cdot\frac{1}{d^{i}_{out}}\|v\|^{2}}}$ $\displaystyle=\cos(u,v).$ $$

Based on Eq. 20, with $\cos(u_{n},v_{n})<\cos(u,v)\approx 0<\cos(u_{c},v_{c})$, there are

$$ $\cos(\mathbf{W}_{i}u_{n}+\mathbf{b}_{i},\mathbf{W}_{i}v_{n}\mathbf{b}_{i})< \cos(\mathbf{W}_{i}u+\mathbf{b}_{i},\mathbf{W}_{i}v+\mathbf{b}_{i})<\cos( \mathbf{W}_{i}u_{c}+\mathbf{b}_{i},\mathbf{W}_{i}v_{c}+\mathbf{b}_{i}).$ (21) $$

Since the activation function $\sigma$ is a monotonically increasing function, it follows

$$ $\cos(f_{i}(u_{n}),f_{i}(v_{n}))<\cos(f_{i}(u),f_{i}(v))<\cos(f_{i}(u_{c}),f_{i }(v_{c})).$ (22) $$

Due to Lemma. 1, $\cos(f_{i}(u),f_{i}(v))$ will be increase with the transmitting layers, and $\cos(f_{i}(u),f_{i}(v))$ will always be a $\beta_{i}>0$, to satisfy:

$$ $\cos(f_{i}(u_{n}),f_{i}(v_{n}))<\cos(f_{i}(u),f_{i}(v))\approx\beta_{i}<\cos(f _{i}(u_{c}),f_{i}(v_{c})).$ (23) $$

After transmitting each layer, Eq. 23 are always satisfied. When transmitting a neural network with $t$ layers, we have

$$ $\cos(F(u_{n}),F(v_{n}))<\cos(F(u),F(v))\approx\beta<\cos(F(u_{c}),F(v_{c})).$ (24) $$

∎

Figure: (a) parameters
Refer to caption: x8.png

###### Lemma 1 .

###### Proof of Theorem. 1 .

### C.3 Proof of Orthogonality Validity in Cone Space

Although we have demonstrated in Appendix. C.1 that in the original high-dimensional space, the cosine similarity between two randomly selected vectors—each dimension following a Gaussian distribution—typically converges near the orthogonal boundary, this property may not necessarily extend to the subspace of the shared embedding space maintained by the trained models. Specifically, for real image-text pairs, the subspace may deviate from the orthogonal characteristics observed in the original space. Thus, it is essential to investigate whether the orthogonality property holds within the cone space for the image-text subdomain post-training.

To explore this, we first analyze the distribution of several dimensions of image and text features from the CC120K dataset, as illustrated in Figure. 3. The results reveal that all vector dimensions, including trained parameters, exhibit a Gaussian distribution with near-zero means. If the dimensions of the trained embedding space follow Gaussian distributions, the process of selecting random vectors within this space would be analogous to that of the original space, thereby preserving the orthogonality property. Here, we present the following theorem: The output features of large-scale models tend to Gaussian distribution. The detailed theorem and proof are provided below.

###### Theorem 2 (Output features tends to Gaussian) .

Given a Neural Network $F(x)=\{f_{t}(f_{t-1}(\dots f_{2}(f_{1}(x))))\}\in\mathbb{R}^{d_{out}}$ with $t$ layers. $f_{l}(x)=\phi_{l}(\mathbf{W}_{l}x+\mathbf{b}_{l})$ denotes the $l^{th}$ layer, where $\phi(\cdot)$ indicates the activation function, and the final layer $f_{t}(x)=\mathbf{W}_{t}x+\mathbf{b}_{t}$ is a fully-connected layer without an activation function for common space projection. Let $x^{k}\in\mathbb{R}^{d_{in}^{k}}$ be the sample feature that will be transmitted into the $k^{th}$ layer, where $x^{1}$ denotes the original feature with an unknown distribution $x^{1}\sim(\mu_{x},\sigma_{x}^{2})$. $\mathbf{W}_{k}\in\mathbb{R}^{d^{k}_{out}\times d^{k}_{in}}$ is a random weight matrix where each element $w^{k}_{ij}\sim\mathcal{N}(0,\sigma_{w}^{2})$ for $i\in[d^{k}_{out}]$, $j\in[d^{k}_{in}]$, and $\mathbf{b}_{k}\in\mathbb{R}^{d^{k}_{out}}$ is a bias vector such that $b_{i}^{k}\sim\mathcal{N}(0,\sigma_{w}^{2})$ for $i\in[d^{k}_{out}]$. In such a neural network, linear layers lead features $x$ gradually to a Gaussian distribution from any initial distribution, and as $|d_{in}|$ is sufficiently large, $F(x)\sim\mathcal{N}(0,\sigma^{2})$.

###### Proof of Theorem. 2 .

For the $k^{th}$ layer ($k\in[t]$), we first calculate the expectation and variance of the linear combination $\sum_{j=1}^{d_{in}^{k}}w_{ij}^{k}x_{j}^{k}$. For the expectation, since $w_{ij}^{k}$ and $x_{j}^{k}$ are independent and $\mathbf{w}^{k}_{ij}\sim\mathcal{N}(0,\frac{1}{d_{out}^{k}})$, we have:

$$ $\mathbb{E}\left[\sum_{j=1}^{d_{in}^{k}}w_{ij}^{k}x_{j}^{k}\right]=\sum_{j=1}^{ d_{in}^{k}}\mathbb{E}[w_{ij}^{k}]\mathbb{E}[x_{j}^{k}]=\sum_{j=1}^{d_{in}^{k}} (0\times\mathbb{E}[x_{j}^{k}])=0.$ (25) $$

For variance, since $w_{ij}^{k}$ and $x_{j}^{k}$ are independent, we have:

$$ $\displaystyle\text{Var}\left(\sum_{j=1}^{d_{in}^{k}}w_{ij}^{k}x_{j}^{k}\right)$ $\displaystyle=\sum_{j=1}^{d_{in}^{k}}\text{Var}(w_{ij}^{k}x_{j}^{k})=\sum_{j=1 }^{d_{in}^{k}}\mathbb{E}\left[(w_{ij}^{k})^{2}(x_{j}^{k})^{2}\right]$ (26) $\displaystyle=\sum_{j=1}^{d_{in}^{k}}\mathbb{E}\left[(w_{ij}^{k})^{2}\right] \mathbb{E}\left[(x_{j}^{k})^{2}\right]$ $\displaystyle=\sum_{j=1}^{d_{in}^{k}}\sigma_{w^{k}}^{2}\left(\text{Var}(x_{j}^ {k})+(\mathbb{E}[x_{j}^{k}])^{2}\right)$ $\displaystyle=\sum_{j=1}^{d_{in}^{k}}\sigma_{w^{k}}^{2}\left(\sigma_{x^{k}}^{2 }+\mu_{x^{k}}^{2}\right)=d_{in}^{k}\sigma_{w^{k}}^{2}\left(\sigma_{x^{k}}^{2}+ \mu_{x^{k}}^{2}\right).$ $$

Since $w_{ij}^{k}$ are independently distributed Gaussian random variables, and $x_{j}^{k}$ has a known mean and variance, the sum of $w_{ij}^{k}x_{j}^{k}$ can apply to a generalized Central Limit Theorem. We have

$$ $\frac{\sum_{j=1}^{d_{in}^{k}}w_{ij}^{k}x_{j}^{k}-\mathbb{E}\left[\sum_{j=1}^{d _{in}^{k}}w_{ij}^{k}x_{j}^{k}\right]}{\sqrt{\text{Var}\left(\sum_{j=1}^{d_{in} ^{k}}w_{ij}^{k}x_{j}^{k}\right)}}\xrightarrow{d}\mathcal{N}(0,1),$ (27) $$

which is equivalent to

$$ $\frac{\sum_{j=1}^{d_{in}^{k}}w_{ij}^{k}x_{j}^{k}-0}{\sqrt{d_{in}^{k}\sigma_{w^ {k}}^{2}(\sigma_{x^{k}}^{2}+\mu_{x^{k}}^{2})}}\xrightarrow{d}\mathcal{N}(0,1).$ (28) $$

Therefore,

$$ $\sum_{j=1}^{d_{in}^{k}}w_{ij}^{k}x_{j}^{k}\xrightarrow{d}\mathcal{N}(0,d_{in}^ {k}\sigma_{w^{k}}^{2}(\sigma_{x^{k}}^{2}+\mu_{x^{k}}^{2})).$ (29) $$

Due to $b^{k}\sim\mathcal{N}(0,\sigma^{2}_{b})$, we finally get

$$ $\sum_{j=1}^{d_{in}^{k}}w_{ij}^{k}x_{j}^{k}+b_{i}^{k}\xrightarrow{d}\mathcal{N} \left(0,d_{in}^{k}\sigma_{w^{k}}^{2}(\sigma_{x^{k}}^{2}+\mu_{x^{k}}^{2})+ \sigma_{b}^{2}\right).$ (30) $$

Although activation functions truncate the Gaussian distribution after each linear layer, the samples still gradually approach a Gaussian distribution from the initial unknown distribution as they pass through the layers. Furthermore, because there is a fully connected layer (
layer) without an activation function before mapping to the final common space, the final feature distribution will approximate a Gaussian distribution, as follows:

$$ $F(x)\sim\mathcal{N}(0,d^{t}_{in}\sigma^{2}_{w^{t}}(\sigma^{2}_{x^{t}}+\mu^{2}_ {x^{t}})+\sigma^{2}_{b}).$ (31) $$

∎

###### Theorem 2 (Output features tends to Gaussian) .

###### Proof of Theorem. 2 .

## Appendix D SDM Visualization

We visualize some representative samples from our synthetic domain originating from COCO by using SDM. The results are shown in Figure. 4. We generate two styles of image based on the MSCOCO caption, and then use pre-trained multimodal models to calculate cosine similarity with the SDM-generated image and original caption.

Figure: Figure 4: Examples of generated SDM dataset. The first row is in sketch style, while the second row is in cartoon style.
Refer to caption: x15.png

## Peer Reviews

confidence: 3
contribution: 2
initial_rating: 5
presentation: 2
questions: See **Weaknesses**.
soundness: 2
strengths: - The paper introduces a novel, model-agnostic method called One-step AntiNoise (OSA) that addresses the issue of noisy labels in a cost-efficient manner, which is an advancement over existing noise mitigation techniques.

- It provides a theoretical framework that explains the stability and precision of the decision boundary in high-dimensional spaces, offering insights into why and how the proposed method works effectively.

- The paper backs up its claims with empirical evidences, demonstrating OSA's superiority across various benchmarks, models, and tasks, which strengthens the credibility of the proposed method.

- The paper shows that OSA introduces minimal additional training time compared to standard training methods, making it suitable for real-world applications.

- The paper demonstrates that OSA is not only effective in standard noise settings but also exhibits strong task transferability and model adaptability, making it a versatile solution applicable to a wide range of scenarios.
summary: The paper addresses the challenge of mitigating the detrimental effects of noisy labels in large-scale pre-training tasks, where obtaining entirely clean data is often impractical. The authors propose a model-agnostic approach called One-step AntiNoise (OSA), which utilizes an estimator model and a scoring function to assess noise levels through single-step inference, significantly reducing computational costs. OSA leverages high-dimensional orthogonality to establish a robust boundary for separating clean and noisy samples, demonstrating enhanced training robustness, improved task transferability, and ease of deployment across various benchmarks and models. The paper provides a theoretical framework explaining the stability of the decision boundary and conducts comprehensive experiments to validate the method's effectiveness and efficiency. The authors conclude that OSA is a novel solution for noise mitigation in practical large-scale training scenarios, with code available for reproducibility.
weaknesses: - Could the authors provide further insights into the design of the scoring function (Equation 5)? Specifically, what is the value of $\beta$ across different datasets and models, and how does the sensitivity of $\beta$ impact performance?

- Regarding Table 4, is it possible to generalize OSA for multi-class classification?

- In Figure 2, is it clear whether the estimator remains fixed or is updated during training? In other words, do the estimator and the target model share the same weights?

- Could the authors include time statistics for more methods in Table 7? Specifically, how is the time recorded? Since convergence time can vary among different methods, it is important to also consider this aspect.

---

confidence: 3
contribution: 2
initial_rating: 5
presentation: 3
questions: - I recommend discussing the relationships with previous works using pretrained models e.g.[1].

[1] Fine tuning pre trained models for robustness under noisy labels. *arXiv preprint arXiv:2310.17668*.
soundness: 3
strengths: - This work is well-written, from the challenges and motivation to the theoretical analysis and method design.
- This paper focuses on an extended scenario from traditional classification tasks to image-text matching task.
- The proposed method also considers the computation consumptions. The efficiency analysis shows its huge potential in practical applications.
summary: This paper introduces a model-agnostic noise mitigation paradigm for the limitations of current noisy label approaches. It leverages cosine similarity measures to distinguish between noisy and clean samples efficiently. It shows robustness across various real-world noisy benchmarks.
weaknesses: - The contribution of the one-step property is weakened due to the common sense that the pre-trained model performs well in distinguishing noise samples because the noisy samples do not damage it. Training a robust model from scratch from noisy datasets is more challenging and attracts more attention.
- I suggest authors conduct more experiments on noise types and noise rates especially extreme noise rates.
- I recommend experiments performed on different scoring functions.

---

confidence: 5
contribution: 2
initial_rating: 5
presentation: 1
questions: See weakness.
soundness: 2
strengths: 1. The problem studied in this paper is very important, especially in the current era when large models are so popular. 
2. Though it has been discussed and proposed before, it is reasonable to use additional models to help with sample selection and reweighting.
summary: This paper focuses on the very practical problem of noisy labels in the dataset. The authors propose a sample weighting mechanism based on pre-trained models, especially visual language models such as CLIP.
weaknesses: 1. The paper is difficult to read, primarily because it lacks a clear problem definition section. For instance, while I understand the intuitive idea, how is "cleanness" mathematically defined? Should we assume that \( x \) and \( y \) come from a shifted joint distribution? Then, how is the noisy distribution structured, and what type of noise is being used? Additionally, what does the noise ratio in the experiments represent? For instance, in the COCO dataset, did you randomly replace a proportion of captions? This lack of clarity also makes it hard for me to understand the significance of Theorem 1 and the related analysis. 

2. The paper lacks a discussion of important related literature. I would list a few representative methods in learning with noisy labels community:
   - *[1]* DivideMix: Learning with Noisy Labels as Semi-supervised Learning.
   
   And sample selection methods based on feature space, which are more relevant to this work:
   - *[2]* Multi-Objective Interpolation Training for Robustness to Label Noise
   - *[3]* FINE Samples for Learning with Noisy Labels

   There are also papers that use the CLIP model:
   - *[4]* CLIPCleaner: Cleaning Noisy Labels with CLIP
   - *[5]* Vision-Language Models are Strong Noisy Label Detectors
   - *[6]* Combating Label Noise With A General Surrogate Model For Sample Selection

   (*Some of these references may be considered concurrent work; The authors are suggested to discuss these papers in the future version.*)

In summary, the method presented in this paper essentially leverages a large pre-trained vision-language model to identify potentially correct samples and exclude likely incorrect ones. As I understand it, the method could be effectively explained within lines 253-258 alone, yet the presentation is overly complex. The authors need to restructure the manuscript to clarify the paper's contribution and explicitly compare it with relevant work.

---

confidence: 3
contribution: 2
initial_rating: 5
presentation: 3
questions: 1. The paper does not seem to describe whether the backbone in the experiment was randomly initialized or trained. As I understand it, the estimator is a trained CLIP and the backbone for the baselines is also a trained CLIP. Is this correct?
soundness: 2
strengths: 1. The writing and presentation of the paper is clear.
2. The boundary principle analysis of the paper is  instructive.
3. The experiments in this paper are detailed and show validity.
summary: This paper focuses on how to mitigate noisy labels, in particular noisy cross-modal matching. Specifically, the authors first use a pre-trained model, such as CLIP, to determine whether a data is noisy, and then give less weight to the noisy label during the training process. And they pointed out that the orthogonal boundary separates the clean and noisy sides. The authors conducted experiments on different tasks and datasets.
weaknesses: 1. I think it would be better if the authors emphasised in the title or elsewhere that the proposed work focuses primarily on noisy cross-modal matching. Otherwise it could be confusing. For example, the authors claim that other methods cause additional computational overhead. However, papers cited by the authors in related work, such as [1], do not incur additional overhead; rather, the proposed work causes additional overhead.
2. The paper doesn't seem to describe how big the CLIP is as an Estimator. If the author uses a trained maximum CLIP as an Estimator, then of course there will be a performance boost because it is a strong model. That doesn't seem fair to the baselines.
3. An approach that relies on trained large models does not seem very interesting. And regarding Eq. 5, the authors do not provide a theoretical analysis.

[1] Generalized Cross Entropy Loss for Training Deep Neural Networks with Noisy Labels, NeurIPS 2018