Learning with Noisy Correspondence for Cross-modal Matching
Year: 2021

Abstract:
Cross-modal matching, which aims to establish the correspondence between two different modalities, is fundamental to a variety of tasks such as cross-modal retrieval and vision-and-language understanding. Although a huge number of crossmodal matching methods have been proposed and achieved remarkable progress in recent years, almost all of these methods implicitly assume that the multimodal training data are correctly aligned. In practice, however, such an assumption is extremely expensive even impossible to satisfy. Based on this observation, we reveal and study a latent and challenging direction in cross-modal matching, named noisy correspondence, which could be regarded as a new paradigm of noisy labels. Different from the traditional noisy labels which mainly refer to the errors in category labels, our noisy correspondence refers to the mismatch paired samples. To solve this new problem, we propose a novel method for learning with noisy correspondence, named Noisy Correspondence Rectifier (NCR). In brief, NCR divides the data into clean and noisy partitions based on the memorization effect of neural networks and then rectifies the correspondence via an adaptive prediction model in a co-teaching manner. To verify the effectiveness of our method, we conduct experiments by using the image-text matching as a showcase. Extensive experiments on Flickr30K, MS-COCO, and Conceptual Captions verify the effectiveness of our method. The code could be accessed from www.pengxi. me.

text: Cross-modal matching, which aims to establish the correspondence between two different modalities, is fundamental to a variety of tasks such as cross-modal retrieval and vision-and-language understanding. Although a huge number of crossmodal matching methods have been proposed and achieved remarkable progress in recent years, almost all of these methods implicitly assume that the multimodal training data are correctly aligned. In practice, however, such an assumption is extremely expensive even impossible to satisfy. Based on this observation, we reveal and study a latent and challenging direction in cross-modal matching, named noisy correspondence, which could be regarded as a new paradigm of noisy labels. Different from the traditional noisy labels which mainly refer to the errors in category labels, our noisy correspondence refers to the mismatch paired samples. To solve this new problem, we propose a novel method for learning with noisy correspondence, named Noisy Correspondence Rectifier (NCR). In brief, NCR divides the data into clean and noisy partitions based on the memorization effect of neural networks and then rectifies the correspondence via an adaptive prediction model in a co-teaching manner. To verify the effectiveness of our method, we conduct experiments by using the image-text matching as a showcase. Extensive experiments on Flickr30K, MS-COCO, and Conceptual Captions verify the effectiveness of our method. The code could be accessed from www.pengxi. me.

heading: 1 Introduction
text: As one of the most fundamental techniques in multimodal learning, cross-modal matching aims to bridge different modalities. In recent years, some cross-modal matching methods [19, 11, 7, 26] have been proposed based on Deep Neural Networks (DNNs), which achieved remarkable progress in a ∗Some parts of the work was done while Zhenyu Huang was an internship at Baidu Inc. †Corresponding author.
35th Conference on Neural Information Processing Systems (NeurIPS 2021).
variety of applications, such as clustering [29, 24], image/video captioning [1, 44, 22], cross-modal retrieval [40, 19, 13], and visual question answering [9].
In general, most existing cross-modal matching methods embed different modalities into a common space wherein the similarity of positive cross-modal pairs is maximized and that of the negative ones is minimized. Although these methods have achieved promising results, their success depends on an implicit data assumption, i.e., the training data are correctly aligned across modalities. For example, in the vision-and-language tasks, the text needs to accurately describe the image content, and vice versa. In practice, however, it is extremely expensive and time-consuming to annotate or collect such data pairs. Especially, considering the data collected from the Internet [35, 14], it is inevitable to collect some mismatched pairs which are wrongly treated as the matched ones. To the best of our knowledge, such a special noisy label (correspondence) problem has been ignored so far, which will remarkably degrade the performance of matching methods as shown in our experiments.
Based on the above observation, we reveal a new paradigm for the noisy labels, named noisy correspondence. Different from the traditional noisy labels, the noisy correspondence refers to the alignment errors in paired data rather than the errors in category annotations (see Fig. 1). To the best of our knowledge, there is no effort has been devoted to study this new problem and the closest paradigm might be the partially view-aligned problem (PVP) [12, 41]. However, PVP is remarkably different from noisy correspondence, and the latter is more practical than the former. To be specific, PVP focuses on that the cross-modal alignment is totally unavailable, whereas the noisy correspondence focuses on that some correspondences are incorrect. In addition, PVP assumes that some correctly aligned data are available for training, whereas our noisy correspondence assumes that the clean and noisy data are mixed.
To solve the noisy correspondence problem in cross-modal matching, we propose a novel method, named Noisy Correspondence Rectifier (NCR). Our method is based on the memorization effect of DNNs observed in [3, 39], i.e., DNNs tend to learn the simple patterns before fitting noisy samples. Motivated by this empirical observation, NCR divides the data into two relative accurate data partitions, i.e., “noisy” and “clean” subsets, based on their loss difference. After that, NCR employs an adaptive prediction function for label rectifying so that the false positives and the true positives could be identified from the “clean” and the “noisy” subsets, respectively. Furthermore, we propose a novel triplet loss for robust cross-modal matching by recasting the rectified labels as the soft margin.
The main contributions and novelties of this paper could be summarized as below. i) We reveal a new problem in cross-modal analysis, which is also a new paradigm for noisy labels, termed noisy correspondence. Different from the traditional noisy labels, the noisy correspondence refers to the alignment errors in paired data instead of the errors in category annotations. To the best of our knowledge, this work could be the first study on this problem. ii) To solve the noisy correspondence problem, we propose a novel method for learning with noisy correspondence, named Noisy Correspondence Rectifier (NCR). One major novelty of NCR is that the rectified label is elegantly recasted as the soft margin of a triplet loss so that the robust cross-modal matching could be achieved. iii) To verify the effectiveness of our method, we conduct experiments on the image-text matching task. Extensive experiments on three challenging datasets verify the effectiveness of our method in the synthesized and real noises.

heading: 2 Related works
text: In this section, we briefly introduce some recent developments in cross-modal matching and learning with noisy labels.

heading: 2.1 Cross-modal Matching
text: Most existing cross-modal matching works seek to learn a common space wherein different modalities are comparable. In general, existing works could be roughly divided into two categories: 1) Coarsegrained Matching. It often utilizes multiple neural networks to compute a global feature and each network is used for a specific modality [17, 37, 8]. For example, Kiros et al. [17] use a Convolutional Neural Network (CNN) and a Gated Recurrent Unit (GRU) to obtain the image and text features, while enforcing the similarity of positive pairs larger than that of the negative ones. To further boost the matching performance, VSE++ [8] uses some representative negatives to improve the discrimination of the model. 2) Fine-grained Matching. It seeks to measure the fine-grained similarity for cross-modal matching [19, 21, 7]. For example, SCAN [19] proposes learning the latent semantic correspondence between the image regions and words that are extracted by bottom-up attention [1] and GRU, respectively. VSRN [21] adopts a graph convolutional network for semantic reasoning. SGRAF [7] proposes constructing a similarity graph to reason the similarity and adopting an attention filtration technique to eliminate the less-meaningful alignments. Recently Chun et al. [6] introduce a new paradigm for cross-modal matching, i.e. possible many-to-many correspondence that existed in the image and captions. To achieve this, they propose to use probabilistic representations to model the possible one-to-many correspondence.
Although promising results have been achieved in recent years, the existing methods heavily rely on the correctly aligned data. In practice, however, such well-matched data is expensive and timeconsuming to collect. Moreover, some recent works [35, 14] show that a large-scale dataset collected from the wild could remarkably improve the performance of the model. However, such a data will inevitably contain some mismatched pairs. Hence, it is highly expected to develop some methods which are robust against the noisy correspondence, which has not been studied as far as we know. Different from the many-to-many correspondence [6] between image and captions, NCR reveals the noisy correspondence problem which refers to the alignment errors of image-text pairs and proposes to eliminate the negative impact from noisy pairs for downstream tasks.

heading: 2.2 Learning with Noisy Labels
text: To handle the possible noisy annotations in the training data, a large number of methods have been proposed and almost all of them focus on the classification task [36, 27]. To reduce the negative impact of the noisy labels, the existing works often resort to robust architecture design, regularization, loss adjustment, or sample selection methods. Here, we mainly introduce the last two approaches which are most related to this work. To be specific, the loss adjustment achieves robustness by adjusting the contribution of clean and noisy samples w.r.t. the loss. For example, Reed et al. [32] proposed a bootstrapping loss based on the model predictions for loss correction. Zhang et al. [45] provided some theoretical explanations for the label correction along with a new label correction algorithm. Different from the loss adjustment methods, sample selection methods aim to select clean samples from a noisy dataset. For example, Arpit et al. [3] showed that DNNs tend to learn simple patterns before fitting noisy samples, namely the memorization effect. Motivated from this, Arazo et al. [2] proposed treating the samples with small loss as the clean samples. To avoid the selection bias of clean samples, Co-teaching methods [10, 43] use the samples with small loss to iteratively train two networks. In recent, DivideMix [20] adopts the MixMatch method [4] for semi-supervised learning with the clean and noisy samples.
Unlike the above noisy label studies, this paper focuses on the noisy correspondence problem which considers mismatched multimodal data pairs instead of incorrectly annotated data points. Besides the difference in the problem, this work is also different from the aforementioned studies in the methodology. To be specific, in cross-modal matching, it is impossible to directly adopt these noise label learning methods to solve the noisy correspondence problem due to the following two reasons. First, most of the noisy label learning methods propose to use the model’s prediction for label rectifying in the scenario of classification, while it is intractable to directly predict the correspondence of given pairs in matching models. Second, even if we can rectify the noisy correspondence somehow,
Soccer p laye r i s beaten to the ball by soccer player.
f
g
Soft Margin
Input Data Embeding Similarity Measuring
+ S
0
1
(a) Training pipeline (b) Robust image-text matching network (A and B)
P(I, T)
Prediction
A B

heading: A BWarmup
text: Divide
Rectify
Training
Prediction
Figure 2: Overview of the proposed method. (a) Training pipeline of NCR. NCR consists of two individual networks (A, B) which work in the manner of co-teaching. In brief, NCR first warmup the networks (A, B) on the original training data using the loss Lw which is also used for per-sample loss computation. Then, based on the memorization effect of DNNs, NCR divides the training data into clean and noisy subsets at each epoch using either A or B, i.e., SA = (SAc ,SAn ) and SB = (SBc ,SBn ). After that, NCR will co-rectify the correspondence of {SA,SB} and obtain {ŜA, ŜB} using an adaptive prediction function. Finally, ŜA and ŜB will be used to train the network B and A in a swapping way. (b) Robust image-text matching network. For example, the network A projects the image and text by the modal-specific networks f and g, respectively. Then the similarity S(I, T ) is computed on the extracted features f(I) and g(T ). To achieve robust image-text matching, the rectified soft labels are recast as the soft margin of our loss Lsoft. As shown, for a given anchor, Lsoft will enforce the true positive to closer to it than the negative by a large margin α̂1, and meanwhile the false positive will has a small margin α̂2.
the rectified real-valued labels are incompatible with the existing matching methods since most of them assume the given labels are binary. To address these problems, NCR proposes an adaptive prediction function and a novel triplet loss by recasting the soft labels as soft margins.

heading: 3 The Proposed Method
text: In this section, we elaborate on the proposed method, i.e., Noisy Correspondence Rectifier (NCR) which could be the first work to solve the noisy correspondence problem in cross-modal matching. In Section 3.1, we introduce the co-divide module which splits the training data into the clean and noisy subsets. After that, we introduce how to rectify the labels with an adaptive prediction function in Section 3.2. Finally, we detail how to combine the co-divide and co-rectify modules to achieve robust cross-modal matching in Section 3.3.

heading: 3.1 Co-divide
text: Without loss of generality, we first introduce the cross-modal matching task by taking the image-text matching as a showcase. Given the training data D = {(Ii, Ti, yi)}Ni=1, where N is the data size, (Ii, Ti) is an image-text pair and yi ∈ {1, 0} indicates that the pair belong to the same instance (positive) or not (negative). For the noisy correspondence case, it defines that an unknown portion of D is mismatched, i.e., (Ii, Ti) is a negative pair but wrongly labeled as yi = 1. To solve such a noisy correspondence problem, we propose NCR to achieve robust cross-modal matching.
To begin, we project the visual and textual modalities into a shared space via two modal-specific networks f and g, respectively. Then the similarity of the given image-text pairs is computed through S(f(I), g(T )). For simplicity, we denote S(f(I), g(T )) as S(I, T ) in the following. Some early empirical studies [3] show that DNNs tend to first learn simple samples and then gradually fit the noisy samples. This so-called memorization effect of DNNs will lead to a relatively low loss for the clean samples. Motivated by this, we utilize the difference of loss distribution between the clean samples and noisy samples to divide the training data like [10, 43, 2, 20]. Specifically, given a matching model (f, g, S), we compute the per-sample loss through:
`(f,g,S) = {`i}Ni=1 = {Lw(Ii, Ti)}Ni=1 (1)
where Lw is defined as: Lw(I, T ) = ∑ T̂ [α− S(I, T ) + S(I, T̂ )]+ + ∑ Î [α− S(I, T ) + S(Î , T )]+, (2)
where (I, T ) is a positive pair, α > 0 denotes a given margin, and [x]+ = max(x, 0). In the loss, the first term treats I as queries taking over all negative text T̂ , while the second term treats T as queries taking over all negative images Î . Then, we fit the per-sample loss of all training data by using a two-component Gaussian Mixture Model [20, 30]:
p(`|θ) = K∑
k=1
βkφ(`|k), (3)
where βk and φ(`|k) are the mixture coefficient and the probability density of the k-th component, respectively. Based on the memorization effect of DNNs, we treat the component with a lower mean value (i.e., lower loss) as the clean set, and the other as the noisy set. To optimize the GMM, we adopt the Expectation-Maximization algorithm. Moreover, we compute the posterior probability wi = p(k|`i) = p(k)p(`i|k)/p(`i) as the clean probability of i-th sample, where k is the Gaussian component with the lower mean. By setting a threshold to {wi}Ni=1, we divide the data into clean and noisy subsets. For simplicity, we set the threshold to 0.5 through all experiments.
As observed in [10], it probably introduces error accumulation if the neural network is trained in a self-divide manner. To avoid such a situation, we adopt the co-teaching paradigm. Specifically, we individually train two networks A = {fA, gA, SA} and B = {fB , gB , SB} with different initializations and batch sequences. At each epoch, the network A or B will model its per-sample loss distribution with a GMM and divide the dataset into clean and noisy subsets which are then used for training the other network, i.e., co-divide. Note that, before co-divide, a warmup process is conducted on all training data to achieve initial convergence with Lw as defined in Eq. 2.

heading: 3.2 Co-Rectify
text: For either of A and B, the data D will be divided into the clean subset Sc = {(Ici , T ci , yci , wi)} Nc i=1 and noisy subset Sn = {Ini , Tni } Nn i=1. Then, the co-rectify module will correct the labels to recall the possible true positives from Sn and eliminate the negative impact of the possible false positives in Sc. Formally, the network k (k ∈ {A,B}) will rectify the labels of {Sc,Sn} into {Ŝc, Ŝn} for training itself. The rectified labels are determined by:{
ŷci = wiy c i + (1− wi)P k(Ici , T ci ), ∀(Ici , T ci , yci , wci ) ∈ Sc
ŷni = (P A(Ini , T n i ) + P B(Ini , T n i ))/2, ∀(Ini , Tni ) ∈ Sn
(4)
where PA(I, T )/PB(I, T ) denotes the predictions given by the network A/B. The roles of Eq. 4 are as below. On the one hand, as most pairs of Sc are true positive, Eq. 4 will use the original labels yci together with the model’s prediction P (I c i , T c i ) to rectify the correspondence. On the other hand, as most pairs of Sn are false positive, Eq. 4 will discard the original labels and rectify the labels by averaging the predictions P (Ini , T n i ) from the networks A and B.
Another key contribution of Eq. 4 is designing the prediction function P (I, T ) that could accurately predict whether the given pairs are positive or negative. Unlike the tasks like classification, image-text matching aims at computing the similarity rather than predicting the label of given image-text pairs. To this end, a straightforward approach is to predict the pairs by setting a threshold on the similarity. However, such a method requires to specify the threshold value, which is a daunting task because the optimal value is actually the similarity boundary of positive and negative pairs and hard to be manually specified. Alternatively, the following adaptive prediction function P (I, T ) is proposed, which could work in a data-driven way,
P (I, T ) = Θ(s)/τ
s = S(I, T )− (1 b ∑ T̂ S(I, T̂ ) + 1 b ∑ Î S(Î , T ))/2, (5)
where b is the batch size, Θ(·) clamps the elements into the range of [0, α], s is the similarity margin between the given pair (I, T ) to the mean of the negatives in a mini-batch, τ is the average similarity
margin of the largest 10% pairs in terms of s. This implies that the data have at least 10% clean pairs which could be regarded as a similarity anchor for prediction. Intuitively, the pairs with the similarity margin larger than τ would be predicted as 1, otherwise [0, 1).
Algorithm 1: Noisy Correspondence Rectifier Input: A given training data D, matching models A = (fA, gA, SA) and B = (fB , gB , SB)
1 Warmup the model (A,B) using Lw. 2 for n=1:num_epoch do 3 WA = {wAi }Ni=0 ← GMM(D, B) 4 WB = {wBi }Ni=0 ← GMM(D, A) 5 for k={A, B} do 6 Skc = {(Ii, Ti, yi, wi)|wi ≥ 0.5,∀(Ii, Ti, yi, wi) ∈ (D,Wk)} 7 Skn = {(Ii, Ti)|wi < 0.5,∀(Ii, Ti) ∈ (D,Wk)} 8 for j=num_steps do 9 Sample a mini-batch (Bcj ,Bnj ) from (Skc , Skn);
10 Rectify the labels of (Bcj ,Bnj ) into (B̂cj , B̂nj ) using Eq. 4–5; 11 Train the network k on (B̂cj , B̂nj ) by optimizing Lsoft.
Result: Matching models (A,B)

heading: 3.3 Robust Cross-modal Matching
text: Exiting cross-modal matching methods can only handle the binary labels which are incompatible with the soft labels rectified by NCR. To achieve robust image-text matching, we propose a novel triplet loss Lsoft by recasting the rectified labels as the soft margin. Mathematically,
Lsoft(Ii, Ti) = [α̂i − S(Ii, Ti) + S(Ii, T̂h)]+ + [α̂i − S(Ii, Ti) + S(Îh, Ti)]+, (6)
where Îh = argmaxIj 6=IiS(Ij , Ti) and T̂h = argmaxTj 6=TiS(Ii, Tj) are the most similar negatives in the mini-batch for a given positive pair (Ii, Ti) similar to VSE++ [8]. The soft margin α̂i is adaptively determined by:
α̂i = mŷi − 1 m− 1 α, (7)
where m is the curve parameter, and ŷi is the rectified label. The above formulation is designed to achieve the following goal, i.e., α̂i will be assigned a small value if ŷi is close to 0, and a large value otherwise. Thanks to Eq. 6–7, the similarity of the pair (I, T ) will be larger than that of the negatives by an adaptive margin α̂i.
Despite the adaptive margin, another major difference between Lw and Lsoft is that Lsoft will use the hard negatives which are the most similar negative pairs. Although the hard negatives are helpful in improving the performance, Lw cannot be beneficial from it due to the existence of noisy correspondence. Specifically, it is expected that only the similarity of the true positives is larger than that of the hard negatives. However, in the case of noisy correspondence, the similarity of the false positives will also be larger than that of the hard negatives, thus leading to the unavailability of the hard negatives for Lw during the co-divide stage. The detail of NCR is presented in Algorithm. 1.

heading: 3.4 Discussions on Matching Loss
text: To achieve robust cross-modal matching with the refined soft labels, we design a soft Triplet loss by recasting the labels into soft margins. Recently, there are some works have been proposed to handle the soft labels in the matching model. For example, Wray et al. [38] recast the soft similarity into binary labels by directly setting a threshold on the predicted similarity. Kim et al. [15] proposes a log-ratio matching loss with a regularization defined by the label distance ratio, which is computed by the continuous labels. Liu et al. [25] introduces the hubness problem in image-text matching and proposes to consider all samples in a mini-batch and weights them according to both local and global statistics. Wray et al. [38] recasts the soft similarity into binary labels by directly setting a threshold on the predicted similarity. Different from them, NCR proposes to recast the rectified soft labels into
soft margins in the triplet loss by assigning large margins to the true positive pairs and small ones to the false positive pairs. As a result and more importantly, our loss is specifically designed to solve the noisy correspondence problem whereas existing ones are not.

heading: 4 Experiment
text: In this section, we carry out experiments to verify the effectiveness of NCR in robust image-text matching. In the experiments, we use three benchmark datasets including Flickr30K [42], MS-COCO [23], and Conceptual Captions [35]. Among them, Conceptual Captions is with real noisy correspondence from the wild, and Flickr30K and MS-COCO are with simulated noisy correspondence.

heading: 4.1 Datasets and Performance Measurements
text: Three datasets are used in our experiments. To be specific, Flickr30K contains 31,000 images collected from the Flickr website with five captions each. Following [19], we use 1,000 images for validation, 1,000 images for testing, and the rest for training. MS-COCO contains 123,287 images with five captions each. We follow the data partition in [19] which consists of 113,287 training images, 5,000 validation images, and 5,000 test images. As Flickr30K and MS-COCO are well annotated, we simulate the noisy correspondence by randomly shuffling the captions of training images for a specific percentage, denoted by noise ratio. Conceptual Captions is a large-scale data consisting of 3.3M images with a single caption each. As this data set is harvested from the Internet, about 3% ∼ 20% correspondences are incorrect [35]. In our experiments, we use a subset of Conceptual Captions for evaluation, named CC152K. Specifically, we randomly select 150,000 samples from the training split for training, 1,000 samples from the validation split for validation, and 1,000 samples from the validation split for testing.
Following [19], for all images, we take the Faster-RCNN [33] detector provided by [1] to extract the top 36 region proposals of which each is encoded as a 2048-dimensional feature. For evaluation, we take the recall at K (R@K) as the measurement. In short, R@K is the fraction of queries for which the correct item is retrieved in the closest K points to the query. In the experiments, we report R@1, R@5, and R@10 for a comprehensive evaluation.

heading: 4.2 Implementation Details
text: NCR is a general framework which could enable almost all existing cross-modal matching methods robust against noisy correspondence. To verify the effectiveness of our framework, SGR [7] is chosen to guarantee the robustness because it is the state of the art in image-text matching. In brief, the image regions and words are projected into a shared embedding space through a full-connected layer (i.e., f ) and a Bi-GRU [34] (i.e., g), respectively. For the similarity function S, it will compute the similarity between the given image and text by combining the local and global features with the help of a graph reasoning technique proposed in [18]. Due to the space limitation, we leave more details and results in the supplemental material.
We train our network using the Adam optimizer [16] with the default parameters and a batch size of 128. For fair comparisons, the networks f and g are the same with SGR, i.e., the word embedding size is 300 and the joint embedding space size is 2048. In addition, we fix the margin α = 0.2 and m = 10 for the soft margin through the experiments. At the inference stage, we average the similarities predicted by network A and B for the retrieval evaluation. To avoid overfitting, we choose the best checkpoint in terms of the sum of the recalls on the validation set.

heading: 4.3 Comparisons with State of The Arts
text: In this section, we conduct comparisons on the three datasets. The baselines include SCAN [19], VSRN [21], IMRAM [5], SGRAF, SGR and SAF [7]. For Flickr30K and MS-COCO, we report the results with three different noise ratios, i.e., 0%, 20%, and 50%. In addition, we also report the results of SGR on the clean Flickr30K and MS-COCO by discarding the noisy pairs, denoted by SGR-C. Clearly, SGR-C is a quite strong baseline since the used data does not contain noisy correspondence. We do not report the results of SGRAF and SAF on the clean datasets since our framework only extends SGR in this paper.
When the noise rate is 0%, we directly refer to the results reported in the corresponding papers. For the noisy cases, we train the baseline models with the recommended settings three times and report the best result. Note that for SGR, we found it is very sensitive to the noisy correspondence, as shown in Table. 2. To obtain a desirable result, we experimentally employ a pre-training process to SGR (denoted by SGR*), namely, training the model with the vanilla triplet loss without hard negatives and then following the standard pipeline of SGR.
Results on Flickr30K & MS-COCO. Table 1 shows the quantitative results on Flickr30K and MS-COCO. Note that for MS-COCO, we only report the results by averaging over 5 folds of 1K test images due to space limitation, and leave the results on the full 5K test images in the supplemental material. From the results, one could observe that NCR is competitive to SGRAF in the noisefree case, namely, NCR could achieve state-of-the-art performance even though it is proposed to achieve robustness. When the data is contaminated by the noisy correspondence, NCR remarkably outperforms all the baselines by a large margin. Even comparing with SGR-C which is trained on the clean data, NCR improves R@1 by 2.2%, 3.1%, 2.3%, and 2.9% in these four valuations.
Results on CC152K. Table 2 shows the quantitative results on the CC152K which is with real noisy correspondences. From the results, one could see that our NCR consistently outperforms the evaluated models by a considerable margin in terms of all metrics. Specifically, NCR is 4.5% and 5.4% higher than the best baseline in terms of R@1 in text and image retrieval, respectively. Moreover, the large performance gap between SGR and SGR* shows the noise sensitivity of the original SGR.

heading: 4.4 Comparison to pre-trained model
text: In this section, we perform comparison to the large pre-trained model CLIP [31]. In brief, CLIP is trained on a massive dataset harvested from the Internet and thus presumably has a lot of noisy
image-text pairs. Such a comparison is helpful in understanding, big data based model (CLIP) or noisy correspondence modeling technique (NCR), which one is more favorable to handle the mismatching problem. More specifically, CLIP claims that using hundreds of million data could ignore the existence of possible noise, while we believe that a well-designed algorithm is essential to solve the noisy correspondence. Noticed, although some existing works including CLIP have slightly/indirectly realized the existence of the noise, NONE of them explicitly give a solution to solve this problem and explores the characteristics of the noise correspondence.
In the experiments, we conduct CLIP on the MS-COCO dataset under the following two settings, i.e., Zero-shot and Fine-tune. In brief, the first setting directly uses the released pre-trained CLIP to perform inference on MS-COCO, and the second fine-tunes the pre-trained model using the noisy training data of MS-COCO. As CLIP only released some pre-trained models and inference code 3, we use the non-official code 4 to fine-tune the model with 32 epochs for the fine-tune setting. Note that CLIP (ViT-L/14†) is unreleased and we report the results from the original paper [31]. One could observe that, although CLIP utilizes 400 million image-text pairs for pre-training, its performance inevitably degenerates during fine-tuning. In contrast, NCR achieves the matching performance with the presence of noisy correspondence, indicating the necessity of algorithm design.

heading: 4.5 Experimental Analysis
text: In this section, we first conduct experiments to show the robustness and generalizability of the proposed method. Then, we investigate the effect of co-divide and co-rectify with the visualization results. After that, we carry out the ablation study to verify different components of NCR. Finally, we visually demonstrate some noisy cases detected by NCR.

heading: 4.5.1 Study on Robustness and Generalizability
text: To show the robustness of NCR, we conduct experiments on Flickr30K by increasing the noise ratio from 0% to 60% with an interval of 10%. In addition, to verify the generalizability of NCR to other image-text matching methods, we extend SCAN [19] by NCR, denoted by NCR-SCAN. Fig. 4 shows that NCR and NCR-SCAN perform more stable than SGR and SCAN with increasing noise ratio. Moreover, NCR (NCR-SCAN) is remarkably superior to SGR (SCAN) in all tests, which shows the generalizability of NCR.

heading: 4.5.2 Visualization on the Co-divide and Co-rectify
text: To further investigate the influence of the co-divide and co-rectify modules in our method, we carry out experiments on the Flickr30K dataset by visualizing the per-sample loss distribution and the model predictions on the noisy data, where the noisy ratio is 20%. For better visualization, here we show the result of NCR-SCAN and leave the result of NCR in the supplemental material. As shown in Fig. 3(b), the loss of most noisy samples is larger than the clean loss, which verifies the memorization effect of DNNs. By fitting the per-sample loss with GMM, NCR could effectively divide the data into clean and noisy splits. Regarding the analysis on the co-rectify, Fig. 3(c) shows
3https://github.com/OpenAI/CLIP 4https://github.com/Zasder3/train-CLIP-FT
that the rectified soft labels of most clean pairs range into [0.3, 1] and those of most noisy pairs range into [0, 0.5]. In other words, one could enforce the similarity of true positives larger than that of the negatives during training, thus eliminating the negative impact of the noisy correspondence.

heading: 4.5.3 Ablation Study
text: In this section, we carry out the ablation study on the Flickr30K with the noise ratio of 50%. As shown in Table 4, all these three components are important to achieving encouraging results.

heading: 4.5.4 Noisy Samples
text: Fig. 4 shows some noisy CC152K examples identified by NCR. As shown, the first four image-text pairs are completely unrelated, which will be successfully detected by NCR. For the last pair, it will also be detected as noisy correspondence even though the visual and textual modalities are correlated at a coarse-grained level, e.g., both the image and text involve “beach".

heading: 5 Conclusion
text: This paper could be the first attempt to study a new problem in cross-modal matching, i.e., the noisy correspondence which could be a potential new direction in noise label. To solve this problem in cross-modal matching, we propose rectifying the noisy correspondence by an adaptive prediction function and a novel triplet loss with a soft margin to achieve robust cross-modal matching. Extensive experiments verify the effectiveness of the proposed method in handling synthesized and real noisy correspondences.
Broader Impact Statement
Cross-modal matching is a fundamental topic in multimodal learning, which could be applied to a wide range of applications including data retrieval, recommender systems, and vision-and-language understanding. This work could be one of the first works to aware of the importance and existence of the noisy correspondence problem in numerous applications. There are many benefits to solving the noisy correspondence problem, e.g., reducing the costs for manually annotating and aligning data; more data could be collected and used even though some of them are incorrectly aligned. Besides the benefits, it should pay attention on the potential negative impacts including but not limited to 1) The risk of automation bias [28] for decision making from the data bias, especially in aviation, health care, and autonomous vehicles. 2) The job loss caused by the NCR since it makes possibility to automatically correct the noisy correspondence, thus remarkably reducing the cost of human labor. We would encourage further work to understand and mitigate the above biases and risks.

heading: Acknowledgements
text: The authors would thank to the anonymous reviewers whose valuable suggestions and constructive comments remarkably improve this work. This work was supported in part by the National Key R&D Program of China under Grant 2020AAA0104500; in part by the Key Research and Development Program of Sichuan Province under Grant 2019YFG0497; in part by NFSC under Grant U21B2040, 62176171, U19A2078, 61625204, and 61836006; in part by the Fund of Sichuan University Tomorrow Advancing Life.

Review summaries:
This paper introduces a novel task, noisy correspondence of cross-modal retrieval tasks. The noisy correspondence is similar to the noisy label in object classification tasks, where the noisy correspondence scenario assumes that some image-caption pairs are incorrectly matched. To solve the problem, this paper proposes a Noisy Correspondence Rectifier (NCR), which could be considered as the novel extension of Co-teaching [9] to cross-modal retrieval tasks. Note that directly applying Co-teaching to cross-modal retrieval is non-trivial as the authors discussed in the paper (L105-115), because noisy label methods assume the existence of the prediction score function of the given input, while cross-modal retrieval methods usually do not have such per-sample score function. Hence, the proposed NCR first selects (co-divides) clean and noisy samples by using triplet loss values and the 2-mean Gaussian Mixture Model (GMM). Using the divided samples, NCR co-rectifies the noisy labels by the prediction function, where the prediction function is defined by the value of triplet distances. Using the extracted clean/noisy samples and rectified labels, NCR applies the hardest negative mining triplet loss to train the model, while the margin value is adaptive to the rectified labels. The experimental results show the effectiveness of NCR in terms of robustness to the noisy correspondence, e.g., achieving nearly dropped retrieval performances when 50% of correspondences are incorrect.

This paper focuses on cross-modal matching with the presence of noisy correspondence. The author proposes a novel method for learning with noisy correspondence to achieve robust cross-modal matching based on the SGR. In brief, it first splits the data into clean and noisy subsets in the manner of co-teaching and rectifies the labels with an adaptive prediction function. Extensive experiments on three datasets show the effectiveness of the proposed method.

The authors propose a method for cross-modal retrieval in the presence of noisy image-text pairings. While there has been a lot of work in learning in the presence of noise in this space, existing works mainly handle the problem of "label" noise - i.e. when semantic categories / labels on the data are available, but are noisy. In contrast, the authors tackle the unsupervised setting where all that is available are image-text pairs harvested from the web. Some of these pairs may be noisy and some may not be. The idea of the paper is straightforward. The authors use a technique commonly used to address noisy labels - i.e. co-teaching. They train two networks, A and B on the same data. After each epoch, the authors collect the similarity scores computed by each of the networks and use it to train a Gaussian Mixture Model. The reason for this is the GMM allows one to calculate the probability that a sample is from a given distribution. They model it with two Gaussians. The idea is that the GMM will fit one to the set of data that has lower similarity (hence noisy) and a separate Gaussian to the one with high similarity (hence clean). They then use the learned similarities predicted by each model to "correct" the labels used for training the other model (for example, treating some data that was considered clean as noisy or vice versa) based on the co-teacher's predictions. Finally, they integrate the similarity / confidence that the model is clean or not into the triplet loss as an adaptive margin, rather than use a static margin. The authors experimentally validate their method on MSCOCO, FlickR30K, and Conceptual Captions and compare against a number of baselines showing reasonable improvement.

This paper proposes a noisy label learning framework for cross-modal learning, where the training data consists of wrongly matched modalities. The proposed NCR first divides the data into clean and noisy splits based on the difference of loss distribution. Then NCR proposes to rectify the labels with a prediction function and further recast them into soft margins in the triplet loss. Extensive experiments and comparisons are conducted to verify the effectiveness of the method.