Learning with Noisy Correspondence for Cross-modal Matching
Authors: Zhenyu Huang
Guocheng Niu
Xiao Liu
Wenbiao Ding
Xinyan Xiao
hua wu
Xi Peng
Venue: NeurIPS 2021 Oral (2021)
Keywords: multimodal learning
noisy labels
cross-modal matching
mismatching

Abstract: Cross-modal matching, which aims to establish the correspondence between two different modalities, is fundamental to a variety of tasks such as cross-modal retrieval and vision-and-language understanding. Although a huge number of cross-modal matching methods have been proposed and achieved remarkable progress in recent years, almost all of these methods implicitly assume that the multimodal training data are correctly aligned. In practice, however, such an assumption is extremely expensive even impossible to satisfy. Based on this observation, we reveal and study a latent and challenging direction in cross-modal matching, named noisy correspondence, which could be regarded as a new paradigm of noisy labels. Different from the traditional noisy labels which mainly refer to the errors in category labels, our noisy correspondence refers to the mismatch paired samples. To solve this new problem, we propose a novel method for learning with noisy correspondence, named Noisy Correspondence Rectifier (NCR). In brief, NCR divides the data into clean and noisy partitions based on the memorization effect of neural networks and then rectifies the correspondence via an adaptive prediction model in a co-teaching manner. To verify the effectiveness of our method, we conduct experiments by using the image-text matching as a showcase. Extensive experiments on Flickr30K, MS-COCO, and Conceptual Captions verify the effectiveness of our method. The code could be accessed from www.pengxi.me .

Learning with Noisy Correspondence for Cross-modal Matching
Zhenyu Huang" College of Computer Science Sichuan University, China zyhuang. gm@gmail. com
Guocheng Niu Baidu Inc., China niuguochengbbaidu.com
Xiao Liu TAL Education Group liuxiao15@tal.com com
Wenbiao Ding TAL Education Group d dingvenbiao@tal.com com
Xinyan Xiao
Hua Wu
Baidu Inc., China
Baidu Inc., China
xi mlaoxinyan@baidd.comm
wu__hu@bbaidu.cm com
Xi Peng
College of Computer Science
Sichuan University, China
margx..g@@mmail..mm com
Abstract
Cross-modee matching, which aims to establish the correspondence between two different modalities, is fundamental variety of tasks such cross-modal retrieval and vision-and-language understanding. Although a huge number of cross- modal matching methods have been proposed and achieved remarkable progress in recent years, almost all of these methods implicitly assume that the multimodal training data are correctly aligned. In practice, however, such an assumption is extremely expensive even impossible to satisfy. Based on this observation, we reveal and study latent and challenging direction in cross- modall matching. named noisy correspondence, which could be regarded as a new paradigm of noisy labels. Different from the traditional noisy labels which mainly refer the errors in category labels, our noisy correspondence refers to the mismatch paired samples. To solve this new problem, we propose novel method for learning with noisy correspondence, named Noisy Correspondence Rectifier (NCR). In brief, NCR divides the data into clean and noisy partitions based on the memorization effect of neural networks and then rectifies the correspondence via adaptive prediction model in co-teaching manner. To verify the effectiveness of our method, we conduct experiments by using the image-text matching as showcase. Extensive experiments on Flickr30K, MS-COCO, and Conceptual Captions verify the effectiveness of our method. The code could be accessed from www. www.xengxi me
1 Introduction
As one of the most fundamental techniques in multimodal learning, cross-modal matching aims to bridge different modalities. In recent years, some cross modal matching methods [19 [I 26 have been proposed based on Deep Neural Networks (DNNs), which achieved remarkable progress in
"Some parts the work was done while Zhenyu Huang was an internship at Baidu Inc.
i "Coresppndiigg author.
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

variety of applications, such as clustering 29 24 image/video captioning [ 44 22 cross-modal
retrieval [40] 19 13,, and visual question answering
In general, most existing cross-modal matching methods embed different modalities into a common space wherein the similarity of positive cross-modal pairs is maximized and that of the negative ones is minimized. Although these methods have achieved promising results, their success depends on an implicit data assumption, i.e., the training data are correctly aligned across modalities. For example, in the vision-and-languaee tasks, the text needs to accurately describe the image content, and vice versa. In practice, however, it extremely expensive and time-consuming to annotate or collect such data pairs. Especially, considering the data collected from the Internet BS [4]. it is inevitable to collect some mismatched pairs which are wrongly treated as the matched ones. To the best of our knowledge, such special noisy label (correspondence) problem has been ignored so far, which will remarkably degrade the performance of matching methods as shown our experiments.
Trae poottee
The segahe
Tale posttoe
Images
Labels
baneh arunng head
ball
nergin
paysz
aqgather
Dog
Simbbnty
Noisy Labels
Noisy Correponndecce
Cross-modal Matching
Noisy labels vs. Noisy Correspondence. We denote the noisy samples with red lines and clean samples with green lines. The traditional noisy labels mainly refer to the errors in category labels, while the noisy correspondence refers to the alignment errors in paired data. For the noisy correspondence in cross-moda matching, the true positive pair correctly guides the cross-modal matching, while the false positive pair causes incorrect supervision for training.
Figure 1:
Based on the above observation, we reveal new paradigm for the noisy labels, named noisy correspondence. Different from the traditional noisy labels, the noisy correspondence refers to the aligmment errors paired data rather than the errors in category annotations (see Fig. D). To the best of our knowledge, there is no effort has been devoted to study this new problem and the closest paradigm might be the partially view-aligned problem (PVP) remarkably different from noisy correspondence, and the latter is more practical than the former. To be specific, PVP focuses on that the cross-mode alignment is totally unavailable, whereas the noisy correspondence focuses on that some correspondences are incorrect. In addition, PVP assumes that some correctly aligned data are available for training, whereas our noisy correspondence assumes that the clean and noisy data are mixed.
4T]. However, PVP is
To solve the noisy correspondence problem in cross-modal matching, we propose a novel method, named Noisy Correspondence Rectifier (NCR). Our method is based on the memorization effect of DNNs observed in [3 39], i.e., DNNs tend to learn the simple patterns before fitting noisy samples. Motivated by this empirical observation, NCR divides the data into two relative accurate data partitions, i.e., "noisy" and "clean" subsets, based on their loss difference. After that, NCR employs an adaptive prediction function for label rectifying so that the false positives and the true positives could be identified from the "clean" and the "noisy" subsets, respectively. Furthermore, we propose novel triplet loss for robust cross-modal matching by recasting the rectified labels as the soft margin.
The main contributions and novelties of this paper could be summarized as below. i) We reveal a new problem in cross-modal analysis, which is also a new paradigm for noisy labels, termed noisy correspondence. Diffeerntt from the traditional noisy labels, the noisy correspondence refers to the alignment errors in paired data instead of the errors in category annotations. To the best of our knowledge, this work could be the first study on this problem. ii) To solve the noisy correspondence problem, we propose novel method for learning with noisy correspondence, named Noisy Corre- spondence Rectifier (NCR). One major novelty of NCR is that the rectified label is elegantly recasted as the soft margin of a triplet loss so that the robust cross-modal matching could be achieved. iii) To verify the effectiveness of our method, we conduct experiments on the image-text matching task. Extensive experiments on three challenging datasets verify the effectiveness of our method in the synthesized and real noises.
2

---

2
Related works
In this section, we briefly introduce some recent developments in cross-modal matching and learning with noisy labels.
2.1 Cross-modal Matching
Most existing cross-modal matching works seek to learn a common space wherein different modalities comparable. In general, existing works could be roughly divided into two categories: 1) Coarse- grained Matching. It often utilizes multiple neural networks to compute a global feature and each network used for a specific modality [I7 37 8.. For example, Kiros al. [17 use a Convolutional Neural Network (CNN) and a Gated Recurrent Unit (GRU) to obtain the image and text features, while enforcing the similarity of positive pairs larger than that of the negative ones. To further boost the matching performance, VSE++ 8 uses some representative negatives to improve the discrimination of the model. 2) Fine-grained Matching. It seeks measure the fine-graing similarity for cross- moosll matching [1 21 [7] For example, SCAN [19] proposes learning the latent semantic correspondence between the image regions and words that are extracted by bottom-up attention [ and GRU, respectively. VSRN (21 adopts a graph convolutional network for semantic reasoning.
SGRAF 7 proposes constructing a similarity graph to reason the similarity and adopting an attention filtration technique to eliminate the less-meaningful alignments. Recently Chun et al. [6] introduce new paradigm for cross-modal matching, i.e. possible many-to-many correspondence that existed in the image and captions. To achieve this, they propose to use probabilistic representations to model the possible one-to-man correspondence.
Although promising results have been achieved in recent years, the existing methods heavily rely on the correctly aligned data. In practice, however, such well-matcher data is expensive and time- consuming to collect. Moreover, some recent works [35 14] show that large-scale dataset collected from the wild could remarkably improve the performance of the model. However, such data will inevitably contain some mismatched pairs. Hence, it is highly expected to develop some methods which are robust against the noisy correspondence, which has not been studied as far as we know. Different from the many-to-many correspondence [6] between image and captions, NCR reveals the noisy correspondence problem which refers to the alignment errors of image-text pairs and proposes to eliminate the negative impact from noisy pairs for downstream tasks.
2.2 Learning with Noisy Labels
To handle the possible noisy annotations in the training data, a large number of methods have been proposed and almost all of them focus on the classification task [36 27]. To reduce the negative impact of the noisy labels, the existing works often resort to robust architecture design, regularization, loss adjustment, or sample selection methods. Here, we mainly introduce the last two approaches which are most related to this work. To be specific, the loss adjustment achieves robustness by adjusting the contribution of clean and noisy samples w.r.t. the loss. For example, Reed et (32 proposed a bootstrapping loss based on the model predictions for loss correction. Zhang et al. 445 provided some theoretical explanations for the label correction along with a new label correction algorithm. Different from the loss adjustment methods, sample selection methods aim to select clean samples from a noisy dataset. For example, Arpit et al. B showed that DNNs tend to learn simple patterns before fitting noisy samples, namely the memorization effect. Motivated from this, Arazo et al. (2 proposed treating the samples with small loss as the clean samples. To avoid the selection bias of clean samples, Co-teaching methods [10 4 use the samples with small loss to iteratively train two networks. In recent, DivideMix (20) adopts the MixMatch method for cemi-supervised learning with the clean and noisy samples.
Unlike the above noisy label studies, this paper focuses on the noisy correspondence problem which considers mismatched multimodal data pairs instead of incorrectly annotated data points. Besides the difference in the problem, this work is also different from the aforementioned studies in the methodology. To be specific, in cross cross-modal matching, it is impossible to directly adopt these noise label learning methods to solve the noisy correspondence problem due to the following two reasons. First, most of the noisy label learning methods propose to use the model's prediction for label rectifying in the scenario of classification, while is intractable to directly predict the correspondence of given pairs in matching models. Second, even if we can rectify the noisy correspondence somehow,
3

---

Iuput
Enkellng
Sumiluity Measuring
Warsup
B
Leaft
Prodiction
Divide
51-(8" &
9+ 18 S
S
SULY)
Andlar
Rectily
(St.8t
(+- (66 &
10
Salt
Training
A
B
9
(a) Training pipeline
(b) Robust image-text mutching network (A and !I)
Figure 2: Overview of the proposed method. (a) Training pipeline of NCR. NCR consists of two individual networks (A, B) which work in the manner of co-teaching. In brief, NCR first warmup the networks (A, B) on the original training data using the loss Lu which is also used for per-sample loss computation. Then, based on the memorization effect of DNNs, NCR divides the training data into clean and noisy subsets at each epoch using either A or B, i... S = (S, S and S = (
After that, NCR will co-rectify the correspondence of {SA, (88) and obtain {SA, , SB) using an adaptive prediction function. Finally, S and 8 will be used to train the network B and A in swapping way. (b) Robust image-text matching network. For example, the network A projects the image and text by the modal-specific networks f and respectively. Then the similarity S(1,7) computed on the extracted features fI) and (T)). To achieve robust image-text matching, the rectified soft labels are recast as the soft margin of our loss Lzoft- As shown, for a given anchor, ft will enforce the true positive closer to it than the negative by large margin d1, and meanwhile the false positive will has a small margin á2
the rectified real-valued labels are incompatible with the existing matching methods since most of them assume the given labels are binary. To address these problems, NCR proposes an adaptive prediction function and a novel triplet loss by recasting the soft labels soft margins.
The Proposed Method
In this section, we elaborate on the proposed method, i.e., Noisy Correspondence Rectifier (NCR) which could be the first work to solve the noisy correspondence problem in cross- matching. In Section [3.T] we introduce the co-divide module which splits the training data into the clean and noisy subsets. After that, we introduce how to rectify the labels with an adaptive prediction function in Section 3.2 Finally, we detail how to combine the co-divide and co-rectify modules to achieve robust cross-modal matching in Section 3.3
3.1 Co-divide
Without loss of generality, we first introduce the cross- matching task by taking the image-text matching as a showcase. Given the training data D {[,,, Te, y/) ) where N the data size, (I,T;) is an image-text pair and yi E {1,0} indicates that the pair belong to the same instance (positive) or not (negative). For the noisy correspondence case, defines that an unknown portion of D is mismatched, i.e., (T,,7 T;) is negative pair but wrongly labeled as y; = 1. To solve such noisy correspondence problem, we propose NCR to achieve robust cross-modal matching.
To begin, we project the visual and textual modalities into a shared space via two modal-specific networks f and g. respectively. Then the similarity of the given image-text pairs computed through S(S(I), (T)). For simplicity, we denote ()) g(T)) as S(1,T) in the following. Some early empirical studies B show that DNNs tend to first learn simple samples and then gradually fit the noisy samples. This so-call so-called memorization effect of DNNs will lead to relatively low loss for the clean samples. Motivated by this, we utilize the difference samples and noisy samples to divide the training data like [10 43 matching model (J. we compute the per-sample loss through:
loss distribution between the clean Specifically, given
2
20
((G..)) ( N
(I)

---

where Lu is defined as:
Lu [w,(T,) = Ela +(1,) T
S(1, 1) + iia- S(T,T) + S(
(2)
where (I.T) is a positive pair, a > 0 denotes given margin, and [x] = mar(< 0). In the loss, the first term treats I as queries taking over all negative text T', while the second term treats T as queries taking over all negative images I. Then, we fit the per-sample loss of all training data by using
two-component Gaussian Mixture Model [20] 30]:
K
0(( Σ (LO(C))) k=1
(3)
where By and (((kk) are the mixture coefficient and the probability density of the k-thh component, respectively. Based on the memorization effect of DNNs, we treat the component with a lower mean value (i.e., lower loss) as the clean set, and the other as the noisy set. To optimize the GMM, we adopt the Expectation-Me Exxpetation-Maxxmiizttinn
algorithm. Moreover, we compute the posterior probability w, = p(k (.) (k))p((, (k)/p({) as the clean probability of i-th sample, where A the Gaussian component with the lower mean. By setting threshold to {w; {w:} J we divide the data into clean and noisy subsets. For simplicity, we set the threshold to 0.5 through all experiments.
As observed in [10], it probably introduces eπror accumulation if the neural network is trained in self-divide manner. To avoid such situation, we adopt the co-teaching paradigm. Specifically, we individually train two networks A = {f .g"" " } and B { ,", with different initializations and batch sequences. loss distribution with a GMM and divide the dataset into clean and noisy subsets which are then used for training the other network, i.e., co-divide. Note that, before co-divide, a warmup process conducted on all training data to achieve initial convergence with L as defined in Eq. 2
each epoch, the network A or B will model its per-sample
3.2 Co- Co-Rectify
For either of A and B. the data D will be divided into the clean subset S. = {14 TY, yft w;) N. and noisy subset S, = {1,"' T Jim1- Then, the co-rectify module will correct the labels to recall the possible true positives from Sn and eliminate the negative impact of the possible false positives in S.- Formally, the network & (k (A, B)) will rectify the labels of (Se, S.) into (S,, for training itself. The rectified labels are determined by:
8f = way. + (1 - w.) Pk (19,
YII TU wf E S.
(4)
807h (T.TT)
A(I TT) E S,
where PA T)/PB((PPPB)) (T,T) denotes the predictions given by the network A/B. B. The roles of Eq. are below. On the one hand, as most pairs of S. are true positive, Eq. wiil use the original labels of together with the model's prediction P( TT) ( to rectify the correspondence. On the other hand, as most pairs of S,, are false positive, Eq. will discard the original labels and rectify the labels by averaging the predictions P(I, T") from the networks A and B.
Another key contribution of Eq. 4 is designing the prediction function PTI, that could accurately predict whether the given pairs are positive or negative. Unlike the tasks like classification, image-text matching aims computing the similarity rather than predicting the label of given image-text pairs. To this end, a straightforward approach is to predict the pairs by setting a threshold on the similarity. However, such method requires specify the threshold value, which is a daunting task because the optimal value is actually the similarity boundary of positive and negative pairs and hard to be manually specified. Alternatively, the following adaptive prediction function PII, is proposed, which could work in a data-driven way.
P(I.T) = ((s)/T
S(ITT) 1
S(I,T)+ 1 E s T)/2, b i
1
(5)
where is the batch size, 0()) clamps the elements into the range of [0, a), s is the similarity margin between the given pair (I,T') to the mean of the negatives in mini-batch, is the average similarity
5

---

margin of the largest 10% pairs in terms of s. This implies that the data have at least 10% clean pairs which could be regarded similarity anchor for prediction. Intuitively, the pairs with the similarity margin larger than would be predicted as 1, otherwise [0, 1).
Algorithm 1:
Noisy Correspondence Rectifier
Input: A given training data D, matching models A = SAA) and B = (f SB) Warmup the model (A, B) using Leg-
2 for n= / :mmmmmmmmpppp
do
3
WA = {w; Ji=0 & GMM B)
4
nd = {w B
Ji=0 + GMM(D,A)
for k=/A, /A, B/do
S [(I,, Vi, wi)lw; 2 0.5, V1,, Bi, w;) W"))
S ((T,, lu) < 0.5, VJ,, (T) E (D, W*
for j=mum__stta stepssteppsspppppppp do
Sample mini-batch (Bj, BB) from (S 8::;;
Rectify the labels of (B5. 13 into 183. using Eq.
10
11
Train the network & on (B8) B by optimizing Leoft-
Result: Matching models (A, B)
3.3 Robust Cross-modal Matching
Exiting cross-modal matching methods can only handle the binary labels which are incompatible with the soft labels rectified by NCR. To achieve robust image-text matching, we propose a novel triplet loss Lsoft by recasting the rectified labels as the soft margin. Mathematically,
Luofe T.) = lā, S(,, T.) + 5(,, 71 + [ās S;), + s(h,,TT)l
(6)
where in = argmax S(L, T:) and Tú argmax FT; S(4,T) are the most similar negatives in the mini-batch for given positive pair (J,; T:) similar VSE++ The soft margin O;, adaptively determined by:
à. m
a
where is the curve parameter, and ÿJ is the rectified label. The above formulation designed to achieve the following goal, i.e., å; will be assigned a small value if ÿi is close to O, and a large value otherwise. Thanks to Eq. 6 the similarity of the pair (I.T) will larger than that of the negatives by an adaptive margin d;
Despite the adaptive margin, another major difference between Tap and Lo is that Lsoft will use the hard negatives which are the most similar negative pairs. Although the hard negatives are helpful in improving the performance, L cannot be beneficial from it due the existence of noisy correspondence. Specifically, it is expected that only the similarity of the true positives is larger than that of the hard negatives. However, the case of noisy correspondence, the similarity of the false positives will also be larger than that of the hard negatives, thus leading to the unavailability of the hard negatives for L during the co-divide stage. The detail of NCR is presented in Algorithm.
3.4 Discussions on Matching Loss
To achieve robust cross-modal matching with the refined soft labels, we design a soft Triplet loss by recasting the labels into soft margins. Recently, there are some works have been proposed to handle the soft labels in the matching model. For example, Wray et al. (38) recast the soft similarity into
binary labels by directly setting threshold on the predicted similarity. Kim et al. [15] proposes log-ratio matching loss with regularization defined by the label distance ratio, which is computed by the continuous labels. Liu et al.
(25) introduces the hubness problem image-text matching and proposes to consider all samples in a mini-batch and weights them according to both local and global statisties. Wray al. (8] recasts the soft similarity into binary labels by directly setting a threshold on the predicted similarity. Different from them, NCR proposes recast the rectified soft labels into

---

soft margins in the triplet loss by assigning large margins to the true positive pairs and small ones to the false positive pairs. As result and more importantly, our loss is specifically designed to solve the noisy correspondence problem whereas existing ones are not.
4 Experiment
In this section, we carry out experiments to verify the effectiivneess of NCR in robust image-text match- ing. In the experiments, we use three benchmark datasets including Flickr30K [42]. MS-COCO 23 and Conceptual Captions [35]. Among them, Conceptual Captions is with real noisy correspondence from the wild, and Flickr30K and MS-COCO are with simulated noisy correspondence.
4.1 Datasets and Performance Measurements
Three datasets are used in our experiments. To be specific, Flickr30K contains 31,000 images collected from the Flickr website with five captions each. Following [19], we use 1,000 images for validation, 1,000 images for testing, and the rest for training. MS-COCO contains 123,287 images with five captions each. We follow the data partition in [19] which consists of 113,287 training images, 5,000 validation images, and 5,000 test images. As Flickr30K and MS- MS-COCO are well annotated, simulate the noisy correspondence by randomly shuffling the captions of training images for specific percentage, denoted by noise ratio. Conceptual Captions is a large-scale data consisting of 3.3M images with single caption each. As this data set is harvested from the Internet, about 3% ~ 20% correspondences are incorrect [35]. In our experiments, we use subset of Conceptual Captions for evaluation, named CC152K. Specifically, we randomly select 150,000 samples from the training split for training, 1,000 samples from the validation split for validation, and 1,000 samples from the validation split for testing.
Following [9], for all images, we take the Faster-RCNi 33 detector provided by Q to extract the top 36 region proposals of which each is encoded as take the recall at K (R @@@ K) as the measurement. In short, @@K the fraction queries for which the correct item is retrieved in the closest K points to the query. In the experiments, we report R @ 1, R@5, and @@ 10 for a comprehensive evaluation.
feature. For evaluation,
4.2 Implementation Details
NCR is general framework which could enable almost all existing cross-modal matching methods robust against noisy correspondence. To verify the effectiveness of our framework, SGR (7] is chosen to guarantee the robustness because it is the state of the art in image-text matching. In brief, the image regions and words are projected into shared embedding space through full-connected layer (i.e., f) and a Bi-GRU (34) (i.e., g). respectively. For the similarity function S, it will compute the similarity between the given image and text by combining the local and global features with the help of a graph reasoning technique proposed in [18]. Due to the space limitation, we leave more details and results in the supplemental material.
We train our network using the Adam optimizer 16 with the default parameters and a batch size of 128. For fair comparisons, the networks and are the same with SGR, i.e., the word embedding size is 300 and the joint embedding space size is 2048. In addition, we fix the margin a = 0.2 and = 10 for the soft margin through the experiments. At the inference stage, average the similarities predicted by network A and for the retrieval evaluation. To avoid overfitting, we choose the best checkpoint in terms of the sum of the recalls on the validation set.
4.3 Comparisons with State of The Arts
In this section, we conduct comparisons on the three datasets. The baselines include SCAN [19] VSRN (1), IMRAM [5). SGRAF, SGR and SAF [7]. For Flickr30K and MO-COCO, we report the results with three different noise ratios, i.e. 0%, 20% and 50%. In addition, we also report the results of SGR on the clean Flickr30K and MS-COCO by discarding the noisy pairs, denoted by SGR-C. Clearly, SGR-C is a quite strong baseline since the used data does not contain noisy correspondence. We do not report the results of SGRAF and SAF on the clean datasets since our framework only extends SGR in this paper.
7

---

When the noise rate is 0%, we directly refer to the results reported in the corresponding papers. For the noisy cases, we train the baseline models with the recommended settings three times and report the best result. Note that for SGR, we found it is very sensitive to the noisy correspondence, as shown in Table. 2 To obtain desirable result, we experimentally employ a pre-training process to SGR (denoted by SGR*), namely, training the model with the vanilla triplet loss without hard negatives and then following the standard pipeline SGR.
Table 1: Image-Terr Text Retrieval on Flickr30K and MS-COCO 1K.
Flickr30K
MS-COCO
Image
Text
Text
Image
Image
Text
Text
Image
Noise
Methods
R@1 1
R@5
R
10
RØ
R@5
RØ 10
R@ 1
R@S
R I0
R@l
R@s
R 10
SCAN
67.4
90.3
95.8
48.6
77.7
85.2
69.2
93.6
97.6
56.0
86.5
93.5
VSRN
71.3
88.2
76.2
89.7
IMRAM
74.1
93.0
96.6
53.9
79.4
87.2
76.7
95.6
98.5
61.7
89.1
95.0
0%
SAF
73.7
93.3
96.3
56.1
81.5
88.0
76.1
95.4
98.3
61.8
89.4
95.3
SGR
75.2
93.3
96.6
56.2
81.0
86.5
78.0
95.8
98.2
61.4
89.3
95.4
SGRAF
77.8
94.1
97.4
58.5
83.0
88.8
79.6
96.2
98.5
63.2
90,7
96.1
NCR
77.3
94.0
97.5
59.6
84.4
89.9
78.7
95.8
98.5
63.3
90.4
95.8
SCAN
59.1
83.4
90.4
36.6
67.0
77.5
66.2
91.0
96.4
45.0
80.2
89.3
VSRN
58.1
82.6
89.3
40.7
68.7
78.2
25511
59.0
74.8
17.6
49.0
64.1
IMRAM
63.0
86.0
41.4
71.2
80.5
68.6
85.0
91.0
20%
SAF
51.0
79.3
88.0
38.3
66.5
76.2
67.3
92.5
96.6
53,4
84.5
92.4
SGR
62.8
86.2
92.2
44.4
72.3
80.4
67.8
91.7
96.2
52,9
83.5
90.1
SGR-C
72.8
95.4
82.1
75.4
88.5
94.8
NCR
75.0
93.9
97.5
58.3
83.0
89.0
77.7
95.5
98.2
62.5
89.3
95.3
SCAN
27.7
57.6
68.8
16.2
39.3
49.8
40.8
73.5
84.9
5.4
15.1
21.0
VSRN
14.3
37.6
50.0
12.1
30.0
39.4
23.5
54.7
69.3
16.0
47.8
65.9
IMRAM
9.1
2.7
8.4
12.7
21.3
22.3
52.8
64.3
50%
SAF
30.3
63.6
75.4
27.9
53.7
65.1
30.4
67.8
82.3
33.5
69.0
82.8
SGR
36.9
68.1
80.2
29.3
56.2
67.0
60.6
87.4
93.6
46.0
74.2
79.0
SGR-C
69.8
90.3
94.8
50.1
77.5
85.2
71.7
94.1
97.7
57.0
86.6
93.7
NCR
72.9
96.3
54.3
79.8
86.5
74.6
97.8
59.1
87.8
94.5
Results on Flickr30K & MS-COCO. Table [ shows the quantitative results on Flickr30K and MS-COCO. Note that for MS-COCO, only report the results by averaging over 5 folds of 1K test images due to space limitation, and leave the results on the full 5K test images in the supplemental material. From the results, one could observe that NCR is competitive to SGRAF in the noise- free case, namely, NCR could achieve state-of-the-a art performance even though it is proposed to achieve robustness. When the data is contaminated by the noisy correspondence, NCR remarkably outperforms all the baselines by large margin. Even comparing with SGR- C which is trained on the clean data, NCR improves R @ by 2.2%, 3.1 ,,, 2.3%. and 2.9% in these four valuations.
Results on CC152K. Table 2 shows the quantitative results on the CC152K which is with real noisy could see that NCR consistently outperforms the evaluated
correspondences. From the results, models by a considerable margin terms of all metrics. Specifically, NCR is 4.5% and 5.4% higher than the best baseline in terms of R@ 1 in text and image retrieval, respectively. Moreover, the large performance gap between SGR and SGR* shows the noise sensitivity of the original SGR.
Table 2: Image- Text Retrieval CC152K.
Image
Text
Text - Image
Methods
R@1
R@5
R@ 10
R@1
R@5
R@ 10
SCAN (ECCV' (ECCV'18)
30.5
55.3
65.3
26.9
53.0
64.7
VSRN (ICCV' 19)
32.6
61.3
70.5
32.5
59.4
70.4
IMRAM (CVPR'20)
33.1
57.6
68.1
29.0
56.8
67.4
SAF (AAAI'21)
31.7
59.3
68.2
31.9
59.0
67.9
SGR (AAAI21)
11.3
29.7
39.6
13.1
30.1
41.6
SGR* (AAAI'2))
35.0
63.4
73.3
34.9
63.0
72,8
NCR
39.5
64.5
73.5
40.3
64.6
73.2
4.4 Comparison to pre-trained model
In this section, we perform comparison to the large pre-trained model CLIP In brief, CLIP is trained on massive dataset harvested from the Internet and thus presumably has lot of noisy

---

image-text text pairs. Such a comparison is helpful in understanding. big data based model (CLIP) or noisy correspondence modeling technique (NCR), which one more favorable handle the mismatching problem. More specifically, CLIP claims that using hundreds of million data could ignore the existence of possible noise, while we believe that a well-designed algorithm is essential to solve the noisy correspondence. Noticed, although some existing works including CLIP have slighdy/indrreetly realized the existence of the noise, NONE of them explicitly give a solution to solve this problem and explores the characteristics of the noise correspondence.
In the experiments, we conduct CLIP on the MS-COCO dataset under the following two settings, i.e. Zero-shot and Fine-ture tine-tune. In brief, the first setting directly uses the released pre-trained CLIP to perform inference on MS- IOCCOOO, and the second fine-tunes the pre-trained model using the noisy training data of IS-COCO. As CLIP only released some pre-trained models and inference code we use the non-official code SadeBto to fine-tune the model with epochs for the fine-tune setting. Note that CLIP (ViT-L//14)) is unreleased and we report the results from the original paper [31]. One could observe that, although CLIP utilizes 400 million image-text pairs for pre-training, inevitably degenerates during fine-tuning, In contrast, NCR achieves the matching performance with the presence of noisy correspondence, indicating the necessity of algorithm design.
performance
Table 3: Comparison with CLIP on MS-COCO 5K.
Image + Text
Text - Image
Noise Ratio
Methods
R@D
R@5
R@
R@
R@5
R@ I0
CLIP (ViT-L/141)
58.4
81.5
88.1
37.8
62.4
72.2
0%, Zero-Shot
CLIP (ViT-B/32)
50.2
74.6
83.6
30.4
56.0
66.8
NCR
58.2
84.2
91.5
41.7
71.0
81.3
CLIP (ViT-B/32)
21.4
49.6
63.3
14.8
37.6
49.6
20%, Fine-tune
NCR
56.9
83.6
91.0
40.6
69.8
80.1
CLIP (ViT-B/32)
10.9
27.8
38.3
7.8
19.5
26.8
50%, Fine-tune
NCR
53.1
80.7
88.5
37.9
66.6
77.8
4.5
Experimental Analysis
In this section, first conduct experiments to show the robustness and generalizability of the proposed method. Then, we investigate the effect co-divide and co-rectify with the visualization results. After that, we carry out the ablation study to verify different components of NCR. Finally, visually demonstrate some noisy cases detected by NCR.
4.5.1 Study Robustness and Generalizability
To show the robustness of NCR, conduct experiments on Flickr30K by increasing the noise ratio from 0% to 60% with an interval of 10%. In addition, to verify the generalizability of NCR to other image-text matching methods, extend SCAN [19] by NCR, denoted by NCR-SČAN. Fig. 4shows that NCR and ICR-SCAN perform more stable than SGR and SCAN with increasing noise ratio. Moreover, NCR (NCR-SCAN) is remarkably superior to SGR (SCAN) in all tests, which shows the generalizability of NCR.
4.5.2 Visualization the Co-divide and Co-rectify
To further investigate the influence of the co-divide and co-rectify modules in our method, we carry out experiments on the Flickr30K dataset by visualizing the per-sample loss distribution and the model predictions on the noisy data, where the noisy ratio is 20%. For better visualization, here we show the result of NCR-SCAN and leave the result of NCR in the supplemental material. As shown in Fig. ((b] the loss of most noisy samples larger than the clean loss, which verifies the memorization effect of DNNs. By fitting the per-sample loss with GMM, NCR could effectively divide the data into clean and noisy splits, Regarding the analysis on the co-rectify, Fig. 3(c) shows

---

Mely Am
0 C.00
(a) The Noise Ratio
(b) Co-divide
(c) Co-rectify
Figure 3: (a) Retrieval performance of NCR and NCR-SCAN Flickr30K with varying noise ratio.
(b) Per-sample loss distribution and GMM fitting visualization after warmup. (c) Model predictions on the noisy subset.
that the rectified soft labels of most clean pairs range into [0.3, and those of most noisy pairs range into [0, 0.5]. In other words, one could enforce the similarity of true positives larger than that of the negatives during training, thus eliminating the negative impact of the noisy correspondence.
4.5.3 Ablation Study
In this section, carry out the ablation study the Flickr30K with the noise ratio of 50%. As shown Table these three components are important to achieving encouraging results.
Table 4: Ablation studies on Flickr30K with 50% noise.
Method
Image -> Text
Text + Image
Co-divide
Co-rectify
Warmup
@
R@5
R@10
@@]
R@5
@@ 10
72.9
93.0
96.3
54.3
79.8
86.5
71.4
90.8
95.7
54.1
80.3
86.5
16.0
38.4
51.7
12.6
31.4
42.8
0.3
0.6
1.0
0.2
0.5
1.1
4.5.4 Noisy Samples
shows some noisy CC152K examples identified by NCR. As shown, the first four image-text pairs completely unrelated, which will be successfully detected by NCR. For the last pair, it will also be detected as noisy correspondence even though the visual and textual modalities are correlated goannee level, e.g., both the image and text involve "beach".
Fig.
BALY
OUTSID
digital relected for the
look him, like work all
take look this
share with
family walking beach
Figure 4: Some noisy examples correctly divided by NCR.
5 Conclusion
This paper could be the first attempt study a new problem in cross-modee modal matching, i.e., the noisy correspondence which could be potential new direction in noise label. To solve this problem in cross-modal matching, we propose rectifying the noisy correspondence by an adaptive prediction function and novel triplet loss with soft margin to achieve robust moosl-modal matching. Extensive experiments verify the effectiveness of the proposed method handling synthesized and real noisy correspondences.
10

---

Broader Impact Statement
Cross-modal matching is a fundamental topic in multimodal learning, which could be applied to wide range of applications including data retrieval, recommender systems, and vision-and-language understanding. This work could be one of the first works aware of the importance and existence of the noisy correspondence problem in numerous applications. There are many benefits to solving the noisy correspondence problem, e.g., reducing the costs for manually annotating and aligning data; more data could be collected and used even though some of them are incorrectly aligned. Besides the benefits, it should pay attention on the potential negative impacts including but not limited to 1) The risk of automation bias (28) for decision making from the data bias, especially in aviation, health care, and autonomous vehicles. 2) The job loss caused by the NCR since makes possibility to automatically correct the noisy correspondence, thus remarkably reducing the cost of human labor. We would encourage further work to understand and mitigate the above biases and risks.
Acknowledgements
The authors would thank to the anonymous reviewers whose valuable suggestions and constructive comments remarkably improve this work. This work was supported in part by the National Key R&D Program of China under Grant 2020AAA0104500; in part by the Key Research and Development Program of Sichuan Province under Grant 2019YFG0497; in part by NFSC under Grant B2040, 62176171, U19A2078, 61625204, and 61 61836006; in part by the Fund of Sichuan University Tomorrow Advancing Life.
References
[1] Peter Anderson, Xiaodong He, Chris Buehler, Damien Teney, Mark Johnson, Stephen Gould, and Lei Zhang. Bottom-uy up and top-down attention for image captioning and visual question answering. In Proceedings of the IEEE conference on computer vision and puttern recognition, pages 6077-6086, 2018 8.
[2] Eric Arazo, Diego Ortego, Paul Albert, Noel O'Connor, and Kevin McGuinness. Unsupervised label noise modeling and loss correction. International Conference on Machine Learning. pages 312-321. PMLR, 2019.
[3] Devansh Arpit, Stanislaw Jastrzębski, Nicolas Ballas, David Krueger, Emmanuel Bengio, Maxinder S Kanwal, Tegan Maharaj, Asja Fischer, Aaron Courville, Yoshua Bengio, al. A closer look memorization in deep networks. In International Conference Machine Learning, pages 233-242. PMLR, 2017.
[4] David Berthelot, Nicholas Carlini, Ian Goodfellow, Nicolas Papernot, Avital Oliver, and Colin Raffel Mixmatch: A holistic approach to semi-supervised learning. arXiv preprint 1XXi:1995.002299 2019.
[5] Hui Chen, Guiguang Ding, Xudong Liu, Zijia Lin, Liu, and Jungong Han. Imram: Iterative matching with recurrent attention memory for cross-modal image-text retrieval. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12655-12663, 2020.
[6] Sanghyuk Chun, Seong Joon Oh, Rafael Sampaio de Rezende, Yannis Kalantidis, and Diane Larlus. Probabilistic embeddings for cross-modal retrieval. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition. pages 8415-8424, 2021.
[7] Haiwen Diao, Ying Zhang, Lin Ma, and Huchuan Lu. Similarity reasoning and filtration for image-text matching. In AAAI, 2021.
[8] Fartash Faghri, David Fleet, Jamie Ryan Kiros, and Sanja Fidler. Vse++: Improving visual- semantic embeddings with hard negatives. arXiv preprint arXiv:1707.05612. 2017.
[9] Dalu Guo, Chang Xu, and Dacheng Tao. Image-question-anss synergistic network for visual dialog. Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pages 10434-10443, 2019.
11

---

[10] Bo Han, Quanming Yao, Xingrui Yu, Gang Niu, Miao Xu, Weihua Hu, Ivor Tsang, and Masashi Sugiyama. Co-teaching: Robust training of deep neural networks with extremely noisy labels. arXiv preprini arXiv: 180.006804 2018.
[11] Yi Hao, Nannan Wang. Xinbo Gao, Jie Li, and Xiaoyu Wang. Dual- Dual-alignment feature embedding for cross- modaliyy person re-identification. In Proceedings of the 27th ACM International Conference on Multimedia, pages 57-65, 2019.
[12] Zhenyu Huang, Peng Hu, Joey Tianyi Zhou, Jiancheng Lv, and Xi Peng. Partially view-aligned clustering. In Proceedings of the 34th Conference on Neural Information Processing Systems, NeurlPS' volume 33, Virtual-only Conference, Dec 2020.
[13] Zhenyu Huang, Joey Tianyi Zhou, Hongyuan Zhu, Changqing Zhang, Jiancheng Lv, and Xi Peng. Deep spectral representation learning from multi-view data. IEEE Transactions on Image Processing, 30:5352-5362, 2021.
[14] Chao Jia, Yinfei Yang. Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoe V Le, Yunhsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and vision-language representation learning with noisy text supervision. arXiv preprint
2021.
[15] Sungyeon Kim, Minkyo Seo, Ivan Laptev, Minsu Cho, and Suha Kwak. Deep metric learning beyond binary supervision. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 2288-2297, 2019.
[16] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint 14Xiv:1412.6980, 2014.
[17] Ryan Kiros, Ruslan Salakhutdinov, and Richard S Zemel. Unifying with multimodal neural language models. arXiv preprint arXiv:1411.2538 2014.
embeddings
[18] Zhanghui Kuang, Yiming Gao, Guanbin Li, Ping Luo, Yimin Chen, Liang Lin, and Wayne Zhang. Fashion retrieval via graph reasoning networks on similarity pyramid. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 3066-3075, 2019.
[19] Kuang-Huei Lee, Xi Chen, Gang Hua, Houdong Hu, and Xiaodong He. Stacked cross attention for image-text matching. In Proceedings of the European Conference on Computer Vision (ECCV), pages 201-216, 2018
[20] Junnan Li, Richard Socher, and Steven CH Hoi. Dividemix: Learning with noisy labels as semi-supervised learning. arXiv preprint arXiv:2002.07394, 2020.
[21] Kunpeng Li, Yulun Zhang, Kai Li, Yuanyuan Li, and Yun Fu. Visual semantic reasoning for image-text matching. In ICCV, 2019.
[22] Sheng Li, Zhiqiang Tao, Kang Li, and Yun Fu. Visual to text: Survey of image and video captioning. IEEE Transactions on Emerging Topics in Computational Intelligence, 3(4):297- 312, 2019.
[23] Tsung-Yi Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European conference on computer vision, pages 740-755. Springer, 2014.
[24] Yijie Lin, Yuanbiao Gou, Zitao Liu, Boyun Li, Jiancheng Lv, and Xi Peng. Completer: Incomplete multi-view clustering via contrastive prediction. In CVPR, June 2021.
[25] Fangyu Liu, Rongtian Ye, Xun Wang, and Shuaipeng Li. Hal: Improved text-image matching by mitigating visual semantic hubs. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 11563-11577 2020.
[26] Risheng Liu, Jinyuan Liu, Zhiying Jiang, Xin Fan, and Zhongxuan Luo. A bilevel integrated model with data-driven layer eusemble for multi-modality image fusion. IEEE Transactions on Image Processing. 30: 10:1261-1274 2020.
(27) Tongliang Liu and Dacheng Tao. Classification with noisy labels by importance reweighting. IEEE Transactions on pattern analysis and machine intelligence, 38734447-441,, 2015. 2015.
12

---

[28] Kathleen L Mosier, Linda J Skitka, Susan Heers, and Mark Burdick. Automation bias: Decision making and performance high-tech cockpits. The International journal of aviation psychology. 8(1):47-63, 1998.
[29] Xi Peng, Zhenyu Huang, Jiancheng Lv, Hongyuan Zhu, and Joey Tianyi Zhou. Comic: Multi- view clustering without parameter selection. In International conference on machine learning, pages 5092-5101. PMLR, 2019.
[30] Haim Permuter, Joseph Francos, and Ian Jermyn. A study of gaussian mixture models of color and texture features for image classification and segmentation. Pattern Recognition, 39(4):695-706, 2006.
[31] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. arXiv preprint arXiv:2103. 0000000 2021.
[32] Scott Reed, Honglak Lee, Dragomir Anguelov, Christian Szegedy, Dumitru Erhan, and Andrew Rabinovich. Training deep neural networks on noisy labels with bootstrapping. arXiv preprint arXiv:1412.6596, 2014.
[33] Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. In NIPS, 2015.
[34] Mike Schuster and Kuldip K Paliwal. Bidirectional recurrent neural networks. IEEE transactions on Signal Processing, 2566722267,2661, 1997.
[35] Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. Conceptual captions: A cleaned, hypemymed, image alt-text dataset for automatic image captioning. In Proceedings of the 56th Anmual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2556-2563 2018
[36] Hwanjun Song, Minseok Kim, Dongmin Park, Yooju Shin, and Jae-Gil Lee. Learning from noisy labels with deep neural networks: A survey. arXiv preprint arXiv:2007.08199 2020.
[37] Liwei Wang, Yin Li, and Svetlana Lazebnik. Learning deep sructure-presenve image-text embeddings. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 5005-5013, 2016.
[38] Michael Wray, Hazel Doughty, and Dima Damen. On semantic similarity in video retrieval. In Proceedings of the IEEE/CVÉ Conference on Computer Vision and Pattern Recognition, pages 3650-3660, 2021.
[39] Xiaobo Xia, Tongliang Liu, Bo Han, Chen Gong, Nannan Wang, Zongyuan Ge, and Yi Chang. Robust early-learnigg Hindering the memorization of noisy labels. International Conference on Learning Representations, 2021.
[40] Erkun Yang, Cheng Deng, Wei Liu, Xianglong Liu, Dacheng Tao, and Xinbo Gao. Pairwise rela- tionship guided deep hashing for cross modal retrieval. In proceedings of the AAAI Conference on Artificial Intelligence, volume 31, 2017.
cross-modal
[41] Mouxing Yang. Yunfan Li, Zhenyu Huang, Zitao Liu, Peng Hu, and Xi Peng. Partially view-aligned representation learning with noise-robust contrastive loss. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1134-1143, 2021.
[42] Peter Young, Alice Lai, Micah Hodosh, and Julia Hockenmaier. From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions. Transactions of the Association for Computational Linguistics, 2:67-78, 2014.
[43] Xingrui Yu, Bo Han, Jiangehao Yao, Gang Niu, Ivor Tsang, and Masashi Sugiyama. How does disagreement help generalization against label corruption? In International Conference on Machine Learning, pages 7164 7173. PMLR, 2019.
[44] Zhou Zhao, Qifan Yang, Deng Cai, Xiaofei He, and Yueting Zhuang. Video question answering via hierarchical spatio-temporal attention networks. In IJCAI, pages 3518-3524, 2017.
13

---

[45] Songzhu Zheng, Pengxiang Wu, Aman Goswami, Mayank Goswami, Dimitris Metaxas, and Chao Chen. Error-bounded correction of noisy labels. In International Conference on Machine Learning, pages 1447-1 1457. PMLR, 2020.
14


Reviews:
[{"review_id": "diZmRfjK6CI", "reviewer": "Reviewer_nty3", "summary": "This paper proposes a noisy label learning framework for cross-modal learning, where the training data consists of wrongly matched modalities. The proposed NCR first divides the data into clean and noisy splits based on the difference of loss distribution. Then NCR proposes to rectify the labels with a prediction function and further recast them into soft margins in the triplet loss. Extensive experiments and comparisons are conducted to verify the effectiveness of the method.", "questions": "", "limitations": "", "rating": 9, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Overall, the paper is generally well written and easy to read. The main strengths and weaknesses of this paper are summarized as follows.\n\nStrengths：\n1.\tThe experiments are comprehensive. To show the effectiveness of the proposed method, the authors take SGR as an example and provide comparisons with other SOTA methods. The results with varying noise ratios and real-world noisy data demonstrated the effectiveness of NCR. Besides the SGR, experimental results of NCR-SCAN also show the superiority of the proposed method.\n2.\tthe technique presented in the paper is novel. To achieve the robust cross-modal matching, the authors propose to turn the rectified soft labels into soft margins which enforce the true positive pairs closer than the negative by a large margin, while the false positive will has a small margin. \n\nWeaknesses:\n1.\tIt seems that the proposed method NCR is only applicable to the triplet loss in cross-modal matching as defined in Eq.5-6. How to achieve robustness to other loss formulations like the softmax loss in ALIGN (Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision)?\n2.\tNCR recast the soft labels as soft margins defined in Eq.7. Why NCR need this procedure for robust matching? Why it could achieve robustness?\n3.\tSince two models are trained individually in the manner of co-teaching. It inevitably needs more computation time for convergence. And how to obtain the final retrieval results with two models?\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The paper provides a comprehensive discussion about the potential negative impact in the Border Impact Statement.", "review_text": "", "extra_scores": {"ethical_concerns": "No", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "_-zZvGcZIX", "reviewer": "Reviewer_peXt", "summary": "This paper focuses on cross-modal matching with the presence of noisy correspondence. The author proposes a novel method for learning with noisy correspondence to achieve robust cross-modal matching based on the SGR. In brief, it first splits the data into clean and noisy subsets in the manner of co-teaching and rectifies the labels with an adaptive prediction function. Extensive experiments on three datasets show the effectiveness of the proposed method.", "questions": "", "limitations": "", "rating": 9, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "In brief, this paper did a good job for solving the noisy problem in cross-modal matching. The proposed method is well verified with dense experiments on three benchmark datasets. Here are some detailed comments.\nPros：\n1.\tThe proposed method for learning with noisy correspondence is novel. To perform label refinement in the scenario of a matching problem, the author proposes an adaptive prediction function for label rectifying. Moreover, to incorporate the rectified soft labels in the triplet loss, they propose to recast the labels as the soft margin in the loss.\n2.\tExperiments on the simulation noisy datasets (COCO and Flickr) and the real-world dataset (Conceptual Captions) show the superiority of the proposed method. Moreover, the ablation study and visualization verify the effectiveness of different modules.\nCons:\n1.\tIn the method, the soft labels are recast as soft margins in the triplet loss. Why are the soft margins computed in the form of Eq. 7? Are there other alternative ways to determine the soft margins?\n2.\tThere is a typo in Eq. 5. S is the similarity margin between the given pair and the mean of negatives. The second minus symbol should be modified as a plus.\n3.\tThe model predictions are based on the average similarity margin of the largest 10% pairs in Eq.6. How to guarantee that the predictions are accurate enough for label rectifying? Why only select 10% pairs?\n4.\tI’m a bit concerned about training time as the NCR trains two models in the manner of co-teaching.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors have discussed the limitation and potential negative impact of the proposed method.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "Xpv8JvGZQF4", "reviewer": "Reviewer_9fra", "summary": "The authors propose a method for cross-modal retrieval in the presence of noisy image-text pairings. While there has been a lot of work in learning in the presence of noise in this space, existing works mainly handle the problem of \"label\" noise - i.e. when semantic categories / labels on the data are available, but are noisy. In contrast, the authors tackle the unsupervised setting where all that is available are image-text pairs harvested from the web. Some of these pairs may be noisy and some may not be. The idea of the paper is straightforward. The authors use a technique commonly used to address noisy labels - i.e. co-teaching. They train two networks, A and B on the same data. After each epoch, the authors collect the similarity scores computed by each of the networks and use it to train a Gaussian Mixture Model. The reason for this is the GMM allows one to calculate the probability that a sample is from a given distribution. They model it with two Gaussians. The idea is that the GMM will fit one to the set of data that has lower similarity (hence noisy) and a separate Gaussian to the one with high similarity (hence clean). They then use the learned similarities predicted by each model to \"correct\" the labels used for training the other model (for example, treating some data that was considered clean as noisy or vice versa) based on the co-teacher's predictions. Finally, they integrate the similarity / confidence that the model is clean or not into the triplet loss as an adaptive margin, rather than use a static margin. The authors experimentally validate their method on MSCOCO, FlickR30K, and Conceptual Captions and compare against a number of baselines showing reasonable improvement.", "questions": "", "limitations": "", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "# Strengths\nThe problem tackled by the authors is a good one - many cross-modal retrieval methods that exist assume some sort of label space is either available, or can be predicted on the data. However, this might not always hold, particularly when one is harvesting web-scale data without labels. Thus, the problem of noisy pairings is important to address. I have done a literature search and haven't found any papers explicitly tackling this problem directly, so I would agree that the problem itself does have some novelty (although, many other papers do learning in the presence of label noise, which is somewhat different).\n\nThe approach taken by the authors is interesting, as they train two models, then try to essentially learn based on the distribution of the learned similarities which are the noisy samples and which are the clean ones - this is essentially a clustering, but done using a GMM. The authors don't have any novelty in that regard though, as the GMM is a standard off-the-shelf model. However, they use it to essentially get a per sample score of how likely the sample is noisy or not. The main idea is that these are then used to update the training set of the other model being used for co-teaching. This is a good idea, as it extends co-teaching methods into the cross-modal retrieval space in the absence of any labeled data. \n\nThe authors' way of integrating this knowledge into the model is an interesting one - they essentially use an adaptive margin in the triplet loss. The main idea is that when we are really confident about a sample being clean, we should enforce a large margin. But when we aren't confident the sample is clean or noisy, we should enforce a small margin. This is a good idea, as it allows the model to learn an embedding space where such uncertainty can naturally be captured.\n\nExperimentally, the authors show reasonable gains in the presence of noise. For the MS-COCO dataset, in the most noisy setting (50% pairings), the authors show ~2.1% gain for T-->I and around ~2.9% gain for I-->T on COCO. However, they are often outperformed in the 0% noise setting. The authors show more convincing gains on the Conceptual Captions dataset.\n\nThe authors include an ablation where they remove various components of their method, such as various pieces of the co-teaching model and show that each has a contribution to improving the performance of the method.\n\n# Weaknesses\nBefore getting into specifics of the details of the model, I would like to point out there is an implicit assumption made by the authors that unclean pairs will end up having a lower similarity after the initial training, while clean pairs will have a higher similarity. The authors note throughout the text that models tend to pick up obvious correspondences first, for example. Thus, the assumption is, if samples have a low similarity, they could possibly be unclean (mismatched image-text). I can buy this argument on some datasets, like COCO and Flickr, where there are clear, defined object categories and objects mentioned in the captions. In this case, the model can quickly pick up the correspondences in caption between \"car\" and the visual object \"car\" (for example). Thus it makes sense that after the initial training such samples will have a high similarity, while noisy samples will have a lower similarity. However, I would argue this assumption is not always applicable - and **most importantly**, it is not applicable to the problem setting the authors' method is designed to address. Specifically, for datasets like Coco and Flickr or other datasets where there is a strong grounding between image and caption, for these types of datasets there *are* often object labels or categories available, or, if not, they can be predicted using many of the other existing methods for learning cross-modal retrieval in the presence of noisy labels (you can, for example, identify a set of objects in the captions, train a model to classify them, and then do some prediction of labels - and voila, you have a self-supervised and noisy set of labeled data). However, the situation that the authors' method is most applicable to are cases where a label set of objects or classes cannot be readily defined, because obviously if such a label set can be defined there are many baselines that deal with that. So, the authors approach is best attuned to situations where a label set can't be defined, like when harvesting a ton of data from the web, with very loose image-text pairings and an unknown set of classes. If the authors wish to argue their method is applicable to the cases where labels can be defined, then they should show results comparing against those noisy self-supervised label methods. But assuming they don't, then the most important dataset the authors test on is the Conceptual Captions dataset. This dataset does not have a definable set of labels, as it is just a huge set of images and paired text crawled from the web. Again, that's the setting the authors are targeting. However, this type of data exhibits properties that break the authors' assumptions. For example, real world data often features images and text that aren't directly overlapping (i.e. the text isn't really a caption, but just goes with the image) - in other words, the image and text are complementary with one another. In other words, the text and image are related, but not directly duplicative of meaning. This is an important aspect of such data to capture, but the authors approach will actually de-emphasize it. For example, after training, these samples exhibiting this complementarity will have a lower similarity score than the obvious caption-like samples, thus the authors' approach may discard them as noisy. \nThere is a recent work that tackles this exact scenario in ECCV20 and doesn't require any labeled data and learns a strong semantic space. I am also interested in hearing the authors thoughts on they can distinguish noisy samples from samples that just have a more indirect relationship between image and text (the so-called complementarity). This is a very challenging problem. Moreover, the authors should have compared against this paper. In my view, the results on FlickR and COCO aren't really the scenario this method is suitable for.\nPaper studying complementarity and dealing with indirect relationship of image-text on real world data:\nThomas, Christopher, and Adriana Kovashka. \"Preserving semantic neighborhoods for robust cross-modal retrieval.\" European Conference on Computer Vision. Springer, Cham, 2020.\n\nAgain, the authors paper is targeting learning on web-scale data where there are no discernable classes and thus there is noise. Thus, I would expect the authors to compare against using CLIP as a baseline. CLIP is trained on a massive dataset of webly harvested data and presumably there are a lot of image-text pairs in there that are noisy. *Why isn't the solution to noisy image-text pairings just to have a lot more data?* That seems to be the answer CLIP provides and it is the state-of-the-art image-text retrieval model at present. Moreover, anybody can take CLIP and finetune it on a dataset if they feel there is too much domain shift.\nThe authors should have showcased a result showing why just having a ton of noisy image-text data isn't enough and that they can do better on less data by explicitly managing / modeling the noise. At present, I am not convinced I should reach for the authors' method over just reaching for a model trained on a large noisy dataset. This is a key point the authors should address.\nCitation for CLIP:\nContrastive Language–Image Pre- training) (Radford et al., 2021)\n\nRelationship with hard / soft negative mining methods, weighting approaches, and adaptive margin methods:\nWhile the experimental comparison against a few recent retrieval methods is OK, the key point is to evaluate your experimental technique against other methods that do the same thing in different ways. For example, there are many papers that do a soft hard-negative mining technique (see Mithun, 2018 for the soft triplet loss), where they weigh each sample according to the distance. Similarly, there are papers that try to measure properties of the surrounding space and integrate it as a weight into the loss (e.g. Hubness aware loss). Others still try different mechanisms for learning an adaptive margin. I would have liked to see a comparison of why your weighting approach is better or at least a thorough discussion of these in related work.\n\nMoreover, onto your approach, I understand that you use the confidence of the sample to compute the margin, but shouldn't we also consider the confidence of the negative samples as well? For example, if the confidence of the sample is very high then your method would say this sample should have a large margin. But if we are in a batch that has noisy negatives where the confidence is very low, then your method might try to enforce a large margin between the positive sample and the negatives - but those negatives could be noisy and actually shouldn't be that far from the positive. In other words, the margin should take into account the relationship of the positive sample as well as each negative to calculate the adaptive margin.\n\nFor the approach, I would also point out that the co-teaching technique used is really an adaptation of what has previously been done in the label space. Other prior work also uses a co-teaching approach to correct noisy data. So, the actual methodology of doing so isn't new, but the application to noisy image-text pairings and the way it is done by integrating it as a weight is new.\n\n# Conclusion\nThe authors present a reasonable technique for correcting noisy image-text pairings. However, the scenario where the proposed method actually would apply has not been thoroughly explored. Moreover, if the authors wish to argue their approach does apply to scenarios where labels could be defined, they should have compared against noisy or self-supervised labeling methods. I believe the authors should also explain why the approach will not ignore the types of samples emphasized by the ECCV20 paper - i.e. the authors approach might learn to ignore the samples that have an indirect image-text relationship, at the expense of losing semantics from the representation space. Moreover, since they wish to argue web-scale data harvested from the web is often noisy, it isn't clear that the solution to that isn't just to use a lot more web-scale data. In that case, they should have compared against CLIP - which is the standard image-text retrieval method at the moment, is trained on noisy image-text pairings, and is available. In sum, while the approach attempts to solve an important and interesting problem, I am concerned about the evaluation scenarios and the lack of discussion of the issues mentioned.\n\n# Final Comment (Post Rebuttal)\nI thank the authors for their very thorough responses. I have reviewed the authors' results on CLIP and I encourage them to include these comparisons in their submission. Given that the performance is superior for some cases, and particularly because the performance is superior on the released model (CLIP) relative to the unreleased model, I would recommend the paper be accepted. I appreciate the authors' discussion of complementarity vs noise. This is an interesting issue that can be discussed in the text. Given the thorough results provided by the authors, I raise my rating to accept.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "This is a technical retrieval work and has no dataset contribution. There are no societal concerns beyond any existing cross-modal retrieval model.", "review_text": "", "extra_scores": {"ethical_concerns": "This is a technical retrieval work and has no dataset contribution. There are no ethical concerns beyond any existing cross-modal retrieval model.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}, {"review_id": "3V07qj7Xg7", "reviewer": "Reviewer_iPic", "summary": "This paper introduces a novel task, noisy correspondence of cross-modal retrieval tasks. The noisy correspondence is similar to the noisy label in object classification tasks, where the noisy correspondence scenario assumes that some image-caption pairs are incorrectly matched. To solve the problem, this paper proposes a Noisy Correspondence Rectifier (NCR), which could be considered as the novel extension of Co-teaching [9] to cross-modal retrieval tasks. Note that directly applying Co-teaching to cross-modal retrieval is non-trivial as the authors discussed in the paper (L105-115), because noisy label methods assume the existence of the prediction score function of the given input, while cross-modal retrieval methods usually do not have such per-sample score function. Hence, the proposed NCR first selects (co-divides) clean and noisy samples by using triplet loss values and the 2-mean Gaussian Mixture Model (GMM). Using the divided samples, NCR co-rectifies the noisy labels by the prediction function, where the prediction function is defined by the value of triplet distances. Using the extracted clean/noisy samples and rectified labels, NCR applies the hardest negative mining triplet loss to train the model, while the margin value is adaptive to the rectified labels. The experimental results show the effectiveness of NCR in terms of robustness to the noisy correspondence, e.g., achieving nearly dropped retrieval performances when 50% of correspondences are incorrect.", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "## Pros\n\n- (+) The introduced noisy correspondence is an interesting and important problem. Although a few previous works [A, B] tackle similar motivation in cross-modal retrieval tasks, as far as the reviewer knows, this paper is the first paper that evaluates synthetic noisy correspondence with state-of-the-art cross-modal retrieval methods.\n    - [A] Chun, Sanghyuk, et al. \"Probabilistic embeddings for cross-modal retrieval.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2021.\n    - [B] Wray, Michael, Hazel Doughty, and Dima Damen. \"On Semantic Similarity in Video Retrieval.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2021.\n- (+) The proposed approach is sound and empirically achieves good results under the noisy correspondence scenario. Especially, The proposed Co-teaching-like algorithm is conceptually well-founded and empirically works well.\n- (+) Experimental results on different noise levels (0%, 20%, 50%) are very interesting; particularly, existing methods perform poorly when noisy correspondence happens.\n- (+) This paper is well-written, well-organized, and easy to follow. I really enjoyed reading this paper.\n\n## Cons\n\nI think this paper has a good problem definition contribution and technical contribution. However, I feel that discussions related to the previous attempts (could be considered as contemporary works) are not enough. Especially if the authors can add discussions related to Chun et al. [A], I think this paper will be stronger submission.\n\n### Precision metric\n\nMusgrave et al. [C] recently showed that \"the use of accuracy metrics that are either misleading, or do not a provide a complete picture of the embedding space\", in other words, recall scores may lead to a wrong conclusion. Chun et al. [A] tackles the same problem in cross-modal retrieval tasks, and proposed Plausible-Match R-Precision (PMRP) for MS-COCO evaluation. I suggest the authors report PMRP scores [A] to show the effectiveness of the proposed NCR method. As far as the reviewer understood, re-evaluating models with PMRP scores does not need extra training budgets but only needs the evaluation procedure which is relatively reasonable computation during the rebuttal period.\n\n- [C] Musgrave, Kevin, Serge Belongie, and Ser-Nam Lim. \"A metric learning reality check.\" European Conference on Computer Vision. Springer, Cham, 2020.\n\n### Possible related works\n\nAs far as the reviewer understood, this paper mainly assumes the underlying dataset has (almost) clean correspondence. However, in practice, the existing widely-used cross-modal datasets, such as MS-COCO already have many noisy correspondences. For example, many images in MS-COCO only contain a big wave and a surfer and their corresponding captions are barely distinguishable (See Figure 4 of [A] as another example). A similar problem, many-to-many correspondence, is actively discussed in Chun et al. [A], which solves the problem by a probabilistic approach. I believe that [A] should be discussed in the main paper because the target problem by [A] is highly related to the proposed noisy correspondence problem. This could be an explanation of why NCR performs better than other methods in 0% noise scenarios in Table 1 and 2. Wray et al. [B] also targeted a similar problem, instance-based retrieval evaluation of video retrieval, and solved the problem by soft labels as rectified soft labels in this paper. Kim et al. [D] also employed soft non-binary labels to solve metric learning tasks. \n\n- [D] Kim, Sungyeon, et al. \"Deep metric learning beyond binary supervision.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2019.\n\nThe listed related works do not hurt the novelty of this paper, but I think adding discussions on these related works, the submission will be stronger.\n\n### Impact of vanilla triplet (or warmup)\n\n- In L191-197, the paper mentions that $L_\\text{soft}$ uses the hardest negative mining, not the vanilla triplet loss without hard negative mining as \"L_w\". I wonder about the performance gap between the vanilla triplet and the hardest negative mining triplet while L192-193 says \"Although the hard negatives are helpful in improving the performance\".\n- On the other hand, it seems that the vanilla triplet loss without hard negatives hurt the performance of SGR in CC152k, as shown in Table 2. Why does it happen? The paper assumes that SGR is very sensitive to noisy correspondence (as denoted in L246-249), but there is no explanation of how the vanilla triplet can mitigate the noisy correspondence. \n  - Similarly, warm-up (with the vanilla triplet) seems the most important ingredient of the method (Table 3). It may imply that the vanilla triplet is one of the most important keys to solve the noisy correspondence. I wonder about the performance of warmup only, i.e., the vanilla triplet trained model.\n\n### More related works\n\nWhile the following related works do not hurt the novelty of this paper, I suggest the authors add more discussions on previous works [A, B] discussing noisy correspondence problems as well. Especially, \n\n\n## Comments not affecting the decision score\n\n- L143-L149 can be replaced by citation of the textbook by Bishop, Christopher M (Bishop, Christopher M. \"Pattern recognition.\" Machine learning 128.9 (2006).)\n- L164: \"On the other hand\" appears two times in sequential sentences. It could be a typo.\n- L186: It would be good to add VSE++ citation for explaining the hardest negative mining strategy\n- Table 3. \"02.\" could be a typo of \"0.2\"\n\n## Overall comment\n\nThis paper introduces an important and undiscovered problem, noisy correspondence. The proposed method motivated by noisy label methods successfully tackles the problem. Although there are some raised concerns, for example, (1) missing precision comparisons in MS-COCO (2) some missing related works (3) study on vanilla triplet without hard negative mining is missing, I think these issues are not a significant issue to reject this paper, but resolving the raised concerns will make the submission stronger. Overall, I vote for accepting this paper.\n\n---\n\n## Post-rebuttal update\n\nI really appreciate the authors addressing my suggestions very well in the response. The precision results (outperforming the baseline with more than +20%) are particularly interesting. I revise my score to clear accept.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "One possible limitation of this paper is the lack of discussions on the existing noisy correspondence in the existing cross-modal retrieval methods. I would suggest adding more related works and their discussions in the main paper.\n\nAnother possible limitation could be the evaluation metric; Recall@K is too noisy and less informative metric than precision. I suggested the authors testing PMRP proposed by [A] in MS-COCO benchmark for measuring precision performances by the proposed method.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper initially received positive reviews from four reviewers (i.e., 6 7 7 8). In the rebuttal, the authors address the concerns from the reviewers very well (e.g., comparison with CLIP), and then all reviewers further raise their scores (to 7 8 9 9). 

After carefully reading the manuscript, comments, and the corresponding response, I agree with the reviewers that this paper makes a significant contribution to the community, namely, revealing a NEW problem (i.e., noisy correspondence/mismatching pairs) in practice for the first time. The work is certainly valuable to the community including but not limited to cross-modal matching because the essence of many tasks such as ReId, visual grounding, and VQA, is exploring the correspondence of paired data. 

Overall, this paper is with strong motivation, a technical sound approach, extensive experiments, and good writing.
Decision: Accept (Oral)