Cross-modal Active Complementary Learning with Self-refining Correspondence
Authors: Yang Qin
Yuan Sun
Dezhong Peng
Joey Tianyi Zhou
Xi Peng
Peng Hu
Venue: NeurIPS 2023 poster (2023)
Keywords: Cross-modal learning
Image-text matching
Noisy correspondence.
TL;DR: A novel contrastive learning method to learn with noisy correspondence for robust image-text matching.

Abstract: Recently, image-text matching has attracted more and more attention from academia and industry, which is fundamental to understanding the latent correspondence across visual and textual modalities. However, most existing methods implicitly assume the training pairs are well-aligned while ignoring the ubiquitous annotation noise, a.k.a noisy correspondence (NC), thereby inevitably leading to a performance drop. Although some methods attempt to address such noise, they still face two challenging problems: excessive memorizing/overfitting and unreliable correction for NC, especially under high noise. To address the two problems, we propose a generalized Cross-modal Robust Complementary Learning framework (CRCL), which benefits from a novel Active Complementary Loss (ACL) and an efficient Self-refining Correspondence Correction (SCC) to improve the robustness of existing methods.   Specifically, ACL exploits active and complementary learning losses to reduce the risk of providing erroneous supervision, leading to theoretically and experimentally demonstrated robustness against NC. SCC utilizes multiple self-refining processes with momentum correction to enlarge the receptive field for correcting correspondences, thereby alleviating error accumulation and achieving accurate and stable corrections. We carry out extensive experiments on three image-text benchmarks, i.e., Flickr30K, MS-COCO, and CC152K, to verify the superior robustness of our CRCL against synthetic and real-world noisy correspondences.

Cross-modal Active Complementary Learning with Self-refining Correspondence
Yang Qin', Yuan Sun', Dezhong Pengl.3. 13,4, Joey Tianyi Zhou',
Xi Peng', Peng Hul-
College of Computer Science, Sichuan University, Chengdu, China.
2 Centre for Frontier Al Research (CFAR) and Institute of High
Performance Computing (IHPC), A*STAR, Singapore.
Chengdu Ruibei Yingte Information Technology Co., Ltd, Chengdu, China.
Sichuan Zhiqian Technology Co., Ltd, Chengdu, China. {qinyang- gm, joey. tianyi zhou, pengx....... gm, penghu. .ml mmlmmmmmmmmmmmmmmmmmmmmmmmmmmm com, sunyuan_ work@163. com, pengdzâscu. edu. cn
Abstract
Recently, image-text matching has attracted more and more attention from academia and industry, which is fundamental to understanding the latent cor- respondence across visual and textual modalities. However, most existing methods implicitly assume the training pairs well-aligned while ignoring the ubiquitous annotation noise, a.k.a noisy correspondence (NC), thereby inevitably leading to a performance drop. Although some methods attempt to address such noise, they still face two challenging problems: excessive memorizing/overfite correction for NC, especially under high noise. To address the two problems, propose a generalized Cross-modal Robust Complementary Learning framework (CRCL), which benefits from novel Active Complementary Loss (ACL) and an efficient Self-refining Correspondence Correction (SCC) to improve the robust- of existing methods. Specifically, ACL exploits active and complementary learning losses to reduce the risk of providing erroneous supervision, leading to theoretically and experimentally demonstrated robustness against NC. SCC utilizes multiple self-refining processes with momentum correction to enlarge the receptive field for correcting correspondences, thereby alleviating error accumulation and achieving accurate and stable corrections. We carry out extensive experiments on three image-text benchmarks, i.e., Flickr30K, MS-COCO, and CC152K, verify the superior robustness of our CRCL against synthetic and real-world noisy correspondences. Code available at https: /ttptggbttttttt///// com/ MianYang79//BBa
and unreliable
Introduction
Image-text matching aims to search the most relevant samples across different modalities, which is fundamental for most cross-modal tasks [12 4 5 6). The core of image-text matching is how to accurately measure the similarity between distinct modalities, however, which is challenging due to the visual-textual discrepancy. To tackle the challenge, numerous deep methods are presented to learn the risual-semantee associations of image-text pairs and achieve remarkable progress, thanks to the powerful representation ability of Deep Neural Networks (DNNs) and some well-designed similarity inference architectures 7 9 They could be roughly divided into two groups, i.e., global-Ievel methods 7 [ and local-level methods [8] 4. which aim at learning the image-to-sentenn and region-to-word correlation to infer the cross-modal similarity, respectively. Although these methods
"Corresponding author.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

achieved promising matching performance, most of them implicitly require large-scale well-aligned data for training, which is expensive or even impossible to collect due to ubiquitous noise in real- world scenarios [1] 11. Therefore, there is inevitably imperfect alignment luring in the data, i.., noisy correspondence (NC) [2], resulting in inferior performance.
To handle the NC problem, some prior arts are presented to alleviate the adverse impact of NC in various tasks, e.g., partially view-aligned clustering [13 14
[5
16) video-text retrieval [7.. visible-
infrared person re-identification [8. and image-text matching 12
19 20 Specifically, inspired by
24 are proposed alleviate the negative impact brought by NC. These works attempt to leverage the memorization effect of DNNs [25] to gradually distinguish the noisy image-text pairs for robust learning in a co-teaching manner. Furthermore, the predicted soft correspondence is used to recast soft margin to replace the scalar margin of triplet ranking loss [7]. which helps to avoid misleading the model by mismatched pairs. However, the soft-margin ranking loss is experimentally found to provide only limited robustness against NC, especially under high noise (as shown Table [T. due to unstable division based on inaccurate predictions. In contrast, some works [19 20 aim to enhance the robustness of cross-modal methods against NC by starting with a robust loss function, i.e., avoiding over-amplification of wrong supervision information to reduce misleading risk. However, the lack of explicitly mitigating the effect of easily separable noise makes them hard to further improve the performance.
learning with noisy labels [21 some works [2
23
To address the aforementioned issues, we propose a generalized robust framework, dubbed CRCL, for learning with noisy correspondences. CRCL could be easily migrated into existing image- text matching methods to enhance their robustness against NC. Our framework introduces novel Active Complementary Loss (ACL) that applies active and complementary learning to mutually boost robustness against NC. Specifically, we present robust complementary learning loss that employs complementary pairs, such as "input image-text pairs are irrelevant", to conduct indirect cross-modal learning with exponential normalization. Due to the low likelihood of selecting incorrect complementary pairs, robust complementary learning could reduce the risk of providing incorrect supervision and smooth the losses, thus embracing robustness against NC. However, the robust loss will face the underfitting problem, leading to suboptimal performance. To overcome this issue, weighted Active learning loss is proposed to enforce the model focus more reliable positive pairs in addition to only complementary pairs. In addition, we propose a Self-refining Correspondence Correction paradigm (SCC) to obtain stable and accurate correspondence correction. SCC utilizes Momentum Correction (MC) to aggregate historical predictions for stable and accurate correspondence corrections. By combining multiple Self-Refining processes (SR) throughout the entire training process, we alleviate over-memorization for NCs. In summary, the key contributions and innovations of this work are as follows:
We propose generalized Cross-modal Robust Contrastive Learning framework (CRCL) to address a pressing and widely-exist problem image-text matching, i.e., noisy correspon- dence. CRCL empowers existing methods with strong robustness through the perspectives robust loss and correction techniques.
A novel Active Complementary Loss (ACL) is presented to balance active and comple- mentary learning, mutually enhancing robustness against NC while encapsulating correct cross- associations in the latent common space.
We design effective Self-refining Correspondence Correction paradigm (SCC) to achieve accurate and stable soft correspondences, which enables the prediction-basee corrections to perceive larger fields and self-refine from the historically learned knowledge.
Extensive experiments verify the effectiveness and superiority of our framework on three benchmark image-text datasets: Flickr30K, MS-COCO, and CC152K. Additionally, compre- hensive ablation studies and insightful analyses demonstrate the reliability and practicability of the proposed CRCL.
2 The Proposed Method
2.1 Preliminaries and Problem Statement
To be specific, we first provide some definitions of instance-level image-text matching so as to conveniently study noisy correspondence. Let D {I, T,T) be an image-text dataset, where I

---

{4)= and T = { N are the training image and text set with size of N. The correspondence label
space is defined as y [Jirli 1.... .N: 1.... N, where 3i represents the correspondence of pair (T; Tj), i.e., if J; and T3 are matched (iee, positive pair), Yij = 1 otherwise V/ij = 0. We assume each pair with the same indices has matched correspondence, i.e, yi = 1, = -, .N.. However, due to the ubiquitous noise during data collection, some negative pairs are mismatched as positives, a.k.a noisy correspondence (NC) [12] 19], which would introduce wrong supervisory information and misguide model training, leading to performance degradation. Mathematically, we define NC as shown in Definition II
Definition 1. Due to the existence of NC, the learner only has access to the noisy training data (Da)) instead of clean data (D). Thus, the correspondence label for pair T,) is reconsidered as
Vii {
with probability (1 - Tk]))
Øis
Yek
with probability Tisk: Vk + j.
For all pairs, conditioned on that i then yij = 1 else yij = O, we have Ejjt Tlik = Thj- Similar to the definitions of noisy labels (26), we assume that NC is uniform, i.e,, Thi = 1) and
Tik = 11 N- tk + j. whheee 1j is constant to represent the noise rate.
The key to learning with noisy correspondence to alleviate the misguiding impact and rectify the mismatched pairs. One direct solution is to enhance the robustness of loss function L against noisy pairs, which can help prevent overfitting on mismatched pairs. The second aims at using the memorization effect of DNNs [25] to discriminate the mismatched pairs, thus removing unreliable supervision from the training data.
For image-text matching, images and texts are first projected into a shared representation space by two mppeiit networks, denoted as and g. respectively. The cross-modal similarity of a given pair (4, T2) is then computed as S (I))), g g(T; )), where S(x) could be the cosine function 7 9 or a relevance inference module R For brevity, S (T(T)),((T))) is denoted as S (Ti, Tj) or Siy in the following. The computed similarities could be considered as supporting evidence for retrieved results. Thus, the learning objective of image-text matching is to maximize the cross-modal similarities of positive pairs while minimizing those of negatives in the latent common space, which is commonly achieved by using contrastive learning
The widely- used triplet ranking loss 7 has shown excellent performance in modal contrastive learning tasks 7 (27 However, recent research [19] has demonstrated that this loss function fails to perform well in image-text data with NCs, especially when using the hardest negative samples as comparison items. To address this issue, some works proposed an adaptive soft margin approach to improve the robustness of the ranking loss [12 24 23), which is defined as follows:
Lsofe (I,, = jā: - S T.) + S (L. [1) + [ - S (I,, + S (Th. (In
(2) where Th and in denote the hardest mooss-modal samples in mini-batch, â; is a soft margin adaptively computed by à; m m1
m-1 a, a is a constant margin, m a curve parameter, and tu the rectified correspondence between 4, and T. However, this approach has two disadvantages: 1) The margin setting a may not be consistent with the empirical setting under NC scenarios. 2) The inaccurate prediction ÿu can still easily produce the risk of misleading gradient, which can cause trivial solutions to fail, especially in the case of high noise (e.g., the results of NCR [12] and BiCro 23 on Flickr30K with 80% noise). To overcome these limitations, we propose a novel Active Complementary Loss (ACL) under the risk minimization theory 28 26 to provide noise tolerance for noisy pairs while ensuring discriminative learning.
2.2 Active Complementary Loss
For the image-text dataset D, our goal is to learn a cross- modal model (M) that can discriminatively identify the positive (matched) and negative (unmatched) pairs well for retrieval, which is intuitively equivalent to maximizing bidirectional matching probabilities of positives. The bidirectional matching probabilities for pair (I,, Tς) are defined as:
e(Sis/T)
((Sis/T)
pi = T() =
Σ e(Su/T)' pē = I.) =
(3)
ΣN e(Sig/T))
3

---

where 7 is a temperature parameter 229 30 311, f is regarded as the cross modal decision function, and "o, means the two retrieval directions, i.e, image-to-te and eext-toimmaee, respectively, For any loss function L. the matching risk of f for image-text matching can be defined as
Rf( B(,,, JDD [ T.), yo)) E (T...))> [C v:).
(4)
where [|| represents the expectation operator. Considering noisy correspondences, the risk of f in noisy data D,, could be formulated as follows:
(S) = (.. >-,, [C T.), 0.)) + + (T,
(5)
where Diy is the noisy correspondence label as shown in Definition T Thus, the cross-mos modal learning objective is learn model M with a global minimizer fú of (f). To achieve robustness, Jü should be also the global minimizer of Rc(J) on the noise-free data.
Inspired by complementary contrastive learning 20,, we propose to optimize the matching probabili- ties of all negative pairs for learning with noisy data indirectly, thereby avoiding fast overfitting to NC. Simultaneously, to further improve the noise tolerance, we introduce an exponential normalization to smooth the complementary loss. Hence, the robust complementary loss for pair (4, Ti) could be formulated as:
C, C1, T. = CI,, T..9) + CTT..
(6)
Lan(pi)) (an(pie))
+ Σ
tan(y)))
(an(pia))
where tan(-) is the tan function and q E [0, 1] is regulatory factor. Theoretically, for any input (.) under noise rate S N
we can show (see proofs in supplementary material) N-1
C S (c* (f) < 0,
(7)
where C= (7((4
Ny N < O. C' increases 9 increases and when q 1, C takes the maximum value 0. Amin and Amax are the maximum and minimum values of Σ Lan
Al )
under the condition Σj=1 Pij 1, where 1 < Amia < Amax. and O S Pij S (pij = pi or pij) f and fú are the global minimizers of Rc. (f) and R2, (f), respectively.
Analysis: The larger the q is. C O, the tighter the bound of Equation 7 is. When q C, is standard complementary contrastive loss [20]. In the extreme case of q = 1, Rc, (f*) R2, (fr)) i.e, C, is noise tolerant since fú; and f are simultancously the minimizers of Rc. (f) (It can also be obtained from Lemma that L, is robust under a = 1). Put another way, as g approaches 1, the optimum f of the noisy risk will be close to f on the clean data more likely, which implies noise tolerance.
we should focus more on reliable data, while less on unreliable data. In other words, smaller value of q is used for more convincing pairs (with larger (is,, while a larger value of q is for less convincing pairs (with smaller (ts). Thus, we could empirically utilize the soft corrected label iis to recast q like the soft margin used in NCR [12]. i.e., q= 1 du- However, indirect learning will face the underfitting problem, resulting in suboptinallinsuffiientt performance. To address this issue, we introduce a weighted active learning loss La to make the model pay more attention to positive/matched pairs, i.e., La Ti, Ssa) = - Dis (log P. + log P This positive learning will mine discrimination from direct supervision, which complements the complementary learning loss. By combining active and complementary learning losses, our active complementary loss is defined as:
Like most robust learning methods 32
33
Lack Ti Tú, (1s) L.a(T,, 0.) + XL, T;, (u):
(8)
where is a scale factor to prevent L. from dominating the cross-modal training and quickly overfitting NC. As shown in Equation (8) when (I, Tí) is a noisy pair and fh: ideally approaches O, the loss emphasizes robust complementary learning, thus mitigating overfitting to NC. Conversely, when (J,, T,) is clean pair and fus ideally approaches 1, the loss focuses on discriminative learning, thereby facilitating the accurate acquisition of
associations. However, due to computational resource constraints, we cannot use the entire training set to perform cross- modsl learning. Therefore,

---

we relax N to the size K of the mini-batche x by Monte Carlo sampling, Without loss of generality, the final loss for cross-modal learning is given by:
K
Lad (x)
K Lad Tú, Dts)
Σ 1=1
Lemma 1. In an instance-level cross-modal matching problem, under uniform NC with noise rate 7 7 when q = L, C, is noise tolerant.
Proof. The proofs of Equation and Lemmal 1 can be found in the supplementary material.
L
2.3
Self-refining Correspondence Correction
Another key to solving NC is how to obtain accurate correspondence estimations so to reduce the adverse effects NC. To this end, we propose an effective Self-refining Correspondence Correction paradigm (SCC). SCC leverages Momentum Correction (MC) aggregate historical predictions, providing stable and accurate correspondence estimations while alleviating the over-memorization to NC. To eliminate the error accumulation against NC, combine multiple independent Self Self-Refining (SR) in the entire training process. Specifically, the MC for the correspondence (I, T,) at the t-th epoch is defined follows:
(1 (1- (I,, Ti),
(10)
where 3 E (0, 1) represents the momentum coefficient, (1,. T:) (Vi +p,.)/ denotes the average matching probability
the /-th epoch. Through adequate cross-modal training, our CRCL will obtain a more stable and accurate soft correspondence label by smoothly evolving and expanding the receptive field of correction based on historical predictions with MC. Notably, training progresses, some pairs would always be incorrectly distinguished as clean or noisy ones, resulting in the error accumulation the estimated labels (see Figure Ic). Additionally, even though the updates performed by MC help reduce the negative influence of NC, the initial correspondence label (v%) still greatly affects the quality subsequent smooth corrections. In other words, providing more accurate initial correspondences weakens the DNN's memorization against NC, thereby reducing the risk of error accumulation.
Algorithm 1: The pseudo-code of CRCL
Input: noisy training dataset D,, image-text matching model .M()); Initialize: O;
for e in [e1, 62. ,m do for in [1,2, e do for x in batches do
Obtain the bidirectional matching probabilities of x with Equation 33:
Update the correspondence labels with Equation (T):
Obtain the corrected labels with Equation (12);
Compute the overall loss (x)(xx (x) with Equation
θ = Optimizer (O, faao(());;
end end end
Re-initialize O;
Output: The learned parameters θ
To achieve accurate initial correspondence estimations, SCC refines updated correspondence labels historically in epochs using MC through multiple concatenated SR pieces during the entire training process. Subsequent SR pieces could gradually aggregate the learned correspondence knowledge from previous pieces, thus improving the quality of estimated correspondences progressively. Fur- thermore, each SR piece is trained from scratch, which aims clear accumulated error/noise that has been memorized, thus providing more accurate correspondence predictions for subsequent training. Mathematically, SCC consists of multiple SR pieces (Je1,-.. ,em) based on MC, where each piece
5

---

.., mz) epochs. Thus, during the j-th SR training, the estimated soft label of the i-th pair at t-th epoch is reconsidered follows:
undergoes robust learning for e; Ú {1....,
(1=1 p (I, Byllh-1)
Vie
if S Ef:
(1.1) Vii
if = and t. = (es + 1):
(11)
Byl + (1- 3) (), otherwise,
where (+) is the average matching probability of pair (4, T:) at the (t-1)-th epoch during the j-th training piece, ey denotes the number of epochs freeze the correspondence label, preventing insufficient model training in the early stage from affecting the correction quality. In our experiments, we set all initial labels to assuming that all training pairs are matched the beginning of the first SR piece. In practice, we assign the label of the confident noisy pair as 0 to reduce the risk of producing erroncous supervision information. Therefore, the final corrected correspondence label
used for Lad is defined as:
{
if (1t)
0, (1.)) Vit
yis <; otherwise,
(12)
where 0.1 is fixed threshold used to filter the confident noisy pairs in experiments.
3 Experiments
In this section, comprehensive experiments are conducted on three widely used benchmarks to demonstrate the robustness and effectiveness of our CRCL under multiple scenarios, including synthetic noise, real-world noise, and well-annotated correspondence.
3.1 Datasets and Protocols
Datasets: For an extensive evaluation, three benchmark datasets (i.e., Flickr3KK [34]] MS- COCO (35] and CC152K [22) in our experiments. More specifically, Flickr30K is a widely-used image-text dataset collected from the Flickr website, which comprises 31,000 images and each one has corresponding textual descriptions. Following [36). 30,000 images are employed for training, 1.000 images for validation, and 1,000 images for testing in experiments. MS-COCO is a large-scale image-text dataset, which has 123,287 images, and 5 captions are given to describe each experiments, i.e., 5000 validation images, 5000 test images, and the rest for training. CC152K a subset of Conceptual Captions (CC) [10] collected in the real world, which selected by [12. Due to the absence of manual annotation, there are about 3% ~ 20% incorrect correspondences in CC, i.e., real-world noisy correspondences. CC152K contains 150,000 image-text pairs for training, 1,000 pairs for validation, and 1,000 pairs for testing.
image. We follow the split of [36]]] g to carry
Evaluation Protocols: Recall at K (R@K=1, 5, and 10) is used to measure the performance of bidirectional retrievals, which is defined as the proportion of the queries with the correct item in the top K retrieved results. Besides, flowing [9 we also take the sum of all Recalls to evaluate the overall performance, iee., rSum.
3.2
Implementation Details
Our CRCL a generalized robust framework that could extend existing methods to confront noisy correspondences. To demonstrate the effectiveness and robustness of CRCL, we extend two repre- sentative methods, i.e., VSExx and SGRAF (SGR and SAF) 4 to perform robust image-text matching, respectively. Specifically, the shared hyper-parameter are set as the same the original works [4] 91. e.g., the batch size is 128, the word embedding size is 300, and the joint embedding and implementation details given in our
dimensionality is 1,024. More specific supplementary material due to the space limitation.
3.3 Comparison with State-of-the-Arts
In this section, we evaluate our CRCL by comparing it with 7 state-of-the-art methods on three benchmarks, i.e., SCAN (ECCV' 18) R], SGRAF (SGR and SAF, AAAI'21) 4, VSEoo (CVPR'21)
[9]. NCR (NeurIPS'2I) [12. DECL (ACM MM'22) [9), BiCro (CVPR'23) (2 and MSCN

---

(CVPR 23) 24 For a fair comparison, all tested approaches adopt the same visual features (BUTD features) and textual backbone Bi-GRU [37]. To comprehensively investigate the robustness of our method, we artificially inject synthetic false correspondence of different ratios by proportionally shuffling the captions on Flickr30K and MS-COCO like [1]. i.e, 20%, 40%, 60%, and 80% noise rates. In addition to synthetic noise, we also evaluate the robustness of tested methods against the real-world noisy correspondences on CC152K. Due to the space limitation, we only provide the results on MS-COCO 5K under well-annotated correspondences in Table 2 the ensemble results CRCL-SGRAF are reported the paper. More extensive comparison results provided in the supplementary material to fully demonstrate the superiority of CRCL.
For faimess, like [9 22)
Table 1: Performance comparison (R @ ((%) and rSum) of image-text matching on Flickr30K and MS-COCO IK. The highest scores are shown in bold. ** means robust methods.
Fhick30K
MS-COCO 1K
Image
Text
Text + Image
Image - Text
Text + Image
Noise
Methods
R@1
R@5
R@ 10
R@
R@5
R@
rSum
R@1
R@5
R@ 10
R@
R@5
R@
rSum
SCAN
56.4
81.7
89.3
34.2
65.1
75.6
402.3
28.9
64.5
79.5
20.6
55.6
73.5
322.6
SAF
51.8
79.5
88.3
38.1
66.8
76.6
401.1
41.0
78.4
89.4
38.2
74.0
85.5
406.5
SGR
61.2
84.3
91.5
44.5
72.1
80.2
433.8
49.1
83.8
92.7
42.5
77.7
88.2
434.0
VSE.0
69.0
89.2
94.8
48.8
76.3
83.8
461.9
73.5
93.3
97.0
57.4
86.5
92.8
500.5
20%
NCR*
76.7
93.9
96.9
57.5
82.8
89.2
497.0
77.0
95.6
98.1
61.5
89.3
95.1
516.6
D
75.6
93.8
97.4
58.5
82.9
89.4
497.6
77.1
95.9
98.4
61.6
89.1
95.2
517.3
BiCro*
78.1
94.4
97.5
60.4
84.4
89.9
504.7
78.8
96.1
98.6
63.7
90.3
95.7
523.2
MSCN
77.4
94.9
97.6
59.6
83.2
89.2
501.9
78.1
97.2
98.8
64.3
95.8
524.6
CRCL
77.9
95.4
98.3
60.9
84.7
90.6
507.8
79.6
96.1
98.7
64.7
90,6
95,9
525.6
SCAN
29.9
60.5
72.5
16.4
38.5
48.6
266.4
30.1
65.2
79.2
18.9
51.1
69.9
314.4
SAF
34.3
65.6
78.4
30.1
58.0
68.5
334.9
36.0
74.4
87.0
33.7
69.4
82.5
383.0
47.2
76.4
83.2
34.5
60.3
70.5
372.1
43.9
78.3
89.3
37.0
72.8
85.1
VSE
30.2
58.3
70.2
22.3
49.6
62.7
293.3
53.3
84.3
92.1
31.4
63.8
75.0
399.9
40%
NCR*
75.3
92.1
95.2
56.2
80.6
87.4
486.8
76.5
95.0
98.2
60.7
88.5
95.0
513.9
DECL*
72.5
93.1
97.0
55.8
81.2
88.1
487.7
77.1
95.7
98.3
61.5
89.2
95.3
517.1
BiCro*
74.6
92.7
96.2
55.5
81.1
87.4
487.5
77.0
95.9
98.3
61.8
89.2
94.9
517.1
MSCNS
74.4
94.44
96.9
57.2
81.7
87.6
492.2
74.8
94.9
60.3
510.9
CRCL
77.8
95.2
98.0
60.0
84.0
90.2
505.2
78.2
95.7
98.3
63.3
90.3
95.7
521.5
SCAN
16.9
39.3
53.9
2.8
7.4
11.4
131.7
27.8
59.8
74.8
16.8
47.8
66.4
293.4
28.3
54.5
67.5
22.1
47.3
59.0
278.7
28.2
63.9
79.4
31.1
65.6
80.5
348.7
SGR
58.0
71.0
23.8
49.5
60.7
291.7
37.6
73.3
86.3
33.8
81.7
381.3
VSE
18.0
44.0
55.7
15.1
38.5
$1.8
223.1
33.4
64.8
79.1
26.0
60.1
76.3
339.7
60%
NCR*
68.7
89.9
95.5
52.0
77.6
84.9
468.6
72.7
94.0
97.6
57.9
87.0
94,1
503.3
DECL
69.4
89.4
95.2
52.6
78.8
85.9
471.3
73.8
94.7
97.7
59.6
87.9
94.5
508.2
BiCro*
67.6
90.8
94.4
51.2
77.6
84.7
466.3
73.9
94.4
97.8
58.3
87.2
93.9
505.5
M
70.4
91.0
94.9
53.4
77.8
84.1
471.6
74.4
95.1
97.9
59.2
87.1
92.8
506.5
CRCL
73.1
93.4
95.8
54.8
81.9
88.3
487.3
76.3
95.1
97.9
60.8
89.0
95.1
514.2
SCAN
5.1
18.1
27.3
3.9
13.1
19.1
86.6
22.2
51.9
67.5
13.8
41.1
58.6
255.1
12.2
32.8
48.4
11.8
30.5
41.5
177.2
24.2
57.5
74.1
24.7
$7.1
73.0
310.6
SGR
13.7
35.1
47.6
12.1
30.9
41.9
181.3
26.7
60.7
75.6
25.3
58.2
72.6
319.1
VSE
8.1
23.1
34.7
7.4
22.6
31.8
127.7
25.4
55.1
70.6
19.2
50.5
68.0
288.8
80%
NCR
1.4
7.1
11.7
1.5
5.4
9.3
36.4
21.6
52.6
67.6
15.1
38.1
49.8
244.8
DECL*
60.7
84.6
91.2
42.1
69.6
78.6
426.8
65.6
91.6
96.6
$2.0
83.0
91.3
480.1
BiCro*
13.9
20.5
1.7
7.5
13.0
40.0
72.6
84.7
22.6
67.2
340.1
MSCN*
1.0
4.4
9.1
0.4
1.4
2.5
18.8
66.8
91.6
96.2
52.7
83.0
90.9
481.2
CRCL#
62.3
86.8
92.8
46.0
73.6
82.2
443.7
72.7
93.5
97.6
57.5
86.8
93.7
501.8
3.3.1 Results under Synthetic Noisy Correspondences
For quantitative evaluation under specific noise levels, we conduct all tested methods under four different noise rates (i.e., 20%, 40%, 60%, and 80%) of synthetic noisy correspondences on the Flickr30K and MS- MSSCOCO datasets. Quantitative results on Flickr30K and MSCOCO IK test set are shown in Table I test images. From the results, one can see that our CRCL could remarkably outperform the robust baselines (NCR, DECL, BiCro, and MSCN) on most of the metrics, which demonstrates the superior robustness of CRCL against NC. Moreover, our CRCL not only performs well in low noise but also achieves the best performance under high noise, especially 80% noise, which provides strong evidence for the stability and robustness of our method.
For MS-COCO, the results are computed by a veraging over folds of 1K
7

---

Table 2: Performance comparison on CC152K and MS-COCO 5K.
CC152K
MS-COCO 5K
Image +
Text
Text
Image
Image
Text
Text
Image
Methods
R@
R@5
R@
10
R@
R@5
R@ 10
iSum
R@1
R@5
R 10
R@)
R@S
@@ 10
rSum
SCAN
30.5
55.3
65.3
26.9
53.0
64.7
295.7
44.7
75.9
86.6
33.3
63.5
75.4
379.4
VSFxo
34.0
64.5
77.0
12.9
19.2
21.6
229.2
56.6
83.6
91.4
39.3
69.9
81.1
421.9
SGRAF
32.5
59.5
70.0
32.5
60.7
68.7
323.9
58.8
84.8
92.1
41.6
70.9
81.5
429.7
NCR*
39.5
64.5
73.5
40.3
64.6
73.2
355.6
58.2
84.2
91.5
41.7
71.0
81.3
427.9
DECL*
39.0
66.1
75.5
40.7
66.3
76.7
364.3
59.2
84.5
91.5
41.7
70.6
81.1
428.6
M
40.1
65.7
76.6
40.6
67.4
76.3
366.7
BiCr+
40.8
67.2
76.1
42.1
67.6
76.4
370.2
59.0
84.4
91.7
42.4
71.2
81.7
430.4
CRCL#
41.8
67.4
76.5
41.6
68.0
78.4
373.7
61.3
85.8
92.7
43.5
72.6
82.7
438.6
3.3.2
Results under Real Noisy Correspondences
In addition synthetic noise, we carry the comparison experiments on the real-world noisy dataset CC152K. The quantitative results are shown in Table 2 From the results, one see that our CRCL is superior to all baselines with the best overall performance of 373.7%, which indicates that the proposed method is robust against real-world noise. Specifically, CRCL outperforms the best baseline BiCro, with absolute performance improvement of 1.0%, 0.2%, 0.4%, 0.4%, and 2.0% across different metries, except for @@) I in text-to-image retrieval.
3.3.3 Results under Well-annotated Correspondences
Besides noisy correspondences, we also evaluate the tested methods trained on the well-aligned MS-COCO dataset for comprchensive comparison. The results MS-COCO 5K shown in Table 2] wherein the results of all baselines are reported by the original papers for a fair comparison, except for the reproduced results of BiCro. From the table, our CRCL remarkably outperforms all baselines terms of rSum. Specifically, CRCL prevails over the best baselines by 8.2% on overall performance (i.e., rSum) absolutely, which shows that CRCL is not only suitable for noisy cases but also performs well in well-aligned ones.
3.4 Comparison to pre-trained model
In this section, we compare our CRCL to the pre-trained model CLIP [3 to further evaluate its effectiveness handling NC. CLIP is a well-known large pre-trained model that is trained from scratch dataset of 400 million image-tex( pairs collected from the Internet, which includes large number of training pairs with real NCs. More specifically, following [12], we report the zero-shot results and fine-tuning results under 20% noise and compare them with that of CRCL-SGRAP in Table From the results, CLIP shows a significant performance drop during fine-tuning under NC. On the contrary, the performance of our CRCL under 20% noise is even better than the zero-shot result of CLIP (ViT-L/14), which shows the strength and potential of our CRCL in dealing with NC.
Table 3: Comparison with CLIP on MS-COCO SK.
Image
Text
Text
Image
Noise Rate
Methods
R@1
R@5
R@ 10
R@1
R@5
R 10
rSum
CLIP (ViT-L/14)
58.4
81.5
88.1
37.8
62.4
72.2
400.4
0%, Zero-Shot
CLIP (VIT-B/32)
50.2
74.6
83.6
30.4
56.0
66.8
361.6
CLIP (ViT-L/14)
36.1
61.3
72.5
22.6
43.2
53.7
289.4
20%, Fine-tune
CLIP (ViT-B/32)
21.4
49.6
63.3
14.8
37.6
49.6
236.3
Our CRCL
59.3
85.2
91.9
42.9
71.9
82.1
433.3
3.5
Ablation Study
In this section, we present ablation studies conducted on Flickr30K with 60% noise, as shown in T From the table, one can find that the full version of our CRCL achieves the best performance, which indicates that each component contributes to our method for performance improvement. By recasting & with the rectified label tu using SCC, C, shows higher potential against NC. Moreover,

---

through the combination of active learning and robust complementary learning, there is further performance improvement, which indicates the effectiveness of our ACL. Note that these results are obtained from single model for comprehensive evaluation, and are not ensemble results as shown Tables 2
Table 4: Ablation studies on Flickr30K with 60% noise.
Configuration
Image +
Text
Text + Image
loss
SCC
R @
R@5 5
R@
10
R@]
R R@5
R@ 10
rSum
Loel Ld
v
70.5
91.3
95.6
52.5
79.4
86.8
476.1
v
69.3
91.1
95.2
51.4
78.2
86.0
471.2
C
V
67.7
91.2
95.0
52.3
79.0
86.4
471.6
(g 1)
25.2
53.9
65.6
20.3
45.0
57.2
267.2
( = 0)
63.5
89.7
93.8
48.2
74.5
82.0
451.7
Ld
65.6
87.5
93.2
47.7
74.6
82.7
451.3
3.6 Visualization Analysis
To further investigate the generalization and robustness of CRCL, we visually study the retrieval performance our CRCL-VSExx, CRCL-SAF, and CRCL-SGR on Flickr30K and MS-COCO with varying noise rates in Figure [a.d). From the figure, one can see that CRCL can enhance the robustness of the original methods and perform well even under high noise. addition, we provide visualization to intuitively showcase the effectiveness of the proposed SCC. Specifically, we visualize the distributions of both cross-modal similarities and corrected correspondences for noisy training pairs using CRCL-SGR results without SCC, where we replace Da with the prediction (I; T:) for the current iteration. The visualizations clearly indicate that SCC prevents the over-accumulation of errors in correspondence correction, thus verifying its effectiveness in mitigating the impact of NC.
Flickr30K with 60% NCs. For comparison, we also visualize the
Clean Parn
50
Cleon Palls
Mokay Parrs
Mo 7a
70
40
Dendty
30
VSEw SAP
20
20
30-
SOA
10-
10
1.
Similartty
Gome,pendence
(a) Flickr30K
(b) Similarities w/o SCC
(c) Correspondences w/o SCC
9.
40
Can farx Rois Parrs
50
Clean
80
35-
40-
140011 R.5. 500
Denuy
30
82-
USE
GAT
5S3
CRCLIVE- CRCL-SOR
0.5
0.4
0.5
0.8
0.5
Simkarity
(d) MS-COCO
(c) Similarities with SCC
(f) Correspondences with SCC
Figure I: (a.d) The performance on Flickr30K and MS-COCO with varying noise rates; (b,c/e,f) The similarities and corrected correspondences of training pairs after learning without/with SCC.
4 Conclusion
In this paper, we propose generalized robust framework CRCL. endow traditional methods with robustness against noisy correspondences. To alleviate the harmful effects brought by NC, present

---

an Active Complementary Loss (ACL) and Self-refining Correspondence Correction technique (SCC). These techniques enable effective learning of visual-semantic associations and progressive correction of correspondences based on historical predictions, thus boosting the robustness of image- text matching. Extensive experiments demonstrate that our CRCL achieves state-of-the-an art robustness to both synthetic and real-world NCs. Furthermore, ablation studies and visualization analyses further verify the effectiveness and reliability of the proposed components.
5 Limitations and Broader Impact Statement
Despite the promising performance of our proposed CRCL, there are some limitations that should be acknowledged. First, we only study the NC problem between two modalities, i.e., image and text. Second, we did not take into account category-level noise. The proposed CRCL likely impacts various applications that require robust image-text matching, e.g., multimedia retrieval, and image annotation. We encourage further study to understand and mitigate the biases and risks potentially brought by image-text matching.
Acknowledgments and Disclosure of Funding
This work was supported by the National Natural Science Foundation of China (U19A2078, U21B2040, 62372315, 62176171, and 62102274), Sichuan Science and Technology Planning Project (2023YFQ0020, 2023YFG0033, 20237HCG0016, 2022YFQ0014, 2022YFH0021), Chengdu Sci- ence and Technology Project 0103--X000000004--0 the SCU-LuZhou Sciences and Technology Coorperation Program (2023CDLZ-16), and Fundamental Research Funds for the Central Universities under Grant YJ202140.
References
[1] Mateusz Malinowski, Marcus Rohrbach, and Mario Fritz. Ask your neurons: A neural-based approach to answering questions about images. In Proceedings of the IEEE international conference computer vision, pages 1-9, 2015.
[2] Oriol Vinyals, Alexander Toshev, Samy Bengio, and Dumitru Erhan. Show and tell: neural image caption generator. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3156-3164, 2015.
[3] Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, Aaron Courville, Ruslan Salakhudinov, Rich Zemel, and Yoshua Bengio. Show, attend and tell: Neural image caption generation with visual attention. In International conference on machine learning, pages 2048-2051 PMLR, 2015.
[4] Haiwen Diao, Ying Zhang, Lin Ma, and Huchuan Lu. Similarity reasoning and filtration for image-text matching. In Proceedings of the AAAI Conference Artificial Intelligence, volume 35, pages 1218-1226, 2021.
[5] Ding Jiang and Mang Ye. Cross-modal implicit relation reasoning and aligning for text-to-image person retrieval. arXiv preprint arXiv:2303. 12501, 2023.
[6] Peng Hu, Hongyuan Zhu, Jie Lin. Dezhong Peng. Yin-Ping Zhao, and Xi Peng. Unsuper- vised contrastive cross modal hashing. IEEE Transactions Pattern Analysis and Machine Intelligence, pages 2022.
[7] Fartash Faghri, David J Fleet, Jamie Ryan Kiros, and Sanja Fidler. Vse++: Improving visual- semantic embeddings with hard negatives. arXiv preprint arXiv: 1t07vv:707.05/1111 07.60 2017.
[8] Kuang-Huei Lee, Xi Chen, Gang Hua, Houdong Hu, and Xiaodong He. Stacked cross attention for image-text matching. In Proceedings of the European conference computer vision (ECCV), pages 201-216, 2018
[9] Jiacheng Chen, Hexiang Hu, Hao Wu, Yuning Jiang, and Changhu Wang. Learning the best pooling strategy for visual semantic embedding. In Proceedings of the IEEF/CVE conference on computer vision and pattern recognition, pages 15789-15798, 2021.
10

---

[10] Piyush Sharma, Nan Ding. Sebastian Goodman, and Radu Soricut. Conceptual captions: A cleaned, hypernmeed, image alt-text dataset for automatic image captioning. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pages 2556- 2565, 2018.
[11] Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoe Le, Yun- Hsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and vision-languee representation learning with noisy text supervision. In International Conference on Machine Learning, pages 4904 4916. PMLR, 2021.
[12] Zhenyu Huang, Guocheng Niu, Xiao Liu, Wenbiao Ding, Xinyan Xiao, Hua Wu, and Xi Peng. Learning with noisy correspondence for cross-modee matching. Advances in Neural Information Processing Systems, 34:29406-2944 2021.
[13] Mouxing Yang. Yunfan Li, Peng Hu, Jinfeng Bai, Jiancheng Lv, and Xi Peng. Robust multi-view clustering with incomplete information. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(1):1055-1069 2022.
[14] Mouxing Yang, Yunfan Li, Zhenyu Huang, Zitao Liu, Peng Hu, and Xi Peng. Partially view-aligned representation learning with noise-robust contrastive loss. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 1134 143, 2021.
[15] Jie Wen, Chengliang Liu, Shijie Deng, Yicheng Liu, Lunke Fei, Ke Yan, and Yong Xu. Deep double incomplete multi-view multi-label learning with incomplete labels and missing views. IEEE Transactions on Neural Networks and Learning Systems, 2023.
[16] Xu Yang, Cheng Deng, Kun Wei, Junchi Yan, and Wei Liu. Adversarial learning for robust deep clustering. Advances in Neural Information Processing Systems, 33:9098-9108, 2020.
[17] Huaiwen Zhang, Yang Yang. Fan Qi, Shengsheng Qian, and Changsheng Xu. Robust video-text retrieval via noisy pair calibration. IEEE Transactions on Multimedia, 2023.
[18] Mouxing Yang, Zhenyu Huang, Peng Hu, Taihao Li, Jiancheng Lv, and Xi Peng. Learning with twin noisy labels for visible-infrared person re- iddentification. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14308-143) 2022.
[19] Yang Qin, Dezhong Peng, Xi Peng, Xu Wang, and Peng Hu. Deep evidential learning with noisy correspondence for cross-modar retrieval. In Proceedings of the 30th ACM International Conference Multimedia, pages 4948 0956649566626 2022.
[20] Peng Hu, Zhenyu Huang, Dezhong Peng, Xu Wang, and Xi Peng. Cross-modal retrieval with partially mismatched pairs. IEEE Transactions on Pattern Analysis and Machine Intelligence, pages 1-15, 2023.
[21] Junnan Li, Richard Socher, and Steven CH Hoi. Dividemix: Learning with noisy labels as semi-supervised learning, arXiv preprint arXiv:2002.07394, 2020.
[22] Bo Han, Quamming Yao, Xingrui Yu, Gang Niu, Miao Xu, Weihua Hu, Ivor Tsang, and Masashi Sugiyama. Co-teaching: Robust training of deep neural networks with extremely noisy labels. Advances in neural information processing systems, 31, 2018.
[23] Shuo Yang. Zhaopan Xu, Kai Wang, Yang You, Hongxun Yao, Tongliang Liu, and Min Xu. Biero: Noisy correspondence rectification for multi-modality data via bi-directional cross-modal similarity consistency. arXiv preprint arXiv:2303.122 12419, 2023.
[24] Haochen Han, Kaiyao Miao, Qinghua Zheng, and Minnan Luo. Noisy correspondence learning with meta similarity correction. arXiv preprint arXiv:2304.06275. 2023.
[25] Devansh Arpit, Stanistaw Jastrzęhski, Nicolas Ballas, David Krueger, Emmanuel Bengio, Maxinder S. Kanwal, Tegan Maharaj, Asja Fischer, Aaron Courville, Yoshua Bengio, and Simon Lacoste-Julien. A closer look at memorization in deep networks. In Doina Precup and Yee Whye Teh, editors, Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pages 233-242, PMLR, 06-11 Aug 2017.
[26] Aritra Ghosh, Himanshu Kumar, and P Shanti Sastry. Robust loss functions under label noise for deep neural networks. Proceedings of the AAAI conference on artificial intelligence, volume 31, 2017.
11

---

[27] Jianfeng Dong, Xirong Li, Chaoxi Xu, Shouling Ji, Yuan He, Gang Yang. and Xun Wang. Dual encoding for zero- example video retrieval. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 3346-9355, 2019.
[28] Naresh Manwani and PS Sastry. Noise tolerance under risk minimization. IEEE Iransactions on cybernetics, 43(3): 146-1111, 2013.
[29] Zhirong Wu, Yuanjun Xiong, Stella X Yu, and Dahua Lin. Unsupervised feature learning via non parametric instance discrimination. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3733-3742, 2018.
[30] Mathilde Caron, Ishan Mísra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised learning of visual features by contrasting cluster assignments. Advances Neural
Information Processing Systems, 33:9912 2020.
[31] Peng Hu, Xi Peng, Hongyuan Zhu, Liangli Zhen, and Jie Lin. Learning cross-modal retrieval with noisy labels. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5403-5413, 2021.
[32] Xingjun Ma, Hanxun Huang, Yisen Wang, Simone Romano, Sarah Erfani, and James Bailey. Normalized loss functions for deep learning with noisy labels. In International conference machine learning, pages 65533. PMLR, 2020.
[33] Zhilu Zhang and Mert Sabuncu. Generalized cross entropy loss for training deep neural networks with noisy labels. Advances in neural information processing systems, 31, 2018.
[34] Peter Young, Alice Lai, Micah Hodosh, and Julia Hockenmaier. From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions.
Transactions of the Association for Computational Linguistics, 2:67-78, 2014.
[35] Tsung-Yi Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European conference on computer vision, pages 740-755. Springer, 2014.
[36] Andrej Karpathy and Li Fei-Fei. Deep visual-semantic alignments for generating image descriptions. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 3128-3137, 2015.
[37] Mike Schuster and Kuldip K Paliwal. Bidirectional recurrent neural networks. IEEE transactions on Signal Processing, 45(1 1):2733-2681 1997.
[38] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning.
pages 8748 8763. PMLR, 2021.
12


Reviews:
[{"review_id": "39jMX6gVQU", "reviewer": "Reviewer_J1KY", "summary": "This paper focuses on the problem of noise correspondence in image-text matching tasks. To address this issue, this paper proposes a generalized cross-modal robust complementary learning framework, which not only reduces the risk of erroneous supervision from the active complementary loss but also obtains stable and accurate correspondence correction through a self-refining correspondence correction. Extensive experiments show that the developed model could significantly improve the effectiveness and robustness compared with the state-of-the-art approaches. The topic of this paper is of great practical interest and the motivation is clear.", "questions": "Please refer to the item \"weaknesses\".", "limitations": "In addition to the effectiveness of the experimental results, I suggest that the authors can apply the proposed framework in more methods to enhance its robustness.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "a.\tThis paper proposes a novel generalized cross-modal robust complementary learning framework to address the noisy correspondence problem in image-text matching tasks, which enhances the effectiveness of existing methods through robust loss and correction techniques.\nb.\tThis paper utilizes the active complementary loss that employs complementary pairs to conduct indirect cross-modal learning with exponential normalization to boost robustness against noisy correspondence. In addition, self-refining correspondence correction is proposed to obtain stable and accurate correspondence correction.\nc.\tExtensive experiments are provided to prove the effectiveness and robustness of the proposed model.\n", "weaknesses": "a.\tEssentially, the authors propose a new loss function to solve the noise correspondence problem in the image-text matching task. The proposed active complementary loss and self-refining correspondence correction both serve the final loss. So I think the novelty may be limited.\nb.\tI noticed in the supplementary that when the framework proposed in this paper is applied to the VSE model, there is a leap in performance improvement (such as 60% noise, Flickr30K dataset, The bidirectional retrieval improved by 50.3% and 35.4% on R@1, respectively.) while the performance improvement was limited when applied to the SGRAF model. Therefore, I think the author should explain why the performance gap is so large. More extensive experiments should also be organized to demonstrate the robustness of the proposed framework.\nc.\tTo my knowledge, in previous studies, the noise injected on the Flickr30K and MS COCO datasets was different. So how is the noise injected by the author generated? \n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your valuable comments. We have carefully looked into all comments. Attached is our point-by-point response.\n\n**Q1. Regarding the concern of novelty .**\n\nThanks for your comment but we disagree with your opinion. Although some methods are proposed to address noisy correspondence (NC), they still face some outstanding issues that limit their performance: overfitting, underfitting, and inefficiency. To be specific, **(1)** Previous works explore different robust losses to alleviate the adverse impact of NC, however, almost all of them face overfitting or underfitting problems, especially under high noise rates. Specifically, active losses own fast-fitting ability but easily overfit noise. On the other hand, some robust losses have high robustness but easily underfit clean data. It is still an open issue of how to balance efficient learning and robustness, however, which is less touched so far. To address this issue, we present a dynamic balance strategy to make active and robust losses promote each other for robust learning. **(2)** Due to the memorization effect of DNNs, cross-modal learning will face the self-sample-selection error accumulation problem, which will degrade the performance. To alleviate the error accumulation, previous robust frameworks (e.g., NCR and BiCro) utilize the co-teaching manner to obtain accurate predictions. However, the co-teaching strategy would increase the number of models,  which will largely increase the training overhead. To address this problem, we present an efficient Self-refining Correspondence Correction (SCC) without any additional model to tackle the error accumulation problem. The efficiency comparison is shown in the response (Q.2) to Reviewer DZ4B.\n\nFurthermore, our method is a generalized framework that can endow existing image-text matching methods with robustness and stability under NC. We have extended several classical image-text matching methods, e.g., VSE$\\infty$, SGR, SAF, and CLIP (see Q.2), all of which have shown superior performance and robustness. In summary, we think that this work is with sufficient novelty and technical contribution. \n\n**Q2. (a) I think the author should explain why the performance gap is so large. (CRCL-VSE$\\infty$/SGRAF vs. VSE$\\infty$/SGRAF) (b) More extensive experiments should also be organized to demonstrate the robustness of the proposed framework.**\n\nThank you for your careful comment and valuable suggestion. **(a)** We think that the discrepancy in performance improvement is due to the differences in the network models. Different network models have different behaviors and sensitivities to noisy correspondences, which result in different absolute performance gains. For example, as shown in Table 2, the performance of VSE$\\infty$ and the SGRAF models varies under different noise rates. Even for SGR and SAF, which have the same framework with a minor difference in a subnetwork, their performance differs significantly in Table 2. Therefore, the difference in network models affects how the methods perform under NC. This can also lead to a gap in absolute performance between different methods (e.g., VSE$\\infty$, SAF, and SGR). Moreover, from the experimental results, we can see that our CRCL consistently improves the methods with different network models, which demonstrates the effectiveness and generalization of our method.\n\n**(b)** As suggested, we applied CRCL on the classic pre-trained model CLIP(ViT/32) to further verify the robustness of our method, in addition to the CRCL-VSE$\\infty$, CRCL-SAF, and CRCL-SGR models reported in the manuscript. We conducted a more extensive evaluation on the Flickr30K dataset and obtained the following partial results:\n|||I2T|I2T|I2T|T2I|T2I|T2I|||\n|-|-|-|-|-|-|-|-|-|-\n|Configurations|Methods|R@1|R@5|R@10|R@1|R@5|R@10|rSum|$\\uparrow$(%)\n|Zero-shot|CLIP|78.6|95.4|97.7|59.8|85.1|90.9|507.5\n|Finetune on 40% Noise|CLIP$_{best}$|81.9|95.3|98.1|66.1|88.7|93.6|523.7\n|Finetune on 40% Noise|CLIP$_{last}$|64.7|86.8|91.1|45.7|68.5|76.2|433.0|-90.7%   \n|Finetune on 40% Noise|CRCL-CLIP$_{best}$|83.2|96.2|98.7|67.3|89.5|94.2|529.1 \n|Finetune on 40% Noise|CRCL-CLIP$_{last}$|83.2|96.2|98.6|67.4|89.5|94.2|529.1|-0.0%\n|Finetune on 80% Noise|CLIP$_{best}$|71.0|90.0|95.0|54.5|79.7|87.1|477.3\n|Finetune on 80% Noise|CLIP$_{last}$|28.8|51.4|63.0|17.5|33.9|42.2|236.8|-240.5%\n|Finetune on 80% Noise|CRCL-CLIP$_{best}$|83.0|96.1|98.4|66.8|89.1|94.0|527.4\n|Finetune on 80% Noise|CRCL-CLIP$_{last}$|83.1|96.0|98.68|66.7|89.1|94.0|527.5|+0.1%\n\nNote that $best$ means the results from testing the checkpoint with the best validation performance, and $last$ means using the last checkpoint to perform testing. By comparing the results of the $best$ and $last$ rows, we can reflect the degree of overfitting NC by the performance degradation, and then understand the robustness of our method. From the results, our CRCL not only improves the performance of CLIP on the dataset with NC, but also alleviates the performance degradation (compare $best$ and $last$ rows) caused by noise overfitting, which demonstrates the strong robustness of CRCL. For more verification of the robustness, we can also refer to Fig. 2 in the supplementary material. Fig. 2 records the variation in validation performance during training. Except for the full version of CRCL, the rest variants suffer from a noise overfitting problem, such as the CCL loss, $\\mathcal{L}_d$, and the TR loss, which also shows that CRCL has better robustness.\n\n**Q3. Regarding the method to generate noisy correspondence.**\n\nThank you for your detailed comments. To generate noisy correspondence, we followed the standard method used in NCR [12], which is **randomly shuffling the captions of training images for a specific percentage**. This way, we can ensure a fair comparison with previous works by using the same noise. For a more comprehensive evaluation, we reproduced the results of some baselines under four noise rates."}, {"review_id": "Xqeit3Jm37", "reviewer": "Reviewer_8spS", "summary": "This manuscript focuses on image-text matching under the noisy correspondence setting. To achieve a noise robust multi-modal representation, the authors propose two components, including a Active Complementary Loss (ACL) and a Self-Refining Correspondence Correction (SRCC). In ACL, a complementary contrastive loss styled fomula is derived, in which a coefficient $q$ is set in seeking for a tighter bound between the divergence between risk of training with noisy correspondence and ideal setting. As for SRCC, the labeled matching score is relaxed by momentum updating to alleviate the noise. Finally, the authors conduct extensive experiments on image-text retrieval to show their performance. ", "questions": "None", "limitations": "None", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The motivation of the manuscript is novel, which is from noise tolerance loss function designing and noisy label correction simultaneously.\n\n2. The proposed Active Complementary Loss has rigorous theoretical proof in evidenting the tighter bound to the divergence between noisy risk with ideal risk.\n\n3. The experiments are adequate and extensive. What' s more, as the Tab. 1 shown, the proposed method is very stable under different noisy ratios, and the improvment is non-trivial under larger noisy ratio.", "weaknesses": "1. Lacks necessary explaination in figures. Actually, the pure text claim to the proposed method could be harder to grab.\n\n2. The Sec. 2.2 is complex and hard to follow. Beyond theoretical proof, how ACL works in intuition should be further discussed.\n\n3. The existing organization to the proposed two components is poor. Actually, I didn' t see a clear connection between Sec. 2.2 with Sec. 2.3. Despite that they are all for noise correspondence, the author fail to discuss the relationship between ACL and SRCC. If not, I could treat this manuscript as a naive A+B technic.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your valuable comments and constructive suggestions. We will address your concerns and questions one by one as follows.\n\n**Q1. Lacks necessary explanation in figures. Actually, the pure text claim to the proposed method could be harder to grab.**\n\nThank you for your constructive comments and suggestion. To make our method more clear and more understandable, we have added a ***pdf file*** to ***the global Author Rebuttal*** that contains an illustration of our framework (Figure 1). Here is a brief explanation of the illustration, as follows:\n\n*This is an illustration of our CRCL framework. CRCL consists of two components: an Active Complementary Loss (ACL) and a Self-refining Correspondence Correction (SCC). ACL balances between active and complementary learning based on the matching probabilities computed by the cross-modal model. It uses the corrected labels from SCC to conduct reasonable direct cross-modal learning by focusing on the active loss of convincing data and also recasting the regulatory factor to provide robust tolerance for possible noisy data under risk minimization theory. SCC exploits multiple self-refining processes with momentum correction to enlarge the receptive field for correcting correspondences, thus obtaining accurate labels and alleviating error accumulation.*\n\n\n**Q2. The Sec. 2.2 is complex and hard to follow. Beyond theoretical proof, how ACL works in intuition should be further discussed.**\n\nThank you for your valuable suggestion. The main idea behind our ACL is to balance between the underfitting property of complementary loss and the noise overfitting disadvantage of active loss, thus improving the robustness against noisy correspondence (NC). Previous works explore different robust losses to alleviate the adverse impact of NC, but they often face overfitting or underfitting problems, especially under high noise rates. For example, active losses can fit the data quickly but they will overfit the noise easily. On the other hand, some robust losses can resist the noise well but they will underfit the clean data. Therefore, we propose our ACL to achieve a balance between efficient learning and robustness. ACL introduces exponential normalization into complementary loss to control the risk difference between noisy and ideal clean data, thus ensuring robustness for noisy data. However, since complementary learning has a long-standing underfitting problem, we still need to pay more attention to positive/matched pairs for direct efficient learning. Therefore, we introduce a weighted active learning loss into ACL that focuses more on convincing pairs. We will give more intuitive explanations in the next version.\n\n**Q3. The existing organization of the proposed two components is poor. Actually, I didn't see a clear connection between Sec. 2.2 with Sec. 2.3. Despite that they are all for noise correspondence, the author fails to discuss the relationship between ACL and SCC. If not, I could treat this manuscript as a naive A+B technic.**\n\nThank you for your detailed review. We would like to clarify that our proposed method is not a simple A+B technique. ACL and SCC are complementary and indispensable components in our proposed framework, which can also be seen from our ablation experiments in Table 4, i.e., SCC without our ACL will yield an inferior performance. More specifically, ACL actually needs to use corrected labels to provide an ideal weighted score for active loss and an ideal regulatory factor for robust complementary loss, which suggests that our ACL actually relies on a very reliable correspondence correction technique. SCC plays such a role and is essential for ACL. To obtain accurate weighted scores, SCC leverages Momentum Correction to aggregate historical predictions, providing stable and accurate correspondence estimations. Furthermore, SCC combines multiple independent self-refining processes in training to eliminate error accumulation against noisy correspondences. The elimination of error accumulation allows ACL to provide better learning objectives for the model, thereby improving performance. So, SCC and ACL are mutually beneficial in our framework actually.\n\nThank you for your time and feedback. We hope this helps you understand our work better. If you have any more questions or comments, please let us know."}, {"review_id": "2fyQXeaPSo", "reviewer": "Reviewer_QLFm", "summary": "This paper presents a novel framework (CRCL) for cross-modal correspondence learning that can handle noisy image-text pairs. The key idea of CRCL is to use a complementary active loss (ACL) that balances between discriminative learning and robust learning. ACL leverages the rectified correspondence labels to adjust the loss function and assign active loss weights, which means that the potentially noisy pairs are more focused on robust learning. To obtain accurate rectified correspondence labels, the paper proposes a self-refining correspondence correction technique (SCC) that estimates the correlation between modalities and avoids error accumulation. The proposed framework is simple yet effective in mitigating the negative impact of noisy training pairs and achieving robust image-text matching. The paper also conducts extensive experiments on three benchmark datasets to demonstrate the effectiveness and rationality of the proposed method.", "questions": "This paper is well-written and easy to follow. It has all the essential elements of a good paper and I do not have any major objections to accept it. I appreciate the authors’ approach, which is more concise and theoretically sound than the existing robust image-text matching methods. However, there are some minor issues that need to be addressed, such as some typos and some recent related works that need to be further discussed. Noisy correspondence is an important research direction in the field of multimodality, as it is similar to noisy label learning. I hope the authors can not only improve the performance of their method, but also explore the potential impact of CRCL on more correspondence learning tasks.", "limitations": "The authors should enrich further, as described in the Questions section.", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "1. The CRCL framework seems to be a general and flexible approach that can be easily integrated with existing methods to enhance the robustness of image-text matching.\n\n2. This paper is well-written and easy to follow. The authors provide clear explanations and motivations for their method.\n\n3. The authors simplify the training process by discarding the co-teaching scheme used in NCR and BiCro, and instead focus on the loss function and the correction technique to make the method concise and clear.\n\n4. The experimental results are impressive and convincing. It is also noteworthy that CRCL can improve the performance of the pre-trained model, e.g., CLIP, as shown in Table 3.\n", "weaknesses": "1. In Eq.3, there is a typo: $i_j$ should be $I_j$.\n\n2. In Eq.11, some symbols are not clearly defined, e.g., $\\hat{p}^(j,t-1)(*)$. Please explain their meanings and notations.\n\n3. The NC problem can be viewed as a special case of the PMP problem. RCL[20] uses complementary contrastive learning (CCL) to deal with PMPs. Similarly, CRCL can also be regarded as a further extension and study of CCL. However, the paper does not provide a direct comparison between CRCL and RCL. It would be interesting to see how they differ in performance and insights.\n\n4. In the supplementary material, there is a capitalization error: “For brevity” should be “for brevity”.\n\n5. Why not include MSCN[A] as a baseline? MSCN seems to be from the same period as BiCro (CVPR’23) and also addresses the NC problem.\n\n6. Some related works are missing: [B-C]. These papers also propose methods for image-text matching and could be relevant for comparison or discussion.\n\n[A] Noisy Correspondence Learning With Meta Similarity Correction, CVPR, 2023.\n[B] Learning Semantic Relationship Among Instances for Image-Text Matching, CVPR, 2023.\n[C] Fine-Grained Image-Text Matching by Cross-Modal Hard Aligning Network, CVPR, 2023.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your valuable comments and insightful suggestions. We will address your concerns and questions one by one as follows.\n\n**Q1. In Eq.3, there is a typo: $i_j$  should be $I_j$.**\n\nThank you for your careful review. We will correct it in the next version.\n\n**Q2. In Eq.11, some symbols are not clearly defined, e.g., $\\hat{p}^{(j,t-1)}(*)$. Please explain their meanings and notations.**\n\nThank you for your suggestion. $\\hat{p}^{(j,t-1)}(I_i,T_i)$ denotes the average matching probability $(p_{ii}^\\circ + p_{ii}^\\diamond)/2$ of pair $(I_i,T_i)$ at the $(t-1)$-th epoch during the $j$-th SR training piece. We will supplement it in the next version.\n\n**Q3. Comparison results with RCL.**\n\nThank you for your valuable suggestion. We followed the same experimental setup in RCL [20] to evaluate our method and compare them. Due to the time limitation, we only report partial results of the SGRAF variants on the Flickr30K dataset, as follows:\n\n|||I2T|I2T|I2T|T2I|T2I|T2I|||\n|-|-|-|-|-|-|-|-|-|-\n|Noise|Methods|R@1|R@5|R@10|R@1|R@5|R@10|rSum|$\\uparrow$(%)\n|40%|RCL-SAF|68.8|89.8|95.0|51.0|76.7|84.8|466.1|\n|40%|CRCL-SAF|71.9|91.9|95.3|54.1|79.6|86.8|479.6|+13.5% \n|40%|RCL-SGR|71.3|91.1|95.3|51.4|78.0|85.2|472.3|\n|40%|CRCL-SGR|73.7|93.0|96.2|53.9|80.5|88.1|485.4|+13.1%\n|80%|RCL-SAF|45.0|72.8|80.8|30.7|56.5|67.3|353.1|\n|80%|CRCL-SAF|49.7|76.6|86.0|34.8|62.5|72.8|382.4|+29.3%\n|80%|RCL-SGR|47.1|70.5|79.4|30.3|56.1|66.3|349.7|\n|80%|CRCL-SGR|49.3|77.4|85.9|34.0|62.8|72.3|381.7|+32.0%\n\nFrom the above results, CRCL outperforms RCL on all metrics, especially on 80% noise. This demonstrates the effectiveness of our method in handling noisy correspondences (NC). RCL uses complementary contrastive learning (CCL) to provide robustness against NC, but as we discussed in the *Introduction* section, it does not explicitly mitigate the effect of easily separable noise to further improve performance. In contrast, our CRCL leverages SCC to provide more reasonable learning objectives for higher performance.\n\n\n**Q4. In the supplementary material, there is a capitalization error: “For brevity” should be “for brevity”.**\n\nThank you for your careful review. We will correct it in the next version.\n\n**Q5. Why not include MSCN[A] as a baseline? MSCN seems to be from the same period as BiCro (CVPR’23) and also addresses the NC problem.**\n\nWe appreciate your valuable comments and helpful suggestions. We agree that MSCN is a strong baseline and should be compared with our CRCL. However, due to the different noise settings, we cannot directly use the reported results of MSCN and need to rerun it under our settings. Unfortunately, training MSCN is very time- and resource-consuming. For instance, training a model of MSCN on MSCOCO requires 4$\\times$V100 32GB cards for 20 days. Therefore, we were not able to finish all the experiments before the submission deadline. After the deadline, we did our best to complete the experiments and report them in the following table. **In the table, the first four columns are the results on Flickr30K, and the last four columns are the results on MS-COCO 1K test.**\n\n|Noise|Methods|I2T(R@1)|$\\uparrow $(%)|T2I(R@1)|$\\uparrow $(%)|I2T(R@1)|$\\uparrow $(%)|T2I(R@1)|$\\uparrow $(%)\n|-|-|-|-|-|-|-|-|-|-\n|20%|MSCN|77.4||59.6||78.1||64.3|\n|20%|CRCL|77.9|+0.5%|60.9|+1.3%|79.6|+1.5%|64.7|+0.4%\n|40%|MSCN|74.4||57.2||74.8||60.3|\n|40%|CRCL|77.8|+4.4%|60.0|+2.8%|78.2|+3.4%|63.3|+3.0%\n|60%|MSCN|70.4||53.4||74.4||59.2||\n|60%|CRCL|73.1|+2.7%|54.8|+1.4%|76.3|+1.9%|60.8|+1.6%\n|80%|MSCN|1.0||0.4||66.8||52.7|\n|80%|CRCL|62.3|+61.3%|46.0|+45.6%|72.7|+5.9%|57.5|+4.8%\n\nAs can be seen from the table, our CRCL consistently outperforms MSCN on both datasets and under all noise levels, which also demonstrates the effectiveness and robustness of our CRCL against NC. Please note that the above table only includes partial comparison results in terms of Recall@1 due to the space limitation, and more experimental results of MSCN will be provided in the next version.\n\n**Q6. Some related works are missing: [B-C]. These papers also propose methods for image-text matching and could be relevant for comparison or discussion.**\n\nWe appreciate your feedback and the related works you mentioned. We will include a discussion of these works in the next version. Briefly, HREM [B] could explicitly capture both fragment-level relations within modality and instance-level relations across different modalities, leading to better retrieval performance. Pan et.al [C] propose a Cross-modal Hard Aligning Network (CHAN) to comprehensively exploit the most relevant region-word pairs and eliminate all other alignments, achieving better retrieval accuracy and efficiency. However, unlike them, our CRCL is a specific framework to address NC. Despite this, compared with them, CRCL still achieves competitive results on well-labeled datasets, as shown in the following table:\n||I2T|I2T|I2T|T2I|T2I|T2I||\n|-|-|-|-|-|-|-|-\n||R@1|R@5|R@10|R@1|R@5|R@10|rSum\n|CHAN|79.7|96.7|98.7|63.8|90.4|95.8|525.0\n|HREM|81.2|96.5|98.9|63.7|90.7|96.0|527.1\n|CRCL|80.7|96.5|98.6|65.1|91.2|96.1|528.2\n\nThe table reports the results of I2T and T2I retrieval tasks on the MS-COCO 5 fold-1K test set.\n\n>[A] Noisy Correspondence Learning With Meta Similarity Correction, CVPR'23.\\\n[B] Learning Semantic Relationship Among Instances for Image-Text Matching, CVPR'23.\\\n[C] Fine-Grained Image-Text Matching by Cross-Modal Hard Aligning Network, CVPR'23."}, {"review_id": "SeFm5fY6gu", "reviewer": "Reviewer_tALj", "summary": "This paper tackles a latent challenge in image-text matching, which is the presence of noisy correspondences between images and texts. The paper introduces a general framework that combines a robust loss function and a correspondence correction technique to enhance the existing models’ ability to cope with noisy correspondences. The paper shows that the proposed CRCL framework outperforms the state-of-the-art methods. The paper also provides both experimental and theoretical evidence to support the effectiveness of the proposed algorithms.", "questions": "- The authors should explain why their results are different from the previous works NCR and DECL. Are there any special settings or hyperparameters that affect the performance? The authors should also report the training efficiency of their method, such as the training time and the computational resources, and compare with the existing methods, such as DECL.\n- The authors should justify their choice of using MS-COCO and Flickr30K datasets, which have one-to-many correspondences (1 to 5), while their problem formulation assumes one-to-one correspondences. How does this affect the validity and applicability of their method?\n- The authors claim that ACL is a general framework that can be applied to any existing model. However, they do not provide any empirical evidence to support this claim under other tasks. For example, can ACL handle noisy labels? The authors should provide more experimental support for the generality of ACL.\n- For Eq.11, it is not clear how the initial labels are obtained. Is it based on the first epoch of the first SR? The authors should clarify this point and explain how they initialize the labels.\n- What does active loss mean? Is it the same as normalized loss [1]? The authors should define this term clearly.\n\n\n[1] Ma X, Huang H, Wang Y, et al. Normalized loss functions for deep learning with noisy labels[C]//International conference on machine learning. PMLR, 2020: 6543-6553.\n", "limitations": "The authors have discussed the potential limitations and implications of their work.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "Significance: Noisy correspondences are a common issue in image-text data, which can arise from alignment errors or weak cross-modal information. Studying how to deal with noisy correspondences can enable more applications of image-text learning, where the alignments are challenging and require domain expertise. This work makes significant contributions both theoretically and empirically, and offers a new perspective for future research on noisy correspondences and related fields.\n\nClarity: This paper is well-written and clear, and the motivation and method are easy to follow. The technical proofs seem to be sound. However, there are some typos, undefined terms, and missing references in the paper. Please see Weaknesses and Questions below.\n\nOriginality: The techniques used in this paper seem to be novel in the image-text matching field, and there are some technical innovations.\n\nQuality: I think the quality of this paper is high, and I did not find any major flaws. The authors conduct comprehensive experiments to demonstrate the effectiveness of the proposed method.\n", "weaknesses": "- The related work section is incomplete. The authors should include and compare with some recent works on image-text matching, such as [1,2,3].\n- Some typos: CRCL in line 48.\n- Eq.1 is confusing. I suggest rewriting it as $1 - y_{ik}$, with probability $\\bar{\\eta}_{ik}, \\forall k\\neq j$. \n- Eq.11 is not clear. The authors should explain the meaning and derivation of each term.\n\n\n[1] Huang Y, Wang Y, Zeng Y, et al. MACK: multimodal aligned conceptual knowledge for unpaired image-text matching[J]. Advances in Neural Information Processing Systems, 2022, 35: 7892-7904.\\\n[2] Goel S, Bansal H, Bhatia S, et al. Cyclip: Cyclic contrastive language-image pretraining[J]. Advances in Neural Information Processing Systems, 2022, 35: 6704-6719.\\\n[3] Pan Z, Wu F, Zhang B. Fine-Grained Image-Text Matching by Cross-Modal Hard Aligning Network[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023: 19275-19284.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your valuable comments and insightful suggestions. Attached is our point-by-point response.\n\n**Q1. Missing related works [1,2,3].**\n\nWe appreciate your valuable feedback and the related works you mentioned.  We will include a discussion and a comparison of these works in the next version. However, due to the time limitation, we are not able to conduct the experiments under our settings. The comparison experiments with [3] can be found in the response (Q.6) to Reviewer QLFm.\n\n**Q2. Eq.11 is not clear.**\n\nEq.11 is a mathematical description of updating binary correspondence labels in multiple SR pieces. Specifically, the next SR uses the updated labels from the previous SR piece as the initial labels for training. Each SR does not update the label before the $e_f$-th epoch, and the rest uses Momentum Correction to update. The only exception is that the first SR uses the predicted matching probabilities at the $e_f$-th epoch as the labels for momentum update. \n\n**Q3. (a) The explanation of why the results are different from the previous works NCR and DECL. (b) The training efficiency comparison between CRCL and DECL.**\n\nWe appreciate your valuable feedback and constructive suggestions. **(a)** This difference in results is mainly caused by the difference in noise settings, i.e., the generation method of noisy correspondences and the noise rates. To fairly evaluate our method, we use the same settings and reproduce some results, which may differ from the results in the original papers of baselines. Specifically, we use the generation method used in NCR [12], i.e., **randomly shuffling the captions of training images for a specific percentage**. For a unified comparison, we set four noise rates, i.e., 20%, 40%, 60%, and 80%. Some of the results in the paper come from original papers, e.g., the results of BiCro (0%, 20%, 40%, and 60%) and NCR (0%, 20%) on Flickr30K and MS-COCO, and the results on CC152K. The others are reproduced by us under the same setting. **(b)** Due to the space limitation, the reply to this question can be found in the response (Q.2) to Reviewer DZ4B.\n\n**Q4. The authors should justify their choice of using MS-COCO and Flickr30K datasets, which have one-to-many correspondences (1 to 5), while their problem formulation assumes one-to-one correspondences. How does this affect the validity and applicability of their method?**\n\nThanks a lot for your valuable comment. Although there are one-to-many correspondences in MS-COCO and Flickr30K datasets, they do not affect the validity and applicability of our method. In practice, we can only train the models by randomly sampling small mini-batches due to the limitation of computing resources. Moreover, the input image-text pairs in each mini-batch basically have one-to-one correspondences due to the sparsity of random sampling, which actually does not violate the assumptions of our problem formulation. Besides, from the experimental results, our CRCL is applicable and effective on MS-COCO and Flickr30K datasets with state-of-the-art performance.\n\n**Q5. (a) Further explanation about ACL is a general framework. (b) Can ACL handle noisy labels? (c) The authors should provide more experimental support for the generality of ACL.**\n\nWe appreciate your thoughtful comment. **(a)** In this paper, we mainly focus on image-text matching, and apply our ACL to three models to demonstrate its generality, i.e., SAF, SGR, and VSE$\\infty$. In the rebuttal phase, we have further applied our ACL to a new task (i.e., noisy labels) to demonstrate its generality as shown in the following table. **(b)** Yes, ACL can handle noisy labels due to the similarity between noisy correspondence and noisy labels. Specifically, we can equate positive pairs to positive classes, such that binary correspondence labels and class labels are actually equivalent. **(c)** To verify this, we use the released code of [4] to conduct a preliminary experiment and compare the performance of ACL with some baselines. Specifically, to adapt to noisy labels, we directly replace the bidirectional matching probabilities used in ACL with unidirectional class probabilities output by the $softmax$ layer. Since ACL needs to have a corrected label $\\hat{y}$ to recast $q$, we simply replace $\\hat{y}$ with the positive class probabilities (possibly false positive class). The comparison baselines are CE, GCE, and NCE + MAE in [4]. The results ($mean±std$)  are reported on CIFAR-10 over 3 random runs, as shown in the following table:\n\n|Mthods|0.2|0.4|0.6|0.8\n|-|-|-|-|-\n|CE|75.90±0.28|60.28±0.27|40.90±0.35|19.65±0.46\n|GCE|87.27±0.21|83.33±0.39|72.00±0.37|29.08±0.80\n|NCE+MAE|87.12±0.21|84.19±0.43|77.61±0.05|49.62±0.72\n|Our ACL|88.64±0.11|85.31±0.14|79.06±0.55|54.26±0.89\n\nFrom the results, our ACL shows superior performance and robustness in dealing with noisy labels, which demonstrates that ACL has the generality to handle noisy labels.\n\n**Q6. For Eq.11, it is not clear how the initial labels are obtained.**\n\nThank you for your careful review. To clarify, the first SR is trained with the initial labels set as 1 and uses the predicted matching probabilities at the $e_f$-th epoch as the labels for subsequent momentum update, as described in Eq.11. \n\n**Q7. What does active loss mean? Is it the same as normalized loss [4]? The authors should define this term clearly.**\n\nThank you for your valuable review. Yes, the active loss is inspired by the normalized loss [4]. We migrated this class-level definition (i.e., “active”) to instance-level image-text matching, i.e., a loss is defined as “active” if it only optimizes at positive pairs.\n\n>[1] MACK: multimodal aligned conceptual knowledge for unpaired image-text matching, NeurIPS'22.\\\n[2] Cyclip: Cyclic contrastive language-image pretraining. NeurIPS'22.\\\n[3] Fine-Grained Image-Text Matching by Cross-Modal Hard Aligning Network, CVPR'23.\\\n[4] Normalized loss functions for deep learning with noisy labels, ICML'20."}, {"review_id": "KiWHTanXlv", "reviewer": "Reviewer_DZ4B", "summary": "This paper tackles a new challenge in image-text matching, namely, noisy correspondence, which refers to the mismatched image-text pairs that can mislead the model to learn incorrect cross-modal associations during training, resulting in a suboptimal cross-modal model that computes inaccurate similarities for retrieval. To address this challenge, the authors propose a general cross-modal robust complementary learning framework CRCL that can enhance the robustness and performance of existing image-text matching methods. The authors provide theoretical and empirical evidence that their method is effective and achieves state-of-the-art performance under the same settings compared with the existing robust baselines, such as NCR, DECL, and BiCro.", "questions": "See Weaknesses.", "limitations": "The authors discussed the limitations and Broader Impacts of CRCL in section 5.", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "1. This paper compares with a comprehensive and fair set of baselines, including the state-of-the-art robust method BiCro (CVPR’23). Moreover, the paper also includes CCR&CCS (WACV’23) and RCL (TPAMI’23) in the appendix, which is commendable.\n2. This paper provides appropriate theoretical analysis in the appendix, which is satisfactory. The authors cleverly adapt the robust theory of the noisy label problem and views NC as an instance-level category noise label problem. The experimental results and the theoretical analysis are consistent and supportive of each other.\n3. This paper shows promising performance on both synthetic NC and real NC datasets. Especially under high noise levels, CRCL demonstrates a strong robustness against NC.\n4. From Figure 1, it is evident that SCC is effective in alleviating the accumulation of noise errors and improving the accuracy of rectified correspondence labels.\n", "weaknesses": "1. There is a typo in lines 48-49: CSRL should be CRCL.\n2. This paper verifies the generality of CRCL on three standard methods, i.e., VSE$\\infty$, SGR, and SAF. How about the training cost of these methods, e.g., compared to the existing robust frameworks BiCro, NCR, and DECL?\n3. The paper seems to lack a discussion of related works in the main text, although I found it in the appendix. I understand that it may be due to the space limitation. However, I think CRCL is a significant improvement over NCR (NeurIPS’2021) for the problem of noisy correspondence. Therefore, for the readers who are not familiar with NCR or noisy correspondence, it would be more helpful to discuss related work in the main text.\n4. Is it possible to add Plausible-Match R-Precision (PMRP)[1] as an evaluation metric, since it may be more suitable for many-to-many matching evaluation? Although I believe that Recall can reflect most of the retrieval performance, PMRP may be more appealing.\n5. I think SCC is a feasible and effective technique. What is the intuition behind it? Can the authors provide more explanation?\n\n[1] Chun S, Oh S J, De Rezende R S, et al. Probabilistic embeddings for cross-modal retrieval[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2021: 8415-8424.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your valuable comments and insightful suggestions. We will address your concerns and questions one by one as follows.\n\n**Q1. here is a typo in lines 48-49: CSRL should be CRCL.**\n\nThank you for your detailed review. We will correct all typos in the next version.\n\n**Q2. This paper verifies the generality of CRCL on three standard methods, i.e., VSE, SGR, and SAF. How about the training cost of these methods, e.g., compared to the existing robust frameworks BiCro, NCR, and DECL?**\n\nThanks for your valuable comment. We performed an efficiency analysis following DECL. For a fair comparison, all experiments are performed on a single GeForce RTX3090 24GB GPU and the same training environment. We report the results on Flickr30K with 20% NC as follows:\n\n|Method|Video memory (GB)|Time/epoch|rSum\n|-|-|-|-\n|NCR|11.9 GB|45.2 min|486.8\n|DECL-SAF|10.9 GB|13.5 min|479.0\n|DECL-SGR|14.6 GB|18.3 min|482.8\n|BiCro-SAF|8.1 GB|15.6 min|478.5\n|BiCro-SGR|11.8 GB|20.0 min|478.9\n|CRCL-VSE$\\infty$|3.3 GB|1.3 min|481.1\n|CRCL-SAF|7.3 GB|12.9 min|492.5\n|CRCL-SGR|10.4 GB|15.2 min|496.6\n\nFrom the experimental results, one can see that our method is more efficient. Especially, compared with the co-teaching architecture NCR and BiCro, our CRCL is more efficient and has a better performance.\n\n**Q3. About the related works.**\n\nThank you for your valuable suggestion. Due to space constraints, we have to move the related works to the supplementary material. If possible, in the next version, we will try to include the related works in the main text.\n\n**Q4. Is it possible to add Plausible-Match R-Precision (PMRP)[1] as an evaluation metric, since it may be more suitable for many-to-many matching evaluation? Although I believe that Recall can reflect most of the retrieval performance, PMRP may be more appealing.**\n\nThank you for your valuable suggestions. We report the results with the PMRP metric on the MS-COCO dataset with 40% noise, as follows (**The first four columns are the results on the MS-COCO 5-fold 1K test, and the last four columns are the results on MS-COCO 5K test.**):\n\n||I2T|I2T|T2I|T2I|I2T|I2T|T2I|T2I \n|-|-|-|-|-|-|-|-|-\n|Methods|R@1|PMRP|R@1|PMRP|R@1|PMRP|R@1|PMRP  \n|PCME (0% noise) [1]|68.8|45.1|54.6|46.0|44.2|34.1|31.9|34.4\n|DECL-SAF|75.8|46.4|60.3|47.8|54.1|35.4|38.8|36.0\n|CRCL-SAF|76.4|46.4|62.1|47.9|54.8|35.5|40.2|36.1\n|DECL-SGR|75.9|46.0|60.2|46.7|54.3|30.8|38.5|35.0   \n|CRCL-SGR|76.8|46.9|61.9|48.3|55.7|35.5|40.3|36.4\n|DECL-SGRAF|77.1|47.0|61.5|48.2|55.8|36.0|40.1|36.4\n|CRCL-SGRAF|78.2|47.3|63.3|48.7|57.6|36.0|41.7|36.8\n\nFrom these results, our CRCL not only achieves better Recall@1 accuracies but also has higher PMRP scores for bidirectional retrieval.\n\n**Q5. I think SCC is a feasible and effective technique. What is the intuition behind it? Can the authors provide more explanation?**\n\nThank you for your insightful suggestion. In fact, our design intuition for SCC comes from addressing the flaws in previous works. Due to the memorization effect of DNNs, cross-modal learning will face the self-sample-selection error accumulation problem, which will degrade the performance. To alleviate the error accumulation, previous robust frameworks (e.g., NCR and BiCro) utilize the co-teaching manner to obtain accurate predictions.  However, the co-teaching strategy would increase the number of models, which will largely increase the training overhead. To address this problem, we present an efficient Self-refining Correspondence Correction (SCC) technique without any additional model to relieve the error accumulation problem. SCC leverages momentum correction to aggregate historical predictions of one cross-modal model, providing stable and accurate correspondence estimations while alleviating the over-memorization to NC. Moreover, SCC combines multiple independent self-refining processes in the entire training life, which could alleviate error accumulation against noisy correspondences.\n\n> [1] Probabilistic embeddings for cross-modal retrieval, CVPR'21."}]

Author rebuttal:
This is a global response. We add the illustration of our method in the attached pdf file.

Meta-review: This paper focuses on cross-modal active complementary learning with self-refining correspondence. It receives all five positive ratings. It seems that all reviewers are satisfied with the writing, interesting idea, theoretical analysis, and experimental results. For the concerns, the authors also present detailed responses. Overall, this is a good paper, and I vote for its acceptance.
Decision: Accept (poster)