Title: Cross-modal Active Complementary Learning with Self-refining Correspondence
Venue: NeurIPS 2023
Accepted: yes
Keywords: Cross-modal learning

Image-text matching

Noisy correspondence.
Abstract: Recently, image-text matching has attracted more and more attention from academia and industry, which is fundamental to understanding the latent correspondence across visual and textual modalities. However, most existing methods implicitly assume the training pairs are well-aligned while ignoring the ubiquitous annotation noise, a.k.a noisy correspondence (NC), thereby inevitably leading to a performance drop. Although some methods attempt to address such noise, they still face two challenging problems: excessive memorizing/overfitting and unreliable correction for NC, especially under high noise. To address the two problems, we propose a generalized Cross-modal Robust Complementary Learning framework (CRCL), which benefits from a novel Active Complementary Loss (ACL) and an efficient Self-refining Correspondence Correction (SCC) to improve the robustness of existing methods.   Specifically, ACL exploits active and complementary learning losses to reduce the risk of providing erroneous supervision, leading to theoretically and experimentally demonstrated robustness against NC. SCC utilizes multiple self-refining processes with momentum correction to enlarge the receptive field for correcting correspondences, thereby alleviating error accumulation and achieving accurate and stable corrections. We carry out extensive experiments on three image-text benchmarks, i.e., Flickr30K, MS-COCO, and CC152K, to verify the superior robustness of our CRCL against synthetic and real-world noisy correspondences.

Cross-modal Active Complementary Learning with Self-refining Correspondence

Abstract

Recently, image-text matching has attracted more and more attention from academia and industry, which is fundamental to understanding the latent correspondence across visual and textual modalities. However, most existing methods implicitly assume the training pairs are well-aligned while ignoring the ubiquitous annotation noise, a . k . a noisy correspondence (NC), thereby inevitably leading to a performance drop. Although some methods attempt to address such noise, they still face two challenging problems: excessive memorizing/overfitting and unreliable correction for NC, especially under high noise. To address the two problems, we propose a generalized Cross-modal Robust Complementary Learning framework (CRCL), which benefits from a novel Active Complementary Loss (ACL) and an efficient Self-refining Correspondence Correction (SCC) to improve the robustness of existing methods. Specifically, ACL exploits active and complementary learning losses to reduce the risk of providing erroneous supervision, leading to theoretically and experimentally demonstrated robustness against NC. SCC utilizes multiple self-refining processes with momentum correction to enlarge the receptive field for correcting correspondences, thereby alleviating error accumulation and achieving accurate and stable corrections. We carry out extensive experiments on three image-text benchmarks, i . e ., Flickr30K, MS-COCO, and CC152K, to verify the superior robustness of our CRCL against synthetic and real-world noisy correspondences. Code is available at https://github.com/QinYang79/CRCL .

1 Introduction

Image-text matching aims to search the most relevant samples across different modalities, which is fundamental for most cross-modal tasks [1, 2, 3, 4, 5, 6]. The core of image-text matching is how to accurately measure the similarity between distinct modalities, however, which is challenging due to the visual-textual discrepancy. To tackle the challenge, numerous deep methods are presented to learn the visual-semantic associations of image-text pairs and achieve remarkable progress, thanks to the powerful representation ability of Deep Neural Networks (DNNs) and some well-designed similarity inference architectures [7, 8, 4, 9]. They could be roughly divided into two groups, i . e ., global-level methods [7, 9] and local-level methods [8, 4], which aim at learning the image-to-sentence and region-to-word correlation to infer the cross-modal similarity, respectively. Although these methods

achieved promising matching performance, most of them implicitly require large-scale well-aligned data for training, which is expensive or even impossible to collect due to ubiquitous noise in realworld scenarios [10, 11]. Therefore, there is inevitably imperfect alignment luring in the data, i . e ., noisy correspondence (NC) [12], resulting in inferior performance.

To handle the NC problem, some prior arts are presented to alleviate the adverse impact of NC in various tasks, e . g ., partially view-aligned clustering [13, 14, 15, 16], video-text retrieval [17], visibleinfrared person re-identification [18], and image-text matching [12, 19, 20]. Specifically, inspired by learning with noisy labels [21, 22], some works [12, 23, 24] are proposed to alleviate the negative impact brought by NC. These works attempt to leverage the memorization effect of DNNs [25] to gradually distinguish the noisy image-text pairs for robust learning in a co-teaching manner. Furthermore, the predicted soft correspondence is used to recast a soft margin to replace the scalar margin of triplet ranking loss [7], which helps to avoid misleading the model by mismatched pairs. However, the soft-margin ranking loss is experimentally found to provide only limited robustness against NC, especially under high noise (as shown in Table 1), due to unstable division based on inaccurate predictions. In contrast, some works [19, 20] aim to enhance the robustness of cross-modal methods against NC by starting with a robust loss function, i . e ., avoiding over-amplification of wrong supervision information to reduce misleading risk. However, the lack of explicitly mitigating the effect of easily separable noise makes them hard to further improve the performance.

To address the aforementioned issues, we propose a generalized robust framework, dubbed CRCL, for learning with noisy correspondences. CRCL could be easily migrated into existing imagetext matching methods to enhance their robustness against NC. Our framework introduces a novel Active Complementary Loss (ACL) that applies active and complementary learning to mutually boost robustness against NC. Specifically, we present a robust complementary learning loss that employs complementary pairs, such as 'input image-text pairs are irrelevant', to conduct indirect cross-modal learning with exponential normalization. Due to the low likelihood of selecting incorrect complementary pairs, robust complementary learning could reduce the risk of providing incorrect supervision and smooth the losses, thus embracing robustness against NC. However, the robust loss will face the underfitting problem, leading to suboptimal performance. To overcome this issue, a weighted Active learning loss is proposed to enforce the model focus on more reliable positive pairs in addition to only complementary pairs. In addition, we propose a Self-refining Correspondence Correction paradigm (SCC) to obtain stable and accurate correspondence correction. SCC utilizes Momentum Correction (MC) to aggregate historical predictions for stable and accurate correspondence corrections. By combining multiple Self-Refining processes (SR) throughout the entire training process, we alleviate over-memorization for NCs. In summary, the key contributions and innovations of this work are as follows:

We propose a generalized Cross-modal Robust Contrastive Learning framework (CRCL) to address a pressing and widely-exist problem in image-text matching, i . e ., noisy correspondence. CRCL empowers existing methods with strong robustness through the perspectives of robust loss and correction techniques.

A novel Active Complementary Loss (ACL) is presented to balance active and complementary learning, mutually enhancing robustness against NC while encapsulating correct cross-modal associations in the latent common space.

We design an effective Self-refining Correspondence Correction paradigm (SCC) to achieve accurate and stable soft correspondences, which enables the prediction-based corrections to perceive larger fields and self-refine from the historically learned knowledge.

Extensive experiments verify the effectiveness and superiority of our framework on three benchmark image-text datasets: Flickr30K, MS-COCO, and CC152K. Additionally, comprehensive ablation studies and insightful analyses demonstrate the reliability and practicability of the proposed CRCL.

2 The Proposed Method

2.1 Preliminaries and Problem Statement

To be specific, we first provide some definitions of instance-level image-text matching so as to conveniently study noisy correspondence. Let D = {I , T , Y} be an image-text dataset, where I =

{ I i } N i =1 and T = { T i } N i =1 are the training image and text set with size of N . The correspondence label space is defined as Y = { y ij | i = 1 , · · · , N ; j = 1 , · · · , N } , where y ij represents the correspondence of pair ( I i , T j ) , i . e ., if I i and T j are matched ( i . e ., positive pair), y ij = 1 otherwise y ij = 0 . We assume each pair with the same indices has matched correspondence, i . e ., y ii = 1 , i = 1 , · · · , N . However, due to the ubiquitous noise during data collection, some negative pairs are mismatched as positives, a . k . a noisy correspondence (NC) [12, 19], which would introduce wrong supervisory information and misguide model training, leading to performance degradation. Mathematically, we define NC as shown in Definition 1.

Definition 1. Due to the existence of NC, the learner only has access to the noisy training data ( D η ), instead of clean data ( D ). Thus, the correspondence label for pair ( I i , T j ) is reconsidered as

̸

̸

For all pairs, conditioned on that if i = j then y ij = 1 else y ij = 0 , we have ∑ j = k ¯ η ik = η ij . Similar to the definitions of noisy labels [26], we assume that NC is uniform, i.e., η ij = η and ¯ η ik = η N -1 , ∀ k = j , where η is a constant to represent the noise rate.

̸

The key to learning with noisy correspondence is to alleviate the misguiding impact and rectify the mismatched pairs. One direct solution is to enhance the robustness of loss function L against noisy pairs, which can help prevent overfitting on mismatched pairs. The second aims at using the memorization effect of DNNs [25] to discriminate the mismatched pairs, thus removing unreliable supervision from the training data.

For image-text matching, images and texts are first projected into a shared representation space by two modality-specific networks, denoted as v and g , respectively. The cross-modal similarity of a given pair ( I i , T j ) is then computed as S ( v ( I i ) , g ( T j )) , where S ( ∗ ) could be the cosine function [7, 9] or a relevance inference module [8, 4]. For brevity, S ( v ( I i ) , g ( T j )) is denoted as S ( I i , T j ) or S ij in the following. The computed similarities could be considered as supporting evidence for retrieved results. Thus, the learning objective of image-text matching is to maximize the cross-modal similarities of positive pairs while minimizing those of negatives in the latent common space, which is commonly achieved by using contrastive learning [7, 8, 4].

The widely-used triplet ranking loss [7] has shown excellent performance in cross-modal contrastive learning tasks [7, 27]. However, recent research [19] has demonstrated that this loss function fails to perform well in image-text data with NCs, especially when using the hardest negative samples as comparison items. To address this issue, some works proposed an adaptive soft margin approach to improve the robustness of the ranking loss [12, 24, 23], which is defined as follows:

where ˆ T h and ˆ I h denote the hardest cross-modal samples in a mini-batch, ˆ α i is a soft margin adaptively computed by ˆ α i = m ˆ y ii -1 m -1 α , α is a constant margin, m is a curve parameter, and ˆ y ii is the rectified correspondence between I i and T i . However, this approach has two disadvantages: 1) The margin setting α may not be consistent with the empirical setting under NC scenarios. 2) The inaccurate prediction ˆ y ii can still easily produce the risk of misleading gradient, which can cause trivial solutions to fail, especially in the case of high noise ( e . g ., the results of NCR [12] and BiCro [23] on Flickr30K with 80% noise). To overcome these limitations, we propose a novel Active Complementary Loss (ACL) under the risk minimization theory [28, 26] to provide noise tolerance for noisy pairs while ensuring discriminative learning.

2.2 Active Complementary Loss

For the image-text dataset D , our goal is to learn a cross-modal model ( M ) that can discriminatively identify the positive (matched) and negative (unmatched) pairs well for retrieval, which is intuitively equivalent to maximizing bidirectional matching probabilities of positives. The bidirectional matching probabilities for pair ( I i , T j ) are defined as:

where τ is a temperature parameter [29, 30, 31], f is regarded as the cross-modal decision function, and ' ◦ , ⋄ ' means the two retrieval directions, i . e ., image-to-text and text-to-image, respectively. For any loss function L , the matching risk of f for image-text matching can be defined as

where E [ · ] represents the expectation operator. Considering noisy correspondences, the risk of f in noisy data D η could be formulated as follows:

where ˜ y ij is the noisy correspondence label as shown in Definition 1. Thus, the cross-modal learning objective is to learn a model M ∗ L with a global minimizer f ∗ η of R η L ( f ) . To achieve robustness, f ∗ η should be also the global minimizer of R L ( f ) on the noise-free data.

Inspired by complementary contrastive learning [20], we propose to optimize the matching probabilities of all negative pairs for learning with noisy data indirectly, thereby avoiding fast overfitting to NC. Simultaneously, to further improve the noise tolerance, we introduce an exponential normalization to smooth the complementary loss. Hence, the robust complementary loss for pair ( I i , T i ) could be formulated as:

̸

̸

where tan( · ) is the tan function and q ∈ [0 , 1] is a regulatory factor. Theoretically, for any input ( I i , T i ) under noise rate η ≤ N -1 N , we can show (see proofs in supplementary material)

where C = 2 η ( A (1 -q ) min -A (1 -q ) max ) / (1 -Nη N -1 ) ≤ 0 . C increases as q increases and when q = 1 , C takes the maximum value 0 . A min and A max are the maximum and minimum values of ∑ N j =1 tan( p ij ) under the condition ∑ N j =1 p ij = 1 , where 1 < A min < A max , and 0 ≤ p ij ≤ 1 ( p ij = p ◦ ij or p ⋄ ij ). f ∗ and f ∗ η are the global minimizers of R L r ( f ) and R η L r ( f ) , respectively.

Analysis: The larger the q is, C → 0 , the tighter the bound of Equation (7) is. When q = 0 , L r is a standard complementary contrastive loss [20]. In the extreme case of q = 1 , R L r ( f ∗ ) = R L r ( f ∗ η ) , i . e ., L r is noise tolerant since f ∗ η and f ∗ are simultaneously the minimizers of R L r ( f ) (It can also be obtained from Lemma 1 that L r is robust under q = 1 ). Put another way, as q approaches 1, the optimum f ∗ η of the noisy risk will be close to f ∗ on the clean data more likely, which implies noise tolerance.

Like most robust learning methods [32, 33], we should focus more on reliable data, while less on unreliable data. In other words, a smaller value of q is used for more convincing pairs (with larger ˆ y ii ), while a larger value of q is for less convincing pairs (with smaller ˆ y ii ). Thus, we could empirically utilize the soft corrected label ˆ y ii to recast q like the soft margin used in NCR [12], i . e ., q = 1 -ˆ y ii . However, indirect learning will face the underfitting problem, resulting in suboptimal/insufficient performance. To address this issue, we introduce a weighted active learning loss L d to make the model pay more attention to positive/matched pairs, i . e ., L d ( I i , T i , ˜ y ii ) = -˜ y ii (log p ◦ ii +log p ⋄ ii ) . This positive learning will mine discrimination from direct supervision, which complements the complementary learning loss. By combining active and complementary learning losses, our active complementary loss is defined as:

where λ is a scale factor to prevent L d from dominating the cross-modal training and quickly overfitting NC. As shown in Equation (8), when ( I i , T i ) is a noisy pair and ˆ y ii ideally approaches 0, the loss emphasizes robust complementary learning, thus mitigating overfitting to NC. Conversely, when ( I i , T i ) is a clean pair and ˆ y ii ideally approaches 1, the loss focuses on discriminative learning, thereby facilitating the accurate acquisition of visual-semantic associations. However, due to computational resource constraints, we cannot use the entire training set to perform cross-modal learning. Therefore,

we relax N to the size K of the mini-batch x by Monte Carlo sampling. Without loss of generality, the final loss for cross-modal learning is given by:

Lemma 1. In an instance-level cross-modal matching problem, under uniform NC with noise rate η ≤ N -1 N , when q = 1 , L r is noise tolerant.

Proof. The proofs of Equation (7) and Lemma 1 can be found in the supplementary material.

2.3 Self-refining Correspondence Correction

Another key to solving NC is how to obtain accurate correspondence estimations so as to reduce the adverse effects of NC. To this end, we propose an effective Self-refining Correspondence Correction paradigm (SCC). SCC leverages Momentum Correction (MC) to aggregate historical predictions, providing stable and accurate correspondence estimations while alleviating the over-memorization to NC. To eliminate the error accumulation against NC, we combine multiple independent Self-Refining (SR) in the entire training process. Specifically, the MC for the correspondence of ( I i , T i ) at the t -th epoch is defined as follows:

where β ∈ (0 , 1) represents the momentum coefficient, ˆ p t ( I i , T i ) = ( p ◦ ii + p ⋄ ii ) / 2 denotes the average matching probability at the t -th epoch. Through adequate cross-modal training, our CRCL will obtain a more stable and accurate soft correspondence label by smoothly evolving and expanding the receptive field of correction based on historical predictions with MC. Notably, as training progresses, some pairs would always be incorrectly distinguished as clean or noisy ones, resulting in the error accumulation of the estimated labels (see Figure 1c). Additionally, even though the updates performed by MC help reduce the negative influence of NC, the initial correspondence label ( y 0 ii ) still greatly affects the quality of subsequent smooth corrections. In other words, providing more accurate initial correspondences weakens the DNN's memorization against NC, thereby reducing the risk of error accumulation.

Algorithm 1: The pseudo-code of CRCL

To achieve accurate initial correspondence estimations, SCC refines updated correspondence labels historically in epochs using MC through multiple concatenated SR pieces during the entire training process. Subsequent SR pieces could gradually aggregate the learned correspondence knowledge from previous pieces, thus improving the quality of estimated correspondences progressively. Furthermore, each SR piece is trained from scratch, which aims to clear accumulated error/noise that has been memorized, thus providing more accurate correspondence predictions for subsequent training. Mathematically, SCC consists of multiple SR pieces ( [ e 1 , · · · , e m ] ) based on MC, where each piece

undergoes robust learning for e j ( j ∈ { 1 , · · · , m } ) epochs. Thus, during the j -th SR training, the estimated soft label of the i -th pair at t -th epoch is reconsidered as follows:

where ˆ p ( j,t -1) ( ∗ ) is the average matching probability of pair ( I i , T i ) at the ( t -1) -th epoch during the j -th training piece, e f denotes the number of epochs to freeze the correspondence label, preventing insufficient model training in the early stage from affecting the correction quality. In our experiments, we set all initial labels to 1, assuming that all training pairs are matched at the beginning of the first SR piece. In practice, we assign the label of the confident noisy pair as 0 to reduce the risk of producing erroneous supervision information. Therefore, the final corrected correspondence label used for L acl is defined as:

where ϵ = 0 . 1 is a fixed threshold used to filter the confident noisy pairs in experiments.

3 Experiments

In this section, comprehensive experiments are conducted on three widely used benchmarks to demonstrate the robustness and effectiveness of our CRCL under multiple scenarios, including synthetic noise, real-world noise, and well-annotated correspondence.

3.1 Datasets and Protocols

Datasets: For an extensive evaluation, we use three benchmark datasets ( i . e ., Flickr30K [34], MSCOCO [35] and CC152K [12]) in our experiments. More specifically, Flickr30K is a widely-used image-text dataset collected from the Flickr website, which comprises 31,000 images and each one has 5 corresponding textual descriptions. Following [36], 30,000 images are employed for training, 1,000 images for validation, and 1,000 images for testing in our experiments. MS-COCO is a large-scale image-text dataset, which has 123,287 images, and 5 captions are given to describe each image. We follow the split of [36, 8] to carry out our experiments, i . e ., 5000 validation images, 5000 test images, and the rest for training. CC152K is a subset of Conceptual Captions (CC) [10] collected in the real world, which is selected by [12]. Due to the absence of manual annotation, there are about 3% ∼ 20% incorrect correspondences in CC, i . e ., real-world noisy correspondences. CC152K contains 150,000 image-text pairs for training, 1,000 pairs for validation, and 1,000 pairs for testing.

Evaluation Protocols: Recall at K (R@K=1, 5, and 10) is used to measure the performance of bidirectional retrievals, which is defined as the proportion of the queries with the correct item in the top K retrieved results. Besides, flowing [19], we also take the sum of all Recalls to evaluate the overall performance, i . e ., rSum.

3.2 Implementation Details

Our CRCL is a generalized robust framework that could extend existing methods to confront noisy correspondences. To demonstrate the effectiveness and robustness of CRCL, we extend two representative methods, i . e ., VSE ∞ [9] and SGRAF (SGR and SAF) [4], to perform robust image-text matching, respectively. Specifically, the shared hyper-parameters are set as the same as the original works [4, 9], e . g ., the batch size is 128, the word embedding size is 300, and the joint embedding dimensionality is 1,024. More specific hyper-parameters and implementation details are given in our supplementary material due to the space limitation.

3.3 Comparison with State-of-the-Arts

In this section, we evaluate our CRCL by comparing it with 7 state-of-the-art methods on three benchmarks, i . e ., SCAN (ECCV'18) [8], SGRAF (SGR and SAF, AAAI'21) [4], VSE ∞ (CVPR'21) [9], NCR (NeurIPS'21) [12], DECL (ACM MM'22) [19], BiCro (CVPR'23) [23] and MSCN

(CVPR'23) [24]. For a fair comparison, all tested approaches adopt the same visual features (BUTD features) [8] and textual backbone Bi-GRU [37]. To comprehensively investigate the robustness of our method, we artificially inject synthetic false correspondence of different ratios by proportionally shuffling the captions on Flickr30K and MS-COCO like [12], i . e ., 20%, 40%, 60%, and 80% noise rates. In addition to synthetic noise, we also evaluate the robustness of tested methods against the real-world noisy correspondences on CC152K. Due to the space limitation, we only provide the results on MS-COCO 5K under well-annotated correspondences in Table 2. For fairness, like [19, 23], the ensemble results of CRCL-SGRAF are reported in the paper. More extensive comparison results are provided in the supplementary material to fully demonstrate the superiority of CRCL.

3.3.1 Results under Synthetic Noisy Correspondences

For quantitative evaluation under specific noise levels, we conduct all tested methods under four different noise rates ( i . e ., 20%, 40%, 60%, and 80%) of synthetic noisy correspondences on the Flickr30K and MS-COCO datasets. Quantitative results on Flickr30K and MSCOCO 1K test set are shown in Table 1. For MS-COCO, the results are computed by a veraging over 5 folds of 1K test images. From the results, one can see that our CRCL could remarkably outperform the robust baselines (NCR, DECL, BiCro, and MSCN) on most of the metrics, which demonstrates the superior robustness of CRCL against NC. Moreover, our CRCL not only performs well in low noise but also achieves the best performance under high noise, especially 80% noise, which provides strong evidence for the stability and robustness of our method.

3.3.2 Results under Real Noisy Correspondences

In addition to synthetic noise, we carry the comparison experiments on the real-world noisy dataset CC152K. The quantitative results are shown in Table 2. From the results, one can see that our CRCL is superior to all baselines with the best overall performance of 373.7%, which indicates that the proposed method is robust against real-world noise. Specifically, CRCL outperforms the best baseline BiCro, with absolute performance improvement of 1.0%, 0.2%, 0.4%, 0.4%, and 2.0% across different metrics, except for R@1 in text-to-image retrieval.

3.3.3 Results under Well-annotated Correspondences

Besides noisy correspondences, we also evaluate the tested methods trained on the well-aligned MS-COCO dataset for a comprehensive comparison. The results on MS-COCO 5K are shown in Table 2, wherein the results of all baselines are reported by the original papers for a fair comparison, except for the reproduced results of BiCro. From the table, our CRCL remarkably outperforms all baselines in terms of rSum. Specifically, CRCL prevails over the best baselines by 8.2% on overall performance ( i . e ., rSum) absolutely, which shows that our CRCL is not only suitable for noisy cases but also performs well in well-aligned ones.

3.4 Comparison to pre-trained model

In this section, we compare our CRCL to the pre-trained model CLIP [38] to further evaluate its effectiveness in handling NC. CLIP is a well-known large pre-trained model that is trained from scratch on a dataset of 400 million image-text pairs collected from the Internet, which includes a large number of training pairs with real NCs. More specifically, following [12], we report the zero-shot results and fine-tuning results under 20% noise and compare them with that of CRCL-SGRAF in Table 3. From the results, CLIP shows a significant performance drop during fine-tuning under NC. On the contrary, the performance of our CRCL under 20% noise is even better than the zero-shot result of CLIP (ViT-L/14), which shows the strength and potential of our CRCL in dealing with NC.

3.5 Ablation Study

In this section, we present ablation studies conducted on Flickr30K with 60% noise, as shown in Table 4. From the table, one can find that the full version of our CRCL achieves the best performance, which indicates that each component contributes to our method for performance improvement. By recasting q with the rectified label ˆ y ii using SCC, L r shows higher potential against NC. Moreover,

through the combination of active learning and robust complementary learning, there is further performance improvement, which indicates the effectiveness of our ACL. Note that these results are obtained from a single model for a comprehensive evaluation, and are not ensemble results as shown in Tables 1 and 2.

3.6 Visualization Analysis

To further investigate the generalization and robustness of CRCL, we visually study the retrieval performance of our CRCL-VSE ∞ , CRCL-SAF, and CRCL-SGR on Flickr30K and MS-COCO with varying noise rates in Figure 1(a,d). From the figure, one can see that CRCL can enhance the robustness of the original methods and perform well even under high noise. In addition, we provide visualization to intuitively showcase the effectiveness of the proposed SCC. Specifically, we visualize the distributions of both cross-modal similarities and corrected correspondences for noisy training pairs using CRCL-SGR on Flickr30K with 60% NCs. For comparison, we also visualize the results without SCC, where we replace ˆ y ii with the prediction ˆ p t ( I i , T i ) for the current iteration. The visualizations clearly indicate that SCC prevents the over-accumulation of errors in correspondence correction, thus verifying its effectiveness in mitigating the impact of NC.

0.0

0.2

0.4

0.6

0.8

10

20

30

40

50

60

70

80

90

AVG(R@1, R@5, R@10)

VSE

SAF

SGR

CRCL-VSE

CRCL-SAF

CRCL-SGR

(a) Flickr30K

0.0

0.25

0.5

0.75

1.0

Similarity

0

10

20

30

40

Density

Clean Pairs

Noisy Pairs

(b) Similarities w/o SCC

0.0

0.25

0.5

0.75

1.0

Correspondence

0

10

20

30

40

50

Density

Clean Pairs

Noisy Pairs

(c) Correspondences w/o SCC

0.0

0.2

0.4

0.6

0.8

40

50

60

70

80

90

AVG(R@1, R@5, R@10)

VSE

SAF

SGR

CRCL-VSE

CRCL-SAF

CRCL-SGR

(d) MS-COCO

0.0

0.25

0.5

0.75

1.0

Similarity

0

5

10

15

20

25

30

35

40

Density

Clean Pairs

Noisy Pairs

(e) Similarities with SCC

0.0

0.25

0.5

0.75

1.0

Correspondence

0

10

20

30

40

50

Density

Clean Pairs

Noisy Pairs

(f) Correspondences with SCC

4 Conclusion

In this paper, we propose a generalized robust framework CRCL to endow traditional methods with robustness against noisy correspondences. To alleviate the harmful effects brought by NC, we present

an Active Complementary Loss (ACL) and a Self-refining Correspondence Correction technique (SCC). These techniques enable effective learning of visual-semantic associations and progressive correction of correspondences based on historical predictions, thus boosting the robustness of imagetext matching. Extensive experiments demonstrate that our CRCL achieves state-of-the-art robustness to both synthetic and real-world NCs. Furthermore, ablation studies and visualization analyses further verify the effectiveness and reliability of the proposed components.

5 Limitations and Broader Impact Statement

Despite the promising performance of our proposed CRCL, there are some limitations that should be acknowledged. First, we only study the NC problem between two modalities, i . e ., image and text. Second, we did not take into account category-level noise. The proposed CRCL likely impacts various applications that require robust image-text matching, e . g ., multimedia retrieval, and image annotation. We encourage further study to understand and mitigate the biases and risks potentially brought by image-text matching.

Acknowledgments and Disclosure of Funding

This work was supported by the National Natural Science Foundation of China (U19A2078, U21B2040, 62372315, 62176171, and 62102274), Sichuan Science and Technology Planning Project (2023YFQ0020, 2023YFG0033, 2023ZHCG0016, 2022YFQ0014, 2022YFH0021), Chengdu Science and Technology Project (2023-XT00-00004-GX), the SCU-LuZhou Sciences and Technology Coorperation Program (2023CDLZ-16), and Fundamental Research Funds for the Central Universities under Grant YJ202140.

References

Mateusz Malinowski, Marcus Rohrbach, and Mario Fritz. Ask your neurons: A neural-based approach to answering questions about images. In Proceedings of the IEEE international conference on computer vision , pages 1-9, 2015.

Oriol Vinyals, Alexander Toshev, Samy Bengio, and Dumitru Erhan. Show and tell: A neural image caption generator. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 3156-3164, 2015.

Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, Aaron Courville, Ruslan Salakhudinov, Rich Zemel, and Yoshua Bengio. Show, attend and tell: Neural image caption generation with visual attention. In International conference on machine learning , pages 2048-2057. PMLR, 2015.

Haiwen Diao, Ying Zhang, Lin Ma, and Huchuan Lu. Similarity reasoning and filtration for image-text matching. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 35, pages 1218-1226, 2021.

Ding Jiang and Mang Ye. Cross-modal implicit relation reasoning and aligning for text-to-image person retrieval. arXiv preprint arXiv:2303.12501 , 2023.

Peng Hu, Hongyuan Zhu, Jie Lin, Dezhong Peng, Yin-Ping Zhao, and Xi Peng. Unsupervised contrastive cross-modal hashing. IEEE Transactions on Pattern Analysis and Machine Intelligence , pages 1-1, 2022.

Fartash Faghri, David J Fleet, Jamie Ryan Kiros, and Sanja Fidler. Vse++: Improving visualsemantic embeddings with hard negatives. arXiv preprint arXiv:1707.05612 , 2017.

Kuang-Huei Lee, Xi Chen, Gang Hua, Houdong Hu, and Xiaodong He. Stacked cross attention for image-text matching. In Proceedings of the European conference on computer vision (ECCV) , pages 201-216, 2018.

Jiacheng Chen, Hexiang Hu, Hao Wu, Yuning Jiang, and Changhu Wang. Learning the best pooling strategy for visual semantic embedding. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 15789-15798, 2021.

Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages 2556-2565, 2018.

Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, YunHsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and vision-language representation learning with noisy text supervision. In International Conference on Machine Learning , pages 4904-4916. PMLR, 2021.

Zhenyu Huang, Guocheng Niu, Xiao Liu, Wenbiao Ding, Xinyan Xiao, Hua Wu, and Xi Peng. Learning with noisy correspondence for cross-modal matching. Advances in Neural Information Processing Systems , 34:29406-29419, 2021.

Mouxing Yang, Yunfan Li, Peng Hu, Jinfeng Bai, Jiancheng Lv, and Xi Peng. Robust multi-view clustering with incomplete information. IEEE Transactions on Pattern Analysis and Machine Intelligence , 45(1):1055-1069, 2022.

Mouxing Yang, Yunfan Li, Zhenyu Huang, Zitao Liu, Peng Hu, and Xi Peng. Partially view-aligned representation learning with noise-robust contrastive loss. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 1134-1143, 2021.

Jie Wen, Chengliang Liu, Shijie Deng, Yicheng Liu, Lunke Fei, Ke Yan, and Yong Xu. Deep double incomplete multi-view multi-label learning with incomplete labels and missing views. IEEE Transactions on Neural Networks and Learning Systems , 2023.

Xu Yang, Cheng Deng, Kun Wei, Junchi Yan, and Wei Liu. Adversarial learning for robust deep clustering. Advances in Neural Information Processing Systems , 33:9098-9108, 2020.

Huaiwen Zhang, Yang Yang, Fan Qi, Shengsheng Qian, and Changsheng Xu. Robust video-text retrieval via noisy pair calibration. IEEE Transactions on Multimedia , 2023.

Mouxing Yang, Zhenyu Huang, Peng Hu, Taihao Li, Jiancheng Lv, and Xi Peng. Learning with twin noisy labels for visible-infrared person re-identification. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 14308-14317, 2022.

Yang Qin, Dezhong Peng, Xi Peng, Xu Wang, and Peng Hu. Deep evidential learning with noisy correspondence for cross-modal retrieval. In Proceedings of the 30th ACM International Conference on Multimedia , pages 4948-4956, 2022.

Peng Hu, Zhenyu Huang, Dezhong Peng, Xu Wang, and Xi Peng. Cross-modal retrieval with partially mismatched pairs. IEEE Transactions on Pattern Analysis and Machine Intelligence , pages 1-15, 2023.

Junnan Li, Richard Socher, and Steven CH Hoi. Dividemix: Learning with noisy labels as semi-supervised learning. arXiv preprint arXiv:2002.07394 , 2020.

Bo Han, Quanming Yao, Xingrui Yu, Gang Niu, Miao Xu, Weihua Hu, Ivor Tsang, and Masashi Sugiyama. Co-teaching: Robust training of deep neural networks with extremely noisy labels. Advances in neural information processing systems , 31, 2018.

Shuo Yang, Zhaopan Xu, Kai Wang, Yang You, Hongxun Yao, Tongliang Liu, and Min Xu. Bicro: Noisy correspondence rectification for multi-modality data via bi-directional cross-modal similarity consistency. arXiv preprint arXiv:2303.12419 , 2023.

Haochen Han, Kaiyao Miao, Qinghua Zheng, and Minnan Luo. Noisy correspondence learning with meta similarity correction. arXiv preprint arXiv:2304.06275 , 2023.

Devansh Arpit, Stanisław Jastrze ¸bski, Nicolas Ballas, David Krueger, Emmanuel Bengio, Maxinder S. Kanwal, Tegan Maharaj, Asja Fischer, Aaron Courville, Yoshua Bengio, and Simon Lacoste-Julien. A closer look at memorization in deep networks. In Doina Precup and Yee Whye Teh, editors, Proceedings of the 34th International Conference on Machine Learning , volume 70 of Proceedings of Machine Learning Research , pages 233-242. PMLR, 06-11 Aug 2017.

Aritra Ghosh, Himanshu Kumar, and P Shanti Sastry. Robust loss functions under label noise for deep neural networks. In Proceedings of the AAAI conference on artificial intelligence , volume 31, 2017.

Jianfeng Dong, Xirong Li, Chaoxi Xu, Shouling Ji, Yuan He, Gang Yang, and Xun Wang. Dual encoding for zero-example video retrieval. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 9346-9355, 2019.

Naresh Manwani and PS Sastry. Noise tolerance under risk minimization. IEEE transactions on cybernetics , 43(3):1146-1151, 2013.

Zhirong Wu, Yuanjun Xiong, Stella X Yu, and Dahua Lin. Unsupervised feature learning via non-parametric instance discrimination. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 3733-3742, 2018.

Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised learning of visual features by contrasting cluster assignments. Advances in Neural Information Processing Systems , 33:9912-9924, 2020.

Peng Hu, Xi Peng, Hongyuan Zhu, Liangli Zhen, and Jie Lin. Learning cross-modal retrieval with noisy labels. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 5403-5413, 2021.

Xingjun Ma, Hanxun Huang, Yisen Wang, Simone Romano, Sarah Erfani, and James Bailey. Normalized loss functions for deep learning with noisy labels. In International conference on machine learning , pages 6543-6553. PMLR, 2020.

Zhilu Zhang and Mert Sabuncu. Generalized cross entropy loss for training deep neural networks with noisy labels. Advances in neural information processing systems , 31, 2018.

Peter Young, Alice Lai, Micah Hodosh, and Julia Hockenmaier. From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions. Transactions of the Association for Computational Linguistics , 2:67-78, 2014.

Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European conference on computer vision , pages 740-755. Springer, 2014.

Andrej Karpathy and Li Fei-Fei. Deep visual-semantic alignments for generating image descriptions. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 3128-3137, 2015.

Mike Schuster and Kuldip K Paliwal. Bidirectional recurrent neural networks. IEEE transactions on Signal Processing , 45(11):2673-2681, 1997.

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning , pages 8748-8763. PMLR, 2021.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper tackles a new challenge in image-text matching, namely, noisy correspondence, which refers to the mismatched image-text pairs that can mislead the model to learn incorrect cross-modal associations during training, resulting in a suboptimal cross-modal model that computes inaccurate similarities for retrieval. To address this challenge, the authors propose a general cross-modal robust complementary learning framework CRCL that can enhance the robustness and performance of existing image-text matching methods. The authors provide theoretical and empirical evidence that their method is effective and achieves state-of-the-art performance under the same settings compared with the existing robust baselines, such as NCR, DECL, and BiCro.\n\nSTRENGTHS:\n1. This paper compares with a comprehensive and fair set of baselines, including the state-of-the-art robust method BiCro (CVPR’23). Moreover, the paper also includes CCR&CCS (WACV’23) and RCL (TPAMI’23) in the appendix, which is commendable.\n2. This paper provides appropriate theoretical analysis in the appendix, which is satisfactory. The authors cleverly adapt the robust theory of the noisy label problem and views NC as an instance-level category noise label problem. The experimental results and the theoretical analysis are consistent and supportive of each other.\n3. This paper shows promising performance on both synthetic NC and real NC datasets. Especially under high noise levels, CRCL demonstrates a strong robustness against NC.\n4. From Figure 1, it is evident that SCC is effective in alleviating the accumulation of noise errors and improving the accuracy of rectified correspondence labels.\n\nWEAKNESSES:\n1. There is a typo in lines 48-49: CSRL should be CRCL.\n2. This paper verifies the generality of CRCL on three standard methods, i.e., VSE$\\infty$, SGR, and SAF. How about the training cost of these methods, e.g., compared to the existing robust frameworks BiCro, NCR, and DECL?\n3. The paper seems to lack a discussion of related works in the main text, although I found it in the appendix. I understand that it may be due to the space limitation. However, I think CRCL is a significant improvement over NCR (NeurIPS’2021) for the problem of noisy correspondence. Therefore, for the readers who are not familiar with NCR or noisy correspondence, it would be more helpful to discuss related work in the main text.\n4. Is it possible to add Plausible-Match R-Precision (PMRP)[1] as an evaluation metric, since it may be more suitable for many-to-many matching evaluation? Although I believe that Recall can reflect most of the retrieval performance, PMRP may be more appealing.\n5. I think SCC is a feasible and effective technique. What is the intuition behind it? Can the authors provide more explanation?\n\n[1] Chun S, Oh S J, De Rezende R S, et al. Probabilistic embeddings for cross-modal retrieval[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2021: 8415-8424.\n\nQUESTIONS:\nSee Weaknesses.\n\nLIMITATIONS:\nThe authors discussed the limitations and Broader Impacts of CRCL in section 5.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper tackles a latent challenge in image-text matching, which is the presence of noisy correspondences between images and texts. The paper introduces a general framework that combines a robust loss function and a correspondence correction technique to enhance the existing models’ ability to cope with noisy correspondences. The paper shows that the proposed CRCL framework outperforms the state-of-the-art methods. The paper also provides both experimental and theoretical evidence to support the effectiveness of the proposed algorithms.\n\nSTRENGTHS:\nSignificance: Noisy correspondences are a common issue in image-text data, which can arise from alignment errors or weak cross-modal information. Studying how to deal with noisy correspondences can enable more applications of image-text learning, where the alignments are challenging and require domain expertise. This work makes significant contributions both theoretically and empirically, and offers a new perspective for future research on noisy correspondences and related fields.\n\nClarity: This paper is well-written and clear, and the motivation and method are easy to follow. The technical proofs seem to be sound. However, there are some typos, undefined terms, and missing references in the paper. Please see Weaknesses and Questions below.\n\nOriginality: The techniques used in this paper seem to be novel in the image-text matching field, and there are some technical innovations.\n\nQuality: I think the quality of this paper is high, and I did not find any major flaws. The authors conduct comprehensive experiments to demonstrate the effectiveness of the proposed method.\n\nWEAKNESSES:\n- The related work section is incomplete. The authors should include and compare with some recent works on image-text matching, such as [1,2,3].\n- Some typos: CRCL in line 48.\n- Eq.1 is confusing. I suggest rewriting it as $1 - y_{ik}$, with probability $\\bar{\\eta}_{ik}, \\forall k\\neq j$. \n- Eq.11 is not clear. The authors should explain the meaning and derivation of each term.\n\n\n[1] Huang Y, Wang Y, Zeng Y, et al. MACK: multimodal aligned conceptual knowledge for unpaired image-text matching[J]. Advances in Neural Information Processing Systems, 2022, 35: 7892-7904.\\\n[2] Goel S, Bansal H, Bhatia S, et al. Cyclip: Cyclic contrastive language-image pretraining[J]. Advances in Neural Information Processing Systems, 2022, 35: 6704-6719.\\\n[3] Pan Z, Wu F, Zhang B. Fine-Grained Image-Text Matching by Cross-Modal Hard Aligning Network[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023: 19275-19284.\n\nQUESTIONS:\n- The authors should explain why their results are different from the previous works NCR and DECL. Are there any special settings or hyperparameters that affect the performance? The authors should also report the training efficiency of their method, such as the training time and the computational resources, and compare with the existing methods, such as DECL.\n- The authors should justify their choice of using MS-COCO and Flickr30K datasets, which have one-to-many correspondences (1 to 5), while their problem formulation assumes one-to-one correspondences. How does this affect the validity and applicability of their method?\n- The authors claim that ACL is a general framework that can be applied to any existing model. However, they do not provide any empirical evidence to support this claim under other tasks. For example, can ACL handle noisy labels? The authors should provide more experimental support for the generality of ACL.\n- For Eq.11, it is not clear how the initial labels are obtained. Is it based on the first epoch of the first SR? The authors should clarify this point and explain how they initialize the labels.\n- What does active loss mean? Is it the same as normalized loss [1]? The authors should define this term clearly.\n\n\n[1] Ma X, Huang H, Wang Y, et al. Normalized loss functions for deep learning with noisy labels[C]//International conference on machine learning. PMLR, 2020: 6543-6553.\n\nLIMITATIONS:\nThe authors have discussed the potential limitations and implications of their work.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a novel framework (CRCL) for cross-modal correspondence learning that can handle noisy image-text pairs. The key idea of CRCL is to use a complementary active loss (ACL) that balances between discriminative learning and robust learning. ACL leverages the rectified correspondence labels to adjust the loss function and assign active loss weights, which means that the potentially noisy pairs are more focused on robust learning. To obtain accurate rectified correspondence labels, the paper proposes a self-refining correspondence correction technique (SCC) that estimates the correlation between modalities and avoids error accumulation. The proposed framework is simple yet effective in mitigating the negative impact of noisy training pairs and achieving robust image-text matching. The paper also conducts extensive experiments on three benchmark datasets to demonstrate the effectiveness and rationality of the proposed method.\n\nSTRENGTHS:\n1. The CRCL framework seems to be a general and flexible approach that can be easily integrated with existing methods to enhance the robustness of image-text matching.\n\n2. This paper is well-written and easy to follow. The authors provide clear explanations and motivations for their method.\n\n3. The authors simplify the training process by discarding the co-teaching scheme used in NCR and BiCro, and instead focus on the loss function and the correction technique to make the method concise and clear.\n\n4. The experimental results are impressive and convincing. It is also noteworthy that CRCL can improve the performance of the pre-trained model, e.g., CLIP, as shown in Table 3.\n\nWEAKNESSES:\n1. In Eq.3, there is a typo: $i_j$ should be $I_j$.\n\n2. In Eq.11, some symbols are not clearly defined, e.g., $\\hat{p}^(j,t-1)(*)$. Please explain their meanings and notations.\n\n3. The NC problem can be viewed as a special case of the PMP problem. RCL[20] uses complementary contrastive learning (CCL) to deal with PMPs. Similarly, CRCL can also be regarded as a further extension and study of CCL. However, the paper does not provide a direct comparison between CRCL and RCL. It would be interesting to see how they differ in performance and insights.\n\n4. In the supplementary material, there is a capitalization error: “For brevity” should be “for brevity”.\n\n5. Why not include MSCN[A] as a baseline? MSCN seems to be from the same period as BiCro (CVPR’23) and also addresses the NC problem.\n\n6. Some related works are missing: [B-C]. These papers also propose methods for image-text matching and could be relevant for comparison or discussion.\n\n[A] Noisy Correspondence Learning With Meta Similarity Correction, CVPR, 2023.\n[B] Learning Semantic Relationship Among Instances for Image-Text Matching, CVPR, 2023.\n[C] Fine-Grained Image-Text Matching by Cross-Modal Hard Aligning Network, CVPR, 2023.\n\nQUESTIONS:\nThis paper is well-written and easy to follow. It has all the essential elements of a good paper and I do not have any major objections to accept it. I appreciate the authors’ approach, which is more concise and theoretically sound than the existing robust image-text matching methods. However, there are some minor issues that need to be addressed, such as some typos and some recent related works that need to be further discussed. Noisy correspondence is an important research direction in the field of multimodality, as it is similar to noisy label learning. I hope the authors can not only improve the performance of their method, but also explore the potential impact of CRCL on more correspondence learning tasks.\n\nLIMITATIONS:\nThe authors should enrich further, as described in the Questions section.", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis manuscript focuses on image-text matching under the noisy correspondence setting. To achieve a noise robust multi-modal representation, the authors propose two components, including a Active Complementary Loss (ACL) and a Self-Refining Correspondence Correction (SRCC). In ACL, a complementary contrastive loss styled fomula is derived, in which a coefficient $q$ is set in seeking for a tighter bound between the divergence between risk of training with noisy correspondence and ideal setting. As for SRCC, the labeled matching score is relaxed by momentum updating to alleviate the noise. Finally, the authors conduct extensive experiments on image-text retrieval to show their performance.\n\nSTRENGTHS:\n1. The motivation of the manuscript is novel, which is from noise tolerance loss function designing and noisy label correction simultaneously.\n\n2. The proposed Active Complementary Loss has rigorous theoretical proof in evidenting the tighter bound to the divergence between noisy risk with ideal risk.\n\n3. The experiments are adequate and extensive. What' s more, as the Tab. 1 shown, the proposed method is very stable under different noisy ratios, and the improvment is non-trivial under larger noisy ratio.\n\nWEAKNESSES:\n1. Lacks necessary explaination in figures. Actually, the pure text claim to the proposed method could be harder to grab.\n\n2. The Sec. 2.2 is complex and hard to follow. Beyond theoretical proof, how ACL works in intuition should be further discussed.\n\n3. The existing organization to the proposed two components is poor. Actually, I didn' t see a clear connection between Sec. 2.2 with Sec. 2.3. Despite that they are all for noise correspondence, the author fail to discuss the relationship between ACL and SRCC. If not, I could treat this manuscript as a naive A+B technic.\n\nQUESTIONS:\nNone\n\nLIMITATIONS:\nNone", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper focuses on the problem of noise correspondence in image-text matching tasks. To address this issue, this paper proposes a generalized cross-modal robust complementary learning framework, which not only reduces the risk of erroneous supervision from the active complementary loss but also obtains stable and accurate correspondence correction through a self-refining correspondence correction. Extensive experiments show that the developed model could significantly improve the effectiveness and robustness compared with the state-of-the-art approaches. The topic of this paper is of great practical interest and the motivation is clear.\n\nSTRENGTHS:\na.\tThis paper proposes a novel generalized cross-modal robust complementary learning framework to address the noisy correspondence problem in image-text matching tasks, which enhances the effectiveness of existing methods through robust loss and correction techniques.\nb.\tThis paper utilizes the active complementary loss that employs complementary pairs to conduct indirect cross-modal learning with exponential normalization to boost robustness against noisy correspondence. In addition, self-refining correspondence correction is proposed to obtain stable and accurate correspondence correction.\nc.\tExtensive experiments are provided to prove the effectiveness and robustness of the proposed model.\n\nWEAKNESSES:\na.\tEssentially, the authors propose a new loss function to solve the noise correspondence problem in the image-text matching task. The proposed active complementary loss and self-refining correspondence correction both serve the final loss. So I think the novelty may be limited.\nb.\tI noticed in the supplementary that when the framework proposed in this paper is applied to the VSE model, there is a leap in performance improvement (such as 60% noise, Flickr30K dataset, The bidirectional retrieval improved by 50.3% and 35.4% on R@1, respectively.) while the performance improvement was limited when applied to the SGRAF model. Therefore, I think the author should explain why the performance gap is so large. More extensive experiments should also be organized to demonstrate the robustness of the proposed framework.\nc.\tTo my knowledge, in previous studies, the noise injected on the Flickr30K and MS COCO datasets was different. So how is the noise injected by the author generated?\n\nQUESTIONS:\nPlease refer to the item \"weaknesses\".\n\nLIMITATIONS:\nIn addition to the effectiveness of the experimental results, I suggest that the authors can apply the proposed framework in more methods to enhance its robustness.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}