Learning with Noisy Correspondence for Cross-modal Matching

Cross-modal matching , which aims to establish the correspondence between two different modalities , is fundamental to a variety of tasks such as cross-modal retrieval and vision-and-language understanding . Although a huge number of crossmodal matching methods have been proposed and achieved remarkable progress in recent years , almost all of these methods implicitly assume that the multimodal training data are correctly aligned . In practice , however , such an assumption is extremely expensive even impossible to satisfy . Based on this observation , we reveal and study a latent and challenging direction in cross-modal matching , named noisy correspondence , which could be regarded as a new paradigm of noisy labels . Different from the traditional noisy labels which mainly refer to the errors in category labels , our noisy correspondence refers to the mismatch paired samples . To solve this new problem , we propose a novel method for learning with noisy correspondence , named Noisy Correspondence Rectifier ( NCR ) . In brief , NCR divides the data into clean and noisy partitions based on the memorization effect of neural networks and then rectifies the correspondence via an adaptive prediction model in a co-teaching manner . To verify the effectiveness of our method , we conduct experiments by using the image-text matching as a showcase . Extensive experiments on Flickr30K , MS-COCO , and Conceptual Captions verify the effectiveness of our method . The code could be accessed from www.pengxi . me . 1 Introduction . As one of the most fundamental techniques in multimodal learning , cross-modal matching aims to bridge different modalities . In recent years , some cross-modal matching methods [ 19 , 11 , 7 , 26 ] have been proposed based on Deep Neural Networks ( DNNs ) , which achieved remarkable progress in a ∗Some parts of the work was done while Zhenyu Huang was an internship at Baidu Inc. †Corresponding author . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . variety of applications , such as clustering [ 29 , 24 ] , image/video captioning [ 1 , 44 , 22 ] , cross-modal retrieval [ 40 , 19 , 13 ] , and visual question answering [ 9 ] . In general , most existing cross-modal matching methods embed different modalities into a common space wherein the similarity of positive cross-modal pairs is maximized and that of the negative ones is minimized . Although these methods have achieved promising results , their success depends on an implicit data assumption , i.e. , the training data are correctly aligned across modalities . For example , in the vision-and-language tasks , the text needs to accurately describe the image content , and vice versa . In practice , however , it is extremely expensive and time-consuming to annotate or collect such data pairs . Especially , considering the data collected from the Internet [ 35 , 14 ] , it is inevitable to collect some mismatched pairs which are wrongly treated as the matched ones . To the best of our knowledge , such a special noisy label ( correspondence ) problem has been ignored so far , which will remarkably degrade the performance of matching methods as shown in our experiments . Based on the above observation , we reveal a new paradigm for the noisy labels , named noisy correspondence . Different from the traditional noisy labels , the noisy correspondence refers to the alignment errors in paired data rather than the errors in category annotations ( see Fig . 1 ) . To the best of our knowledge , there is no effort has been devoted to study this new problem and the closest paradigm might be the partially view-aligned problem ( PVP ) [ 12 , 41 ] . However , PVP is remarkably different from noisy correspondence , and the latter is more practical than the former . To be specific , PVP focuses on that the cross-modal alignment is totally unavailable , whereas the noisy correspondence focuses on that some correspondences are incorrect . In addition , PVP assumes that some correctly aligned data are available for training , whereas our noisy correspondence assumes that the clean and noisy data are mixed . To solve the noisy correspondence problem in cross-modal matching , we propose a novel method , named Noisy Correspondence Rectifier ( NCR ) . Our method is based on the memorization effect of DNNs observed in [ 3 , 39 ] , i.e. , DNNs tend to learn the simple patterns before fitting noisy samples . Motivated by this empirical observation , NCR divides the data into two relative accurate data partitions , i.e. , “ noisy ” and “ clean ” subsets , based on their loss difference . After that , NCR employs an adaptive prediction function for label rectifying so that the false positives and the true positives could be identified from the “ clean ” and the “ noisy ” subsets , respectively . Furthermore , we propose a novel triplet loss for robust cross-modal matching by recasting the rectified labels as the soft margin . The main contributions and novelties of this paper could be summarized as below . i ) We reveal a new problem in cross-modal analysis , which is also a new paradigm for noisy labels , termed noisy correspondence . Different from the traditional noisy labels , the noisy correspondence refers to the alignment errors in paired data instead of the errors in category annotations . To the best of our knowledge , this work could be the first study on this problem . ii ) To solve the noisy correspondence problem , we propose a novel method for learning with noisy correspondence , named Noisy Correspondence Rectifier ( NCR ) . One major novelty of NCR is that the rectified label is elegantly recasted as the soft margin of a triplet loss so that the robust cross-modal matching could be achieved . iii ) To verify the effectiveness of our method , we conduct experiments on the image-text matching task . Extensive experiments on three challenging datasets verify the effectiveness of our method in the synthesized and real noises . 2 Related works . In this section , we briefly introduce some recent developments in cross-modal matching and learning with noisy labels . 2.1 Cross-modal Matching . Most existing cross-modal matching works seek to learn a common space wherein different modalities are comparable . In general , existing works could be roughly divided into two categories : 1 ) Coarsegrained Matching . It often utilizes multiple neural networks to compute a global feature and each network is used for a specific modality [ 17 , 37 , 8 ] . For example , Kiros et al . [ 17 ] use a Convolutional Neural Network ( CNN ) and a Gated Recurrent Unit ( GRU ) to obtain the image and text features , while enforcing the similarity of positive pairs larger than that of the negative ones . To further boost the matching performance , VSE++ [ 8 ] uses some representative negatives to improve the discrimination of the model . 2 ) Fine-grained Matching . It seeks to measure the fine-grained similarity for cross-modal matching [ 19 , 21 , 7 ] . For example , SCAN [ 19 ] proposes learning the latent semantic correspondence between the image regions and words that are extracted by bottom-up attention [ 1 ] and GRU , respectively . VSRN [ 21 ] adopts a graph convolutional network for semantic reasoning . SGRAF [ 7 ] proposes constructing a similarity graph to reason the similarity and adopting an attention filtration technique to eliminate the less-meaningful alignments . Recently Chun et al . [ 6 ] introduce a new paradigm for cross-modal matching , i.e . possible many-to-many correspondence that existed in the image and captions . To achieve this , they propose to use probabilistic representations to model the possible one-to-many correspondence . Although promising results have been achieved in recent years , the existing methods heavily rely on the correctly aligned data . In practice , however , such well-matched data is expensive and timeconsuming to collect . Moreover , some recent works [ 35 , 14 ] show that a large-scale dataset collected from the wild could remarkably improve the performance of the model . However , such a data will inevitably contain some mismatched pairs . Hence , it is highly expected to develop some methods which are robust against the noisy correspondence , which has not been studied as far as we know . Different from the many-to-many correspondence [ 6 ] between image and captions , NCR reveals the noisy correspondence problem which refers to the alignment errors of image-text pairs and proposes to eliminate the negative impact from noisy pairs for downstream tasks . 2.2 Learning with Noisy Labels . To handle the possible noisy annotations in the training data , a large number of methods have been proposed and almost all of them focus on the classification task [ 36 , 27 ] . To reduce the negative impact of the noisy labels , the existing works often resort to robust architecture design , regularization , loss adjustment , or sample selection methods . Here , we mainly introduce the last two approaches which are most related to this work . To be specific , the loss adjustment achieves robustness by adjusting the contribution of clean and noisy samples w.r.t . the loss . For example , Reed et al . [ 32 ] proposed a bootstrapping loss based on the model predictions for loss correction . Zhang et al . [ 45 ] provided some theoretical explanations for the label correction along with a new label correction algorithm . Different from the loss adjustment methods , sample selection methods aim to select clean samples from a noisy dataset . For example , Arpit et al . [ 3 ] showed that DNNs tend to learn simple patterns before fitting noisy samples , namely the memorization effect . Motivated from this , Arazo et al . [ 2 ] proposed treating the samples with small loss as the clean samples . To avoid the selection bias of clean samples , Co-teaching methods [ 10 , 43 ] use the samples with small loss to iteratively train two networks . In recent , DivideMix [ 20 ] adopts the MixMatch method [ 4 ] for semi-supervised learning with the clean and noisy samples . Unlike the above noisy label studies , this paper focuses on the noisy correspondence problem which considers mismatched multimodal data pairs instead of incorrectly annotated data points . Besides the difference in the problem , this work is also different from the aforementioned studies in the methodology . To be specific , in cross-modal matching , it is impossible to directly adopt these noise label learning methods to solve the noisy correspondence problem due to the following two reasons . First , most of the noisy label learning methods propose to use the model ’ s prediction for label rectifying in the scenario of classification , while it is intractable to directly predict the correspondence of given pairs in matching models . Second , even if we can rectify the noisy correspondence somehow , Soccer p laye r i s beaten to the ball by soccer player . f g Soft Margin Input Data Embeding Similarity Measuring + S 0 1 ( a ) Training pipeline ( b ) Robust image-text matching network ( A and B ) P ( I , T ) Prediction A B A BWarmup . Divide Rectify Training Prediction Figure 2 : Overview of the proposed method . ( a ) Training pipeline of NCR . NCR consists of two individual networks ( A , B ) which work in the manner of co-teaching . In brief , NCR first warmup the networks ( A , B ) on the original training data using the loss Lw which is also used for per-sample loss computation . Then , based on the memorization effect of DNNs , NCR divides the training data into clean and noisy subsets at each epoch using either A or B , i.e. , SA = ( SAc , SAn ) and SB = ( SBc , SBn ) . After that , NCR will co-rectify the correspondence of { SA , SB } and obtain { ŜA , ŜB } using an adaptive prediction function . Finally , ŜA and ŜB will be used to train the network B and A in a swapping way . ( b ) Robust image-text matching network . For example , the network A projects the image and text by the modal-specific networks f and g , respectively . Then the similarity S ( I , T ) is computed on the extracted features f ( I ) and g ( T ) . To achieve robust image-text matching , the rectified soft labels are recast as the soft margin of our loss Lsoft . As shown , for a given anchor , Lsoft will enforce the true positive to closer to it than the negative by a large margin α̂1 , and meanwhile the false positive will has a small margin α̂2 . the rectified real-valued labels are incompatible with the existing matching methods since most of them assume the given labels are binary . To address these problems , NCR proposes an adaptive prediction function and a novel triplet loss by recasting the soft labels as soft margins .

Summary: This paper introduces a novel task, noisy correspondence of cross-modal retrieval tasks. The noisy correspondence is similar to the noisy label in object classification tasks, where the noisy correspondence scenario assumes that some image-caption pairs are incorrectly matched. To solve the problem, this paper proposes a Noisy Correspondence Rectifier (NCR), which could be considered as the novel extension of Co-teaching [9] to cross-modal retrieval tasks. Note that directly applying Co-teaching to cross-modal retrieval is non-trivial as the authors discussed in the paper (L105-115), because noisy label methods assume the existence of the prediction score function of the given input, while cross-modal retrieval methods usually do not have such per-sample score function. Hence, the proposed NCR first selects (co-divides) clean and noisy samples by using triplet loss values and the 2-mean Gaussian Mixture Model (GMM). Using the divided samples, NCR co-rectifies the noisy labels by the prediction function, where the prediction function is defined by the value of triplet distances. Using the extracted clean/noisy samples and rectified labels, NCR applies the hardest negative mining triplet loss to train the model, while the margin value is adaptive to the rectified labels. The experimental results show the effectiveness of NCR in terms of robustness to the noisy correspondence, e.g., achieving nearly dropped retrieval performances when 50% of correspondences are incorrect.