Learning with Noisy Correspondence for Cross-modal Matching

Cross-modal matching , which aims to establish the correspondence between two different modalities , is fundamental to a variety of tasks such as cross-modal retrieval and vision-and-language understanding . Although a huge number of crossmodal matching methods have been proposed and achieved remarkable progress in recent years , almost all of these methods implicitly assume that the multimodal training data are correctly aligned . In practice , however , such an assumption is extremely expensive even impossible to satisfy . Based on this observation , we reveal and study a latent and challenging direction in cross-modal matching , named noisy correspondence , which could be regarded as a new paradigm of noisy labels . Different from the traditional noisy labels which mainly refer to the errors in category labels , our noisy correspondence refers to the mismatch paired samples . To solve this new problem , we propose a novel method for learning with noisy correspondence , named Noisy Correspondence Rectifier ( NCR ) . In brief , NCR divides the data into clean and noisy partitions based on the memorization effect of neural networks and then rectifies the correspondence via an adaptive prediction model in a co-teaching manner . To verify the effectiveness of our method , we conduct experiments by using the image-text matching as a showcase . Extensive experiments on Flickr30K , MS-COCO , and Conceptual Captions verify the effectiveness of our method . The code could be accessed from www.pengxi . me . 1 Introduction . As one of the most fundamental techniques in multimodal learning , cross-modal matching aims to bridge different modalities . In recent years , some cross-modal matching methods [ 19 , 11 , 7 , 26 ] have been proposed based on Deep Neural Networks ( DNNs ) , which achieved remarkable progress in a ∗Some parts of the work was done while Zhenyu Huang was an internship at Baidu Inc. †Corresponding author . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . variety of applications , such as clustering [ 29 , 24 ] , image/video captioning [ 1 , 44 , 22 ] , cross-modal retrieval [ 40 , 19 , 13 ] , and visual question answering [ 9 ] . In general , most existing cross-modal matching methods embed different modalities into a common space wherein the similarity of positive cross-modal pairs is maximized and that of the negative ones is minimized . Although these methods have achieved promising results , their success depends on an implicit data assumption , i.e. , the training data are correctly aligned across modalities . For example , in the vision-and-language tasks , the text needs to accurately describe the image content , and vice versa . In practice , however , it is extremely expensive and time-consuming to annotate or collect such data pairs . Especially , considering the data collected from the Internet [ 35 , 14 ] , it is inevitable to collect some mismatched pairs which are wrongly treated as the matched ones . To the best of our knowledge , such a special noisy label ( correspondence ) problem has been ignored so far , which will remarkably degrade the performance of matching methods as shown in our experiments . Based on the above observation , we reveal a new paradigm for the noisy labels , named noisy correspondence . Different from the traditional noisy labels , the noisy correspondence refers to the alignment errors in paired data rather than the errors in category annotations ( see Fig . 1 ) . To the best of our knowledge , there is no effort has been devoted to study this new problem and the closest paradigm might be the partially view-aligned problem ( PVP ) [ 12 , 41 ] . However , PVP is remarkably different from noisy correspondence , and the latter is more practical than the former . To be specific , PVP focuses on that the cross-modal alignment is totally unavailable , whereas the noisy correspondence focuses on that some correspondences are incorrect . In addition , PVP assumes that some correctly aligned data are available for training , whereas our noisy correspondence assumes that the clean and noisy data are mixed . To solve the noisy correspondence problem in cross-modal matching , we propose a novel method , named Noisy Correspondence Rectifier ( NCR ) . Our method is based on the memorization effect of DNNs observed in [ 3 , 39 ] , i.e. , DNNs tend to learn the simple patterns before fitting noisy samples . Motivated by this empirical observation , NCR divides the data into two relative accurate data partitions , i.e. , “ noisy ” and “ clean ” subsets , based on their loss difference . After that , NCR employs an adaptive prediction function for label rectifying so that the false positives and the true positives could be identified from the “ clean ” and the “ noisy ” subsets , respectively . Furthermore , we propose a novel triplet loss for robust cross-modal matching by recasting the rectified labels as the soft margin . The main contributions and novelties of this paper could be summarized as below . i ) We reveal a new problem in cross-modal analysis , which is also a new paradigm for noisy labels , termed noisy correspondence . Different from the traditional noisy labels , the noisy correspondence refers to the alignment errors in paired data instead of the errors in category annotations . To the best of our knowledge , this work could be the first study on this problem . ii ) To solve the noisy correspondence problem , we propose a novel method for learning with noisy correspondence , named Noisy Correspondence Rectifier ( NCR ) . One major novelty of NCR is that the rectified label is elegantly recasted as the soft margin of a triplet loss so that the robust cross-modal matching could be achieved . iii ) To verify the effectiveness of our method , we conduct experiments on the image-text matching task . Extensive experiments on three challenging datasets verify the effectiveness of our method in the synthesized and real noises . 2 Related works . In this section , we briefly introduce some recent developments in cross-modal matching and learning with noisy labels . 2.1 Cross-modal Matching . Most existing cross-modal matching works seek to learn a common space wherein different modalities are comparable . In general , existing works could be roughly divided into two categories : 1 ) Coarsegrained Matching . It often utilizes multiple neural networks to compute a global feature and each network is used for a specific modality [ 17 , 37 , 8 ] . For example , Kiros et al . [ 17 ] use a Convolutional Neural Network ( CNN ) and a Gated Recurrent Unit ( GRU ) to obtain the image and text features , while enforcing the similarity of positive pairs larger than that of the negative ones . To further boost the matching performance , VSE++ [ 8 ] uses some representative negatives to improve the discrimination of the model . 2 ) Fine-grained Matching . It seeks to measure the fine-grained similarity for cross-modal matching [ 19 , 21 , 7 ] . For example , SCAN [ 19 ] proposes learning the latent semantic correspondence between the image regions and words that are extracted by bottom-up attention [ 1 ] and GRU , respectively . VSRN [ 21 ] adopts a graph convolutional network for semantic reasoning . SGRAF [ 7 ] proposes constructing a similarity graph to reason the similarity and adopting an attention filtration technique to eliminate the less-meaningful alignments . Recently Chun et al . [ 6 ] introduce a new paradigm for cross-modal matching , i.e . possible many-to-many correspondence that existed in the image and captions . To achieve this , they propose to use probabilistic representations to model the possible one-to-many correspondence . Although promising results have been achieved in recent years , the existing methods heavily rely on the correctly aligned data . In practice , however , such well-matched data is expensive and timeconsuming to collect . Moreover , some recent works [ 35 , 14 ] show that a large-scale dataset collected from the wild could remarkably improve the performance of the model . However , such a data will inevitably contain some mismatched pairs . Hence , it is highly expected to develop some methods which are robust against the noisy correspondence , which has not been studied as far as we know . Different from the many-to-many correspondence [ 6 ] between image and captions , NCR reveals the noisy correspondence problem which refers to the alignment errors of image-text pairs and proposes to eliminate the negative impact from noisy pairs for downstream tasks . 2.2 Learning with Noisy Labels . To handle the possible noisy annotations in the training data , a large number of methods have been proposed and almost all of them focus on the classification task [ 36 , 27 ] . To reduce the negative impact of the noisy labels , the existing works often resort to robust architecture design , regularization , loss adjustment , or sample selection methods . Here , we mainly introduce the last two approaches which are most related to this work . To be specific , the loss adjustment achieves robustness by adjusting the contribution of clean and noisy samples w.r.t . the loss . For example , Reed et al . [ 32 ] proposed a bootstrapping loss based on the model predictions for loss correction . Zhang et al . [ 45 ] provided some theoretical explanations for the label correction along with a new label correction algorithm . Different from the loss adjustment methods , sample selection methods aim to select clean samples from a noisy dataset . For example , Arpit et al . [ 3 ] showed that DNNs tend to learn simple patterns before fitting noisy samples , namely the memorization effect . Motivated from this , Arazo et al . [ 2 ] proposed treating the samples with small loss as the clean samples . To avoid the selection bias of clean samples , Co-teaching methods [ 10 , 43 ] use the samples with small loss to iteratively train two networks . In recent , DivideMix [ 20 ] adopts the MixMatch method [ 4 ] for semi-supervised learning with the clean and noisy samples . Unlike the above noisy label studies , this paper focuses on the noisy correspondence problem which considers mismatched multimodal data pairs instead of incorrectly annotated data points . Besides the difference in the problem , this work is also different from the aforementioned studies in the methodology . To be specific , in cross-modal matching , it is impossible to directly adopt these noise label learning methods to solve the noisy correspondence problem due to the following two reasons . First , most of the noisy label learning methods propose to use the model ’ s prediction for label rectifying in the scenario of classification , while it is intractable to directly predict the correspondence of given pairs in matching models . Second , even if we can rectify the noisy correspondence somehow , Soccer p laye r i s beaten to the ball by soccer player . f g Soft Margin Input Data Embeding Similarity Measuring + S 0 1 ( a ) Training pipeline ( b ) Robust image-text matching network ( A and B ) P ( I , T ) Prediction A B A BWarmup . Divide Rectify Training Prediction Figure 2 : Overview of the proposed method . ( a ) Training pipeline of NCR . NCR consists of two individual networks ( A , B ) which work in the manner of co-teaching . In brief , NCR first warmup the networks ( A , B ) on the original training data using the loss Lw which is also used for per-sample loss computation . Then , based on the memorization effect of DNNs , NCR divides the training data into clean and noisy subsets at each epoch using either A or B , i.e. , SA = ( SAc , SAn ) and SB = ( SBc , SBn ) . After that , NCR will co-rectify the correspondence of { SA , SB } and obtain { ŜA , ŜB } using an adaptive prediction function . Finally , ŜA and ŜB will be used to train the network B and A in a swapping way . ( b ) Robust image-text matching network . For example , the network A projects the image and text by the modal-specific networks f and g , respectively . Then the similarity S ( I , T ) is computed on the extracted features f ( I ) and g ( T ) . To achieve robust image-text matching , the rectified soft labels are recast as the soft margin of our loss Lsoft . As shown , for a given anchor , Lsoft will enforce the true positive to closer to it than the negative by a large margin α̂1 , and meanwhile the false positive will has a small margin α̂2 . the rectified real-valued labels are incompatible with the existing matching methods since most of them assume the given labels are binary . To address these problems , NCR proposes an adaptive prediction function and a novel triplet loss by recasting the soft labels as soft margins .

Summary: This paper focuses on cross-modal matching with the presence of noisy correspondence. The author proposes a novel method for learning with noisy correspondence to achieve robust cross-modal matching based on the SGR. In brief, it first splits the data into clean and noisy subsets in the manner of co-teaching and rectifies the labels with an adaptive prediction function. Extensive experiments on three datasets show the effectiveness of the proposed method.