PC$^2$: Pseudo-Classification Based Pseudo-Captioning for Noisy Correspondence Learning in Cross-Modal Retrieval

Authors: zhangxuan  gu
zhenzhe  ying
changhua  meng
yinghuan  shi
Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-03-19 14:30:38

Abstract: In the realm of cross-modal retrieval, seamlessly integrating diverse modalities within multimedia remains a formidable challenge, especially given the complexities introduced by noisy correspondence learning (NCL). Such noise often stems from mismatched data pairs, which is a significant obstacle distinct from traditional noisy labels. This paper introduces Pseudo-Classification based Pseudo-Captioning (PC$^2$) framework to address this challenge. PC$^2$ offers a threefold strategy: firstly, it establishes an auxiliary "pseudo-classification" task that interprets captions as categorical labels, steering the model to learn image-text semantic similarity through a non-contrastive mechanism. Secondly, unlike prevailing margin-based techniques, capitalizing on PC$^2$'s pseudo-classification capability, we generate pseudo-captions to provide more informative and tangible supervision for each mismatched pair. Thirdly, the oscillation of pseudo-classification is borrowed to assistant the correction of correspondence. In addition to technical contributions, we develop a realistic NCL dataset called Noise of Web (NoW), which could be a new powerful NCL benchmark where noise exists naturally. Empirical evaluations of PC$^2$ showcase marked improvements over existing state-of-the-art robust cross-modal retrieval techniques on both simulated and realistic datasets with various NCL settings. The contributed dataset and source code are released at https://github.com/alipay/PC2-NoiseofWeb.

Main paper: 
 introduction: 
 Cross-modal retrieval, a cornerstone of multimodal learning, is a vibrant domain tasked with bridging diverse modalities in the vast realm of multimedia [36, 52] . Yet, the tangible success of these methods hinges on a critical presumption: the training data must be in harmonious alignment across modalities. The hitch, however, lies in obtaining such perfectly matched data pairs. Manual annotation is not only a huge task but also prone to subjective errors. A potential alternative, often adopted, is mining co-occurring image-text pairs from the vast expanse of the internet [26, 38, 45] . But this convenience comes at a cost: the introduction of noise in the form of mismatched data pairs. This brings us to the crux of our discoursenoisy correspondence [25] . Unlike traditional noisy labels, which are about incorrect category labels [32, 35, 48] , noisy correspondence is the mismatch between different modalities in paired data (an example is shown in the upper part of Fig. 1 ). The collected data, riddled with a mix of clean and noisy data pairs, can diminish the effectiveness of cross-modal retrieval techniques [25, 37, 57] .Noisy correspondence learning (NCL) mentioned above still holds vast potential for development. Since it is first introduced by NCR [25] , only a handful of works have ventured further exploration and they are mainly evaluated on artificially simulated NCL datasets [25, 57] . Thus, we collect 100K website image-meta description pairs from the web to construct a large-scale NCL-specific dataset: Noise of Web (NoW), which has more complex, natural, and challenging noisy correspondences. Back to the main topic, the previous NCL solutions can be summarized as adjusting the correspondence labels, which can be recasted as the soft margin of triplet loss,Negative CaptionSelected Soccer player is beaten to the ball by soccer player.A sad man is crying silently in a narrow room.Pseudo-Caption-Based Method ( )Figure 1 : Illustrations for noisy correspondence and difference between currently popular margin-based methods and proposed pseudo-caption based PC 2 . PC 2 aims to provide direct supervision for false positive pairs with pseudo-caption and larger margin (i.e., 𝛼 large ) in triplet loss, rather than adjusting a smaller margin (i.e., 𝛼 small ) to alleviate the negative influence of false positive pairs in margin-based methods.thereby mitigating the negative impact on the training caused by the mismatched image-text pairs [25, 57] . We refer to these methods as margin-based methods, which is showcased on the left side of Fig. 1 . Although these methods demonstrates viability and efficacy, it possesses certain limitations. Adjusting the margin doesn't directly provide beneficial supervisory information for those false positive pairs but rather alleviates their incorrect supervision. Meanwhile, although these methods strive to resist noisy data, they are still affected by noticeable adverse impacts, as exemplified in Fig. 2 provided. The learning process of NCR [25] mentioned above presents an oscillatory pattern, especially pronounce during initial encounters with noisy data, thereby inducing notable volatility in the loss associated with clean data. In response to this observed phenomenon, we proffer a novel NCL framework, named Pseudo-Classification based Pseudo-Captioning (PC 2 ), engineered for robust cross-modal retrieval with noisy correspondence. This framework can be divided into three integrated solutions:(1) Inspired by non-contrastive learning [6, 7, 61] , we initially design an auxiliary task named "pseudo-classification" to reinforce the model's learning from clean data. Simplistically, the caption is interpreted as a categorical label, thereby driving the model to internalize image semantic categories through a refined cross-entropy paradigm. Utilizing cross-entropy loss brings the benefit of a explicit optimization objective for the model, without the need for negative samples. Pseudo-classification enables automatic grouping of visual concepts from image-text pairs, ultimately inducing additional semantic information. (2) Sparked by pseudo-labeling used in various semi-supervised learning tasks [9, 14, 15, 17, 56, 62, 63] and image captioning used in multimodal learning [39, 58] , in contrast to the margin-based NCL methods, we propose that offering more informative supervision for each mismatched pair by generating pseudo-captions, which is illustrated in the right side of Our method shows a more robust learning performance on clean data, maintaining a gradually converging trend with minimal influence from noisy data. In contrast, NCR exhibits a more oscillating pattern in learning clean data, especially when starting to learn from noisy data, causing noticeable fluctuations in the loss of clean data. Fig. 1 . Many studies have highlighted the importance of accurate captions [5, 43, 46] . When mismatched pairs persist in training, their adverse impact on model performance is profound. Consequently, we strive to produce captions for these mismatched pairs as correctly as possible. For specific, capitalizing on the pseudoclassification prowess of PC 2 , our strategy concurrently compute pseudo-predictions for both clean data and noisy data. These predictions, informed by their intrinsic similarities, serve as the basis for assigning pseudo-captions to the noisy data. Our primary goal remains to ensure correct correspondences across all pairs, thereby steering the model towards a better learning trajectory. (3) We make use of the pseudo-classifier, capitalizing on its oscillatory prediction behavior across different epochs, to perform a simple yet effective correspondence correction for clean data.In summary, our contributions are as follows: (1) We introduce a NCL-robust framework: PC 2 , offering a threefold strategy: an auxiliary "pseudo-classification" task using cross-entropy loss; a novel use of pseudo-captions for richer supervision of mismatched pairs, and a correspondence correction mechanism, all rooted in pseudo-classification. (2) PC 2 shows promising NCL performance on both the simulated and the realistic datasets, outperforming both popular cross-modal retrieval approaches and NCL-robust methods across a variety of NCL settings. (3) We introduce a realistic dataset Noise of Web (NoW), which can serve as a powerful benchmark for future evaluations of NCL. 
related work: 
 Bridging the semantic divide between diverse modalities is the cornerstone in multimedia research [24, 28, 53] . Such cross-modal endeavors predominantly revolve around mapping these disparate modalities into a unified, learnable space, ensuring measurable semantic correlations. However, the methodologies and challenges associate with this goal vary based on the data modalities and the alignment strategies in play, e.g., image captioning [39, 58] , video captioning [51] . For the focus of this article, image-text matching, the crux lies in deriving representations from images and aligning these with their textual counterparts [10, 16, 41] .Although previous image-text matching work has achieved considerable success [8, 12, 33] , a recurrent concern in these studies is the assumption of perfectly aligned training data pairs, which is hard to guarantee due to extensive collection and annotation expenses. Noisy correspondence learning (NCL), a relatively novel problem, delves into this issue [21, 25, 37, 57] . It addresses the mismatched pairs inaccurately considered positive. Initial research in this domain is NCR [25] , which trains image-text matching models robustly with adaptively rectified soft correspondence label. Subsequent to NCR, its successors have ushered in enhancements on NCL. For instance, BiCro [57] introduces an innovative approach to rectify noisy correspondence labels by leveraging the bidirectional cross-modal similarity consistency. This methodology capitalizes on the inherent consistency present within paired data. On the other hand, DECL [37] exploits cross-modal evidential learning to estimate the uncertainty brought by noise to isolate the noisy pairs, A salient feature uniting these methodologies is their conciliatory strategy towards handling misaligned image-text pairs; their designs primarily revolve around mitigating the detrimental impacts of mismatched pairs by isolating them or adjusting a smaller margin in triplet ranking loss. Contrasting these approaches, PC 2 furnishes direct supervisory signals for images in mismatched pairs, which enriches the learning process. 
dataset contribution: noise of web 3.1 motivation: 
 The aim of noisy correspondence learning (NCL) is building robust models based on large-scale noisy data, which can be easily obtained on website and apps. However, although there exist some noisy correspondence learning datasets such as MS-COCO [34] and Flickr30K [59] as the benchmarks, the noise in them is human generated and picked, which limits noisy correspondence models' generalization ability towards real-world applications. Randomly replacing some images' caption with others in one dataset is not a perfect choice for noise generating since there may be multiple positive and reasonable captions to one image. Another disadvantage of existing datasets is the huge human labor for writing meaningful captions for images with various different representations. For example, MS-COCO has 616,435 captions for 123,287 images, and all these captions are given by human. Although Conceptual Captions [45] (a realistic datasets) is used for NCL [25] , but its low noise ratio (3% ∼ 20%) makes it insufficient for a comprehensive evaluation.Motivated by the above mentioned, we develop a new dataset named Noise of Web (NoW) for NCL. It contains 100K cross-modal pairs consisting of website images and multilingual website metadescriptions (98,000 pairs for training, 1,000 for validation, and 1,000 for testing). NoW has two main characteristics: without human annotations and the noisy pairs are naturally captured. The source image data of NoW is obtained by taking screenshots when accessing web pages on mobile user interface (MUI) with 720×1280 resolution, and we parse the meta-description field in the HTML source code as the captions. In NCR [25] (predecessor of NCL), each image in all datasets are preprocessed using Faster-RCNN [40] detector provided by [1] to generate 36 region proposals, and each proposal is encoded as a 2048-dimensional feature. Thus, following NCR, we release our the features instead of raw images for fair comparison. However, we can not just use detection methods like Faster-RCNN [40] to extract image features since it is trained on real-world animals and objects on MS-COCO. To tackle this, we adapt APT [19] as the detection model since it is trained on MUI data. Then, we capture the 768-dimensional features of top 36 objects for one image. Using local objects' feature could contribute more to the contrastive learning and pseudo-caption generating, as explained in [12, 25, 31] . Due to the automated and non-human curated data collection process, the noise in NoW is highly authentic and intrinsic. For example, semantic inconsistencies between page content and descriptions (e.g., the third column in Fig. 3 ), nonsensical garbled description resulting from improper website maintenance (e.g., the fourth column in Fig. 3 ). The estimated noise ratio of this dataset is nearly 70%. More details of NoW can be found in Sec. A of Supplementary Material. 
method 4.1 overview: 
 In the domain of cross-modal retrieval, ensuring accurate correspondence between different modalities, such as images and text, is crucial. To comprehensively study this challenge, we take imagetext retrieval as a representative task to delve into the issue of noisy correspondence. At the heart of this task is a training set denoted as D = {(𝐼 𝑖 ,𝑇 𝑖 , 𝑐 𝑖 )} 𝑁 𝑖=1 , where each tuple represents an image-text pair. Here, 𝐼 𝑖 and 𝑇 𝑖 are the image and text components of the 𝑖-th pair, respectively. The label 𝑐 𝑖 ∈ {0, 1} signifies whether the pair is matched (𝑐 𝑖 = 1) or mismatched (𝑐 𝑖 = 0). 𝑁 represents the total count of data pairs in the training set. In the conventional setting c ...White dog playing with a red ball on the shore near the water.singleperson vehicle.A young woman pouring the contents of a bottle.Image-Text Encoder 𝑖 ,𝑇 n 𝑖 ), we first discard its caption 𝑇 n 𝑖 . We input the embedding of 𝐼 n 𝑖 into C to obtain its pseudo-prediction 𝑝 n 𝑖 , then find the most similar one (denoted as 𝑝 c 𝑗 ) to 𝑝 n 𝑖 among 𝑝 c 𝑖 from the aforementioned batch of clean data being trained synchronously. We assign the corresponding caption of 𝑝 c 𝑗 (i.e., 𝑇 c 𝑗 ) to 𝐼 n 𝑖 as the pseudo-caption 𝑇 p 𝑖 , and also utilize a margin based on pseudo-prediction similarity to train the matching model with a triplet ranking loss. of image-text retrieval, it is often assumed that all image-text pairs in the dataset are matching (i.e., ∀𝑖 ∈ {1, • • • , 𝑁 }, 𝑐 𝑖 = 1). However, multimodal datasets might be imprecisely annotated in real-world, especially if they are sourced from the internet or created using cost-effective methods (i.e., ∃𝑖 ∈ {1, • • • , 𝑁 }, 𝑐 𝑖 = 0), which we refer to as noisy correspondence learning (NCL). In general, we do not have sufficient resources to accurately identify the matching status of all image-text pairs, as 𝑐 𝑖 can be considered inaccessible.Given D, we use two modal-specific encoder 𝑓 (•) and 𝑔(•) to respectively compute the feature embedding 𝑓 (𝐼 ) and 𝑔(𝑇 ). The fundamental aim of cross-modal retrieval is to map different modalities into a unified feature space, where positive pairs should exhibit higher feature similarities, while negative pairs should manifest lower similarities. The similarity between given image-text pairs is determined using the function 𝑆 (𝐼,𝑇 ), which is a shorthand for 𝑆 (𝑓 (𝐼 ), 𝑔(𝑇 )). Generally, the primary objective is to optimize 𝑓 and 𝑔 by minimizing a triplet ranking loss function, which is influenced by the similarity measure and a distance margin 𝛼:EQUATIONwhere [𝑥] + = max(𝑥, 0), (𝐼 𝑖 ,𝑇 𝑖 ), (𝐼 𝑖 , T ℎ ) and ( Ĩ ℎ ,𝑇 𝑖 ) are the positive pair, negative pair treating image as query and negative pair treating text as query, respectively. Ĩℎ = arg max 𝐼 𝑗 ≠𝐼 𝑖 𝑆 (𝐼 𝑗 ,𝑇 𝑖 ) and Tℎ = arg max 𝑇 𝑗 ≠𝑇 𝑖 𝑆 (𝐼 𝑖 ,𝑇 𝑗 ) are the hardest negatives in the minibatch [16] . Dynamic margin plays a crucial role in NCL. Previous margin-based approaches [25, 37, 57] mitigate the impact of mismatched pairs on model training by cleverly adjusting it. The general adjustment strategy is to set a larger 𝛼 for matched pairs and a smaller 𝛼 for mismatched pairs. However, our focus is on 𝑇 𝑖 , i.e., we aim to ensure that all images in the pairs have the correct corresponding captions., as optimizing this loss will help the model converge towards a better direction.In NCL, both noisy and clean data are intermixed. Therefore, the first thing we need to consider is how to distinguish the two as correctly as possible. For simplicity, we directly utilize the memorization effect1 based co-dividing module in [25] to predict the clean probability 𝑤 𝑖 of (𝐼 𝑖 ,𝑇 𝑖 , 𝑐 𝑖 ) ∈ D. Setting a threshold 𝜏, we divide D into clean subsetD c = {(𝐼 c 𝑖 ,𝑇 c 𝑖 , 𝑐 𝑖 )} 𝑁 c 𝑖=1 and noisy subset D n = {(𝐼 n 𝑖 ,𝑇 n 𝑖 , 𝑐 𝑖 )} 𝑁 n 𝑖=1 , i.e., D = D c ∪ D n and D c ∩ D n = ∅. For specific, (𝐼 𝑖 ,𝑇 𝑖 , 𝑐 𝑖 ) with 𝑤 𝑖 > 𝜏 is selected into D c , otherwise it is selected into D n .In this paper, we refer to (𝐼 c 𝑖 ,𝑇 c 𝑖 ) ∈ D c as clean data, while (𝐼 n 𝑖 ,𝑇 n 𝑖 ) ∈ D n is referred to as noisy data (because 𝑐 𝑖 is solely used for defining the task of NCL, it will be omitted in the subsequent discussions). Furthermore, following [25, 57] , we adopt the co-training manner [20] to alleviate the error accumulation problem. Due to space limitation, the details of co-dividing and co-training are placed in Sec. B of Supplementary Material.In NCL, addressing mismatched data is paramount. However, many approaches often overlook the protection of learning from clean data. As previously discussed in Sec. EQUATIONwhere 𝐻 (𝑃, 𝑄) denotes the standard cross-entropy loss between distribution 𝑄 and 𝑃. The hard pseudo-label is widely leveraged in semi-supervised learning [13, 47] to achieve entropy minimization [18] , which encourages the model to make highly confident predictions. Moreover, to avoid C from assigning all samples to a single class, we minimize an entropy loss to spreads the pseudopredictions uniformly across the all classes [3, 4, 50] :EQUATIONOur pseudo-classification loss additionally helps the model capture similarity relationships between samples. It strengthens the model's learning from clean data in D c , enhancing its ability to resist the interference of noisy data in D n .The framework of PC 2 is shown in Fig. 4 . With pseudo-classifier C, we design a simple and effective approach to assign pseudocaptions to 𝐼 n 𝑖 . Given a mini-batch of data {(𝐼 EQUATIONwhere 𝑆 p (•, •) is a function that can be used to compute the similarity between two distributions. Then, we assemble 𝐼 n 𝑖 and 𝑇 p 𝑖 into a pseudo-pair (𝐼 n 𝑖 ,𝑇 p 𝑖 ) and substitute them into Eq. ( 1), aiming to provide more accurate supervision signals for model training. As we cannot guarantee that the found pseudo-caption accurately reflects the semantic information of 𝐼 n 𝑖 , we dynamically adjust the margin to ensure that the model benefits from a more accurate level of correspondence during training. For specific, we adaptively adjust 𝛼 in Eq. ( 1) with selected 𝑗 in Eq. ( 4) :EQUATION)where 𝑚 is a pre-defined curve parameter. The underlying principle here is that if the similarity of the pseudo-predictions 𝑆 p (𝑝 n 𝑖 , 𝑝 c 𝑗 ) is higher, then the similarity between 𝐼 n 𝑖 and 𝐼 𝑐 𝑗 (i.e., the image in the original pair where 𝑇 p 𝑖 is present) should also be higher, indicating a stronger correspondence between 𝐼 n 𝑖 and 𝑇 p 𝑖 . Then, for the noisy data {(𝐼 n 𝑖 ,𝑇 n 𝑖 )} 𝐵 𝑖=1 in the given mini-batch, we train the model by minimizing the following loss:EQUATIONIn addition to paying special attention to noisy data, learning from clean data cannot be taken lightly, because we cannot guarantee that mismatched pairs have not been erroneously included in D c . Thus, we introduce a correspondence correction module with the following core idea: the pseudo-classification results of images, learned from pseudo-labels based on captions with correct correspondences, should be stable, i.e., oscillating pseudo-predictions indicate low correspondence in the image-caption pair.We define prediction oscillation as the difference between predictions for the same sample between adjacent epochs. A larger difference indicates a higher oscillation, indicating that the model is less confident about the sample and is resisting the supervision provided by the caption-based classification labels, i.e., implying a weaker correspondence between the image and caption. This pattern is very similar to the DNN's memorization effect mentioned in Sec. ), (7) where 𝐷 𝐾𝐿 (𝑃 ∥ 𝑄) is the KL-divergence between distribution 𝑄 and 𝑃. We input {𝑜 (𝑒 )𝑖 } 𝐵 𝑖=1 into the co-dividing module described in Sec. 4.1 and obtain the prediction oscillation based clean probabilities {𝑤 o 𝑖 } 𝐵 𝑖=1 . Following Eq. ( 5), we recast the strength of correspondence to the margin of Eq. ( 1), to assist the learning of clean data, i.e.,EQUATIONwhere 1(•) is the indicator function. More explanations of Eqs. ( 7)and ( 8) can be found in Sec. B of Supplementary Material. Next, for the clean data {(𝐼 c 𝑖 ,𝑇 c 𝑖 )} 𝐵 𝑖=1 in the given mini-batch, we minimize the following loss:EQUATIONIn sum, the total loss of PC 2 can be presented asEQUATION)where 𝜆 n , 𝜆 pse and 𝜆 ent are pre-defined loss weights. Performance Metrics. The primary metric for assessing retrieval performance is the recall rate at 𝑘 (R@𝑘). We use both images and text as query entities and report on R@1, R@5, and R@10 for the evaluation. For the well-annotated datasets Flickr30K and MS-COCO, we introduce artificial noise by randomly mixing the training images and captions at five noise levels: 0%, 20%, 40%, 50%, and 60%. For all evaluations, the best checkpoint is selected based on the validation set, and its test set performance is reported.Baselines. For a comprehensive comparison, we extensively employ the following baselines: (1) generic image-text matching approaches: SCAN [31] , VSRN [33] , IMRAM [8] , SASGR, SGRAF [12] (specially, SGR * and SGR-C [25] are SGR pre-training without hard negatives and SGR training on clean data without noisy data, respectively) and ( 2) noisy-correspondence-resistant techniques: NCR [25] , DECL [37] , BiCro [57] and L2RM [22] . 
implementation details: 
 Just like the previous state-of-the-art (SOTA) NCL methods [25, 57] , PC 2 can also be universally extended to various cross-modal retrieval models. For a fair comparison, we adopt the same crossmodal retrieval backbone, SGR [12] , as used in [25, 57] , i.e., a fullconnected layer is adopted for 𝑓 (•), Bi-GRU [42] is adopted for 𝑔(•) and a graph reasoning technique proposed in [30] is adopted for 𝑆 (•, •). Similarly, the training details (e.g., batch size 𝐵 = 128, threshold 𝜏 = 0.5, margin 𝛼 = 0.2, 𝑚 = 10) are kept consistent with [25, 57] . For the additional hyper-parameters in PC 2 , we set 𝐾 = 128 for pseudo-classification and adopt cosine similarity for 𝑆 p used in pseudo-captioning. For loss weight, we set 𝜆 n = 𝜆 pse = 1 and 𝜆 ent = 10. Following [25] , we firstly warm up the model for 5, 10 and 10 epochs for Flickr30K, MS-COCO and NoW, respectively. Then, we train the model for 50 epochs in all experiments. We use the same Adam optimizer [27] with the default parameters for training as in [25, 57] . The complete list of hyper-parameters can be found in Sec. B of Supplementary Material.Main Results. We summarize the main comparisons in Tab. 1, where SoC shows promising results on both Flickr30K and MS-COCO. In the most NCL settings, PC 2 outperforms all baseline methods on the indicator Rsum by a tangible margin, e.g., PC 2 outperforms the best baseline method on Flickr30K at noise ratios of 40%, 50%, and 60% by 3.3, 10.2 and 5.9, respectively. Further, it is noteworthy that even in settings without noisy correspondences, PC 2 still achieves competitive performance, which to some extent outperform the best generic method: SGRAF (504.8 vs. 499.6 on Flickr30K). Conversely, NCR may be defeated by SGRAF (522.5 vs. 524.3 on MS-COCO). From the perspective of general image-text matching methods, they all suffer a significant setback at high noise ratios (e.g., NCR with 60%), highlighting the importance of NCL methods. From the viewpoint of noise-robust methods, margin-based approaches are generally weaker than the pseudocaption-based PC 2 . The core enhancement of our method lies in its ability to provide the correct supervisory signal for mismatched pairs as much as possible, enabling the model to make better use of noisy data. This offers a richer imagination space for NCL. Although the pseudo-captions assigned by PC 2 may not be completely consistent with the semantics of the noisy images, a certain degree of semantic overlap is sufficient to provide effective supervision. Additionally, we show the comparison with BiCro * [57], a variant of BiCro that uses mismatch thresholds to filter out mismatched pairs (the performance of PC 2 can also benefit from this technique), in Sec. C.2 of Supplementary Material. Results on NoW. Results on our challenging NCL benchmark, NoW, in Tab. 2, show our method's consistent performance advantage. Since a significant portion of captions in NoW are in Chinese, we first consider using JiebaTokenizer [23] to conduct tokenization. Moreover, we provide the additional results on BPETokenizer [44] and BertTokenizer [11, 55] that can be applied to multilingual texts in Sec. C.3 of Supplementary Material. Compared to meticulously organized datasets like MS-COCO, NoW better mirrors real-world industry scenarios. The lower success of existing methods on NoW reveals NCL research gaps, opening new exploration avenues for the community. Challenges of NoW are twofold: (1) high noise levels and sparse visual elements in images (web pages), with overly verbose or less informative captions; (2) overly abstract Table 1 : Performance comparison of image-text retrieval on Flickr30K and MS-COCO with recall at 1, 5, and 10 (R@1, R@5, R@10), along with Rsum (sum of recall values). We mark out the best results in bold and the second best results in underline. For a fair comparison, we adopt the noise ratio protocol of NCR (0%, 20% and 50%) [25] and BiCro (20%, 40% and 60%) [57] . Image → Text Text → Image Methods R@1 R@5 R@10 R@1 R@5 R@10 Rsum SGR [ Flickr30K MS-COCO Image → Text Text → Image Image → Text Text → Image Noise Methods R@1 R@5 R@10 R@1 R@5 R@10 Rsum R@1 R@5 R@10 R@1 R@5 R@The construction of PC 2 relies on components pseudo-classification (P-Cls), pseudo-captioning (P-Cap), and correspondence rectification (CR). We conduct ablation experiments on these three components to demonstrate their effectiveness. As shown in Tab. 3, the default PC 2 (the first row) achieves the most superior performance compared with other settings. The second and third rows illustrate the effectiveness of P-Cap and CR, respectively, while the fourth row indicates that even without transforming margin-based methods into pseudo-caption-based methods, relying solely on P-Cls can still greatly benefit the model from enhanced learning on clean data. Furthermore, to explore the effectiveness of P-Cls based pseudocaptioning, we strip away P-Cls and employ an alternative, more straightforward implementation to assign pseudo-captions (the fifth row): directly calculating the similarity between the embeddings of noisy and clean data images within the same batch and, like PC 2 , using the most similar pair to transfer the pseudo-caption. Default PC 2 exceeds this design, demonstrating that learning through pseudoclassification can more effectively refine the semantic information of images, thereby aiding in more rational pseudo-captioning.As P-Cls is the foundation of all PC 2 's components, we should carefully select the value of 𝐾. As shown in Tab. 4, a moderately sized 𝐾 allows C to learn the images with appropriate granularity, thereby better enhancing the learning from clean data and the selection of pseudo-captions. Following previous methods [21, 25, 57] , we set curve parameter 𝑚 = 10 for the margin adjustment functions Eqs. ( 5) and ( 8) in PC 2 . As shown in Fig. 5 , we verify the suitability of this default setting for 𝑚. More ablations on other hyper-parameters can be found in Sec. C.4 of Supplementary Material. 
discussion and future work: 
 Methodology. The design of batch-level pseudo-caption search is a trade-off between ease of implementation, efficiency, and performance. A larger batch size could make it easier for PC 2 to find the appropriate pseudo-captions, thereby improving the performance. Likewise, global search for pseudo-captions could further enhance PC 2 . In our future work, improving the caption search space of PC 2 or using image captioning solution for noisy data is our focus. In addition, it is an indisputable fact that vision-language model [38, 60] and multimodal large language model [29, 49, 54] have great potential as backbones in cross-modal retrieval tasks, but their application in NCL has not been fully explored [25] . This will also be our future direction of progress. Dataset. In the future, we will increase the overall size of our dataset, and improve the validation and test sets by manually reannotating the captions of the images, rather than just picking pairs that are manually considered to match from the original dataset. 
conclusion: 
 In this paper, we introduce Pseudo-Classification based Pseudo-Captioning (PC 2 ) framework to enhance cross-modal retrieval in the presence of noisy correspondence learning. PC 2 innovatively employs pseudo-classification and pseudo-captions for richer supervision of mismatched pairs and experiments showcases PC 2 's superiority over existing techniques. This study further contributes by open-sourcing Noise of Web (NoW) dataset, a new powerful benchmark for NCL. In the future, we will explore PC 2 's potential in other areas of multimodal learning. 
 
 Appendix: Deep neural networks (DNNs) tend to have relatively high loss for the noisy data and relatively low loss for the noisy clean in the training[2]. 


Decision: All four reviewers agree that the authors addressed their concerns in the rebuttal. Authors need to implement reviewers' feedback for the final version.

Reviews: review: paper_summary: paper_summary: This paper proposes a cross-modal retrieval method targeting noisy correspondence learning. The method initially employs an auxiliary task termed "Pseudo-classification," where captions are used as classification labels to guide the model in learning the semantic relationships between image-text pairs. Subsequently, leveraging its classification capabilities, the method constructs pseudo-captions for mismatched pairs, thereby providing richer supervision. Finally, the oscillation of pseudo-classification is utilized to assist in the correction of correspondence. Additionally, a more complex and natural dataset, Noise of Web, has been constructed to train and evaluate the model's performance.
strength_weakness: strength_weakness: - The paper introduces a more complex and natural dataset for training and evaluating cross-modal retrieval models.
- The paper employs an auxiliary "pseudo-classification" task to learn the semantic relationships between image-text pairs, a novel use of pseudo-captions for richer supervision of mismatched pairs, and a correspondence correction mechanism. This approach demonstrates significant innovation.
- The paper is very well written and easily understood. The organization is also clear.
- This paper conducts a comparative analysis with existing methods and proves the superiority of the proposed method in cross-moal retrieval task.
limitations: limitations: - In Section 4.1, setting the threshold at 0.5 may be overly simplistic and may not accurately distinguish between clean and noisy data. There are no experimental results provided to validate this setting. 
- How are captions used as classification labels? Are different captions used directly as labels? Could there be cases where captions have the same semantics or are missing? 
- In Section 5.3, what does SoC represent?
paper_summary: paper_summary: In this paper, the authors present a noise correspondence learning framework, which is a threefold strategy. It first establishes an auxiliary pseudo-classification task to interpret captions as categorical labels. Thereafter, the proposed method generates pseudo-captions to provide more information for mismatched pairs. Finally, it adopts the oscillation of pseudo-classification to assistant the correction of correspondence. The authors demonstrate the effectiveness of the proposed method by experiments. In addition, the authors also develop a dataset for noise correspondence learning.
strength_weakness: strength_weakness: 1.	The idea of this work is technically sound. This paper is well organized. 
2.	Extensive experiments are conducted. And the proposed method achieves competitive results. 
3.	A new dataset is developed.
limitations: limitations: 1.	Some sentences are confusing. For example, in 5.3 Results and analysis, the authors state that “SoC” shows promising results on……”. What is SoC?
2.	There are many writing problems, e.g.. grammars.
paper_summary: paper_summary: This paper proposes a Pseudo-Classification based Pseudo-Captioning framework to address the noisy correspondence challenge. The authors establish a pseudo-classification task to provide more information and tangible supervision for each mismatched pair. Building upon this, the oscillations of pseudo-classification are used to assist the correction of correspondence. Moreover, the authors constructed a new dataset named Noise of Web (NoW) to simulate real-world scenarios better. The proposed method exhibits significant performance improvement compared to state-of-the-art baseline methods on the NOW dataset.
strength_weakness: strength_weakness: 1: This paper leverages the pseudo-classification capability of the model to provide more usable information for samples, offering more intuitive supervision for mismatched pairs.

2: The model uses the correspondence correction mechanism, to perform effective correspondence correction for clean data.

3: This paper constructs a new dataset that better simulates the noisy correspondence in the real world, providing a basis for future research to explore new methods and solve new problems.
limitations: limitations: 1: Could the authors elaborate on how the curve parameter m in Eq. 5 is defined?

2: In Section 4.4, can the correspondence correction mechanism be applied to the re-matched dataset for further refinement?

3: The authors did not conduct the parametric sensitivity analysis for the parameters in Eq.10.

4: Does mini-batch training introduce errors in global matching?

5: In Section 3.2, the authors mentioned that the noise ratio in the dataset approaches 70%. However, they did not validate its performance under extreme conditions. 

6: The comparison methods are not consistent across this paper. 7 methods are compared at 0% noise, 8 methods are compared at 50% noise, and 9 methods are compared for other scenarios.

7: The experiments are insufficient, lacking visualization results, the compared methods are outdated, and the argumentation is not persuasive.
paper_summary: paper_summary: This paper proposed a new method to solve the noisy correspondence problem, named $\text{PC}^2$, which inherits a variety of strategies that are beneficial to solving NC, such as Pseudo-Classification, Pseudo-captions, and Oscillation. From the ablation experiments, each module used contributes. And, the authors proposed a new benchmark, which will make a certain contribution to the community.
strength_weakness: strength_weakness: 1. The method is simple and clear.
2. Experiments prove that the method is effective.
3. Writing meets standards.
limitations: limitations: 1. I think innovation may be limited. Among them, Pseudo-Classification is very similar to ID loss technology (Instance loss) in Reid. For example: Dual-path convolutional image-text embeddings with instance loss, and Cross-Modal Implicit Relation Reasoning Aligning for Text-to-Image Person Retrieval.
2. Pseudo-subtitles Logically speaking, the larger the mini-batch, the greater the probability that the retrieval may be correct, and the performance should be higher. However, the results presented in Table 5 are opposite.
3. The performance of the newly proposed benchmark is very low. Has the author considered refining the test set? Because the NC problem seems to be specific to the training set.
4. The author mentioned too many places and asked me to find supplementary materials, which greatly affected my reading.
5. The comparison is not completely SOTA and lacks extensive citations of recent related works on noisy correspondence.