Robust Contrastive Multi-view Clustering against Dual Noisy Correspondence

Authors: ruiming  guo
mouxing  yang
yijie  lin
xi  peng
peng  hu
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-15 17:22:25

Abstract: Recently, contrastive multi-view clustering (MvC) has emerged as a promising avenue for analyzing data from heterogeneous sources, typically leveraging the off-the-shelf instances as positives and randomly sampled ones as negatives. In practice, however, this paradigm would unavoidably suffer from the Dual Noisy Correspondence (DNC) problem, where noise compromises the constructions of both positive and negative pairs.
Specifically, the complexity of data collection and transmission might mistake some unassociated pairs as positive (namely, false positive correspondence), while the intrinsic one-to-many contrast nature of contrastive MvC would sample some intra-cluster samples as negative (namely, false negative correspondence).
To handle this daunting problem, we propose a novel method, dubbed Contextually-spectral based correspondence refinery (CANDY). 
CANDY dexterously exploits inter-view similarities as \textit{context} to uncover false negatives. Furthermore, it employs a spectral-based module to denoise correspondence, alleviating the negative influence of false positives. 
Extensive experiments on five widely-used multi-view benchmarks, in comparison with eight competitive multi-view clustering methods, verify the effectiveness of our method in addressing the DNC problem.
The code is available at https://github.com/XLearning-SCU/2024-NeurIPS-CANDY.

Main paper: 
 introduction: 
 In real-world applications, data are often presented in various modalities or views, including but not limited to visible images, thermal images, text, and audio [1, 2] . Multi-view Clustering (MvC), a fundamental tool in multi-view data analysis, aimed at learning a common space in which data are grouped into distinct clusters, attracts significant attention across various research communities [3] [4] [5] [6] [7] [8] . In recent years, contrastive MvC methods have emerged as a central focus in multi-view clustering researches [9, 10] . The typical implementation of these methods involves leveraging the off-the-shelf data pairs as positives and randomly sampling cross-view pairs as negatives, followed by employing contrastive learning upon them [11] [12] [13] . As a result, the cross-view discrepancy could be eliminated, revealing the underlying cluster structure.Although existing contrastive MvC methods have achieved promising performance, their success heavily relies on the assumption of faultless cross-view correspondence. In practice, however, this assumption is hard or even impossible to meet [14] [15] [16] [17] [18] , leading to inevitable contamination of the cross-view correspondence, as shown in Fig. 1a . More specifically, the complexity of data collection and transmission might mistake certain unaligned cross-view pairs for positive pairs, leading to false positive correspondence. Conversely, the inherent one-to-many contrast characteristic of contrastive MvC would inevitably result in semantically-associated cross-view positives being wrongly treated as negatives, thus producing false negative correspondence.Based on the above observations, this paper reveals a novel problem for contrastive MvC called Dual Noisy Correspondence (DNC). Formally, DNC refers to the noise present in both cross-view positive and negative pairs. This problem is akin to the partially view-aligned problem (PVP), yet differs in that PVP presupposes the availability of some correctly-associated instances for training, while DNC breaks through this impractical assumption and remains agnostic to any clean correspondence [19, 20] . Thus, DNC could be regarded as a more practical yet challenging variant of PVP, resulting in the infeasibility of PVP-oriented methods to address the DNC problem. Notably, our experimental findings, detailed in Section 4, support this claim.To tackle the DNC problem, we present a novel robust method, dubbed ContextuAlly-spectral based correspoNDence refinerY (CANDY), for learning to cluster with noisy positive and negative correspondences. As illustrated in Fig. 1b , CANDY consists of two core modules: i) the Contextbased Semantic Mining (CSM) module for recalling the false negatives, and ii) the Spectral-based Correspondence Denoising (SCD) module for alleviating the adverse impact of false positives. To be specific, CANDY first constructs a cross-view affinity graph from the multi-view data. Subsequently, CANDY calculates the connection probabilities from each node to all others, forming the context, and exploits CSM to induce a high-order contextual affinity graph. Thanks to the properties of high-order affinity, CSM could facilitate the discovery of semantically-associated positives hidden in the negatives. After that, inspired by singular value decomposition techniques used in image denoising [21, 22] , CANDY performs spectral decomposition on the contextual affinity graph and employs SCD to filter noise in the graphical spectrum, thus mitigating overfitting to false positives. Finally, CANDY employs the denoised contextual affinities to weight arbitrary contrastive losses to achieve robust MvC against DNC.In summary, the main contributions and novelties of this work could be summarized as follows.• We reveal and study a new practical problem in contrastive multi-view clustering, namely, dual noisy correspondence (DNC). Unlike prior PVP-oriented studies that rely on quite a few correctly-associated pairs, DNC refers to noise inherent in both cross-view positive and negative pairs. To the best of our knowledge, this could be one of the first investigations into noisy correspondence within MvC, particularly the more practical and challenging DNC problem.• We propose a novel robust method called CANDY for enhancing the robustness of contrastive MvC against DNC, embracing the following novelties: i) The formulation of affinity from one data point to others as context, facilitating the revelation of false negatives; and ii) Spectral denoising upon the high-order affinity graph, preventing overfitting to false positives.• Extensive experiments verify the effectiveness and superiority of CANDY. Moreover, we demonstrate the generalizability of CANDY, showing that it could serve as a plug-and-play solution to enhance the robustness of most contrastive MvC methods against DNC. 
related work: 
 In this section, we present a brief review of two topics related to this work: multi-view clustering and noisy correspondence learning.The inherent pairing characteristic of the multi-view data renders the contrastive learning paradigm a natural fit for MvC, giving rise to the established paradigm of contrastive MvC. Existing MvC methods could be roughly classified into the following three groups: i) Vanilla contrastive MvC methods [23] , which directly exploit contrastive learning to enhance the discrimination of learned representations by maximizing the mutual information between distinct views. ii) Robust contrastive MvC methods against incomplete instances [24] [25] [26] , which employ contrastive learning to learn the cross-view consistency, thereby facilitating the recovery of missing samples. iii) Robust contrastive MvC methods against false negatives [17, 27] , which redesign dedicated loss functions or similarity estimation techniques to conquer false negatives inherent in contrastive learning, thus boosting clustering performance.Our CANDY, alongside the works of [17, 27] , is devoted to addressing false negatives, while having the following significant distinctions. Different from [17] , which utilizes a false-negative-robust loss, CANDY presents a Context-based Semantic Mining (CSM) module to induce a context-aware and high-order affinity graph, benefiting the discovery of false negatives from a global perspective. Moreover, [27] proposes modeling the probability of false negatives by resorting to random walks while being susceptible to cross-view false positives. In contrast, thanks to the SCM module, CANDY embraces a more robust performance in uncovering false negatives, as verified in our experimental results.In the era of big data, millions of multimodal data are crawled from the Internet, often requiring extensive curation, which is time-intensive and cost-prohibitive [14] [15] [16] . Nevertheless, it is almost impossible to eliminate misalignment in a large quantity of multimodal data, leading to noisy correspondence. To handle this problem, noisy correspondence learning is presented to alleviate the negative influence of false positive and negative correspondences within data pairs, which has achieved promising results across various applications, such as cross-modal retrieval [28] [29] [30] , object re-identification [31] [32] [33] , multi-view learning [20, 34] , graph matching [35] , video reasoning [36] , image-text pre-training [37] .To the best of our knowledge, this work could be one of the first studies on learning to cluster with noisy correspondence. Unlike most existing approaches focusing solely on either false positives or negatives [29, 20] , our CANDY addresses the more general challenge called Dual Noisy Correspondence (DNC). Extensive experiments reveal the impracticality of applying the existing approaches to DNC in MvC, highlighting the necessity of a tailored solution to MvC against the DNC problem.Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes] Justification: The limitations are discussed in the Conclusion section. Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
method: 
 In this section, we elaborate on the proposed ContextuAlly-spectral based correspoNDence refinerY (CANDY), which aims to enhance the robustness of contrastive MvC against the Dual Noisy Correspondence (DNC) problem. As illustrated in Fig. 2 , our CANDY consists of two novel modules: Given the multi-view dataset D = {(xi , . . . , x (V ) i )} N i=1 with N instances observed from V views, the objective of contrastive MvC is to group these instances into K clusters.To this end, contrastive MvC methods construct the sets of positive and negative pairs asN i=1 x (v1) i , x (v2) i , c i | c i = 1, 1 ≤ v 1 , v 2 ≤ V, v 1 ̸ = v 2 and N i=1 i̸ =j N j=1 x (v1) i , x (v2) j , c i | c i = 0, 1 ≤ v 1 , v 2 ≤ V, v 1 ̸ = v 2by utilizing the off-the-shelf instances and perform random sampling across views respectively, where c denotes the established cross-view correspondence. Subsequently, the contrastive loss [38, 39] is applied to eliminate the cross-view discrepancy and reveal the cluster structure. However, as elaborated in the Introduction, cross-view correspondence could often be contaminated by both false positives and negatives. More specifically, a certain amount of unassociated (ĉ = 0) and associated (ĉ = 1) pairs would be wrongly treated as positives (c = 1) and negatives (c = 0) respectively, while the ground-truth correspondence ĉ is unknown. In particular, the ratio of false negatives would reach up to 1/k when the categories of the dataset D are uniformly distributed, where k is the number of classes.To counter the DNC challenge, we introduce a soft contrastive loss:EQUATIONwhere H denotes the row-wise cross-entropy function with mean reduction, C (v1,v2) ∈ R n×n is the pseudo target (Eq. 6), Z (v1) Z (v2) ⊤ represents the affinity matrix between views v 1 and v 2 , and ρ (•) signifies the softmax function. The batch-wise representation matrix Z (v) ∈ R n×d encapsulates features extracted by the view-specific encoder f (v) , with n denoting the batch size. The softmax function (ρ (•)) is applied row-wise to ensure each row sums to one as follows:EQUATIONIn general, traditional contrastive MvC methods assume that the cross-view correspondence is faultless, typically adopting an identity matrix I ∈ R n×n as the target. As verified in our experiments, such a vanilla target not only misleads the model to overfit false positives but also neglects numerous semantically associated false negatives. Therefore, the goal of CANDY becomes generating a robust pseudo target resilient against the DNC problem.The crux of uncovering false negatives lies in accurately modeling the semantic association between data points. Therefore, the widely-used strategy is based on the point-to-point similarity in the affinity graph. Specifically, a fully-connected affinity graph A is first constructed using the feature Z (v1) and Z (v2) as nodes in a mini-batch, with edge weights defined by Gaussian kernel similarity. Mathematically,EQUATIONwhere σ is a scale parameter and v 1 is the anchor view. After that, a cross-view graph Â(v1→v2) , where each edge represents the probability of semantic association between the corresponding two nodes, could be obtained by normalizing A in a row-wise manner. This strategy, however, tends to be short-sighted, potentially neglecting the out-of-neighborhood yet semantically-associated false negatives, as shown in Fig. 1b and supported by our experiments.In contrast, a simple yet effective semantic modeling strategy is presented to formulate the connection probability from one node to all others as a context, thereby redefining the context as a special representation for semantic mining. Intuitively, the contextEQUATIONin serves as a new embedding for the node i, facilitating the construction of a cross-view high-order affinity graph G (v1→v2) as follows:EQUATIONThanks to context modeling, our CSM embraces two distinct advantages: i) it encapsulates the structural information of nodes into the graph, enhancing the ability of global semantic mining, and ii) it provides a novel basis for data representation to project nodes into a new affinity space, potentially better uncovering semantically-associated false negatives.The false positive correspondence would emerge in both the off-the-shelf positive pairs, as elaborated in Section 1, and the wrongly associated negatives during the construction of G (v1→v2) . To address this, we propose a correspondence-denoising mechanism for the high-order affinity graph G (v1→v2) based on the spectral denoising theorems [21, 22] . In brief, it is widely acknowledged that the eigenvectors of signals corresponding to larger eigenvalues represent principal components, while smaller ones are apt to be noise. By selectively discarding the information tied to the minor eigenvalues, one could filter out the noise, thereby revealing the underlying structures. Inspired by these preliminary insights, we propose refining G (v1→v2) by resorting to singular value decomposition.Mathematically,EQUATIONwhere Σ denotes a diagonal matrix consisting of the singular values, U and V is the left singular matrix and the right singular matrix, respectively.After that, the denoised pseudo target could be obtained viaEQUATIONwhereΣ = diag(λ 1 , • • • , λ L ) is a diagonal matrix consisting of the retained singular values (λ 1 > • • • > λ L ≥ η), with η being a denoising hyper-parameter fixed as 0.2 in our experiments.By combining the denoised pseudo target with the vanilla target (I), we obtain the noise-resisted pseudo target (C (v1,v2) ) for the proposed soft contrastive loss (Eq. 1) viaEQUATIONwhere λ is fixed as 0.2 in our experiments.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof? Answer: [NA] Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?Answer: [Yes] Justification: This study uses public datasets. The URL of the code repository is included in the Abstract.Guidelines:• The answer NA means that paper does not include experiments requiring code. • Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted. 
experiments: 
 In this section, we verify the effectiveness of our CANDY against the DNC problem through extensive experiments by addressing the following questions:1. Performance Superiority: Does CANDY outperform the existing state-of-the-art (SOTA) MvC methods, including those designed for PVP?2. Component Indispensability: Are all components crucial for maintaining robustness against DNC?3. Working Mechanism: How does CANDY achieve robustness against DNC?4. Approach Necessity: Why is it necessary to design an approach for the DNC problem instead of using existing noisy correspondence learning methods?5. Approach Generalizability: Can CANDY be used in a plug-and-play manner to endow other contrastive MvC methods with robustness against DNC?CANDY is designed as a plug-and-play solution to endow most existing contrastive MvC methods with robustness against the DNC problem. Therefore, we choose the SOTA contrastive MvC method, namely, DIVIDE [27] , as our baseline. Specifically, we retain the architecture and pipeline of DIVIDE, modifying only the loss function. Following DIVIDE, to obtain a good initialization for the neural networks, we use the vanilla contrastive loss by setting the target C (v1,v2) in Eq. 1 as the identity matrix I for the first 20 epochs of training. To endow DIVIDE with robustness against DNC, we incorporate context-based semantic mining and spectral-based correspondence denoising modules, alongside the soft contrastive loss (Eq. 1). Since MvC requires training and clustering on the same dataset, we conduct the view realignment strategy on the learned representation by following the PVP studies [19, 20] . For achieving clustering, we concatenate the realigned representations across views to form a common representation of the MvC data and then apply the k-means algorithm by following [24] .In the experiment, CANDY is implemented with PyTorch 2.1.2, and the model is optimized with the Adam [40] optimizer with a learning rate of 0.002 across all experiments, with a batch size fixed to 1024. All evaluations are conducted on Ubuntu 20.04 OS with NVIDIA 3090 GPUs. The scale parameter σ in Eq. 3 is fixed as 0.07 across all experiments. The experiments are carried out on the following five widely-used multi-view learning datasets.• Scene-15 [41] includes 4,485 images across 15 categories. We employ PHOG and GIST as two distinct views following [17] .• Caltech-101 [42] consists 8,677 images collected from 101 classes. We use two kinds of deep features extracted by the DECAF and VGG19 neural networks as two views following [43] .• LandUse-21 [44] contains 2,100 satellite imagery samples in 21 categories. We employ the PHOG and LBP features as two views following Lin et al. [45] .• Reuters [46] is a repository of news content in multiple languages with 18,758 samples. Following [47] , we transform the texts into a 10-dimensional latent space with a conventional autoencoder and use English and French as two different views. • NUS-WIDE [48] includes 9,000 images paired with their respective captions from 10 classes. We adopt a VGG19 neural network for the extraction of visual features, and a Sentence CNN to extract the text features by following [49] .For comprehensive evaluations, we vary the noise ratio in the datasets by adopting the following protocols. For the false positive correspondence, we select one view as the anchor and randomly shuffle samples in other views according to the specified FP ratio η which is varied from 0%, 20%, 50%, to 80%. For the false negative correspondence, we adhere to the inherent FN ratio in each dataset. For clarity, we present the statistics of FP and FN ratios for different datasets in Table 1 .Notably, as the samples within the same instance would be regarded as negative if they do not belong to the same class, the practical FP ratios might be slightly lower than the specified η.In this section, we compare CANDY with eight SOTA MvC methods including the typical MvC methods (DCCAE [50] , BMVC [51] ), the PVP-oriented MvC methods (MvCLN [20] , PVC [19] , SURE [17] , CGCN [52] ), the false-negative-robust contrastive MvC (GCFAgg [53] , and DI-VIDE [27] ). Following the widely-used evaluation protocols, we adopt "ACC", "NMI" and "ARI" as the metrics.Table 2 presents the comparison results for each dataset and the average results overall, where one could have the following observations. First, our CANDY outperforms all baselines in terms of the average ACC and ARI when the FP ratio is 0%, which could be attributed to the powerful semantic mining capacity on the false negatives. Second, all baselines experience heavy performance degradation when encountering false positives. In contrast, CANDY achieves significant robustness and remarkably outperforms all baselines by a large margin. The above two observations could verify the effectiveness of CANDY against the DNC problem.Furthermore, we explore the capacity of CANDY on handling the other important problem in MvC, namely, missing views. To this end, we follow DIVIDE [27] to recover the missing views. We conduct experiments on four widely-used incomplete MvC benchmarks and compare CANDY with other baseline methods [50, 51, 54-56, 17, 57-59, 27] . As demonstrated in Table 3 , CANDY could achieve competitive results comparable to SOTA methods, even though it is primarily designed for handing DNC rather than missing modalities.In this section, we conduct ablation studies and parameter analysis to investigate the indispensable role and robustness of our modules.As shown in Table 4 , we design the following four method variants for the ablation studies: i) Warmup Only: using the identity matrix I as the target for Eq. 1 throughout the training process; ii) Re-alignment: adopting re-alignment strategy like the PVP studies; iii) SCD: performing the SCD module to denoise the vanilla affinity graph Â(v1→v2) and using the resulting graph as the target for Eq. 1. iv) CSM: the complete version of CANDY, adopting the CSM module to induce G (v1→v2) for recalling the false negatives and performing Eq. 6 to obtain the final pseudo target. From the results, one could observe that both the SCD and CSM modules play important roles in achieving robustness against DNC. Figures 5a and 5b demonstrate that our method is robust to the selection of the denoising hyperparameter η. Notably, setting η too high would destroy the structural information of the high-order graph G (v1→v2) . Therefore, we fix η at 0.2 for all experiments without elaborated tuning.To shed light on the working mechanism behind CANDY, we visualize the achieved robustness against the false positive and negative correspondences, respectively. Fig. 3 depicts the distribution of true and false positive pairs, where one could observe that the SCD module could remarkably distinguish the noisy correspondence from the clean one, thus supporting the robustness against false positive correspondence. Meanwhile, Fig. 4 presents the false negative recalling effects of different method variants, which demonstrate the significant semantic mining capacity of our CSM module and the polishing ability of the SCD module. The visualization of the cross-view similarity matrix, where each block is ordered using the ground-truth labels. For quantitative comparisons, we report the MSE between each result and the ground truth.As claimed in Related Works, we argue that the existing noisy correspondence learning cannot address the DNC problem well. In this section, we verify the necessity to devise a new approach to the DNC problem. To this end, we adopt the SOTA noisy correspondence learning method [60] in the cross-modal retrieval area for the MvC task using the same architecture (namely, DIVIDE) as CANDY. Table 5 summarizes the comparison results, highlighting the necessity of developing DNC-robust methods for contrastive MvC.CANDY aims at generating a DNC-robust pseudo target for the existing contrastive MvC methods.To verify the generalizability of CANDY, in this section, we apply CANDY on another contrastive MvC baseline, namely, AECoKM. The results of "AECoKM" and "AECoKM+Ours" are shown in Fig. 5c , where the two methods are conducted with the false positive ratio varying from 0.0 to 0.9 with an interval of 0.1. As one can observe, our CANDY could remarkably enhance the robustness and effectiveness of the baseline, demonstrating the plug-and-play role of our method.  
conclusion: 
 In this paper, we reveal and study a novel and practical problem within the field of contrastive Multiview Clustering (MvC): Dual Noisy Correspondence (DNC). In brief, DNC involves both the false positive correspondences that arise during data collection, and the false negative correspondences that are inherent in the random sampling of contrastive MvC. To address this issue, we present CANDY comprising two novel modules: Context-based Semantic Mining (CSM) and Spectralbased Correspondence Denoising (SCD). On the one hand, CSM dexterously leverages contextual information to transform distinct views into a common contextual affinity space, thereby uncovering the semantically-associated false negatives. On the other hand, SCD refines the pseudo target to mitigate the adverse impact of false positives by using the spectral denoising technique. By integrating these models, our method provides a plug-and-play solution that could enhance the robustness of the most contrastive MvC methods against DNC. Extensive experiments on a broad spectrum of scenarios have validated the effectiveness of CANDY. In the future, we plan to extend CANDY to address more practical scenarios, such as simultaneously handling both noisy correspondence and missing modalities. 
experimental setting/details: 
 Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?Answer: [Yes] Justification: This paper provide the details of data preprocessing steps, hyperparameters and optimization in the Experiments section.Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
experiment statistical significance: 
 Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [No] Justification: Almost all compared baselines do not include the statistical significance in experiments thus we do not report it.Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.• The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors). • It should be clear whether the error bar is the standard deviation or the standard error of the mean. Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [NA] Justification: The proposed algorithm have no societal impact. All datasets used in this paper are publicly available, and the algorithm only produces clustering assignments.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA]Justification: All datasets used in this paper are publicly available, and the algorithm only produces clustering assignments.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes]Justification: Original papers and datasets are properly cited.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.• If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 
new assets: 
 Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?Answer: [NA] Justification: This paper does not release new assets.Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file. 
crowdsourcing and research with human subjects: 
 Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?Answer: [NA] Justification: This work does not involve crowdsourcing nor research with human subjects.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper. • According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector. 
institutional review board (irb) approvals or equivalent for research with human subjects: 
 Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?Answer: [NA] Justification: This work does not involve crowdsourcing nor research with human subjects.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.• We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: 1. Claims Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? Answer: [Yes] Justification: The scope and contributions of this paper are clearly stated in the abstract and introduction.Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. 


Decision: This paper makes a valuable contribution to the multi-view clustering (MvC) community by introducing the Dual Noisy Correspondence (DNC) problem and proposing a novel solution based on a contextually-spectral mechanism. The work is supported by strong experimental validation and demonstrates significant technical innovation. During the rebuttal process, the authors effectively addressed the reviewers' concerns, leading two out of four reviewers to raise their scores to clear acceptance. Overall, there is no dispute regarding the paper's contribution and technical novelty, and I recommend its acceptance.

Reviews: review: paper_summary: paper_summary: The manuscript "Robust Contrastive Multi-view Clustering against Dual Noisy Correspondence" addresses the Dual Noisy Correspondence (DNC) issue in contrastive multi-view clustering (MvC), where noise affects both positive and negative data pairs. The authors propose CANDY (Contextually-spectral based correspondence refinery), a method that uses inter-view similarities to identify false negatives and a spectral-based module to denoise correspondences, thus mitigating the impact of false positives. Extensive experiments on five multi-view benchmarks demonstrate CANDY's effectiveness over existing methods.
strength_weakness: strength_weakness: -The method's applicability to other types of multi-view learning tasks, such as classification or retrieval, is not explored, limiting its broader impact within the multi-view learning domain. Including classification results would enhance the paper.

-Visualization results, such as those in Figure 3, should be compared with other baseline methods to provide detailed insights into how CANDY's clustering performance or robustness visually differs from that of other methods, particularly in handling noisy correspondences.
questions: questions: How is the pseudo target C defined in Equation 1 of the proposed method? Please provide a detailed explanation of its construction and role within the CANDY framework.
limitations: limitations: Please see weaknesses.
paper_summary: paper_summary: This paper addresses a new problem called dual noisy correspondence, which the authors claim is practical and underexplored in the multi-view learning community. Dual noisy correspondence refers to two challenges: 1) false positive correspondences induced by irrelevant multi-view data and 2) false negative correspondences caused by the random sampling characteristic of contrastive learning. To address this problem, the authors propose a novel metric for cross-view similarity estimation that recalls more false negative pairs, and a spectral-based denoising method to address false positive correspondences. Extensive experiments on multiple datasets validate the superiority of the proposed method compared to existing approaches. Overall, this paper makes an important contribution by tackling the practical and underexplored problem of dual noisy correspondence in multi-view learning. The technical solutions proposed, including the new similarity metric and spectral denoising, demonstrate strong empirical performance.
strength_weakness: strength_weakness: The authors claim to address a new problem, dual noisy correspondence (false positive and false negative). However, reviewers noted that similar issues have been explored in works [A] of other fields. Additional claims and discussions are needed to clarify the novelty of the proposed setting, which the authors regard as a significant contribution. [A] Cross-modal retrieval with noisy correspondence via consistency refining and mining, TIP.
-How about the time cost of the proposed method? It is essential to compare this with other methods, particularly the peer method (e.g., DIVIDE) with the same network architecture.
The reviewer suggests that the noise ratios for both false positive and false negative correspondences should be explicitly presented in the main table for a more comprehensive understanding of performance and comparisons.
questions: questions: (1) What is the novelty of the proposed setting?

 (2) What is the additional time cost?
limitations: limitations: see weakness
paper_summary: paper_summary: The authors delve into the study of contrastive multi-view clustering (MVC) and aim to address the false positive and false negative correspondence issues, collectively referred to as dual noisy correspondence. To tackle this problem, they propose a two-fold solution named CANDY. Firstly, CANDY exploits inter-view similarities as context to uncover false negatives. Secondly, it employs a spectral-based module to denoise correspondences, thereby mitigating the negative impact of false positives.

One of the most interesting insights in this paper is the observation that context can serve as a new axis for transforming data similarity into a high-order affinity space. 

To verify the effectiveness of CANDY, the authors conduct experiments across various datasets and settings.
strength_weakness: strength_weakness: i) I have some doubts regarding the results in Figure 4. Firstly, how is the cross-view similarity matrix arranged? The current form appears to be arranged by the ground truth, whereas my understanding is that the ground truth should be agnostic during training. Secondly, how is robustness compared in the figure? While the proposed method can recall more false negative samples, it seems to introduce more wrongly recalled samples. How is the trade-off between truly and falsely recalled samples balanced?

ii)  Regarding the references in Line 31, is the observation in Fig. 1(a) first proposed in this paper?

iii) Two open-world questions. Firstly, robustness against incomplete views is another important topic in the MVC community. Some baselines in the paper address this issue. Can the proposed method be extended to handle incomplete views? If so, this would significantly enhance the paper’s contribution. Secondly, the approach is mainly presented in scenarios involving two views. How could the method be extended to handle more views (≥ 3)?
questions: questions: Please see the weaknesses.
limitations: limitations: No.
paper_summary: paper_summary: The paper presents a novel method, Contextually-spectral based correspondence refinery (CANDY), to address the Dual Noisy Correspondence (DNC) problem in contrastive multi-view clustering (MvC). CANDY utilizes inter-view similarities as context and employs a spectral-based module for denoising correspondence, effectively mitigating the influence of false positives and uncovering false negatives. The method's effectiveness is demonstrated through extensive experiments on five multi-view benchmarks, outperforming eight competitive MvC methods.
strength_weakness: strength_weakness: 1. How will the choice of the Gaussian kernel parameter $\sigma$ influence the performance of CANDY? The sensitivity of the clustering results to this parameter needs to be thoroughly examined, as different values of $\sigma $ might affect the affinity graph’s construction.
2. Can the authors provide insights into the computational overhead introduced by the CANDY method? 
3. How are the false negative and false positive proportions handled in the experiments? Further clarification is needed on how this proportion is maintained across different datasets and experimental conditions. Does the false negative proportion keep fixed as 1/K in Table 1?
4. An interesting question arises from the paper’s claim that CANDY can serve as a plug-and-play solution. Is it possible to integrate the pseudo target generated by CANDY into other contrastive multi-view clustering methods beyond DIVIDE? Exploring this potential would demonstrate the versatility of CANDY and its applicability to a broader range of clustering techniques, providing valuable insights into how the pseudo target can enhance other methods’ robustness against noisy correspondences.
questions: questions: Please refer to weaknesses.
limitations: limitations: Please refer to weaknesses.

Comments: comment: Thank you for your recognition and positive assessment of our work. We sincerely appreciate your time and effort.
The authors have addressed my concerns; therefore, I will maintain my score.
We sincerely appreciate your positive recognition and assessment of our work!
Thank you for raising your score! We appreciate the time and effort you dedicated to reviewing this work.
Thank you for the author's response, which has effectively addressed my concerns. I will raise my score on the paper.
Thanks very much for the detailed response. The authors have provided solid and convincing experimental results to answer my problems.