Robust Variational Contrastive Learning for Partially View-unaligned Clustering

Authors: peng  hu
Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-07 15:10:41

Abstract: Although multi-view learning has achieved remarkable progress over the past decades, most existing methods implicitly assume that all views (or modalities) are well-aligned. In practice, however, collecting fully aligned views is challenging due to complexities and discordances in time and space, resulting in the Partially View-unaligned Problem (PVP), such as audio-video asynchrony caused by network congestion. While some methods are proposed to align the unaligned views by learning view-invariant representations, almost all of them overlook specific information across different views for complementarity, limiting performance improvement. To address these problems, we propose a robust framework, dubbed \textbf{V}ariat\textbf{I}onal Con\textbf{T}r\textbf{A}stive \textbf{L}earning (VITAL), designed to learn both common and specific information simultaneously. To be specific, each data sample is first modeled as a Gaussian distribution in the latent space, where the mean estimates the most probable common information, while the variance indicates view-specific information. Second, by using variational inference, VITAL conducts intra- and inter-view contrastive learning to preserve common and specific semantics in the distribution representations, thereby achieving comprehensive perception. As a result, the common representation (mean) could be used to guide category-level realignment, while the specific representation (variance) complements sample semantic information, thereby boosting overall performance. Finally, considering the abundance of False Negative Pairs (FNPs) generated by unsupervised contrastive learning, we propose a robust loss function that seamlessly incorporates FNP rectification into the contrastive learning paradigm. Empirical evaluations on eight benchmark datasets reveal that VITAL outperforms ten state-of-the-art deep clustering baselines, demonstrating its efficacy in both partially and fully aligned scenarios. The Code is available at \url{https://github.com/He-Changhao/2024-MM-VITAL}.

Main paper: 
 introduction: 
 Multi-view learning aims at exploiting the common and specific information of different views (or modalities) to achieve comprehensive perception [13, 34] . The success of existing multi-view learning methods heavily relies on an implicit assumption that all views are aligned perfectly [4, 28, 31] . In practice, however, this assumption is not always feasible, due to sensor discrepancies or communication disruptions, resulting in unaligned views, i.e., Partially View-unaligned Problem (PVP) [15, 39] . That is to say, the collected views lack proper alignment, which will adversely affect multi-view representation learning and, consequently, downstream tasks [29, 43] .Traditional solutions to PVP involve graph matching techniques, such as the Hungarian algorithm [20] , to align unaligned views in a common space. However, their non-differentiable nature precludes integration with deep neural networks (DNNs) for end-toend learning. To overcome this issue, Huang et al. [15] presented Partially View-unaligned Clustering (PVC), a differentiable alternative that learns common representations across different views while achieving differentiable alignment by using these representations. Nonetheless, PVC, akin to the Hungarian algorithm, relies on instance-level alignment, which is excessive for tasks requiring only category-level alignment [38, 39] , such as clustering. Recent advancements, such as MvCLN [39] , introduced category-level alignment, enhancing efficiency and reducing matching errors brought For variational contrastive learning, we utilize a probabilistic encoder (𝑞 𝜙 ) to approximate the true posterior distribution of observed samples, where the mean and variance of the distribution are modeled as common and specific information, respectively. Then, L 𝑖𝑛𝑡𝑒𝑟 is used for contrasting between different views in the common space to maximize the shared semantics, while L 𝑖𝑛𝑡𝑟𝑎 is used for contrasting between reconstructed and original views to retain intrinsic semantics. As for Dynamic Rectification, we employ a probabilistic model to fit the loss and derive a confidence mask for FNPs (see section 3.4). Finally, the common representations are utilized for realigning views, and the features ultimately used for downstream tasks are obtained from the combination of common and specific representations.by instance-level alignment. SURE [38] extended MvCLN by reconstructing each original view from integrated common representations, aiming to preserve high-dimensional semantic information. Yet, these methods focused predominantly on common information, often at the expense of unique view-specific information, leading to suboptimal representations for downstream applications.To address this problem, we propose a robust framework, dubbed VariatIonal ConTrAstive Learning (VITAL), to learn common representations for category-level realignment and view-specific representations for information complementation. The pipeline of VITAL is shown in Figure 1 , which consists of two modules: Variational Contrastive Learning and Dynamic Rectification. Specifically, VI-TAL first models each data sample as a Gaussian distribution in the latent space, with the mean representing likely common information and the variance capturing specific information. Second, to preserve comprehensive semantic information, VITAL employs intra-and inter-view contrastive learning coupled with variational inference to encapsulate intrinsic semantics into the distribution representations and maximize the shared semantics across different views, respectively. Thanks to the explicit learning of both viewinvariant and view-specific information, our VITAL could infuse comprehensive semantics into the latent space, thereby enhancing the discrimination of the fused representations. Finally, as the labels are unavailable, all unpaired samples across different views are considered as negative pairs, which inevitably leads to some samples from the same class being incorrectly treated as negatives, namely False Negative Pairs (FNPs). Clearly, FNPs will mislead the models, resulting in suboptimal or even erroneous solutions. To alleviate or even eliminate the adverse impact of FNPs, we present a robust contrastive loss to adaptively focus more on clean pairs and less on noisy ones, thus avoiding overemphasizing FNPs and boosting the model's robustness. Our main contributions are as follows:• We propose a comprehensive framework to tackle the Partially View-unaligned Problem (PVP), which initially aligns views using common representations, and subsequently enriches the fused representations with specific information.  
related works: 
 This section provides a brief review of some related works in the domains of multi-view learning and contrastive learning.Most existing multi-view learning methods implicitly take the view completeness assumption to learn from multiple views or modalities [13, 14, 28] . However, this ideal assumption would be easily broken in practice, resulting in incomplete views, i.e., Partially Samplemissing Problem (PSP) and Partially View-unaligned Problem (PVP).To address PSP, numerous methods were proposed to learn viewinvariant representations between views through filling in the missing parts using information from complete portions [22, [24] [25] [26] 30] . Additionally, the PVP-oriented approaches aimed to establish the cross-view correspondences of the view-unaligned samples by learning view-invariant representations [15, 38] . The pioneering PVP-oriented method PVC utilized DNNs to reconcile unaligned views by learning latent common representations and employing a differentiable bi-graph matching algorithm for instance-level alignment [15] . Alternatively, MvCLN presented a category-level alignment paradigm, enhancing the efficiency and accuracy of crossview alignment for partially view-unaligned clustering [39] . Subsequently, SURE further extended MvCLN to preserve comprehensive sample information by jointly reconstructing the original samples [38] . Despite their advancements, these methods predominantly focused on latent common representations, neglecting the specific information of each view, and thus failing to fully exploit the available comprehensive information [13, 41] .Contrastive learning, renowned for its efficacy in representation learning tasks, has been widely integrated into numerous algorithms. SimCLR established a foundational framework by pairing two augmented views of the same sample as positive pairs, with other samples forming negative ones [5] . Although this cross-view contrastive learning achieved promising performance in various tasks, it risked introducing False Negative Pairs (FNPs), which share the same category while coming from different instances, within a batch in unsupervised settings, leading to performance degradation. Intuitively, addressing FNPs is crucial for boosting the robustness and effectiveness of contrastive learning. Various techniques have been proposed to tackle this issue: Yang et al. divide FNPs using a calculated threshold based on the distance between positives and negatives [38] , while Huynh et al. employ a threshold and a top-k strategy to filter FNPs. Other approaches, like those by Chuang et al. [7] and Caron et al. [2] , focus on increasing the weight of positive pairs to enhance the robustness. However, most of these methods roughly separate the training pairs into positives and negatives, which may not guarantee the correctness of the hard binary identification for all pairs, thereby limiting their performance improvement. 
methodology: 
 In this section, we first formulate the Partially View-unaligned Problem (PVP) in subsection 3.1, and then provide an overview of VITAL in subsection 3.2. Subsequently, in the next two subsections, we elaborate on the two modules involved in VITAL, namely Variational Contrastive Learning and Dynamic Rectification.Given a multi-view dataset X = {𝑋 1 , 𝑋 2 , . . . , 𝑋 𝑉 }, where 𝑉 is the number of views, 𝑋 𝑘 = {𝒙 𝑘 1 , 𝒙 𝑘 2 , . . . , 𝒙 𝑘 𝑁 } represents the sample set of the 𝑘-th view, 𝑁 is the total number of instances, and 𝒙 𝑘 𝑖 denotes the 𝑖-th sample of the 𝑘-th view. Without loss of generality, considering the case of binary views, for a fully aligned dataset, we assume that 𝒙 𝑘 1 𝑖 and 𝒙 𝑘 2 𝑖 have alignment between 𝑘1-th view and 𝑘2-th view. However, the PVP arises when this alignment is disrupted, thus splitting the dataset into a fully aligned subset𝐴 = {𝒂 1 𝑖 , 𝒂 2 𝑖 , . . . , 𝒂 𝑉 𝑖 } 𝑁 1 𝑖=1and an unaligned subset𝑈 = {𝒖 1 𝑖 , 𝒖 2 𝑖 , . . . , 𝒖 𝑉 𝑖 } 𝑁 2 𝑖=1 .To address the mentioned PVP, existing methods primarily focus on seeking a latent common space for realignment by learning a mapping 𝑓 : X → Y [15, 38, 39] . However, this often overlooks view-specific information from each view, consequently resulting in performance degradation. Drawing inspiration from the characteristics of Gaussian distribution, where the mean encapsulates the central feature and the variance reflects its diversity, we could model the shared information as the mean and the view-specific information as the variance. To achieve this, each sample is embedded as a Gaussian distribution instead of a fixed point in the latent space. More specifically, our VITAL employs variational inference to approximate the posterior Gaussian distribution of observed samples, minimizing the divergence from the true posterior through the proposed intra-and inter-view contrastive learning. Thus, the learned common representations (means) could facilitate category-level realignment due to its central semantics, while the view-specific representations (variances) provide complementary information due to the diverse specificity. The loss function of this methodology is formulated as:EQUATIONFurthermore, to mitigate the adverse influence of FNPs, we propose a dynamic rectification module that integrates seamlessly with VITAL. This module employs a probabilistic model to infer a confidence mask for FNPs while dynamically correlating L 𝑖𝑛𝑡𝑒𝑟 with this mask to establish a robust loss L 𝑖𝑛𝑡𝑒𝑟 -𝐷𝑅 . The details of our VITAL will be elaborated in the following sections.For an observed sample 𝒙 𝑘 𝑖 , let 𝒛 𝑘 𝑖 represent its latent variables in the semantic space following [19] . We assume that 𝒛 𝑘 𝑖 adheres to a posterior Gaussian distribution 𝑝 Θ (𝒛 𝑘 𝑖 |𝒙 𝑘 𝑖 ), where Θ indicates the true parameters. Due to sampling from 𝑝 Θ may yield latent variables semantically approximate to 𝒛 𝑘 𝑖 , 𝑝 Θ encapsulates the essence of the observed sample. However, the sparsity of 𝒛 𝑘 𝑖 renders direct optimization of 𝑝 Θ (𝒛 𝑘 𝑖 |𝒙 𝑘 𝑖 ) impractical [42] . To tackle this, variational inference is employed to approximate the distribution, thereby deriving intra-and inter-contrastive learning to enhance the discrimination of representations. More specifically, for each sample 𝒙 𝑘 𝑖 , a recognition model 𝑞 𝜙 (𝒛 𝑘 𝑖 |𝒙 𝑘 𝑖 ) ∼ N (𝒛 𝑘 𝑖 ; 𝝁 𝑘 𝑖 , (𝝈 𝑘 𝑖 ) 2 𝐼 ) serves as the approximation solution to 𝑝 Θ (𝒛 𝑘 𝑖 |𝒙 𝑘 𝑖 ), where 𝜙 is the variational parameters [19] . Given that various views of an instance may serve as mutual priors, the objective is to minimize the Kullback-Leibler (KL) divergence between the approximate and true posterior across all views:EQUATIONwhere L 𝐸𝐿𝐵𝑂 denotes the evidence lower bound (ELBO) [17] . Since log 𝑝 Θ (𝒙 𝑛 𝑖 ) remains constant for the observed sample, minimizing Equation ( 2) is equivalent to maximizing L 𝐸𝐿𝐵𝑂 :EQUATIONwhere 𝑝 Θ (𝒛 𝑖 ) could be a preset arbitrary distribution. Assuming that 𝑝 Θ (𝒛 𝑖 ) follows a standard Gaussian distribution N (𝒛 𝑖 ; 0, 𝐼 ), the first term in Equation ( 3) could be optimized via a KL loss:EQUATIONMoreover, the second term in Equation ( 3) is used to measure the generative quality of the approximate posterior 𝑞 𝜙 , which can be calculated through intra-and inter-view contrastive losses:EQUATIONwhere L ′ 𝑖𝑛𝑡𝑟𝑎 and L ′ 𝑖𝑛𝑡𝑒𝑟 are intra-and inter-contrastive losses respectively. By using the SGVB estimator [19] , the two terms in Equation ( 5) can be approximated as a sampling process:EQUATIONwhere 𝒛 𝑘 (𝑖,𝑙 ) ∼ 𝑞 𝜙 (𝒛 𝑖 |𝒙 𝑘 𝑖 ) andEQUATIONwhere 𝒛 𝑛 (𝑖,𝑙 ) ∼ 𝑞 𝜙 (𝒛 𝑖 |𝒙 𝑛 𝑖 ), and 𝐿 is the number of Monte Carlo samples in the SGVB estimator, which can be set to 1 if the batch size is large enough [19] . Accordingly, Equation ( 6) could be relaxed as follows:EQUATIONwhere H denotes cross-entropy, 𝑇 𝑖𝑛𝑡𝑟𝑎 represents the intra-view ground truth, x𝑘 𝑖 = 𝑞 𝜃 (𝒛 𝑘 (𝑖,𝑙 ) ) is the result obtained through the reparameterization trick [19] via a generative model 𝑞 𝜃 , and 𝑝 (•, •) is the likelihood between two points, which is computed as follows:EQUATIONwhere 𝑠𝑖𝑚(•, •) is the calculation function of cosine similarity, and 𝜏 denotes the temperature parameter.For Equation (7) , due to the inconsistency of the cross-view variables, direct computation of the expectation through sampling is not feasible. Since the specificity (variance) of each view is independent of each other, we relax the optimization of Equation ( 7) by maximizing the shared semantics across different views as follows:EQUATIONwhere 𝑇 𝑖𝑛𝑡𝑒𝑟 is the inter-view ground truth. Therefore, the final variational contrastive learning loss function can be written as Equation (1) . Notably, 𝑇 𝑖𝑛𝑡𝑒𝑟 = 𝐼 may inadvertently introduce False Negative Pairs (FNPs) since L 𝑖𝑛𝑡𝑒𝑟 is conducted at category level, potentially impending model efficacy [27] . The following section will introduce a Dynamic Rectification module designed to produce good soft rather than hard partitions of the pairs, thus enhancing the robustness.In unsupervised contrastive learning, handling the challenge posed by FNPs is crucial for enhancing model performance. To this end, inspired by [45] , we introduce a robust mechanism to improve the traditional contrastive learning by rewriting 𝑇 𝑖𝑛𝑡𝑒𝑟 as follows:EQUATIONwhere 𝛼 is a parameter sensitive to FNPs. Equation ( 11) could produce gradients in different directions within two distinct intervals, whose dividing point can be computed by setting the gradient of L 𝑖𝑛𝑡𝑒𝑟 with respect to 𝑝 to zero:EQUATIONthen 𝑝 = 𝑒 -1/𝛼 . As a result, the optimization surface is divided into two areas, i.e., (0 < 𝑝 < 𝑒 -1/𝛼 ) and (𝑒 -1/𝛼 < 𝑝 < 1). In the two areas, the optimizer performs different optimization directions:• (0 < 𝑝 < 𝑒 -1/𝛼 ): Pairs in this interval are prone to be FNPs, mistakenly considered as hard positive samples due to low similarity by traditional contrastive learning. This misclassification will mislead the learning process, resulting in an overfitting issue [12, 45] . In contrast, our L 𝑖𝑛𝑡𝑒𝑟 applies a reverse gradient to these pairs, facilitating their correct optimization. • (𝑒 -1/𝛼 < 𝑝 < 1): Pairs within this range are likely true positive pairs, thus receiving the correct optimization direction. Before dynamically rectifying FNPs, we initially set 𝛼 to 0.1 to mitigate the impact of FNPs during the rectification process and guarantee correct optimization. After the aforementioned steps, the next challenge is to rectify the FNPs caused by category-level mismatching. In the past, probabilistic models have been proven to be useful for decoupling loss distributions between noisy labels and clean labels [1, 16, 23] . Inspired by this, we similarly propose to fit the loss distribution of all pairs using a two-component Gaussian Mixture Model (GMM):EQUATIONwhere 𝑙 represents the contrastive loss for a specific pair in order to ensure a fair comparison, 𝛾 𝑘 and 𝜙 (𝑙 |𝑘) are the mixture coefficient and the probability density of the 𝑘-th component [16] respectively, which can be simply solved using the EM algorithm [9] . Moreover, since the loss of positives is small, we consider the component with a lower mean as the distribution of positives, while the other as negatives. After that, we can easily compute the posterior probability of all pairs with respect to a specific component as follows:EQUATION)If 𝑘 represents the component with a lower mean, then {𝑤 𝑖 } 𝐵 2 𝑖=1 can be regarded as the probability that all pairs belong to positives, or alternatively, as soft labels for FNPs. As positives should have smaller losses, we dynamically associate {𝑤 𝑖 } 𝐵 2 𝑖=1 with the sensitivity parameter in 𝑇 𝑖𝑛𝑡𝑒𝑟 through 𝛼 ′ = 0.1× 𝑤 𝑖 -min( {𝑤 𝑖 } 𝐵 2 𝑖=1 ) max( {𝑤 𝑖 } 𝐵 2 𝑖=1 ) -min( {𝑤 𝑖 } 𝐵 2 𝑖=1 ), where 𝐵 represents the batch size. This implies utilizing the posterior probabilities provided by the GMM model to guide the optimization of L 𝑖𝑛𝑡𝑒𝑟 .On the other hand, we can easily separate the positive set and negative set by setting a threshold of {𝑤 𝑖 } 𝐵 2 𝑖=1 [16] . However, in practical applications, we found that directly using the GMM model for predictions in the overlapping region between two components is often inaccurate, as illustrated in Figure 2 (a) and (b). To address this issue, we propose a solution called confidence mask to dynamically focus more on clean pairs and less on noisy ones. Specifically, for the fitting result 𝜙 (𝑙 |𝑘) ∼ N (𝜇 𝑘 , 𝜎 2 𝑘 𝐼 ), we first calculate the confidence through the distance between the means of the two components:EQUATION)Then, for the initial version of the positive set 𝑀 𝐺𝑀𝑀 obtained by dividing {𝑤 𝑖 } 𝐵 2 𝑖=1 with a threshold of 0.5 [16] , we pick out the subset with higher confidence byEQUATION)where 𝑄 𝑑 represents the 𝑑 quantile of the loss value set {𝑙 𝑖 } 𝐵 2and we can finally take 𝑀 𝑐𝑜𝑛𝑓 𝑖𝑑𝑒𝑛𝑐𝑒 as the final positive set. This approach avoids pairs in the overlapping part between the two components and improves the rectification accuracy for FNPs. Figure 2 (c) and (d) demonstrate the effectiveness of 𝑀 𝑐𝑜𝑛𝑓 𝑖𝑑𝑒𝑛𝑐𝑒 through experiments. Therefore, the dynamically robust version of L 𝑖𝑛𝑡𝑒𝑟 can be written as:EQUATIONwhere 𝑇 ′ 𝑖𝑛𝑡𝑒𝑟 = 𝑝 𝛼 ′ × 𝑀, 𝑀 represents the mask of the final positive set, and × denotes element-wise multiplication. In summary, the final loss function of this module can be rewritten as:EQUATION)Through Equation ( 18), we integrate the process of dynamically rectifying FNPs with our robust contrastive loss. In the next section, we will demonstrate the specific implementation details of applying this framework to PVP.Given an observed set 𝑋 , we first use L 𝑉 𝐶𝐿 to perform the first stage training. Subsequently, we replace L 𝑉 𝐶𝐿 with L 𝑉 𝐶𝐿-𝐷𝑅 to rectify the FNPs, thereby securing a more robust model. The mean 𝝁 𝑘 𝑖 and standard deviation 𝝈 𝑘 𝑖 of each sample are utilized as its common and specific representations, respectively:EQUATIONFinally, {𝑪 𝑘 } 𝑉 𝑘=1 facilitates category-level realignment across different views, while the fused representations {𝒛 𝑖 } 𝑁 𝑖=1 are used for clustering, where 𝒛 𝑖 = [𝝁 𝑘 𝑖 + 𝝈 𝑘 𝑖 ] 𝑉 𝑘=1 . The fused representations incorporate both common and specific view information, offering comprehensiveness and interpretability. Classical clustering algorithms, such as K-means, can then be applied to the fused representations to obtain the final clustering results. 
experiment: 
 In this section, we validate the effectiveness of VITAL by evaluating the clustering performance on eight widely used multi-view datasets in both partially and fully aligned scenarios. Firstly, we introduce the experimental setups in Section 4.1, including datasets and training settings. Subsequently, in Section 4.2, we compare the clustering performance of VITAL with ten state-of-the-art deep clustering methods. After that, Section 4.3 demonstrates the effectiveness of the Dynamic Rectification module in VITAL through Eight widely used multi-view datasets were utilized in our experiments, including CUB [33] , Scene15 [11] , WIKI [8] , NUS-WIDE [6] ,Deep Animal [21] , Deep Caltech-101 [10] , MNIST-USPS [28] , and NoisyMNIST [35] . For the partially aligned scenario, we simulate it by randomly shuffling the correspondence of a portion of the original dataset. The aligned rate is defined as 𝜂 = 𝑁 1 𝑁 , where 𝑁 1 represents the number of samples with correct alignments, and 𝑁 is the total number of instances.To train VITAL, we used PyTorch version 1.12.1 with an NVIDIA 3090Ti for all experiments. Our training strategy consists of two stages: initially, we perform variational contrastive learning using L 𝑉 𝐶𝐿 to obtain the approximate posterior distribution of samples, for a total of 100 epochs. Subsequently, we replace L 𝑉 𝐶𝐿 with L 𝑉 𝐶𝐿-𝐷𝑅 for dynamic rectification of FNPs, conducting additional 10 epochs to enhance model performance. As for the hyperparameters, We employed the Adam [18] optimizer with a learning rate of 2e-3 for the first stage and 1e-4 for the second, and we fixed the contrastive temperature 𝜏 to 0.4 for training on all datasets. Regarding the network architecture, we adopted the same four-layer fully connected network for the probabilistic encoder/decoder across all datasets. It is worth noting that in VITAL, all datasets share the same model structure, which allows it to be seamlessly applied to various PVP downstream task scenarios but not limited to the field of clustering.In this section, we conducted a comparison between VITAL and ten state-of-the-art deep clustering methods in both partially and fully aligned scenarios. The methods include AE2-Nets [44] , PVC [15] , MvCLN [39] , DSIMVC [30] , MFLVC [36] , DCP [25] , GCFAgg [37] , DealMVC [40] , SURE [38] and ICMVC [3] . As only PVC, MvCLN, and SURE can directly handle partially aligned scenarios, before applying other algorithms for comparison, we first reduce the dimensionality of the original features using PCA and then realign the representations via the Hungarian algorithm [20] . After that, we run all methods five times and report the average performance in terms of three widely used metrics, including clustering Accuracy (ACC), Normalized Mutual Information (NMI), and Adjusted Rand Index (ARI), where larger values indicate better clustering performance. Table 1 and Table 2 show the performance in partially and fully aligned scenarios, respectively. Figure 3 displays the performance curves of each method under different aligned rates from 10% to 100%. From these experimental results, we can easily observe the following:• In the partially aligned scenario, VITAL achieves state-ofthe-art performance on all datasets, with significant gaps observed between it and other baselines on CUB, NUS-WIDE, Deep Animal, and Deep Caltech-101. This verifies the importance of leveraging specific information for complementary sample information, as the selected baselines do not explicitly consider it. • For PVP tasks, the comprehensiveness of the view representations learned by our framework ensures consistent superiority across different aligned rates. This advantage persists even under extreme scenarios (10% aligned). • Superior performance are obtained by VITAL in both partially and fully aligned scenarios, indicating its versatility beyond being a specific method tailored solely for PVP. We attribute this success to the contributions of both common and specific information, as well as the effective rectification of FNPs in unsupervised contrastive learning.In this section, we conduct ablation studies on the Dynamic Rectification module in VITAL. Specifically, for all eight datasets used in the experiments, we analyze the impact of the L 𝑉 𝐶𝐿-𝐷𝑅 loss term on model performance in both partially and fully aligned scenarios. The clustering performance are shown in Table 3 , where it is evident that when L 𝑉 𝐶𝐿-𝐷𝑅 is not used to rectify False Negative Pairs (FNPs) in the second dynamic training stage, there is a noticeable decline in performance across all datasets to varying degrees. This verifies the effectiveness of L 𝑉 𝐶𝐿-𝐷𝑅 .In Figure 4 , we present the t-SNE [32] visualization results of SURE [38] (the state-of-the-art method for addressing PVP) and our VI-TAL on the MNIST-USPS dataset. It is evident that VITAL exhibits larger clusters and fewer outliers compared to SURE. We attribute this to VITAL's consideration of both common and specific information of the samples. In contrast to SURE, which only utilizes common information for clustering, VITAL can achieve higher clustering accuracy. 
conclusions: 
 In this paper, we propose a framework explicitly considering the common and specific information across different views to address the Partially View-unaligned Problem (PVP). Diverging from existing works, our contributions are primarily twofold: i) a novel variational contrastive learning paradigm is proposed to learn the common and specific information from multi-view data. ii) to tackle the challenge of False Negative Pairs (FNPs) in unsupervised contrastive learning, we present a robust loss that dynamically associates the rectification process of FNPs with contrastive learning. We demonstrate the effectiveness of our framework in multi-view clustering in both partially and fully aligned scenarios. In future research, we aim to refine our method to handle cases involving Partially Sample-missing Problem (PSP) and combinations such as PSP+PVP. 
 
 Appendix:  


Decision: The initial ratings are all positive, and two reviewers raise their ratings after rebuttal. Since all ratings after rebuttal are above weak accept, this paper is accepted.

Reviews: review: paper_summary: paper_summary: The paper proposes a robust framework called VariatIonal ConTrAstive Learning (VITAL) to address the Partially View-unaligned Problem (PVP) in multi-view learning. VITAL models each data sample as a Gaussian distribution in the latent space, where the mean represents the common information across views, and the variance captures the view-specific information. To mitigate the negative impact of False Negative Pairs (FNPs) in unsupervised contrastive learning, VITAL introduces a robust contrastive loss that adaptively focuses more on clean pairs and less on noisy ones.
strength_weakness: strength_weakness: 1. The method has a certain degree of innovation.
2. It achieves significant performance improvements.
limitations: limitations: 1. There is a lack of explanation regarding the rationale behind the design. For example, why is it modeled as a Gaussian distribution? Why is variational inference introduced?
2. In Section 2.2, you point out the drawbacks of previous methods for solving the FNPs problem. What are the advantages of your method? Why does it avoid the mentioned drawbacks?
3. How does the performance vary with different missing rates?
4. Can you provide more analysis on why the proposed method is robust?
paper_summary: paper_summary: The paper proposes a contrastive learning method based on variational autoencoder, named VITAL, to solve the problem of Partially View-unaligned Problem (PVP). By integrating common information and specific information to achieve comprehensive perception, and combining confidence masks to alleviate the negative impact of FNP during the training process.
strength_weakness: strength_weakness: 1.	The Idea is simple and novel, and the motivation is clear.
2.	The article is well-organized and the sections are arranged reasonably.
3.	Good experimental results.
limitations: limitations: 1.	ELBO as the evidence lower bound is supposed to maximize. It is necessary to adjust the position of the numerator and denominator inside the log to change the addition of LELBO to the subtraction of LELBO.
2.	Equations 6 and 7 represent reconstruction and consistency semantics, respectively. But I'm not quite sure how Equations 8 and 10 are derived based on Equations 6 and 7. It feels more like a direct implementation based on the semantics expressed by Equations 6 and 7 is provided here. Why adopt this approach? There are various reconstruction and consistency losses that exist.
3.	For the derivation formula given in the appendix. I think the first term of the fifth "equal sign" in Equation 1 should be multiplied by the coefficient V. The second item of the seventh "equal sign" should also be like this.
4.	In Equations 8 and 10, The p-function returns a number that does not match the dimensions of Tintra and Tinter. Therefore, this representation is not appropriate.
paper_summary: paper_summary: This paper presents a robust framework for partially view-unaligned clustering, named VITAL. It addresses the challenges of incomplete and unaligned views by learning both common and specific information simultaneously. VITAL uses variational contrastive learning to model each data sample as a Gaussian distribution in the latent space, capturing both shared and view-specific semantics. The framework also includes a dynamic rectification module to handle false negative pairs generated during unsupervised contrastive learning.
strength_weakness: strength_weakness: - Novelty: This paper introduce variational inference to model both common and specific information across different views, which is a novel idea.
- Technical correctness: The proposed method is based on solid theoretical foundations, including variational inference and contrastive learning.
- Adequate evaluation: The authors perform extensive experiments on eight benchmark datasets in both partially aligned and fully aligned scenarios, comparing their method with ten state-of-the-art baselines. The results show consistent improvements over existing methods, demonstrating the effectiveness of VITAL.
limitations: limitations: - This paper not provide a clear ablation study to demonstrate the individual contributions of the contrastive learning and dynamic rectification modules to the overall performance of the VITAL framework.
- This paper lacks a discussion on the scalability of the proposed method to larger datasets or higher-dimensional data, which could be important for real world applications.
- This paper does provide a detailed analysis of the learned Gaussian distributions in the latent, which could offer insights into the effectiveness of the proposed in capturing common and specific information.
- This paper lacks the parameter analysis about 𝛼, which is critical to observe the influence of FNPs.
paper_summary: paper_summary: This paper proposes a robust framework called VITAL for partially view-unaligned clustering. VITAL learns both common and specific information simultaneously by modeling each data sample as a Gaussian distribution in the latent space and employing variational inference and contrastive learning. The framework also incorporates a dynamic rectification module to handle false negative pairs generated during unsupervised contrastive learning.
strength_weakness: strength_weakness: 1. The paper addresses the problem of partially view-unaligned clustering, which is more realistic than the fully aligned assumption in many real-world scenarios.
2. The proposed VITAL framework is novel in its approach to learning both common and specific information simultaneously by modeling data samples as Gaussian distributions in the latent space.
3. The use of variational inference and contrastive learning to preserve comprehensive semantic information is a theoretically sound approach.
4. The introduction of a robust contrastive loss to handle false negative pairs is an innovative solution to a common problem in unsupervised contrastive learning.
5. The experimental evaluation is adequate, with comparisons to ten state-of-the-art deep clustering methods on eight benchmark datasets, covering both partially and fully aligned scenarios.
6. The paper is well-written and clearly organized, making it easy to follow the proposed methodology and results.
limitations: limitations: 1. While the paper mentions that the proposed VITAL framework outperforms state-of-the-art methods, it would be beneficial to provide more detailed comparisons and analyses of the performance gaps, especially on the CUB, NUS-WIDE, Deep Animal, and Deep Caltech-101 datasets.
2. The paper does not provide a thorough discussion on the computational complexity of the proposed framework, which could be an important factor in real-world applications.
3. The paper lacks a sensitivity analysis of the hyperparameters involved in the VITAL framework, which could help practitioners better understand the robustness and generalizability of the proposed method.
4. Although the paper addresses the partially view-unaligned problem, it does not discuss how the proposed framework would perform in scenarios with extreme view misalignment or missing views.