COPER: Correlation-based Permutations for Multi-View Clustering
Authors: Ran Eisenberg
Jonathan Svirsky
Ofir Lindenbaum
Venue: ICLR 2025 Spotlight (2025)
Keywords: clustering
canonical correlation analysis
self supervision
multiview
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning

Abstract: Combining data from different sources can improve data analysis tasks such as clustering. However, most of the current multi-view clustering methods are limited to specific domains or rely on a suboptimal and computationally intensive two-stage process of representation learning and clustering. We propose an end-to-end deep learning-based multi-view clustering framework for general data types (such as images and tables). Our approach involves generating meaningful fused representations using a novel permutation-based canonical correlation objective. We provide a theoretical analysis showing how the learned embeddings approximate those obtained by supervised linear discriminant analysis (LDA). Cluster assignments are learned by identifying consistent pseudo-labels across multiple views. Additionally, we establish a theoretical bound on the error caused by incorrect pseudo-labels in the unsupervised representations compared to LDA. Extensive experiments on ten multi-view clustering benchmark datasets provide empirical evidence for the effectiveness of the proposed model.

Published as a conference paper at ICLR 2025
COPER: CORRELATION-BASED PERMUTATIONS FOR MULTI-VIEW CLUSTERING
Ran Eisenberg', Jonathan Svirsky', Ofir Lindenbaum Faculty of Engineering
Bar Ila University
Ramat Gan, 5290002, Israel
Tran.eisenber so ofd.....ee.. wbiu.ui....ee.........eeeae ac
ABSTRACT
Combining data from different sources can improve data analysis tasks such as clustering. However, most the current multi-view clustering methods are limited to specific domains or rely on suboptimal and computationally intensive two-stage process of representation learning and clustering. We propose an end-to-end deep learning-base baaiin multi-view clustering framework for general data types (such as images and tables). Our approach involves generating meaningful fused representa- tions using novel permutation-base canonical correlation objective. We provide a theoretical analysis showing how the learned embeddings approximate those obtained by supervised linear discriminant amalysis (LDA). Cluster assignments learned by identifying consistent pseudo-labels across multiple views. Additionally, we establish theoretical bound the error caused by incorrect pseudo-labels in the unsupervised representations compared to LDA. Extensive experiments on ten multi-view clustering benchmark datasets provide empirical evidence for the effectiveness of the proposed model.
INTRODUCTION
Clustering is an important task in data-driven scientific discovery that focuses on categorizing samples into groups based semantic relationships. This allows for deeper understanding of complex datasets and is used in various domains such as gene expression analysis in bioinformatics (Armingol
al. 2021), efficient categorizing of large-scale medical images (Kart et al. (02T), and in collider physics (Mikuni & Canelli 2021). Existing clustering approaches can be broadly categorized as
centroid-based (Boley et 1999 Jain 2010, Velmurugan & Santhanam, (201)). density-based
(Januzaj et al. 2004] Kriegel & Pfeifle 22055 Chen & |007]0 Duan el al.) 2207)), distribution -based (Preheim al., 2011
a 201 Ia), and hierarchical (Murtagh 1983 |es3]/ann Carlsson al., 2010).
Multi-view clustering an extension of the clustering paradigm that simultancously leverages diverse
views of the same observations (Sun Tao 2014 Kumar et al. 2011: Li al., 2018b).
The main idea behind multi-view clustering is to combine information from multiple data facets (or views) to obtain more comprehensive and accurate understanding of the underlying data structures. Each view may capture distinct aspects or facets of the data, and by integrating them, we can discover hidden patterns and relationships that might be obscured in any single view (Huang et al. 2012 Xu
al. 2013). This approach has great potential in applications like communication systems content delivery V Vázquez& & Pérez-Nee Neiral 2020). community detection social networks (Zhao et al. 2022
Shi et al. 2022). cancer subtype identification in bioinformatics (Wen et al.] (021). and personalize
genetic analysis through multi-modal clustering frameworks (Li et al. 2023 Kuang et al. 2024
Existing multiview clustering (MVC) methods can be divided into traditional (non-deep) and deep learning-based methods. Traditional MVC methods include: subspace (Cao et al., 2015 [uo et al.)
2018 Li al. 2019a), matrix factorization (Zhao al. 2017 Wen et 2018 Yang al. [021] and graph learning-base methods (Nie et al. 2017 Zhan et al. 2017 et al. 2018). The main drawbacks of the traditional methods are poor representation ability, high computation complexity, and limited performance in real-world data (Guo & Ye 2019).
"Equal contribution

---

Published as a conference paper at ICLR 2025
Recently, several deep leaming-based MVC schemes have demonstrated promising representation and clustering capabilities (Abavisami & Patel 2018 Alwassel et al. 2019 Li et al. 2019b; Yin eet al. 2020 Wen et al. 2020| Xu et al.] 202 by Most of these methods adopt two-stage approach, where they first learn representations, followed by clustering, as seen in works such as Lin et al. (202T) Xu et al. (2021by; Abavisani & Patel (20T8); Alwassel et al. (2019): : et al. (20Tbb); Cao et al. (2015); Zhang et al. (2017a); Brbić & Kopriva (2018); Tang et al. (2018). However, such two-stage procedure can be computationally expensive and does not directly update the model's weights based on cluster assignments; therefore, it may lead to suboptimal results.
few studies presented an end-to-end scheme for MV representation learning and clustering (Tagg & Liu 2022: Trosten et al. 20211; Chen et al.] 2020) 2023] Sun et al. [0244 Chao et al.] [2024 Gupta
al. 2024 ). By performing both tasks simultaneously. MVC can improve the data embeddings
by making them more suited for cluster assignments. However, the multi-view fusion process
used these studies may not be adaptable to all types of data, which can limit their generalization capabilities across wide range of datasets.
We introduce a new approach called COrrelation-based PERmutations (COPER) that aims to address the main challenges of multi-view clustering. Our deep learning model combines clustering and rep- resentation learning tasks, providing an end-to-end MVC framework for fusion and clustering. This climinates the need for an additional step. The approach involves learning data representations through a novel self-supervision technique, where within-class pseudo-labels are permuted (PER) across different views for canonical correlation (CO) analysis loss. The proposed framework approximates the same projection that would have been achieved by the (supervised) linear discriminant analysis (LDA) method (Fisher 1936), under some mild assumptions. This projection enhances clustering capabilities as it maximizes between-class variance while minimizing within-class variation.
Our main contributions are summarized as follows: (i) Develop a deep learning model that exploits self-supervision and a CCA- based objective for end-to-end MVC. (ii) Present multi-view pseudo-labeling procedure for identifying consistent labels across views. (iii) Demonstrate empirically and theoretically that within-cluster permutation can improve the usability of CCA-based representations for MVC by enhancing cluster separation. (iv) Analyze the relation between the solution of our new permutation- obtained by the supervised LDA. Which further justifies the usability of our model for MVC. This analysis indicates that our method can be applied to a broader range of CCA-basee methods. (v) Conduct comprehensive experimental evaluation to demonstrate the superiority of our proposed model over the state-of-the-ee deep MVC models. In Table [ we summarize the average performance for all baselines on ten datasets: COPER improves the best baselines in ACC, ARI, and NMI metrics.
Table I: Average performance.
Methad
ACC
ARI
NMI
DSMVC
48.6
47.2
CVCL
51.1
37.1
48.5
ICMVC
45.5
30.3
41.6
RMCNC
40.7
22.1
31.0
OPMC
55.9
41.4
52.9
MVCAN
44.1
29.5
40.9
38.6
19.4
30.9
DCCA-AE
47.8
29.3
41.4
CCA procedure and the solution
&CDAC
44.2
40.8
COPER
61.6
47.8
54.0
BACKGROUND
2.1 CANONICAL CORRELATION ANALYSIS (CCA)
Canonical Correlation Analysis (CCA) (Harold 1936 Thompson 1984 is a well-celebrated statistical framework for multill representation learning. CCA can help analyze the associations
between two sets of paired observations. This framework and its nonlinear extensions (Bach &
Jordan 2002 Michaeli et al. 2016 Lindenbaum al.) 2020, Salhov al. 2020 Andrew al.
2013) have been applied in various domains, including biology (Pimentel et al. 2018 neuroscience Sha-Shargie al., (017), medicine (Zhang et al.) (2177), and engineering (Chen et al. 2017).
The main goal of CCA is to find linear combinations of variables from each view, aiming to maximize their correlation. Formally, denoting the observations x (1) E I xN and X (2) e IR D2xN where both modalities are centered and encompass N samples with D1 and D2 attributes, respectively. CCA seeks for canonical vectors a E R and b E R22 such that u = a" ( and e = (2). The objective is maximize correlations between these canonical variates, as represented by the
2

---

Published as a conference paper at ICLR 2025
H
99
g
(
Fmender
H
View specilic pooudd labula
Within clostr
g
Leorr
Fasion
Chaster
Head f
P
xx(0)
((xz))
g n.
permutation
Corr
x = (xx,,)))))
g! Lx
x
Encober
200(0),00000
Encoder yru
H
T
H ((w)
cc
Figure 1: Our proposed deep learning model, COrrelation-ee based PERmutations (COPER). Two modalities and w, are processed through view-specific encoders, creating latent embeddings (H
and H ())). The embeddings are learned using correlation maximization loss and then fused to serve as input for the clustering head. The clustering head estimates probability matrix, denoted as P. which is used to derive multi-view pseudo-labels. We then generate within-cluster permutations based on these pseudo-labels. The permuted samples are used to update the CCA representation. By changing the pairing of observations fed into the CCA objective, extract embeddings that theoretically approximate the solution of supervised Linear Discriminant Analysis (LDA), as demonstrated Section
enhance cluster separation and
4.3
following optimization task:
a x (1 (X
(1)
a.bş0
a a (x (a b" x5'xt x (x
These canonical vectors be found by solving the generalized eigenpair problem
C12C C21a Xa,
Ab,
where C1, C2 are within view sample covariance matrices and C12, C21 are cross-view sample covariance matrices.
Various extensions of CCA have been proposed to study non-linear relationships between the observed modalities. Some kernel-based methods, such as Kernel CCA (Bach & Jordan 22022) Non- parameeettii CCA (Michaeli et al., 2016), and Multi-view Diffusion maps (Lindenbaum al. 2015 Salhov et al., 2020), explore non-linear connections within reproducing Hilbert spaces. However, these methods limited by pre-defined kernels, have restricted interpolation capabilities, and do not scale well with large datasets.
To overcome these limitations Andrew et al. (2013) introduced Deep CCA (DCCA), which extends traditional CCA by leveraging neural networks to model non-linear interactions between input features. This enables more flexible and scalable modeling of complex relationships in large datasets. Section 4.4.1 describes how we incorporate a DCCA objective to embed the multi-view data.
2.2 LINEAR DISCRIMINANT ANALYSIS (LDA)
Fisher's linear discriminant analysis (LDA) aims preserve variance while seeking the optimal linear discriminant function (Fisher 1936). Unlike unsupervised techniques such principal component analysis (PCA) or canonical correlation analysis (CCA), LDA a supervised method that incorporates categorical class label information to identify meaningful projections. LDA''s objective function is designed to be maximized through projection that increases the between-class seatter and reduces the within-class scatter.
For a dataset X NDD and it's covariance matrix C, we denote the within-class covariance matrix as C. and the between-cluster covariance matrix Ca.
The optimization for LDA can formulated as the following form (left), and its corresponding generalized eigenproblem (right):
max CaCa h h² C.h.
C.h CC.h, C-1Cah- Xh,
3

---

Published as a conference paper at ICLR 2025
where h is the eigenvector that maximizes the LDA objective. The representation obtained by h is ideal for clustering as aims to maximize the distance between class means while minimizing the variance within each class. A detailed formulation of LDA can be found in Appendix [.I Despite the ideal properties of LDAS representation, it not traditionally used for unsupervised learning tasks such clustering it requires labeled data for its objective. Several existing works use LDA adaptations for feature extraction and dimensionality reduction for downstream clustering (Pang et al.
2014
Zhu & Hastie 2003).
2.3
SELF-SUPERVISION FOR CLUSTERING
Self-supervised learning is method for acquiring meaningful data representations without using labeled data. It leverages information from unlabeled samples to create tasks that do not require manual annotations. In clustering tasks, self-supervision improves data representation learning by assigning pseudo-labels to unlabeled data based on semantic similarities between samples.
In single-view data, two-stage clustering frameworks proposed by Caron et al. (2018 2020); Nousi & Tefas (2020) alternate between clustering and using the cluster assignments as pseudo-labels to revise image representations, Niu et al. (2022) have introduced a pseudo-labeling method that encourages the formation of more mcaningful ful and coherent clusters that align with the semantic content of the images. Their framework treats some pseudo-labels reliable, synergizing the similarity and discrepancy of the samples. For multi-view datasets, clustering and self-supervision frameworks (Xin) al. 2021 Alwassel al. 2019 Li 2018al 8a) offer loss components that enforce consensus the latent multi- view representations. We propose a novel pseudo-labeling scheme suited for a multi-view setting, this scheme is suited for general data (image, tabular, etc.) and does not required any prior, domain specific knowledge. One of our method's 's critical and novel components is a simple update of the corresponding pairing of samples in multi-view data. Furthermore, we show a theoretical analysis that the obtained, updated CCA representation enhances cluster separation and is, therefore, suited for clustering.
RELATED WORK
Several existing MVC methods incorporate Deep Canonical correlation analysis (DCCA) during their representation learning phase. These methods obtain a useful representation by transforming multiple views into maximally correlated embeddings using nonlinear transformations (Xin et al. 2021; Chandar et al., 2016 Cao et al., 2015| Tang et al.) 2018). However, these methods use a suboptimal two-stage procedure, first creating the representation learned through DCCA and then independently applying the clustering scheme.
A few end-to-end, multi-view DCCA-based clustering solutions have been proposed: these include Tang et al. (201)), which update their representations to improve the clustering capabilities. We have developed model that belongs to the category of end-to-end representation learning and clustering. However, we have introduced several new elements that set our approach apart from existing schemes. These include novel self-supervised permutation procedure that enhances the representation, as well as a multi-view pseudo-label selection scheme. Empirical results show that these new components significantly improve clustering capabilities, presented in Section 5.1 In addition, our self-supervise permutation procedure is invariant to any CCA-based method.
4 THE PROPOSED METHOD
4.1 PROBLEM SETUP
We are given a set of multi-view data X = {x (v) E RddxNN Rd you with ny views and N samples.
(v) Each view is defined by x (v) [a] .22 .... x N and each x; is d, dimdinmnsio instance.
(v)
(0)
Our objective is to predict cluster assignment y; for each tuple of instances (xx x (2) x
(no)),
1. 1..., N in X' and the number of clusters is K.
4

---

Published as a conference paper at ICLR 2025
4.2 HIGH LEVEL SOLUTION
At the core of our proposed solution are three complementary components based on multi-view observations: (i) end-to-end representation learning, (ii) multi-view reliable pseudo-labels prediction, and (iii) within-cluster sample permutations.
Our representation learning component aims create aligned data embedding by using a maximum correlation objective. This embedding captures the shared information across multiple views of the data. More details about this component can be found in Section 4.4.I] The second component involves fusing the embeddings and predicting pseudo labels using a clustering head. We then present a filtration procedure to select reliable representatives in each cluster. Both pseudo and reliable labels are used for self-supervision as they enforce agreement between the embeddings of different views. This component is described in Section 4.4.2
Our final component uses the selected reliable label representatives and introduces permutation to the samples belonging to the same aseudo-labels. The permuted samples are then used to update the CCA representation, approximating LDA and thereby improving clustering performance (see Section 4.3 Through both theoretical and empirical analysis, we show that these permutations can enhance cluster separation when using CCA-based objectives. An overview of model, called COPER, is presented in Fig. | and a complete description of the method appears in Section [..4 We now introduce the theoretical justification for the within-cluster permutations, which are core component of our method.
4.3 THEORETICAL GUARANTEES
COPER incorporates within-cluster permutation to enhance the learned multi-view embedding. The justification for this technique lies in two theoretical aspects. The first follows Kursun et al. (201T). which artificially created multi-view data from a single view by pairing samples with other samples from the same class. The authors then demonstrate that applying CCA to such data is equivalent to linear discriminant analysis (LDA) (Fisher] 1936) (described in 2.2). Here, as part of our contribution, show theoretically that within-cluster permutations of multi data lead similar results (see Section I.3.I and 4.3.2). This also backed up by empirical evidence presented in Figure
The second theoretical justification for our permutations was presented by Chaudhuri al. 22099). The authors showed that CCA-based objectives can improve cluster separation the views are uncorrelated for samples within given cluster. Our within-cluster permutations help obtain this property, as we demonstrate empirically in Figure in Appendix F
Definition 4.1 A permutation is defined for the vector of indices I L....,,,s These are indices of N& samples whose pseudo-labele labels correspond cluster k. The within-cluster permutation defined as the application of the following operator II; It. where II, is randomly sampled from the set of all permutation matrices of size N₃ X Nç. A permutation for all clusters is denoted as II'
Where represents the permutation index, permutations may be applied multiple times across different training batches and epochs (refer to Figure 2 This flexibility allows our model to adapt to various scenarios. Once these permutations
applied to samples across the different views (v I...., we can augment the data with new artificially paired samples across views. In the following subsection, show connection between the solution of CCA induced by this within-cluster permutation augmentation procedure and the solution of LDA.
4.3.1 APPROXIMATION OF LDA
In Kursun et al. (20T)), the authors constructed an artificial multi-view dataset from single view by pairing samples with other samples from the same class labels. This procedure creates a multi-view dataset which the shared information across views the class label. They proved that applying CCA to such augmented data converges to the solution of LDA (described in Section 2.2
We assume all views capture a common source, a shared underlying variable denoted as 0. For instance, 0 can describe a mixture model that captures the true cluster structure in the data. Similar assumptions are also present in [Benton et al. (2017); Lyu Fu (2020). Based on this, we can prove a relation between our scheme and L.DA. Formally, we start from the following assumption:
5

---

Published as a conference paper at ICLR 2025
Within-clster
Within-custer permutation 11=
permuration T1
CCA v
CCA
Multiview Panado Aahels
Multivlew Preudo-tabell
Corraspondenee
CCA W
CCA
Lower Half
K-Means All: 0.598
K-Mcans ABc 0.724
0.87
Figure 2: Illustration of how within-cluster permutations enhance the embeddings learned by CCA. We use a binary subset FashionMNIST and split the images to create the two views. Next, we embed the data by applying CCA to both views, X (v) and X (w), As described in Subsection 4.4.2 the embeddings are used extract multi-view pseudo-labels, then within-cluster permutations
I1- used create new corresponding pairs of samples 21(0) and (((w), They then used as augmentations to perform a second CCA (middle pair of images). This process is repeated with II² (right-most pair of images). As shown by this example, using within-cluster permutations enhance the representations learned by CCA, improving clustering performance from an adjusted Rand index (ARI) of 0.598 to 0.872.
Assumption 4.2 For two different views v, w. the observations created by pushforward function (with noise) of some latent common parameter O that is shared across the views. Specifically, X ( E²) and e" is view specific noise.
Following our assumption, f is shared across all views and contains the cluster information. Therefore, applying within-cluster permutations on v, w (assuming there are no false cluster assignments) would be equivalent to constructing an artificial multi-view dataset in Kursun al. (2011). We show that the following proposition holds:
The embedding learned through the CCA objective using within-cluster permutation for and w converges to the same representation extracted when applying the LDA objective from to 0.
Proposition 4.3
A proof of this proposition appears in Appendix G where we show that hLDA = lCCA The proof follows the analysis of Kursun et al. (201T). Intuitively, this result indicates that the label information is "leaked" into embedding learned by CCA once apply the within-cluster permutations. This enhances cluster separation in the obtained representation and is, therefore, suitable for clustering. In Section 4.3.3 we conduct experiment using Fashion MNIST to demonstrate how within-cluster permutation can enhance the embedding learned by CCA. We visually illustrate how permutations can improve cluster embeddings in Figure 2
4.3.2 ERROR BOUND
Since our model relies on pseudo-labels, these can induce errors in the within-cluster permutation, influencing the learned representations. If such label errors induced, Proposition 4.3 breaks, and the solution of CCA with permutations is no longer equivalent to the solution of LDA.
To quantify this effect, we treat these induced errors perturbation matrix. We substitute C--Ca from Eq. with and denote D as the perturbation noise such that A A + D. In addition, we use tools from perturbation theory (Stewart & Sun 1990) to provide the following upper bound for the approximated IDA eigenvalues: [X A
2, 1 .... n,, where X; is the i'th LDA eigenvalue

---

Published as a conference paper at ICLR 2025
(a)
Reperisedd
0.4
Insuperiiedd
Mean [ -Al
10
0.3
(b)
Noise
Mean M --
Supervsed Uisupervied
0.5
0.0
0.6
46% 55% 64% 73% 82% 91%100%
77%
Percent to Permute
Percent of Data to Include (ii)
Figure 3: (i) Case study of permutation CCA using Fashion MNIST. (a) Permuting more samples within cluster improves cluster separation as measured by the Adjusted Rand Index (ARI). We compare labeled permutation (supervised) to pseudo-label-bsses (unsupervised) and random. (b) Permuting more samples also pushes the representation obtained by CCA towards LDA, as indicated by the gap between cigenvalues. (ii) An experiment on LDA approximation with induced label noise. We perform LDA on different subsets of F-MNIST. We gradually increased the number of samples starting from 20% and analyzed the effect the resulting eigenvalues År compared to the eigenvalues obtained from LDA on the entire dataset A; As expected adding more correctly annotated samples reduces the eigenvalue gap, while noisy annotations increase it.
obtained from A; A is the i'th LDA approximated eigenvalue obtained from Á; and is the total number of eigenvalues. In Appendix G.3 we provide a complete derivation of this approximation. In subsection 4.3.3 conduct a controlled experiment to evaluate how label noise influences this approximation. This implies that the more the pseudo-labels resemble the ground truth labels, the closer the representation is to LDA.
4.3.3 CASE STUDY ON FASHION MNIST
We conduct controlled experiment using F-MNIST Xiao al. 2017) corroborate theoretical results presented in this section, as shown in Figure 2 First, we create two coupled views by horizontally splitting the images. CCA is subsequently performed on the multi-view dataset, with different versions of within-cluster sample permutations.
First, we only permute samples with the same label. We show in Fig. 3 (a) that such supervised per- mutations improve cluster separation, as evidenced by the increased ARI. Moreover, our experiments panel (b) indicate that the CCA solution becomes more similar to LDA solution.
Next, we repeated the evaluations in panels (a) and (b) using permutations based on pseudo-labels. We also included the results based on random permutations and the original data (no permutation) as baselines. The results show that bring the representation of CCA closer to the solution of LDA.
permutations also improve cluster separation and
We conducted an additional experiment to measure the error induced by falsely annotated pseudo- labels. This experiment complements our LDA approximation presented in subsection 4.3.2] We perform LDA different subsets of F-MNIST. We gradually increased the number of samples starting from 20% and analyzed the effect on the resulting cigenvalues, compared to the eigenvalues obtained from LDA the entire dataset. The results, shown in Fig. 3 demonstrate that introducing false annotations increases the gap between the eigenvalues. On the other hand, including samples with accurate annotations rapidly converges the solution the precise LDA solution.
rapi
4.4
METHOD DETAILS
4.4.1
DEEP CANONICALLY CORRELATED ENCODERS
To learn meaningful representations from the multi-view observations, encoders with maximum correlation objective (Andrew et al. 2013). Specifically, we train view-specific encoders F()) which extract latent representations h
We utilize a correlation loss to encourage the embeddings to be correlated (see Subsection 2.1). This enables our model extract shared information across views and reduce view-specific noise, proving
7

---

Published as a conference paper at ICLR 2025
useful for multi-view clustering (Xin et al. 2021 Chandar et al. 2016 Cao et al. 2013 Tang et al. 2018).
For two different views, v, w we denote the latent representations by H()) E xd,,, and (w)
NNxd generated by the encoders F(r) and F(w) respectively. H H and F (") are denoted as the centered representations of H and H respectively. The covariance matrix between these
representations can be expressed C tng = H (H (T)) T / 1). Similarly, the covariance
matrices H (1) and { as Cy
/ - 1) and C = H (10) (H (1) /T////((N /(N 1).
The correlation loss is defined by Eq.
Con
Trace
1 C, C C
(3)
4.4.2 MULTI-VIEW PSEUDO-LABELING
A key aspect of our model is the generation of pscudo-labels, which are utilized for self-supervision to improve the representation learned by our model. Below are the main steps for creating pseudo- labels in multi-view setting, illustrated in Figure 4 We present them concisely and provide additional details and a detailed example of these steps in Appendix
Label Prediction The cluster head 9 predicts clus- ter assignments by fusing latent embeddings from multiple views Σ. We H H()), where wy are learnable weights. The output P RNN is probability matrix where each row corresponds to the cluster probabilities for sample. For each cluster k, we select the top B samples with the highest probabili- ties to form the set Tk of confidently labeled samples. The union of such sets is denoted as T.
H
Label Refinement Since samples assigned to the same cluster may have different representations across views, we refine the cluster assignments for each view separately. We compute view-specific clus-
Yae Sarppetpppe BA4i
ter centers (e) and evaluate the similarity of each sample to these centers using cosine similarity. Sam- ples with similarity above a threshold are retained, and pseudo-labels o!" (() are created based on their sim- ilarity scores. This ensures that each sample is more accurately assigned to clusters in each view. As part of our experiment's evaluation, detailed analysis of is described in Section E,3
Figure 4: A high-level illustration of pseudo-labeling scheme for single view w. Appendix provides a complimentary Fig- ure (5) for the entire process, with correspond- ing samples in view v and additional details.
Multi-view Agreement For samples assigned to clusters in multiple views, we enforce agreement between views by retaining only those where the cluster predictions are consistent across views (i.c., argmax(p) (v)) is the same for all views).
View-Specifie Probabilities To train the model, view-specific probability matrices p()) are com- puted using H (v) The remaining samples in T and their pscudo-labels e are used for optimizing
the cluster assignments. The final training set for each view (²))) ((x,, 9 ((")) TT).
EXPERIMENTS
5.1 DEEP BASELINES COMPARISON
We conduct extensive experiments with ten publicly available multi-viww datasets used in recent
works (Chen al. 2023 Tang Liu 2022 Chao
al. 2024 Sun al. 2024). The properties
8

---

Published as a conference paper at ICLR 2025
Table 2: Clustering ACC evaluation using ten datasets. Our model (COPER) is compared against four
recent end-to-end MVC models (DSMVC Tang & Liu 2022). CVCL (Chen et al. (2023). ICMVC
(Chao et al.) 2024). RMCNC (Sun al. 2024), OPMC (Liu et al. 202T). MVCAN (Xu et al. 2024
and two two-stage schemes. The ARI, and NMI evaluations are in Table 6
Method
METABRIC
Reuters
Caltech10--80
VOC
CallechS
RBGD
MNIST-USPS
CCV
MSRVCI
Scmne15
ACC
DSMVE
19.352224
S7.823500
10.20
39,75153
17.90112
00.-12115
24.30:29
CVCL
42.66+5.2
45 0.000-8
33.50-144
36.88=3.1
78.58-5.0
51,04+188
99 3838++0.1
26.23+1.0
77.00+12.3
40116-18
IOMYC
12.12=1.36
38.18=11.78
26.54+0.52
1..k-1.29
60.97=2.33 57
32.97=1.25 1.23
99.29-0008
2086:1.21
66.28.6611 11
41.26=0.89
RMENC
32.55=1.300 30
37.61=1.50 1.30
36.73661.35
39 18
56.77:2295
33.22=1335
76.3724.13
21.46::1.99
32.38 35
40.03:0.98
OPMC
41.25220.00
47.50-11655
48.2511155
62 1.776
59.51
43.96:1.44 1.40
72.3120224
26.9120444
83.000:111 10
40.57:1.009
MYCAN
48.3221.28
25.44
43.56:11677
17:55.0.35
75.6320.14
21,93111449
85.57&114
19,2910.0
42.95:11.114 14
38.51
AE
33.921225
43.20.40
20.39.2.3
36 2.66
36,48-3.2
32.16114
33.83336
15.58-106
36.76.13.88
33.59-2.3
DCCA-AE
25.39228
23:1522
18.18:14
46.15227
38:31=119
32.50113
18.25400
59.02:2111
35:70=2200
4,DCCA
43.64:28
34,44:3.69
20.244:2.04
44.46=11.18
47,46-2.38
25.10=1174
99.54-0109
25.41=0.55
42.57::2.00
38.67:1.88
COPER
49.13832
53.15:33
54.53:339
64.45:266
#2.81:55.1
49,89:808
99.886800
28.066111
92.57::0..8
400688::116
Table 3: Clustering ACC evaluation on linear embedding schemes. We applied K-means to the Raw dataset, PCA or CCA-transformed, and cCa-transformed with permutations (CCA perm). The
ARI, and NMI evaluations are in Table 7
Method
METABRIC
Renters
Calcch101-20
VOC
Calissh
RBCD
MINISTUSPS
CCV
MIRVCCT
Sccnels
3.16-59
4008+31
2834163
0.76-63
1635007
7400-1
PCA
37.2+14
42.56299
3436663
16.17405
74.4--611
CCA
40.0612.1
4285.25
42.311111
S3.1141
7586152
4147122
80.1515.0
16.1610.7
73.29
37.59113
CCA W.
4066116
4316217
4008238
53522644
77.4556
4629:5211
849936.3
16.39506
75.00-4.1
38.2511.2
of the datasets are presented in Table B and a complete description appears in Appendix B The implementation details are detailed in Appendix E.1
We assess the clustering performance with three commonly used metrics: Clustering Accuracy (ACC), adjusted Rand index (ARI), and Normalized Mutual Information (NMI). ACC and ARI are scored between 0 and 1, with higher values indicating better clustering performance. We conducted each experiment 10 times and reported the mean and standard deviation for each metric. We compare our
model to deep end-to-end models DSMVC (Tang & Liu| 2022). CVCL (Chen et al.
(023), ICMVC CCao et al. 2024) and RMCNC [Sun et al. 2024), OPMC Liu et al. (202)). MVCAN Xu et al. (2024) in addition to two-stage baselines, where K K-means was applied: Autoencoder (AE) transformations;
Deep CCA with autoencoders (DCCA-AE) (Chandar et al.) 2016 Wang et al. 2015) and LO-CCA
Lindenbaum al. ( transformations. The last was successfully applied for clustering in (Benton)
et al. [2017] Gao et al., 2020). We present our ACC results in Table 2 and ACC, ARI, NMI in
Our model surpasses sfate-of-the-art models in both accuracy and the adjusted Rand index across all
datasets. The accuracy improvement reaches up to 7%
5.2 PERMUTATIONS ENIIANCE PERFORMANCE IN LINEAR BASELINES
We now evaluate the effect of applying permutations on representations learned with linear models. We present Table 3 ACC evaluation of K-means applied on raw samples (Raw) without any feature transformations, PCA transformations (PCA), linear CCA transformations (CCA), and CCA- transformed samples with permutations (CCA w perm). These results demonstrate that the clustering accuracy can be improved by introducing our within-cluster permutation procedure. We further show ARI and NMI results in Table
5.3 ABLATION STUDY
We performed an ablation study to assess the impact of different components of our model. We created five variations of our model, each including different subsets of the components of COPER. These variations were: (i) COPER with linear encoder, (ii) COPER without the correlation loss, (iii)
'The code implementation provided by (Sun et al. 2024 Lindenbaum et al. (021) is restricted to two views, hence randomly chose views from each dataset with more than two views
WW note that the results of Chen et al. 2023 : Tang & Liu 2002) are different from the values reported by the authors. We report the mean over ten runs, while they report the best result out of ten runs. We argue that our evaluation, which includes the standard deviation, provides more informative indication of the capabilities of
each method. The hest results can be found in the Appendix, Table

---

Published as a conference paper at ICLR 2025
COPER without the within-cluster permutations, (iv) COPER without the multi view agreement term, and (v) COPER. We conducted this ablation study using the MSRVC1 dataset.
Table Ablation study on the MSRVC1 datasets.
Model
ACC
ARI
NMI
COPER with linear
48,00+44.6
22,22-5.0
32.26+5..3
COPER w/o fewrs
79.
64.46:444
70.76.14.1
COPER w/o permutations
85.33+7.5
14.80-10.2
79.87+7.2
COPER w/o multi-view agroc. COPER
86 86.29:77.7
16.10-10.6
80.81=7.6
92.57+0.8
84.57=14
86.91+1.3
Table 4 presents the clustering metrics across variations of COPER, The results indicate that the correlatoon loss, permutations, and multi view agreements are essential components of our model and boost the accuracy by 6- 6-12%.
5.4 SCALABILITY STUDY
In this section, we assess the scalability of our model using large-scale dataset. We generated two different views from first 300,000 samples of the infinite MNIST dataset (Loosli et al. 2007) one view with random background noise and the other with random background images from CIFARIO (Kizhevskyy) 2009). Ta shows the clustering performance 10 experiments, along with the mean and standard deviation of each metric. Since deep learning methods are known to be suitable for large-scale datasets, we compared our model to two recent end-to-end deep MVC models: DSMVC (Chao al., (024)) and RMCNC (Sun et al., (024). Other methods were not included due to practical limitations imposed by the scale of the dataset Our evaluation demonstrates that our method provides significantly more accurate cluster assignments for this large and noisy dataset.
Table 5: Clustering performance metrics large datasets generated using infinite MNIST.
Model
ACC
ARI
NMI
DSMVC 11.69+0.02
00.22:10.05
00.42:10.10
RMCNC 54.47-6..3
47.5145.26
57.73+43.33
COPER 81.93+5.44
66.7545.22
70.08+44.08
Results show that COPER outperforms both DSMVC (Chao al. 2024) and RMCNC (Sun et al. 2024). Indicating that it is scalable compared to other baselines.
5.5
CONCLUSION AND LIMITATIONS
Our work presents a new approach called COrrelation-base PERmutations (COPER), a deep learning model for multi-view clustering (MVC). COPER integrates clustering and representation tasks into an end-to-end framework, eliminating the need for separate clustering step. The model employs a unique self- ceff-supervision task, where within-cluster pseudo-labels are permuted across views for canonical correlation analysis loss, contributing to the maximization of between-class variance and minimization of within-class variation in the shared embedding space. We demonstrate that, under mild assumptions,
model approximates the projection achieved by linear discriminant analysis (LDA). Finally, we perform an extensive experimental evaluation showing that our model can cluster diverse data types accurately.
Our model has limitations, such as the potential need for relatively large batch sizes due to the DCCA loss (Andrew et al. 2013). This be addressed by alternative objectives like soft decorrelation CCang et al. 2018). Additionally, the loss combination presented here is suboptimal, and exploring smarter multitask schemes such as (Achituve et al. 2024) is promising direction for improving the method. Other directions for future work include introducing interpretability modules (Svirsky &
Lindenbaum 2024) and extending the model cluster data under domain shift (Rozner et 2023).
ACKNOWLEDGMENTS AND DISCLOSURE OF FUNDING
The work of OL. is supported by the MOST grant number 0007341.
'The code provided by CVCL. (Chen
CVCI.,Cheneee et al. 2023 and ICMVC Chao 2024 could scale this data.
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Mahdi Abavisani and Vishal M Patel. Deep multimodal subspace clustering networks. IEEE Journal of Selected Topics in Signal Processing, 12(6):1601-1114 2018.
Idan Achituve, Idit Diamant, Amnn Netzer, Gal Chechik, and Ethan Fetaya. Bayesian uncertainty for gradient aggregation in multi-task learning, arXiv preprint arXiv:2402.04005 2024.
Fares Al-Shargie, Tong Boon Tang, and Masashi Kiguchi. Assessment of mental stress effects on prefrontal cortical activities using canonical correlation analysis: an fnirs-eeg study. Biomedical optics express, 8(5):2583-5588, 2017.
Humam Alwassel, Dhruv Mahajan, Bruno Korbar, Lorenzo Torresani, Bernard Ghanem, and Du Tran. Self-supervised learning by cross-modal audio-video clustering. In NeurIPS, pp. 9758-9770, 2019.
Massih R Amini, Nicolas Usunier, and Cyril Goutte. Learning from multiple partially observed views-an application multilingual text categorization. Advances in neural information processing systems, 22, 2009.
Galen Andrew, Raman Arora, Jeff Bilmes, and Karen Livescu. Deep canonical correlation analysis. In ICMI. pp. 1247-1255, 2013.
Brick Armingol, Adam Officer, Olivier Harismendy, and Nathan E Lewis. Deciphering cell-cell interactions and communication from gene expression. Nature Reviews Genetics, 22(2):71-85) 2021.
Arthur Asuncion and David Newman. Uci machine learning repository, 2007.
Francis R Bach and Michael Jordan. Kernel independent component analysis. Journal of machine learning research, 3(Jul):1-48, 2002.
Adrian Benton, Huda Khayrallah, Biman Gujral, Dee Ann Reisinger, Sheng Zhang, and Raman Arora. Deep generalized canonical correlation analysis. arXiv preprint arXiv: 1702.00599 2017.
Daniel Boley, Maria Gini, Robert Gross, Eui-Hong Sam Han, Kyle Hastings, George Karypis, Vipin Kumar, Bamshad Mobasher, and Jerome Moore. Partitioning-based clustering for web document categorization. Decision Support Systems, 27(3):32-3341, 1999.
Maria Brbić and Ivica Kopriva. Multi-viww low-rank sparse subspace clustering. Pattern Recognition, 73:247-258, 2018.
Xiaochun Cao, Changqing Zhang, Huazhu Fu, Si Liu, and Hua Zhang. Diversity-induced multi-view subspace clustering. In CVPR, pp. 586-594, 2015.
Gunnar E Carlsson, Facundo Mémoli, et al. Characterization, stability and convergence of hierarchical clustering methods. J. Mach. Learn. Res., 11(App):1425-1470, 2010.
Mathilde Caron, Piotr Bojanowski, Armand Joulin, and Mathhij Douze. Deep clustering for unsuper- vised learning of visual features. In Proceedings of the European conference on computer vision (ECCV), pp. 132-149 149, 2018.
Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised learning of visual features by contrasting cluster assignments. Advances in neural information processing systems, 33:9912 999994299994,, 2020.
Sarath Chandar, Mitesh M Khapra, Hugo Larochelle, and Balaraman Ravindran. Correlational neural networks. Neural computation, 28(2)2577 2016.
Xiaobin Chang, Tao Xiang, and Timothy M Hospedales. Scalable and effective deep cca via soft decorrelation, In Proceedings the pp. 1488-1497, 2018.
the IEEE Conference on Computer Vision and Pattern Recognition,
Guoqing Chao, Yi Jiang, and Dianhui Chu. Incomplete contrastive multi-view clustering with high- confidence guiding. Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp. 11221-11229, 2024.
11

---

Published as a conference paper at ICLR 2025
Kamalika Chaudhuri, Sham M Kakade, Karen Livescu, and Karthik Sridharan. Multi-view clustering via canonical correlation analysis. In Proceedings of the 26th annual international conference on machine learning, pp. 129-136. 2009.
Jie Chen, Hua Mao, Wai Lok Woo, and Xi Peng. Deep multiview clustering by contrasting cluster assignments. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 16752-16761, October 2023.
Mansheng Chen, Ling Huang, Chang-Done Dong Wang, and Dong Huang. Multi-view clustering in latent embedding space. In AAAI, pp. 151--3220, 2020.
Yixin Chen and Li Tu. Density- based clustering for real-time stream data. In Proceedings of the 13th ACM SIGKDD international conference on Knowledge discovery and data mining, pp. 133-142, 2007.
Zhiwen Chen, Steven X Ding, Tao Peng, Chunhua Yang, and Weihua Gui. Fault detection for non- gaussian processes using generalized canonical correlation analysis and randomized algorithms. IEEE Transactions on Industrial Electronics, 65(2):155911577, 2017.
Christina Curtis, Sohrab P Shah, Suet-Feung Chin, Gulisa Turashvili. Oscar M Rueda, Mark Dunning, Doug Speed, Andy G Lynch, Shamith Samarajiwa, Yinyin Yuan, et al. The genomic and transcriptomic architecture of 2,000 breast tumours reveals novel subgroups. Nature, 486(7403): 346-352, 2012.
Sarah-Jam Dawson, Oscar M Rueda, Samuel Aparicio, and Carlos Caldas. A new genome-driven integrated classification of breast cancer and its implications. The EMBO journal, 32(5):617-628, 2013.
Lian Duan, Lida Xu, Feng Guo, Jun Lee, and Baopin Yan. local-density based spatial clustering algorithm with noise. Information systems, 32(7):978-986, 2007.
Delbert Dueck and Brendan J Frey. Non-metric affinity propagation for unsupervised image cat- egorization. 2007 IEEE 11th international conference on computer vision, pp. 1-8. IEEE, 2007.
Mark Everingham, Lue Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman. The pascal visual object classes (voc) challenge. International journal of computer vision, 88:303-338, 2010.
Li Fei-Fei and Pietro Perona. A bayesian hierarchical model for learning natural scene categories. In 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'0S) 05), volume 2, pp. 524-531. IEEE, 2005.
Ronald A Fisher. The use of multiple measurements in taxonomic problems. Annals of eugenics, (2):179-188, 1936.
Quanxue Gao, Huanhuan Lian, Qianqian Wang, and Gan Sun. Cross-modal subspace clustering via deep canonical correlation analysis. In Proceedings of the AAAI Conference on artificial intelligence, volume 34, pp.
2020.
Jun Guo and Jiahui Ye. Anchors bring case: An embarrassingly simple approach to partial multi-view clustering. In AAAI, pp. 118-125, 2019.
Pooja Gupta, Anurag Goel, Angshul Majumdar, Emilie Chouzenoux, and Giovanni Chierchia. De- confeluster: Deep convolutional transform learning based multiview clustering fusion framework. Signal Processing. 224:109597, 2024.
Hotelling Harold. Relations between two sets of variables. Biometrika, 28(3):321-377 1936.
Xudong Huang, Yanan Gu, Kang Liu, and Xudong Gao. A self-training co-training algorithm for multiview spectral clustering. Pattern Recognition Letters, 33(13):1690-1700 2012.
Anil K Jain. Data clustering: 50 years beyond k-means. Pattern recognition letters. 31(8):651-666, 2010.
12

---

Published as a conference paper at ICLR 2025
Eshref Januzaj, Hans-Peter Kriegel, and Martin Pfeifle. Scalable density-basee distributed clustering. In Knowledge Discovery in Databases: PKDD 2004: 8th European Conference on Principles and Practice of Knowledge Discovery in Databases, Pisa, Italy, September 20-24, 2004. Proceedings 8, pp. 231-244. Springer, 2004.
Bin Jiang, Jian Pei, Yufei Tao, and Xuemin Lin. Clustering uncertain data based probability distribution similarity. IEEE Transactions on Knowledge and Data Engineering, 25(4):751-763,
2011a.
Yu-Gang Jiang, Guangnan Ye, Shih-Fu Chang, Daniel Ellis, and Alexander C Loui. Consumer video understanding: A benchmark datahase and an evaluation of human and machine performance. In ICMR, pp. 1-8, 201
Turkay Kart, Wenjia Bai, Ben Glocker, and Daniel Rueckert. Deepmcat: large-scale deep clustering for medical image categorization. In Deep Generative Models, and Data Augmentation, Labelling, and Imperfections: First Workshop, DGM4MICCAI 1021, and First Workshop. DALI 2021, Held in Conjunction with MICCAI 2021, Strasbourg, France, October 1, 2021, Proceedings 1, pp. 259-267. Springer, 2021.
Chen Kong, Dahua Lin, Mohit Bansal, Raquel Urtasun, and Sanja Fidler. What are you talking about? texi-to-image coreference. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3558-3565, 2014.
Hans-Peter Kriegel and Martin Pfeifle. Density-based clustering of uncertain data. In Proceedings of the eleventh ACM SIGKDD international conference on Knowledge discovery in data mining, pp. 672-677, 2005.
Alex Krizhevsky. Learning multiple layers of features from tiny images. Technical Report TR-2009, University of Toronto, 2009, URL httpp::/www.cs. cs toronto..dd..... edukkkkoee/// fearning-features- pdE
Yabin Kuang, Minzhu Xie, Zhanhong Zhao, Dongze Deng, and Ergude Bao. Multi-view contrastive clustering for cancer subtyping using fully and weakly paired multi-omics data. Methods, 232:1-8, 2024.
Ankit Kumar, Hal Daumé III, and Amol Saha. Co-regularized multi-view spectral clustering. Proceedings of the 28th international conference on machine learning (ICML-11), pp. 521-528, 2011.
Olcay Kursun, Ethem Alpaydin, and Oleg V Favorov. Canonical correlation analysis using within- class coupling. Pattern Recognition Letters, 3242):1344144, 2011.
Chao Li, Cheng Deng, Ning Li, Wei Liu, Xinbo Gao, and Dacheng Tao. Self-supervised adversarial hashing networks for cross-modal retrieval. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4242-4251 2018a.
Ruihuang Li, Changqing Zhang, Huazhu Fu, Xi Peng, Tianyi Zhou, and Qinghua Hu. Reciprocal multi-layer subspace learning for multi-view clustering. In ICCV, pp. 8172-8180, 2019a.
Xiang Li, Xiaojie Guo, and Zhi Zhang. Multiview clustering: A survey. In 2018 24th International Conference on Paltern Recognition (ICPR), pp. 387-392. IEEE, 2018b.
Zhaoyang Li, Qianqian Wang, Zhiqiang Tao, Quanxue Gao, and Zhaohua Yang. Deep adversarial multi-view clustering network. In IJCAI, pp. 2952-2958, 2019b.
Zuqi Li, Federico Melograna, Hanne Hoskens, Diane Duroux, Mary L Marazita, Susan Walsh, Seth M Weinberg, Mark D Shriver, Bertram Müller-Myhsoke Peter Claes, et al. netmug: novel network-guided multi-view clustering workflow for dissecting genetic and facial heterogeneity. Frontiers in Genetics, 14:1 2868000 2023.
Yijie Lin, Yuanbiao Gou, Zitao Liu, Boyun Li, Jiancheng Lv, and Xi Peng. COMPLETER: Incomplete multi-view clustering via contrastive prediction. In CVPR, 2021.
13

---

Published as a conference paper at ICLR 2025
Ofir Lindenbaum, Arie Yeredor, and Moshe Salhov. Learning coupled embedding using multiview diffusion maps. In Latent Variable Analysis and Signal Separation: 12th International Conference, LVA/ICA 2015. Liberec, Czech Republic, August 25-28, 2015, Proceedings 12, pp. 127-134. Springer, 2015.
Ofir Lindenbaum, Arie Yeredor, Moshe Salhov, and Amir Averbuch. Multi-view diffusion maps. Information Fusion, 15:127-149, 2020.
Ofir Lindenbaum, Moshe Salhov, Amir Averbuch, and Yuval Kluger. LO-sparse canonical correlation analysis. In International Conference Learning Representations, 2021.
Jiyuan Liu, Xinwang Liu, Yuexiang Yang, Li Liu, Siqi Wang, Weixuan Liang, and Jiangyong Shi. One-pass multi-view clustering for scale data. In ICCV, pp. 12344-12353, 2021.
Gaël R Loosli, Stéphane Canu, and Léon Bottou. Training invariant support vector machines using selective sampling. In Large Scale Kernel Machines, pp. 301-320. MIT Press, 2007.
Shirui Luo, Changqing Zhang, Wei Zhang, and Xiaochun Cao. Consistent and specific multi-view subspace clustering. In AAAI, 2018.
Qi Lyu and Xiao Fu. Nonlinear multiview analysis: Identifiability and neural network-assisted implementation. IEEE Transactions Signal Processing, 68:2697-2712, 2020.
Tomer Michaeli, Weiran Wang, and Karen Livescu. Nonparametric canonical correlation analysis. In International conference on machine learning, pp. 1967-1976. PMLR, 2016.
Vinicius Mikuni and Florencia Canelli. Unsupervised clustering for collider physics. Physical Review D, 103(9):02200,, 2021.
Fionn Murtagh. A survey of recent advances in hierarchical clustering algorithms. The computer journal, 26(4):354-359 1983.
Feiping Nie, Jing Li, and Xuelong Li. Self-weighted multiview clustering with multiple graphs. In IJCAI, pp. 2564-2570, 2017.
Chuang Niu, Hongming Shan, and Ge Wang. Spice: Semantic pseudo-labeling for image clustering. IEEE Transactions on Image Processing, 31:7264-7278, 2022.
Paraskevi Nousi and Anastasios Tefas. Self-supervised autoencoders for clustering and classification. Evolving Systems, 11(3):533446, 2020.
Yanwei Pang, Shuang Wang, and Yuan Yuan. Learning regularized Ida by clustering. IEEE transac- fions on neural networks and learning systems, 25(12):119-2201, 2014.
Harold Pimentel, Zhiyue Hu, and Haiyan Huang. Biclustering by sparse canonical correlation analysis. Quantitative Biology, 6:56-67 67, 2018.
Sarah P Preheim, Allison R Perrotta, Antonio M Martin-Platero, Anika Gupta, and Eric Alm. instriutionn-based clustering: using ecology to refine the operational taxonomic unit. Applied and environmental microbiology, 79(21):6593-66
2013.
Peter Rousseeuw. Silhouettes: graphical aid to the interpretation and validation of cluster analysis. Journal of computational and applied mathematics, 20:53-65 65, 1987.
Amit Rozner, Barak Battash, Lior Wolf, and Ofir Lindenbaum. Domain Gonain-generaalizalee multiple- domain clustering. arXiv preprint arXiv:2301. 13 2023.
Moshe Salhov, Ofir Lindenbaum, Yariv Aizenbud, Avi Silberschatz, Yoel Shkolnisky, and Amir Averbuch. Multi-view kernel consensus for data analysis. Applied and Computational Harmonic Analysis, (0(1):20-220) 2020.
Xiaofeng Shi, Cheng Liang. and Hong Wang. Multiview robust graph-based clustering for cancer subtype identification. IEEE/ACM Transactions on Computational Biology and Bioinformatics, 20 (1):544-556 2022.
14

---

Published as a conference paper at ICLR 2025
Gilbert W Stewart and Ji-guang Sun. Matrix perturbation theory. (No Title), 1990.
Xiaoshuai Sun and Dacheng Tao. A survey of multiview machine learning. Neurocomputing, 128: 22-45, 2014.
Yuan Sun, Yang Qin, Yongxiang Li, Dezhong Peng, Xi Peng, and Peng Hu. Robust multi-view clustering with noisy correspondence. IEEE Transactions on Knowledge and Data Engineering, 2024.
Jonathan Svirsky and Ofir Lindenbaum. Interpretable deep clustering for tabular data. In International Conference on Machine Learning, pp. 47314-47330. PMLR, 2024.
Huayi Tang and Yong Liu. Deep safe multi-view clustering: Reducing the risk of clustering performance degradation caused by view increase. Proceedings of the IEEF/CVF Conference Computer Vision and Pattern Recognition, pp. 202-211, 2022.
Xiaoliang Tang, Xuan Tang, Wanli Wang, Li Fang, and Xian Wei. Deep multi-view sparse subspace clustering. In Proceedings of the 2018 VII International Conference on Network, Communication and Computing, pp. 115-119, 2018
Bruce Thompson. Canonical correlation analysis: Uses and interpretation. Number 47. Sage, 1984.
Daniel Trosten, Sigurd Lokse, Robert Jenssen, and Michael Kampffmeyer. Reconsidering repre- sentation alignment for multi-view clustering. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 1255-1265, 2021a a.
Daniel J. Trosten, Sigurd Løkse, Robert Jenssen, and Michael Kampffmeyer. Reconsidering repre- sentation alignment for multi-view clustering. In CVPR, pp. 1255-1265, 2021b.
Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine
learning research, 9(1)), 2008.
Miguel Ángel Vázquez and Ana Pérez-Neira. Multigraph spectral clustering for joint content delivery and scheduling in beam-free satellite communications. In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 8802-8806. IEEE, 2020.
Velmurugan and Santhanam. A survey of partition based clustering algorithms in data mining: An experimental approach. Information Technology Journal, 10(3):478 484, 2011.
Weiran Wang, Raman Arora, Karen Livescu, and Jeff Bilmes. On deep multi-view representation learning. In ICML, pp. 1083-1092, 2015.
Jie Wen, Zheng Zhang, Yong Xu, and Zuofeng Zhong. Incomplete multi-view clustering via graph regularized matrix factorization. ECCV Workshops, 2018.
Jie Wen, Zheng Zhang, Yong Xu, Bob Zhang, Lunke Fei, and Guo-Sen Xie. CDIMC-net: Cognitive deep incomplete multi-view clustering network. In IJCAI, pp. 3230-322- 020000233,220 2020.
Yuqi Wen, Xinyu Song, Bowei Yan, Xiaoxi Yang, Lianlian Wu, Dongjin Leng, Song He, and Xiaochen Bo. Multi-dimensional data integration algorithm based on random walk with restart. BMC bioinformatics, 22:1-22, 2021.
Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms. arXiv preprint 17X17:1008.07747, 2017.
Bowen Xin, Shan Zeng, and Xiuying Wang. Self-supervised deep correlational multi-view clustering. In 2021 International Joint Conference on Neural Networks (IJCNN) pp. 1-8. IEEE, 2021.
Chang Xu, Dacheng Tao, and Chao Xu. A survey on multi-view learning. arXiv preprint w.Xiv:134.5662. 2013.
Jie Xu, Yazhou Ren, Guofeng Li, Lili Pan, Ce Zhu, and Zenglin Xu. Deep embedded multi-view clustering with collaborative training. Information Sciences, 573:279-290,, 2021a.
15

---

Published as a conference paper at ICLR 2025
Jie Xu. Yazhou Ren, Huayi Tang. Xiaorong Pu, Xiaofeng Zhu, Ming Zeng, and Lifang He. Multi- VAE: Learning disentangled view-common and view-peculiar visual representations for multi-view clustering. In ICCV, pp.
2021b.
Jie Xu, Yazhou Ren, Xiaolong Wang, Lei Feng, Zheng Zhang, Gang Niu, and Xiaofeng Zhu. Investigating and mitigating the side effects of noisy views for self-supervised clustering algorithms in practical multi-view scenarios. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp. 22957-22966, 2024.
Zuyuan Yang. Naiyao Liang, Wei Yan, Zhenni Li, and Shengli Xie. Uniform distribution non-negative matrix factorization for multiview clustering. IEEE Transactions on Cybernetics, pp. 3249-3262, 2021.
Ming Yin, Weitian Huang, and Junbin Gao. Shared generative latent representation learning for multi-view clustering. In AAAI, pp. 6688-6695, 2020.
Kun Zhan, Changqing Zhang, Junpeng Guan, and Junsheng Wang. Graph learning for multiview clustering. IEEE Transactions on Cyberneties, 48(10)2887722855 2017.
Changqing Zhang, Qinghua Hu, Huazhu Fu, Pengfei Zhu, and Xiaochun Cao. Latent multi-view subspace clustering. In CVPR, pp. 4279-4287, 2017a.
Yu Zhang, Guoxu Zhou, Jing Jin, Yangsong Zhang, Xingyu Wang, and Andrzej Cichocki. Sparse bayesian multiway canonical correlation analysis for eeg pattern recognition. Neurocomputinng 225:103-110, 2017b.
Handong Zhao, Zhengming Ding, and Yun Fu. Multi-view clustering via deep matrix factorization. In AAAI, pp. 2921-2927, 2017.
Wei Zhao, Cai Xu, Ziyu Guan, and Ying Liu. Multiview concept learning via deep matrix factorization.
IEEE transactions on neural networks and learning systems, 32(2):814-825, 2020.
Xusheng Zhao, Qiong Dai, Jia Wu, Hao Peng, Mingsheng Liu, Xu Bai, Jianlong Tan, Senzhang Wang, and S Yu Philip. Multi-view tensor graph neural networks through reinforced aggregation. IEEE Transactions on Knowledge and Data Engineering, 35(4):407774001, 2022.
Runwu Zhou and Yi-Dong Shen. End-to-end anvversarilllattention network for multi-modal clustering. In CVPR, pp. 146188 2020.
Miaoyu Zhu and Trevor Hastie. Feature extraction for nonparametric discriminant analysis. Journal of Machine Learning Research, 3(May): 1-00, 2003.
Xiaofeng Zhu, Shichao Zhang, Wei He, Rongyao Hu, Cong Lei, and Pengfei Zhu. One-step multi-view spectral clustering. IEEE Transactions on Knowledge and Data Engineering, 31(10): 2022-2034, 2018.
16

---

Published as a conference paper at ICLR 2025
A RESULTS
Table 6: Clustering evaluation using ten datasets. Our model (COPER) is compared against four
recent end-to-end MVC models (DSMVC (Tang & Liu 2022), CVCL CChn et al.) 2023), ICMVC
(Chao et al. 2024). RMCNC (Sun al. 2024), OPMC (Lu et al.
2021 MVCAN (Xu et al. 2024
and two stogs schemes.
Method
METABRIC
Keuters
Callechyv
MINIST
KCMVC
32.12=1.16
38.18800.7
26.54:0.52
36.94.1.20
60.9762.37
32.97:1.255
99.2910.05
20.8813.21
66.2826.11
41.26::0.59
RMCNC
32.55_133 30
37,6121150
36.73=1.38
39.06:11.40
56.77:2..08 98
76.371:411
1144640.05
52.38-1.35
40.03.0.96
OPMC
41:35-040
47.50 42.50-00.5
1.955-195
62.86 1.776
41.96-1 1.0
72.31-0034 34
16.4100.44
18.001100
40997-149
MYCAN
48.32-122 28
45.4622881
4..56=1667
17.85:0.35
756360014
21.03:0.49
65.57=1 14
19.2940.00
42.55:114
36.1:2185
AB
28524225
43.7554.
00.39:235
36.00:2..00
51.08132
32.16:16
15 15.88106
56 56,76838
33.59:23
DCUA-AL
85.19228
45188228
48188:34
26.3222
56337439
32,59.13
92.19 92.19:24
1828:405
5962611
31:00:20
AAKCAA
41 8
99 29:54-204
46.6611:188
47.8642.88
28 14:10:124
99.56-009
18411083
67.572200 200
38.671188
COPER
5315E33
54
6466822
928115.1
8130208
99.881000
28.062111
92.5728.8
00.88:166
ARI
DSMPVC
25
3121222
2377222
1631
5.93555
122SE 1500
1USS22
CVCL
32.99543
2435-09
22,0923
63.25663
1.
9863463
12:72+13
6427-198
24.01-17
KMVC
10:46:121
0500
25
41578255
45-75-221
MVCAN
OPMC
22.69
0.59
23,25 4
2.76
0.30
61,55
1 200
0.00
1557 17.57-007
25.51 2:
23.89
24.4240022
AE
14.1--00
10.36143
18.992
1998-166
4343148
12.00-17
DCCA-AE
S7.9E53
N2.753A
3461541
15503200
1841815
17:2211
GEDCCA
21.74 = 1.94
0.65 200
27.37 = 1.93
3 1:92
030
98.90
11 0.56
17.56 8 2.64
22.57 = 0.56
COPER
9935:53
5336349
99758800
1237
$4576314
25:0:111
KM
19
57.77-110
W.55-13
CVCL
32.9 4.5
20 10.1-1111
56.18-10
22.05-1.3
69 89.66-48
3r60-16 1.
982146.
2625-108
7265+8.9
41.33-17
KMYC
18,50:100 102
20 20.5250020
0.590
42.72:40.33
503232210 10
28372987
97
15.1330.06
56.42:4.37
4399::0.77
RMUNC
2.00
17:15:1106
29:31-0044
64.49:2300
160510555
41.22::0:4
QPMC
66.16
62.05
0.73
77.38 0.22
0.40
79.08
41.50
MYCAN
34
1.01
30.25
62.52
11.95
70.46
0.46
1.42
1.55
32.74
42.46 1.16
ICCAAAF
35041311
40.30-40
$2.494411
G,CCCA
1.38
4.30
43.75
0.67
08.73
1.20
42.04
COPER
3400752.6
31.1054.1
49225262
38.1300
9966880.1
25.22300
36.1333.3
11
Table 7: Clustering evaluation on linear embedding schemes. We applied K-means to the Raw dataset, PCA or CCA-transformed, and CCA-transformed with permutations (CCA w perm).
Mothod
METABRIC
Renters
Callrch001220
VOC
CalleehAAe
RBGD
VNIIS-USP
CCV
MISVCC
Sccne15
ACC
Row PCA
35.18129
37,16:5.9
42,9813.1
5241173
73541633
4315513
68.78:63
16.2510.7
74.00-33
3791309
37.72114
42.56:29
41,4713.4
43.66149
24226663
4264119
68.1439
16.3710.5
74.14-6.1
57.53112
4000++2.1
4288-225
42.34-3.1
531+411
7506452
4147422
80.15-500
16.16+0.7
73.29-7.3
37.59+1.3
406681.6
4316=17
4308338
5852244
7744456
46299221
$499-6.3
16,399065
75.00-411
38.25112
1861F13
58.3
PCA
19.01115
19.88.14
31.75135
214415
5791147
23.98114
$4.55128
5.45.0.2
58.39
22.01004
CCA
19.41117
78.7
32.68.188
346915
6010119
24.9
71.09 4.5
5.30.0.33
56.97 7.2
22.57.105
CCA Wi pecm
20.66:15
17.8--17
32.993433
6L45143
2504
76.02-59
5.3950.3
$7.9--377
22.638044
NMI
Row
15:38-90
$339433
6434410
668--1 1.8
PCA
25.4+117
27.92-1.2
60:50:144
44 29+99
63.79+255
3743=10
15.27=111 1
15.17+0.3
67.57=4.0
40 40.3340055
CCA
26.8512.0
27.16:1.0
61 13:220
63011136
65.72119 1
38 38.91:10
77,58-2.2
14.38104
65 66.52-2
40. 40.7510.5
CCA
27.821116
26.96:2006
6100222
54.54133
663992
39.322008
797
14,16004
68.89-3.1
41.02203
B DATASETS DESCRIPTION
METABRIC Curtis et al. 2012): Consists of 1, 440 samples from breast cancer patients, which are annotated by subtypes based on InClust Dawson et al. (0013). We observe two modalities, namely the RNA gene expression data and Copy Number Alteration (CNA) data. The dimensions of these modalities are 15, 709 and 47, 127, respectively.
Reuters Amini et al. (009): Consists of 18, 758 documents from 6 different classes. Documents are represented as a bag of words using TFIDF-based weighting scheme. This dataset a subset of the Reuters database, comprising the English version as well as translations in four distinct languages: French, German, Spanish, and Italian. Each language is treated as different view. To further reduce the input dimensions, we preprocess the data with a truncated version of SVD, turning all input dimensions to 3, 000.
Caltech101-20 Zhao et al. (011); Consists of 386 images of 20 classes. This dataset is a subset of Caltech101. Each view is an extracted handcrafted feature, including the
17

---

Published as a conference paper at ICLR 2025
Gabor feature, Wavelet Moments, CENTRIST feature, HOG feature, GIST feature, and LBP feature.
VOC Everingham et al. 2010): Consists of 9, 963 image and text pairs from 20 different
classes. Following the conventions by TTrssen et al. (2021a); Van der Maaten & Hinton
(2008), 5, 649 instances are selected to construct two-View dataset, where the first and the second view is 512 Gist features and 399 word frequency count of the instance, respectively.
Caltech-5V-77 Dueck Frey] (2007): Consists of 1, 400 images of classes. Same as Caltech101 this dataset is also a subset of Caltech101 and is comprised of the same views apart from the Gabor feature.
RBGD Kong et al. (2014) Consists of 1, 449 samples of indoor scenes image-text of 13 classes. We follow the version provided in Trosten et al. (202Ia); Zhou & Shen (2020), where image features are extracted from ResNer50 model pre-trained on the ImageNet dataset and text features from a doc2vec model pre-trained on the Wikipedia dataset.
MNIST-USPS Asuncion Newman (2007): Consists of 5, 000 digits from 10 different classes (digits). MNIST and USPS are both handwritten digital datasets and are treated as two different views.
CCV Jiang et al. (b)1 Consists of 6, 773 samples of indoor scenes image-text of 20 classes. Following the convention in Li et al. (2019b) we use the subset of the original CCV data. The views comprise of three hand-crafted features: STIP features with 5, 000 dimen- sional Bag-of- Words (BoWs) representation, SIFT features extracted every two seconds with 5, 000 dimensional BoWs representation, and MFCC features with 4, 000 dimensional BoWs.
MSRCv1 Consists of 210 scene recognition images belonging to 7 categories Zhao et al. 2020). Each image is described by five different types of features.
Scene15 Consists of 4,485 scene images belonging to 15 classes Fei-Fee Fei-Fei & Perona 2005).
300K-MNIST-CIFARIO Looshi et al. (2007); Krizhevsky (2009): Consists of 300, 000 samples generated from the infinite MNIST dataset. Two different views are created: with random background noise and the other with random background images from CIFAR10. This dataset is used to assess the scalability and robustness of clustering methods under noisy and large-scale conditions. Deep learning-basee models such DSMVC Chao et al. (2024) and RMCNC Sun et al. (2024) demonstrate improved performance over traditional approaches on this dataset.
Table 8: Datasets used in our experiments.
Dataset
Samples
# lasses (K) # Views
Dimensions
Ke
METABRIC
1440
[15709, 471271
Curtis at
2012
Reulers
18758
[21531, 24892, 34251, 15506, 11547]
Caltech101-20
2386
20
(48, 40, 254, 1984. 512, 928]
Zhao 21
2017
VOC
5649
20
[512. 399]
Everinghan
2010
Caltech-SV.7
1400
[40, 254, 1984, 512, 928]
Dieck & Frey
1007
REGD
1449
13
[2048, 300]
201
MNIST-USPS
5000
10
[784, 784]
Asunciinn & Newman
2007
CCV
6773
20
[4000, 5000, 5000]
MSRVCI
210
[24, 576, 512, 256, 254]
7han al 21200
Scene15
4485
15
[20, 59, 40)
Fei-Fot Ferona
2005
C COMPLEXITY ANALYSIS
Let Nmb represent the number of samples in mini batch and N the total number of samples in the dataset, L denote the maximum number of neurons in the model's hidden layers across all views, and I indicates the dimension of the low-dimensional embedding space. Additionally, L' stands for the number of neurons in the cluster head, and the number of clusters is denoted as K. For reliable labels, we denote Nms as the number of samples in the permuted mini batch.
The time complexity of training each autoencoder is O(L Nma TN 1). With M views, the total complexity for the pretraining phase is O(M -- N
N Nmm 1). Calculating the CCA loss for each
The creation of both Caltech 101-20 and Caltech-5V-7 due the unhalanced classes in Caltech-101.
18

---

Published as a conference paper at ICLR 2025
pair of views has complexity of (N Now 1) for each pair. Since there are (2)) possible pairs, the total complexity for the CCA loss O(M2 Non [ N 1)-
The time complexity for training the cluster head is O(L' Nmm Nma 1). Computing reliable labels N
for each view and each cluster also involves a complexity of O(M Nmm the total complexity for the multi-view reliable labels tuning phase is O(M L' -mm
[ K). Therefore, NNa.
N K).
Computing CCA loss for reliable labels involves complexity of O( -.N-1) 1). The overall time complexity of our multi-view CCC model is the sum of the complexities of the individual phases. Therefore, the total time complexity is:
Nom
N Nus
N Nmb
O(M - L. Nmb
Nmb 1 M². N mb
+ M L', Nmb 1. K + N
D)-
Nmb
Where the dominant factor is:
N N
O(M2.
N
1)-
mb
D MULTI-VIEW PSEUDO-LABELING
D.1 DETAILED STEPS
Labels prediction Our cluster head G accepts fusion of latent embeddings. The fusion is weighted sum E, (o)) where { + are learnable weights. For all samples, G predicts clusters probabilities g(E. w. H()) P N where each row a probability vector p; for samples a, for all E 1,.., n. (as is shared between all views). Each cluster is represented by column index in P. and by selecting the top probabilities, which are the most confident samples for each & we form To {ili argtopk :.A: B), Vi = 1,2,... ..., N). We denote T Us the set of samples that were assigned with at least one label. At this stage, we note that (a) samples that are assigned to the same cluster based T have embedding vectors that have different geometric relations to cluster centers in different views, and (b) some samples may be assigned to more than one cluster. Therefore, we present the following steps that refine the list of pseudo-labels identified in T.
Labels refinement To deal with (a), we refine the clusters separately for each view We start by and the semantic similarity between them and all other embedded samples h Vi E Th Semantic similarity expressed by cosine similarity 85,k = 1. F.(") /| h | [o) For each cluster, we preserve samples that share a similarity greater than a predefined threshold X. Next, for each remaining sample, we construct corresponding pseudo-label vector ")")) If sample is assigned a single pseudo-label then o." is one hot vector, otherwise v.()) is multi pseudo-label vector where values for the assigned pseudo-labels are $;,6 if si 2 À and otherwise. Multi pseudo-label vectors are then normalized to obtain probability vectors. Due to label filtering and issue (b), different viewpoints may now disagree on cluster assignments.
computing the view specific cluster centers in the embedded space (e) = 1 h
Multi-view agreements We now establish an agreement between views for cluster assignments. For each i E T, if a sample is assigned to at least one pseudo-label in more than one view, it will be
kept only the views form agreement expressed by argmax(p; (")) argmax (w)),, and otherwise discarded. If sample is assigned in one view only, it will be retained for training since the cluster head optimization performed for each view separately, as described below.
View specific probabilities Since our optimization is performed on each view, we compute view- specific probabilities matrices p(e) by feeding H(() (v) to G.
(r) and their corresponding pseudo-labels g)" (0) The probabilities for remaining samples in T, P.
used to train the model. We denote (v)) = {(x,, g ( hee as the set of samples with corresponding pseudo-labels in view
19

---

Published as a conference paper at ICLR 2025
D.2 EXAMPLE
H
H
Fusion
Guster Head
(a)
(b)
Fitme Unillrity
(c)
Mutklee
agrammnnt
(d)
Figure 5: Illustration of our pseudo-labeling scheme.
Figure S
illustrates our pseudo-labeling scheme. As shown in part (a), the embeddings H (t) and ( first fused and fed into the clustering head G predict clusters probabilities 9(E, W.H
P. Next, (b), the top B (in this case, 4) probabilities are selected for each of the two clusters: Tk = {ilk E argtopk( P. c,k, B), where T Us Tx. and the centers of the clusters
calculated by:
(1) 1/B TET h (w) Note that the samples hgg, hy and h h have been assigned to more than one pseudo-label. In the next step, see part (c), we filter samples if their cosine similarity from the
cluster center $ h! hg /l (*) ||] does not surpass predefined threshold A. Here for view v (left) sample hg was filtered from the blue cluster and samples h h2 were filtered only from the red cluster. For view (right) sample h was filtered from the red cluster and sample hy was filtered, but only from the blue cluster. Next, we achieve multi-view agreement by filtering out h2 and hg since they have different labels. Despite that h h and h hī do have an agreement, he and h are retained since the cross-entropy optimization is performed on each view separately. They will later be retained for within-cluster permutation only if the pseudo- agrees with argmax (Ps) and argmax(p,).
(v)
E EXPERIMENTS
E. IMPLEMENTATION DETAILS
We implement our model using Py PyTorch, and the code is available for public use All experiments were conducted using an Nvidia A100 GPU server with Intel(R) Xeon(R) Gold 6338 CPU @ 2.00GHz. The training is done with Adam optimizer with learning rate 10 4 and its additional default parameters in Pytorch.
To improve convergence stability, we add decoder modules defined for each view that reconstruct the original samples and are optimized jointly with the main model by adding mean squared error objective in addition to Lowr-
E.2 HYPERPARAMETERS TUNING
To tune the parameters for each dataset, utilize the Silhouette Coefficient Rousseeuw 19887 as an unsupervised metric for cluster separability. The Silhouette Coefficient is calculated using each sample's 's mean within-cluster distance and the mean nearest-cluster distance. The distances are calculated on the fusion of the learned representations 1 Coefficient after each epoch and pick the configuration that produces the maximal average Silhouette
n. Σ, H((), We calculate the Silhouette
'The code will be released at Githuh.
20

---

Published as a conference paper at ICLR 2025
0.7
0.5
0.4
0.3
0.2
0.1
ACC ARI NMI
Figure 6: Box-plot for measured ACC, ARI, and NMI metries on model trained with different batch sizes on MSRCvI dataset.
Coefficient value across a limited set of options. We present the correlation between the Silhouette Coefficient value and clustering accuracy metrics in Table E.2 on datasets VOC and METABRIC. In addition, set the le for the argtopk function to be the batch size divided by number of clusters. In practice, we observed that for some datasets, the parameter value should be increased due to the filtration we apply pseudo labels. We use a fixed cosine similarity threshold for all datasets 0.5.
Table 9: Silhouette Coefficient for different batch sizes.
Ratch Size
ACC
ART
NMI Silhoouttte Confficient
VCC
256
58.77
43.24
57.19
0.063
360
62.45
52,06
59.3
0.161
60.4
47.33
56.06
0.138
METABRIC
128
45.3
23.7
34,1
0.0396
256
53.9
27.8
37.7
0.0133
360
49.2
23.1
35.3
0.0569
46.4
27.9
38.9
0,1088
E.3 A SENSITIVITY ANALYSIS
In our experiments, we initially set the default = 0.5. For three datasets (Scene15, Reuters, and CCV), we have decided to increase it after observing the convergence of cross-entropy loss. Furthermore, we provide an analysis our chosen X's sensitivity on the Caltech5V dataset in Table E.3 below:
Table 10: Performance metrics (ACC, ARI, NMI) for different values of X.
A
ACC()
(STD)
NMI (STD)
0
80,94
1 4.3
67.12 3.9
71.63 2,8
0.45
82.19
+ 2.5
69.08 0.9
74.57 2.9
0.5
82,81
5.1
69.53 6.7
74.03 4.6
0.55
82.81
1 5.1
69.53 6.7
74.03 4.6
0.85
76.56 + 5.6
60.16 6.7
66.64 5.6
This analysis shows that thresholding the pseudo-labels could have a positive impact up some value, although a too high threshold may have a negative impact.
In addition, provide sensitivity analysis of the batch size hyperparameter. We train model with batch sizes [128, 256, 512, 1024 on the MSRCv1 dataset, 5 times for each batch size with different random initialization seeds. We present the box plot in Figure E.3
21

---

Published as a conference paper at ICLR 2025
E.4 GRADUAL TRAINING
We train the model by gradually introducing additional loss terms during the training. We allow training the model with additional decoders and reconstruction loss that could enhance model stability between different random initialization seeds. We start with Leorr loss and optionally with reconstruction loss mse- Next, after few epochs, we add cross entropy loss Lee being minimized with predicted with pseudo labels. Finally, we introduce the within-cluster permutations, and the model optimized with all loss terms during the next epochs. order to tune the number of epochs for each step, we start with 100 epochs for the first step, 50 epochs for the second step, and total of 1000 epochs for training in total. During the experiments, we found that some of the datasets could be trained with fewer epochs by analyzing the training loss dynamics.
E.5 NEURAL NETWORKS ARCHITECTRRES
We present in Table E.5 the dimensions of the fully connected non-linear neural networks for each dataset. In case the decoder is applied, its architecture is mirrored version of the encoder. The clustering head accepts the same dimension as the last encoder dimension, and we present only the single hidden dimension use for each dataset.
Table 11: Model architecture for different datasets used in our experiments.
Dataset
J Hidden Dimensions
g Hidden Dimensions
METABRIC
[512, 2048, 128]
[2048]
Reuters
[512, 1024, 10)
[1024]
Caltech101-20
$12,512, 1024, 20]
[1024]
VOC
[512, 512, 2048, 20]
[1024]
Caltech-5V-7
[512, 256, 128]
[1024]
RBGD
[512, 2048, 128]
[1024]
MNIST-USPS
[1024, 512, 128]
[1024]
CCV
[1024, 2048. 128]
[2048]
MSRVCI
[256, 512, 128]
[1024]
Scene
[256, 512, 1024, 2048, 128]
[1024]
F CASE STUDY USING FASHION MNIST
We conduct a controlled experiment using F-MNIST (Xiao et al. 2017). First, we create two coupled views by horizontally splitting the images. CCA is subsequently performed on the multi-view dataset, with different versions of within-cluster sample permutations.
We only permute samples with the same label. Fig. 7 we confirmed our hypothesis that these permutations decrease the mean within-class correlation across views (please refer to the second paragraph in Section for more details).
Superssed
0.80
0.75
0.70
0.65
11%
22%
88%
100%
Percent to Permute
Figure 7: Permutations decrease the mean within-class correlation across views.
22

---

Published as a conference paper at ICLR 2025
G RELATION BETWEEN COPER AND LDA
G.1 REMINDER: LINEAR DISCRIMINANT ANALYSIS (LDA)
For dataset X RN and it's covariance matrix C, denote the within-class covariance matrix as Ce and the between-cluster covariance matrix as Ca:
N MΣ (at
1 = N
C.
Ms) M&)"
Where N₂ are the samples from class a is the i'th sample, and My is the mean. The between-class covariance is:
N N P
and C' = Ce + Ca. The optimization for LDA can be formulated as:
h T Cah
max h=0 h C.h
This could be solved using generalized eigenproblem:
Cah XCch,
Cah = Ah.
G.2 FROM COPER TO LDA
is treated as two different views 0¹ and where each view is comprised of the same samples from f but different, within-class permutations.
To create the views, different permutations IT' of f for 1 = 0, 1,2..... oo are stacked, where the order of stacked permutations is different for the two different views.
Lemma G.1 Applying CCA on 0² and 0² produces the same projection as applying LDA on 0 with the (unknown) labels.
Since both 0¹ and 6 are comprised of the same samples, it follows:
C Ce Cg2,
(4)
and
Cørga Coa
we can use equations in 2.1 to find to solution for CCA:
CT Co. ACCARCCA,
(6)
while from equation
2 we know that the solution for LDA is:
CahLDA = ALDACEhLDA.
We can further plug equation and get:
CahLDA ALDACC Ca)ALDA
M. = ALDA AADA (1+ AADDA)CCaLLDA ALDACHLDA
ALDA ChLDA
C.hLDA= 1 ALDA
ALDA hLDA:
C-lCaliDA = 1 ALDA
Next, if we substitute C-1C, A and ALDA
X" in G.2 get:
T+ALDA
AhLDA X NhLDA:
(7)
23

---

Published as a conference paper at ICLR 2025
In addition if we substitute this in 6 get:
Co Co, AANcCA
AAboCA = A(AhocA)
A(AhccA) - A(X"hLDA)
ADA = ("(ALDDA)
X (AhLDA) ('(AALDA))
X (A"ALDA)) = ACCAhсCA.
Since the canonical correlation coefficients are the square root of the eigenvalue obtained from the generalized eigenvalue problem, it follows that hLDA = hCCA-
G.3 LDA APPROXIMATION
First, to simplify notation, we denoted and h the LDA representation and representation, which is based pseudo-labels that are potentially incorrect. In the previous sections we saw that C and Čo_ used compute h in equation Hence, to assess this equivalence, we draw attention to potential errors in estimating Co from equation where Co Co. Co..
Let Ng be the estimated samples for each class k. and jig it's estimated mean. N₂ are the indices of samples from class & not included in N₁. N₂ are samples not class k, which are incorrectly included in Nr. and Ñú are samples in N₂ which are correctly included, the corresponding class for samples in N₂ are assumed to be known for this analysis.
For Co Ni.. EK (x ax)) ) Incomplete inclusion of all between-class samples may cause error. As this depends on the batch size and the pseudo-labels batch size. We denote this type of error as:
E
(af (an Pxx)
k=
ki
In addition, false pscudo-labels cause samples from diffcrent classes to be permuted together. We denote this type of error
2
ΣΣ R= VEN JeNz
Σ (x (x)(2) - Pr)
where ş the true class of samples j.
For Ćo. N Ek Nin k= (Tx Errors in estimating Ax. denoted by Apt: 14 fa can be further propagated to the third type of error, denoted as:
N
E
A
Nmb
Together all three types of errors can be expressed as E = E 2 E Taking these errors into account,
treat them perturbation. Formally:
Će Ce. + E
Će. Co. E +
C. Co + E.
Since the latent dimension is significantly smaller than the input dimension, it is plausible to assume that Öo is invertible. In addition, express the first order approximation of the inverse as:
Co¹ (Co + EE- Co C EC
24

---

Published as a conference paper at ICLR 2025
This is accurate in terms of order ||E|| Stewart & Sun 1990)
This means that the estimated matrix A from Eq.
7 À can be written as:
A= Cg Co.
C = =Co - BC (Co, E
(Cø Co (CC"")(C, E (C,1 Cē EC, (C Co. + +E
(Co Co EC (Ca Co. + Ed) C Co Co E Ca- CC-
C Co. + Co EC, Co. Co E Co EC, E
C Co Cp C EC, C. - C Coat
C EC Co. C E Co EC E
I- C- EI C Co. + Co Co + CT'E" E -C E
and A can be expressed as:
A Co
C Co. Cp 'Co Coc),
(Co Co.) Co Co Co.Co
T Co C Co. = Co.
Now we can estimate the perturbation from A, denoted as D:
D = A A,
A A-Á
- = =I - T - C Ca. EC Co. + C - C EC, E²),
(I CΦ Co.) - -(I CI- EI C Co Co. +C E Co. CØ E Cp EC E
TT Cp EC Co. - C + +C + EC,
This perturbation can used as a bound for the approximated eigenvalues according Stewart Sun 1990):
Ad < 1DD 1...n.
H COMPARING THE BEST RESULTS
In the Table H. we provide the results obtained by our model in the same setup as in previous works where the best accuracy values are selected from multiple runs with random initialization.
Table Best clustering results. Our model (COPER) is compared against two recent deep MVC
models.
Method/Datase
METABRIC
Renters
Callech1or-20
VOC
Callechs..7
RBGD
MINISTUSPS
CCV
MIRVC!
Scene
ACC
CVCL
52.48
55.41
35.43
39.82
89.14
33.21
99.68
26.27
93.33
42.72
ICMVC
33.96
39.18
27.11
38.80
61.36
31.92
99.12
22.04
77.14
12.36
RMCNC
34.58
44.63
43.17
42.49
58.36
34.37
83.00
23,90
35.24
41.32
ART
DSSIVC
22.38
26.72
35.31
5804
85.10
20.01
76.86
6.50
64.05
23.48
JCMVC
12:24
18.00
26.12
44.42
17.37
98.72
6.23
55.88
26.47
RMCNC
11.90
14.06
30.85
21.61
39.89
16.46
66.41
9.98
12.62
25.46
COPER
33.01
30.59
$1.55
58.28
76.26
36.68
99.87
12:58
86.74
26.40
DSMVC
30.07
33.55
62.34
69.11
87.11
41.63
83.54
17.44
68.43
41.93
CVCL
39.16
42.22
56.95
33.29
79.76
27.68
99.05
22.52
86.38
42.64
JCMVC
44.42
49.7
COPER
40.54
37:42
56.15
60.82
79.17
37.78
99.81
26.93
88.71
40.53
25

---

end for Published as a conference paper at ICLR 2025
I BROADER IMPACT
Our work presents a new deep learning-based solution for multi-view clustering, along with a theoretical foundation for its performance. This research can positively impact various domains by providing researchers and practitioners with a versatile data analysis tool for use across heterogeneous datasets, thereby facilitating advancements in knowledge discovery and decision- maniing processes. However, the proposed method has ethical implications and potential societal consequences that must be considered. It is crucial to pay attention to bias and fairness to prevent the amplification biases across modalities. Transparency and explainability are essential to ensure user understanding and mitigate the perceived black-box nature of deep learning models.
J COPER ALGORITHM
The COPER Algorithm processes multi-view data using deep canonically correlated encoders and clusters the data. It iterates through several epochs, performing different tasks based on the current epoch. The following is a detailed explanation of the algorithm:
Algorithm COPER
Inputs: Multi- View Data: == (X E RRdxxN] Ju=1: Deep Canonically Correlated Encoders: {F(()) Ju=1
Cluster Head: G, Number clusters K, Number Epochs Nepochs: Epoch to Start Pscudo-Labeling and
Permutations Natert-
Outputs: Cluster Assignments y for each instance tuple (m!) (1) 2 2
(2)
(nc) 1,...
1: for epoch in Nepochs do
2: for cach pair of views e, w0 do
3: Train encoders F=()) and fF(m) with data X and (w) by minimizing Lcorr
4: end for
5: epoch > Nstart then
6:
Predict cluster probabilities G(E, v W.
7:
Select top B samples per cluster Te = (il E P.. B)} Filter samples in Tr using cosine similarity À and multi-view agreement
8:
9:
Create view specific pseudo-lahels x²() ((x, 9 heT for cach view v do Minimize L.. with pseudo-labels for view
10:
11:
12:
end for
Create permuted multi-view data p for cach pair views v, w do
Train encoders Fr()) and FF()) with data x (*) and X (()) by minimizing Cearr
15:
16:
end for
17:
end
26


Reviews:
[{"review_id": "77PG0dQE1K", "reviewer": "Reviewer_aFs1", "summary": "The proposed approach involves generating meaningful fused representations using a novel permutation-based canonical correlation objective. Cluster assignments are learned by identifying consistent pseudo-labels across multiple views.", "questions": "See above.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The proposed approach involves a novel permutation-based canonical correlation objective. Simulanteously, the authors provide a theoretical analysis showing how the learned embeddings approximate those obtained by supervised linear discriminant analysis (LDA).", "weaknesses": "1）In Line 41, multi-view clustering holds immense potential in various applications, however, the methods mentioned are not updated to recent literature. Please update these references to ensure your work is current.\n\n2）The selected comparison methods are not enough. It is recommended to add some comparison methods, otherwise this may have a negative impact on the reliability of the experimental results. \n\n3）Some of the selected comparison methods are for incomplete multi-view data, and some are for noise correspondence. These methods have special properties and are not recommended as comparison methods.\n\n4）Considering that the proposed method is derived from the CCA objective, it is recommended to classify the compared methods into CCA-derived methods and other non-CCA-derived methods. This can directly demonstrate the effectiveness of the new elements introduced by the proposed method compared to previous similar frameworks and its competitiveness compared to other MVC paradigms.\n\n5) Check for all possible errors in the statement, e.g. the missing serial number for Figure in Line 244,“using using within-cluster permutations”in Line 292.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "c6ZKAdo2wo", "reviewer": "Reviewer_gjrx", "summary": "This paper proposes a deep learning model for multi-view clustering framework, namely, COPER. The proposed model integrates clustering and representation tasks into an end-to-end framework, eliminating the need for a separate clustering step. Extensive experimental evaluation across various benchmark datasets validates the efficiency of the proposed algorithm.", "questions": "As mentioned above.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.\tThis paper is well organized, and the proposed methodology is enlightening.\n2.\tThe motivation behind the paper is clear, and the theoretical analysis is complete.\n3.\tThe comparison experiments are comprehensive, encompassing datasets of varying sizes and multiple types of baseline methods.", "weaknesses": "1.\tUnlike general methods, the proposed approach generates pseudo-labels for each view to enable self-supervised learning. However, in multi-view clustering, aligning the labels across views can pose challenges that may impact subsequent self-supervised learning.\n2.\tAlthough the paper includes theoretical analysis, the proposed method offers limited innovation. The correlation maximization loss has already been proposed, and generating pseudo-labels by estimating the probability matrix is also a common approach.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "64UE5lLqtX", "reviewer": "Reviewer_bbo9", "summary": "In literature, most of the current multi-view clustering methods are limited to specific domains or rely on a sub-optimal and computationally intensive two-stage process of representation learning and clustering. To address this issue, the authors propose an end-to-end deep learning-based multi-view clustering framework which is validated to be effective in experiments.", "questions": "Please see Weakness", "limitations": "", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The theory of LDA is analyzed. \n2. The proposed method is validated to be effective in experiments.", "weaknesses": "1. The authors say that most of existing multi-view clustering methods are composed of two-stage process of representation learning and clustering, therefore they propose an end-to-end method. However, the authors also claim that a few end-to-end methods are proposed in literature. So, the motivation should be further clarified. \n\n2. The paper [1] is an classical end-to-end multi-view clustering method and should be compared or discussed.\n\n3. The parameter study should be included.\n\n[1] Liu J, et al. One-pass Multi-view Clustering for Large-scale Data, ICCV 2021.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "0SflkczVsE", "reviewer": "Reviewer_Yjr1", "summary": "This paper proposed an end-to-end MVC method that leveraged CCA-based correlation maximization and self-supervised pseudo-labels to learn multi-view representations and clusters jointly. In the proposed method, the key components are the Sample Selection and Label Refinement-Agreement of multi-view pseudo-labelling, which follows some technologies of semi-supervised learning and yields a simple unsupervised MVC architecture. Then, the authors present experimental and theoretical results to support the effectiveness of their method.", "questions": "1. Please see above weakness. It is encouraged to use a uniform model architecture to test the clustering performance of the proposed method on different datasets, for a fair comparison and availability.\n\n2. It is encouraged to compare some latest self-supervised deep MVC approaches, e.g., On the effects of self-supervision and contrastive alignment in deep multi-view clustering [CVPR 2023], Investigating and Mitigating the Side Effects of Noisy Views for Self-Supervised Clustering Algorithms in Practical Multi-View Scenarios [CVPR 2024]...", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This paper presents a deep MVC method which has the advantages of simpleness and end-to-end. The method leverages a cross-entropy loss and a CCA-like maximization loss to train the deep model. It transfers the two-step training schedule in previous methods into step-by-step one, among which it conducts sample selection by high confidence and label correction by multi-view agreement.\n\n2. The paper is well-written and easy to follow, which introduces theoretical proofs and visualization to support its method.\n\n3. The illustration of cluster permutations is interesting, and it can enhance the embeddings learned by CCA (verified by ablation study).", "weaknesses": "I have the following concern and hope they are useful for improving this manuscript:\n\nAs for unsupervised clustering task, the robust model is needed when it processes different datasets in practical scenarios. However, we can observe that the proposed method is sensitive to model architecture settings (Table 7&9). For different datasets, the proposed method has different settings of model architecture and batch size. Moreover, in D.4 GRADUAL TRAINING, the training epochs in each step are determined by human. Since we have no labelled data to tune the model settings in practical applications, the proposed might have limited practical application value.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Spotlight)