Generalizable and Robust Spectral Method for Multi-view Representation Learning
Authors: Amitai Yacobi
Ofir Lindenbaum
Uri Shaham
Venue: Accepted by TMLR ()

Abstract: Multi-view representation learning (MvRL) has garnered substantial attention in recent years, driven by the increasing demand for applications that can effectively process and analyze data from multiple sources. In this context, graph Laplacian-based MvRL methods have demonstrated remarkable success in representing multi-view data. However, these methods often struggle with generalization to new data and face challenges with scalability. Moreover, in many practical scenarios, multi-view data is contaminated by noise or outliers. In such cases, modern deep-learning-based MvRL approaches that rely on alignment or contrastive objectives present degraded performance in downstream tasks, as they may impose incorrect consistency between clear and corrupted data sources. We introduce *SpecRaGE*, a novel fusion-based framework that integrates the strengths of graph Laplacian methods with the power of deep learning to overcome these challenges. SpecRage uses neural networks to learn parametric mapping that approximates a joint diagonalization of graph Laplacians. This solution bypasses the need for alignment while enabling generalizable and scalable learning of informative and meaningful representations. Moreover, it incorporates a meta-learning fusion module that dynamically adapts to data quality, ensuring robustness against outliers and noisy views. Our extensive experiments demonstrate that SpecRaGE outperforms state-of-the-art methods, particularly in scenarios with data contamination, paving the way for more reliable and efficient multi-view learning. Our code will be made publicly available upon acceptance.

Published in Transactions on Machine Learning Research (04/2025)
Generalizable and Robust Spectral Method for Multi-view Representation Learning
Amitai Yacobi
amitaiyocooii@gmall.cm
Deportment of Computer Science
Bar-Hlan University
Ofir Lindenbaum
Rirdeendenaamm @biu.ac.i
Faculty Engincering
Bar-Hlan University
Uri Shaham
wri.hahamm@biu
Department of Computer Science
Bar-Nla, University
Reviewed on OpenReview: /ttps://openneen opwrrevee......ee.oo net/ forum? X6IY04Aku)
Abstract
Multi-view representation learning (MvRL) has garnered substantial attention in recent years, driven by the increasing demand for applications that can effectively process and analyze data from multiple sources. In this context, graph Taplacian-based MvRL methods have demonstrated remarkable success in representing multi-view data. However, these methods often struggle with generalization to new data and face challenges with scalability. Moreover, in many practical scenarios, multi-view data contaminated by noise or outliers. In such cases, modern doep-learning-basee MvRL approaches that rely on alignment or contrastive objectives present degraded performance in downstream lasks, as they may impose incorrect consistency between clear and corrupted data sources. We introduce Sp novel fusion- based framework that integrates the strengths of graph Laplacian methods with the power of deep learning to overcome these challenges. SpecRage uses neural networks learn parametric mapping that approximates a joint diagonalization of graph Laplacians. This solution bypasses the need for alignment while enabling generalizable and scalable learning of informative and meaningful representations. Moreover, incorporates fusion module that dynamically adapts data quality, ensuring robustness against outliers and noisy views. Our extensive experiments demonstrate that SpecRaGE outperforms state-of-the-art methods, particularly in scenarios with data contamination, paving the way for more reliable and efficient multi-view learning. Our code available at: httpss //github. com/ swwsahhhallaaaaa -aaahan-1bb/SpecRaaa
1 Introduction
Multi-view representation learning (MvRL) has become a crucial paradigm in recent years. Its primary objective is integrate data from multiple sources into unified representation, which can be used for various tasks such clustering and classification. The demand for MvRL methods has surged more applications require analyzing objects or phenomena from diverse perspectives. For instance, streaming platforms rely on the fusion of visual, audio, and textual features to enhance content recommendations, while healthcare systems combine genetic data, imaging, and clinical records to provide a comprehensive view of patient. health.
Graph Laplacian methods have demonstrated notable effectiveness in representing both single- view and multi-view data (Ng et al. 2001 Belkin & Niyogi 2003 Coifman & Lafon 2006a Cai et. al. 2011 Kumar

---

Published in Transactions on Machine Learning Research (04/2025)
et al. 2011 Eynard et al., 2012 2015
This success largely stems from the ability of Laplacian eigenvectors
to preserve similarity and capture the underlying cluster structure of the data (Ng et al. 2001 Belkin &
Niyogi 2003). Moreover, by focusing on the eigenvectors corresponding to the smallest eigenvalues, one can
uncover the intrinsic low-dimensional manifold structure, as demonstrated in Laplacian eigenmaps (Belkin &
Niyogi 2003).
However, in the multi-view setting, traditional graph Laplacian based methods face two major limitations: Generalizabllity the ability to map new test points into the representation space after the training set has been processed (i.e,, out-of-sample extension); Scalability the capability to process large datasets within a practical time frame efficiently. Current graph Laplacian-based MvRL approaches often fail to address these two challenges (see discussion in Section 2 These limitations in generalizability and scalability hinder the full potential of graph Laplacians for representing multi-view data in world applications.
Another critical challenge in the multi-view setting is handling noise or outliers, which are pervasive in real-world applications. For example, in autonomous driving systems that integrate data from multiple sensors, adverse weather conditions can lead to unreliable noisy data from certain sensors. Most modern
deep learning- based MvRL approaches (Andrew et al. 2013 Huang et al. 2019 2021 Federici et al. 2020
Trosten et al. 2021 Xu et al. 2022 Trosten et al., 2023
Wang et al. 2023 Yan et al. 2023) rely on aligmment
or contrastive objectives to enforce consistency between view-specific representations. These methods assume
that all views of the same sample are of similar quality, making consistency enforcement reasonable under ideal conditions. However, this assumption breaks down in the presence of contaminated data, where enforcing consistency between clear and degraded views can lead to erroneous representations. These limitations underscore the urgent need for robust MvRL methods capable of handling real-world data imperfections.
To address the challenges of Generalizabilily and Scalability in graph Laplacian-based methods, as well as ensuring Robustness in the presence of contaminated views, we propose SpecRoGE, a novel fusion-based MvRL framework that combines the strengths of graph Laplacians with the power of deep learning. At its core, SpecRaGE provides a deep-learning solution to the approximate joint diagonalization of Laplacians problem, by exlending SpectvalNet (Shaham et al. 2018) to multi-view settings. The use of joint. diagonalization is our key strategy for avoiding the need for alignment between views (see Section 4.1 for further discussion). SpecRaGE is inherently scalable, as it is trained on mini-batches in stochastic manner, allowing it to efficiently procoss large datasets. Furthermore, the resulting model provides a parametric map that approximates the leading joint eigenvectors of the multi-view graph Laplacians. This parametric map enables efficient application to new data, addressing the generalizability challenge of traditional graph Laplacian mothods.
Moreover, SpecRaGE incorporates a flexible fusion technique that overcomes the rigid limitations of traditional aligument- methods when dealing with contaminated multi-view data. Specifically, SpeeRaGE introduces fusion mechanism, that generates sample-specific weight vectors, allowing the model to dynamically down- weight anomalous or noisy views.
Our extensive experiments demonstrate that SpecRaGE not only achioves statc-of-tho-art performance on standard multi-view benchmarks but also significantly outperforms existing methods when dealing with outliers and noisy views.
The main contributions of this work are: (1) We introduce a gencralizable and scalable, graph Laplacian- MyRL framework that. exlends the power of spectral methods to large-scale multi-view data. (2) We propose adaptive fusion module that dynamically weights different views, providing robust performance in the face of data contamination. (3) We present extensive experimental results demonstrating SpecRaGE'ss superior performance across various benchmarks, particularly in scenarios involving outliers and noisy views.
2 Related Work
Graph Laplacian-based MvRL Methods. Various graph Laplacian-based methods (also known as multi-view spectral representation learning methods) have been proposed to extract compact and informative representations from multi-view data (Cai et al.] 2011 Kumar al. 2011 Eynard et al. 2012 2015 Li let al. 2015 Lindenbaum et al.) 2020 Yang et al. 2023). These approaches typically aim to learn a fused representation based multiple graph Laplacians, one for each view. While these methods produce meaningful
2

---

Published in Transactions on Machine Learning Research (04/2025)
representations, they often face challenges with generalizability (out-of-sample extension), requiring the recomputation of graph Laplacians to embed new, unseen data into the fused representation space. Some approaches address this issue through out-of-sample extension methods, such as the Nystrom extension [Nyström 1930) or Geometric Harmonics (Coifman & Lafon 2006bb. However, these techniques were originally developed for single-view data and provide only local extensions, functioning effectively only near existing training points. Additionally, they are computationally intensive and memory-demanding as they require calculating distances between each new test point and all training points. Furthermore, many graph Laplacian- based MvRL approaches face significant scalability challenges due to their computational complexity and memory requirements. These methods typically rely on iterative optimization procedures that demand substantial computations and storage for large Laplacians (Cai et al. 2011 Kumar et al. 2011 Eynard et al._ 2012: 2015; Zhan el. al.) 2017 2018). To address scalability, some approaches, such as (Li et al.) 2015 Tao et al. 2024), introduce large-scale techniques for multi-view spectral clustering. (Li et al, 2015) improves efficiency by constructing a bipartite graph to approximate the full affinity matrix, roducing the computational complexity of spectral clustering. [Tao et al. (2024) enhances scalability by leveraging tensor factorization to decompose multi-view data into compact shared latent space, avoiding the need for explicit pairwise similarity computations. However, the method presented in (Tao et al., 2024 still requires quadratic time complexity, and both methods are specifically designed for clustering rather than learning representations. As a result, they cannot, directly map new, unseen points into the learned representation space. These limitations in generalizability and scalability hinder the practical use of graph Laplacians for multi-view representation learning in weal-world applications.
Deep-learning based MvRL Methods. Modern deep-learning based MvRL methods attempt to design a loss function that is useful for extracting meaningful representations from multi-view data. One category of methods includes deep extensions of the Canonical Correlation Analysis (CCA) algorithm, such as DCCA,
DCCAE, and 4-DDCA [Hotelling] 1936: Andrew et al., 2013 Wang et al., 2015 Lindenbaum et al. 2021 p. These methods utilize deep networks to learn non-linear mapping for two views, maximizing their correlations
Another set of algorithms relies on information-theory-bassse
metrics (Federici et al. 2020 Lin et al.
2022),
aiming to maximize the mutual information between views while minimizing redundant information unique to
each view. Contrastive learning methods (Trosten et al. 2021 Xu et al., 2022: Yan et al. 2023 Wang el al. 2023 Guo et al. 2024) represent another group of deep learning approaches, utilizing a contrastive alignment objective to achieve vicw-specific alignment. Deep learning techniques have also been applied to address the multi-view spectral clustering problem (Huang el al., 2019 2021). These methods are closely related to our work, as they also extend SpectralNet to multi-view settings by incorporating alignment objectives between the spectral embeddings from each view. Despite their promising performance, these approaches rely on some form of alignment between the view-specific reprosentations, making them vulnerable to data contamination. Their underlying assumptions regarding data quality and view consistency may struggle in the presence of noise, outliers, or asymmetric corruption across views, as demonstrated in Section 5.3
Robustness in Multi-view Learning. Recent works have explored various approaches to handling corruption in multi-view settings. Geng et al. (2021): Zhang et al. (2023) introduce an uncertainty estimation mechanism to model reliability across views through probabiliste ii modeling. However, these methods are specifically designed for Gaussian noise, making strong assumptions about the nature of uncertainty that may not hold in real-world scenarios with complex corruption patterns. Xu et al. (2020) approaches robustness through modal regression and low-rank matrix recovery, formulating multi-viers learning as a structured matrix recovery problem. While this method can handle outliers, it requires solving complex optimization problems involving alternate minimization with multiple terms. Other approaches address different aspects of multi-view noise that are less relevant to our work. Sun et al. (0244 focuses on scenarios where sample correspondences between views are noisy, meaning that some instances are incorrectly paired across views. While their probabilistic model corrects such correspondences during clustering, this method addresses a fundamentally different problem from our goal of handling corrupted observations within properly aligned views. Similarly, Xu et al. (2024) primarily deals with label noise rather than feature-level corruption, introducing a trustworthiness mechanism to reweight samples based on their likelihood of having incorrect labels. This focus on supervision noise makes it orthogonal to our problem of handling corruption in the views themselves.
3

---

Published in Transactions on Machine Learning Research (04/2025)
3 Preliminaries
3.1 Graph Laplacian and SpectralNet
Graph Laplacian. Given dataset of n points 21, ,2, an affinity matrix W is an n x symmetric matrix with non-negative entries, where Wi, represents the similarity between D; and zy. The unnormalized graph Laplacian is defined as D W where correspond to the degrees of the points zi. 1...., The cigenvectors corresponding to the smallest eigenvalues of the Laplacian provide valuable low-dimensional representations, capturing structural information like relationships and similarities between data points. These eigenvectors are widely used in applications such as spectral clustcring [gg et al. 2001). dimensionality reduction (Belkin & Niyogi 2003). , graph partitioning
diagonal matrix in which the element Di ΣØ= W.,
Karypis & Kumar 1998), and image segmentation (Shi & Malik 2000 Melas-Kyriazi et al. 2022
SpectralNet. SpectralNeet (Shaham et al. 2018 is a deep-learning model that effectively maps single-view data to the approximate eigenvectors huge single-view datasets since the loss is amenable parallelized training. Furthermore, the model can be easily and accurately used to generalize the representation to unseen test data. To learn the eigenvectors, SpectralNet minimizes the following Rayleigh-quotient loss: Tr (Y (YTLY) s.t. Y = where Y E [gn* is the network's output and L is the graph Laplacian.
Laplacian. This enables the performance of spectral clustering on
3.2 Joint Diagonalization
Definition 1. A set of diagonalizable malrices A ...... A(V is said to be sineallaneously dingonalizable if there erists single invertible matriz U such that for all 1 S < V, U-1 is a diagonal metria.
Intuitively, joint diogonalization (or simultaneous diagonalization) seeks to find a common basis by which all matrices could be represented in a diagonal form.
However, exact joint diagonalization can be achieved if and only if A A A commute. Nevertheless, when commutativity cannot be guaranteed, optimizing joint diagonality criterion and approximating the solution remains possible. This defines the approzimate joint diagonalization problem.
Approximate Joint Diagonalization of Laplacians. In terms of graph Laplacians, for set of graph Laplacians ( L(2) _(2),... ..... LIV) the objective of joint diagonalization is to find a set of orthogonal eigenvectors
U such that for each Laplacian I(("), the matrix U L is diagonal and contains the eigenvalues of I(()) on the diagonal. has been demonstrated (Eynard et al, 2012 2015), that it is possible to find an approximate joint diagonalization of Laplacians by solving the following objective:
min Tr LU), s.t. UT-=1, UERaxk
(1)
where L represents a form of average of the graph Laplacians. This average can be computed, for example, as where (*) represents the contribution of the e-th view. In Section 4.5 we provide more detailed discussion of this objective and its connection approximate joint diagonalization.
weighted arithmetic mean: I. = -y a(v)
4 SpecRaGE
Problem Statement. Let X = (X(() Raxd.jy J be multi-view dataset where is the number of samples, V is the number of views, and d, denotes the dimensionality of samples within the v-th view. MyRL aims to leverage the multi-view information to learn a high-quality unified representation that facilitates downstream tasks such as clustering, manifold learning, or classification.
In this scction, we introduce SpecRaGE, our MyRL framework designed tackle the challenges of general- izability and scalability in graph Laplacian methods, and robustness to contaminated views. We begin by outlining the rationale behind method. Next, we detail how SpecRaGE efficiently learns the approximate
4

---

Published in Transactions on Machine Learning Research (04/2025)
(X
(()
Weighting Model
y
(1)
1. (P)
x( mxd
g6
(3) Cm
(P)
Y
Fusion
YR
YM
OR-Decomposition
v m
x E R maddy
(V) 99
Co
Σ
W 000
l1
mmV
W
W (v)
(v)
(iii) Unified representation: Joint eigenvectors
Modality-ppecific representations
(ii) Fusion
Figure 1: SpecRaGE architecture: Given imput of V batches, each containing samples, V view-specific representations are generated by V corresponding neural networks. Subsequently, the concatenated input is passed through the fusion model, which computes the weights for performing the weighted sum fusion. The resulting fused representation undergoes QR decomposition to enforce orthogonality. Finally, the loss function in Eq. 4 is computed, and the weights of all networks are updated via the gradients.
joint eigenvectors, facilitating generalization new data. Finally, we explore the fusion module, which enhances robustness in the presence of contaminated views. The overall framework of method is illustrated in Fig.
4.1 Rationale
To effectively harness the strengths of graph Laplacians while ensuring robustness to data contamination, we aim to: (1) adopt less contaminatiion-sensiiive stralegy that eliminates the need for aligument objectives, (2) implement a dynamic weighting mechanism to reduce the influence of low-quality views, and (3) provide a scalable and generalizable graph Laplacian solution.
SpecRaGE addresses the first requirement. through joint diagonalization of Taplacians, which fuses information from all views into a unified representation that approximates the joint eigenvectors. This fusion process involves averaging Laplacians across views, which naturally reduces the impact of corrupted data, as noise in one view typically independent of others. This makes SpecRaGE less sensitive to contamination because the impact of noisy views is diluted when averaged with cleaner views. In contrast, alignment-based approaches directly enforce consistency between representations of all views, which can lead to clean views being compromised when forced to align with contaminated ones.
To meet the second requirement, SpecRaGE incorporates dynamic fusion module that adjusts the weighting of views based on their quality, effectively reducing the influence of noisy or outlier views in sample- manner. This module prioritizes cleaner views while down-weighting those affected by noise or corruption, ensuring robust and accurate fused representations.
Overall, SpecRaGE parametric (hence generalizable) map, trained in a stochastic fashion (hence scalable), and therefore meets the third requirement. well.
5

---

Published in Transactions on Machine Learning Research (04/2025)
4.2
Generalizable and Scalable Approximation of Joint Eigenvectors
Let Fo: Rd R&d x... x Rav +R R* be a parametric mapping (i.e,, neural network model) that transforms multi-view imput into the corresponding coordinates in a fused representation. That is, for a multi-view input & = (2(") (2) ".......!!)) Fo produces ys = Fg (f;) where yi E IR* represenis coordinates in the fused reprosentation.
Given a batch of m multi-view data points, let W() denote the m x m affinity matrix for the w-th view, where Wofe) represents an affinity measure between (v) and afth, We propose the following loss:
V V
Lo
lus y3
(2)
by
1J==1
This loss encourages points with high affinity (as measured by the affinity matrices) to be close in the fused representation space. However, it is evident that this loss can be minimized trivially by mapping all points to the same output, i.e, Fo(G,) = yo for all i. To avoid this trivial solution, an orthogonality constraint is added to the outputs:
YT Lacks
where Y is an m x k matrix of outputs, with its i-th row corresponding to yt- To satisly the orthogonality constraint, we follow the same technique from (Shaham et al.) 2018) and construct an orthogonalization layer that computes the QR decomposition specifically, let Y denote the m x matrix obtained from the fusion step; the weights of this layer are defined to be the matrix R-1 from the QR decomposition of Y. The final orthogonal output then Y = -==RR! YR-!. For further details about the training process with the orthogonalization layer, see Appendix F
the network output and returns the orthogonal Q matrix. More
With some mathematical transitions (see Appendix Cb, the loss in Eq. can be written in the following Rayleigh quotient form:
V 7,00))
2 Tr m²V
Lg
3
v=1
where ((v) is the m x m graph Laplacian of the e-th view. One can observe that this loss is exactly the arithmetic mean version of the objective in Eq. [ In Section 4.5 we further explore the connection between this loss function and the approximate joint diagonalization Laplacians problem.
The choice of affinity measure plays a crucial role determining the quality of the generated representations. An appropriate affinity measure can enhance the ability of the model to capture the underlying relationships within the data, while an inadequate one may lead to poor representation quality and misinterpretation of the data structure. In Section 4.4 we provide discussion of the technique we employed to construct the affinity matrices.
Generalizability and Scalability. Once the framework is trained, it provides a mapping function Fo that transforms each multi-view sample directly into its coordinates in the final unified representation, facilitating efficient generalization for new samples from the same distribution. Notably, all our experiments were conducted using test sets, which illustrates the method's generalizability. Additionally, the stochastic mini-batch training in SpecRaGE avoids computing the full Laplacian eigenvectors, enabling scalability for large datasets. For example, SpecRaGE processed the 1-million-sample InfiniteMNIST dataset (see Section 5.1 in about 15 minutes on a MacBook with M1 CPU, whereas traditional graph Laplacian methods faced out-of-memory errors or much longer runtimes. The overall running time complexity of SpecRaGE is O(n(k² + mV)), where is the output dimension, V is the number of views, and m is the batch size. Since k, V, and m are typically much smaller than n and independent of n, our method exhibits near-lincar time complexity. In Appendix D.3 we compare the runtime of our method against several existing graph Laplacian-based approaches. Tn Appendix F. we present. the full time complexity analysis.
6

---

Published in Transactions on Machine Learning Research (04/2025)
4.3 Robust Weighting Fusion Model
View-specific Representations. As described above, the loss functions in Eq. 2 and Eq. Opperate a fused representation Y E _=* derived from a mini-batch of m multi-view samples. To construct this fused representation, we first need to generate intermediate representations for each individual view. The primary goal of these representations is to embed numeric and categorical features into a common space and ensure that all views are of the same size. To extract the view-specific representations, we introduce an individual IR* (e.g,, an MLP), for each view. Specifically, given a multi-view input (2) ....,,.("). the intermediate representations are 34 (2) (V) ,.. where (e) = -90 22
neural network, Pg (1) RRd, +
i 2- [ 2
(e)
For batch of m multi-view samples, we obtain V matrices Y y(2) ,,,, where y(v) is an m X matrix of the outputs, and its i-th row corresponds to y,
Y()) into unified representation Y presents significant challenge. A key difficulty in this process arises from the potential presence of contaminated samples in some views, such as outliers or noisy data. In such cases, it is desirable to give less weight to the low-quality views in order to obtain a more accurate representation. Specifically, we need an approach that can evaluate the quality of each view directly from the data and determine its degree of contribution accordingly.
Merging the view-specifie representations
Dynamic Weighting Model for Fusion. To dynamically weight views based on their quality, we introduce another neural network model (specifically, an MLP) that takes a multi-viow sample d; as input and prodicts a weights vector a E Av, where Av fa E RVV : (v()) 2 0, for = I,.... V, ΣV a( = 1} is the V dimensional probability simplex. Each entry of a; indicates the quality of the corresponding view for instance 4. The process works as follows: First, the view-specific representations y," y/2), .....yy"" (V) are obtained from the multi-view sample 4.. Then, & is concatenated and passed through the weighting model, generating a weights vector og, Finally, the fused representation is computed as y/1 = = (() " "y
To provide meaningful feedback to the weighting model, we integrate the view-specific contributions directly into the similarity matrices in Eq. g Specifically, for a batch of size m, weight vectors Q1, 01,02...... ............ Om are first generated using the weighting model, where a.") represents the reliability of sample in the w-th view. These weights are incorporated by modifying each entry in the similarity matrix: W:(() := w This multiplication adjusts the strength of pairwise relationships based on the reliability of both samples involved. The resulting loss function is:
ig aa(") -aj
(0) (v)
V
La
W ( 13:
-y113.
(4)
T
2= 11j=1
This formulation allows the weighting model to dynamically adjust the contribution of each view. By multiplying wr(e) with af") and af the model effectively reduces the influence of noisy or corrupted samples, ensuring that their similarities to other samples are dowi-weighted while preserving the structure of reliable ones. If the weighting model assigned high weights to contaminated views, the loss would increase, noisy samples would distort the similarity structure in the corresponding W(v) and lead to suboptimal representations, reinforcing the need for correctly learned weights. A complete algorithm summarizing our method can be found in Alg.
To demonstrate the capability of our fusion mechanism to identify observations of low quality and downscale their importance in the overall objective, we conducted an experiment. with the seikit-learn 2D Blobs datase D We created two views, where the second view is random rotation of the first, and randomly injected outlicrs independently into each view, comprising 20% of the samples. After training SpecRaGE on this dataset, we analyzed the weight distributions of clean and contaminated samples in a test set. As illustrated in Fig. 2 our method effectively distinguishes between clean and contaminated samples through their assigned weights. Clean samples consistently receive weights centered around 0.5, indicating balanced and reliable information.
httee://ecikt---ammrrrrrrtttaabe htw.://////aaallaaaaaaaaaaaaaaaa http://aaaaaara.ddatttttttee datasets. make
hww.saaaaaaabaaaabbbaaaaea www.b.bbbb........... .htll
7

---

Published in Transactions on Machine Learning Research (04/2025)
Distribution Weights
70
tcmmal
Density
40-
20
0
Weights
Figure 2: Distribution of weights assigned by our weighting model on the scikit-learn 2D Blobs datasct. Clean samples (blue) receive moderate weights centered around 0.5, while outliers (red), are assigned very low weights, effectively reducing their influence on the learning process.
In contrast, outlier samples, are assigned significantly lower weights (close to 0), effectively reducing their influence on the learned representations. This clear separation in weight distributions demonstrates the model's ability to automatically identify and downweight potentially corrupted samples. Additionally, Section demonstrates that this approach not only achieves state-of-the-arl performance on standard multi-view benchmarks but also significantly outperforms existing methods in handling outliers and noisy views.
5.3
4.4
Self- sedf-super Affinity Learning.
To perform approximate joint diagonalization of Laplacians, it is necessary to know how to construct affinity matrix for each view. A widely used approach is to use Gaussian kernel with specific scale parameter a > O. For instance, for the v-th view, the affinity matrix is defined as follows:
- ) [1x 1100-19010 20
(e) is one of the / nearest neighbors of a("). F3
Wl() bg
exp
(5)
0,
otherwise.
However, Euclidean distance may offer limited measure of similarity, particularly for high-ddmennsionll data [eyer et al. 1999 Aggarwal et al. 2001). Therefore, instead of directly computing the Euclidean distance between di and Ky, we choose to train a SiameseNet (Koch al. 2015) for each view v, denoted as h (e) Osiamesse
This replaces the Euclidean distance ||=;5) 1 in Eq. 5 with ||
112, where 2.(0) = =h Maaammsse (0) (0)
(1)
Training a SiameseNet typically requires positive and negative pairs. When labels are provided, pairs from the same class considered positive, while those from different classes negative. In unsupervised setup, we determine positive pairs based on small Euclidean distances between points xx(e) and a and negative pairs otherwise. Specifically, positive pairs are formed from the nearest neighbors of each point, while negative pairs are generated from points that not included in the nearest neighbors set. The parameters and are syper-parameet
discussed in Appendix F
The training of the SiameseNets serves as pre-processing step. Once the SiamescNets are trained, we use them to construct batch affinity matrices for each view during the training. As in (Shaham et al. 2018), we empirically found that using SiameseNet for the affinities improves the quality of the representations.
4.5 Theoretical Analysis
In this section, we provide additional theoretical insight into the relationship between our loss function Eq. 3 and the approximate joint diagonalization problem. We begin by establishing fundamental property of jointly diagonalizable matrices:
8

---

Published in Transactions on Machine Learning Research (04/2025)
Proposition 1. Let I. = - 1 be sum of these matrices. Then, the joint eigenvectors of I,(,).... ,.... I,(V) are also the
be n x n real, symmetric, and patrwise commutting matrices. Let
eigenvectors of L.
The proof of this proposition can be found in Appendix
B
Due to the uniqueness of the eigendecomposition of L, these joint eigenvectors, are the only eigenvectors of L (up to sign). Therefore, the joint cigenvoctors of L(4,, ( ,..,,... L(V) can be recovered by analyzing the eigendecomposstionn of their sum, L. Building on this result, we can now characterize the optimal solution for our loss functions:
Corollary 1. When joint eigenvectors exist, the minimizer of the loss functions in Eq. n consists of the joint eigenvectors (or an orthogonal rotation of them) of L(), L (.8),..., LVV corresponding to the smallest eigenvalues of -= LV-
This corollary follows from both Proposition [ and classical Rayleigh-quotient optimization. Classical Rayleigh-quotient optimization tells us that minimizing Eq. 3 is equivalent to finding the eigenvectors of L corresponding to its smallest eigenvalues. Proposition the joint eigenvectors of the matrices L((1, [,(2),... ...., L(V) Consequently, the minimizers of the loss functions are intrinsically tied to the joint eigenvectors, thereby highlighting the connection between our optimization problem and the joint diagonalization of these matrices.
| further establishes that these eigenvectors are
In practical scenarios, the graph Laplacians L(", _ L(V) rarely satisfy the commutativity condition perfectly due to contaminations and inherent differences between modalities. In such cases, our loss functions serve to find approximate joint eigenvectors, providing an approach to handling real-world data where exact joint diagonalization may not be possible. Appendix D.4 shows that our method effectively approximates the eigenvectors of L and performs approximate joint diagonalization.
5
Experiments
5.1 Experimental Settings
Datasets. We assess the performance of SpecRaGE using five well-studied multi-view datasets. Our selection prioritizes datasets that exhibit diversity in the types and number of views, as well as the number of classes, as depicted in Table 3 in Appendix A The datasets are listed as follows: (1) BDGP (Cai et al. 2012h contains 2500 images of Drosophila embryos divided into five categories with two extracted features. One view has 1750-dimensional visual fcatures, and the other view has 79-dimensional textual features. (2) Reuters is a multilingual datasel comprising more than 11,000 articles across six categories and five languages: English, French, German, Italian, and Spanish. We used a subset of this datasot containing 18,758 samples for our analysis.
(3) Caltech20 is a subset of 2386 examples derived from the object recognition dataset Fei et al. 2004), which comprises 20 classes viewed from six different perspectives. The dataset encompasses various features such Gabor features, wavelet moments, CENTRIST features, histogram of oriented gradients, GIST features, and local binary patterns. (4) Handwritten (Asuncion & Newman 2007 contains 2,000 digital images of handwrilten numerals ranging from O to 9. This dataset employs (wo lypes of descriptors: a -40-dimmnsional pixel average within 2 x 3 windows, and 216-dimensional profile correlations method, serving as two distinct vicwpoints. (5) InfinitcMNIST is large-scale variant of MNIST (LeCun) et al. 1998 with 1 million samples. The first view contains the original images, while the second adds Gaussian noise (a 0.2) to images from the same class, as in (Trssten et al. 2023)- Created similarly to a this dataset applies small random affine transformations to expand the original data. More details about the different datasets can be found in Appendix A
We compared the performance of SpecRaGE with seven multi-view representation learning methods including two classic deep methods (DCCA (Andrew el. al. 2013) and DCCAE (Wang et al. 2015 and five state-of-the-art methods (MvSCN (Huang et al.) 2019). MIB (Federici et al. 2020), Multi VAE (Xu
Baselines.
httpps:/www..c www.nto.ttttoo..ooooooo wtw.roottt..chhttiiineeaaaaaatat

---

Published in Transactions on Machine Learning Research (04/2025)
Table 1: Clustering results of all methods on five datascts. The best result in each row is shown in bold and the second-best is underlined.
Datasets
Metrics
DCCA
DCCAE
MvSCN
MIB
Muld--VA
AECoKM
MetaViewer
SpecRaGE
ACC
75.8 +1.31
80.1 +1.50
81.9 +-4.3
86.9 +0.75
53.8 -6.11
76.6 +5.33
90.4 -0.43
97.6 +0.53
BDGP
NMI
67.9 +1.15
73.2 #1.04
76.9
+4.15
80.9 #0.91
29.9 =6.44
08.7 +5.89
86.3 =1.0
93.4 +0.83
ARI
49.2
+1.31
55.1
+1.02
70.3
+5.42
74.1 11.8
28.9
=4.22
54.7 +6.34
89.3 1121
94.1 +1.29
ACC
47.9
+0.93
42.0 +1.23
49.5
+2250
49.5 +1.10
36.6
2.34
26.5 +0.52
47.8 -0.08
56.7 +3.22
Renters
NMI
26.6
+1.32
20.3 +1.13
27.6
+:1.2
28.4 +1.81
21.1
2220
16.4 +1.54
24.2 0006
38.3 4:2.45
ARI
12.7
1:144
8.5 1:1.06
23.1 4:1.12
24.4 1:123
12.4
22.34
10.1 4:2.54
19.19 40.81
29.6 4:2.64
ACC
39.7
+1.10
36.1 +1.50
42.1 +-2.9
36.1 +1123
32.4
-2.2
42.2 +3.65
45.1 -1.33
50.1 +2.61
Caltech20
NMT
51.2
+1.21
52.4 +1.71
53.7 +-3.49
47.5 +1.55
46.2 -2.8
57.6 +0.18
60.9 -2.15
66.0 +1.72
ART
23.5
1:2.23
25.3 1:1114
29.7
4:3.26
23.2 1.004
24.8 1.77
32.2 1:1.20
35.0
1.24
40.1 4:3.49
ACC
58.3
1:2.10
63.8 1:121
68.3
144.6
67.5
1:2.55
74.5 =3.34
66.4 1:234
86.3 2251
91.9 1:351
Handwritten
NMI
70.2
11.69
75.2
11.34
70.9 42.90
64.7 12.18
70.0 =3.65
70.3 12.70
78.9 1.44
86.5 11.74
ARI
52.2 +1.33
60.8
11.87
59.6
44.25
48.9 +2.56
60.5 =3.15
61.4 +2.70
72.3 22.44
83.0 11.98
ACC
95.6 11000
95.1
11121
99.1
280.8
68.6 13.95
98.1 0050
99.3
10.20
80.0
0.22
90.1 10.27
InfiniteMNIST
NMI
90.0 1.33
88.9 11411
97.7
11.54
67.2 13.15
96.0 11.20
98.1 40.42
72.3
=0.15
97.5 10.54
ARI
90.1 1.33
89.3 11.01
97.9 +0.48
66.9 13,56
96.4 =1.05
98.5
20.20
65.2 -0224
97.9 +0.21
et al. 2021 , AFCoKM (Trosten al. 2023 and MetaViewer (Wang et al. 2023 DCCA and DCCAE are deep extensions of classic correlation analysis. MvSCN learns spectral embeddings for each view while aligning view-specific representations with an MSE objective. MIB uses information theory to separate shared and view-specific information. Multi- VAE disentangles visual representations into view-common and view-peculiar variables. AECoKM combines autoencoders with contrastive learning for view alignment. Meta Viewer learns to fuse representations by observing reconstruction from unified representations to specific views and employs contrastive learning in its improved version. To ensure fairness, we ran each algorithm ten times on the aforementioned datasets using the same backbones, recording the mean and standard deviation of their performance. For clustering, we employed k-means, while Support Vector Machines (SVM) were used for classification. For aligument-based methods, we concatenated the view-specific representations before applying clustering and classification. More details on hyper- hyper-paramene parameters and training are in Appendix F
Evaluation metrics. For clustering, we employed three widely used metrics: Normalized Mutual Infor- mation (NMI), Accuracy (ACC), and Adjusted Rand Index (ARI). For classification, we used Accuracy, Precision, and F-score. Higher values of these metrics indicate superior performance.
5.2 Evaluation of the Learned Representations
Clustering Results. In the clustering experiment, we aimed to evaluate SpecRaGE's ability to capture the underlying cluster structure of the data in comparison to the baseline methods. To achieve this, we applied the k-means algorithm to the final representations learned by each method. Table I summarizes the clustering results across the five datasets. SpecRaGE, achieves top-performing results on most datasets and evaluation metries, with particularly notable improvements on BDGP and Reuters. On these datascts, SpecRaGE outperforms the second-best method by more than 7%. The consistent clustering performance across different metrics (ACC, NMI, and ARI) suggests that SpecRaGE learns representations that preserve both global cluster structure and local neighborhood relationships. Only on InfiniteMNIST, where most methods already perform well due to the dataset's 's relative simplicity, does SpecRaGE perform comparably lo the best baseline. In Figure 3 we provide an additional illustration of the efficacy of SpecRaGE in representing the data and capturing its inherent cluster structure.
Classification Results. In the classification experiment, we aimed to assess the quality of the learned representations by running an SVM on lop of the final embeddings produced by each method. We chose to use SVM because it does not introduce any additional non non-linear transformations to the feature spaces, ensuring that the comparison between different algorithms remains unbiasod. SVM and linear classifiers are widely accepted as standard benchmarks for evaluating learned representations and embeddings (See, for example, (Chen et al. 2020 Federici et al. 2020: Bardes et al. 2021: Wang et al.] 2023
10

---

Published in Transactions on Machine Learning Research (04/2025)
Table 2: Classification results of all methods on five datasets. The best result in each row is shown in bold and the second- best is underlined.
Datasets
Metrics
DOCA
DCCAE
MvSCN
MIB
Muli- VAE
AECoKM
MetaViewer
SpecRaGE
ACC
98. 10 40.81
98.65 #0.26
98.76
#0.10
90.36
11.55
88.87 + 51
98.92
#0.21
98.00 #0.11
99.01 #0.30
BDGP
F-score
98. 10 10.82
98.50
=0.21
98.76
#0.10
90.10
+0.00
88.87
#2.54
95.92
#0.20
98.02 #0.11
99.00
10.23
Precision
10.61
10.30
10.15
1:132
99.01
10.10
99.01
ACC
74.40
4-0.22
74.10
+0087
75.52
4-0.12
71.96
+-1.02
62.06
4-3.40
68.01
58.17 +-010
+1.70
Renters
F-score
74.50
+1.10
74.21
+0.90
75.51
+-0112
70.78
+-0.1
59.60
+3395
65.21
+-0.8
51.19 +0.09
76.60 1.77
Precision
74.72
+0.92
74.35
+1.01
75.53
#0.14
70.78
+1.10
61.31
+3. 22
67.21
+0.85
56. 36.43 #0.12
78.59
1.77
ACC
72.60
+-0.51
72.58
+-0.44
86.54
+-0.35
73.52
+:2.41
87.73
+0.63
93.38
+1.07
92.16 +-0.05
96.42
+-0.22
Caltech20
F-score
10.12 +0050
43.26
=0.81
86.75
+0.37
72.52
+2 10
87.30
+ 91
93.03
0.997
85.72 10.10
96.93
+:0.93
Precision
16.30
10044
60.66
+0.69
85.32
+0333
73.25
+1.93
88. $8.90-1
1.60
11.2 93.09
90.68 00.15
10.90 95.44
ACC
25,25-22 +:091
90.01
+0.45
96.21
+0.21
96.01
+1.04
95.36
+0.95 95.5
96.91
+0.65
97.75 00.21
97.80
1.27
Handwritten
F-scoze
+1.03
96.21
#0.21
11.20
95.36
96.95
73 =0020
97.77
1.27
Precision
89.20
1:0885
90.48
10.56
96.24
#0.21
96.03
+0,90
+0.92 95.38
96.91
#0.34
97. 97.90 10.19
97.80
=1.26
ACC
97.21
1.0.12
97.60
1.0.1
99.12 1.0100
95.52
1.011
98.75
0.002
99.50
1.0.09
95.71 1.0110
99.35
0.04
InfiniteMNIST
F-score
97.21
+-0.14
97.60
+-0.1
99.12
+-0.10
96.51
+-0.11
98.75
+-0.02
99.50
+-0.08
95.71 +00100
99.34
0.04
Precissonn
97.21 +0.12
97.60
#0.41
99.12 10.10
95.56
+0.11
98.78
+0.00 03
99.51
+.009
74 +0.15
99.35
=0.04
(a) Epoch
(b) Epoch
(c) Epoch 20
(d) Epoch
Figure 3: Visualization of the unified representation Y during training on the BDGP dataset.
Table 2 presents the classification results across the five datasets. As expected, all methods perform better in classification compared to clustering, since classification benefits from supervised training with label information, whereas clustering relies on uncovering the underlying data structure without supervision, making it inherently more challenging. Notably, SpecRaGE consistently achieves top performance in most datasets and evaluation metrics, with particularly strong results more complex datasets like Caltech20 and Reuters. Similarly to the clustering results, on InfiniteNISTT, SpecRaGE performs competitively but slightly below the best baseline, suggesting that for simpler, well-structured datasets, multiple methods can achieve near-optimal performance.
Visualization. To further demonstrate the effectiveness of the learned unified representation, we utilize the I-SNE algorithm on the representation obtained from the validation set during the training, As depicted in Fig. 3 our method successfully separates the data into distinct clusters with increasing training epochs.
5.3 Evaluation of Robustness Data Contamination
In real-world scenarios, data contamination is a pervasive challenge, whether the form of noise or anomalous outliers, often introduced by faulty sensors, human errors, external environmental factors. These lypes of contamination severely degrade the performance of multi-view models, making robustness critical capability. To evaluate SpecRaGE's robustness we designed two contamination experiments: one targeting robustness to outliers and the other to Gaussian noise.
the outlier experiment, global anomalies were randomly injected into one view, following the approach from (Han al. (022). For the noise experiment, Gaussian noise with o 1.2 was injected into portion of the samples one randomly selected view. We conducted both experiments at contamination ratios of 10%, 20%, 30,, and 40%, and measured clustering ACC.
11

---

Published in Transactions on Machine Learning Research (04/2025)
Synthetic Outllers
BDGP- Outllers
Handwritten Outllers
-10
-10
Degrodation (%)
Deg Degradation (%)
-00
Dog Dogradation
-30
SpecRaGE
MI8
-1
SpecRaGE
140
SpecRaGE
MIB
Vecay over
W:SCM
MYSSC
Recavfewer
KVSCN
AECORM
DOCA
AECORM
DCCA
-00
AECDON
DOCA
MNAE
DOCAE
MUAE
DCCAE
MADE
DOCAE
40
29
Outliers Level 196)
Outliers Level (9)
Outliers Level (%)
(a) Synthetic Outliers
(b) BDGP Outliers
(c) Handwrilten Outliers
Synthetic Noise
BDGP Noise
Handwritten Noise
-10
Degracatton 1%)
Degracatton 176)
Dograstation
-20
SpecRaGE
500
SpecRaGE
340
SpecTaGt
KID
MattV over
MUSEM
WASCN
Natactnner
KVSCN
ARCaiDe
DCCA
DCCA
ADCHON
DCCA
MWAT
DCCAT
MWAE
DCCAF
MvAr
DOCAF
Noise Level 196)
Noise Level (%)
Noise Level (%6
(d) Synthetic Noise
(e) BDGP- Noise
(f) Handwritten Noise
Figure 4: Clustering performance (ACC) degradation across various datasets under increasing levels of outliers (top) and noise (bottom). Each curve represents a different method, illustrating the relative performance drop (%) as the proportion of outliers noise rises
These tests were performed on two real-world datasets and synthetic 2-view version of the scikit-learn Blobs dataset. We chose to include this simple synthetic 2D data in this experiment, as it allows us to more clearly distinguish the effect of the noise or outliers on the model's performance. In these experiments, we report the relative degradation percentage with respect to the uncontaminated baseline, where no contamination is injected.
As shown in Fig SpecRaGE consistently outperforms other methods in terms of robustness, maintaining stable performance even under high levels of contamination. In both the outlier and noisy view experiments, SpecRaGE demonstrates significantly smaller relative degradation percentages across all datasets, even at the highest contamination ratios. This superior robustness can be attributed to our fusion-based approach, which fundamentally differs from methods like MvSCN, AECoKM, and MetaViewer Viewer that rely on aligument or contrastive objectives. These alignment-based methods attempt to enforce consistency between view-spocific representations, which becomes problematic when one view is contaminated, as they may erroneously align clean data with corrupted information. In contrast, SpecRaGE's fusion module dynamically dowh-weight contaminated views, preventing their corruption from propagating to the final representation.
5.4 Comparison with other Fusion Methods
To evaluate the effectiveness and the relevance of our dynamic fusion module, we conducted a comparison with several popular fusion techniques (Li et al., 2018): Simple Average, Concatenation, Lincar layer, and Cross-view Attention (Wu et al., 2022). Among these, the latter two also learnable fusion approaches. The evaluation was performed with contamination ratios of 10%, 20%, 30%, and 40%, considering both outliers and noise corruption scenarios.
Similar to our robustness analysis in Section 5.3 we measured the relative degradation in clustering performance with respect to an uncontaminated baseline. Figure 5) presents the results of this comparison on the BDGP datasct. In the outlier scenario (Figure 5p), our weighting fusion module demonstrates superior robustness, with performance degrading by only 12% at 40% contamination, while other methods show higher degradation (27.3% for Concat, 23.9% for Simple Average).
12

---

Published in Transactions on Machine Learning Research (04/2025)
Degradation (96)
Degradation (%6
-10
-6
Weighting Model
Weighting Model
-20
Simple Average
Simple Average
Linear layer
Linear layer
-25
Ciows-view Attention
Cross-view Attention
Concal
Concat
Outliers Level (%)
Noise Level
(a) BDGP- Outliers
(b) BDGP Noise
Figure 5: Comparison of clustering performance (ACC) degradation among different fusion methods on the BDGP dataset with increasing outlier and noise ralios.
When examining noise corruption (Figure) p) all methods show better resilience compared to the outlier scenario, but still, our fusion maintains remarkably stable performance with only 1.9% degradation at 40% noise, while other methods show higher vulnerability (14.3% for Concal, 9.9% for Simple Average).
These results highlight the relevance and necessity of our fusion module for maintaining robustness across various contamination scenarios.
6 Conclusion
In this work, we introduced SpecRaGE, a novel fusion-based framework for multi-view spectral representation learning that effectively integrates graph Laplacian methods with deep learning techniques. By efficiently learning parametric map to uncover the approximate joint eigenvectors from diverse graph Laplacians, SpecRaGE addresses the challenges of generalizability and scalability in graph Laplacian methods, enabling it to handle large datasets while generalizing to new samples. Moreover, SpecRaGE employs a dynamic fusion technique that enhances robustness against outliers and noise in contaminated multi-view data. Extensive experiments validate that SpecRaGE achieves state-of-the-art performance on standard multi-view benchmarks and significantly purperfomms existing methods when faced with outliers and corrupted views. These results highlight SpecRaGE's potential transform multi-view learning in practical applications where data quality is often compromised.
Reproducibility Statement. A complete explanation of SpecRaGE's logic is provided in Section 4 and Algorithm 0 All key details for reproducibility, including the training methodology with the orthogonality constraint, network backbones for both the view-specific and weighting fusion networks, hyperparameters, data splits, operating system, and hardware specifications, are available in Appendix Further discussion on the construction of affinity matrices found in Appendix 4.4
Limitations. While our approach enables scalable and generalizable joint eigenvector approximation, it relies sufficiently large mini-batches for effective convergence. As discussed in Appendix F small mini-batches that fail to adequately represent the data distribution can hinder the orthogonalization process, limiting generalization without explicit QR decompasition. Additionally, we observed some sensitivity to high learning rates during the orthogonalization step. Excessively high learning rates can lead to network outputs that contain nearly linearly dependent vectors, causing the QR dccomposition to become numerically unstable or fail. To mitigate these limitations, a relatively low learning rate and large batch size are required ensure stable training.
References
Charu Aggarwal, Alexander Hinneburg, and Daniel A Keim. On the surprising behavior of distance metrics in high dimensional space. In Database Theory- ICDT 2001: 8th International Conference London, UK, January 4-6 2001 Proceedings 8. Springer, 2001.
13

---

Published in Transactions on Machine Learning Research (04/2025)
Galen Andrew, Raman Arora, Jeff Bilmes, and Karen Livescu. Deep canonical correlation analysis. In International conference on machine learning, pp- 1247-1255. PMTRR 2013.
Arthur Asuncion and David Newman. Uci machine learning repository, 2007.
Adrien Bardes, Jean Ponce, and Yann LeCun. Vicreg: Varianceimmarrann-ccvaaaaa regularization for self-supervised learning. arXiv preprint arXim:2 05. 04906, 2021.
Mikhail Belkin and Partha Niyogi. Laplacian eigenmaps for dimensionality reduction and data representation. Neural computation, 15()):1333-1396, 2003.
Kevin Beyer, Jonathan Goldstein, Raghu Ramakrishnan, and Uri Shaft. When is "nearest. neighbor" meaningful? In Database Theory-ICDTT ICDT 7th International Conference Jerusalem, Israel, January 10-12, 1999 Proceedings 7. Springer, 1999.
Xiao Cai, Feiping Nie, Heng Huang, and Farhad Kamangar. Heterogeneous image feature integration via multi-modal spectral clustering. In CVPR 2011, pp- 1977-1984. TEFE, 2011.
Xiao Cai, Hua Wang, Heng Huang, and Chris Ding. Joint stage recognition and anatomical annotation of drosophila gene expression patterns. Bioinformatics, pp. i16 124, 2012.
Jean-François Cardoso and Antoinc Souloumiac. Jacobi angles for simultancous diagonalization. SIAM journal on malrix analysis and applicalions, 17(1):161 -1611611166,,,11 1996.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. simple framework for contrastive learning of visual representations. In International conference on machine learning, pp. 1597-1607, 2020.
Ronald R Coifman and Stéphane Lafon. Diffusion maps. Applied and computational harmonic analysis, 21 (1):5-30, 2006a.
Ronald Coifman and Stéphane Lafon. Geometric harmonics: a novel tool for multiscale out-of-sample extension of empirical functions. Applied and Computational Harmonác Analysis, 2006b.
Davide Eynard, Klaus Glashoff, Michael M Bronstein, and Alexander M Bronstein. Multimodal diffusion geometry by joint diagonalization of laplacians. ar preprint 1rXiv:1209.200 2295, 2012.
Davide Eynard, Artiom Kovnatsky, Michael M Bronstein, Klaus Glashoff, and Alexander M Bronstein. Multimodal manifold analysis by simultaneous diagonalization of laplacians. IEEE transactions on pollern analysis and machine intelligence, (72) -5117, 2015.
Marco Federici, Anjan Dutta, Patrick Forré, Nate Kushman, and Zeynep Akata. Learning robust representa- tions via multi-view information bottleneck. arXiv preprint ar urXiv:2002.07011, 2020.
Li Fei-Fei, Rob Fergus, and Pictro Perona. Learning generative visual models from few training examples: An incremental bayesian approach tested on 101 object categories. In 2004 conference compuler vision and pottern recognition workshop, pp. 178-178. IEEE, 2004.
Yu Geng, Zongbo Han, Changqing Zhang, and Qinghua Hu. Uncertainty-aware multi-view representation learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pp. 7545 7553, 2021.
Ruiming Guo, Mouxing Yang, Yijie Lin, Xi Peng, and Peng Hu. Robust contrastive multi-view clustering against dual noisy correspondence. 37, 2024.
Songqiao Han, Xiyang Hu, Hailiang Huang, Minqi Jiang, and Yue Zhao. Adbench: Anomaly detection benchmark. Advances in Neural Information Processing Systems, 35:32145 42-32159, 2022.
Harold Hotelling. Relations between two sets of variales. Biometrika, 28:321 377, 1936.
Zhenyu Huang, Joey Tianyi Zhou, Xi Peng, Changqing Zhang, Hongyuan Zhu, and Jiancheng Lv. Multi-view spectral clustering network. In IJCAT, pp. 4, 2019.
14

---

Published in Transactions on Machine Learning Research (04/2025)
Zhenyu Huang, Joey Tianyi Zhou, Hongyuan Zhu, Changqing Zhang, Jiancheng Lv, and Xi Peng. Deep spectral representation learning from multi-view data. IEEE Transactions on Image Processing, 30: 5352 5362, 2021.
George Karypis and Vipin Kumar. A fast and high quality multilevel scheme for partitioning irregular graphs. STAM Journal on scientific Computing, 1998.
Gregory Koch, Richard Zemel, Ruslan Salakhutdinov, et al. Siamese neural networks for one-shot image rocognition. In ICML deep learning workshop. Lille, 2015.
Abhishek Kumar, Piyush Rai, and Hal Daume. Co-regularized multi-view spectral clustering. Advances in neural information processing systems, 24, 2011.
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient- based learning applied to document recognition. Proceedings of the IEEE, pp. 2278 2324, 1998.
Yeqing Li, Feiping Nie, Heng Huang, and Junzhou Huang. Large-scale multi-view spectral clustering via bipartite graph. In Proceedings of the AAAI conference on artificial intelligence, 2015.
Yingming Li, Ming Yang, and Zhongfei Zhang. A survey of multi-view representation learning. IEEE Iransactions on knowledge and dala engineering, 31(10):1863 1883, 2018.
Yijic Lin, Yuanbiao Gou, Xiaotian Liu, Jinfeng Bai, Jiancheng Lv, and Xi Peng. Dual contrastive prodiction for incomplete multi-view represenlation learning. IEEE Transuctions on Pallern Analysis and Machine Intelligence, pp. 4447-4461, 2022.
Ofir Lindenbaum, Arie Yeredor, Moshe Salhov, and Amir Averbuch. Multi-view diffusion maps. Information Fusion, 55:127-149, 2020.
Ofir Lindenbaum, Moshe Salhov, Amir Averbuch, and Yuval Kluger. LO-sparse canonical correlation analysis. In International Conference on Learning Representations, 2021.
Luke Melas-Kyriazi, Christian Rupprecht, Tro Laina, and Andrea Vedaldi. Deep spectral methods: A surprisingly strong baseline for unsupervised semantic segmentation and localization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognation, pp. 8364-8375, 2022.
Andrew Ng, Michael Jordan, and Yair Weiss. On spectral clustering: Analysis and an algorithm. Advances in neural information processing systems, 14, 2001.
Evert J Nyström. Über die praktische auflösung von intogralgleichungen mit anwendungen auf randwertauf- gaben. 1930.
Uri Shaham, Kelly Stanton, Henry Li, Boaz Nadler, Ronen Basri, and Yuval Kluger. Spectralnet: Spectral clustering using deep neural networks. Xixiv preprint 1xXi:180100058 2018.
Jianbo Shi and Jitendra Malik. Normalized cuts and image segmentation. IEEE Transaclions on pallern analysis and machine intelligence, 2000.
Yuan Sun, Yang Qin, Yongxiang Li, Dezhong Peng, Xi Peng, and Peng Hu. Robust multi-view clustering with noisy correspondence. JEEE Transactions on Knowledge and Data Engineering, 2024.
Qinghua Tao, Francesco Tonin, Panagiotis Patrinos, and Johan AK Suykens. Tensor-based multi-view spectral clustering via. shared latent space. Information Fusion, pp. 102405, 2024.
Daniel J Trosten, Sigurd Lokse, Robert Jenssen, and Michael Kampffmeyer. Reconsidering representation alignment for multi-view clustering. In Proceedings of the LEEE/CVF conference on computer vision and pattern recognition, pp. 1255-1265, 2021.
Daniel Trosten, Sigurd Løkse, Robert Jenssen, and Michael C Kampffmeyer. On the effects of self-supervision and contrastive aligument in deep multi-view clustering. In Proceedings of the IEEE/CV /CVF Conference on Computer Vision and Patlern Recognition, pp. 23976 23985, 2023.
15

---

Published in Transactions on Machine Learning Research (04/2025)
Ren Wang, Haoliang Sun, Yuling Ma, Xiaoming Xi, and Yilong Yin. Metaviewer: Towards unified amulti-ieww representation. In Proceedings of the IEEE/CVE /VV Conference on Computer Vision and Pattern Recognilion, pp. 11590 11599, 2023.
Weiran Wang, Raman Arora, Karen Livescu, and Jeff Bilmes. On deep multi-view view representation learning. In International conference on machine learning, pp. 1083 1092. PMLR, 2015.
Danyang Wu, Xia Dong, Feiping Nie, Rong Wang. and Xuclong Li. An attention-based framework for multi-view clustering on grassmann manifold. Pallern Recognition, 128:108610, 2022.
Cai Xu, Yilin Zhang, Ziyu Guan, and Wei Zhao. Trusted multi-view learning with label noise. arXiv preprint 240.441111444 2024.
Jiamiao Xu, Fangzhao Wang, Qinmu Peng, Xinge You, Shuo Wang, Xiao Yuan Jing, and CL Philip Chen. Modal-rogrresion--aased structured low-rank matrix recovery for multiview learning. IEEE Transactions on Neural Networks and Learning Systems, 32(3):1204 1216, 2020.
Jie Xu, Yazhou Ren, Huayi Tang, Xiaorong Pu, Xinofeng Zhu, Ming Zeng, and Lifang Hc. Multi-vac: Learning disentangled view-common and view-peculiar visual representations for multi-view clustering. In Proceedings of the IEEE/CY CCF international conference computer vision, pp. 9234-9243, 2021.
Jie Xu, Huayi Tang, Yazhou Ren, Liang Peng, Xiaofeng Zhu, and Lifang He. Multi-level feature learning for contrastive multi-viow clustering. In Proceedings of the IEEE/CVF conference on computer vision and pallern recognition, pp.
2022.
Weiqing Yan, Yuanyang Zhang, Chenlei Lv, Chang Tang, Guanghui Yue, Liang Liao, and Weisi Lin. Gcfagg: Global and cross-view feature aggregation for multi-view clustering. In Proceedings of the TEEE/CVF Conference on Computer Vision and Pottern Recognition, pp. 19863 19872, 2023.
Junchen Yang, Ofir Lindenbaum, Yuval Kluger, and Ariel Jaffe. Multi-modal differentiable unsupervised feature selection. In Uncerlainly in Artificial Intelligence, pp. 2400 2410. PMLR, 2023.
Kun Zhan, Changqing Zhang, Jumpeng Guan, and Junsheng Wang. Graph learning for multiview clustering. IEFE transactions on cyberneties, 48(10):2887 2895, 2017.
Kun Zhan, Feiping Nie, Jing Wang, and Yi Yang. Multiview consensus graph clustering. IEEE Transactions on Image Processing, 2863:122,, 2018.
Qingyang Zhang, Haitao Wu, Changqing Zhang, Qinghua Hu, Huazhu Fu, Joey Tianyi Zhou, and Xi Peng. Provable dynamic fusion for low-quality multimodal data. International conference on machine learning, pp. 41753 41769. PMLR, 2023.
A Datasets Characteristics
In Table 3 we provide additional information regarding the sample size, number of views, and dimensions of the various datasets.
Table 3: Characteristics of the datasets in experiments
Dataset
# Samples
# Classes
# Views
# Dimensioos
BDGP
2500
5
1750; 79
Reuters
18,758
6
21531; 24892; 34251; 15506; 11547
Caltech20
2386
20
48; 40; 254; 1984; 512; 928
Handwritten
2000
10
240; 216
InfIniteMNIST
1,000,000
10
784:784
16

---

Published in Transactions on Machine Learning Research (04/2025)
B Proof of Proposition
Proposition Let L(", L(2), ..... L() be real, symmetric, and pairwise commuting matrices. Let
L = -= ((") be a sum of these matrices. Then, the joint eigenvectors of L(), ..... L(V) are also the eigenvectors of L.
Proof. Since L ,..,,.., L(V are real, symmetric, and pairwise commuting, they can be simultancously diago- nalized by an orthogonal matrix U. This means there exists an orthogonal matrix U such that:
L A for I,...,V,
where A(r) are diagonal matrices containing the eigenvalues of L and the columns of U are the joint eigenvectors of ((), .V.),
Now, consider the sum = EV= L(e). Using the property of linearity and the simultaneous diagonalization, we have:
v D ΣnO
L -U T
E U
Since the of diagonal matrices is also diagonal matrix, U T. is diagonal. This implies that U diagonalizes L. Thus, the columns of U, which are the joint eigenvectors of ((), ...., LVV,, also eigenvectors of L.
U.U
C Equivalence of the loss functions
In Section 4.2 claim that the loss function in Eq.
2
equivalent the joint-diagonalization loss in Eq.
that is:
Σ
Σ
m vsll2
W W
- 2Tr
Y Enoyy
1j=1
Here, we provide proof for this equation.
Proof. As demonstrated in Belkin & Niyogi 2003) the following relation holds:
Welllu 1,113 2 2 TT (Y
Therefore, we can derive the following expression:
Wo llue
2ı = 2 > Tr y ((0
== m==1
e=1
Σ TT 1 = 2Tr
YT (
2T
17

---

Published in Transactions on Machine Learning Research (04/2025)
Table 4: Comparison of clustering performance of SpecRaGE against two additional basclines: concatenated Siamese embeddings (Siamese) and spectral embedding on the average Laplacian (SE on L)
Datasets
Metrics
Siamese
SE L SpecRaGE
ACC
88.20
95.81
97.62
BDGP
NMI
75.42
91.54
93.43
ARI
72.21
91.79
94.10
ACC
46.90
53.40
56.74
Reuters
NMI
29.61
34.43
38.31
ARI
24.32
24.58
29.63
ACC
38.91
46.60
50.11
Caltech20
NMI
61.48
65.37
66.02
ARI
30.10
38.02
40.09
ACC
87.60
88.91
91.89
Handwritten
NM1
82.31
84.30
86.54
ARI
78.89
80.92
83.02
ACC
82.41
98.13
99.09
InfiniteMNSTT
NMI
85.28
96.75
97.49
ARI
78.62
97.23
97.90
Further experiments
D.1 Additional Baselines
To further validate our approach, we introduce two additional baselines that help contextualize the performance of SpecRaGE. The first baseline directly concatenates the embeddings obtained from our pre-trained siamese networks across views. This simple approach helps us evaluate whether the additional complexity of our framework is beneficial compared simply combining the view-specific representations learned by the siamese networks.
The second baseline computes the spectral embedding directly the average Laplacian L using the same affinity matrices obtained from our pre-trained siamese networks. This baseline represents the direct, non- parametric solution to the problem that SpecRaGE' aims to solve in a learnable, generalizable way. For fair comparison, as in SpecRaGE, we first construct view-specific similarity matrices using the siamese networks, compule their corresponding Laplacians, average them to obtain T,, and then compute its first eigenvectors 10 times using the LOBPCG algorithm on different test sets. While this approach provides excellent results as it computes the exact solution, it lacks the ability to generalize to new samples and scale to large datascts key advantages that our parametric approach provides.
Table presents the clustering performance comparison across all datasets. The concatenated siamese embeddings consistently underperform both SpecRaGE and SE on L, demonstrating the value of our spectral embedding approach. As expected, SE on L achieves strong results across all datasets, as it directly computes the exact eigenvectors that our method aims approximate. Remarkably, SpecRaGE still outperforms SE on L across all datasets. This superior performance can be attributed to SpEc adaptive weighting mechanism, which learns to create more reliable fused representations by automatically adjusting the contribution of each view based its quality. The consistent improvement over the direct solution demonstrates that our learnable approach not only successfully approximates the optimal solution but can actually enhance it through intelligent fusion, while providing crucial benefits of generalization and scalability, These results validate both the effectiveness of our approach over simple feature concatenation and show that our parametric approximation not only matches but surpasses the dircct solution, benefiting from robust weighting mechanism while offering the additional advantages of generalization new samples and scalability to larger datasets.
18

---

Published in Transactions on Machine Learning Research (04/2025)
Table 5: Comparison of clustering performance of SpecRaGE with three Info-CCE-based multi-viww methods: CoMVC, AECoKM, and Meta Viewer.
Datasets
Metrics
CoMVC
AECoKM
Meta Viewer
SpecRaGE
ACC
73.96
76.61
90.40
97.62
BDGP
NM1
59.10
68.70
86.32
93.43
ARI
49.63
54.71
89.33
94.10
D.2 Comparison with InfoNCE-based Methods
To strengthen our evaluation of SpecRaGE against widely recognized InfoNCE- multi-view learning methods, we include an additional basclinc: CoMVC Trosten et al. (2021): a contrastive learning method that employs an aligument objective to maximize invariance across view-specific representations. In addition to CoMVC, our baseline set already includes AECoKM and Meta Viewer, both of which incorporate InfoNCE-styi objectives for aligning multi-view representations. Table 5 presents clustering results the BDGP dataset, showing that our proposed SpecRaGE significantly outperforms all CHoNCE-based baselines, including the newly added CoMVC. This highlights the strength of our method, even in comparison to contrastive learning-basee approaches, in learning robust and discrininative multi-view representations.
D.3 Scalability
As described in Sections 12 traditional graph Laplacian-based methods struggle with scalability to large datasets (e.g. Cai et al. ) Kumar et al. (2011): Eynard et al. (2012)). In the following experiment, we demonstrate the scalability our method compared to several well-known traditional approaches in multi-view spectral representation and joint diagonalization of Laplacians. To evaluate scalability, we measure the runtime of each method the InfiniteMNIST dataset with increasing number of points. The methods compare include Jacobi angles for simultaneous diagonalization (JADE) Cardoso & Souloumiac (1966) utilized in Eynard et al. (2012), Co-regularized multi-view spectral clustering (CoRegMVSCC) Kumar et al. (2011), and Sarge-Scale Multi- View Spectral Clustering via Bipartite Graph (LSMVSC) Li et al. (2015). For fairness, the runtime was measured for all methods until convergence (no change clustering ACC) using an
MI MacBook CPU, For our algorithm, convergence typically occurred after 25 epochs.
Algarithm Running Times Comparison
1000
Running limee (seconds)
JADE
CallghhiS
200
50
20
10a
10'
10
Number Data Points
Figure 6: Running time (seconds), as function of the number of points (dataset size).
As depicted in Fig. 6 SpecRaGE is able to efficiently process large datasets containing up to 1, 1,000, 000 data points within a reasonable timeframe. This scalability achieved through the implementation a stochastic training approach on mini-batches. In contrast, JADE terminates after 50, 000 points, CoRegMVSC after 100, 000 points, and LSMVSC after 500, 000 points due memory constraints, demonstrating the superior scalability of our approach.
19

---

Published in Transactions on Machine Learning Research (04/2025)
D.4 Approximation of the Joint Eigenvectors
We begin by demonstrating that our unilied representation approximales the joint eigenvectors. To show this approximation, we compute the Grassmann distance between the subspace of SpecRaGE's output and that of the truc cigenvectors of the matrix L = >. L(t), These true cigenvectors, referred to as the joint eigenvectors, are computed on a validation set after each training epoch. The squared Grassmann distance measures the sum of squared sines of the angles between two k-dimensional subspaces, yielding values within the [0, k] range. To demonstrate this approximation, we used the BDGP dataset and the scikit-learn 2D Blobs datasct, where the second view is generated by applying random rotation transformation to the first view.
2.
0.35
0.30
3.8
825
2.6
0.0
Grossmarn
3.4
8.05
0.00
Feechs
(a) BDGP
(b) Blobs
Figure The Grassmann distance between SpecRaGE's output and the true eigenvectors of the matrix I L= [(*) is plotted as a function of the number of epochs. A smaller distance indicates a more accurate approximation of the joint eigenvectors.
In the BDGP dataset, our model has an output dimension 5, so the maximum distance between the subspace of SpecRaGE's output and that of the joint eigenvectors is also 5. As shown in Fig. 7 this distance significantly decreases early in training and stabilizes around 0.9, indicating that SpecRaGE effectively approximates the joint cigenvectors. For the Blobs dataset, the Grassmann distance quickly approaches zero, indicating that the subspace of SpecRaGE output is extremely close, not identical, to the subspace spanned by the exact true joint eigenvect.rr.
In addition, Figure X demonstrates that our model learns representations Y that approximately jointly diagonalize the graph Laplacians of both views. Specifically, the matrices Y L exhibit a predominantly diagonal structure for both the image view (Figure (a) and the text view (Figure Kp) of the BDGP dataset.
20

---

Published in Transactions on Machine Learning Research (04/2025)
Diegonaiiation Values
Deegenalzztion
3-
(a) Image view Laplacian diagonalization
(b) Text view Laplacian diaaonalliaaiion
Figure 8: Approximate joint diagonalization. Visualization of Y y for both views of the BDGP datasct, showing the approximate diagonal structure achieved by our learned represenlations.
E
Time Complexity Analysis
Algorithm SpecRaGE
1: Input: Multi-view data 1 22 i output dimension k batch size number of epochs T
2: Output: Fused representation ,,...., = = E guxk, the approximate joint eigenvectors.
3: for each epoch t E (1,2..... 7) do for each mini-batch size m do
Obtain view-specific representations y(), ( ..... (())
Generate weights vectors 01, O2,..... Om using the weighting fusion model affinity matrices W((), (2)) ..... w(V) using Eq. 5
Construct
Compute Wx(') == W a aj for v 1,..,
f) Obtain the fused representation Y using the weights vectors
10:
Apply orthogonality constraint using QR decomposition: Y = 8 R
11:
Compute loss in Eq.
12:
end for
13:
end for
14:
Forward propagate $1, ..,, &n and obtain outputs V1 U2.... Un R
The time complexity of the algorithm in can be analyzed as follows:
Given that:
The size of the networks and the number of epochs are constant.
Batch size is m.
The number of views is V.
The total number of samples is
The output dimension is k.
The running time breakdown per batch is:
line 5: O(mV).
21

---

Published in Transactions on Machine Learning Research (04/2025)
line 6: (mmV).
lines 7-8: V).²V).
line 9: (.mV).
line 10: (m.)2).
line 12: Vm.
The Overall complexity:
Per batch: O(3mV + mk? + 2m²
(3mV + mk? + 2m² = V(2( -2))))))) 3))) = O(n(k² + mm)). Per epoch: 0 m
Now since m, V, and & are much smaller than n, this method presents almost linear running time complexity.
F Technical Details
For fairness, we run each of the compared algorithms ten times on the above datasets, recording both the mean and standard deviation of their performance. The same backbones are employed across all methods and datasets. Specifically, we used ML. with hidden layers of sizes 1024, 1024, and 512 for all view-specific networks 90 across all datasets and methods. The weighting fusion model also uses an MLP backbone with three hidden layers, each containing 100 units.
Additionally, for certain datasets such as BDGP, InfiniteMNIST, and Reuters, we initially embedded the features of each view X((1) using pre-trained Autoencoder (AE) with hidden layers of sizes 512, 512, and 2048. This pre-trained AE is used to obtain a lower-dimensional input, typically containing less nuisance information, as shown in (Shaham et al. 2018).
The size the output of our model (k), is determined by the number of categories the data. In the case of coupled view methods like DCCA, DCCAF, and MIB, we present results based on the two best-performing views. For alignment-base Subsequently, clustering and classification tasks are conducted using k-means and SVM classifiers, respectively. Training typically took 35-50 epochs for each dataset.
methods, the final unified representations
produced through concatenation.
Training with Orthogonalization Layer. To train the model with the orthogonalization layer, we adopt technique similar to that used in SpectralNet (Shaham al.) 2018)). This technique employs coordinate descent training approach comprising two main optimization steps: the "orthogonalization step" and the 'gradient step". Each step involves processing different mini-batch.
During the orthogonalization step, we forward a mini-batch through the model and compute the QR decomposition of the fused representation to update the weights the orthogonalization layer. In the gradient step, we pass another mini-batch through the model and use the orthogonalization weights from the preceding orthogonalization step to orthogonal the output. Following this, we compute the loss and update the network weights via backpropagation, while keeping the weights of the orthogonalization layer unchanged.
After training the model, all weights, including those of the orthogonal layer, are fixed. Consistent with obser- vations from SpectralNet, we empirically found that when employing large mini-batches, the orthogonalization layer can also approximately orthogonalize the output of other mini-batches towards the end of training, For instance, in the Blobs dataset, when a random batch of size 1024 passes through the model with the fixed orthogonalization layer (i.c., after training), the resulting output Y, exhibits approximate orthogonality. This is evident in Y where the average deviation of off- diagonal elements from 0 is merely 0.04. In the BDGP dataset, we got an average deviation of ~ 0.05, and in the INfINiteMNSST we got 0.045.
22

---

Published in Transactions on Machine Learning Research (04/2025)
Table 6: Hyper-parameters.
Hyper-params params
BDGP
Reuters
Caltech20
Handwritten
InfiniteMNIST
LR
10
10
10
10
10-3
Batch size
1024
2048
1024
1024
1024
#Neighbors (1)
22
18
18
22
30
scale (7)
Global; Median
Global; Median
Global; Median
Global; Median
Global; Median
Softmax temperture
250
500
250
250
250000
Syper-parameters. In Table 6 we provide a breakdown of the hyperparameters utilized for the various datasets. We ensured that the same hyperparameter tuning procedure was applied to all baselines for a fair comparison. Specifically, for each method, we conducted multiple runs (with the number of runs varying based on the method's specific hyperparameters), each using different hyperparameter configuration. We then selected the setting that yielded the best results across all metrics on validation set. For the unsupervised case, used ACC, NMI, and ARI evaluation metrics, while for classification, relied on ACC, F-score, and precision.
The #Neighbors parameter corresponds to the number of nearest neighbors (l) used for each data point in the Gaussian kernel, as outlined Appendix 4.4 We select the same number of neighbors for each view X((r), The scale parameter (a) is also utilized for the Gaussian kernel and was chosen for each view X as the median of the distances from any point in xx()) to l-nearest neighbors (across all points in the view), resulting in global scale. The temperature parameter is applied in the softmax function used on the weight vector generated by the weighting fusion model. Using temperature greater than in the softmax function helps to smooth the weight distribution across the views, reducing the absolute dominance of any single view. This is particularly useful when we want to ensure that all views contribute to the final representation, even if some might be slightly less informative.
The initial learning rate (LR) was uniformly set to 10-3 for all datasets, with a decay policy in place. This decay policy contingent on monitoring the validation loss. the validation loss fails to improve over epochs, the LR is multiplied by 0.1. Furthermore, if the LR decreases to 10-8, training is terminated. Adam optimizer is used for training.
Data Split. For each dataset, we initially divide it into an 80% training set and 20% testing set. Subsequently, for training, we further divide the training set into 90% training subset and a 10% validation subset.
OS and Hardware. The training procedures were executed on both macOS Sequoia (15.0) and Rocky Linux 9.3, utilizing MacBook M1 processor and Nvidia GPUs including GeForce GTX 1080 Ti and A100 80GB PCle.
23


Reviews:
[{"review_id": "We6ikuaAVT", "reviewer": "Reviewer_aZ3f", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "(+)This is a very interesting job, and in this job, contribution is enough. The writing and organization of the paper are very well-organized. \n\nThere are some minor issues that require the author's explanation and supplementation. \n\n(-) It is suggested that the author discuss the limitations of some methods.\n\n(-) It is recommended that the author show or compare the training time, testing time, and number of parameters with the current well-known methods.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "In this paper, authors  introduced  SpecRaGE, a novel fusion-based framework that integrates the strengths of graph Laplacian methods with the power of deep learning to overcome these challenges. SpecRage uses neural networks to learn parametric mapping that approximates a joint diagonalization of graph Laplacians. This solution bypasses the need for alignment while enabling generalizable and scalable learning of informative and meaningful representations. Moreover, it incorporates a meta-learning fusion module that dynamically adapts to data quality, ensuring robustness against outliers and noisy views. Extensive experiments demonstrate that SpecRaGE outperforms state-of-the-art methods, particularly in scenarios with data contamination, paving the way for more reliable and efficient multi-view learning.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "NaN", "broader_impact_concerns": "NAN", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "B6pC3NJocM", "reviewer": "Reviewer_p3ct", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "S1.\tThe introduction of approximate joint diagonalization of graph Laplacian using deep learning is novel and eliminates the need for explicit alignment between views, reducing sensitivity to noise.\nS2.\tThe meta-learning fusion module effectively re-weights unreliable views based on learned data quality, mitigating the negative impact of contaminated views.\nS3.\tThe paper includes a mathematical formulation connecting its loss function to the joint diagonalization problem, reinforcing its conceptual rigor.\nS4.\tThe method is extensively evaluated on five diverse datasets, demonstrating consistent improvements in clustering and classification performance over strong baselines.\n\nW1.\tThe paper lacks a detailed theoretical justification for choosing Graph Laplacian-based MvRL and its specific formulation, making the methodological foundation less clear.\nW2.\tThe weighting process of the meta-learning fusion module is not well explained, particularly in terms of how it interprets contamination levels across different views.\nW3.\tThe paper could benefit from a more detailed discussion on scenarios where the proposed method may not perform as expected. Understanding the limitations and failure cases can provide more nuanced insights and guide practical implementations.\nW4.\tThe Related Work is not sufficiently comprehensive, as it lacks a detailed discussion on scalability challenges in traditional Laplacian-based methods", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper presents SpecRaGE, a spectral-based multi-view representation learning (MvRL) framework that integrates graph Laplacians with deep learning to improve robustness. By leveraging approximate joint diagonalization via a parametric neural network, SpecRaGE avoids alignment constraints used in contrastive learning. Additionally, a meta-learning fusion module dynamically reweights views based on data quality, enhancing robustness to noise and outliers. Empirical results demonstrate better performance in clustering and classification, particularly under data contamination.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1.\tExpand the Related Work section to explicitly discuss scalability challenges in Laplacian-based methods and compare with large-scale spectral techniques.\n2.\tThe meta-learning module assigns weights to views based on learned reliability, but what are the learned patterns? How is the loss function for this meta-learning module defined? Does the meta-learning network receive supervision signals directly?\n3.\tThe paper claims that SpecRaGE is effective in handling incomplete views, yet no experiments validate this. Since missing data is fundamentally different from noisy or corrupted data, additional empirical validation is necessary to support this claim.\n4.\tWhile each step of SpecRaGE algorithm is well described in Section 4, it would greatly help readers better understand the algorithm if the pseudo-code for SpecRaGE is presented.\n5.\tAlthough the paper compares the proposed method with existing approaches, it could be strengthened by including more recent or alternative state-of-the-art methods that also address similar issues presented in paper.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "lBx8h9IN3R", "reviewer": "Reviewer_txCb", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strength**\nThe method as a multiview extension of SpectralNet is straightforward and worthwhile contribution that incorporates the softmax weighting to provide a instance-wise weighting during learning. \n\nThe papers is easy to read. \n\nResults seem promising with considerable improvements over baselines. \n\nThe work could lead to further developments that incorporate the graph Laplacian formulation as a way to distill multiview similarity information into a single representation. That is, there doesn't seem a reason that the approach couldn't be combined with other self-supervised learning approaches like those based on InfoNCE.\n\n**Weaknesses**\nThe paper adopts many of the approaches from SpectralNet without mention or discussion in the main body (self-supervised pretraining and orthogonalization update). \n\nA key point in SpectralNet and this paper is that 'The choice of affinity measure plays a crucial role in determining the quality of the generated representations.' However, an explanation that this is based on pertaining with self-supervision is left until Appendix D... It is misleading to not mention this in the main body. \n\nThe use of self-supervised pretraining for the affinity matrix, introduces a form of distillation into the multiview learning, which alone is interesting, but it does not seem quite fair without baselines that also use this pretraining step. \n\nThe theoretical analysis does not justify approximation. \n\nHyper-parameter selection is not clear. \n\nSome key points of the data processing and methodology are missing or unclear.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper presents a way to learn multi-view embedding functions based on the concepts of spectral embedding using the graph Laplacian of different views.  This extends the approach of SpectralMap (Shaham et al., ICLR 2018), which uses a mini-batch formulation of the spectral embedding cost and an optimization that applies a QR decomposition of alternate batches to approximately orthogonalize batch embeddings. That is, a neural network is optimized to provide a mapping that gives coordinates for any input point that act as the coordinates of the $k$ eigenvector associated to the smallest eignenvalues of the Laplacian. The proposed multiview case follows SpectralMap in that the affinity matrix underlying the Laplacians uses a nearest-neighbor truncated heat kernel on top of a representation learned for each view through self-supervised SiameseNets (Koch et al., 2015).  To create robustness on the individual data point the multi-view setting, a multilayer perceptron with softmax activation is applied to each instance to generate weights on the simplex for each view. This enables both weighted averages of the view-specific embeddings to create a unified embedding as well as weighted average of the affinity matrices during learning. \n\nThe results use multi view data of varying modality (image, image features, and text for example), quality or difficulty and the proposed outperforms the selected baseline methods. Results also show resilience to the presence of synthetically added noise or outliers to toy and real-world data. \n\nCompared to traditional Laplacian eigenmaps the main contributions are scalability due to mini-batch samples (at the cost of suboptimality), explicit embedding which does not require maintaining entire or quantized representation of training set for out of sample embedding, and a way to downweight outlier views. The first two contributions are shared by SpectralNet, but the last is specific to the multiview case of approximate spectral embedding.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "**Fairness of evaluation.**\nFrom the paper, it seems that all methods use the same architecture for the view processing backbone, but there is a subtle concern I have with the results for the proposed method that exploit self-supervised learning for the affinity matrix discussed in Appendix D. Namely, it should be made clear in the main body that this is possibly an information advantage since \"The training of the SiameseNets serves as a pre-processing step. Once the SiameseNets are trained, we use them to construct batch affinity matrices for each view during the training.\"  This means that the proposed work is **distilling** into a multi-view representation the information learned by the SiameseNet. Why not simple learn on top of the representations learned by the SiameseNet to obtain the multi-view embedding? How would a direct concatenation of the SiameseNet outputs across the views perform on the clustering and classification? \n\nIn any case, this self-supervised information on what is similar in a view that is used by the proposed method, means the proposed method is actually a two-stage approach. In the first stage contrastive learning could be substituted to the SiameseNet. If this single-view representation is then processed further, say with a linear layer and fine-tuning, my guess is that it would outperform training from scratch. Furthermore, if the other baselines started with this pretraining step per view they may all be more competitive. I understand that the backbone doesn't use SiameseNet directly, but if it did, it and other methods would generally be improved. To be fair, the self-supervised single-view representations could be considered additional views for all methods while learning. This way all methods that are multi-view could benefit from this information. It would also be useful for the community to connect the spectral embedding as a form of multi-view knowledge distillation. \n\nFinally, modern self-supervised learning methods that have been published since SiameseNet, namely SimCLR and CLIP. Perhaps if these were substituted for SiameseNet all methods may benefit more through this pertraining stage. This could be mentioned.\n\n*Adjusting the main body text regarding the pretraining is critical, suggested baseline of concatenated Siamese Net is critical (and should be simple to test). Other additions could strengthen.*\n\n**Broad claims about other methods and contaminated data.**\nThe paper claims that contrastive learning make assumptions that all views are of similar quality, and \"this assumption breaks down in the presence of contaminated data, where enforcing consistency between clear and degraded views can lead to erroneous representations\" or later ``Despite their promising performance, these approaches rely on some form of alignment between the view-specific representations, making them vulnerable to data contamination.''. Essentially, this statement claims that contrastive learning are not robust. While the paper's approach of directly learning an instance-wise view weighting is explicitly considering this possibility, there is no ablation study where the 'meta-learning' is removed and SpectralNet is applied to the average affinity. Would the average affinity be more robust? Or is the performance difference due to other differences in methodology?  \n\n\n*An ablation study is necessary to show directly the benefit of 'meta-learning'. Additionally, this ablation should be tested on both the real (no added noise) as well as the synthetically noised data (Figure 4).*\n\n**Cost-benefit of mini-batch.**\nMost of the datasets are reasonably small such that standard Laplacian eigenmaps could be applied to the average affinity across views for a baseline. Getting the bottom $k$ eigenvectors of a few thousand instances is not difficult. *The inclusion of the performance of this explicit solution would provide an additional baseline in the paper.*\n\n**Meta-learning and possibility for alternating explicit solution.**  \nThe choice of 'meta-learning' is a bit misleading, as it would initially imply a different or level of learning. How about 'view consistency weights'? A theoretical discussion of Equation 4 is lacking, in terms of the impacts on the optimization or learning. Firstly, while I understand the alpha values are given by an MLP with a softmax layer, it is never very clear that the parameters are part of  $\\theta$  in Equation 4. Secondly, if the alpha were additional parameters it appears Equation 4 would be a polynomial of factor 4 in terms of them. However, if the goal was to obtain the spectral embedding of the multiview then only the $\\alpha$ in the combined affinity would be needed. That is, it seems completely possible to find a spectral embedding of the reweighted affinity matrix without an explicit network. The idea is to use iterative reweighting where for fixed $\\alpha$ weights it is standard Laplacian eigenmaps. Then, to update the weights it would be a linearly constrained (weights for each point on a simplex) quadratic program which could be optimized via Frank-Wolfe. My thought is that this has been done before... In any case, it would shed light on the proposed network solution which can exploit mini-batch.  *It would seem analyzing this, or looking into possibly similar work would be helpful, especially as the `meta-learning' is the main contribution of this work.*\n\n\n**Hyperparameter selection criteria**\nIt is critical that the paper is more clear about how hyperparameters were chosen and if similar search was performed on baseline. What is the criterion (performance metric) for hyper-parameter tuning? Is the same criterion used for all tasks (unsupervised clustering and classification decoding)? In the unsupervised case, it is not clear what would be available in the real-world.  Do baseline methods select hyper-parameters, and if not wouldn't it be more fair to use the same criterion? \n\n(One minor note about hyper-parameters, is that the temperature for InfiniteMNIST seems to indicate that all views are given equal weight since this high of temperature surely means a uniform weight... This doesn't seem to make much sense as the second view is just a noisy version. )\n\n\n**The theoretical analysis is not very applicable**\nThe theoretical analysis of Proposition 1 is limited to matrices that commute, it does not provide any insight for the approximate case as claimed \"we provide additional theoretical insight into the relationship between our loss function in Eq. 3 and the approximate joint diagonalization problem.\" As stated Corollary 1 is standard Rayleigh-quotient and the eigenvectors of $\\bar{L}$ with smallest eigenvalues will be optimal even if the matrices do not commute, which makes it confusing why it is a corollary of Proposition 1. The paragraph concludes with a reference to Appendix E.2 which plots a Grassmanian distance  to the true eigenvectors on the validation sets of two datasets.  For the first dataset the distance converges to a non-zero value, and for the synthetic data 'Blobs' it converges to zero probably should be log scale. \nThus, there is a lack of theoretical analysis to the actual approximation, which isn't required, but *claims should be tempered*. \n\n**Missing baselines**\nTrosten et al. (2023) actually proposes multiple methods and the one that is reported to work the best overall is AECoDDC. It is not clear why only one of the methods was compared. \n\n**Clarity on underlying data processing**\n Representation and processing of text Reuters and text in BDGP is missing.\n\n**Synthetic noise**\nIn the evaluation of Robustness to data contamination, is the noise and anomalies only added to the image view? It is not clear how they would be added in the text view of BDGP dataset.\n\n**Effect of orthogonalization update**\nOne concern I have stems from the adoption of the orthogonalization algorithm from prior work without inclusion of justification that it is stable and/or optimal. The QR decomposition does not in general give the closest orthogonal representation to the input, so it is not clear how stable this would be across iterations... In the Gram-Schmidt orthogonalization, the first eigenvector which is unmodified in the QR would be stable and ideally this corresponds to the (Fiedler vector), but the subsequent coordinates may note be stable. But the QR in the SpectralNet uses Cholesky decomposition, which may be faster than SVD, but can cause sign flips I believe which would complicate learning batch to batch. \n\n*The paper could benefit form more discussion of the approximate orthogonalization.*\n\n**Blobs**\nI am concerned when the paper uses the synthetic Blobs dataset for justifying multiple aspects: the quality of the joint diagonalization, the robustness to contamination, and the approximate orthogonalization... While it is a toy dataset and offer insight, it doesn't offer confirmation that the observations will hold on real data. The paper should be clear about claims only demonstrated on this data.  \n\n**Minor points:**\n\nSection 3.1 Minor point is that the affinity matrix needs not be semi-definite and generally won't be if the values between data points that aren't nearest neighbor are set to zero. The resulting unnormalized graph Laplacian will be positive semi-definite for any positive symmetric affinity matrix. \n\n\nSmall error in Appendix D after equation 5 when referring to SiameseNet, it says 'for each view $p$ and then uses $v$ to index view as done elsewhere.", "broader_impact_concerns": "none.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This work introduces a framework called SpecRaGE, which integrates the strengths of graph Laplacian methods with the power of deep learning to address the reliability and efficiency challenges in multi-view representation learning. Three reviewers have provided positive feedback regarding its novelty, contributions, mathematical rigor, extensive evaluation, readability, and significant improvements over baseline methods. However, they also raised several concerns, including the lack of detailed justification, explanation, or discussion, the comprehensiveness of the related work, the extensiveness and fairness of the experimental study, and the need for an additional ablation study to better support the role of meta-learning in this work, among others. The authors have provided thoughtful responses to address each of these concerns and have revised the work accordingly. Reviewer txCb, who provided the most detailed review, commented that "the revision improves the paper and addresses many of my concerns."

In their official recommendation, all three reviewers recommended "Leaning Accept," which are consistent with each other. Additionally, Reviewer txCb suggested that more comparisons could be made with respect to InfoNCE-based multi-view learning methods, and that the theoretical analysis could be further refined to directly address the approximation. 

It is evident that certain members of TMLR's audience would find the findings of this paper valuable. Furthermore, the claims made in the submission are overall sufficiently supported by clear, convincing, and accurate evidence. Considering the reviews, the authors' responses, and the final recommendations, the AE recommends "Accept with minor revisions." This will also allow the authors to address the above comments raised by Reviewer txCb.
Decision: Accept (poster)