Revisiting Orbital Minimization Method for Neural Operator Decomposition
Authors: Jongha Jon Ryu
Samuel Zhou
Gregory W. Wornell
Venue: NeurIPS 2025 poster ()
Keywords: spectral decomposition
self-adjoint positive definite operator
structured representation learning
streaming algorithm
Primary area: general_machine_learning
TL;DR: We revisit the orbital minimization method (OMM) from computational chemistry and demonstrate its utility as a simple, principled, and scalable approach for modern machine learning.

Abstract: Spectral decomposition of linear operators plays a central role in many areas of machine learning and scientific computing. Recent work has explored training neural networks to approximate eigenfunctions of such operators, enabling scalable approaches to representation learning, dynamical systems, and partial differential equations (PDEs). In this paper, we revisit a classical optimization framework from the computational physics literature known as the *orbital minimization method* (OMM), originally proposed in the 1990s for solving eigenvalue problems in computational chemistry. We provide a simple linear-algebraic proof of the consistency of the OMM objective, and reveal connections between this method and several ideas that have appeared independently across different domains. Our primary goal is to justify its broader applicability in modern learning pipelines. We adapt this framework to train neural networks to decompose positive semidefinite operators, and demonstrate its practical advantages across a range of benchmark tasks. Our results highlight how revisiting classical numerical methods through the lens of modern theory and computation can provide not only a principled approach for deploying neural networks in numerical simulation, but also effective and scalable tools for machine learning.

Revisiting Orbital Minimization Method for Neural Operator Decomposition
Jon Ryu, Samuel Zhou, Gregory W. Wornell
Department of EECS, MIT, Cambridge, MA 02139, United States (jongha Btongha.samtaii samtzhou, gww.@mit.ed
Abstract
Spectral decomposition of linear operators plays central role in many areas of machine learning and scientific computing. Recent work has explored training neural networks to approximate eigenfunctions of such operators, enabling scalable approaches to representation learning, dynamical systems, and partial differential equations (PDEs). In this paper, we revisit classical optimization framework from the computational physics literature known as the orbital minimization method (OMM), originally proposed the 1990s for solving eigenvalue problems in com- putational chemistry. We provide simple linear-aleebre proof of the consistency the OMM objective, and reveal connections between this method and several ideas that have appeared independently across different domains. Our primary goal is to justify its broader applicability in modern learning pipelines. We adapt this framework to train neural networks to decompose positive semidefinite operators, and demonstrate its practical advantages across a range of benchmark tasks. Our results highlight how revisiting classical numerical methods through the lens of modern theory and computation can provide not only a principled approach for deploying neural networks in numerical simulation, but also effective and scalable tools for machine learning.
Introduction
Spectral decomposition linear operators is a foundational tool in applied mathematics, with far- reaching implications across machine learning (ML) and scientific computing. Eigenfunctions of differential, integral, and graph based operators capture essential geometric and dynamical structures, playing central role in problems ranging from representation learning solving partial differential equations (PDEs). As such, the design of efficient and scalable methods for approximating these eigenfunctions has become core pursuit in modern computational science.
Recent advances have leveraged neural networks to approximate spectral components of operators, offering promising avenues for scaling classical techniques high-dimensional or irregular domains scientific simulation. Notable examples include recent breakthroughs in quantum chemistry based 9 42). Beyond quantum chemistry, neural approaches have also proven effective in diverse domains such as spectral embeddings [15 43 Koopman operator theory for analyzing dynamical systems 32 23 20), and neural solvers for PDEs [26]. For an overview of the broader literature, we refer the reader to 43 Appendix B]. However, many existing methods rely on surrogate losses architectural constraints that lack clear variational foundation, often leading to brittle optimization or limited extensibility.
ansatzes 41
In this work, revisit classical optimization framework from computational quantum chemistry known as the orbital minimization method (OMM), Originally developed in the 1990s for finding the ground state in electronic-structure theory 38 approach to approximating the top eigenspace of a positive-definit operator, without requiring
37 M OMM offers direct and elegant
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

explicit orthonormalization. Despite its heuristic, Nomain--seciinn origins, we derive OMM rigorously from a simple variational principle valid for general positive-semidefine matrices and operators.
We leverage this reinterpretation to construct a modern neural variant of OMM, suitable for decom- posing wide class of linear operators. Our contributions are threefold:
1.
We provide new derivation of the OMM objective with simple linear-algebraic proof for finiteddimensional cases, clarifying its key idea and extending its theoretical foundation. When specialized for streaming PCA, we identify its connection the celebrated Sanger's rule, which is often known as the generalized Hebbian algorithm [44].
2. We adapt this framework to train neural networks that learn eigenspaces of positive-definice operators, eliminating the need for explicit orthogonalization or cigensolvers.
3. We empirically demonstrate the effectiveness of our method across range of tasks, including learning Laplacian- -aplacian-based based representations in reinforcement learning settings, solving PDEs, and wtt--sseerviiidleariiii representation of images and graphs.
We emphasize that properly positioning the OMM within the modern ML literature is of both theoretical and practical significance. Although the resulting formulation may appear natural in hindsight, recent work ML has proposed a variety of alternative, and often considerably more complex, approaches for computing eigenvectors of matrices eigenfunctions of operators. In this work, we draw connections between OMM and several existing lines of research, including streaming PCA 44 from theoretical statisties and computational science, well as recent neural parameterizations used for operator learning in deep learning [43), and independent development of spectral techniques in reinforcement learning literature [51 [3]. These connections provide unified perspective on seemingly disparate methods. Our findings underscore the value of revisiting classical numerical techniques through the lens of modern ML, offering both conceptual clarity and practical advantages for scalable spectral learning.
Notation. For an integer 2 1, [nx] := (1....., n... We use capital sans-serif fonts, such as A and V, to denote matrices. In particular, 12 E R denotes the identity matrix. For square matrix A, tr(A) denotes its trace. For matrix E RR xd2 = [Vi...., Vdgl. where V, E Rd for each j E [dg]. we use the subscript notation Visk [Vi,..., vu] for 1 S k S d2 denote the subset of columns.
2 Methods
In this section, we revisit the orbital minimization method and provide a new variational perspective for the fiite-dimensionall matrix case. We then present its application to an infiniteddimensional problem, i.e., an operator problem.
2.1 Preliminary: Matrix Eigenvalue Problem
Let E R be a real symmetric matrix of interest We wish to compute the top-k eigenvectors of A. By the spectral theorem. A admits an eigenvalue decomposition, i.c., there exists orthogonal eigenbasis W [wi,... wa] E dxdd such that we can write A WAWT Σí >, w/ where A = diag(Ai, Aa), where A > A2 2 2 Ag. Our goal is to find the top-k eigensubspace, i.c., span Wi,..., w. and ideally the top-k eigenvectors w W Wa and corresponding eigenvalues \1 AR. We use a shorthand notation a A == min b) for a, b.be E R.
For a symmetric matrix E RRaxd the most standard variational characterization of the top-k eigensubspace is the multicolumn Rayleigh quotient maximization formulation, which is
max
tr(V
VER4xE.VT
Note that it is natural extension of the maximum Rayleigh quotient characterization of the largest eigenvalue. While can derive its unconstrained version
max
tr( tr(VVT)--VVAAV)
(2)
VER
the inversion (VTV)-1 renders the optimization less practical. For example, if V is not full rank (or nearly degenerate), computing (VTV)-1 may be infeasible or numerically unstable.
We focus real matrices for simplicity, but the technique can be naturally adapted for Hermitian matrices.

---

2.2 Orbital Minimization Method: Matrix Version
We now restrict our focus on a positive-semidefiite (PSD) matrix A E Rdxd The orbital minimiza-
tion method (OMM) [8] 37 34 33 30] aims to solve the following maximization problem:
min VERUX
Lamm(V) (V), where Lomm (V) tr(((22 VTV)VTAV)
(3)
While it an unconstrained objective, it is widely known that its minimizer V satisfies V*(V*)T WI. like i... V corresponds to the top-k eigenvectors up rotation. Interestingly, the OMM objective does not have any spurious local minima, that is, any local minima a global optima; see [30] Theorem 2]. This approach was originally proposed to develop linear-scaling algorithms for electronic-struccure calculations, i.e., algorithms whose complexity scales linearly with the number of electrons in the structure, by avoiding explicit enforcement of the orthogonality constraint VTV =
2.2.1 Existing Derivations
There exist two standard ways motivate the OMM objective [T]] The first is Lagrange- multiplier approach (38). The Lagrangian of the constrained optimization formulation in Eq. ( is
C)V, tr(VTAV) tr(E(VTV lk)), where E E IR kx denotes the Lagrange multiplier. From the KKT conditions, the optimal Lagrange multiplier is E = VTAV, and the Lagrangian E*)
yields the OMM objective.
Another argument is based on the finite-order approximation of the Neumann series expansion of the multicolumn Rayliegh quotient tr((VTV)-IVTA)) VVVV) (TO). Assuming that the spectrum of VTV is bounded by 1, we have the Neumann series expansion of the inverse matrix (VTV)-1 =
(Ix - (Is VTV))-
EE= (1g VTV)*. we consider the first-order approximation, then (VTV)-+ z la - (Ix - VTV) = 21 - VTV, which yields the OMM objective:
tr((VTV)- -
-VTTV AV) tr((21, VTV)VTAV) Lomm (V).
Interestingly, in their original paper, Mauri al. 334 showed that higher-order extension following the same idea possible. For any integer p 2 L, define Qp := Σ - VTV)', which can be understood the order (2p- 1)-th order approximation of (VTV) -I Then, the global minimizers of
2p-1
omm (V) VTAVV tr (1- (IVVVV))VTAV).
(4)
which we will refer to by the OMM-p objective, also span the same top-k eigensubspace. The original argument prove this fact in B4 is domain-specifer and rather obscure. Below, we provide purely
algear-algebraic and simple proof for the consistency of the OMM-p objective.
2.2.2 New Derivation with Simple Proof
To introduce a more intuitive way to derive the general OMM objective without resorting to the Neumann series expansion, we start from noting that we can rewrite the OMM objective function as Lomm (V) = =2r((22,; VTV)VTAV) = tr((1a VVT) tr(A).
Hence, Eq. is equivalent to minimizing tr((la VVT)²A). With this reformulation, we can extend
the optimization problem in Eq. (3) to a higher order as
(
2p (E-ry
(3))
2p (VTV)--1 (TTVV)
C(p) (V) = tr((la VVT) tr(A)
(5)
for integer p 1. While it looks different from the expression in Eq. we can show that they exactly same, i.e., <(e) (V) = č(p) (V): we include its proof in Appendix A for completeness. As
stated below, the global minima of the OMM-p objective characterizes the desired top-k cigensubspace for any p 2 1. Here we outline the main idca, leaving the rigorous proof to Appendix A
Theorem (Consistency of the OMM-p objective). Let S d denote the rank of a PSD matrix with EVD A Ei= A.w.w. Then,
kAr Σ
min (()) (V)
Ai.
VERaX
In particular, its minimizer V* satisfies Viskar (Visker)" W1 WT
3

---

Proof sketch. The proof crucially relies on Eq. ( which shows that the objective depends on V only through the PSD matrix VT E Rdxd of rank at most k. Hence, we can reparameterize the optimization problem with a matrix P e d such that PTP and diagonal matrix E E T with nonnegative entries, by considering the reduced SVD of VVT = PΣPT. We can then show that tr((la A) 2
Ei=kt1 X; and the equality is achieved if and only if VVT = WWT.
2.2.3 Discussions
Can We Apply OMM to Non-PSD Matrices? Consider a rank-1 1 case with p = 1, i.e., Comm (v) = tr((1 - vV A)) tr()). Suppose that A has negative eigenvalue X with normalized eigenvector W.. If we restrict V = cWj for some e E IR, it is easy to show that we can simplify the objective as Com (CW)) = (1- 2)2, - Aj. Since X₃ < O, the objective will diverge to negative infinity when e² - co. This explains why the OMM cannot be applied to non-PSD symmetric matrices. If lower bound on the smallest eigenvalue is known a priori, i.e., Ad 2 -K, we can shift the spectrum by considering the positive semidefinite matrix A + kla, to which the OMM can then be applied.
Equivalence Regularization and Spectrum Shift. The OMM implicitly enforces orthogonality without requiring any explicit regularization. Nevertheless, additional regularization term that promotes the orthonormality of V. i.c., VTV = 1g. we adopt 2 to the OMM-1 objective, it is straightforward to verify that this is equivalent to applying the 0MM-1 to the spectrum-shifted matrix + kla. This reveals peculiar property of the OMM-1: unlike standard regularization techniques that modify the optimal solution to enhance stability, the OMM-1 with this form of regularization preserves the global optima.
might be tempted to introduce an
squared-Frobenius Frobenius penalty <|| |TV - Ixlli
Connection to PCA. For a PSD matrix A E Rgd consider a random vector x E Rd with mean zero and covariance Ex(x) [xxT] = A. Then, we can express tr( (l,, VVT)RA)= = [xxx)]]x VVTx [2]
This is comparable to the standard characterization of PCA:
min VERaXE
Epp1I V(VTV)-
(6)
V E TRixk were orthogonal, then VVTx is the best projection of x onto the column subspace of V, and the OMM objective could be interpreted as the corresponding approximation error. It is worth emphasizing that, even without explicitly enforcing the orthogonality constraint on V E dx the global optima of the OMM remain unchanged, in sharp contrast the common understanding that the whitening operation (VTV)-1 in Eq. (6) is required to characterize the top-k eigensubspace.
2.2.4 Nesting for Learning Ordered Eigenvectors
To learn the ordered eigenvectors, we can apply the idea of nesting in 43 which was originally
proposed for their low-rank approximation (LoRA) objective; see Section 2.3. for the comparison of
the LoRA and OMM. The same logic applies to the OMM. There exist two versions. joint nesting and sequential nesting, which we explain below.
The joint nesting aims to minimize a single objective C6p) (V; := Ei=l Cg (V1s) for choice of positive weights a1,... ag > O, aiming to solve the OMM problem for Vix for each i [k)] When p
>
use the same masking technique of (43), without needing to compute
the objective going over a for loop over E [k.]. Similar to [4] Theorem 3.3], we have:
Theorem 2. Let V* E dR be a global minimizer of c(p) CV: For any positive weights a E R if the top-(k + 1) eigenvalues are all distinct, V* = W.
The proof straightforward: if the objective can have the minimum possible value if and only clmi (V1x) is minimized by satisfying V1.. VT. WT:: for each i E [k), which is equivalent
to V; w; for each 1 Per the suggestion in we use the uniform weighting a = ( ))
throughout.
Here, the strict spectral gap is assumed for simplicity. With degenerate eigenvalues, the columns correspond-
ing to the degenerate part of an optimal V* will be orthonormal eigenbasis of the degenerate eigensubspace.

---

The idea of sequential nesting is to iteratively update v,, using its gradient of Comm (V1.), i.e.,
V cmm (Vis) = - Vi.
+ A(la V1. T.Vvi.
as if V V [vi....., vi-1] already converged to the top-(i - 1) eigensubspace, for each i [A]. The intuition for convergence is based on an inductive argument.
We refer to the nested variants of OMM collectively as NestedOMM, and denote them by OMMjnt jnt and OMMyeq for brevity. We note that the 2MM for inite-dimensional matrices was proposed as the triangularized orthogomalization method (OFM) in the numerical linear algebra literature [T)]
2.2.5 Connection to Sanger's Algorithm for Streaming PCA
There is rather separate line of literature on streaming PCA with long history, which aims to solve the essentially same eigenproblem for a finite-dimensionll case. The goal of the streaming PCA problem is to find the leading eigenvectors of the covariance matrix A t Ep(x) [xx]] of zero-mean random variable x ~ p(x). In the streaming case, are restricted receive minibatch of independent and identically distributed (i.i.d.) samples {x1 :..., B at each time I, and can (xg )T)
JT. In this setup, Sanger 44] proposed the following iterative update procedure, often called Sanger's 's rule or generalized Hebbian algorithm [1] 2] at each time step t, for each I.....
thus only compute the empirical estimate A, :- _े Eb= X (t)
(1+1)
v(()) + + v (2)
We remark that the OMM gradient in Eq. 77) is derived from the well-defined objective and can be viewed as the symmetrized version of the Sanger update term, while the update term itself cannot be viewed as a gradient of any function, as its Jacobian is not symmetrie; see 11 Proposition K.2). With this connection between the OMM and Sanger's rule, a practical variant, we can also treat the Sanger update (I - v(a TAAy!" as a based optimization algorithm, such Adam [22]. Interestingly, in our experiment with operator learning, the Sanger variant works well in one PDE example where the OMM exhibits numerical instability. We attribute the instability of the OMM gradient to statistical noise, which may make the operator appear to have eigenfunctions with negative eigenvalues, thus triggering the divergent See Section 3.2 for a resolution and further discussion.
and plug-in to an off-the shelf gradient-
behavior discussed in Section 2.2.3
We note that this connection is rather surprising, given the extensive body of work over the past few decades developing efficient streaming spectral decomposition algorithms under orthogonality 51]. It is remarkable that a classical yet simple idea originating from computational chemistry provides such elegant solution to this long-studies spectral decomposition problem-yee to the best of our knowledge, it has remained largely unnoticed in the modern machine learning literature.
constraints; see, e.g.. [36 25| 44, 31 48
2.3 Orbital Minimization Method: Operator Version
In many applications, we are often tasked to compute the leading eigenfunctions of a positive- semidefinite operator I: (2( + (x). Here, L2,(X) & (f: X + R fx f(x)² oo} is a Hilbert space equipped with the inner product Stp fx (x(x))((x))((dx) By the spectral theorem, if r is compact, then there exists a sequence of orthonormal eigenfunctions (ft)21 with non-increasing eigenvalues a d2 2 2 O. When X' (1....,() d) and p is the counting measure, the space (x)) is naturally identified with RRd and T reduces to d x d PSD matrix, recovering the standard dimensionll eigenvalue problem in the previous section.
We can easily extend the OMM objective in Eq. (5) to this objective compactly using only k x matrices. Note that the objective in Eq. ( a function the overlap matrix VTV and projected matrix VTAV. In the infinite-dimensions function case, the overlap matrix and the projected matrix become the second moment matrices [ == f ( and M, f. rf] := f (f)) p(dx), respectively. Hence, by plugging-in this expression to Eq. (5) we can easily derive the corresponding objective for the infinite- dimensional case:
case, expressing the
C(p) omm (f) tr EE(-
(-1))
(2p (M, (M,JETJ--I,,, M,IF.)
(8)
5

---

In particular, if p = 1, it boils down to C()) (f) 2tr TT) + tr(M,]]]M[[[,, If)). If the point speetrum of I is isolated and separated by a spectral gap from its essential spectrum, we can also argue that the global minimizer of the OMM objective corresponds to the top-k eigensubspace.
2.3.1 Nesting for Operator OMM
As explained in the finiee-dimensional case, the idea of sequential nesting is to update the i-th eigenfunction fü by the gradient Ø comm (fi:) for each = 1..... k. We can implement these gradients succinctly via autograd for the special case of p 1. order to do so, we define a partially stop-gradiens second moment matrix
(f1.91); (f2, sg 91
(2) (f2.
(sg. 93)
(sg fil.
(sg 03)p
sg f2 Iklp
[ [f.g] ==
(fs. sg 9ile
sglgghh
(3s. 13)p
(sg 9
(fn, sglan))
sgigel sgiga)l
(fk: gh)p
Then, if we define the surrogate objective
Coom (f) == 2tr(Mp Tf]) + [ [f,
(9)
then 0j, Comm (f - Õr Coll (f).)) ) for each E [k]. In other words, the gradient for OMM with sequential nesting can be efficiently implemented by this single surrogate objective function in Eq. (
Note that the surrogate objective is equivalent to the OMM objective Cohm (f1:s) in its nominal value, but the gradients different due the stop-gru poop-graieents. The Sanger variant can be implemented
similarly with autograd; see Appendix B
This can be implemented efficiently with almost no additional computational overhead compared to computing M, [f.. g) without nesting. Note that the simple implementation is made possible thanks to the fact that tr tr(M, [f),, TT)) = (Mp(f), Mp(f, TF)) For p >> 1, however, we can no longer apply a similar, partial stop-gre googggradeett trick to the higher-order interaction terms tr [f MM,... Tf) for j 2 3, as they cannot be simply written as a matrix inner product with soop-gaddeet terms. Thus, it is harder to utilize the advantages of high-order OMM in the nested case. We thus only experiment with the higher-order extension in the self-supervised representation learning setting (Section 3.3 where the ordered structure is not often necessary.
Similar to the argument in [3 we also implement OMMjnt for operator with p 1 as follows:
Cint (f:
Σ
as C(l (fr..) = tr tr(po (-2Mplf, of + M,IF(,,(r.TF)))
Here, we define the matrix mask P E R as Pij mixx(is) with m; = E}=:'0 aj and O
denotes the entrywise multiplication. While Ryu et al. 45 suggest to use joint nesting for jointly parameterized eigenfunctions, we have empirically found that the convergence with sequential nesting is comparable to or sometimes better than joint nesting even with joint parameterization in general.
2.3.2 Comparison Low-Rank Approximation
As alluded to earlier, closely related approach is the www-rank-approximatie (LoRA) approach based on the Youun -Mirsky theorem [8] (5) (or Schmidt theorem [45]). This approach has been widely studied in both numerical linear algebra 29 55 I0 and machine learning [52 1S 43
55 24 20). For a self-adjoint operator T, LoRA A seeks rank-k approximation minimizing the error in the
norm T Σm_ f. @ liis ||T which simplifies to:
Ciora (f) 2 tr M, [f, Tf]) + (fM,,IIM,LL
((1) (f) := 2 2r (M2 Tf) - Lr(M, Tf)).
where we repeat (()) (f) for direct comparison. While both objectives are unconstrained and admit unbiased gradient estimates with minibatch samples, they differ in their geometric and spectral inter- pretations. Let Th E A @ o denote the rank-k truncation of T. If T is PSD, the optimizer

---

fomm of the OMM objective satisfies Ei=1 Somm, @ famm,i = =E== Oi @ Φ;, whereas the LoRA optimizer satisfies Σx S & flora 4 = TeAry, where r+ is the number of positive eigenvalues of T, for any self-adjoint, compact 43 Theorem C.5]. In words, OMM approximates the projection
operator onto the top-k cigensubspace, whereas LoRA reconstructs the best rank-k approximation of
T itself. Moreover, the LoRA principle naturally extends to the singular value decomposition of gen- eral compact operators, leading to the objective Civr (f, g) = -2r (M, [f, Tg)) + tr (M, [f] (g).
as studied in 43 In contrast, extending the OMM to handle general operators is nontrivial.
Interestingly, as shown in Appendix C.3.2] OMM can outperform LoRA in certain settings, par- ticularly when the target matrix is sparse, such as the graph Laplacians of real-world networks, highlighting its robustness in structured problems. We finally remark that sequentially nested for- mulations of both LoRA and OMM were previously proposed and analyzed for finite-dimensional matrices [10). yet their operator-level and learning-based generalizations remain largely unexplored.
3 Experiments
We evaluate the OMM and its variants (including the Sanger variant) across three experimental setups. In the first two, the eigenfunctions are well defined with clear operational meaning, and estimating them in the order of eigenvalues is desirable. These results underscore the strong potential of NestedOMM for both modern machine learning applications and ML-based scientific simulations. In the third setup, we explore the applicability of OMM in self-supervised representation learning, where preserving the underlying spectral structure is often secondary to optimizing downstream task performance. Our PyTorch implementation is available at at/htps:: /ttt:::://gtubbbbb com/ jonharyggary neural- svd
We defer the experiment details and additional numerical results to Appendix C
3.1 Laplacian Representation Learning for Reinforcement Learning
We consider the representation learning problem in reinforcement learning (RL), which is known as the successor representation [1] or more recently the Laplacian representation [13.. The high-level idea is that, given transition kernel from an RL environment, we aim to find good representation of a state in the given state space such that it reflects the intrinsic geometry of the environment.
More formally, we consider discrete state space S [N] for simplicity, but the treatment below can be easily extended to continuous state space. Suppose that we are given an environment p(ylar, a). which denotes the transition probability from 8; = x to 8t+1 = y given an action a a, from an RL. problem. For a given policy π (a)ar), we consider the policy-dependent transition kernel denote this by P,, E RSSS where (Pr)ay == Pa (z) In the literature [54] 51 [3] Laplacian is defined as any matrix L 1- (P) where f is some function that maps P.: to symmetric matrix, such as f(P) == 1 (P + PT). In the experiment below, P., is constructed to be symmetric. Like in the graph Laplacian, it is shown in the literature that the top-eigenveetors of L (or equivalently bottom-eigenvectors of - L f(p, x )) well capture the geometry of the RL problem 31 [44.
defined as Pr (v(zr) = E (e) [p(y|ar, a)}. In matrix notation,
In this experiment, we compare NestedOMM (OMM,eq and OMMjat) with a recently proposed
algorithm called the augmented Lagrangian Laplacian objective (ALLO) method [33. While the OMM involves no tunable hyperparameters, the ALLO requires selecting barrier coefficient b and barrier growth rate Obarrier via grid search on subset of grid environments, after which the chosen values are used for full-scale training with more epochs and transition samples.
We consider the same suite of experiments from [33, which consists of several grid environments as visualized in Appendix C.I For each environment, we generated 10t transition samples from a uniform random policy and uniform initial state distribution, used the (x,z) coordinates as inputs to a neural network, and aimed to learn the top k = 11 eigenfunctions. We trained for 80, 000 epochs using Adam (22) with learning rate 10-3, Additionally, we reproduce the ALLO training with identical number of transition samples, epochs, and optimizer, with the suggested initial barrier coefficient& b 2.0 and Obbamier = 0.01 reported in [3]. We trained with NestedOMM for four random runs after shifting the spectrum by and ALLO with 10 runs. The performance is measured by the average of cosine similarities of each mode, where degenerate spaces are handled by an oracle knowing the degeneracy.
7

---

GrdMate-7
meenn cosite sim Lertt
0.58
GrilMace-17
0.8
GriMMaze
--- GrdMare-26
GrilMare-31 32
0.94
Gridluam-
GridRoom44
GrdRoom
Gridaom 32
0.2
0.90
Griduumm-e 64
gridmace
gridnasis
gidinace 17
gridnace 10
gridmare_ gridmare 32
grittom,
grittpom
gitrrom 16
uoddram
pridiomm 54 ondrronayy
100
GridoamSyys
eigenndex
Figure 1: Summary of experimental results of the Laplacian representation learning. On the right panel, plot the top-11 eigenvalues of the Laplacians. shows that the hard instances (i.e., (aadMazze-2662222 and GridRoom (32,64)) have very small spectral gaps (marked with x). Error bar indicates one standard deviation.
7
N -2
W
LoRA...
K-)
Songer
K+
*
tree
Figure 2: Visualization of learned cigenfunctions for the 2D hydrogen atom. OMM performs as well as LoRAug. which is also known NeuralSVD.ss 43
We summarize the results in Figure D Across different environments, both OMMssq and OMMjnt perform comparably to the complex optimization dynamics of ALLLO, without any hyperparameter tuning. To understand the failure cases, we plot the top-11 eigenvalues of the Laplacians on the right panel. It shows that these hard instances have very small spectral gaps, suggesting that a small cosine similarity does not necessarily indicate a failure of learning in such cases.
3.2 Solving Schrödinger Equations
Following 43 we now consider applying OMM to solve some simple instances of time-independenn Schrödinger's equation [4]. The equation is (x)
(x(x)) where म -2 V Vxx is the Hamiltonian operator of a system, where V(x) is potential function. Since low-energy eigenfunctions correspond the most stable states, we are primarily interested in retrieving the smallest eigenvalues of S. Accordingly, in our convention, we aim to retrieve the eigenvalues and eigenfunctions of the operator T -f from the top.
3.2.1 Bounded-Spectrun Case
The OMM directly applicable to certain problems, where the Hamiltonian ' has eigenenergies bounded above, such that T = -H (or its shifted version) is PSD. To assess the potential in such settings, we consider the simplest example this kind: the two-dimensional hydrogen atom, where the potential function is given by V(x)
|x|2 for E R
40 43
In this case, the true eigenvalues
of II are expressed as >,, = (2n + 1) with two quantum numbers n 2 and -n S f S n.
We compare the performance of OMMseq to LoRAseg We first note that we empirically found that the OMM applied to - leads unstable training. We conjecture that this instability is due to the fast-decaying spectrum and statistical noise during minibatch optimization. That is, in the beginning of optimization, some small eigenvalue modes are captured by the model, and the moment matrices formed by minibatch samples yield negative eigenvalues due to statistical noise, model may diverge, as we discussed in Section 2.2 found that this misbehavior of OMM in this example can be rectified by decomposing the shifted operator T + k for some >> 0, such that the smallest eigenvalue bounded away from zero. Instead of reporting results with OMM with identity shift, here report the result with the Sanger variant. Rather surprisingly, find that the

---

Sanger variant does not exhibit optimization instability and performs on par with LoRAgeq- which is shown to outperform existing baselines such as SpIN [40] and NeuralEF 77. As shown in Figure the learned eigenfunctions from both Sanger and I LoRAseq are well well-aligned with the ground truth, showing the competitiveness of the OMM compared to the state-of-the approach. Figure in Appendix C provides the quantitative evaluation.
3.2.2 Unbounded-Spectrune Case
As alluded to earlier, the OMM is not directly applicable when the energy spectrum of H is unbounded above, as the cases of the harmonic oscillator or the infinite well 14 43
To render the OMM applicable in such settings, we introduce an inverse-operator trick, analogous to that used in the inverse iteration [49]. Let & be a self-adjoint operator on a Hilbert space with a purely discrete, positive spectrum 0 d I A2 ... with X - co, and corresponding orthonormal cigenfunctions {or h21- This assumption holds, for example, for Schrödinger operators on bounded domains with Dirichlet boundary conditions, or more generally, for confining potentials V(x) - oc as ||||| + 00, such the harmonic oscillator on R In this case, L is invertible, and its inverse --1 is a bounded, self-adjoint, and compact operator with eigenvalues A 4 0 and the same eigenfunctions. Consequently, the top-k eigenfunctions of --1 coincide with the bottom-k eigenfunctions of c. Thus, applying the OMM to - enables recovery of the lowest-k modes of 4
practical difficulty is that -- is rarely available in closed form; for differential operators, corresponds to an integration operator. To avoid explicit inversion, we can parameterize f := Lg with g represented by a neural network. Substituting this into the OMM-1 objective yields
Ciny (g: L) -2t L (f)) + to [ M (f
2 2n (M.MMM/Lg.g) g)) + tn Mp[[g] Mp[Ag, gl),
which preserves the same computational complexity the original OMM. Since the i-th eigenfunction of L-1 is di. Lg: corresponds to di, and ideally 9i A, Φ; holds for each i E [A]]
In Appendix C.2 we empirically validate this approach on the 2D infinite-well and harmonic-oscillator systems, both of which admit analytical solutions. We find that while the OMM with the inverse- operator trick accurately recovers the top eigenfunctions, it can introduce numerical instabilities and requires careful hyperparameter tuning, likely due to the nature of the parametrization.
3.3 Sele-Supervised Contrastive Representation Learning
We now apply the OMM for contrastive representation learning. Here, we primarily consider self- supervised image representation learning using OMM, comparing to SimCLR [3), deferring its application to graph data to Appendix C.3.2. In this setup, images x ~ p(z), and our goal is to learn a network f-(-) parametrized by such that it yields a useful representation for downstream tasks such classification. Following SimCLR, we consider random augmentation p(t[ær) and draw two random transformations (2,, ~ p(1 (z)(2))((t)))) (x) to produce two views of data entry By repeating this procedure for each image. obtain the full dataset, which can be viewed as samples from the joint distribution p(t1, t2) == Epcr) [p(t)) [=))=(2p(((()))
given an unlabeled dataset of
The key object is then the canonical dependence kernel (CDK) t2)
p(t, p((t)) where p(t) :=
p(ti.t2)
Ep(x)[p(t(zr) [] 443. Most contrastive learning approaches aim to learn fo(-) such that it factorizes
the log CDK (or pointwise mutual information) fa (fz (fg( = log k(t,, f2) 27 28 50]. Few exceptions include [15] 55| 43). which instead aim to factorize the CDK itself fg ) fo R (2), such that the optimal (4) can be interpreted as the low-rank approximation of the CDK. Application of the OMM to CDK provides a rather unique way to learn representations, as it does not directly aim to approximate function of the CDK, but rather trying to find normalized eigenfunctions purely from linear-algebre point of view. Note that we can apply the OMM framework in this
For instance, consider the Schrödinger equation under zero Dirichlet boundary conditions with the square infinite-wer well potential. V Vxx) for [|1, 1J2 and V Vxx
otherwise. Its eigenvalues Ang, x n + ng. parameterized by quantum numbers Azz fly N, which diverge as fz or ny - oc.
given by
When the spectrum includes continuous component whose infimum is zero (as in unconfined systems), may not be bounded. In such cases, may instead use (L + +e))= with some > 0.

---

Table 1: Top-1 and top-5 classification accuracies (%) for the SimCLR and OMM variants CIFAR-100.
With projector
DirectCLR (top-64 dim.)
Method
Top-1
Top-5
Top-1
Top-5
OMM (p = 1)
60.02
87.13
59.83
86.65
OMM (p = 1)
61.30
85.22
59.92
85.13
OMMseq (p 1)
59.91
87.02
52.96
81.22
OMM (p =0=2]
63.92
89.08
61.27
87.07
OMM (p 1) OMM (p=2)
64.77
89.18
63.99
88.88
SimCLR
66.50
89.28
N/A
N/A
symmetric supe--uue setup, as the CDK is symmetric and PSD by construction. The purpose of this experiment is not to establish the state-of-the-ar performance, but to assess the potential.
We followed the standard setup of 4. We used ResNet- 18 as our backbone model and adopted two different feature encoding strategies: (1) use a nonlinear projector with 2048 hidden dimensions and k 256 output dimensions as our input to the OMM; (2) in vein similar to DirectCLR [21), take the top & = 64 dimensions as the feature fed to the OMM objective without a projector. In all cases, normalized features along the feature dimension before computing any loss following the standard convention. We report the results of top- |,5] classification accuracy with linear probe.
The results are summarized in Table I We first observe that the OMM-1 yields reasonably effective representation for classification, achieving a top-1 accuracy of approximately 60%. We then examined the effect of nesting representation quality and observed no improvement, with slight degradation in some cases. Subsequently, we evaluated the effect of the higher-order OMM with p 2 as well as the mixed-order OMM combining p E {1, 2). Somewhat surprisingly, both variants substantially improved performance, reaching approximately 64% top-1 accuracy. In Appendix D we provide a theoretical explanation for the benefit of higher-order OMM, based on a gradient analysis. In short, the higher-order OMM gradient may provide an additional gradient to escape an immature flat local minima. A similar trend is observed with the DirectCLR variant, and notably, the performance remains comparable even without a projector. While these methods do not surpass the SimCLR baseline (~ 66.5%), the preliminary findings underscore the promise of this linear-algebraic representation learning framework and motivate further investigation.
4 Concluding Remarks
In this paper, we revisited a classical optimization framework from computational chemistry for computing the top-k eigensubspace of a PSD matrix. Despite its appealing properties that align well with modern machine learning objectives. this approach remains underexplored in the current literature. We hope that this work stimulates further research into this ainear-algebraic perspective and inspires the development of more principled learning methods. We also note that existing parametric approaches to spectral decomposition exhibit notable limitations, particularly a lack of theoretical understanding regarding their convergence properties compared to classical numerical methods. We refer an interested reader to general discussion on the potential advantage and limitation of the parametric spectral decomposition approach over classical methods in
The experimental results presented in this work are preliminary, and we believe that a more com- prehensive investigation across diverse application domains could yield deeper insights and more impactful findings. In particular, extending the current framework to research-level quantum excited- state computation problems, as recently explored via alternative variational principles 42 g be of significant theoretical and practical interest. Furthermore, advancing the linear-algebraic perspective on modern representation learning may open up a pathway toward more structured, interpretable, and theoretically grounded representations.
would
10

---

Acknowledgments
This work was supported part by the MIT-IBM Watson Al Lab under Agreement No. W1771646.
References
[1] David Bowler and Tsuyoshi Miyazaki. Methods in electronic structure calculations. Rep. Prog. Phys., 75(3):036503, 2012. P.
[2] James Chapman, Lennie Wells, and Ana Lawry Aguila. Unconstrained stochastic CCA: Unifying multiview and self-supervised learning. In Int. Conf. Learn. Repr., 2024. URL
https://openreview..u net. fortt? wit.
p.
[3] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In Proc. Int. Conf. Mach. Learn., pages 1597-1607. PMLR, 2020. pp. 0. and
[4] Victor Guilherme Turrisi da Costa, Enrico Fini, Moin Nabi, Nicu Sebe, and Elisa Ricci. solo- learn: A library of supe-vupe methods for visual representation learning. Res., 23(56):1-6, 2022. URL http:///mmnll.oorrappprr///////s
Mach. Learn.
html
pp. 10 and 22
[5] Peter Dayan. Improving generalization for temporal difference learning: The successor repre- sentation. Neural Comput., 5(4):613-63) 1993. p.
[6] Zhijie Deng, Jiaxin Shi, Hao Zhang, Peng Cui, Cewu Lu, and Jun Zhu. Neural Figenfunctions
Are Structured Representation Learners. arXiv, October 2022. pp. 23 and 24
[7] Zhijie Deng, Jiaxin Shi, and Jun Zhu. NeuralEF: Deconstructing kernels by deep neural networks. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato, editors, Proc. Int. Conf. Mach. Learn., volume 162 of Proceedings of Machine Learning Research, pages
PMLR, 17-23 Jul 2022. URL https:://proceeedingg
mlr. wwwppss/vvv2/ddmg22....bff html
p.
[8] Carl Eckart and Gale Young. The approximation of one matrix by another of lower rank. Psychometrika, 1(3):211-218, September 1936. ISSN 0033-3123, 1860-0980. doi: 10.1007/ BF02288367. p.
[9] M T Entwistle, Z Schätzle. P A Erdman, J Hermann, and F Noé. Electronic excited states in deep variational Monte Carlo. Nat. Commun., 14(1):274, January 2023. ISSN 2041-1723. doi: 10. 1038//414477002-3555522
pp. and
[10] Weiguo Gao, Yingzhou Li, and Bichen Lu. Triangularized OrthogonalizzaioonFFeee Method for Solving Extreme Eigenvalue Problems. Sci. Comput., 93(63):1--28 October 2022. doi: 10.10077:100115-022-002222
pp. 6 and
[11] Ian Gemp, Brian McWilliams, Claire Vernade, and Thore Graepel. EigenGame: PCA as Nash equilibrium. In Int. Conf. Learn. Repr., 2021. p.
[12] Gemp, Brian McWilliams, Claire Vernade, and Thore Graepel. BigenGame unloaded: When playing games is better than optimizing. In Int. Conf. Learn. Repr. 2022. URL https
//openreviwwnne//ffeee SosooonnnniissssAAAsgggggn
p.
[13] Diego Gomez, Michael Bowling, and Marlos C. Machado. Progeraaapaaccneeeeneentttttoto learn- ing. In Int. Conf. Learn. Repr. 2024. URL http:://wooneeeeew.eeeee//////ra pp.
18 and 19
[14] David J Griffiths and Darrell F Schroeter. Introduction to quantum mechanics. Cambridge university press, 2018. pp- and 21
[15] Jeff Z. HaoChen, Colin Wei, Adrien Gaidon, and Tengyu Ma. Provable guarantees for self- supervised deep learning with spectral contrastive loss. In Ady. Neural Inf. Proc. Syst., 2021.
pp. 6 and 9
11

---

010166/00010-4655597)0000000 [16] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proc. IEEE Comput. Soc. Conf. Comput. Vis. Pattern Recognit,, pages 770-778,
2016. pp. 22 and 24
[17] Geoffrey Hinton, Nitish Srivastava, and Kevin Swersky. Neural networks for machine learning lecture 6a overview of mini-batch gradient descent. 2012.
20
[18] Weihua Hu, Matthias Fey, Marinka Zitnik, Yuxiao Dong, Hongyu Ren, Bowen Liu, Michele Catasta, and Jure Leskovec. Open graph benchmark: Datasets for machine learning on graphs. In Adv. Neural Inf. Proc. Syst., volume 33, pages 22118 228-222133,,1 2020. p. 23
[19] Qian Huang, Horace He, Abhay Singh, Ser-Nam Lim, and Austin Benson. Combining label propagation and simple models out out-performs graph neural networks. In Int. Conf. Learn. Repr.,
2021. p.
[20] Minchan Jeong, J. Jon Ryu, Se- Yo-n Yun, and Gregory W. Wornell. Efficient parametric SVD of Koopman operator for stochastic dynamical systems. In Adv. Neural Inf. Proc. Syst., 2025.
pp. ppnda
[21] Li Jing, Pascal Vincent, Yann LeCun, and Yuandong Tian. Understanding dimensional collapse contrastive self-supervised learning. In Int. Conf. Learn. Repr, 2022. URL https://
pp. a0
[22] Diederik Kingma and Jimmy Ba. Adam: method for stochastic optimization. In Int. Conf. Learn. Repr, 2015. pp.
20 and 21
[23] Vladimir R Kostic, Pietro Novelli, Riccardo Grazzi, Karim Lounici, and Massimiliano Pontil. Learning invariant representations of hime-homogeneous stochastic dynamical systems. In Int. Conf. Learn. Repr., 2024. URL http:::://ppeeeeeeeeeeeee/ooooop htttt//oorrrmmdddts22i000o 1d=twSn2wi01m p
[24] Vladimir Kostic, Gregoire Pacreau, Giacomo Turri, Pietro Novelli, Karim Lounici, and Massimiliano Pontil. Neural conditional probability for uncertainty quantification. In Adv. p.
Neural Inf. Proc. Syst., 2024. URL httts:///pppmeevieee.e///frrrrrr
[25] T P Krasulina. The method of stochastic approximation for the determination of the least eigenvalue of a symmetrical matrix. USSR Computational Mathematics and Mathernatical Physics,
January 1969. ISSN 0041-5553, doi: 10. 10.101660041155536999911111
p
5
[26] IE Lagaris, A Likas, and D I Fotiadis. Artificial neural network methods in quantum mechanics. Comput. Phys. Commun., 104(1-3):--1, August 1997. ISSN 0010-4655, 1879-2944. doi:
[.
[27] Omer Levy and Yoav Goldberg. Dependency- word embeddings. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 302-308, 2014. p.
[28] Omer Levy and Yoav Goldberg. Neural word embedding implicit matrix factorization. In Adv. Neural Inf. Proc. Syst., volume 27, pages 2177-2185, 2014.
[29] Xin Liu, Zaiwen Wen, and Yin Zhang. An efficient low-rank product matrix approximations. SIAM J. Optim., 25(3):1571-1603 2015. p.
algorithm for symmetric
[30] Jianfeng Lu and Kyle Thicke. Orbital minimization method with el regularization. J. Comput. Phys., 336:87-103, May 2017. ISSN 0021-9991. doi: 20.1016/jjcc..2017.02.. p.
[31] Marlos C. Machado, Clemens Rosenbaum, Xiaoxiao Guo, Miao Liu, Gerald Tesauro, and Murray Campbell. Eigenoption discovery through the deep successor representation. In Int. Conf. Learn Repr., 2018.
pp_ and 7
[32] Andreas Mardt, Luca Pasquali, Hao Wu, and Frank Noé. VAMPnets for deep learning of molecular kinetics. Nature Comm., 91):5, January 2018. ISSN 2041-1723, doi: 10. 1031038/ s41467-017-02388- URL htttp::///oi.... 103. http:///ww...o..///////////000/0
p.
12

---

[33] Francesco Mauri and Giulia Galli. Electronic-structur calculations and molecular-dynamics simulations with linear system-size scaling. Phys. Rev. Condens. Matter, 50(7):4316, 1994.
pp. and
[34] Francesco Mauri, Giulia Galli, and Roberto Car. Orbital formulation for electronic- calculations with linear system-size scaling, Phys. Rev. B Condens. Matter, 47(15):9973, 1993.
Pp.
1
and
[35] L Mirsky, Symmetric gauge functions and unitarily invariant norms. Q. J. Math., 11(1):50-59, January 1960. ISSN 0033-5606. doi: 10. 10.1099//math/11 1.1.50. p.
[36] Erkki Oja. A simplified neuron model as a principal component analyzer. J. Math. Biol., 15(3): 267-273, 1982. ISSN 0303-6812, doi: 10. 10000//000225687. p.5
[37] Pablo Ordejón, David A Drabold, Richard M Martin, and Matthew P Grumbach. Linear system- size scaling methods for electronii--trructure calculations. Phys. Rev. B Condens. Matter, 51(3):
1456, 1995. pp-
and
[38] Pablo Ordejón, David A Drabold, Matthew P Grumbach, and Richard M Martin. Unconstrained minimization approach for electronic computations that scales linearly with system size. Phys. Rev. B Condens. Matter, 48(19): 1446-1664444 14649, November 1993. ISSN 0163-1829. doi: 10.
1 1103/hhsseev..45 48. 14646. pp. and
[39] Adam Paszke, Sam Gross, Franeisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Köpf, Edward Yang. Zach DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang. Junjie Bai, and Soumith Chintala. PyTorrc: An imperative style, high-performance deep learning library. In Adv. Neural Inf. Proc. Syst., Red Hook, NY, USA, 2019. Curran Associates Inc. p. 17
[40] David Pfau, Stig Petersen, Ashish Agarwal, David GT Barrett, and Kimberly L Stachenfeld. Spectral inference networks: Unifying deep and spectral learning. In Int. Conf. Learn. Repr., 2019. pp. [
[41] David Pfau, James S Spencer, Alexander G D G Matthews, and W M C Foulkes. Ab initio solution of the many-electron Schrödinger equation with deep neural networks. Phys. Rev. Res., 2(3):033429, September 2020. doi: 10. Rewe/Phyyeeeeeeecchhh22.223 2.033442, pp. 21
[42] David Pfau, Simon Axelrod, Halvard Sutterud, Ingrid von Glehn, and James S Spencer. Accurate computation of quantum excited states with neural networks. Science, 385(6711)
2024. pp. and 10
[43] Jon Ryu, Xiangxiang Xu, Hasan Sabri Melihcan Erol, Yuheng Bu, Lizhong Zheng, and Gregory W. Wornell. Operator SVD with neural networks via nested low-rank approximation. In Proc. Int. Conf. Mach. Learn., 2024. URL https:
PP.
20
and 24
[44] Terence Sanger. Optimal unsupervised learning in a single-layer linear feedforward neural network. Neural Netv,, 2(6):459-477 January 1989. ISSN 0893-6080. doi: (1.00160893--00008899900000
pp-2 and
[45] Erhard Schmidt. Zur Theorie der linearen und nichtlinearen Integralgleichungen Math. Ann., 63(4):433-477 December 1907. ISSN 0025-5831, 1832-1807. doi: 10.1007/BFO01449770.
6
[46] Daniel Spielman. Spectral and algebraic graph theory. Lecture notes (Yale University), 4:47,
2019. p. 23
[47] Matthew Tancik, Pratul Srinivasan, Ben Mildenhall, Sara Fridovich-Keil. Nithin Raghavan, Utkarsh Singhal, Ravi Ramamoorthi, Jonathan Barron, and Ren Ng. Fourier features let networks learn high frequency funetions in low dimensional domains. In Adv. Neural Inf. Proc. Syst., volume 33, pages 7537-7547, 2020. p. 20
13

---

[48] Cheng Tang. Exponentially convergent stochastic k-PCA without variance reduction. In
editor, Adv. Neural Inf. Proc. Syst. volume 32. Curran Associates, Inc., April 2019. p.
[49] Lloyd N Trefethen and David Bau. Numerical linear algebra, volume 181. SIAM, 2022. p.
9
[50] Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint arXiv: /rXiv: w00..0007.00000 00000000.78... 2018.
[51] Kaixin Wang, Kuangqi Zhou, Qixin Zhang. Jie Shao, Bryan Hooi, and Jiashi Feng. Towards better Laplacian representation in reinforcement learning with generalized graph drawing. In
Proc. Int. Conf. Mach. Learn., pages 11003-10012. PMLR, 2021. pp and 7
[52] Lichen Wang, Jiaxiang Wu, Shao-Lut Huang, Lizhong Zheng, Xiangxiang Xu, Lin Zhang, and Junzhou Huang. An Efficient Approach to Informative Feature Extraction from Multimodal Data. In Proc. AAAI Conf. Artif. Int., volume 33, pages 2281-5288, July 2019. doi:
aaai.v33i01 10001033312288 p- 6
[53] Zaiwen Wen, Chao Yang, Xin Liu, and Yin Zhang. Trace- Seacctpenalty minimization for large-scale eigenspace computation. J. Sci. Comput., 66:1175-111 2016. p.
[54] Yifan Wu, George Tucker, and Ofir Nachum. The Laplacian in RL: Learning representations with efficient approximations. In Int. Conf. Learn. Repr. 2019. URL https:///operrvvieew
[55] Xiangxiang Xu and Lizhong Zheng, Neural feature learning in function space. J. Mach. Learn Res., 25(142)::-76, 2024. pp. and
[56] Yang You, Igor Gitman, and Boris Ginsburg. Large batch training of convolutional networks. arXiv preprint /70::::7083888888 2017. pp. 23 and 24
14

---

Appendix
Deferred Proafs
15
A.1
Equivalence of c(p) (V) and E(p) (V)
15
A.2
Proot of Theorem
15
A.3 Proof of Theorem
16
On Implementation
16
B.1
Implementation of Sanger's 's Variant
16
Pseudocode for NMM-edOMM-1I.
17
B.3
Computational Complexity of OMM
18
C
Details on Experimental Setups and Additional Results
18
C.1
Laplacian Representation I Learning for Reinforcement Learning
18
C.1.1
Experimental Setup
18
C.1.2
Ablation Study: Hyperparameter Sensitivity of ALLO vs OMM
19
C.2
Solving Schrödinger Equations
19
C.2.1
Experimental Setup for 2D Hydrogen Atom
20
C.2.2
Results on Unbounded-Spectra Case
21
Supe-Supervieed Contrastive Representation Learning
22
C.3.1
Representation Learning for Images
22
C33.2
Representation Learning with Graph Data
23
On the Benefit of Higher-Order OMM
25
Deferred Proofs
A.1 Equivalence of Comm (V) and (6m)
(V)
To establish the equivalence between the original proposal cém (V) in Eq. and our derivation a(p)
Comm(V) in Eq. (5) we need to show that
tr(QpTTAN = tr((1a (VT)2 tr(A),
where we recall
2p-1
(le VTV)'.
To show this, we first note that we can write
2p-1
O Σ (1k S)'- S-' (lk S)
by letting S = VTV. Hence, we have
VTAV) -tr 5 (Ix S-
-(s -Ia-01-5)))))) (S)DOVTAAAAA)
-tr(S VVTVV -VTAVS-VVAVV + trsss-1155 1--1(1 (1x - -S)PPVTAV)
--tr(S-VVTAV) + tr(S-IVTAV) + to tr((1a -VTTJ2AA - -r((A)
-tr( (r()).
This concludes the proof.
A.2 Proof of Theorem 1
Proof of Theorem Since VVT of rank most & and PSD, the optimization can reparam-
eterized based on the reduced SVD of VVT -PPP, where P = [p...... P&] dd and
15

---

= diag(H1..... mb) E R such that PTP = ly and μl ... 2 0. For given orthog- onal matrix P e Raxk let P1 (d= denote matrix whose columns form an orthonormal basis of the orthogonal complement of the column subspace of Note PPT + P1PI and PIP (d=))xx by construction, so that we can write
- VVT = (P(I: - 2:PT + P
= P(1 - +
Hence, the objective function can be lower bounded as
tr((1a - (VT) tr(P(12
+
(4) tr(PIAPL)
M
de.
Here, (a) follows since EP) and A are both PSD and the inner product of two PSD matrices is always nonnegative. Further, (b) follows since tr(PI AP1) is minimized as Σi= òe when optimized over an orthogonal matrix P1 E [ddx(d=)), We now consider the equality condition. First, (b) holds with equality if and only if P1 consists of bottom- k) eigenbasis of A, or equivalently can be written as P = VQT by orthogonal matrix Q E R Given that, we can write
tr(P(l=: P)PT))))
tr((I: 2 QVTAVTT)
-tr((l - Q)*
Σ - px) 8=1
This implies that (a) holds with equality and only if μe = 1 for all l E [kr.].
A.3 Proof of Theorem
Proof of Theorem 2 The global minima of the jointly nested objective Coom (V:
Σ (o (Vis) ) is achieved if and only if cle) (V1s) is minimized for each E [b)]. Hence, the V* E d achieves the global minima, then by the optimality condition from (ip) (Vi.),
must satisfy
((y) Σ W
j=1
for each E [k). By telescoping, this leads to V, w, for each i.
B On Implementation
In this section, we present omitted details on the implementation of MestedOMM.
B.1 Implementation Sanger's Variant
Similar to the implementation of the sequential nesting by the following partially stop arogigradient objective
Coom (f) [= Tf]) + tr(M Tf)),
can implement the Sanger variant by auto-differenniiaing the following objective
Csanger (f) (f, Tf) + tf(M)" (f, Tf])).
With the appropriate stop-gradient operation, its gradient implements the Sanger variant.
16

---

B.2 Pseudocode for NestedOMM-11
Here, we include a unified PyTorch 33) implementation various versions of original OMM with (which we call OMM-1): OMM-1 without any nesting (when nesting is None), OMM-1 with the joint nesting (when nesting is jnt), OMM-1 with the sequential nesting (when nesting is seq), and OMM-1 with the Sanger variant (when nesting is sanger).
class NestedOrbitalLoss: def _nnit_ self, nesting=None, n_ n_modes=None, ):
5
6
assert nesting in [None, 'jnt', 'seq", 'sanger']
self. relfinnsting nesting self.nesting 'jnt':
assert n_modes is not None
self.vec_mae self.mat_mask = ogt..pppni....iiiggggaaaaausaaso m_medsss / n_modes)
else:
self. vec_mask, self.mat wam.mmmtttttt... None, None
14
def
___ f: Tensor, Tf: torch. Tensor):
# [b, k]
# Tf: [b, k]
self.nesting == 'jnt':
M_f compute_ secitee_seood___ee (e__s_seenn__m_ne(())
M_ M_f_Tf compute secpteeesecood___eeee moment(f, Tf)
operator_tera
(torch.diag(self..w wwt.:eewewetto...eeeeeeeeee M_f_Tf) mean
(6). sum()
22
metric term (self.mm feelf.mat_mask.cee mask.to(f..eviin
M_f_Tf M_f)).um()
23
else: # if self nesting in [None, 'seq', 'sanger']:
M_f compute second cictond_momen(ff.. mea.nestingssel..nesttiin in ['seq', 'sanger'
M_f_Tf Sompute_secomee (f Tf, seq_nesting-self.neet == 'seq') if self self.nesting == 'sanger':
26
M_f_Tf M_f_Tf.detach
operator_
torch. trwacee__c_ffFr_e
29
metric_tent term (M_f_Tf M_f).sum()
30
31
return aperator_term metric_term
def compute compute_second second [
35
f: torch. Tensor, torch. Tensor None, seq_nesting: bool False
-> torch. Tensor:
39
40
compute (optionally sequentially nested) second-momen matrix
M_ij <f_i, g_j>
with partial stop-gradient handling when seq_nesting True.
43
44
args
f: (n, k) tensor
g: (n, k) tensor None
48
seq_nesting bool
29
if is None:
g=f s.shape[o] if not seq_nesting:
53
return (f.T @ g) /
else:
17

---

# apply partial stop-gradient
# trian-triangula:: <f_i, sg[g_j]> for i > lower torch. Tarch.tril(f.T @ g.detach(), diagonal--1) # upper-triangular: esg[f_i), 8_j> for < upper torch. triu(ff.etacch().. g, diagonal=+1)
58
59
60
# diagonal:
<f_i, g_i> (no stop-grad)
diag torch.diag(() ())ssm(dimm)))
63
return (lower diag upper)
64
def ret_joint_neetiing.om www.inntnnnn...n (weighht: np.ndarray):
57
vector_mast List(np. wwtp:ccccuumssssse (innsuu((ist(weiightt::::: (::-))) vector_mase torch.
68
69
matrix_mask torch.minimum;
vector_mask. (aask.unsqueeze()) vector_masks uus.uskkuusuuueee()........ f.float()
72
return vector_mask, mask, matrix_mast
B.3
Computational Complexity of OMM
In terms of complexity in computing a given objective, the complexity of OMM-1 similar to LoRA 43 and ALLO [3). For each minibatch of size B, the dominant factor is in computing the empirical moment matrices, which takes O(B)²) complexity for a given minibatch of size and for k eigenfunctions. When 1, the complexity becomes O(Bk² + pk*), as we need to recursively compute the powers the overlap matrix in Eq. (
For a detailed comparison between the parametric spectral decomposition approach and a conventional numerical eigensolver, we refer to 43 Appendix A]. There, the authors report an undesirable scaling behavior exhibited by numerical eigensolvers and highlight the efficiency advantage of the LoRA- based parametric formulation. Given that the computational complexity of gradient evaluation in LoRA and OMM is equivalent, the same argument applies here.
C Details on Experimental Setups and Additional Results
In this section, we describe the detail on the experimental setups. All codebases to reproduce the experiments will be made public upon acceptance of the paper. All experiments were conducted single GPU, either a NVIDIA GeForce RTX 3090 (24GB) or NVIDIA RTX A6е96 (48GB).
C.1 Laplacian Representation Learning for Reinforcement Learning
Our implementation was built upon the codebase of Gomez et al. 13
C.1.1 Experimental Setup
We followed the experimental setup in closely. We visualize the RL environments in Figure 3
For each experiment, we generated = 10 transition samples from uniform random policy and uniform initial state distribution. The (I, y) coordinates are given to a neural network as inputs.
Data generation:
Architecture: We used a fully connected neural network with hidden layers of 256 units with ReLU activations, with output layer of k 11 to learn the top-k Laplacian cigenfunction representation.
Optimization: We trained for 80,000 epochs using the Adam optimizer with a learning rate of 10-3 In the first 10% of training steps, we also add a linear warm-up from 0 to 10-3 Additionally, we reproduce the ALLO training with identical number of transition samples, epochs, and optimizer, with the suggested initial barrier coefficient = 2.0 and Obarrier 0.01 reported [I3]. We trained with NestedOMM for four random runs, and ALLO with 10 runs. Since the smallest eigenvalue of each Laplacian matrix can be negative
GGithub repository: httos://gttioo.cc////aoatttttppp _
18

---

in this case, but being bounded below by -1 (by the Gershgorin circle theorem), we add the identity matrix to the Laplacian matrix to shift the spectrum.
Performance metric: The performance is measured by the average of cosine similarities of each mode, where degenerate spaces are handled by an oracle knowing the degeneracy.
GrAIMaze-11
GridMarm-05
Gritioomm1
GridRoom4
Grddoom-le 16
Crddtoom-
+
Figure 3: Grid environments for Laplacian representation learning.
C.1.2 Ablation Study: Hyperparameter Sensitivity of ALLO vs OMM
We continue with the experimental setup in [3] but attempt the more difficult task of learning the top & = 50 rather than top k = 11 Laplacian eigenfunctions. Our goal is to examine differences in h
hyperparameter sensitivity between ALLO and OMM.
Revised Setup: For both OMM and ALLO training in the ablation study, use the following common modifications to the experimental setup: we generate = 5 10 transition samples from uniform random policy and uniform initial state distribution. Our architecture keeps the 3 hidden layers of 256 units with ReLU activations, but the output layer's dimension is l- 50 to learn the top-50 Laplacian eigenfunctions. Additionally, omit the GridMaze-7 and GridMaze-9 environments, as they have less than 50 eigenmodes. The optimization recipe and performance metric remain the same.
Initial Evaluation of ALLO Hyperparameters: We begin by examining how well the 50. In particular, we use the suggested initial barrier coefficient b = 2.0 and Oharrier 0.01. After training each environment for four runs, we obtain the results displayed in Table (a). It is clear that the learned eigenfunctions do not capture the top-50 modes well, and that require additional hyperparameter tuning to effectively learn the eigenmodes with ALLO.
tuned ALLO hyperparameters transfer from k 11 to
sweep the initial barrier coefficient b over [0.5, 1.0, 2.0, 10.0). while Obarrier sweeps over [0.001, 0.01, 0.05, 0.1, 0.2, 0.5, 1.0]. We train over the GridRoom-l, GridRoom-16 16, and GridMaze- 19 environments, and also use 10 transition samples instead of 2 1 for hyperparameter selection. This yields the best pair as (b, Obarrier) (1.0, 0.1... Using the updated hyperparameters, we rerun ALLO training for k = 50 with 3 independent runs per environment. The results are displayed in Table bb). We can observe a significant jump in performance, but the expense of costly hyperparameter search.
ALLO Hyperparameter Sweep: As in [3].
OMM Retraining With Same Hyperparameters: In contrast, we train OMM for k: = 50 with the same recipe as k 11, with the only difference being the increase to 5 1 transition samples. We train for 5 independent runs per environment and get the results see, the representational power of OMM comparable with ALLO, but we were able to train without the costly pre-training hyperparameter sweep.
displayed in Table ()). As
C.2 Solving Schrödinger Equations
We followed the setup in 4 closely, implemnnnn code based top of theirs
GGithbb repository:
19

---

Table 2: Ablation study results with ALLO and OMM for & = 50.
(a) ALLO for 50 with optimal hyperparameters tuned for 11.
Env
GridMaze-17 GridMaze-19 GridMaze-26 GridMaze-32
GridRoom-1
Mean
0,8150
0.7616
0,4361
0,3340 0.0303
0.8081
Std
0.0234
0.0277
0.0361
0.0077
Env
GridRoom-4 GridRoom-16 GridRaom-32 GridRoom-64 GridRoomSym-4
Mean
0.6414
0.6692
0,4837
0.3420
0.6297
Std
0.0190
0.0153
0.0451
0.0164
0.0321
(b) ALL.O for & 50 with re-tuned hyperparameters (b 1.0, Oburier 0.1).
Env GridMaze-17 GridMaze-19 GridMaze-26
GridMaze-32
GridRoom-
Mean
0,9798
0.9669
0.8233
0.7359
0.9088
Std
0.0022
0.0001
0.0165
0.0176
0.0328
Env
GridRoom-4 GridRoom-Ie 16 GridRoom-32 GridRoom-64 GridRoomSym-4
Mean
0.8681
0.8885
0.7452
0.7417
0.7880
Std
0.0004
0.0127
0.0093
0.0473
0.0241
(c) OMM for k = 50 with same training recipe for k = 11.
Env
GridMaze-17 GridMaze-19 2ridMaze-26 GridMaze-32
GridRoom- I
Mean
0,9435
0.9594
0.8226 0.0150
0,6536
0,9532
Std
0.0111
0.0128
0.0370
0.0090
Env
GridRoom-4 GridRoom-16 16 GridRoom-32 GridRoom-64 GridRoomSym-4
Mean 0.8697 0.0045
0.9191
0.7433
0.6985
0.8107
Std
0.0015
0.0219
0.0350
0.0041
C.2.1 Experimental Setup for 2D Hydrogen Atom
Exactly same configurations were used for both the Sanger variant and LoRAgaq with sequential nesting. Since the configuration is almost same as [43] Appendix E.1.I], we note the setup succinctly, and refer an interested reader to therein for the detailed setup.
Data generation:
We opted to Gaussian distribution (O, 1612) sampling distri- bution, and generated new minibatch sample of size 512.
Architecture: We used 16 separate fully connected neural networks with hidden layers of 128 units with the softplus activation. We also used the multi-scale Fourier features [47].
Optimization: We trained the neural networks for 10 iterations. To enable a direct comparison between OMM and LoRA under consistent computational settings, used five times fewer iterations than in 43 While longer training may improve performance, our choice facilitates controlled evaluation of relative efficacy. We note that we used Adam optimizer
22 with learning rate 10-4 with the cosine learning rate scheduler, instead of RMSprop [17.. which we found perform worse than Adam. We multiplied the operator by 100, but did not shift it by a multiple of identity.
Performance metric: Following [43], we report the relative errors eigenvalue estimates, angle distances for each mode (also similar to the Laplacian representation learning for RL), and subspace distances within each degenerate subspace. For each metric, we followed the same procedure defined in 43 We ran 10 different training runs for each method and report average values with standard deviations.
The numerical comparisons are summarized Figure quickly without any additional identity shift, as we alluded to earlier. The Sanger variant is comparable to or sometimes even better than LoRAwe (i.e., NeuralSVD..a) in terms of different metrics. Since LoRAgeg was the strongest method from (43), this result suggests that OMM can be also alternatively
4 First, we note that the OMMsaq diverged
20

---

Relative entors eigervaiues
Angle distances
didancee
1000 orarN
26
12-4
10
Serger
12
10m4
Figure 4: Summary of the 2D hydrogen experiment Figure deviations.
The shaded region indicates = standard
used in place LoRA when we wish to decompose a PSD operator. alluded earlier, however, OMM cannot deal with unbounded differential operators (such as harmonic oscillators) inherently, while LoRA is capable of that.
C.2.2 Results on Unbounded-Spectee Case
In this section, we present numerical results for Schrödinger equations with unbounded spectra with the inverse-operator trick.
2D Harmonic Oscillator. When V(x) = |x|| in the Schrödinger, it is called the quantum harmonic oscillator. This another classical textbook example in quantum mechanies [1]. For simplicity, we consider the wwo-dimensional case. Similar to the 2D hydrogen atom case, we closely followed the setting in 443 except the followings.
Data generation: We used a Gaussian distribution A(0, 412) as sampling distribution with batch size 512.
Architecture: We used k = 28 disjoint neural networks essentially same as those used for the 2D hydrogen atom. On top of that, we applied the Dirichlet boundary mask 44
the bounded box [-10, 10, 10)2, to ensure that the parametric eigenfunctions vanish outside the box. We found that this is essential to enable successful training.
Optimization: We trained for
10 iterations. We used Adam optimizer [22 with learning rate 10-4 and the cosine learning rate scheduler. We note that RMSProp did not lead successful training in this case.
The results are shown in Figures 5 and 6 Figure visualizes the top-15 learned eigenfunctions. After applying subspace alignment via the orthogonal Procrustes procedure for fair comparison, the learned eigenfunctions closely match the ground-trutt solutions. The quantitative results in further confirm that the corresponding eigenvalues are accurately recovered.
2D Infinite Well. Yet another simple example is the infinite-well problem, where V(x) 0 for x E and (xx) = oo otherwise for some domain Q. In this case, the stationary Schrödinger equation reduces to the Laplace equation on f2 with zero Dirichlet boundary conditions. We consider the two-dimensional problem with ! [-L, L)2 where = 5. The eigenfunctions can be explicitly written
1
TT (x L)
sin nly T + +L)
mnn.. (x,, v)
sin
(10)
2T.
2L.
with eigenvalues
πT²
Ang.
(m² + n
(11)
4L2
Here, (nz, nyy) N are the quantum numbers.
Data generation: We used uniform distribution for sampling with batch size 512. We used A 15 disjoint neural networks without random Fourier features, but with the Dirichlet boundary mask.
Architecture:
Optimization: We trained for x 103 iterations. We used Adam optimizer 22 with learning rate 10 without any learning rate scheduler. RMSProp also not effective
21

---

OMM
OMM
(nieed))
tre
Figure 5: Visualization of 15 learned eigenfunctions with OMMas on 2D harmonic oscillator.
The first row shows the raw learned parametric cigenfunctions. The second row presents the eigenfunctions aligned to the mound-truth degenerate subspaces via the orthogonal Procrustes procedure as instructed in [43 The third row shows the ground ground-truth eigenfunctions.
Eigenvalues
Anglee distances
Subspace distances
10
10
Ground
12-1
10
10
Figure 6: Summary of the 2D harmonic oscillator experiment in Figure The shaded region indicates +
standard deviations.
in this Also, we found that the autograd implementation is very crucial with the Laplacian operator in the Hamiltonian, while the fiiit-ddifference-based approximation of the Laplacian, which proven work for other experiments and in [43) leads to slow convergence. We conjecture that this behavior due to the operator-dependent parameterization in the operator-inverse trick, which could amplify the approximation gap of the operator, if there is any.
The results are shown in Figures 7 and 8 While the training process can be sensitive certain hyper- parameter configurations, the learned eigenfunctions exhibit high quality once properly optimized. Such sensitivity is a common issue in deep learning practice, though it may make the OMM less appealing in comparison, as the LoRA approach is considerably more straightforward to apply.
C.3 Self-Supervised Contrastive Representation Learning
In this section, we describe the experimental setup for the image experiment in the main text, as well as an additional graph experiment.
C.3.1 Representation Learning for Images
For this experiment, we used the solo- learn codebase of da Costa et al.
Data generation: We used the default data augmentations for CIFAR-100 in the codebase.
The exact configurations to reproduce the results will be shared upon acceptance.
backbone model and adopted two different feature encoding strategies: (1) we used a nonlinear projector of shape Linear(featur _ituure__im,,__- (o6.8)-BatchNoomaD-- -ew.mmm---------aaeeeee- 2048, 2048)-BatchNorm.D RRLLU-Linear(20)) 2554; (2) similar to DirectCLR (2I), we removed the projector and simply train the top 64 dimensions of the ResNet-18 feature the top-k eigenfunctions using the OMM objective. In both cases, each feature vector is normalized by its l norm following the standard convention.
Architecture: We used ResNet-18 [16]
"Githb repository: http:://Ekhu...cw/wuurrrss/ooooo
22

---

OMM
OMM
(seed))
tre
Figure 7: Visualization of 15 learned eigenfunctions with OMMssq on 2D harmonic oscillator.
The first row shows the raw learned parametric eigenfunctions. The second row presents the eigenfunctions aligned to the ground-truth degenerate subspaces via the orthogonal Procrustes procedure instructed in 43 The third row shows the ground ground-truth eigenfunctions.
Eigenvalues
Aiggle
Subspace distancess
25
16
10
2.
10t4
10
Goundtrih OVN..
10
6
10
12
14
30
32
34
5
6
agenstree niee
eipestate intex
subquace InDex
Figure 8: Summary of the 2D harmonic oscillator experiment in Figure standard deviations.
The shaded region indicates +
We used the default optimization configuration of the codebase R We use the LARS optimizer (56) with weight decay set to 0, initial learning rate of 0.3 governed by cosine decay schedule, batch size 256, and 1000 epochs.
Optimization:
Evaluation: We evaluate the representation based on the linear probe accuracy on the test split, trained by SGD with learning 0.1, batch size 256, and 100 epochs.
C.3.2
Representation Learning with Graph Data
We can apply the orbital minimization principle to graph data. Suppose that we are given an adjacency matrix A R. N where Ag encodes the connectivity between nodes and j. the spectral graph theory [6). it is well known that the lowest cigenvectors of (symmetrically normalized) graph Laplacian Laym := D 1-12A 11/2 encodes important properties of the underlying graph, and it is also well known that the spectrum is bounded within [0, 2].
When the graph is large and when each node E [N] associated with a feature vector x,, computing the eigenvectors numerically might be cumbersome and extrapolation to points is nontrivial. Hence, this case, it is natural to learn eigenfunctions f(x) of the graph Laplacian as a function of the feature vector In this section, show the applicability of OMM in this scenario, and assess the quality of the learned eigenfunctions the graph Laplacian in the node classification task.
We closely followed the experimental setup in the Neural Eigenmaps paper 66. implementing our code based on the codebase of Deng et al. (6] Neural Eigenmaps was proposed as method find eigenfunctions of PSD operator similar OMM, but it is regularization-bass approach and thus does not have the sharp global optimality that OMM enjoys. Moreover, practitioners also need to tune the regularization parameter in the framework, while OMM is hyperparameter-ffee.
Data: We used the sebnn-produce dataset [1]] where the feature vector has 100 dimensions and the classification task has 47 classes. benchmark, and the accompanied graph consists of 2,449,029 nodes and 61,859, edges. The density of this graph 2.06 x 10 1 suggesting that the underlying graph is extremely sparse.
a large-scale node property prediction
'We refer reader to configuration SimCLR pretraining:
https://giitub..co///urrrs
solo- sctel//amm/hbbb/aaaa///pssprrrrr "Github repository:
23

---

Architecture: We parameterize the eigenfunctions by an 11-layer MLP encoder with width of 2048 and residual connections [16], followed by a projector of the same architecture as in the image experiment. We used 8192-4096 hidden units for OMM and LoRA, and 8192-8192 for Neural Eigenmaps. We note that, with OMM, we did not require the feature normalization by Z--norm. In contrast, Neural Eigenmaps quickly diverged without the 2--norm normalization and thus used the normalization so that the feature has f-rnr 10. Even worse, we also trained model with LoRA 443 (i.e., NeuralSVD without nesting), but the training dynamics were unstable and diverged regardless of Ez-normalization.
Optimization: Training was conducted over 20 epochs on the full set of nodes using the LARS optimizer [56] with batch size of 16384, weight decay set to O, and an initial learning rate of 0.3 with a cosine learning rate scheduler. We used the default hyperparameter a = 0.3 for Neural Eigenmaps suggested in the paper [6], which selected based linear probe accuracy on the validation set.
Evaluation: Similar to the image experiment, we evaluate the representation based on the linear probe accuracy the test split, trained by SGD with learning rate 0.01 and weight decay 10 batch size 256, and 100 epochs. We consider two evaluation strategies. The first is train linear classifier directly on the original training labels. The second follows the Correct & Smooth (C&S) method of Huang et al. [19), which enhances node classification by first correcting the training labels using a graph-based error estimation and then smoothing the corrected labels via feature propagation. This procedure produces refined supervision signal for training, often leading improved downstream performance. We used the default configuration in the Neural Eigenmaps codebase for C&S.
Results. We summarize our result Table
First, unlike the standard image contrastive representation learning setting and Neural Eigenmaps, we found that OMM was capable of training the final embedding trained to fit the eigenfunctions to become highly performant on the classification task. That is, remarkably, the linear probe performance from the embedding is better than the intermediate feature, which is the output of the MLP. We note that the drastic performance drop in Neural Eigenmaps from ~ 74% (representation) to ~ 50% (embedding) a typical behavior. This implies that while Neural Eigenmaps might provide sufficient signal for the intermediate feature to capture relevant information about each node, the embedding might not be truly trained fit the underlying eigenfunctions and thus provides worse discriminative power. On the other hand, the good classification performance of embedding (even better than representation) of OMM suggests that the OMM objective may behave better than competitors in the context of capturing true eigenfunctions.
Second, we observe that the C&S postprocessing boosts the classification accuracy for all cases to be relatively close. Nonetheless, even after the application of C&S, we find that the performance of the OMM embedding is clearly the best.
Table 3: Summary of OGBN- CGBN-products experiment (%). The model was trained once for each method, but the linear probe were trained for 10 different times for each case. The + 's indicate the standard deviations. "Representation" refers the lincar probe accuracy based on the output of the MLP backbone, and "Embedding" refers to that based the output of the projector, which is trained to fit the underlying eigenfunctions. The LoRA objective failed to yield convergent training dynamics.
Finetuning
Correct & Smooth [19
representation
embedding
representation
embedding
LoRA 43
N/A 74.05 73.664128
N/A 50.76 20.7 74.17
N/A 82.40 82.06+10s
N/A 80.90
Neural Eigenmaps 6
OMM (ours)
84.11
24

---

D On the Benefit of Higher-Order OMM
In the self-supervised image representation learning experiment, we observe sharp increase in the downstream task performance by using the OMM-2 objective Lomm (V), and even better by using the mixed objective Comm (V) Comm (V). In this section, we provide theoretical argument on the practical benefit of the higher-order OMM based on gradient analysis.
We analyze the gradient for the finiteddimensional case for simplicity, but the same argument is readily extended to the function case. Let R := la VVT. Then, we can show, by chain rule, that the gradient of the OMM-p objective is
Vv (V) Vvtr((la (VT)
-RRt(R""A)VVRR
R'AR R'AR
-2
- A R2 PAR
R AR
For the purpose of our analysis, we restrict our attention to the case where VVT idempotent,
ie., R² = R. Then the gradient expression simplifies to
Vv (V) 2(RA + AR + (2p 2)RAR)V
Vv C())
omm (V) - 4(p - 1)RAR
where we have the base p = case of
(1) (V) = 2(RA + AR]V Vv
Hence, we consider the Frobenius norm of the gradient,
Vv (V = VV (V) + +A,
where we let
A = 16(p - 1) - -(pp - 1itp-1ir((V)) (i--it(VVcc..)) (V)TTm(())RRR))) (V)
This shows that when p > sufficiently large, the gradient norm can be made strictly larger than the norm of Cy (V). This can improve convergence speed when convergence, especially when || VTVI|F is close to O, since the gradient norm the original OMM gradient || ( [V) becomes small proportional to || VTV||E. In practical optimization, the flat minima may cause immature convergence, and the additional gradient signal from the OMM with p > 1 can help escape the flat minima.
25

---

NeurIPS Paper Checklist
1. Claims
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Question:
Answer: [Yes]
Justification: The main claim of this paper is to posit the orbital minimization method in modern machine learning literature and assess its applicability. The abstract and introduction accurately reflect this goal.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: We clearly discuss the limitations of the orbital minimization method and parametric spectral decomposition methods in general in the concluding remarks.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "I "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and prool's
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
26

---

Answer: [Yes]
Justification:
We provide all proofs of the theoretical claims in the paper in the Appendix.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We include all the information to reproduce the results reported in the paper including the Py PyTorch code snippet for the main objective. We will disclose the code and data upon acceptance.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reprodueibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results. 5. Open access to data and code
27

---

reproducible, they Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification:
We open-source the code and data online httpp://gitthb. com/
jongharyu/neural-sv
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.cc/
for more details.
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: The paper has all the details for reproducing experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: We include error bars the first and second experiments, but do not include for the third experiment due to the large amount of computation required.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
28

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: We provide the amount of compute resources we used for experiments in Appendix.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't I make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics httppp////errrrg...oooppo/tttttt
Answer: [Yes]
Justification: The scope of this paper is mostly theoretical and its application as a generic tool, and there is no component of potentially violating the code of ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative socictal impacts of the work performed?
Answer: [NA]
Justification: There is no societal impact of the work performed we foresee, as the con- tribution is rather purely theoretical and methodological in accurately estimating spectral properties of operator.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
29

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), faimess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [Yes]
Justification: The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but encourage authors to take this into account and make best. faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We cite all the original creators and owners of assets used in this paper for their credit.
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include a URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
30

---

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets. paperswi thcode. wwt.coom/aaaa. datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
We do not introduce new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmode as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: This paper does involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
31

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology. scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: LLMs were not used affect the core method development in this paper.
Guidelines:
The answer NA means that the method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy (ttess///eerriscccc/oofeeeeeeess for what should or should not be deseribed.
32


Reviews:
[{"review_id": "WmBUEWroHF", "reviewer": "Reviewer_9r7G", "summary": "The paper includes three experiments:\n1) Successor (Laplacian) representation in grid environments, which both ALLO and OMM can solve almost perfectly, unless when the spectral gap is extremely small.\n2) Solve the 2D hydrogen atom Schrödinger equation, that a Sanger variant of OMM solves almost exactly, like NeuralSVD.\n3) Contrastive representation learning (by learning eigenfunction of the canonical dependence kernel of random views) on CIFAR-100, achieving ~64% top-1 accuracy with a higher-order variant (OMM p=2), compared to SimCLR's ~66.5%", "questions": "- Can the authors provide experiments where  OMM has advantages over existing methods? Or argue for which properties a problem should have for OMM to be advantageous?\n- What is the computational cost of OMM compared to other methods, especially with p=1 or higher-order variants?\n\nMinor notes:\n- There seems to be a small typo in line 62 in $\\mathcal{R}^{d_1}$\n- Add to the caption of figure 2: what is depicted (solution of the H2 Schroedinger equation) and what to take away from it (that both learn the same true representation?)", "limitations": "yes", "rating": 3, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strengths:\n- The paper is clearly written, and could serve as a useful resource for anyone who wants to explore OMM\n- The paper transparently discusses challenges in application and optimization (non-PSD with OMM are only possible if the eigenvalues can be bounded, numerical instability with near-zero eigenvalues, p=1 vs higher-order)\n\n\nWeaknesses:\nI understand the goal of the paper is to motivate, rather than beating baselines. Nevertheless, I currently I do not find it very motivating:\n- The experiments do not show that OMM has any advantages over the baseline\n    - The successor representation learning on GridMaze and GridRoom seems saturated. Both OMM and the baseline perform equally well, with a close to perfect cosine similarity of one, except for poor performance on few environments where the spectral gap is extremely small.\n    - Similarly the 2D hydrogen atom seems to be perfectly solved by the baseline. Does OMM have an advantage for other Hamiltonians?\n\n- From the discussion it is not clear to me when OMM would be useful. The only advantage seems to be that the baselines lack theoretical convergence guarantees?", "was_revised": true, "final_justification": "I raise my overall score to 3, and the significance rating to 3. I slightly recommend rejection as I do not find the advantages of OMM apparent from the theory or experiments, but remain low confidence as I am not very familiar with the matter, specifically the use of OMM in quantum chemistry.", "rebuttal": "We appreciate the reviewer's time and effort in evaluating our manuscript. The primary concern raised appears to be that the advantages of OMM over existing methodologies are not sufficiently clear. Below, we clarify the theoretical and practical merits of OMM in comparison to existing methods, and explain why it warrants further attention from the community. We hope this will allow the reviewer to reassess the contribution of our work, and we will carefully update our manuscript accordingly.\n\n---\n\n1. **On Theoretical Merits**\n\nBefore addressing the concerns related to experiments, we emphasize why repositioning OMM represents a significant contribution. Although numerical linear algebra is a well-established field, researchers in diverse areas (such as computational physics and chemistry for solving linear PDEs, statistics for correlation analysis, theoretical computer science for streaming PCA/CCA, and machine learning for representation learning) have often developed their own computational methods. These methods are sometimes tailored to specific types of matrices or operators relevant to their domains, and sometimes these methods can be more generic and broadly applicable in disguise. In many cases, however, such methods inadvertently rediscover techniques already developed in other fields, often in more primitive or suboptimal forms. \n\nAs a particular example developed in the field of density functional theory, OMM has been substantially studied in specialized cases, with some highlights by numerical linear algebraists (see, e.g., \\[9\\], \\[29\\]). **However, the idea has never been explicitly considered as a generic tool for spectral decomposition of PSD operators**, which may result in efficient algorithms for various problems including streaming PCA, learning representation for RL, and solving PDEs, as we highlight in the manuscript. While we acknowledge that our experimental results do not outperform some existing baselines, the experiments show competitive, if not outperforming,  performance to the state of the art methods developed for such problems. \n\nBelow, we highlight the theoretical and conceptual merits we bring by repositioning the OMM.\n\n- `Within streaming PCA literature`: As we alluded to in Section 2.2.1, the literature for streaming PCA has devoted subsequent effort in trying to develop an efficient algorithm that learns top-k “orthogonal” eigenvectors. To our understanding, however, the literature unfortunately has narrowly focused on Oja’s algorithm and its variants or constrained optimization framework such as Rayleigh quotient (RQ) maximization (in Eq. (1)), both of which require explicit handling of orthogonality constraints. This inherently complicates their analysis and limits applicability to more high-dimensional problems. One recent example is EigenGame \\[11\\], which won a best paper award from ICLR 2021; however, EigenGame still requires explicit normalization, derived from the RQ maximization framework.\n\n    Given this, we found it noteworthy that OMM can result in a simpler and more efficient (i.e., without explicitly handling orthogonality constraint) algorithm. Thanks to the implicit orthogonalization embedded in OMM, this can potentially lead to more scalable spectral algorithms for large-scale, high-dimensional problems. We note that LoRA has the same property and benefit (see Section 2.4), and its generic modern application was studied in \\[41\\].\n\n- `Within representation learning for RL literature`: In RL, there exists a rather independent literature for the “successor representation learning” framework. Their literature stems from the graph drawing algorithm (GDO), which essentially aims to solve the RQ maximization in Eq. (1), where the orthogonality constraint is handled via a Lagrangian multiplier. It was later generalized to the “generalized GDO (GGDO)” to learn top-k eigenvectors in the order of eigenvalues. ALLO \\[13\\] was proposed to improve GGDO, removing the need of tuning the Lagrangian multiplier in GDO/GGDO, by the augmented Lagrangian method. We emphasize, however, that (1) ALLO needs to solve max-min problem which require tuning two separate learning rate scales, and (2) ALLO still has a hyperparameter to be tuned, the barrier parameter $b$, which is mentioned in the paper that they proposed to increase it in a gradient ascent fashion as a heuristic. In contrast, OMM is a simple minimization framework, and does not have any parameters that must be tuned or handled heuristically. In practice, the parameter-free nature allows a much easier implementation, which implies a wider applicability of OMM than ALLO. In direct contrast, our experiment demonstrates that OMM can perform on par with ALLO, even with a much simpler idea and implementation. \n\n    - **Additional ablation study on ALLO**: To better highlight the issue of the hyperparameter tuning, we performed an additional ablation study for the RL experiment, by increasing d=11 to d=50. In summary, we demonstrate that the optimal hyperparameters of ALLO (the initial value of $b$ and the learning rate to update $b$) tuned for d=11 result in poor performance for d=50; this suggests that ALLO requires expensive hyperparameter tuning for not just for new problem instances, but even for different numbers of modes to be retrieved. In contrast, OMM was able to successfully train the network with the same optimization configuration for d=11. This provides a strong case for why OMM may be preferred over ALLO in general. We will include this new ablation study in our revision.\n\n2. **Questions**\n\n- `Can the authors provide experiments where OMM has advantages over existing methods? Or argue for which properties a problem should have for OMM to be advantageous?` \n\n    As explained above, we demonstrate how OMM performs comparably to ALLO, without any additional technique such as the augmented Lagrangian. In the 2D hydrogen example, we demonstrate how OMM performs comparably to LoRA (also known as NeuralSVD, see [41]), which was shown in \\[41\\] to outperform existing methods such as spectral inference networks \\[A\\] and NeuralEF \\[B\\]. \n\n    The reviewer might question then when OMM may stand out compared to LoRA. As alluded to in line 239, we include an experiment in Appendix C.3.2 to demonstrate when LoRA can miserably fail, while OMM can still learn good representation. In a nutshell, LoRA may not be able to learn eigenfunctions when the target matrix/operator to decompose is derived from a “sparse” kernel. We explain the reasoning behind this failure mode below.\n\n    - Consider an operator defined by a kernel. Recall the objectives of OMM and LoRA after line 227, and note that the term $\\\\mathrm{tr}(\\\\mathsf{M}\\_{\\\\rho}\\[\\\\mathbf{f},\\\\mathcal{T}\\\\mathbf{f}\\])=\\\\mathbb{E}\\_{\\\\rho(\\\\mathbf{x})\\\\rho(\\\\mathbf{x}')}\\[\\\\mathbf{f}(\\\\mathbf{x})^\\\\intercal k(\\\\mathbf{x},\\\\mathbf{x'}) \\\\mathbf{f}(\\\\mathbf{x}')\\]$ appears only in the first term of LoRA objective. Given this, suppose that the kernel is extremely sparse, i.e., $k(\\\\mathbf{x},\\\\mathbf{x}')\\\\approx 0$ with high probability for $\\\\mathbf{x},\\\\mathbf{x}’\\\\sim p(\\\\mathbf{x})p(\\\\mathbf{x}')$. (This can happen when the underlying matrix is extremely sparse, like a graph Laplacian matrix of a real-world graph.) In such cases, if we perform stochastic gradient updates with minibatch samples, the optimization signal from the LoRA objective will only be based on the second term, which only encourages orthogonality. By contrast, OMM can still learn useful signals in this scenario, as the term $\\\\mathrm{tr}(\\\\mathsf{M}\\_{\\\\rho}[\\\\mathbf{f},\\\\mathcal{T}\\\\mathbf{f}])$ appears both terms in the OMM objective and thus the scales of the two terms are “balanced”. \n\n    - We demonstrate this case for a real-world graph Laplacian in Appendix C.3.2, and evaluate the quality of representations in the downstream task of node classification. In this example, we observe that LoRA failed to converge during training, while OMM demonstrates decent linear probe performance on the task. It even outperforms one baseline (Neural Eigenmaps), demonstrating the outstanding discriminative performance of learned eigenfunctions (marked as “embedding”), whereas the learned eigenfunctions in Neural Eigenmaps do not show good discriminative power and thus they had to resort to the intermediate layer representation (marked as “representation”).   \n    \n    We believe that this demonstrates the superiority of OMM against the existing parametric spectral solvers such as NeuralSVD (based on LoRA) and Neural Eigenmaps.\n\n    ```\n    [A] Pfau, David, et al. \"Spectral inference networks: Unifying deep and spectral learning.\" ICLR, 2019.  \n    [B] Deng, Zhijie, Jiaxin Shi, and Jun Zhu. \"NeuralEF: Deconstructing kernels by deep neural networks.\" ICML, 2022.\n    ```\n- `What is the computational cost of OMM compared to other methods, especially with p=1 or higher-order variants?`\n\n    In terms of complexity in computing a given objective, OMM’s complexity is similar to ALLO and LoRA when $p=1$, where the dominant factor is in computing the empirical moment matrices, which takes $O(BK^2)$ complexity for a given minibatch of size $B$ and for $K$ eigenfunctions. When $p\\>1$, the complexity becomes $O(BK^2+pK^3)$, as we need to explicitly compute the matrix powers in Eq. (7). \n\n3. **On Minor Notes**\n\nWe appreciate the detailed suggestions after careful reading. We will revise the paper accompanying the comments.\n\n---\n\nIf the responses provided address the reviewer's concerns, we would appreciate reconsideration of the score, as we believe the paper makes significant contributions to the field."}, {"review_id": "9TwEsgpRJh", "reviewer": "Reviewer_YTWU", "summary": "The authors extend the orbital minimization method (OMM) to the decomposition of positive semi-definite (psd) operators and show its applications to a range of benchmark tasks. The extension of OMM from matrices to operators is done elegantly, and the authors contextualize their work very well.", "questions": "See weaknesses above. Could the authors address this both in the response and in the text?\n\nI'd be particularly curious in the case of the (negative) Laplacian on a 2D domain with some boundary conditions, where we know the true eigenfunctions (for [-1,1]^2, for instance). If you recovered the eigenfunctions using operator OMM, what accuracy would those have after sampling on a regular grid and comparing to the eigenvectors computed from, say, a p-th order finite difference discretization of that Laplacian?\n\nI'm willing to significantly increase my score based on responses.", "limitations": "The authors do not have an explicit limitations statement. Basically, the work needs more intuitive experimental evidence that can allow sci comp and SciML practitioners to decide tradeoffs.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 4, "originality": 4, "strengths_and_weaknesses": "Strengths:\n\n1. The paper is incredibly well-written, the language among the clearest I've seen, and the contexualization in the body of literature is fantastic.\n2. Operator OMM will likely have many applications in scientific computing and scientific machine learning.\n3. Connecting iterative PCA updates to gradient descent for OMM is also a very scientifically novel idea.\n\nWeaknesses:\nDespite the clarity, the authors skip over a few details without citations or exposition. I would have also liked to see more SciML or sci comp friendly experiments.\n\n1. What is the partially stop-gradiented second moment matrix, as in, why is this matrix called that? \n2. What is the object actually being parametrized by a neural network in the operator OMM? Is it the eigenfunction? If so, the paper needs a clear step where the operator OMM goes from continuous to discrete, with all the parametrizations and sampling details clearly written down (before we get to the results).\n3. Does a discretized operator OMM on a continuous operator recover a particular discrete OMM result for some discretization of that operator? Is there an error estimate or perhaps a numerical guidance rule?\n4. I'd be curious to see how this recovers eigenfunctions of the Laplacian Beltrami operator or of positive-definite reproducing kernels. These would be important examples for sci comp and SciML. \n5. A more basic benchmark was definitely needed! For instance, in the quantum mechanics case, connecting to the Laplace-Beltrami operator and showing the recovery of orbitals for the hydrogren atom.", "was_revised": true, "final_justification": "The authors addressed my questions and have run the additional experiments I requested, specifically about error estimates, numerical guidance, the transition from continuous to discrete, and the recovery of known eigenvalues and eigenvectors.", "rebuttal": "We appreciate the reviewer’s effort in reviewing our paper and providing constructive feedback. We will incorporate the suggestions in our revision to improve the clarity and better demonstrate the significance of our work.\nBelow, we provide answers and clarifications to the points in Weaknesses / Questions.\n\n---\n1. `What is the partially stop-gradiented second moment matrix, as in, why is this matrix called that?`  \nWe acknowledge that the explanation around the implementation of sequential nesting may be too terse. We will revise the paragraph to improve readability by adding the following explanation.\n\n    In a nutshell, the partially stop-gradiented second matrix is introduced to efficiently implement the gradients of sequential nesting via autograd, unlike the custom gradient implementation of sequential nesting for LoRA in \\[41\\]. As explained in the finite-dimensional case, the idea of sequential nesting is to update the $k$-th eigenfunction $f\\_k$ by the gradient $\\\\partial\\_{f\\_k} \\\\mathcal{L}\\_{\\\\mathsf{orbital}}(\\\\mathbf{f}\\_{1:k})$ for each $k=1,\\\\ldots,K$.   \n    To implement these gradients succinctly via autograd, we introduce the “partially stop-gradiented” moment matrix as in after line 205 and define a surrogate objective function for sequential nesting as in after line 204. Then, we can formally show that $$\\\\partial\\_{f\\_1,\\\\ldots,f\\_K}\\\\mathcal{L}\\_{\\\\mathsf{orbital}}^{\\\\mathsf{seq}}(\\\\mathbf{f}\\_{1:K})=\\\\begin{bmatrix}   \\\\partial\\_{f\\_1} \\\\mathcal{L}\\_{\\\\mathsf{orbital}}(f\\_{1})\\\\\\\\  \\\\vdots\\\\\\\\  \\\\partial\\_{f\\_K} \\\\mathcal{L}\\_{\\\\mathsf{orbital}}(\\\\mathbf{f}\\_{1:K})  \\\\end{bmatrix},$$ which implies that we can simply compute this surrogate objective function to update a model using the gradients from sequential nesting. Note that the surrogate loss is equivalent to the OMM objective (with p=1) in its nominal value, but the gradients are different due to the stop gradient operations.  \n    We call the matrix “partially stop-gradiented”, as we apply stop-gradient operations “partially” to the definition of the second moment matrix to implement the sequential nesting’s gradient.\n\n2. `What is the object actually being parametrized by a neural network in the operator OMM? Is it the eigenfunction? If so, the paper needs a clear step where the operator OMM goes from continuous to discrete, with all the parametrizations and sampling details clearly written down (before we get to the results).`\n\n    For the operator case, a neural network is meant to parameterize the top-K eigensubspace (or the top-K eigenfunctions when nesting is applied). We agree that the exact parameterization of the eigenfunctions are of great importance in SciML, and we will revise the manuscript to clarify them in the main text. Currently all the detailed settings for the experiments can be found in Appendix C. \n\n    `Regarding “when the operator OMM goes from continuous to discrete”`:  We understand the motivation behind this question, and here is how we can understand the transition. \n    - In the ML setup, where we are given a finite (yet possibly large) dataset of size $N$, the “discretized” matrix of size $N\\\\times N$ is readily (but implicitly) defined, and there exists a full batch version of the objective function we wish to optimize. When we optimize it in practice, in the modern ML paradigm, we implement using minibatch, only looking at its submatrix at a time.  \n    - In the SciML setup, there is a delicate difference. Consider solving a linear PDE over a given domain. In this case, instead of discretizing the domain by a fixed grid, we can freely sample from the domain according to a certain probability measure to optimize the population OMM objective in a minibatch sense. In this case, there is no “prior” discretization, and we update the model using freshly sampled data points from the domain for every minibatch. Since any practical optimization proceeds over a finite number of iterations, we may view the set of all sampled points encountered during training as inducing a discretized version of the operator, which the model effectively decomposes. \n\n3. `Does a discretized operator OMM on a continuous operator recover a particular discrete OMM result for some discretization of that operator? Is there an error estimate or perhaps a numerical guidance rule?`\n\n    We believe the first part of this question is addressed in the previous item. The second part is particularly relevant in the SciML setup, where we have the flexibility to choose the sampling distribution. See Appendix D.3 of \\[41\\] for a discussion on the importance sampling variant of the LoRA objective, which applies directly to OMM. A high-level rule of thumb in choosing a good sampling distribution is that it is beneficial to choose a distribution that has more probability mass around where the true eigenfunctions have non-zero values. However, since this requires a prior knowledge of the eigenfunctions to some extent, this does not always lead to a practical numerical guidance. We believe that, as the reviewer implied, a principled error estimate or numerical rule to guide optimization would be highly desirable to make this approach more applicable, and we leave this as a future research direction.\n\n4. `Eigenfunctions of the Laplacian Beltrami operator or of positive-definite reproducing kernels? More examples?`\n\n    We acknowledge that having more examples for PDEs will further strengthen the contribution of the paper. In short, we performed additional experiments on decomposing the 2D Laplacian operator over a bounded domain as well as 2D quantum harmonic oscillators with OMM, and were able to recover the underlying eigenfunctions to high accuracy. Below, we explain a bit of details and additional technicalities in this experiment, and we will revise the manuscript to highlight and strengthen this contribution.\n\n    - The reviewer particularly asked about solving a Laplacian over a 2D square, which is essentially a Schrodinger equation for square infinite-well potential if the boundary condition is of Dirichlet type. In this case, we have analytical solutions to the underlying eigenvalues, which is characterized by two quantum numbers $n\\_x,n\\_y\\\\in \\\\mathbb{N}$ as $\\\\lambda\\_{n\\_x,n\\_y}\\\\propto n\\_x^2+n\\_y^2$. At first glance, as the eigenvalues are increasing and diverging to infinity as $n\\_x,n\\_y\\\\to\\\\infty$, it seems that we cannot apply the OMM as discussed in the main text, since we cannot negate the operator and shift it, as we cannot avoid negative parts. \n\n    - To address the issue in such operators, we propose to apply the “inverse operator” trick as follows. Let $\\\\mathcal{L}$ be a linear operator that has positive eigenvalues $0\\<\\\\lambda\\_1\\\\le \\\\lambda\\_2\\\\le \\\\ldots $. Since this implies the invertibility of the operator, we can consider decomposing the inverse $\\\\mathcal{L}^{-1}$, which has eigenvalues $\\\\lambda\\_1^{-1}\\\\ge \\\\lambda\\_2^{-1} \\\\ge \\\\ldots \\\\ge 0$, and the top-$k$ eigenfunction $\\\\psi\\_k$ corresponds to the bottom-$k$ eigenfunction of the original operator $\\\\mathcal{L}$. This implies that we can apply OMM to learn top-$k$ eigenfunctions of $\\\\mathcal{L}^{-1}$ to learn the bottom-$k$ eigenfunctions of $\\\\mathcal{L}$.\n\n    - A technical challenge is then that we may not know how to compute the inverse operator $\\\\mathcal{L}^{-1}$. For example, the inverse would correspond to an integration operator for the case of differential operators. To detour this challenge, we can set $\\\\mathbf{f}\\gets \\\\mathcal{L}\\\\mathbf{g}$, where $\\\\mathbf{g}$ is the function we parameterize by neural networks. After all, this yields the following objective function:  \n    $$  \\\\begin{aligned}  \\\\mathcal{L}\\_{\\\\mathsf{orbital}}^{\\\\mathsf{inv}}(\\\\mathbf{g};\\\\mathcal{L})  &\\\\triangleq \\-2\\\\mathrm{tr}(\\\\mathsf{M}\\_{\\\\rho}\\[\\\\mathbf{f},\\\\mathcal{L}^{-1}\\\\mathbf{f}\\]) \\+ \\\\mathrm{tr}(\\\\mathsf{M}\\_{\\\\rho}\\[\\\\mathbf{f}\\] \\\\mathsf{M}\\_{\\\\rho}\\[\\\\mathbf{f},\\\\mathcal{L}^{-1}\\\\mathbf{f}\\])\\\\\\\\  &=-2\\\\mathrm{tr}(\\\\mathsf{M}\\_{\\\\rho}\\[\\\\mathcal{L}\\\\mathbf{g},\\\\mathbf{g}\\]) \\+ \\\\mathrm{tr}(\\\\mathsf{M}\\_{\\\\rho}\\[\\\\mathcal{L}\\\\mathbf{g}\\] \\\\mathsf{M}\\_{\\\\rho}\\[\\\\mathcal{L}\\\\mathbf{g},\\\\mathbf{g}\\]).  \\\\end{aligned}  $$  \n    The computational complexity remains the same. We note that, as we parameterize the $k$-th eigenfunction of $\\\\mathcal{L}^{-1}$ by $\\\\mathcal{L}g\\_k$, at convergence, $\\\\mathcal{L}g\\_k$ would correspond to the normalized eigenfunction $\\\\phi\\_k$, and thus $g\\_k(\\\\mathbf{x})=\\\\frac{1}{\\\\lambda\\_k}\\\\phi\\_k(\\\\mathbf{x})$ should ideally hold. \n\n    - This enables OMM to be used with a broad class of operators whose spectra diverge, including the Laplacian and other elliptic operators. We applied this technique to the 2D infinite well and 2D harmonic oscillator problems which have analytical solutions, and observed that OMM can successfully recover the top \\~30 eigenfunctions accurately, to the similar order of what's reported for the 2D hydrogen atom. We will update the manuscript with these experiments. We will also add an ablation study for higher-dimensional versions of these basic problems to demonstrate the scalability.\n\n---\n\nIf these clarifications adequately address the reviewer’s concerns, we would greatly appreciate it if the reviewer could consider updating the score to reflect what we believe are meaningful contributions to the community. We would be happy to incorporate  any additional suggestions and comments, if any."}, {"review_id": "F8RpzApTE5", "reviewer": "Reviewer_piiF", "summary": "This paper breathes new life into the Orbital Minimization Method (OMM) -- a classical idea from quantum chemistry -- and repositions it as a powerful, modern tool for learning spectral representations with neural networks. The authors offer a clean and elegant re-derivation of OMM from first principles in linear algebra, showing that its objective naturally leads to orthogonal eigenvectors without any explicit constraints. This key property makes OMM uniquely well-suited for integration with deep learning. It also generalizes OMM to handle both finite-dimensional matrices and infinite-dimensional operators, and introduces neural OMM variants using a novel nesting technique to extract ordered eigenspaces. Across diverse domains -- from reinforcement learning (Laplacian-based representations), to quantum physics (solving Schrödinger equations), to self-supervised learning (representation learning from unlabeled data) -- the authors demonstrate that OMM is not only theoretically grounded but also practically robust. By unifying it with various existing spectral methods like streaming PCA and LoRA, the paper successfully convinces me that revisiting old ideas through a modern lens can yield surprisingly powerful solutions.", "questions": "Below are not suggestions to make changes, but something I, as a reader, would like to see more discussions with:\n\n1. Given that OMM minimizes a surrogate objective without explicit orthogonality constraints, does the optimization introduce any systematic bias toward specific eigenvectors or bases (e.g., favoring more localized or smoother modes), especially in high multiplicity eigenspaces?\n\n2. Can the higher-order OMM variants (p > 1) be interpreted through a statistical lens, such as variance regularization or higher-order moment matching?", "limitations": "N/A", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 4, "originality": 3, "strengths_and_weaknesses": "Overall, this is a high-quality, well-written paper that contributes both theoretical insight and practical methodology. I also think the work is important, mainly because it touches one of the foundations in ML -- eigenspace solving problem. In particular, it eliminates the need for orthogonality constraints (common in PCA/SVD-based methods). This work presents a method that integrates naturally with gradient-based deep learning, which enables scalable and stable eigenspace learning, even in operator or infinite-dimensional settings.\n\nWeakness are minor - \n1. although the core idea behind OMM was originally proposed in the 1990s within the context of quantum chemistry -- somewhat weakening the perceived algorithmic originality -- the authors’ new derivation (w/ simpler proof), reinterpretation, and adaptation to modern machine learning settings (esp. the nesting usage), bring genuine novelty and practical value. \n2. Experiments are solid but not exhaustive; more large-scale results or ablations could have strengthened the case.", "was_revised": true, "final_justification": "The authors have addressed all my questions, I am keeping my positive rating.", "rebuttal": "We appreciate the reviewer’s effort in reviewing our paper and asking thought-provoking questions. Below, we address the reviewer’s questions.\n\n---\n1. `Given that OMM minimizes a surrogate objective without explicit orthogonality constraints, does the optimization introduce any systematic bias toward specific eigenvectors or bases (e.g., favoring more localized or smoother modes), especially in high multiplicity eigenspaces?`\n    This is an interesting question! We observe that OMM does not appear to introduce systematic bias of this nature, and it is capable of fitting to the true eigenfunctions provided the neural network architecture is sufficiently expressive. That being said, we can observe “smoother” modes as the reviewer suggested if we restrict the capacity of the neural network, which is independent of the OMM objective. More concretely, consider a physical system (like the hydrogen example) where the first eigenfunctions correspond to the most stable configurations. In this case, higher modes have (spatially) higher frequency components, and thus we need more “expressive” neural networks to fit to higher modes. We empirically observe that by limiting the capacity of the network via decreasing the width from 128 to 32, we can see that the learned eigenfunctions are smoother than the true eigenfunctions, while the orthogonality is still approximately preserved due to the implicit orthogonalization induced by the OMM objective. We believe that this type of experiment can provide further insights on the choice of neural network architecture, and thus we will include a similar ablation study in the appendix.\n\n2. `Can the higher-order OMM variants (p > 1) be interpreted through a statistical lens, such as variance regularization or higher-order \nmoment matching?`\n\n    This is again a very interesting question. To our understanding, since higher-order OMM variants are based on powers of the second moment matrix, they do not directly encode higher-order statistical moments. The variance regularization perspective, if there is such an effect, would be pleasantly surprising; we leave such theoretical exploration as a future direction. \n\n    We wish to remark, however, that the higher-order version has a statistical drawback in practical optimization. Namely, to estimate its gradient in an unbiased manner, which is essential in scalable optimization, we need to use at least $p$ independent samples to handle the $p$-th power of the moment matrix. Given a finite minibatch size, splitting a fixed-size minibatch into $p$ subsets may increase the variance of each individual moment estimate. Therefore, for a finite batch setting, there might exist a sweet spot in the choice of optimal $p$. We also leave such an investigation to future work."}, {"review_id": "0DDIgk4ueQ", "reviewer": "Reviewer_cjx3", "summary": "This paper revisits the orbital minimization method (OMM), a classical optimization framework from computational chemistry originally developed in the 1990s, and adapts it for modern neural network-based spectral decomposition tasks. The authors provide a new theoretical derivation of the OMM objective using simple linear algebraic arguments, extending beyond the original domain-specific context to general positive semidefinite matrices and operators. They demonstrate the method's effectiveness across three application domains: Laplacian representation learning for reinforcement learning, solving Schrödinger equations, and self-supervised contrastive representation learning. To me atleast the key insight is that OMM offers an unconstrained optimization approach to eigenspace computation without requiring explicit orthogonalization, which I think makes it particularly appealing for neural network implementations.", "questions": "I have several questions that could help clarify the contribution and potentially improve my assessment:\n\n1. Numerical Stability: The authors mention stability issues with the direct OMM approach for the Schrödinger equation but don't provide a thorough analysis. Can you characterize when these stability problems occur and provide theoretical or empirical guidance on when practitioners should expect issues?\n2. Computational Complexity: How does the computational cost of OMM-based neural training compare to traditional eigensolvers or other spectral decomposition methods? The paper claims to avoid explicit orthogonalization, but what about the overall wall-clock time and memory requirements?\n3. Hyperparameter Sensitivity: The experiments show competitive performance, but there's limited discussion of hyperparameter sensitivity. How robust is the method to choices like learning rate, network architecture, and the order parameter p in higher-order OMM variants?\n4. Scalability: The experimental domains are relatively modest in scale. How do you expect the method to perform on very high-dimensional problems where traditional eigensolvers become prohibitive?", "limitations": "I think the authors do a reasonable job addressing limitations in their concluding remarks. They acknowledge that the experimental results in representation learning are preliminary and note the broader limitations of parametric spectral decomposition methods. However, I would have liked to see more discussion of the numerical stability issues and clearer guidance on when practitioners should consider using OMM versus alternatives. The restriction to PSD operators is appropriately highlighted, though I think more analysis of the spectrum shifting workaround would strengthen the paper.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 4, "strengths_and_weaknesses": "Strengths: The theoretical contribution is quite solid - I appreciate how the authors provide a clean, intuitive derivation of the OMM objective that doesn't rely on the somewhat obscure Neumann series arguments from the original computational chemistry literature. The connection they draw between OMM and other methods like streaming PCA and Sanger's algorithm is illuminating and helps position the work within the broader ML landscape. The experimental validation is reasonably comprehensive, spanning three quite different application domains. I think the authors did a good job showing that the method can work across diverse settings, from discrete grid environments to quantum mechanics to image representation learning. The comparison with established baselines like ALLO in the RL experiments and LoRA in the PDE setting provides useful context. The writing is generally clear and the paper is well-structured. The authors do a nice job motivating why one might want to revisit classical numerical methods in the context of modern ML, and they're appropriately honest about limitations.\n\nWeaknesses: The restriction to positive semidefinite operators is a significant limitation that constrains the method's applicability. While the authors acknowledge this and suggest spectrum shifting as a workaround, this feels somewhat ad-hoc and may affect the theoretical guarantees. I found some of the experimental results to be preliminary. In the representation learning experiments, the performance doesn't quite match SimCLR, and the authors seem to treat this as acceptable since it's not their main focus. However, this makes it harder to assess the practical value of the approach in this domain. The numerical stability issues mentioned in the Schrödinger equation experiments are concerning. The fact that they had to switch to the Sanger variant suggests that the core OMM method may have robustness problems that aren't fully characterized. The computational complexity analysis is somewhat lacking. While the authors mention that OMM avoids explicit orthogonalization, they don't provide a detailed comparison of computational costs versus traditional eigensolvers or other spectral methods.", "was_revised": true, "final_justification": "The authors' rebuttal successfully addressed my key concerns by providing clear explanations for numerical stability issues with practical solutions, adequate computational complexity context, and honest scalability assessments. While experimental results remain somewhat preliminary and the method is limited to PSD operators, these are appropriately acknowledged limitations. The solid theoretical contribution (clean OMM derivation, connections to existing methods) combined with comprehensive experimental validation across diverse domains outweighs the preliminary aspects, making this a valuable contribution that merits acceptance.", "rebuttal": "We appreciate the reviewer's time and effort in evaluating our paper and providing thoughtful feedback. We acknowledge that the concerns raised are valid and believe that addressing them will substantially enhance the quality of the manuscript. Below, we respond to each point individually.\n\n---\n1. `Numerical Stability: The authors mention stability issues with the direct OMM approach [...] Can you characterize when these stability problems occur and provide theoretical or empirical guidance on when practitioners should expect issues?`   \nWe appreciate the reviewer for raising this question. We agree that this warrants a better clarification, as we posited this as a fundamental flaw of OMM in the current writing. After we submitted our manuscript, we were able to identify that such instability can occur due to the statistical noise, if the underlying operator has diminishingly eigenvalues close to 0. (Recall that the Hamiltonian of the 2D hydrogen atom has such a property.) While the operator is PD, since we estimate the second moment matrices for step only with finite minibatch samples, there is a chance due to statistical noise and variance in estimation that the empirical moment matrix may falsely signal a negative eigenvalue. Preventing this failure mode can be done easily by adding a small amount of the “identity” shift to the target operator, such that the spectrum is strictly bounded away from zero. We empirically verified that this trick works effectively in practice. We will add a thorough ablation study on this matter in our revision.\n\n2. `Computational Complexity: How does the computational cost of OMM-based neural training compare to traditional eigensolvers or other spectral decomposition methods? The paper claims to avoid explicit orthogonalization, but what about the overall wall-clock time and memory requirements?`   \n    OMM has a similar computational complexity, as in computing the objective function, to LoRA and ALLO. A more critical question would be its comparison to the numerical eigensolver. In \\[35\\], where the authors advocated the LoRA objective for scientific computing and representation learning, they performed such an empirical analysis on computational complexity. As we demonstrated the comparable performance of OMM to LoRA in the example, the same conclusion can be deduced for OMM. While we borrow the discussion from \\[35\\] for now, we will add a similar yet more in-depth analysis on the comparison of OMM to numerical eigensolvers.\n\n    In \\[Appendix A.3, 35\\], the authors performed an empirical comparison of the computational complexity of the neural-network solution obtained by LoRA against one of the most representative, generic numerical eigensolver LOBPCG, for the 2D hydrogen atom problem.  They computed the eigenvalues of the Hamiltonian by increasing the grid size, and plot the accuracy of the eigenvalues and the wall-clock time. They showed that the accuracy of the numerical solver gets improved with increasing resolution, but the time complexity scaled cubically. Given this, they showed that LoRA+neural network solution achieved much higher accuracy with a shorter wall-clock time. While there exist some caveats in this comparison as noted therein (about, e.g., that such apple-to-apple comparison may not be fair and some structure of a problem at hand can be exploited), the scaling behavior of the naive application of the generic numerical eigensolver even for this simple problem clearly demonstrates the benefit of the parametric approach based on LoRA and OMM.\n\n3. `Hyperparameter Sensitivity: The experiments show competitive performance, but there's limited discussion of hyperparameter sensitivity. How robust is the method to choices like learning rate, network architecture, and the order parameter p in higher-order OMM variants?` \n\n     We acknowledge that we currently do not have any sensitivity analysis in the empirical results, especially mostly borrowing neural network architectures and optimization configurations from prior works. In our revision, however, we will add some ablation study on the choices as well as on the effect of $p$ beyond representation learning, to provide better insights for researchers who wish to apply this methodology to new problems. Below, we briefly overview the architectures and optimization configuration.\n\n    - The architectures used in our experiments were a 3-layer MLP for Laplacian representation learning, ResNet-18 for image representation learning, disjoint 3-layer MLPs for solving Schrodinger’s equation, and a 11-layer MLP encoder with skip connections for the graph experiment in the Appendix. For the 3-layer MLP for Laplacian representation learning, we observed significantly stronger gradient signals at the beginning of training than the middle, but we opted to fix this with a linear warmup rather than architectural changes. \n\n    - We did not explicitly sweep learning rates and usually used the default learning rate/scheduler for similar approaches. However, as suggested by the strong initial gradients and linear warmup fix in the Laplacian representation learning experiment, using a scheduler to keep gradient update magnitudes comparable is important.\n\n4. `Scalability: The experimental domains are relatively modest in scale. How do you expect the method to perform on very high-dimensional problems where traditional eigensolvers become prohibitive?` \n\n    This is an important question. To demonstrate the scalability of the method for more challenging problems, we will add some more controlled experiments with analytical solutions, such as N-dim infinite well and N-dim harmonic oscillator problems, as described in our response to **Reviewer YTWU**; see Item 4 therein. We expect the method to extend well to these simple examples, albeit high-dimensional, fairly straightforwardly, as long as we scale the architecture properly. A more challenging problem would be to solve the computational chemistry problem, which is to solve the Schrodinger equation for an atom with multiple electrons. For such a challenging problem with a much more complicated potential, the current methodology may not work as is, as efficient sampling from the high-dimensional becomes the main bottleneck of the problem. The recent breakthroughs in the field handle the issue by the quantum (variational) Monte Carlo (QMC), which resorts to a good architecture and sampling from the distribution induced by the neural network ansatz. We believe that one may be able to modify OMM to apply to this challenging problem, which could result in a very neat solution to the important problem. We leave this direction as an exciting future work."}]

Author rebuttal:

Meta-review: Machine learning has a long and fruitful history of translating methods from other field to the context of machine learning, and this paper does so for a spectral method known as orbital minimization in computational chemistry.

The paper is well-written and makes an interesting contribution in a topical area. The reviews and the discussion clearly support publication. The revisions promised by the authors, in particular the promised ablation studies on spectral shifts, hyperparameter sensitivity etc, will strenghten the paper further.
Decision: Accept (poster)