Neural Isometries: Taming Transformations for Equivariant ML
Authors: Thomas Mitchel
Michael Taylor
Vincent Sitzmann
Venue: NeurIPS 2024 poster (2024)
Keywords: Equivariance
Geometric Deep Learning
Representation Learning
Primary area: machine_vision
TL;DR: Neural Isometries find latent spaces where complicated transformations become tractable for downstream tasks.

Abstract: Real-world geometry and 3D vision tasks are replete with challenging symmetries that defy tractable analytical expression. In this paper, we introduce Neural Isometries, an autoencoder framework which learns to map the observation space to a general-purpose latent space wherein encodings are related by isometries whenever their corresponding observations are geometrically related in world space. Specifically, we regularize the latent space such that maps between encodings preserve a learned inner product and commute with a learned functional operator, in the same manner as rigid-body transformations commute with the Laplacian. This approach forms an effective backbone for self-supervised representation learning, and we demonstrate that a simple off-the-shelf equivariant network operating in the pre-trained latent space can achieve results on par with meticulously-engineered, handcrafted networks designed to handle complex, nonlinear symmetries. Furthermore, isometric maps capture information about the respective transformations in world space, and we show that this allows us to regress camera poses directly from the coefficients of the maps between encodings of adjacent views of a scene.

Neural Isometries:
Taming Transformations for Equivariant ML
Thomas W. Mitchel PlayStation tommy. commy.iitchel@sory. com
Michael Taylor
Vincent Sitzmann MIT sitzmann@mit. edu
PlayStation mike. www.ttaalyooo..oooo com com
Abstract
Real-world geometry and 3D vision tasks replete with challenging symmetries that defy tractable analytical expression. In this paper, we introduce Neural Isome- tries, autoencoder framework which learns map the observation space general-purpose latent space wherein encodings are related by isometries whenever their corresponding observations are geometrically related in world space. Specifi- cally, we regularize the latent space such that maps between encodings preserve learned inner product and commute with a learned functional operator, the same manner as rigid-body transformations commute with the Laplacian. This approach forms an effective backbone for superrieeeiee representation learning, and we demonstrate that a simple fff-the-hhelf equivariant network operating in the pre-trained latent space can achieve results on par with waaccculuss--enginneeee handcrafted networks designed to handle complex, nonlinear symmetries. Further- more, isometric maps capture information about the respective transformations world space, and we show that this allows us to regress camera poses directly from the coefficients of the maps between encodings of adjacent views of a scene.
Introduction
We constantly capture lossy observations of our world images, for instance, 2D projections of the 3D world. Observations captured secutively in time are often related by transformations which are casily described in world space, but are intractable in the space of observations. For instance, vidco frames captured by a camera moving through a static scene fully described by a combination of the 3D scene geometry and SE(3) camera poses. In contrast, the image space transformations between these frames only be characterized by optical flow, high- dimensional vector field that does not itself have any easily tractable low-dimensional representation.
Tv
Encuder E
Latent Space
E(V)
8(T
Geometric deep learning seeks to build neural network architectures that provably robust to transformations acting their inputs, such tations [1-3], dilations [4, 5], and projective transformations [6, 7]. How- ever, such approaches are only tractable for transformations that have group structure, and, even in those cases, still require meticulously hand- crafted and complex architectures. Yet, many real-world transformations of interest, for instance in vision and geometry processing, altogether lack identifiable group structure, such the effect of camera motion in image space-see Fig. to the right. Even when group-structured they often non-linear and non-compact, such is the case of image
Esoneety
Input generic equivariant
Figure Neural Isome- tries find latent spaces where complex transfor- mations become tractable.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

homographies and non-rigid shape deformations where existing approaches can be prohibitively expensive.
In this paper, we take a first step towards a class of models that learns to be equivariant to unknown and difficult geometric transformations in world space. We propose Neural Isometries (NIso), an autoencoder framework which learns to map the observation space to a latent space in which encodings are related by tractable, highly-ttructured linear maps whenever their corresponding observations are geometrically related in world space.
Specifically, observations encoded into latents preserving their spatial dimensions. Images, for instance, can be encoded into latent functions defined over a lower resolution grid or the patch tokens of VIT. For observations sharing some potentially unknown relationship in world space, enforce their encodings be related by a functional map 7 a linear transformation on the space of latent functions. In particular, require that is an isometry such that it preserves learned inner product and commutes with a learned functional operator, in the sense that rigid body transformations commute with the Laplace operator in Euclidean space.
Neural Isometries exhibit unique properties that make them a promising step towards an architecture- agnostic regime for self-supervise equivariant representation learning. We experimentally validate two principle claims regarding the efficacy and applicability of our approach:
Neural Isometries recover general-purre purpose latent space in which challenging symmetries in the observation space can be reduced to compact, tractable maps in the latent space. We show that this can be exploited by simple hand-crafted equivariant networks in tasks with complex non-linear symmetries.
networks to achieve results on par with leading
The latent space constructed is geometrically informative in that it encodes information about the transformations in world space. We demonstrate that robust camera poses can be regressed directly from the isometric functional maps between encodings of adjacent views of a scene.
2 Related Work
Geometric Deep Learning. Geometric deep learning is generally concerned with hand-crafting architectures that are equivariant to (i.e. that commute with) known transformations acting on the data [1-4, 6-17]. To this end, many successful architectures exploit group representations by using established mappings, such as the Fourier or spherical harmonic transforms, to map features onto domains where the group actions manifest equivalently as linear transformations [1-3, 7]. In most cases, the representations considered are finite-dimensional and irreducible which, loosely speaking, means that the group action in observation space can be expressed exactly by frequency-preserviig block-diagonal matrices acting on the transform coefficients. While finit-diimensional irreducible representations (IRs) are attractive building blocks for equivariance due to their computationally exploitable structure, they often don't I exist for non-compact groups, precluding generalizations to most non-linear symmetries. let alone those ill-modeled by groups. We instead avoid heuristic choice of symmetry model and seek an approach that enables robustness to arbitrary transformations that may not even be group-structure
Functional Maps. Essential to our approach is the parameterization of transformations between ob- servations in the latent space not as group representations, but instead as functional maps. Introduced in the seminal work of Ovsjanikov el al. [18]. functional maps (FMs) provide a powerful medium for interpreting, manipulating, and constructing otherwise intractable mappings between 3D shapes via their realization as linear transformations on spaces of functions on meshes, forming the basis for sfte-of-thh-arr pipelines in shape matching and correspondence [19-23]. Beyond 3D shapes, FMs can be seen as tool to parameterize transformations between observations viewed as functions, in the sense that images, for instance, are functions that map 2D pixel coordinates RGB colors. Integral to their study and implementation is the Laplace- Laplace-Beltrami operator (the generalization of the Laplace operator to function spaces on manifolds), and in particular the expression of FMs in its eigenbasis which can expose specific geometric properties of deformations. In particular, isometries ddistance-preservingg transformations) manifest as highly-structured matrices not unlike IRs, being orthogonal and commuting with the diagonal matrix of eigenvalues and are thus approximately black-diagonal. That said, FMs are recovered through regularized linear solves and as such lack the consistency inherent in the analytical expressions of IRs for compact groups However, freed

---

+
M HE
En(+t)
T Tq En (T+)
Tb Ln- project
Spec. dropout
Eigenbasis Projection
Image
Latent
Functional Map Ts!
Recon.
Equire Loss
Ce (((((T
# 'M
n'v= admAs Solve for Isometry
Recon, Le (P(()(()
Tel
poy A((r))
Learned Spectral Operator
Multip. L. |Axl
Image
Latent
Recon.
M Eigenbasis Projection
T Un- project
Spee. dropoun
(T(
Enbb)
Figure 2: Overview of Neural Isometries (NTso). Nīso learn latent space where transformations of observations manifest as isometries, achieved by regularizing the functional maps T between latents to commute with learned operator S2, parameterized via its spectral decomposition into mass matrix MI, cigenfunctions L and eigenvalues A (sec. 4.1). Given two observations v and T'th related by some unknown transformation T (in this case, camera motion in 3D scene), they are first encoded into latent functions {tb) and (TT)) and projected into the operator eigenbasis. An isometric functional map TQ is estimated between them, and used to map one to the other. Losses promote isometry- equivariance in the latent space, reconstruction of transformed latents, and distinct, low- low-multiplicity eigenvalues A, with the latter encouraging a diagonal as possible TQ- An optional spectral dropout layer can be applied before the basis unprojection to encourage a physically meaningful ordering of the learned spectrum (sec, 4.2).
from nagging theoretical constraints, FMs have displayed remarkable representational capacity, well modeling variety of highly-complex non-rigid deformations [24] including those without group structure, such as partial [25] and inter-genus correspondence [26]. Please see [27] for an outstanding introduction to functional maps.
Discovering Latent Symmetries. A number of recent approaches have proposed autoencoder frameworks wherein given symmetries in the base space manifest as simple operations in the latent space [28-35]. Perhaps most similar our approach is recent work on Neural Fourier Transform (NFT) that has sought to manifest group actions in observation space as IRs in the the latent space [3]. These methods offer impressive theoretical guarantees under the conditions that the observed transformations are either known or are a group action, although these assumptions may not hold for real-world data with complex symmetries. In contrast, our method is wholly unsupervised, and assumes no knowledge of the transformations in observation space nor that they even form group.
3 Method Overview
Neural Isometries are an architecture-agee agnossic autoencoder framework which learns to map pairs of observations related in world space to latents which are related by an approximately isometric FM T see Fig. We first formulate encoding and decoding, and define a FM in the latent space. Next, we show how a functional operator Ω and mass matrix MI can be learned in the latent space to regularize by requiring
be an isometry. Specifically, for observations sharing some potentially unknown relationship in world space, we enforce there exist FM between their encodings satisfying two key properties: 1) That т preserves the functional inner product in the latent space determined by MI; and 2). 7 commutes with the functional operator Q. Subsequently, we show that such maps can be recovered analytically through a differentiable, closed-fore least squares solve the operator eigenbasis. Last, we formulate NIso
optimization problem incorporating both the strictness of the isometric correspondence between latents and the eigenvalue multiplicity of the operator, the latter of which controls the structure of the maps.
In experiments, we demonstrate that NIso are capable both discovering approximations of known operators and constructing latent spaces where complex, non-lineat symmetries in the observation space manifest equivalently as isometries. We show how the latter property can be exploited by demonstrating that simple vector neuron MIP [38] acting in our pre-trained latent space can achieve results on par with state-of-the-ar handcrafted equivariant networks operating in observation space. Subsequently, consider the task of pose estimation, and demonstrate that robust SE(3) camera

---

poses can be extracted from latent transformations, serving as evidence that NIso encourages models encode information about transformations in world space.
4 Neural Isometries
We consider an observation space O C [M,, ) consisting of functions defined over some domain M (e.g. with M the plane and n for RGB images). Here, elements of are fact captures from some world space W with W - O representing the mechanism from which O is formed from W. Furthermore, we assume there is a potentially unknown collection phenomena {T) acting on the world space that relates observations. That is, for some TΦ = (Tw), we assume that T is also in and that we are able to associate with vb.
W, & = (w) E O and denoting
Additionally, we consider an autoencoder consisting of an encoder and decoder E LM, [=*) + L2
(I) mapping between observation space and a space of latent functions over some domain N. In practice, we operate over discretizations of M and N, with v E and (vb) E T,2 (N,,,, IR)) represented as tensors &b M| and {()) E R| For example, if M consists of the pixel indices of an image, then N could be grid or token indices if & is ConvNet or ViT, respectively.
and
L, RT) - (2(,
Goal: Equivariance of Latent Functions. Our aim is to train the autoencoder such that for any acting in the world space and corresponding observations vb. Tab E O, there exists a linear map T: T,2 (N, (N, IR) (N, Rd) such that
((T = TE)))
In other words, we desire our latent space be equivariant under world space transformations. As our problem is discrete, T is a functional map an |N] x |N] matrix representation of maps on (N,,, ). ). the case of latents with [N] = H x W pixels, 7 is matrix whose rows express each pixel in ((T+) linear combination of pixels in & (16), similar to the weight matrix one might obtain from a cross-attention operation.
4.1 Regularization Through Isometries
We will find r by solving a least-squares problem of the form -= min,;|(((Tv) - Unfor- tunately, as we will show in experiments, direct solve without additional regularization leads to uninformative maps that capture little information about the actual world-space transformations T. To add structure, we might ask that be orthogonal with 7 T = [NN), generating gradients promoting latent codes having the property [[(())]=[((((T))
However, we can obtain more structure yet. We propose to learn a representation of the latent geometry by jointly regressing diagonal mass matrix M and positive semi-definite (PSD) operator Ω R x[N] such that manifests isometry. That is, preserves the functional inner product defined by M (f, g)m = Mg for f.g E I,2,N, Rd) and is 2-commutative with
T TMT = M
and
r QT.
Together, the conditions in Equation (3) form a strong regularizer, the effects of which are best seen in the expression of in the eigenspectrum of Ω. As J a PSD matrix with respect the inner product defined by M. it can be expressed in terms of its spectral decomposition as
Ω = Φ T M
with
$ MM = [|M]
(4)
where A = diag( (Ax h()S is the diagonal matrix of (non-negative) eigenvalues and & E || is the matrix whose columns are the Mlorthogonal cigenfunctions of Ωt. Denoting
Tn = + M r +
(5)
as the projection of r into the eigenbasis, can shown that the conditions in Eq. (3) reduce to To being orthogonal and A-commutative [27]. This is equivalent to asking that be both sparse and condensed in that it forms an orthogonal, block-diagonal matrix, with the size of each block determined by the multiplicity of the eigenvalues in A.

---

4.2 Estimating and End-to-End Optimization
Fig. 2 visualizes Neural Isometries's training loop. First, is estimated between pairs of encoded T-related observations such that it approximately satisfies the conditions for an -lissmmetry as in Eq. (3). Second, the weights of the autoencoder, M, and I are jointly updated with respect combined loss term, promoting: a) latent equivariance as in Eq. (2), b) the ability of the decoder to reconstruct observations, and e) distinet eigenvalues A which encourage
slagonal-as-possible TQ.
Recovering r Between Latents. Instead of estimating 7 directly, we equivalently estimate TQ in the eigenbasis of Ω2 motivated by the corresponding simplification of the conditions in Eq. (3). Let
E + M 3
(6)
be the map given by encoding followed by projection into the eigenbasis of Ω. Then, given observa- tions d and Tıb, we define TQ to be the solution to the least squares problem
T
minimum
|| En(w) - & (TVll)
While Eq. (7) has an exact analytical solution [39], we instead approximate TQ with a fuzzy analogue which we find better facilitates backwards gradient flow to the parameters of f2.
Specifically, letting x R NI + O([N)) denote the Procrustes projection to the nearest orthogonal matrix (e.g. through the SVD), we recover TQ via the approximation
PA
T2
O
To R x (PA @ (T() [n(+))]),
(8)
where [PA]e exp(-IA, Xyl) is a smooth multiplicity mask over the eigenvalues applied element-wise. See the supplement for details.
To facilitate the recovery of TR, parameterize M directly by its diagonal elements and Q in terms of its spectral decomposition, learning a M-orthogonal matrix of eigenfunctions and non- negative eigenvalues A. This has the added benefit of enabling a low-rank approximation of f by parameterizing only the first k eigenvalues and eigenfunctions, i.e. & E RIN and A = ( with k S |N], mirroring similar approaches SoTA FM pipelines [19, 40, 41]. This reduces the complexity of the orthogonal projection in Eq. (8) from | X [N] to k x k.
Optimization. During training, the autoencoder is given pairs of relltte observations (v. TV). which are mapped to the latent space and nn is estimated as in Eq. (8) giving T Ta & M. First, an equivariance loss is formed between the eigenspace projections of the encodings.
LF En(t) (T(
We note that for full rank Ωt this loss is equivalent to measuring the degree to which the equivariance condition Eq. (2) holds due to the orthogonality of 4. Next we compute reconstruction loss,
LR |D((T T + |D((-- E(TV)) - +b1,,
(10)
with --1 = क7n +T Mi, forcing the decoder to map the transformed latents to the corresponding T-related observations. Last, we formulate a multiplicity loss which promotes distinct eigenvalues and ensures 12 is "interesting" by preventing it from regressing to the identity. We observe that the cigenvalue mask PA can be viewed as a graph in which the number of connected components (i.e. the number of distinct eigenvalues) is equivalent to the dimension of the nullspace of the graph Laplacian Ap, formed from the mask [42]. As a measure the nullspace dimension, we use the norm of the eigenvalues of P given by Lm | ||- The NFT [36, 37] takes similar approach, wherein diagonalization loss is imposed on estimated transformations themselves, though our experiments show it be far less effective in enforcing structure. The total loss is the of aforementioned terms
L = Ln + aLg + BLM,
(11)
with a, 9 weighting the contributions of the equivariance and multiplicity losses.
In experiments, we also consider a similar triplet regime as proposed in [36, 37], where the autoen- coder is given triples of T-related observations (w), Twè T2 (assuming 7 is composable) and the estimated map T between the encodings of (15, Tv) used form equivariance and reconstruction losses between (T'b (2)) and vice-versa. This works to prevent 7 from "cheating" by encoding
5

---

Data
Niso Outputs Latent Map
Reference GT 52: Laplacian
T
Learned Figenfinctions Operator ft
Torie
Spherical
Figure 3: Approximating the Laplacian. Forced map between shifted images on the torus (first row, left) and rotated images on the sphere (second row, left), NIso regress operators (center right) structurally similar to the toric and spherical Laplacian (right). Maps TQ between projected images are strongly diagonal (center left), with individual blocks (inset) preserving the subspaces spanned by eigenfunctions (center, first 64 shown) sharing nearly the same eigenvalues. These experiments result in the discovery of basis with the similar properties to the the toric and spherical harmonics. In particular, the estimated spherical Tp manifest exactly Wigner-D -D matrices corresponding to the rotation, with square blocks of size (26 + x (2t + 1) for the (-th distinct eigenvalue. Please zoom in to view structural details.
same structure as the ground truth
privileged information about the relationship between pairs beyond T, property we show to be critical in enforcing a useful notion of latent equivariance. However, triples of T-related observations are rare in practical settings, and we show in experiments that a major benefit of our isometric regularization that our multiplicity loss (promoting a diagonal-as as--sssiblle and thus sparse and condensed T3) can serve effective substitute for access to triples.
Spectral Dropout. While the composite loss in Equation (11) promotes latent equivariance and distinct eigenvalues, does not, however, promote physically meaningful ordering of the eigenvalues -ie. that small cigenvalues correspond smooth, low-frequency eigenfunctions and larger eigenvalues to eigenfunctions with sharper, high-frequency details. Though such an ordering amounts to permutation the spectral dimension and is not necessary condition for the existence of diagonalized isometric maps, it could be potentially useful in downstream applications where classical notion of eigenvalues frequencies desired.
To this end we propose an optional spectral dropout layer applied after computing the equivariance loss LE and before the basis unprojection. The layer is implemented there is a 50% chance that dropout will be applied to given example in a batch. Then, for each such example, spectral index 1 < le is randomly chosen and all features with index j 2 are masked out. Intuitively, this forces the decoder to produce the best possible reconstructions with the lowest frequency eigenfunctions, which appear most often and thus must capture large scale features, while reserving higher frequencies for filling in fine details.
follows: During training,
4.3 A Simple Example: Approximating the Toric and Spherical Laplacians
We demonstrate that NIso are able to learn a compact representation of isometries that reflect the dynamics of transformations in world space. To do so, we perform two experiments in which consider pairs of observations formed by 16 x 16 images from ImageNet [43] viewed functions on toric and spherical grids, and respectively transformed by random circular shifts and SO(3) rotations to form pairs. Thus pairs are related by the isometries of the torus and sphere which commute with the Laplacian on each domain. Taking the encoder and decoder to be the identity map (making the equivariance and reconstruction losses equivalent) we optimize for M,, S E IR 2526x226 via the pairwise training procedure described in sec. 4.2, including the use of spectral dropout. For the toric experiments, we learn a full-rank parameterization of Ω with k 256; for the sphere, we learn

---

low-rank approximation with k = 64. As seen in Fig 3, NIso regresses operators with significant structural similarities to the Laplacian matrices formed by the standard 3 stencil on the torus and the low rank approximation using the first 64 spherical harmonics. In both cases, NIso recovers an eigenspace that diagonalizes TQ between shifted images, with eigenfunctions ordered by their energy. As our approach is data-driven and our estimated maps are only approximately isometric, our learned operator and its eigenspeetrum do not perfectly correspond to the ground-truth Laplacian. Instead, we are able to characterize similar, non-trivial spatial relationships that are preserved under shifts.
4.4 Representation Learning with Niso
Viewed in terms of representation learning, NIso can be seen recipe for the self-supervised
pre-training of a network backbone & satisfying the equivariance condition in Eq. (2) such that transformations T in the world space manifest as isometries 7 in the sense of Eq. (3).
Exploiting Equivariance in Latent Space As we demonstrate in experiments, a simple off-the- shelf
head can be appended to the pre-trained backbone and fine-tuned to achieve competitive results in tasks with challenging symmetries. We employ simple strategy wherein low rank & approximation of Ω is learned during the pre-training stage. Thus, the eigenspace projections of the encodings of T-related observations are k tensors that are nearly equivalent up to an orthogonal transformation Th- As such, we pass the projected encodings to head consisting of an O(k) eqqiivariat vector neuron (VN) MLP [38]. We note that for large-scal scale tasks NIso is potentially well-suited to pair with DiffusionNer [44], which can make use of the learned eigenbasis to perform accelerated operations in the latent space, though we do not consider this regime here.
Pose Extraction from Latent Isometries. We propose that the recovered functional maps T encode information about world-space transformations T. To test this, we consider a simple pose estimation paradigm consisting of a pre-training phase and fine-tuning phase. In the first phase, NIso autoencoder is trained using T-related pairs of observations consisting of adjacent frames from video sequences. Subsequently, the decoder is discarded and the same pairs of observations considered during fine-tuning. In the second phase, isometries Tn are estimated between the eigenspace projections of the encoded observations, vectorized, and passed directly to an MLP which predicts the parameters of the SE(3) transformation corresponding to the relative camera motion in world space between the adjacent frames. In our experiments, the weights of the Niso backbone frozen during fine-tuning to better evaluate the information about world space transformations encoded during the unsupervised pre-training phase. At evaluation, trajectories are recovered by composing estimated frame-to-frame poses over the length of the sequence.
5 Experiments
In this section, we provide empirical evidence through experiments that NIso 1) recovers a general- purpose latent space that can be exploited by symmetries (5.1, 5.2); and 2) NIso encodes information about transformations in world space through the construction of isometric maps in the latent space from which geometric quantities such as camera poses can be directly regressed (5.3). Here we pre-train NIso without spectral dropout, as the ordering of the learned spectrum is irrelevant in our target applications and we find it slightly decreases accuracy of the prediction head. We provide reproducibility details in the supplement in addition to experiments quantitatively evaluating the degree of learned equivariance. We note that consistent theme in our experiments are comparisons against the unsupervised variant of the NFT [37] (the semi- supervised variants cannot be applied because the symmetries we consider have no IRs). Like our approach, the NFT seeks to relate latents via linear transformations though, it differs fundamentally that maps are guaranteed additional structure only if the world space transformations are a compact group action. While not originally proposed by the authors, we evaluate it in place of our approach in the same self-supervised representation learning regimes discussed in sec. 4.4. Thus the role of these comparisons is to show that our proposed isometric regularization better and more consistently provides a tractable and informative latent space.
networks to handle challenging
5.1 Homography-Perturree MNIST
In our first set of experiments, we consider classification on the homNIST dataset [6] consisting of aamoorrahy--erttrbee MNIST digits. Following the procedure outlined in see. 4.4, the classification
7

---

network consists of pre-trained Niso encoder backbone followed by VN-MLP. Specifically, pre-training performed by randomly sampling homographies from the distribution proposed in [6] which are applied to the elements of the standard MNIST training set to create pairs of observations. Here the weights of the encoder backbone are frozen and the equivariant head is trained only on the original (unperturbed) MNIST training set and evaluated on the perturbed test set. Thus the aim of these experiments is to directly quantify the degree to which pre-trained latent space is both equivariant and distinguishable.
With this in mind, we perform three ablations. In the first, we train the Niso autoencoder in triplet regime (4.2) made possible by the synthetic parameterization of T homographies. In the second and third, we train the autoencoder in the standard pairwise regime without considering the equivariance loss LE and multiplicity loss LM. respectively. Additionally we compare the efficacy of our approach versus an NFT backbone pre-trained in the same manner. Last, we pre-train and evaluate baseline backbone which considers only a reconstruction loss without respect to T rrelttdd pairs, but trains with data-augmentation during the fine-tuning phase by applying randomly sampled homographies.
V
O
3
2
9
Tv
3
9
Acc.
NIso
92.52 (1 0.91)
w/ triplet
97.38 (1 0.23)
w/o LE
77.30 (1: 2.56)
w/o LM
45.27 (# 1.20)
Results are shown in Tab. 1, averaged over five randomly initial- ized pre-training and fine-tuning runs with standard errors. Also included are those reported by homConv [6] and LieDecomp [45], networks which serve top-pert popppeformmin ammography-equivariinn
NFT [37]
41.93 (#: 0.84)
w/ triplet
67.15 (#: 1.10)
AE w/ aug.
80.96 (# 1.95)
handcrafted baseline. While NIso pre-trained with the triplet regime produces results on par with the handcrafted baselines, pre-training with the pairwise regime-which reflects a real-world scenario-achiewe a classification accuracy above 90%, signifi- cantly better than all but the three aforementioned approaches. Critically, performance drops when the multiplicity loss L is omitted, which corresponds to a regime where 7 must only preserve the inner product and 12 converges to multiple of the identity operator. This suggests that sparsifying To by filtering it through a low- mow-multiplccity cigenspace (i.e. enforcing that it is an "interesting" isometry) is fundamental in forcing the network to disentangle the structure of the observed transformations from the content of the observations themselves. In the same vein, the equivariance loss Las is also clearly instrumental, as the reconstruction loss alone does not explicitly enforce that ((t) is in fact mapped to ((Tb) under the estimated r. Furthermore, while the authors report that latent maps tend converge to orthogonal maps for compact group actions in world space [36, 37J, both NFT regimes preform poorly, implying that the learned maps do not replicate the properties of finite-dimensional IRs when the group non-compact.
homConv [6]
95.71 (#: 0.09)
LieDecomp [45]
98.30 (#: 0.10)
Table 1: Hom. MNIST.
5.2 Conformal Shape Classification
Next, we apply Niso to classify conformally-related 3D shapes from the augmented SHREC '11 dataset [7, 46]. We follow [7] by mapping each mesh to the sphere and subsequently rasterizing to grid. During pre-training, T-related pairs are selected from the sets of conformally- augmented meshes derived from the same base shape in the train split. In the fine-tuning phase, the encoder weights are unfrozen and are jointly optimized with the equivariant head, representing the practical implementation of our proposed approach for equivariant tasks.
2b
T
Results are shown in Tab. 2, averaged over five randomly initialized pre-training and fine-tuning runs with standard errors. NIso outper- forms the NET and the autoencoder baseline (with random Möbius transformations applied during the fine-tuning phase) in addition to Möbius Convolutions (MC) [7], SoTA handerafted spherical network equivariant to Möbius transformations. We consider this dataset to present significant challenge as shape classes are roughly conformally-relateed and thus the maps between their spherical pa- rameterizations are only approximated by Möbius transformations.
Acc.
NIso
90.26 (+: 1.27)
NFT [37]
83.24 (#: 2.03)
AE w/ aug.
69.36 (#: 2.81)
MC [7]
86.5
Table 2: Conf. SHREC 11.
8

---

Predicted Trajectories
Skip 0
5
Nis
hirt
VIT
Latent Maps
Select Eigenfunctions
752
NET
CO3D Sequence ATE
Frame Skip
0
5
Niso
0.023 (+: 0.001)
0.031 (# 0.001)
0.044 (+: 0.001)
0.057 (+: 0.002)
0.071 (# 0.002)
0.081 (+ 0.002)
w/o
0.027 0.001)
0.045 0.001)
0.068 (+ 0.000)
0.084 0.000)
0.000)
0.110 0.000)
NFT
0.022 0.003)
0.035 (# 0.002)
0.059 (#: 0.001)
0.077 0.001)
0.091 0.001)
0.104 (#: 0.001)
DINOv2 [47]
0.027 (=: 0.003)
0.045 (# 0.002)
0.068 (4: 0.001)
0.084 (#: 0.001)
0.097 (# 0.001)
0.110 000000)))))
BelT [48]
0.028 (++ 0.003)
0.046 (+ 0.002)
0.068 (+ 0.001)
0.085 (+ 0.001)
0.088 (+ 0.001)
0.110 0.001)
VIT
0.020 0.001)
0.042 0.001)
0.066 (#: 0.000)
0.083 0.000) 0.097
0.109 (# 0.000)
Table 3: Pose Estimation Comparison. Mean ATE for each method across all frame skips on CO3Dv2 evaluation sequences. Top: Examples of predicted trajectories, with ground truth in black, NIso in red, the NFT in blue, and the transformer baseline in green. Additional examples are shown in the supplement. Center Left: Representative examples of latent maps formed by NIso and the NFT, with the latter shown after applying the authors' proposed diagonalization procedure. NIso is able to form highly sparse, block diagonal maps. In contrast, the NFT struggles to form similarly condensed maps. Center Right: Selected learned eigenfunctions. Each column forms subspace, reflecting fundamental modes of symmetry identified by Niso.
This makes the performance of Niso particularly notable it suggests that framework has the potential to offer a more flexible and effective alternative than specialized, handcrafted equivariant networks whose underlying group-based architectures, though elegant, can only approximate inexact symmetries. Furthermore, while the NFT is relatively more competitive with the encoder backbone unfrozen, its performance and that of the baseline indicate that lack of existing equivariant structure cannot easily be overcome in the fine-tuning phase.
5.3 Camera Pose Estimation from World Video
Last, we apply NIso to the task of camera pose estimation from real-world video on the CO3Dv2 dataset [49] following the procedure described in sec. 4.4. Due to the varying quality of the ground- truth trajectories in the dataset, we create train and test sets from the top 25% of sequences in the dataset as ranked by the provided pose quality scores. In particular, we are interested in the general ability of our method to encode information about both small and large scale world space transformations. Thus we train and evaluate in a variable baseline regime, randomly skipping between and 9 frames between pairs during the pre-training and fine-tuning phases. Evaluation performed by computing the mean absolute trajectory error (ATE) between the ground truth trajectories and those recovered by our method over the sequences in the evaluation set. To understand the efficacy of our approach at different scales, report the mean ATE over six splits consisting of the same sequences the evaluation set with frame skips of 0, 1, 3, 5, 7, and 9.
We compare against the NET using the same pre- pre-training and fine-tuning procedure along with transformer baseline. The latter is inspired by the recent success of such models in 3D vision tasks [50, 51], and uses the same ViT-based architecture proposed in DUSt3R [51], with the decoder modified to directly predict the parameters of relative camera pose. As the weights of the NIso and NET backbones are frozen during the fine-tuning phase, the transformer baseline is trained from scratch the fine-turning phase to more directly compare the descriptiveness of the latent representations relative to the information contained in the raw observations. We also compare

---

against two strong representation learning baselines. We extract features from both images using two pre-trained state-of-the- vision foundation models DIN0v2[47] and BeIT[48] and pass the tokens into the modified DUSt3R-styk decoder which is trained to predict the pose.
We additionally train and evaluate ablative version of Niso which does not learn operator. Here т is predicted directly between encodings and is required only to be orthogonal with the equivariance and reconstruction losses alone enforced during training. We note that computing Dlock-diagnnalizatin loss on directly lacks justification as it would promote maps that preserve contiguous chunks of spatial indices in the latent tensors which are ordered arbitrarily. Thus, this regime serves to evaluate the degree to which the regularization through learned f2commutativvty forces the latent transfornations to reflect geometric relationships in world space.
Results are shown in Tab. 3, averaged over five randomly initialized pre-training and fine-tuning runs with standard errors. All methods perform similarly with 0 frame skip but diverge afterwards, with Niso achieving significantly lower ATE values as the skip length increases. Notably, the ablative version of our method is consistently among the worst performers, suggesting that isometric regularization is also critical to encode information about world space transformations. Overall the NFT achieves the second best performance, slightly outperforming NIso 0 frame skip but diverging thereafter. However, the latent maps it recovers are neither sparse nor exhibit condensed structure (Tab. 3, center left), and we hypothesize that its inability to effectively regularize maps beyond linearity makes it difficult for the network to discover consistent transformation model that applies across scales. This could cause the network focus on a specific regime at the expense of others, which may explain its relatively strong performance at O frame skip. The transformer baseline and representation learners are also highly flexible, and an analogous line of reasoning could explain their similar error profile.
6 Discussion
Limitations. A key factor limiting the broader applicability of our approach to geometry processing and graph-based tasks is an inability to learn and transfer operator between domains with varying connectivity. In addition, Niso does not explicitly handle partiality or occlusion between input pairs which are ubiquitous in real-world data and likely degrade its performance in the pose estimation task. We seek to address these limitations in future work.
Conclusion. In this paper we introduce Neural Isometries, method which converts challenging observed symmetries into isometries in the latent space. Our approach forms an effective backbone for sule-supe representation learning, enabling simple off-the-shelf equivariant networks to achieve strong results in tasks with complex, non- linear symmetries. Furthermore, isometric regularization produces latent representations that are geometrically informative by encoding information about transformations in world space, and we demonstrate that robust camera poses can be extracted from the isometric maps between latents in general baseline setting.
Acknowledgements. Use of the CO3Dv2 dataset is solely for benchmarking purposes and limited exclusively the experiments described in sec. 5.3. We thank Ishaan Chandratreya and Hyunwoo Ryu for helpful discussions and David Charatan for his aesthetic oversight and for granting us permission to use his images captured for FlowMap [52] in Fig. 1-2.
Vincent Sitzmann was supported by the National Science Foundation under Grant No. 2211259, by the Singapore DSTA under DST000EC120300823 (New Representations for Vision and 3D Self-Supervised Learning for Label-Efficient Vision), by the Intelligence Advanced Research Projects Activity (IARPA) via Department of Interior/ Interior Business Center (DOI/IBC) under 140D0423C0075, by the Amazon Science Hub, and by IBM.
References
[1] Taco Cohen and Max Welling. Group equivariant convolutional networks. In Proceedings of the Interna- tional Conference on Machine Learning (ICML), pages 2990-2999 2999. PMLR, 2016.
[2] Daniel E Worrall, Stephan Garbin, Daniyar Turmukhambetov, and Gabriel Brostow. Harmonic networks: Deep translation and rotation equivariance. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages
2017.
10

---

[3] Carlos Esteves, Ameesh Makadia, and Kostas Danilidis. Spin-weighted spherical cmns. Advances in Neural Information Processing Systems (NeurIPS), 2020.
[4] Mare Finzi, Samuel Stanton, Pavel Izmailov, and Andrew Gordon Wilson, Generalizing convolutional neural networks for equivariance to lie groups on arbitrary continuous data. In Proceedings of the International Conference on Machine Learning (ICML), pages 3165-3176, PMIRR, 2020.
[5] Daniel E Worrall and Max Welling, Deep scale-spaces Equivariance over scale. arXiv preprint arXiv: 13Xiv:9905.11997. 2019.
[6] Lachlan E MacDonald, Sameera Ramasinghe, and Simon Lucey. Enabling equivariance for arbitrary lie groups. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 8183-8192, 2022.
[7] Thomas W Mitchel, Noam Aigerman, Vladimir G Kim, and Michael Kazhdan. Möbius convolutions for spherical enns. In ACM Trans. Graph., pages 1-9, 2022.
[8] Taco S Cohen, Mario Geiger. Jonas Köhler, and Max Welling. Spherical cnns. arXiv: 10130,0100. 2018.
arXiv preprint
[9] Mare Finzi, Max Welling, and Andrew Gordon Wilson. A practical method for constructing equivariant multilayer perceptrons for arbitrary matrix groups. In International conference on machine learning, pages 3318-3322 PMLR, 2021.
2021.
[10] Simon Batzner, Albert Musaclian, Lixin Sun, Mario Geiger, Jonathan P Mailoa, Mordechai Kornbluth, Nicola Molinari, Tess E Smidt, and Boris Kozinsky. E (3)-cquivariant graph neural networks for data- efficient and accurate interatomic potentials. Nature communications, 13(1):2453, 2022.
[11] Victor Garcia Satorras, Emiel Hoogeboom, and Max Welling. E (a) equivariant graph neural networks. In International conference muchine learning, pages 9323-9332. PMLR, 2021.
[12] Johannes Brandstetter, Roh Hesselink, Elise van der Pol, Erik Bekkers, and Max Welling. Geometric and physical quantities improve e (3) equivariant message passing. arXiv preprint arXiv:21 10,02905, 2021.
[13] Alexander Bogatskiy, Brandon Anderson, Jan Offermann, Marwah Roussi, David Miller, and Risi Kondor. Lorentz group equivariant neural network for particle physics. In International Conference on Machine Learning, pages 992 1002. PMLR, 2020.
[14] Erik Bekkers. B-spline cnns on groups arXiv preprint 19Kiv:1909.12000 22017,20771
[15] Yue Wang. Yongbin Sun, Ziwei Liu, Sanjay E Sarma, Michael M Bronstein, and Justin M Solomon. Dynamic graph enn for learning on point clouds. ACM Transactions on Graphics (tog), 38(5):1-12 12, 2019.
[16] Michael M Bronstein, Joan Bruna, Taco Cohen, and Petar Veličković. Geometric deep learning: Grids, groups, graphs, geodesics, and gauges. arXiv preprint arXiv:2 13478, 2021.
[17] Omri Puny, Matan Atzmon, Heli Ben-Hamu, Ishan Misra, Aditya Grover, Edward J Smith, and Yaron Lipman. Frame averaging for invariant and equivariant network design. arXiv preprint 2 2021.
[18] Maks Ovsjanikov. Mirela Ben-Chen, Justin Solomon, Adrian Butscher, and Leonidas Guibas. Functional maps: flexible representation of maps between shapes. ACM Transactions on Graphics (ToG), 3 31():: 1-11, 2012.
[19] Simone Melzi, Jing Ren, Fmanuele Rodola, Abhishek Sharma, Peter Wonka, and Maks Ovsjanikov. 7oomout: Spectral upsampling for efficient shape correspondence. arXiv preprint arXiv: 1904.07865, 2019.
[20] Robin Magnet and Maks Ovsjanikov. Mcmory-scalable and simplified functional map learning, arXiv preprint arXiv:2404. 00330, 2024.
[21] Mingze Sun, Shiwei Mao, Puhua Jiang, Maks Ovsjanikov, and Ruqi Huang. Spatially and spectrally consistent deep functional maps. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 14497-14507, 2023.
[22] Ahmed AhdeIrcheem, Ahdelrahman Eldesokey, Maks Ovsjanikov, and Peter Wonka. Zero-shot 3d shape correspondence. In SIGGRAPH Asia 2023 Conference Papers, pages 1-11,201 2023.
[23] Or Litany, Tal Remez, Emanuele Rodola, Alex Bronstein, and Michael Bronstein. Deep functional maps: Structured prediction for dense shape correspondence. In Proceedings of the IEEE international conference 2017.
computer vision, pages

---

[24] Mikhail Panine, Maxime Kirgo, and Maks Ovsjanikov. Non-isometric shape matching via functional maps on landmark-adapted bases. In Computer graphicx forum, volume 41, pages 394-417. Wiley Online Library, 2022.
[25] Souhaib Attakii Gautam Pai, and Maks Ovsjanikov. Dpfm: Deep partial functional maps. In 2021 International Conference on 3D Vision (3DV), pages 175-185. IEEE, 2021.
[26] Dongliang Cao, Paul Roetzer, and Florian Bernard. Revisiting map relations for unsupervised non-rigid shape matching. arXiv preprint arXiv:2300.111111 11400, 2023.
[27] Maks Ovsjanikov. Etienne Corman, Michael Bronstein, Emanuele Rodolà, Mirela Ben-Chen, Leonidas Guibas, Frederic Chazal, and Alex Bronstcin. Computing and processing correspondences with functional maps. In SIGGRAPH ASIA 2016 Courses, pages 10. 2016.
[28] Yilun Du Katie Collins. Josh Tenenbaum, and Vincent Sitzmann. Learning signal agnostic manifolds of neural fields. Advances in Neural Information Processing Systems (NeurIPS), 34:8320-8331, 2021.
[29] Sharut Gupta, Joshua Robinson, Derek Lim, Soledad Villar, and Stefanic Jegelka. Learning structured representations with equivariant contrastive learning. 2023.
[30] Sangnie Bhardwaj, Willie McClinton, Tongzhou Wang, Guillaume Lajoie, Chen Sun, Phillip Isola, and Dilip Krishnan. Steerable equivariant representation learning. arXiv preprint arXiv:2302.11349 2023.
[31] Nima Dehmamy, Robin Walters, Yanchen Liu, Dashun Wang, and Rose Yu. Automatic symmetry discovery with lie algehra convolutional network. Advances in Neural Information Processing Systems, 34:2503-2515,
2021.
[32] Artem Moskalev, Anna Sepliarskaia, Ivan Sosnovik, and Aroold Smeulders. Liegg: Studying learned lie group generators. Advances in Neural Information Processing Systems, 35:25212-25223, 2022.
[33] Jianke Yang, Robin Walters, Nima Dehmamy, and Rose Yu. Generative adversarial symmetry discovery. In International Conference on Machine Learning, pages 39488-39508. PMLR, 2023.
[34] Jianke Yang, Nima Dehmamy, Robin Walters, and Rose Yu. Latent space symmetry discovery. arXiv preprint 21X1v22310001005. 2023.
[35] Ho Yin Chau, Frank Qiu, Yubei Chen, and Bruno Olshausen. Disentangling images with lie group transformations and sparse coding. arXiv preprint arXiv:2012.12071, 2020.
[36] Takeru Miyato, Masanori Koyama, and Kenji Fukumizu. Unsupervised learning equivariant structure from sequences. Advances in Neural Information Processing Systems (NeurIPS), 35:768-78887 781, 2022.
[37] Masanori Koyama, Kenji Fukumizu, Kohei Hayashi, and Takeru Miyato. Neural fourier transform: general approach to cquivariant representation learning. Proceedings of the International Conference on Learning Representations (ICLR), 2023.
[38] Congyue Deng, Or Litany, Yueqi Duan, Adrien Poulenard, Andrea Tagliasacchi, and Lconidas Guibas. Vector neurons: A general framework for so (3)-cquivariant networks. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 12200-12209, 2021.
[39] user1551
faacckexchange.com/usrr////
Solve | - bll subject to xe =
Mathematics Stack Exchange. URL. https: htap:::://aatsttteeeee stackexchange........ com/q/ ctttteee.co////88/11/e /etp////8881..... cz.
(version: 2024-03-23).
[40] Nicolas Donati, Abhishek Sharma, and Maks Ovsjanikov. Deep geometric functional maps: Robust feature learning for shape correspondence. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 8592-8601, 2020.
[41] Souhaib Attaiki and Maks Ovsjanikov. Understanding and improving features learned in deep functional maps. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1316-1326, 2023.
[42] Jiashi Feng. Zhouchen Lin, Huan Xu, and Shuicheng Yan. Robust subspace segmentation with block- diagonal prior. In Proceedings of the IEEE conference on computer vision and pultern recognition, pages 3818-3825, 2014.
[43] Jia Deng. Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages leee, 2009.
12

---

[44] Nicholas Sharp. Souhaib Attaiki, Keenan Crane, and Maks Ovsjanikov. Diffusionnet: Discretization agnostic learning surfaces. ACM Transactions Graphics (TOG). 41(3):1-16, 2022.
[45] Mircea Mironenco and Patrick Forré. Lie group decompositions for equivariant neural networks. In The Twelfth International Conference Learning Representations, 2024. URL /tRLLtttp://ppaaeriii...ee
2orum?sdrp34fRRKpp..
[46] Z Lian, A Godil, B Bustos, M Daoudi, J Hermans, S Kawamura, Y Kurita, G Lavoua, Dp Suetens, et al. Shape retrieval on non-rigid 3d watertight meshes, Eurographics workshop on 3d object retrieval (3DOR). Citeseer, 2011.
[47] Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Mare Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. arXiv preprint WxXiv2334.07/999 2023.
[48] Hangbo Bao, Li Dong, Songhao Piao, and Furu Wei. Beit: Bert pre-training of image transformers. arXiv preprint arXiv:2106.088 08254, 2021.
[49] Jeremy Reizenstein, Roman Shapovalov, Philipp Henzler, Luca Sbordone, Patrick Labatut, and David Novotny. Common objects in 3d: Large-scale learning and evaluation of real-life 3d category reconstruction. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 10901-10911 2021.
[50] Philippe Weinzaepfel, Vincent Leroy, Thomas Lucas, Romain Brégier. Yohann Cabon, Vaibhav Arora, Leonid Antsfeld, Boris Chidlovskii, Gabricla Csurka, and Jérôme Revaud. Croco: Self-supervised pre- training for 3d vision tasks by cross-view completion. Advances Neural Information Processing Systems, 35:3502 3516, 2022.
[51] Shuzhe Wang, Vincent Leroy. Yohann Cabon, Boris Chidlovskii, Jerome Revaud. Dust3r: Geometric 3d vision made easy, 2023.
[52] Cameron Smith, David Charatan, Ayush Tewari, and Vincent Sitzmann. Flowmap: High-quality camera poses, intrinsics, and depth via gradient descent. arXiv preprint arXiv:2404.15256 2024.
[53] Ilya Loshchilov Frank Hutter. 100501 2017.
Decoupled weight decay regularization.
arXiv preprint
[54] Michael Kazhdan, Jake Solomon, and Mirela Ben-Chen. mean curvature flow modified be non-singular? In Computer Graphics Forum, volume 31, pages 1745- 1754. Wiley Online Library, 2012.
[55] Jian Sun, Maks Ovsjanikov, and Leonidas Guibas. A concise and provably informative multi-scale signature based on heat diffusion. In Computer graphics forum, volume 28, pages 1383-1392, Wiley Online Library, 2009.
13

---

A Derivations and Implementation Details
A.1 Isometries in the Eigenbasis
Here we show that isometries manifest as orthogonal matrices that commute with the diagonal matrix of eigenvalues in the operator eigenbasis. Suppose mass matrix M, with the former expressed in terms of its eigendecomposition as in Equation (4) for eigenfunctions T and eigenvalues A.
given PSD operator Ω and diagonal
Let an isometric functional map satisfying the conditions Equation (3) with TQ its projection into the eigenbasis in Equation (5). Noting that the condition & MD 1/81 implies that O
s T M and $ MG, it follows that
{$b T - M. M/
M M
+ Ml +
+ MM$ IN1
and thus Tn is orthogonal. Furthermore, it follows that
τΩ + MMM ΦA
+ Mr 4A MMP
"MM 12
T "MM +
AΦ M 1 ATB.
s0 TqA = ATA-
A.2
Recovering TQ
Here we derive the approximate solution to the least squares problem in Equations (7 8). Specifically. let A diag( (Achssgs)) E k be a diagonal matrix of non-negative eigenvalues with Ag S 14 Furthermore, suppose that there are distinct eigenvalues {X: d s Xisı each with multiplicity m, Thus, A can expressed as the direct sum
A
X Tmm:
(12)
1=1
with Im, denoting the m, x my identity matrix. It follows that any A x matrix i satisfying πλ Am must be of the form
π
Ty
IR XM,, 15isg.
(13)
That is, π must be a block diagonal matrix with the size of each block determined by the multiplicity of the eigenvalues of A.
Now, given any A, B Rexd consider the minimization problem minimum || A B|],
n
(14)
==/,
consisting of finding an orthogonal, A A-momutting map minimizing the distance between and B under the Frobenius norm. Writing
A
B
A
and
B
(15)
B
the solution to the minimization problem in Equation (14) is equivalently expressed as the direct sum of the solutions the orthogonal sub-problems corresponding each block [39] with
minimum||aa [T A, -Bll--
(16)
x x==1
14

---

Now, for arbitrary n let x R - O(n) denote the Procrustes projection to the nearest orthogonal matrix such that for any M E lkxx with SVD M = UEV x(M) = UVT. Thus, the solution to the minimization problem is given by
(B
(17)
i=1
which, following from the properties of the SVD, can be expressed equivalently via a single &s x k Procrustes projection such that
== :()UAA))
(18)
Here we observe that by defining PA E Blck b to be the eigenvalue mask given by
11 [Pxla
=
(19)
00
otherwise
we have
P PABBAT
+ B,A, i=1
(20)
and thus
T' x(P O BAT)
(21)
In practice we substitute PA in Equation (19) with the fuzzy analogue [Palas cxp(-1>; X which we find better facilitates the flow of backwards gradients to the parameters of Ω. Replacing A and B with (n( and (T(T)) we arrive the approximate solution for TQ in Equation (8).
A.3 Graph Laplacian of PA
Given the k x eigenvalue mask PA with [Pxlij = exp(-1AA, - -2s)), we form its graph Laplacian as
A = diag PA11 - PA.
(22)
with 1 E & the vector of ones.
B
Reproducibility
Code and experiments are available at httpp:///ittub..cmm httpp:///gthhb..cm//sssssaaa/ara
B.1
Hardware
All experiments were performed on single NVIDIA A6000 GPU with 48 GB of memory.
B.2
Approxímating the Laplacian
In the toric/spherical experiments, the model consists of a single NIso layer. which learns a full- rank/low-rank operator and mass matrix, with the former parameterized by orthogonal matrix of eigenvectors E 252x2266/ E 2526666 and diagonal matrix of non-negative eigenvalues A. During training, input features are created by stacking 86 16 x 16 3 examples from ImageNet into single 16 x 16 x 258 tensor representing an observation vb. In the toric experiments, random circular shifts about both spatial axes are applied form T T's. In the spherical experiments, the images are assumed to lie on 16 x 16 Priscoll-Heat, spherical grid and random rotations in SO(3) are sampled and applied to form Tęb As the encoder and decoders are the identity maps, the equivariance and reconstruction losses are equivalent, and the objective in Equation (11) is minimized with a 0 and = 0.1. The models is trained with spectral dropout for 100,000 iterations with a batch size of using the AdamW optimizer [53] with a weight decay of 10-1 The learning rate follows schedule consisting of a 2,000 step warm up from 0.0 cosine anncaling. Training takes approximately 6 hours.
10-1 and afterwards decays to 5 x 10-5 via
15

---

B.3 Homography-Perturber MNIST
Architecture. The encoder and decoder are mirrored 2-level ConvNets. Specifically, each level consists of three convolutional ResNet blocks with 128 and 256 channels at the finest and coarsest resolutions respectively. A mean pool (nearest neighbor unpool) layer halving (doubling) the spatial resolution bridges the two levels.
The latent space has 32 dimensions. A rank k 32 operator Ω and mass matrix M. are learned, with the former parameterized by 32 eigenfunctions and non-negative eigenvalues.
The equivariant classification head consists of 2-layer VN-MLP [38] with 128 channels, followed by an output layer which computes invariant features from the inner products between vectors which passed to standard linear layer to output class predictions.
Pre-Training. Following [6], MNIST digits are padded to 40 x 40, and pairs are produced by warping digits with respect to homographies T' sampled from the proposed distribution. In the triplet regimes, tuples are created by applying T in addition to T. Both the pairwise and triplet regimes are trained with respect to the composite loss with a = 0.5 and / = 0.1. The two ablative regimes O, respectively. In the triplet regime, denoting and a to be the
are trained with a = 0 and maps estimated between (E($), (TT)) and (E(TV), E(T+)) respectively, the equivariance and reconstruction losses are formulated as
CE | En(v) En(TT)) Ir (T(V) En(TT
(23)
and
CR = I - Tbl I ((TT)) Tr
(24)
The autoencoders are trained without spectral dropout for 50, 000 steps with a batch size of 16 using the AdamW optimizer with weight decay of 10-4, The learning rate follows schedule consisting of a 2,000 step warm up from 0.0 to 5 x 10-4 and afterwards decays to 5 x 10-6 via cosine annealing. Training takes approximately 30 minutes.
Fine- Tuning During the fine tuning phase, the decoder discarded, and the weights of the encoder, operator Q2 and mass MI are frozen. Examples & encoded, and their eigenspace projections En are passed to the classification head to form class predictions under a standard softmax cross-entropy loss. Training is performed for 10,000 iterations with batch size of 16 using the AdamW optimizer the same weight decay and learning rate schedule as used previously. Training takes approximately one minute. Evaluation is performed on the fixed test set proposed in [6].
Baselines. We implement the NFT following the procedure described in the paper [37]. While ViT-based encoder is proposed in the original paper, we find that it produces worse evaluation results than the 2-layer ConvNet encoder described above and thus use the latter our evaluations to provide a fair comparison. We follow the authors' proposed implementation and consider only reconstruction loss and learn a diagonalization transform in cost-processing step. Otherwise, the NFT is pre- are-trained, fine-tuned. and evaluated identically to own approach, and uses the same cquivariant classification head. We note that the dimension of the latent map constructed by the NFT is the same as Tn (32 x 32) and that the tensor passed to the classification head is the same size for both our method and the NFT.
The autoencoder baseline consists of the same encoder and decoder, pre-trained identically but with respect to the standard reconstruction loss (+)() xbهl without considering T-related pairs. During the fine-tuning phase, we apply data augmentation to the input images by transforming them by homographies sampled from the distribution proposed in [6] before encoding. Latents are passed directly to a standard non-equivariant 2-layer MLP with 128 channels to form class predictions. Otherwise, fine-tuning and evaluation are performed in the same manner.
B.4 Conformal Shape Classification
Experiments are performed using the conformally extended SHREC '11 shape classifica- tion dataset proposed in [7], with 30 random conformal transformations are applied to each shape to extend the original dataset [46]. The augmented dataset contains 25 distinct shape classes. Each shape mapped to the sphere using the method of [54]. Input features are taken to be the values of the heat
Dataset.
kernel signature [55] computed on the original mesh at 16 timescales logarithmically distributed in the
16

---

range [-2, 0J and rasterized to 96 x 192 spherical grids. Pairs of -erelted observations are formed by randomly selecting two conformal augmentations of the same shape. For each set of training and fine-tuning runs, train and evaluation splits are randomly generated by selecting respectively 10 and 4 of the 20 sets of conformally augmented shapes per class.
Architecture. The encoder and decoder are mirrored 4-level ConvNets, with two ResNet blocks per level and 32, 64, 128, 256 channels per layer from finest to coarsest resolution. Mean pooling and nearest neighbor upsampling are used to halve and double the resolution between layers.
The latent space has 64 dimensions. A rank k 64 operator ! and mass matrix M. are learned, with the former parameterized by 64 eigenfunctions and non-negative eigenvalues.
The equivariant classification head consists of 4-layer VN-MIP with 128 channels, using the same output layer as described in sec. (B.3) to produce class predictions.
Pre- Traininnn Training is performed with respect to the composite loss with a = 0.5 and / = 0.01. The model is trained without spectral dropout for 50,000 iterations with a batch size of 16 using the AdamW optimizer with a weight decay of 10 The learning rate follows schedule consisting of a 2,000 step warm up from 0.0 5 x 10 and afterwards decays 5 x 10-5 via cosine annealing. Training takes approximately four hours.
Funn-n During the fine tuning phase, the decoder is discarded and the encoder, operator Ω and mass matrix M are jointly optimized with the equivariant classification head using the standard softmax cross-entropy loss. Training is performed for 10,000 iterations with a batch size of 16 using the Adam W optimizer, with the same weight decay and learning rate schedule as used previously. Training takes approximately one hour.
Baselines. We pre-train, fine-tune, and evaluate the NFT and autoencoder baseline using the same autoencoder architecture, classification head, and training regimes in the same manner sec. (B.3). Here data augmentation is applied during the fine-tuning phase by randomly sampling and applying Möbius transformations with scale factors proportional to those estimated between the spherical parameterizations of the shapes. Again, the both the size of the latent map and size of the tensor passed to the classification are the same size for both our method and the NFT. Here, the weights of the encoder are left unfrozen, and are jointly optimized with the classification head during the fine-tuning phase.
B.5 Camera Pose Estimation from Real-World Video
Dataset. Experiments are performed with a subsection of the CO3Dv2 dataset [49]. Given the high-degree of variability in the quality of the ground truth trajectories, we consider only the top 25% of sequences in the dataset as ranked by the provided pose quality score, slightly under 9,000 trajectories in total. An evaluation set is formed by withholding 10% of said trajectories, with the rest used for training. The list sequences in the train and evaluation sets will be made available along with the code release. As the videos in the dataset are of diffferent resolutions, we center crop each frame to the minimum of its height and width dimensions and resize to 144 x 144.
Architecture. The encoder and decoder are mirrored 3-vell ConvNets, with four ResNet blocks per level and 64, 128, 256 channels per layer from finest to coarsest resolution. Mean pooling and nearest neighbor resolution are used to halve and double the resolution between layers.
The latent space has 128 dimensions. A rank k 128 operator Ω parameterized by and mass matrix MI are learned, with the former parameterized by 128 eigenvectors and eigenvalues.
The pose extraction network consists of standard 5-layer MLP with 512 channels, which outputs 9-dimensional tensor representing the parameters of an SE(3) transformation (translation two vectors used to form the rotation matrix through cross products).
Pre-Traie Training. During pre-training -eellaedd pairs of observations are formed by randomly select- ing adjacent frames from sequences with a frame skip between O and 10. Training is performed with respect to the composite loss with a = 0.5 and 8 = 0.025. The model is trained for 200,000 iterations with a batch size of 8 using the AdamW optmizer with a weight decay of 10-1 The learning rate follows schedule consisting 2,000 step warm up from 0.0 5 x 10-1 and afterwards decays to 5 x 10-5 via cosine annealing. Training takes approximately 12 hours.
17

---

Fine-Tuning, During the fine-tuning phase, the weights of the encoder, operator S2 and mass matrix M. are frozen. As in the pre-traine pre-training phase, pairs of observations are passed to the decoder and 7Q E 12 is estimated between the eigenspace projection of the encodings. Then, is vectorized and passed to the MIPP head to predict the parameters of the camera pose.
Training is performed with respect to two term loss measuring the degree to which the predicted translation and rotation deviate from those of the ground truth pose. Denoting Ry and It be the predicted and ground truth poses respectively, the rotational component of the loss is given by
L = R Rall-
(25)
The transitional component of the loss must be scale invariant as the depth scale factor unknown between sequences. To handle this, we consider sub- batcha formed from pairs of frames all belonging to the same sequence. Denoting tn and to be the matrices consisting of the sub- uub-batched predicted and ground truth translations stacked column volumn-wise, we form the translational component of the loss via
LT |t tall
(26)
where
tr(t tq)
(27)
||ta|1
the scale factor that minimizes Cr over the pairs of frames in the sub-batch. The total loss given by [ LR + LT
The prediction head is trained without spectral dropout for 20,000 iterations with batch size of 16 (consisting of 4 sub-batches of pairs of frames from 4 sequences) using the Adam optimizer with the same weight decay and learning rate schedule used previously. Training takes approximately one 80 minutes.
Baselines. The NFT is pre-trained, fine-tuned, and evaluated using the same autoencoder architec- ture, prediction head, and training regime as our own method preceding experiments. We note that both our method and the NFT construct latent maps of the same dimension, which are passed to identical prediction heads.
same manner as is done in the
The transformer baseline is based on the CroCo/DUSC3R550, 51] architectures. As with our approach, input frames are passed to the same decoder. Here the encoder consists of 12-layer VITs with a patch size of 16, with 16 heads and 512 channels per layer. Adjacent frames are independently encoded, and then concatenated along the spatial dimension in addition tensor is passed to 12-layer transformer decoder which mirrors the encoder with 16 heads and 512 channels per layer. Subsequently, the learned token is extracted from the output and passed through a linear layer to recover the predicted pose parameters.
single learned token. The resulting
Additionally, we also evaluate the efficacy of DINOv2[47] and BeIT[48] as representation learners for the pose estimation task. Pairs of adjacent frames are passed through the pre-trained models and the feature tokens for each image from the final layers are extracted. These are used as input the same DUSt3R[4]]ssty decoder described above which is trained to predict the parameters of the relative camera pose.
The VIT model and DINOv2/BelT prediction heads are trained from scratch in the fine-tuning phase, using the same loss and optimization procedure described above.
C Additional Results
Below we provide additional results including a quantiative evaluation of learned equivariance, visu- alizations of the eigenfunctions & and mass matrices MI learned in each experiment, and comparisons of estimated camera trajectories recovered by each method in the pose prediction experiments. For the latter, examples of failure cases are also included.
18

---

Equivariance Error
Hom. MNIST
Conf. SHREC '11
Niso homConv [6] MobiusConv
4.06% (# 3.79)
5.91% (= 3.49)
6.86%
2. 6.6%%
CO3D Equivariance Error
Frame Skip
5
7
Niso
5.25% (+: 1 1.61) 5.84% (+: 1.41) 7.44% (+: 1.43) 8.23% (+ 1.48)
8.79 % (#t: 1.56)
9.24% (+: 1.67)
Table Quantitative Evaluation of Learned Equivariance. Mcan NIso equivariance error in the latent space across all experiments. Following [6, 7, 45], measure the learned equivariance of the NIso latent space in the standard way via the average of || ( - E (T v))l" Ea(()) over all pairs in the test for five randomly initialized pre-training runs. For Hom. MNIST, randomly sample homographies from the distribution proposed in [6] which are applied to the elements of the standard MNIST test set; for Conf. SHREC 11, pairs formed from the sets of conformally- augmented meshes derived from the same base shape in the test split. With CO3D we measure the error between encodings of adjacent frames in test set, with increasing frame skip. We also list the errors reported by competing hand-crafted methods.
Homography MNIST
Conformal SHREC '11
CO3D
Leurned Eigenfunctions +
Learned Masses M
Figure 4: Visualizing the Learned Eigenfunctions and Mass Matrices M. Visualizations the eigenfunctions & learned in each experiment shown the top row. Eigenfunctions are sorted by eigenvalue in ascending order along rows in C-style indexing. Here, experiments were performed without spectral dropout the ordering random. The elements of the learned diagonal mass matrices MI are shown on the bottom row, in terms of the magnitude of the deviation from the mean value each grid index the latent space. White indicates little deviation from the mean, with green-blue indicating mass values above the mean and orange-red indicating mass values below. In the MNIST experiments (sec. 5.1), the distribution of mass appears to segment null space from the central region most often occupied by the digits. In the conformal shape classification experiments (sec. 5.2), the larger deviations from the mean values appear closer the poles (the top-most and bottom-most rows of the spherical grid). For the pose estimation experiments (sec. 5.3), larger deviations appear at the boundaries, with the lower half of the grid having slightly higher values.
19

---

Predicted Trajectories
Skip 0
Figure Qualitative Pose Estimation Comparisons. Example predicted trajectories for each method select CO3Dv2 evaluation sequences. Ground truth shown in black, NIso in red, the NFT in blue, and the transformer bascline green. NIso appears to consistently better capture rotational (curvature) information about the world space transformations, helping better track the the camera motion across scales.
20

---

Predicted Trajectories
Skip 0
O
Figure 6: Qualitative Pose Estimation Comparisons Cont.
21

---

Predicted Trajectories
Skip
5
Figure 7: Failure Cases. Select failure cases. Interestingly. NIso appears to consistently fail when the background and foreground objects are respectively far from and close the camera. In cases, depth maps are often ill-defined, suggesting NIso may encourage models to reason about the underlying 3D structure the scene.
such
22

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The claims made in the abstract and introduction supported by experimental results and are contextualized with respect to competing methods.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Limitations are discussed in the final section of the paper (6).
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model sell-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
23

---

Justification:
Derivations of relevant expressions are provided in the supplement.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: Yes)
Justification:
We provide reproducibility details for each experiment in the supplement.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
24

---

Answer: [Yes]
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.c cc/
http://uieee/eoeuuuiiiiiiiiioooe (oossiccyn for more details.
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https:
/ cttps/cccccccccccccccccccccccccc / /CodeeSubiissiiiiieeeee ssi asionPolicyy for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes]
Justification: Experiment details are provided the paper and supplement. Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: We report all quantitative results in terms of the mean over five random initializations (and dataset splits, where applicable) with the standard error.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
25

---

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of enrors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative rates).
error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time execution) needed to reproduce the experiments?
Answer: [Yes]
Justification: All experiments were performed the same device, the details of which described in the supplement.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g.. preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https https://neett /ttp:/:///......///// hcc//pllic/Ehhiscliiieeeeeeeeeee
Answer: [Ys]]
Justification: To the best of our knowledge, conform to the Code of Ethics. At this time we do not see our method providing a straightforward avenue for abuse by bad actors.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No. they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is a special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: Due to the low-level nature of our method, we do not see it directly facilitating any negative social impacts.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
26

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: All data used is cited in accordance with the provided licenses.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
27

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
28


Reviews:
[{"review_id": "BoQAWAVHLj", "reviewer": "Reviewer_2Npe", "summary": "The main motivation behind Neural Isometry is as follows: most of real world transformations in vision and geometry processing lack identifiable group structure and therefore challenging for prior work in equivariant learning that assumes such knowledge apriori. The paper proposes an auto encoder framework that maps observations (related by some geometric transformations) to a latent space where embeddings are  related by a linear transform. The framework leverages the existing functional map framework and corresponding regularization to learn this latent space. The paper validates and compares their performance with the baseline NFT (Neural Fourier Transform) that assumes the knowledge of the group a priori.", "questions": "please see above.", "limitations": "The submission should state the scalability of this approach given its reliance on Laplacian Eigen basis.", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "-  The problem of learning structured latent space in a self supervised equivariant learning set up is relevant.\n\n- The proposed framework is formalized clearly and in detail in Section 4.", "weaknesses": "As mentioned many times in paper, Instead of theoretical justification ( as in NFT), this paper seeks to validate the efficacy of approach experimentally on real world tasks.\n\n- Limited Experimental Set up: The datasets used are still very small scale (MNIST,) or synthetic (SHREC) and therefore far from the real world data.  The submission mainly follows and compares with one baseline (NFT) that was implemented by authors themselves.  The submission could motivate the problem/solution by probing/playing with the learned structured space e.g. in geometry processing tasks [1]. \n\n- Limitation of Functional map: One of the biggest limitation of functional map framework is its applicability to real world data where partiality is ubiquitous (due to occlusion etc). Therefore, Neural Isometry by design also inherits this limitation.  experiments are shown on MNIST/SHREC where there is no occlusion or partiality. In the third benchmark, instead of skipping the odd frame, does the performance decays rapidly between distant frames?\n\n- Presentation: The submission contains several typos, convoluted sentences (Line 85-88) and statements without a citation. e.g.\n\n-- Line 49: < Lorentz transformations with d'almbert operator of Minkowski space> citation is missing here. Also, why do we need to know this fact in the introduction of this paper? \n\n-- Line 42: <by preserving the spatial dimensions> not sure what does this mean after reading the next 2 lines. Please clarify\n\n-- Line 89: <orthogonal relaxation of FM> citation or please detail what those are.\n\n- Related work: The submission cites a dozen variants of deep functional map paper [2] but not the deep functional map paper itself [2] that inspired them. Please justify how [20,21,22,39] are related to this work or inspired this work given other deep functional pipeline cited in paper. The reviewer would instead relate/distinguish this work with [3].\n\n\n1. Composite Shape Modeling via Latent Space Factorization, Dubrovina et al, 2021\n2. Deep Functional Maps Litany et al. 2017\n3. Map-based exploration of intrinsic shape differences and variability  Rustamov et al. 2013", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## Limited experimental setup\nWe would like to respectfully push back on the notion that we are considering too few baselines. We note that there is very little work on equivariant machine learning *without* prior knowledge of the symmetry group - the NFT is the most relevant baseline in this space. Further, the reviewer is suggesting that we are only comparing “small-scale” or “synthetic” experiments”. However, camera pose estimation on Co3D is *not* - it is a large, real-world dataset that is actively used for benchmarking of applications from pose estimation to novel view synthesis.\n\nMore generally,  we sought geometric deep-learning baselines for difficult, non-compact symmetries in vision tasks. To the best of our knowledge, the only existing models handling these symmetries are homConv [6] and LieDecomp [43] (both for homographies) and MobiusConv [7] (for Mobius Transformations). Thus, benchmarking on MNIST and SHREC is necessary to support our claims. These baselines are not scalable and will not run on any more complex or real-world datasets - this is not a limitation of *our* method, but a limitation of the baselines.\n\nWe would be happy to compare with additional baselines if you could clarify what they should be.\n\n## Limitation of functional maps (partiality)\nWe agree that functional maps can be sensitive to occlusion and partiality.  However,  we note that NIso is already *surprisingly robust* to partiality in its inputs, as is evident in the Co3D experiments. Please see the discussion in the general rebuttal.\n\n## Presentation\n\n### Line 49: < Lorentz transformations with d'almbert operator of Minkowski space...\n\nThank you - we agree that this is confusing and removed it.\n\n### Line 42: <by preserving the spatial dimensions...\nBy this, we mean that the encoder encodes images into 2D feature maps with height and width higher than 1x1 pixel. This is opposed to *collapsing* the spatial dimension, as done in the NFT, which encodes the whole image into a single, global latent vector.\n\nHowever, we agree that this is unclear as written, and will change the wording.\n\n### Line 89: <orthogonal relaxation of FM>\n\nReference [23] (cited later in the same sentence) seeks functional maps that represent near conformal, rather than isometric deformations. Conformal transformations preserve the Dirichlet inner product, and thus manifest as orthogonal transformations in the eigenbasis of the Dirichlet Laplacian.  \n\nThat said, we agree that the sentence is overly long and that “orthogonal relaxation” is vague as written. This will be addressed in the revision.\n\n### Does not cite the “Deep Functional Maps” paper inspiring cited work.\nWe will add this in the revision and make clear its influence.\n\n### Justify how [20, 21, 22, 39] are related to or inspire present work\nThese represent recent work that either seeks to address existing drawbacks of FM, including scalability [20] and spatial consistency [21], investigate the properties of deep FMs frameworks [39], or explore composability with other powerful tools including LLMs [22]. All methods achieve state-of-the-art results, and we include them to show that increasing the efficacy and flexibility of FMs is an active topic.\n\nHowever, we would be happy to instead discuss in more detail those works most closely related to our own, including Litany et al. 2017 and Rustamov et al. 2013.\n\n### Relate/distinguish with the work of Rustamov et al. 2013\nWe thank the reviewer for suggesting we examine this work, as we now understand it provides an interesting comparison we did not previously realize!\n\nThis work shows how performing PCA on shape difference operators constructed with FMs forms a kind of “latent space” where codes corresponding to given shapes share a notion of closeness whenever said shapes can be related by specific transformations, including conformal and  authalic (area-preserving). Like NIso, this forms a framework for symmetry discovery by observing the clusterings in the latent space. However, in NIso similar observations lie on the same orbit formed by the transformations $\\tau$, rather than being “close” in the Euclidean sense.\n\nPerhaps the biggest difference between NIso and this work is that NIso also recovers a dimensionally-reduced representation of the transformations between observations. In addition,  we demonstrate in experiments that NIso can recover a meaningfully structured latent space for various types of transformations the observation space, whereas this work only reveals meaningful structure as it relates to conformal and authalic transformations.\n\n## Scalability of the eigenbasis\nWe agree that the ultimate scalability of the eigenbasis could be a limitation and we will note this in the revision. We believe the limiting factor is the parameterization of the basis directly via learned weights, which would result in large model sizes for high-res latent spaces. Currently, we find this can be mitigated by encoding to a lower resolution latent space, where the dimensions of the weight matrices are manageable, though this likely comes at the cost of some expressivity due to aliasing.  Currently, we are working on a follow up to address this problem in tandem with partiality by defining an *observation-dependent* eigenbasis via the output of a second encoder inspired by the concurrent work of [Cheng et al. 2024].\n\nHowever, we note that in other Deep FM frameworks, the principal computational bottleneck is the solve for the functional map. For example, the method proposed in the influential [38] to compute a near isometric functional map requires solving $K$  $K \\times K$ linear systems, with $K$ the dimension of the eigenbasis. For NIso, the solve requires only a single $K \\times K$ SVD computation. This is due to the closed form solution derived in Equation (8) which, while not mathematically significant, is to the best of our knowledge novel in the functional maps literature.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "LzvkhIG4Nz", "reviewer": "Reviewer_JQqD", "summary": "This paper proposes a generic equivariant ML framework by learning latent representations are are modelled to be related by an isometry. There are several important design choices made by the authors: (1.) An autoencoder framework that keeps the spatial structure (i.e. images get encoded into images) (2.) The isometric map is represented compactly with a basis that is the outcome of an eigen- -decomposition of a PSD operator (3.) The isometry manifests as an orthogonal matrix with a sparse block diagonal structure in its reduced projected form. \n\nExperiments are shown for (1.) Learning the spectrum of a Toric Laplacian and comparing it qualitatively to the real one  in Fig 3 (2.) Homography perturbed MNIST (3.) Conformal shape classification and (4.) Camera Pose Estimation from real-world video. The results (1) show a good proof of concept and (2), (3) and (4) demonstrate a clear superiority over the closest conceptual baseline NFT. (2.) Also makes the case for achieving comparable performance with simpler equivariant architectures using the NIso latent space.", "questions": "- The basis functions visualized in Fig3 (third column) are learned or from the actual toric laplacian? what are the various rows? \n- It would also be useful to report some feature of complexity in Table 1 indicating \"meticulously-engineered, handcrafted networks\"\n- More examples of evidence showing good basis functions \n- What is a perturbation in Section 5.1?", "limitations": "I think the authors make a fair declaration of the limitations of their work in Section 6.\nI would make a stronger vote after the rebuttal. I think this paper has many ingredients to warrant acceptance: clever idea, good argumentation, and decent proof of concept. It lacks clarity in some places and the evaluation does not appear equally general as is the description of the method. For this reason, I maintain a borderline acceptance with a full intention to re-assess in the next round.", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 4, "contribution": 3, "strengths": "- Overall the biggest strength of this paper is that it was an enjoyable read. I found the hypothesis (i.e. learning latent codes related by an isometry for arbitrary transformations in the observation space) to be interesting and reasonable. The overall writing and description of the method is very nice. \n- The Experiments especially (1.) was a good proof of concept.", "weaknesses": "- I am actually quite confused on how the operator \\Omega and Mass matrix M is learned? The Laplace Beltrami is a very structured object. See all requirements enumerated in Wardetzky, Max, et al. \"Discrete Laplace operators: no free lunch.\" Symposium on Geometry processing. Vol. 33. 2007.). It is unclear what the nature of this operator is and its learning feels ad-hoc and unmotivated. \n- Do the authors make sure that the optimization for the functional map is well-posed? Typically in the original formulation a \"good\" functional map is the outcome of using many-many descriptors in addition to regularization constraints like commutativity with laplacian. If I understand correctly the proposed formulation seems to be confident in learning a structured functional map (like Fig 2) using just one pair of corresponding functions ? \n- As a high-level opinion: Overall the experimental section could be stronger. The \"discovery\" of the toric laplacian was nice, but a demonstration on roto-translations perhaps with scale and/or spherical images with SO(3) would be extremely convincing. I would also recommend an evaluation that focuses beyond NFT and actually quantifying the complexity of different equivariant models. \n- As mentioned by the authors, the requirement of generalizing to domains with diverse connectivity is indeed a hard one, but important.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your detailed review, and for your kind words on our hypothesis being interesting and plausible, our writing enjoyable, and our validation with the laplacian helpful! We are happy to report that we executed your key task -- recovering the harmonics with spherical images – with exciting results! \n\n## Learning the operator $\\Omega$ and mass matrix $\\mathbb{M}$. \n\n### Motivation\n\nThe goal of NIso is to discover symmetries (i.e. what is preserved between observations) of complex transformations in image space. For instance, in the toric laplacian experiment, the image transformation is a shift. Under shifts, a particular property of the images is preserved: their frequencies.  For any transformation that preserves some property, there exists a basis such that its functional map (FM) is block-diagonal in that basis. The functional basis is defined as the eigendecomposition of some operator $\\Omega$. The FM being block diagonal in the basis – with the size of each block corresponding to the eigenvalue multiplicity –  means it will commute with $\\Omega$ and equivalently preserve its frequencies (i.e. an isometry).  Hence, we can view the problem of symmetry discovery as equivalent to jointly finding a set of maps relating observations *as well as* the operator they commute with. \n\n### Parameterization and learning\n\n[Wang and Solomon, 2019] establish that the key property characterizing functional operators is semi-positive-definiteness (SPD), and we impose no other constraints so as not to limit the space of discoverable operators. (We will cite this work in the revision). SPD is defined with respect to some  inner product, and thus we learn a diagonal mass matrix (with positive entries) which defines a functional inner product. Then, we seek to simultaneously regress the parameters for an operator $\\Omega$ that is SPD w.r.t. the inner product defined by $\\mathbb{M}$. We observe that *any* $\\mathbb{M}$-SPD operator can be expressed in the form of Equation (4) for some matrix of $\\mathbb{M}$-orthogonal eigenfunctions $\\Phi$ and non-negative eigenvalues $\\Lambda$. Thus we learn a set of weights parameterizing $\\Phi$ and $\\Lambda$ which together with learned $\\mathbb{M}$ form $\\Omega$.  In practice, the weights of  $\\Phi$ are projected to the nearest $\\mathbb{M}$-orthogonal matrix via the SVD. \n\n## Well-posedness of functional map\nWe carefully ensure that the solve for the functional map is well-posed. \n\nFirst, we leverage a number of regularizers that jointly ensure well-posedness: 1). Orthogonality; 2). Operator commutativity;  and 3). The “Equivariance loss” is a version of the widely used “descriptor preservation” loss in the FM. These are several of the most common principled regularizers widely disseminated and discussed in the FM literature [21, 26, 38]  \n\nSecond, we note that our latent codes double as multi-channel pointwise descriptors, of the same kind as used in the Deep FM family of work [Litany et al. 2017, 38] , which forms the backbone of many SoTA shape correspondence pipelines.\n\n## Discovering the Spherical Harmonic Transform\nWe are happy to report that we have performed this experiment with *spectacular* results! Please see the discussion in the general rebuttal under “Discovering the spherical harmonics” and Fig. 1 of the PDF. We hope you find these convincing! \n\n## Quantifying the complexity of different equivariant models.\nThere are two dimensions of “complexity” that we failed to clearly delineate in the text: (1) *human* complexity of hand-crafting the method and (2) compute / memory complexity.\n\nWhile approaches for rotation equivariance are wide-spread, there are no standard equivariant networks for most difficult symmetries. Any attempt at building such a network is associated with extensive, difficult mathematical labor. MobiusConv [7], for instance, essentially derives what to the best of our knowledge is a previously unknown representation on a specialized group of filters (which is no mean feat), simply to facilitate a tractable discretization.\n\nhomConv [6] and LieDecomp [43] seek to sidestep this via a recipe for general equivariance using Monte-carlo integration. However, these methods struggle with *compute* complexity. For $N$ samples (typically at least 10-25), the computational complexity of a forward pass scales as $N^L$ where $L$ is the number of layers!  In practice, this means that these methods cannot be scaled past incredibly low-dimensional images, which is why both homConv and LieDecomp are run only on MNIST.\n\nIn contrast, NIso relies on recovering a latent space such that the transformations can be represented as an isometry, which can thus be exploited by far more efficient and scalable isometry-equivariant networks such as [36] and [42]. .\n\n## Evaluation that focuses beyond NFT.\nWe agree and have since added two additional baselines to the submission - see the point “Additional Baselines” in the general reply above! However, we would like to note that besides the NFT, *we are not aware of any other methods that aim to perform equivariant representation learning without prior knowledge of the group*. Thus, we see the NFT as a important benchmark with which to compare. \n\n## Generalizing across connectivity\nWe completely agree and are currently exploring this as a direction of future work.\n\n## The basis functions visualized in Fig3 (third column) are learned or from the actual toric laplacian? What are the various rows?\nThey are learned. At the time we did not discover them ordered by a classical notion of frequency so the order is random.  Rows are eigenfunctions in C-style indexing. \n\n## More examples of evidence showing good basis functions\nIn addition to the learned spherical harmonics, we have also visualized examples of the eigenfunctions discovered in each of the three other experiments.  Please find them in the PDF, in Fig. 2.\n\n## What is a perturbation in Section 5.1?\nPerturbation refers to applying a homography to the image.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "PWtlQYu34t", "reviewer": "Reviewer_FFcP", "summary": "This paper introduces Neural Isometries, which is an autoencoder framework learning to map the observation space to a general-purpose latent space wherein encodings are related by isometries whenever their corresponding observations are geometrically related in world space. Several experiments, including camera pose estimation, are conducted.", "questions": "1. If the observation space is already isometric, e.g. rotate by 90 degrees, what can be said about the latent sapce?\n2. How robust is this model? For example, how will the latent vector change when the inputs are partially observed (masked)?", "limitations": "See the weakness section.", "rating": 4, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The idea of transforming complicated equivariances in observation space into isometries in latent space is intuitive and interesting. Experiments have demonstrated the potentials of this method in practical tasks. The paper is well written and easy to understand.", "weaknesses": "This is an interesting paper but in-depth discussion is lacking. My main concern is that there is no quantitative or theoretical description of the equivariance. For example, \n1.  How equivariant is the model? i.e. How large is the equivariance loss?\n2. What type of equivariance in the observation space can be modeled? Will the model fail when the pertubations are too strong?\n\nIn addition, the experiment results are relatively weak. Only 3 small scale experiments are conducted, and only a few baselines are considered.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for validating our motivation and our communication - we appreciate it! \n\n## Quantitative evaluation of equivariance: How equivariant is the model?\n\nWe are happy to report that we were able to quantitatively evaluate the equivariance of our model following standard procedures [6-7, 43].  Please see the discussion in the general rebuttal and Tab. 1 in the attached PDF. \n\n## What type of equivariance in the observation space can be modeled? \n\nNIso can *exactly* model unitary transformations. Any other transformation can in theory be approximated with a sufficiently expressive latent space. We also note that some transformations that are not unitary in image space can be made unitary by estimating the correct latent space: consider the case of camera pose estimation from video. From frame to frame, there are occlusions so there exists no unitary map between the two images. However, there exists a unitary map between the state of the underlying 3D scenes. Our motivation with NIso is to take a first step towards a model that can discover the correct symmetry of the underlying problem.\n\n## Will the model fail when the pertubations are too strong? How robust is the model? \n\nAs remarked in the general discussion, functional maps can be sensitive to occlusion and partiality.  However, we note that NIso is already *surprisingly robust* to partiality in its inputs, as is evident in the Co3D experiments. Please see the discussion in the general rebuttal under the “NIso under the presence of occlusions / masking / partiality” section.\n\n\n## Lack of Baselines + Only Small-Scale / Synthetic Experimental Results\n\nWe are happy to report that we have provided comparisons with additional baselines, including with two SoTA representation learners (DINOv2 and BeIT) in the pose estimation experiments on the Co3D dataset. Please see the \"Additional baselines\" section in the general rebuttal.\n\nHowever, we would like to respectfully push back on the notion that we are considering too few baselines. \n\nWe note that there is not much work on the topic of equivariant machine learning *without* prior knowledge of the symmetry group - the NFT is the most relevant baseline in this space. Further, the reviewer is suggesting that we are only comparing “small-scale” or “synthetic” experiments”. However, camera pose estimation on Co3D is *not* a small-scale experiment - it is a large, real-world dataset that is actively used for benchmarking of applications from pose estimation to novel view synthesis. \n\nMore generally,  we sought geometric deep-learning baselines for difficult, non-compact symmetry groups. To the best of our knowledge, the only existing SoTA equivariant models handling such symmetries in vision-related tasks are are homConv [6] and LieDecomp [43] (both for homographies) and MobiusConv [7] (for Mobius Transformations). Thus, benchmarking on MNIST and SHREC is absolutely vital to support our claims. These baselines are not scalable and will not run on any more complex or real-world datasets - this is not a limitation of *our* method, but a limitation of the baselines.  Please also see our discussion in the general rebuttal under the section \"Scope of experimental evaluations\". \n\nWe would be happy to compare with additional baselines if you could clarify what these baselines should be. \n\n\n## If the observation space is already isometric, e.g. rotate by 90 degrees, what can be said about the latent space?\n\nNote that our toy experiment - discovering the toric laplacian - is *exactly* an instance of this problem. We parameterize images to lie on a torus, such that shifts become rotations, and hence, the shift is exactly an isometric transformation! In this case, our formulation is likely to recover a close approximation of the *correct* operator and corresponding irreducible unitary representation of the transformation in the eigenbasis - in Exp. 1, we recover a very good approximation of the laplacian operator and the block-diagonal shift representation. \n\nWe further demonstrate this in the new experiment where we discover the spherical harmonic transform (see the “Discovering the spherical harmonic transform” section in the general rebuttal and Fig. 1 of the attached PDF).  This experiment results in the discovery of a basis with almost *exactly* the properties of the spherical harmonics. In particular, the estimated $\\tau_\\Omega$ manifest *exactly* the same structure as the ground truth Wigner-D matrices corresponding to the rotation (which are the IURs of $SO(3)$), with square blocks of size $(2\\ell + 1) \\times (2\\ell + 1)$, for the $\\ell$-th distinct eigenvalue", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "X6aO8vTLy6", "reviewer": "Reviewer_Sfsw", "summary": "The paper proposes an autoencoder framework that encodes the input symmetries into isometries in the latent space. The equivariance in the latent space is captured by a functional map $\\tau$, which is regularized to be an isometry. Instead of hard constraints, the equivariance of the system is encouraged via the optimized $\\tau$ and the loss function design. The proposed method is evaluated on a homography-perturbed MNIST, conformal shape classification, and camera pose estimation tasks. The experiments show the proposed method performs on par with the handcrafted equivariant baselines and outperforms a baseline with a similar approach (Neural Fourier Transform). \n\nI’m positive about the paper. However, there are some main concerns that I would like to know the answers to. I’m willing to raise the scores once they are addressed.", "questions": "1. Since there are no hard-coded constraints (except for the regularization) in the system, the proposed method can work with different input transformations. Do you think this work can be applied to automatic symmetry discovery tasks? \n2. I understand that the inner product needs to be preserved to construct an isometry, but why does $\\tau$ have to commute with $\\Omega$?\n3. Ln 207, “we show in experiments that a major benefit of our isometric regularization is that our multiplicity loss (promoting a diagonal-as-possible and thus sparse and compact $\\tau_\\Omega$) can serve as an effective substitute for access to triples.” Is this true for all three experiments, or is it only validated on homNIST?", "limitations": "As mentioned in the paper, the proposed network is unable to perform graph-based tasks. In addition, the equivariance in the latent space is not evaluated. As a result, it is unclear how the equivariance is preserved in the latent space. Lastly, by restricting the functional maps to be isometries, the transformations that the network can model might be limited.", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper is well-written and easy to follow. \n2. The paper proposes a novel framework that softly models the latent space equivariance as functional maps. \n3. Unlike handcrafted equivariant networks that are limited to certain groups, the proposed framework can model different input transformations. \n4. The proposed method is shown to perform on par with the baselines in homNIST and camera pose estimation tasks, and outperforms the baselines in the conformal shape classification task.", "weaknesses": "1. Although being shown to be effective in the experiments, the constraint of $\\tau$ being an isometry seems a bit heuristic. The paper can benefit from some theoretical investigation on why constraining to being isometries helps the performance.\n2. Following up on the previous point, can all transformations be modeled as isometric functional maps in the latent space? That being said, does restricting $\\tau$ to isometries affect the types of transformations the network is able to model? \n3. Since the equivariance is only obtained softly via the loss function design, it’s essential to see how much equivariant is maintained/lost in the latent space. I recommend the authors report some measures of equivariance and compare them with the handcrafted methods. (For example, equivariant error/loss in the latent space.)\n4. There are several heuristic designs in the proposed framework, and some of them affect the performance significantly. For example, it is not explained clearly why a smooth multiplicative mask is required to ensure $\\tau_\\Omega$ is semi-diagonal. The effect of such approximation is also not discussed/evaluated. The second is the multiplicity loss, which greatly affects the performance of the framework. \n5. I appreciate the comparisons with handcrafted equivariant networks. However, I believe the paper can be greatly strengthened by comparing it with the same autoencoder framework while not enforcing the latent equivariance and training with data augmentation. This can demonstrate the benefits of the proposed latent equivariant design.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for highlighting that our paper is well-written and easy to follow, novel, and competitive with baselines, and for providing valuable feedback! We further appreciate your commitment to reconsidering your score - we have tried our best to address your feedback, but please let us know if we missed anything!\n\n## Clarification: Automatic Symmetry Discovery\nWe would like to clarify that this is *exactly* the goal of NIso. It *does not assume* upfront knowledge of the symmetry, but instead learns to be equivariant *automatically*. \n\n## Why commute with $\\Omega$?\n\nThe goal of NIso is to discover symmetries (i.e. what is preserved between observations) of transformations. For instance, if the image transformation is a shift, then a a particular property of the images is preserved: their frequencies.  For any transformation that preserves some property, there exists a basis such that its functional map (FM) is block-diagonal in that basis. The basis is defined as the eigendecomposition of some operator $\\Omega$. The FM being block diagonal in the basis – with the size of each block corresponding to the eigenvalue multiplicity –  means it will commute with $\\Omega$ and equivalently preserve its frequencies (i.e. an isometry).  Hence, we can view the problem of symmetry discovery as equivalent to jointly finding a set of maps relating observations *as well as* the operator they commute with.\n\nWe will be happy to include a detailed theoretical exposition of these properties in the revision. \n\n## Can all transformations be modeled?\nNIso can *exactly* model unitary transformations. Any other transformation can in theory be approximated with a sufficiently expressive latent space. We also note that some transformations that are not unitary in image space can be made so by estimating the correct latent space: consider the case of camera pose estimation from video. From frame to frame, there are occlusions so there exists no unitary map between the two images. However, there exists a unitary map between the state of the underlying 3D scenes. Our motivation with NIso is to take a first step towards a model that can discover the correct symmetries of the underlying problem.\n\nFunctional maps can nevertheless be sensitive to occlusion and partiality.  However,  we note that NIso is already *surprisingly robust* to partiality in its inputs, as is evident in the Co3D experiments. Please see the related discussion in the general rebuttal.\n\n## Report measures of equivariance and compare them with the handcrafted methods.\nWe are happy to report that we were able to quantitatively evaluate the equivariance of our model following standard procedures [6-7, 43].  Please see the discussion in the general rebuttal and Tab. 1 in the attached PDF. \n \n## Explanation of why a multiplicative mask is required to ensure diagonality.\n\nThe multiplicative mask $P_\\Lambda$ appears in the derivation of the closed form solution to the constrained minimization problem in Equation (7) (section A.2 of the supplement).  Remarkably, the derivation reveals that enforcing commutativity with the diagonal matrix of eigenvalues $\\Lambda$ (and thus block diagonality) simply reduces to element-wise multiplication with the “sharp” eigenvalue mask in Equation (19) before the Procrustes projection. \n\n## Evaluate effect of smooth multiplicative mask.\nThe “sharp” eigenvalue mask in Equation (19) is difficult to work with.  Evaluating equality between floating point values is hard without some heuristic measure of closeness. Furthermore, we found that this approach results in poor backwards gradient flow to the eigenvalues, to the point where the NIso fails completely.\n\nThus, we replace the “exact” mask with its “soft” analogue, approximating the Kronecker deltas via the exponential. While it induces some degree of error, it allows backprop through the mask. We note that we are not the first to introduce a “soft” eigenvalue mask with alternatives explored extensively in [Ren and Panine et al. 2019] and concurrent work [Ceng and Deng et al. 2024] employing a similar soft mask as a regularizer. \n\n## Explanation of Multiplicity Loss.\nThe point of the multiplicity loss is to make the operator “interesting” and force our framework to recover the symmetry of the transformation. Without the multiplicity loss, all eigenvalues could collapse to a single one. The operator is then  a multiple of the identity, which would not constrain the FM in a meaningful way. The more distinct eigenvalues the operator has, the more it is forced to discover subspaces of the latent space that are preserved, i.e., symmetries. Intuitively, the multiplicity loss can also be seen as forcing the network to discover an \"as-simple-as-possible\" representation for the transformations by promoting diagonality.\n\n\n## Effect of multiplicity loss across experiments\n\nWithout the multiplicity loss our method achieves poor results in all three experiments. On Conf. SHREC`11, without the multiplicity loss NIso fails to discover a meaningful notion of equivariance in the latent space and archives a classification accuracy of only approximately 36%.  In the CO3D experiments, we also indirectly evaluate its absence in the regime where we train and evaluate a version of NIso that does not learn an operator (and thus only enforces orthogonality). We find that this version performs worse than all other baselines. Together these results suggest that forcing the network to discover an operator with a diverse eigenspectrum via the multiplicity loss is integral to gaining a meaningful notion of equivariance in the practical sense. \n\n\n## Compare to the baseline autoencoder framework with data augmentation.\n\nThank you for suggesting this experiment and we are happy to report we executed it with strong results -- NIso still outperforms these baselines by a significant margin. Please see the \"Comparison with data-augmentation\" section in the general rebuttal and Table 2 of the PDF.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank the reviewers for their careful reading, and detailed and considerate feedback. We are glad that reviewers deem our paper “a clever idea”, “relevant”, “interesting and reasonable”, and “intuitive and interesting”, and the writing to be “very nice”, “easy to follow”, an “enjoyable read”, and a “clear formalization”. We are glad that reviewers recognize the relevance of this first step towards self-supervised symmetry discovery and equivariant ML!

Key outstanding concerns revolve around measuring the equivariance error, clarity, and experimental evaluation. We are glad to present new results and analysis to address these concerns!

## Measuring Equivariance Error
We measure the standard equivariance error in the latent space [6-7, 43], with the results shown in Tab. 1 of the PDF. On both HomMNIST and SHREC, NIso is on par with expert-designed baselines. Similarly for Co3D, we find that NIso achieves low errors. While the error rises with increasing frame skip, it remains under 10%, and NIso remains the best-performing method, demonstrating that NIso remains equivariant even under increasing partiality.

## Discovering the spherical harmonic transform
Following JQqD’s suggestion, we perform an experiment identical to “discovering the toric laplacian”, by mapping ImageNet to the sphere and acting on it via SO(3). We further realized that adding a simple dropout layer which randomly masks out coefficients corresponding to large eigenvalues before the basis unprojection yields eigenfunctions ordered by their energy. This experiment results in the discovery of a basis and maps $\tau_{\Omega}$ with almost *exactly* the properties of the spherical harmonics and the Wigner-D matrices, see the attached PDF, Fig. 1!

## Additional baselines
### Pose estimation
In the camera pose estimation experiment, we added two strong representation-learning baselines. We extract image features from both images using two state-of-the-art vision foundation models -  DINOv2 [Oquab et al. 2023] and BeIT [Bao et al. 2021]. We then pass the tokens into the DUSt3R[47]-style decoder described in section 5.3 to predict the pose. The results are shown in Table 2 of the PDF.  NIso outperforms both significantly.

### Comparison with data augmentation 
In line with Sfsw’s comments, we compare NIso to the AE baseline with dataset augmentation during both the pre-training and fine-tuning phases in the SHREC and MNIST experiments. The results are shown in the PDF, Tab. 2. On both MNIST and SHREC, training with augmentation improves the baseline performance by approximately 34% and 7%. NIso still outperforms this baseline significantly. 

## Scope of experimental evaluations
2Npe remarks a “limited experimental setup”. We would like to contextualize the scope of our evaluation with that of comparable work at top conferences. 

HomConv [CVPR] and LieDecomp [ICRL] benchmark *exclusively* on MNIST. MobiusConv [SIGGRAPH] benchmarks *only* on SHREC to demonstrate advantages of Mobius-equivariance. Our experiments are also commensurate with those in the NFT [NeurIPS, ICLR]. Among all these methods, we alone present results on Co3D, a large-scale, real-world dataset, in contrast to 2Npe's review ("datasets are very small scale, synthetic, not real world."). 

Further, 2Npe prefaces their criticism with the statement "As mentioned many times in paper, instead of theoretical justification, this paper seeks to validate the efficacy of approach experimentally on real world tasks." We respectfully point out that this is a misrepresentation of our claims.  In fact, *only once*, on lines 101-102 do we make a similar statement in which we state our intention to "validate the efficacy of
our approach experimentally, including in geometry processing and real-world 3D vision tasks" (referencing the conformal SHREC and Co3D experiments).  At no point do we say our goal is to validate NIso on real-world tasks, only that our experiments *include* a real-world task (Co3D pose estimation).

2Npe also remarks that we could "motivate the problem/solution by probing/playing with the learned structured space". As our chief goal is self-supervised symmetry discovery, we respectfully note that our toric and spherical Laplacian experiments serve *exactly* this end.

In terms of baselines, we benchmarked with the *best-performing* ones for homography and mobius equivariance, and SOTA representation learners for Co3D. To the best of our knowledge, the NFT is the only applicable baseline for equivariant representation learning without prior knowledge of the group. We contacted the authors of the NFT, who confirmed our implementation is faithful.

It is challenging to pick the right evaluation when working on the new problem of self-supervised symmetry discovery / equivariant ML. We are hence grateful for the reviewers' suggestions, which we did our very best to incorporate!

## NIso under the presence of occlusions / masking / partiality
We agree that showing robustness to partiality would make the paper stronger. That said, several recent works have sought to make functional maps robust to partiality [Attaiki et al. 2021, Cheng et al. 2024, Bracha et al. 2024] and we are excited to incorporate these techniques in our next steps! However, we strongly believe that NIso without this improvement remains extremely valuable. First, the limitation of not modeling partiality is not unique to NIso, but to most equivariant networks, including all of NIso’s baselines. Second, we note that NIso is already robust to partiality, as is evident in the Co3D experiments. At a frameskip 9, only about 80% of the pixels are shared across frames, and NIso outperforms two foundation model baselines with the equivariance error remaining below 10%. Third, a major application for NIso is the continuous-time regime, i.e., representation learning from video. In this case, the overlap between consecutive frames is generally large,and NIso remains acutely practical.

Meta-review: This paper received mixed reviews in the first round, was intensively discussed between reviewers and authors, and between reviewers. After the discussion phases, the scores were 3x borderline reject and 1x weak accept.

The overall consensus is that the paper is a great read and proposes an interesting angle for obtaining equivariant representations. On the negative side, there were several concerns with respect to the experimental setup and if they support the general claims of the work. Some of them were successfully rebutted by the authors:
- the paper does indeed evaluate on a relevant real world task (CO3D camera pose estimation)
- it also successfully shows in the ablations in Section 5.1 that the proposed losses are necessary to recover the correct transformations
- the claim that the method performs on par with hand-designed methods is supported (if only on limited experiments)

However, there are also concerns left, which have not been adequately resolved:
- It is true that the paper lacks theoretical justifications for the many "intuitive" heuristics that it employs
- It is also true that the paper mostly invents its own baselines and datasets. Only the measurements of equivariance error (rebuttal document) seem to be in the same framework as previous works. Meanwhile, the NFT authors provide a benchmark set (https://github.com/masomatics/NFTPublic/tree/main), which is not used by the authors of this work.

Regarding the last point it has to be said that the presented approach is indeed quite novel and the used datasets and evaluation methods seem to be quite creative to me. In this borderline situation, I tend to recommend acceptance, favoring the explorative nature of this work.
Decision: Accept (poster)